1 An informal guide to reading and working on the rustc compiler.
2 ==================================================================
4 If you wish to expand on this document, or have a more experienced
5 Rust contributor add anything else to it, please get in touch:
7 * https://internals.rust-lang.org/
8 * https://chat.mibbit.com/?server=irc.mozilla.org&channel=%23rust
12 https://github.com/rust-lang/rust/issues
14 Your concerns are probably the same as someone else's.
16 You may also be interested in the
17 [Rust Forge](https://forge.rust-lang.org/), which includes a number of
18 interesting bits of information.
20 Finally, at the end of this file is a GLOSSARY defining a number of
21 common (and not necessarily obvious!) names that are used in the Rust
22 compiler code. If you see some funky name and you'd like to know what
23 it stands for, check there!
28 Rustc consists of a number of crates, including `syntax`,
29 `rustc`, `rustc_back`, `rustc_trans`, `rustc_driver`, and
30 many more. The source for each crate can be found in a directory
31 like `src/libXXX`, where `XXX` is the crate name.
33 (NB. The names and divisions of these crates are not set in
34 stone and may change over time -- for the time being, we tend towards
35 a finer-grained division to help with compilation time, though as
36 incremental improves that may change.)
38 The dependency structure of these crates is roughly a diamond:
46 rustc_trans rustc_borrowck ... rustc_metadata
61 The idea is that `rustc_driver`, at the top of this lattice, basically
62 defines the overall control-flow of the compiler. It doesn't have much
63 "real code", but instead ties together all of the code defined in the
64 other crates and defines the overall flow of execution.
66 At the other extreme, the `rustc` crate defines the common and
67 pervasive data structures that all the rest of the compiler uses
68 (e.g., how to represent types, traits, and the program itself). It
69 also contains some amount of the compiler itself, although that is
72 Finally, all the crates in the bulge in the middle define the bulk of
73 the compiler -- they all depend on `rustc`, so that they can make use
74 of the various types defined there, and they export public routines
75 that `rustc_driver` will invoke as needed (more and more, what these
76 crates export are "query definitions", but those are covered later
79 Below `rustc` lie various crates that make up the parser and error
80 reporting mechanism. For historical reasons, these crates do not have
81 the `rustc_` prefix, but they are really just as much an internal part
82 of the compiler and not intended to be stable (though they do wind up
83 getting used by some crates in the wild; a practice we hope to
86 Each crate has a `README.md` file that describes, at a high-level,
87 what it contains, and tries to give some kind of explanation (some
93 The Rust compiler is comprised of six main compilation phases.
96 2. Configuration & expanding (cfg rules & syntax extension expansion)
97 3. Running analysis passes
98 4. Translation to LLVM
102 Phase one is responsible for parsing & lexing the input to the compiler. The
103 output of this phase is an abstract syntax tree (AST). The AST at this point
104 includes all macro uses & attributes. This means code which will be later
105 expanded and/or removed due to `cfg` attributes is still present in this
106 version of the AST. Parsing abstracts away details about individual files which
107 have been read into the AST.
109 Phase two handles configuration and macro expansion. You can think of this
110 phase as a function acting on the AST from the previous phase. The input for
111 this phase is the unexpanded AST from phase one, and the output is an expanded
112 version of the same AST. This phase will expand all macros & syntax
113 extensions and will evaluate all `cfg` attributes, potentially removing some
114 code. The resulting AST will not contain any macros or `macro_use` statements.
116 The code for these first two phases is in [`libsyntax`][libsyntax].
118 After this phase, the compiler allocates IDs to each node in the AST
119 (technically not every node, but most of them). If we are writing out
120 dependencies, that happens now.
122 The third phase is analysis. This is the most complex phase in the compiler,
123 and makes up much of the code. This phase included name resolution, type
124 checking, borrow checking, type & lifetime inference, trait selection, method
125 selection, linting and so on. Most of the error detection in the compiler comes
126 from this phase (with the exception of parse errors which arise during
127 parsing). The "output" of this phase is a set of side tables containing
128 semantic information about the source program. The analysis code is in
129 [`librustc`][rustc] and some other crates with the `librustc_` prefix.
131 The fourth phase is translation. This phase translates the AST (and the side
132 tables from the previous phase) into LLVM IR (intermediate representation).
133 This is achieved by calling into the LLVM libraries. The code for this is in
134 [`librustc_trans`][trans].
136 Phase five runs the LLVM backend. This runs LLVM's optimization passes on the
137 generated IR and generates machine code resulting in object files. This phase
138 is not really part of the Rust compiler, as LLVM carries out all the work.
139 The interface between LLVM and Rust is in [`librustc_llvm`][llvm].
141 The final phase, phase six, links the object files into an executable. This is
142 again outsourced to other tools and not performed by the Rust compiler
143 directly. The interface is in [`librustc_back`][back] (which also contains some
144 things used primarily during translation).
146 A module called the driver coordinates all these phases. It handles all the
147 highest level coordination of compilation from parsing command line arguments
148 all the way to invoking the linker to produce an executable.
150 Modules in the librustc crate
151 =============================
153 The librustc crate itself consists of the following submodules
154 (mostly, but not entirely, in their own directories):
156 - session: options and data that pertain to the compilation session as
158 - middle: middle-end: name resolution, typechecking, LLVM code
160 - metadata: encoder and decoder for data required by separate
162 - plugin: infrastructure for compiler plugins
163 - lint: infrastructure for compiler warnings
164 - util: ubiquitous types and helper functions
165 - lib: bindings to LLVM
167 The entry-point for the compiler is main() in the [`librustc_driver`][driver]
170 The 3 central data structures:
171 ------------------------------
173 1. `./../libsyntax/ast.rs` defines the AST. The AST is treated as
174 immutable after parsing, but it depends on mutable context data
175 structures (mainly hash maps) to give it meaning.
177 - Many – though not all – nodes within this data structure are
178 wrapped in the type `spanned<T>`, meaning that the front-end has
179 marked the input coordinates of that node. The member `node` is
180 the data itself, the member `span` is the input location (file,
181 line, column; both low and high).
183 - Many other nodes within this data structure carry a
184 `def_id`. These nodes represent the 'target' of some name
185 reference elsewhere in the tree. When the AST is resolved, by
186 `middle/resolve.rs`, all names wind up acquiring a def that they
187 point to. So anything that can be pointed-to by a name winds
190 2. `middle/ty.rs` defines the datatype `sty`. This is the type that
191 represents types after they have been resolved and normalized by
192 the middle-end. The typeck phase converts every ast type to a
193 `ty::sty`, and the latter is used to drive later phases of
194 compilation. Most variants in the `ast::ty` tag have a
195 corresponding variant in the `ty::sty` tag.
197 3. `./../librustc_llvm/lib.rs` defines the exported types
198 `ValueRef`, `TypeRef`, `BasicBlockRef`, and several others.
199 Each of these is an opaque pointer to an LLVM type,
200 manipulated through the `lib::llvm` interface.
202 [libsyntax]: https://github.com/rust-lang/rust/tree/master/src/libsyntax/
203 [trans]: https://github.com/rust-lang/rust/tree/master/src/librustc_trans/
204 [llvm]: https://github.com/rust-lang/rust/tree/master/src/librustc_llvm/
205 [back]: https://github.com/rust-lang/rust/tree/master/src/librustc_back/
206 [rustc]: https://github.com/rust-lang/rust/tree/master/src/librustc/
207 [driver]: https://github.com/rust-lang/rust/tree/master/src/librustc_driver
212 The compiler uses a number of...idiosyncratic abbreviations and
213 things. This glossary attempts to list them and give you a few
214 pointers for understanding them better.
216 - AST -- the **abstract syntax tree** produced the `syntax` crate; reflects user syntax
218 - cx -- we tend to use "cx" as an abbrevation for context. See also tcx, infcx, etc.
219 - HIR -- the **High-level IR**, created by lowering and desugaring the AST. See `librustc/hir`.
220 - `'gcx` -- the lifetime of the global arena (see `librustc/ty`).
221 - generics -- the set of generic type parameters defined on a type or item
222 - infcx -- the inference context (see `librustc/infer`)
223 - MIR -- the **Mid-level IR** that is created after type-checking for use by borrowck and trans.
224 Defined in the `src/librustc/mir/` module, but much of the code that manipulates it is
225 found in `src/librustc_mir`.
226 - obligation -- something that must be proven by the trait system.
227 - sess -- the **compiler session**, which stores global data used throughout compilation
228 - substs -- the **substitutions** for a given generic type or item
229 (e.g., the `i32, u32` in `HashMap<i32, u32>`)
230 - tcx -- the "typing context", main data structure of the compiler (see `librustc/ty`).
231 - trans -- the code to **translate** MIR into LLVM IR.
232 - trait reference -- a trait and values for its type parameters (see `librustc/ty`).
233 - ty -- the internal representation of a **type** (see `librustc/ty`).