1 # rustbuild - Bootstrapping Rust
3 This is an in-progress README which is targeted at helping to explain how Rust
4 is bootstrapped and in general, some of the technical details of the build
9 The rustbuild build system has a primary entry point, a top level `x.py` script:
15 Note that if you're on Unix, you should be able to execute the script directly:
21 The script accepts commands, flags, and arguments to determine what to do:
23 * `build` - a general purpose command for compiling code. Alone, `build` will
24 bootstrap the entire compiler, and otherwise, arguments passed indicate what to
28 # build the whole compiler
29 ./x.py build --stage 2
31 # build the stage1 compiler
35 ./x.py build --stage 0 library/std
37 # build a particular crate in stage0
38 ./x.py build --stage 0 library/test
41 If files that would normally be rebuilt from stage 0 are dirty, the rebuild can be
42 overridden using `--keep-stage 0`. Using `--keep-stage n` will skip all steps
43 that belong to stage n or earlier:
46 # build stage 1, keeping old build products for stage 0
47 ./x.py build --keep-stage 0
50 * `test` - a command for executing unit tests. Like the `build` command, this
51 will execute the entire test suite by default, and otherwise, it can be used to
52 select which test suite is run:
61 # execute the UI test suite
62 ./x.py test src/test/ui
64 # execute only some tests in the UI test suite
65 ./x.py test src/test/ui --test-args substring-of-test-name
67 # execute tests in the standard library in stage0
68 ./x.py test --stage 0 library/std
70 # execute tests in the core and standard library in stage0,
71 # without running doc tests (thus avoid depending on building the compiler)
72 ./x.py test --stage 0 --no-doc library/core library/std
74 # execute all doc tests
78 * `doc` - a command for building documentation. Like above, can take arguments
81 ## Configuring rustbuild
83 rustbuild offers a TOML-based configuration system with a `config.toml`
84 file. An example of this configuration can be found at `config.toml.example`,
85 and the configuration file can also be passed as `--config path/to/config.toml`
86 if the build system is being invoked manually (via the python script).
88 You can generate a config.toml using `./configure` options if you want to automate creating the file without having to edit it.
90 Finally, rustbuild makes use of the [cc-rs crate] which has [its own
91 method][env-vars] of configuring C compilers and C flags via environment
94 [cc-rs crate]: https://github.com/alexcrichton/cc-rs
95 [env-vars]: https://github.com/alexcrichton/cc-rs#external-configuration-via-environment-variables
99 The rustbuild build system goes through a few phases to actually build the
100 compiler. What actually happens when you invoke rustbuild is:
102 1. The entry point script, `x.py` is run. This script is
103 responsible for downloading the stage0 compiler/Cargo binaries, and it then
104 compiles the build system itself (this folder). Finally, it then invokes the
105 actual `bootstrap` binary build system.
106 2. In Rust, `bootstrap` will slurp up all configuration, perform a number of
107 sanity checks (whether compilers exist, for example), and then start building the
109 3. The stage0 `cargo`, downloaded earlier, is used to build the standard library
110 and the compiler, and then these binaries are then copied to the `stage1`
111 directory. That compiler is then used to generate the stage1 artifacts which
112 are then copied to the stage2 directory, and then finally, the stage2
113 artifacts are generated using that compiler.
115 The goal of each stage is to (a) leverage Cargo as much as possible and failing
116 that (b) leverage Rust as much as possible!
118 ## Incremental builds
120 You can configure rustbuild to use incremental compilation with the
121 `--incremental` flag:
124 $ ./x.py build --incremental
127 The `--incremental` flag will store incremental compilation artifacts
128 in `build/<host>/stage0-incremental`. Note that we only use incremental
129 compilation for the stage0 -> stage1 compilation -- this is because
130 the stage1 compiler is changing, and we don't try to cache and reuse
131 incremental artifacts across different versions of the compiler.
133 You can always drop the `--incremental` to build as normal (but you
134 will still be using the local nightly as your bootstrap).
138 This build system houses all output under the `build` directory, which looks
142 # Root folder of all output. Everything is scoped underneath here
145 # Location where the stage0 compiler downloads are all cached. This directory
146 # only contains the tarballs themselves, as they're extracted elsewhere.
153 # Output directory for building this build system itself. The stage0
154 # cargo/rustc are used to build the build system into this location.
159 # Output of the dist-related steps like dist-std, dist-rustc, and dist-docs
162 # Temporary directory used for various input/output as part of various stages
165 # Each remaining directory is scoped by the "host" triple of compilation at
167 x86_64-unknown-linux-gnu/
169 # The build artifacts for the `compiler-rt` library for the target that
170 # this folder is under. The exact layout here will likely depend on the
171 # platform, and this is also built with CMake, so the build system is
172 # also likely different.
176 # Output folder for LLVM if it is compiled for this target
179 # build folder (e.g. the platform-specific build system). Like with
180 # compiler-rt, this is compiled with CMake
183 # Installation of LLVM. Note that we run the equivalent of 'make install'
184 # for LLVM, to setup these folders.
191 # Output folder for all documentation of this target. This is what's filled
192 # in whenever the `doc` step is run.
195 # Output for all compiletest-based test suites
201 # Location where the stage0 Cargo and Rust compiler are unpacked. This
202 # directory is purely an extracted and overlaid tarball of these two (done
203 # by the bootstrap python script). In theory, the build system does not
204 # modify anything under this directory afterwards.
207 # These to-build directories are the cargo output directories for builds of
208 # the standard library and compiler, respectively. Internally, these may also
209 # have other target directories, which represent artifacts being compiled
210 # from the host to the specified target.
212 # Essentially, each of these directories is filled in by one `cargo`
213 # invocation. The build system instruments calling Cargo in the right order
214 # with the right variables to ensure that these are filled in correctly.
220 # This is a special case of the above directories, **not** filled in via
221 # Cargo but rather the build system itself. The stage0 compiler already has
222 # a set of target libraries for its own host triple (in its own sysroot)
223 # inside of stage0/. When we run the stage0 compiler to bootstrap more
224 # things, however, we don't want to use any of these libraries (as those are
225 # the ones that we're building). So essentially, when the stage1 compiler is
226 # being compiled (e.g. after libstd has been built), *this* is used as the
227 # sysroot for the stage0 compiler being run.
229 # Basically, this directory is just a temporary artifact used to configure the
230 # stage0 compiler to ensure that the libstd that we just built is used to
231 # compile the stage1 compiler.
234 # These output directories are intended to be standalone working
235 # implementations of the compiler (corresponding to each stage). The build
236 # system will link (using hard links) output from stageN-{std,rustc} into
237 # each of these directories.
239 # In theory, there is no extra build output in these directories.
247 The current build is unfortunately not quite as simple as `cargo build` in a
248 directory, but rather the compiler is split into three different Cargo projects:
250 * `library/std` - the standard library
251 * `library/test` - testing support, depends on libstd
252 * `compiler/rustc` - the actual compiler itself
254 Each "project" has a corresponding Cargo.lock file with all dependencies, and
255 this means that building the compiler involves running Cargo three times. The
256 structure here serves two goals:
258 1. Facilitating dependencies coming from crates.io. These dependencies don't
259 depend on `std`, so libstd is a separate project compiled ahead of time
260 before the actual compiler builds.
261 2. Splitting "host artifacts" from "target artifacts". That is, when building
262 code for an arbitrary target, you don't need the entire compiler, but you'll
263 end up needing libraries like libtest that depend on std but also want to use
264 crates.io dependencies. Hence, libtest is split out as its own project that
265 is sequenced after `std` but before `rustc`. This project is built for all
268 There is some loss in build parallelism here because libtest can be compiled in
269 parallel with a number of rustc artifacts, but in theory, the loss isn't too bad!
273 We've actually got quite a few tools that we use in the compiler's build system
274 and for testing. To organize these, each tool is a project in `src/tools` with a
275 corresponding `Cargo.toml`. All tools are compiled with Cargo (currently having
276 independent `Cargo.lock` files) and do not currently explicitly depend on the
277 compiler or standard library. Compiling each tool is sequenced after the
278 appropriate libstd/libtest/librustc compile above.
280 ## Extending rustbuild
282 So, you'd like to add a feature to the rustbuild build system or just fix a bug.
283 Great! One of the major motivational factors for moving away from `make` is that
284 Rust is in theory much easier to read, modify, and write. If you find anything
285 excessively confusing, please open an issue on this, and we'll try to get it
286 documented or simplified, pronto.
288 First up, you'll probably want to read over the documentation above, as that'll
289 give you a high level overview of what rustbuild is doing. You also probably
290 want to play around a bit yourself by just getting it up and running before you
291 dive too much into the actual build system itself.
293 After that, each module in rustbuild should have enough documentation to keep
294 you up and running. Some general areas that you may be interested in modifying
297 * Adding a new build tool? Take a look at `bootstrap/tool.rs` for examples of
299 * Adding a new compiler crate? Look no further! Adding crates can be done by
300 adding a new directory with `Cargo.toml` followed by configuring all
301 `Cargo.toml` files accordingly.
302 * Adding a new dependency from crates.io? This should just work inside the
303 compiler artifacts stage (everything other than libtest and libstd).
304 * Adding a new configuration option? You'll want to modify `bootstrap/flags.rs`
305 for command line flags and then `bootstrap/config.rs` to copy the flags to the
307 * Adding a sanity check? Take a look at `bootstrap/sanity.rs`.
309 If you make a major change, please remember to:
311 + Update `VERSION` in `src/bootstrap/main.rs`.
312 * Update `changelog-seen = N` in `config.toml.example`.
313 * Add an entry in `src/bootstrap/CHANGELOG.md`.
315 A 'major change' includes
318 * A change in the default options.
320 Changes that do not affect contributors to the compiler or users
321 building rustc from source don't need an update to `VERSION`.
323 If you have any questions, feel free to reach out on the `#t-infra` channel in
324 the [Rust Zulip server][rust-zulip] or ask on internals.rust-lang.org. When
325 you encounter bugs, please file issues on the rust-lang/rust issue tracker.
327 [rust-zulip]: https://rust-lang.zulipchat.com/#narrow/stream/242791-t-infra