1 # rustbuild - Bootstrapping Rust
3 This is an in-progress README which is targeted at helping to explain how Rust
4 is bootstrapped and in general some of the technical details of the build
9 The rustbuild build system has a primary entry point, a top level `x.py` script:
15 Note that if you're on Unix you should be able to execute the script directly:
21 The script accepts commands, flags, and arguments to determine what to do:
23 * `build` - a general purpose command for compiling code. Alone `build` will
24 bootstrap the entire compiler, and otherwise arguments passed indicate what to
28 # build the whole compiler
31 # build the stage1 compiler
32 ./x.py build --stage 1
35 ./x.py build --stage 0 src/libstd
37 # build a particular crate in stage0
38 ./x.py build --stage 0 src/libtest
41 If files are dirty that would normally be rebuilt from stage 0, that can be
42 overridden using `--keep-stage 0`. Using `--keep-stage n` will skip all steps
43 that belong to stage n or earlier:
46 # keep old build products for stage 0 and build stage 1
47 ./x.py build --keep-stage 0 --stage 1
50 * `test` - a command for executing unit tests. Like the `build` command this
51 will execute the entire test suite by default, and otherwise it can be used to
52 select which test suite is run:
58 # execute the run-pass test suite
59 ./x.py test src/test/run-pass
61 # execute only some tests in the run-pass test suite
62 ./x.py test src/test/run-pass --test-args substring-of-test-name
64 # execute tests in the standard library in stage0
65 ./x.py test --stage 0 src/libstd
67 # execute all doc tests
71 * `doc` - a command for building documentation. Like above can take arguments
74 ## Configuring rustbuild
76 There are currently two methods for configuring the rustbuild build system.
78 First, rustbuild offers a TOML-based configuration system with a `config.toml`
79 file. An example of this configuration can be found at `config.toml.example`,
80 and the configuration file can also be passed as `--config path/to/config.toml`
81 if the build system is being invoked manually (via the python script).
83 Next, the `./configure` options serialized in `config.mk` will be
84 parsed and read. That is, if any `./configure` options are passed, they'll be
85 handled naturally. `./configure` should almost never be used for local
86 installations, and is primarily useful for CI. Prefer to customize behavior
89 Finally, rustbuild makes use of the [gcc-rs crate] which has [its own
90 method][env-vars] of configuring C compilers and C flags via environment
93 [gcc-rs crate]: https://github.com/alexcrichton/gcc-rs
94 [env-vars]: https://github.com/alexcrichton/gcc-rs#external-configuration-via-environment-variables
98 The rustbuild build system goes through a few phases to actually build the
99 compiler. What actually happens when you invoke rustbuild is:
101 1. The entry point script, `x.py` is run. This script is
102 responsible for downloading the stage0 compiler/Cargo binaries, and it then
103 compiles the build system itself (this folder). Finally, it then invokes the
104 actual `bootstrap` binary build system.
105 2. In Rust, `bootstrap` will slurp up all configuration, perform a number of
106 sanity checks (compilers exist for example), and then start building the
108 3. The stage0 `cargo` downloaded earlier is used to build the standard library
109 and the compiler, and then these binaries are then copied to the `stage1`
110 directory. That compiler is then used to generate the stage1 artifacts which
111 are then copied to the stage2 directory, and then finally the stage2
112 artifacts are generated using that compiler.
114 The goal of each stage is to (a) leverage Cargo as much as possible and failing
115 that (b) leverage Rust as much as possible!
117 ## Incremental builds
119 You can configure rustbuild to use incremental compilation. Because
120 incremental is new and evolving rapidly, if you want to use it, it is
121 recommended that you replace the snapshot with a locally installed
122 nightly build of rustc. You will want to keep this up to date.
124 To follow this course of action, first thing you will want to do is to
125 install a nightly, presumably using `rustup`. You will then want to
126 configure your directory to use this build, like so:
129 # configure to use local rust instead of downloading a beta.
130 # `--local-rust-root` is optional here. If elided, we will
131 # use whatever rustc we find on your PATH.
132 > configure --enable-rustbuild --local-rust-root=~/.cargo/ --enable-local-rebuild
135 After that, you can use the `--incremental` flag to actually do
139 > ../x.py build --incremental
142 The `--incremental` flag will store incremental compilation artifacts
143 in `build/<host>/stage0-incremental`. Note that we only use incremental
144 compilation for the stage0 -> stage1 compilation -- this is because
145 the stage1 compiler is changing, and we don't try to cache and reuse
146 incremental artifacts across different versions of the compiler. For
147 this reason, `--incremental` defaults to `--stage 1` (though you can
148 manually select a higher stage, if you prefer).
150 You can always drop the `--incremental` to build as normal (but you
151 will still be using the local nightly as your bootstrap).
155 This build system houses all output under the `build` directory, which looks
159 # Root folder of all output. Everything is scoped underneath here
162 # Location where the stage0 compiler downloads are all cached. This directory
163 # only contains the tarballs themselves as they're extracted elsewhere.
170 # Output directory for building this build system itself. The stage0
171 # cargo/rustc are used to build the build system into this location.
176 # Output of the dist-related steps like dist-std, dist-rustc, and dist-docs
179 # Temporary directory used for various input/output as part of various stages
182 # Each remaining directory is scoped by the "host" triple of compilation at
184 x86_64-unknown-linux-gnu/
186 # The build artifacts for the `compiler-rt` library for the target this
187 # folder is under. The exact layout here will likely depend on the platform,
188 # and this is also built with CMake so the build system is also likely
193 # Output folder for LLVM if it is compiled for this target
196 # build folder (e.g. the platform-specific build system). Like with
197 # compiler-rt this is compiled with CMake
200 # Installation of LLVM. Note that we run the equivalent of 'make install'
201 # for LLVM to setup these folders.
208 # Output folder for all documentation of this target. This is what's filled
209 # in whenever the `doc` step is run.
212 # Output for all compiletest-based test suites
219 # Location where the stage0 Cargo and Rust compiler are unpacked. This
220 # directory is purely an extracted and overlaid tarball of these two (done
221 # by the bootstrapy python script). In theory the build system does not
222 # modify anything under this directory afterwards.
225 # These to build directories are the cargo output directories for builds of
226 # the standard library and compiler, respectively. Internally these may also
227 # have other target directories, which represent artifacts being compiled
228 # from the host to the specified target.
230 # Essentially, each of these directories is filled in by one `cargo`
231 # invocation. The build system instruments calling Cargo in the right order
232 # with the right variables to ensure these are filled in correctly.
238 # This is a special case of the above directories, **not** filled in via
239 # Cargo but rather the build system itself. The stage0 compiler already has
240 # a set of target libraries for its own host triple (in its own sysroot)
241 # inside of stage0/. When we run the stage0 compiler to bootstrap more
242 # things, however, we don't want to use any of these libraries (as those are
243 # the ones that we're building). So essentially, when the stage1 compiler is
244 # being compiled (e.g. after libstd has been built), *this* is used as the
245 # sysroot for the stage0 compiler being run.
247 # Basically this directory is just a temporary artifact use to configure the
248 # stage0 compiler to ensure that the libstd we just built is used to
249 # compile the stage1 compiler.
252 # These output directories are intended to be standalone working
253 # implementations of the compiler (corresponding to each stage). The build
254 # system will link (using hard links) output from stageN-{std,rustc} into
255 # each of these directories.
257 # In theory there is no extra build output in these directories.
265 The current build is unfortunately not quite as simple as `cargo build` in a
266 directory, but rather the compiler is split into three different Cargo projects:
268 * `src/libstd` - the standard library
269 * `src/libtest` - testing support, depends on libstd
270 * `src/rustc` - the actual compiler itself
272 Each "project" has a corresponding Cargo.lock file with all dependencies, and
273 this means that building the compiler involves running Cargo three times. The
274 structure here serves two goals:
276 1. Facilitating dependencies coming from crates.io. These dependencies don't
277 depend on `std`, so libstd is a separate project compiled ahead of time
278 before the actual compiler builds.
279 2. Splitting "host artifacts" from "target artifacts". That is, when building
280 code for an arbitrary target you don't need the entire compiler, but you'll
281 end up needing libraries like libtest that depend on std but also want to use
282 crates.io dependencies. Hence, libtest is split out as its own project that
283 is sequenced after `std` but before `rustc`. This project is built for all
286 There is some loss in build parallelism here because libtest can be compiled in
287 parallel with a number of rustc artifacts, but in theory the loss isn't too bad!
291 We've actually got quite a few tools that we use in the compiler's build system
292 and for testing. To organize these, each tool is a project in `src/tools` with a
293 corresponding `Cargo.toml`. All tools are compiled with Cargo (currently having
294 independent `Cargo.lock` files) and do not currently explicitly depend on the
295 compiler or standard library. Compiling each tool is sequenced after the
296 appropriate libstd/libtest/librustc compile above.
298 ## Extending rustbuild
300 So you'd like to add a feature to the rustbuild build system or just fix a bug.
301 Great! One of the major motivational factors for moving away from `make` is that
302 Rust is in theory much easier to read, modify, and write. If you find anything
303 excessively confusing, please open an issue on this and we'll try to get it
304 documented or simplified pronto.
306 First up, you'll probably want to read over the documentation above as that'll
307 give you a high level overview of what rustbuild is doing. You also probably
308 want to play around a bit yourself by just getting it up and running before you
309 dive too much into the actual build system itself.
311 After that, each module in rustbuild should have enough documentation to keep
312 you up and running. Some general areas that you may be interested in modifying
315 * Adding a new build tool? Take a look at `bootstrap/tool.rs` for examples of
317 * Adding a new compiler crate? Look no further! Adding crates can be done by
318 adding a new directory with `Cargo.toml` followed by configuring all
319 `Cargo.toml` files accordingly.
320 * Adding a new dependency from crates.io? This should just work inside the
321 compiler artifacts stage (everything other than libtest and libstd).
322 * Adding a new configuration option? You'll want to modify `bootstrap/flags.rs`
323 for command line flags and then `bootstrap/config.rs` to copy the flags to the
325 * Adding a sanity check? Take a look at `bootstrap/sanity.rs`.
327 If you have any questions feel free to reach out on `#rust-infra` on IRC or ask on
328 internals.rust-lang.org. When you encounter bugs, please file issues on the
329 rust-lang/rust issue tracker.