1 # Compiler Test Documentation
3 In the Rust project, we use a special set of commands embedded in
4 comments to test the Rust compiler. There are two groups of commands:
9 Both types of commands are inside comments, but header commands should
10 be in a comment before any code.
12 ## Summary of Error Info Commands
14 Error commands specify something about certain lines of the
15 program. They tell the test what kind of error and what message you
18 * `~`: Associates the following error level and message with the
20 * `~|`: Associates the following error level and message with the same
21 line as the previous comment
22 * `~^`: Associates the following error level and message with the
23 previous line. Each caret (`^`) that you add adds a line to this, so
24 `~^^^^^^^` is seven lines up.
26 The error levels that you can have are:
31 4. `HELP` and `SUGGESTION`*
33 \* **Note**: `SUGGESTION` must follow immediately after `HELP`.
35 ## Summary of Header Commands
37 Header commands specify something about the entire test file as a
38 whole. They are normally put right after the copyright comment, e.g.:
41 // Copyright blah blah blah
42 // except according to those terms.
44 // ignore-test This doesn't actually work
49 These are used to ignore the test in some situations, which means the test won't
52 * `ignore-X` where `X` is a target detail or stage will ignore the test accordingly (see below)
53 * `ignore-pretty` will not compile the pretty-printed test (this is done to test the pretty-printer, but might not always work)
54 * `ignore-test` always ignores the test
55 * `ignore-lldb` and `ignore-gdb` will skip a debuginfo test on that debugger.
57 Some examples of `X` in `ignore-X`:
59 * Architecture: `aarch64`, `arm`, `asmjs`, `mips`, `wasm32`, `x86_64`, `x86`, ...
60 * OS: `android`, `emscripten`, `freebsd`, `ios`, `linux`, `macos`, `windows`, ...
61 * Environment (fourth word of the target triple): `gnu`, `msvc`, `musl`.
62 * Pointer width: `32bit`, `64bit`.
63 * Stage: `stage0`, `stage1`, `stage2`.
65 ### Other Header Commands
67 * `min-{gdb,lldb}-version`
69 * `must-compile-successfully` for UI tests, indicates that the test is supposed
70 to compile, as opposed to the default where the test is supposed to error out.
71 * `compile-flags` passes extra command-line args to the compiler,
72 e.g. `compile-flags -g` which forces debuginfo to be enabled.
73 * `should-fail` indicates that the test should fail; used for "meta testing",
74 where we test the compiletest program itself to check that it will generate
75 errors in appropriate scenarios. This header is ignored for pretty-printer tests.
76 * `gate-test-X` where `X` is a feature marks the test as "gate test" for feature X.
77 Such tests are supposed to ensure that the compiler errors when usage of a gated
78 feature is attempted without the proper `#![feature(X)]` tag.
79 Each unstable lang feature is required to have a gate test.
83 Certain classes of tests support "revisions" (as of the time of this
84 writing, this includes run-pass, compile-fail, run-fail, and
85 incremental, though incremental tests are somewhat
86 different). Revisions allow a single test file to be used for multiple
87 tests. This is done by adding a special header at the top of the file:
90 // revisions: foo bar baz
93 This will result in the test being compiled (and tested) three times,
94 once with `--cfg foo`, once with `--cfg bar`, and once with `--cfg
95 baz`. You can therefore use `#[cfg(foo)]` etc within the test to tweak
96 each of these results.
98 You can also customize headers and expected error messages to a particular
99 revision. To do this, add `[foo]` (or `bar`, `baz`, etc) after the `//`
103 // A flag to pass in only for cfg `foo`:
104 //[foo]compile-flags: -Z verbose
108 let x: usize = 32_u32; //[foo]~ ERROR mismatched types
112 Note that not all headers have meaning when customized to a revision.
113 For example, the `ignore-test` header (and all "ignore" headers)
114 currently only apply to the test as a whole, not to particular
115 revisions. The only headers that are intended to really work when
116 customized to a revision are error patterns and compiler flags.
118 ## Guide to the UI Tests
120 The UI tests are intended to capture the compiler's complete output,
121 so that we can test all aspects of the presentation. They work by
122 compiling a file (e.g., `ui/hello_world/main.rs`), capturing the output,
123 and then applying some normalization (see below). This normalized
124 result is then compared against reference files named
125 `ui/hello_world/main.stderr` and `ui/hello_world/main.stdout`. If either of
126 those files doesn't exist, the output must be empty. If the test run
127 fails, we will print out the current output, but it is also saved in
128 `build/<target-triple>/test/ui/hello_world/main.stdout` (this path is
129 printed as part of the test failure message), so you can run `diff` and
132 Normally, the test-runner checks that UI tests fail compilation. If you want
133 to do a UI test for code that *compiles* (e.g. to test warnings, or if you
134 have a collection of tests, only some of which error out), you can use the
135 `// must-compile-successfully` header command to have the test runner instead
136 check that the test compiles successfully.
138 ### Editing and updating the reference files
140 If you have changed the compiler's output intentionally, or you are
141 making a new test, you can use the script `ui/update-references.sh` to
142 update the references. When you run the test framework, it will report
143 various errors: in those errors is a command you can use to run the
144 `ui/update-references.sh` script, which will then copy over the files
145 from the build directory and use them as the new reference. You can
146 also just run `ui/update-all-references.sh`. In both cases, you can run
147 the script with `--help` to get a help message.
151 The normalization applied is aimed at eliminating output difference
152 between platforms, mainly about filenames:
154 - the test directory is replaced with `$DIR`
155 - all backslashes (`\`) are converted to forward slashes (`/`) (for Windows)
156 - all CR LF newlines are converted to LF
158 Sometimes these built-in normalizations are not enough. In such cases, you
159 may provide custom normalization rules using the header commands, e.g.
162 // normalize-stdout-test: "foo" -> "bar"
163 // normalize-stderr-32bit: "fn\(\) \(32 bits\)" -> "fn\(\) \($$PTR bits\)"
164 // normalize-stderr-64bit: "fn\(\) \(64 bits\)" -> "fn\(\) \($$PTR bits\)"
167 This tells the test, on 32-bit platforms, whenever the compiler writes
168 `fn() (32 bits)` to stderr, it should be normalized to read `fn() ($PTR bits)`
169 instead. Similar for 64-bit. The replacement is performed by regexes using
170 default regex flavor provided by `regex` crate.
172 The corresponding reference file will use the normalized output to test both
173 32-bit and 64-bit platforms:
178 = note: source type: fn() ($PTR bits)
179 = note: target type: u16 (16 bits)
183 Please see `ui/transmute/main.rs` and `.stderr` for a concrete usage example.
185 Besides `normalize-stderr-32bit` and `-64bit`, one may use any target
186 information or stage supported by `ignore-X` here as well (e.g.
187 `normalize-stderr-windows` or simply `normalize-stderr-test` for unconditional