1 % The Rust Testing Guide
3 > Program testing can be a very effective way to show the presence of bugs, but
4 > it is hopelessly inadequate for showing their absence.
6 > Edsger W. Dijkstra, "The Humble Programmer" (1972)
8 Let's talk about how to test Rust code. What we will not be talking about is
9 the right way to test Rust code. There are many schools of thought regarding
10 the right and wrong way to write tests. All of these approaches use the same
11 basic tools, and so we'll show you the syntax for using them.
13 # The `test` attribute
15 At its simplest, a test in Rust is a function that's annotated with the `test`
16 attribute. Let's make a new project with Cargo called `adder`:
23 Cargo will automatically generate a simple test when you make a new project.
24 Here's the contents of `src/lib.rs`:
32 Note the `#[test]`. This attribute indicates that this is a test function. It
33 currently has no body. That's good enough to pass! We can run the tests with
38 Compiling adder v0.0.1 (file:///home/you/projects/adder)
39 Running target/adder-91b3e234d4ed382a
44 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
50 test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured
53 Cargo compiled and ran our tests. There are two sets of output here: one
54 for the test we wrote, and another for documentation tests. We'll talk about
55 those later. For now, see this line:
61 Note the `it_works`. This comes from the name of our function:
68 We also get a summary line:
71 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
74 So why does our do-nothing test pass? Any test which doesn't `panic!` passes,
75 and any test that does `panic!` fails. Let's make our test fail:
84 `assert!` is a macro provided by Rust which takes one argument: if the argument
85 is `true`, nothing happens. If the argument is false, it `panic!`s. Let's run
90 Compiling adder v0.0.1 (file:///home/you/projects/adder)
91 Running target/adder-91b3e234d4ed382a
94 test it_works ... FAILED
98 ---- it_works stdout ----
99 task 'it_works' panicked at 'assertion failed: false', /home/steve/tmp/adder/src/lib.rs:3
106 test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured
108 task '<main>' panicked at 'Some tests failed', /home/steve/src/rust/src/libtest/lib.rs:247
111 Rust indicates that our test failed:
114 test it_works ... FAILED
117 And that's reflected in the summary line:
120 test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured
123 We also get a non-zero status code:
130 This is useful if you want to integrate `cargo test` into other tooling.
132 We can invert our test's failure with another attribute: `should_fail`:
142 This test will now succeed if we `panic!` and fail if we complete. Let's try it:
146 Compiling adder v0.0.1 (file:///home/you/projects/adder)
147 Running target/adder-91b3e234d4ed382a
152 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
158 test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured
161 Rust provides another macro, `assert_eq!`, that compares two arguments for
168 assert_eq!("Hello", "world");
172 Does this test pass or fail? Because of the `should_fail` attribute, it
177 Compiling adder v0.0.1 (file:///home/you/projects/adder)
178 Running target/adder-91b3e234d4ed382a
183 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
189 test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured
192 `should_fail` tests can be fragile, as it's hard to guarantee that the test
193 didn't fail for an unexpected reason. To help with this, an optional `expected`
194 parameter can be added to the `should_fail` attribute. The test harness will
195 make sure that the failure message contains the provided text. A safer version
196 of the example above would be:
200 #[should_fail(expected = "assertion failed")]
202 assert_eq!("Hello", "world");
206 That's all there is to the basics! Let's write one 'real' test:
209 pub fn add_two(a: i32) -> i32 {
215 assert_eq!(4, add_two(2));
219 This is a very common use of `assert_eq!`: call some function with
220 some known arguments and compare it to the expected output.
224 There is one way in which our existing example is not idiomatic: it's
225 missing the test module. The idiomatic way of writing our example
229 pub fn add_two(a: i32) -> i32 {
239 assert_eq!(4, add_two(2));
244 There's a few changes here. The first is the introduction of a `mod tests` with
245 a `cfg` attribute. The module allows us to group all of our tests together, and
246 to also define helper functions if needed, that don't become a part of the rest
247 of our crate. The `cfg` attribute only compiles our test code if we're
248 currently trying to run the tests. This can save compile time, and also ensures
249 that our tests are entirely left out of a normal build.
251 The second change is the `use` declaration. Because we're in an inner module,
252 we need to bring our test function into scope. This can be annoying if you have
253 a large module, and so this is a common use of the `glob` feature. Let's change
254 our `src/lib.rs` to make use of it:
259 pub fn add_two(a: i32) -> i32 {
269 assert_eq!(4, add_two(2));
274 Note the `feature` attribute, as well as the different `use` line. Now we run
279 Updating registry `https://github.com/rust-lang/crates.io-index`
280 Compiling adder v0.0.1 (file:///home/you/projects/adder)
281 Running target/adder-91b3e234d4ed382a
284 test test::it_works ... ok
286 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
292 test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured
297 The current convention is to use the `test` module to hold your "unit"-style
298 tests. Anything that just tests one small bit of functionality makes sense to
299 go here. But what about "integration"-style tests instead? For that, we have
300 the `tests` directory
302 # The `tests` directory
304 To write an integration test, let's make a `tests` directory, and
305 put a `tests/lib.rs` file inside, with this as its contents:
312 assert_eq(4, adder::add_two(2));
316 This looks similar to our previous tests, but slightly different. We now have
317 an `extern crate adder` at the top. This is because the tests in the `tests`
318 directory are an entirely separate crate, and so we need to import our library.
319 This is also why `tests` is a suitable place to write integration-style tests:
320 they use the library like any other consumer of it would.
326 Compiling adder v0.0.1 (file:///home/you/projects/adder)
327 Running target/adder-91b3e234d4ed382a
330 test test::it_works ... ok
332 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
334 Running target/lib-c18e7d3494509e74
339 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
345 test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured
348 Now we have three sections: our previous test is also run, as well as our new
351 That's all there is to the `tests` directory. The `test` module isn't needed
352 here, since the whole thing is focused on tests.
354 Let's finally check out that third section: documentation tests.
356 # Documentation tests
358 Nothing is better than documentation with examples. Nothing is worse than
359 examples that don't actually work, because the code has changed since the
360 documentation has been written. To this end, Rust supports automatically
361 running examples in your documentation. Here's a fleshed-out `src/lib.rs`
365 //! The `adder` crate provides functions that add numbers to other numbers.
370 //! assert_eq!(4, adder::add_two(2));
375 /// This function adds two to its argument.
380 /// use adder::add_two;
382 /// assert_eq!(4, add_two(2));
384 pub fn add_two(a: i32) -> i32 {
394 assert_eq!(4, add_two(2));
399 Note the module-level documentation with `//!` and the function-level
400 documentation with `///`. Rust's documentation supports Markdown in comments,
401 and so triple graves mark code blocks. It is conventional to include the
402 `# Examples` section, exactly like that, with examples following.
404 Let's run the tests again:
408 Compiling adder v0.0.1 (file:///home/steve/tmp/adder)
409 Running target/adder-91b3e234d4ed382a
412 test test::it_works ... ok
414 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
416 Running target/lib-c18e7d3494509e74
421 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
426 test add_two_0 ... ok
429 test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured
432 Now we have all three kinds of tests running! Note the names of the
433 documentation tests: the `_0` is generated for the module test, and `add_two_0`
434 for the function test. These will auto increment with names like `add_two_1` as
435 you add more examples.
439 Rust also supports benchmark tests, which can test the performance of your
440 code. Let's make our `src/lib.rs` look like this (comments elided):
447 pub fn add_two(a: i32) -> i32 {
458 assert_eq!(4, add_two(2));
462 fn bench_add_two(b: &mut Bencher) {
463 b.iter(|| add_two(2));
468 We've imported the `test` crate, which contains our benchmarking support.
469 We have a new function as well, with the `bench` attribute. Unlike regular
470 tests, which take no arguments, benchmark tests take a `&mut Bencher`. This
471 `Bencher` provides an `iter` method, which takes a closure. This closure
472 contains the code we'd like to benchmark.
474 We can run benchmark tests with `cargo bench`:
478 Compiling adder v0.0.1 (file:///home/steve/tmp/adder)
479 Running target/release/adder-91b3e234d4ed382a
482 test tests::it_works ... ignored
483 test tests::bench_add_two ... bench: 1 ns/iter (+/- 0)
485 test result: ok. 0 passed; 0 failed; 1 ignored; 1 measured
488 Our non-benchmark test was ignored. You may have noticed that `cargo bench`
489 takes a bit longer than `cargo test`. This is because Rust runs our benchmark
490 a number of times, and then takes the average. Because we're doing so little
491 work in this example, we have a `1 ns/iter (+/- 0)`, but this would show
492 the variance if there was one.
494 Advice on writing benchmarks:
497 * Move setup code outside the `iter` loop; only put the part you want to measure inside
498 * Make the code do "the same thing" on each iteration; do not accumulate or change state
499 * Make the outer function idempotent too; the benchmark runner is likely to run
501 * Make the inner `iter` loop short and fast so benchmark runs are fast and the
502 calibrator can adjust the run-length at fine resolution
503 * Make the code in the `iter` loop do something simple, to assist in pinpointing
504 performance improvements (or regressions)
506 There's another tricky part to writing benchmarks: benchmarks compiled with
507 optimizations activated can be dramatically changed by the optimizer so that
508 the benchmark is no longer benchmarking what one expects. For example, the
509 compiler might recognize that some calculation has no external effects and
517 fn bench_xor_1000_ints(b: &mut Bencher) {
519 range(0u, 1000).fold(0, |old, new| old ^ new);
524 gives the following results
528 test bench_xor_1000_ints ... bench: 0 ns/iter (+/- 0)
530 test result: ok. 0 passed; 0 failed; 0 ignored; 1 measured
533 The benchmarking runner offers two ways to avoid this. Either, the closure that
534 the `iter` method receives can return an arbitrary value which forces the
535 optimizer to consider the result used and ensures it cannot remove the
536 computation entirely. This could be done for the example above by adjusting the
541 # impl X { fn iter<T, F>(&self, _: F) where F: FnMut() -> T {} } let b = X;
543 // note lack of `;` (could also use an explicit `return`).
544 range(0u, 1000).fold(0, |old, new| old ^ new)
548 Or, the other option is to call the generic `test::black_box` function, which
549 is an opaque "black box" to the optimizer and so forces it to consider any
557 # impl X { fn iter<T, F>(&self, _: F) where F: FnMut() -> T {} } let b = X;
559 test::black_box(range(0u, 1000).fold(0, |old, new| old ^ new));
564 Neither of these read or modify the value, and are very cheap for small values.
565 Larger values can be passed indirectly to reduce overhead (e.g.
566 `black_box(&huge_struct)`).
568 Performing either of the above changes gives the following benchmarking results
572 test bench_xor_1000_ints ... bench: 1 ns/iter (+/- 0)
574 test result: ok. 0 passed; 0 failed; 0 ignored; 1 measured