3 > Program testing can be a very effective way to show the presence of bugs, but
4 > it is hopelessly inadequate for showing their absence.
6 > Edsger W. Dijkstra, "The Humble Programmer" (1972)
8 Let's talk about how to test Rust code. What we will not be talking about is
9 the right way to test Rust code. There are many schools of thought regarding
10 the right and wrong way to write tests. All of these approaches use the same
11 basic tools, and so we'll show you the syntax for using them.
13 # The `test` attribute
15 At its simplest, a test in Rust is a function that's annotated with the `test`
16 attribute. Let's make a new project with Cargo called `adder`:
23 Cargo will automatically generate a simple test when you make a new project.
24 Here's the contents of `src/lib.rs`:
32 Note the `#[test]`. This attribute indicates that this is a test function. It
33 currently has no body. That's good enough to pass! We can run the tests with
38 Compiling adder v0.0.1 (file:///home/you/projects/adder)
39 Running target/adder-91b3e234d4ed382a
44 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
50 test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured
53 Cargo compiled and ran our tests. There are two sets of output here: one
54 for the test we wrote, and another for documentation tests. We'll talk about
55 those later. For now, see this line:
61 Note the `it_works`. This comes from the name of our function:
68 We also get a summary line:
71 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
74 So why does our do-nothing test pass? Any test which doesn't `panic!` passes,
75 and any test that does `panic!` fails. Let's make our test fail:
84 `assert!` is a macro provided by Rust which takes one argument: if the argument
85 is `true`, nothing happens. If the argument is false, it `panic!`s. Let's run
90 Compiling adder v0.0.1 (file:///home/you/projects/adder)
91 Running target/adder-91b3e234d4ed382a
94 test it_works ... FAILED
98 ---- it_works stdout ----
99 thread 'it_works' panicked at 'assertion failed: false', /home/steve/tmp/adder/src/lib.rs:3
106 test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured
108 thread '<main>' panicked at 'Some tests failed', /home/steve/src/rust/src/libtest/lib.rs:247
111 Rust indicates that our test failed:
114 test it_works ... FAILED
117 And that's reflected in the summary line:
120 test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured
123 We also get a non-zero status code:
130 This is useful if you want to integrate `cargo test` into other tooling.
132 We can invert our test's failure with another attribute: `should_panic`:
142 This test will now succeed if we `panic!` and fail if we complete. Let's try it:
146 Compiling adder v0.0.1 (file:///home/you/projects/adder)
147 Running target/adder-91b3e234d4ed382a
152 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
158 test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured
161 Rust provides another macro, `assert_eq!`, that compares two arguments for
168 assert_eq!("Hello", "world");
172 Does this test pass or fail? Because of the `should_panic` attribute, it
177 Compiling adder v0.0.1 (file:///home/you/projects/adder)
178 Running target/adder-91b3e234d4ed382a
183 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
189 test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured
192 `should_panic` tests can be fragile, as it's hard to guarantee that the test
193 didn't fail for an unexpected reason. To help with this, an optional `expected`
194 parameter can be added to the `should_panic` attribute. The test harness will
195 make sure that the failure message contains the provided text. A safer version
196 of the example above would be:
200 #[should_panic(expected = "assertion failed")]
202 assert_eq!("Hello", "world");
206 That's all there is to the basics! Let's write one 'real' test:
209 pub fn add_two(a: i32) -> i32 {
215 assert_eq!(4, add_two(2));
219 This is a very common use of `assert_eq!`: call some function with
220 some known arguments and compare it to the expected output.
224 There is one way in which our existing example is not idiomatic: it's
225 missing the test module. The idiomatic way of writing our example
229 pub fn add_two(a: i32) -> i32 {
239 assert_eq!(4, add_two(2));
244 There's a few changes here. The first is the introduction of a `mod test` with
245 a `cfg` attribute. The module allows us to group all of our tests together, and
246 to also define helper functions if needed, that don't become a part of the rest
247 of our crate. The `cfg` attribute only compiles our test code if we're
248 currently trying to run the tests. This can save compile time, and also ensures
249 that our tests are entirely left out of a normal build.
251 The second change is the `use` declaration. Because we're in an inner module,
252 we need to bring our test function into scope. This can be annoying if you have
253 a large module, and so this is a common use of the `glob` feature. Let's change
254 our `src/lib.rs` to make use of it:
258 pub fn add_two(a: i32) -> i32 {
268 assert_eq!(4, add_two(2));
273 Note the different `use` line. Now we run our tests:
277 Updating registry `https://github.com/rust-lang/crates.io-index`
278 Compiling adder v0.0.1 (file:///home/you/projects/adder)
279 Running target/adder-91b3e234d4ed382a
282 test test::it_works ... ok
284 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
290 test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured
295 The current convention is to use the `test` module to hold your "unit-style"
296 tests. Anything that just tests one small bit of functionality makes sense to
297 go here. But what about "integration-style" tests instead? For that, we have
298 the `tests` directory
300 # The `tests` directory
302 To write an integration test, let's make a `tests` directory, and
303 put a `tests/lib.rs` file inside, with this as its contents:
310 assert_eq!(4, adder::add_two(2));
314 This looks similar to our previous tests, but slightly different. We now have
315 an `extern crate adder` at the top. This is because the tests in the `tests`
316 directory are an entirely separate crate, and so we need to import our library.
317 This is also why `tests` is a suitable place to write integration-style tests:
318 they use the library like any other consumer of it would.
324 Compiling adder v0.0.1 (file:///home/you/projects/adder)
325 Running target/adder-91b3e234d4ed382a
328 test test::it_works ... ok
330 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
332 Running target/lib-c18e7d3494509e74
337 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
343 test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured
346 Now we have three sections: our previous test is also run, as well as our new
349 That's all there is to the `tests` directory. The `test` module isn't needed
350 here, since the whole thing is focused on tests.
352 Let's finally check out that third section: documentation tests.
354 # Documentation tests
356 Nothing is better than documentation with examples. Nothing is worse than
357 examples that don't actually work, because the code has changed since the
358 documentation has been written. To this end, Rust supports automatically
359 running examples in your documentation. Here's a fleshed-out `src/lib.rs`
363 //! The `adder` crate provides functions that add numbers to other numbers.
368 //! assert_eq!(4, adder::add_two(2));
371 /// This function adds two to its argument.
376 /// use adder::add_two;
378 /// assert_eq!(4, add_two(2));
380 pub fn add_two(a: i32) -> i32 {
390 assert_eq!(4, add_two(2));
395 Note the module-level documentation with `//!` and the function-level
396 documentation with `///`. Rust's documentation supports Markdown in comments,
397 and so triple graves mark code blocks. It is conventional to include the
398 `# Examples` section, exactly like that, with examples following.
400 Let's run the tests again:
404 Compiling adder v0.0.1 (file:///home/steve/tmp/adder)
405 Running target/adder-91b3e234d4ed382a
408 test test::it_works ... ok
410 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
412 Running target/lib-c18e7d3494509e74
417 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
422 test add_two_0 ... ok
425 test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured
428 Now we have all three kinds of tests running! Note the names of the
429 documentation tests: the `_0` is generated for the module test, and `add_two_0`
430 for the function test. These will auto increment with names like `add_two_1` as
431 you add more examples.
435 Rust also supports benchmark tests, which can test the performance of your
436 code. Let's make our `src/lib.rs` look like this (comments elided):
441 pub fn add_two(a: i32) -> i32 {
452 assert_eq!(4, add_two(2));
456 fn bench_add_two(b: &mut Bencher) {
457 b.iter(|| add_two(2));
462 We've imported the `test` crate, which contains our benchmarking support.
463 We have a new function as well, with the `bench` attribute. Unlike regular
464 tests, which take no arguments, benchmark tests take a `&mut Bencher`. This
465 `Bencher` provides an `iter` method, which takes a closure. This closure
466 contains the code we'd like to benchmark.
468 We can run benchmark tests with `cargo bench`:
472 Compiling adder v0.0.1 (file:///home/steve/tmp/adder)
473 Running target/release/adder-91b3e234d4ed382a
476 test tests::it_works ... ignored
477 test tests::bench_add_two ... bench: 1 ns/iter (+/- 0)
479 test result: ok. 0 passed; 0 failed; 1 ignored; 1 measured
482 Our non-benchmark test was ignored. You may have noticed that `cargo bench`
483 takes a bit longer than `cargo test`. This is because Rust runs our benchmark
484 a number of times, and then takes the average. Because we're doing so little
485 work in this example, we have a `1 ns/iter (+/- 0)`, but this would show
486 the variance if there was one.
488 Advice on writing benchmarks:
491 * Move setup code outside the `iter` loop; only put the part you want to measure inside
492 * Make the code do "the same thing" on each iteration; do not accumulate or change state
493 * Make the outer function idempotent too; the benchmark runner is likely to run
495 * Make the inner `iter` loop short and fast so benchmark runs are fast and the
496 calibrator can adjust the run-length at fine resolution
497 * Make the code in the `iter` loop do something simple, to assist in pinpointing
498 performance improvements (or regressions)
500 ## Gotcha: optimizations
502 There's another tricky part to writing benchmarks: benchmarks compiled with
503 optimizations activated can be dramatically changed by the optimizer so that
504 the benchmark is no longer benchmarking what one expects. For example, the
505 compiler might recognize that some calculation has no external effects and
513 fn bench_xor_1000_ints(b: &mut Bencher) {
515 (0..1000).fold(0, |old, new| old ^ new);
520 gives the following results
524 test bench_xor_1000_ints ... bench: 0 ns/iter (+/- 0)
526 test result: ok. 0 passed; 0 failed; 0 ignored; 1 measured
529 The benchmarking runner offers two ways to avoid this. Either, the closure that
530 the `iter` method receives can return an arbitrary value which forces the
531 optimizer to consider the result used and ensures it cannot remove the
532 computation entirely. This could be done for the example above by adjusting the
537 # impl X { fn iter<T, F>(&self, _: F) where F: FnMut() -> T {} } let b = X;
539 // note lack of `;` (could also use an explicit `return`).
540 (0..1000).fold(0, |old, new| old ^ new)
544 Or, the other option is to call the generic `test::black_box` function, which
545 is an opaque "black box" to the optimizer and so forces it to consider any
555 # impl X { fn iter<T, F>(&self, _: F) where F: FnMut() -> T {} } let b = X;
557 let n = test::black_box(1000);
559 (0..n).fold(0, |a, b| a ^ b)
564 Neither of these read or modify the value, and are very cheap for small values.
565 Larger values can be passed indirectly to reduce overhead (e.g.
566 `black_box(&huge_struct)`).
568 Performing either of the above changes gives the following benchmarking results
572 test bench_xor_1000_ints ... bench: 131 ns/iter (+/- 3)
574 test result: ok. 0 passed; 0 failed; 0 ignored; 1 measured
577 However, the optimizer can still modify a testcase in an undesirable manner
578 even when using either of the above.