3 > Program testing can be a very effective way to show the presence of bugs, but
4 > it is hopelessly inadequate for showing their absence.
6 > Edsger W. Dijkstra, "The Humble Programmer" (1972)
8 Let's talk about how to test Rust code. What we will not be talking about is
9 the right way to test Rust code. There are many schools of thought regarding
10 the right and wrong way to write tests. All of these approaches use the same
11 basic tools, and so we'll show you the syntax for using them.
13 # The `test` attribute
15 At its simplest, a test in Rust is a function that's annotated with the `test`
16 attribute. Let's make a new project with Cargo called `adder`:
23 Cargo will automatically generate a simple test when you make a new project.
24 Here's the contents of `src/lib.rs`:
27 # // The next line exists to trick play.rust-lang.org into running our code as a
39 For now, let's remove the `mod` bit, and focus on just the function:
42 # // The next line exists to trick play.rust-lang.org into running our code as a
51 Note the `#[test]`. This attribute indicates that this is a test function. It
52 currently has no body. That's good enough to pass! We can run the tests with
57 Compiling adder v0.1.0 (file:///home/you/projects/adder)
58 Finished debug [unoptimized + debuginfo] target(s) in 0.15 secs
59 Running target/debug/deps/adder-941f01916ca4a642
64 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
70 test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured
73 Cargo compiled and ran our tests. There are two sets of output here: one
74 for the test we wrote, and another for documentation tests. We'll talk about
75 those later. For now, see this line:
81 Note the `it_works`. This comes from the name of our function:
90 We also get a summary line:
93 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
96 So why does our do-nothing test pass? Any test which doesn't `panic!` passes,
97 and any test that does `panic!` fails. Let's make our test fail:
100 # // The next line exists to trick play.rust-lang.org into running our code as a
110 `assert!` is a macro provided by Rust which takes one argument: if the argument
111 is `true`, nothing happens. If the argument is `false`, it will `panic!`. Let's
116 Compiling adder v0.1.0 (file:///home/you/projects/adder)
117 Finished debug [unoptimized + debuginfo] target(s) in 0.17 secs
118 Running target/debug/deps/adder-941f01916ca4a642
121 test it_works ... FAILED
125 ---- it_works stdout ----
126 thread 'it_works' panicked at 'assertion failed: false', src/lib.rs:5
127 note: Run with `RUST_BACKTRACE=1` for a backtrace.
133 test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured
138 Rust indicates that our test failed:
141 test it_works ... FAILED
144 And that's reflected in the summary line:
147 test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured
150 We also get a non-zero status code. We can use `$?` on OS X and Linux:
157 On Windows, if you’re using `cmd`:
163 And if you’re using PowerShell:
166 > echo $LASTEXITCODE # the code itself
167 > echo $? # a boolean, fail or succeed
170 This is useful if you want to integrate `cargo test` into other tooling.
172 We can invert our test's failure with another attribute: `should_panic`:
175 # // The next line exists to trick play.rust-lang.org into running our code as a
186 This test will now succeed if we `panic!` and fail if we complete. Let's try it:
190 Compiling adder v0.1.0 (file:///home/you/projects/adder)
191 Finished debug [unoptimized + debuginfo] target(s) in 0.17 secs
192 Running target/debug/deps/adder-941f01916ca4a642
197 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
203 test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured
206 Rust provides another macro, `assert_eq!`, that compares two arguments for
210 # // The next line exists to trick play.rust-lang.org into running our code as a
217 assert_eq!("Hello", "world");
221 Does this test pass or fail? Because of the `should_panic` attribute, it
226 Compiling adder v0.1.0 (file:///home/you/projects/adder)
227 Finished debug [unoptimized + debuginfo] target(s) in 0.21 secs
228 Running target/debug/deps/adder-941f01916ca4a642
233 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
239 test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured
242 `should_panic` tests can be fragile, as it's hard to guarantee that the test
243 didn't fail for an unexpected reason. To help with this, an optional `expected`
244 parameter can be added to the `should_panic` attribute. The test harness will
245 make sure that the failure message contains the provided text. A safer version
246 of the example above would be:
249 # // The next line exists to trick play.rust-lang.org into running our code as a
254 #[should_panic(expected = "assertion failed")]
256 assert_eq!("Hello", "world");
260 That's all there is to the basics! Let's write one 'real' test:
263 # // The next line exists to trick play.rust-lang.org into running our code as a
267 pub fn add_two(a: i32) -> i32 {
273 assert_eq!(4, add_two(2));
277 This is a very common use of `assert_eq!`: call some function with
278 some known arguments and compare it to the expected output.
280 # The `ignore` attribute
282 Sometimes a few specific tests can be very time-consuming to execute. These
283 can be disabled by default by using the `ignore` attribute:
286 # // The next line exists to trick play.rust-lang.org into running our code as a
290 pub fn add_two(a: i32) -> i32 {
296 assert_eq!(4, add_two(2));
301 fn expensive_test() {
302 // Code that takes an hour to run...
306 Now we run our tests and see that `it_works` is run, but `expensive_test` is
311 Compiling adder v0.1.0 (file:///home/you/projects/adder)
312 Finished debug [unoptimized + debuginfo] target(s) in 0.20 secs
313 Running target/debug/deps/adder-941f01916ca4a642
316 test expensive_test ... ignored
319 test result: ok. 1 passed; 0 failed; 1 ignored; 0 measured
325 test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured
328 The expensive tests can be run explicitly using `cargo test -- --ignored`:
331 $ cargo test -- --ignored
332 Finished debug [unoptimized + debuginfo] target(s) in 0.0 secs
333 Running target/debug/deps/adder-941f01916ca4a642
336 test expensive_test ... ok
338 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
344 test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured
347 The `--ignored` argument is an argument to the test binary, and not to Cargo,
348 which is why the command is `cargo test -- --ignored`.
352 There is one way in which our existing example is not idiomatic: it's
353 missing the `tests` module. You might have noticed this test module was
354 present in the code that was initially generated with `cargo new` but
355 was missing from our last example. Let's explain what this does.
357 The idiomatic way of writing our example looks like this:
360 # // The next line exists to trick play.rust-lang.org into running our code as a
364 pub fn add_two(a: i32) -> i32 {
374 assert_eq!(4, add_two(2));
379 There's a few changes here. The first is the introduction of a `mod tests` with
380 a `cfg` attribute. The module allows us to group all of our tests together, and
381 to also define helper functions if needed, that don't become a part of the rest
382 of our crate. The `cfg` attribute only compiles our test code if we're
383 currently trying to run the tests. This can save compile time, and also ensures
384 that our tests are entirely left out of a normal build.
386 The second change is the `use` declaration. Because we're in an inner module,
387 we need to bring our test function into scope. This can be annoying if you have
388 a large module, and so this is a common use of globs. Let's change our
389 `src/lib.rs` to make use of it:
392 # // The next line exists to trick play.rust-lang.org into running our code as a
396 pub fn add_two(a: i32) -> i32 {
406 assert_eq!(4, add_two(2));
411 Note the different `use` line. Now we run our tests:
415 Updating registry `https://github.com/rust-lang/crates.io-index`
416 Compiling adder v0.1.0 (file:///home/you/projects/adder)
417 Running target/debug/deps/adder-91b3e234d4ed382a
420 test tests::it_works ... ok
422 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
428 test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured
433 The current convention is to use the `tests` module to hold your "unit-style"
434 tests. Anything that tests one small bit of functionality makes sense to
435 go here. But what about "integration-style" tests instead? For that, we have
436 the `tests` directory.
438 # The `tests` directory
440 Each file in `tests/*.rs` directory is treated as an individual crate.
441 To write an integration test, let's make a `tests` directory and
442 put a `tests/integration_test.rs` file inside with this as its contents:
445 # // The next line exists to trick play.rust-lang.org into running our code as a
449 # // Sadly, this code will not work in play.rust-lang.org, because we have no
450 # // crate adder to import. You'll need to try this part on your own machine.
455 assert_eq!(4, adder::add_two(2));
459 This looks similar to our previous tests, but slightly different. We now have
460 an `extern crate adder` at the top. This is because each test in the `tests`
461 directory is an entirely separate crate, and so we need to import our library.
462 This is also why `tests` is a suitable place to write integration-style tests:
463 they use the library like any other consumer of it would.
469 Compiling adder v0.1.0 (file:///home/you/projects/adder)
470 Running target/debug/deps/adder-91b3e234d4ed382a
473 test tests::it_works ... ok
475 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
477 Running target/debug/integration_test-68064b69521c828a
482 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
488 test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured
491 Now we have three sections: our previous test is also run, as well as our new
494 Cargo will ignore files in subdirectories of the `tests/` directory.
495 Therefore shared modules in integrations tests are possible.
496 For example `tests/common/mod.rs` is not separately compiled by cargo but can
497 be imported in every test with `mod common;`
499 That's all there is to the `tests` directory. The `tests` module isn't needed
500 here, since the whole thing is focused on tests.
502 Let's finally check out that third section: documentation tests.
504 # Documentation tests
506 Nothing is better than documentation with examples. Nothing is worse than
507 examples that don't actually work, because the code has changed since the
508 documentation has been written. To this end, Rust supports automatically
509 running examples in your documentation (**note:** this only works in library
510 crates, not binary crates). Here's a fleshed-out `src/lib.rs` with examples:
513 # // The next line exists to trick play.rust-lang.org into running our code as a
517 //! The `adder` crate provides functions that add numbers to other numbers.
522 //! assert_eq!(4, adder::add_two(2));
525 /// This function adds two to its argument.
530 /// use adder::add_two;
532 /// assert_eq!(4, add_two(2));
534 pub fn add_two(a: i32) -> i32 {
544 assert_eq!(4, add_two(2));
549 Note the module-level documentation with `//!` and the function-level
550 documentation with `///`. Rust's documentation supports Markdown in comments,
551 and so triple graves mark code blocks. It is conventional to include the
552 `# Examples` section, exactly like that, with examples following.
554 Let's run the tests again:
558 Compiling adder v0.1.0. (file:///home/you/projects/adder)
559 Running target/debug/deps/adder-91b3e234d4ed382a
562 test tests::it_works ... ok
564 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
566 Running target/debug/integration_test-68064b69521c828a
571 test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured
576 test add_two_0 ... ok
579 test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured
582 Now we have all three kinds of tests running! Note the names of the
583 documentation tests: the `_0` is generated for the module test, and `add_two_0`
584 for the function test. These will auto increment with names like `add_two_1` as
585 you add more examples.
587 We haven’t covered all of the details with writing documentation tests. For more,
588 please see the [Documentation chapter](documentation.html).
590 # Testing and concurrency
592 It is important to note that tests are run concurrently using threads. For this
593 reason, care should be taken to ensure your tests do not depend on each-other,
594 or on any shared state. "Shared state" can also include the environment, such
595 as the current working directory, or environment variables.
597 If this is an issue it is possible to control this concurrency, either by
598 setting the environment variable `RUST_TEST_THREADS`, or by passing the argument
599 `--test-threads` to the tests:
602 $ RUST_TEST_THREADS=1 cargo test # Run tests with no concurrency
604 $ cargo test -- --test-threads=1 # Same as above
610 By default Rust's test library captures and discards output to standard
611 out/error, e.g. output from `println!()`. This too can be controlled using the
612 environment or a switch:
616 $ RUST_TEST_NOCAPTURE=1 cargo test # Preserve stdout/stderr
618 $ cargo test -- --nocapture # Same as above
622 However a better method avoiding capture is to use logging rather than raw
623 output. Rust has a [standard logging API][log], which provides a frontend to
624 multiple logging implementations. This can be used in conjunction with the
625 default [env_logger] to output any debugging information in a manner that can be
626 controlled at runtime.
628 [log]: https://crates.io/crates/log
629 [env_logger]: https://crates.io/crates/env_logger