The general approach here is to scan TLS, all locals, and the main memory map for all provenance, accumulating a `HashSet` of all pointer tags which are stored anywhere (we also have a special case for panic payloads). Then we iterate over every borrow stack and remove tags which are not in said `HashSet`, or which could be terminating a SRW block.
Runtime of benchmarks decreases by between 17% and 81%.
GC off:
```
Benchmark 1: cargo +miri miri run --manifest-path /home/ben/miri/bench-cargo-miri/backtraces/Cargo.toml
Time (mean ± σ): 7.080 s ± 0.249 s [User: 6.870 s, System: 0.202 s]
Range (min … max): 6.933 s … 7.521 s 5 runs
Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet PC without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.
Benchmark 1: cargo +miri miri run --manifest-path /home/ben/miri/bench-cargo-miri/mse/Cargo.toml
Time (mean ± σ): 1.875 s ± 0.031 s [User: 1.630 s, System: 0.245 s]
Range (min … max): 1.825 s … 1.910 s 5 runs
Benchmark 1: cargo +miri miri run --manifest-path /home/ben/miri/bench-cargo-miri/serde1/Cargo.toml
Time (mean ± σ): 2.785 s ± 0.075 s [User: 2.536 s, System: 0.168 s]
Range (min … max): 2.698 s … 2.851 s 5 runs
Benchmark 1: cargo +miri miri run --manifest-path /home/ben/miri/bench-cargo-miri/serde2/Cargo.toml
Time (mean ± σ): 6.267 s ± 0.066 s [User: 6.072 s, System: 0.190 s]
Range (min … max): 6.152 s … 6.314 s 5 runs
Benchmark 1: cargo +miri miri run --manifest-path /home/ben/miri/bench-cargo-miri/slice-get-unchecked/Cargo.toml
Time (mean ± σ): 4.733 s ± 0.080 s [User: 4.177 s, System: 0.513 s]
Range (min … max): 4.681 s … 4.874 s 5 runs
Benchmark 1: cargo +miri miri run --manifest-path /home/ben/miri/bench-cargo-miri/unicode/Cargo.toml
Time (mean ± σ): 3.770 s ± 0.034 s [User: 3.549 s, System: 0.211 s]
Range (min … max): 3.724 s … 3.819 s 5 runs
```
GC on:
```
Benchmark 1: cargo +miri miri run --manifest-path /home/ben/miri/bench-cargo-miri/backtraces/Cargo.toml
Time (mean ± σ): 5.886 s ± 0.054 s [User: 5.696 s, System: 0.182 s]
Range (min … max): 5.799 s … 5.937 s 5 runs
Benchmark 1: cargo +miri miri run --manifest-path /home/ben/miri/bench-cargo-miri/mse/Cargo.toml
Time (mean ± σ): 936.4 ms ± 7.0 ms [User: 815.4 ms, System: 119.6 ms]
Range (min … max): 925.7 ms … 945.0 ms 5 runs
Benchmark 1: cargo +miri miri run --manifest-path /home/ben/miri/bench-cargo-miri/serde1/Cargo.toml
Time (mean ± σ): 2.126 s ± 0.022 s [User: 1.979 s, System: 0.146 s]
Range (min … max): 2.089 s … 2.143 s 5 runs
Benchmark 1: cargo +miri miri run --manifest-path /home/ben/miri/bench-cargo-miri/serde2/Cargo.toml
Time (mean ± σ): 4.242 s ± 0.066 s [User: 4.051 s, System: 0.160 s]
Range (min … max): 4.196 s … 4.357 s 5 runs
Benchmark 1: cargo +miri miri run --manifest-path /home/ben/miri/bench-cargo-miri/slice-get-unchecked/Cargo.toml
Time (mean ± σ): 907.4 ms ± 2.4 ms [User: 788.6 ms, System: 118.2 ms]
Range (min … max): 903.5 ms … 909.4 ms 5 runs
Benchmark 1: cargo +miri miri run --manifest-path /home/ben/miri/bench-cargo-miri/unicode/Cargo.toml
Time (mean ± σ): 1.821 s ± 0.011 s [User: 1.687 s, System: 0.133 s]
Range (min … max): 1.802 s … 1.831 s 5 runs
```
But much more importantly for me this drops the peak memory usage of the first 1 minute of running `regex`'s tests from 103 GB to 1.7 GB.
Thanks to `@oli-obk` for suggesting a while ago that this was possible and `@darksonn` for reminding me that we can just search through memory to find Provenance to locate pointers.
disable extern-so ffi support for now due to licensing situation
libffi depends on abort_on_panic which has a [very unfortunate license situation going on](https://rust-lang.zulipchat.com/#narrow/stream/231349-t-core.2Flicensing). For now, in order to let us update Miri in rustc, I see no way but to disable our FFI support again. Sorry `@emarteca` :(
Auto merge of #2523 - saethlin:protector-test, r=RalfJung
Add a protector test that demonstrates the base tag diagnostic
Per https://github.com/rust-lang/miri/pull/2519#issuecomment-1232736295, this demonstrates this case for protector diagnostics:
```
help: <3131> was created here, as a base tag for alloc1623
--> tests/fail/stacked_borrows/invalidate_against_protector3.rs:10:19
|
10 | let ptr = std::alloc::alloc(std::alloc::Layout::for_value(&0i32)) as *mut i32;
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
```
This diagnostic is inspired by what Miri used to do with https://github.com/rust-lang/rust/issues/60076#issuecomment-1214169179
bors [Wed, 31 Aug 2022 10:06:05 +0000 (10:06 +0000)]
Auto merge of #2519 - saethlin:rustup, r=RalfJung
Use the better FnEntry spans in protector errors
Example error, from `tests/fail/stacked_borrows/invalidate_against_protector1.rs`:
```
error: Undefined Behavior: not granting access to tag <3095> because that would remove [Unique for <3099>] which is protected because it is an argument of call 943
--> tests/fail/stacked_borrows/invalidate_against_protector1.rs:5:25
|
5 | let _val = unsafe { *x }; //~ ERROR: protect
| ^^ not granting access to tag <3095> because that would remove [Unique for <3099>] which is protected because it is an argument of call 943
|
= help: this indicates a potential bug in the program: it performed an invalid operation, but the Stacked Borrows rules it violated are still experimental
= help: see https://github.com/rust-lang/unsafe-code-guidelines/blob/master/wip/stacked-borrows.md for further information
help: <3095> was created by a SharedReadWrite retag at offsets [0x0..0x4]
--> tests/fail/stacked_borrows/invalidate_against_protector1.rs:10:16
|
10 | let xraw = &mut x as *mut _;
| ^^^^^^
help: <3099> is this argument
--> tests/fail/stacked_borrows/invalidate_against_protector1.rs:1:23
|
1 | fn inner(x: *mut i32, _y: &mut i32) {
| ^^
= note: backtrace:
= note: inside `inner` at tests/fail/stacked_borrows/invalidate_against_protector1.rs:5:25
note: inside `main` at tests/fail/stacked_borrows/invalidate_against_protector1.rs:12:5
--> tests/fail/stacked_borrows/invalidate_against_protector1.rs:12:5
|
12 | inner(xraw, xref);
| ^^^^^^^^^^^^^^^^^
```
bors [Mon, 29 Aug 2022 11:46:26 +0000 (11:46 +0000)]
Auto merge of #2517 - saethlin:zst-field-retagging, r=RalfJung
Skip field retagging on ZSTs, it can take forever
I just tried running the `alloc`'s tests with `miri-test-libstd` with field retagging enabled. The test suite eventually hangs on a few tests which pass around ZSTs that have a lot of fields.
I don't really know how to test this effectively. The test passes, but if you remove this fast-path it effectively just hangs the interpreter. And since it hangs _inside_ a step, there's no hope for doing some kind of timeout within the test.
bors [Sun, 28 Aug 2022 16:01:49 +0000 (16:01 +0000)]
Auto merge of #2513 - RalfJung:protected, r=saethlin
slightly improve protector-related error messages
I find the current retag messages confusing, since they sound like the item *was* protected, when it still actively *is* protected (and that is, in fact, the issue).
Example error message:
```
error: Undefined Behavior: not granting access to tag <3095> because incompatible item [Unique for <3099>] is protected by call 943
--> tests/fail/stacked_borrows/invalidate_against_barrier1.rs:5:25
|
5 | let _val = unsafe { *x }; //~ ERROR: protect
| ^^ not granting access to tag <3095> because incompatible item [Unique for <3099>] is protected by call 943
|
= help: this indicates a potential bug in the program: it performed an invalid operation, but the Stacked Borrows rules it violated are still experimental
= help: see https://github.com/rust-lang/unsafe-code-guidelines/blob/master/wip/stacked-borrows.md for further information
help: <3095> was created by a SharedReadWrite retag at offsets [0x0..0x4]
--> tests/fail/stacked_borrows/invalidate_against_barrier1.rs:10:16
|
10 | let xraw = &mut x as *mut _;
| ^^^^^^
help: <3095> cannot be used for memory access because that would remove protected tag <3099>, protected by this function call
--> tests/fail/stacked_borrows/invalidate_against_barrier1.rs:1:1
|
1 | / fn inner(x: *mut i32, _y: &mut i32) {
2 | | // If `x` and `y` alias, retagging is fine with this... but we really
3 | | // shouldn't be allowed to use `x` at all because `y` was assumed to be
4 | | // unique for the duration of this call.
5 | | let _val = unsafe { *x }; //~ ERROR: protect
6 | | }
| |_^
help: <3099> was derived from <3098>, which in turn was created here
--> tests/fail/stacked_borrows/invalidate_against_barrier1.rs:12:17
|
12 | inner(xraw, xref);
| ^^^^
= note: backtrace:
= note: inside `inner` at tests/fail/stacked_borrows/invalidate_against_barrier1.rs:5:25
note: inside `main` at tests/fail/stacked_borrows/invalidate_against_barrier1.rs:12:5
--> tests/fail/stacked_borrows/invalidate_against_barrier1.rs:12:5
|
12 | inner(xraw, xref);
| ^^^^^^^^^^^^^^^^^
```
bors [Sun, 28 Aug 2022 13:20:54 +0000 (13:20 +0000)]
Auto merge of #2512 - cbeuw:scfix, r=RalfJung
Strengthen C++20 SC accesses
`@SabrinaJewson` noted in #2301 that Miri could produce behaviours forbidden under C++20 even without SC fences. Due to the added coherence-ordered before relationship which is created from read from and read before, plus the fact that coherence-ordered before between SC operations must be consistent with the Global Total Order S, in C++20 if there's an SC load that reads from any store, then a later SC load cannot read before that store. This PR adds this restriction
### Implementation
Adding support for calling external C functions that have any number of integer arguments (types of integers: `i8`, `i16`, `i32`, `i64`, `u8`, `u16`, `u32`, `u64`) and an integer return type (or `void`).
As suggested in `@https://github.com/rust-lang/miri/issues/11,` the [`libffi` crate](https://docs.rs/libffi/latest/libffi/index.html) is used to dispatch the calls to external C functions.
#### Modifications
Main modifications are to:
* [helper.rs](https://github.com/emarteca/miri/blob/int-function-args-returns/src/helpers.rs) : adding a function `call_and_add_external_c_fct_to_context` to read the code pointer to the external C function, dispatch the call, and save the return in MIRI internal memory. Handles all conversions between MIRI and C values (using some macros, also defined in this file).
* [foreign_items.rs](https://github.com/emarteca/miri/blob/int-function-args-returns/src/shims/foreign_items.rs) : handles the calling of `call_and_add_external_c_fct_to_context` in [helper.rs](https://github.com/emarteca/miri/blob/int-function-args-returns/src/helpers.rs) when a foreign item is encountered. Also adds some structs to model C representations of arguments, and the signature of the external C call.
### Testing
Adds tests for the following external functions which are now supported:
* [int tests](https://github.com/emarteca/miri/blob/int-function-args-returns/tests/pass/external_C/int_c_tests.rs):
- adds 2 to a provided int (no type of int specified, so autocasts)
- takes the sum of its 12 arguments (tests stack spill)
- adds 3 to a 16 bit int
- adds an `i16` to an `i64`
- returns -10 as an unsigned int
* [void tests](https://github.com/emarteca/miri/blob/int-function-args-returns/tests/pass/external_C/print_from_c.rs)
- void function that prints from C
### Code review
The code in this PR was reviewed by `@maurer` on [another fork](https://github.com/maurer/miri/pull/1) -- thanks!
bors [Thu, 25 Aug 2022 15:35:16 +0000 (15:35 +0000)]
Auto merge of #2449 - oli-obk:ui_test_subtree_sync, r=RalfJung
Use ui_test from crates.io instead of having it in-tree
I have moved a copy of the `ui_test` crate into [a separate repo](https://github.com/oli-obk/ui_test) to facilitate the further non-miri development of it. I will keep syncing until we have reached a point where we don't touch it anymore for miri. At that point we can remove the in-tree version and do further development out of tree.
One thing that I'm wondering is if we should do away with running each folder individually and make `ui_test` support running all tests in parallel (so building deps becomes a thread, and all tests needing deps are blocked on it)
bors [Mon, 22 Aug 2022 14:11:46 +0000 (14:11 +0000)]
Auto merge of #2441 - RalfJung:arithmetic, r=oli-obk
pass clippy::integer_arithmetic in our shims
`@oli-obk` [raised some concerns](https://github.com/rust-lang/miri/pull/2422#discussion_r928220546) about this one. I still think it is the right call, since I don't see a good way to enable overflow checks for our official release builds. I'm open to suggestions though!
bors [Sat, 20 Aug 2022 13:30:34 +0000 (13:30 +0000)]
Auto merge of #2495 - RalfJung:ra, r=RalfJung
add ./miri cargo for RA to invoke
RA expects a check command to check *individual workspaces*, whereas `./miri` is designed to check/build/test all parts of Miri. So add a new `./miri cargo` that performs just a single cargo invocation, but with the right env vars so that the build cache can be shared with `./miri check`.
bors [Thu, 18 Aug 2022 20:37:50 +0000 (20:37 +0000)]
Auto merge of #2454 - saethlin:diagnostics-cleanup, r=RalfJung
Improve information sharing across SB diagnostics
Previous Stacked Borrows diagnostics were missing a lot of information about the state of the interpreter, and it was difficult to add additional state because it was threaded through all the intervening function signatures.
This change factors a lot of the arguments which used to be passed individually to many stacked borrows functions into a single `DiagnosticCx`, which is built in `Stacks::for_each`, and since it wraps a handle to `AllocHistory`, we can now handle more nuanced things like heterogeneous borrow of `!Freeze` types.
Ben Kimock [Sun, 22 May 2022 23:39:09 +0000 (19:39 -0400)]
Improve information sharing across SB diagnostics
Previous Stacked Borrows diagnostics were missing a lot of information
about the state of the interpreter, and it was difficult to add
additional state because it was threaded through all the intervening
function signatures.
This change factors a lot of the arguments which used to be passed
individually to many stacked borrows functions into a single
`DiagnosticCx`, which is built in `Stacks::for_each`, and since it
wraps a handle to `AllocHistory`, we can now handle more nuanced
things like heterogeneous borrow of `!Freeze` types.
projects / rust.git / log