Alex Crichton [Thu, 20 Sep 2018 16:31:14 +0000 (09:31 -0700)]
std: Check for overflow in `str::repeat`
This commit fixes a buffer overflow issue in the standard library
discovered by Scott McMurray where if a large number was passed to
`str::repeat` it may cause and out of bounds write to the buffer of a `Vec`.
This bug was accidentally introduced in #48657 when optimizing the
`str::repeat` function. The bug affects stable Rust releases 1.26.0 to
1.29.0. We plan on backporting this fix to create a 1.29.1 release, and
the 1.30.0 release onwards will include this fix.
The fix in this commit is to introduce a deterministic panic in the case of
capacity overflow. When repeating a slice where the resulting length is larger
than the address space, there’s no way it can succeed anyway!
The standard library and surrounding libraries were briefly checked to see if
there were othere instances of preallocating a vector with a calculation that
may overflow. No instances of this bug (out of bounds write due to a calculation
overflow) were found at this time.
Note that this commit is the first steps towards fixing this issue,
we'll be making a formal post to the Rust security list once these
commits have been merged.
Auto merge of #54301 - alexcrichton:update-curl, r=Mark-Simulacrum
Update some `*-sys` dependencies of Cargo/RLS
This is intended to help solve #54206 on nightly where the RLS on MinGW is
having build issues with accidentally building a `curl` library which links to
pthread symbols on Windows (where it should use native mutex locking instead).
The build system for these `*-sys` crates have all been rewritten to be based on
`cc` to bypass native build systems and platform detection to make sure we
configure them correctly.
Alex Crichton [Mon, 17 Sep 2018 16:52:02 +0000 (09:52 -0700)]
Update some `*-sys` dependencies of Cargo/RLS
This is intended to help solve #54206 on nightly where the RLS on MinGW is
having build issues with accidentally building a `curl` library which links to
pthread symbols on Windows (where it should use native mutex locking instead).
The build system for these `*-sys` crates have all been rewritten to be based on
`cc` to bypass native build systems and platform detection to make sure we
configure them correctly.
Auto merge of #54211 - nnethercote:keccak-Liveness-memory, r=nikomatsakis
Split `Liveness::users` into three.
This reduces memory usage on some benchmarks because no space is wasted
for padding. For a `check-clean` build of `keccak` it reduces `max-rss`
by 20%.
r? @nikomatsakis, but I want to do a perf run. Locally, I had these results:
- instructions: slight regression
- max-rss: big win on "Clean" builds
- faults: big win on "Clean" and "Nll" builds
- wall-time: small win on "Clean" and "Nll" builds
So I want to see how a different machine compares.
Auto merge of #54174 - parched:park, r=alexcrichton
Fix `thread` `park`/`unpark` synchronization
Previously the code below would not be guaranteed to exit when the
second unpark took the `return, // already unparked` path because there
was no write to synchronize with a read in `park`.
EDIT: doesn't actually require third thread
```
use std::sync::atomic::{AtomicBool, Ordering};
use std::thread::{current, spawn, park};
while !FLAG.load(Ordering::Relaxed) {
park();
}
}
```
I have some other ideas on how to improve the performance of `park` and `unpark` using fences, avoiding any atomic RMW when the state is already `NOTIFIED`, and also how to avoid calling `notify_one` without the mutex locked. But I need to write some micro benchmarks first, so I'll submit those changes at a later date if they prove to be faster.
Fixes https://github.com/rust-lang/rust/issues/53366 I hope.
Auto merge of #54318 - nnethercote:use-HybridBitSet-in-SparseBitMatrix, r=pnkfelix
Use `HybridBitSet` in `SparseBitMatrix`.
This fixes most of the remaining NLL memory regression.
r? @pnkfelix, because you reviewed #54286.
cc @nikomatsakis, because NLL
cc @Mark-Simulacrum, because this removes `array_vec.rs`
cc @lqd, because this massively improves `unic-ucd-name`, and probably other public crates
David Wood [Thu, 6 Sep 2018 13:48:33 +0000 (15:48 +0200)]
De-duplicate moved variable errors.
By introducing a new map that tracks the errors reported and the
`Place`s that spawned those errors against the move out that the error
was referring to, we are able to silence duplicate errors by emitting
only the error which corresponds to the most specific `Place` (that which
other `Place`s which reported errors are prefixes of).
This generally is an improvement, however there is a case -
`liveness-move-in-while` - where the output regresses.
Auto merge of #54319 - GuillaumeGomez:rollup, r=GuillaumeGomez
Rollup of 9 pull requests
Successful merges:
- #53522 (Add doc for impl From for Addr)
- #54097 (rustdoc: Remove namespace for keywords)
- #54205 (Add treat-err-as-bug flag in rustdoc)
- #54225 (Regression test for rust-lang/rust#53675.)
- #54232 (add `-Z dont-buffer-diagnostics`)
- #54273 (Suggest to change numeric literal instead of casting)
- #54299 (Issue 54246)
- #54311 (Remove README with now-out-of-date docs about docs.)
- #54313 (OsStr: Document that it's not NUL terminated)
Rollup merge of #54313 - cgwalters:osstr-ref-cstr, r=joshtriplett
OsStr: Document that it's not NUL terminated
I somehow got confused into thinking this was the case, but
it's definitely not. Let's help the common case of people who
have an `OsStr` and need to call e.g. Unix APIs.
Rollup merge of #54299 - snaedis:issue-54246, r=varkor
Issue 54246
I added the option of providing a help message for deprecated features, that takes precedence over the default `help: remove this attribute` message, along with messages for the features that mention replacements in the reason for deprecation.
Rollup merge of #54232 - pnkfelix:add-way-to-disable-diagnostic-buffering, r=nikomatsakis
add `-Z dont-buffer-diagnostics`
Add `-Z dont-buffer-diagnostics`, a way to force NLL to immediately its diagnostics.
This is mainly intended for developers who want to see the error in its original context in the control flow. Two uses cases for that are:
1. `-Z treat-err-as-bug` (which then allows extraction of a stack-trace to the origin of the error)
2. RUST_LOG=... rustc, in which case it is often useful to see the logging statements that occurred immediately prior to the point where the diagnostic was signalled.
Rollup merge of #54225 - pnkfelix:issue-53675-add-test-called-panic, r=petrochenkov
Regression test for rust-lang/rust#53675.
(Includes a couple variations on the theme. I confirmed that the ones
in `in_expression_position` and `what_if_we_use_panic_directly_in_expr`
both failed back on "rustc 1.30.0-nightly (0f063aef6 2018-09-03)".)
Use `HybridBitSet` for rows within `SparseBitMatrix`.
This requires adding a few extra methods to `HybridBitSet`. (These are
tested in a new unit test.)
This commit reduces the `max-rss` for `nll-check` builds of `html5ever`
by 46%, `ucd` by 45%, `clap-rs` by 23%, `inflate` by 14%. And the
results for the `unic-ucd-name` crate are even more impressive: a 21%
reduction in instructions, a 60% reduction in wall-time, a 96%
reduction in `max-rss`, and a 97% reduction in faults!
`SparseBitSet` is the only remaining user of `ArrayVec`. This commit
switches it to using `SmallVec`, and removes `array_vec.rs`.
Why the switch? Although `SparseBitSet` is size-limited and doesn't need
the ability to spill to the heap, `SmallVec` has many more features than
`ArrayVec`. In particular, it's now possible to keep `SparseBitSet`'s
elements in sorted order, which gives in-order iteration, which is a
requirement for the next commit.
Auto merge of #54286 - nnethercote:BitSet, r=pnkfelix
Merge `bitvec.rs` and `indexed_set.rs`
Because it's not good to have two separate implementations. Also, I will combine the best parts of each to improve NLL memory usage on some benchmarks significantly.
Colin Walters [Tue, 18 Sep 2018 01:10:36 +0000 (21:10 -0400)]
OsStr: Document that it's not NUL terminated
I somehow got confused into thinking this was the case, but
it's definitely not. Let's help the common case of people who
have an `OsStr` and need to call e.g. Unix APIs.
`BitwiseOperator` is an unnecessarily low-level thing. This commit
replaces it with `BitSetOperator`, which works on `BitSet`s instead of
words. Within `bit_set.rs`, the commit eliminates `Intersect`, `Union`,
and `Subtract` by instead passing a function to `bitwise()`.
Merge indexed_set.rs into bitvec.rs, and rename it bit_set.rs.
Currently we have two files implementing bitsets (and 2D bit matrices).
This commit combines them into one, taking the best features from each.
This involves renaming a lot of things. The high level changes are as
follows.
- bitvec.rs --> bit_set.rs
- indexed_set.rs --> (removed)
- BitArray + IdxSet --> BitSet (merged, see below)
- BitVector --> GrowableBitSet
- {,Sparse,Hybrid}IdxSet --> {,Sparse,Hybrid}BitSet
- BitMatrix --> BitMatrix
- SparseBitMatrix --> SparseBitMatrix
The changes within the bitset types themselves are as follows.
```
OLD OLD NEW
BitArray<C> IdxSet<T> BitSet<T>
-------- ------ ------
grow - grow
new - (remove)
new_empty new_empty new_empty
new_filled new_filled new_filled
- to_hybrid to_hybrid
clear clear clear
set_up_to set_up_to set_up_to
clear_above - clear_above
count - count
contains(T) contains(&T) contains(T)
contains_all - superset
is_empty - is_empty
insert(T) add(&T) insert(T)
insert_all - insert_all()
remove(T) remove(&T) remove(T)
words words words
words_mut words_mut words_mut
- overwrite overwrite
merge union union
- subtract subtract
- intersect intersect
iter iter iter
```
In general, when choosing names I went with:
- names that are more obvious (e.g. `BitSet` over `IdxSet`).
- names that are more like the Rust libraries (e.g. `T` over `C`,
`insert` over `add`);
- names that are more set-like (e.g. `union` over `merge`, `superset`
over `contains_all`, `domain_size` over `num_bits`).
Also, using `T` for index arguments seems more sensible than `&T` --
even though the latter is standard in Rust collection types -- because
indices are always copyable. It also results in fewer `&` and `*`
sigils in practice.
Auto merge of #54277 - petrochenkov:afterder, r=alexcrichton
Temporarily prohibit proc macro attributes placed after derives
... and also proc macro attributes used together with `#[test]`/`#[bench]`.
Addresses item 6 from https://github.com/rust-lang/rust/pull/50911#issuecomment-411605393.
The end goal is straightforward predictable left-to-right expansion order for attributes.
Right now derives are expanded last regardless of their relative ordering with macro attributes and right now it's simpler to temporarily prohibit macro attributes placed after derives than changing the expansion order.
I'm not sure whether the new beta is already released or not, but if it's released, then this patch needs to be backported, so the solution needs to be minimal.
How to fix broken code (derives):
- Move macro attributes above derives. This won't change expansion order, they are expanded before derives anyway.
Using attribute macros on same items with `#[test]` and `#[bench]` is prohibited for similar expansion order reasons, but this one is going to be reverted much sooner than restrictions on derives.
How to fix broken code (test/bench):
- Enable `#![feature(plugin)]` (don't ask why).
Add -Z dont-buffer-diagnostics, a way to force NLL to immediately emit its diagnostics.
This is mainly intended for `rustc` developers who want to see a
diagnostic in its original context in the control flow. Two uses
cases for that are:
* `-Z treat-err-as-bug` which then allows extraction of a stack-trace to the origin of the error
(a case that is so important that we make that flag imply this one, effectively).
* `RUST_LOG=... rustc`, in which case it is often useful to see the logging statements that
occurred immediately prior to the point where the diagnostic was signalled.
Drive-by: Added some documentation pointing future devs at
HandlerFlags, and documented the fields of `HandlerFlags` itself.
Auto merge of #54260 - maxdeviant:public-scope-fields, r=petrochenkov
Make rustc::middle::region::Scope's fields public
This PR makes the following changes to `rustc::middle::region::Scope`:
- [x] Makes `region::Scope`'s fields public
- [x] Removes the `impl Scope` block with constructors (as per [this comment](https://github.com/rust-lang/rust/pull/54032#discussion_r216618208))
- [x] Updates call sites throughout the compiler
Auto merge of #54254 - RalfJung:miri-dangling, r=eddyb
miri engine: keep around some information for dead allocations
We use it to test if a dangling ptr is aligned and non-NULL. This makes some code pass that should pass (writing a ZST to a properly aligned dangling pointer), and makes some code fail that should fail (writing a ZST to a pointer obtained via pointer arithmetic from a real location, but ouf-of-bounds -- that pointer could be NULL, so we cannot allow writing to it).
CTFE does not allow these operations; tests are added to miri with https://github.com/solson/miri/pull/453.
Auto merge of #54247 - Munksgaard:better-error-message-in-no_lookup_host_duplicates, r=alexcrichton
Improve output if no_lookup_host_duplicates test fails
If the test fails, output the offending addresses and a helpful error message.
Also slightly improve legibility of the preceding line that puts the addresses
into a HashMap.
Auto merge of #53461 - petrochenkov:pmu, r=alexcrichton
resolve: Do not error on access to proc macros imported with `#[macro_use]`
This error is artificial, but previously, when `#[macro_use] extern crate x;` was stable, but non-derive proc macros were not, it worked like kind of a feature gate. Now both features are stable, so the error is no longer necessary.
This PR simplifies how `#[macro_use] extern crate x;` works - it takes all items from macro namespace of `x`'s root and puts them into macro prelude from which they all can now be accessed.
Auto merge of #53804 - RalfJung:ptr-invalid, r=nagisa
fix some uses of pointer intrinsics with invalid pointers
[Found by miri](https://github.com/solson/miri/pull/446):
* `Vec::into_iter` calls `ptr::read` (and the underlying `copy_nonoverlapping`) with an unaligned pointer to a ZST. [According to LLVM devs](https://bugs.llvm.org/show_bug.cgi?id=38583), this is UB because it contradicts the metadata we are attaching to that pointer.
* `HashMap` creation calls `ptr:.write_bytes` on a NULL pointer with a count of 0. This is likely not currently UB *currently*, but it violates the rules we are setting in https://github.com/rust-lang/rust/pull/53783, and we might want to exploit those rules later (e.g. with more `nonnull` attributes for LLVM).
Probably what `HashMap` really should do is use `NonNull::dangling()` instead of 0 for the empty case, but that would require a more careful analysis of the code.
It seems like ideally, we should do a review of usage of such intrinsics all over libstd to ensure that they use valid pointers even when the size is 0. Is it worth opening an issue for that?
Auto merge of #54157 - euclio:structured-suggestion, r=estebank
use structured suggestion for "missing mut" label
Fixes #54133 for both NLL and non-NLL.
r? @estebank
I'm not super happy with the existing wording here, since it's now a suggestion. I wonder if the message would work better as something like "help: make binding mutable: `mut foo`"?
Also, are the `HELP` and `SUGGESTION` comments necessary?
Auto merge of #53754 - RalfJung:slice_align_to, r=alexcrichton
stabilize slice_align_to
This is very hard to implement correctly, and leads to [serious bugs](https://github.com/llogiq/bytecount/pull/42) when done incorrectly. Moreover, this is needed to be able to run code that opportunistically exploits alignment on miri. So code using `align_to`/`align_to_mut` gets the benefit of a well-tested implementation *and* of being able to run in miri to test for (some kinds of) UB.
This PR also clarifies the guarantee wrt. the middle part being as long as possible. Should the docs say under which circumstances the middle part could be shorter? Currently, that can only happen when running in miri.
Auto merge of #54116 - eddyb:extern-prelude, r=petrochenkov
rustc_resolve: allow only core, std, meta and --extern in Rust 2018 paths.
As per https://github.com/rust-lang/rust/issues/53166#issuecomment-419265401:
* Rust 2018 imports can no longer refer to crates not in "extern prelude"
* `::foo` won't load a crate named `foo` unless `foo` is in the "extern prelude"
* `extern crate foo;`, however, remains unchanged (can load arbitrary crates)
* `--extern crate_name` is added (note the lack of `=path`) as an unstable option
* adds `crate_name` to the "extern prelude" (see above)
* crate is searched in sysroot & library paths, just like `extern crate crate_name`.
* `Cargo` support will be added later
* `core`, `std` and ~~`proc_macro`~~ `meta` are *always* available in the extern prelude
* warning for interaction with `no_std` / `no_core` will be added later
* **EDIT**: `proc_macro` was replaced by `meta`, see https://github.com/rust-lang/rust/issues/53166#issuecomment-421137230
* note that there is no crate named `meta` being added, so `use meta::...;` will fail, we're only whitelisting it so we can start producing `uniform_paths` compatibility errors
Fixes #54006 (as the example now requires `--extern alloc`, which is unstable).
Fixes #54253 (hit during fixing RLS).
r? @petrochenkov cc @aturon @alexcrichton @Centril @joshtriplett