Auto merge of #44620 - zackmdavis:rfc_1940_housekeeping, r=nikomatsakis
RFC 1940 housekeeping
* move test to own directory, as requested in https://github.com/rust-lang/rust/issues/43302#issuecomment-329579185
* exercise trait methods in test
* unstable book section
Auto merge of #44601 - alexcrichton:lower-attributes-in-hir, r=nrc
rustc: Forbid interpolated tokens in the HIR
Right now the HIR contains raw `syntax::ast::Attribute` structure but nowadays
these can contain arbitrary tokens. One variant of the `Token` enum is an
"interpolated" token which basically means to shove all the tokens for a
nonterminal in this position. A "nonterminal" in this case is roughly analagous
to a macro argument:
macro_rules! foo {
($a:expr) => {
// $a is a nonterminal as an expression
}
}
Currently nonterminals contain namely items and expressions, and this poses a
problem for incremental compilation! With incremental we want a stable hash of
all HIR items, but this means we may transitively need a stable hash *of the
entire AST*, which is certainly not stable w/ node ids and whatnot. Hence today
there's a "bug" where the "stable hash" of an AST is just the raw hash value of
the AST, and this only arises with interpolated nonterminals. The downside of
this approach, however, is that a bunch of errors get spewed out during
compilation about how this isn't a great idea.
This PR is focused at fixing these warnings, basically deleting them from the
compiler. The implementation here is to alter attributes as they're lowered from
the AST to HIR, expanding all nonterminals in-place as we see them. This code
for expanding a nonterminal to a token stream already exists for the
`proc_macro` crate, so we basically just reuse the same implementation there.
After this PR it's considered a bug to have an `Interpolated` token and hence
the stable hash implementation simply uses `bug!` in this location.
Auto merge of #44026 - QuietMisdreavus:trimmed-std, r=steveklabnik
hide internal types/traits from std docs via new #[doc(masked)] attribute
Fixes #43701 (hopefully for good this time)
This PR introduces a new parameter to the `#[doc]` attribute that rustdoc looks for on `extern crate` statements. When it sees `#[doc(masked)]` on such a statement, it hides traits and types from that crate from appearing in either the "Trait Implementations" section of many type pages, or the "Implementors" section of trait pages. This is then applied to the `libc`/`rand`/`compiler_builtins` imports in libstd to prevent those crates from creating broken links in the std docs.
Like in #43348, this also introduces a feature gate, `doc_masked`, that controls the use of this parameter.
To view the std docs generated with this change, head to https://tonberry.quietmisdreavus.net/std-43701/std/index.html.
In <https://github.com/rust-lang/rust/pull/42998>, we added an
uninstantiable type for the internal `UNICODE_VERSION` value,
`UnicodeVersion`, but it was not made public to the outside of the
crate, resulting in the value becoming less useful. Here we make the
type accessible from the outside.
Also add a run-pass test to make sure the type and value can be accessed
as intended.
Alex Crichton [Fri, 15 Sep 2017 15:28:34 +0000 (08:28 -0700)]
rustc: Forbid interpolated tokens in the HIR
Right now the HIR contains raw `syntax::ast::Attribute` structure but nowadays
these can contain arbitrary tokens. One variant of the `Token` enum is an
"interpolated" token which basically means to shove all the tokens for a
nonterminal in this position. A "nonterminal" in this case is roughly analagous
to a macro argument:
macro_rules! foo {
($a:expr) => {
// $a is a nonterminal as an expression
}
}
Currently nonterminals contain namely items and expressions, and this poses a
problem for incremental compilation! With incremental we want a stable hash of
all HIR items, but this means we may transitively need a stable hash *of the
entire AST*, which is certainly not stable w/ node ids and whatnot. Hence today
there's a "bug" where the "stable hash" of an AST is just the raw hash value of
the AST, and this only arises with interpolated nonterminals. The downside of
this approach, however, is that a bunch of errors get spewed out during
compilation about how this isn't a great idea.
This PR is focused at fixing these warnings, basically deleting them from the
compiler. The implementation here is to alter attributes as they're lowered from
the AST to HIR, expanding all nonterminals in-place as we see them. This code
for expanding a nonterminal to a token stream already exists for the
`proc_macro` crate, so we basically just reuse the same implementation there.
After this PR it's considered a bug to have an `Interpolated` token and hence
the stable hash implementation simply uses `bug!` in this location.
Auto merge of #44680 - infinity0:master, r=Mark-Simulacrum
rustbuild: with --no-fail-fast, report the specific commands that failed
I'm not sure this is the most elegant way of doing it, I'm still a bit of a rust noob. I tried `Vec<Command>` and keeping `Cell` instead of `RefCell` but couldn't fight my way past the borrow errors, this was the first arrangement that I could make work.
Fold allows iterators to implement a different (non-resumable) internal
iteration when it is more efficient than the external iteration
implemented through the next method. (Common examples are VecDeque and
.chain()).
Introduce rfold() so that the same customization is available for
reverse iteration. This is achieved by both adding the method, and by
having the Rev<I> adaptor connect Rev::rfold -> I::fold, Rev::fold -> I::rfold.
Adaptors are things that take iterators and adapt them into other
iterators. With this definition, fold is just a usual method, because it
doesn't normally make an iterator.
Alex Crichton [Mon, 18 Sep 2017 16:04:25 +0000 (11:04 -0500)]
Rollup merge of #44657 - Ixrec:patch-1, r=eddyb
Replace str's transmute() calls with pointer casts
After the following conversation in #rust-lang:
```
[14:43:50] <Ixrec> TIL the implementation of from_utf_unchecked is literally just "mem::transmute(x)"
[14:43:59] <Ixrec> no wonder people keep saying transmute is overpowered
[15:15:30] <eddyb> Ixrec: it should be a pointer cast lol
[15:15:46] <eddyb> unless it doesn't let you
[16:50:34] <Ixrec> https://play.rust-lang.org/?gist=d1e6b629ad9ec1baf64ce261c63845e6&version=stable seems like it does let me
[16:52:35] <eddyb> Ixrec: yeah that's the preferred impl
[16:52:46] <eddyb> Ixrec: it just wasn't in 1.0
[16:52:50] <eddyb> IIRC
[16:53:00] <eddyb> (something something fat pointers)
```
Since I already wrote half of the preferred impls in the playground, might as well make an actual PR.
Alex Crichton [Mon, 18 Sep 2017 16:04:19 +0000 (11:04 -0500)]
Rollup merge of #44364 - michaelwoerister:hash-all-the-things2, r=nikomatsakis
incr.comp.: Compute fingerprint for all query results.
This PR enables query result fingerprinting in incremental mode. This is an essential piece of infrastructure for red/green tracking. We don't do anything with the fingerprints yet but merging the infrastructure should protect it from bit-rotting and will make it easier to start measuring its performance impact (and thus let us determine if we should switch to a faster hashing algorithm rather sooner than later).
Note, this PR also includes the changes from https://github.com/rust-lang/rust/pull/43887 which I'm therefore closing. No need to re-review the first commit though.
- Don't hash traits in scope as part of HIR hashing any more.
- Some queries returned DefIndexes from other crates.
- Provide a generic way of stably hashing maps (not used everywhere yet).
incr.comp.: Move result fingerprinting to DepGraph::with_task().
This makes sure that we don't introduce strange cases where we have
nodes outside the query system that could break red/green tracking
and it will allow to keep red/green neatly encapsulated within the
DepGraph implementation.
Auto merge of #43628 - oli-obk:orbital_standard_library, r=alexcrichton
Run the miri test suite on the aux builder and travis
Reopen of #38350
see https://github.com/rust-lang/rust/pull/43340#issuecomment-316940762 for earlier discussion
Rationale for running miri's test suite in rustc's CI is that miri currently contains many features that we want in const eval in the future, and these features would break if the test suite is not run.
Auto merge of #44529 - alexcrichton:trans-query, r=michaelwoerister
Refactor translation unit partitioning/collection as a query
This commit is targeted at #44486 with the ultimate goal of making the `collect_and_partition_translation_items` function a query. This mostly just involved query-ifying a few other systems along with plumbing the tcx instead of `SharedCrateContext` in a few locations.
Currently this only tackles the first bullet of #44486 and doesn't add a dedicated query for a particular codegen unit. I wasn't quite sure how to do that yet but figured this was good to put up.
Alex Crichton [Thu, 14 Sep 2017 03:26:39 +0000 (20:26 -0700)]
rustc: Move codegen to a query
This commit moves the actual code generation in the compiler behind a query
keyed by a codegen unit's name. This ended up entailing quite a few internal
refactorings to enable this, along with a few cut corners:
* The `OutputFilenames` structure is now tracked in the `TyCtxt` as it affects a
whole bunch of trans and such. This is now behind a query and threaded into
the construction of the `TyCtxt`.
* The `TyCtxt` now has a channel "out the back" intended to send data to worker
threads in rustc_trans. This is used as a sort of side effect of the codegen
query but morally what's happening here is the return value of the query
(currently unit but morally a path) is only valid once the background threads
have all finished.
* Dispatching work items to the codegen threads was refactored to only rely on
data in `TyCtxt`, which mostly just involved refactoring where data was
stored, moving it from the translation thread to the controller thread's
`CodegenContext` or the like.
* A new thread locals was introduced in trans to work around the query
system. This is used in the implementation of `assert_module_sources` which
looks like an artifact of the old query system and will presumably go away
once red/green is up and running.
Auto merge of #44607 - alexcrichton:rustbuild-no-j, r=Mark-Simulacrum
rustbuild: Don't pass `-j` if called by `make`
In these situations Cargo just prints out a warning about ignoring the flag
anyway, so let `make` take care of jobs and whatnot instead of getting warnings
printed.
Alex Crichton [Wed, 13 Sep 2017 23:03:24 +0000 (16:03 -0700)]
rustc: Attach an mpsc channel to TyCtxt
This commit attaches a channel to the LLVM workers to the `TyCtxt` which will
later be used during the codegen query to actually send work to LLVM workers.
Otherwise this commit is just plumbing this channel throughout the compiler to
ensure it reaches the right consumers.
Alex Crichton [Wed, 13 Sep 2017 22:24:13 +0000 (15:24 -0700)]
rustc: Remove another global map from trans
This commit removes the `crate_trans_items` field from the `CrateContext` of
trans. This field, a big map, was calculated during partioning and was a set of
all translation items. This isn't quite incremental-friendly because the map may
change a lot but not have much effect on downstream consumers.
Instead a new query was added for the one location this map was needed, along
with a new comment explaining what the location is doing!
Alex Crichton [Wed, 13 Sep 2017 20:22:20 +0000 (13:22 -0700)]
rustc: Mostly remove `ExportedSymbols`
This is a big map that ends up inside of a `CrateContext` during translation for
all codegen units. This means that any change to the map may end up causing an
incremental recompilation of a codegen unit! In order to reduce the amount of
dependencies here between codegen units and the actual input crate this commit
refactors dealing with exported symbols and such into various queries.
The new queries are largely based on existing queries with filled out
implementations for the local crate in addition to external crates, but the main
idea is that while translating codegen untis no unit needs the entire set of
exported symbols, instead they only need queries about particulare `DefId`
instances every now and then.
The linking stage, however, still generates a full list of all exported symbols
from all crates, but that's going to always happen unconditionally anyway, so no
news there!
Alex Crichton [Tue, 12 Sep 2017 19:18:11 +0000 (12:18 -0700)]
rustc: Move some attr methods to queries
Otherwise we may emit double errors related to the `#[export_name]` attribute,
for example, and using a query should ensure that it's only emitted at most
once.
Alex Crichton [Tue, 12 Sep 2017 18:04:46 +0000 (11:04 -0700)]
rustc: Make trans collect/partition a query
This commit moves the `collect_and_partition_translation_items` function into a
query on `TyCtxt` instead of a free function in trans, allowing us to track
dependencies and such of the function.
Alex Crichton [Tue, 12 Sep 2017 16:32:37 +0000 (09:32 -0700)]
rustc: Calculate `ExportedSymbols` in a query
This commit moves the definition of the `ExportedSymbols` structure to the
`rustc` crate and then creates a query that'll be used to construct the
`ExportedSymbols` set. This in turn uses the reachablity query exposed in the
previous commit.
Alex Crichton [Tue, 12 Sep 2017 15:32:50 +0000 (08:32 -0700)]
rustc: Refactor trans paritioning to use tcx
This commit refactors the the `partitioning::partition` function to operate with
a `TyCtxt` instead of a `SharedCrateContext` in preparation for making it a
query.
Alex Crichton [Tue, 12 Sep 2017 15:28:17 +0000 (08:28 -0700)]
rustc_trans: Refactor collection to use tcx
This commit refactors the `collect_crate_translation_items` function to only
require the `TyCtxt` instead of a `SharedCrateContext` in preparation for
query-ifying this portion of trans.
Alex Crichton [Fri, 15 Sep 2017 16:40:35 +0000 (09:40 -0700)]
rustbuild: Don't pass `-j` if called by `make`
In these situations Cargo just prints out a warning about ignoring the flag
anyway, so let `make` take care of jobs and whatnot instead of getting warnings
printed.
Replace str's transmute() calls with pointer casts
After the following conversation in #rust-lang:
```
[14:43:50] <Ixrec> TIL the implementation of from_utf_unchecked is literally just "mem::transmute(x)"
[14:43:59] <Ixrec> no wonder people keep saying transmute is overpowered
[15:15:30] <eddyb> Ixrec: it should be a pointer cast lol
[15:15:46] <eddyb> unless it doesn't let you
[16:50:34] <Ixrec> https://play.rust-lang.org/?gist=d1e6b629ad9ec1baf64ce261c63845e6&version=stable seems like it does let me
[16:52:35] <eddyb> Ixrec: yeah that's the preferred impl
[16:52:46] <eddyb> Ixrec: it just wasn't in 1.0
[16:52:50] <eddyb> IIRC
[16:53:00] <eddyb> (something something fat pointers)
```
Since I already wrote half of the preferred impls in the playground, might as well make an actual PR.
Tim Neumann [Sun, 17 Sep 2017 11:19:15 +0000 (13:19 +0200)]
Rollup merge of #44647 - tmerr:fix-44645, r=dtolnay
Ensure tcp test case passes when disconnected from network
net::tcp::tests::connect_timeout_unroutable fails when the network
is unreachable, like on a laptop disconnected from wifi. Check for
this error and allow the test to pass.
Tim Neumann [Sun, 17 Sep 2017 11:19:11 +0000 (13:19 +0200)]
Rollup merge of #44617 - alexcrichton:download-from-us-west-1, r=aidanhs
ci: Upload/download from a new S3 bucket
Moving buckets from us-east-1 to us-west-1 because us-west-1 is where
rust-central-station itself runs and in general is where we have all our other
buckets.
Tim Neumann [Sun, 17 Sep 2017 11:19:09 +0000 (13:19 +0200)]
Rollup merge of #44595 - budziq:stabilize_compiler_fences, r=alexcrichton
stabilized compiler_fences (fixes #41091)
I did not know what to proceed with "unstable-book" entry. The feature would no longer be unstable so I have deleted it. If it was the wrong call I'll revert it (unfortunately his case is not described in the CONTRIBUTING.md).