]> git.lizzy.rs Git - rust.git/commit
Make `TokenStream` less recursive.
authorNicholas Nethercote <nnethercote@mozilla.com>
Wed, 19 Dec 2018 03:53:52 +0000 (14:53 +1100)
committerNicholas Nethercote <nnethercote@mozilla.com>
Tue, 8 Jan 2019 04:08:46 +0000 (15:08 +1100)
commite80a93040ffbbb7eb8013f1dcd3b594ce8a631cd
tree7dab947607d393e7d4fdc35ffa95a6f101f3a549
parentb92552d5578e4544006da0dd5e793a19c2149321
Make `TokenStream` less recursive.

`TokenStream` is currently recursive in *two* ways:

- the `TokenTree` variant contains a `ThinTokenStream`, which can
  contain a `TokenStream`;

- the `TokenStream` variant contains a `Vec<TokenStream>`.

The latter is not necessary and causes significant complexity. This
commit replaces it with the simpler `Vec<(TokenTree, IsJoint)>`.

This reduces complexity significantly. In particular, `StreamCursor` is
eliminated, and `Cursor` becomes much simpler, consisting now of just a
`TokenStream` and an index.

The commit also removes the `Extend` impl for `TokenStream`, because it
is only used in tests. (The commit also removes those tests.)

Overall, the commit reduces the number of lines of code by almost 200.
src/libsyntax/attr/mod.rs
src/libsyntax/ext/quote.rs
src/libsyntax/ext/tt/transcribe.rs
src/libsyntax/parse/lexer/tokentrees.rs
src/libsyntax/parse/parser.rs
src/libsyntax/tokenstream.rs
src/libsyntax_ext/proc_macro_server.rs