]> git.lizzy.rs Git - rust.git/commit
rustc: Fix joint-ness of stringified token-streams
authorAlex Crichton <alex@alexcrichton.com>
Thu, 17 May 2018 16:30:43 +0000 (09:30 -0700)
committerAlex Crichton <alex@alexcrichton.com>
Fri, 18 May 2018 17:36:24 +0000 (10:36 -0700)
commit0ee031ab9668c52c05e189c4e9380901d7d1e579
treea24008d8ddeb80756c7e33da37bd268630c010cb
parentba64edb3edb9d54f24507255d6aaca5a567e2f40
rustc: Fix joint-ness of stringified token-streams

This commit fixes `StringReader`'s parsing of tokens which have been stringified
through procedural macros. Whether or not a token tree is joint is defined by
span information, but when working with procedural macros these spans are often
dummy and/or overridden which means that they end up considering all operators
joint if they can!

The fix here is to track the raw source span as opposed to the overridden span.
With this information we can more accurately classify `Punct` structs as either
joint or not.

Closes #50700
src/libsyntax/parse/lexer/mod.rs
src/libsyntax/parse/lexer/tokentrees.rs
src/libsyntax/parse/token.rs
src/test/run-pass-fulldeps/proc-macro/auxiliary/not-joint.rs [new file with mode: 0644]
src/test/run-pass-fulldeps/proc-macro/not-joint.rs [new file with mode: 0644]