proc_macro: implement `TokenTree`, `TokenKind`, hygienic `quote!`, and other API
All new API is gated behind `#![feature(proc_macro)]` and may be used with `#[proc_macro]`, `#[proc_macro_attribute]`, and `#[proc_macro_derive]` procedural macros.
More specifically, this PR adds the following in `proc_macro`:
```rust
// `TokenStream` constructors:
impl TokenStream { fn empty() -> TokenStream { ... } }
impl From<TokenTree> for TokenStream { ... }
impl From<TokenKind> for TokenStream { ... }
impl<T: Into<TokenStream>> FromIterator<T> for TokenStream { ... }
macro quote($($t:tt)*) { ... } // A hygienic `TokenStream` quoter
// `TokenStream` destructuring:
impl TokenStream { fn is_empty(&self) -> bool { ... } }
impl IntoIterator for TokenStream { type Item = TokenTree; ... }
struct TokenTree { span: Span, kind: TokenKind }
impl From<TokenKind> for TokenTree { ... }
impl Display for TokenTree { ... }
struct Span { ... } // a region of source code along with expansion/hygiene information
impl Default for Span { ... } // a span from the current procedural macro definition
impl Span { fn call_site() -> Span { ... } } // the call site of the current expansion
fn quote_span(span: Span) -> TokenStream;
enum TokenKind {
Group(Delimiter, TokenStream), // A delimited sequence, e.g. `( ... )`
Term(Term), // a unicode identifier, lifetime ('a), or underscore
Op(char, Spacing), // a punctuation character (`+`, `,`, `$`, etc.).
Literal(Literal), // a literal character (`'a'`), string (`"hello"`), or number (`2.3`)
}
enum Delimiter {
Parenthesis, // `( ... )`
Brace, // `[ ... ]`
Bracket, // `{ ... }`
None, // an implicit delimiter, e.g. `$var`, where $var is `...`.
}
struct Term { ... } // An interned string
impl Term {
fn intern(string: &str) -> Symbol { ... }
fn as_str(&self) -> &str { ... }
}
enum Spacing {
Alone, // not immediately followed by another `Op`, e.g. `+` in `+ =`.
Joint, // immediately followed by another `Op`, e.g. `+` in `+=`
}
struct Literal { ... }
impl Display for Literal { ... }
impl Literal {
fn integer(n: i128) -> Literal { .. } // unsuffixed integer literal
fn float(n: f64) -> Literal { .. } // unsuffixed floating point literal
fn u8(n: u8) -> Literal { ... } // similarly: i8, u16, i16, u32, i32, u64, i64, f32, f64
fn string(string: &str) -> Literal { ... }
fn character(ch: char) -> Literal { ... }
fn byte_string(bytes: &[u8]) -> Literal { ... }
}
```
For details on `quote!` hygiene, see [this example](https://github.com/rust-lang/rust/pull/40939/commits/
20a90485c040df87a667e9b6ee38e4d8a7d7fc5d) and [declarative macros 2.0](https://github.com/rust-lang/rust/pull/40847).
r? @nrc