diff options
Diffstat (limited to 'src/doc/rustc-dev-guide')
| -rw-r--r-- | src/doc/rustc-dev-guide/src/macro-expansion.md | 172 |
1 files changed, 130 insertions, 42 deletions
diff --git a/src/doc/rustc-dev-guide/src/macro-expansion.md b/src/doc/rustc-dev-guide/src/macro-expansion.md index 77f764d6a63..dd735ed230b 100644 --- a/src/doc/rustc-dev-guide/src/macro-expansion.md +++ b/src/doc/rustc-dev-guide/src/macro-expansion.md @@ -2,56 +2,136 @@ Macro expansion happens during parsing. `rustc` has two parsers, in fact: the normal Rust parser, and the macro parser. During the parsing phase, the normal -Rust parser will call into the macro parser when it encounters a macro. The -macro parser, in turn, may call back out to the Rust parser when it needs to -bind a metavariable (e.g. `$my_expr`). There are a few aspects of this system to -be explained. The code for macro expansion is in `src/libsyntax/ext/tt/`. +Rust parser will call into the macro parser when it encounters a macro +definition or macro invocation (TODO: verify). The macro parser, in turn, may +call back out to the Rust parser when it needs to bind a metavariable (e.g. +`$my_expr`) while parsing the contents of a macro invocation. The code for macro +expansion is in [`src/libsyntax/ext/tt/`][code_dir]. This chapter aims to +explain how macro expansion works. + +### Example + +It's helpful to have an example to refer to. For the remainder of this chapter, +whenever we refer to the "example _definition_", we mean the following: + +```rust +macro_rules! printer { + (print $mvar:ident) => { + println!("{}", $mvar); + } + (print twice $mvar:ident) => { + println!("{}", $mvar); + println!("{}", $mvar); + } +} +``` + +`$mvar` is called a _metavariable_. Unlike normal variables, rather than binding +to a value in a computation, a metavariable binds _at compile time_ to a tree of +_tokens_. A _token_ zero or more symbols that together have some meaning. For +example, in our example definition, `print`, `$mvar`, `=>`, `{` are all tokens +(though that's not an exhaustive list). There are also other special tokens, +such as `EOF`, which indicates that there are no more tokens. The process of +producing a stream of tokens from the raw bytes of the source file is called +_lexing_. For more information about _lexing_, see the [Parsing +chapter][parsing] of this book. + +Whenever we refer to the "example _invocation_", we mean the following snippet: + +```rust +printer!(print foo); // Assume `foo` is a variable defined somewhere else... +``` + +The process of expanding the macro invocation into the syntax tree +`println!("{}", foo)` and then expanding that into a call to `Display::fmt` is +called _macro expansion_, it is the topic of this chapter. ### The macro parser +There are two parts to macro expansion: parsing the definition and parsing the +invocations. Interestingly, both are done by the macro parser. + Basically, the macro parser is like an NFA-based regex parser. It uses an algorithm similar in spirit to the [Earley parsing algorithm](https://en.wikipedia.org/wiki/Earley_parser). The macro parser is -defined in `src/libsyntax/ext/tt/macro_parser.rs`. - -In a traditional NFA-based parser, one common approach is to have some pattern -which we are trying to match an input against. Moreover, we may try to capture -some portion of the input and bind it to variable in the pattern. For example: -suppose we have a pattern (borrowing Rust macro syntax) such as `a $b:ident a` --- that is, an `a` token followed by an `ident` token followed by another `a` -token. Given an input `a foo a`, the _metavariable_ `$b` would bind to the -`ident` `foo`. On the other hand, an input `a foo b` would be rejected as a -parse failure because the pattern `a <ident> a` cannot match `a foo b` (or as -the compiler would put it, "no rules expected token `b`"). - -The macro parser does pretty much exactly that with one exception: in order to -parse different types of metavariables, such as `ident`, `block`, `expr`, etc., -the macro parser must sometimes call back to the normal Rust parser. - -Interestingly, both definitions and invokations of macros are parsed using the -macro parser. This is extremely non-intuitive and self-referential. The code to -parse macro _definitions_ is in `src/libsyntax/ext/tt/macro_rules.rs`. It -defines the pattern for matching for a macro definition as `$( $lhs:tt => -$rhs:tt );+`. In other words, a `macro_rules` defintion should have in its body -at least one occurence of a token tree followed by `=>` followed by another -token tree. When the compiler comes to a `macro_rules` definition, it uses this -pattern to match the two token trees per rule in the definition of the macro -_using the macro parser itself_. - -When the compiler comes to a macro invokation, it needs to parse that -invokation. This is also known as _macro expansion_. The same NFA-based macro -parser is used that is described above. Notably, the "pattern" (or _matcher_) -used is the first token tree extracted from the rules of the macro _definition_. -In other words, given some pattern described by the _definition_ of the macro, -we want to match the contents of the _invokation_ of the macro. - -The algorithm is exactly the same, but when the macro parser comes to a place in -the current matcher where it needs to match a _non-terminal_ (i.e. a -metavariable), it calls back to the normal Rust parser to get the contents of -that non-terminal. Then, the macro parser proceeds in parsing as normal. +defined in [`src/libsyntax/ext/tt/macro_parser.rs`][code_mp]. + +The interface of the macro parser is as follows (this is slightly simplified): + +```rust +fn parse( + sess: ParserSession, + tts: TokenStream, + ms: &[TokenTree] +) -> NamedParseResult +``` + +In this interface: + +- `sess` is a "parsing session", which keeps track of some metadata. Most + notably, this is used to keep track of errors that are generated so they can + be reported to the user. +- `tts` is a stream of tokens. The macro parser's job is to consume the raw + stream of tokens and output a binding of metavariables to corresponding token + trees. +- `ms` a _matcher_. This is a sequence of token trees that we want to match + `tts` against. + +In the analogy of a regex parser, `tts` is the input and we are matching it +against the pattern `ms`. Using our examples, `tts` could be the stream of +tokens containing the inside of the example invocation `print foo`, while `ms` +might be the sequence of token (trees) `print $mvar:ident`. + +The output of the parser is a `NamedParserResult`, which indicates which of +three cases has occured: + +- Success: `tts` matches the given matcher `ms`, and we have produced a binding + from metavariables to the corresponding token trees. +- Failure: `tts` does not match `ms`. This results in an error message such as + "No rule expected token _blah_". +- Error: some fatal error has occured _in the parser_. For example, this happens + if there are more than one pattern match, since that indicates the macro is + ambiguous. + +The full interface is defined [here][code_parse_int]. + +The macro parser does pretty much exactly the same as a normal regex parser with +one exception: in order to parse different types of metavariables, such as +`ident`, `block`, `expr`, etc., the macro parser must sometimes call back to the +normal Rust parser. + +As mentioned above, both definitions and invocations of macros are parsed using +the macro parser. This is extremely non-intuitive and self-referential. The code +to parse macro _definitions_ is in +[`src/libsyntax/ext/tt/macro_rules.rs`][code_mr]. It defines the pattern for +matching for a macro definition as `$( $lhs:tt => $rhs:tt );+`. In other words, +a `macro_rules` defintion should have in its body at least one occurence of a +token tree followed by `=>` followed by another token tree. When the compiler +comes to a `macro_rules` definition, it uses this pattern to match the two token +trees per rule in the definition of the macro _using the macro parser itself_. +In our example definition, the metavariable `$lhs` would match the patterns of +both arms: `(print $mvar:ident)` and `(print twice $mvar:ident)`. And `$rhs` +would match the bodies of both arms: `{ println!("{}", $mvar); }` and `{ +println!("{}", $mvar); println!("{}", $mvar); }`. The parser would keep this +knowledge around for when it needs to expand a macro invocation. + +When the compiler comes to a macro invocation, it parses that invocation using +the same NFA-based macro parser that is described above. However, the matcher +used is the first token tree (`$lhs`) extracted from the arms of the macro +_definition_. Using our example, we would try to match the token stream `print +foo` from the invocation against the matchers `print $mvar:ident` and `print +twice $mvar:ident` that we previously extracted from the definition. The +algorithm is exactly the same, but when the macro parser comes to a place in the +current matcher where it needs to match a _non-terminal_ (e.g. `$mvar:ident`), +it calls back to the normal Rust parser to get the contents of that +non-terminal. In this case, the Rust parser would look for an `ident` token, +which it finds (`foo`) and returns to the macro parser. Then, the macro parser +proceeds in parsing as normal. Also, note that exactly one of the matchers from +the various arms should match the invocation (otherwise, the macro is +ambiguous). For more information about the macro parser's implementation, see the comments -in `src/libsyntax/ext/tt/macro_parser.rs`. +in [`src/libsyntax/ext/tt/macro_parser.rs`][code_mp]. ### Hygiene @@ -64,3 +144,11 @@ TODO ### Custom Derive TODO + + + +[code_dir]: https://github.com/rust-lang/rust/tree/master/src/libsyntax/ext/tt +[code_mp]: https://github.com/rust-lang/rust/tree/master/src/libsyntax/ext/tt/macro_parser.rs +[code_mp]: https://github.com/rust-lang/rust/tree/master/src/libsyntax/ext/tt/macro_rules.rs +[code_parse_int]: https://github.com/rust-lang/rust/blob/a97cd17f5d71fb4ec362f4fbd79373a6e7ed7b82/src/libsyntax/ext/tt/macro_parser.rs#L421 +[parsing]: ./the-parser.md |