Overhaul `MacArgs::Eq`.

The value in `MacArgs::Eq` is currently represented as a `Token`.
Because of `TokenKind::Interpolated`, `Token` can be either a token or
an arbitrary AST fragment. In practice, a `MacArgs::Eq` starts out as a
literal or macro call AST fragment, and then is later lowered to a
literal token. But this is very non-obvious. `Token` is a much more
general type than what is needed.

This commit restricts things, by introducing a new type `MacArgsEqKind`
that is either an AST expression (pre-lowering) or an AST literal
(post-lowering). The downside is that the code is a bit more verbose in
a few places. The benefit is that makes it much clearer what the
possibilities are (though also shorter in some other places). Also, it
removes one use of `TokenKind::Interpolated`, taking us a step closer to
removing that variant, which will let us make `Token` impl `Copy` and
remove many "handle Interpolated" code paths in the parser.

Things to note:
- Error messages have improved. Messages like this:
  ```
  unexpected token: `"bug" + "found"`
  ```
  now say "unexpected expression", which makes more sense. Although
  arbitrary expressions can exist within tokens thanks to
  `TokenKind::Interpolated`, that's not obvious to anyone who doesn't
  know compiler internals.
- In `parse_mac_args_common`, we no longer need to collect tokens for
  the value expression.

1 files changed, 16 insertions, 16 deletions

diff --git a/compiler/rustc_ast_lowering/src/lib.rs b/compiler/rustc_ast_lowering/src/lib.rs
index 526679e5315..d6bbd23cdce 100644
--- a/compiler/rustc_ast_lowering/src/lib.rs
+++ b/compiler/rustc_ast_lowering/src/lib.rs
@@ -38,7 +38,6 @@
 #![recursion_limit = "256"]
 #![allow(rustc::potential_query_instability)]
 
-use rustc_ast::token::{self, Token, TokenKind};
 use rustc_ast::tokenstream::{CanSynthesizeMissingTokens, TokenStream};
 use rustc_ast::visit;
 use rustc_ast::{self as ast, *};
@@ -874,23 +873,24 @@ impl<'a, 'hir> LoweringContext<'a, 'hir> {
                 )
             }
             // This is an inert key-value attribute - it will never be visible to macros
-            // after it gets lowered to HIR. Therefore, we can synthesize tokens with fake
-            // spans to handle nonterminals in `#[doc]` (e.g. `#[doc = $e]`).
-            MacArgs::Eq(eq_span, ref token) => {
-                // In valid code the value is always representable as a single literal token.
-                // Otherwise, a dummy token suffices because the error is handled elsewhere.
-                let token = if let token::Interpolated(nt) = &token.kind
-                    && let token::NtExpr(expr) = &**nt
-                {
-                    if let ExprKind::Lit(Lit { token, span, .. }) = expr.kind {
-                        Token::new(TokenKind::Literal(token), span)
-                    } else {
-                        Token::dummy()
-                    }
+            // after it gets lowered to HIR. Therefore, we can extract literals to handle
+            // nonterminals in `#[doc]` (e.g. `#[doc = $e]`).
+            MacArgs::Eq(eq_span, MacArgsEq::Ast(ref expr)) => {
+                // In valid code the value always ends up as a single literal. Otherwise, a dummy
+                // literal suffices because the error is handled elsewhere.
+                let lit = if let ExprKind::Lit(lit) = &expr.kind {
+                    lit.clone()
                 } else {
-                    unreachable!()
+                    Lit {
+                        token: token::Lit::new(token::LitKind::Err, kw::Empty, None),
+                        kind: LitKind::Err(kw::Empty),
+                        span: DUMMY_SP,
+                    }
                 };
-                MacArgs::Eq(eq_span, token)
+                MacArgs::Eq(eq_span, MacArgsEq::Hir(lit))
+            }
+            MacArgs::Eq(_, MacArgsEq::Hir(ref lit)) => {
+                unreachable!("in literal form when lowering mac args eq: {:?}", lit)
             }
         }
     }