Implement token-based handling of attributes during expansion

This PR modifies the macro expansion infrastructure to handle attributes
in a fully token-based manner. As a result:

* Derives macros no longer lose spans when their input is modified
  by eager cfg-expansion. This is accomplished by performing eager
  cfg-expansion on the token stream that we pass to the derive
  proc-macro
* Inner attributes now preserve spans in all cases, including when we
  have multiple inner attributes in a row.

This is accomplished through the following changes:

* New structs `AttrAnnotatedTokenStream` and `AttrAnnotatedTokenTree` are introduced.
  These are very similar to a normal `TokenTree`, but they also track
  the position of attributes and attribute targets within the stream.
  They are built when we collect tokens during parsing.
  An `AttrAnnotatedTokenStream` is converted to a regular `TokenStream` when
  we invoke a macro.
* Token capturing and `LazyTokenStream` are modified to work with
  `AttrAnnotatedTokenStream`. A new `ReplaceRange` type is introduced, which
  is created during the parsing of a nested AST node to make the 'outer'
  AST node aware of the attributes and attribute target stored deeper in the token stream.
* When we need to perform eager cfg-expansion (either due to `#[derive]` or `#[cfg_eval]`),
we tokenize and reparse our target, capturing additional information about the locations of
`#[cfg]` and `#[cfg_attr]` attributes at any depth within the target.
This is a performance optimization, allowing us to perform less work
in the typical case where captured tokens never have eager cfg-expansion run.

1 files changed, 5 insertions, 7 deletions

diff --git a/compiler/rustc_parse/src/parser/expr.rs b/compiler/rustc_parse/src/parser/expr.rs
index 02ee268b88c..e155b3fa773 100644
--- a/compiler/rustc_parse/src/parser/expr.rs
+++ b/compiler/rustc_parse/src/parser/expr.rs
@@ -2581,19 +2581,17 @@ impl<'a> Parser<'a> {
         attrs: AttrWrapper,
         f: impl FnOnce(&mut Self, Vec<ast::Attribute>) -> PResult<'a, P<Expr>>,
     ) -> PResult<'a, P<Expr>> {
-        // FIXME - come up with a nice way to properly forward `ForceCollect`from
-        // the nonterminal parsing code. TThis approach iscorrect, but will cause
-        // us to unnecessarily capture tokens for exprs that have only builtin
-        // attributes. Revisit this before #![feature(stmt_expr_attributes)] is stabilized
-        let force_collect = if attrs.is_empty() { ForceCollect::No } else { ForceCollect::Yes };
-        self.collect_tokens_trailing_token(attrs, force_collect, |this, attrs| {
+        self.collect_tokens_trailing_token(attrs, ForceCollect::No, |this, attrs| {
             let res = f(this, attrs)?;
             let trailing = if this.restrictions.contains(Restrictions::STMT_EXPR)
                 && this.token.kind == token::Semi
             {
                 TrailingToken::Semi
             } else {
-                TrailingToken::None
+                // FIXME - pass this through from the place where we know
+                // we need a comma, rather than assuming that `#[attr] expr,`
+                // always captures a trailing comma
+                TrailingToken::MaybeComma
             };
             Ok((res, trailing))
         })