Rollup merge of #66935 - petrochenkov:attrtok2, r=Centril

syntax: Unify macro and attribute arguments in AST

The unified form (`ast::MacArgs`) represents parsed arguments instead of an unstructured token stream that was previously used for attributes.
It also tracks some spans and delimiter kinds better for fn-like macros and macro definitions.

I've been talking about implementing this with @nnethercote in https://github.com/rust-lang/rust/pull/65750#issuecomment-546517322.
The parsed representation is closer to `MetaItem` and requires less token juggling during conversions, so it potentially may be faster.

r? @Centril

1 files changed, 8 insertions, 0 deletions

diff --git a/src/libsyntax/tokenstream.rs b/src/libsyntax/tokenstream.rs
index 6a0523dd655..491b9a9ade4 100644
--- a/src/libsyntax/tokenstream.rs
+++ b/src/libsyntax/tokenstream.rs
@@ -225,6 +225,14 @@ impl TokenStream {
         self.0.len()
     }
 
+    pub fn span(&self) -> Option<Span> {
+        match &**self.0 {
+            [] => None,
+            [(tt, _)] => Some(tt.span()),
+            [(tt_start, _), .., (tt_end, _)] => Some(tt_start.span().to(tt_end.span())),
+        }
+    }
+
     pub fn from_streams(mut streams: SmallVec<[TokenStream; 2]>) -> TokenStream {
         match streams.len() {
             0 => TokenStream::default(),