Properly capture trailing 'unglued' token

If we try to capture the `Vec<u8>` in `Option<Vec<u8>>`, we'll
need to capture a `>` token which was 'unglued' from a `>>` token.
The processing of unglueing a token for parsing purposes bypasses the
usual capturing infrastructure, so we currently lose the trailing `>`.
As a result, we fall back to the reparsed `TokenStream`, causing us to
lose spans.

This commit makes token capturing keep track of a trailing 'unglued'
token. Note that we don't need to care about unglueing except at the end
of the captured tokens - if we capture both the first and second unglued
tokens, then we'll end up capturing the full 'glued' token, which
already works correctly.

2 files changed, 48 insertions, 0 deletions

diff --git a/src/test/ui/proc-macro/capture-unglued-token.rs b/src/test/ui/proc-macro/capture-unglued-token.rs
new file mode 100644
index 00000000000..727b779776b
--- /dev/null
+++ b/src/test/ui/proc-macro/capture-unglued-token.rs
@@ -0,0 +1,20 @@
+// aux-build:test-macros.rs
+// compile-flags: -Z span-debug
+// check-pass
+
+// Tests that we properly handle parsing a nonterminal
+// where we have two consecutive angle brackets (one inside
+// the nonterminal, and one outside)
+
+#![no_std] // Don't load unnecessary hygiene information from std
+extern crate std;
+extern crate test_macros;
+
+macro_rules! trailing_angle {
+    (Option<$field:ty>) => {
+        test_macros::print_bang_consume!($field);
+    }
+}
+
+trailing_angle!(Option<Vec<u8>>);
+fn main() {}
diff --git a/src/test/ui/proc-macro/capture-unglued-token.stdout b/src/test/ui/proc-macro/capture-unglued-token.stdout
new file mode 100644
index 00000000000..7e6b540332c
--- /dev/null
+++ b/src/test/ui/proc-macro/capture-unglued-token.stdout
@@ -0,0 +1,28 @@
+PRINT-BANG INPUT (DISPLAY): Vec<u8>
+PRINT-BANG RE-COLLECTED (DISPLAY): Vec < u8 >
+PRINT-BANG INPUT (DEBUG): TokenStream [
+    Group {
+        delimiter: None,
+        stream: TokenStream [
+            Ident {
+                ident: "Vec",
+                span: $DIR/capture-unglued-token.rs:19:24: 19:27 (#0),
+            },
+            Punct {
+                ch: '<',
+                spacing: Alone,
+                span: $DIR/capture-unglued-token.rs:19:27: 19:28 (#0),
+            },
+            Ident {
+                ident: "u8",
+                span: $DIR/capture-unglued-token.rs:19:28: 19:30 (#0),
+            },
+            Punct {
+                ch: '>',
+                spacing: Alone,
+                span: $DIR/capture-unglued-token.rs:19:30: 19:31 (#0),
+            },
+        ],
+        span: $DIR/capture-unglued-token.rs:15:42: 15:48 (#4),
+    },
+]