Implement span quoting for proc-macros

This PR implements span quoting, allowing proc-macros to produce spans
pointing *into their own crate*. This is used by the unstable
`proc_macro::quote!` macro, allowing us to get error messages like this:

```
error[E0412]: cannot find type `MissingType` in this scope
  --> $DIR/auxiliary/span-from-proc-macro.rs:37:20
   |
LL | pub fn error_from_attribute(_args: TokenStream, _input: TokenStream) -> TokenStream {
   | ----------------------------------------------------------------------------------- in this expansion of procedural macro `#[error_from_attribute]`
...
LL |             field: MissingType
   |                    ^^^^^^^^^^^ not found in this scope
   |
  ::: $DIR/span-from-proc-macro.rs:8:1
   |
LL | #[error_from_attribute]
   | ----------------------- in this macro invocation
```

Here, `MissingType` occurs inside the implementation of the proc-macro
`#[error_from_attribute]`. Previosuly, this would always result in a
span pointing at `#[error_from_attribute]`

This will make many proc-macro-related error message much more useful -
when a proc-macro generates code containing an error, users will get an
error message pointing directly at that code (within the macro
definition), instead of always getting a span pointing at the macro
invocation site.

This is implemented as follows:
* When a proc-macro crate is being *compiled*, it causes the `quote!`
  macro to get run. This saves all of the sapns in the input to `quote!`
  into the metadata of *the proc-macro-crate* (which we are currently
  compiling). The `quote!` macro then expands to a call to
  `proc_macro::Span::recover_proc_macro_span(id)`, where `id` is an
opaque identifier for the span in the crate metadata.
* When the same proc-macro crate is *run* (e.g. it is loaded from disk
  and invoked by some consumer crate), the call to
`proc_macro::Span::recover_proc_macro_span` causes us to load the span
from the proc-macro crate's metadata. The proc-macro then produces a
`TokenStream` containing a `Span` pointing into the proc-macro crate
itself.

The recursive nature of 'quote!' can be difficult to understand at
first. The file `src/test/ui/proc-macro/quote-debug.stdout` shows
the output of the `quote!` macro, which should make this eaier to
understand.

This PR also supports custom quoting spans in custom quote macros (e.g.
the `quote` crate). All span quoting goes through the
`proc_macro::quote_span` method, which can be called by a custom quote
macro to perform span quoting. An example of this usage is provided in
`src/test/ui/proc-macro/auxiliary/custom-quote.rs`

Custom quoting currently has a few limitations:

In order to quote a span, we need to generate a call to
`proc_macro::Span::recover_proc_macro_span`. However, proc-macros
support renaming the `proc_macro` crate, so we can't simply hardcode
this path. Previously, the `quote_span` method used the path
`crate::Span` - however, this only works when it is called by the
builtin `quote!` macro in the same crate. To support being called from
arbitrary crates, we need access to the name of the `proc_macro` crate
to generate a path. This PR adds an additional argument to `quote_span`
to specify the name of the `proc_macro` crate. Howver, this feels kind
of hacky, and we may want to change this before stabilizing anything
quote-related.

Additionally, using `quote_span` currently requires enabling the
`proc_macro_internals` feature. The builtin `quote!` macro
has an `#[allow_internal_unstable]` attribute, but this won't work for
custom quote implementations. This will likely require some additional
tricks to apply `allow_internal_unstable` to the span of
`proc_macro::Span::recover_proc_macro_span`.

8 files changed, 235 insertions, 6 deletions

diff --git a/src/test/ui/proc-macro/auxiliary/custom-quote.rs b/src/test/ui/proc-macro/auxiliary/custom-quote.rs
new file mode 100644
index 00000000000..714417deee5
--- /dev/null
+++ b/src/test/ui/proc-macro/auxiliary/custom-quote.rs
@@ -0,0 +1,31 @@
+// force-host
+// no-prefer-dynamic
+// ignore-tidy-linelength
+
+#![feature(proc_macro_quote)]
+#![crate_type = "proc-macro"]
+
+extern crate proc_macro;
+use std::str::FromStr;
+use proc_macro::*;
+
+#[proc_macro]
+pub fn custom_quote(input: TokenStream) -> TokenStream {
+    let mut tokens: Vec<_> = input.into_iter().collect();
+    assert_eq!(tokens.len(), 1, "Unexpected input: {:?}", tokens);
+    match tokens.pop() {
+        Some(TokenTree::Ident(ident)) => {
+            assert_eq!(ident.to_string(), "my_ident");
+
+            let proc_macro_crate = TokenStream::from_str("::proc_macro").unwrap();
+            let quoted_span = proc_macro::quote_span(proc_macro_crate, ident.span());
+            let prefix = TokenStream::from_str(r#"let mut ident = proc_macro::Ident::new("my_ident", proc_macro::Span::call_site());"#).unwrap();
+            let set_span_method = TokenStream::from_str("ident.set_span").unwrap();
+            let set_span_arg = TokenStream::from(TokenTree::Group(Group::new(Delimiter::Parenthesis, quoted_span)));
+            let suffix = TokenStream::from_str(";proc_macro::TokenStream::from(proc_macro::TokenTree::Ident(ident))").unwrap();
+            let full_stream: TokenStream = std::array::IntoIter::new([prefix, set_span_method, set_span_arg, suffix]).collect();
+            full_stream
+        }
+        _ => unreachable!()
+    }
+}
diff --git a/src/test/ui/proc-macro/auxiliary/span-from-proc-macro.rs b/src/test/ui/proc-macro/auxiliary/span-from-proc-macro.rs
new file mode 100644
index 00000000000..49292acfea7
--- /dev/null
+++ b/src/test/ui/proc-macro/auxiliary/span-from-proc-macro.rs
@@ -0,0 +1,49 @@
+// force-host
+// no-prefer-dynamic
+
+#![feature(proc_macro_quote)]
+#![feature(proc_macro_internals)] // FIXME - this shouldn't be necessary
+#![crate_type = "proc-macro"]
+
+extern crate proc_macro;
+extern crate custom_quote;
+
+use proc_macro::{quote, TokenStream};
+
+macro_rules! expand_to_quote {
+    () => {
+        quote! {
+            let bang_error: bool = 25;
+        }
+    }
+}
+
+#[proc_macro]
+pub fn error_from_bang(_input: TokenStream) -> TokenStream {
+    expand_to_quote!()
+}
+
+#[proc_macro]
+pub fn other_error_from_bang(_input: TokenStream) -> TokenStream {
+    custom_quote::custom_quote! {
+        my_ident
+    }
+}
+
+#[proc_macro_attribute]
+pub fn error_from_attribute(_args: TokenStream, _input: TokenStream) -> TokenStream {
+    quote! {
+        struct AttributeError {
+            field: MissingType
+        }
+    }
+}
+
+#[proc_macro_derive(ErrorFromDerive)]
+pub fn error_from_derive(_input: TokenStream) -> TokenStream {
+    quote! {
+        enum DeriveError {
+            Variant(OtherMissingType)
+        }
+    }
+}
diff --git a/src/test/ui/proc-macro/meta-macro-hygiene.stdout b/src/test/ui/proc-macro/meta-macro-hygiene.stdout
index b7a37ab10ed..d9337fb361b 100644
--- a/src/test/ui/proc-macro/meta-macro-hygiene.stdout
+++ b/src/test/ui/proc-macro/meta-macro-hygiene.stdout
@@ -45,10 +45,10 @@ fn main /* 0#0 */() { ; }
 Expansions:
 0: parent: ExpnId(0), call_site_ctxt: #0, def_site_ctxt: #0, kind: Root
 1: parent: ExpnId(0), call_site_ctxt: #0, def_site_ctxt: #0, kind: AstPass(StdImports)
-2: parent: ExpnId(0), call_site_ctxt: #0, def_site_ctxt: #0, kind: Macro(Bang, "produce_it")
+2: parent: ExpnId(0), call_site_ctxt: #0, def_site_ctxt: #0, kind: Macro { kind: Bang, name: "produce_it", proc_macro: false }
 3: parent: ExpnId(0), call_site_ctxt: #0, def_site_ctxt: #0, kind: AstPass(StdImports)
-4: parent: ExpnId(2), call_site_ctxt: #4, def_site_ctxt: #0, kind: Macro(Bang, "meta_macro::print_def_site")
-5: parent: ExpnId(4), call_site_ctxt: #5, def_site_ctxt: #0, kind: Macro(Bang, "$crate::dummy")
+4: parent: ExpnId(2), call_site_ctxt: #4, def_site_ctxt: #0, kind: Macro { kind: Bang, name: "meta_macro::print_def_site", proc_macro: true }
+5: parent: ExpnId(4), call_site_ctxt: #5, def_site_ctxt: #0, kind: Macro { kind: Bang, name: "$crate::dummy", proc_macro: true }
 
 SyntaxContexts:
 #0: parent: #0, outer_mark: (ExpnId(0), Opaque)
diff --git a/src/test/ui/proc-macro/nonterminal-token-hygiene.stdout b/src/test/ui/proc-macro/nonterminal-token-hygiene.stdout
index ba3b3ee7827..54dc856bfb0 100644
--- a/src/test/ui/proc-macro/nonterminal-token-hygiene.stdout
+++ b/src/test/ui/proc-macro/nonterminal-token-hygiene.stdout
@@ -69,10 +69,10 @@ fn main /* 0#0 */() { }
 Expansions:
 0: parent: ExpnId(0), call_site_ctxt: #0, def_site_ctxt: #0, kind: Root
 1: parent: ExpnId(0), call_site_ctxt: #0, def_site_ctxt: #0, kind: AstPass(StdImports)
-2: parent: ExpnId(0), call_site_ctxt: #0, def_site_ctxt: #0, kind: Macro(Bang, "outer")
+2: parent: ExpnId(0), call_site_ctxt: #0, def_site_ctxt: #0, kind: Macro { kind: Bang, name: "outer", proc_macro: false }
 3: parent: ExpnId(0), call_site_ctxt: #0, def_site_ctxt: #0, kind: AstPass(StdImports)
-4: parent: ExpnId(2), call_site_ctxt: #4, def_site_ctxt: #4, kind: Macro(Bang, "inner")
-5: parent: ExpnId(4), call_site_ctxt: #6, def_site_ctxt: #0, kind: Macro(Bang, "print_bang")
+4: parent: ExpnId(2), call_site_ctxt: #4, def_site_ctxt: #4, kind: Macro { kind: Bang, name: "inner", proc_macro: false }
+5: parent: ExpnId(4), call_site_ctxt: #6, def_site_ctxt: #0, kind: Macro { kind: Bang, name: "print_bang", proc_macro: true }
 
 SyntaxContexts:
 #0: parent: #0, outer_mark: (ExpnId(0), Opaque)
diff --git a/src/test/ui/proc-macro/quote-debug.rs b/src/test/ui/proc-macro/quote-debug.rs
new file mode 100644
index 00000000000..e0304a01405
--- /dev/null
+++ b/src/test/ui/proc-macro/quote-debug.rs
@@ -0,0 +1,18 @@
+// check-pass
+// force-host
+// no-prefer-dynamic
+// compile-flags: -Z unpretty=expanded
+//
+// This file is not actually used as a proc-macro - instead,
+// it's just used to show the output of the `quote!` macro
+
+#![feature(proc_macro_quote)]
+#![crate_type = "proc-macro"]
+
+extern crate proc_macro;
+
+fn main() {
+    proc_macro::quote! {
+        let hello = "world";
+    }
+}
diff --git a/src/test/ui/proc-macro/quote-debug.stdout b/src/test/ui/proc-macro/quote-debug.stdout
new file mode 100644
index 00000000000..4bdc04b9ac4
--- /dev/null
+++ b/src/test/ui/proc-macro/quote-debug.stdout
@@ -0,0 +1,52 @@
+#![feature(prelude_import)]
+#![no_std]
+// check-pass
+// force-host
+// no-prefer-dynamic
+// compile-flags: -Z unpretty=expanded
+//
+// This file is not actually used as a proc-macro - instead,
+// it's just used to show the output of the `quote!` macro
+
+#![feature(proc_macro_quote)]
+#![crate_type = "proc-macro"]
+#[prelude_import]
+use ::std::prelude::rust_2015::*;
+#[macro_use]
+extern crate std;
+
+extern crate proc_macro;
+
+fn main() {
+    [crate::TokenStream::from(crate::TokenTree::Ident(crate::Ident::new("let",
+                                                                        crate::Span::recover_proc_macro_span(0)))),
+     crate::TokenStream::from(crate::TokenTree::Ident(crate::Ident::new("hello",
+                                                                        crate::Span::recover_proc_macro_span(1)))),
+     crate::TokenStream::from(crate::TokenTree::Punct(crate::Punct::new('\u{3d}',
+                                                                        crate::Spacing::Alone))),
+     crate::TokenStream::from(crate::TokenTree::Literal({
+                                                            let mut iter =
+                                                                "\"world\"".parse::<crate::TokenStream>().unwrap().into_iter();
+                                                            if let (Some(crate::TokenTree::Literal(mut lit)),
+                                                                    None) =
+                                                                   (iter.next(),
+                                                                    iter.next())
+                                                               {
+                                                                lit.set_span(crate::Span::recover_proc_macro_span(2));
+                                                                lit
+                                                            } else {
+                                                                {
+                                                                    ::core::panicking::panic("internal error: entered unreachable code")
+                                                                }
+                                                            }
+                                                        })),
+     crate::TokenStream::from(crate::TokenTree::Punct(crate::Punct::new('\u{3b}',
+                                                                        crate::Spacing::Alone)))].iter().cloned().collect::<crate::TokenStream>()
+}
+const _: () =
+    {
+        extern crate proc_macro;
+        #[rustc_proc_macro_decls]
+        #[allow(deprecated)]
+        static _DECLS: &[proc_macro::bridge::client::ProcMacro] = &[];
+    };
diff --git a/src/test/ui/proc-macro/span-from-proc-macro.rs b/src/test/ui/proc-macro/span-from-proc-macro.rs
new file mode 100644
index 00000000000..ecff2d72587
--- /dev/null
+++ b/src/test/ui/proc-macro/span-from-proc-macro.rs
@@ -0,0 +1,17 @@
+// aux-build:custom-quote.rs
+// aux-build:span-from-proc-macro.rs
+// compile-flags: -Z macro-backtrace
+
+#[macro_use]
+extern crate span_from_proc_macro;
+
+#[error_from_attribute] //~ ERROR cannot find type `MissingType`
+struct ShouldBeRemoved;
+
+#[derive(ErrorFromDerive)] //~ ERROR cannot find type `OtherMissingType`
+struct Kept;
+
+fn main() {
+    error_from_bang!(); //~ ERROR mismatched types
+    other_error_from_bang!(); //~ ERROR cannot find value `my_ident`
+}
diff --git a/src/test/ui/proc-macro/span-from-proc-macro.stderr b/src/test/ui/proc-macro/span-from-proc-macro.stderr
new file mode 100644
index 00000000000..2cbe91afacd
--- /dev/null
+++ b/src/test/ui/proc-macro/span-from-proc-macro.stderr
@@ -0,0 +1,62 @@
+error[E0412]: cannot find type `MissingType` in this scope
+  --> $DIR/auxiliary/span-from-proc-macro.rs:37:20
+   |
+LL | pub fn error_from_attribute(_args: TokenStream, _input: TokenStream) -> TokenStream {
+   | ----------------------------------------------------------------------------------- in this expansion of procedural macro `#[error_from_attribute]`
+...
+LL |             field: MissingType
+   |                    ^^^^^^^^^^^ not found in this scope
+   | 
+  ::: $DIR/span-from-proc-macro.rs:8:1
+   |
+LL | #[error_from_attribute]
+   | ----------------------- in this macro invocation
+
+error[E0412]: cannot find type `OtherMissingType` in this scope
+  --> $DIR/auxiliary/span-from-proc-macro.rs:46:21
+   |
+LL | pub fn error_from_derive(_input: TokenStream) -> TokenStream {
+   | ------------------------------------------------------------ in this expansion of procedural macro `#[derive(ErrorFromDerive)]`
+...
+LL |             Variant(OtherMissingType)
+   |                     ^^^^^^^^^^^^^^^^ not found in this scope
+   | 
+  ::: $DIR/span-from-proc-macro.rs:11:10
+   |
+LL | #[derive(ErrorFromDerive)]
+   |          --------------- in this macro invocation
+
+error[E0425]: cannot find value `my_ident` in this scope
+  --> $DIR/auxiliary/span-from-proc-macro.rs:29:9
+   |
+LL | pub fn other_error_from_bang(_input: TokenStream) -> TokenStream {
+   | ---------------------------------------------------------------- in this expansion of procedural macro `other_error_from_bang!`
+LL |     custom_quote::custom_quote! {
+LL |         my_ident
+   |         ^^^^^^^^ not found in this scope
+   | 
+  ::: $DIR/span-from-proc-macro.rs:16:5
+   |
+LL |     other_error_from_bang!();
+   |     ------------------------- in this macro invocation
+
+error[E0308]: mismatched types
+  --> $DIR/auxiliary/span-from-proc-macro.rs:16:36
+   |
+LL |             let bang_error: bool = 25;
+   |                             ----   ^^ expected `bool`, found integer
+   |                             |
+   |                             expected due to this
+...
+LL | pub fn error_from_bang(_input: TokenStream) -> TokenStream {
+   | ---------------------------------------------------------- in this expansion of procedural macro `error_from_bang!`
+   | 
+  ::: $DIR/span-from-proc-macro.rs:15:5
+   |
+LL |     error_from_bang!();
+   |     ------------------- in this macro invocation
+
+error: aborting due to 4 previous errors
+
+Some errors have detailed explanations: E0308, E0412, E0425.
+For more information about an error, try `rustc --explain E0308`.