Implement span quoting for proc-macros

This PR implements span quoting, allowing proc-macros to produce spans
pointing *into their own crate*. This is used by the unstable
`proc_macro::quote!` macro, allowing us to get error messages like this:

```
error[E0412]: cannot find type `MissingType` in this scope
  --> $DIR/auxiliary/span-from-proc-macro.rs:37:20
   |
LL | pub fn error_from_attribute(_args: TokenStream, _input: TokenStream) -> TokenStream {
   | ----------------------------------------------------------------------------------- in this expansion of procedural macro `#[error_from_attribute]`
...
LL |             field: MissingType
   |                    ^^^^^^^^^^^ not found in this scope
   |
  ::: $DIR/span-from-proc-macro.rs:8:1
   |
LL | #[error_from_attribute]
   | ----------------------- in this macro invocation
```

Here, `MissingType` occurs inside the implementation of the proc-macro
`#[error_from_attribute]`. Previosuly, this would always result in a
span pointing at `#[error_from_attribute]`

This will make many proc-macro-related error message much more useful -
when a proc-macro generates code containing an error, users will get an
error message pointing directly at that code (within the macro
definition), instead of always getting a span pointing at the macro
invocation site.

This is implemented as follows:
* When a proc-macro crate is being *compiled*, it causes the `quote!`
  macro to get run. This saves all of the sapns in the input to `quote!`
  into the metadata of *the proc-macro-crate* (which we are currently
  compiling). The `quote!` macro then expands to a call to
  `proc_macro::Span::recover_proc_macro_span(id)`, where `id` is an
opaque identifier for the span in the crate metadata.
* When the same proc-macro crate is *run* (e.g. it is loaded from disk
  and invoked by some consumer crate), the call to
`proc_macro::Span::recover_proc_macro_span` causes us to load the span
from the proc-macro crate's metadata. The proc-macro then produces a
`TokenStream` containing a `Span` pointing into the proc-macro crate
itself.

The recursive nature of 'quote!' can be difficult to understand at
first. The file `src/test/ui/proc-macro/quote-debug.stdout` shows
the output of the `quote!` macro, which should make this eaier to
understand.

This PR also supports custom quoting spans in custom quote macros (e.g.
the `quote` crate). All span quoting goes through the
`proc_macro::quote_span` method, which can be called by a custom quote
macro to perform span quoting. An example of this usage is provided in
`src/test/ui/proc-macro/auxiliary/custom-quote.rs`

Custom quoting currently has a few limitations:

In order to quote a span, we need to generate a call to
`proc_macro::Span::recover_proc_macro_span`. However, proc-macros
support renaming the `proc_macro` crate, so we can't simply hardcode
this path. Previously, the `quote_span` method used the path
`crate::Span` - however, this only works when it is called by the
builtin `quote!` macro in the same crate. To support being called from
arbitrary crates, we need access to the name of the `proc_macro` crate
to generate a path. This PR adds an additional argument to `quote_span`
to specify the name of the `proc_macro` crate. Howver, this feels kind
of hacky, and we may want to change this before stabilizing anything
quote-related.

Additionally, using `quote_span` currently requires enabling the
`proc_macro_internals` feature. The builtin `quote!` macro
has an `#[allow_internal_unstable]` attribute, but this won't work for
custom quote implementations. This will likely require some additional
tricks to apply `allow_internal_unstable` to the span of
`proc_macro::Span::recover_proc_macro_span`.

3 files changed, 88 insertions, 12 deletions

diff --git a/compiler/rustc_expand/src/base.rs b/compiler/rustc_expand/src/base.rs
index 59505842816..0e966d0bf79 100644
--- a/compiler/rustc_expand/src/base.rs
+++ b/compiler/rustc_expand/src/base.rs
@@ -14,7 +14,7 @@ use rustc_lint_defs::builtin::PROC_MACRO_BACK_COMPAT;
 use rustc_lint_defs::BuiltinLintDiagnostics;
 use rustc_parse::{self, nt_to_tokenstream, parser, MACRO_ARGUMENTS};
 use rustc_session::{parse::ParseSess, Limit, Session};
-use rustc_span::def_id::DefId;
+use rustc_span::def_id::{CrateNum, DefId};
 use rustc_span::edition::Edition;
 use rustc_span::hygiene::{AstPass, ExpnData, ExpnId, ExpnKind};
 use rustc_span::source_map::SourceMap;
@@ -810,8 +810,16 @@ impl SyntaxExtension {
         descr: Symbol,
         macro_def_id: Option<DefId>,
     ) -> ExpnData {
+        use SyntaxExtensionKind::*;
+        let proc_macro = match self.kind {
+            // User-defined proc macro
+            Bang(..) | Attr(..) | Derive(..) => true,
+            // Consider everthing else to be not a proc
+            // macro for diagnostic purposes
+            LegacyBang(..) | LegacyAttr(..) | NonMacroAttr { .. } | LegacyDerive(..) => false,
+        };
         ExpnData::new(
-            ExpnKind::Macro(self.macro_kind(), descr),
+            ExpnKind::Macro { kind: self.macro_kind(), name: descr, proc_macro },
             parent,
             call_site,
             self.span,
@@ -873,6 +881,10 @@ pub trait ResolverExpand {
     fn take_derive_resolutions(&mut self, expn_id: ExpnId) -> Option<DeriveResolutions>;
     /// Path resolution logic for `#[cfg_accessible(path)]`.
     fn cfg_accessible(&mut self, expn_id: ExpnId, path: &ast::Path) -> Result<bool, Indeterminate>;
+
+    /// Decodes the proc-macro quoted span in the specified crate, with the specified id.
+    /// No caching is performed.
+    fn get_proc_macro_quoted_span(&self, krate: CrateNum, id: usize) -> Span;
 }
 
 #[derive(Clone, Default)]
diff --git a/compiler/rustc_expand/src/proc_macro.rs b/compiler/rustc_expand/src/proc_macro.rs
index 3f84979ac05..494b3fb61ee 100644
--- a/compiler/rustc_expand/src/proc_macro.rs
+++ b/compiler/rustc_expand/src/proc_macro.rs
@@ -9,12 +9,14 @@ use rustc_data_structures::sync::Lrc;
 use rustc_errors::ErrorReported;
 use rustc_parse::nt_to_tokenstream;
 use rustc_parse::parser::ForceCollect;
+use rustc_span::def_id::CrateNum;
 use rustc_span::{Span, DUMMY_SP};
 
 const EXEC_STRATEGY: pm::bridge::server::SameThread = pm::bridge::server::SameThread;
 
 pub struct BangProcMacro {
     pub client: pm::bridge::client::Client<fn(pm::TokenStream) -> pm::TokenStream>,
+    pub krate: CrateNum,
 }
 
 impl base::ProcMacro for BangProcMacro {
@@ -24,7 +26,7 @@ impl base::ProcMacro for BangProcMacro {
         span: Span,
         input: TokenStream,
     ) -> Result<TokenStream, ErrorReported> {
-        let server = proc_macro_server::Rustc::new(ecx);
+        let server = proc_macro_server::Rustc::new(ecx, self.krate);
         self.client.run(&EXEC_STRATEGY, server, input, ecx.ecfg.proc_macro_backtrace).map_err(|e| {
             let mut err = ecx.struct_span_err(span, "proc macro panicked");
             if let Some(s) = e.as_str() {
@@ -38,6 +40,7 @@ impl base::ProcMacro for BangProcMacro {
 
 pub struct AttrProcMacro {
     pub client: pm::bridge::client::Client<fn(pm::TokenStream, pm::TokenStream) -> pm::TokenStream>,
+    pub krate: CrateNum,
 }
 
 impl base::AttrProcMacro for AttrProcMacro {
@@ -48,7 +51,7 @@ impl base::AttrProcMacro for AttrProcMacro {
         annotation: TokenStream,
         annotated: TokenStream,
     ) -> Result<TokenStream, ErrorReported> {
-        let server = proc_macro_server::Rustc::new(ecx);
+        let server = proc_macro_server::Rustc::new(ecx, self.krate);
         self.client
             .run(&EXEC_STRATEGY, server, annotation, annotated, ecx.ecfg.proc_macro_backtrace)
             .map_err(|e| {
@@ -64,6 +67,7 @@ impl base::AttrProcMacro for AttrProcMacro {
 
 pub struct ProcMacroDerive {
     pub client: pm::bridge::client::Client<fn(pm::TokenStream) -> pm::TokenStream>,
+    pub krate: CrateNum,
 }
 
 impl MultiItemModifier for ProcMacroDerive {
@@ -97,7 +101,7 @@ impl MultiItemModifier for ProcMacroDerive {
             nt_to_tokenstream(&item, &ecx.sess.parse_sess, CanSynthesizeMissingTokens::No)
         };
 
-        let server = proc_macro_server::Rustc::new(ecx);
+        let server = proc_macro_server::Rustc::new(ecx, self.krate);
         let stream =
             match self.client.run(&EXEC_STRATEGY, server, input, ecx.ecfg.proc_macro_backtrace) {
                 Ok(stream) => stream,
diff --git a/compiler/rustc_expand/src/proc_macro_server.rs b/compiler/rustc_expand/src/proc_macro_server.rs
index 1ea26b4eab4..f6645448fcc 100644
--- a/compiler/rustc_expand/src/proc_macro_server.rs
+++ b/compiler/rustc_expand/src/proc_macro_server.rs
@@ -1,4 +1,4 @@
-use crate::base::ExtCtxt;
+use crate::base::{ExtCtxt, ResolverExpand};
 
 use rustc_ast as ast;
 use rustc_ast::token;
@@ -7,6 +7,7 @@ use rustc_ast::token::NtIdent;
 use rustc_ast::tokenstream::{self, CanSynthesizeMissingTokens};
 use rustc_ast::tokenstream::{DelimSpan, Spacing::*, TokenStream, TreeAndSpacing};
 use rustc_ast_pretty::pprust;
+use rustc_data_structures::fx::FxHashMap;
 use rustc_data_structures::sync::Lrc;
 use rustc_errors::Diagnostic;
 use rustc_lint_defs::builtin::PROC_MACRO_BACK_COMPAT;
@@ -14,6 +15,8 @@ use rustc_lint_defs::BuiltinLintDiagnostics;
 use rustc_parse::lexer::nfc_normalize;
 use rustc_parse::{nt_to_tokenstream, parse_stream_from_source_str};
 use rustc_session::parse::ParseSess;
+use rustc_span::def_id::CrateNum;
+use rustc_span::hygiene::ExpnId;
 use rustc_span::hygiene::ExpnKind;
 use rustc_span::symbol::{self, kw, sym, Symbol};
 use rustc_span::{BytePos, FileName, MultiSpan, Pos, RealFileName, SourceFile, Span};
@@ -355,22 +358,34 @@ pub struct Literal {
 }
 
 pub(crate) struct Rustc<'a> {
+    resolver: &'a dyn ResolverExpand,
     sess: &'a ParseSess,
     def_site: Span,
     call_site: Span,
     mixed_site: Span,
     span_debug: bool,
+    krate: CrateNum,
+    expn_id: ExpnId,
+    rebased_spans: FxHashMap<usize, Span>,
 }
 
 impl<'a> Rustc<'a> {
-    pub fn new(cx: &'a ExtCtxt<'_>) -> Self {
+    pub fn new(cx: &'a ExtCtxt<'_>, krate: CrateNum) -> Self {
         let expn_data = cx.current_expansion.id.expn_data();
+        let def_site = cx.with_def_site_ctxt(expn_data.def_site);
+        let call_site = cx.with_call_site_ctxt(expn_data.call_site);
+        let mixed_site = cx.with_mixed_site_ctxt(expn_data.call_site);
+        let sess = cx.parse_sess();
         Rustc {
-            sess: &cx.sess.parse_sess,
-            def_site: cx.with_def_site_ctxt(expn_data.def_site),
-            call_site: cx.with_call_site_ctxt(expn_data.call_site),
-            mixed_site: cx.with_mixed_site_ctxt(expn_data.call_site),
+            resolver: cx.resolver,
+            sess,
+            def_site,
+            call_site,
+            mixed_site,
             span_debug: cx.ecfg.span_debug,
+            krate,
+            expn_id: cx.current_expansion.id,
+            rebased_spans: FxHashMap::default(),
         }
     }
 
@@ -713,6 +728,51 @@ impl server::Span for Rustc<'_> {
     fn source_text(&mut self, span: Self::Span) -> Option<String> {
         self.sess.source_map().span_to_snippet(span).ok()
     }
+    /// Saves the provided span into the metadata of
+    /// *the crate we are currently compiling*, which must
+    /// be a proc-macro crate. This id can be passed to
+    /// `recover_proc_macro_span` when our current crate
+    /// is *run* as a proc-macro.
+    ///
+    /// Let's suppose that we have two crates - `my_client`
+    /// and `my_proc_macro`. The `my_proc_macro` crate
+    /// contains a procedural macro `my_macro`, which
+    /// is implemented as: `quote! { "hello" }`
+    ///
+    /// When we *compile* `my_proc_macro`, we will execute
+    /// the `quote` proc-macro. This will save the span of
+    /// "hello" into the metadata of `my_proc_macro`. As a result,
+    /// the body of `my_proc_macro` (after expansion) will end
+    /// up containg a call that looks like this:
+    /// `proc_macro::Ident::new("hello", proc_macro::Span::recover_proc_macro_span(0))`
+    ///
+    /// where `0` is the id returned by this function.
+    /// When `my_proc_macro` *executes* (during the compilation of `my_client`),
+    /// the call to `recover_proc_macro_span` will load the corresponding
+    /// span from the metadata of `my_proc_macro` (which we have access to,
+    /// since we've loaded `my_proc_macro` from disk in order to execute it).
+    /// In this way, we have obtained a span pointing into `my_proc_macro`
+    fn save_span(&mut self, mut span: Self::Span) -> usize {
+        // Throw away the `SyntaxContext`, since we currently
+        // skip serializing `SyntaxContext`s for proc-macro crates
+        span = span.with_ctxt(rustc_span::SyntaxContext::root());
+        self.sess.save_proc_macro_span(span)
+    }
+    fn recover_proc_macro_span(&mut self, id: usize) -> Self::Span {
+        let resolver = self.resolver;
+        let krate = self.krate;
+        let expn_id = self.expn_id;
+        *self.rebased_spans.entry(id).or_insert_with(|| {
+            let raw_span = resolver.get_proc_macro_quoted_span(krate, id);
+            // Ignore the deserialized `SyntaxContext` entirely.
+            // FIXME: Preserve the macro backtrace from the serialized span
+            // For example, if a proc-macro crate has code like
+            // `macro_one!() -> macro_two!() -> quote!()`, we might
+            // want to 'concatenate' this backtrace with the backtrace from
+            // our current call site.
+            raw_span.with_def_site_ctxt(expn_id)
+        })
+    }
 }
 
 // See issue #74616 for details
@@ -722,7 +782,7 @@ fn ident_name_compatibility_hack(
     rustc: &mut Rustc<'_>,
 ) -> Option<(rustc_span::symbol::Ident, bool)> {
     if let NtIdent(ident, is_raw) = nt {
-        if let ExpnKind::Macro(_, macro_name) = orig_span.ctxt().outer_expn_data().kind {
+        if let ExpnKind::Macro { name: macro_name, .. } = orig_span.ctxt().outer_expn_data().kind {
             let source_map = rustc.sess.source_map();
             let filename = source_map.span_to_filename(orig_span);
             if let FileName::Real(RealFileName::Named(path)) = filename {