Auto merge of #106505 - Nilstrieb:format-args-string-literal-episode-2, r=petrochenkov

Properly allow macro expanded `format_args` invocations to uses captures

Originally, this was kinda half-allowed. There were some primitive checks in place that looked at the span to see whether the input was likely a literal. These "source literal" checks are needed because the spans created during `format_args` parsing only make sense when it is indeed a literal that was written in the source code directly.

This is orthogonal to the restriction that the first argument must be a "direct literal", not being exanpanded from macros. This restriction was imposed by [RFC 2795] on the basis of being too confusing. But this was only concerned with the argument of the invocation being a literal, not whether it was a source literal (maybe in spirit it meant it being a source literal, this is not clear to me).

Since the original check only really cared about source literals (which is good enough to deny the `format_args!(concat!())` example), macros expanding to `format_args` invocations were able to use implicit captures if they spanned the string in a way that lead back to a source string.

The "source literal" checks were not strict enough and caused ICEs in certain cases (see #106191). So I tightened it up in #106195 to really only work if it's a direct source literal.

This caused the `indoc` crate to break. `indoc` transformed the source literal by removing whitespace, which made it not a "source literal" anymore (which is required to fix the ICE). But since `indoc` spanned the literal in ways that made the old check think that it's a literal, it was able to use implicit captures (which is useful and nice for the users of `indoc`).

This commit properly seperates the previously introduced concepts of "source literal" and "direct literal" and therefore allows `indoc` invocations, which don't create "source literals" to use implicit captures again.

Fixes #106191

[RFC 2795]: https://rust-lang.github.io/rfcs/2795-format-args-implicit-identifiers.html#macro-hygiene

9 files changed, 144 insertions, 19 deletions

diff --git a/tests/ui/fmt/auxiliary/format-string-proc-macro.rs b/tests/ui/fmt/auxiliary/format-string-proc-macro.rs
index 1b7ef93f41d..0c39ade721f 100644
--- a/tests/ui/fmt/auxiliary/format-string-proc-macro.rs
+++ b/tests/ui/fmt/auxiliary/format-string-proc-macro.rs
@@ -28,25 +28,41 @@ pub fn err_with_input_span(input: TokenStream) -> TokenStream {
     TokenStream::from(TokenTree::Literal(lit))
 }
 
+fn build_format(args: impl Into<TokenStream>) -> TokenStream {
+    TokenStream::from_iter([
+        TokenTree::from(Ident::new("format", Span::call_site())),
+        TokenTree::from(Punct::new('!', Spacing::Alone)),
+        TokenTree::from(Group::new(Delimiter::Parenthesis, args.into())),
+    ])
+}
 
 #[proc_macro]
 pub fn respan_to_invalid_format_literal(input: TokenStream) -> TokenStream {
     let mut s = Literal::string("{");
     s.set_span(input.into_iter().next().unwrap().span());
-    TokenStream::from_iter([
-        TokenTree::from(Ident::new("format", Span::call_site())),
-        TokenTree::from(Punct::new('!', Spacing::Alone)),
-        TokenTree::from(Group::new(Delimiter::Parenthesis, TokenTree::from(s).into())),
-    ])
+
+    build_format(TokenTree::from(s))
 }
 
 #[proc_macro]
 pub fn capture_a_with_prepended_space_preserve_span(input: TokenStream) -> TokenStream {
     let mut s = Literal::string(" {a}");
     s.set_span(input.into_iter().next().unwrap().span());
-    TokenStream::from_iter([
-        TokenTree::from(Ident::new("format", Span::call_site())),
-        TokenTree::from(Punct::new('!', Spacing::Alone)),
-        TokenTree::from(Group::new(Delimiter::Parenthesis, TokenTree::from(s).into())),
-    ])
+
+    build_format(TokenTree::from(s))
+}
+
+#[proc_macro]
+pub fn format_args_captures(_: TokenStream) -> TokenStream {
+    r#"{ let x = 5; format!("{x}") }"#.parse().unwrap()
+}
+
+#[proc_macro]
+pub fn bad_format_args_captures(_: TokenStream) -> TokenStream {
+    r#"{ let x = 5; format!(concat!("{x}")) }"#.parse().unwrap()
+}
+
+#[proc_macro]
+pub fn identity_pm(input: TokenStream) -> TokenStream {
+    input
 }
diff --git a/tests/ui/fmt/format-args-capture-first-literal-is-macro.rs b/tests/ui/fmt/format-args-capture-first-literal-is-macro.rs
new file mode 100644
index 00000000000..bf5c0dcb54d
--- /dev/null
+++ b/tests/ui/fmt/format-args-capture-first-literal-is-macro.rs
@@ -0,0 +1,21 @@
+// aux-build:format-string-proc-macro.rs
+
+#[macro_use]
+extern crate format_string_proc_macro;
+
+macro_rules! identity_mbe {
+    ($tt:tt) => {
+        $tt
+        //~^ ERROR there is no argument named `a`
+    };
+}
+
+fn main() {
+    let a = 0;
+
+    format!(identity_pm!("{a}"));
+    //~^ ERROR there is no argument named `a`
+    format!(identity_mbe!("{a}"));
+    format!(concat!("{a}"));
+    //~^ ERROR there is no argument named `a`
+}
diff --git a/tests/ui/fmt/format-args-capture-first-literal-is-macro.stderr b/tests/ui/fmt/format-args-capture-first-literal-is-macro.stderr
new file mode 100644
index 00000000000..4cf3afad7b8
--- /dev/null
+++ b/tests/ui/fmt/format-args-capture-first-literal-is-macro.stderr
@@ -0,0 +1,30 @@
+error: there is no argument named `a`
+  --> $DIR/format-args-capture-first-literal-is-macro.rs:16:26
+   |
+LL |     format!(identity_pm!("{a}"));
+   |                          ^^^^^
+   |
+   = note: did you intend to capture a variable `a` from the surrounding scope?
+   = note: to avoid ambiguity, `format_args!` cannot capture variables when the format string is expanded from a macro
+
+error: there is no argument named `a`
+  --> $DIR/format-args-capture-first-literal-is-macro.rs:8:9
+   |
+LL |         $tt
+   |         ^^^
+   |
+   = note: did you intend to capture a variable `a` from the surrounding scope?
+   = note: to avoid ambiguity, `format_args!` cannot capture variables when the format string is expanded from a macro
+
+error: there is no argument named `a`
+  --> $DIR/format-args-capture-first-literal-is-macro.rs:19:13
+   |
+LL |     format!(concat!("{a}"));
+   |             ^^^^^^^^^^^^^^
+   |
+   = note: did you intend to capture a variable `a` from the surrounding scope?
+   = note: to avoid ambiguity, `format_args!` cannot capture variables when the format string is expanded from a macro
+   = note: this error originates in the macro `concat` (in Nightly builds, run with -Z macro-backtrace for more info)
+
+error: aborting due to 3 previous errors
+
diff --git a/tests/ui/fmt/format-args-capture-from-pm-first-arg-macro.rs b/tests/ui/fmt/format-args-capture-from-pm-first-arg-macro.rs
new file mode 100644
index 00000000000..f67edf5e167
--- /dev/null
+++ b/tests/ui/fmt/format-args-capture-from-pm-first-arg-macro.rs
@@ -0,0 +1,8 @@
+// aux-build:format-string-proc-macro.rs
+
+extern crate format_string_proc_macro;
+
+fn main() {
+    format_string_proc_macro::bad_format_args_captures!();
+    //~^ ERROR there is no argument named `x`
+}
diff --git a/tests/ui/fmt/format-args-capture-from-pm-first-arg-macro.stderr b/tests/ui/fmt/format-args-capture-from-pm-first-arg-macro.stderr
new file mode 100644
index 00000000000..bb6a14d88b3
--- /dev/null
+++ b/tests/ui/fmt/format-args-capture-from-pm-first-arg-macro.stderr
@@ -0,0 +1,12 @@
+error: there is no argument named `x`
+  --> $DIR/format-args-capture-from-pm-first-arg-macro.rs:6:5
+   |
+LL |     format_string_proc_macro::bad_format_args_captures!();
+   |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+   |
+   = note: did you intend to capture a variable `x` from the surrounding scope?
+   = note: to avoid ambiguity, `format_args!` cannot capture variables when the format string is expanded from a macro
+   = note: this error originates in the macro `concat` (in Nightly builds, run with -Z macro-backtrace for more info)
+
+error: aborting due to previous error
+
diff --git a/tests/ui/fmt/format-args-capture-issue-106408.rs b/tests/ui/fmt/format-args-capture-issue-106408.rs
new file mode 100644
index 00000000000..0fd195416ee
--- /dev/null
+++ b/tests/ui/fmt/format-args-capture-issue-106408.rs
@@ -0,0 +1,10 @@
+// check-pass
+// aux-build:format-string-proc-macro.rs
+
+extern crate format_string_proc_macro;
+
+fn main() {
+    // While literal macros like `format_args!(concat!())` are not supposed to work with implicit
+    // captures, it should work if the whole invocation comes from a macro expansion (#106408).
+    format_string_proc_macro::format_args_captures!();
+}
diff --git a/tests/ui/fmt/format-args-capture-macro-hygiene-pass.rs b/tests/ui/fmt/format-args-capture-macro-hygiene-pass.rs
new file mode 100644
index 00000000000..7553fcc4e01
--- /dev/null
+++ b/tests/ui/fmt/format-args-capture-macro-hygiene-pass.rs
@@ -0,0 +1,16 @@
+// run-pass
+
+macro_rules! format_mbe {
+    ($tt:tt) => {
+        {
+            #[allow(unused_variables)]
+            let a = 123;
+            format!($tt)
+        }
+    };
+}
+
+fn main() {
+    let a = 0;
+    assert_eq!(format_mbe!("{a}"), "0");
+}
diff --git a/tests/ui/fmt/respanned-literal-issue-106191.rs b/tests/ui/fmt/respanned-literal-issue-106191.rs
index 5a18983a3fa..44642a10fc0 100644
--- a/tests/ui/fmt/respanned-literal-issue-106191.rs
+++ b/tests/ui/fmt/respanned-literal-issue-106191.rs
@@ -1,15 +1,10 @@
 // aux-build:format-string-proc-macro.rs
-// check-fail
-// known-bug: #106191
-// unset-rustc-env:RUST_BACKTRACE
-// had to be reverted
-// error-pattern:unexpectedly panicked
-// failure-status:101
-// dont-check-compiler-stderr
 
 extern crate format_string_proc_macro;
 
 fn main() {
     format_string_proc_macro::respan_to_invalid_format_literal!("¡");
+    //~^ ERROR invalid format string: expected `'}'` but string was terminated
     format_args!(r#concat!("¡        {"));
+    //~^ ERROR invalid format string: expected `'}'` but string was terminated
 }
diff --git a/tests/ui/fmt/respanned-literal-issue-106191.stderr b/tests/ui/fmt/respanned-literal-issue-106191.stderr
index 16717f42253..73a3af65a38 100644
--- a/tests/ui/fmt/respanned-literal-issue-106191.stderr
+++ b/tests/ui/fmt/respanned-literal-issue-106191.stderr
@@ -1,2 +1,19 @@
-                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  query stack during panic:
-end of query stack
+error: invalid format string: expected `'}'` but string was terminated
+  --> $DIR/respanned-literal-issue-106191.rs:6:65
+   |
+LL |     format_string_proc_macro::respan_to_invalid_format_literal!("¡");
+   |                                                                 ^^^ expected `'}'` in format string
+   |
+   = note: if you intended to print `{`, you can escape it using `{{`
+
+error: invalid format string: expected `'}'` but string was terminated
+  --> $DIR/respanned-literal-issue-106191.rs:8:18
+   |
+LL |     format_args!(r#concat!("¡        {"));
+   |                  ^^^^^^^^^^^^^^^^^^^^^^^ expected `'}'` in format string
+   |
+   = note: if you intended to print `{`, you can escape it using `{{`
+   = note: this error originates in the macro `concat` (in Nightly builds, run with -Z macro-backtrace for more info)
+
+error: aborting due to 2 previous errors
+