Rollup merge of #78610 - petrochenkov:nostriptok, r=Aaron1011

Do not remove tokens before AST json serialization

`TokenStripper` is error-prone and introduces one more use of `MutVisitor`.
It's much simpler to treat serialization as just one more place that wants lazy token stream to turn into a real token stream.
Also, no code is better than more code, in general.
r? @Aaron1011

(I also merged tests for `TokenStripper` ICEs into one.)

1 files changed, 3 insertions, 2 deletions

diff --git a/compiler/rustc_ast/src/tokenstream.rs b/compiler/rustc_ast/src/tokenstream.rs
index b53acb97aeb..98da20af8f6 100644
--- a/compiler/rustc_ast/src/tokenstream.rs
+++ b/compiler/rustc_ast/src/tokenstream.rs
@@ -153,8 +153,9 @@ impl fmt::Debug for LazyTokenStream {
 }
 
 impl<S: Encoder> Encodable<S> for LazyTokenStream {
-    fn encode(&self, _s: &mut S) -> Result<(), S::Error> {
-        panic!("Attempted to encode LazyTokenStream");
+    fn encode(&self, s: &mut S) -> Result<(), S::Error> {
+        // Used by AST json printing.
+        Encodable::encode(&self.create_token_stream(), s)
     }
 }