rust/compiler/rustc_parse/src/parser/attr_wrapper.rs, branch 1.89.0

Use `ThinVec` to shrink `LazyAttrTokenStreamInner`.

2025-04-29T21:12:09+00:00

Simplify `LazyAttrTokenStream`.

2025-04-29T21:10:56+00:00

This commit does the following. - Changes it from `Lrc>` to `Lrc`. - Reworks `LazyAttrTokenStreamImpl` as `LazyAttrTokenStreamInner`, which is a two-variant enum. - Removes the `ToAttrTokenStream` trait and the two impls of it. The recursion limit must be increased in some crates otherwise rustdoc aborts.

Move various token stream things from `rustc_parse` to `rustc_ast`.

2025-04-29T02:14:27+00:00

Specifically: `TokenCursor`, `TokenTreeCursor`, `LazyAttrTokenStreamImpl`, `FlatToken`, `make_attr_token_stream`, `ParserRange`, `NodeRange`. `ParserReplacement`, and `NodeReplacement`. These are all related to token streams, rather than actual parsing. This will facilitate the simplifications in the next commit.

Remove `token::{Open,Close}Delim`.

2025-04-20T21:35:56+00:00

By replacing them with `{Open,Close}{Param,Brace,Bracket,Invisible}`. PR #137902 made `ast::TokenKind` more like `lexer::TokenKind` by replacing the compound `BinOp{,Eq}(BinOpToken)` variants with fieldless variants `Plus`, `Minus`, `Star`, etc. This commit does a similar thing with delimiters. It also makes `ast::TokenKind` more similar to `parser::TokenType`. This requires a few new methods: - `TokenKind::is_{,open_,close_}delim()` replace various kinds of pattern matches. - `Delimiter::as_{open,close}_token_kind` are used to convert `Delimiter` values to `TokenKind`. Despite these additions, it's a net reduction in lines of code. This is because e.g. `token::OpenParen` is so much shorter than `token::OpenDelim(Delimiter::Parenthesis)` that many multi-line forms reduce to single line forms. And many places where the number of lines doesn't change are still easier to read, just because the names are shorter, e.g.: ``` - } else if self.token != token::CloseDelim(Delimiter::Brace) { + } else if self.token != token::CloseBrace { ```

Impl `Copy` for `Token` and `TokenKind`.

2025-04-02T05:16:49+00:00

Rustfmt

2025-02-08T22:12:13+00:00

Introduce `InvisibleOrigin` on invisible delimiters.

2024-11-20T21:16:54+00:00

It's not used meaningfully yet, but will be needed to get rid of interpolated tokens.

Revert "Avoid nested replacement ranges" from #129346.

2024-11-04T04:57:35+00:00

It caused a test regression in the `cfg_eval.rs` crate. (The bugfix in #129346 was in a different commit; this commit was just a code simplification.)

Rollup merge of #130551 - nnethercote:fix-break-last-token, r=petrochenkov

2024-09-23T14:54:44+00:00

Fix `break_last_token`. It currently doesn't handle the three-char tokens `>>=` and `<<=` correctly. These can be broken twice, resulting in three individual tokens. This is a latent bug that currently doesn't cause any problems, but does cause problems for #124141, because that PR increases the usage of lazy token streams. r? `@petrochenkov`

Fix `break_last_token`.

2024-09-22T23:14:30+00:00

It currently doesn't handle the three-char tokens `>>=` and `<<=` correctly. These can be broken twice, resulting in three individual tokens. This is a latent bug that currently doesn't cause any problems, but does cause problems for #124141, because that PR increases the usage of lazy token streams.