| Age | Commit message (Collapse) | Author | Lines |
|---|
|
Delay unmatched delimiter errors until after the parser has run to
deduplicate them when parsing and attempt recovering intelligently.
|
|
|
|
`TokenStream` is currently recursive in *two* ways:
- the `TokenTree` variant contains a `ThinTokenStream`, which can
contain a `TokenStream`;
- the `TokenStream` variant contains a `Vec<TokenStream>`.
The latter is not necessary and causes significant complexity. This
commit replaces it with the simpler `Vec<(TokenTree, IsJoint)>`.
This reduces complexity significantly. In particular, `StreamCursor` is
eliminated, and `Cursor` becomes much simpler, consisting now of just a
`TokenStream` and an index.
The commit also removes the `Extend` impl for `TokenStream`, because it
is only used in tests. (The commit also removes those tests.)
Overall, the commit reduces the number of lines of code by almost 200.
|
|
|
|
Remove `TokenStream::JointTree`.
This is done by adding a new `IsJoint` field to `TokenStream::Tree`,
which simplifies a lot of `match` statements. And likewise for
`CursorKind`.
The commit also adds a new method `TokenTree:stream()` which can replace
a choice between `.into()` and `.joint()`.
|
|
This is done by adding a new `IsJoint` field to `TokenStream::Tree`,
which simplifies a lot of `match` statements. And likewise for
`CursorKind`.
The commit also adds a new method `TokenTree:stream()` which can replace
a choice between `.into()` and `.joint()`.
|
|
Do not point at delim spans for complete correct blocks
Fix #56834.
|
|
|
|
`TokenStream::new` is a better name for the former, and the latter is
now just equivalent to `TokenStream::Stream`.
|
|
Because it's an extra type layer that doesn't really help; in a couple
of places it actively gets in the way, and overall removing it makes the
code nicer. It does, however, move `tokenstream::TokenTree` further away
from the `TokenTree` in `quote.rs`.
More importantly, this change reduces the size of `TokenStream` from 48
bytes to 40 bytes on x86-64, which is enough to slightly reduce
instruction counts on numerous benchmarks, the best by 1.5%.
Note that `open_tt` and `close_tt` have gone from being methods on
`Delimited` to associated methods of `TokenTree`.
|
|
|
|
proc_macro::Group::span_open and span_close
Before this addition, every delimited group like `(`...`)` `[`...`]` `{`...`}` has only a single Span that covers the full source location from opening delimiter to closing delimiter. This makes it impossible for a procedural macro to trigger an error pointing to just the opening or closing delimiter. The Rust compiler does not seem to have the same limitation:
```rust
mod m {
type T =
}
```
```console
error: expected type, found `}`
--> src/main.rs:3:1
|
3 | }
| ^
```
On that same input, a procedural macro would be forced to trigger the error on the last token inside the block, on the entire block, or on the next token after the block, none of which is really what you want for an error like above.
This commit adds `group.span_open()` and `group.span_close()` which access the Span associated with just the opening delimiter and just the closing delimiter of the group. Relevant to Syn as we implement real error messages for when parsing fails in a procedural macro: https://github.com/dtolnay/syn/issues/476.
```diff
impl Group {
fn span(&self) -> Span;
+ fn span_open(&self) -> Span;
+ fn span_close(&self) -> Span;
}
```
Fixes #48187
r? @alexcrichton
|
|
|
|
|
|
* When encountering EOF, point at the last opening brace that does not
have the same indentation level as its close delimiter.
* When encountering the wrong type of close delimiter, point at the
likely correct open delimiter to give a better idea of what went
wrong.
|
|
|
|
Tweak unclosed delimiter parser error
|
|
|
|
|
|
|
|
|
|
This commit fixes `StringReader`'s parsing of tokens which have been stringified
through procedural macros. Whether or not a token tree is joint is defined by
span information, but when working with procedural macros these spans are often
dummy and/or overridden which means that they end up considering all operators
joint if they can!
The fix here is to track the raw source span as opposed to the overridden span.
With this information we can more accurately classify `Punct` structs as either
joint or not.
Closes #50700
|
|
|
|
Like #43008 (f668999), but _much more aggressive_.
|
|
|
|
|
|
|
|
|
