| Age | Commit message (Collapse) | Author | Lines |
|---|
|
|
|
Because they're very similar, and this will allow some follow-up
changes.
|
|
|
|
This is a small performance win, alas.
|
|
For alignment with `rust_ast::TokenKind::Eof`. Plus it's a bit faster,
due to less `Option` manipulation in `StringReader::next_token`.
|
|
This is a case where a small amount of repetition results in code that
is faster and easier to read.
|
|
`Cursor` is currently hidden, and the main tokenization path uses
`rustc_lexer::first_token` which involves constructing a new `Cursor`
for every single token, which is weird. Also, `first_token` also can't
handle empty input, so callers have to check for that first.
This commit makes `Cursor` public, so `StringReader` can contain a
`Cursor`, which results in a simpler structure. The commit also changes
`StringReader::advance_token` so it returns an `Option<Token>`,
simplifying the the empty input case.
|
|
`TokenTreesReader` wraps a `StringReader`, but the `into_token_trees`
function obscures this. This commit moves to a more straightforward
control flow.
|
|
The spacing computation is done in two parts. In the first part
`next_token` and `bump` use `Spacing::Alone` to mean "preceded by
whitespace" and `Spacing::Joint` to mean the opposite. In the second
part `parse_token_tree_other` then adjusts the `spacing` value to mean
the usual thing (i.e. "is the following token joinable punctuation?").
This shift in meaning is very confusing and it took me some time to
understand what was going on.
This commit changes the first part to use a bool, and adds some
comments, which makes things much clearer.
|
|
Currently does the "is this a `#!` at the start of the file?" check for
every single token(!)
This commit moves it so it only happens once.
|
|
It has no useful effect.
|
|
by module
|
|
From 72 bytes to 12 bytes (on x86-64).
There are two parts to this:
- Changing various source code offsets from 64-bit to 32-bit. This is
not a problem because the rest of rustc also uses 32-bit source code
offsets. This means `Token` is no longer `Copy` but this causes no
problems.
- Removing the `RawStrError` from `LiteralKind`. Raw string literal
invalidity is now indicated by a `None` value within
`RawStr`/`RawByteStr`, and the new `validate_raw_str` function can be
used to re-lex an invalid raw string literal to get the `RawStrError`.
There is one very small change in behaviour. Previously, if a raw string
literal matched both the `InvalidStarter` and `TooManyHashes` cases,
the latter would override the former. This has now changed, because
`raw_double_quoted_string` now uses `?` and so returns immediately upon
detecting the `InvalidStarter` case. I think this is a slight
improvement to report the earlier-detected error, and it explains the
change in the `test_too_many_hashes` test.
The commit also removes a couple of comments that refer to #77629 and
say that the size of these types don't affect performance. These
comments are wrong, though the performance effect is small.
|
|
|
|
It not needed, always being set to the end of the text.
|
|
Avoid doing stuff until it's necessary.
|
|
|
|
Change `span_suggestion` (and variants) to take `impl ToString` rather
than `String` for the suggested code, as this simplifies the
requirements on the diagnostic derive.
Signed-off-by: David Wood <david.wood@huawei.com>
|
|
|
|
r=petrochenkov
Improve diagnostics for unterminated nested block comment
close #95283
(This is my first time try to messing around with rust compiler and might get a lot of things wrong... :bow: )
|
|
|
|
|
|
|
|
Reduce max hash in raw strings from u16 to u8
[Relevant discussion](https://rust-lang.zulipchat.com/#narrow/stream/237824-t-lang.2Fdoc/topic/Max.20raw.20string.20delimiters)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
In the lexer, consider emojis to be valid identifiers and reject
them later to avoid knock down parse errors.
|
|
|
|
|
|
|
|
|
|
Address CVE-2021-42574.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Co-authored-by: Niko Matsakis <niko@alum.mit.edu>
|
|
|
|
|
|
|
|
This commit denies any identifiers immediately followed by
one of three tokens `"`, `'` or `#`, which is stricter than
the requirements of RFC 3101 but may be necessary according
to the discussion at [Zulip].
[Zulip]: https://rust-lang.zulipchat.com/#narrow/stream/268952-edition-2021/topic/reserved.20prefixes/near/238470099
|
|
It had no callers which didn't immediately call `raise()`, and this
unifies the behavior with `Session`.
|
