summary refs log tree commit diff
path: root/compiler/rustc_lexer/src/cursor.rs
AgeCommit message (Collapse)AuthorLines
2022-08-01Shrink `Token`.Nicholas Nethercote-2/+2
From 72 bytes to 12 bytes (on x86-64). There are two parts to this: - Changing various source code offsets from 64-bit to 32-bit. This is not a problem because the rest of rustc also uses 32-bit source code offsets. This means `Token` is no longer `Copy` but this causes no problems. - Removing the `RawStrError` from `LiteralKind`. Raw string literal invalidity is now indicated by a `None` value within `RawStr`/`RawByteStr`, and the new `validate_raw_str` function can be used to re-lex an invalid raw string literal to get the `RawStrError`. There is one very small change in behaviour. Previously, if a raw string literal matched both the `InvalidStarter` and `TooManyHashes` cases, the latter would override the former. This has now changed, because `raw_double_quoted_string` now uses `?` and so returns immediately upon detecting the `InvalidStarter` case. I think this is a slight improvement to report the earlier-detected error, and it explains the change in the `test_too_many_hashes` test. The commit also removes a couple of comments that refer to #77629 and say that the size of these types don't affect performance. These comments are wrong, though the performance effect is small.
2021-12-01Replace `nth_char(0)` with `next()` in `cursor.first()`Julian Wollersberger-12/+21
and optimize the iterator returned by `tokenize(). This improves lexer performance by 35%
2021-01-07Return EOF_CHAR constant instead of magic char.Hanzhen Liang-1/+1
2020-08-30mv compiler to compiler/mark-0/+84