about summary refs log tree commit diff
path: root/compiler/rustc_lexer/src
AgeCommit message (Collapse)AuthorLines
2023-12-13Rename some unescaping functions.Nicholas Nethercote-9/+7
`unescape_raw_str_or_raw_byte_str` only does checking, no unescaping. And it also now handles C string literals. `unescape_raw_str` is used for all the non-raw strings.
2023-12-13Adjust `Mode::is_unicode_escape_disallowed`.Nicholas Nethercote-2/+3
Some cases are unreachable.
2023-12-09Tweak `Mode`.Nicholas Nethercote-34/+38
- Add `use Mode::*` to avoid all the qualifiers. - Reorder the variants. The existing order makes no particular sense, which has bugged me for some time. I've chosen an order that makes sense to me.
2023-12-09Remove an unnecessary `into`.Nicholas Nethercote-1/+1
2023-12-09Eliminate `is_byte: bool` args in unescaping code.Nicholas Nethercote-21/+19
These don't really make sense since C string literals were added. This commit removes them in favour for `mode: Mode` args. `ascii_check` still has a `characters_should_be_ascii: bool` arg. Also, `characters_should_be_ascii` is renamed to be shorter.
2023-12-09Remove explicit `\n` and `\t` handling in `unescape_str_common`.Nicholas Nethercote-2/+0
The fallback `_` case works for these chars, no need to treat them specially.
2023-12-09Add some useful comments.Nicholas Nethercote-4/+8
2023-12-07Fix an out-of-date comment.Nicholas Nethercote-1/+1
2023-12-07Identify impossible cases in `ascii_escapes_should_be_ascii`.Nicholas Nethercote-2/+3
Raw strings (of all kinds) don't support escapes, so this function should never be called on them.
2023-12-07De-pub some functions.Nicholas Nethercote-3/+3
2023-07-29Update lexer emoji diagnostics to Unicode 15.0Charles Lew-7/+4
2023-07-25Auto merge of #113476 - fee1-dead-contrib:c-str-lit, r=petrochenkovbors-0/+11
Reimplement C-str literals This reverts #113334, cc `@fmease.` While converting lexer tokens to ast Tokens in `rustc_parse`, we check the edition of the span of the token. If the edition < 2021, we split the token into two, one being the identifier and other being the str literal.
2023-07-23fix some clippy::style findingsMatthias Krüger-1/+1
comparison_to_empty iter_nth_zero for_kv_map manual_next_back redundant_pattern
2023-07-23reimplement C string literalsDeadbeef-0/+11
2023-07-05Revert the lexing of c_str_literalsLeón Orell Valerian Liehr-7/+0
2023-05-15Don't try to eat non-existent decimal digits.Nicholas Nethercote-2/+6
After seeing a `0`, if it's followed by any of `[0-9]`, `_`, `.`, `e`, or `E`, we consume all the digits. But in the `.`, `e` and `E` cases this is pointless because we know there aren't any digits.
2023-05-15Make `Cursor::number` less DRY.Nicholas Nethercote-10/+10
A tiny bit of repetition makes this easier to read, and avoids a test on the "Not a base prefix" match arm.
2023-05-02address commentsDeadbeef-1/+1
2023-05-02refactor unescapeDeadbeef-30/+61
2023-05-02fix TODO commentsDeadbeef-57/+51
2023-05-02initial step towards implementing C string literalsDeadbeef-78/+151
2023-04-10Revert "Don't recover lifetimes/labels containing emojis as character literals"Michael Goulet-34/+11
Reverts PR #108031 Fixes (doesnt close until beta backported) #109746 This reverts commit e3f9db5fc319c6d8eee5d47d216ea6a426070c41. This reverts commit 98b82aedba3f3f581e89df54352914b27f42c6f7. This reverts commit 380fa264132ad481e73cbbf0f3a0feefd99a1d78.
2023-03-09fix(lexer): not skipped whitespace warning for '\x0c'bohan-3/+3
2023-03-03Rustdoc-ify LiteralKind noteest31-5/+7
2023-02-14Update lexer lifetime test许杰友 Jieyou Xu (Joe)-1/+1
2023-02-14Don't recover lifetimes/labels containing emojis as character literals许杰友 Jieyou Xu (Joe)-10/+33
Note that at the time of this commit, `unic-emoji-char` seems to have data tables only up to Unicode 5.0, but Unicode is already newer than this. A newer emoji such as `🥺` will not be recognized as an emoji but older emojis such as `🐱` will.
2023-01-17Remove double spaces after dots in commentsMaybe Waffle-2/+2
2022-12-27Recover `fn` keyword as `Fn` trait in boundsMichael Goulet-1/+1
2022-12-12minor code cleanupsMatthias Krüger-2/+1
2022-12-10compiler: remove unnecessary imports and qualified pathsKaDiWa-1/+0
2022-11-24Auto merge of #104507 - WaffleLapkin:asderefsyou, r=wesleywiserbors-4/+2
Use `as_deref` in compiler (but only where it makes sense) This simplifies some code :3 (there are some changes that are not exacly `as_deref`, but more like "clever `Option`/`Result` method use")
2022-11-16Use `as_deref` in compiler (but only where it makes sense)Maybe Waffle-4/+2
2022-11-16Use `token::Lit` in `ast::ExprKind::Lit`.Nicholas Nethercote-2/+6
Instead of `ast::Lit`. Literal lowering now happens at two different times. Expression literals are lowered when HIR is crated. Attribute literals are lowered during parsing. This commit changes the language very slightly. Some programs that used to not compile now will compile. This is because some invalid literals that are removed by `cfg` or attribute macros will no longer trigger errors. See this comment for more details: https://github.com/rust-lang/rust/pull/102944#issuecomment-1277476773
2022-11-09Rollup merge of #103919 - nnethercote:unescaping-cleanups, r=matkladDylan DPC-103/+66
Unescaping cleanups Some code improvements, and some error message improvements. Best reviewed one commit at a time. r? ````@matklad````
2022-11-08Simplify `unescape_{char,byte}`.Nicholas Nethercote-17/+9
The `usize` isn't needed in the error case.
2022-11-07Make underscore_literal_suffix a hard error.Nicholas Nethercote-3/+6
It's been a warning for 5.5 years. Time to make it a hard error. Closes #42326.
2022-11-05Rename some `result` variables as `res`, for consistency.Nicholas Nethercote-6/+6
2022-11-05Remove `unescape_byte_literal`.Nicholas Nethercote-21/+7
It's easy to just use `unescape_literal` + `byte_from_char`.
2022-11-04Inline and remove `validate_int_literal`.Nicholas Nethercote-5/+5
It has a single callsite, and is fairly small. The `Float` match arm already has base-specific checking inline, so this makes things more consistent.
2022-11-04Improve comments.Nicholas Nethercote-4/+1
Remove a low-value comment, remove a duplicate comment, and correct a third comment.
2022-11-04Make non-ASCII errors more consistent.Nicholas Nethercote-10/+4
There are three kinds of "byte" literals: byte literals, byte string literals, and raw byte string literals. None are allowed to have non-ASCII chars in them. Two `EscapeError` variants exist for when that constraint is violated. - `NonAsciiCharInByte`: used for byte literals and byte string literals. - `NonAsciiCharInByteString`: used for raw byte string literals. As a result, the messages for raw byte string literals use different wording, without good reason. Also, byte string literals are incorrectly described as "byte constants" in some error messages. This commit eliminates `NonAsciiCharInByteString` so the three cases are handled similarly, and described correctly. The `mode` is enough to distinguish them. Note: Some existing error messages mention "byte constants" and some mention "byte literals". I went with the latter here, because it's a more correct name, as used by the Reference.
2022-11-03Use `Mode` less.Nicholas Nethercote-24/+22
It's passed to numerous places where we just need an `is_byte` bool. Passing the bool avoids the need for some assertions. Also rename `is_bytes()` as `is_byte()`, to better match `Mode::Byte`, `Mode::ByteStr`, and `Mode::RawByteStr`.
2022-11-03Clarify range calculations.Nicholas Nethercote-10/+12
There is some subtlety here.
2022-11-03Rename some variables.Nicholas Nethercote-38/+32
These have been bugging me for a while. - `literal_text`: `src` is also used and is shorter and better. - `first_char`: used even when "first" doesn't make sense; `c` is shorter and better. - `curr`: `c` is shorter and better. - `unescaped_char`: `result` is also used and is shorter and better. - `second_char`: these have a single use and can be elided.
2022-10-26rustc_lexer::TokenKind improve docsTshepang Mbambo-5/+18
2022-09-28Auto merge of #102302 - nnethercote:more-lexer-improvements, r=matkladbors-35/+34
More lexer improvements A follow-up to #99884. r? `@matklad`
2022-09-28Address review comments.Nicholas Nethercote-2/+4
2022-09-28Small cleanups in unescaping code.Nicholas Nethercote-14/+11
- Rename `unescape_raw_str_or_raw_byte_str` as `unescape_raw_str_or_byte_str`, which is more accurate. - Remove the unused `Mode::in_single_quotes` method. - Make some assertions more precise, and add a missing one to `unescape_char_or_byte`. - Change all the assertions to `debug_assert!`, because this code is reasonably hot, and the assertions aren't required for memory safety, and any violations are likely to be sufficiently obvious that normal tests will trigger them.
2022-09-27Fix an incorrect comment.Nicholas Nethercote-1/+1
If a `\x` escape occurs in a non-byte literals (e.g. char literal, string literal), it must be <= 0xff.
2022-09-26Add `rustc_lexer::TokenKind::Eof`.Nicholas Nethercote-4/+13
For alignment with `rust_ast::TokenKind::Eof`. Plus it's a bit faster, due to less `Option` manipulation in `StringReader::next_token`.
generated by cgit-pink 1.4.1-3-g733a5 (git 2.37.1) at 2026-02-22 18:25:57 +0000