about summary refs log tree commit diff
path: root/compiler/rustc_lexer
AgeCommit message (Collapse)AuthorLines
2024-12-09Add test to check unicode identifier versionEric Huss-0/+1
2024-12-01Revert "Reject raw lifetime followed by \' as well"Michael Goulet-11/+1
This reverts commit 1990f1560801ca3f9e6a3286e58204aa329ee037.
2024-11-25Clean up `c_or_byte_string`.Nicholas Nethercote-4/+4
- Rename a misleading local `mk_kind` as `single_quoted`. - Use `fn` for all three arguments, for consistency.
2024-11-25Improve `strip_shebang` testing.Nicholas Nethercote-41/+31
It's currently a bit ad hoc. This commit makes it more methodical, with pairs of match/no-match tests for all the relevant cases.
2024-11-19Remove `TokenKind::InvalidPrefix`.Nicholas Nethercote-13/+8
It was added in #123752 to handle some cases involving emoji, but it isn't necessary because it's always treated the same as `TokenKind::InvalidIdent`. This commit removes it, which makes things a little simpler.
2024-11-19Improve `TokenKind` comments.Nicholas Nethercote-41/+39
- Improve wording. - Use backticks consistently for examples.
2024-11-19Improve `fake_ident_or_unknown_prefix`.Nicholas Nethercote-6/+5
- Rename it as `invalid_ident_or_prefix`, which matches the possible outputs (`InvalidIdent` or `InvalidPrefix`). - Use the local wrapper for `is_xid_continue`, for consistency. - Make it clear what `\u{200d}` means.
2024-10-30Reject raw lifetime followed by \' as wellMichael Goulet-1/+11
2024-10-08Reserve guarded string literals (RFC 3593)Peter Jaszkowiak-8/+85
2024-09-22Reformat using the new identifier sorting from rustfmtMichael Goulet-35/+17
2024-09-06Add initial support for raw lifetimesMichael Goulet-1/+12
2024-09-06Format lexerMichael Goulet-1/+1
2024-09-06Reserve prefix lifetimes tooMichael Goulet-9/+17
2024-08-27Add `warn(unreachable_pub)` to `rustc_lexer`.Nicholas Nethercote-0/+1
2024-07-29Reformat `use` declarations.Nicholas Nethercote-4/+4
The previous commit updated `rustfmt.toml` appropriately. This commit is the outcome of running `x fmt --all` with the new formatting options.
2024-06-12Use `tidy` to sort crate attributes for all compiler crates.Nicholas Nethercote-0/+2
We already do this for a number of crates, e.g. `rustc_middle`, `rustc_span`, `rustc_metadata`, `rustc_span`, `rustc_errors`. For the ones we don't, in many cases the attributes are a mess. - There is no consistency about order of attribute kinds (e.g. `allow`/`deny`/`feature`). - Within attribute kind groups (e.g. the `feature` attributes), sometimes the order is alphabetical, and sometimes there is no particular order. - Sometimes the attributes of a particular kind aren't even grouped all together, e.g. there might be a `feature`, then an `allow`, then another `feature`. This commit extends the existing sorting to all compiler crates, increasing consistency. If any new attribute line is added there is now only one place it can go -- no need for arbitrary decisions. Exceptions: - `rustc_log`, `rustc_next_trait_solver` and `rustc_type_ir_macros`, because they have no crate attributes. - `rustc_codegen_gcc`, because it's quasi-external to rustc (e.g. it's ignored in `rustfmt.toml`).
2024-04-24Improved the compiler code with clippyMichael Scholten-5/+5
2024-04-10Properly handle emojis as literal prefix in macrosEsteban Küber-1/+5
Do not accept the following ```rust macro_rules! lexes {($($_:tt)*) => {}} lexes!(🐛"foo"); ``` Before, invalid emoji identifiers were gated during parsing instead of lexing in all cases, but this didn't account for macro expansion of literal prefixes. Fix #123696.
2024-03-17Silence redundant error on char literal that was meant to be a string in ↵Esteban Küber-0/+9
2021 edition
2024-03-17Handle str literals written with `'` lexed as lifetimeEsteban Küber-1/+1
Given `'hello world'` and `'1 str', provide a structured suggestion for a valid string literal: ``` error[E0762]: unterminated character literal --> $DIR/lex-bad-str-literal-as-char-3.rs:2:26 | LL | println!('hello world'); | ^^^^ | help: if you meant to write a `str` literal, use double quotes | LL | println!("hello world"); | ~ ~ ``` ``` error[E0762]: unterminated character literal --> $DIR/lex-bad-str-literal-as-char-1.rs:2:20 | LL | println!('1 + 1'); | ^^^^ | help: if you meant to write a `str` literal, use double quotes | LL | println!("1 + 1"); | ~ ~ ``` Fix #119685.
2024-02-06Invert diagnostic lints.Nicholas Nethercote-2/+1
That is, change `diagnostic_outside_of_impl` and `untranslatable_diagnostic` from `allow` to `deny`, because more than half of the compiler has be converted to use translated diagnostics. This commit removes more `deny` attributes than it adds `allow` attributes, which proves that this change is warranted.
2024-01-25Use `unescape_unicode` for raw C string literals.Nicholas Nethercote-19/+14
They can't contain `\x` escapes, which means they can't contain high bytes, which means we can used `unescape_unicode` instead of `unescape_mixed` to unescape them. This avoids unnecessary used of `MixedUnit`.
2024-01-25Rename the unescaping functions.Nicholas Nethercote-9/+13
`unescape_literal` becomes `unescape_unicode`, and `unescape_c_string` becomes `unescape_mixed`. Because rfc3349 will mean that C string literals will no longer be the only mixed utf8 literals.
2024-01-25Rename and invert sense of `Mode` predicates.Nicholas Nethercote-32/+24
I find it easier if they describe what's allowed, rather than what's forbidden. Also, consistent naming makes them easier to understand.
2024-01-25Rework `CStrUnit`.Nicholas Nethercote-34/+45
- Rename it as `MixedUnit`, because it will soon be used in more than just C string literals. - Change the `Byte` variant to `HighByte` and use it only for `\x80`..`\xff` cases. This fixes the old inexactness where ASCII chars could be encoded with either `Byte` or `Char`. - Add useful comments. - Remove `is_ascii`, in favour of `u8::is_ascii`.
2024-01-25Use `from` instead of `into` in unescaping code.Nicholas Nethercote-4/+4
The `T` type in these functions took me some time to understand, and I find the explicit `T` in the use of `from` makes the code easier to read, as does the `u8` annotation in `scan_escape`.
2024-01-22Rollup merge of #118639 - fmease:deny-features-in-stable-rustc-crates, ↵Matthias Krüger-2/+3
r=WaffleLapkin Undeprecate lint `unstable_features` and make use of it in the compiler See also #117937. r? compiler
2024-01-12Detect `NulInCStr` error earlier.Nicholas Nethercote-2/+15
By making it an `EscapeError` instead of a `LitError`. This makes it like the other errors produced when checking string literals contents, e.g. for invalid escape sequences or bare CR chars. NOTE: this means these errors are issued earlier, before expansion, which changes behaviour. It will be possible to move the check back to the later point if desired. If that happens, it's likely that all the string literal contents checks will be delayed together. One nice thing about this: the old approach had some code in `report_lit_error` to calculate the span of the nul char from a range. This code used a hardwired `+2` to account for the `c"` at the start of a C string literal, but this should have changed to a `+3` for raw C string literals to account for the `cr"`, which meant that the caret in `cr"` nul error messages was one short of where it should have been. The new approach doesn't need any of this and avoids the off-by-one error.
2023-12-20Undeprecate and use lint `unstable_features`León Orell Valerian Liehr-2/+3
2023-12-13Rename some unescaping functions.Nicholas Nethercote-9/+7
`unescape_raw_str_or_raw_byte_str` only does checking, no unescaping. And it also now handles C string literals. `unescape_raw_str` is used for all the non-raw strings.
2023-12-13Adjust `Mode::is_unicode_escape_disallowed`.Nicholas Nethercote-2/+3
Some cases are unreachable.
2023-12-09Tweak `Mode`.Nicholas Nethercote-34/+38
- Add `use Mode::*` to avoid all the qualifiers. - Reorder the variants. The existing order makes no particular sense, which has bugged me for some time. I've chosen an order that makes sense to me.
2023-12-09Remove an unnecessary `into`.Nicholas Nethercote-1/+1
2023-12-09Eliminate `is_byte: bool` args in unescaping code.Nicholas Nethercote-21/+19
These don't really make sense since C string literals were added. This commit removes them in favour for `mode: Mode` args. `ascii_check` still has a `characters_should_be_ascii: bool` arg. Also, `characters_should_be_ascii` is renamed to be shorter.
2023-12-09Remove explicit `\n` and `\t` handling in `unescape_str_common`.Nicholas Nethercote-2/+0
The fallback `_` case works for these chars, no need to treat them specially.
2023-12-09Add some useful comments.Nicholas Nethercote-4/+8
2023-12-07Fix an out-of-date comment.Nicholas Nethercote-1/+1
2023-12-07Identify impossible cases in `ascii_escapes_should_be_ascii`.Nicholas Nethercote-2/+3
Raw strings (of all kinds) don't support escapes, so this function should never be called on them.
2023-12-07De-pub some functions.Nicholas Nethercote-3/+3
2023-10-30Clean up `rustc_*/Cargo.toml`.Nicholas Nethercote-1/+0
- Sort dependencies and features sections. - Add `tidy` markers to the sorted sections so they stay sorted. - Remove empty `[lib`] sections. - Remove "See more keys..." comments. Excluded files: - rustc_codegen_{cranelift,gcc}, because they're external. - rustc_lexer, because it has external use. - stable_mir, because it has external use.
2023-10-18Use v0.0.0 in compiler cratesMichael Goulet-1/+1
2023-07-29Update lexer emoji diagnostics to Unicode 15.0Charles Lew-8/+9
2023-07-25Auto merge of #113476 - fee1-dead-contrib:c-str-lit, r=petrochenkovbors-0/+11
Reimplement C-str literals This reverts #113334, cc `@fmease.` While converting lexer tokens to ast Tokens in `rustc_parse`, we check the edition of the span of the token. If the edition < 2021, we split the token into two, one being the identifier and other being the str literal.
2023-07-23fix some clippy::style findingsMatthias Krüger-1/+1
comparison_to_empty iter_nth_zero for_kv_map manual_next_back redundant_pattern
2023-07-23reimplement C string literalsDeadbeef-0/+11
2023-07-05Revert the lexing of c_str_literalsLeón Orell Valerian Liehr-7/+0
2023-05-15Don't try to eat non-existent decimal digits.Nicholas Nethercote-2/+6
After seeing a `0`, if it's followed by any of `[0-9]`, `_`, `.`, `e`, or `E`, we consume all the digits. But in the `.`, `e` and `E` cases this is pointless because we know there aren't any digits.
2023-05-15Make `Cursor::number` less DRY.Nicholas Nethercote-10/+10
A tiny bit of repetition makes this easier to read, and avoids a test on the "Not a base prefix" match arm.
2023-05-02address commentsDeadbeef-1/+1
2023-05-02refactor unescapeDeadbeef-30/+61