summary refs log tree commit diff
path: root/compiler/rustc_parse/src/lexer
AgeCommit message (Collapse)AuthorLines
2024-11-21Enforce that raw lifetime identifiers must be valid raw identifiersMichael Goulet-4/+10
(cherry picked from commit 9785c7cf94f5e30742f886764f2d25b6a4da66e8)
2024-10-08Reserve guarded string literals (RFC 3593)Peter Jaszkowiak-1/+83
2024-09-22Reformat using the new identifier sorting from rustfmtMichael Goulet-5/+5
2024-09-17Store raw ident span for raw lifetimeMichael Goulet-0/+3
2024-09-09Remove needless returns detected by clippy in the compilerEduardo Sánchez Muñoz-1/+1
2024-09-06Add some more testsMichael Goulet-1/+1
2024-09-06Add initial support for raw lifetimesMichael Goulet-3/+36
2024-09-06Format lexerMichael Goulet-19/+22
2024-09-06Reserve prefix lifetimes tooMichael Goulet-0/+10
2024-08-14Use `impl PartialEq<TokenKind> for Token` more.Nicholas Nethercote-1/+1
This lets us compare a `Token` with a `TokenKind`. It's used a lot, but can be used even more, avoiding the need for some `.kind` uses.
2024-07-30Auto merge of #127955 - ↵bors-3/+18
chenyukang:yukang-fix-mismatched-delimiter-issue-12786, r=nnethercote Add limit for unclosed delimiters in lexer diagnostic Fixes #127868 The first commit shows the original diagnostic, and the second commit shows the changes.
2024-07-29Reformat `use` declarations.Nicholas Nethercote-19/+23
The previous commit updated `rustfmt.toml` appropriately. This commit is the outcome of running `x fmt --all` with the new formatting options.
2024-07-25add limit for unclosed delimiters in lexer diagnosticyukang-3/+18
2024-06-18Use a dedicated type instead of a reference for the diagnostic contextOli Scherer-4/+4
This paves the way for tracking more state (e.g. error tainting) in the diagnostic context handle
2024-06-18Prefer `dcx` methods over fields or fields' methodsOli Scherer-8/+7
2024-06-05Remove `stream_to_parser`.Nicholas Nethercote-1/+2
It's a zero-value wrapper of `Parser::new`.
2024-06-05Don't use the word "parse" for lexing operations.Nicholas Nethercote-27/+24
Lexing converts source text into a token stream. Parsing converts a token stream into AST fragments. This commit renames several lexing operations that have "parse" in the name. I think these names have been subtly confusing me for years. This is just a `s/parse/lex/` on function names, with one exception: `parse_stream_from_source_str` becomes `source_str_to_stream`, to make it consistent with the existing `source_file_to_stream`. The commit also moves that function's location in the file to be just above `source_file_to_stream`. The commit also cleans up a few comments along the way.
2024-06-05`UNICODE_ARRAY` and `ASCII_ARRAY` fixes.Nicholas Nethercote-37/+38
- Avoid unnecessary escaping of single quotes within string literals. - Add a missing blank line between two `UNICODE_ARRAY` sections.
2024-05-23Remove `#[macro_use] extern crate tracing` from `rustc_parse`.Nicholas Nethercote-0/+2
2024-05-21Rename buffer_lint_with_diagnostic to buffer_lintXiretza-2/+2
2024-05-21Generate lint diagnostic message from BuiltinLintDiagXiretza-3/+1
Translation of the lint message happens when the actual diagnostic is created, not when the lint is buffered. Generating the message from BuiltinLintDiag ensures that all required data to construct the message is preserved in the LintBuffer, eventually allowing the messages to be moved to fluent. Remove the `msg` field from BufferedEarlyLint, it is either generated from the data in the BuiltinLintDiag or stored inside BuiltinLintDiag::Normal.
2024-05-17Clarify that the diff_marker is talking about version control systemardi-1/+1
conflicts specifically and a few more improvements.
2024-05-07narrow down visibilities in `rustc_parse::lexer`Lin Yihai-6/+6
2024-04-18Rollup merge of #123752 - estebank:emoji-prefix, r=wesleywiserJubilee-1/+4
Properly handle emojis as literal prefix in macros Do not accept the following ```rust macro_rules! lexes {($($_:tt)*) => {}} lexes!(🐛"foo"); ``` Before, invalid emoji identifiers were gated during parsing instead of lexing in all cases, but this didn't account for macro pre-expansion of literal prefixes. Fix #123696.
2024-04-18Simplify `static_assert_size`s.Nicholas Nethercote-1/+1
We want to run them on all 64-bit platforms.
2024-04-12Rollup merge of #123223 - estebank:issue-123079, r=pnkfelixMatthias Krüger-13/+7
Fix invalid silencing of parsing error Given ```rust macro_rules! a { ( ) => { impl<'b> c for d { e::<f'g> } }; } ``` ensure an error is emitted. Fix #123079.
2024-04-10Properly handle emojis as literal prefix in macrosEsteban Küber-1/+4
Do not accept the following ```rust macro_rules! lexes {($($_:tt)*) => {}} lexes!(🐛"foo"); ``` Before, invalid emoji identifiers were gated during parsing instead of lexing in all cases, but this didn't account for macro expansion of literal prefixes. Fix #123696.
2024-04-08parser: reduce visibility of unnecessary public `UnmatchedDelim`Yutaro Ohno-5/+2
`lexer::UnmatchedDelim` struct in `rustc_parse` is unnecessary public outside of the crate. This commit reduces the visibility to `pub(crate)`. Beside, this removes unnecessary field `expected_delim` that causes warnings after changing the visibility.
2024-04-07Fix invalid silencing of parsing errorEsteban Küber-13/+7
Given ```rust macro_rules! a { ( ) => { impl<'b> c for d { e::<f'g> } }; } ``` ensure an error is emitted. Fix #123079.
2024-04-03Check `x86_64` size assertions on `aarch64`, tooZalathar-1/+1
This makes it easier for contributors on aarch64 workstations (e.g. Macs) to notice when these assertions have been violated.
2024-03-17fix rustdoc testEsteban Küber-1/+1
2024-03-17Silence redundant error on char literal that was meant to be a string in ↵Esteban Küber-1/+10
2021 edition
2024-03-17review comment: `str` -> string in messagesEsteban Küber-1/+1
2024-03-17Use shorter span for existing `'` -> `"` structured suggestionEsteban Küber-5/+15
2024-03-17Handle str literals written with `'` lexed as lifetimeEsteban Küber-4/+42
Given `'hello world'` and `'1 str', provide a structured suggestion for a valid string literal: ``` error[E0762]: unterminated character literal --> $DIR/lex-bad-str-literal-as-char-3.rs:2:26 | LL | println!('hello world'); | ^^^^ | help: if you meant to write a `str` literal, use double quotes | LL | println!("hello world"); | ~ ~ ``` ``` error[E0762]: unterminated character literal --> $DIR/lex-bad-str-literal-as-char-1.rs:2:20 | LL | println!('1 + 1'); | ^^^^ | help: if you meant to write a `str` literal, use double quotes | LL | println!("1 + 1"); | ~ ~ ``` Fix #119685.
2024-03-05Rename `BuiltinLintDiagnostics` as `BuiltinLintDiag`.Nicholas Nethercote-3/+3
Not the dropping of the trailing `s` -- this type describes a single diagnostic and its name should be singular.
2024-03-05Rename all `ParseSess` variables/fields/lifetimes as `psess`.Nicholas Nethercote-36/+36
Existing names for values of this type are `sess`, `parse_sess`, `parse_session`, and `ps`. `sess` is particularly annoying because that's also used for `Session` values, which are often co-located, and it can be difficult to know which type a value named `sess` refers to. (That annoyance is the main motivation for this change.) `psess` is nice and short, which is good for a name used this much. The commit also renames some `parse_sess_created` values as `psess_created`.
2024-02-29Rollup merge of #121724 - nnethercote:LitKind-Err-for-floats, r=fmeaseMatthias Krüger-3/+7
Use `LitKind::Err` for malformed floats #121120 changed `StringReader::cook_lexer_literal` to return `LitKind::Err` for malformed integer literals. This commit does the same for float literals, for consistency. r? ``@fmease``
2024-02-28Use `LitKind::Err` for floats with unsupported bases.Nicholas Nethercote-1/+3
This slightly changes error messages in `float-field.rs`, but nothing of real importance.
2024-02-28Use `LitKind::Err` for floats with empty exponents.Nicholas Nethercote-2/+4
This prevents a follow-up type error in a test, which seems fine.
2024-02-28Rename `DiagnosticBuilder` as `Diag`.Nicholas Nethercote-8/+5
Much better! Note that this involves renaming (and updating the value of) `DIAGNOSTIC_BUILDER` in clippy.
2024-02-25Rollup merge of #121060 - clubby789:bool-newtypes, r=cjgillotMatthias Krüger-5/+5
Add newtypes for bool fields/params/return types Fixed all the cases of this found with some simple searches for `*/ bool` and `bool /*`; probably many more
2024-02-20Add newtype for raw identsclubby789-5/+5
2024-02-19Prefer `DiagnosticBuilder` over `Diagnostic` in diagnostic modifiers.Nicholas Nethercote-3/+3
There are lots of functions that modify a diagnostic. This can be via a `&mut Diagnostic` or a `&mut DiagnosticBuilder`, because the latter type wraps the former and impls `DerefMut`. This commit converts all the `&mut Diagnostic` occurrences to `&mut DiagnosticBuilder`. This is a step towards greatly simplifying `Diagnostic`. Some of the relevant function are made generic, because they deal with both errors and warnings. No function bodies are changed, because all the modifier methods are available on both `Diagnostic` and `DiagnosticBuilder`.
2024-02-15Add `ErrorGuaranteed` to `ast::LitKind::Err`, `token::LitKind::Err`.Nicholas Nethercote-10/+12
This mostly works well, and eliminates a couple of delayed bugs. One annoying thing is that we should really also add an `ErrorGuaranteed` to `proc_macro::bridge::LitKind::Err`. But that's difficult because `proc_macro` doesn't have access to `ErrorGuaranteed`, so we have to fake it.
2024-02-15Make `emit_unescape_error` return `Option<ErrorGuaranteed>`.Nicholas Nethercote-40/+34
And use the result in `cook_common` to decide whether to return an error token.
2024-02-15Remove `LitError::LexerError`.Nicholas Nethercote-15/+15
`cook_lexer_literal` can emit an error about an invalid int literal but then return a non-`Err` token. And then `integer_lit` has to account for this to avoid printing a redundant error message. This commit changes `cook_lexer_literal` to return `Err` in that case. Then `integer_lit` doesn't need the special case, and `LitError::LexerError` can be removed.
2024-01-29Stop using `String` for error codes.Nicholas Nethercote-8/+8
Error codes are integers, but `String` is used everywhere to represent them. Gross! This commit introduces `ErrCode`, an integral newtype for error codes, replacing `String`. It also introduces a constant for every error code, e.g. `E0123`, and removes the `error_code!` macro. The constants are imported wherever used with `use rustc_errors::codes::*`. With the old code, we have three different ways to specify an error code at a use point: ``` error_code!(E0123) // macro call struct_span_code_err!(dcx, span, E0123, "msg"); // bare ident arg to macro call \#[diag(name, code = "E0123")] // string struct Diag; ``` With the new code, they all use the `E0123` constant. ``` E0123 // constant struct_span_code_err!(dcx, span, E0123, "msg"); // constant \#[diag(name, code = E0123)] // constant struct Diag; ``` The commit also changes the structure of the error code definitions: - `rustc_error_codes` now just defines a higher-order macro listing the used error codes and nothing else. - Because that's now the only thing in the `rustc_error_codes` crate, I moved it into the `lib.rs` file and removed the `error_codes.rs` file. - `rustc_errors` uses that macro to define everything, e.g. the error code constants and the `DIAGNOSTIC_TABLES`. This is in its new `codes.rs` file.
2024-01-25Use `unescape_unicode` for raw C string literals.Nicholas Nethercote-1/+1
They can't contain `\x` escapes, which means they can't contain high bytes, which means we can used `unescape_unicode` instead of `unescape_mixed` to unescape them. This avoids unnecessary used of `MixedUnit`.
2024-01-25Rename the unescaping functions.Nicholas Nethercote-12/+12
`unescape_literal` becomes `unescape_unicode`, and `unescape_c_string` becomes `unescape_mixed`. Because rfc3349 will mean that C string literals will no longer be the only mixed utf8 literals.