about summary refs log tree commit diff
path: root/compiler/rustc_parse/src/lexer
AgeCommit message (Collapse)AuthorLines
2024-07-25add limit for unclosed delimiters in lexer diagnosticyukang-3/+18
2024-06-18Use a dedicated type instead of a reference for the diagnostic contextOli Scherer-4/+4
This paves the way for tracking more state (e.g. error tainting) in the diagnostic context handle
2024-06-18Prefer `dcx` methods over fields or fields' methodsOli Scherer-8/+7
2024-06-05Remove `stream_to_parser`.Nicholas Nethercote-1/+2
It's a zero-value wrapper of `Parser::new`.
2024-06-05Don't use the word "parse" for lexing operations.Nicholas Nethercote-27/+24
Lexing converts source text into a token stream. Parsing converts a token stream into AST fragments. This commit renames several lexing operations that have "parse" in the name. I think these names have been subtly confusing me for years. This is just a `s/parse/lex/` on function names, with one exception: `parse_stream_from_source_str` becomes `source_str_to_stream`, to make it consistent with the existing `source_file_to_stream`. The commit also moves that function's location in the file to be just above `source_file_to_stream`. The commit also cleans up a few comments along the way.
2024-06-05`UNICODE_ARRAY` and `ASCII_ARRAY` fixes.Nicholas Nethercote-37/+38
- Avoid unnecessary escaping of single quotes within string literals. - Add a missing blank line between two `UNICODE_ARRAY` sections.
2024-05-23Remove `#[macro_use] extern crate tracing` from `rustc_parse`.Nicholas Nethercote-0/+2
2024-05-21Rename buffer_lint_with_diagnostic to buffer_lintXiretza-2/+2
2024-05-21Generate lint diagnostic message from BuiltinLintDiagXiretza-3/+1
Translation of the lint message happens when the actual diagnostic is created, not when the lint is buffered. Generating the message from BuiltinLintDiag ensures that all required data to construct the message is preserved in the LintBuffer, eventually allowing the messages to be moved to fluent. Remove the `msg` field from BufferedEarlyLint, it is either generated from the data in the BuiltinLintDiag or stored inside BuiltinLintDiag::Normal.
2024-05-17Clarify that the diff_marker is talking about version control systemardi-1/+1
conflicts specifically and a few more improvements.
2024-05-07narrow down visibilities in `rustc_parse::lexer`Lin Yihai-6/+6
2024-04-18Rollup merge of #123752 - estebank:emoji-prefix, r=wesleywiserJubilee-1/+4
Properly handle emojis as literal prefix in macros Do not accept the following ```rust macro_rules! lexes {($($_:tt)*) => {}} lexes!(🐛"foo"); ``` Before, invalid emoji identifiers were gated during parsing instead of lexing in all cases, but this didn't account for macro pre-expansion of literal prefixes. Fix #123696.
2024-04-18Simplify `static_assert_size`s.Nicholas Nethercote-1/+1
We want to run them on all 64-bit platforms.
2024-04-12Rollup merge of #123223 - estebank:issue-123079, r=pnkfelixMatthias Krüger-13/+7
Fix invalid silencing of parsing error Given ```rust macro_rules! a { ( ) => { impl<'b> c for d { e::<f'g> } }; } ``` ensure an error is emitted. Fix #123079.
2024-04-10Properly handle emojis as literal prefix in macrosEsteban Küber-1/+4
Do not accept the following ```rust macro_rules! lexes {($($_:tt)*) => {}} lexes!(🐛"foo"); ``` Before, invalid emoji identifiers were gated during parsing instead of lexing in all cases, but this didn't account for macro expansion of literal prefixes. Fix #123696.
2024-04-08parser: reduce visibility of unnecessary public `UnmatchedDelim`Yutaro Ohno-5/+2
`lexer::UnmatchedDelim` struct in `rustc_parse` is unnecessary public outside of the crate. This commit reduces the visibility to `pub(crate)`. Beside, this removes unnecessary field `expected_delim` that causes warnings after changing the visibility.
2024-04-07Fix invalid silencing of parsing errorEsteban Küber-13/+7
Given ```rust macro_rules! a { ( ) => { impl<'b> c for d { e::<f'g> } }; } ``` ensure an error is emitted. Fix #123079.
2024-04-03Check `x86_64` size assertions on `aarch64`, tooZalathar-1/+1
This makes it easier for contributors on aarch64 workstations (e.g. Macs) to notice when these assertions have been violated.
2024-03-17fix rustdoc testEsteban Küber-1/+1
2024-03-17Silence redundant error on char literal that was meant to be a string in ↵Esteban Küber-1/+10
2021 edition
2024-03-17review comment: `str` -> string in messagesEsteban Küber-1/+1
2024-03-17Use shorter span for existing `'` -> `"` structured suggestionEsteban Küber-5/+15
2024-03-17Handle str literals written with `'` lexed as lifetimeEsteban Küber-4/+42
Given `'hello world'` and `'1 str', provide a structured suggestion for a valid string literal: ``` error[E0762]: unterminated character literal --> $DIR/lex-bad-str-literal-as-char-3.rs:2:26 | LL | println!('hello world'); | ^^^^ | help: if you meant to write a `str` literal, use double quotes | LL | println!("hello world"); | ~ ~ ``` ``` error[E0762]: unterminated character literal --> $DIR/lex-bad-str-literal-as-char-1.rs:2:20 | LL | println!('1 + 1'); | ^^^^ | help: if you meant to write a `str` literal, use double quotes | LL | println!("1 + 1"); | ~ ~ ``` Fix #119685.
2024-03-05Rename `BuiltinLintDiagnostics` as `BuiltinLintDiag`.Nicholas Nethercote-3/+3
Not the dropping of the trailing `s` -- this type describes a single diagnostic and its name should be singular.
2024-03-05Rename all `ParseSess` variables/fields/lifetimes as `psess`.Nicholas Nethercote-36/+36
Existing names for values of this type are `sess`, `parse_sess`, `parse_session`, and `ps`. `sess` is particularly annoying because that's also used for `Session` values, which are often co-located, and it can be difficult to know which type a value named `sess` refers to. (That annoyance is the main motivation for this change.) `psess` is nice and short, which is good for a name used this much. The commit also renames some `parse_sess_created` values as `psess_created`.
2024-02-29Rollup merge of #121724 - nnethercote:LitKind-Err-for-floats, r=fmeaseMatthias Krüger-3/+7
Use `LitKind::Err` for malformed floats #121120 changed `StringReader::cook_lexer_literal` to return `LitKind::Err` for malformed integer literals. This commit does the same for float literals, for consistency. r? ``@fmease``
2024-02-28Use `LitKind::Err` for floats with unsupported bases.Nicholas Nethercote-1/+3
This slightly changes error messages in `float-field.rs`, but nothing of real importance.
2024-02-28Use `LitKind::Err` for floats with empty exponents.Nicholas Nethercote-2/+4
This prevents a follow-up type error in a test, which seems fine.
2024-02-28Rename `DiagnosticBuilder` as `Diag`.Nicholas Nethercote-8/+5
Much better! Note that this involves renaming (and updating the value of) `DIAGNOSTIC_BUILDER` in clippy.
2024-02-25Rollup merge of #121060 - clubby789:bool-newtypes, r=cjgillotMatthias Krüger-5/+5
Add newtypes for bool fields/params/return types Fixed all the cases of this found with some simple searches for `*/ bool` and `bool /*`; probably many more
2024-02-20Add newtype for raw identsclubby789-5/+5
2024-02-19Prefer `DiagnosticBuilder` over `Diagnostic` in diagnostic modifiers.Nicholas Nethercote-3/+3
There are lots of functions that modify a diagnostic. This can be via a `&mut Diagnostic` or a `&mut DiagnosticBuilder`, because the latter type wraps the former and impls `DerefMut`. This commit converts all the `&mut Diagnostic` occurrences to `&mut DiagnosticBuilder`. This is a step towards greatly simplifying `Diagnostic`. Some of the relevant function are made generic, because they deal with both errors and warnings. No function bodies are changed, because all the modifier methods are available on both `Diagnostic` and `DiagnosticBuilder`.
2024-02-15Add `ErrorGuaranteed` to `ast::LitKind::Err`, `token::LitKind::Err`.Nicholas Nethercote-10/+12
This mostly works well, and eliminates a couple of delayed bugs. One annoying thing is that we should really also add an `ErrorGuaranteed` to `proc_macro::bridge::LitKind::Err`. But that's difficult because `proc_macro` doesn't have access to `ErrorGuaranteed`, so we have to fake it.
2024-02-15Make `emit_unescape_error` return `Option<ErrorGuaranteed>`.Nicholas Nethercote-40/+34
And use the result in `cook_common` to decide whether to return an error token.
2024-02-15Remove `LitError::LexerError`.Nicholas Nethercote-15/+15
`cook_lexer_literal` can emit an error about an invalid int literal but then return a non-`Err` token. And then `integer_lit` has to account for this to avoid printing a redundant error message. This commit changes `cook_lexer_literal` to return `Err` in that case. Then `integer_lit` doesn't need the special case, and `LitError::LexerError` can be removed.
2024-01-29Stop using `String` for error codes.Nicholas Nethercote-8/+8
Error codes are integers, but `String` is used everywhere to represent them. Gross! This commit introduces `ErrCode`, an integral newtype for error codes, replacing `String`. It also introduces a constant for every error code, e.g. `E0123`, and removes the `error_code!` macro. The constants are imported wherever used with `use rustc_errors::codes::*`. With the old code, we have three different ways to specify an error code at a use point: ``` error_code!(E0123) // macro call struct_span_code_err!(dcx, span, E0123, "msg"); // bare ident arg to macro call \#[diag(name, code = "E0123")] // string struct Diag; ``` With the new code, they all use the `E0123` constant. ``` E0123 // constant struct_span_code_err!(dcx, span, E0123, "msg"); // constant \#[diag(name, code = E0123)] // constant struct Diag; ``` The commit also changes the structure of the error code definitions: - `rustc_error_codes` now just defines a higher-order macro listing the used error codes and nothing else. - Because that's now the only thing in the `rustc_error_codes` crate, I moved it into the `lib.rs` file and removed the `error_codes.rs` file. - `rustc_errors` uses that macro to define everything, e.g. the error code constants and the `DIAGNOSTIC_TABLES`. This is in its new `codes.rs` file.
2024-01-25Use `unescape_unicode` for raw C string literals.Nicholas Nethercote-1/+1
They can't contain `\x` escapes, which means they can't contain high bytes, which means we can used `unescape_unicode` instead of `unescape_mixed` to unescape them. This avoids unnecessary used of `MixedUnit`.
2024-01-25Rename the unescaping functions.Nicholas Nethercote-12/+12
`unescape_literal` becomes `unescape_unicode`, and `unescape_c_string` becomes `unescape_mixed`. Because rfc3349 will mean that C string literals will no longer be the only mixed utf8 literals.
2024-01-12Detect `NulInCStr` error earlier.Nicholas Nethercote-0/+3
By making it an `EscapeError` instead of a `LitError`. This makes it like the other errors produced when checking string literals contents, e.g. for invalid escape sequences or bare CR chars. NOTE: this means these errors are issued earlier, before expansion, which changes behaviour. It will be possible to move the check back to the later point if desired. If that happens, it's likely that all the string literal contents checks will be delayed together. One nice thing about this: the old approach had some code in `report_lit_error` to calculate the span of the nul char from a range. This code used a hardwired `+2` to account for the `c"` at the start of a C string literal, but this should have changed to a `+3` for raw C string literals to account for the `cr"`, which meant that the caret in `cr"` nul error messages was one short of where it should have been. The new approach doesn't need any of this and avoids the off-by-one error.
2024-01-11Stop using `DiagnosticBuilder::buffer` in the parser.Nicholas Nethercote-4/+4
One consequence is that errors returned by `maybe_new_parser_from_source_str` now must be consumed, so a bunch of places that previously ignored those errors now cancel them. (Most of them explicitly dropped the errors before. I guess that was to indicate "we are explicitly ignoring these", though I'm not 100% sure.)
2024-01-11Fix lifetimes in `StringReader`.Nicholas Nethercote-23/+27
Two different lifetimes are conflated. This doesn't matter right now, but needs to be fixed for the next commit to work. And the more descriptive lifetime names make the code easier to read.
2024-01-10Rename consuming chaining methods on `DiagnosticBuilder`.Nicholas Nethercote-6/+6
In #119606 I added them and used a `_mv` suffix, but that wasn't great. A `with_` prefix has three different existing uses. - Constructors, e.g. `Vec::with_capacity`. - Wrappers that provide an environment to execute some code, e.g. `with_session_globals`. - Consuming chaining methods, e.g. `Span::with_{lo,hi,ctxt}`. The third case is exactly what we want, so this commit changes `DiagnosticBuilder::foo_mv` to `DiagnosticBuilder::with_foo`. Thanks to @compiler-errors for the suggestion.
2024-01-10Rename `{create,emit}_warning` as `{create,emit}_warn`.Nicholas Nethercote-2/+2
For consistency with `warn`/`struct_warn`, and also `{create,emit}_err`, all of which use an abbreviated form.
2024-01-08Remove `DiagnosticBuilder::delay_as_bug_without_consuming`.Nicholas Nethercote-4/+4
The existing uses are replaced in one of three ways. - In a function that also has calls to `emit`, just rearrange the code so that exactly one of `delay_as_bug` or `emit` is called on every path. - In a function returning a `DiagnosticBuilder`, use `downgrade_to_delayed_bug`. That's good enough because it will get emitted later anyway. - In `unclosed_delim_err`, one set of errors is being replaced with another set, so just cancel the original errors.
2024-01-08Remove all eight `DiagnosticBuilder::*_with_code` methods.Nicholas Nethercote-36/+37
These all have relatively low use, and can be perfectly emulated with a simpler construction method combined with `code` or `code_mv`.
2024-01-08Use chaining in `DiagnosticBuilder` construction.Nicholas Nethercote-3/+3
To avoid the use of a mutable local variable, and because it reads more nicely.
2024-01-08Make `DiagnosticBuilder::emit` consuming.Nicholas Nethercote-1/+1
This works for most of its call sites. This is nice, because `emit` very much makes sense as a consuming operation -- indeed, `DiagnosticBuilderState` exists to ensure no diagnostic is emitted twice, but it uses runtime checks. For the small number of call sites where a consuming emit doesn't work, the commit adds `DiagnosticBuilder::emit_without_consuming`. (This will be removed in subsequent commits.) Likewise, `emit_unless` becomes consuming. And `delay_as_bug` becomes consuming, while `delay_as_bug_without_consuming` is added (which will also be removed in subsequent commits.) All this requires significant changes to `DiagnosticBuilder`'s chaining methods. Currently `DiagnosticBuilder` method chaining uses a non-consuming `&mut self -> &mut Self` style, which allows chaining to be used when the chain ends in `emit()`, like so: ``` struct_err(msg).span(span).emit(); ``` But it doesn't work when producing a `DiagnosticBuilder` value, requiring this: ``` let mut err = self.struct_err(msg); err.span(span); err ``` This style of chaining won't work with consuming `emit` though. For that, we need to use to a `self -> Self` style. That also would allow `DiagnosticBuilder` production to be chained, e.g.: ``` self.struct_err(msg).span(span) ``` However, removing the `&mut self -> &mut Self` style would require that individual modifications of a `DiagnosticBuilder` go from this: ``` err.span(span); ``` to this: ``` err = err.span(span); ``` There are *many* such places. I have a high tolerance for tedious refactorings, but even I gave up after a long time trying to convert them all. Instead, this commit has it both ways: the existing `&mut self -> Self` chaining methods are kept, and new `self -> Self` chaining methods are added, all of which have a `_mv` suffix (short for "move"). Changes to the existing `forward!` macro lets this happen with very little additional boilerplate code. I chose to add the suffix to the new chaining methods rather than the existing ones, because the number of changes required is much smaller that way. This doubled chainging is a bit clumsy, but I think it is worthwhile because it allows a *lot* of good things to subsequently happen. In this commit, there are many `mut` qualifiers removed in places where diagnostics are emitted without being modified. In subsequent commits: - chaining can be used more, making the code more concise; - more use of chaining also permits the removal of redundant diagnostic APIs like `struct_err_with_code`, which can be replaced easily with `struct_err` + `code_mv`; - `emit_without_diagnostic` can be removed, which simplifies a lot of machinery, removing the need for `DiagnosticBuilderState`.
2024-01-04Inline and remove `StringReader::struct_fatal_span_char`.Nicholas Nethercote-22/+11
It has a single call site.
2024-01-03Rename some `Diagnostic` setters.Nicholas Nethercote-1/+1
`Diagnostic` has 40 methods that return `&mut Self` and could be considered setters. Four of them have a `set_` prefix. This doesn't seem necessary for a type that implements the builder pattern. This commit removes the `set_` prefixes on those four methods.
2023-12-24Remove `Session` methods that duplicate `DiagCtxt` methods.Nicholas Nethercote-8/+8
Also add some `dcx` methods to types that wrap `TyCtxt`, for easier access.