about summary refs log tree commit diff
path: root/compiler/rustc_parse/src/lexer/mod.rs
AgeCommit message (Collapse)AuthorLines
2023-12-19Add `EmitResult` associated type to `EmissionGuarantee`.Nicholas Nethercote-2/+4
This lets different error levels share the same return type from `emit_*`. - A lot of inconsistencies in the `DiagCtxt` API are removed. - `Noted` is removed. - `FatalAbort` is introduced for fatal errors (abort via `raise`), replacing the `EmissionGuarantee` impl for `!`. - `Bug` is renamed `BugAbort` (to avoid clashing with `Level::Bug` and to mirror `FatalAbort`), and modified to work in the new way with bug errors (abort via panic). - Various diagnostic creators and emitters updated to the new, better signatures. Note that `DiagCtxt::bug` no longer needs to call `panic_any`, because `emit` handles that. Also shorten the obnoxiously long `diagnostic_builder_emit_producing_guarantee` name.
2023-12-18Rename `ParseSess::span_diagnostic` as `ParseSess::dcx`.Nicholas Nethercote-10/+10
2023-12-11Add spacing information to delimiters.Nicholas Nethercote-1/+1
This is an extension of the previous commit. It means the output of something like this: ``` stringify!(let a: Vec<u32> = vec![];) ``` goes from this: ``` let a: Vec<u32> = vec![] ; ``` With this PR, it now produces this string: ``` let a: Vec<u32> = vec![]; ```
2023-12-01Auto merge of #117472 - jmillikin:stable-c-str-literals, r=Nilstriebbors-3/+0
Stabilize C string literals RFC: https://rust-lang.github.io/rfcs/3348-c-str-literal.html Tracking issue: https://github.com/rust-lang/rust/issues/105723 Documentation PR (reference manual): https://github.com/rust-lang/reference/pull/1423 # Stabilization report Stabilizes C string and raw C string literals (`c"..."` and `cr#"..."#`), which are expressions of type [`&CStr`](https://doc.rust-lang.org/stable/core/ffi/struct.CStr.html). Both new literals require Rust edition 2021 or later. ```rust const HELLO: &core::ffi::CStr = c"Hello, world!"; ``` C strings may contain any byte other than `NUL` (`b'\x00'`), and their in-memory representation is guaranteed to end with `NUL`. ## Implementation Originally implemented by PR https://github.com/rust-lang/rust/pull/108801, which was reverted due to unintentional changes to lexer behavior in Rust editions < 2021. The current implementation landed in PR https://github.com/rust-lang/rust/pull/113476, which restricts C string literals to Rust edition >= 2021. ## Resolutions to open questions from the RFC * Adding C character literals (`c'.'`) of type `c_char` is not part of this feature. * Support for `c"..."` literals does not prevent `c'.'` literals from being added in the future. * C string literals should not be blocked on making `&CStr` a thin pointer. * It's possible to declare constant expressions of type `&'static CStr` in stable Rust (as of v1.59), so C string literals are not adding additional coupling on the internal representation of `CStr`. * The unstable `concat_bytes!` macro should not accept `c"..."` literals. * C strings have two equally valid `&[u8]` representations (with or without terminal `NUL`), so allowing them to be used in `concat_bytes!` would be ambiguous. * Adding a type to represent C strings containing valid UTF-8 is not part of this feature. * Support for a hypothetical `&Utf8CStr` may be explored in the future, should such a type be added to Rust.
2023-11-21Fix `clippy::needless_borrow` in the compilerNilstrieb-3/+3
`x clippy compiler -Aclippy::all -Wclippy::needless_borrow --fix`. Then I had to remove a few unnecessary parens and muts that were exposed now.
2023-11-01Stabilize C string literalsJohn Millikin-3/+0
2023-10-30When encountering unclosed delimiters during parsing, check for diff markersEsteban Küber-6/+8
Fix #116252.
2023-08-13Remove reached_eof from ParseSessbjorn3-1/+0
It was only ever set in a function which isn't called anywhere.
2023-07-25extract common codeDeadbeef-11/+10
2023-07-23reimplement C string literalsDeadbeef-1/+25
2023-05-16Avoid `&format("...")` calls in error message code.Nicholas Nethercote-1/+1
Error message all end up passing into a function as an `impl Into<{D,Subd}iagnosticMessage>`. If an error message is creatd as `&format("...")` that means we allocate a string (in the `format!` call), then take a reference, and then clone (allocating again) the reference to produce the `{D,Subd}iagnosticMessage`, which is silly. This commit removes the leading `&` from a lot of these cases. This means the original `String` is moved into the `{D,Subd}iagnosticMessage`, avoiding the double allocations. This requires changing some function argument types from `&str` to `String` (when all arguments are `String`) or `impl Into<{D,Subd}iagnosticMessage>` (when some arguments are `String` and some are `&str`).
2023-05-05Rollup merge of #108801 - fee1-dead-contrib:c-str, r=compiler-errorsDylan DPC-3/+60
Implement RFC 3348, `c"foo"` literals RFC: https://github.com/rust-lang/rfcs/pull/3348 Tracking issue: #105723
2023-05-03Restrict `From<S>` for `{D,Subd}iagnosticMessage`.Nicholas Nethercote-2/+2
Currently a `{D,Subd}iagnosticMessage` can be created from any type that impls `Into<String>`. That includes `&str`, `String`, and `Cow<'static, str>`, which are reasonable. It also includes `&String`, which is pretty weird, and results in many places making unnecessary allocations for patterns like this: ``` self.fatal(&format!(...)) ``` This creates a string with `format!`, takes a reference, passes the reference to `fatal`, which does an `into()`, which clones the reference, doing a second allocation. Two allocations for a single string, bleh. This commit changes the `From` impls so that you can only create a `{D,Subd}iagnosticMessage` from `&str`, `String`, or `Cow<'static, str>`. This requires changing all the places that currently create one from a `&String`. Most of these are of the `&format!(...)` form described above; each one removes an unnecessary static `&`, plus an allocation when executed. There are also a few places where the existing use of `&String` was more reasonable; these now just use `clone()` at the call site. As well as making the code nicer and more efficient, this is a step towards possibly using `Cow<'static, str>` in `{D,Subd}iagnosticMessage::{Str,Eager}`. That would require changing the `From<&'a str>` impls to `From<&'static str>`, which is doable, but I'm not yet sure if it's worthwhile.
2023-05-02make cook genericDeadbeef-37/+27
2023-05-02make it semantic errorDeadbeef-0/+3
2023-05-02initial step towards implementing C string literalsDeadbeef-0/+64
2023-04-25Fix static string lintsclubby789-5/+1
2023-04-17Spelling - compilerJosh Soref-1/+1
* account * achieved * advising * always * ambiguous * analysis * annotations * appropriate * build * candidates * cascading * category * character * clarification * compound * conceptually * constituent * consts * convenience * corresponds * debruijn * debug * debugable * debuggable * deterministic * discriminant * display * documentation * doesn't * ellipsis * erroneous * evaluability * evaluate * evaluation * explicitly * fallible * fulfill * getting * has * highlighting * illustrative * imported * incompatible * infringing * initialized * into * intrinsic * introduced * javascript * liveness * metadata * monomorphization * nonexistent * nontrivial * obligation * obligations * offset * opaque * opportunities * opt-in * outlive * overlapping * paragraph * parentheses * poisson * precisely * predecessors * predicates * preexisting * propagated * really * reentrant * referent * responsibility * rustonomicon * shortcircuit * simplifiable * simplifications * specify * stabilized * structurally * suggestibility * translatable * transmuting * two * unclosed * uninhabited * visibility * volatile * workaround Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>
2023-04-12Rollup merge of #110135 - compiler-errors:revert-108031, r=davidtwcoMatthias Krüger-7/+2
Revert "Don't recover lifetimes/labels containing emojis as character literals" Reverts PR #108031 per https://github.com/rust-lang/rust/pull/109754#issuecomment-1490452045 Fixes (doesnt close until beta backported) #109746 This reverts commit e3f9db5fc319c6d8eee5d47d216ea6a426070c41. This reverts commit 98b82aedba3f3f581e89df54352914b27f42c6f7. This reverts commit 380fa264132ad481e73cbbf0f3a0feefd99a1d78.
2023-04-10Revert "Don't recover lifetimes/labels containing emojis as character literals"Michael Goulet-7/+2
Reverts PR #108031 Fixes (doesnt close until beta backported) #109746 This reverts commit e3f9db5fc319c6d8eee5d47d216ea6a426070c41. This reverts commit 98b82aedba3f3f581e89df54352914b27f42c6f7. This reverts commit 380fa264132ad481e73cbbf0f3a0feefd99a1d78.
2023-04-09Remove identity castsNilstrieb-2/+2
2023-04-04Replace another lock with an append-only vecOli Scherer-1/+1
2023-02-28refactor parse_token_trees to not return unmatched_delimsyukang-3/+26
2023-02-28rename unmatched_braces to unmatched_delimsyukang-2/+2
2023-02-14Don't recover lifetimes/labels containing emojis as character literals许杰友 Jieyou Xu (Joe)-2/+7
Note that at the time of this commit, `unic-emoji-char` seems to have data tables only up to Unicode 5.0, but Unicode is already newer than this. A newer emoji such as `🥺` will not be recognized as an emoji but older emojis such as `🐱` will.
2023-02-06Migrate `rustc_parse` to derive diagnosticsclubby789-112/+47
2023-01-27Improve unexpected close and mismatch delimiter hint in TokenTreesReaderyukang-0/+1
2023-01-14Emit only one nbsp error per fileDavid Tolnay-5/+26
2023-01-12Emit a single error for contiguous sequences of Unicode homoglyphsclubby789-4/+24
2022-11-16Use `token::Lit` in `ast::ExprKind::Lit`.Nicholas Nethercote-1/+12
Instead of `ast::Lit`. Literal lowering now happens at two different times. Expression literals are lowered when HIR is crated. Attribute literals are lowered during parsing. This commit changes the language very slightly. Some programs that used to not compile now will compile. This is because some invalid literals that are removed by `cfg` or attribute macros will no longer trigger errors. See this comment for more details: https://github.com/rust-lang/rust/pull/102944#issuecomment-1277476773
2022-11-09Rollup merge of #103919 - nnethercote:unescaping-cleanups, r=matkladDylan DPC-59/+48
Unescaping cleanups Some code improvements, and some error message improvements. Best reviewed one commit at a time. r? ````@matklad````
2022-11-07Make underscore_literal_suffix a hard error.Nicholas Nethercote-11/+1
It's been a warning for 5.5 years. Time to make it a hard error. Closes #42326.
2022-11-04Inline and remove `validate_int_literal`.Nicholas Nethercote-18/+13
It has a single callsite, and is fairly small. The `Float` match arm already has base-specific checking inline, so this makes things more consistent.
2022-11-04Refactor `cook_lexer_literal`.Nicholas Nethercote-42/+36
It deals with eight cases: ints, floats, and the six quoted types (char/byte/strings). For ints and floats we have an early return, and the other six types fall through to the code at the end, which makes the function hard to read. This commit rearranges things to avoid the early returns.
2022-10-22Recover unclosed char literal being parsed as lifetimeMichael Goulet-2/+7
2022-10-03Merge `parse_token_trees_until_close_delim` and `parse_all_token_trees`.Nicholas Nethercote-1/+1
Because they're very similar, and this will allow some follow-up changes.
2022-09-28Address review comments.Nicholas Nethercote-1/+1
2022-09-26Inline and remove `cook_lexer_token`.Nicholas Nethercote-171/+174
This is a small performance win, alas.
2022-09-26Add `rustc_lexer::TokenKind::Eof`.Nicholas Nethercote-8/+2
For alignment with `rust_ast::TokenKind::Eof`. Plus it's a bit faster, due to less `Option` manipulation in `StringReader::next_token`.
2022-09-26Use less DRY in `cook_lexer_token`.Nicholas Nethercote-19/+19
This is a case where a small amount of repetition results in code that is faster and easier to read.
2022-09-26Make `rustc_lexer::cursor::Cursor` public.Nicholas Nethercote-10/+13
`Cursor` is currently hidden, and the main tokenization path uses `rustc_lexer::first_token` which involves constructing a new `Cursor` for every single token, which is weird. Also, `first_token` also can't handle empty input, so callers have to check for that first. This commit makes `Cursor` public, so `StringReader` can contain a `Cursor`, which results in a simpler structure. The commit also changes `StringReader::advance_token` so it returns an `Option<Token>`, simplifying the the empty input case.
2022-09-26[ui] Rearrange `StringReader`/`TokenTreesReader` creation.Nicholas Nethercote-1/+2
`TokenTreesReader` wraps a `StringReader`, but the `into_token_trees` function obscures this. This commit moves to a more straightforward control flow.
2022-09-26Clarify spacing computation.Nicholas Nethercote-7/+8
The spacing computation is done in two parts. In the first part `next_token` and `bump` use `Spacing::Alone` to mean "preceded by whitespace" and `Spacing::Joint` to mean the opposite. In the second part `parse_token_tree_other` then adjusts the `spacing` value to mean the usual thing (i.e. "is the following token joinable punctuation?"). This shift in meaning is very confusing and it took me some time to understand what was going on. This commit changes the first part to use a bool, and adds some comments, which makes things much clearer.
2022-09-26Move `#!` checking.Nicholas Nethercote-9/+8
Currently does the "is this a `#!` at the start of the file?" check for every single token(!) This commit moves it so it only happens once.
2022-09-26Remove unnecessary `spacing` assignment.Nicholas Nethercote-1/+0
It has no useful effect.
2022-09-01Always import all tracing macros for the entire crate instead of piecemeal ↵Oli Scherer-2/+0
by module
2022-08-01Shrink `Token`.Nicholas Nethercote-23/+29
From 72 bytes to 12 bytes (on x86-64). There are two parts to this: - Changing various source code offsets from 64-bit to 32-bit. This is not a problem because the rest of rustc also uses 32-bit source code offsets. This means `Token` is no longer `Copy` but this causes no problems. - Removing the `RawStrError` from `LiteralKind`. Raw string literal invalidity is now indicated by a `None` value within `RawStr`/`RawByteStr`, and the new `validate_raw_str` function can be used to re-lex an invalid raw string literal to get the `RawStrError`. There is one very small change in behaviour. Previously, if a raw string literal matched both the `InvalidStarter` and `TooManyHashes` cases, the latter would override the former. This has now changed, because `raw_double_quoted_string` now uses `?` and so returns immediately upon detecting the `InvalidStarter` case. I think this is a slight improvement to report the earlier-detected error, and it explains the change in the `test_too_many_hashes` test. The commit also removes a couple of comments that refer to #77629 and say that the size of these types don't affect performance. These comments are wrong, though the performance effect is small.
2022-08-01Add a size assertion for `Token`.Nicholas Nethercote-0/+7
2022-08-01Remove `StringReader::end_src_index`.Nicholas Nethercote-5/+2
It not needed, always being set to the end of the text.
2022-08-01Improve shebang handling.Nicholas Nethercote-8/+5
Avoid doing stuff until it's necessary.
generated by cgit-pink 1.4.1-3-g733a5 (git 2.37.1) at 2026-02-24 06:47:32 +0000