about summary refs log tree commit diff
path: root/compiler/rustc_lexer
AgeCommit message (Collapse)AuthorLines
2025-09-22Fix a crash/mislex when more than one frontmatter closing possibility is ↵Alex-1/+3
considered
2025-09-03Rollup merge of #146106 - epage:whitespace, r=fee1-deadStuart Cook-15/+28
fix(lexer): Only allow horizontal whitespace in frontmatter In writing up the reference for frontmatter, I realized that we probably shouldn't be accepting Unicode Line Ending characters between the code fence and infostring or trailing after the infostring or a code fence. In digging into the unicode specification we use for Whitespace, it divides it up into categories, so I'm deferring to what it says for horizontal whitespace for what should be used within a line. Note, I am leaving out support for Unicode Default Ignorable characters. I figure that can be discussed outside of this change within the reference and tracking issue. Fixes rust-lang/rust#145971 Frontmatter tracking issue: rust-lang/rust#136889
2025-09-02Revert introduction of `[workspace.dependencies]`.Nicholas Nethercote-1/+1
This was done in #145740 and #145947. It is causing problems for people using r-a on anything that uses the rustc-dev rustup package, e.g. Miri, clippy. This repository has lots of submodules and subtrees and various different projects are carved out of pieces of it. It seems like `[workspace.dependencies]` will just be more trouble than it's worth.
2025-09-01fix(lexer): Only allow horizontal whitespace in frontmatterEd Page-3/+18
In writing up the reference for frontmatter, I realized that we probably shouldn't be accepting Unicode Line Ending characters between the code fence and infostring or trailing after the infostring or a code fence. In digging into the unicode specification we use for Whitespace, it divides it up into categories, so I'm deferring to what it says for horizontal whitespace for what should be used within a line. Note, I am leaving out support for Unicde Default Ignorable characters. I figure that can be discussed outside of this change within the reference and tracking issue.
2025-09-01docs(lexer): Organize and document whitespace by Pattern_White_SpaceEd Page-12/+10
2025-08-28fix(lexer): Don't require frontmatters to be escaped with indented fencesEd Page-21/+13
The RFC only limits hyphens at the beginning of lines and not if they are indented or embedded in other content. Sticking to that approach was confirmed by the T-lang liason at https://github.com/rust-lang/rust/issues/141367#issuecomment-3202217544 There is a regression in error message quality which I'm leaving for someone if they feel this needs improving.
2025-08-28Remove unnecessary `[dependencies.unicode-properties]` entries.Nicholas Nethercote-5/+1
The Cargo style guide says to put dependencies on a single line if they fit.
2025-08-27Add `memchr` to `[workspace.dependencies]`.Nicholas Nethercote-1/+1
2025-08-22fix(lexer): Allow '-' in the infostring continue setEd Page-2/+2
This more closely matches the RFC and what our T-lang contact has asked for, see https://github.com/rust-lang/rust/issues/136889#issuecomment-3212715312
2025-07-10test(lexer): Add frontmatter unit testEd Page-2/+84
2025-07-09feat(lexer): Allow including frontmatter with 'tokenize'Ed Page-11/+21
2025-07-03setup CI and tidy to use typos for spellchecking and fix few typosklensy-1/+1
2025-06-06rustc_lexer: typo fix + small cleanupsMarijn Schouten-10/+8
2025-06-04Fix parsing of frontmatters with inner hyphensMatthew Jasper-1/+3
2025-05-05Implement RFC 3503: frontmattersDeadbeef-9/+162
Supercedes #137193
2025-04-04Replace `rustc_lexer/unescape` with `rustc-literal-escaper` crateGuillaume Gomez-725/+0
2025-03-18Revert "Rollup merge of #136355 - ↵Ralf Jung-4/+725
GuillaumeGomez:proc-macro_add_value_retrieval_methods, r=Amanieu" This reverts commit 08dfbf49e30d917c89e49eb14cb3f1e8b8a1c9ef, reversing changes made to 10bcdad7df0de3cfb95c7bdb7b16908e73cafc09.
2025-03-17Rollup merge of #136355 - ↵Jacob Pratt-725/+4
GuillaumeGomez:proc-macro_add_value_retrieval_methods, r=Amanieu Add `*_value` methods to proc_macro lib This is the implementation of https://github.com/rust-lang/libs-team/issues/459. It allows to get the actual value (unescaped) of the different string literals. Part of https://github.com/rust-lang/rust/issues/136652. r? libs-api
2025-03-11Remove `#![warn(unreachable_pub)]` from all `compiler/` crates.Nicholas Nethercote-1/+0
It's no longer necessary now that `-Wunreachable_pub` is being passed.
2025-03-10Revert "Use workspace lints for crates in `compiler/` #138084"许杰友 Jieyou Xu (Joe)-3/+1
Revert <https://github.com/rust-lang/rust/pull/138084> to buy time to consider options that avoids breaking downstream usages of cargo on distributed `rustc-src` artifacts, where such cargo invocations fail due to inability to inherit `lints` from workspace root manifest's `workspace.lints` (this is only valid for the source rust-lang/rust workspace, but not really the distributed `rustc-src` artifacts). This breakage was reported in <https://github.com/rust-lang/rust/issues/138304>. This reverts commit 48caf81484b50dca5a5cebb614899a3df81ca898, reversing changes made to c6662879b27f5161e95f39395e3c9513a7b97028.
2025-03-08Remove `#![warn(unreachable_pub)]` from all `compiler/` crates.Nicholas Nethercote-1/+0
(Except for `rustc_codegen_cranelift`.) It's no longer necessary now that `unreachable_pub` is in the workspace lints.
2025-03-08Specify rust lints for `compiler/` crates via Cargo.Nicholas Nethercote-0/+3
By naming them in `[workspace.lints.rust]` in the top-level `Cargo.toml`, and then making all `compiler/` crates inherit them with `[lints] workspace = true`. (I omitted `rustc_codegen_{cranelift,gcc}`, because they're a bit different.) The advantages of this over the current approach: - It uses a standard Cargo feature, rather than special handling in bootstrap. So, easier to understand, and less likely to get accidentally broken in the future. - It works for proc macro crates. It's a shame it doesn't work for rustc-specific lints, as the comments explain.
2025-02-22Greatly simplify lifetime captures in edition 2024Michael Goulet-1/+1
2025-02-22Upgrade the compiler to edition 2024Michael Goulet-1/+1
2025-02-10Reexport `literal-escaper` from `rustc_lexer` to allow rust-analyzer to compileGuillaume Gomez-0/+4
2025-02-10Extract `unescape` from `rustc_lexer` into its own crateGuillaume Gomez-725/+0
2025-02-08Rustfmtbjorn3-16/+34
2025-02-05implement `eat_until` leveraging memchr in lexergvozdvmozgu-2/+10
2024-12-09Add test to check unicode identifier versionEric Huss-0/+1
2024-12-01Revert "Reject raw lifetime followed by \' as well"Michael Goulet-11/+1
This reverts commit 1990f1560801ca3f9e6a3286e58204aa329ee037.
2024-11-25Clean up `c_or_byte_string`.Nicholas Nethercote-4/+4
- Rename a misleading local `mk_kind` as `single_quoted`. - Use `fn` for all three arguments, for consistency.
2024-11-25Improve `strip_shebang` testing.Nicholas Nethercote-41/+31
It's currently a bit ad hoc. This commit makes it more methodical, with pairs of match/no-match tests for all the relevant cases.
2024-11-19Remove `TokenKind::InvalidPrefix`.Nicholas Nethercote-13/+8
It was added in #123752 to handle some cases involving emoji, but it isn't necessary because it's always treated the same as `TokenKind::InvalidIdent`. This commit removes it, which makes things a little simpler.
2024-11-19Improve `TokenKind` comments.Nicholas Nethercote-41/+39
- Improve wording. - Use backticks consistently for examples.
2024-11-19Improve `fake_ident_or_unknown_prefix`.Nicholas Nethercote-6/+5
- Rename it as `invalid_ident_or_prefix`, which matches the possible outputs (`InvalidIdent` or `InvalidPrefix`). - Use the local wrapper for `is_xid_continue`, for consistency. - Make it clear what `\u{200d}` means.
2024-10-30Reject raw lifetime followed by \' as wellMichael Goulet-1/+11
2024-10-08Reserve guarded string literals (RFC 3593)Peter Jaszkowiak-8/+85
2024-09-22Reformat using the new identifier sorting from rustfmtMichael Goulet-35/+17
2024-09-06Add initial support for raw lifetimesMichael Goulet-1/+12
2024-09-06Format lexerMichael Goulet-1/+1
2024-09-06Reserve prefix lifetimes tooMichael Goulet-9/+17
2024-08-27Add `warn(unreachable_pub)` to `rustc_lexer`.Nicholas Nethercote-0/+1
2024-07-29Reformat `use` declarations.Nicholas Nethercote-4/+4
The previous commit updated `rustfmt.toml` appropriately. This commit is the outcome of running `x fmt --all` with the new formatting options.
2024-06-12Use `tidy` to sort crate attributes for all compiler crates.Nicholas Nethercote-0/+2
We already do this for a number of crates, e.g. `rustc_middle`, `rustc_span`, `rustc_metadata`, `rustc_span`, `rustc_errors`. For the ones we don't, in many cases the attributes are a mess. - There is no consistency about order of attribute kinds (e.g. `allow`/`deny`/`feature`). - Within attribute kind groups (e.g. the `feature` attributes), sometimes the order is alphabetical, and sometimes there is no particular order. - Sometimes the attributes of a particular kind aren't even grouped all together, e.g. there might be a `feature`, then an `allow`, then another `feature`. This commit extends the existing sorting to all compiler crates, increasing consistency. If any new attribute line is added there is now only one place it can go -- no need for arbitrary decisions. Exceptions: - `rustc_log`, `rustc_next_trait_solver` and `rustc_type_ir_macros`, because they have no crate attributes. - `rustc_codegen_gcc`, because it's quasi-external to rustc (e.g. it's ignored in `rustfmt.toml`).
2024-04-24Improved the compiler code with clippyMichael Scholten-5/+5
2024-04-10Properly handle emojis as literal prefix in macrosEsteban Küber-1/+5
Do not accept the following ```rust macro_rules! lexes {($($_:tt)*) => {}} lexes!(🐛"foo"); ``` Before, invalid emoji identifiers were gated during parsing instead of lexing in all cases, but this didn't account for macro expansion of literal prefixes. Fix #123696.
2024-03-17Silence redundant error on char literal that was meant to be a string in ↵Esteban Küber-0/+9
2021 edition
2024-03-17Handle str literals written with `'` lexed as lifetimeEsteban Küber-1/+1
Given `'hello world'` and `'1 str', provide a structured suggestion for a valid string literal: ``` error[E0762]: unterminated character literal --> $DIR/lex-bad-str-literal-as-char-3.rs:2:26 | LL | println!('hello world'); | ^^^^ | help: if you meant to write a `str` literal, use double quotes | LL | println!("hello world"); | ~ ~ ``` ``` error[E0762]: unterminated character literal --> $DIR/lex-bad-str-literal-as-char-1.rs:2:20 | LL | println!('1 + 1'); | ^^^^ | help: if you meant to write a `str` literal, use double quotes | LL | println!("1 + 1"); | ~ ~ ``` Fix #119685.
2024-02-06Invert diagnostic lints.Nicholas Nethercote-2/+1
That is, change `diagnostic_outside_of_impl` and `untranslatable_diagnostic` from `allow` to `deny`, because more than half of the compiler has be converted to use translated diagnostics. This commit removes more `deny` attributes than it adds `allow` attributes, which proves that this change is warranted.
2024-01-25Use `unescape_unicode` for raw C string literals.Nicholas Nethercote-19/+14
They can't contain `\x` escapes, which means they can't contain high bytes, which means we can used `unescape_unicode` instead of `unescape_mixed` to unescape them. This avoids unnecessary used of `MixedUnit`.
generated by cgit-pink 1.4.1-3-g733a5 (git 2.37.1) at 2026-01-14 00:59:21 +0000