rust/compiler/rustc_lexer/src, branch masterhttps://github.com/rust-lang/rust
http://git.dreamy.place/mirrors/rust/atom?h=master2025-09-22T19:10:41+00:00Fix a crash/mislex when more than one frontmatter closing possibility is considered2025-09-22T19:10:41+00:00Alexhuangalex409@gmail.com2025-09-22T19:10:41+00:00urn:sha1:2d18c886f5754258109fad2e1e0c3bcca0cc9fa5fix(lexer): Only allow horizontal whitespace in frontmatter2025-09-02T01:51:39+00:00Ed Pageeopage@gmail.com2025-08-28T20:15:41+00:00urn:sha1:6f0da976c5f422ff5b1124d409e7d7e063fc4225
In writing up the reference for frontmatter, I realized that we probably
shouldn't be accepting Unicode Line Ending characters between the code
fence and infostring or trailing after the infostring or a code fence.
In digging into the unicode specification we use for Whitespace, it
divides it up into categories, so I'm deferring to what it says for
horizontal whitespace for what should be used within a line.
Note, I am leaving out support for Unicde Default Ignorable characters.
I figure that can be discussed outside of this change within the
reference and tracking issue.docs(lexer): Organize and document whitespace by Pattern_White_Space2025-09-02T01:51:39+00:00Ed Pageeopage@gmail.com2025-08-28T19:18:42+00:00urn:sha1:428e413414e0719a7758676ac49e07b85eb86decfix(lexer): Don't require frontmatters to be escaped with indented fences2025-08-28T19:08:33+00:00Ed Pageeopage@gmail.com2025-08-22T15:46:47+00:00urn:sha1:142e25e35621b7043cc86bdc3d74f463ca2032d1
The RFC only limits hyphens at the beginning of lines and not if they
are indented or embedded in other content.
Sticking to that approach was confirmed by the T-lang liason at
https://github.com/rust-lang/rust/issues/141367#issuecomment-3202217544
There is a regression in error message quality which I'm leaving for
someone if they feel this needs improving.
fix(lexer): Allow '-' in the infostring continue set2025-08-22T14:26:19+00:00Ed Pageeopage@gmail.com2025-08-22T14:26:19+00:00urn:sha1:f43f974b9e14af328b7982d6320cc4d7a9104410
This more closely matches the RFC and what our T-lang contact has asked
for, see https://github.com/rust-lang/rust/issues/136889#issuecomment-3212715312test(lexer): Add frontmatter unit test2025-07-10T15:25:29+00:00Ed Pageeopage@gmail.com2025-07-10T15:25:29+00:00urn:sha1:df53b3dc04950d980bd28ec4b04f48cd0cd01237feat(lexer): Allow including frontmatter with 'tokenize'2025-07-09T21:42:27+00:00Ed Pageeopage@gmail.com2025-07-09T21:40:22+00:00urn:sha1:45a1e492b1adfc6f7664d1da736dff147e5c3168setup CI and tidy to use typos for spellchecking and fix few typos2025-07-03T07:51:06+00:00klensyklensy@users.noreply.github.com2024-12-07T10:51:08+00:00urn:sha1:c76d032f0144b650a438ee1efba89c475e0b115brustc_lexer: typo fix + small cleanups2025-06-06T13:08:16+00:00Marijn Schoutenmhkbst@gmail.com2025-06-06T13:06:59+00:00urn:sha1:2a5225a369e87ebaac3d34583603479064e2623dFix parsing of frontmatters with inner hyphens2025-06-04T15:51:36+00:00Matthew Jaspermjjasper1@gmail.com2025-06-04T15:51:36+00:00urn:sha1:55f59fb0e329d0815a06d51dd469d774eb455cba