summary refs log tree commit diff
path: root/compiler/rustc_lexer/src/lib.rs
AgeCommit message (Collapse)AuthorLines
2021-12-03Auto merge of #91393 - Julian-Wollersberger:lexer_optimization, r=petrochenkovbors-13/+7
Optimize `rustc_lexer` The `cursor.first()` method in `rustc_lexer` now calls the `chars.next()` method instead of `chars.nth_char(0)`. This allows LLVM to optimize the code better. The biggest win is that `eat_while()` is now fully inlined and generates better assembly. This improves the lexer's performance by 35% in a micro-benchmark I made (Lexing all 18MB of code in the compiler directory). But lexing is only a small part of the overall compilation time, so I don't know how significant it is. Big thanks to criterion and `cargo asm`.
2021-12-01Replace `nth_char(0)` with `next()` in `cursor.first()`Julian Wollersberger-13/+7
and optimize the iterator returned by `tokenize(). This improves lexer performance by 35%
2021-11-23udpate comment to be more accurateEsteban Kuber-2/+1
2021-11-23Tokenize emoji as if they were valid indentifiersEsteban Kuber-0/+25
In the lexer, consider emojis to be valid identifiers and reject them later to avoid knock down parse errors.
2021-08-22Fix more “a”/“an” typosFrank Steffahn-1/+1
2021-07-26Remove ASCII fast path from rustc_lexer::{is_id_continue, is_id_start}Ibraheem Ahmed-12/+2
2021-06-26Rename 'bad prefix' to 'unknown prefix'.Mara Bos-8/+8
2021-06-26Improve comments for reserved prefixes.Mara Bos-2/+7
Co-authored-by: Niko Matsakis <niko@alum.mit.edu>
2021-06-26Reserve prefixed identifiers and string literals (RFC 3101)lrh2000-5/+12
This commit denies any identifiers immediately followed by one of three tokens `"`, `'` or `#`, which is stricter than the requirements of RFC 3101 but may be necessary according to the discussion at [Zulip]. [Zulip]: https://rust-lang.zulipchat.com/#narrow/stream/268952-edition-2021/topic/reserved.20prefixes/near/238470099
2021-04-08Fix outdated crate names in compiler docspierwill-3/+3
Changes `librustc_X` to `rustc_X`, only in documentation comments. Plain code comments are left unchanged. Also fix incorrect file paths.
2020-12-03Fix some clippy lintsJoshua Nelson-5/+5
2020-10-30Add back missing commentsJoshua Nelson-0/+1
2020-10-30Fix even more clippy warningsJoshua Nelson-6/+4
2020-10-09Noticed a potential bug in `eat_while()`: it doesn't account for number of ↵Julian Wollersberger-20/+14
UTF8 bytes. Fixed it by inlining it in the two places where the count is used and simplified the logic there.
2020-09-21Fix typo in rustc_lexer docsLingMan-1/+1
Also add an Oxford comma while we're editing that line.
2020-09-02lexer: Tiny improvement to shebang detectionVadim Petrochenkov-5/+9
Lexer now discerns between regular comments and doc comments, so use that. The change only affects the choice of reported errors.
2020-08-30mv compiler to compiler/mark-0/+819