about summary refs log tree commit diff
path: root/src/libsyntax/parse/lexer/mod.rs
AgeCommit message (Collapse)AuthorLines
2019-08-19remove composite tokens support from the lexerAleksey Kladov-25/+0
2019-08-18Auto merge of #62948 - matklad:failable-file-loading, r=petrochenkovbors-66/+15
Normalize newlines when loading files Fixes #62865
2019-08-15syntax_pos: `NO_EXPANSION`/`SyntaxContext::empty()` -> `SyntaxContext::root()`Vadim Petrochenkov-2/+2
For consistency with `ExpnId::root`. Also introduce a helper `Span::with_root_ctxt` for creating spans with `SyntaxContext::root()` context
2019-08-14remove special handling of \r\n from the lexerAleksey Kladov-66/+15
2019-08-05add unknown tokenAleksey Kladov-1/+1
2019-08-05remove special code path for unknown tokensAleksey Kladov-60/+13
2019-08-02libsyntax: Unconfigure tests during normal buildVadim Petrochenkov-259/+3
2019-07-31cleanup StringReader fieldsAleksey Kladov-12/+11
2019-07-27syntax_ext: `proc_macro_decls` -> `proc_macro_harness`Vadim Petrochenkov-1/+2
Few other minor renamings for consistency. Remove one unused dependency from `rustc_passes`. Fix libsyntax tests. Fix rebase.
2019-07-25review comments: add FIXME comments and formattingEsteban Küber-4/+11
2019-07-24Allow lexer to recover from some homoglyphsEsteban Küber-1/+4
2019-07-23Rollup merge of #62851 - matklad:unescape, r=petrochenkovMark Rousskov-1/+1
move unescape module to rustc_lexer It makes sense to keep the definition of escape sequences closer to the lexer itself, and it is also a bit of code that I would like to share with rust-analyzer. r? @petrochenkov
2019-07-22fix lexing of comments with many \rAleksey Kladov-1/+1
closes #62863
2019-07-21move unescape module to rustc_lexerAleksey Kladov-1/+1
2019-07-20Introduce rustc_lexerAleksey Kladov-1014/+433
The idea here is to make a reusable library out of the existing rust-lexer, by separating out pure lexing and rustc-specific concerns, like spans, error reporting an interning. So, rustc_lexer operates directly on `&str`, produces simple tokens which are a pair of type-tag and a bit of original text, and does not report errors, instead storing them as flags on the token.
2019-07-16normalize use of backticks in compiler messages for libsyntax/parseSamy Kacimi-1/+1
https://github.com/rust-lang/rust/issues/60532
2019-07-07syntax: Pre-intern names of all built-in macrosVadim Petrochenkov-1/+1
They always end up interned anyway
2019-07-06Rollup merge of #62329 - matklad:no-peeking, r=petrochenkovMazdak Farrokhzad-144/+74
Remove support for 1-token lookahead from the lexer `StringReader` maintained `peek_token` and `peek_span_src_raw` for look ahead. `peek_token` was used only by rustdoc syntax coloring. After moving peeking logic into highlighter, I was able to remove `peek_token` from the lexer. I tried to use `iter::Peekable`, but that wasn't as pretty as I hoped, due to buffered fatal errors. So I went with hand-rolled peeking. After that I've noticed that the only peeking behavior left was for raw tokens to test tt jointness. I've rewritten it in terms of trivia tokens, and not just spans. After that it became possible to simplify the awkward constructor of the lexer, which could return `Err` if the first peeked token contained error.
2019-07-05Rollup merge of #62292 - Centril:split-async-closures, r=cramertjMazdak Farrokhzad-0/+1
Move `async || ...` closures into `#![feature(async_closure)]` The `async || expr` syntax is moved out from `#![feature(async_await)]` into its own gate `#![feature(async_closure)]`. New tracking issue: https://github.com/rust-lang/rust/issues/62290 Closes https://github.com/rust-lang/rust/issues/62214. cc https://github.com/rust-lang/rust/issues/62149 r? @varkor
2019-07-04make unwrap_or_abort non-generic againAleksey Kladov-1/+1
2019-07-04remove unused mk_sp_and_rawAleksey Kladov-10/+3
2019-07-04don't rely on spans when checking tokens for jointnessAleksey Kladov-35/+11
2019-07-04slightly comment lexer APIAleksey Kladov-11/+18
2019-07-04move constructors to topAleksey Kladov-51/+51
2019-07-04cleanup lexer constructorsAleksey Kladov-13/+5
2019-07-04remove peek_span_src_raw from StringReaderAleksey Kladov-61/+37
2019-07-04remove peek_token from StringReaderAleksey Kladov-17/+7
2019-07-04remove StringReader::peekAleksey Kladov-4/+0
The reader itself doesn't need ability to peek tokens, so it's better if clients implement this functionality. This hopefully becomes especially easy once we use iterator interface for lexer, but this is not too easy at the moment, because of buffered errors.
2019-07-04Rollup merge of #62297 - matklad:peek-delimited, r=petrochenkovMazdak Farrokhzad-22/+0
refactor check_for_substitution No behavior change, just flatter and simpler code. r? @petrochenkov
2019-07-03Add separate 'async_closure' feature gate.Mazdak Farrokhzad-0/+1
2019-07-02refactor check_for_substitutionAleksey Kladov-22/+0
No behavior change, just flatter and simpler code
2019-06-25cleanup: rename name_from to symbol_fromAleksey Kladov-20/+19
Lexer uses Symbols for a lot of stuff, not only for identifiers, so the "name" terminology is just confusing.
2019-06-25refactor lexer to use idiomatic borrowingAleksey Kladov-121/+104
2019-06-23let_chains: Inline visit_expr_with_let_maybe_allowed.Mazdak Farrokhzad-0/+1
2019-06-12Auto merge of #60669 - c410-f3r:attrs-fn, r=petrochenkovbors-0/+1
Allow attributes in formal function parameters Implements https://github.com/rust-lang/rust/issues/60406. This is my first contribution to the compiler and since this is a large and complex project, I am not fully aware of the consequences of the changes I have made. **TODO** - [x] Forbid some built-in attributes. - [x] Expand cfg/cfg_attr
2019-06-10Auto merge of #60793 - Xanewok:raw-string-cleanup, r=petrochenkovbors-100/+58
lexer: Disallow bare CR in raw byte strings Handles bare CR ~but doesn't translate `\r\n` to `\n` yet in raw strings yet~ and translates CRLF to LF in raw strings. As a side-note I think it'd be good to change the `unescape_` to return plain iterators to reduce some boilerplate (e.g. `has_error` could benefit from collecting `Result<T>` and aborting early on errors) but will do that separately, unless I missed something here that prevents it. @matklad @petrochenkov thoughts?
2019-06-09Actually translate CRLF in raw byte strings and unify unescape implIgor Matuszewski-2/+2
2019-06-09Add a doc comment for scan_raw_stringIgor Matuszewski-0/+2
2019-06-09Allow attributes in formal function parametersCaio-0/+1
2019-06-09Translate CRLF -> LF in raw (byte) stringsIgor Matuszewski-2/+2
2019-06-08Prohibit bare CRs in raw byte stringsIgor Matuszewski-70/+24
2019-06-08Validate and transcribe raw strings via unescape moduleIgor Matuszewski-21/+24
2019-06-08syntax: Move most of the `TokenKind` methods to `Token`Vadim Petrochenkov-3/+3
2019-06-08Remove redundant, commented out codeIgor Matuszewski-6/+0
It was commented out as part of https://github.com/rust-lang/rust/commit/8a8e497ae786ffc032c1e68fc23da0edcf6fa5e3. Done probably by accident, since the code in question was moved to a match arm, along with newly introduced logic to detect bare CRs in raw strings.
2019-06-08Separate a `scan_raw_string` (similar `raw_byte` variant)Igor Matuszewski-77/+82
2019-06-08Clean up minor bitsIgor Matuszewski-1/+1
2019-06-06syntax: Remove duplicate span from `token::Ident`Vadim Petrochenkov-16/+6
2019-06-06syntax: Remove duplicate span from `token::Lifetime`Vadim Petrochenkov-9/+5
2019-06-06syntax: Add some helper methods to `Token`Vadim Petrochenkov-37/+24
2019-06-06syntax: Use `Token` in `StringReader` and `TokenTreesReader`Vadim Petrochenkov-23/+12
generated by cgit-pink 1.4.1-3-g733a5 (git 2.37.1) at 2026-02-26 11:44:50 +0000