about summary refs log tree commit diff
path: root/src/libsyntax/parse/lexer
AgeCommit message (Collapse)AuthorLines
2019-06-12Auto merge of #60669 - c410-f3r:attrs-fn, r=petrochenkovbors-0/+1
Allow attributes in formal function parameters Implements https://github.com/rust-lang/rust/issues/60406. This is my first contribution to the compiler and since this is a large and complex project, I am not fully aware of the consequences of the changes I have made. **TODO** - [x] Forbid some built-in attributes. - [x] Expand cfg/cfg_attr
2019-06-10Auto merge of #60793 - Xanewok:raw-string-cleanup, r=petrochenkovbors-100/+58
lexer: Disallow bare CR in raw byte strings Handles bare CR ~but doesn't translate `\r\n` to `\n` yet in raw strings yet~ and translates CRLF to LF in raw strings. As a side-note I think it'd be good to change the `unescape_` to return plain iterators to reduce some boilerplate (e.g. `has_error` could benefit from collecting `Result<T>` and aborting early on errors) but will do that separately, unless I missed something here that prevents it. @matklad @petrochenkov thoughts?
2019-06-09Actually translate CRLF in raw byte strings and unify unescape implIgor Matuszewski-2/+2
2019-06-09Add a doc comment for scan_raw_stringIgor Matuszewski-0/+2
2019-06-09Allow attributes in formal function parametersCaio-0/+1
2019-06-09Translate CRLF -> LF in raw (byte) stringsIgor Matuszewski-2/+2
2019-06-08Prohibit bare CRs in raw byte stringsIgor Matuszewski-70/+24
2019-06-08Validate and transcribe raw strings via unescape moduleIgor Matuszewski-21/+24
2019-06-08syntax: Remove `Deref` impl from `Token`Vadim Petrochenkov-1/+1
2019-06-08syntax: Move most of the `TokenKind` methods to `Token`Vadim Petrochenkov-3/+3
2019-06-08Remove redundant, commented out codeIgor Matuszewski-6/+0
It was commented out as part of https://github.com/rust-lang/rust/commit/8a8e497ae786ffc032c1e68fc23da0edcf6fa5e3. Done probably by accident, since the code in question was moved to a match arm, along with newly introduced logic to detect bare CRs in raw strings.
2019-06-08Separate a `scan_raw_string` (similar `raw_byte` variant)Igor Matuszewski-77/+82
2019-06-08Clean up minor bitsIgor Matuszewski-1/+1
2019-06-06syntax: Remove duplicate span from `token::Ident`Vadim Petrochenkov-16/+6
2019-06-06syntax: Remove duplicate span from `token::Lifetime`Vadim Petrochenkov-9/+5
2019-06-06syntax: Add some helper methods to `Token`Vadim Petrochenkov-39/+26
2019-06-06syntax: Use `Token` in `StringReader` and `TokenTreesReader`Vadim Petrochenkov-46/+32
2019-06-06syntax: Use `Token` in `TokenTree::Token`Vadim Petrochenkov-23/+19
2019-06-06syntax: Rename `TokenAndSpan` into `Token`Vadim Petrochenkov-79/+64
2019-06-06syntax: Rename `Token` into `TokenKind`Vadim Petrochenkov-17/+17
2019-05-27Pre-intern "0", "1", ..., "9", and use where appropriate.Nicholas Nethercote-2/+2
2019-05-23syntax: Turn `token::Lit` into a structVadim Petrochenkov-54/+51
2019-05-21Move `edition` outside the hygiene lock and avoid accessing itJohn Kåre Alsaker-15/+16
2019-05-16Auto merge of #60763 - matklad:tt-parser, r=petrochenkovbors-37/+56
Move token tree related lexer state to a separate struct Just a types-based refactoring. We only used a bunch of fields when tokenizing into a token tree, so let's move them out of the base lexer
2019-05-13move raw span to tt readerAleksey Kladov-1/+2
See https://github.com/rust-lang/rust/pull/50838/files#r283296243 for explanation how jointness checking works with *next* pair
2019-05-13move span and token to tt readerAleksey Kladov-38/+34
2019-05-13Move token tree related lexer state to a separate structAleksey Kladov-44/+66
We only used a bunch of fields when tokenizing into a token tree, so let's move them out of the base lexer
2019-05-11Move literal parsing code into a separate fileVadim Petrochenkov-13/+6
Remove some dead code
2019-05-11Eliminate `comments::Literal`Vadim Petrochenkov-26/+3
2019-05-09Rollup merge of #60188 - estebank:recover-block, r=varkorMazdak Farrokhzad-1/+2
Identify when a stmt could have been parsed as an expr There are some expressions that can be parsed as a statement without a trailing semicolon depending on the context, which can lead to confusing errors due to the same looking code being accepted in some places and not others. Identify these cases and suggest enclosing in parenthesis making the parse non-ambiguous without changing the accepted grammar. Fix #54186, cc #54482, fix #59975, fix #47287.
2019-05-06review comments: fix typo and add commentsEsteban Küber-1/+1
2019-05-02introduce unescape moduleAleksey Kladov-443/+142
Currently, we deal with escape sequences twice: once when we lex a string, and a second time when we unescape literals. This PR aims to remove this duplication, by introducing a new `unescape` mode as a single source of truth for character escaping rules
2019-04-29Identify when a stmt could have been parsed as an exprEsteban Küber-1/+2
There are some expressions that can be parsed as a statement without a trailing semicolon depending on the context, which can lead to confusing errors due to the same looking code being accepted in some places and not others. Identify these cases and suggest enclosing in parenthesis making the parse non-ambiguous without changing the accepted grammar.
2019-04-23simplify and avoid allocationAleksey Kladov-2/+2
2019-04-23remove obsolete and incorrect commentAleksey Kladov-3/+1
2019-04-20Auto merge of #59700 - matklad:simplify, r=eddybbors-17/+5
Simplify doc comment lexing is_doc_comment function checks the first four chars, but this is redundant, `doc_comment` local var has the same info.
2019-04-17Rollup merge of #59128 - oli-obk:colorful_json, r=mark-i-m,eddybMazdak Farrokhzad-0/+1
Emit ansi color codes in the `rendered` field of json diagnostics cc @ljedrz Implemented for https://github.com/rust-lang/rust/pull/56595#issuecomment-447645115 (x.py clippy)
2019-04-05Rollup merge of #59687 - matklad:shebang, r=petrochenkovMazdak Farrokhzad-11/+5
cleanup shebang handling in the lexer
2019-04-04Simplify doc comment lexingAleksey Kladov-17/+5
is_doc_comment function checks the first four chars, but this is redundant, `doc_comment` local var has the same info.
2019-04-04cleanup shebang handling in the lexerAleksey Kladov-11/+5
2019-04-03make StringReader methods privateAleksey Kladov-2/+2
2019-04-03make StringReader fields privateAleksey Kladov-6/+6
2019-04-02Update more unit test to new APIOliver Scherer-0/+1
2019-03-20Tweak incorrect escaped char diagnosticEsteban Küber-19/+19
2019-03-16syntax: Introduce `Ident::can_be_raw`Vadim Petrochenkov-8/+4
2019-03-13Rollup merge of #58876 - estebank:numeric-lifetime, r=petrochenkovMazdak Farrokhzad-3/+15
Parse lifetimes that start with a number and give specific error Fix #58786.
2019-03-09review commentsEsteban Küber-4/+4
2019-03-08Parse lifetimes that start with a number and give specific errorEsteban Küber-6/+18
2019-03-07Fix with_emitter callersEsteban Küber-1/+1
2019-02-16help suggestion when trying to delimit string literals with directed unicode ↵Patrick McCarter-9/+44
quotes #58436
generated by cgit-pink 1.4.1-3-g733a5 (git 2.37.1) at 2026-02-24 19:29:13 +0000