about summary refs log tree commit diff
path: root/src/libsyntax/parse/lexer/mod.rs
AgeCommit message (Collapse)AuthorLines
2019-06-08Prohibit bare CRs in raw byte stringsIgor Matuszewski-70/+24
2019-06-08Validate and transcribe raw strings via unescape moduleIgor Matuszewski-21/+24
2019-06-08syntax: Move most of the `TokenKind` methods to `Token`Vadim Petrochenkov-3/+3
2019-06-08Remove redundant, commented out codeIgor Matuszewski-6/+0
It was commented out as part of https://github.com/rust-lang/rust/commit/8a8e497ae786ffc032c1e68fc23da0edcf6fa5e3. Done probably by accident, since the code in question was moved to a match arm, along with newly introduced logic to detect bare CRs in raw strings.
2019-06-08Separate a `scan_raw_string` (similar `raw_byte` variant)Igor Matuszewski-77/+82
2019-06-08Clean up minor bitsIgor Matuszewski-1/+1
2019-06-06syntax: Remove duplicate span from `token::Ident`Vadim Petrochenkov-16/+6
2019-06-06syntax: Remove duplicate span from `token::Lifetime`Vadim Petrochenkov-9/+5
2019-06-06syntax: Add some helper methods to `Token`Vadim Petrochenkov-37/+24
2019-06-06syntax: Use `Token` in `StringReader` and `TokenTreesReader`Vadim Petrochenkov-23/+12
2019-06-06syntax: Use `Token` in `TokenTree::Token`Vadim Petrochenkov-22/+18
2019-06-06syntax: Rename `TokenAndSpan` into `Token`Vadim Petrochenkov-77/+62
2019-06-06syntax: Rename `Token` into `TokenKind`Vadim Petrochenkov-16/+16
2019-05-27Pre-intern "0", "1", ..., "9", and use where appropriate.Nicholas Nethercote-2/+2
2019-05-23syntax: Turn `token::Lit` into a structVadim Petrochenkov-54/+51
2019-05-21Move `edition` outside the hygiene lock and avoid accessing itJohn Kåre Alsaker-15/+16
2019-05-16Auto merge of #60763 - matklad:tt-parser, r=petrochenkovbors-25/+1
Move token tree related lexer state to a separate struct Just a types-based refactoring. We only used a bunch of fields when tokenizing into a token tree, so let's move them out of the base lexer
2019-05-13move span and token to tt readerAleksey Kladov-12/+0
2019-05-13Move token tree related lexer state to a separate structAleksey Kladov-13/+1
We only used a bunch of fields when tokenizing into a token tree, so let's move them out of the base lexer
2019-05-11Move literal parsing code into a separate fileVadim Petrochenkov-13/+6
Remove some dead code
2019-05-09Rollup merge of #60188 - estebank:recover-block, r=varkorMazdak Farrokhzad-1/+2
Identify when a stmt could have been parsed as an expr There are some expressions that can be parsed as a statement without a trailing semicolon depending on the context, which can lead to confusing errors due to the same looking code being accepted in some places and not others. Identify these cases and suggest enclosing in parenthesis making the parse non-ambiguous without changing the accepted grammar. Fix #54186, cc #54482, fix #59975, fix #47287.
2019-05-06review comments: fix typo and add commentsEsteban Küber-1/+1
2019-05-02introduce unescape moduleAleksey Kladov-443/+142
Currently, we deal with escape sequences twice: once when we lex a string, and a second time when we unescape literals. This PR aims to remove this duplication, by introducing a new `unescape` mode as a single source of truth for character escaping rules
2019-04-29Identify when a stmt could have been parsed as an exprEsteban Küber-1/+2
There are some expressions that can be parsed as a statement without a trailing semicolon depending on the context, which can lead to confusing errors due to the same looking code being accepted in some places and not others. Identify these cases and suggest enclosing in parenthesis making the parse non-ambiguous without changing the accepted grammar.
2019-04-23simplify and avoid allocationAleksey Kladov-2/+2
2019-04-23remove obsolete and incorrect commentAleksey Kladov-3/+1
2019-04-20Auto merge of #59700 - matklad:simplify, r=eddybbors-17/+5
Simplify doc comment lexing is_doc_comment function checks the first four chars, but this is redundant, `doc_comment` local var has the same info.
2019-04-17Rollup merge of #59128 - oli-obk:colorful_json, r=mark-i-m,eddybMazdak Farrokhzad-0/+1
Emit ansi color codes in the `rendered` field of json diagnostics cc @ljedrz Implemented for https://github.com/rust-lang/rust/pull/56595#issuecomment-447645115 (x.py clippy)
2019-04-05Rollup merge of #59687 - matklad:shebang, r=petrochenkovMazdak Farrokhzad-11/+5
cleanup shebang handling in the lexer
2019-04-04Simplify doc comment lexingAleksey Kladov-17/+5
is_doc_comment function checks the first four chars, but this is redundant, `doc_comment` local var has the same info.
2019-04-04cleanup shebang handling in the lexerAleksey Kladov-11/+5
2019-04-03make StringReader methods privateAleksey Kladov-2/+2
2019-04-03make StringReader fields privateAleksey Kladov-6/+6
2019-04-02Update more unit test to new APIOliver Scherer-0/+1
2019-03-20Tweak incorrect escaped char diagnosticEsteban Küber-19/+19
2019-03-16syntax: Introduce `Ident::can_be_raw`Vadim Petrochenkov-8/+4
2019-03-13Rollup merge of #58876 - estebank:numeric-lifetime, r=petrochenkovMazdak Farrokhzad-3/+15
Parse lifetimes that start with a number and give specific error Fix #58786.
2019-03-09review commentsEsteban Küber-4/+4
2019-03-08Parse lifetimes that start with a number and give specific errorEsteban Küber-6/+18
2019-03-07Fix with_emitter callersEsteban Küber-1/+1
2019-02-16help suggestion when trying to delimit string literals with directed unicode ↵Patrick McCarter-0/+22
quotes #58436
2019-02-13Rollup merge of #58273 - taiki-e:rename-dependency, r=matthewjasperMazdak Farrokhzad-3/+2
Rename rustc_errors dependency in rust 2018 crates I think this is a better solution than `use rustc_errors as errors` in `lib.rs` and `use crate::errors` in modules. Related: rust-lang/cargo#5653 cc #58099 r? @Centril
2019-02-13Cleanup importsTaiki Endo-2/+2
2019-02-13Rename rustc_errors dependency in rust 2018 cratesTaiki Endo-2/+1
2019-02-10rustc: doc commentsAlexander Regueiro-6/+6
2019-02-07Deduplicate mismatched delimiter errorsEsteban Küber-0/+11
Delay unmatched delimiter errors until after the parser has run to deduplicate them when parsing and attempt recovering intelligently.
2019-02-07libsyntax => 2018Taiki Endo-16/+19
2019-01-26remove `_with_applicability` from suggestion fnsAndy Russell-2/+2
2019-01-20Auto merge of #56884 - euclio:codeblock-diagnostics, r=QuietMisdreavusbors-13/+0
rustdoc: overhaul code block lexing errors Fixes #53919. This PR moves the reporting of code block lexing errors from rendering time to an early pass, so we can use the compiler's error reporting mechanisms. This dramatically improves the diagnostics in this situation: we now de-emphasize the lexing errors as a note under a warning that has a span and suggestion instead of just emitting errors at the top level. Additionally, this PR generalizes the markdown -> source span calculation function, which should allow other rustdoc warnings to use better spans in the future. Last, the PR makes sure that the code block is always emitted in the docs, even if it fails to highlight correctly. Of note: - The new pass unfortunately adds another pass over the docs to gather the doc blocks for syntax-checking. I wonder if this could be combined with the pass that looks for testable blocks? I'm not familiar with that code, so I don't know how feasible that is. - `pulldown_cmark` doesn't make it easy to find the spans of the code blocks, so the code that calculates the spans is a little nasty. It works for all the test cases I threw at it, but I wouldn't be surprised if an edge case would break it. Should have a thorough review. - This PR worsens the state of #56885, since those certain fatal lexing errors are now emitted before docs get generated at all.
2019-01-20Revert changeYuki Okushi-3/+2
generated by cgit-pink 1.4.1-3-g733a5 (git 2.37.1) at 2026-02-23 22:32:39 +0000