about summary refs log tree commit diff
path: root/src/libsyntax/parse/lexer
AgeCommit message (Collapse)AuthorLines
2019-05-21Move `edition` outside the hygiene lock and avoid accessing itJohn Kåre Alsaker-15/+16
2019-05-16Auto merge of #60763 - matklad:tt-parser, r=petrochenkovbors-37/+56
Move token tree related lexer state to a separate struct Just a types-based refactoring. We only used a bunch of fields when tokenizing into a token tree, so let's move them out of the base lexer
2019-05-13move raw span to tt readerAleksey Kladov-1/+2
See https://github.com/rust-lang/rust/pull/50838/files#r283296243 for explanation how jointness checking works with *next* pair
2019-05-13move span and token to tt readerAleksey Kladov-38/+34
2019-05-13Move token tree related lexer state to a separate structAleksey Kladov-44/+66
We only used a bunch of fields when tokenizing into a token tree, so let's move them out of the base lexer
2019-05-11Move literal parsing code into a separate fileVadim Petrochenkov-13/+6
Remove some dead code
2019-05-11Eliminate `comments::Literal`Vadim Petrochenkov-26/+3
2019-05-09Rollup merge of #60188 - estebank:recover-block, r=varkorMazdak Farrokhzad-1/+2
Identify when a stmt could have been parsed as an expr There are some expressions that can be parsed as a statement without a trailing semicolon depending on the context, which can lead to confusing errors due to the same looking code being accepted in some places and not others. Identify these cases and suggest enclosing in parenthesis making the parse non-ambiguous without changing the accepted grammar. Fix #54186, cc #54482, fix #59975, fix #47287.
2019-05-06review comments: fix typo and add commentsEsteban Küber-1/+1
2019-05-02introduce unescape moduleAleksey Kladov-443/+142
Currently, we deal with escape sequences twice: once when we lex a string, and a second time when we unescape literals. This PR aims to remove this duplication, by introducing a new `unescape` mode as a single source of truth for character escaping rules
2019-04-29Identify when a stmt could have been parsed as an exprEsteban Küber-1/+2
There are some expressions that can be parsed as a statement without a trailing semicolon depending on the context, which can lead to confusing errors due to the same looking code being accepted in some places and not others. Identify these cases and suggest enclosing in parenthesis making the parse non-ambiguous without changing the accepted grammar.
2019-04-23simplify and avoid allocationAleksey Kladov-2/+2
2019-04-23remove obsolete and incorrect commentAleksey Kladov-3/+1
2019-04-20Auto merge of #59700 - matklad:simplify, r=eddybbors-17/+5
Simplify doc comment lexing is_doc_comment function checks the first four chars, but this is redundant, `doc_comment` local var has the same info.
2019-04-17Rollup merge of #59128 - oli-obk:colorful_json, r=mark-i-m,eddybMazdak Farrokhzad-0/+1
Emit ansi color codes in the `rendered` field of json diagnostics cc @ljedrz Implemented for https://github.com/rust-lang/rust/pull/56595#issuecomment-447645115 (x.py clippy)
2019-04-05Rollup merge of #59687 - matklad:shebang, r=petrochenkovMazdak Farrokhzad-11/+5
cleanup shebang handling in the lexer
2019-04-04Simplify doc comment lexingAleksey Kladov-17/+5
is_doc_comment function checks the first four chars, but this is redundant, `doc_comment` local var has the same info.
2019-04-04cleanup shebang handling in the lexerAleksey Kladov-11/+5
2019-04-03make StringReader methods privateAleksey Kladov-2/+2
2019-04-03make StringReader fields privateAleksey Kladov-6/+6
2019-04-02Update more unit test to new APIOliver Scherer-0/+1
2019-03-20Tweak incorrect escaped char diagnosticEsteban Küber-19/+19
2019-03-16syntax: Introduce `Ident::can_be_raw`Vadim Petrochenkov-8/+4
2019-03-13Rollup merge of #58876 - estebank:numeric-lifetime, r=petrochenkovMazdak Farrokhzad-3/+15
Parse lifetimes that start with a number and give specific error Fix #58786.
2019-03-09review commentsEsteban Küber-4/+4
2019-03-08Parse lifetimes that start with a number and give specific errorEsteban Küber-6/+18
2019-03-07Fix with_emitter callersEsteban Küber-1/+1
2019-02-16help suggestion when trying to delimit string literals with directed unicode ↵Patrick McCarter-9/+44
quotes #58436
2019-02-13Rollup merge of #58273 - taiki-e:rename-dependency, r=matthewjasperMazdak Farrokhzad-4/+3
Rename rustc_errors dependency in rust 2018 crates I think this is a better solution than `use rustc_errors as errors` in `lib.rs` and `use crate::errors` in modules. Related: rust-lang/cargo#5653 cc #58099 r? @Centril
2019-02-13Cleanup importsTaiki Endo-2/+2
2019-02-13Rename rustc_errors dependency in rust 2018 cratesTaiki Endo-3/+2
2019-02-10rustc: doc commentsAlexander Regueiro-9/+9
2019-02-07Deduplicate mismatched delimiter errorsEsteban Küber-15/+26
Delay unmatched delimiter errors until after the parser has run to deduplicate them when parsing and attempt recovering intelligently.
2019-02-07libsyntax => 2018Taiki Endo-33/+41
2019-01-26remove `_with_applicability` from suggestion fnsAndy Russell-3/+3
2019-01-20Auto merge of #56884 - euclio:codeblock-diagnostics, r=QuietMisdreavusbors-13/+0
rustdoc: overhaul code block lexing errors Fixes #53919. This PR moves the reporting of code block lexing errors from rendering time to an early pass, so we can use the compiler's error reporting mechanisms. This dramatically improves the diagnostics in this situation: we now de-emphasize the lexing errors as a note under a warning that has a span and suggestion instead of just emitting errors at the top level. Additionally, this PR generalizes the markdown -> source span calculation function, which should allow other rustdoc warnings to use better spans in the future. Last, the PR makes sure that the code block is always emitted in the docs, even if it fails to highlight correctly. Of note: - The new pass unfortunately adds another pass over the docs to gather the doc blocks for syntax-checking. I wonder if this could be combined with the pass that looks for testable blocks? I'm not familiar with that code, so I don't know how feasible that is. - `pulldown_cmark` doesn't make it easy to find the spans of the code blocks, so the code that calculates the spans is a little nasty. It works for all the test cases I threw at it, but I wouldn't be surprised if an edge case would break it. Should have a thorough review. - This PR worsens the state of #56885, since those certain fatal lexing errors are now emitted before docs get generated at all.
2019-01-20Revert changeYuki Okushi-3/+2
2019-01-20[WIP] Improve error behaviorYuki Okushi-6/+8
2019-01-18Continue chekingYuki Okushi-1/+1
2019-01-16Cancel processYuki Okushi-1/+1
2019-01-14rustdoc: check code block syntax in early passAndy Russell-13/+0
2019-01-14Rollup merge of #57540 - estebank:eval-more, r=petrochenkovMazdak Farrokhzad-1/+1
Modify some parser diagnostics to continue evaluating beyond the parser Continue evaluating further errors after parser errors on: - trailing type argument attribute - lifetime in incorrect location - incorrect binary literal - missing `for` in `impl Trait for Foo` - type argument in `where` clause - incorrect float literal - incorrect `..` in pattern - associated types - incorrect discriminator value variant error and others. All of these were found by making `continue-parse-after-error` `true` by default to identify errors that would need few changes. There are now only a handful of errors that have any change with `continue-parse-after-error` enabled. These changes make it so `rust` _won't_ stop evaluation after finishing parsing, enabling type checking errors to be displayed on the existing code without having to fix the parse errors. Each commit has an individual diagnostic change with their corresponding tests. CC #48724.
2019-01-11Continue evaluating after parsing incorrect binary literalEsteban Küber-1/+1
2019-01-08Make `TokenStream` less recursive.Nicholas Nethercote-4/+4
`TokenStream` is currently recursive in *two* ways: - the `TokenTree` variant contains a `ThinTokenStream`, which can contain a `TokenStream`; - the `TokenStream` variant contains a `Vec<TokenStream>`. The latter is not necessary and causes significant complexity. This commit replaces it with the simpler `Vec<(TokenTree, IsJoint)>`. This reduces complexity significantly. In particular, `StreamCursor` is eliminated, and `Cursor` becomes much simpler, consisting now of just a `TokenStream` and an index. The commit also removes the `Extend` impl for `TokenStream`, because it is only used in tests. (The commit also removes those tests.) Overall, the commit reduces the number of lines of code by almost 200.
2018-12-31Do not use unicode character in diagnostic helpEsteban Küber-1/+1
2018-12-31Use structured suggestion for braceless unicode escape squenceEsteban Küber-6/+30
2018-12-25Remove licensesMark Rousskov-40/+0
2018-12-23Rollup merge of #56964 - nnethercote:TokenStream-IsJoint, r=petrochenkovMazdak Farrokhzad-3/+2
Remove `TokenStream::JointTree`. This is done by adding a new `IsJoint` field to `TokenStream::Tree`, which simplifies a lot of `match` statements. And likewise for `CursorKind`. The commit also adds a new method `TokenTree:stream()` which can replace a choice between `.into()` and `.joint()`.
2018-12-20Remove `TokenStream::JointTree`.Nicholas Nethercote-3/+2
This is done by adding a new `IsJoint` field to `TokenStream::Tree`, which simplifies a lot of `match` statements. And likewise for `CursorKind`. The commit also adds a new method `TokenTree:stream()` which can replace a choice between `.into()` and `.joint()`.
2018-12-19Rollup merge of #56910 - estebank:unclosed-eof, r=oli-obkPietro Albini-1/+9
Do not point at delim spans for complete correct blocks Fix #56834.
generated by cgit-pink 1.4.1-3-g733a5 (git 2.37.1) at 2026-02-25 06:00:42 +0000