summary refs log tree commit diff
path: root/src/librustc_lexer
AgeCommit message (Collapse)AuthorLines
2020-07-10rustc_lexer: Simplify shebang parsing once moreVadim Petrochenkov-15/+12
2020-05-29rustc_lexer: Optimize shebang detection slightlyVadim Petrochenkov-19/+18
2020-05-25Fix bug in shebang handlingRussell Cohen-7/+76
Shebang handling was too agressive in stripping out the first line in cases where it is actually _not_ a shebang, but instead, valid rust (#70528). This is a second attempt at resolving this issue (the first attempt was flawed, for, among other reasons, causing an ICE in certain cases (#71372, #71471). The behavior is now codified by a number of UI tests, but simply: For the first line to be a shebang, the following must all be true: 1. The line must start with `#!` 2. The line must contain a non whitespace character after `#!` 3. The next character in the file, ignoring comments & whitespace must not be `[` I believe this is a strict superset of what we used to allow, so perhaps a crater run is unnecessary, but probably not a terrible idea.
2020-05-13Replace some usages of the old `unescape_` functions in AST, clippy and tests.Julian Wollersberger-52/+6
2020-05-13Unified `unescape_{char,byte,str,byte_str,raw_str,raw_byte_str}` methods ↵Julian Wollersberger-4/+42
into one method `unescape_literal` with a mode argument.
2020-05-09Small doc improvements.Julian Wollersberger-2/+11
The phrasing is from the commit description of 395ee0b79f23b90593b01dd0a78451b8c93b0aa6 by @Matklad.
2020-04-28Revert "Rollup merge of #71372 - ayushmishra2005:shebang_stripping, r=estebank"Eduard-Mihai Burtescu-24/+1
This reverts commit 46a8dcef5c9e4de0d412c6ac3c4765cb4aef4f7f, reversing changes made to f28e3873c55eb4bdcfc496e1f300b97aeb0d189c.
2020-04-21Minor refactoringAyush Kumar Mishra-1/+1
2020-04-21Fix formatting issueAyush Kumar Mishra-1/+0
2020-04-21Refactoring and added test-cases #70528Ayush Kumar Mishra-1/+25
2020-04-21Fix #! (shebang) stripping account space issue #70528Ayush Kumar Mishra-1/+1
2020-04-02Handle unterminated raw strings with no #s properlyRussell Cohen-1/+35
The modified code to handle parsing raw strings didn't properly account for the case where there was no "#" on either end and erroneously reported this strings as complete. This lead to a panic trying to read off the end of the file.
2020-03-31Fix tests to handle debug_assertRussell Cohen-0/+2
2020-03-30Clean up redudant conditions and match exprsRussell Cohen-1/+1
2020-03-29Cleanup error messages, improve docstringsRussell Cohen-17/+32
2020-03-29Improve error messages for raw strings (#60762)Russell Cohen-23/+229
This diff improves error messages around raw strings in a few ways: - Catch extra trailing `#` in the parser. This can't be handled in the lexer because we could be in a macro that actually expects another # (see test) - Refactor & unify error handling in the lexer between ByteStrings and RawByteStrings - Detect potentially intended terminators (longest sequence of "#*" is suggested)
2020-03-20remove redundant returns (clippy::needless_return)Matthias Krüger-3/+3
2020-03-06fix various typosMatthias Krüger-2/+2
2020-02-13Spelling error "represening" to "representing"Drew Ripberger-1/+1
2019-12-22Format the worldMark Rousskov-18/+9
2019-11-10move syntax::parse -> librustc_parseMazdak Farrokhzad-1/+1
also move MACRO_ARGUMENTS -> librustc_parse
2019-11-04librustc_lexer: Make nth_char method privateIgor Aleksanov-1/+1
2019-11-04librustc_lexer: Reorder imports in lib.rsIgor Aleksanov-3/+2
2019-11-04librustc_lexer: Simplify "lifetime_or_char" methodIgor Aleksanov-30/+37
2019-11-03librustc_lexer: Simplify "raw_double_quoted_string" methodIgor Aleksanov-25/+34
2019-11-03librustc_lexer: Simplify "double_quoted_string" methodIgor Aleksanov-6/+6
2019-11-03librustc_lexer: Simplify "single_quoted_string" methodIgor Aleksanov-11/+15
2019-11-03librustc_lexer: Make "eat_float_exponent" return bool instead of resultIgor Aleksanov-4/+6
2019-11-03librustc_lexer: Introduce "eat_while" and "eat_identifier" methodsIgor Aleksanov-22/+28
2019-11-03librustc_lexer: Add methods "first" and "second" to the "Cursor"Igor Aleksanov-20/+30
2019-10-27librustc_lexer: Enhance documentationIgor Aleksanov-24/+225
Apply review suggestions Apply review suggestions
2019-09-05fill metadata in rustc_lexer's Cargo.tomlAleksey Kladov-0/+6
We publish this to crates.io, so having non-empty meta is useful
2019-09-04flatten rustc_lexer::character_properties moduleAleksey Kladov-65/+70
On the call site, `rustc_lexer::is_whitespace` reads much better than `character_properties::is_whitespace`.
2019-09-04remove XID and Pattern_White_Space unicode tables from libcoreAleksey Kladov-37/+34
They are only used by rustc_lexer, and are not needed elsewhere. So we move the relevant definitions into rustc_lexer (while the actual unicode data comes from the unicode-xid crate) and make the rest of the compiler use it.
2019-08-19remove composite tokens support from the lexerAleksey Kladov-162/+14
2019-08-14remove special handling of \r\n from the lexerAleksey Kladov-38/+11
2019-08-02librustc_lexer: Unconfigure tests during normal buildVadim Petrochenkov-280/+279
2019-07-27Add commenttopecongiro-0/+2
2019-07-27Add lib section to rustc_lexer's Cargo.tomltopecongiro-0/+4
2019-07-23Rollup merge of #62869 - matklad:feature-gate, r=Mark-SimulacrumMark Rousskov-2/+1
add rustc_private as a proper language feature gate At the moment, `rustc_private` as a (library) feature exists by accident: `char::is_xid_start`, `char::is_xid_continue` methods in libcore define it. cc https://rust-lang.zulipchat.com/#narrow/stream/131828-t-compiler/topic/How.20to.20declare.20new.20langauge.20feature.3F I don't know if this is at all reasonable, but at least tests seem to pass locally. That probably means that we can remove/rename to something more resonable the feature in libcore in the next release?
2019-07-23Update src/librustc_lexer/src/lib.rsAleksey Kladov-1/+1
Co-Authored-By: Ralf Jung <post@ralfj.de>
2019-07-22add rustc_private as a proper language feature gateAleksey Kladov-1/+0
At the moment, `rustc_private` as a (library) feature exists by accident: `char::is_xid_start`, `char::is_xid_continue` methods in libcore define it.
2019-07-21move unescape module to rustc_lexerAleksey Kladov-0/+603
2019-07-20Introduce rustc_lexerAleksey Kladov-0/+776
The idea here is to make a reusable library out of the existing rust-lexer, by separating out pure lexing and rustc-specific concerns, like spans, error reporting an interning. So, rustc_lexer operates directly on `&str`, produces simple tokens which are a pair of type-tag and a bit of original text, and does not report errors, instead storing them as flags on the token.