about summary refs log tree commit diff
path: root/src/librustc_lexer
AgeCommit message (Collapse)AuthorLines
2020-08-30mv compiler to compiler/mark-1711/+0
2020-08-24Introduce expect snapshot testing library into rustcAleksey Kladov-0/+49
Snapshot testing is a technique for writing maintainable unit tests. Unlike usual `assert_eq!` tests, snapshot tests allow to *automatically* upgrade expected values on test failure. In a sense, snapshot tests are inline-version of our beloved UI-tests. Example: ![expect](https://user-images.githubusercontent.com/1711539/90888810-3bcc8180-e3b7-11ea-9626-d06e89e1a0bb.gif) A particular library we use, `expect_test` provides an `expect!` macro, which creates a sort of self-updating string literal (by using `file!` macro). Self-update is triggered by setting `UPDATE_EXPECT` environmental variable (this info is printed during the test failure). This library was extracted from rust-analyzer, where we use it for most of our tests. There are some other, more popular snapshot testing libraries: * https://github.com/mitsuhiko/insta * https://github.com/aaronabramov/k9 The main differences of `expect` are: * first-class snapshot objects (so, tests can be written as functions, rather than as macros) * focus on inline-snapshots (but file snapshots are also supported) * restricted feature set (only `assert_eq` and `assert_debug_eq`) * no extra runtime (ie, no `cargo insta`) See https://github.com/rust-analyzer/rust-analyzer/pull/5101 for a an extended comparison. It is unclear if this testing style will stick with rustc in the long run. At the moment, rustc is mainly tested via integrated UI tests. But in the library-ified world, unit-tests will become somewhat more important (that's why use use `rustc_lexer` library-ified library as an example in this PR). Given that the cost of removal shouldn't be too high, it probably makes sense to just see if this flies!
2020-08-21Minor, remove double nesting of a test moduleAleksey Kladov-136/+121
2020-08-21Auto merge of #75642 - matklad:lexer-comments, r=petrochenkovbors-5/+30
Move doc comment parsing to rustc_lexer Plain comments are trivia, while doc comments are not, so it feels like this belongs to the rustc_lexer. The specific reason to do this is the desire to use rustc_lexer in rustdoc for syntax highlighting, without duplicating "is this a doc comment?" logic there. r? @ghost
2020-08-20Rename rustc_lexer::TokenKind::Not to BangAleksey Kladov-2/+2
All other tokens are named by the punctuation they use, rather than by semantics operation they stand for. `!` is the only exception to the rule, let's fix it.
2020-08-19Move doc comment parsing to rustc_lexerAleksey Kladov-5/+30
Plain comments are trivial, while doc comments are not, so it feels like this belongs to the rustc_lexer. The specific reason to do this is the desire to use rustc_lexer in rustdoc for syntax highlighting, without duplicating "is this a doc comment?" logic there.
2020-08-11Introduce `rustc_lexer::is_ident` and use it in couple of placesVadim Petrochenkov-0/+10
2020-07-06Rollup merge of #73856 - pierwill:pierwill-lexer-doc, r=jonas-schievinkManish Goregaokar-4/+5
Edit librustc_lexer top-level docs Minor edit, and adds link to librustc_parse::lexer.
2020-07-06Edit librustc_lexer top-level docspierwill-4/+5
Add link to librustc_parse::lexer
2020-07-01Rollup merge of #73848 - pierwill:pierwill-lexer-block-doc, r=jonas-schievinkManish Goregaokar-2/+3
Fix markdown rendering in librustc_lexer docs Use back-ticks instead of quotation marks in docs for the block comment variant of TokenKind. ## [Before](https://doc.rust-lang.org/nightly/nightly-rustc/rustc_lexer/enum.TokenKind.html#variant.BlockComment) and after <img width="1103" alt="Screen Shot 2020-06-28 at 1 22 30 PM" src="https://user-images.githubusercontent.com/19642016/85957562-446a8380-b943-11ea-913a-442cf7744083.png"> <img width="1015" alt="Screen Shot 2020-06-28 at 1 28 29 PM" src="https://user-images.githubusercontent.com/19642016/85957566-4af8fb00-b943-11ea-8fef-a09c1d586772.png"> ## Question For visual consistency, should we use back-ticks throughout the docs for these enum variants?
2020-06-28Edit cursor.prev() method docs in lexerpierwill-2/+2
Fix missing punctuation
2020-06-28Fix markdown rendering in librustc_lexer docspierwill-2/+3
Use back-ticks instead of quotation marks in docs for the block comment variant of TokenKind.
2020-06-26rustc_lexer: Simplify shebang parsing once moreVadim Petrochenkov-15/+12
2020-06-12Rollup merge of #72906 - lzutao:migrate-numeric-assoc-consts, r=dtolnayDylan DPC-1/+1
Migrate to numeric associated consts The deprecation PR is #72885 cc #68490 cc rust-lang/rfcs#2700
2020-06-10Migrate to numeric associated constsLzu Tao-1/+1
2020-06-09Fix more clippy warningsMatthias Krüger-2/+2
Fixes more of: clippy::unused_unit clippy::op_ref clippy::useless_format clippy::needless_return clippy::useless_conversion clippy::bind_instead_of_map clippy::into_iter_on_ref clippy::redundant_clone clippy::nonminimal_bool clippy::redundant_closure clippy::option_as_ref_deref clippy::len_zero clippy::iter_cloned_collect clippy::filter_next
2020-06-01Simplify raw string error reporting.Julian Wollersberger-189/+61
This makes `UnvalidatedRawStr` and `ValidatedRawStr` unnecessary and removes 70 lines.
2020-05-29rustc_lexer: Optimize shebang detection slightlyVadim Petrochenkov-19/+18
2020-05-25Fix bug in shebang handlingRussell Cohen-7/+76
Shebang handling was too agressive in stripping out the first line in cases where it is actually _not_ a shebang, but instead, valid rust (#70528). This is a second attempt at resolving this issue (the first attempt was flawed, for, among other reasons, causing an ICE in certain cases (#71372, #71471). The behavior is now codified by a number of UI tests, but simply: For the first line to be a shebang, the following must all be true: 1. The line must start with `#!` 2. The line must contain a non whitespace character after `#!` 3. The next character in the file, ignoring comments & whitespace must not be `[` I believe this is a strict superset of what we used to allow, so perhaps a crater run is unnecessary, but probably not a terrible idea.
2020-05-13Replace some usages of the old `unescape_` functions in AST, clippy and tests.Julian Wollersberger-52/+6
2020-05-13Unified `unescape_{char,byte,str,byte_str,raw_str,raw_byte_str}` methods ↵Julian Wollersberger-4/+42
into one method `unescape_literal` with a mode argument.
2020-05-09Small doc improvements.Julian Wollersberger-2/+11
The phrasing is from the commit description of 395ee0b79f23b90593b01dd0a78451b8c93b0aa6 by @Matklad.
2020-04-28Revert "Rollup merge of #71372 - ayushmishra2005:shebang_stripping, r=estebank"Eduard-Mihai Burtescu-24/+1
This reverts commit 46a8dcef5c9e4de0d412c6ac3c4765cb4aef4f7f, reversing changes made to f28e3873c55eb4bdcfc496e1f300b97aeb0d189c.
2020-04-21Minor refactoringAyush Kumar Mishra-1/+1
2020-04-21Fix formatting issueAyush Kumar Mishra-1/+0
2020-04-21Refactoring and added test-cases #70528Ayush Kumar Mishra-1/+25
2020-04-21Fix #! (shebang) stripping account space issue #70528Ayush Kumar Mishra-1/+1
2020-04-02Handle unterminated raw strings with no #s properlyRussell Cohen-1/+35
The modified code to handle parsing raw strings didn't properly account for the case where there was no "#" on either end and erroneously reported this strings as complete. This lead to a panic trying to read off the end of the file.
2020-03-31Fix tests to handle debug_assertRussell Cohen-0/+2
2020-03-30Clean up redudant conditions and match exprsRussell Cohen-1/+1
2020-03-29Cleanup error messages, improve docstringsRussell Cohen-17/+32
2020-03-29Improve error messages for raw strings (#60762)Russell Cohen-23/+229
This diff improves error messages around raw strings in a few ways: - Catch extra trailing `#` in the parser. This can't be handled in the lexer because we could be in a macro that actually expects another # (see test) - Refactor & unify error handling in the lexer between ByteStrings and RawByteStrings - Detect potentially intended terminators (longest sequence of "#*" is suggested)
2020-03-20remove redundant returns (clippy::needless_return)Matthias Krüger-3/+3
2020-03-06fix various typosMatthias Krüger-2/+2
2020-02-13Spelling error "represening" to "representing"Drew Ripberger-1/+1
2019-12-22Format the worldMark Rousskov-18/+9
2019-11-10move syntax::parse -> librustc_parseMazdak Farrokhzad-1/+1
also move MACRO_ARGUMENTS -> librustc_parse
2019-11-04librustc_lexer: Make nth_char method privateIgor Aleksanov-1/+1
2019-11-04librustc_lexer: Reorder imports in lib.rsIgor Aleksanov-3/+2
2019-11-04librustc_lexer: Simplify "lifetime_or_char" methodIgor Aleksanov-30/+37
2019-11-03librustc_lexer: Simplify "raw_double_quoted_string" methodIgor Aleksanov-25/+34
2019-11-03librustc_lexer: Simplify "double_quoted_string" methodIgor Aleksanov-6/+6
2019-11-03librustc_lexer: Simplify "single_quoted_string" methodIgor Aleksanov-11/+15
2019-11-03librustc_lexer: Make "eat_float_exponent" return bool instead of resultIgor Aleksanov-4/+6
2019-11-03librustc_lexer: Introduce "eat_while" and "eat_identifier" methodsIgor Aleksanov-22/+28
2019-11-03librustc_lexer: Add methods "first" and "second" to the "Cursor"Igor Aleksanov-20/+30
2019-10-27librustc_lexer: Enhance documentationIgor Aleksanov-24/+225
Apply review suggestions Apply review suggestions
2019-09-05fill metadata in rustc_lexer's Cargo.tomlAleksey Kladov-0/+6
We publish this to crates.io, so having non-empty meta is useful
2019-09-04flatten rustc_lexer::character_properties moduleAleksey Kladov-65/+70
On the call site, `rustc_lexer::is_whitespace` reads much better than `character_properties::is_whitespace`.
2019-09-04remove XID and Pattern_White_Space unicode tables from libcoreAleksey Kladov-37/+34
They are only used by rustc_lexer, and are not needed elsewhere. So we move the relevant definitions into rustc_lexer (while the actual unicode data comes from the unicode-xid crate) and make the rest of the compiler use it.
generated by cgit-pink 1.4.1-3-g733a5 (git 2.37.1) at 2026-03-01 18:03:59 +0000