about summary refs log tree commit diff
path: root/src/libsyntax/parse/lexer
AgeCommit message (Collapse)AuthorLines
2016-11-30Rename 'librustc_unicode' crate to 'libstd_unicode'.Corey Farwell-1/+1
Fixes #26554.
2016-11-21Fix fallout in `rustdoc` and tests.Jeffrey Seyfried-4/+5
2016-11-20Move `syntax::util::interner` -> `syntax::symbol`, cleanup.Jeffrey Seyfried-33/+34
2016-11-04Remove field `TtReader::next_tok`.Jeffrey Seyfried-2/+2
2016-11-03Move doc comment desugaring into the parser.Jeffrey Seyfried-1/+1
2016-11-03Avoid recontructing the `Parser` in `macro_parser.rs`.Jeffrey Seyfried-21/+0
2016-10-20Refactor parser lookahead buffer and increase its sizeVadim Petrochenkov-0/+6
2016-10-19Rollup merge of #37208 - jseyfried:fix_partially_consumed_tokens_in_macros, ↵Eduard-Mihai Burtescu-3/+24
r=nrc macros: fix partially consumed tokens in macro matchers Fixes #37175. This PR also avoids re-transcribing the tokens consumed by a matcher (and cloning the `TtReader` once per matcher), which improves expansion performance of the test case from #34630 by ~8%. r? @nrc
2016-10-18Fix some pretty printing testsVadim Petrochenkov-16/+30
2016-10-17Fix partially consumed tokens in macro matchers.Jeffrey Seyfried-3/+24
2016-10-05Clarify StringReader::bump.Nicholas Nethercote-13/+16
This commit renames the variables to make it clearer which char each one refers to. It also slightly reorders and rearranges some statements.
2016-10-05Rename StringReader::curr as ch.Nicholas Nethercote-99/+99
Likewise, rename StringReader::curr_is as ch_is. This is a [breaking-change] for libsyntax.
2016-10-05Rename StringReader::last_pos as pos.Nicholas Nethercote-100/+100
This is a [breaking-change] for libsyntax.
2016-10-05Rename StringReader::pos as next_pos.Nicholas Nethercote-18/+18
This is a [breaking-change] for libsyntax.
2016-10-03Simplify `start_bpos` calculation in scan_comment().Nicholas Nethercote-5/+1
The two branches of this `if` compute the same value. This commit gets rid of the first branch, which makes this calculation identical to the one in scan_block_comment().
2016-10-03Streamline StringReader::bump.Nicholas Nethercote-3/+1
First, assert! is redundant w.r.t. the unwrap() immediately afterwards. Second, `byte_offset_diff` is effectively computed as `current_byte_offset + ch.len_utf8() - current_byte_offset` (with `next` as an intermediate) which is silly and can be simplified.
2016-09-23reviewer comments and rebasingNick Cameron-23/+0
2016-09-22Adds a `ProcMacro` form of syntax extensionNick Cameron-6/+52
This commit adds syntax extension forms matching the types for procedural macros 2.0 (RFC #1566), these still require the usual syntax extension boiler plate, but this is a first step towards proper implementation and should be useful for macros 1.1 stuff too. Supports both attribute-like and function-like macros.
2016-08-07Fix old call in lexer testsJonathan Turner-3/+1
2016-07-14Remove BasicEmitterJonathan Turner-1/+1
2016-07-03prefer `if let` to match with `None => {}` arm in some placesZack M. Davis-8/+5
This is a spiritual succesor to #34268/8531d581, in which we replaced a number of matches of None to the unit value with `if let` conditionals where it was judged that this made for clearer/simpler code (as would be recommended by Manishearth/rust-clippy's `single_match` lint). The same rationale applies to matches of None to the empty block.
2016-06-23make old school mode a bit more configurableJonathan Turner-1/+4
2016-06-23Address more travis errorsJonathan Turner-1/+1
2016-06-23Move errors from libsyntax to its own crateJonathan Turner-25/+27
2016-06-16Add an abs_path member to FileMap, use it when writing debug info.Ted Mielczarek-2/+2
When items are inlined from extern crates, the filename in the debug info is taken from the FileMap that's serialized in the rlib metadata. Currently this is just FileMap.name, which is whatever path is passed to rustc. Since libcore and libstd are built by invoking rustc with relative paths, they wind up with relative paths in the rlib, and when linked into a binary the debug info uses relative paths for the names, but since the compilation directory for the final binary, tools trying to read source filenames will wind up with bad paths. We noticed this in Firefox with source filenames from libcore/libstd having bad paths. This change stores an absolute path in FileMap.abs_path, and uses that if available for writing debug info. This is not going to magically make debuggers able to find the source, but it will at least provide sensible paths.
2016-05-05Auto merge of #33128 - xen0n:more-confusing-unicode-chars, r=nagisabors-6/+53
Add more aliases for Unicode confusable chars Building upon #29837, this PR: * added aliases for space characters, * distinguished square brackets from parens, and * added common CJK punctuation characters as aliases. This will especially help CJK users who may have forgotten to switch off IME when coding.
2016-05-02lexer: do not display char confusingly in error messageGeorg Brandl-5/+4
Current code leads to messages like "... use a \xHH escape: \u{e4}" which is confusing. The printed span already points to the offending character, which should be enough to identify the non-ASCII problem. Fixes: #29088
2016-04-27Make some fatal lexer errors recoverablemitaa-66/+103
2016-04-24syntax: Check paths in visibilities for type parametersVadim Petrochenkov-10/+4
syntax: Merge PathParsingMode::NoTypesAllowed and PathParsingMode::ImportPrefix syntax: Rename PathParsingMode and its variants to better express their purpose syntax: Remove obsolete error message about 'self lifetime syntax: Remove ALLOW_MODULE_PATHS workaround syntax/resolve: Adjust some error messages resolve: Compare unhygienic (not renamed) names with keywords::Invalid, invalid identifiers may appear to be valid after renaming
2016-04-24syntax: Get rid of token::IdentStyleVadim Petrochenkov-25/+10
2016-04-21add more confusable CJK square bracket aliasesWang Xuerui-0/+12
2016-04-21correct aliases for square bracketsWang Xuerui-6/+8
2016-04-21add confusable space charactersWang Xuerui-0/+17
2016-04-21add more characters easily inputtable with CJK IMEsWang Xuerui-0/+16
2016-03-07Auto merge of #29734 - Ryman:whitespace_consistency, r=Aatchbors-12/+12
libsyntax: be more accepting of whitespace in lexer Fixes #29590. Perhaps this may need more thorough testing? r? @Aatch
2016-01-16libsyntax: accept only whitespace with the PATTERN_WHITE_SPACE propertyKevin Butler-9/+12
This aligns with unicode recommendations and should be stable for all future unicode releases. See http://unicode.org/reports/tr31/#R3. This renames `libsyntax::lexer::is_whitespace` to `is_pattern_whitespace` so potentially breaks users of libsyntax.
2016-01-14libsyntax: use char::is_whitespace instead of custom implementationsKevin Butler-4/+1
Fixes #29590.
2016-01-14Display better snippet for invalid char literalGreg Chapple-10/+15
Given this code: fn main() { let _ = 'abcd'; } The compiler would give a message like: error: character literal may only contain one codepoint: '; let _ = 'abcd'; ^~ With this change, the message now displays: error: character literal may only contain one codepoint: 'abcd' let _ = 'abcd' ^~~~~~ Fixes #30033
2016-01-12re-instate comment that was mysteriously disappearedTshepang Lekhonkhobe-0/+1
2016-01-04address review commentTshepang Lekhonkhobe-10/+2
2016-01-03fix "make tidy" failureTshepang Lekhonkhobe-1/+2
2016-01-03run rustfmt on syntax::parse::lexerTshepang Lekhonkhobe-513/+764
2015-12-30use structured errorsNick Cameron-23/+47
2015-12-17test errorsNick Cameron-27/+51
2015-12-17move error handling from libsyntax/diagnostics.rs to libsyntax/errors/*Nick Cameron-11/+10
Also split out emitters into their own module.
2015-11-18Add some unicode aliases for ".Huon Wilson-0/+17
2015-11-17Detect confusing unicode characters and show the alternativeRavi Shankar-1/+189
2015-11-05Improve error message for char literalsSteve Klabnik-5/+6
If you try to put something that's bigger than a char into a char literal, you get an error: fn main() { let c = 'ஶ்ரீ'; } error: unterminated character constant: This is a very compiler-centric message. Yes, it's technically 'unterminated', but that's not what you, the user did wrong. Instead, this commit changes it to error: character literal may only contain one codepoint As this actually tells you what went wrong. Fixes #28851
2015-10-27Start pushing panics outward in lexer.Eli Friedman-34/+38
2015-10-10Prevent `/**/` from being parsed as a doc commentBarosl Lee-2/+3
Previously, `/**/` was incorrectly regarded as a doc comment because it starts with `/**` and ends with `*/`. However, this caused an ICE because some code assumed that the length of a doc comment is at least 5. This commit adds an additional check to `is_block_doc_comment` that tests the length of the input. Fixes #28844.
generated by cgit-pink 1.4.1-3-g733a5 (git 2.37.1) at 2026-02-23 17:06:29 +0000