about summary refs log tree commit diff
path: root/src/libsyntax/parse/lexer/mod.rs
AgeCommit message (Collapse)AuthorLines
2014-10-28Use an enum rather than a bool in token::IdentBrendan Zabarauskas-25/+36
2014-10-28Convert some token functions into methodsBrendan Zabarauskas-5/+4
2014-10-28Use PascalCase for token variantsBrendan Zabarauskas-94/+94
2014-10-19Remove a large amount of deprecated functionalityAlex Crichton-3/+3
Spring cleaning is here! In the Fall! This commit removes quite a large amount of deprecated functionality from the standard libraries. I tried to ensure that only old deprecated functionality was removed. This is removing lots and lots of deprecated features, so this is a breaking change. Please consult the deprecation messages of the deleted code to see how to migrate code forward if it still needs migration. [breaking-change]
2014-09-30librustc: Forbid `..` in range patterns.Patrick Walton-1/+1
This breaks code that looks like: match foo { 1..3 => { ... } } Instead, write: match foo { 1...3 => { ... } } Closes #17295. [breaking-change]
2014-09-18Fix fallout in tests from removing the use of Gc in ExpnInfo.Eduard Burtescu-3/+3
2014-09-03Fix spelling errors and capitalization.Joseph Crail-1/+1
2014-08-14librustc: Implement simple `where` clauses.Patrick Walton-12/+12
These `where` clauses are accepted everywhere generics are currently accepted and desugar during type collection to the type parameter bounds we have today. A new keyword, `where`, has been added. Therefore, this is a breaking change. Change uses of `where` to other identifiers. [breaking-change]
2014-08-13quote_expr macro: embed Ident using special encoding that preserves hygiene.Felix S. Klock II-0/+105
This adds support to `quote_expr!` and friends for round-trip hygienic preservation of Ident. Here are the pieces of the puzzle: * adding a method for encoding Ident for re-reading into token tree. * Support for reading such encoded Idents in the lexer. Note that one must peek ahead for MOD_SEP after scan_embedded_hygienic_ident. * To ensure that encoded Idents are only read when we are in the midst of expanding a `quote_expr` or similar, added a `read_embedded_ident` flag on `StringReader`. * pprust support for encoding Ident's as (uint,uint) pairs (for hygiene).
2014-08-06Use byte literals in libsyntaxnham-2/+2
2014-07-11Add scaffolding for assigning alpha-numeric codes to rustc diagnosticsJakub Wieczorek-1/+1
2014-07-09Fix all the test falloutCorey Richardson-7/+17
2014-07-09ast: make Name its own typeCorey Richardson-25/+25
2014-07-09lexer: lex WS/COMMENT/SHEBANG rather than skippingCorey Richardson-53/+81
Now, the lexer will categorize every byte in its input according to the grammar. The parser skips over these while parsing, thus avoiding their presence in the input to syntax extensions.
2014-07-09syntax: don't parse numeric literals in the lexerCorey Richardson-180/+162
This removes a bunch of token types. Tokens now store the original, unaltered numeric literal (that is still checked for correctness), which is parsed into an actual number later, as needed, when creating the AST. This can change how syntax extensions work, but otherwise poses no visible changes. [breaking-change]
2014-07-09syntax: don't process string/char/byte/binary litsCorey Richardson-47/+53
This shuffles things around a bit so that LIT_CHAR and co store an Ident which is the original, unaltered literal in the source. When creating the AST, unescape and postprocess them. This changes how syntax extensions can work, slightly, but otherwise poses no visible changes. To get a useful value out of one of these tokens, call `parse::{char_lit, byte_lit, bin_lit, str_lit}` [breaking-change]
2014-07-09lexer: add ident_from and ident_from_to methodsCorey Richardson-0/+14
2014-07-09lexer: shuffle around some functionsCorey Richardson-99/+100
2014-07-09syntax: use a better Show impl for IdentCorey Richardson-1/+1
Rather than just dumping the id in the interner, which is useless, actually print the interned string. Adjust the lexer logging to use Show instead of Poly.
2014-07-09syntax: doc comments all the thingsCorey Richardson-6/+6
2014-07-08Change DST syntax: type -> Sized?Nick Cameron-0/+1
closes #13367 [breaking-change] Use `Sized?` to indicate a dynamically sized type parameter or trait (used to be `type`). E.g., ``` trait Tr for Sized? {} fn foo<Sized? X: Share>(x: X) {} ```
2014-07-01rustc: Remove `&str` indexing from the language.Brian Anderson-2/+2
Being able to index into the bytes of a string encourages poor UTF-8 hygiene. To get a view of `&[u8]` from either a `String` or `&str` slice, use the `as_bytes()` method. Closes #12710. [breaking-change]
2014-06-24Remove the quad_precision_float feature gateAlex Crichton-8/+1
The f128 type has very little support in the compiler and the feature is basically unusable today. Supporting half-baked features in the compiler can be detrimental to the long-term development of the compiler, and hence this feature is being removed.
2014-06-18Handle CRLF properly in the lexerKevin Ballard-23/+118
The lexer already ignores CRLF in between tokens, but it doesn't properly handle carriage returns inside strings and doc comments. Teach it to treat CRLF as LF inside these tokens, and to disallow carriage returns that are not followed by linefeeds. This includes handling an escaped CRLF inside a regular string token the same way it handles an escaped LF. This is technically a breaking change, as bare carriage returns are no longer allowed, and CRLF sequences are now treated as LF inside strings and doc comments, but it's very unlikely to actually affect any real-world code. This change is necessary to have Rust code compile on Windows the same way it does on Unix. The mozilla/rust repository explicitly sets eol=lf for Rust source files, but other Rust repositories don't. Notably, rust-http cannot be compiled on Windows without converting the CRLF line endings back to LF. [breaking-change]
2014-06-18Don't require mutable StringReader to emit lexer errorsKevin Ballard-42/+52
Teach StringReader how to emit errors for arbitrary spans, so we don't need to modify peek_span. This allows for emitting errors without having a &mut borrow of the StringReader.
2014-06-18Fix spans for doc commentsKevin Ballard-3/+3
2014-06-18Fix some violations of stronger guarantees for mutable borrows.Simon Sapin-9/+17
See 159e27aebb940926ccf1bad0b2b12087d36ad903
2014-06-17Refactor backslash-escape parsing to share similar code.Simon Sapin-151/+69
Move into a new syntax::parse::lexer::StringReader method the code that was almost duplicated for parsing backslash-escapes in byte, byte string, char, and string literals.
2014-06-17Add br##"xx"## raw byte string literals.Simon Sapin-3/+53
2014-06-17Add a b"xx" byte string literal of type &'static [u8].Simon Sapin-48/+111
2014-06-17Add a b'x' byte literal of type u8.Simon Sapin-3/+65
2014-06-13Fix all violations of stronger guarantees for mutable borrowsCameron Zwarich-28/+54
Fix all violations in the Rust source tree of the stronger guarantee of a unique access path for mutable borrows as described in #12624.
2014-06-13librustc: Fix the issue with labels shadowing variable names by makingPatrick Walton-12/+27
the leading quote part of the identifier for the purposes of hygiene. This adopts @jbclements' solution to #14539. I'm not sure if this is a breaking change or not. Closes #12512. [breaking-change]
2014-06-04syntax: methodify the lexerCorey Richardson-0/+1153