about summary refs log tree commit diff
path: root/src/libsyntax/parse/lexer
AgeCommit message (Collapse)AuthorLines
2016-01-14Display better snippet for invalid char literalGreg Chapple-10/+15
Given this code: fn main() { let _ = 'abcd'; } The compiler would give a message like: error: character literal may only contain one codepoint: '; let _ = 'abcd'; ^~ With this change, the message now displays: error: character literal may only contain one codepoint: 'abcd' let _ = 'abcd' ^~~~~~ Fixes #30033
2016-01-12re-instate comment that was mysteriously disappearedTshepang Lekhonkhobe-0/+1
2016-01-04address review commentTshepang Lekhonkhobe-10/+2
2016-01-03fix "make tidy" failureTshepang Lekhonkhobe-1/+2
2016-01-03run rustfmt on syntax::parse::lexerTshepang Lekhonkhobe-513/+764
2015-12-30use structured errorsNick Cameron-23/+47
2015-12-17test errorsNick Cameron-27/+51
2015-12-17move error handling from libsyntax/diagnostics.rs to libsyntax/errors/*Nick Cameron-11/+10
Also split out emitters into their own module.
2015-11-18Add some unicode aliases for ".Huon Wilson-0/+17
2015-11-17Detect confusing unicode characters and show the alternativeRavi Shankar-1/+189
2015-11-05Improve error message for char literalsSteve Klabnik-5/+6
If you try to put something that's bigger than a char into a char literal, you get an error: fn main() { let c = 'ஶ்ரீ'; } error: unterminated character constant: This is a very compiler-centric message. Yes, it's technically 'unterminated', but that's not what you, the user did wrong. Instead, this commit changes it to error: character literal may only contain one codepoint As this actually tells you what went wrong. Fixes #28851
2015-10-27Start pushing panics outward in lexer.Eli Friedman-34/+38
2015-10-10Prevent `/**/` from being parsed as a doc commentBarosl Lee-2/+3
Previously, `/**/` was incorrectly regarded as a doc comment because it starts with `/**` and ends with `*/`. However, this caused an ICE because some code assumed that the length of a doc comment is at least 5. This commit adds an additional check to `is_block_doc_comment` that tests the length of the input. Fixes #28844.
2015-10-01Stop re-exporting AttrStyle's variants and rename them.Ms2ger-2/+2
2015-09-03std: Account for CRLF in {str, BufRead}::linesAlex Crichton-1/+1
This commit is an implementation of [RFC 1212][rfc] which tweaks the behavior of the `str::lines` and `BufRead::lines` iterators. Both iterators now account for `\r\n` sequences in addition to `\n`, allowing for less surprising behavior across platforms (especially in the `BufRead` case). Splitting *only* on the `\n` character can still be achieved with `split('\n')` in both cases. The `str::lines_any` function is also now deprecated as `str::lines` is a drop-in replacement for it. [rfc]: https://github.com/rust-lang/rfcs/blob/master/text/1212-line-endings.md Closes #28032
2015-09-03Use consistent terminology for byte string literalsVadim Petrochenkov-4/+4
Avoid confusion with binary integer literals and binary operator expressions in libsyntax
2015-07-29Replace illegal with invalid in most diagnosticsSimonas Kazlauskas-8/+8
2015-07-21Use a span from the correct file for the inner span of a moduleNick Cameron-0/+1
This basically only affects modules which are empty (or only contain comments). Closes #26755
2015-07-13Auto merge of #26947 - nagisa:unicode-escape-error, r=nrcbors-17/+27
Inspired by the now-mysteriously-closed https://github.com/rust-lang/rust/pull/26782. This PR introduces better error messages when unicode escapes have invalid format (e.g. `\uFFFF`). It also makes rustc always tell the user that escape may not be used in byte-strings and bytes and fixes some spans to not include unecessary characters and include escape backslash in some others.
2015-07-13Tell unicode escapes can’t be used as bytes earlier/moreSimonas Kazlauskas-16/+14
2015-07-10Change some instances of .connect() to .join()Wesley Wiser-1/+1
2015-07-10Improve some of the string escape diagnostic spansSimonas Kazlauskas-6/+4
2015-07-10Improve incomplete unicode escape reportingSimonas Kazlauskas-5/+19
This improves diagnostic messages when \u escape is used incorrectly and { is missing. Instead of saying “unknown character escape: u”, it will now report that unicode escape sequence is incomplete and suggest what the correct syntax is.
2015-07-01Make the unused_mut lint smarter with respect to locals.Ariel Ben-Yehuda-1/+1
Fixes #26332
2015-06-22fix minor indentation issuesYongqian Li-13/+13
2015-05-17Auto merge of #25387 - eddyb:syn-file-loader, r=nikomatsakisbors-2/+2
This allows compiling entire crates from memory or preprocessing source files before they are tokenized. Minor API refactoring included, which is a [breaking-change] for libsyntax users: * `ParseSess::{next_node_id, reserve_node_ids}` moved to rustc's `Session` * `new_parse_sess` -> `ParseSess::new` * `new_parse_sess_special_handler` -> `ParseSess::with_span_handler` * `mk_span_handler` -> `SpanHandler::new` * `default_handler` -> `Handler::new` * `mk_handler` -> `Handler::with_emitter` * `string_to_filemap(sess source, path)` -> `sess.codemap().new_filemap(path, source)`
2015-05-14Fix stupid mistake from previous commitLee Jeffery-2/+5
2015-05-14Make BytePos calculation same as originalLee Jeffery-1/+2
2015-05-14syntax: refactor (Span)Handler and ParseSess constructors to be methods.Eduard Burtescu-2/+2
2015-05-13Added test to check that newlines are stripped from commentsLee Jeffery-0/+9
2015-05-13Fix byte offset and error message inconsistenciesLee Jeffery-3/+3
2015-05-08Fix CRLF line-ending parsing for comments.Lee Jeffery-24/+26
2015-04-25Interpolate AST nodes in quasiquote.Geoffry Song-107/+0
This changes the `ToTokens` implementations for expressions, statements, etc. with almost-trivial ones that produce `Interpolated(*Nt(...))` pseudo-tokens. In this way, quasiquote now works the same way as macros do: already-parsed AST fragments are used as-is, not reparsed. The `ToSource` trait is removed. Quasiquote no longer involves pretty-printing at all, which removes the need for the `encode_with_hygiene` hack. All associated machinery is removed. A new `Nonterminal` is added, NtArm, which the parser now interpolates. This is just for quasiquote, not macros (although it could be in the future). `ToTokens` is no longer implemented for `Arg` (although this could be added again) and `Generics` (which I don't think makes sense). This breaks any compiler extensions that relied on the ability of `ToTokens` to turn AST fragments back into inspectable token trees. For this reason, this closes #16987. As such, this is a [breaking-change]. Fixes #16472. Fixes #15962. Fixes #17397. Fixes #16617.
2015-04-24Change name of unit test sub-module to "tests".Johannes Oertel-2/+2
Changes the style guidelines regarding unit tests to recommend using a sub-module named "tests" instead of "test" for unit tests as "test" might clash with imports of libtest.
2015-04-21syntax: Copy unstable str::char_at into libsyntaxErick Tryzelaar-12/+14
2015-04-21syntax: Replace String::from_str with the stable String::fromErick Tryzelaar-1/+1
2015-04-21syntax: remove uses of `.into_cow()`Erick Tryzelaar-4/+4
2015-04-14Negative case of `len()` -> `is_empty()`Tamir Duberstein-2/+2
`s/([^\(\s]+\.)len\(\) [(?:!=)>] 0/!$1is_empty()/g`
2015-04-13Added a help span which informs the user about the escaping of curly braces ↵Manuel Hoffmann-1/+7
in a format string if a wrongly escaped one is detected in a string.
2015-04-05Work towards a non-panicing parser (libsyntax)Phil Dawes-2/+2
- Functions in parser.rs return PResult<> rather than panicing - Other functions in libsyntax call panic! explicitly for now if they rely on panicing behaviour. - 'panictry!' macro added as scaffolding while converting panicing functions. (This does the same as 'unwrap()' but is easier to grep for and turn into try!()) - Leaves panicing wrappers for the following functions so that the quote_* macros behave the same: - parse_expr, parse_item, parse_pat, parse_arm, parse_ty, parse_stmt
2015-03-31rollup merge of #23872: huonw/eager-lexingAlex Crichton-14/+26
Conflicts: src/libsyntax/parse/lexer/mod.rs
2015-03-31Stabilize std::numAaron Turon-4/+3
This commit stabilizes the `std::num` module: * The `Int` and `Float` traits are deprecated in favor of (1) the newly-added inherent methods and (2) the generic traits available in rust-lang/num. * The `Zero` and `One` traits are reintroduced in `std::num`, which together with various other traits allow you to recover the most common forms of generic programming. * The `FromStrRadix` trait, and associated free function, is deprecated in favor of inherent implementations. * A wide range of methods and constants for both integers and floating point numbers are now `#[stable]`, having been adjusted for integer guidelines. * `is_positive` and `is_negative` are renamed to `is_sign_positive` and `is_sign_negative`, in order to address #22985 * The `Wrapping` type is moved to `std::num` and stabilized; `WrappingOps` is deprecated in favor of inherent methods on the integer types, and direct implementation of operations on `Wrapping<X>` for each concrete integer type `X`. Closes #22985 Closes #21069 [breaking-change]
2015-03-31Lex binary and octal literals more eagerly.Huon Wilson-13/+25
Previously 0b12 was considered two tokens, 0b1 and 2, as 2 isn't a valid base 2 digit. This patch changes that to collapse them into one (and makes `0b12` etc. an error: 2 isn't a valid base 2 digit). This may break some macro invocations of macros with `tt` (or syntax extensions) that rely on adjacent digits being separate tokens and hence is a [breaking-change] The fix is to separate the tokens, e.g. `0b12` -> `0b1 2`. cc https://github.com/rust-lang/rfcs/pull/879
2015-03-28Rollup merge of #23803 - richo:unused-braces, r=ManishearthManish Goregaokar-1/+1
Pretty much what it says on the tin.
2015-03-28cleanup: Remove unused braces in use statementsRicho Healey-1/+1
2015-03-27Prevent ICEs when parsing invalid escapes, closes #23620Florian Hahn-11/+29
2015-03-25Add trivial cast lints.Nick Cameron-2/+2
This permits all coercions to be performed in casts, but adds lints to warn in those cases. Part of this patch moves cast checking to a later stage of type checking. We acquire obligations to check casts as part of type checking where we previously checked them. Once we have type checked a function or module, then we check any cast obligations which have been acquired. That means we have more type information available to check casts (this was crucial to making coercions work properly in place of some casts), but it means that casts cannot feed input into type inference. [breaking change] * Adds two new lints for trivial casts and trivial numeric casts, these are warn by default, but can cause errors if you build with warnings as errors. Previously, trivial numeric casts and casts to trait objects were allowed. * The unused casts lint has gone. * Interactions between casting and type inference have changed in subtle ways. Two ways this might manifest are: - You may need to 'direct' casts more with extra type information, for example, in some cases where `foo as _ as T` succeeded, you may now need to specify the type for `_` - Casts do not influence inference of integer types. E.g., the following used to type check: ``` let x = 42; let y = &x as *const u32; ``` Because the cast would inform inference that `x` must have type `u32`. This no longer applies and the compiler will fallback to `i32` for `x` and thus there will be a type error in the cast. The solution is to add more type information: ``` let x: u32 = 42; let y = &x as *const u32; ```
2015-03-20don't use Result::ok just to be able to use unwrap/unwrap_orOliver Schneider-2/+2
2015-03-17std: Tweak some unstable features of `str`Alex Crichton-11/+12
This commit clarifies some of the unstable features in the `str` module by moving them out of the blanket `core` and `collections` features. The following methods were moved to the `str_char` feature which generally encompasses decoding specific characters from a `str` and dealing with the result. It is unclear if any of these methods need to be stabilized for 1.0 and the most conservative route for now is to continue providing them but to leave them as unstable under a more specific name. * `is_char_boundary` * `char_at` * `char_range_at` * `char_at_reverse` * `char_range_at_reverse` * `slice_shift_char` The following methods were moved into the generic `unicode` feature as they are specifically enabled by the `unicode` crate itself. * `nfd_chars` * `nfkd_chars` * `nfc_chars` * `graphemes` * `grapheme_indices` * `width`
2015-03-13Fallout of std::old_io deprecationAlex Crichton-2/+2