about summary refs log tree commit diff
path: root/src/libsyntax/token.rs
AgeCommit message (Collapse)AuthorLines
2020-02-29Move directory `libsyntax` -> `librustc_ast`Vadim Petrochenkov-761/+0
2020-02-28Rollup merge of #69384 - petrochenkov:nounnorm, r=CentrilMazdak Farrokhzad-29/+25
parser: `token` -> `normalized_token`, `nonnormalized_token` -> `token` So, after https://github.com/rust-lang/rust/pull/69006, its follow-ups and an attempt to remove `Parser::prev_span` I came to the conclusion that the unnormalized token and its span is what you want in most cases, so it should be default. Normalization only makes difference in few cases where we are checking against `token::Ident` or `token::Lifetime` specifically. This PR uses `normalized_token` for those cases. Using normalization explicitly means that people writing code should remember about `NtIdent` and `NtLifetime` in general. (That is alleviated by the fact that `token.ident()` and `fn parse_ident_*` are already written.) Remembering about `NtIdent`, was, however, already the case, kind of, because the implicit normalization was performed only for the current/previous token, but not for things like `look_ahead`. As a result, most of token classification methods in `token.rs` already take `NtIdent` into account (this PR fixes a few pre-existing minor mistakes though). The next step is removing `normalized(_prev)_token` entirely and replacing it with `token.ident()` (mostly) and `token.normalize()` (occasionally). I want to make it a separate PR for that and run it though perf. `normalized_token` filled on every bump has both a potential to avoid repeated normalization, and to do unnecessary work in advance (it probably doesn't matter anyway, the normalization is very cheap). r? @Centril
2020-02-24syntax: Remove `Nt(Impl,Trait,Foreign)Item`Vadim Petrochenkov-9/+0
2020-02-24Add some missing support for `NtIdent`Vadim Petrochenkov-29/+25
2020-02-22parser: Cleanup `Parser::bump_with` and its usesVadim Petrochenkov-0/+33
2020-02-13parser: fuse free `fn` parsing together.Mazdak Farrokhzad-3/+5
2020-01-30Use `P` for `NtMeta`.Nicholas Nethercote-2/+2
This commit reduces the size of `Nonterminal` from a 72 bytes to 40 bytes (on x86-64).
2020-01-30Use `P` for `NtTraitItem`, `NtImplItem`, and `NtForeignItem`.Nicholas Nethercote-3/+7
This commit reduces the size of `Nonterminal` from a whopping 240 bytes to 72 bytes (on x86-64), which gets it below the `memcpy` threshold. It also removes some impedance mismatches with `Annotatable`, which already uses `P` for these variants.
2020-01-02Normalize `syntax::symbol` imports.Mazdak Farrokhzad-4/+3
2020-01-01Rename `syntax_pos` to `rustc_span` in source codeVadim Petrochenkov-2/+2
2019-12-22Format the worldMark Rousskov-119/+117
2019-12-12Remove `ast::{Impl,Trait}{Item,ItemKind}`.Mazdak Farrokhzad-2/+2
2019-11-23Derive HashStable for TokenKind.Camille GILLOT-9/+10
2019-11-23Use proc-macro for TokenTree.Camille GILLOT-0/+1
2019-11-23Rename StableHashingContextLike to HashStableContext.Camille GILLOT-1/+1
2019-11-22Invert implementations for TokenKind.Camille GILLOT-1/+10
Also export a bunch of Token-related impls.
2019-11-17Remove extern crate.Camille GILLOT-0/+1
2019-11-17HashStable literals in libsyntax.Camille GILLOT-2/+2
2019-11-11Tiny cleanup to size assertionsVadim Petrochenkov-3/+1
2019-11-10move syntax::parse -> librustc_parseMazdak Farrokhzad-22/+22
also move MACRO_ARGUMENTS -> librustc_parse
2019-11-07syntax::parser::token -> syntax::tokenMazdak Farrokhzad-0/+728