https://github.com/rust-lang/rust http://git.dreamy.place/mirrors/rust/atom?h=try 2020-02-29T17:47:10+00:00 2020-02-29T17:47:10+00:00 Vadim Petrochenkov vadim.petrochenkov@gmail.com 2020-02-29T17:10:08+00:00 urn:sha1:e94d3b70cb495df83230cff9ef5bb00f235883e4 2020-02-28T16:17:26+00:00 Mazdak Farrokhzad twingoow@gmail.com 2020-02-28T16:17:26+00:00 urn:sha1:3828fa2852bf5e622267b24c4eda74e4fa6cb5f9 parser: `token` -> `normalized_token`, `nonnormalized_token` -> `token` So, after https://github.com/rust-lang/rust/pull/69006, its follow-ups and an attempt to remove `Parser::prev_span` I came to the conclusion that the unnormalized token and its span is what you want in most cases, so it should be default. Normalization only makes difference in few cases where we are checking against `token::Ident` or `token::Lifetime` specifically. This PR uses `normalized_token` for those cases. Using normalization explicitly means that people writing code should remember about `NtIdent` and `NtLifetime` in general. (That is alleviated by the fact that `token.ident()` and `fn parse_ident_*` are already written.) Remembering about `NtIdent`, was, however, already the case, kind of, because the implicit normalization was performed only for the current/previous token, but not for things like `look_ahead`. As a result, most of token classification methods in `token.rs` already take `NtIdent` into account (this PR fixes a few pre-existing minor mistakes though). The next step is removing `normalized(_prev)_token` entirely and replacing it with `token.ident()` (mostly) and `token.normalize()` (occasionally). I want to make it a separate PR for that and run it though perf. `normalized_token` filled on every bump has both a potential to avoid repeated normalization, and to do unnecessary work in advance (it probably doesn't matter anyway, the normalization is very cheap). r? @Centril 2020-02-24T17:25:32+00:00 Vadim Petrochenkov vadim.petrochenkov@gmail.com 2020-02-24T09:06:45+00:00 urn:sha1:d134385823525255546deaa930af2d5b1eada0df 2020-02-24T10:03:58+00:00 Vadim Petrochenkov vadim.petrochenkov@gmail.com 2020-02-22T17:19:49+00:00 urn:sha1:59261f0a7cf264f16e2b6a4e87d4249b212e920e 2020-02-22T14:28:10+00:00 Vadim Petrochenkov vadim.petrochenkov@gmail.com 2020-02-22T13:22:38+00:00 urn:sha1:4356d18e4ab262a6703fa3a901c7cf00e9d27cc7 2020-02-13T09:39:24+00:00 Mazdak Farrokhzad twingoow@gmail.com 2020-01-30T12:02:06+00:00 urn:sha1:a833be21626890de406e12f2561d2ffbda4aadb4 2020-01-30T00:19:40+00:00 Nicholas Nethercote nnethercote@mozilla.com 2019-12-13T00:25:08+00:00 urn:sha1:7d2173ed27c1cddc4d4a7a9755f244b66cf1ec81 This commit reduces the size of `Nonterminal` from a 72 bytes to 40 bytes (on x86-64). 2020-01-30T00:18:56+00:00 Nicholas Nethercote nnethercote@mozilla.com 2019-12-12T05:41:18+00:00 urn:sha1:0d69fe8308a76630a104504c14e1d3d74e2a3f15 This commit reduces the size of `Nonterminal` from a whopping 240 bytes to 72 bytes (on x86-64), which gets it below the `memcpy` threshold. It also removes some impedance mismatches with `Annotatable`, which already uses `P` for these variants. 2020-01-02T12:57:04+00:00 Mazdak Farrokhzad twingoow@gmail.com 2020-01-01T18:30:57+00:00 urn:sha1:4ff12ce4c14d348dfe53d1f74ab53c878f4fb4f8 2020-01-01T06:15:18+00:00 Vadim Petrochenkov vadim.petrochenkov@gmail.com 2019-12-31T17:15:40+00:00 urn:sha1:70f1d57048d2c73c8c018d0d65ceb0e2ca5d9dae