about summary refs log tree commit diff
path: root/src/libsyntax/tokenstream.rs
AgeCommit message (Collapse)AuthorLines
2019-11-11Tiny cleanup to size assertionsVadim Petrochenkov-3/+1
2019-11-10move config.rs to libsyntax_expandMazdak Farrokhzad-4/+1
2019-11-07syntax::parser::token -> syntax::tokenMazdak Farrokhzad-1/+1
2019-11-06Rollup merge of #66086 - RalfJung:smallvec, r=nagisaMazdak Farrokhzad-1/+1
bump smallvec to 1.0 This includes https://github.com/servo/rust-smallvec/pull/162, fixing an unsoundness in smallvec. See https://github.com/servo/rust-smallvec/pull/175 for the 1.0 release announcement. Cc @mbrubeck @emilio
2019-11-04bump smallvec to 1.0Ralf Jung-1/+1
2019-11-03syntax: Avoid span arithmetics for delimiter tokensVadim Petrochenkov-15/+5
2019-10-21Derive `Rustc{En,De}codable` for `TokenStream`.Nicholas Nethercote-15/+2
`TokenStream` used to be a complex type, but it is now just a newtype around a `Lrc<Vec<TreeAndJoint>>`. Currently it uses custom encoding that discards the `IsJoint` and custom decoding that adds `NonJoint` back in for every token tree. This requires building intermediate `Vec<TokenTree>`s. This commit makes `TokenStream` derive `Rustc{En,De}codable`. This simplifies the code, and avoids the creation of the intermediate vectors, saving up to 3% on various benchmarks. It also changes the AST JSON output in one test.
2019-10-18Change `MetaItem::tokens()` to `MetaItem::token_trees_and_joints()`.Nicholas Nethercote-4/+0
Likewise for `NestedMetaItem::tokens()`. Also, add `MetaItemKind::token_trees_and_joints()`, which `MetaItemKind::tokens()` now calls. This avoids some unnecessary `TokenTree` to `TokenStream` conversions, and removes the need for the clumsy `TokenStream::append_to_tree_and_joint_vec()`.
2019-10-18Make `TokenStream::from_iter` less general and more efficient.Nicholas Nethercote-3/+3
The current code has this impl: ``` impl<T: Into<TokenStream>> iter::FromIterator<T> for TokenStream ``` If given an `IntoIterator<Item = TokenTree>`, it will convert each individual `TokenTree` to a `TokenStream` (at the cost of two allocations: a `Vec` and an `Lrc`). It will then merge those `TokenStream`s into a single `TokenStream`. This is inefficient. This commit changes the impl to this less general one: ``` impl iter::FromIterator<TokenTree> for TokenStream ``` It collects the `TokenTree`s into a single `Vec` first and then converts that to a `TokenStream` by wrapping it in a single `Lrc`. The previous generality was unnecessary; no other code needs changing. This change speeds up several benchmarks by up to 4%.
2019-10-16move syntax::ext to new crate syntax_expandMazdak Farrokhzad-1/+1
2019-10-14Rollup merge of #65261 - nnethercote:rm-Option-from-TokenStream, r=petrochenkovTyler Mandry-133/+91
Remove `Option` from `TokenStream` A code simplification. r? @petrochenkov
2019-10-14Remove the `Option` in `TokenStream`.Nicholas Nethercote-133/+91
It means an allocation is required to create an empty `TokenStream`, but all other operations are simpler and marginally faster due to not having to check for `None`. Overall it simplifies the code for a negligible performance effect. The commit also removes `TokenStream::empty` by implementing `Default`, which is now possible.
2019-10-13tokenstream: don't depend on pprustMazdak Farrokhzad-8/+1
2019-10-08Optimize `TokenStreamBuilder::push`.Nicholas Nethercote-51/+43
Currently, when two tokens must be glued together, this function duplicates large chunks of the existing streams. This can cause quadratic behaviour. This commit changes the function so that it overwrites the last token with a glued token, which avoids the quadratic behaviour. This removes the need for `TokenStreamBuilder::push_all_but_{first,last}_tree`. The commit also restructures `push` somewhat, by removing `TokenStream::{first_tree_and_joint,last_tree_if_joint}` in favour of more pattern matching and some comments. This makes the code shorter, and in my opinion, more readable.
2019-10-08Optimize `TokenStream::from_streams`.Nicholas Nethercote-10/+37
Currently, this function creates a new empty stream, and then appends the elements from each given stream onto that stream. This can cause quadratic behaviour. This commit changes the function so that it modifies the first stream (which can be long) by extending it with the elements from the subsequent streams (which are almost always short), which avoids the quadratic behaviour.
2019-09-22push TokenTree::parse downAleksey Kladov-15/+0
2019-09-22pull mbe token tree definition upAleksey Kladov-2/+2
2019-09-22rename tt -> mbe, part 2Aleksey Kladov-1/+1
2019-09-22reduce visibility of a bunch of stuff in ext::ttAleksey Kladov-1/+1
2019-09-07Aggregation of cosmetic changes made during work on REPL PRs: libsyntaxAlexander Regueiro-4/+3
2019-09-03use TokenStream rather than &[TokenTree] for built-in macrosAleksey Kladov-1/+1
That way, we don't loose the jointness info
2019-08-23hygiene: Require passing transparency explicitly to `apply_mark`Vadim Petrochenkov-8/+1
2019-08-19glue tokens when building token streamAleksey Kladov-1/+1
2019-08-02libsyntax: Unconfigure tests during normal buildVadim Petrochenkov-111/+3
2019-07-23cleanup: Remove `extern crate serialize as rustc_serialize`sVadim Petrochenkov-1/+1
2019-07-19Adjust other names after the `Mark` renamingVadim Petrochenkov-3/+3
2019-07-19libsyntax: Remove `Mark` into `ExpnId`Vadim Petrochenkov-2/+2
2019-07-04Switch master to 1.38Mark Rousskov-11/+0
2019-06-14put back the workarounds for #60846Felix S Klock II-0/+11
based on https://github.com/rust-lang/rust/pull/61754#issuecomment-501743750 I am adding `bootstrap` to the cfg-preconditions for the two manual `unsafe impls`'s of `Send` and `Sync` for `TokenTree`.
2019-06-12remove hacks that are no longer neededNiko Matsakis-11/+0
2019-06-08syntax: Remove `Deref` impl from `Token`Vadim Petrochenkov-8/+0
2019-06-08syntax: Move most of the `TokenKind` methods to `Token`Vadim Petrochenkov-3/+2
2019-06-06Some code cleanup and tidy/test fixesVadim Petrochenkov-2/+3
2019-06-06syntax: Switch function parameter order in `TokenTree::token`Vadim Petrochenkov-9/+9
2019-06-06syntax: Remove duplicate span from `token::Ident`Vadim Petrochenkov-2/+2
2019-06-06syntax: Add some helper methods to `Token`Vadim Petrochenkov-1/+1
2019-06-06syntax: Use `Token` in `TokenTree::Token`Vadim Petrochenkov-57/+54
2019-06-06syntax: Rename `TokenAndSpan` into `Token`Vadim Petrochenkov-1/+0
2019-06-06syntax: Rename `Token` into `TokenKind`Vadim Petrochenkov-7/+7
2019-06-06Always use token kinds through `token` module rather than `Token` typeVadim Petrochenkov-12/+12
2019-05-27Short circuit Send and Sync impls for TokenTreeJohn Kåre Alsaker-0/+22
2019-05-21Move `edition` outside the hygiene lock and avoid accessing itJohn Kåre Alsaker-16/+18
2019-05-19Improve type size assertionsVadim Petrochenkov-2/+2
Now they - Tell what the new size is, when it changes - Do not require passing an identifier
2019-04-24Add guard for missing comma in macro call suggestionEsteban Küber-2/+4
2019-03-29Use `SmallVec` in `TokenStreamBuilder`.Nicholas Nethercote-6/+8
This reduces by 12% the number of allocations done for a "clean incremental" of `webrender_api`, which reduces the instruction count by about 0.5%. It also reduces instruction counts by up to 1.4% across a range of rustc-perf benchmark runs.
2019-03-13Fix operator precedenceEsteban Küber-2/+2
2019-03-11Be more discerning on when to attempt suggesting a comma in a macro invocationEsteban Küber-3/+5
2019-02-25Restrict value in key-value attributes to literalsVadim Petrochenkov-1/+1
2019-02-18Make `interpolated_to_tokenstream` a method on `Nonterminal`.Nicholas Nethercote-2/+2
2019-02-10rustc: doc commentsAlexander Regueiro-2/+3
generated by cgit-pink 1.4.1-3-g733a5 (git 2.37.1) at 2026-02-26 09:34:22 +0000