about summary refs log tree commit diff
path: root/src/libsyntax/tokenstream.rs
AgeCommit message (Collapse)AuthorLines
2020-02-29Move directory `libsyntax` -> `librustc_ast`Vadim Petrochenkov-486/+0
2020-01-01Rename `syntax_pos` to `rustc_span` in source codeVadim Petrochenkov-1/+1
2019-12-22Format the worldMark Rousskov-29/+23
2019-12-02syntax: Remove redundant span from `ast::Mac`Vadim Petrochenkov-0/+8
Also remove a couple of redundant `visit_mac` asserts
2019-11-23Use proc-macro for TokenTree.Camille GILLOT-21/+1
2019-11-23Rename StableHashingContextLike to HashStableContext.Camille GILLOT-2/+2
2019-11-22Fix rebase fallout.Camille GILLOT-0/+1
2019-11-22Invert implementations for TokenKind.Camille GILLOT-0/+31
Also export a bunch of Token-related impls.
2019-11-22Export HashStable for DelimSpan, Lit and Path.Camille GILLOT-1/+1
2019-11-11Tiny cleanup to size assertionsVadim Petrochenkov-3/+1
2019-11-10move config.rs to libsyntax_expandMazdak Farrokhzad-4/+1
2019-11-07syntax::parser::token -> syntax::tokenMazdak Farrokhzad-1/+1
2019-11-06Rollup merge of #66086 - RalfJung:smallvec, r=nagisaMazdak Farrokhzad-1/+1
bump smallvec to 1.0 This includes https://github.com/servo/rust-smallvec/pull/162, fixing an unsoundness in smallvec. See https://github.com/servo/rust-smallvec/pull/175 for the 1.0 release announcement. Cc @mbrubeck @emilio
2019-11-04bump smallvec to 1.0Ralf Jung-1/+1
2019-11-03syntax: Avoid span arithmetics for delimiter tokensVadim Petrochenkov-15/+5
2019-10-21Derive `Rustc{En,De}codable` for `TokenStream`.Nicholas Nethercote-15/+2
`TokenStream` used to be a complex type, but it is now just a newtype around a `Lrc<Vec<TreeAndJoint>>`. Currently it uses custom encoding that discards the `IsJoint` and custom decoding that adds `NonJoint` back in for every token tree. This requires building intermediate `Vec<TokenTree>`s. This commit makes `TokenStream` derive `Rustc{En,De}codable`. This simplifies the code, and avoids the creation of the intermediate vectors, saving up to 3% on various benchmarks. It also changes the AST JSON output in one test.
2019-10-18Change `MetaItem::tokens()` to `MetaItem::token_trees_and_joints()`.Nicholas Nethercote-4/+0
Likewise for `NestedMetaItem::tokens()`. Also, add `MetaItemKind::token_trees_and_joints()`, which `MetaItemKind::tokens()` now calls. This avoids some unnecessary `TokenTree` to `TokenStream` conversions, and removes the need for the clumsy `TokenStream::append_to_tree_and_joint_vec()`.
2019-10-18Make `TokenStream::from_iter` less general and more efficient.Nicholas Nethercote-3/+3
The current code has this impl: ``` impl<T: Into<TokenStream>> iter::FromIterator<T> for TokenStream ``` If given an `IntoIterator<Item = TokenTree>`, it will convert each individual `TokenTree` to a `TokenStream` (at the cost of two allocations: a `Vec` and an `Lrc`). It will then merge those `TokenStream`s into a single `TokenStream`. This is inefficient. This commit changes the impl to this less general one: ``` impl iter::FromIterator<TokenTree> for TokenStream ``` It collects the `TokenTree`s into a single `Vec` first and then converts that to a `TokenStream` by wrapping it in a single `Lrc`. The previous generality was unnecessary; no other code needs changing. This change speeds up several benchmarks by up to 4%.
2019-10-16move syntax::ext to new crate syntax_expandMazdak Farrokhzad-1/+1
2019-10-14Rollup merge of #65261 - nnethercote:rm-Option-from-TokenStream, r=petrochenkovTyler Mandry-133/+91
Remove `Option` from `TokenStream` A code simplification. r? @petrochenkov
2019-10-14Remove the `Option` in `TokenStream`.Nicholas Nethercote-133/+91
It means an allocation is required to create an empty `TokenStream`, but all other operations are simpler and marginally faster due to not having to check for `None`. Overall it simplifies the code for a negligible performance effect. The commit also removes `TokenStream::empty` by implementing `Default`, which is now possible.
2019-10-13tokenstream: don't depend on pprustMazdak Farrokhzad-8/+1
2019-10-08Optimize `TokenStreamBuilder::push`.Nicholas Nethercote-51/+43
Currently, when two tokens must be glued together, this function duplicates large chunks of the existing streams. This can cause quadratic behaviour. This commit changes the function so that it overwrites the last token with a glued token, which avoids the quadratic behaviour. This removes the need for `TokenStreamBuilder::push_all_but_{first,last}_tree`. The commit also restructures `push` somewhat, by removing `TokenStream::{first_tree_and_joint,last_tree_if_joint}` in favour of more pattern matching and some comments. This makes the code shorter, and in my opinion, more readable.
2019-10-08Optimize `TokenStream::from_streams`.Nicholas Nethercote-10/+37
Currently, this function creates a new empty stream, and then appends the elements from each given stream onto that stream. This can cause quadratic behaviour. This commit changes the function so that it modifies the first stream (which can be long) by extending it with the elements from the subsequent streams (which are almost always short), which avoids the quadratic behaviour.
2019-09-22push TokenTree::parse downAleksey Kladov-15/+0
2019-09-22pull mbe token tree definition upAleksey Kladov-2/+2
2019-09-22rename tt -> mbe, part 2Aleksey Kladov-1/+1
2019-09-22reduce visibility of a bunch of stuff in ext::ttAleksey Kladov-1/+1
2019-09-07Aggregation of cosmetic changes made during work on REPL PRs: libsyntaxAlexander Regueiro-4/+3
2019-09-03use TokenStream rather than &[TokenTree] for built-in macrosAleksey Kladov-1/+1
That way, we don't loose the jointness info
2019-08-23hygiene: Require passing transparency explicitly to `apply_mark`Vadim Petrochenkov-8/+1
2019-08-19glue tokens when building token streamAleksey Kladov-1/+1
2019-08-02libsyntax: Unconfigure tests during normal buildVadim Petrochenkov-111/+3
2019-07-23cleanup: Remove `extern crate serialize as rustc_serialize`sVadim Petrochenkov-1/+1
2019-07-19Adjust other names after the `Mark` renamingVadim Petrochenkov-3/+3
2019-07-19libsyntax: Remove `Mark` into `ExpnId`Vadim Petrochenkov-2/+2
2019-07-04Switch master to 1.38Mark Rousskov-11/+0
2019-06-14put back the workarounds for #60846Felix S Klock II-0/+11
based on https://github.com/rust-lang/rust/pull/61754#issuecomment-501743750 I am adding `bootstrap` to the cfg-preconditions for the two manual `unsafe impls`'s of `Send` and `Sync` for `TokenTree`.
2019-06-12remove hacks that are no longer neededNiko Matsakis-11/+0
2019-06-08syntax: Remove `Deref` impl from `Token`Vadim Petrochenkov-8/+0
2019-06-08syntax: Move most of the `TokenKind` methods to `Token`Vadim Petrochenkov-3/+2
2019-06-06Some code cleanup and tidy/test fixesVadim Petrochenkov-2/+3
2019-06-06syntax: Switch function parameter order in `TokenTree::token`Vadim Petrochenkov-9/+9
2019-06-06syntax: Remove duplicate span from `token::Ident`Vadim Petrochenkov-2/+2
2019-06-06syntax: Add some helper methods to `Token`Vadim Petrochenkov-1/+1
2019-06-06syntax: Use `Token` in `TokenTree::Token`Vadim Petrochenkov-57/+54
2019-06-06syntax: Rename `TokenAndSpan` into `Token`Vadim Petrochenkov-1/+0
2019-06-06syntax: Rename `Token` into `TokenKind`Vadim Petrochenkov-7/+7
2019-06-06Always use token kinds through `token` module rather than `Token` typeVadim Petrochenkov-12/+12
2019-05-27Short circuit Send and Sync impls for TokenTreeJohn Kåre Alsaker-0/+22
generated by cgit-pink 1.4.1-3-g733a5 (git 2.37.1) at 2026-02-19 21:51:29 +0000