about summary refs log tree commit diff
path: root/src/libsyntax/tokenstream.rs
AgeCommit message (Collapse)AuthorLines
2018-04-10proc_macro: Avoid cached TokenStream more oftenAlex Crichton-3/+5
This commit adds even more pessimization to use the cached `TokenStream` inside of an AST node. As a reminder the `proc_macro` API requires taking an arbitrary AST node and transforming it back into a `TokenStream` to hand off to a procedural macro. Such functionality isn't actually implemented in rustc today, so the way `proc_macro` works today is that it stringifies an AST node and then reparses for a list of tokens. This strategy unfortunately loses all span information, so we try to avoid it whenever possible. Implemented in #43230 some AST nodes have a `TokenStream` cache representing the tokens they were originally parsed from. This `TokenStream` cache, however, has turned out to not always reflect the current state of the item when it's being tokenized. For example `#[cfg]` processing or macro expansion could modify the state of an item. Consequently we've seen a number of bugs (#48644 and #49846) related to using this stale cache. This commit tweaks the usage of the cached `TokenStream` to compare it to our lossy stringification of the token stream. If the tokens that make up the cache and the stringified token stream are the same then we return the cached version (which has correct span information). If they differ, however, then we will return the stringified version as the cache has been invalidated and we just haven't figured that out. Closes #48644 Closes #49846
2018-03-18Initial implementation of RFC 2151, Raw IdentifiersLymia Aluysia-1/+1
2018-03-14Remove syntax and syntax_pos thread localsJohn Kåre Alsaker-33/+50
2018-01-10Glued tokens can themselves be joint.Geoffry Song-6/+24
When gluing two tokens, the second of which is joint, the result should also be joint. This fixes an issue with joining three `Dot` tokens to make a `DotDotDot` - the intermediate `DotDot` would not be joint and therefore we would not attempt to glue the last `Dot` token, yielding `.. .` instead of `...`.
2017-08-30Make fields of `Span` privateVadim Petrochenkov-7/+3
2017-07-21Review commentsEsteban Küber-1/+1
2017-07-20Use the macro structure spans instead of the invocationEsteban Küber-0/+32
2017-06-26Address review comments.Jeffrey Seyfried-4/+6
2017-06-26Add `LazyTokenStream`.Jeffrey Seyfried-12/+33
2017-06-26Implement `quote!` and other `proc_macro` API.Jeffrey Seyfried-16/+136
2017-06-26Clean up `tokenstream::Cursor` and `proc_macro`.Jeffrey Seyfried-27/+27
2017-06-11Learn to parse `a as usize < b`Esteban Küber-0/+3
Parsing `a as usize > b` always works, but `a as usize < b` was a parsing error because the parser would think the `<` started a generic type argument for `usize`. The parser now attempts to parse as before, and if a DiagnosticError is returned, try to parse again as a type with no generic arguments. If this fails, return the original `DiagnosticError`.
2017-05-18Add an option to the parser to avoid parsing out of line modulesNick Cameron-1/+1
This is useful if parsing from stdin or a String and don't want to try and read in a module from another file. Instead we just leave a stub in the AST.
2017-05-12Fix some clippy warnings in libsyntaxAndre Bogus-7/+7
This is mostly removing stray ampersands, needless returns and lifetimes.
2017-03-29Merge `ExpnId` and `SyntaxContext`.Jeffrey Seyfried-7/+9
2017-03-19Rollup merge of #40532 - jseyfried:improve_tokenstream_quoter, r=nrcCorey Farwell-0/+6
macros: improve the `TokenStream` quoter This PR - renames the `TokenStream` quoter from `qquote!` to `quote!`, - uses `$` instead of `unquote` (e.g. `let toks: TokenStream = ...; quote!([$toks])`), - allows unquoting `Token`s as well as `TokenTree`s and `TokenStream`s (fixes #39746), and - to preserve syntactic space, requires that `$` be followed by - a single identifier to unquote, or - another `$` to produce a literal `$`. r? @nrc
2017-03-15Improve the `TokenStream` quoter.Jeffrey Seyfried-0/+6
2017-03-14Refactor `Attribute` to use `Path` and `TokenStream` instead of `MetaItem`.Jeffrey Seyfried-1/+1
2017-03-03Fix fallout in unit tests.Jeffrey Seyfried-11/+3
2017-03-03Integrate `TokenStream`.Jeffrey Seyfried-79/+67
2017-03-03Introduce `syntax::parse::parser::TokenCursor`.Jeffrey Seyfried-68/+46
2017-03-03Optimize `syntax::tokenstream::Cursor`.Jeffrey Seyfried-57/+59
2017-03-03Remove lifetime parameter from `syntax::tokenstream::Cursor`.Jeffrey Seyfried-26/+32
2017-02-28Add `syntax::ext::tt::quoted::{TokenTree, ..}` and remove ↵Jeffrey Seyfried-57/+5
`tokenstream::TokenTree::Sequence`.
2017-02-28Clean up `ext::tt::transcribe::TtFrame`, rename to `Frame`.Jeffrey Seyfried-2/+2
2017-01-28Auto merge of #39360 - osa1:typos, r=GuillaumeGomezbors-1/+1
Fix typos in libsyntax/tokenstream.rs
2017-01-28Fix typos in libsyntax/tokenstream.rsÖmer Sinan Ağacan-1/+1
2017-01-23Remove `open_span` and `close_span` from `Delimited`.Jeffrey Seyfried-14/+16
2017-01-22Refactor `TokenStream`.Jeffrey Seyfried-684/+127
2017-01-17Refactor the parser to consume token trees.Jeffrey Seyfried-5/+8
2016-12-07macros: fix the expected paths for a non-inline module matched by an `item` ↵Jeffrey Seyfried-2/+6
fragment.
2016-11-21Fix fallout in `rustdoc` and tests.Jeffrey Seyfried-15/+18
2016-11-21Use `Symbol` instead of `InternedString` in the AST, HIR, and various other ↵Jeffrey Seyfried-3/+3
places.
2016-11-20Move `syntax::util::interner` -> `syntax::symbol`, cleanup.Jeffrey Seyfried-4/+5
2016-11-03Move doc comment desugaring into the parser.Jeffrey Seyfried-4/+2
2016-11-03Revert "macros: Improve `tt` fragments"Jeffrey Seyfried-7/+0
This reverts commit 41745f30f751364bdce14428b7d3ffa5dd028903.
2016-11-03Reduce the size of `Token` and make it cheaper to clone by refactoringJeffrey Seyfried-3/+9
`Token::Interpolated(Nonterminal)` -> `Token::Interpolated(Rc<Nonterminal>)`.
2016-10-29Move `CrateConfig` from `Crate` to `ParseSess`.Jeffrey Seyfried-1/+1
2016-10-19Improve `$crate`.Jeffrey Seyfried-7/+0
2016-09-22Adds a `ProcMacro` form of syntax extensionNick Cameron-0/+7
This commit adds syntax extension forms matching the types for procedural macros 2.0 (RFC #1566), these still require the usual syntax extension boiler plate, but this is a first step towards proper implementation and should be useful for macros 1.1 stuff too. Supports both attribute-like and function-like macros.
2016-08-16Auto merge of #35538 - cgswords:libproc_macro, r=nrcbors-0/+6
Kicking off libproc_macro This PR introduces `libproc_macro`, which is currently quite bare-bones (just a few macro construction tools and an initial `quote!` macro). This PR also introduces a few test cases for it, and an additional `shim` file (at `src/libsyntax/ext/proc_macro_shim.rs` to allow a facsimile usage of Macros 2.0 *today*!
2016-08-16Proc_macro is alivecgswords-0/+6
2016-08-10Implemented a smarter concatenation system that will hopefully produce more ↵cgswords-20/+92
efficient tokenstream usages.
2016-08-01Reimplemented tokenstreams as ropes and reduced the exposed TokenStream API.cgswords-609/+356
2016-07-23macros: Improve `tt` fragmentsJeffrey Seyfried-0/+4
2016-07-07Preliminary implementation for TokenStreams and TokenSlices, including unit ↵cgswords-39/+1121
tests and associated operations.
2016-07-04Revert "Change `fold_tt` and `fold_tts` to take token trees by value ↵Jeffrey Seyfried-4/+6
(instead of by reference)" This reverts commit 5bf7970ac70b4e7781e7b2f3816720aa62fac6fd.
2016-06-26Rollup merge of #33943 - jseyfried:libsyntax_cleanup, r=nrcJeffrey Seyfried-5/+4
Miscellaneous low priority cleanup in `libsyntax`.
2016-06-21Refactored tokentrees into their own files in preparation for tokenstreams. ↵cgswords-0/+211
Modified tests to point to the new file now.
generated by cgit-pink 1.4.1-3-g733a5 (git 2.37.1) at 2026-02-24 01:12:41 +0000