| Age | Commit message (Collapse) | Author | Lines |
|---|
|
Discovered in #50061 we're falling off the "happy path" of using a stringified
token stream more often than we should. This was due to the fact that a
user-written token like `0xf` is equality-different from the stringified token
of `15` (despite being semantically equivalent).
This patch updates the call to `eq_unspanned` with an even more awful solution,
`probably_equal_for_proc_macro`, which ignores the value of each token and
basically only compares the structure of the token stream, assuming that the AST
doesn't change just one token at a time.
While this is a step towards fixing #50061 there is still one regression
from #49154 which needs to be fixed.
|
|
This commit adds even more pessimization to use the cached `TokenStream` inside
of an AST node. As a reminder the `proc_macro` API requires taking an arbitrary
AST node and transforming it back into a `TokenStream` to hand off to a
procedural macro. Such functionality isn't actually implemented in rustc today,
so the way `proc_macro` works today is that it stringifies an AST node and then
reparses for a list of tokens.
This strategy unfortunately loses all span information, so we try to avoid it
whenever possible. Implemented in #43230 some AST nodes have a `TokenStream`
cache representing the tokens they were originally parsed from. This
`TokenStream` cache, however, has turned out to not always reflect the current
state of the item when it's being tokenized. For example `#[cfg]` processing or
macro expansion could modify the state of an item. Consequently we've seen a
number of bugs (#48644 and #49846) related to using this stale cache.
This commit tweaks the usage of the cached `TokenStream` to compare it to our
lossy stringification of the token stream. If the tokens that make up the cache
and the stringified token stream are the same then we return the cached version
(which has correct span information). If they differ, however, then we will
return the stringified version as the cache has been invalidated and we just
haven't figured that out.
Closes #48644
Closes #49846
|
|
|
|
|
|
When gluing two tokens, the second of which is joint, the result should also be
joint.
This fixes an issue with joining three `Dot` tokens to make a `DotDotDot` - the
intermediate `DotDot` would not be joint and therefore we would not attempt to
glue the last `Dot` token, yielding `.. .` instead of `...`.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Parsing `a as usize > b` always works, but `a as usize < b` was a
parsing error because the parser would think the `<` started a generic
type argument for `usize`. The parser now attempts to parse as before,
and if a DiagnosticError is returned, try to parse again as a type with
no generic arguments. If this fails, return the original
`DiagnosticError`.
|
|
This is useful if parsing from stdin or a String and don't want to try and read in a module from another file. Instead we just leave a stub in the AST.
|
|
This is mostly removing stray ampersands, needless returns and lifetimes.
|
|
|
|
macros: improve the `TokenStream` quoter
This PR
- renames the `TokenStream` quoter from `qquote!` to `quote!`,
- uses `$` instead of `unquote` (e.g. `let toks: TokenStream = ...; quote!([$toks])`),
- allows unquoting `Token`s as well as `TokenTree`s and `TokenStream`s (fixes #39746), and
- to preserve syntactic space, requires that `$` be followed by
- a single identifier to unquote, or
- another `$` to produce a literal `$`.
r? @nrc
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
`tokenstream::TokenTree::Sequence`.
|
|
|
|
Fix typos in libsyntax/tokenstream.rs
|
|
|
|
|
|
|
|
|
|
fragment.
|
|
|
|
places.
|
|
|
|
|
|
This reverts commit 41745f30f751364bdce14428b7d3ffa5dd028903.
|
|
`Token::Interpolated(Nonterminal)` -> `Token::Interpolated(Rc<Nonterminal>)`.
|
|
|
|
|
|
This commit adds syntax extension forms matching the types for procedural macros 2.0 (RFC #1566), these still require the usual syntax extension boiler plate, but this is a first step towards proper implementation and should be useful for macros 1.1 stuff too.
Supports both attribute-like and function-like macros.
|
|
Kicking off libproc_macro
This PR introduces `libproc_macro`, which is currently quite bare-bones (just a few macro construction tools and an initial `quote!` macro).
This PR also introduces a few test cases for it, and an additional `shim` file (at `src/libsyntax/ext/proc_macro_shim.rs` to allow a facsimile usage of Macros 2.0 *today*!
|
|
|
|
efficient tokenstream usages.
|
|
|
|
|
|
tests and associated operations.
|
|
(instead of by reference)"
This reverts commit 5bf7970ac70b4e7781e7b2f3816720aa62fac6fd.
|
|
Miscellaneous low priority cleanup in `libsyntax`.
|
|
Modified tests to point to the new file now.
|
