Rollup merge of #35539 - cgswords:ts_concat, r=nrc

Implemented a smarter TokenStream concatenation system

The new algorithm performs 'aggressive compacting' during concatenation as follows:

- If the nodes' combined total total length is less than 32, we copy both of
   them into a new vector and build a new leaf node.
- If one node is an internal node and the other is a 'small' leaf (length<32),
   we recur down the internal node on the appropriate side.
 - Otherwise, we construct a new internal node that points to them as left and
 right.

This should produce notably better behavior than the current concatenation implementation.

0 files changed, 0 insertions, 0 deletions