Merge #10995

10995: internal: switch from trait-based TokenSource to simple struct of arrays r=matklad a=matklad

cc #10765 

The idea here is to try to simplify the interface as best as we can. The original trait-based approach is a bit over-engineered and hard to debug. Here, we replace callback with just data. The next PR in series will replace the output `TreeSink` trait with data as well. 


The biggest drawback here is that we now require to materialize all parser's input up-front. This is a bad fit for macro by example: when you parse `$e:expr`, you might consume only part of the input. However, today's trait-based solution doesn't really help -- we were already materializing the whole thing! So, let's keep it simple!

Co-authored-by: Aleksey Kladov <aleksey.kladov@gmail.com>

0 files changed, 0 insertions, 0 deletions