diff options
| author | Chris Simpkins <git.simpkins@gmail.com> | 2020-04-05 22:50:28 -0400 |
|---|---|---|
| committer | Who? Me?! <mark-i-m@users.noreply.github.com> | 2020-04-18 11:11:33 -0500 |
| commit | 35f766a50c430da02e87db08fbc8eacc2015afa2 (patch) | |
| tree | 14ee4faaa6744120357861f6b969609acef1400a /src/doc/rustc-dev-guide | |
| parent | e8ba37d8d9ef212ca248ce6e7ef368406d43252d (diff) | |
| download | rust-35f766a50c430da02e87db08fbc8eacc2015afa2.tar.gz rust-35f766a50c430da02e87db08fbc8eacc2015afa2.zip | |
[overview.md] add documentation of lexer support for Unicode encoding
Diffstat (limited to 'src/doc/rustc-dev-guide')
| -rw-r--r-- | src/doc/rustc-dev-guide/src/overview.md | 4 |
1 files changed, 2 insertions, 2 deletions
diff --git a/src/doc/rustc-dev-guide/src/overview.md b/src/doc/rustc-dev-guide/src/overview.md index 751a01ec08a..d641718c8ee 100644 --- a/src/doc/rustc-dev-guide/src/overview.md +++ b/src/doc/rustc-dev-guide/src/overview.md @@ -28,8 +28,8 @@ we'll talk about that later. to the rest of the compilation process as a [`rustc_interface::Config`]. - The raw Rust source text is analyzed by a low-level lexer located in [`librustc_lexer`]. At this stage, the source text is turned into a stream of - atomic source code units known as _tokens_. (**TODO**: chrissimpkins - Maybe - discuss Unicode handling during this stage?) + atomic source code units known as _tokens_. The lexer supports the Unicode + character encoding. - The token stream passes through a higher-level lexer located in [`librustc_parse`] to prepare for the next stage of the compile process. The [`StringReader`] struct is used at this stage to perform a set of validations |
