about summary refs log tree commit diff
path: root/src/doc/rustc-dev-guide
diff options
context:
space:
mode:
authorChris Simpkins <git.simpkins@gmail.com>2020-04-05 22:50:28 -0400
committerWho? Me?! <mark-i-m@users.noreply.github.com>2020-04-18 11:11:33 -0500
commit35f766a50c430da02e87db08fbc8eacc2015afa2 (patch)
tree14ee4faaa6744120357861f6b969609acef1400a /src/doc/rustc-dev-guide
parente8ba37d8d9ef212ca248ce6e7ef368406d43252d (diff)
downloadrust-35f766a50c430da02e87db08fbc8eacc2015afa2.tar.gz
rust-35f766a50c430da02e87db08fbc8eacc2015afa2.zip
[overview.md] add documentation of lexer support for Unicode encoding
Diffstat (limited to 'src/doc/rustc-dev-guide')
-rw-r--r--src/doc/rustc-dev-guide/src/overview.md4
1 files changed, 2 insertions, 2 deletions
diff --git a/src/doc/rustc-dev-guide/src/overview.md b/src/doc/rustc-dev-guide/src/overview.md
index 751a01ec08a..d641718c8ee 100644
--- a/src/doc/rustc-dev-guide/src/overview.md
+++ b/src/doc/rustc-dev-guide/src/overview.md
@@ -28,8 +28,8 @@ we'll talk about that later.
   to the rest of the compilation process as a [`rustc_interface::Config`].
 - The raw Rust source text is analyzed by a low-level lexer located in
   [`librustc_lexer`]. At this stage, the source text is turned into a stream of
-  atomic source code units known as _tokens_. (**TODO**: chrissimpkins - Maybe
-  discuss Unicode handling during this stage?)
+  atomic source code units known as _tokens_.  The lexer supports the Unicode
+  character encoding.
 - The token stream passes through a higher-level lexer located in
   [`librustc_parse`] to prepare for the next stage of the compile process. The
   [`StringReader`] struct is used at this stage to perform a set of validations