Lex binary and octal literals more eagerly.

Previously 0b12 was considered two tokens, 0b1 and 2, as 2 isn't a valid
base 2 digit. This patch changes that to collapse them into one (and
makes `0b12` etc. an error: 2 isn't a valid base 2 digit).

This may break some macro invocations of macros with `tt` (or syntax
extensions) that rely on adjacent digits being separate tokens and hence
is a

[breaking-change]

The fix is to separate the tokens, e.g. `0b12` -> `0b1 2`.

cc https://github.com/rust-lang/rfcs/pull/879

1 files changed, 14 insertions, 1 deletions

diff --git a/src/libsyntax/parse/mod.rs b/src/libsyntax/parse/mod.rs
index bea42a88bf5..f8820999c9d 100644
--- a/src/libsyntax/parse/mod.rs
+++ b/src/libsyntax/parse/mod.rs
@@ -735,7 +735,20 @@ pub fn integer_lit(s: &str, suffix: Option<&str>, sd: &SpanHandler, sp: Span) ->
 
     let res: u64 = match ::std::num::from_str_radix(s, base).ok() {
         Some(r) => r,
-        None => { sd.span_err(sp, "int literal is too large"); 0 }
+        None => {
+            // small bases are lexed as if they were base 10, e.g, the string
+            // might be `0b10201`. This will cause the conversion above to fail,
+            // but these cases have errors in the lexer: we don't want to emit
+            // two errors, and we especially don't want to emit this error since
+            // it isn't necessarily true.
+            let already_errored = base < 10 &&
+                s.chars().any(|c| c.to_digit(10).map_or(false, |d| d >= base));
+
+            if !already_errored {
+                sd.span_err(sp, "int literal is too large");
+            }
+            0
+        }
     };
 
     // adjust the sign