Auto merge of #33588 - nikomatsakis:compiletest-ui, r=acrichto

add UI testing framework

This adds a framework for capturing and tracking the precise output of rustc, which allows us to check all manner of minor details with the output. It's pretty strict right now -- the output must match almost exactly -- and hence maybe a bit too strict. But I figure we can add wildcards or whatever later. There is also a script intended to make updating the references easy, though the script could make things a *bit* easier (in particular, it'd be nice if it would find the build directory for you automatically).

One thing I was wondering about is the best way to test colors. Since windows doesn't embed those in the output stream, this test framework can't test colors on windows -- so I figure we can just write tests that are ignored on windows and which pass `--color=always` or whatever to rustc.

cc @jonathandturner
r? @alexcrichton

6 files changed, 152 insertions, 0 deletions

diff --git a/src/test/ui/README.md b/src/test/ui/README.md
new file mode 100644
index 00000000000..dcdeabd8032
--- /dev/null
+++ b/src/test/ui/README.md
@@ -0,0 +1,31 @@
+# Guide to the UI Tests
+
+The UI tests are intended to capture the compiler's complete output,
+so that we can test all aspects of the presentation. They work by
+compiling a file (e.g., `hello_world/main.rs`), capturing the output,
+and then applying some normalization (see below). This normalized
+result is then compared against reference files named
+`hello_world/main.stderr` and `hello_world/main.stdout`. If either of
+those files doesn't exist, the output must be empty. If the test run
+fails, we will print out the current output, but it is also saved in
+`build/<target-triple>/test/ui/hello_world/main.stdout` (this path is
+printed as part of the test failure mesage), so you can run `diff` and
+so forth.
+
+# Editing and updating the reference files
+
+If you have changed the compiler's output intentionally, or you are
+making a new test, you can use the script `update-references.sh` to
+update the references. When you run the test framework, it will report
+various errors: in those errors is a command you can use to run the
+`update-references.sh` script, which will then copy over the files
+from the build directory and use them as the new reference. You can
+also just run `update-all-references.sh`. In both cases, you can run
+the script with `--help` to get a help message.
+
+# Normalization
+
+The normalization applied is aimed at filenames:
+
+- the test directory is replaced with `$DIR`
+- all backslashes (\) are converted to forward slashes (/) (for windows)
diff --git a/src/test/ui/hello_world/main.rs b/src/test/ui/hello_world/main.rs
new file mode 100644
index 00000000000..61183975577
--- /dev/null
+++ b/src/test/ui/hello_world/main.rs
@@ -0,0 +1,15 @@
+// Copyright 2013-2014 The Rust Project Developers. See the COPYRIGHT
+// file at the top-level directory of this distribution and at
+// http://rust-lang.org/COPYRIGHT.
+//
+// Licensed under the Apache License, Version 2.0 <LICENSE-APACHE or
+// http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your
+// option. This file may not be copied, modified, or distributed
+// except according to those terms.
+
+// Test that compiling hello world succeeds with no output of any kind.
+
+fn main() {
+    println!("Hello, world!");
+}
diff --git a/src/test/ui/mismatched_types/main.rs b/src/test/ui/mismatched_types/main.rs
new file mode 100644
index 00000000000..85d9fa53fcf
--- /dev/null
+++ b/src/test/ui/mismatched_types/main.rs
@@ -0,0 +1,17 @@
+// Copyright 2013-2014 The Rust Project Developers. See the COPYRIGHT
+// file at the top-level directory of this distribution and at
+// http://rust-lang.org/COPYRIGHT.
+//
+// Licensed under the Apache License, Version 2.0 <LICENSE-APACHE or
+// http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your
+// option. This file may not be copied, modified, or distributed
+// except according to those terms.
+
+// rustc-env:RUST_NEW_ERROR_FORMAT
+
+fn main() {
+    let x: u32 = (
+    );
+}
+
diff --git a/src/test/ui/mismatched_types/main.stderr b/src/test/ui/mismatched_types/main.stderr
new file mode 100644
index 00000000000..98bc11752e0
--- /dev/null
+++ b/src/test/ui/mismatched_types/main.stderr
@@ -0,0 +1,8 @@
+error: mismatched types [--explain E0308]
+  --> $DIR/main.rs:14:18
+14 |>     let x: u32 = (
+   |>                  ^ expected u32, found ()
+note: expected type `u32`
+note:    found type `()`
+
+error: aborting due to previous error
diff --git a/src/test/ui/update-all-references.sh b/src/test/ui/update-all-references.sh
new file mode 100755
index 00000000000..ddd69c399a5
--- /dev/null
+++ b/src/test/ui/update-all-references.sh
@@ -0,0 +1,31 @@
+#!/bin/bash
+#
+# Copyright 2015 The Rust Project Developers. See the COPYRIGHT
+# file at the top-level directory of this distribution and at
+# http://rust-lang.org/COPYRIGHT.
+#
+# Licensed under the Apache License, Version 2.0 <LICENSE-APACHE or
+# http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+# <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your
+# option. This file may not be copied, modified, or distributed
+# except according to those terms.
+
+# A script to update the references for all tests. The idea is that
+# you do a run, which will generate files in the build directory
+# containing the (normalized) actual output of the compiler. You then
+# run this script, which will copy those files over. If you find
+# yourself manually editing a foo.stderr file, you're doing it wrong.
+#
+# See all `update-references.sh`, if you just want to update a single test.
+
+if [[ "$1" == "--help" || "$1" == "-h" || "$1" == "" || "$2" != "" ]]; then
+    echo "usage: $0 <build-directory>"
+    echo ""
+    echo "For example:"
+    echo "   $0 ../../../build/x86_64-apple-darwin/test/ui"
+fi
+
+BUILD_DIR=$PWD/$1
+MY_DIR=$(dirname $0)
+cd $MY_DIR
+find . -name '*.rs' | xargs ./update-references.sh $BUILD_DIR
diff --git a/src/test/ui/update-references.sh b/src/test/ui/update-references.sh
new file mode 100755
index 00000000000..f0a6f8a3d44
--- /dev/null
+++ b/src/test/ui/update-references.sh
@@ -0,0 +1,50 @@
+#!/bin/bash
+#
+# Copyright 2015 The Rust Project Developers. See the COPYRIGHT
+# file at the top-level directory of this distribution and at
+# http://rust-lang.org/COPYRIGHT.
+#
+# Licensed under the Apache License, Version 2.0 <LICENSE-APACHE or
+# http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+# <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your
+# option. This file may not be copied, modified, or distributed
+# except according to those terms.
+
+# A script to update the references for particular tests. The idea is
+# that you do a run, which will generate files in the build directory
+# containing the (normalized) actual output of the compiler. This
+# script will then copy that output and replace the "expected output"
+# files. You can then commit the changes.
+#
+# If you find yourself manually editing a foo.stderr file, you're
+# doing it wrong.
+
+if [[ "$1" == "--help" || "$1" == "-h" || "$1" == "" || "$2" == "" ]]; then
+    echo "usage: $0 <build-directory> <relative-path-to-rs-files>"
+    echo ""
+    echo "For example:"
+    echo "   $0 ../../../build/x86_64-apple-darwin/test/ui *.rs */*.rs"
+fi
+
+MYDIR=$(dirname $0)
+
+BUILD_DIR="$1"
+shift
+
+while [[ "$1" != "" ]]; do
+    STDERR_NAME="${1/%.rs/.stderr}"
+    STDOUT_NAME="${1/%.rs/.stdout}"
+    shift
+    if [ -f $BUILD_DIR/$STDOUT_NAME ] && \
+           ! (diff $BUILD_DIR/$STDOUT_NAME $MYDIR/$STDOUT_NAME > /dev/null); then
+        echo updating $MYDIR/$STDOUT_NAME
+        cp $BUILD_DIR/$STDOUT_NAME $MYDIR/$STDOUT_NAME
+    fi
+    if [ -f $BUILD_DIR/$STDERR_NAME ] && \
+           ! (diff $BUILD_DIR/$STDERR_NAME $MYDIR/$STDERR_NAME > /dev/null); then
+        echo updating $MYDIR/$STDERR_NAME
+        cp $BUILD_DIR/$STDERR_NAME $MYDIR/$STDERR_NAME
+    fi
+done
+
+