travis: Parallelize tests on Android

Currently our slowest test suite on android, run-pass, takes over 5 times longer
than the x86_64 component (~400 -> ~2200s). Typically QEMU emulation does indeed
add overhead, but not 5x for this kind of workload. One of the slowest parts of
the Android process is that *compilation* happens serially. Tests themselves
need to run single-threaded on the emulator (due to how the test harness works)
and this forces the compiles themselves to be single threaded.

Now Travis gives us more than one core per machine, so it'd be much better if we
could take advantage of them! The emulator itself is still fundamentally
single-threaded, but we should see a nice speedup by sending binaries for it to
run much more quickly.

It turns out that we've already got all the tools to do this in-tree. The
qemu-test-{server,client} that are in use for the ARM Linux testing are a
perfect match for the Android emulator. This commit migrates the custom adb
management code in compiletest/rustbuild to the same qemu-test-{server,client}
implementation that ARM Linux uses.

This allows us to lift the parallelism restriction on the compiletest test
suites, namely run-pass. Consequently although we'll still basically run the
tests themselves in single threaded mode we'll be able to compile all of them in
parallel, keeping the pipeline much more full and using more cores for the work
at hand. Additionally the architecture here should be a bit speedier as it
should have less overhead than adb which is a whole new process on both the host
and the emulator!

Locally on an 8 core machine I've seen the run-pass test suite speed up from
taking nearly an hour to only taking 6 minutes. I don't think we'll see quite a
drastic speedup on Travis but I'm hoping this change can place the Android tests
well below 2 hours instead of just above 2 hours.

Because the client/server here are now repurposed for more than just QEMU,
they've been renamed to `remote-test-{server,client}`.

Note that this PR does not currently modify how debuginfo tests are executed on
Android. While parallelizable it wouldn't be quite as easy, so that's left to
another day. Thankfully that test suite is much smaller than the run-pass test
suite.

As a final fix I discovered that the ARM and Android test suites were actually
running all library unit tests (e.g. stdtest, coretest, etc) twice. I've
corrected that to only run tests once which should also give a nice boost in
overall cycle time here.

2 files changed, 287 insertions, 0 deletions

diff --git a/src/tools/remote-test-client/Cargo.toml b/src/tools/remote-test-client/Cargo.toml
new file mode 100644
index 00000000000..54739101f1e
--- /dev/null
+++ b/src/tools/remote-test-client/Cargo.toml
@@ -0,0 +1,6 @@
+[package]
+name = "remote-test-client"
+version = "0.1.0"
+authors = ["The Rust Project Developers"]
+
+[dependencies]
diff --git a/src/tools/remote-test-client/src/main.rs b/src/tools/remote-test-client/src/main.rs
new file mode 100644
index 00000000000..265354ff800
--- /dev/null
+++ b/src/tools/remote-test-client/src/main.rs
@@ -0,0 +1,281 @@
+// Copyright 2017 The Rust Project Developers. See the COPYRIGHT
+// file at the top-level directory of this distribution and at
+// http://rust-lang.org/COPYRIGHT.
+//
+// Licensed under the Apache License, Version 2.0 <LICENSE-APACHE or
+// http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your
+// option. This file may not be copied, modified, or distributed
+// except according to those terms.
+
+/// This is a small client program intended to pair with `remote-test-server` in
+/// this repository. This client connects to the server over TCP and is used to
+/// push artifacts and run tests on the server instead of locally.
+///
+/// Here is also where we bake in the support to spawn the QEMU emulator as
+/// well.
+
+use std::env;
+use std::fs::{self, File};
+use std::io::prelude::*;
+use std::io::{self, BufWriter};
+use std::net::TcpStream;
+use std::path::{Path, PathBuf};
+use std::process::{Command, Stdio};
+use std::thread;
+use std::time::Duration;
+
+macro_rules! t {
+    ($e:expr) => (match $e {
+        Ok(e) => e,
+        Err(e) => panic!("{} failed with {}", stringify!($e), e),
+    })
+}
+
+fn main() {
+    let mut args = env::args().skip(1);
+
+    match &args.next().unwrap()[..] {
+        "spawn-emulator" => {
+            spawn_emulator(&args.next().unwrap(),
+                           Path::new(&args.next().unwrap()),
+                           Path::new(&args.next().unwrap()),
+                           args.next().map(|s| s.into()))
+        }
+        "push" => {
+            push(Path::new(&args.next().unwrap()))
+        }
+        "run" => {
+            run(args.next().unwrap(), args.collect())
+        }
+        cmd => panic!("unknown command: {}", cmd),
+    }
+}
+
+fn spawn_emulator(target: &str,
+                  server: &Path,
+                  tmpdir: &Path,
+                  rootfs: Option<PathBuf>) {
+    if target.contains("android") {
+        start_android_emulator(server);
+    } else {
+        let rootfs = rootfs.as_ref().expect("need rootfs on non-android");
+        start_qemu_emulator(rootfs, server, tmpdir);
+    }
+
+    // Wait for the emulator to come online
+    loop {
+        let dur = Duration::from_millis(100);
+        if let Ok(mut client) = TcpStream::connect("127.0.0.1:12345") {
+            t!(client.set_read_timeout(Some(dur)));
+            t!(client.set_write_timeout(Some(dur)));
+            if client.write_all(b"ping").is_ok() {
+                let mut b = [0; 4];
+                if client.read_exact(&mut b).is_ok() {
+                    break
+                }
+            }
+        }
+        thread::sleep(dur);
+    }
+}
+
+fn start_android_emulator(server: &Path) {
+    println!("waiting for device to come online");
+    let status = Command::new("adb")
+                    .arg("wait-for-device")
+                    .status()
+                    .unwrap();
+    assert!(status.success());
+
+    println!("pushing server");
+    let status = Command::new("adb")
+                    .arg("push")
+                    .arg(server)
+                    .arg("/data/tmp/testd")
+                    .status()
+                    .unwrap();
+    assert!(status.success());
+
+    println!("forwarding tcp");
+    let status = Command::new("adb")
+                    .arg("forward")
+                    .arg("tcp:12345")
+                    .arg("tcp:12345")
+                    .status()
+                    .unwrap();
+    assert!(status.success());
+
+    println!("executing server");
+    Command::new("adb")
+                    .arg("shell")
+                    .arg("/data/tmp/testd")
+                    .spawn()
+                    .unwrap();
+}
+
+fn start_qemu_emulator(rootfs: &Path, server: &Path, tmpdir: &Path) {
+    // Generate a new rootfs image now that we've updated the test server
+    // executable. This is the equivalent of:
+    //
+    //      find $rootfs -print 0 | cpio --null -o --format=newc > rootfs.img
+    t!(fs::copy(server, rootfs.join("testd")));
+    let rootfs_img = tmpdir.join("rootfs.img");
+    let mut cmd = Command::new("cpio");
+    cmd.arg("--null")
+       .arg("-o")
+       .arg("--format=newc")
+       .stdin(Stdio::piped())
+       .stdout(Stdio::piped())
+       .current_dir(rootfs);
+    let mut child = t!(cmd.spawn());
+    let mut stdin = child.stdin.take().unwrap();
+    let rootfs = rootfs.to_path_buf();
+    thread::spawn(move || add_files(&mut stdin, &rootfs, &rootfs));
+    t!(io::copy(&mut child.stdout.take().unwrap(),
+                &mut t!(File::create(&rootfs_img))));
+    assert!(t!(child.wait()).success());
+
+    // Start up the emulator, in the background
+    let mut cmd = Command::new("qemu-system-arm");
+    cmd.arg("-M").arg("vexpress-a15")
+       .arg("-m").arg("1024")
+       .arg("-kernel").arg("/tmp/zImage")
+       .arg("-initrd").arg(&rootfs_img)
+       .arg("-dtb").arg("/tmp/vexpress-v2p-ca15-tc1.dtb")
+       .arg("-append").arg("console=ttyAMA0 root=/dev/ram rdinit=/sbin/init init=/sbin/init")
+       .arg("-nographic")
+       .arg("-redir").arg("tcp:12345::12345");
+    t!(cmd.spawn());
+
+    fn add_files(w: &mut Write, root: &Path, cur: &Path) {
+        for entry in t!(cur.read_dir()) {
+            let entry = t!(entry);
+            let path = entry.path();
+            let to_print = path.strip_prefix(root).unwrap();
+            t!(write!(w, "{}\u{0}", to_print.to_str().unwrap()));
+            if t!(entry.file_type()).is_dir() {
+                add_files(w, root, &path);
+            }
+        }
+    }
+}
+
+fn push(path: &Path) {
+    let client = t!(TcpStream::connect("127.0.0.1:12345"));
+    let mut client = BufWriter::new(client);
+    t!(client.write_all(b"push"));
+    send(path, &mut client);
+    t!(client.flush());
+
+    // Wait for an acknowledgement that all the data was received. No idea
+    // why this is necessary, seems like it shouldn't be!
+    let mut client = client.into_inner().unwrap();
+    let mut buf = [0; 4];
+    t!(client.read_exact(&mut buf));
+    assert_eq!(&buf, b"ack ");
+    println!("done pushing {:?}", path);
+}
+
+fn run(files: String, args: Vec<String>) {
+    let client = t!(TcpStream::connect("127.0.0.1:12345"));
+    let mut client = BufWriter::new(client);
+    t!(client.write_all(b"run "));
+
+    // Send over the args
+    for arg in args {
+        t!(client.write_all(arg.as_bytes()));
+        t!(client.write_all(&[0]));
+    }
+    t!(client.write_all(&[0]));
+
+    // Send over env vars
+    //
+    // Don't send over *everything* though as some env vars are set by and used
+    // by the client.
+    for (k, v) in env::vars() {
+        match &k[..] {
+            "PATH" |
+            "LD_LIBRARY_PATH" |
+            "PWD" => continue,
+            _ => {}
+        }
+        t!(client.write_all(k.as_bytes()));
+        t!(client.write_all(&[0]));
+        t!(client.write_all(v.as_bytes()));
+        t!(client.write_all(&[0]));
+    }
+    t!(client.write_all(&[0]));
+
+    // Send over support libraries
+    let mut files = files.split(':');
+    let exe = files.next().unwrap();
+    for file in files.map(Path::new) {
+        send(&file, &mut client);
+    }
+    t!(client.write_all(&[0]));
+
+    // Send over the client executable as the last piece
+    send(exe.as_ref(), &mut client);
+
+    println!("uploaded {:?}, waiting for result", exe);
+
+    // Ok now it's time to read all the output. We're receiving "frames"
+    // representing stdout/stderr, so we decode all that here.
+    let mut header = [0; 5];
+    let mut stderr_done = false;
+    let mut stdout_done = false;
+    let mut client = t!(client.into_inner());
+    let mut stdout = io::stdout();
+    let mut stderr = io::stderr();
+    while !stdout_done || !stderr_done {
+        t!(client.read_exact(&mut header));
+        let amt = ((header[1] as u64) << 24) |
+                  ((header[2] as u64) << 16) |
+                  ((header[3] as u64) <<  8) |
+                  ((header[4] as u64) <<  0);
+        if header[0] == 0 {
+            if amt == 0 {
+                stdout_done = true;
+            } else {
+                t!(io::copy(&mut (&mut client).take(amt), &mut stdout));
+                t!(stdout.flush());
+            }
+        } else {
+            if amt == 0 {
+                stderr_done = true;
+            } else {
+                t!(io::copy(&mut (&mut client).take(amt), &mut stderr));
+                t!(stderr.flush());
+            }
+        }
+    }
+
+    // Finally, read out the exit status
+    let mut status = [0; 5];
+    t!(client.read_exact(&mut status));
+    let code = ((status[1] as i32) << 24) |
+               ((status[2] as i32) << 16) |
+               ((status[3] as i32) <<  8) |
+               ((status[4] as i32) <<  0);
+    if status[0] == 0 {
+        std::process::exit(code);
+    } else {
+        println!("died due to signal {}", code);
+        std::process::exit(3);
+    }
+}
+
+fn send(path: &Path, dst: &mut Write) {
+    t!(dst.write_all(path.file_name().unwrap().to_str().unwrap().as_bytes()));
+    t!(dst.write_all(&[0]));
+    let mut file = t!(File::open(&path));
+    let amt = t!(file.metadata()).len();
+    t!(dst.write_all(&[
+        (amt >> 24) as u8,
+        (amt >> 16) as u8,
+        (amt >>  8) as u8,
+        (amt >>  0) as u8,
+    ]));
+    t!(io::copy(&mut file, dst));
+}