| Age | Commit message (Collapse) | Author | Lines |
|---|
|
|
|
As we continue to add more crates to the compiler and use them to implement
various features we want to be sure we're not accidentally expanding the API
surface area of the compiler! To that end this commit adds a new `run-make` test
which will attempt to `extern crate foo` all crates in the sysroot, verifying
that they're all unstable.
This commit discovered that the `std_shim` and `test_shim` crates were
accidentally stable and fixes the situation by deleting those shims. The shims
are no longer necessary due to changes in Cargo that have happened since they
were originally incepted.
|
|
|
|
LeakSanitizer, ThreadSanitizer, AddressSanitizer and MemorySanitizer support
```
$ cargo new --bin leak && cd $_
$ edit Cargo.toml && tail -n3 $_
```
``` toml
[profile.dev]
opt-level = 1
```
```
$ edit src/main.rs && cat $_
```
``` rust
use std::mem;
fn main() {
let xs = vec![0, 1, 2, 3];
mem::forget(xs);
}
```
```
$ RUSTFLAGS="-Z sanitizer=leak" cargo run --target x86_64-unknown-linux-gnu; echo $?
Finished dev [optimized + debuginfo] target(s) in 0.0 secs
Running `target/debug/leak`
=================================================================
==10848==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 16 byte(s) in 1 object(s) allocated from:
#0 0x557c3488db1f in __interceptor_malloc /shared/rust/checkouts/lsan/src/compiler-rt/lib/lsan/lsan_interceptors.cc:55
#1 0x557c34888aaa in alloc::heap::exchange_malloc::h68f3f8b376a0da42 /shared/rust/checkouts/lsan/src/liballoc/heap.rs:138
#2 0x557c34888afc in leak::main::hc56ab767de6d653a $PWD/src/main.rs:4
#3 0x557c348c0806 in __rust_maybe_catch_panic ($PWD/target/debug/leak+0x3d806)
SUMMARY: LeakSanitizer: 16 byte(s) leaked in 1 allocation(s).
23
```
```
$ cargo new --bin racy && cd $_
$ edit src/main.rs && cat $_
```
``` rust
use std::thread;
static mut ANSWER: i32 = 0;
fn main() {
let t1 = thread::spawn(|| unsafe { ANSWER = 42 });
unsafe {
ANSWER = 24;
}
t1.join().ok();
}
```
```
$ RUSTFLAGS="-Z sanitizer=thread" cargo run --target x86_64-unknown-linux-gnu; echo $?
==================
WARNING: ThreadSanitizer: data race (pid=12019)
Write of size 4 at 0x562105989bb4 by thread T1:
#0 racy::main::_$u7b$$u7b$closure$u7d$$u7d$::hbe13ea9e8ac73f7e $PWD/src/main.rs:6 (racy+0x000000010e3f)
#1 _$LT$std..panic..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..FnOnce$LT$$LP$$RP$$GT$$GT$::call_once::h2e466a92accacc78 /shared/rust/checkouts/lsan/src/libstd/panic.rs:296 (racy+0x000000010cc5)
#2 std::panicking::try::do_call::h7f4d2b38069e4042 /shared/rust/checkouts/lsan/src/libstd/panicking.rs:460 (racy+0x00000000c8f2)
#3 __rust_maybe_catch_panic <null> (racy+0x0000000b4e56)
#4 std::panic::catch_unwind::h31ca45621ad66d5a /shared/rust/checkouts/lsan/src/libstd/panic.rs:361 (racy+0x00000000b517)
#5 std::thread::Builder::spawn::_$u7b$$u7b$closure$u7d$$u7d$::hccfc37175dea0b01 /shared/rust/checkouts/lsan/src/libstd/thread/mod.rs:357 (racy+0x00000000c226)
#6 _$LT$F$u20$as$u20$alloc..boxed..FnBox$LT$A$GT$$GT$::call_box::hd880bbf91561e033 /shared/rust/checkouts/lsan/src/liballoc/boxed.rs:605 (racy+0x00000000f27e)
#7 std::sys::imp::thread::Thread::new::thread_start::hebdfc4b3d17afc85 <null> (racy+0x0000000abd40)
Previous write of size 4 at 0x562105989bb4 by main thread:
#0 racy::main::h23e6e5ca46d085c3 $PWD/src/main.rs:8 (racy+0x000000010d7c)
#1 __rust_maybe_catch_panic <null> (racy+0x0000000b4e56)
#2 __libc_start_main <null> (libc.so.6+0x000000020290)
Location is global 'racy::ANSWER::h543d2b139f819b19' of size 4 at 0x562105989bb4 (racy+0x0000002f8bb4)
Thread T1 (tid=12028, running) created by main thread at:
#0 pthread_create /shared/rust/checkouts/lsan/src/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:902 (racy+0x00000001aedb)
#1 std::sys::imp::thread::Thread::new::hce44187bf4a36222 <null> (racy+0x0000000ab9ae)
#2 std::thread::spawn::he382608373eb667e /shared/rust/checkouts/lsan/src/libstd/thread/mod.rs:412 (racy+0x00000000b5aa)
#3 racy::main::h23e6e5ca46d085c3 $PWD/src/main.rs:6 (racy+0x000000010d5c)
#4 __rust_maybe_catch_panic <null> (racy+0x0000000b4e56)
#5 __libc_start_main <null> (libc.so.6+0x000000020290)
SUMMARY: ThreadSanitizer: data race $PWD/src/main.rs:6 in racy::main::_$u7b$$u7b$closure$u7d$$u7d$::hbe13ea9e8ac73f7e
==================
ThreadSanitizer: reported 1 warnings
66
```
```
$ cargo new --bin oob && cd $_
$ edit src/main.rs && cat $_
```
``` rust
fn main() {
let xs = [0, 1, 2, 3];
let y = unsafe { *xs.as_ptr().offset(4) };
}
```
```
$ RUSTFLAGS="-Z sanitizer=address" cargo run --target x86_64-unknown-linux-gnu; echo $?
=================================================================
==13328==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7fff29f3ecd0 at pc 0x55802dc6bf7e bp 0x7fff29f3ec90 sp 0x7fff29f3ec88
READ of size 4 at 0x7fff29f3ecd0 thread T0
#0 0x55802dc6bf7d in oob::main::h0adc7b67e5feb2e7 $PWD/src/main.rs:3
#1 0x55802dd60426 in __rust_maybe_catch_panic ($PWD/target/debug/oob+0xfe426)
#2 0x55802dd58dd9 in std::rt::lang_start::hb2951fc8a59d62a7 ($PWD/target/debug/oob+0xf6dd9)
#3 0x55802dc6c002 in main ($PWD/target/debug/oob+0xa002)
#4 0x7fad8c3b3290 in __libc_start_main (/usr/lib/libc.so.6+0x20290)
#5 0x55802dc6b719 in _start ($PWD/target/debug/oob+0x9719)
Address 0x7fff29f3ecd0 is located in stack of thread T0 at offset 48 in frame
#0 0x55802dc6bd5f in oob::main::h0adc7b67e5feb2e7 $PWD/src/main.rs:1
This frame has 1 object(s):
[32, 48) 'xs' <== Memory access at offset 48 overflows this variable
HINT: this may be a false positive if your program uses some custom stack unwind mechanism or swapcontext
(longjmp and C++ exceptions *are* supported)
SUMMARY: AddressSanitizer: stack-buffer-overflow $PWD/src/main.rs:3 in oob::main::h0adc7b67e5feb2e7
Shadow bytes around the buggy address:
0x1000653dfd40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x1000653dfd50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x1000653dfd60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x1000653dfd70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x1000653dfd80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x1000653dfd90: 00 00 00 00 f1 f1 f1 f1 00 00[f3]f3 00 00 00 00
0x1000653dfda0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x1000653dfdb0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x1000653dfdc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x1000653dfdd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x1000653dfde0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Heap right redzone: fb
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack partial redzone: f4
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
==13328==ABORTING
1
```
```
$ cargo new --bin uninit && cd $_
$ edit src/main.rs && cat $_
```
``` rust
use std::mem;
fn main() {
let xs: [u8; 4] = unsafe { mem::uninitialized() };
let y = xs[0] + xs[1];
}
```
```
$ RUSTFLAGS="-Z sanitizer=memory" cargo run; echo $?
==30198==WARNING: MemorySanitizer: use-of-uninitialized-value
#0 0x563f4b6867da in uninit::main::hc2731cd4f2ed48f8 $PWD/src/main.rs:5
#1 0x563f4b7033b6 in __rust_maybe_catch_panic ($PWD/target/debug/uninit+0x873b6)
#2 0x563f4b6fbd69 in std::rt::lang_start::hb2951fc8a59d62a7 ($PWD/target/debug/uninit+0x7fd69)
#3 0x563f4b6868a9 in main ($PWD/target/debug/uninit+0xa8a9)
#4 0x7fe844354290 in __libc_start_main (/usr/lib/libc.so.6+0x20290)
#5 0x563f4b6864f9 in _start ($PWD/target/debug/uninit+0xa4f9)
SUMMARY: MemorySanitizer: use-of-uninitialized-value $PWD/src/main.rs:5 in uninit::main::hc2731cd4f2ed48f8
Exiting
77
```
|
|
|
|
|
|
Add support for test suites emulated in QEMU
This commit adds support to the build system to execute test suites that cannot
run natively but can instead run inside of a QEMU emulator. A proof-of-concept
builder was added for the `arm-unknown-linux-gnueabihf` target to show off how
this might work.
In general the architecture is to have a server running inside of the emulator
which a local client connects to. The protocol between the server/client
supports compiling tests on the host and running them on the target inside the
emulator.
Closes #33114
|
|
|
|
This commit adds support to the build system to execute test suites that cannot
run natively but can instead run inside of a QEMU emulator. A proof-of-concept
builder was added for the `arm-unknown-linux-gnueabihf` target to show off how
this might work.
In general the architecture is to have a server running inside of the emulator
which a local client connects to. The protocol between the server/client
supports compiling tests on the host and running them on the target inside the
emulator.
Closes #33114
|
|
I've been noticing some spurious recompiles of the final stage on Travis lately
and in debugging them I found a case where we were a little to eager to update
a stamp file due to the build_helper library being introduced during the testing
phase.
Part of the rustbuild system detects when libstd is recompiled and automatically
cleans out future directories to ensure that dirtyness propagation works. To do
this rustbuild doesn't know the artifact name of the standard library so it just
probes everything in the target directory, looking to see if anything changed.
The problem here happened where:
* First, rustbuild would compile everything (a normal build)
* Next, rustbuild would run all tests
* During testing, the libbuild_helper library was introduced into the target
directory, making it look like a change happened because a file is newer
than the newest was before
* Detecting a change, the next compilation would then cause rustbuild to clean
out old artifacts and recompile everything again.
This commit fixes this problem by correcting rustbuild to just not test the
build_helper crate at all. This crate doesn't have any unit tests, nor is it
intended to. That way the target directories should stay the same throughout
testing after a previous build.
|
|
This commit starts adding the infrastructure for uploading release artifacts
from AppVeyor/Travis on each commit. The idea is that eventually we'll upload a
full release to AppVeyor/Travis in accordance with plans [outlined earlier].
Right now this configures Travis/Appveyor to upload all tarballs in the `dist`
directory, and various images are updated to actually produce tarballs in these
directories. These are nowhere near ready to be actual release artifacts, but
this should allow us to play around with it and test it out. Once this commit
lands we should start seeing artifacts uploaded on each commit.
[outlined earlier]: https://internals.rust-lang.org/t/rust-ci-release-infrastructure-changes/4489
|
|
This commit optimizes the compile time for creating tarballs of cross-host
compilers and as a proof of concept adds two to the standard Travis matrix. Much
of this commit is further refactoring and refining of the `step.rs` definitions
along with the interpretation of `--target` and `--host` flags. This has gotten
confusing enough that I've also added a small test suite to
`src/bootstrap/step.rs` to ensure what we're doing works and doesn't regress.
After this commit when you execute:
./x.py dist --host $MY_HOST --target $MY_HOST
the build system will compile two compilers. The first is for the build platform
and the second is for the host platform. This second compiler is then packaged
up and placed into `build/dist` and is ready to go. With a fully cached LLVM and
docker image I was able to create a cross-host compiler in around 20 minutes
locally.
Eventually we plan to add a whole litany of cross-host entries to the Travis
matrix, but for now we're just adding a few before we eat up all the extra
capacity.
cc #38531
|
|
Gate on distcheck on Travis
This commit adds a new entry to the Travis matrix to gate on distcheck, the illustrious test process that has historically taken *8 hours* to complete and also breaks all the time on nightly. By adding it to Travis we should hope to never see nightly breakage (like https://github.com/rust-lang/rust/issues/38690) because of this ever again!
"But wait, surely we can't wait 8 hours for all PRs!" you might be thinking, and you are indeed correct. The distcheck added here is much more optimized for speed than the old buildbot instances for a number of reasons:
* We're not building *two host compilers* beforehand. The current distcheck bot does a cross for i686 Linux and x86_64 Linux before it actually runs distcheck, building 6 compilers and LLVM twice. None of this is done in parallel as well (e.g. `-j1`). Not doing any of this work will be a huge win!
* We're using sccache to compile LLVM, so it should be much faster. Distcheck on the bots didn't cache LLVM well and rebuilt it every time.
All in all, this version of "distcheck" should be exactly like other matrix entries that run tests except that it's a *little* slower to start as it has to create the source tarball then rebuild the build system in the distcheck dir. Overall this should be well under the 2 hours that Android is currently taking anyway.
Closes https://github.com/rust-lang/rust/issues/38691
|
|
rustbuild: Compile all support tools in stage0
This commit changes all tools and such to get compiled in stage0, not in
later stages. The purpose of this commit is to cut down dependencies on later
stages for future modifications to the build system. Notably we're going to be
adding builders that produce a full suite of cross-compiled artifacts for a
particular host, and that shouldn't compile the `x86_64-unknown-linux-gnu`
compiler more than once. Currently dependencies on, for example, the error index
end up compiling the `x86_64-unknown-linux-gnu` compiler more than necessary.
As a result here we move many dependencies on these tools to being produced by a
stage0 compiler, not a stage1+ compiler. None of these tools actually need to be
staged at all, so they'll exhibit consistent behavior across the stages.
|
|
The android-copy-libs step is crucial for running tests on the Android target as
it copies necessary scripts and such to the emulator. We must run that before
running any tests there, but we erroneously only did it for compiletest test
suites!
|
|
This commit adds a new entry to the Travis matrix which performs a "distcheck",
which basically means that we create a tarball, extract that tarball, and then
build/test inside there. This ensures that the tarballs we produce are actually
able to be built/tested!
Along the way this also updates the rustbuild distcheck definition to propagate
the configure args from the top-level invocation.
Closes #38691
|
|
This commit changes all tools and such to get compiled in stage0, not in
later stages. The purpose of this commit is to cut down dependencies on later
stages for future modifications to the build system. Notably we're going to be
adding builders that produce a full suite of cross-compiled artifacts for a
particular host, and that shouldn't compile the `x86_64-unknown-linux-gnu`
compiler more than once. Currently dependencies on, for example, the error index
end up compiling the `x86_64-unknown-linux-gnu` compiler more than necessary.
As a result here we move many dependencies on these tools to being produced by a
stage0 compiler, not a stage1+ compiler. None of these tools actually need to be
staged at all, so they'll exhibit consistent behavior across the stages.
|
|
This commit switches the rustbuild build system to compiling the
compiler twice for a normal bootstrap rather than the historical three
times.
Rust is a bootstrapped language which means that a previous version of
the compiler is used to build the next version of the compiler. Over
time, however, we change many parts of compiler artifacts such as the
metadata format, symbol names, etc. These changes make artifacts from
one compiler incompatible from another compiler. Consequently if a
compiler wants to be able to use some artifacts then it itself must have
compiled the artifacts.
Historically the rustc build system has achieved this by compiling the
compiler three times:
* An older compiler (stage0) is downloaded to kick off the chain.
* This compiler now compiles a new compiler (stage1)
* The stage1 compiler then compiles another compiler (stage2)
* Finally, the stage2 compiler needs libraries to link against, so it
compiles all the libraries again.
This entire process amounts in compiling the compiler three times.
Additionally, this process always guarantees that the Rust source tree
can compile itself because the stage2 compiler (created by a freshly
created compiler) would successfully compile itself again. This
property, ensuring Rust can compile itself, is quite important!
In general, though, this third compilation is not required for general
purpose development on the compiler. The third compiler (stage2) can
reuse the libraries that were created during the second compile. In
other words, the second compilation can produce both a compiler and the
libraries that compiler will use. These artifacts *must* be compatible
due to the way plugins work today anyway, and they were created by the
same source code so they *should* be compatible as well.
So given all that, this commit switches the default build process to
only compile the compiler three times, avoiding this third compilation
by copying artifacts from the previous one. Along the way a new entry in
the Travis matrix was also added to ensure that our full bootstrap can
succeed. This entry does not run tests, though, as it should not be
necessary.
To restore the old behavior of a full bootstrap (three compiles) you can
either pass:
./configure --enable-full-bootstrap
or if you're using config.toml:
[build]
full-bootstrap = true
Overall this will hopefully be an easy 33% win in build times of the
compiler. If we do 33% less work we should be 33% faster! This in turn
should affect cycle times and such on Travis and AppVeyor positively as
well as making it easier to work on the compiler itself.
|
|
adaptation to rustbuild for openbsd
Since the switch to rustbuild, the build for openbsd is broken:
- [X] `ar` inference based on compiler name is wrong (OpenBSD usually use `egcc`, but `ear` doesn't exist)
- [X] `make` isn't GNU-make under OpenBSD (and others BSD platforms)
- [x] `stdc++` isn't the right stdc++ library to link with (it should be `estdc++`)
- [x] corrects tests that don't pass anymore (problems related to rustbuild)
r? @alexcrichton
|
|
For example:
- we now support `-vv` to get very verbose output.
- RUSTFLAGS is respected by `x.py`
- better error messages for some cases
|
|
the diff extends build_helper to provide an function to return the
expected name of GNU-make on the host: "make" or "gmake".
Fixes #38429
|
|
This ensures that stable releases produced by rustbuild will succeed in testing
as some of the rustdoc tests use unstable features.
|
|
rustbuild: Print out failing commands
Just ensure that we always print out the command line which should aid in
debugging.
Closes #38228
|
|
This commit implements the `distcheck` target for rustbuild which is only ever
run on our nightly bots. This essentially just creates a tarball, un-tars it,
and then runs a full build, validating that the release tarballs do indeed have
everything they need to build Rust.
|
|
Just ensure that we always print out the command line which should aid in
debugging.
Closes #38228
|
|
This commit switches the default build system for Rust from the makefiles to
rustbuild. The rustbuild build system has been in development for almost a year
now and has become quite mature over time. This commit is an implementation of
the proposal on [internals] which slates deletion of the makefiles on
2016-01-02.
[internals]: https://internals.rust-lang.org/t/proposal-for-promoting-rustbuild-to-official-status/4368
This commit also updates various documentation in `README.md`,
`CONTRIBUTING.md`, `src/bootstrap/README.md`, and throughout the source code of
rustbuild itself.
Closes #37858
|
|
Now that we've got a beta build, let's use it!
|
|
Add command `./x.py bench`; use `./x.py bench --help -v` to list all
available benchmark targets.
|
|
Add a configuration key to `config.toml`, read it from `./configure`, and add
auto-detection if none of those were specified.
Closes #35760
|
|
If it ran too soon there wasn't a `test` directory lying around but we'll need
one!
|
|
|
|
This commit is a rewrite of the user-facing interface to the rustbuild build
system. The intention here is to make it much easier to compile/test the project
without having to remember weird rule names and such. An overall view of the new
interface is:
# build everything
./x.py build
# document everyting
./x.py doc
# test everything
./x.py test
# test libstd
./x.py test src/libstd
# build libcore stage0
./x.py build src/libcore --stage 0
# run stage1 run-pass tests
./x.py test src/test/run-pass --stage 1
The `src/bootstrap/bootstrap.py` script is now aliased as a top-level `x.py`
script. This `x` was chosen to be both short and easily tab-completable (no
collisions in that namespace!). The build system now accepts a "subcommand" of
what to do next, the main ones being build/doc/test.
Each subcommand then receives an optional list of arguments. These arguments are
paths in the source repo of what to work with. That is, if you want to test a
directory, you just pass that directory as an argument.
The purpose of this rewrite is to do away with all of the arcane renames like
"rpass" is the "run-pass" suite, "cfail" is the "compile-fail" suite, etc. By
simply working with directories and files it's much more intuitive of how to run
a test (just pass it as an argument).
The rustbuild step/dependency management was also rewritten along the way to
make this easy to work with and define, but that's largely just a refactoring of
what was there before.
The *intention* is that this support is extended for arbitrary files (e.g.
`src/test/run-pass/my-test-case.rs`), but that isn't quite implemented just yet.
Instead directories work for now but we can follow up with stricter path
filtering logic to plumb through all the arguments.
|
|
|
|
Fixes https://github.com/rust-lang/rust/issues/36788.
|
|
This will make it easier for packagers to bootstrap rustc when they happen
to have a bootstrap compiler with a slightly different version number.
It's not ok for anything other than the build system to set this environment variable.
|
|
This updates the commit to use workspaces to use `cargo metadata` instead of
hardcoded lists about what to test. This should help us be resilient to updates
in the future on behalf of the crate DAG and minimize the amount of files that
need to be touched.
|
|
This involves hacking the code used to run cargo test on various
packages, because it reads Cargo.lock to determine which packages should
be tested. This change implements a blacklist, since that will catch new
crates when they are added in the future.
|
|
|
|
|
|
|
|
|
|
|
|
We've got tests which require a particular version of LLVM to run as they're
testing bug fixes. Our build system, however, supports multiple LLVM versions,
so we can't run these tests on all LLVM versions.
This adds a new `min-llvm-version` directive for tests so they can opt out of
being run on older versions of LLVM. This then namely applies that logic to the
`issue-36023.rs` test case and...
Closes #36138
|
|
The organization in rustbuild was a little odd at the moment where the `lib.rs`
was quite small but the binary `main.rs` was much larger. Unfortunately as well
there was a `build/` directory with the implementation of the build system, but
this directory was ignored by GitHub on the file-search prompt which was a
little annoying.
This commit reorganizes rustbuild slightly where all the library files (the
build system) is located directly inside of `src/bootstrap` and all the binaries
now live in `src/bootstrap/bin` (they're small). Hopefully this should allow
GitHub to index and allow navigating all the files while maintaining a
relatively similar layout to the other libraries in `src/`.
|
