| Age | Commit message (Collapse) | Author | Lines |
|---|
|
This writes an extra `dist/image-$image.txt` which contains the S3 URL
of the cached image and the `sha256` digest of the docker entry point.
This will be uploaded with the rest of the deployed artifacts in the
Travis `after_success` script.
|
|
`src/ci/docker`, so that when files under `dist-x86_64-linux` is changed,
its dependent image `dist-i686-linux` will also be rebuilt.
However, this ultraconservative solution caused the `dist-i686-linux` to
be rebuilt every time an irrelevant Dockerfile (e.g. the PowerPC ones) is
changed, which increases the building time beyond 3 hours and forcing
a spurious but expected failure.
This commit instead parses the Dockerfile itself and look for the actual
dependencies. The scripts needs to be copied into the Docker image, which
must be done with the COPY command, so we just need to find all lines with
a COPY command and add the source file into the hash calculator.
Note: this script only handles single-lined COPY command in the form
`COPY src1 src2 src3 dst`, since these are the only variant used inside
this repository.
|
|
|
|
Before this commit the hash used to cache docker images was calculated
from the image's files and the files in the scripts/ directory. This
worked fine when all the files used by an image were in those
directories, but some images pull files from other images, causing hash
collisions in some cases.
This commit changes the hash to include the files of all the docker
images, causing a rebuild of all the images when a single one changes.
That's a bit heavy-handed, but we have no way to track which files an
image pulls in and hash collisions are really painful to deal with.
|
|
In theory we shouldn't require trusty so long as docker continues to
work!
|
|
|
|
This reverts commit 32e8bda4e35ada92beab7236d62520e6c6df8596.
|
|
Switch a `mkdir $foo` to `mkdir -p $foo` to handle the case that this script is
being run locally and has previously executed.
|
|
|
|
Tries to workaround travis-ci/travis-ci#9696.
|
|
Prevent spuriously needing to rebuild the docker image when the network
was down.
Also, adjusted the retry function to insert a sleep between retries,
because retrying immediately will often just hit the same issue.
|
|
Bots that read the log can simply look for `[CI_JOB_NAME=...]` to find out
the job's name.
|
|
Ideally I'd like to soon enable sccache for rustbuild itself and some of the
stage0 tools, but for that to work we'll need some better Rust support than the
pretty old version we were previously using!
|
|
This commit moves away from caching on Travis to our own caching on S3 for
caching docker layers between builds. Unfortunately the Travis caches have over
time had a few critical pain points:
* Caches are only updated for successful builds, meaning that if a build times
out or fails in a different location the sucessfully-created docker images
isn't always cached. While this makes sense as a general rule of caches it
hurts our use cases.
* Caches are per-branch and builder which means that we don't have a separate
cache on each release channel. All our merges go through the `auto` branch
which means that they're all sharing the same cache, even those for merging to
master/beta. This means that PRs which switch between master/beta will keep
rebuilting and having cache misses.
* Caches have historically been invaliated somewhat regularly a little more
aggressively than we'd want (I think).
* We don't always need to update the contents of the cache if the Docker image
didn't change at all, and saving off the docker layers can sometimes be quite
expensive.
For all these reasons this commit drops the usage of Travis's built-in caching
support. Instead our own caching is used by storing blobs to S3. Normally this
would be a very risky endeavour but we're basically priming a cache for a cache
(docker) so if we get this wrong the failure mode is longer builds, not stale
caches. We'll notice that pretty quickly and hopefully fix it!
The logic here is inserted directly into the `src/ci/docker/run.sh` script to
download an image based on a shasum of the `Dockerfile` and other assorted files.
This blob, if found, is loaded into docker and we record what layers were
inserted. After docker finishes the build (hopefully quickly with lots of cache
hits) we then see the sha of the final image. If it's one of the layers we
loaded then there's no need to update the cache. Otherwise we upload our layers
to the global cache, possibly overwriting what we previously just downloaded.
This is hopefully a step towards mitigating #49278 although it doesn't
completely fix it as it means we'll still probably have to retry builds that
bust the cache.
|
|
|
|
On some systems, the bash command could be available in another
directory than /bin. As such, to offer an env shebang is more
convenient.
This make sense even for docker scripts, as you can use Docker
on FreeBSD or SmartOS for example.
|
|
|
|
ci: Fix building disabled containers
* Change the context into the disabled directory. Now you can test
containers which are disabled.
|
|
* Change the context into the disabled directory. Now you can test
containers which are disabled.
|
|
Most of the other rust-lang buckets are in us-west-1 and I think the original
bucket was just accidentally created in the us-east-1 region. Let's consolidate
by moving it to the same location as the rest of our buckets.
|
|
Newer versions of Docker have a `--init` argument which spawns an init process
in the container, which we should be able to use everywhere now.
|
|
When upgrading to LLVM 5.0 it was found that the leak sanitizer tests were
failing with fatal errors, but they were passing locally when run. Turns out it
looks like they may be using new ptrace-like syscalls so the docker container
now needs `--privileged` when executing to complete the test.
|
|
|
|
|
|
* Bring back colors on Travis, which was disabled since #39036.
Append --color=always to cargo when running in CI environment.
* Removed `set -x` in the shell scripts. The `retry` function already
prints which command it is running, add `-x` just add noise to the
output.
* Support travis_fold/travis_time. Matching pairs of these allow Travis CI
to collapse the output in between. This greatly cut down the unnecessary
"successful" output one need to scroll through before finding the failed
statement.
|
|
|
|
|
|
|
|
|
|
I have a suspicion that this caused a large regression in cycle times by forcing
the compiler to perform more checks on every `debug!` statement, so let's test
this out by removing the `RUST_LOG` env var globally.
This regression in cycle time was witnessed between [two] [builds] where the
[PR] in question didn't do much suspicious. Judging by how the stage0 times
*also* regressed though then this is my best guess.
[two]: https://travis-ci.org/rust-lang/rust/builds/210149932
[builds]: https://travis-ci.org/rust-lang/rust/builds/210179995
[PR]: https://github.com/rust-lang/rust/pull/40446
|
|
rustbuild: Build documentation for `proc_macro`
This commit fixes #38749 by building documentation for the `proc_macro` crate by
default for configured hosts. Unfortunately did not turn out to be a trivial
fix. Currently rustbuild generates documentation into multiple locations: one
for std, one for test, and one for rustc. The initial fix for this issue simply
actually executed `cargo doc -p proc_macro` which was otherwise completely
elided before.
Unfortunately rustbuild was the left to merge two documentation trees together.
One for the standard library and one for the rustc tree (which only had docs for
the `proc_macro` crate). Rustdoc itself knows how to merge documentation files
(specifically around search indexes, etc) but rustbuild was unaware of this, so
an initial fix ended up destroying the sidebar and the search bar from the
libstd docs.
To solve this issue the method of documentation has been tweaked slightly in
rustbuild. The build system will not use symlinks (or directory junctions on
Windows) to generate all documentation into the same location initially. This'll
rely on rustdoc's logic to weave together all the output and ensure that it ends
up all consistent.
Closes #38749
|
|
This commit fixes #38749 by building documentation for the `proc_macro` crate by
default for configured hosts. Unfortunately did not turn out to be a trivial
fix. Currently rustbuild generates documentation into multiple locations: one
for std, one for test, and one for rustc. The initial fix for this issue simply
actually executed `cargo doc -p proc_macro` which was otherwise completely
elided before.
Unfortunately rustbuild was the left to merge two documentation trees together.
One for the standard library and one for the rustc tree (which only had docs for
the `proc_macro` crate). Rustdoc itself knows how to merge documentation files
(specifically around search indexes, etc) but rustbuild was unaware of this, so
an initial fix ended up destroying the sidebar and the search bar from the
libstd docs.
To solve this issue the method of documentation has been tweaked slightly in
rustbuild. The build system will not use symlinks (or directory junctions on
Windows) to generate all documentation into the same location initially. This'll
rely on rustdoc's logic to weave together all the output and ensure that it ends
up all consistent.
Closes #38749
|
|
I can't find anything that'd cause unexpected EOF in the source, so let's try
taking a look at the error logs on failures.
|
|
This commit attempts to move more network operations to being retryable through
various operations. For example git submodule updates, downloading snapshots,
etc, are now all in retryable steps.
Hopefully this commit can cut down on the number of network failures we've been
seeing!
|
|
This commit adds three new builders, one OSX, one Linux, and one MSVC, which
will produce "nightlies" with LLVM assertions disabled. Currently all nightly
releases have LLVM assertions enabled to catch bugs before they reach the
beta/stable channels. The beta/stable channels, however, do not have LLVM
assertions enabled.
Unfortunately though projects like Servo are stuck on nightlies for the near
future at least and are also suffering very long compile times. The purpose of
this commit is to provide artifacts to these projects which are not distributed
through normal channels (e.g. rustup) but are provided for developers to use
locally if need be.
Logistically these builds will all be uploaded to `rustc-builds-alt` instead of
the `rustc-builds` folder of the `rust-lang-ci` bucket. These builds will stay
there forever (until cleaned out if necessary) and there are no plans to
integrate this with rustup and/or the official release process.
|
|
This commit updates the compilers for many of the artifacts that we're producing
on Travis. These compilers are all compiled by crosstool-ng as they're currently
done for the images in which we're building all our cross compiled compilers.
The purpose of this commit is that when we ship binaries the artifacts won't
require a newer glibc, but rather be as compatible as possible with Linux
distributions by working with a very old version of glibc.
This commit always allocates a new matrix entry for the i686/x86_64 builder.
This builder is dedicated to just producing artifacts and eventually we'll
expand it to building other tools like Cargo and the RLS. The other builders
testing i686 and x86_64 won't use these historical toolchains.
|
|
This commit passes the `--release-channel=nightly` flag to all images which have
the `DEPLOY` flag set. This means that we'll name artifacts and the compiler
appropriately.
This reworks a bit how arguments are passed, but for now doesn't change what's
already being passed. Eventually we'll want to avoid enabling debug assertions
and llvm assertions for *all* releases, but I figure we can tackle that a little
bit more down the road.
|
|
travis: Expand dist builder coverage
This commit adds six new travis matrix entires for doing cross-compiled
distribution builds of the compiler. The support added in #38731 allows us to
quickly compile a complete suite of distribution artifacts for cross-compiled
platforms, and currently each matrix entry (when fully cached) clocks in around
an hour to finish. Note that a full test run typically takes about two hours
right now.
With further optimizations coming down the pike in #39026 this commit also
starts doubling up cross-compiled distribution builders on each matrix entry. We
initially planned to do one build per entry, but it's looking like we may be
able to get by with more than one in each entry. Depending on how long these
builds take we may even be able to up it to three, but we'll start with two
first.
This commit then completes the suite of cross-compiled compilers that we're
going to compile, adding it for a whole litany of platforms detailed in the
changes to the docker files here. The existing `cross` image is also trimmed
down quite a bit to avoid duplicate work, and we'll eventually provision it for
far more cross compilation as well.
Note that the gcc toolchains installed to compile most of these compilers are
inappropriate for actualy distribution. The glibc they pull in is much newer
than we'd like, so before we turn nightlies off we'll need to tweak these docker
files to custom build toolchains like the current `linux-cross` docker image
does.
|
|
This commit adds six new travis matrix entires for doing cross-compiled
distribution builds of the compiler. The support added in #38731 allows us to
quickly compile a complete suite of distribution artifacts for cross-compiled
platforms, and currently each matrix entry (when fully cached) clocks in around
an hour to finish. Note that a full test run typically takes about two hours
right now.
With further optimizations coming down the pike in #39026 this commit also
starts doubling up cross-compiled distribution builders on each matrix entry. We
initially planned to do one build per entry, but it's looking like we may be
able to get by with more than one in each entry. Depending on how long these
builds take we may even be able to up it to three, but we'll start with two
first.
This commit then completes the suite of cross-compiled compilers that we're
going to compile, adding it for a whole litany of platforms detailed in the
changes to the docker files here. The existing `cross` image is also trimmed
down quite a bit to avoid duplicate work, and we'll eventually provision it for
far more cross compilation as well.
Note that the gcc toolchains installed to compile most of these compilers are
inappropriate for actualy distribution. The glibc they pull in is much newer
than we'd like, so before we turn nightlies off we'll need to tweak these docker
files to custom build toolchains like the current `linux-cross` docker image
does.
|
|
|
|
This commit configures local sccache directory to get used if you're running
builds locally to enjoy the benefits of caching when running inside the
containers.
|
|
This commit adds support for sccache, a ccache-like compiler which works on MSVC
and stores results into an S3 bucket. This also switches over all Travis and
AppVeyor automation to using sccache to ensure a shared and unified cache over
time which can be shared across builders.
The support for sccache manifests as a new `--enable-sccache` option which
instructs us to configure LLVM differently to use a 'sccache' binary instead of
a 'ccache' binary. All docker images for Travis builds are updated to download
Mozilla's tooltool builds of sccache onto various containers and systems.
Additionally a new `rust-lang-ci-sccache` bucket is configured to hold all of
our ccache goodies.
|
|
This commit switches the default build system for Rust from the makefiles to
rustbuild. The rustbuild build system has been in development for almost a year
now and has become quite mature over time. This commit is an implementation of
the proposal on [internals] which slates deletion of the makefiles on
2016-01-02.
[internals]: https://internals.rust-lang.org/t/proposal-for-promoting-rustbuild-to-official-status/4368
This commit also updates various documentation in `README.md`,
`CONTRIBUTING.md`, `src/bootstrap/README.md`, and throughout the source code of
rustbuild itself.
Closes #37858
|
|
This commit configures our `.travis.yml` to test the full suite of tests we have
on Buildbot right now. A whole mess of docker images are added to the `src/ci`
directory which represent all the build environments for each configuration.
Each of these environments is then configured in `.travis.yml` to run on the
auto branch.
Note that the full matrix of tests aren't intended to be run on all PRs.
Instead, we continue to run only one entry in the matrix, forcing all others to
finish quickly. Only the `auto` branch should run the full matrix of builds.
Also note that the infrastructure hasn't quite been allocated yet to the
rust-lang/rust repository, so everything is disabled for now except for the one
build that happens on PRs. Once that infrastructure is allocated though we can
enable this and let it fly!
Notable modifications from the current test suite today:
* Android tests are run in rustbuild instead of the makefiles, for whatever
reason I couldn't get the makefiles to work on Travis.
* A debuginfo test was updated to work with the current version of the Android
NDK.
* Some dependencies in `mk/tests.mk` were fixed to allow running tests in
parallel.
|
