| Age | Commit message (Collapse) | Author | Lines |
|---|
|
Co-Authored-By: YANG Xiaojuan <yangxiaojuan@loongson.cn>
|
|
Remove `tidy` key in PR CI
This avoids confusing error messages when adding an `auto` job to CI (as recommended in the dev-guide: https://rustc-dev-guide.rust-lang.org/tests/ci.html#using-ci-to-test).
cc https://rust-lang.zulipchat.com/#narrow/stream/182449-t-compiler.2Fhelp/topic/Do.20.60needs-profiler-support.60.20tests.20not.20run.20in.20CI.3F/near/355302998
r? `@ghost`
|
|
Remove aws cli install.
All runner images have the AWS CLI 2 installed, so there isn't a really strong reason to install our own version anymore.
The version we were installing was 1.27.122. The runner images currently have 2.11.x (the exact version varies by image).
I do not have the means to really test if the new version has any issues. I looked at all the `aws` commands, and none of them seem to be doing anything unusual. The page at https://docs.aws.amazon.com/cli/latest/userguide/cliv2-migration-changes.html contains a list of all the breaking changes, and I didn't see anything that looked important.
|
|
This avoids confusing error messages when adding an `auto` job to CI (as
recommended in the dev-guide: https://rustc-dev-guide.rust-lang.org/tests/ci.html#using-ci-to-test).
|
|
Optimize builder sizes
The infra-team is continuously monitoring the efficiency of the CI system in an effort to improve overall build times and resource usage. Some builders have used much less than their allocated resources, so we are testing smaller builder sizes for them.
r? `@pietroalbini`
|
|
|
|
|
|
The infra-team is continuously monitoring the efficiency of the build
system in an effort to improve overall build times and resource usage.
The builders for some of the `x86_64-gnu` targets have used much less
resources than allocated in the past, so we are testing a smaller
builder size for them.
|
|
The infra-team is continuously monitoring the efficiency of the build
system in an effort to improve overall build times and resource usage.
The builder for the `mingw-check` target have used much less resources
than allocated in the past, so we are testing a smaller builder size for
it.
|
|
The infra-team is continuously monitoring the efficiency of the build
system in an effort to improve overall build times and resource usage.
The builders for the `i686-gnu` targets have used much less resources
than allocated in the past, so we are testing a smaller builder size for
them.
|
|
|
|
Like #107044, this will let us track compatibility with LLVM 16 going
forward, especially after we eventually upgrade our own to the next.
This also drops `tidy` here and in `x86_64-gnu-llvm-15`, syncing with
that change in #106085.
|
|
These builders aren't particularly high on overall average CPU usage and finish in typically around
30 minutes. Cutting their core counts will hopefully not significantly increase wall-time while
cutting costs, allowing us to shift some of the wins into our slower builders.
|
|
This will allow moving jobs between 8 and 16 core VMs in the next commit
more easily.
|
|
Before: `CI / PR (mingw-check, false, ubuntu-20.04-16core-64gb) (pull_request)`
After: `CI / PR - mingw-check (pull_request)`
|
|
For now this keeps all the configuration identical (AFAICT) but we'll
likely want to play with the specifics to move some of the slower
builders to larger machines and the faster builders to smaller machines,
likely reducing overall usage and improving CI times.
|
|
This lead to a miscompilation in at least `char::is_whitespace` and
probably in more unknown places.....
This reverts commit 684663ed380d0e6a6e135aed9c6055ab4ba94ac8.
|
|
|
|
This reverts commit e63ec2e1402eaff949e5c53b8f6062b152010fcc.
|
|
|
|
Previously, it would only run on changes to subtrees, submodules, or select directories.
That made it so that changes to the compiler that broke tools would only be detected on a full bors merge.
This makes it so the tools builder runs by default, making it easier to catch breaking changes to clippy (which was the most effected).
|
|
Enable Cargo's sparse protocol in CI
This enables the sparse protocol in CI in order to exercise and dogfood it. This is intended test the production server in a real-world situation.
Closes #107342
|
|
This reverts commit fcbae989ae790d5b9a0a23ceba242d0b0d4e6c5b.
|
|
|
|
Port pgo.sh to Python
This PR ports the `pgo.sh` multi stage build file from bash to Python, to make it easier to add new functionality and gather statistics. Main changes:
1) `pgo.sh` rewritten from Bash to Python. Jump from ~200 Bash LOC to ~650 Python LOC. Bash is, unsurprisingly, more concise for running scripts and binaries.
2) Better logging. Each separate stage is now clearly separated in logs, and the logs can be quickly grepped to find out which stage has completed or failed, and how long it took.
3) Better statistics. At the end of the run, there is now a table that shows the duration of the individual stages, along with a percentual ratio of the total workflow run:
```
2023-01-15T18:13:49.9896916Z stage-build INFO: Timer results
2023-01-15T18:13:49.9902185Z ---------------------------------------------------------
2023-01-15T18:13:49.9902605Z Build rustc (LLVM PGO): 1815.67s (21.47%)
2023-01-15T18:13:49.9902949Z Gather profiles (LLVM PGO): 418.73s ( 4.95%)
2023-01-15T18:13:49.9903269Z Build rustc (rustc PGO): 584.46s ( 6.91%)
2023-01-15T18:13:49.9903835Z Gather profiles (rustc PGO): 806.32s ( 9.53%)
2023-01-15T18:13:49.9904154Z Build rustc (LLVM BOLT): 1662.92s (19.66%)
2023-01-15T18:13:49.9904464Z Gather profiles (LLVM BOLT): 715.18s ( 8.46%)
2023-01-15T18:13:49.9914463Z Final build: 2454.00s (29.02%)
2023-01-15T18:13:49.9914798Z Total duration: 8457.27s
2023-01-15T18:13:49.9915305Z ---------------------------------------------------------
```
A sample run can be seen [here](https://github.com/rust-lang/rust/actions/runs/3923980164/jobs/6707932029).
I tried to keep the code compatible with Python 3.6 and don't use dependencies, which required me to reimplement some small pieces of functionality (like formatting bytes). I suppose that it shouldn't be so hard to upgrade to a newer Python or install dependencies in the CI container, but I'd like to avoid it if it won't be needed.
The code is in a single file `stage-build.py`, so it's a bit cluttered. I can also separate it into multiple files, although having it in a single file has some benefits. The code could definitely be nicer, but I'm a bit wary of introducing a lot of abstraction and similar stuff, as long as the code is stuffed into a single file.
Currently, the Python pipeline should faithfully mirror the bash pipeline one by one. After this PR, I'd like to try to optimize it, e.g. by caching the LLVM builds on S3.
r? `@Mark-Simulacrum`
|
|
|
|
|
|
|
|
|
|
Nilstrieb:no-merge-commits-for-you-only-bors-is-allowed-to-do-that, r=jyn514"
This reverts commit 4839886f0abe208ab8f2bb73a3076a59fe2ab60c, reversing
changes made to ce85c98575e3016cf2007d90a85be321e592aa96.
|
|
|
|
This duplicates mingw-check into two jobs where one job
runs `tidy` only while the other job does not. The tidy
job will not cancel other jobs on failure.
|
|
This reverts commit ddb6fe2e1d270d23ec09c2cb435c22229830e97c.
|
|
This reverts commit 3a085f769545e5f3327d29460060520d59766ba7.
|
|
Enable ThinLTO for rustc on `x86_64-apple-darwin`
Local measurements seemed to show an improvement on a couple benchmarks, so I'd like to test real CI builds, and see if the builder doesn't timeout with the expected slight increase in build times.
Let's start with x64 rustc ThinLTO, and then figure out the file structure to configure LLVM ThinLTO. Maybe we'll then try `aarch64` builds since that also looked good locally.
|
|
Enable ThinLTO for rustc on x64 msvc
This applies the great work from `@bjorn3` and `@Kobzol` in https://github.com/rust-lang/rust/pull/101403 to x64 msvc.
Here are the local results for the try build `68c5c85ed759334a11f0b0e586f5032a23f85ce4`, compared to its parent `0a6b941df354c59b546ec4c0d27f2b9b0cb1162c`. Looking better than my previous local builds.
(I can't show cycle counts, as that option is failing on the windows version of the perf collector, but I'll try to analyze and debug this soon)
This will be the first of a few tests for rustc / llvm / both ThinLTO on the windows and mac targets.
|
|
|
|
Do this because XCode 14 no longer supports a macOS deployment target
of anything before 10.13. We need 10.7+(-ish, really 10.9+) for now.
|
|
This reverts commit 8514b0097b28f84ee5cfead059a5771c6a693fb0.
|
|
|
|
|
|
Distribute bootstrap in CI
This pre-compiles bootstrap from source and adds it to the existing `rust-dev` component. There are two main goals here:
1. Make it faster to build rust from source, both the first time and incrementally
2. Make it easier to add non-python entrypoints, since they can call out to bootstrap directly rather than having to figure out the right flags to pre-compile it. This second part is still in a bit of flux, see the tracking issue below for more information.
There are also several changes to make bootstrap able to run on a machine other than the one it was built (particularly around `config.src` and `config.out` detection). I (`@jyn514)` am slightly concerned these will regress unless tested - maybe we should add an automated test that runs bootstrap in a chroot or something? Unclear whether the effort is worth the test coverage.
Helps with https://github.com/rust-lang/rust/issues/94829.
|
|
GitHub Workflows security hardening
This PR adds explicit [permissions section](https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#permissions) to workflows. This is a security best practice because by default workflows run with [extended set of permissions](https://docs.github.com/en/actions/security-guides/automatic-token-authentication#permissions-for-the-github_token) (except from `on: pull_request` [from external forks](https://securitylab.github.com/research/github-actions-preventing-pwn-requests/)). By specifying any permission explicitly all others are set to none. By using the principle of least privilege the damage a compromised workflow can do (because of an [injection](https://securitylab.github.com/research/github-actions-untrusted-input/) or compromised third party tool or action) is restricted.
It is recommended to have [most strict permissions on the top level](https://github.com/ossf/scorecard/blob/main/docs/checks.md#token-permissions) and grant write permissions on [job level](https://docs.github.com/en/actions/using-jobs/assigning-permissions-to-jobs) case by case.
|
|
This commit removes an allow-list for the dynamic linking of the LLVM
tools and instead relies on the builder's linking preference only.
|
|
- Add a new `bootstrap` component
Originally, we planned to combine this with the `rust-dev` component.
However, I realized that would force LLVM to be redownloaded whenever bootstrap is modified.
LLVM is a much larger download, so split this to get better caching.
- Build bootstrap for all tier 1 and 2 targets
|
|
|
|
|
|
Signed-off-by: sashashura <93376818+sashashura@users.noreply.github.com>
|
|
|
|
See comment added for details on the test builder restriction. This is primarily
intended for macOS CI, but is likely to be a slight win on other builders too.
|
