| Age | Commit message (Collapse) | Author | Lines |
|---|
|
refactor `fn bootstrap::builder::Builder::compiler_for` logic
- check compiler stage before forcing for stage2.
- check if download_rustc is not set before forcing for stage1.
resolves #109286
|
|
|
|
This option was added to LLVM in
https://reviews.llvm.org/D121750?id=416339. It makes `llvm_unreachable`
in builds without assertions compile to an `LLVM_BUILTIN_TRAP` instead
of `LLVM_BUILTIN_UNREACHABLE` (which causes undefined behavior and is
equivalent to `std::hint::unreachable_unchecked`).
Having compiler bugs triggering undefined behavior generally seems
undesirable and inconsistent with Rust's goals. There is a check in
`src/tools/tidy/src/style.rs` to reject code using `llvm_unreachable`.
But it is used a lot within LLVM itself.
For instance, this changes a failure I get compiling `libcore` for m68k
from a `SIGSEGV` to `SIGILL`, which seems better though it still doesn't
provide a useful message without switching to an LLVM build with asserts.
It may be best not to do this if it noticeably degrades compiler
performance, but worthwhile if it doesn't do so in any significant way. I
haven't looked into what benchmarks there are for Rustc. That should be
considered before merging.
|
|
Include executed tests in the build metrics (and use a custom test display impl)
The main goal of this PR is to include all tests executed in CI inside the build metrics JSON files. I need this for Ferrocene, and `@Mark-Simulacrum` expressed desire to have this as well to ensure all tests are executed at least once somewhere in CI.
Unfortunately implementing this required rewriting inside of bootstrap all of the code to render the test output to console. libtest supports outputting JSON instead of raw text, which we can indeed use to populate the build metrics. Doing that suppresses the console output though, and compared to rustc and Cargo the console output is not included as a JSON field.
Because of that, this PR had to reimplement both the "pretty" format (one test per line, with `rust.verbose-tests = true`), and the "terse" format (the wall of dots, with `rust.verbose-tests = false`). The current implementation should have the exact same output as libtest, except for the benchmark output. libtest's benchmark output is broken in the "terse" format, so since that's our default I slightly improved how it's rendered.
Also, to bring parity with libtest I had to introduce support for coloring output from bootstrap, using the same dependencies `annotate-snippets` uses. It's now possible to use `builder.color_for_stdout(Color::Red, "text")` and `builder.color_for_stderr(Color::Green, "text")` across all of bootstrap, automatically respecting the `--color` flag and whether the stream is a terminal or not.
I recommend reviewing the PR commit-by-commit.
r? `@Mark-Simulacrum`
|
|
|
|
Add `dist.compression-profile` option to control compression speed
PR #108534 reduced the size of compressed archives, but (as expected) it also resulted in way longer compression times and memory usage during compression.
It's desirable to keep status quo (smaller archives but more CI usage), but it should also be configurable so that downstream users don't have to waste that much time on CI. As a data point, this resulted in doubling the time of Ferrocene's dist jobs, and required us to increase the RAM allocation for one of such jobs.
This PR adds a new `config.toml` setting, `dist.compression-profile`. The values can be:
* `fast`: equivalent to the gzip and xz preset of "1"
* `balanced`: equivalent to the gzip and xz preset of "6" (the CLI defaults as far as I'm aware)
* `best`: equivalent to the gzip present of "9", and our custom xz profile
The default has also been moved back to `balanced`, to try and avoid the compression time regression for downstream users. I don't feel too strongly on the default, and I'm open to changing it.
Also, for the `best` profile the XZ settings do not match the "9" preset used by the CLI, and it might be confusing. Should we create a `custom-rustc-ci`/`ultra` profile for that?
r? ``@Mark-Simulacrum``
|
|
|
|
Distribute libntdll.a with windows-gnu toolchains
This allows the OS loader to load essential functions (e.g. read/write file) at load time instead of lazily doing so at runtime.
r? libs
|
|
migrate compiler, bootstrap and compiletest to windows-rs
This PR migrates the compiler, bootstrap, and compiletest to use [windows-rs](https://github.com/microsoft/windows-rs) instead of winapi-rs. windows-rs is the bindings crate provided by Microsoft, and is actively maintained compared to winapi-rs. Not all ecosystem crates have migrated over yet, so there will be a period of time where both crates are used.
windows-rs also provides some nice ergonomics over winapi-rs to convert return values to `Result`s (which found a case where we forgot to check the return value of `CreateFileW`).
|
|
|
|
Add tests for configure.py
I highly recommend reviewing this with whitespace disabled.
Notably, verifying that we generate valid toml relies on python 3.11 so
we can use `tomllib`, so this also switches`x86_64-gnu-llvm-14` (one of the PR builders) to use 3.11.
While fixing that, I noticed that we stopped testing python2.7 support on PR CI in https://github.com/rust-lang/rust/pull/106085. `@fee1-dead` `@pietroalbini` please be more careful in the future, there is no CI for CI itself that verifies we are testing everything we should be.
- Separate out functions so that each unit test doesn't create a file on disk
- Add a few unit tests
|
|
Set `CMAKE_SYSTEM_NAME` for Linux targets
When bootstrap compiles native dependencies like LLVM, it should set `CMAKE_SYSTEM_NAME` for the target system; otherwise cmake may not identify that it is cross-compiling.
In particular, when building a Linux rustc on a macOS host, cmake was including `-isysroot /path/to/macOS.sdk` options that caused things to break. By setting `CMAKE_SYSTEM_NAME=Linux` when building for Linux targets, the macOS SDK is no longer passed as sysroot to the compiler.
r? bootstrap
|
|
|
|
Upgrade to LLVM 16
This updates Rust to LLVM 16. It also updates our host compiler for dist-x86_64-linux to LLVM 16. The reason for that is that Bolt from LLVM 15 is not capable of compiling LLVM 16 (https://github.com/llvm/llvm-project/issues/61114).
LLVM 16.0.0 has been [released](https://discourse.llvm.org/t/llvm-16-0-0-release/69326) on March 18, while Rust 1.70 will become stable on June 1.
Tested images: `dist-x86_64-linux`, `dist-riscv64-linux` (alt), `dist-x86_64-illumos`, `dist-various-1`, `dist-various-2`, `dist-powerpc-linux`, `wasm32`, `armhf-gnu`
Tested images until the usual IPv6 failures: `test-various`
|
|
- check compiler stage before forcing for stage2.
- check if download_rustc is not set before forcing for stage1.
Signed-off-by: ozkanonur <work@onurozkan.dev>
|
|
Speed up tidy quite a lot
I highly recommend reviewing this commit-by-commit. Based on #106440 for convenience.
## Timings
These were collected by running `x test tidy -v` to copy paste the command, then using [`samply record`](https://github.com/mstange/samply).
before (8 threads)
after (8 threads) 
before (64 threads) 
after (64 threads) 
The last commit makes tidy use more threads, so comparing "before (8 threads)" to "after (64 threads)" is IMO the most realistic comparison. Locally, that brings the time for me to run tidy down from 4 to .9 seconds, i.e. the majority of the time for `x test tidy` is now spend running `fmt --check`.
r? `@the8472`
|
|
This has a significant speedup for me locally, from about 1.3 seconds to
.9 seconds.
|
|
Check for llvm-tools before install
Fixes #108948
````@jpalus```` Please review
|
|
This also fixes a regression from
https://github.com/rust-lang/rust/pull/106085 which stopped testing that
we support python2 in PR CI.
|
|
- Separate out functions so that each unit test doesn't create a file on disk
- Add a few unit tests
Notably, verifying that we generate valid toml relies on python 3.11 so
we can use `tomllib`.
|
|
|
|
|
|
Prevent stable `libtest` from supporting `-Zunstable-options`
Took a while for me to get around to this but seems trivial (unless I'm missing some reason this will break all our tests). Fixes #75526
Basically `libtest` already tries to handle this in https://github.com/rust-lang/rust/blob/501ad021b9a4fb2cd6a39e0302d22f169f6166b0/library/test/src/cli.rs#L310-L318
But that env var was not passed. I'm guessing at one point [this code](https://github.com/rust-lang/rust/blob/501ad021b9a4fb2cd6a39e0302d22f169f6166b0/src/bootstrap/compile.rs#L842-L844) (or a common ancestor) was used to compile the standard library/libtest, but that is no longer the case (or perhaps it never worked, I don't have time to go digging).
I don't love that this is a "allow unstable by default" situation, as it means things like [`rustc-build-sysroot`](https://github.com/RalfJung/rustc-build-sysroot) could accidentally get unstable (CC ````@RalfJung)```` even if this is fixed here, but it's consistent with what happens in `rustc_feature`, so... yeah.
This is user-facing after all, even if it's hard to imagine the outcome of that conversation being "lets continue allowing use of `-Zunstable-features` from stable rust" (especially since a `RUSTC_BOOTSTRAP=1`-shaped loophole remains)... I think it probably should get a vibe check in the t-libs meeting (and plausibly a relnote along the lines of "hey `cargo test -- -Zunstable-options --some --unstable --stuff=here` used to work on stable, that's been fixed, sorry").
I'll nominate it for that after CI comes up green (I've done a smoke check but don't know what (if anything) will need `bootstrap` to enable `RUSTC_BOOTSTRAP=1` when running tests)
r? ````@jyn514````
|
|
When bootstrap compiles native dependencies like LLVM, it should set
CMAKE_SYSTEM_NAME for the target system; otherwise cmake may not
identify that it is cross-compiling.
In particular, when building a Linux rustc on a macOS host, cmake was
including `-isysroot /path/to/macOS.sdk` options that caused things to
break. By setting `CMAKE_SYSTEM_NAME=Linux` when building for Linux
targets, the macOS SDK is no longer passed as sysroot to the compiler.
|
|
Create dirs for build_triple
Fixes #109103
|
|
Signed-off-by: ozkanonur <work@onurozkan.dev>
|
|
|
|
|
|
create `config::tests::detect_src_and_out` test for bootstrap
Resolves one of the `FIXME` in bootstrap
|
|
Don't use fd-lock on Solaris in bootstrap
...as Solaris is missing flock()
fixes #103630
|
|
...as Solaris is missing flock()
fixes #103630
|
|
Signed-off-by: ozkanonur <work@onurozkan.dev>
|
|
Rename `config.toml.example` to `config.example.toml`
This had bothered me for a while as it leads to bad (missing) syntax highlighting in most editors I've used, and `@jyn514` suggested I just make the change and that the compatibility concerns I had don't really matter.
I suspect it will be a contentious one, so will not be offended if the outcome of this is to close the PR.
|
|
|
|
bootstrap: document tidy
Enable documentation of tidy, as suggested in #106803. Jyn mentioned they should probably be added to `doc.rust-lang.org`, how should that be done?
|
|
|
|
|
|
only call git on git checkouts during bootstrap
fixes https://github.com/rust-lang/rust/issues/108959
|
|
|
|
x fmt: Don't print all modified files if there's more than 10
This avoids spam for dozens of modified files.
Fixes https://github.com/rust-lang/rust/issues/106920.
|
|
This avoids spam for dozens of modified files.
|
|
Set `LIBC_CHECK_CFG=1` when building Rust code in bootstrap
Downstream forks of the Rust compiler might want to use a custom `libc` to add support for targets that are not yet available upstream. Adding a patch to replace `libc` with a custom one would cause compilation errors though, because Cargo would interpret the custom `libc` as part of the workspace, and apply the check-cfg lints on it.
Since https://github.com/rust-lang/libc/pull/3037, the `libc` build script emits check-cfg flags only when the `LIBC_CHECK_CFG` environment variable is set, so this PR allows the use of custom `libc`s.
|
|
|
|
Import rust-installer & adjust compression settings
This brings in rust-lang/rust-installer#123, which enables a larger compression window (8 -> 64MB) amongst other changes to the xz compression settings. The net effect should be smaller compressed tarballs which will decrease bandwidth usage for
static.rust-lang.org, download times, and decompression time.
This comes at the cost of higher baseline requirements for running rustup to use these files, which we believe should be largely acceptable (running rustc is likely to use at least this much memory) but if we get specific reports we may explore options to decrease impact (e.g., using the gzip tarballs automatically in rustup).
To simplify iteration on compression settings this also imports the rust-lang/rust-installer submodule, it is now hosted fully inside rust-lang/rust. Once we land this I'll file a followup to add a note to that repo and we can subsequently archive it.
--
CI times for dist-x86_64-linux builds:
* threads=6, master - 2h 50m
* threads=1, new - 3h 40m
* threads=6, new - 2h 50m
|
|
Force to use the stage 2 compiler when config.download_rustc is set
Fixes #108768
|
|
Add a new config flag, dist.include-mingw-linker.
The flag controls whether to copy the linker, DLLs, and various libraries from MinGW into the rustc toolchain.
It applies only when the host or target is pc-windows-gnu.
The flag is true by default to preserve existing behavior.
|
|
config.download_rustc is set
|
|
Remove the option to disable `llvm-version-check`
We don't support old versions of LLVM; there's no reason to have an easy way to force bootstrap to use them anyway. If someone really needs to use an unsupported version, they can modify bootstrap to change the version range.
r? ``@cuviper`` on whether we want to do this or not, since you maintain rust on Fedora and touched this config last.
|
|
|
|
|
