| Age | Commit message (Collapse) | Author | Lines |
|---|
|
|
|
ci: Increase `rust.debuginfo-level-tests` to `2` in `x86_64-gnu-debug` job
Simply to increase the scope of the testing.
Part of https://github.com/rust-lang/rust/issues/61117.
cc rust-lang/rust#145967 and rust-lang/rust#146025 which prepared for this. And rust-lang/rust#144499 that set to level to `1`
try-job: x86_64-gnu-debug
|
|
Simply to increase the scope of the testing.
Force debuginfo=0 for a handful of tests so that we can have CI prevent
regressing on more tests.
|
|
v6.17-rc5 contains the equivalent of the two commits we had here, thus
move the Rust for Linux job forward to that so that we don't need the
temporary commits anymore.
Signed-off-by: Miguel Ojeda <ojeda@kernel.org>
|
|
Split `run-make` into two {`run-make`,`run-make-cargo`} test suites
## Summary
Split `tests/run-make` into two test suites, to make it faster and more convenient for contributors to run run-make tests that do not need in-tree `cargo`.
| New test suites | Explanation |
| ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `tests/run-make` | The "fast path" test suite intended for run-make tests that do not need in-tree `cargo`. These tests may not use `cargo`. |
| `tests/run-make-cargo` | The "slow path" test suite that requires checking out `cargo` submodule and building in-tree `cargo`, and thus will have access to in-tree `cargo`. In practice, these constitute a very small portion of the original `run-make` tests. |
This PR carries out [MCP 847: Split run-make test suite into slower-building test suite with suitably-staged cargo and faster-building test suite without cargo](https://github.com/rust-lang/compiler-team/issues/847).
Fixes rust-lang/rust#135573 (for the tests that do not need in-tree `cargo`).
Fixes rust-lang/rust#134109.
## Remarks
- I considered if we want to split by in-tree tools previously. However, as discussed rust-lang/rust#134109, in practice `rustdoc` is not very slow to build, but `cargo` takes a good few minutes. So, the partition boundary was determined to be along in-tree `cargo` availability.
- The `run-make` tests previously that wanted to use `cargo` cannot just use the bootstrap `cargo`, otherwise they would run into situations where bootstrap `cargo` can significantly diverge from in-tree `cargo` (see https://github.com/rust-lang/rust/pull/130642).
---
try-job: aarch64-msvc-1
try-job: test-various
try-job: x86_64-gnu-debug
try-job: aarch64-gnu-debug
try-job: aarch64-apple
try-job: dist-various-1
|
|
To catch obvious build failures of the `src/tools/bump-stage0` early,
before we find out it can't even build when we really need it to work.
|
|
For the ones that explicitly picks which test suite to run.
|
|
This upgrades the Rust CI from v6.16-rc1 plus a temporary commit for
the >= 1.91 target spec [1] to v6.17-rc3 with two commits pending to
be merged upstream -- one for the same target spec format change [1]
and another for the `file_as_c_str` change [2].
Link: https://github.com/rust-lang/rust/pull/144443 [1]
Link: https://github.com/rust-lang/rust/pull/145928 [2]
Signed-off-by: Miguel Ojeda <ojeda@kernel.org>
|
|
Enforce in bootstrap that test must have stage at least 1 (except for compiletest)
This PR cleans up a bunch of test steps and adds metadata to them. I didn't yet touch the most complicated step (`CompileTest`), I'm leaving that for another PR.
Testing anything on stage 0 is only possible for compiletest and with `build.allow-compiletest-stage0`. Testing anything else on stage 0 will either produce a nice error or crash with a stage being subtracted below zero.
r? `@jieyouxu`
try-job: dist-x86_64-linux
try-job: aarch64-gnu
try-job: arm-android
try-job: `x86_64-gnu-llvm-20*`
try-job: `x86_64-msvc-*`
try-job: aarch64-apple
try-job: test-various
|
|
|
|
To unblock the Rust CI in PR [1], use a temporary commit from Rust for
Linux that supports the future target spec format.
Link: https://github.com/rust-lang/rust/pull/144443 [1]
Signed-off-by: Miguel Ojeda <ojeda@kernel.org>
|
|
Improve dist for gnullvm hosts
LLVM tools cross-compilation has been fixed by rust-lang/rust#145763 and LLVM downloading from CI no longer causes build error, so let's enable them both.
|
|
ci: Begin running ui tests with `rust.debuginfo-level-tests=1`
To reduce risk of regressing on generating debuginfo e.g. in the form of ICE:s. This will also ensure that future ui tests work with different debuginfo levels. See https://github.com/rust-lang/rust/issues/61117.
When I looked at run time for different CI jobs, **x86_64-gnu-debug** was far from the bottleneck, so it should be fine to make it perform more work.
A handful of tests are failing so we need to force debuginfo=0 on those for now.
We'll start small with debuginfo=1. We'll step up to debuginfo=2 once most (all?) tests can handle debuginfo=1. There are more failures with debuginfo=2 than with debuginfo=1.
|
|
Make `x test distcheck` self-contained
Before, the checked components were extracted under the checked out source root, which caused us to test some weird combination of tarball + checkout sources/aritfacts/configuration.
Now `x test distcheck` works with an external temporary directory instead, which should make it self-contained. I also moved some config from the Dockerfile to the test itself, and fixed an issue in tidy that caused `x test tidy` to fail on tarball sources.
I also removed `.args(&builder.config.configure_args)`, because it was passing all kinds of crap from the CI config to the distcheck step, which was making it less reproducible.
Fixes: https://github.com/rust-lang/rust/issues/145183
r? ```@jieyouxu```
try-job: x86_64-gnu-distcheck
|
|
Wait for DPkg frontend lock when trying to remove packages
Hopefully this helps with [#t-infra > pr-check-1 fails on "free up disk space"](https://rust-lang.zulipchat.com/#narrow/channel/242791-t-infra/topic/pr-check-1.20fails.20on.20.22free.20up.20disk.20space.22/with/535794424).
As suggested by riking in [#t-infra > pr-check-1 fails on "free up disk space" @ đź’¬](https://rust-lang.zulipchat.com/#narrow/channel/242791-t-infra/topic/pr-check-1.20fails.20on.20.22free.20up.20disk.20space.22/near/535791579), thanks!
r? infra-ci
|
|
|
|
|
|
|
|
Signed-off-by: houpo-bob <houpocun@outlook.com>
|
|
It no longer fails with an error locally.
|
|
This issue has been fixed by
<https://github.com/rust-lang/rust/pull/145763>.
|
|
|
|
Fix host code appearing in Wasm binaries
This is a direct fix for issue [132802](https://github.com/rust-lang/rust/issues/132802).
Followed the outline as follows:
> * give a hard error in bootstrap when using gcc to compile for wasm
> * change our CI to use clang instead of gcc
> * add a test that compiling a sample program for wasm32-unknown doesn't give any linker warnings
The `test-various` ci job was also changed.
try-job: test-various
try-job: dist-various-1
try-job: dist-various-2
try-job: x86_64-msvc-1
|
|
To reduce risk of regressing on generating debuginfo e.g. in the form of
ICE:s. This will also ensure that future ui tests support different
debuginfo levels.
When I looked at run time for different CI jobs, **x86_64-gnu-debug**
was far from the bottle neck, so it should be fine to make it perform
more work.
|
|
|
|
r=Kobzol,madsmtm
Demote x86_64-apple-darwin to Tier 2 with host tools
Switch to only using aarch64 runners (implying we are now cross-compiling) and stop running tests. In the future, we could enable (some?) tests via Rosetta 2.
This implements the decision from https://github.com/rust-lang/rfcs/pull/3841.
|
|
run spellcheck as a tidy extra check in ci
This is probably how it should've been done from the start.
r? ``@Kobzol``
|
|
r=lolbinarycat,GuillaumeGomez
rustdoc-search: search backend with partitioned suffix tree
Before:
- https://notriddle.com/windows-docs-rs/doc-old/windows/
- https://doc.rust-lang.org/1.89.0/std/index.html
- https://doc.rust-lang.org/1.89.0/nightly-rustc/rustc_hir/index.html
After:
- https://notriddle.com/windows-docs-rs/doc/windows/
- https://notriddle.com/rustdoc-html-demo-12/stringdex/doc/std/index.html
- https://notriddle.com/rustdoc-html-demo-12/stringdex/compiler-doc/rustc_hir/index.html
## Summary
Rewrites the rustdoc search engine to use an indexed data structure, factored out as a crate called [stringdex](https://crates.io/crates/stringdex), that allows it to perform modified-levenshtein distance calculations, substring matches, and prefix matches in a reasonably efficient, and, more importantly, *incremental* algorithm.
## Motivation
Fixes https://github.com/rust-lang/rust/issues/131156
While the windows-rs crate is definitely the worst offender, I've noticed performance problems with the compiler crates as well. It makes no sense for rustdoc-search to have poor performance: it's basically a spell checker, and those have been usable since the 90's.
Stringdex is particularly designed to quickly return exact matches, to always report those matches first, and to never, ever [place new matches on top of old ones](https://web.dev/articles/cls). It also tries to yield to the event loop occasionally as it runs. This way, you can click the exactly-matched result before the rest of the search finishes running.
## Explanation
A longer description of how name search works can be found in stringdex's [HACKING.md](https://gitlab.com/notriddle/stringdex/-/blob/main/HACKING.md) file.
Type search is done by performing a name search on each element, then performing bitmap operations to narrow down a list of potential matches, then performing type unification on each match.
## Drawbacks
It's rather complex, and takes up more disk space than the current flat list of strings.
## Rationale and alternatives
Instead of a suffix tree, I could've used a different [approximate matching data structure](https://en.wikipedia.org/wiki/Approximate_string_matching). I didn't do that because I wanted to keep the current behavior (for example, a straightforward trigram index won't match [oepn](https://doc.rust-lang.org/nightly/std/?search=oepn) like the current system does).
## Prior art
[Sherlodoc](https://github.com/art-w/sherlodoc) is based on a similar concept, but they:
- use edit distance over a suffix tree for type-based search, instead of the binary matching that's implemented here
- use substring matching for name-based search, [but not fuzzy name matching](https://github.com/art-w/sherlodoc/issues/21)
- actually implement body text search, which is a potential-future feature, but not implemented in this PR
## Future possibilities
### Low-level optimization in stringdex
There are half a dozen low-level optimizations that I still need to explore. I haven't done them yet, because I've been working on bug fixes and rebasing on rustdoc's side, and a more solid and diverse test suite for stringdex itself.
- Stringdex decides whether to bundle two nodes into the same file based on size. To figure out a node's size, I have to run compression on it. This is probably slower than it needs to be.
- Stack compression is limited to the same 256-slot sliding windows as backref compression, and it doesn't have to be. (stack and backref compression are used to optimize the representation of the edge pointer from a parent node to its child; backref uses one byte, while stack is entirely implicit)
- The JS-side decoder is pretty naive. It performs unnecessary hash table lookups when decoding compressed nodes, and retains a list of hashes that it doesn't need. It needs to calculate the hashes in order to construct the merkle tree correctly, but it doesn't need to keep them.
- Data compression happens at the end, while emitting the node. This means it's not being counted when deciding on how to bundle, which is pretty dumb.
### Improved recall in type-driven search
Right now, type-driven search performs very strict matching. It's very precise, but misses a lot of things people would want.
What I'm not sure about is whether to focus more on edit-distance-based approaches, or to focus on type-theoretical approaches. Both gives avenues to improve, but edit distance is going to be faster while type checking is going to be more precise.
For example, a type theoretical improvement would fix [`Iterator<T>, (T -> U) -> Iterator<U>`](https://doc.rust-lang.org/nightly/std/?search=Iterator%3CT%3E%2C%20(T%20-%3E%20U)%20-%3E%20Iterator%3CU%3E) to give [`Iterator::map`](https://doc.rust-lang.org/nightly/std/iter/trait.Iterator.html#method.map), because it would recognize that the Map struct implements the Iterator trait. I don't know of any clean way to get this result to work without implementing significant type checking logic in search.js, and an edit-distance-based "dirty" approach would likely give a bunch of other results on top of this one.
## Full-text search
Once you've got this fuzzy dictionary matching to work, the logical next step is to implement some kind of information retrieval-based approach to phrase matching.
Like applying edit distance to types, phrase search gets you significantly better recall, but with a few major drawbacks:
- You have to pick between index bloat and the use of stopwords. Stopwords are bad because they might actually be important (try searching "if let" in mdBook if you're feeling brave), but without them, you spend a lot of space on text that doesn't matter.
- Example code also tends to have a lot of irrelevant stuff in it. Like stop words, we'd have to pick potentially-confusing or bloat.
Neither of these problems are deal-breakers, but they're worth keeping in mind.
|
|
|
|
|
|
|
|
Switch to only using aarch64 runners (implying we are now
cross-compiling) and stop running tests. In the future, we could
enable (some?) tests via Rosetta 2.
|
|
ci: clean windows disk space in background
|
|
|
|
Enforce in bootstrap that clippy must have stage at least 1
This mostly piggybacks on the previous `x check` [rework](https://github.com/rust-lang/rust/pull/143048).
The new "rules" follow the new staging logic. So `x clippy <foo>` lints `foo` using stage0 Clippy. `x clippy --stage 2 <foo>` lints `foo` using stage1 Clippy (which is built from in-tree sources).
I had to fix some latent issues with `prepare_compiler_for_check` along the way.
Checking `rustc_private` tools should now check less compiler crates (or rather not check compiler examples/tests/etc.), potentially speeding it up slightly.
I also had to make some manual adjustments to `x clippy ci` so that it doesn't do needless work.
r? `@jieyouxu`
|
|
Update to LLVM 21.1.0 rc3
|
|
|
|
|
|
Consolidate stage directories and group logs in bootstrap
My post-stage-0-redesign bootstrap fixes aren't done yet, but I think that enough steps have been migrated to the new system that it makes sense to actually modify the directories on disk, and what gets printed when bootstrap runs, so that it actually corresponds to the new system. Before, the printed stages didn't always make sense.
This PR:
- Fixes the numbering of `stageN` directories in the build directory. It was not corresponding to the correct stages before; notice that I did not modify `bootstrap/README.md`, as it was essentially describing what happens after this PR (first commit).
- Unifies all steps that output a build group to use the `Builder::msg` method. It's probably not the final stage, and some of the test steps might not be fully accurate yet, because I didn't fix test step numbering yet, but I think that it's a clear improvement from before, and now that everything uses the same method, we can easily make changes across the board, to ensure that it stays unified (second commit).
r? `@jieyouxu`
try-job: dist-x86_64-msvc
try-job: dist-x86_64-linux
|
|
|
|
Document compiler and stdlib in stage1 in `pr-check-2` CI job
This restores the original behavior pre-https://github.com/rust-lang/rust/pull/145011 (I thought that stage 2 makes more sense here, but it made the job ~30m slower, which is bad).
Let's see what will be the "new" duration, it should be ~55 minutes.
r? ```````@jieyouxu```````
|
|
|
|
Enable RUST_BACKTRACE=1 on CI
We should really see the backtrace if something in bootstrap fails on CI. https://github.com/rust-lang/rust/pull/145011#issuecomment-3172664934 doesn't show many details.
|
|
We should really see the backtrace if something in bootstrap fails on CI.
|
|
As they were previously.
|
|
|
|
The project build for compiler-rt is deprecated.
The runtimes build will use the just-built clang. As such, we
also need to pass --gcc-toolchain to the runtimes build, so that
it can find the GCC installation.
|
|
Necessary to avoid a bolt-related crash.
|
|
Remove install Rust script from CI
Windows ARM images should contain Rust now (https://github.com/actions/partner-runner-images/issues/77#issuecomment-3082613685).
CC dpaoliello
try-job: `*aarch64-msvc*`
|
|
Reject running `compiletest` self-tests against stage 0 rustc unless explicitly allowed
Currently, in `pr-check-1`, we run `python3 ../x.py test --stage 0 src/tools/compiletest`, which is `compiletest` self-tests against stage 0 rustc. This makes it very annoying for PRs wanting to change target spec JSON format, which `compiletest` depends on for target information, as otherwise `compiletest` would have to know how to handle 2 different target spec JSON formats and know when to pick which.
Instead of doing that, we change `compiletest` self-tests to reject running against stage 0 `rustc` *unless* explicitly allowed with `build.compiletest-allow-stage0=true`. `build.compiletest-allow-stage0` is a proper bootstrap config option in favor of the ad-hoc `COMPILETEST_FORCE_STAGE0` env var. This means that:
- `./x test src/tools/compiletest --stage=0` is not allowed, unless `build.compiletest-allow-stage0=true` is set. In this scenario, `compiletest` self-tests should be expected to fail unless the stage 0 `rustc` as provided is like codegen_cranelift where it's *actually* built from in-tree `rustc` sources.
- In CI, we change `./x test src/tools/compiletest --stage=0` to `./x test src/tools/compiletest --stage=1`, and move it to `pr-check-2`. Yes, this involves building the stage 1 compiler, but `pr-check-2` already has to build stage 1 compiler to test stage 1 library crates.
- Crucially, this means that **`compiletest` is only intended to support one target spec JSON format**, namely the one corresponding to the in-tree `rustc`.
- This should preserve the `compiletest-use-stage0-libtest` UX optimization where changing `compiler/` tree should still not require rebuilding `compiletest` as long as `build.compiletest-use-stage0-libtest=true`, as that should remain orthogonal. This is completely unlike my previous attempt at https://github.com/rust-lang/rust/pull/144563 that tries to do a way more invasive change which would cause the rebuild problem.
Best reviewed commit-by-commit.
---
r? `@Kobzol`
|
