| Age | Commit message (Collapse) | Author | Lines |
|---|
|
|
|
CI: dynamic jobs
This PR modifies our CI workflows to be dynamic. This means that when a GitHub event is generated, we will run a Python script (`calculate-job-matrix.py`), which decides which CI jobs should be generated. These jobs are defined in `src/ci/github-actions/jobs.yml`).
This should provide a few benefits:
- Once the migration to dynamic jobs is complete, we shouldn't need `expand-yaml-anchors` anymore.
- The job table on PRs (and also the left job column on auto/try builds) should be much cleaner and contain only the jobs that are actually relevant/executed.
- It should be much easier to support dynamic try builds, i.e. to run an arbitrary CI job on a try build.
See [this Zulip discussion](https://rust-lang.zulipchat.com/#narrow/stream/242791-t-infra/topic/job.20matrix.20re-ordered.20PR.20list) for more context.
r? `@ghost`
|
|
|
|
|
|
|
|
|
|
enable clippy for bootstrap on CI PRs (in `mingw-check` image)
Let's keep the bootstrap codebase cleaner.
|
|
CI: add script for installing NodeJS and update it to v20
I centralized the installation on a single place to make it simple to update the NodeJS version across the board.
Fixes: https://github.com/rust-lang/rust/issues/123965
r? `@Mark-Simulacrum`
|
|
r=Mark-Simulacrum
Provide prebuilt std for gnullvm targets
Revival of https://github.com/rust-lang/rust/pull/114346 which waiting on MCP that was accepted recently: https://github.com/rust-lang/compiler-team/issues/710#issuecomment-1942014308
|
|
|
|
|
|
|
|
|
|
|
|
Update how WASI toolchains are used in CI and bootstrap
This commit updates how the WASI targets are configured with their toolchain. Long ago a `config.toml` option of `wasi-root` was added to enable building with the WASI files produced by wasi-libc. Additionally for CI testing and release building the Rust toolchain has been using a hard-coded commit of wasi-libc which is bundled with the release of the `wasm32-wasip1` target, for example.
Nowadays though the wasi-sdk project, the C/C++ toolchain for WASI, is the go-to solution for compiling/linking WASI code and contains the more-or-less official releases of wasi-libc. This commit migrates CI to using wasi-sdk releases and additionally updates `bootstrap` to recognize when this is configured. This means that with `$WASI_SDK_PATH` configured there's no further configuration necessary to get a working build. Notably this also works better for the new targets of WASI as well, such as `wasm32-wasip2` and `wasm32-wasip1-threads` where the wasi-sdk release now has libraries for all targets bundled within it.
|
|
Signed-off-by: onur-ozkan <work@onurozkan.dev>
|
|
Improve the experience of running Docker locally
When running locally, the absence of the `GITHUB_STEP_SUMMARY` environment variable will lead to the following error:
```
::endgroup::
./src/ci/docker/run.sh: line 349: : No such file or directory
```
I've also changed the output artifacts directory to `obj/$image_name`, allowing me to easily run all images locally. We always encounter various strange issues when modifying the test cases in the `codegen` directory.
r? Kobzol cc `@saethlin`
|
|
Previously this command was linting compiler and library together.
As we no longer run clippy on the entire tree unless it's explicitly
requested, we need to update this command by adding `library` path.
Signed-off-by: onur-ozkan <work@onurozkan.dev>
|
|
|
|
CI: add a script for dynamically computing CI job matrix
It would be great if was easier to run specific CI workflows locally, and also to allow us to spawn a specific CI workflow by bors, to enable running arbitrary try builds. See discussion [here](https://rust-lang.zulipchat.com/#narrow/stream/242791-t-infra/topic/CI.20workflows.20refactoring).
This PR is a first step in that direction.
- Moves the definition of CI runners and (for now) PR jobs into a separate `jobs.yml` file.
- Adds a simple Python script that reads the file, decides which jobs should be active for the current CI workflow, and prints them as JSON to their output.
- The PR job then reads this output and generates its job matrix based on it.
By moving the job definitions from `ci.yml` into a separate file, we can handle it programmatically, which should make it easier to both do local execution of CI jobs and also to do arbitrary try builds.
|
|
This commit updates how the WASI targets are configured with their
toolchain. Long ago a `config.toml` option of `wasi-root` was added to
enable building with the WASI files produced by wasi-libc. Additionally
for CI testing and release building the Rust toolchain has been using a
hard-coded commit of wasi-libc which is bundled with the release of the
`wasm32-wasip1` target, for example.
Nowadays though the wasi-sdk project, the C/C++ toolchain for WASI, is
the go-to solution for compiling/linking WASI code and contains the
more-or-less official releases of wasi-libc. This commit migrates CI to
using wasi-sdk releases and additionally updates `bootstrap` to
recognize when this is configured. This means that with `$WASI_SDK_PATH`
configured there's no further configuration necessary to get a working
build. Notably this also works better for the new targets of WASI as
well, such as `wasm32-wasip2` and `wasm32-wasip1-threads` where the
wasi-sdk release now has libraries for all targets bundled within it.
|
|
|
|
|
|
ci: test cargo on `aarch64-gnu`
Since `aarch64-unknown-linux-gnu` is a tier-1 target, we should also test cargo on it, especially since cargo's own CI doesn't cover this yet. This might have helped us discover #123733 sooner, which is not a cargo problem but was uncovered by a new cargo test (which we'll have to skip for now). Everything else passes in my local run, so at least we'll have a guard against future regressions.
|
|
Co-authored-by: Eric Huss <eric@huss.org>
|
|
Enable building tier2 target riscv32im-unknown-none-elf
riscv32im-unknown-none-elf was promoted to tier2 in
https://github.com/rust-lang/rust/pull/117874
but it has not yet been added to the list of build targets.
By adding riscv32im-unknown-none-elf to the list of build targets, this PR enables end-users to install this target via rustup.
|
|
This updates the Fuchsia commit used in `auto - x86_64-gnu-integration`
CI bot to use the Rust commit 703dc9ce64d9b31a239a7280d9b5f9ddd85ffed6.
This should help improve the coverage of this builder.
It also updates the SDK version to F20.20240412.3.1, and the Fuchsia Clang
version to c777c011a709dffd4fa5e79cad7947b7c3405d02.
|
|
|
|
|
|
|
|
[rustdoc] [GUI tests] Make theme switching closer to reality
Better to actually perform actions user do rather than only testing the change through local storage.
As for `browser-ui-test` update: I updated `puppeteer` version (to `0.19.4`) and fixed a bug when displaying the file if it came from an `include`.
r? `@notriddle`
|
|
|
|
|
|
|
|
Rollup of 4 pull requests
Successful merges:
- #114788 (impl get_mut_or_init and get_mut_or_try_init for OnceCell and OnceLock)
- #122291 (Stabilize `const_caller_location` and `const_location_fields`)
- #123357 (CI: Redirect stderr to stdout to order GHA logs)
- #123504 (bootstrap: split cargo-miri test into separate Step)
r? `@ghost`
`@rustbot` modify labels: rollup
|
|
bootstrap: split cargo-miri test into separate Step
This makes it easier to test just the driver or the cargo-miri integration.
````@rust-lang/miri```` this means to test both you now need to do `./x.py test miri cargo-miri`.
|
|
CI: Redirect stderr to stdout to order GHA logs
This PR modifies the main CI workflow so that its stderr is redirected to stdout. This should make it so that stderr and stdout output is not interleaved in the wrong order in GHA logs (see discussion [here](https://rust-lang.zulipchat.com/#narrow/stream/242791-t-infra/topic/Github.20actions.20logs.20show.20lines.20in.20the.20wrong.20order)).
|
|
Bump dependencies
Follow up for #123252
Unfortunately this file needs to be manually bumped when any dependencies are bumped in the main lockfile
```
Updating autocfg v1.1.0 -> v1.2.0
Updating chrono v0.4.35 -> v0.4.37
Updating clap v4.5.3 -> v4.5.4
Updating clap_derive v4.5.3 -> v4.5.4
Updating handlebars v5.1.0 -> v5.1.2
Updating itoa v1.0.10 -> v1.0.11
Updating memoffset v0.9.0 -> v0.9.1
Updating openssl-sys v0.9.101 -> v0.9.102
Updating pin-project-lite v0.2.13 -> v0.2.14
Updating r-efi v4.3.0 -> v4.4.0
Updating regex-syntax v0.8.2 -> v0.8.3
Updating security-framework v2.9.2 -> v2.10.0
Updating security-framework-sys v2.9.1 -> v2.10.0
Updating serde_json v1.0.114 -> v1.0.115
Updating syn v2.0.55 -> v2.0.57
Updating tokio v1.36.0 -> v1.37.0
```
|
|
|
|
|
|
Support running library tests in Miri
This adds a new bootstrap subcommand `./x.py miri` which can test libraries in Miri. This is in preparation for eventually doing that as part of bors CI, but this PR only adds the infrastructure, and doesn't enable it yet.
`@rust-lang/bootstrap` should this be `x.py test --miri library/core` or `x.py miri library/core`? The flag has the advantage that we don't have to copy all the arguments from `Subcommand::Test`. It has the disadvantage that most test steps just ignore `--miri` and still run tests the regular way. For clippy you went the route of making it a separate subcommand. ~~I went with a flag now as that seemed easier, but I can change this.~~ I made it a new subcommand. Note however that the regular cargo invocation would be `cargo miri test ...`, so `x.py` is still going to be different in that the `test` is omitted. That said, we could also make it `./x.py miri-test` to make that difference smaller -- that's in fact more consistent with the internal name of the command when bootstrap invokes cargo.
`@rust-lang/libs` ~~unfortunately this PR does some unholy things to the `lib.rs` files of our library crates.~~
`@m-ou-se` found a way that entirely avoids library-level hacks, except for some new small `lib.miri.rs` files that hopefully you will never have to touch. There's a new hack in cargo-miri but there it is in good company...
|
|
Default to light theme if JS is enabled but not working
It doesn't [fix] #123399 but it allows to reduce the problem:
* if JS is completely disabled, then `noscript.css` will be applied
* if JS failed for any reason, then the light theme will be applied (because `noscript.css` won't be applied)
r? `@notriddle`
|
|
|
|
|
|
|
|
|
|
Update to new browser-ui-test version
This new version brings a lot of new internal improvements (mostly around validating the commands input).
It also improved some command names and arguments.
r? `@notriddle`
|
|
rustdoc-search: shard the search result descriptions
## Preview
This makes no visual changes to rustdoc search. It's a pure perf improvement.
<details><summary>old</summary>
Preview: <http://notriddle.com/rustdoc-html-demo-10/doc/std/index.html?search=vec>
WebPageTest Comparison with before branch on a sort of worst case (searching `vec`, winds up downloading most of the shards anyway): <https://www.webpagetest.org/video/compare.php?tests=240317_AiDc61_2EM,240317_AiDcM0_2EN>
Waterfall diagram:
</details>
Preview: <http://notriddle.com/rustdoc-html-demo-10/doc2/std/index.html?search=vec>
WebPageTest Comparison with before branch on a sort of worst case (searching `vec`, winds up downloading most of the shards anyway): <https://www.webpagetest.org/video/compare.php?tests=240322_BiDcCH_13R,240322_AiDcJY_104>
## Description
r? `@GuillaumeGomez`
The descriptions are, on almost all crates[^1], the majority of the size of the search index, even though they aren't really used for searching. This makes it relatively easy to separate them into their own files.
Additionally, this PR pulls out information about whether there's a description into a bitmap. This allows us to sort, truncate, *then* download.
This PR also bumps us to ES8. Out of the browsers we support, all of them support async functions according to caniuse.
https://caniuse.com/async-functions
[^1]:
<https://microsoft.github.io/windows-docs-rs/>, a crate with
44MiB of pure names and no descriptions for them, is an outlier
and should not be counted. But this PR should improve it, by replacing a long line of empty strings with a compressed bitmap with a single Run section. Just not very much.
## Detailed sizes
```console
$ cat test.sh
set -ex
cp ../search-index*.js search-index.js
awk 'FNR==NR {a++;next} FNR<a-3' search-index.js{,} | awk 'NR>1 {gsub(/\],\\$/,""); gsub(/^\["[^"]+",/,""); print} {next}' | sed -E "s:\\\\':':g" > search-index.json
jq -c '.t' search-index.json > t.json
jq -c '.n' search-index.json > n.json
jq -c '.q' search-index.json > q.json
jq -c '.D' search-index.json > D.json
jq -c '.e' search-index.json > e.json
jq -c '.i' search-index.json > i.json
jq -c '.f' search-index.json > f.json
jq -c '.c' search-index.json > c.json
jq -c '.p' search-index.json > p.json
jq -c '.a' search-index.json > a.json
du -hs t.json n.json q.json D.json e.json i.json f.json c.json p.json a.json
$ bash test.sh
+ cp ../search-index1.78.0.js search-index.js
+ awk 'FNR==NR {a++;next} FNR<a-3' search-index.js search-index.js
+ awk 'NR>1 {gsub(/\],\\$/,""); gsub(/^\["[^"]+",/,""); print} {next}'
+ sed -E 's:\\'\'':'\'':g'
+ jq -c .t search-index.json
+ jq -c .n search-index.json
+ jq -c .q search-index.json
+ jq -c .D search-index.json
+ jq -c .e search-index.json
+ jq -c .i search-index.json
+ jq -c .f search-index.json
+ jq -c .c search-index.json
+ jq -c .p search-index.json
+ jq -c .a search-index.json
+ du -hs t.json n.json q.json D.json e.json i.json f.json c.json p.json a.json
64K t.json
800K n.json
8.0K q.json
4.0K D.json
16K e.json
192K i.json
544K f.json
4.0K c.json
36K p.json
20K a.json
```
These are, roughly, the size of each section in the standard library (this tool actually excludes libtest, for parsing-json-with-awk reasons, but libtest is tiny so it's probably not important).
t = item type, like "struct", "free fn", or "type alias". Since one byte is used for every item, this implies that there are approximately 64 thousand items in the standard library.
n = name, and that's now the largest section of the search index with the descriptions removed from it
q = parent *module* path, stored parallel to the items within
D = the size of each description shard, stored as vlq hex numbers
e = empty description bit flags, stored as a roaring bitmap
i = parent *type* index as a link into `p`, stored as decimal json numbers; used only for associated types; might want to switch to vlq hex, since that's shorter, but that would be a separate pr
f = function signature, stored as lists of lists that index into `p`
c = deprecation flag, stored as a roaring bitmap
p = parent *type*, stored separately and linked into from `i` and `f`
a = alias, as [[key, value]] pairs
## Search performance
http://notriddle.com/rustdoc-html-demo-11/perf-shard/index.html
For example, in stm32f4:
<table><thead><tr><th>before<th>after</tr></thead>
<tbody><tr><td>
```
Testing T -> U ... in_args = 0, returned = 0, others = 200
wall time = 617
Testing T, U ... in_args = 0, returned = 0, others = 200
wall time = 198
Testing T -> T ... in_args = 0, returned = 0, others = 200
wall time = 282
Testing crc32 ... in_args = 0, returned = 0, others = 0
wall time = 426
Testing spi::pac ... in_args = 0, returned = 0, others = 0
wall time = 673
```
</td><td>
```
Testing T -> U ... in_args = 0, returned = 0, others = 200
wall time = 716
Testing T, U ... in_args = 0, returned = 0, others = 200
wall time = 207
Testing T -> T ... in_args = 0, returned = 0, others = 200
wall time = 289
Testing crc32 ... in_args = 0, returned = 0, others = 0
wall time = 418
Testing spi::pac ... in_args = 0, returned = 0, others = 0
wall time = 687
```
</td></tr><tr><td>
```
user: 005.345 s
sys: 002.955 s
wall: 006.899 s
child_RSS_high: 583664 KiB
group_mem_high: 557876 KiB
```
</td><td>
```
user: 004.652 s
sys: 000.565 s
wall: 003.865 s
child_RSS_high: 538696 KiB
group_mem_high: 511724 KiB
```
</td></tr>
</table>
This perf tester is janky and unscientific enough that the apparent differences might just be noise. If it's not an order of magnitude, it's probably not real.
## Future possibilities
* Currently, results are not shown until the descriptions are downloaded. Theoretically, the description-less results could be shown. But actually doing that, and making sure it works properly, would require extra work (we have to be careful to avoid layout jumps).
* More than just descriptions can be sharded this way. But we have to be careful to make sure the size wins are worth the round trips. Ideally, data that’s needed only for display should be sharded while data needed for search isn’t.
* [Full text search](https://internals.rust-lang.org/t/full-text-search-for-rustdoc-and-doc-rs/20427) also needs this kind of infrastructure. A good implementation might store a compressed bloom filter in the search index, then download the full keyword in shards. But, we have to be careful not just of the amount readers have to download, but also of the amount that [publishers](https://gist.github.com/notriddle/c289e77f3ed469d1c0238d1d135d49e1) have to store.
|
|
|
|
Pass RUST_BACKTRACE when running docker.
|
