| Age | Commit message (Collapse) | Author | Lines |
|---|
|
CI: Redirect stderr to stdout to order GHA logs
This PR modifies the main CI workflow so that its stderr is redirected to stdout. This should make it so that stderr and stdout output is not interleaved in the wrong order in GHA logs (see discussion [here](https://rust-lang.zulipchat.com/#narrow/stream/242791-t-infra/topic/Github.20actions.20logs.20show.20lines.20in.20the.20wrong.20order)).
|
|
Bump dependencies
Follow up for #123252
Unfortunately this file needs to be manually bumped when any dependencies are bumped in the main lockfile
```
Updating autocfg v1.1.0 -> v1.2.0
Updating chrono v0.4.35 -> v0.4.37
Updating clap v4.5.3 -> v4.5.4
Updating clap_derive v4.5.3 -> v4.5.4
Updating handlebars v5.1.0 -> v5.1.2
Updating itoa v1.0.10 -> v1.0.11
Updating memoffset v0.9.0 -> v0.9.1
Updating openssl-sys v0.9.101 -> v0.9.102
Updating pin-project-lite v0.2.13 -> v0.2.14
Updating r-efi v4.3.0 -> v4.4.0
Updating regex-syntax v0.8.2 -> v0.8.3
Updating security-framework v2.9.2 -> v2.10.0
Updating security-framework-sys v2.9.1 -> v2.10.0
Updating serde_json v1.0.114 -> v1.0.115
Updating syn v2.0.55 -> v2.0.57
Updating tokio v1.36.0 -> v1.37.0
```
|
|
|
|
|
|
Support running library tests in Miri
This adds a new bootstrap subcommand `./x.py miri` which can test libraries in Miri. This is in preparation for eventually doing that as part of bors CI, but this PR only adds the infrastructure, and doesn't enable it yet.
`@rust-lang/bootstrap` should this be `x.py test --miri library/core` or `x.py miri library/core`? The flag has the advantage that we don't have to copy all the arguments from `Subcommand::Test`. It has the disadvantage that most test steps just ignore `--miri` and still run tests the regular way. For clippy you went the route of making it a separate subcommand. ~~I went with a flag now as that seemed easier, but I can change this.~~ I made it a new subcommand. Note however that the regular cargo invocation would be `cargo miri test ...`, so `x.py` is still going to be different in that the `test` is omitted. That said, we could also make it `./x.py miri-test` to make that difference smaller -- that's in fact more consistent with the internal name of the command when bootstrap invokes cargo.
`@rust-lang/libs` ~~unfortunately this PR does some unholy things to the `lib.rs` files of our library crates.~~
`@m-ou-se` found a way that entirely avoids library-level hacks, except for some new small `lib.miri.rs` files that hopefully you will never have to touch. There's a new hack in cargo-miri but there it is in good company...
|
|
Default to light theme if JS is enabled but not working
It doesn't [fix] #123399 but it allows to reduce the problem:
* if JS is completely disabled, then `noscript.css` will be applied
* if JS failed for any reason, then the light theme will be applied (because `noscript.css` won't be applied)
r? `@notriddle`
|
|
|
|
|
|
|
|
|
|
Update to new browser-ui-test version
This new version brings a lot of new internal improvements (mostly around validating the commands input).
It also improved some command names and arguments.
r? `@notriddle`
|
|
rustdoc-search: shard the search result descriptions
## Preview
This makes no visual changes to rustdoc search. It's a pure perf improvement.
<details><summary>old</summary>
Preview: <http://notriddle.com/rustdoc-html-demo-10/doc/std/index.html?search=vec>
WebPageTest Comparison with before branch on a sort of worst case (searching `vec`, winds up downloading most of the shards anyway): <https://www.webpagetest.org/video/compare.php?tests=240317_AiDc61_2EM,240317_AiDcM0_2EN>
Waterfall diagram:
</details>
Preview: <http://notriddle.com/rustdoc-html-demo-10/doc2/std/index.html?search=vec>
WebPageTest Comparison with before branch on a sort of worst case (searching `vec`, winds up downloading most of the shards anyway): <https://www.webpagetest.org/video/compare.php?tests=240322_BiDcCH_13R,240322_AiDcJY_104>
## Description
r? `@GuillaumeGomez`
The descriptions are, on almost all crates[^1], the majority of the size of the search index, even though they aren't really used for searching. This makes it relatively easy to separate them into their own files.
Additionally, this PR pulls out information about whether there's a description into a bitmap. This allows us to sort, truncate, *then* download.
This PR also bumps us to ES8. Out of the browsers we support, all of them support async functions according to caniuse.
https://caniuse.com/async-functions
[^1]:
<https://microsoft.github.io/windows-docs-rs/>, a crate with
44MiB of pure names and no descriptions for them, is an outlier
and should not be counted. But this PR should improve it, by replacing a long line of empty strings with a compressed bitmap with a single Run section. Just not very much.
## Detailed sizes
```console
$ cat test.sh
set -ex
cp ../search-index*.js search-index.js
awk 'FNR==NR {a++;next} FNR<a-3' search-index.js{,} | awk 'NR>1 {gsub(/\],\\$/,""); gsub(/^\["[^"]+",/,""); print} {next}' | sed -E "s:\\\\':':g" > search-index.json
jq -c '.t' search-index.json > t.json
jq -c '.n' search-index.json > n.json
jq -c '.q' search-index.json > q.json
jq -c '.D' search-index.json > D.json
jq -c '.e' search-index.json > e.json
jq -c '.i' search-index.json > i.json
jq -c '.f' search-index.json > f.json
jq -c '.c' search-index.json > c.json
jq -c '.p' search-index.json > p.json
jq -c '.a' search-index.json > a.json
du -hs t.json n.json q.json D.json e.json i.json f.json c.json p.json a.json
$ bash test.sh
+ cp ../search-index1.78.0.js search-index.js
+ awk 'FNR==NR {a++;next} FNR<a-3' search-index.js search-index.js
+ awk 'NR>1 {gsub(/\],\\$/,""); gsub(/^\["[^"]+",/,""); print} {next}'
+ sed -E 's:\\'\'':'\'':g'
+ jq -c .t search-index.json
+ jq -c .n search-index.json
+ jq -c .q search-index.json
+ jq -c .D search-index.json
+ jq -c .e search-index.json
+ jq -c .i search-index.json
+ jq -c .f search-index.json
+ jq -c .c search-index.json
+ jq -c .p search-index.json
+ jq -c .a search-index.json
+ du -hs t.json n.json q.json D.json e.json i.json f.json c.json p.json a.json
64K t.json
800K n.json
8.0K q.json
4.0K D.json
16K e.json
192K i.json
544K f.json
4.0K c.json
36K p.json
20K a.json
```
These are, roughly, the size of each section in the standard library (this tool actually excludes libtest, for parsing-json-with-awk reasons, but libtest is tiny so it's probably not important).
t = item type, like "struct", "free fn", or "type alias". Since one byte is used for every item, this implies that there are approximately 64 thousand items in the standard library.
n = name, and that's now the largest section of the search index with the descriptions removed from it
q = parent *module* path, stored parallel to the items within
D = the size of each description shard, stored as vlq hex numbers
e = empty description bit flags, stored as a roaring bitmap
i = parent *type* index as a link into `p`, stored as decimal json numbers; used only for associated types; might want to switch to vlq hex, since that's shorter, but that would be a separate pr
f = function signature, stored as lists of lists that index into `p`
c = deprecation flag, stored as a roaring bitmap
p = parent *type*, stored separately and linked into from `i` and `f`
a = alias, as [[key, value]] pairs
## Search performance
http://notriddle.com/rustdoc-html-demo-11/perf-shard/index.html
For example, in stm32f4:
<table><thead><tr><th>before<th>after</tr></thead>
<tbody><tr><td>
```
Testing T -> U ... in_args = 0, returned = 0, others = 200
wall time = 617
Testing T, U ... in_args = 0, returned = 0, others = 200
wall time = 198
Testing T -> T ... in_args = 0, returned = 0, others = 200
wall time = 282
Testing crc32 ... in_args = 0, returned = 0, others = 0
wall time = 426
Testing spi::pac ... in_args = 0, returned = 0, others = 0
wall time = 673
```
</td><td>
```
Testing T -> U ... in_args = 0, returned = 0, others = 200
wall time = 716
Testing T, U ... in_args = 0, returned = 0, others = 200
wall time = 207
Testing T -> T ... in_args = 0, returned = 0, others = 200
wall time = 289
Testing crc32 ... in_args = 0, returned = 0, others = 0
wall time = 418
Testing spi::pac ... in_args = 0, returned = 0, others = 0
wall time = 687
```
</td></tr><tr><td>
```
user: 005.345 s
sys: 002.955 s
wall: 006.899 s
child_RSS_high: 583664 KiB
group_mem_high: 557876 KiB
```
</td><td>
```
user: 004.652 s
sys: 000.565 s
wall: 003.865 s
child_RSS_high: 538696 KiB
group_mem_high: 511724 KiB
```
</td></tr>
</table>
This perf tester is janky and unscientific enough that the apparent differences might just be noise. If it's not an order of magnitude, it's probably not real.
## Future possibilities
* Currently, results are not shown until the descriptions are downloaded. Theoretically, the description-less results could be shown. But actually doing that, and making sure it works properly, would require extra work (we have to be careful to avoid layout jumps).
* More than just descriptions can be sharded this way. But we have to be careful to make sure the size wins are worth the round trips. Ideally, data that’s needed only for display should be sharded while data needed for search isn’t.
* [Full text search](https://internals.rust-lang.org/t/full-text-search-for-rustdoc-and-doc-rs/20427) also needs this kind of infrastructure. A good implementation might store a compressed bloom filter in the search index, then download the full keyword in shards. But, we have to be careful not just of the amount readers have to download, but also of the amount that [publishers](https://gist.github.com/notriddle/c289e77f3ed469d1c0238d1d135d49e1) have to store.
|
|
|
|
Pass RUST_BACKTRACE when running docker.
|
|
|
|
|
|
|
|
Refactor the way bootstrap invokes `cargo miri`
Instead of basically doing `cargo run --manifest-path=<cargo-miri's manifest> -- miri`, let's invoke the `cargo-miri` binary directly. That means less indirections, and also makes it easier to e.g. run the libcore test suite in Miri. (But there are still other issues with that.)
Also also adjusted Miri's stage numbering so that it is consistent with rustc/rustdoc.
This also makes `./x.py test miri` honor `--no-doc`.
And this fixes https://github.com/rust-lang/rust/issues/123177 by moving where we handle parallel_compiler.
|
|
|
|
Update ninja on Windows
Errors started showing up, and I read somewhere that this might be because of old ninja versions. This ninja version is indeed *ancient*.
```
multiple outputs aren't (yet?) supported by depslog; bring this up on the mailing list if it affects you
```
This requires someone uploading https://github.com/ninja-build/ninja/releases/download/v1.11.1/ninja-win.zip as `2024-03-28-v1.11.1-ninja-win.zip`.
should end https://github.com/rust-lang/rust/pull/122671#issuecomment-2026178791
|
|
Errors started showing up, and I read somewhere that this might be
because of old ninja versions. This ninja version is indeed *ancient*.
```
multiple outputs aren't (yet?) supported by depslog; bring this up on the mailing list if it affects you
```
|
|
Update upload-artifact to v4
This contains a breaking change around artifact merging no longer being done. This was not relied on, so it's fine.
|
|
ci: Build gccjit from a git archive
A full `git clone` of GCC includes quite a lot of history, and it's
completely unnecessary for building it in CI. We can use a GitHub
archive URL to get a simple tarball that is much faster to download.
Also, the `gcc-build` directory can be removed after install to reduce
the image size even further.
|
|
This contains a breaking change around artifact merging no longer being
done. This was not relied on, so it's fine.
|
|
This is implemented with the freshly-released Wasmtime 19 and should
prevent beta breakage from wasm tests that was observed and fixed
in #122640 again.
|
|
Co-authored-by: Urgau <3616612+Urgau@users.noreply.github.com>
|
|
Co-authored-by: Urgau <3616612+Urgau@users.noreply.github.com>
|
|
A full `git clone` of GCC includes quite a lot of history, and it's
completely unnecessary for building it in CI. We can use a GitHub
archive URL to get a simple tarball that is much faster to download.
Also, the `gcc-build` directory can be removed after install to reduce
the image size even further.
|
|
|
|
The descriptions are, on almost all crates[^1], the majority
of the size of the search index, even though they aren't really
used for searching. This makes it relatively easy to separate
them into their own files.
This commit also bumps us to ES8. Out of the browsers we support,
all of them support async functions according to caniuse.
https://caniuse.com/async-functions
[^1]:
<https://microsoft.github.io/windows-docs-rs/>, a crate with
44MiB of pure names and no descriptions for them, is an outlier
and should not be counted.
|
|
`mv` tools off the path instead of `rm -r`-ing them in `install-msys2.sh`
This is a follow up patch to https://github.com/rust-lang/rust/pull/121182
r? `@Mark-Simulacrum`
|
|
Check library crates for all tier 1 targets in PR CI
Let's try checking all tier 1 targets. Shouldn't take much time.
Not sure if this is the right place to put it or not but let's see if it works first.
|
|
CI: cache PR CI Docker builds
An attempt to restore Docker caching for PR CI workflows that share the Docker image with an auto/try workflow.
This was broken by my previous [PR](https://github.com/rust-lang/rust/pull/119290) that changed how we cache the Docker builds.
[Before](https://github.com/rust-lang/rust/pull/122545):
After:
r? ``@ghost``
|
|
|
|
|
|
|
|
|
|
|
|
Rename `wasm32-wasi-preview1-threads` to `wasm32-wasip1-threads`
This commit renames the current `wasm32-wasi-preview1-threads` target to `wasm32-wasip1-threads`. The need for this rename is a bit unfortunate as the previous name was chosen in an attempt to be future-compatible with other WASI targets. Originally this target was proposed to be `wasm32-wasi-threads`, and that's what was originally implemented in wasi-sdk as well. After discussion though and with the plans for the upcoming component-model target (now named `wasm32-wasip2`) the "preview1" naming was chosen for the threads-based target. The WASI subgroup later decided that it was time to drop the "preview" terminology and recommends "pX" instead, hence previous PRs to add `wasm32-wasip2` and rename `wasm32-wasi` to `wasm32-wasip1`.
So, with all that history, the "proper name" for this target is different than its current name, so one way or another a rename is required. This PR proposes renaming this target cold-turkey, unlike `wasm32-wasi` which is having a long transition period to change its name. The threads-based target is predicted to see only a fraction of the traffic of `wasm32-wasi` due to the unstable nature of the WASI threads proposal itself.
While I was here I updated the in-tree documentation in the target spec file itself as most of the documentation was copied from the original WASI target and wasn't as applicable to this target.
Also, as an aside, I can at least try to apologize for all the naming confusion here, but this is hopefully the last WASI-related rename.
|
|
Build `rustc` with 1CGU on `x86_64-apple-darwin`
Distribute `x86_64-apple-darwin` artifacts built with `rust.codegen-units=1`, like we already do on Linux.
1) effect on code size on `x86_64-apple-darwin`: it's a 11.14% reduction on `librustc_driver.dylib`
- before, [`41d97c8a5dea2731b0e56fe97cd7cb79e21cff79`](https://ci-artifacts.rust-lang.org/rustc-builds/41d97c8a5dea2731b0e56fe97cd7cb79e21cff79/rustc-nightly-x86_64-apple-darwin.tar.xz): 161232048
- after, [`7549dbdc09f0c4f6cc84002ac03081828054784b`](https://ci-artifacts.rust-lang.org/rustc-builds/7549dbdc09f0c4f6cc84002ac03081828054784b/rustc-nightly-x86_64-apple-darwin.tar.xz): 143256928
2) time it took on CI:
- the [first `try` build](https://github.com/rust-lang-ci/rust/actions/runs/8155512915/job/22291187124) took: 1h 33m
- the [second `try` build](https://github.com/rust-lang-ci/rust/actions/runs/8157057880/job/22295839911) took: 1h 45m
3) most recent perf results on (a noisy) x64 mac are [here](https://github.com/rust-lang/rust/pull/112268#issuecomment-1575635073).
Related tracking issue for build configuration: #103595
|
|
Test wasm32-wasip1 in CI, not wasm32-unknown-unknown
This commit changes CI to no longer test the `wasm32-unknown-unknown` target and instead test the `wasm32-wasip1` target. There was some discussion of this in a [Zulip thread], and the motivations for this PR are:
* Runtime failures on `wasm32-unknown-unknown` print nothing, meaning all you get is "something failed". In contrast `wasm32-wasip1` can print to stdout/stderr.
* The unknown-unknown target is missing lots of pieces of libstd, and while `wasm32-wasip1` is also missing some pieces (e.g. threads) it's missing fewer pieces. This means that many more tests can be run.
Overall my hope is to improve the debuggability of wasm failures on CI and ideally be a bit less of a maintenance burden.
This commit specifically removes the testing of `wasm32-unknown-unknown` and replaces it with testing of `wasm32-wasip1`. Along the way there were a number of other archiectural changes made as well, including:
* A new `target.*.runtool` option can now be specified in `config.toml` which is passed as `--runtool` to `compiletest`. This is used to reimplement execution of WebAssembly in a less-wasm-specific fashion.
* The default value for `runtool` is an ambiently located WebAssembly runtime found on the system, if any. I've implemented logic for Wasmtime.
* Existing testing support for `wasm32-unknown-unknown` and Emscripten has been removed. I'm not aware of Emscripten testing being run any time recently and otherwise `wasm32-wasip1` is in theory the focus now.
* I've added a new `//@ needs-threads` directive for `compiletest` and classified a bunch of wasm-ignored tests as needing threads. In theory these tests can run on `wasm32-wasi-preview1-threads`, for example.
* I've tried to audit all existing tests that are either `ignore-emscripten` or `ignore-wasm*`. Many now run on `wasm32-wasip1` due to being able to emit error messages, for example. Many are updated with comments as to why they can't run as well.
* The `compiletest` output matching for `wasm32-wasip1` automatically uses "match a subset" mode implemented in `compiletest`. This is because WebAssembly runtimes often add extra information on failure, such as the `unreachable` instruction in `panic!`, which isn't able to be matched against the golden output from native platforms.
* I've ported most existing `run-make` tests that use custom Node.js wrapper scripts to the new run-make-based-in-Rust infrastructure. To do this I added `wasmparser` as a dependency of `run-make-support` for the various wasm tests to use that parse wasm files. The one test that executed WebAssembly now uses `wasmtime`-the-CLI to execute the test instead. I have not ported over an exception-handling test as Wasmtime doesn't implement this yet.
* I've updated the `test` crate to print out timing information for WASI targets as it can do that (gets a previously ignored test now passing).
* The `test-various` image now builds a WASI sysroot for the WASI target and additionally downloads a fixed release of Wasmtime, currently the latest one at 18.0.2, and uses that for testing.
[Zulip thread]: https://rust-lang.zulipchat.com/#narrow/stream/131828-t-compiler/topic/Have.20wasm.20tests.20ever.20caused.20problems.20on.20CI.3F/near/424317944
|
|
Build `rustc` with 1CGU on `x86_64-pc-windows-msvc`
Distribute `x86_64-pc-windows-msvc` artifacts built with `rust.codegen-units=1`, like we already do on Linux.
1) effect on code size on `x86_64-pc-windows-msvc`: it's a 3.67% reduction on `rustc_driver.dll`
- before, [`41d97c8a5dea2731b0e56fe97cd7cb79e21cff79`](https://ci-artifacts.rust-lang.org/rustc-builds/41d97c8a5dea2731b0e56fe97cd7cb79e21cff79/rustc-nightly-x86_64-pc-windows-msvc.tar.xz): 137605632
- after, [`704aaa875e4acccc973cbe4579e66afbac425691`](https://ci-artifacts.rust-lang.org/rustc-builds/704aaa875e4acccc973cbe4579e66afbac425691/rustc-nightly-x86_64-pc-windows-msvc.tar.xz): 132551680
2) time it took on CI
- the [first `try` build](https://github.com/rust-lang-ci/rust/actions/runs/8155647651/job/22291592507) took: 1h 31m
- the [second `try` build](https://github.com/rust-lang-ci/rust/actions/runs/8157043594/job/22295790552) took: 1h 32m
3) most recent perf results:
- on a slightly noisy desktop [here](https://github.com/rust-lang/rust/pull/112267#issuecomment-1575758155)
- ChrisDenton's results [here](https://github.com/rust-lang/rust/pull/112267#issuecomment-1583295338)
Related tracking issue for build configuration: #103595
|
|
Drop testing of `wasm32-unknown-unknown` and instead only test a WASI
target which enables more debugging utilities such as printing.
|
|
This commit renames the current `wasm32-wasi-preview1-threads` target to
`wasm32-wasip1-threads`. The need for this rename is a bit unfortunate
as the previous name was chosen in an attempt to be future-compatible
with other WASI targets. Originally this target was proposed to be
`wasm32-wasi-threads`, and that's what was originally implemented in
wasi-sdk as well. After discussion though and with the plans for the
upcoming component-model target (now named `wasm32-wasip2`) the
"preview1" naming was chosen for the threads-based target. The WASI
subgroup later decided that it was time to drop the "preview"
terminology and recommends "pX" instead, hence previous PRs to add
`wasm32-wasip2` and rename `wasm32-wasi` to `wasm32-wasip1`.
So, with all that history, the "proper name" for this target is
different than its current name, so one way or another a rename is
required. This PR proposes renaming this target cold-turkey, unlike
`wasm32-wasi` which is having a long transition period to change its
name. The threads-based target is predicted to see only a fraction of
the traffic of `wasm32-wasi` due to the unstable nature of the WASI
threads proposal itself.
While I was here I updated the in-tree documentation in the target spec
file itself as most of the documentation was copied from the original
WASI target and wasn't as applicable to this target.
Also, as an aside, I can at least try to apologize for all the naming
confusion here, but this is hopefully the last WASI-related rename.
|
|
|
|
|
|
Revert back to Git-for-Windows for MinGW CI builds
Following discussion in https://github.com/rust-lang/rust/pull/121182 it was decided to revert using MSYS2 Git for mingw builds.
|
|
r=MarkSimulacrum
Subtree update cg gcc 2024 03 05
Reopening of #121390.
r? `@ghost`
|
|
|
|
Update host LLVM on x64 Linux to LLVM 18
Updates host LLVM on Linux to `18.1.0`.
|
