| Age | Commit message (Collapse) | Author | Lines |
|---|
|
bootstrap: config: fix version comparison bug
Rust requires a previous version of Rust to build, such as the current version, or the previous version. However, the version comparison logic did not take patch releases into consideration when doing the version comparison for the current branch, e.g. Rust 1.71.1 could not be built by Rust 1.71.0 because it is neither an exact version match, or the previous version.
Adjust the version comparison logic to tolerate mismatches in the patch version.
|
|
Strip unexpected debuginfo from `libLLVM.so` and `librustc_driver.so` when not requesting any debuginfo
As seen in #114175 and in [this zulip discussion](https://rust-lang.zulipchat.com/#narrow/stream/247081-t-compiler.2Fperformance/topic/Artifact.20sizes/near/379302655), there's still some small amount of debuginfo in LLVM's shared library on linux, even when not requesting it (nightly CI), coming from `libstdc++`.
```
$ readelf --debug-dump=info ~/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/libLLVM-16-rust-1.73.0-nightly.so | grep DW_TAG_compile_unit -A5 | grep DW_AT_comp_dir | cut -d ":" -f 2- | counts
101 counts
( 1) 39 (38.6%, 38.6%): (indirect string, offset: 0x7): /tmp/gcc-build/x86_64-pc-linux-gnu/libstdc++-v3/libsupc++
( 2) 38 (37.6%, 76.2%): (indirect string, offset: 0x43fb2): /tmp/gcc-build/x86_64-pc-linux-gnu/libstdc++-v3/src/c++11
( 3) 23 (22.8%, 99.0%): (indirect string, offset: 0x18ed8): /tmp/gcc-build/x86_64-pc-linux-gnu/libstdc++-v3/src/c++98
( 4) 1 ( 1.0%,100.0%): (indirect string, offset: 0x53f04): /tmp/gcc-build/x86_64-pc-linux-gnu/libstdc++-v3/src
```
Similarly, here's `librustc_driver.so` when not requesting debuginfo from either rustc or the tools (nightly CI), coming e.g. from our LLVM wrapper:
```
$ readelf --debug-dump=info ~/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/librustc_driver-e534b3a316089f5f.so | grep DW_TAG_compile_unit -A5 | grep DW_AT_comp_dir | cut -d ":" -f 2- | counts
116 counts
( 1) 34 (29.3%, 29.3%): (indirect string, offset: 0x3c11): /tmp/gcc-build/x86_64-pc-linux-gnu/libstdc++-v3/libsupc++
( 2) 32 (27.6%, 56.9%): (indirect string, offset: 0x9753c): /tmp/gcc-build/x86_64-pc-linux-gnu/libstdc++-v3/src/c++11
( 3) 25 (21.6%, 78.4%): (indirect string, offset: 0x393bd): /tmp/gcc-build/x86_64-pc-linux-gnu/libstdc++-v3/src/c++98
( 4) 23 (19.8%, 98.3%): (indirect string, offset: 0x33ed3): /cargo/registry/src/index.crates.io-6f17d22bba15001f/compiler_builtins-0.1.98
( 5) 1 ( 0.9%, 99.1%): (indirect string, offset: 0xaffff): /rustc/0d95f9132909ae7c5f2456748d0ffd1c3ba4a8e8
( 6) 1 ( 0.9%,100.0%): (indirect string, offset: 0xb604a): /tmp/gcc-build/x86_64-pc-linux-gnu/libstdc++-v3/src
```
To reduce the size of distributed artifacts, this PR strips debuginfo from the LLVM and `rustc_driver` shared libraries, when:
- no debuginfo is requested when building LLVM: `link-shared` is true, `optimize` is true and `release-debuginfo` is false
- no debuginfo is requested when building the rustc driver:
- `debuginfo-level-rustc` and `debuginfo-level-tools` are off.
- when building with a stage != 0 compiler: since this is about the distributed artifacts, there's no need to do this at other stages.
- for both: on a x64 linux host and target where `strip -g` is available and fixes the issue (I don't know how to strip debuginfo from a `.dylib` on mac). The LLVM BOLTed .so, and `librustc_driver.so` are big there, and this will help a little. Other targets/hosts can be added in the future if we want to.
#114175 did the same thing unconditionally in `opt-dist`, prior to BOLTing LLVM. But this should only be used in conjunction with the other config options mentioned above, and which `opt-dist` doesn't know about. Therefore, it makes more sense as in bootstrap when building LLVM and rustc when applicable and no debuginfo is requested.
This shouldn't interact badly with CI caching builds and artifacts, right?
---
From the other PR, `libLLVM-16-rust-1.73.0-nightly.so` prior to #114141:
- master: 173.13 MiB
- stripped debuginfo: 165.12 MiB (-8 MiB, -4.6%)
`libLLVM-16-rust-1.73.0-nightly.so` after #114141:
- master: 121.13 MiB
- stripped debuginfo: 113.12 MiB (still -8 MiB, -6.6%)
`librustc_driver.so`:
- master: 118.58 MiB
- stripped debuginfo: 106.46 MiB (-12 MiB, -10.2%)
(Results are also available in this most recent [perf run's artifact sizes](https://perf.rust-lang.org/compare.html?start=b321edd1b2d4bd00c7b4611e8f20a03ee7b77023&end=810ab570d5d27facb91806e5d9847815d9dac22a&stat=instructions%3Au&tab=artifact-size))
|
|
Rust requires a previous version of Rust to build, such as the current version, or the
previous version. However, the version comparison logic did not take patch releases
into consideration when doing the version comparison for the current branch, e.g.
Rust 1.71.1 could not be built by Rust 1.71.0 because it is neither an exact version
match, or the previous version.
Adjust the version comparison logic to tolerate mismatches in the patch version.
Signed-off-by: Ariadne Conill <ariadne@dereferenced.org>
|
|
|
|
archive
|
|
|
|
|
|
bootstrap: Clean up try_run
r? `@ozkanonur` since you reviewed `@GuillaumeGomez's` PR
i recommend reviewing commit-by-commit
|
|
This does three things:
1. Remove `forward!(Build, fn try_run())`. Having `try_run` behave differently as a free function than an associated function is confusing, and `Builder::try_run` is a very desirable name.
2. Move `test::try_run` and `run::try_run` to `Builder::try_run`. These functions are different than `Config::try_run` - they delay the failure and print it out at the end of the build.
3. Mark `Config::try_run` as deprecated to encourage people to use `Builder::try_run` instead.
|
|
`detail` and `macro` weren't adding any info.
|
|
various download-rustc fixes
separated out from https://github.com/rust-lang/rust/pull/112143 because it keeps getting stuck in limbo.
best reviewed commit-by-commit
|
|
|
|
Signed-off-by: hi-rustin <rustin.liu@gmail.com>
|
|
by using `rustc-builds-alt` if download-rustc is set
this also changes the download code to use a separate build/cache/ directory and .rustc-stamp stamp file depending on whether assertions are enabled.
|
|
|
|
Previously, it would always treat download-rustc as set to false, which made bootstrap issues with
download-rustc hard to debug.
|
|
Signed-off-by: hi-rustin <rustin.liu@gmail.com>
|
|
Signed-off-by: hi-rustin <rustin.liu@gmail.com>
|
|
|
|
|
|
This has real differences in the effective debuginfo: in particular, it omits the module-level information and breaks perf.
Allow passing `line-tables-only` directly in config.toml instead.
|
|
bootstrap: Disallow `--exclude test::std`
Use the top-level Kind to determine whether Steps are excluded.
Previously, this would use the `Kind` passed to `--exclude` (and not do any filtering at all if no kind was passed).
That meant that `x test linkchecker --exclude std` would fail - you had to explicitly say `--exclude test::std`.
Change bootstrap to use the top-level Kind instead, which does the right thing automatically.
Note that this breaks things like `x test --exclude doc::std`, but I'm not sure why you'd ever want to do that.
There's a lot of churn here, but the 1-line change in the first commit is the actual behavior change, the rest is just cleanup.
Fixes https://github.com/rust-lang/rust/issues/103201. Note that this effectively reverts most of https://github.com/rust-lang/rust/pull/91965.
cc `@pietroalbini`
|
|
|
|
|
|
|
|
Signed-off-by: ozkanonur <work@onurozkan.dev>
|
|
chenyukang:yukang-fix-110067-version-issue, r=jyn514"
This reverts commit 9267843e72c8692f934c2ef5505c8a7496c0f06e, reversing
changes made to e52fbff5e8c2de7ba085ef84d7d50500b695d521.
This breaks our ability to bump the src/version where we're bootstrapping with an older compiler
than usual (according to version number). It's not clear whether the intended use case has a clean
solution given this constraint, so reverting for now - we can reland with a fix of some kind implemented.
|
|
Override config.toml options from command line
https://rust-lang.zulipchat.com/#narrow/stream/326414-t-infra.2Fbootstrap/topic/Running.20tests.20on.20precompiled.20rustc/near/357763280
cc `@jyn514`
|
|
|
|
|
|
|
|
|
|
|
|
r=albertlarsan68
support set `rpath` option for each target independently
Currently the `rpath` option is a global config and it's effect on all targets.
But sometimes when developers edit the rustc code and try to release rust toolchains themselves, they may not want to add `rpath` in all targets to avoid dynamically linked shared object library privilege escalation attack.
This PR supports set `rpath` option for each target independently .
Common developers are not aware of the existence of this configuration option and do not affect the existing development process. This configuration option takes effect only after developers explicitly sets .
r? ``@albertlarsan68``
|
|
Migrate bootstrap to Clap-based argument parsing
Supercedes #108083
I chose to re-do the work rather than rebase the onto the large changes since the original PR. If it's preferred I can instead force-push the original PR to this version.
cc `@jyn514` `@albertlarsan68`
|
|
|
|
|
|
Previously, this was only passed to miri and compiletest. Extended it to
all other tests and binaries as well.
|
|
|
|
Previously, this would give an error because the shell script had a
different working directory:
```
test: basic_install
$ sh /home/jyn/src/rust/src/tools/rust-installer/gen-installer.sh --image-dir=/home/jyn/src/rust/src/tools/rust-installer/test/image1 --work-dir=/home/jyn/src/rust/build/x86_64-unknown-linux-gnu/test/rust-installer/workdir --output-dir=/home/jyn/src/rust/build/x86_64-unknown-linux-gnu/test/rust-installer/outdir
/home/jyn/src/rust/src/tools/rust-installer/gen-installer.sh: 15: ../rust3/build/host/stage2-tools-bin/cargo: not found
TEST FAILED!
```
|
|
|
|
ignore_git → omit_git_hash
Fixes #110020
|
|
|
|
r=Mark-Simulacrum
Download beta compiler toolchain in bootstrap if it doesn't yet exist
Blocker for #107812 and #99989
See: https://github.com/rust-lang/rust/pull/107812#discussion_r1143492202
r? `@jyn514`
|
|
This is needed for when the shell scripts bypass python altogether and run the downloaded
bootstrap directly. Changes are mainly provided from @jyn514, I just fixed the review notes.
Signed-off-by: ozkanonur <work@onurozkan.dev>
|
|
|
|
Renamed 'native.rs' to 'llvm.rs', also moved `TestHelpers` to `test.rs`.Replaced all the `native.rs` ocurrences at `src/bootstrap` files to `llvm.rs`
|
|
Include executed tests in the build metrics (and use a custom test display impl)
The main goal of this PR is to include all tests executed in CI inside the build metrics JSON files. I need this for Ferrocene, and `@Mark-Simulacrum` expressed desire to have this as well to ensure all tests are executed at least once somewhere in CI.
Unfortunately implementing this required rewriting inside of bootstrap all of the code to render the test output to console. libtest supports outputting JSON instead of raw text, which we can indeed use to populate the build metrics. Doing that suppresses the console output though, and compared to rustc and Cargo the console output is not included as a JSON field.
Because of that, this PR had to reimplement both the "pretty" format (one test per line, with `rust.verbose-tests = true`), and the "terse" format (the wall of dots, with `rust.verbose-tests = false`). The current implementation should have the exact same output as libtest, except for the benchmark output. libtest's benchmark output is broken in the "terse" format, so since that's our default I slightly improved how it's rendered.
Also, to bring parity with libtest I had to introduce support for coloring output from bootstrap, using the same dependencies `annotate-snippets` uses. It's now possible to use `builder.color_for_stdout(Color::Red, "text")` and `builder.color_for_stderr(Color::Green, "text")` across all of bootstrap, automatically respecting the `--color` flag and whether the stream is a terminal or not.
I recommend reviewing the PR commit-by-commit.
r? `@Mark-Simulacrum`
|
|
Add `dist.compression-profile` option to control compression speed
PR #108534 reduced the size of compressed archives, but (as expected) it also resulted in way longer compression times and memory usage during compression.
It's desirable to keep status quo (smaller archives but more CI usage), but it should also be configurable so that downstream users don't have to waste that much time on CI. As a data point, this resulted in doubling the time of Ferrocene's dist jobs, and required us to increase the RAM allocation for one of such jobs.
This PR adds a new `config.toml` setting, `dist.compression-profile`. The values can be:
* `fast`: equivalent to the gzip and xz preset of "1"
* `balanced`: equivalent to the gzip and xz preset of "6" (the CLI defaults as far as I'm aware)
* `best`: equivalent to the gzip present of "9", and our custom xz profile
The default has also been moved back to `balanced`, to try and avoid the compression time regression for downstream users. I don't feel too strongly on the default, and I'm open to changing it.
Also, for the `best` profile the XZ settings do not match the "9" preset used by the CLI, and it might be confusing. Should we create a `custom-rustc-ci`/`ultra` profile for that?
r? ``@Mark-Simulacrum``
|
|
|
