| Age | Commit message (Collapse) | Author | Lines |
|---|
|
|
|
Signed-off-by: onur-ozkan <work@onurozkan.dev>
|
|
Make sure lld is enabled for dist profile unless it is explicitly disabled.
Signed-off-by: onur-ozkan <work@onurozkan.dev>
|
|
make cargo submodule optional
Right now, we fetch the cargo submodule no matter what, even if the command we are running doesn't need it (e.g., `x build compiler library`). This PR changes that to only fetch the cargo submodule when it's necessary.
For more context, see the zulip thread: https://rust-lang.zulipchat.com/#narrow/stream/326414-t-infra.2Fbootstrap/topic/Why.20is.20cargo.20always.20checked.20out.3F
|
|
Currently bootstrap doesn't use any inner paths from rust-analyzer and
bootstrap with `ShouldRun::create_or_deps`.
Signed-off-by: onur-ozkan <work@onurozkan.dev>
|
|
rustdoc: use current stage if download-rustc enabled
When using download-rustc, using stage 1 rustdoc results in the wrong librustc_driver being used.
```sh
$ ./build/host/stage1/bin/rustdoc --version
./build/host/stage1/bin/rustdoc: error while loading shared libraries: librustc_driver-7ff02ed05016d515.so: cannot open shared object file: No such file or directory
```
This change fixes that by not cutting the stage if download-rustc is enabled.
|
|
|
|
|
|
Fixes for 32-bit SPARC on Linux
This PR fixes a number of issues which previously prevented `rustc` from being built
successfully for 32-bit SPARC using the `sparc-unknown-linux-gnu` triplet.
In particular, it adds linking against `libatomic` where necessary, uses portable `AtomicU64`
for `rustc_data_structures` and rewrites the spec for `sparc_unknown_linux_gnu` to use
`TargetOptions` and replaces the previously used `-mv8plus` with the more portable
`-mcpu=v9 -m32`.
To make `rustc` build successfully, support for 32-bit SPARC needs to be added to the `object`
crate as well as the `nix` crate which I will be sending out later as well.
r? nagisa
|
|
patch `rust-lld` and `ld.lld` on NixOS
When `rustc` uses its self-contained lld, we also need to patch `rust-lld` and `ld.lld`.
The `rpath` for `rust-lld` is `$ORIGIN/../../../:$ORIGIN/../lib`, so I use `--add-rpath` instead of `--set-rpath`, which should be easier to maintain.
I also changed `src/bootstrap/src/core/download.rs`, even this doesn't fix any known issues.
For the `lld-wrapper.sh` of lld, refer to: https://github.com/rust-lang/rustc-dev-guide/pull/1999.
|
|
Currently rustdoc breaks the build cache (due to having different rustflags) when building
rustdoc before building another tool (e.g., `x test miri && x test rustdoc && x test miri`).
This change fixes that by moving `on-broken-pipe` into `prepare_cargo_tool` so it is
set for all tools.
Signed-off-by: onur-ozkan <work@onurozkan.dev>
|
|
Signed-off-by: onur-ozkan <work@onurozkan.dev>
|
|
Signed-off-by: onur-ozkan <work@onurozkan.dev>
|
|
Signed-off-by: onur-ozkan <work@onurozkan.dev>
|
|
Signed-off-by: onur-ozkan <work@onurozkan.dev>
|
|
While at it, order the list of architectures alphabetically.
|
|
Promote loongarch64-unknown-linux-musl to Tier 2 with host tools
MCP: https://github.com/rust-lang/compiler-team/issues/753
Tracking issue: https://github.com/rust-lang/rust/issues/122592
try-job: dist-loongarch64-musl
|
|
less bootstrap warnings
This is how the build logs looks like currently:
```sh
$ x build
Building bootstrap
Compiling bootstrap v0.0.0 (/home/nimda/devspace/onur-ozkan/rust/src/bootstrap)
Finished `dev` profile [unoptimized] target(s) in 3.43s
WARNING: no codegen-backends config matched the requested path to build a codegen backend. HELP: add backend to codegen-backends in config.toml.
WARNING: creating symbolic link `/home/nimda/devspace/.other/rustc-builds/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/rustc-src/rust` to `/home/nimda/devspace/onur-ozkan/rust` failed with File exists (os
error 17)
Creating a sysroot for stage2 compiler (use `rustup toolchain link 'name' build/host/stage2`)
WARNING: creating symbolic link `/home/nimda/devspace/.other/rustc-builds/build/x86_64-unknown-linux-gnu/ci-rustc-sysroot/lib/rustlib/rustc-src/rust` to `/home/nimda/devspace/onur-ozkan/rust` failed with File e
xists (os error 17)
Building tool rustdoc (stage1 -> stage2, x86_64-unknown-linux-gnu)
Compiling rustdoc v0.0.0 (/home/nimda/devspace/onur-ozkan/rust/src/librustdoc)
Compiling rustdoc-tool v0.0.0 (/home/nimda/devspace/onur-ozkan/rust/src/tools/rustdoc)
Finished `release` profile [optimized + debuginfo] target(s) in 13.57s
Build completed successfully in 0:00:17
```
This PR removes artifact linking warnings and only shows the codegen-backend warning if explicitly called or during Dist or Install steps.
|
|
tidy: skip submodules if not present for non-CI environments
Right now tidy requires rustc-perf to be fetched as it checks its license, but this doesn't make sense for most contributors since rustc-perf is a dist-specific tool and its license will not change frequently, especially during compiler development. This PR makes tidy to skip rustc-perf if it's not fetched and if it's not running in CI.
Applies https://github.com/rust-lang/rust/pull/126225#issuecomment-2158143674 and reverts #126225.
|
|
MCP: https://github.com/rust-lang/compiler-team/issues/753
|
|
r=onur-ozkan
Support absolute source paths in bootstrap
Fixes https://github.com/rust-lang/rust/issues/126765
`x test [PATHS]` should work when each path
1. Is the name of a build step, such as `tidy` in `x test tidy` or
2. Points to an existing file that is a descendant of the builder's source directory (root of this repository).
|
|
|
|
|
|
Add a `x perf` command for integrating bootstrap with `rustc-perf`
This PR adds a new `x perf` command to bootstrap. The idea is to let rustc developers profile (`profile_local`) and benchmark (`bench_local`) a stage1/stage2 compiler directly from within `rust`.
Before, if you wanted to use `rustc-perf`, you had to clone it, set it up, copy the `rustc` sysroot after every change to `rust` etc. This is an attempt to automate that.
I opened this PR mostly for discussion. My idea is to offer an interface that looks something like this (a random sample of commands):
```bash
x perf --stage 2 profile eprintln
x perf --stage1 profile cachegrind
x perf benchmark --id baseline
x perf benchmark --id after-edit
x perf cmp baseline after-edit
```
In this PR, I'd like to only implement the simplest case (`profile_local (eprintln)`), because that only requires a single sysroot (you don't compare anything), and it's relatively easy to set up. Also, I'd like to avoid forcing developers to deal with the rustc-perf UI, so more complex use-cases (like benchmarking two sysroots and comparing the results) should probably wait for https://github.com/rust-lang/rustc-perf/issues/1734 (which is hopefully coming along soon-ish).
I'm not sure if it's better to do this in bootstrap directly, or if I should create some shim tool that will receive a `rustc` sysroot, and offer a simplified CLI on top of `rustc-perf`.
## Why is a separate CLI needed?
We definitely need to add some support to bootstrap to automate preparing `rustc-perf` and the `rustc` sysroot, but in theory after that we could just let people invoke `rustc-perf` manually. While that is definitely possible, you'd need to manually figure out where is your sysroot located, which seems annoying to me. The `rustc-perf` CLI is also relatively complex, and for this use-case it makes sense to only use a subset of it. So I thought that it would be better to offer a simplified interface on top of it that would make life easier for contributors. But maybe it's not worth it.
CC `@onur-ozkan`
|
|
Bootstrap command refactoring: refactor `BootstrapCommand` (step 1)
This PR is a first step towards https://rust-lang.zulipchat.com/#narrow/stream/326414-t-infra.2Fbootstrap.
It refactors `BoostrapCommand` to get it closer to a state where it is an actual command wrapper that can be routed through a central place of command execution, and also to make the distinction between printing output vs handling output programatically clearer (since now it's a mess).
The existing usages of `BootstrapCommand` are complicated primarily because of different ways of handling output. There are commands that:
1) Want to eagerly print stdout/stderr of the executed command, plus print an error message if the command fails (output mode `PrintAll`). Note that this error message attempts to print stdout/stderr of the command when `-v` is enabled, but that will always be empty, since this mode uses `.status()` and not `.output()`.
2) Want to eagerly print stdout/stderr of the executed command, but do not print any additional error message if it fails (output mode `PrintOutput`)
3) Want to capture stdout/stderr of the executed command, but print an error message if it fails (output mode `PrintFailure`). This means that the user wants to either ignore the output or handle it programatically, but that's not obvious from the name.
The difference between 1) and 2) (unless explicitly specified) is determined dynamically based on the bootstrap verbosity level.
It is very difficult for me to wrap my head around all these modes. I think that in a future PR, we should split these axes into e.g. this:
1) Do I want to handle the output programmatically or print it to the terminal? This should be a separate axis, true/false. (Note that "hiding the output" essentially just means saying that I handle it programmatically, and then I ignore the output).
2) Do I want to print a message if the command fails? Yes/No/Based on verbosity (which would be the default).
Then there is also the failure mode, but that is relatively simple to handle, the command execution will just shutdown bootstrap (either eagerly or late) when the command fails.
Note that this is just a first refactoring steps, there are a lot of other things to be done, so some things might not look "final" yet. The next steps are (not necessarily in this order):
- Remove `run` and `run_cmd` and implement everything in terms of `run_tracked` and rename `run_tracked` to `run`
- Implement the refactoring specified above (change how output modes work)
- Modify `BootstrapCmd` so that it stores `Command` instead of `&mut Command` and remove all the annoying `BootstrapCmd::from` by changing `Command::new` to `BootstrapCmd::new`
- Refactor the rest of command executions not currently using `BootstrapCmd` that can access Builder to use the correct output and failure modes. This will include passing Builder to additional places.
- Handle the most complex cases, such as output streaming. That will probably need to be handled separately.
- Refactor the rest of commands that cannot access builder (e.g. `Config::parse`) by introducing a new command context that will be passed to these places, and then stored in `Builder`. Move certain fields (such as `fail_fast`) from `Builder` to the command context.
- Handle the co-operation of `Builder`, `Build`, `Config` and command context. There are some fields and logic used during command execution that are distributed amongst `Builder/Build/Config`, so it will require some refactoring to make it work if the execution will happen on a separate place (in the command context).
- Refactor logging of commands, so that it is either logged to a file or printed in a nice hierarchical way that cooperates with the `Step` debug hierarchical output.
- Implement profiling of commands (add command durations to the command log, print a log of slowest commands and their execution counts at the end of bootstrap execution, perhaps store command executions to `metrics.json`).
- Implement caching of commands.
- Implement testing of commands through snapshot tests/mocking.
Best reviewed commit by commit.
r? ``@onur-ozkan``
|
|
|
|
|
|
Replace sort implementations
This PR replaces the sort implementations with tailor-made ones that strike a balance of run-time, compile-time and binary-size, yielding run-time and compile-time improvements. Regressing binary-size for `slice::sort` while improving it for `slice::sort_unstable`. All while upholding the existing soft and hard safety guarantees, and even extending the soft guarantees, detecting strict weak ordering violations with a high chance and reporting it to users via a panic.
* `slice::sort` -> driftsort [design document](https://github.com/Voultapher/sort-research-rs/blob/main/writeup/driftsort_introduction/text.md), includes detailed benchmarks and analysis.
* `slice::sort_unstable` -> ipnsort [design document](https://github.com/Voultapher/sort-research-rs/blob/main/writeup/ipnsort_introduction/text.md), includes detailed benchmarks and analysis.
#### Why should we change the sort implementations?
In the [2023 Rust survey](https://blog.rust-lang.org/2024/02/19/2023-Rust-Annual-Survey-2023-results.html#challenges), one of the questions was: "In your opinion, how should work on the following aspects of Rust be prioritized?". The second place was "Runtime performance" and the third one "Compile Times". This PR aims to improve both.
#### Why is this one big PR and not multiple?
* The current documentation gives performance recommendations for `slice::sort` and `slice::sort_unstable`. If for example only one of them were to be changed, this advice would be misleading for some Rust versions. By replacing them atomically, the advice remains largely unchanged, and users don't have to change their code.
* driftsort and ipnsort share a substantial part of their implementations.
* The implementation of `select_nth_unstable` uses internals of `slice::sort_unstable`, which makes it impractical to split changes.
---
This PR is a collaboration with `@orlp.`
|
|
The previous message omits which of the dozens of tools called tidy is
meant. And it's written in a way that one can easily miss the *not*,
thinking it reads "Note that `tidy` is the in-tree `src/tools/tidy` but
needs to be installed". The error message should hopefully help future
contributors.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
This function both handles error printing and early/late failures, but it also always returns the actual output of the command
|
|
When using download-rustc, using stage 1 rustdoc results in the wrong librustc_driver being used.
```sh
$ ./build/host/stage1/bin/rustdoc --version
./build/host/stage1/bin/rustdoc: error while loading shared libraries: librustc_driver-7ff02ed05016d515.so: cannot open shared object file: No such file or directory
```
This change fixes that by not cutting the stage if download-rustc is enabled.
Signed-off-by: onur-ozkan <work@onurozkan.dev>
|
|
People are having trouble when they don't want to build `lld` for their custom
distribution tarballs even with `lld = false` in their config.toml. This is because
it is not controlled by `lld_enabled` flag. This change ensures that `llvm:Lld`
is controlled by lld configuration.
Signed-off-by: onur-ozkan <work@onurozkan.dev>
|
|
|
|
|
|
override user defined channel when using precompiled rustc
We need to override `rust.channel` if it's manually specified when using the CI rustc. This is because if the compiler uses a different channel than the one specified in config.toml, tests may fail due to using a different channel than the one used by the compiler during tests.
For more context, see https://github.com/rust-lang/rust/pull/122709#issuecomment-2165246281.
|
|
|
|
If `codegen-backends` is missing "cranelift" and "gcc" (which is common),
bootstrap will now only show this warning during `dist` and `install` steps,
or if codegen-backends was explicitly called for build.
Signed-off-by: onur-ozkan <work@onurozkan.dev>
|
|
When using `symlink_dir`, it first removes the existing link with `remove_dir`.
However, if the path isn't a link and contains files, `remove_dir` fails
with "DirectoryNotEmpty", which causes the symbolic linking to fail as well.
We have this problem on linking 'rustlib/rust' because it contains files as an
actual directory.
Signed-off-by: onur-ozkan <work@onurozkan.dev>
|
|
|
|
This is already handled at the config parsing level, so we can simplify it.
Signed-off-by: onur-ozkan <work@onurozkan.dev>
|
|
Signed-off-by: onur-ozkan <work@onurozkan.dev>
|
|
Signed-off-by: onur-ozkan <work@onurozkan.dev>
|
|
bump few deps
This bump deps to cut dupes:
Updating html5ever 0.27
Updating derive_more 0.99.18
Updating crossbeam-deque v0.8.5
Updating crossbeam-epoch v0.9.18
Updating crossbeam-utils v0.8.20
Updating junction v1.1.0
No interesting changes in changelogs.
|
|
unify git command preperation
Due to https://github.com/rust-lang/rust/issues/125954, we had to modify git invocations with certain flags in https://github.com/rust-lang/rust/pull/126255. However, because there are so many instances of `Command::new("git")` in bootstrap, it is difficult to apply these solutions to all of them.
This PR creates a helper function that unifies the git usage in bootstrap. Meaning, whenever special flags or hacks are needed, we can apply them to this single function which makes things much simpler for the bootstrap team.
|
