| Age | Commit message (Collapse) | Author | Lines |
|---|
|
|
|
Fixes https://github.com/rust-lang/rust/issues/99083
|
|
|
|
Allow using `download-ci-llvm = true` outside the git checkout
`@bjorn3` noticed that this is already allowed today when download-llvm is disabled, but breaks with it enabled:
```
$ ./rust2/x.py build
fatal: not a git repository (or any of the parent directories): .git
thread 'main' panicked at 'command did not execute successfully: "git" "rev-list" "--author=bors@rust-lang.org" "-n1" "--first-parent" "HEAD" "--" "/home/jnelson/rust-lang/rust2/src/llvm-project" "/home/jnelson/rust-lang/rust2/src/bootstrap/download-ci-llvm-stamp" "/home/jnelson/rust-lang/rust2/src/version"
expected success, got: exit status: 128', src/bootstrap/native.rs:134:20
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
```
Support it too for consistency. It's unclear to me when anyone would need to use this, but `@bjorn3`
feels we should support it, and it's not much additional effort to get it working.
|
|
@bjorn3 noticed that this is already allowed today when download-llvm is disabled, but breaks with it enabled:
```
$ ./rust2/x.py build
fatal: not a git repository (or any of the parent directories): .git
thread 'main' panicked at 'command did not execute successfully: "git" "rev-list" "--author=bors@rust-lang.org" "-n1" "--first-parent" "HEAD" "--" "/home/jnelson/rust-lang/rust2/src/llvm-project" "/home/jnelson/rust-lang/rust2/src/bootstrap/download-ci-llvm-stamp" "/home/jnelson/rust-lang/rust2/src/version"
expected success, got: exit status: 128', src/bootstrap/native.rs:134:20
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
```
Support it too for consistency. It's unclear to me when anyone would need to use this, but @bjorn3
feels we should support it, and it's not much additional effort to get it working.
This also updates a bunch of other git commands that were similarly depending on the current directory.
|
|
implement detail_exit but I'm not sure it is right.
not create new file and write detail exit in lib.rs
replace std::process::exit to detail_exit
that is not related to code runnning.
remove pub
|
|
Move locking from bootstrap.py to rust bootstrap, using fd-lock
Helps with https://github.com/rust-lang/rust/issues/94829.
|
|
Allow macOS to build LLVM as shared library
Inspired by how [homebrew](https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/llvm.rb) builds and distributes llvm, here we manually create a symlink with a versioned dylib path to make `llvm-config` work properly. Note, the resulting `rustc` executable and `librustc_driver-<hash>.dylib` still links to the un-versioned `libLLVM.dylib` as expected when distributed in the final output. I have confirmed this by checking `otool -L` on both binaries.
After the change, enabling `llvm.link-shared` and `llvm.thin-lto` will be possible on macOS.
|
|
|
|
Create symlinks to workaround file missing error in llvm-config
|
|
|
|
These submodules were previously updated in python because Cargo gives a hard error if toml files
are missing from the workspace:
```
error: failed to load manifest for workspace member `/home/jnelson/rust-lang/rust/src/tools/rls`
Caused by:
failed to read `/home/jnelson/rust-lang/rust/src/tools/rls/Cargo.toml`
Caused by:
No such file or directory (os error 2)
failed to run: /home/jnelson/rust-lang/rust/build/x86_64-unknown-linux-gnu/stage0/bin/cargo build --manifest-path /home/jnelson/rust-lang/rust/src/bootstrap/Cargo.toml
```
However, bootstrap doesn't actually need to be part of the workspace.
Remove it so we can move submodule handling fully to Rust, avoiding duplicate code between Rust and Python.
Note that this does break `cargo run`; it has to be `cd src/bootstrap && cargo run` now.
Given that we're planning to make the main entrypoint a shell script (or rust binary),
I think this is a good tradeoff for reduced complexity in bootstrap.py.
|
|
Make "Assemble stage1 compiler" orders of magnitude faster (take 2)
This used to take upwards of 5 seconds for me locally. I found that the culprit was copying the downloaded LLVM shared object:
```
[22:28:03] Install "/home/jnelson/rust-lang/rust/build/x86_64-unknown-linux-gnu/ci-llvm/lib/libLLVM-14-rust-1.62.0-nightly.so" to "/home/jnelson/rust-lang/rust/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/libLLVM-14-rust-1.62.0-nightly.so"
[22:28:09] c Sysroot { compiler: Compiler { stage: 1, host: x86_64-unknown-linux-gnu(x86_64-unknown-linux-gnu) } }
```
It turned out that `install()` used full copies unconditionally. Change it to try using a hard-link before falling back to copying.
- Panic if we generate a symbolic link in a tarball
- Change install to use copy internally, like in my previous PR
- Change copy to dereference symbolic links, which avoids the previous regression in #96803.
I also took the liberty of fixing `x dist llvm-tools` to work even if you don't call `x build` previously.
|
|
This used to take upwards of 5 seconds for me locally. I found that the
culprit was copying the downloaded LLVM shared object:
```
[22:28:03] Install "/home/jnelson/rust-lang/rust/build/x86_64-unknown-linux-gnu/ci-llvm/lib/libLLVM-14-rust-1.62.0-nightly.so" to "/home/jnelson/rust-lang/rust/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/libLLVM-14-rust-1.62.0-nightly.so"
[22:28:09] c Sysroot { compiler: Compiler { stage: 1, host: x86_64-unknown-linux-gnu(x86_64-unknown-linux-gnu) } }
```
It turned out that `install()` used full copies unconditionally. Change
it to use `copy()` internally, which uses hard links instead when
available.
Note that this has a change in behavior: Installing a file will also
change permissions on the source, not just the destination, if hard
links are used.
To avoid changing the behavior on symlinks for existing code, I
introduce a new function `copy_internal` which only dereferences
symlinks when told to do so.
|
|
Pass all paths to `Step::run` at once when using `ShouldRun::krate`
Helps with https://github.com/rust-lang/rust/pull/95503. The goal is to run `cargo test -p rustc_data_structures -p rustc_lint_defs` instead of `cargo test -p rustc_data_structures; cargo test -p rustc_lint_defs`, which should both recompile less and avoid replaying cached warnings.
This was surprisingly complicated. The main changes are:
1. Invert the order of iteration in `StepDescription::run`.
Previously, it did something like:
```python
for path in paths:
for (step, should_run) in should_runs:
if let Some(set) = should_run.pathset_for_path(path):
step.run(builder, set)
```
That worked ok for individual paths, but didn't allow passing more than one path at a time to `Step::run`
(since `pathset_for_paths` only had one path available to it).
Change it to instead look at the intersection of `paths` and `should_run.paths`:
```python
for (step, should_run) in should_runs:
if let Some(set) = should_run.pathset_for_paths(paths):
step.run(builder, set)
```
2. Change `pathset_for_path` to take multiple pathsets.
The goal is to avoid `x test library/alloc` testing *all* library crates, instead of just alloc.
The changes here are similarly subtle, to use the intersection between the paths rather than all
paths in `should_run.paths`. I added a test for the behavior to try and make it more clear.
Note that we use pathsets instead of just paths to allow for sets with multiple aliases (*cough* `all_krates` *cough*).
See the documentation added in the next commit for more detail.
3. Change `StepDescription::run` to explicitly handle 0 paths.
Before this was implicitly handled by the `for` loop, which just didn't excute when there were no paths.
Now it needs a check, to avoid trying to run all steps (this is a problem for steps that use `default_condition`).
4. Change `RunDescription` to have a list of pathsets, rather than a single path.
5. Remove paths as they're matched
This allows checking at the end that no invalid paths are left over.
Note that if two steps matched the same path, this will no longer run both;
but that's a bug anyway.
6. Handle suite paths separately from regular sets.
Running multiple suite paths at once instead of in separate `make_run` invocations is both tricky and not particularly useful.
The respective test Steps already handle this by introspecting the original paths.
Avoid having to deal with it by moving suite handling into a seperate loop than `PathSet::Set` checks.
`@rustbot` label +A-rustbuild
|
|
This was surprisingly complicated. The main changes are:
1. Invert the order of iteration in `StepDescription::run`.
Previously, it did something like:
```python
for path in paths:
for (step, should_run) in should_runs:
if let Some(set) = should_run.pathset_for_path(path):
step.run(builder, set)
```
That worked ok for individual paths, but didn't allow passing more than one path at a time to `Step::run`
(since `pathset_for_paths` only had one path available to it).
Change it to instead look at the intersection of `paths` and `should_run.paths`:
```python
for (step, should_run) in should_runs:
if let Some(set) = should_run.pathset_for_paths(paths):
step.run(builder, set)
```
2. Change `pathset_for_path` to take multiple pathsets.
The goal is to avoid `x test library/alloc` testing *all* library crates, instead of just alloc.
The changes here are similarly subtle, to use the intersection between the paths rather than all
paths in `should_run.paths`. I added a test for the behavior to try and make it more clear.
Note that we use pathsets instead of just paths to allow for sets with multiple aliases (*cough* `all_krates` *cough*).
See the documentation added in the next commit for more detail.
3. Change `StepDescription::run` to explicitly handle 0 paths.
Before this was implicitly handled by the `for` loop, which just didn't excute when there were no paths.
Now it needs a check, to avoid trying to run all steps (this is a problem for steps that use `default_condition`).
4. Change `RunDescription` to have a list of pathsets, rather than a single path.
5. Remove paths as they're matched
This allows checking at the end that no invalid paths are left over.
Note that if two steps matched the same path, this will no longer run both;
but that's a bug anyway.
6. Handle suite paths separately from regular sets.
Running multiple suite paths at once instead of in separate `make_run` invocations is both tricky and not particularly useful.
The respective test Steps already handle this by introspecting the original paths.
Avoid having to deal with it by moving suite handling into a seperate loop than `PathSet::Set` checks.
|
|
Allow configuring where artifacts are downloaded from
Bootstrap has support for downloading prebuilt LLVM and rustc artifacts to speed up local builds, but that currently works only for users working on `rust-lang/rust`. Forks of the repository (for example Ferrocene) might have different URLs to download artifacts from, or might use a different email address on merge commits, breaking both LLVM and rustc artifact downloads.
This PR refactors bootstrap to load the download URLs and other constants from `src/stage0.json`, allowing downstream forks to tweak those values. It also future-proofs the download code to easily allow forks to add their own custom protocols (like `s3://`).
This PR is best reviewed commit-by-commit.
|
|
|
|
rustc_session"
This reverts commit 2d854f9c340df887e30896f49270ae81feb3e227.
|
|
This simplifies the arguments to `download_component` in config.rs.
It also moves stage0.json metadata handling to `Build::new`, making it easier to download the stage0
compiler in rustbuild later if necessary.
|
|
|
|
Add build metrics to rustbuild
This PR adds a new module of rustbuild, `ci_profiler`, whose job is to gather as much information as possible about the CI build as possible and store it in a JSON file uploaded to `ci-artifacts`. Right now for each step it collects:
* Type name and debug representation of the `Step` object.
* Duration of the step (excluding child steps).
* Systemwide CPU stats for the duration of the step (both single core and all cores).
* Which child steps were executed.
This is capable of replacing both the scripts to collect CPU stats and the `[TIMING]` lines in build logs (not yet removed, until we port our tooling to use the CI profiler). The format is also extensible to be able in the future to collect more information.
r? `@Mark-Simulacrum`
|
|
r=Mark-Simulacrum
Print stderr consistently
Solves https://github.com/rust-lang/rust/issues/96712
I tried to follow what I perceived as the general consensus for error messages in boostrap i.e messages that were ..
* resulting from an Err(...) =>
* literally called as "Error: ...."
* by the end of the block scope forced to run a panic! or process::exit with a guaranteed non-zero error code.
|
|
|
|
|
|
I don't know why anyone would turn this off; doing so makes builds much slower (nearly a 60x slowdown according to #49057).
Remove the option to do so, which makes bootstrap a little easier to maintain.
Bootstrap continues to allow you to manage submodules manually by setting `submodules = false`.
|
|
boostrap.py use curl by default
Fixes #61611
|
|
|
|
Make "Assemble stage1 compiler" orders of magnitude faster
This used to take upwards of 5 seconds for me locally. I found that the culprit was copying the downloaded LLVM shared object:
```
[22:28:03] Install "/home/jnelson/rust-lang/rust/build/x86_64-unknown-linux-gnu/ci-llvm/lib/libLLVM-14-rust-1.62.0-nightly.so" to "/home/jnelson/rust-lang/rust/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/libLLVM-14-rust-1.62.0-nightly.so"
[22:28:09] c Sysroot { compiler: Compiler { stage: 1, host: x86_64-unknown-linux-gnu(x86_64-unknown-linux-gnu) } }
```
It turned out that `install()` used full copies unconditionally. Change it to try using a hard-link before falling back to copying.
|
|
This used to take upwards of 5 seconds for me locally. I found that the culprit was copying the downloaded LLVM shared object:
```
[22:28:03] Install "/home/jnelson/rust-lang/rust/build/x86_64-unknown-linux-gnu/ci-llvm/lib/libLLVM-14-rust-1.62.0-nightly.so" to "/home/jnelson/rust-lang/rust/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/libLLVM-14-rust-1.62.0-nightly.so"
[22:28:09] c Sysroot { compiler: Compiler { stage: 1, host: x86_64-unknown-linux-gnu(x86_64-unknown-linux-gnu) } }
```
It turned out that `install()` used full copies unconditionally. Change it to try using a hard-link before falling back to copying.
|
|
Initialize rust-analyzer submodule on bootstrap
Fixes #96456
|
|
This tool will generate a JSON file with statistics about each
individual step to disk. It will be used in rust-lang/rust's CI to
replace the mix of scripts and log scraping we currently have to gather
this data.
|
|
|
|
|
|
|
|
This attempts to keep the logic as close to the original python as possible.
`probably_large` has been removed, since it was always `True`, and UTF-8 paths are no longer supported when patching files for NixOS.
I can readd UTF-8 support if desired.
Note that this required making `llvm_link_shared` computed on-demand,
since we don't know whether it will be static or dynamic until we download LLVM from CI.
|
|
Always use system `python3` on MacOS
This PR includes 2 changes:
1. Always use the system Python found at `/usr/bin/python3` on MacOS
2. Removes the hard requirement on having `python` in your system path if you didn't specify alternatives. The proposed change will instead attempt to find and use in order: `python` -> `python3` -> `python2`. This change isn't strictly necessary but without any change to this check, the original issue inspiring this change will still exist.
Fixes #95204
r? ```@jyn514```
|
|
|
|
Make it possible to run `cargo test` for bootstrap
Note that this only runs bootstrap's self-tests, not compiler or library tests.
Helps with https://github.com/rust-lang/rust/issues/94829.
|
|
Fix `x doc compiler/rustc`
This also has a few cleanups to `doc.rs`. The last two commits I don't care about, but the first commit I'd like to keep - it will be very useful for https://github.com/rust-lang/rust/issues/44293.
Fixes https://github.com/rust-lang/rust/issues/95447.
|
|
Note that this only runs bootstrap's self-tests, not compiler or library tests.
|
|
of calculating them after the fact
|
|
Fix `cargo run tidy`
When I implemented rust-only bootstrapping in https://github.com/rust-lang/rust/pull/92260,
I neglected to test stage0 tools - it turns out they were broken because
they couldn't find the sysroot of the initial bootstrap compiler.
This fixes stage0 tools by using `rustc --print sysroot` instead of assuming rustc is already in a
sysroot and hard-coding the relative directory.
Fixes https://github.com/rust-lang/rust/issues/94797 (properly, without having to change rustup).
|
|
|
|
feature gives change of builded crates: 238 -> 224.
|
|
Enable conditional checking of values in the Rust codebase
This pull-request enable conditional checking of (well known) values in the Rust codebase.
Well known values were added in https://github.com/rust-lang/rust/pull/94362. All the `target_*` values are taken from all the built-in targets which is why some extra values were needed do be added as they are not (yet ?) defined in any built-in targets.
r? `@Mark-Simulacrum`
|
|
When I implemented rust-only bootstrapping in https://github.com/rust-lang/rust/pull/92260,
I neglected to test stage0 tools - it turns out they were broken because
they couldn't find the sysroot of the initial bootstrap compiler.
This fixes stage0 tools by using `rustc --print sysroot` instead of assuming rustc is already in a
sysroot and hard-coding the relative directory.
|
|
Add miri to the well known conditional compilation names and values
This pull request adds `miri` to the list of well known names and values of the conditional compilation checking (`--check-cfg`).
This was brought up in [Zulip](https://rust-lang.zulipchat.com/#narrow/stream/246057-t-cargo/topic/RFC.203013.3A.20Checking.20conditional.20compilation.20at.20compile.20time/near/274513827) when discussing about the future of `--check-cfg`.
r? ``@petrochenkov``
|
|
|
|
|
