| Age | Commit message (Collapse) | Author | Lines |
|---|
|
Allow configuring where artifacts are downloaded from
Bootstrap has support for downloading prebuilt LLVM and rustc artifacts to speed up local builds, but that currently works only for users working on `rust-lang/rust`. Forks of the repository (for example Ferrocene) might have different URLs to download artifacts from, or might use a different email address on merge commits, breaking both LLVM and rustc artifact downloads.
This PR refactors bootstrap to load the download URLs and other constants from `src/stage0.json`, allowing downstream forks to tweak those values. It also future-proofs the download code to easily allow forks to add their own custom protocols (like `s3://`).
This PR is best reviewed commit-by-commit.
|
|
|
|
|
|
|
|
rustc_session"
This reverts commit 2d854f9c340df887e30896f49270ae81feb3e227.
|
|
This simplifies the arguments to `download_component` in config.rs.
It also moves stage0.json metadata handling to `Build::new`, making it easier to download the stage0
compiler in rustbuild later if necessary.
|
|
|
|
|
|
|
|
It was confusing to have two functions with the same name but different behavior.
|
|
Add build metrics to rustbuild
This PR adds a new module of rustbuild, `ci_profiler`, whose job is to gather as much information as possible about the CI build as possible and store it in a JSON file uploaded to `ci-artifacts`. Right now for each step it collects:
* Type name and debug representation of the `Step` object.
* Duration of the step (excluding child steps).
* Systemwide CPU stats for the duration of the step (both single core and all cores).
* Which child steps were executed.
This is capable of replacing both the scripts to collect CPU stats and the `[TIMING]` lines in build logs (not yet removed, until we port our tooling to use the CI profiler). The format is also extensible to be able in the future to collect more information.
r? `@Mark-Simulacrum`
|
|
Remove dead code
Missing }
./x.py fmt
Remove duplicate check
Recursively remove all usage of help_on_error
|
|
Move download-rustc from python to rustbuild
- Remove download-rustc handling from bootstrap.py
- Allow a custom `pattern` in `builder.unpack()`
- Only download rustc once another part of bootstrap depends on it.
This is somewhat necessary since the download functions rely on having a full
`Builder`, which isn't available until after config parsing finishes.
Helps with https://github.com/rust-lang/rust/issues/94829.
|
|
r=Mark-Simulacrum
Print stderr consistently
Solves https://github.com/rust-lang/rust/issues/96712
I tried to follow what I perceived as the general consensus for error messages in boostrap i.e messages that were ..
* resulting from an Err(...) =>
* literally called as "Error: ...."
* by the end of the block scope forced to run a panic! or process::exit with a guaranteed non-zero error code.
|
|
|
|
- Remove download-rustc handling from bootstrap.py
- Allow a custom `pattern` in `builder.unpack()`
- Only download rustc once another part of bootstrap depends on it.
This is somewhat necessary since the download functions rely on having a full
`Builder`, which isn't available until after config parsing finishes.
|
|
|
|
I don't know why anyone would turn this off; doing so makes builds much slower (nearly a 60x slowdown according to #49057).
Remove the option to do so, which makes bootstrap a little easier to maintain.
Bootstrap continues to allow you to manage submodules manually by setting `submodules = false`.
|
|
This tool will generate a JSON file with statistics about each
individual step to disk. It will be used in rust-lang/rust's CI to
replace the mix of scripts and log scraping we currently have to gather
this data.
|
|
Move `download-ci-llvm` out of bootstrap.py
This is ready for review. It has been tested on Windows, Linux, and NixOS.
The second commit ports the changes from https://github.com/rust-lang/rust/pull/95234 to Rust; I can remove it if desired.
Helps with https://github.com/rust-lang/rust/issues/94829.
As a follow-up, this makes it possible to avoid downloading llvm until it's needed for building `rustc_llvm`; it would be nice to do that, but it shouldn't go in the first draft. It might also be possible to avoid requiring python until tests run (currently there's a check in `sanity.rs`), but I haven't looked too much into that.
`@rustbot` label +A-rustbuild
|
|
This attempts to keep the logic as close to the original python as possible.
`probably_large` has been removed, since it was always `True`, and UTF-8 paths are no longer supported when patching files for NixOS.
I can readd UTF-8 support if desired.
Note that this required making `llvm_link_shared` computed on-demand,
since we don't know whether it will be static or dynamic until we download LLVM from CI.
|
|
I didn't know that the `test::` syntax was valid before, and it doesn't
seem to be documented anywhere. Add a test so it doesn't regress accidentally,
and as executable documentation.
|
|
Signed-off-by: David Wood <david.wood@huawei.com>
|
|
Replace `run-dysutil` option with more general `split-debuginfo` option
that works on all platforms.
Signed-off-by: David Wood <david.wood@huawei.com>
|
|
In particular, this makes the output of `x build -vv` easier to read.
Before:
```
c Sysroot { compiler: Compiler { stage: 0, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } }
```
After:
```
c Sysroot { compiler: Compiler { stage: 0, host: x86_64-unknown-linux-gnu } }
```
|
|
bootstrap: untangle static-libstdcpp & llvm-tools
Previously, the static-libstdcpp setting was tied to llvm-tools such
that enabling the latter always enabled the latter. This seems
unfortunate, since it is entirely reasonable for someone to want to
_not_ statically link stdc++, but _also_ want to build the llvm-tools.
This patch therefore separates the two settings such that neither
implies the other.
On its own, that would change the default behavior in a way that's
likely to surprise users. Specifically, users who build llvm-tools
_likely_ want those tools to be statically compiled against libstdc++,
since otherwise users with older GLIBCXX will be unable to run the
vended tools. So, we also flip the default for the `static-libstdcpp`
setting such that builds always link statically against libstdc++ by
default, but it's _possible_ to opt out.
See also #94719.
|
|
|
|
This reduces binary size from 9.7MiB (5.8MiB for just rustbuild code)
to 9.3MiB (5.3MiB for just rustbuild code).
This doesn't reduce compile time in a statistically significant way.
|
|
This reduces binary size from 10.1MiB (6.2MiB for just rustbuild code)
to 9.7MiB (5.8MiB for just rustbuild code).
This also reduces compile time from ~6.1s for incr recompilation to ~5.6s.
There is still a lot of
unnecessary code due to the toml crate monomorphizing every
deserialization impl 5 times.
|
|
|
|
Previously, the static-libstdcpp setting was tied to llvm-tools such
that enabling the latter always enabled the latter. This seems
unfortunate, since it is entirely reasonable for someone to want to
_not_ statically link stdc++, but _also_ want to build the llvm-tools.
This patch therefore separates the two settings such that neither
implies the other.
On its own, that would change the default behavior in a way that's
likely to surprise users. Specifically, users who build llvm-tools
_likely_ want those tools to be statically compiled against libstdc++,
since otherwise users with older GLIBCXX will be unable to run the
vended tools. So, we also flip the default for the `static-libstdcpp`
setting such that builds always link statically against libstdc++ by
default, but it's _possible_ to opt out.
See also #94719.
|
|
This also preserves the behavior where x.py will only give a hard error on a missing config file
if it was configured through `--config` or RUST_BOOTSTRAP_CONFIG.
It also removes the top-level fallback for everything except the default path.
|
|
this also fixes a bug where bootstrap would try to use the fake `rustc` binary built by bootstrap -
cargo puts it in a different directory when using `cargo run` instead of x.py
|
|
|
|
|
|
Same rationale as https://github.com/rust-lang/rust/pull/76544;
it would be nice to make python entirely optional at some point.
This also removes $ROOT as an option for the build directory; I haven't been using it, and like Alex
said in https://github.com/rust-lang/rust/pull/76544#discussion_r488248930 it seems like a
misfeature.
This allows running `cargo run` from src/bootstrap, although that still gives
lots of compile errors if you don't use the beta toolchain.
|
|
|
|
The majority of the code is only used by either rustbuild or
rustc_llvm's build script. Rust_build is compiled once for rustbuild and
once for every stage. This means that the majority of the code in this
crate is needlessly compiled multiple times. By moving only the code
actually used by the respective crates to rustbuild and rustc_llvm's
build script, this needless duplicate compilation is avoided.
|
|
Remove some dead code from rustbuild
This should reduce build time a tiny bit.
|
|
|
|
|
|
|
|
x.py has support for excluding some steps from the invocation, but
unfortunately that's not granular enough: some steps have the same name
in different modules, and that prevents excluding only *some* of them.
As a practical example, let's say you need to run everything in `./x.py
test` except for the standard library tests, as those tests require IPv6
and need to be executed on a separate machine. Before this commit, if
you were to just run this:
./x.py test --exclude library/std
...the execution would fail, as that would not only exclude running the
tests for the standard library, it would also exclude generating its
documentation (breaking linkchecker).
This commit adds support for an optional module annotation in --exclude
paths, allowing the user to choose which module to exclude from:
./x.py test --exclude test::library/std
This maintains backward compatibility, but also allows for more ganular
exclusion. More examples on how this works:
| `--exclude` | Docs | Tests |
| ------------------- | ------- | ------- |
| `library/std` | Skipped | Skipped |
| `doc::library/std` | Skipped | Run |
| `test::library/std` | Run | Skipped |
Note that the new behavior only works in the `--exclude` flag, and not
in other x.py arguments or flags yet.
|
|
|
|
|
|
The task of the macro is simple enough that a decl macro is almost ten
times shorter than the original proc macro. The proc macro is 159 lines
while the decl macro is just 18 lines.
This reduces the amount of dependencies of rustbuild from 45 to 37. It
also slight reduces compilation time from 47s to 44s for debug builds.
|
|
|
|
r=Mark-Simulacrum
Make new symbol mangling scheme default for compiler itself.
As suggest in https://github.com/rust-lang/rust/pull/89917#issuecomment-945888574, this PR enables the new symbol mangling scheme for the compiler itself. The standard library is still compiled using the legacy mangling scheme so that the new symbol format does not show up in user code (yet).
r? `@Mark-Simulacrum`
|
|
|
|
I'm working on some LLVM patches in concert with a Rust patch, and it's
helping me quite a bit to have this as an option. It doesn't seem that
hard, so I figured I'd formalize it in x.py and send it upstream.
|
