| Age | Commit message (Collapse) | Author | Lines |
|---|
|
|
|
This reverts commit 3acb505ee560770c62bad5362f6caf7567d467b9
(PR #101833).
The changes in this commit caused several bugs or at least
incompatibilies. For now we're reverting this commit and will re-land it
alongside fixes for those bugs.
|
|
Use BOLT in CI to optimize LLVM
This PR adds an optimization step in the Linux `dist` CI pipeline that uses [BOLT](https://github.com/llvm/llvm-project/tree/main/bolt) to optimize the `libLLVM.so` library built by boostrap.
Steps:
- [x] Use LLVM 15 as a bootstrap compiler and use it to build BOLT
- [x] Compile LLVM with support for relocations (`-DCMAKE_SHARED_LINKER_FLAGS="-Wl,-q"`)
- [x] Gather profile data using instrumented LLVM
- [x] Apply profile to LLVM that has already been PGOfied
- [x] Run with BOLT profiling on more benchmarks
- [x] Decide on the order of optimization (PGO -> BOLT?)
- [x] Decide how we should get `bolt` (currently we use the host `bolt`)
- [x] Clean up
The latest perf results can be found [here](https://github.com/rust-lang/rust/pull/94381#issuecomment-1258269440). The current CI build time with BOLT applied is around 1h 55 minutes.
|
|
|
|
|
|
Package `rust-docs-json` into nightly components (take 3)
`dist` creates a `rust-docs-json.tar.xz` tarfile. But build-manifest expected it to be named `rust-docs-json-preview.tar.xz`. Change build-manifest to allow the name without the `-preview` suffix.
I haven't actually tested this :( build-manifest is a pain to run locally.
|
|
rust-lang/rust#100557 removed the `git-commit-hash` file and replaced it
with `git-commit-info`. However, build-manifest relies on the
`git-commit-hash` file being present, so this adds it back.
|
|
fix: use git-commit-info for version information
Fixes #33286.
Fixes #86587.
This PR changes the current `git-commit-hash` file that `./x.py` dist puts in the `rustc-{version}-src.tar.{x,g}z` to contain the hash, the short hash, and the commit date from which the tarball was created, assuming git was available when it was. It uses this for reading the version so that rustc has all the appropriate metadata.
# Testing
Testing this is kind of a pain. I did it with something like
```sh
./x.py dist # ensure that `ignore-git` is `false` in config.toml
cp ./build/dist/rustc-1.65.0-dev-src.tar.gz ../rustc-1.65.0-dev-src.tar.gz
cd .. && tar -xzf rustc-1.65.0-dev-src && cd rustc-1.65.0-dev-src
./x.py build
```
Then, the output of `rustc -vV` with the stage1 compiler should have the `commit-hash` and `commit-date` fields filled, rather than be `unknown`. To be completely sure, you can use `rustc --sysroot` with the stdlib that the original `./x.py dist` made, which will require that the metadata matches.
|
|
This PR adds support for fetching version information from the
`git-commit-info` file when building the compiler from a source tarball.
|
|
`dist` creates a `rust-docs-json.tar.xz` tarfile. But build-manifest expected it to be named
`rust-docs-json-preview.tar.xz`. Change build-manifest to allow the name without the `-preview` suffix.
This also adds `rust-docs-json` to the `rust` component. I'm not quite sure why it exists,
but rustup uses it to determine which components are available.
|
|
r=Mark-Simulacrum
Make the `c` feature for `compiler-builtins` an explicit opt-in
Its build script doesn't support cross-compilation. I tried fixing it, but the cc crate itself doesn't appear to support cross-compiling to windows either unless you use the -gnu toolchain:
```
error occurred: Failed to find tool. Is `lib.exe` installed?
```
Fixes https://github.com/rust-lang/rust/issues/101172.
|
|
The build script for `compiler_builtins` doesn't support cross-compilation. I tried fixing it, but the cc crate itself
doesn't appear to support cross-compiling to windows either unless you use the -gnu toolchain:
```
error occurred: Failed to find tool. Is `lib.exe` installed?
```
Rather than trying to fix it or special-case the platforms without bugs,
make it opt-in instead of automatic.
|
|
Distribute bootstrap in CI
This pre-compiles bootstrap from source and adds it to the existing `rust-dev` component. There are two main goals here:
1. Make it faster to build rust from source, both the first time and incrementally
2. Make it easier to add non-python entrypoints, since they can call out to bootstrap directly rather than having to figure out the right flags to pre-compile it. This second part is still in a bit of flux, see the tracking issue below for more information.
There are also several changes to make bootstrap able to run on a machine other than the one it was built (particularly around `config.src` and `config.out` detection). I (`@jyn514)` am slightly concerned these will regress unless tested - maybe we should add an automated test that runs bootstrap in a chroot or something? Unclear whether the effort is worth the test coverage.
Helps with https://github.com/rust-lang/rust/issues/94829.
|
|
|
|
Add `llvm-dis` to the set of tools in `ci-llvm`
The LLVM disassembler is needed for the test introduced in https://github.com/rust-lang/rust/pull/97550.
Signed-off-by: Miguel Ojeda <ojeda@kernel.org>
|
|
Also require other subtrees to always build successfully
r? ``@jyn514``
|
|
The LLVM disassembler is needed for the test introduced in
https://github.com/rust-lang/rust/pull/97550.
Signed-off-by: Miguel Ojeda <ojeda@kernel.org>
|
|
|
|
r=Mark-Simulacrum
Distribute rust-docs-json via rustup.
I am not 100% sure on how to treat `rust-json-docs` in `target_host_combination`. I went along with a similar strategy to the one used for `rust-docs`, but looking for guidance there.
|
|
|
|
|
|
documentation for std crates in nightly toolchains.
We also add a new flag to `x doc`, `--json`, to render the JSON-formatted version alongside the HTML-formatted one.
|
|
|
|
- Add a new `bootstrap` component
Originally, we planned to combine this with the `rust-dev` component.
However, I realized that would force LLVM to be redownloaded whenever bootstrap is modified.
LLVM is a much larger download, so split this to get better caching.
- Build bootstrap for all tier 1 and 2 targets
|
|
Change implementation of `-Z gcc-ld` and `lld-wrapper` again
This PR partially reverts https://github.com/rust-lang/rust/pull/97375 and uses the strategy described in https://github.com/rust-lang/rust/issues/97402#issuecomment-1147404520 instead, thus fixes https://github.com/rust-lang/rust/issues/97755.
|
|
These generally aren't used too much, and I feel like aren't really
helpful for installing the RLS stub.
|
|
|
|
When running `x.py dist`, rustfmt was being allowed to fail when
missing-tools is true. This isn't much of an issue in practice
since other CI jobs will fail if rustfmt fails. This code was just
leftovers from when rustfmt was tracked in toolstate, and this removes
it to make it clear that it no longer works that way.
|
|
|
|
This builds `src/tools/rust-analyzer/crates/proc-macro-srv-cli` and
ships it as part of Rustc's dist component. This allows rust-analyzer's
proc macro support to work on all rustc versions (stable, beta and
nightly) starting now.
|
|
|
|
Before:
```
thread 'main' panicked at 'Unable to build RLS', dist.rs:42:9
```
After:
```
thread 'main' panicked at 'Unable to build submodule tool RLS (use `missing-tools = true` to ignore this failure)
note: not all tools are available on all nightlies
help: see https://forge.rust-lang.org/infra/toolstate.html for more information', dist.rs:43:9
```
|
|
bootstrap: Allow building individual crates
This aims to be as unintrusive as possible, but did still require adding a new `tail_args` field to all `Rustc` and `Std` steps.
New library and compiler crates are added to the sysroot as they are built, since it's useful to have e.g. just alloc and not std.
Fixes https://github.com/rust-lang/rust/issues/44293.
|
|
- Add `Interned<Vec<String>>` and use it for tail args
- Refactor `cache.rs` not to need a separate impl for each internable type
|
|
Fix `x dist rust-dev` on a fresh checkout
Previously, it required you to manually run `x build` first, because it
assumed the LLVM binaries were already present.
|
|
Previously, it required you to manually run `x build` first, because it
assumed the LLVM binaries were already present.
|
|
|
|
This is difficult to support without submodule handling in bootstrap.py, because cargo will refuse
to vendor sources unless it knows the Cargo.toml files of all tools in tree. Moving vendor support
to rustbuild means that rustbuild will be built without vendoring.
Rather than trying to solve this, just remove support altogether and require
people to use `rustc-src` if they want vendoring (or run `cargo vendor` manually).
|
|
Make "Assemble stage1 compiler" orders of magnitude faster (take 2)
This used to take upwards of 5 seconds for me locally. I found that the culprit was copying the downloaded LLVM shared object:
```
[22:28:03] Install "/home/jnelson/rust-lang/rust/build/x86_64-unknown-linux-gnu/ci-llvm/lib/libLLVM-14-rust-1.62.0-nightly.so" to "/home/jnelson/rust-lang/rust/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/libLLVM-14-rust-1.62.0-nightly.so"
[22:28:09] c Sysroot { compiler: Compiler { stage: 1, host: x86_64-unknown-linux-gnu(x86_64-unknown-linux-gnu) } }
```
It turned out that `install()` used full copies unconditionally. Change it to try using a hard-link before falling back to copying.
- Panic if we generate a symbolic link in a tarball
- Change install to use copy internally, like in my previous PR
- Change copy to dereference symbolic links, which avoids the previous regression in #96803.
I also took the liberty of fixing `x dist llvm-tools` to work even if you don't call `x build` previously.
|
|
This used to take upwards of 5 seconds for me locally. I found that the
culprit was copying the downloaded LLVM shared object:
```
[22:28:03] Install "/home/jnelson/rust-lang/rust/build/x86_64-unknown-linux-gnu/ci-llvm/lib/libLLVM-14-rust-1.62.0-nightly.so" to "/home/jnelson/rust-lang/rust/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/libLLVM-14-rust-1.62.0-nightly.so"
[22:28:09] c Sysroot { compiler: Compiler { stage: 1, host: x86_64-unknown-linux-gnu(x86_64-unknown-linux-gnu) } }
```
It turned out that `install()` used full copies unconditionally. Change
it to use `copy()` internally, which uses hard links instead when
available.
Note that this has a change in behavior: Installing a file will also
change permissions on the source, not just the destination, if hard
links are used.
To avoid changing the behavior on symlinks for existing code, I
introduce a new function `copy_internal` which only dereferences
symlinks when told to do so.
|
|
This avoids regressions in rustup-toolchain-install-master
|
|
Rollup of 5 pull requests
Successful merges:
- #94640 (Partially stabilize `(const_)slice_ptr_len` feature by stabilizing `NonNull::len`)
- #97034 (Implement `Hash` for `core::alloc::Layout`)
- #97327 (macros: introduce `fluent_messages` macro )
- #97448 (docs: Don't imply that OsStr on Unix is always UTF-8)
- #97466 ([bootstrap] Move `sanitize_sh` from `dist` to `install`)
Failed merges:
r? `@ghost`
`@rustbot` modify labels: rollup
|
|
This is the only place it's used, so there's no need for it to be public in another module.
In general, `dist` shouldn't ever touch shell scripts.
|
|
Previously, this would break because the submodule wasn't checked out.
|
|
- The logic is now unified for all targets (wasm targets should also be supported now)
- Additional "symlink" files like `ld64` are eliminated
- lld-wrapper is used for propagating the correct lld flavor
- Cleanup "unwrap or exit" logic in lld-wrapper
|
|
Omit stdarch workspace from rust-src
The path `library/stdarch/crates/Cargo.toml` does not exist.
In Rust 1.61.0, `rust-src` still includes `src/rust/library/stdarch/Cargo.toml` (but not `stdarch-verify`), which includes
```toml
[workspace]
members = [
"crates/stdarch-verify"
```
This didn't show up when testing with `-Zbuild-std` in https://github.com/rust-lang/rust/pull/94907 since the [standard list of crates](https://github.com/rust-lang/cargo/blob/f624095e1c98228a74a165ddb702078c0dd8b81e/src/cargo/core/compiler/standard_lib.rs#L26-L30) to include when building `std` does not include `stdarch`, but it will show up if a user explicitly requests `stdarch`. Or, perhaps more importantly, because of https://github.com/rust-lang/rust/issues/95736, many editors (like IntelliJ) won't treat the root of `rust-src` as a workspace, and will instead recurse into all the sub-crates directly, which then includes `stdarch`.
Also related to https://github.com/rust-lang/rust/issues/94906.
|
|
|
|
The path `library/stdarch/crates/Cargo.toml` does not exist.
This was introduced in #94907.
|
|
|
|
|
