| Age | Commit message (Collapse) | Author | Lines |
|---|
|
Make "Assemble stage1 compiler" orders of magnitude faster (take 2)
This used to take upwards of 5 seconds for me locally. I found that the culprit was copying the downloaded LLVM shared object:
```
[22:28:03] Install "/home/jnelson/rust-lang/rust/build/x86_64-unknown-linux-gnu/ci-llvm/lib/libLLVM-14-rust-1.62.0-nightly.so" to "/home/jnelson/rust-lang/rust/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/libLLVM-14-rust-1.62.0-nightly.so"
[22:28:09] c Sysroot { compiler: Compiler { stage: 1, host: x86_64-unknown-linux-gnu(x86_64-unknown-linux-gnu) } }
```
It turned out that `install()` used full copies unconditionally. Change it to try using a hard-link before falling back to copying.
- Panic if we generate a symbolic link in a tarball
- Change install to use copy internally, like in my previous PR
- Change copy to dereference symbolic links, which avoids the previous regression in #96803.
I also took the liberty of fixing `x dist llvm-tools` to work even if you don't call `x build` previously.
|
|
This used to take upwards of 5 seconds for me locally. I found that the
culprit was copying the downloaded LLVM shared object:
```
[22:28:03] Install "/home/jnelson/rust-lang/rust/build/x86_64-unknown-linux-gnu/ci-llvm/lib/libLLVM-14-rust-1.62.0-nightly.so" to "/home/jnelson/rust-lang/rust/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/libLLVM-14-rust-1.62.0-nightly.so"
[22:28:09] c Sysroot { compiler: Compiler { stage: 1, host: x86_64-unknown-linux-gnu(x86_64-unknown-linux-gnu) } }
```
It turned out that `install()` used full copies unconditionally. Change
it to use `copy()` internally, which uses hard links instead when
available.
Note that this has a change in behavior: Installing a file will also
change permissions on the source, not just the destination, if hard
links are used.
To avoid changing the behavior on symlinks for existing code, I
introduce a new function `copy_internal` which only dereferences
symlinks when told to do so.
|
|
This avoids regressions in rustup-toolchain-install-master
|
|
Rollup of 5 pull requests
Successful merges:
- #94640 (Partially stabilize `(const_)slice_ptr_len` feature by stabilizing `NonNull::len`)
- #97034 (Implement `Hash` for `core::alloc::Layout`)
- #97327 (macros: introduce `fluent_messages` macro )
- #97448 (docs: Don't imply that OsStr on Unix is always UTF-8)
- #97466 ([bootstrap] Move `sanitize_sh` from `dist` to `install`)
Failed merges:
r? `@ghost`
`@rustbot` modify labels: rollup
|
|
This is the only place it's used, so there's no need for it to be public in another module.
In general, `dist` shouldn't ever touch shell scripts.
|
|
Previously, this would break because the submodule wasn't checked out.
|
|
- The logic is now unified for all targets (wasm targets should also be supported now)
- Additional "symlink" files like `ld64` are eliminated
- lld-wrapper is used for propagating the correct lld flavor
- Cleanup "unwrap or exit" logic in lld-wrapper
|
|
Omit stdarch workspace from rust-src
The path `library/stdarch/crates/Cargo.toml` does not exist.
In Rust 1.61.0, `rust-src` still includes `src/rust/library/stdarch/Cargo.toml` (but not `stdarch-verify`), which includes
```toml
[workspace]
members = [
"crates/stdarch-verify"
```
This didn't show up when testing with `-Zbuild-std` in https://github.com/rust-lang/rust/pull/94907 since the [standard list of crates](https://github.com/rust-lang/cargo/blob/f624095e1c98228a74a165ddb702078c0dd8b81e/src/cargo/core/compiler/standard_lib.rs#L26-L30) to include when building `std` does not include `stdarch`, but it will show up if a user explicitly requests `stdarch`. Or, perhaps more importantly, because of https://github.com/rust-lang/rust/issues/95736, many editors (like IntelliJ) won't treat the root of `rust-src` as a workspace, and will instead recurse into all the sub-crates directly, which then includes `stdarch`.
Also related to https://github.com/rust-lang/rust/issues/94906.
|
|
|
|
The path `library/stdarch/crates/Cargo.toml` does not exist.
This was introduced in #94907.
|
|
|
|
|
|
Remove DebbugerScripts from bootstrap CLI
This PR implements #95992 and removes the debugger scripts from the bootstrap CLI.
I could not find a lot of documentation on the bootstrap binary so perhaps there's still some documentation to be updated.
|
|
|
|
This attempts to keep the logic as close to the original python as possible.
`probably_large` has been removed, since it was always `True`, and UTF-8 paths are no longer supported when patching files for NixOS.
I can readd UTF-8 support if desired.
Note that this required making `llvm_link_shared` computed on-demand,
since we don't know whether it will be static or dynamic until we download LLVM from CI.
|
|
This has two benefits:
1. There is a clearer mental model of how bootstrap works. Steps correspond to paths on disk unless it's strictly impossible for them to do so (e.g. dist components).
2. Bootstrap has better checks for internal consistency. This caught several issues:
- `src/sanitizers` doesn't exist; I changed it to just be a `sanitizers` alias.
- `src/tools/lld` doesn't exist; I removed it, since `lld` alone already works.
- `src/llvm` doesn't exist; removed it since `llvm` and `src/llvm-project` both work.
- `src/lldb_batchmode.py` doesn't exist, it was moved to `src/etc`.
- `install` was still using `src/librustc` instead of `compiler/rustc`.
- None of the tools in `dist` / `install` allowed using `src/tools/X` to build them. This might be intentional - I can change them to aliases if you like.
|
|
|
|
Signed-off-by: codehorseman <cricis@yeah.net>
|
|
These crates aren't necessary for building the standard library. This
saves 30MB of disk space.
Fixes #94906
|
|
|
|
The majority of the code is only used by either rustbuild or
rustc_llvm's build script. Rust_build is compiled once for rustbuild and
once for every stage. This means that the majority of the code in this
crate is needlessly compiled multiple times. By moving only the code
actually used by the respective crates to rustbuild and rustc_llvm's
build script, this needless duplicate compilation is avoided.
|
|
adapt static-nobundle test to use llvm-nm
No functional changes intended.
This updates the test case to use llvm-nm instead of the system nm.
This fixes an instance over at the experimental build of rustc with HEAD LLVM:
https://buildkite.com/llvm-project/rust-llvm-integrate-prototype/builds/8380#ef6f41b5-8595-49a6-be37-0eff80e0ccb5
It is related to https://github.com/rust-lang/rust/pull/94001.
The issue is that this test uses the system nm, which may not be recent
enough to understand the update to uwtable. This replaces the test to
use the llvm-nm that should be recent enough (consistent with the LLVM
sources we use to build rustc).
|
|
Remove CFG_PLATFORM
It seems to be unused and it is incorrect for arm/aarch64 anyway.
|
|
It seems to be unused and it is incorrect for arm/aarch64 anyway.
|
|
Upgrade to LLVM 14
LLVM patch state:
* [x] https://github.com/llvm/llvm-project/commit/a55727f334b39600bfc71144b11b42aae6b94e0b Backported.
* [x] https://github.com/rust-lang/llvm-project/commit/c3c82dc12402dd41441180c0c6cf7aed7e330c53 Backported as https://github.com/llvm/llvm-project/commit/917c47b3bf0dfc45a2a5ba12c1397d647ecf4017.
* [x] https://github.com/rust-lang/llvm-project/commit/6e8f9ab632d12271355d10d34c9835a7ba14e4b9 No plan to upstream.
* [x] https://github.com/llvm/llvm-project/commit/319f4b2d52e31b000db75a0a2484b5f2ab90534a Backported.
* [x] https://github.com/rust-lang/llvm-project/commit/8b2c25d321f877161f85218479e2d1317d770e18 No plan to upstream.
* [x] https://github.com/rust-lang/llvm-project/commit/75fef2efd427362c8f16b2d09e6ebf44069e3919 No plan to upstream.
* [ ] https://github.com/rust-lang/llvm-project/commit/adef757547de5a570d9f6a00d3e6ac16c666ab79 Upstreamed as https://github.com/llvm/llvm-project/commit/2d2ef384b2f6e723edb793d08f52e7f4dc94ba3a. Needs backport.
* [x] https://github.com/rust-lang/llvm-project/commit/4b7c1b4910e9fa9e04f23f06be078e168ef4c0ee No plan to upstream.
* [x] https://github.com/rust-lang/llvm-project/commit/3f5ab0c061adb723f25b94243828b6b5407720c8 No plan to upstream.
* [x] https://github.com/rust-lang/llvm-project/commit/514d05500e0e15e358f05f5c4cec78a805858f8e No plan to upstream.
* [ ] https://github.com/rust-lang/llvm-project/commit/54c586958564582b3341d1838a5de86541e5fecf Under review at https://reviews.llvm.org/D119695 and https://reviews.llvm.org/D119856.
Release timeline:
* LLVM 14.0.0 final planned for Mar 15.
* Rust 1.60.0 planned for Apr 7.
Compile-time:
* https://perf.rust-lang.org/compare.html?start=250384edc5d78533e993f38c60d64e42b21684b2&end=b87df8d2c7c5d9ac448c585de10927ab2ee1b864
* A slight improvement on average, though no big changes either way.
* There are some larger max-rss improvements.
r? `@ghost`
|
|
This directory is required for building LLVM now.
|
|
|
|
This was only used for the inclusion of 'current' dates into our manpages, but
it is not clear that this is practically necessary. The manpage is essentially
never updated, and so we can likely afford to keep a manual date in these files.
It also seems possible to just omit it, but that may cause other tools trouble,
so avoid doing that for now.
|
|
x.py has support for excluding some steps from the invocation, but
unfortunately that's not granular enough: some steps have the same name
in different modules, and that prevents excluding only *some* of them.
As a practical example, let's say you need to run everything in `./x.py
test` except for the standard library tests, as those tests require IPv6
and need to be executed on a separate machine. Before this commit, if
you were to just run this:
./x.py test --exclude library/std
...the execution would fail, as that would not only exclude running the
tests for the standard library, it would also exclude generating its
documentation (breaking linkchecker).
This commit adds support for an optional module annotation in --exclude
paths, allowing the user to choose which module to exclude from:
./x.py test --exclude test::library/std
This maintains backward compatibility, but also allows for more ganular
exclusion. More examples on how this works:
| `--exclude` | Docs | Tests |
| ------------------- | ------- | ------- |
| `library/std` | Skipped | Skipped |
| `doc::library/std` | Skipped | Run |
| `test::library/std` | Run | Skipped |
Note that the new behavior only works in the `--exclude` flag, and not
in other x.py arguments or flags yet.
|
|
This builder is the slowest in the fleet. This should cut a considerable
amount of time. The manifest should now include the docs from
x86_64-apple-darwin. Although those docs are slightly different, it
should be close enough. When aarch64-apple-darwin heads towards tier 1,
we can revisit whether or not to re-enable the docs.
|
|
This reverts commit 241160de72b5b55187ca54243e2a6e82e336d07c.
|
|
On Windows, `x.py dist` command panics without proper error message if 'WIX' environment variable is not set. This patch changes `Option::unwrap()` to `Option::expect()`.
|
|
Fixes #88941
This will allow using `download-ci-llvm` while still having LLD
available.
|
|
Before, you could have the confusing situation where the command to
generate a component had no relation to the name of that component (e.g.
the `rustc` component was generated with `src/librustc`). This changes
the name to make them match up.
|
|
This also fixes `x doc src/tools/clippy` when compiler-docs is disabled.
|
|
The wrapper is installed as `ld` and `ld64` in the `lib\rustlib\<host_target>\bin\gcc-ld`
directory and its sole purpose is to invoke `rust-lld` in the parent directory with
the correct flavor.
|
|
Make `-Z gcc-ld=lld` work for Apple targets
`-Z gcc-ld=lld` was introduced in #85961. It does not work on Macos because lld needs be either named `ld64` or passed `-flavor darwin` as the first two arguments in order to select the Mach-O flavor. Rust invokes cc (=clang) on Macos for linking which calls `ld` as linker binary and not `ld64`, so just creating an `ld64` binary and modifying the search path with `-B` does not work.
In order to solve this patch does:
* Set the `lld_flavor` for all Apple-derived targets to `LldFlavor::Ld64`. As far as I can see this actually works towards fixing `-Xlinker=rust-lld` as all those targets use the Mach-O object format.
* Copy/hardlink rust-lld to the gcc-ld subdirectory as ld64 next to ld.
* If `-Z gcc-ld=lld` is used and the target lld flavor is Ld64 add `-fuse-ld=/path/to/ld64` to the linker invocation.
Fixes #86945.
|
|
This shows up to 5% less instruction counts on multiple benchmarks, and up to
19% wins on the -j1 wall times for rustc self-compilation.
We can afford to spend the extra cycles building LLVM essentially once more for
the x86_64-unknown-linux-gnu CI build today. The builder finishes in around 50
minutes on average, and this adds just 10 more minutes. Given the sizeable
improvements in compiler performance, this is definitely worth it.
|
|
|
|
Ensure `./x.py dist` adheres to `build.tools`
According to `config.toml.example`, the way to produce dist artifacts for both the compiler and a *subset* of tools would be to enable the extended build and manually specify the list of tools to build:
```toml
[build]
extended = true
tools = ["cargo", "rustfmt"]
```
This works as expected for `./x.py build` and `./x.py install`, but *not* for `./x.py dist`. Before this PR `./x.py dist` simply ignored the contents of `build.tools`, building just rustc/rustdoc if `build.extended = false` and all of the tools otherwise. This PR does two things:
* Changes `./x.py dist extended` to only build the tools defined in `build.tools`, if `build.tools` is not empty. The rest of the extended step was refactored to simplify the code.
* Changes how dist jobs for tools are gated: instead of `assert!(builder.config.extended)` to prevent tools from being built with `build.extended = false`, tools are simply built by default depending on `build.extended` and `build.tools`. This also enables to **explicitly** dist tools even with `build.extended = false`.
This PR is best reviewed commit-by-commit.
Fixes #86436
|
|
The default is then overridden by `should_run`.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
