| Age | Commit message (Collapse) | Author | Lines |
|---|
|
Proper macOS libLLVM symlink when cross compiling
Follow up of #98418
When cross compiling on macOS with `llvm.link-shared` enabled, the symlink creation will fail after compiling LLVM for the target architecture, because it will attempt to create the symlink in the host LLVM directory, which was already created when being built.
This commit changes the symlink path to the actual LLVM output.
r? `@jyn514`
|
|
bootstrap: Allow building individual crates
This aims to be as unintrusive as possible, but did still require adding a new `tail_args` field to all `Rustc` and `Std` steps.
New library and compiler crates are added to the sysroot as they are built, since it's useful to have e.g. just alloc and not std.
Fixes https://github.com/rust-lang/rust/issues/44293.
|
|
bootstrap.py: Always use `.exe` for Windows
This ensures that it will run the Windows executable if other files in the directory (such as Linux executables) have the same file name minus the extension. I've been tripped up by this a few times where both `bootstrap` and `bootstrap.exe` end up in the same directory.
This PR avoids ever having to see the following message:
`OSError: [WinError 193] %1 is not a valid Win32 application`
|
|
Co-authored-by: Marco Colombo <mar.colombo13@gmail.com>
|
|
This ensure that it will run the Windows executable if other files in the directory (such as Linux executables) have the same file name minus the extension.
|
|
Only obey optimize-tests flag on UI tests that are run-pass
stage1 UI tests walltime on my machine:
```
optimize-tests = false, master
25.98s
optimize-tests = true, master
34.69s
optimize-tests = true, patched
28.79s
```
Effects:
- faster UI tests
- llvm asserts get exercised less on build-pass tests
- the difference between opt and nopt builds shrinks a bit
- aux libs don't get optimized since they don't have a pass mode and almost never have explicit compile flags
|
|
When cross compiling on macOS with `llvm.link-shared` enabled,
the symlink creation will fail after compiling LLVM for the target
architecture, because it will attempt to create the symlink in the
host LLVM directory, which was already created when being built.
This commit changes the symlink path to the actual LLVM output.
|
|
When cross compiling rustc with `llvm.clang = true`, CLANG_TABLEGEN
has to be set to the host clang-tblgen executable to build clang.
|
|
When cross compiling LLVM on an arm64 machine to x86_64, CMake will
produce universal binaries by default, causing link errors. Explicitly
set CMAKE_OSX_ARCHITECTURES to the one single target architecture.
|
|
Revert #95993 fix
This reverts the temporary fix implemented by https://github.com/rust-lang/rust/pull/95993 since a permanent fix has been implemented by https://github.com/rust-lang/cargo/pull/10594
Fixes https://github.com/rust-lang/rust/issues/98728
|
|
|
|
Signed-off-by: InfRandomness <infrandomness@gmail.com>
|
|
This adds miri to https://doc.rust-lang.org/nightly/nightly-rustc/
Signed-off-by: InfRandomness <infrandomness@gmail.com>
|
|
Move locking from bootstrap.py to rust bootstrap, using fd-lock
Helps with https://github.com/rust-lang/rust/issues/94829.
|
|
|
|
- Add `Interned<Vec<String>>` and use it for tail args
- Refactor `cache.rs` not to need a separate impl for each internable type
|
|
This makes it much simpler to add new interned types, rather than having
to add 4+ impl blocks for each type.
|
|
|
|
Fix `x dist rust-dev` on a fresh checkout
Previously, it required you to manually run `x build` first, because it
assumed the LLVM binaries were already present.
|
|
```
optimize-tests = false, master
25.98s
optimize-tests = true, master
34.69s
optimize-tests = true, patched
28.79s
```
Effects:
- faster UI tests
- llvm asserts get exercised less on build-pass tests
- the difference between opt and nopt builds shrinks a bit
- aux libs don't get optimized since they don't have a pass mode and almost never have explicit compile flags
|
|
bootstrap: illumos platform flags for split-debuginfo
Bootstrap currently provides `-Zunstable-options` for platforms
when using split debuginfo - this commit adds it for the illumos
target too.
|
|
When moving this to rustbuild, I introduced a bug: if you had the file already downloaded, but
deleted the sysroot for whatever reason, rustbuil would fail to unpack the cached tarball.
This only affects people if they have a cached tarball, which is probably why we haven't seen an issue yet -
wiping `build/cache` would work around the issue, or just not deleting `build/$TARGET/stage2`.
|
|
Allow macOS to build LLVM as shared library
Inspired by how [homebrew](https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/llvm.rb) builds and distributes llvm, here we manually create a symlink with a versioned dylib path to make `llvm-config` work properly. Note, the resulting `rustc` executable and `librustc_driver-<hash>.dylib` still links to the un-versioned `libLLVM.dylib` as expected when distributed in the final output. I have confirmed this by checking `otool -L` on both binaries.
After the change, enabling `llvm.link-shared` and `llvm.thin-lto` will be possible on macOS.
|
|
Bootstrap currently provides `-Zunstable-options` for platforms
when using split debuginfo - this commit adds it for the illumos
target too.
|
|
Add a `--build-dir` flag to rustbuild
This adds an optional `--build-dir <path>` flag to rustbuild (to both the python and rust code in src/bootstrap). If provided, it overrides build directory from the config file (if any was provided).
My reason for wanting this is that I often will make a change, save, and then go run `x.py check` or `x.py test` (or something). Because I've saved, vscode will start doing its thing in the background, but this will take the file lock, preventing `x.py` from running until vscode finishes whatever it's doing (since the manually invoked x.py won't be able to acquire said file lock). This is annoying, because I'd rather the command I explicitly invoke *not* wait for r-a to complete, as r-a's check is conceptually a background task (and one which can take quite some time to complete).
Anyway, while there are likely other ways this could be handled, if you have the disk space an easy way is to just have vscode be configured to use a different build directory, and then they never have to block each-other.
This can currently be arranged without this patch, by maintaining two `config.toml`s, one of which has a different build dir, and just exists to be passed into the overridden check command in vscode.
Unfortunately, this has the downside of requiring I maintain two `config.toml`s and keep them (at least somewhat) in sync, aside from the build dir. I dislike for several reasons, not the least of which because I know myself well enough to know that these will inevitably get out of sync and confuse me in the future (perhaps this case would be different since I've thought about it enough to write this patch? Who knows, I'd rather not find out).
Either way, it would be much easier for me to have a way for *only* the build directory to differ, which this patch provides by way of a new flag. I suggested this to `@jyn514` who indicated it sounded reasonable so long as it didn't add too much complexity, which I think I've achieved, but he can be the judge.
Anyway, with this patch I can just use something like `["python3", "x.py", "check", "--build-dir", "build-vscode", "--json-output"]` as the overridden check command to rust-analyzer, and do not need to futz with any additional `config.toml`s. Which is very nice!
I've tested this manually, and can confirm that it works. I'm not sure if it needs automated tests, or where I should add them if so.
r? `@jyn514` (who has had to put up with my complaints about this... many times. <3)
|
|
Let rust-analyzer ship on stable, non-preview
The consensus on rust-lang/rust-analyzer#12432 seems to be that we are ready for `rust-analyzer` to ship as a rustup component on the beta and stable channels. This won't always be the preferred distribution method, e.g. the VS Code extension will probably still independently update to its weekly releases, but it's still useful to have a component that follows the release train with the rest of the Rust toolchain. So this removes the nightly-only gating on the bundled component, and removes the "-preview" suffix as well by the usual renaming mechanism.
cc ``@rust-lang/wg-rls-2`` ``@rust-lang/release``
|
|
Previously, it required you to manually run `x build` first, because it
assumed the LLVM binaries were already present.
|
|
|
|
This reverts the temporary fix implemented by https://github.com/rust-lang/rust/pull/95993 since a permanent fix has been implemented by https://github.com/rust-lang/cargo/pull/10594
|
|
|
|
Ensure that `static_crt` is set in the bootstrapper whenever using `cc-rs` to get a compiler command line.
When attempting to build rustc with LLVM on Windows, I noticed that the CRT flag provided to the C and C++ Compilers was inconsistent:
```
"-DCMAKE_C_FLAGS=-nologo -MT -Brepro" "-DCMAKE_CXX_FLAGS=-nologo -MD -Brepro"
```
Since the bootstrapper also sets the various `LLVM_USE_CRT` variables, this resulted in cl.exe reporting a bunch of warnings:
```
cl : Command line warning D9025 : overriding '/MD' with '/MT'
```
The root cause for this is that `cc_detect::find` was creating a `cc::Build` twice, but didn't set `static_crt` the second time.
It's possible that this what is also causing #81381
|
|
|
|
get a compiler command line.
When attempting to build rustc with LLVM on Windows, I noticed that the CRT flag provided to the C and C++ Compilers was inconsistent:
```
"-DCMAKE_C_FLAGS=-nologo -MT -Brepro" "-DCMAKE_CXX_FLAGS=-nologo -MD -Brepro"
```
Since the bootstrapper also sets the various `LLVM_USE_CRT` variables, this resulted in cl.exe reporting a bunch of warnings:
```
cl : Command line warning D9025 : overriding '/MD' with '/MT'
```
The root cause for this is that `cc_detect::find` was creating a `cc::Build` twice, but didn't set `static_crt` the second time.
It's possible that this what is also causing #81381
|
|
|
|
|
|
|
|
Running steps multiple times defeats the whole point of #96501,
since lint messages will be duplicated.
|
|
|
|
Fix LLVM rebuild with download-ci-llvm.
This fixes an issue where updating a local checkout that includes a change in `src/version` causes a linking failure.
The cause is that the `rustc_llvm` build script uses `rerun-if-changed` of `llvm-config` to know if it needs to rerun. Cargo only compares the timestamp of the last time the build script to the file. However, extracting the tar files retains the timestamps in the tarball which may be some time in the past. Since `src/version` is included in the LLVM `.so` filename, `rustc` attempts to load the wrong shared library since the `rustc_llvm` build script doesn't rerun.
https://github.com/rust-lang/cargo/issues/10791 contains a more detailed explanation.
The solution here is a hack which updates the timestamp of `llvm-config` to the current time when it is extracted.
This is a bit of a hack, but seems to be the best solution I can think of until https://github.com/rust-lang/cargo/issues/10791 is fixed. There are likely several other situations where this is a problem (such as using system LLVM), and this isn't really a complete fix.
Note that apple platforms are not directly affected by this problem because they don't have a version in the dylib filename.
How to test this:
1. On a linux host, enable download-ci-llvm
2. Check out 7036449c774860a5b348dbbe01c20704c557382e (the commit just before the last version bump)
3. `./x.py build library/std`
4. Check out 5f015a24f99f52ea9b67beb420aff24f82acf1af (the commit that bumped the version)
5. `./x.py build library/std`
Fixes #98495
|
|
|
|
Create symlinks to workaround file missing error in llvm-config
|
|
This is difficult to support without submodule handling in bootstrap.py, because cargo will refuse
to vendor sources unless it knows the Cargo.toml files of all tools in tree. Moving vendor support
to rustbuild means that rustbuild will be built without vendoring.
Rather than trying to solve this, just remove support altogether and require
people to use `rustc-src` if they want vendoring (or run `cargo vendor` manually).
|
|
|
|
These submodules were previously updated in python because Cargo gives a hard error if toml files
are missing from the workspace:
```
error: failed to load manifest for workspace member `/home/jnelson/rust-lang/rust/src/tools/rls`
Caused by:
failed to read `/home/jnelson/rust-lang/rust/src/tools/rls/Cargo.toml`
Caused by:
No such file or directory (os error 2)
failed to run: /home/jnelson/rust-lang/rust/build/x86_64-unknown-linux-gnu/stage0/bin/cargo build --manifest-path /home/jnelson/rust-lang/rust/src/bootstrap/Cargo.toml
```
However, bootstrap doesn't actually need to be part of the workspace.
Remove it so we can move submodule handling fully to Rust, avoiding duplicate code between Rust and Python.
Note that this does break `cargo run`; it has to be `cd src/bootstrap && cargo run` now.
Given that we're planning to make the main entrypoint a shell script (or rust binary),
I think this is a good tradeoff for reduced complexity in bootstrap.py.
|
|
Make "Assemble stage1 compiler" orders of magnitude faster (take 2)
This used to take upwards of 5 seconds for me locally. I found that the culprit was copying the downloaded LLVM shared object:
```
[22:28:03] Install "/home/jnelson/rust-lang/rust/build/x86_64-unknown-linux-gnu/ci-llvm/lib/libLLVM-14-rust-1.62.0-nightly.so" to "/home/jnelson/rust-lang/rust/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/libLLVM-14-rust-1.62.0-nightly.so"
[22:28:09] c Sysroot { compiler: Compiler { stage: 1, host: x86_64-unknown-linux-gnu(x86_64-unknown-linux-gnu) } }
```
It turned out that `install()` used full copies unconditionally. Change it to try using a hard-link before falling back to copying.
- Panic if we generate a symbolic link in a tarball
- Change install to use copy internally, like in my previous PR
- Change copy to dereference symbolic links, which avoids the previous regression in #96803.
I also took the liberty of fixing `x dist llvm-tools` to work even if you don't call `x build` previously.
|
|
This used to take upwards of 5 seconds for me locally. I found that the
culprit was copying the downloaded LLVM shared object:
```
[22:28:03] Install "/home/jnelson/rust-lang/rust/build/x86_64-unknown-linux-gnu/ci-llvm/lib/libLLVM-14-rust-1.62.0-nightly.so" to "/home/jnelson/rust-lang/rust/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/libLLVM-14-rust-1.62.0-nightly.so"
[22:28:09] c Sysroot { compiler: Compiler { stage: 1, host: x86_64-unknown-linux-gnu(x86_64-unknown-linux-gnu) } }
```
It turned out that `install()` used full copies unconditionally. Change
it to use `copy()` internally, which uses hard links instead when
available.
Note that this has a change in behavior: Installing a file will also
change permissions on the source, not just the destination, if hard
links are used.
To avoid changing the behavior on symlinks for existing code, I
introduce a new function `copy_internal` which only dereferences
symlinks when told to do so.
|
|
This avoids regressions in rustup-toolchain-install-master
|
|
Don't build the compiler before building rust-demangler
This saves a lot of time compiling, since rust-demangler doesn't actually use any unstable features.
This is not quite ideal because it uses ToolStd, not ToolBootstrap, so rust-demangler would be able to add unstable library features in the future. But it's a lot better than before, and `builder.cargo` doesn't currently know how to handle stages other than 0.
|
|
Pass all paths to `Step::run` at once when using `ShouldRun::krate`
Helps with https://github.com/rust-lang/rust/pull/95503. The goal is to run `cargo test -p rustc_data_structures -p rustc_lint_defs` instead of `cargo test -p rustc_data_structures; cargo test -p rustc_lint_defs`, which should both recompile less and avoid replaying cached warnings.
This was surprisingly complicated. The main changes are:
1. Invert the order of iteration in `StepDescription::run`.
Previously, it did something like:
```python
for path in paths:
for (step, should_run) in should_runs:
if let Some(set) = should_run.pathset_for_path(path):
step.run(builder, set)
```
That worked ok for individual paths, but didn't allow passing more than one path at a time to `Step::run`
(since `pathset_for_paths` only had one path available to it).
Change it to instead look at the intersection of `paths` and `should_run.paths`:
```python
for (step, should_run) in should_runs:
if let Some(set) = should_run.pathset_for_paths(paths):
step.run(builder, set)
```
2. Change `pathset_for_path` to take multiple pathsets.
The goal is to avoid `x test library/alloc` testing *all* library crates, instead of just alloc.
The changes here are similarly subtle, to use the intersection between the paths rather than all
paths in `should_run.paths`. I added a test for the behavior to try and make it more clear.
Note that we use pathsets instead of just paths to allow for sets with multiple aliases (*cough* `all_krates` *cough*).
See the documentation added in the next commit for more detail.
3. Change `StepDescription::run` to explicitly handle 0 paths.
Before this was implicitly handled by the `for` loop, which just didn't excute when there were no paths.
Now it needs a check, to avoid trying to run all steps (this is a problem for steps that use `default_condition`).
4. Change `RunDescription` to have a list of pathsets, rather than a single path.
5. Remove paths as they're matched
This allows checking at the end that no invalid paths are left over.
Note that if two steps matched the same path, this will no longer run both;
but that's a bug anyway.
6. Handle suite paths separately from regular sets.
Running multiple suite paths at once instead of in separate `make_run` invocations is both tricky and not particularly useful.
The respective test Steps already handle this by introspecting the original paths.
Avoid having to deal with it by moving suite handling into a seperate loop than `PathSet::Set` checks.
`@rustbot` label +A-rustbuild
|
|
|
