| Age | Commit message (Collapse) | Author | Lines |
|---|
|
Only look for commits by bors that are merge commits, because
those are the only ones with CI artifacts. Also, use
`--first-parent` to avoid traversing stuff like rollup branches.
|
|
Use `git rev-list` instead of `git log` to be more robust against
UI changes in git. Also, use the full email address for bors,
because `--author` uses a substring match.
|
|
When determining which LLVM artifacts to download, bootstrap.py
calls: `git log --author=bors --format=%H -n1 -m --first-parent --
src/llvm-project src/bootstrap/download-ci-llvm-stamp src/version`.
However, the `-m` option has no effect, per the `git log` help:
> -m
> This option makes diff output for merge commits to be shown in the
> default format. -m will produce the output only if -p is given as
> well. The default format could be changed using log.diffMerges
> configuration parameter, which default value is separate.
Accordingly, this commit removes use of the -m option in favor of
`--no-patch`, to make clear that this command should never output
diff information, as the SHA-1 hash is the only desired output.
Tested using git 2.32, this does not change the
output of the command.
The motivation for this change is that some patched versions of git
change the behavior of the `-m` flag to imply `-p`, rather than to do
nothing unless `-p` is passed. These patched versions of git lead to
this script not working. Google's corp-provided git is one such example.
|
|
Fix NixOS detection
Use `/etc/os-release` instead of `/etc/NIXOS` for detection.
The latter one does not exist on NixOS when using tmpfs as root.
|
|
This only updates the submodules the first time they're needed, instead
of unconditionally the first time you run x.py.
Ideally, this would move *all* submodules and not exclude some tools and
backtrace. Unfortunately, cargo requires all `Cargo.toml` files in the
whole workspace to be present to build any crate.
On my machine, this takes the time for an initial submodule clone (for
`x.py --help`) from 55.70 to 15.87 seconds.
This uses exactly the same logic as the LLVM update used, modulo some
minor cleanups:
- Use a local variable for `src.join(relative_path)`
- Remove unnecessary arrays for `book!` macro and make the macro simpler to use
- Add more comments
|
|
Use `/etc/os-release` instead of `/etc/NIXOS`.
The latter one does not exist on NixOS when using tmpfs as root.
|
|
- Don't print the exact command run by rustbuild unless `--verbose` is set.
This is almost always unhelpful, since it's just cargo with a lot of
arguments.
- Don't print "Build completed unsuccessfully" unless --verbose is set.
You can already tell the build failed by the errors above, and the
time isn't particularly helpful.
- Don't print the full path to bootstrap. This is useless to everyone,
even including when working on x.py itself. You can still opt-in to
this being shown with `--verbose`, since it will throw an exception.
Before:
```
error[E0432]: unresolved import `x`
--> library/std/src/lib.rs:343:5
|
343 | use x;
| ^ no external crate `x`
error: aborting due to previous error
For more information about this error, try `rustc --explain E0432`.
error: could not compile `std`
To learn more, run the command again with --verbose.
command did not execute successfully: "/home/joshua/rustc4/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "check" "--target" "x86_64-unknown-linux-gnu" "-Zbinary-dep-depinfo" "-j" "8" "--release" "--features" "panic-unwind backtrace" "--manifest-path" "/home/joshua/rustc4/library/test/Cargo.toml" "--message-format" "json-render-diagnostics"
expected success, got: exit status: 101
failed to run: /home/joshua/rustc4/build/bootstrap/debug/bootstrap check
Build completed unsuccessfully in 0:00:13
```
After:
```
error[E0432]: unresolved import `x`
--> library/std/src/lib.rs:343:5
|
343 | use x;
| ^ no external crate `x`
error: aborting due to previous error
For more information about this error, try `rustc --explain E0432`.
error: could not compile `std`
To learn more, run the command again with --verbose.
```
|
|
Use HTTPS links where possible
While looking at #86583, I wondered how many other (insecure) HTTP links were in `rustc`. This changes most other `http` links to `https`. While most of the links are in comments or documentation, there are a few other HTTP links that are used by CI that are changed to HTTPS.
Notes:
- I didn't change any to or in licences
- Some links don't support HTTPS :(
- Some `http` links were dead, in those cases I upgraded them to their new places (all of which used HTTPS)
|
|
Move LLVM submodule updates back to native.rs
Time to find more bugs!
The first commit is a straight revert of https://github.com/rust-lang/rust/pull/85647, the second is a fix for https://rust-lang.zulipchat.com/#narrow/stream/182449-t-compiler.2Fhelp/topic/x.2Epy.20always.20updates.20LLVM.20submodule/near/240113320 and https://github.com/rust-lang/rust/pull/82653#issuecomment-846755631. I haven't been able to replicate https://github.com/rust-lang/rust/pull/82653#issuecomment-849013698.
|
|
|
|
This reverts commit ad308264a38531bc8d2179324bac3652a1cda640.
|
|
`download-rustc`
Previously, changing the standard library with `download-rustc =
"if-unchanged"` would incorrectly reuse the cached compiler and standard
library from CI, which was confusing and led to incorrect test failures
or successes.
|
|
|
|
This enables better caching, since LLVM is only updated when needed, not
whenever x.py is run. Before, bootstrap.py had to use heuristics to
guess if LLVM would be needed, and updated the module more often than
necessary as a result.
This syncs the LLVM submodule only just before building the compiler, so
people working on the standard library never have to worry about it.
Example output:
```
Copying stage0 std from stage0 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu / x86_64-unknown-linux-gnu)
Updating submodule src/llvm-project
Submodule 'src/llvm-project' (https://github.com/rust-lang/llvm-project.git) registered for path 'src/llvm-project'
Submodule path 'src/llvm-project': checked out 'f9a8d70b6e0365ac2172ca6b7f1de0341297458d'
```
- Don't try to update the LLVM submodule when using system LLVM
Previously, this would try to update LLVM unconditionally. Now the
submodule is only initialized if `llvm-config` is not set.
- Don't update LLVM submodule in dry runs
This prevents the following test failures:
```
running 17 tests
fatal: invalid gitfile format: /checkout/src/llvm-project/.git
test builder::tests::defaults::build_cross_compile ... FAILED
---- builder::tests::defaults::build_default stdout ----
thread 'main' panicked at 'command did not execute successfully: "git" "rev-parse" "HEAD"
expected success, got: exit code: 128', src/build_helper/lib.rs:139:9
```
- Try running git without --progress if it fails the first time
This avoids having to do version detection to see if --progress is
supported or not.
- Don't try to update submodules when the source repository isn't managed by git
- Update LLVM submodules that have already been checked out
- Only check for whether the submodule should be updated in lib.rs; update
it unconditionally in native.rs
|
|
Now that stdarch has been updated, we can do this without breaking the
build.
|
|
Previously, this caused a bug on NixOS:
1. bootstrap.py would download and patch stage0/cargo
2. bootstrap.py would download nightly cargo, but extract it to
stage0/cargo instead of ci-rustc/cargo.
3. bootstrap.py would fail to build rustbuild because stage0/cargo
wasn't patched.
The "proper" fix is to extract nightly cargo to ci-rustc instead, but it
doesn't seem to be necessary at all, so this just skips downloading it
instead.
|
|
Moving the `.nix-deps` has resulted in rpath links being broken and
therefore bootstrap on NixOS broken entirely.
This PR still produces a `.nix-deps` but only for the purposes of
producing a gc root. We rpath a symlink-resolved result instead.
For purposes of simplicity we also use joinSymlink to produce a single
merged output directory so that we don't need to update multiple
locations every time we add a library or something.
|
|
Add `download-rustc = "if-unchanged"`
This allows keeping the setting to a fixed value without having to
toggle it when you want to work on the compiler instead of on tools.
This sets `BOOTSTRAP_DOWNLOAD_RUSTC` in bootstrap.py so rustbuild doesn't have to try and replicate its logic.
Helps with https://github.com/rust-lang/rust/issues/81930.
r? `@Mark-Simulacrum` cc `@camelid`
|
|
This allows keeping the setting to a fixed value without having to
toggle it when you want to work on the compiler instead of on tools.
|
|
Use the beta compiler for building bootstrap tools when `download-rustc` is set
## Motivation
This avoids having to rebuild bootstrap and tidy each time you rebase
over master. In particular, it makes rebasing and running `x.py fmt` on
each commit in a branch significantly faster. It also avoids having to
rebuild bootstrap after setting `download-rustc = true`.
## Implementation
Instead of extracting the CI artifacts directly to `stage0/`, extract
them to `ci-rustc/` instead. Continue to copy them to the proper
sysroots as necessary for all stages except stage 0.
This also requires `bootstrap.py` to download both stage0 and CI
artifacts and distinguish between the two when checking stamp files.
Note that since tools have to be built by the same compiler that built
`rustc-dev` and the standard library, the downloaded artifacts can't be
reused when building with the beta compiler. To make sure this is still
a good user experience, warn when building with the beta compiler, and
default to building with stage 2.
I tested this by rebasing this PR from edeee915b1c52f97411e57ef6b1a8bd46548a37a over 1c77a1fa3ca574f2a40056f64d498db8efe0d8a8 and confirming that only the bootstrap library itself had to be rebuilt, not any dependencies and not `tidy`. I also tested that a clean build with `x.py build` builds rustdoc exactly once and does no other work, and that `touch src/librustdoc/lib.rs && x.py build` works. `x.py check` still behaves as before (checks using the beta compiler, even if there are changes to `compiler/`).
Helps with https://github.com/rust-lang/rust/issues/81930.
r? `@Mark-Simulacrum`
|
|
## Motivation
This avoids having to rebuild bootstrap and tidy each time you rebase
over master. In particular, it makes rebasing and running `x.py fmt` on
each commit in a branch significantly faster. It also avoids having to
rebuild bootstrap after setting `download-rustc = true`.
## Implementation
Instead of extracting the CI artifacts directly to `stage0/`, extract
them to `ci-rustc/` instead. Continue to copy them to the proper
sysroots as necessary for all stages except stage 0.
This also requires `bootstrap.py` to download both stage0 and CI
artifacts and distinguish between the two when checking stamp files.
Note that since tools have to be built by the same compiler that built
`rustc-dev` and the standard library, the downloaded artifacts can't be
reused when building with the beta compiler. To make sure this is still
a good user experience, warn when building with the beta compiler, and
default to building with stage 2.
|
|
When bumping the bootstrap version, the name of the generated LLVM
shared object file is changed, even though it's the same contents as
before. If bootstrap tries to use an older version, it will get linking
errors:
```
Building rustdoc for stage1 (x86_64-unknown-linux-gnu)
Compiling rustdoc-tool v0.0.0 (/home/joshua/rustc/src/tools/rustdoc)
error: linking with `cc` failed: exit code: 1
|
= note: "cc" "-Wl,--as-needed" ... lots of args ...
= note: /usr/bin/ld: cannot find -lLLVM-12-rust-1.53.0-nightly
clang: error: linker command failed with exit code 1 (use -v to see invocation)
error: could not compile `rustdoc-tool`
```
|
|
On reflection on the issue in https://github.com/rust-lang/rust/pull/79540#discussion_r572572280, I think the bug was actually using the `compiler/` filter, not using `--author=bors`. https://github.com/rust-lang/rust/commit/9a1d6174c925f54c923599e29b09d6855e6b3a78 has no CI artifacts because it was merged as part of a rollup:
```
$ curl -I https://ci-artifacts.rust-lang.org/rustc-builds/96e843ce6ae42e0aa519ba45e148269de347fd84/rust-std-nightly-x86_64-unknown-linux-gnu.tar.xz
HTTP/2 404
```
So 9a1d6174c925f54c923599e29b09d6855e6b3a78 is the correct commit to download, and that's what `--author=bors` does:
$ git log --author=bors 4aec8a5da5547d6e1c24e99dad0003b7cac107f5
commit 9a1d6174c925f54c923599e29b09d6855e6b3a78
Ideally it would look for "the most recent bors commit not followed by a change to `compiler/`", which would exclude things like documentation changes and avoid redownloading more than necessary, but
- Redownloading isn't the end of the world,
- That metric is hard to implement, and
- Documentation-only or library-only changes are very rare anyway since they're usually rolled up with changes to the compiler.
|
|
|
|
|
|
- Use the same compiler for stage0 and stage1. This should be fixed at
some point (so bootstrap isn't constantly rebuilt).
- Make sure `x.py build` and `x.py check` work.
- Use `git merge-base` to determine the most recent commit to download.
- Copy stage0 to the various sysroots in `Sysroot`, and delegate to
Sysroot in Assemble. Leave all other code unchanged.
- Rename date -> key
This can also be a commit hash, so 'date' is no longer a good name.
- Add the commented-out option to config.toml.example
- Disable all steps by default when `download-rustc` is enabled
Most steps don't make sense when downloading a compiler, because they'll
be pre-built in the sysroot. Only enable the ones that might be useful,
in particular Rustdoc and all `check` steps.
At some point, this should probably enable other tools, but rustdoc is
enough to test out `download-rustc`.
- Don't print 'Skipping' twice in a row
Bootstrap forcibly enables a dry run if it isn't already set, so
previously it would print the message twice:
```
Skipping bootstrap::compile::Std because it is not enabled for `download-rustc`
Skipping bootstrap::compile::Std because it is not enabled for `download-rustc`
```
Now it correctly only prints once.
## Future work
- Add FIXME about supporting beta commits
- Debug logging will never work. This should be fixed.
|
|
Don't clone LLVM submodule when download-ci-llvm is set
Previously, `downloading_llvm` would check `self.build` while it was
still an empty string, and think it was always false. This fixes the
check.
This addresses the worst part of https://github.com/rust-lang/rust/issues/76653. There are still some large submodules being downloaded (in particular, `rustc-by-example` is 146 MB, and all the submodules combined are 311 MB), but this is a lot better than the whopping 1.4 GB before.
|
|
Previously, `downloading_llvm` would check `self.build` while it was
still an empty string, and think it was always false. This fixes the
check.
|
|
Before, it could print this error if no toolchain was configured:
```
error: no default toolchain configured
error: backtrace:
error: stack backtrace:
0: error_chain::backtrace::imp::InternalBacktrace::new
1: rustup::config::Cfg::toolchain_for_dir
2: rustup_init::run_rustup_inner
3: rustup_init::main
4: std::rt::lang_start::{{closure}}
5: main
6: __libc_start_main
7: _start
```
|
|
cc #79813
This PR adds an allow-by-default future-compatibility lint
`SEMICOLON_IN_EXPRESSIONS_FROM_MACROS`. It fires when a trailing semicolon in a
macro body is ignored due to the macro being used in expression
position:
```rust
macro_rules! foo {
() => {
true; // WARN
}
}
fn main() {
let val = match true {
true => false,
_ => foo!()
};
}
```
The lint takes its level from the macro call site, and
can be allowed for a particular macro by adding
`#[allow(semicolon_in_expressions_from_macros)]`.
The lint is set to warn for all internal rustc crates (when being built
by a stage1 compiler). After the next beta bump, we can enable
the lint for the bootstrap compiler as well.
|
|
... and update the comment in `config.toml.example`
|
|
In particular, the CI built `libLLVM-*.so` needs to have `libz.so`
RPATHed so that binaries like `llvm-config` work at all.
|
|
Don't use `self.date` unconditionally for `program_out_of_date()`
This avoids unnecessary cache invalidations for programs not affected by
the stage0 version (which is everything except the stage0 compiler
itself).
The redundant invalidations weren't noticed until now because they only
showed up on stage0 bumps, at which point people are used to rebuilding
everything anyway. I noticed it in https://github.com/rust-lang/rust/pull/79540
because I wasn't adding `self.date` to the stamp file (because I didn't realize it was necessary). Rather than
adding self.date I thought it was better to remove it from the cache key.
|
|
Previously:
```
curl: (22) The requested URL returned error: 404
failed to run: curl -# -y 30 -Y 10 --connect-timeout 30 --retry 3 -Sf -o /tmp/tmp6ptXJV https://ci-artifacts.rust-lang.org/rustc-builds/99ad5a1a2824fea1ecf60068fd3636beae7ea2da/rust-dev-nightly-x86_64-unknown-linux-gnu.tar.gz
```
Now:
```
error: XZ support is required to download LLVM
help: consider disabling `download-ci-llvm` or using a different version of python
Build completed unsuccessfully in 0:00:00
```
|
|
This avoids unnecessary cache invalidations for programs not affected by
the stage0 version (which is everything except the stage0 compiler
itself).
The redundant invalidations weren't noticed until now because they only
showed up on stage0 bumps, at which point people are used to rebuilding
everything anyway. I noticed it because I wasn't adding `self.date` to
the stamp file (because I didn't realize it was necessary). Rather than
adding self.date I thought it was better to remove it from the cache
key.
|
|
Previously, `os.remove` would always give a FileNotFound error the first
time you called it, causing bootstrap to make unnecessary copies. This
now only calls `remove()` if the file exists, avoiding the unnecessary
error.
|
|
Misc rustbuild improvements when the LLVM backend isn't used
* Don't checkout llvm-project
* Don't require cmake and ninja
Fixes #78564
|
|
|
|
bootstrap completely ignores all errors when detecting a rustup version,
so this wasn't noticed before.
Fixes the following error:
```
rustup not detected: a bytes-like object is required, not 'str'
falling back to auto-detect
```
This also takes the opportunity to only call rustup and other external
commands only once during startup.
|
|
Allow using `download-ci-llvm` from directories other than the root
Previously, bootstrap.py would attempt to find the LLVM commit from
`src/llvm-project`. However, it assumed it was always being run from the
top-level directory, which isn't always the case.
Before:
```
downloading https://ci-artifacts.rust-lang.org/rustc-builds//rust-dev-nightly-x86_64-unknown-linux-gnu.tar.gz
curl: (22) The requested URL returned error: 404
failed to run: curl -# -y 30 -Y 10 --connect-timeout 30 --retry 3 -Sf -o /tmp/tmppyh4w8 https://ci-artifacts.rust-lang.org/rustc-builds//rust-dev-nightly-x86_64-unknown-linux-gnu.tar.gz
Build completed unsuccessfully in 0:00:02
```
After:
```
downloading https://ci-artifacts.rust-lang.org/rustc-builds/430feb24a46993e5073c1bb1b39da190d83fa2bf/rust-dev-nightly-x86_64-unknown-linux-gnu.tar.gz
###################################################################################################################################################################### 100.0%
extracting /home/joshua/rustc/src/bootstrap/build/cache/llvm-430feb24a46993e5073c1bb1b39da190d83fa2bf-False/rust-dev-nightly-x86_64-unknown-linux-gnu.tar.gz
```
r? ```@Mark-Simulacrum```
cc ```@pnkfelix```
|
|
Previously, bootstrap.py would attempt to find the LLVM commit from
`src/llvm-project`. However, it assumed it was always being run from the
top-level directory, which isn't always the case.
Before:
```
downloading https://ci-artifacts.rust-lang.org/rustc-builds//rust-dev-nightly-x86_64-unknown-linux-gnu.tar.gz
curl: (22) The requested URL returned error: 404
failed to run: curl -# -y 30 -Y 10 --connect-timeout 30 --retry 3 -Sf -o /tmp/tmppyh4w8 https://ci-artifacts.rust-lang.org/rustc-builds//rust-dev-nightly-x86_64-unknown-linux-gnu.tar.gz
Build completed unsuccessfully in 0:00:02
```
After:
```
downloading https://ci-artifacts.rust-lang.org/rustc-builds/430feb24a46993e5073c1bb1b39da190d83fa2bf/rust-dev-nightly-x86_64-unknown-linux-gnu.tar.gz
###################################################################################################################################################################### 100.0%
extracting /home/joshua/rustc/src/bootstrap/build/cache/llvm-430feb24a46993e5073c1bb1b39da190d83fa2bf-False/rust-dev-nightly-x86_64-unknown-linux-gnu.tar.gz
```
|
|
Historically the stable tarballs were named after the version number of
the specific tool, instead of the version number of Rust. For example,
both of the following tarballs were part of the same release:
rustc-1.48.0-x86_64-unknown-linux-gnu.tar.xz
cargo-0.49.0-x86_64-unknown-linux-gnu.tar.xz
PR #77336 changed the dist code to instead use Rust's version number for
all the tarballs, regardless of the tool they contain:
rustc-1.48.0-x86_64-unknown-linux-gnu.tar.xz
cargo-1.48.0-x86_64-unknown-linux-gnu.tar.xz
Because of that there is no need anymore to have a separate `cargo`
field in src/stage0.txt, as the Cargo version will always be the same as
the rustc version. This PR removes the field, simplifying the code and
the maintenance work required while producing releases.
|
|
This fixes ongoing issues where x.py will detect the wrong host triple
between MSVC and GNU.
- Add line to changelog
|
|
Add cg_clif as optional codegen backend
Rustc_codegen_cranelift is an alternative codegen backend for rustc based on Cranelift. It has the potential to improve compilation times in debug mode. In my experience the compile time improvements over debug mode LLVM for a clean build are about 20-30% in most cases.
This PR adds cg_clif as optional codegen backend. By default it is only enabled for `./x.py check`. It can be enabled for `./x.py build` too by adding `cranelift` to the `rust.codegen-backends` array in `config.toml`.
MCP: https://github.com/rust-lang/compiler-team/issues/270
r? `@Mark-Simulacrum`
|
|
|
|
Sync LLVM submodule if it has been initialized
Since having enabled the download-ci-llvm option,
and having rebased on top of #76864,
I've noticed that I had to update the llvm-project
submodule manually if it was checked out.
Orignally, the submodule update logic was
introduced to reduce the friction for contributors
to manage the submodules, or in other words, to prevent
getting PRs that have unwanted submodule rollbacks
because the contributors didn't run git submodule update.
This commit adds logic to ensure there is no inadvertent
LLVM submodule rollback in a PR if download-ci-llvm
(or llvm-config) is enabled. It will detect whether the
llvm-project submodule is initialized, and if so, update
it in any case. If it is not initialized, behaviour is
kept to not do any update/initialization.
An alternative to the chosen implementation would
be to not pass the --init command line arg to
`git submodule update` for the src/llvm-project
submodule. This would show a confusing error message
however on all builds with an uninitialized repo.
We could pass the --silent param, but we still want
it to print something if it is initialized and has
to update something.
So we just do a manual check for whether the
submodule is initialized.
|
|
Since having enabled the download-ci-llvm option,
and having rebased on top of f05b47ccdfa63f8b4b9fb47a9aa92381801d3ff1,
I've noticed that I had to update the llvm-project
submodule manually if it was checked out.
Orignally, the submodule update logic was
introduced to reduce the friction for contributors
to manage the submodules, or in other words, to prevent
getting PRs that have unwanted submodule rollbacks
because the contributors didn't run git submodule update.
This commit adds logic to ensure there is no inadvertent
LLVM submodule rollback in a PR if download-ci-llvm
(or llvm-config) is enabled. It will detect whether the
llvm-project submodule is initialized, and if so, update
it in any case. If it is not initialized, behaviour is
kept to not do any update/initialization.
An alternative to the chosen implementation would
be to not pass the --init command line arg to
`git submodule update` for the src/llvm-project
submodule. This would show a confusing error message
however on all builds with an uninitialized repo.
We could pass the --silent param, but we still want
it to print something if it is initialized and has
to update something.
So we just do a manual check for whether the
submodule is initialized.
|
|
|
|
|
|
|
