| Age | Commit message (Collapse) | Author | Lines |
|---|
|
When bumping the bootstrap version, the name of the generated LLVM
shared object file is changed, even though it's the same contents as
before. If bootstrap tries to use an older version, it will get linking
errors:
```
Building rustdoc for stage1 (x86_64-unknown-linux-gnu)
Compiling rustdoc-tool v0.0.0 (/home/joshua/rustc/src/tools/rustdoc)
error: linking with `cc` failed: exit code: 1
|
= note: "cc" "-Wl,--as-needed" ... lots of args ...
= note: /usr/bin/ld: cannot find -lLLVM-12-rust-1.53.0-nightly
clang: error: linker command failed with exit code 1 (use -v to see invocation)
error: could not compile `rustdoc-tool`
```
|
|
On reflection on the issue in https://github.com/rust-lang/rust/pull/79540#discussion_r572572280, I think the bug was actually using the `compiler/` filter, not using `--author=bors`. https://github.com/rust-lang/rust/commit/9a1d6174c925f54c923599e29b09d6855e6b3a78 has no CI artifacts because it was merged as part of a rollup:
```
$ curl -I https://ci-artifacts.rust-lang.org/rustc-builds/96e843ce6ae42e0aa519ba45e148269de347fd84/rust-std-nightly-x86_64-unknown-linux-gnu.tar.xz
HTTP/2 404
```
So 9a1d6174c925f54c923599e29b09d6855e6b3a78 is the correct commit to download, and that's what `--author=bors` does:
$ git log --author=bors 4aec8a5da5547d6e1c24e99dad0003b7cac107f5
commit 9a1d6174c925f54c923599e29b09d6855e6b3a78
Ideally it would look for "the most recent bors commit not followed by a change to `compiler/`", which would exclude things like documentation changes and avoid redownloading more than necessary, but
- Redownloading isn't the end of the world,
- That metric is hard to implement, and
- Documentation-only or library-only changes are very rare anyway since they're usually rolled up with changes to the compiler.
|
|
|
|
|
|
- Use the same compiler for stage0 and stage1. This should be fixed at
some point (so bootstrap isn't constantly rebuilt).
- Make sure `x.py build` and `x.py check` work.
- Use `git merge-base` to determine the most recent commit to download.
- Copy stage0 to the various sysroots in `Sysroot`, and delegate to
Sysroot in Assemble. Leave all other code unchanged.
- Rename date -> key
This can also be a commit hash, so 'date' is no longer a good name.
- Add the commented-out option to config.toml.example
- Disable all steps by default when `download-rustc` is enabled
Most steps don't make sense when downloading a compiler, because they'll
be pre-built in the sysroot. Only enable the ones that might be useful,
in particular Rustdoc and all `check` steps.
At some point, this should probably enable other tools, but rustdoc is
enough to test out `download-rustc`.
- Don't print 'Skipping' twice in a row
Bootstrap forcibly enables a dry run if it isn't already set, so
previously it would print the message twice:
```
Skipping bootstrap::compile::Std because it is not enabled for `download-rustc`
Skipping bootstrap::compile::Std because it is not enabled for `download-rustc`
```
Now it correctly only prints once.
## Future work
- Add FIXME about supporting beta commits
- Debug logging will never work. This should be fixed.
|
|
Don't clone LLVM submodule when download-ci-llvm is set
Previously, `downloading_llvm` would check `self.build` while it was
still an empty string, and think it was always false. This fixes the
check.
This addresses the worst part of https://github.com/rust-lang/rust/issues/76653. There are still some large submodules being downloaded (in particular, `rustc-by-example` is 146 MB, and all the submodules combined are 311 MB), but this is a lot better than the whopping 1.4 GB before.
|
|
Previously, `downloading_llvm` would check `self.build` while it was
still an empty string, and think it was always false. This fixes the
check.
|
|
Before, it could print this error if no toolchain was configured:
```
error: no default toolchain configured
error: backtrace:
error: stack backtrace:
0: error_chain::backtrace::imp::InternalBacktrace::new
1: rustup::config::Cfg::toolchain_for_dir
2: rustup_init::run_rustup_inner
3: rustup_init::main
4: std::rt::lang_start::{{closure}}
5: main
6: __libc_start_main
7: _start
```
|
|
cc #79813
This PR adds an allow-by-default future-compatibility lint
`SEMICOLON_IN_EXPRESSIONS_FROM_MACROS`. It fires when a trailing semicolon in a
macro body is ignored due to the macro being used in expression
position:
```rust
macro_rules! foo {
() => {
true; // WARN
}
}
fn main() {
let val = match true {
true => false,
_ => foo!()
};
}
```
The lint takes its level from the macro call site, and
can be allowed for a particular macro by adding
`#[allow(semicolon_in_expressions_from_macros)]`.
The lint is set to warn for all internal rustc crates (when being built
by a stage1 compiler). After the next beta bump, we can enable
the lint for the bootstrap compiler as well.
|
|
... and update the comment in `config.toml.example`
|
|
In particular, the CI built `libLLVM-*.so` needs to have `libz.so`
RPATHed so that binaries like `llvm-config` work at all.
|
|
Don't use `self.date` unconditionally for `program_out_of_date()`
This avoids unnecessary cache invalidations for programs not affected by
the stage0 version (which is everything except the stage0 compiler
itself).
The redundant invalidations weren't noticed until now because they only
showed up on stage0 bumps, at which point people are used to rebuilding
everything anyway. I noticed it in https://github.com/rust-lang/rust/pull/79540
because I wasn't adding `self.date` to the stamp file (because I didn't realize it was necessary). Rather than
adding self.date I thought it was better to remove it from the cache key.
|
|
Previously:
```
curl: (22) The requested URL returned error: 404
failed to run: curl -# -y 30 -Y 10 --connect-timeout 30 --retry 3 -Sf -o /tmp/tmp6ptXJV https://ci-artifacts.rust-lang.org/rustc-builds/99ad5a1a2824fea1ecf60068fd3636beae7ea2da/rust-dev-nightly-x86_64-unknown-linux-gnu.tar.gz
```
Now:
```
error: XZ support is required to download LLVM
help: consider disabling `download-ci-llvm` or using a different version of python
Build completed unsuccessfully in 0:00:00
```
|
|
This avoids unnecessary cache invalidations for programs not affected by
the stage0 version (which is everything except the stage0 compiler
itself).
The redundant invalidations weren't noticed until now because they only
showed up on stage0 bumps, at which point people are used to rebuilding
everything anyway. I noticed it because I wasn't adding `self.date` to
the stamp file (because I didn't realize it was necessary). Rather than
adding self.date I thought it was better to remove it from the cache
key.
|
|
Previously, `os.remove` would always give a FileNotFound error the first
time you called it, causing bootstrap to make unnecessary copies. This
now only calls `remove()` if the file exists, avoiding the unnecessary
error.
|
|
Misc rustbuild improvements when the LLVM backend isn't used
* Don't checkout llvm-project
* Don't require cmake and ninja
Fixes #78564
|
|
|
|
bootstrap completely ignores all errors when detecting a rustup version,
so this wasn't noticed before.
Fixes the following error:
```
rustup not detected: a bytes-like object is required, not 'str'
falling back to auto-detect
```
This also takes the opportunity to only call rustup and other external
commands only once during startup.
|
|
Allow using `download-ci-llvm` from directories other than the root
Previously, bootstrap.py would attempt to find the LLVM commit from
`src/llvm-project`. However, it assumed it was always being run from the
top-level directory, which isn't always the case.
Before:
```
downloading https://ci-artifacts.rust-lang.org/rustc-builds//rust-dev-nightly-x86_64-unknown-linux-gnu.tar.gz
curl: (22) The requested URL returned error: 404
failed to run: curl -# -y 30 -Y 10 --connect-timeout 30 --retry 3 -Sf -o /tmp/tmppyh4w8 https://ci-artifacts.rust-lang.org/rustc-builds//rust-dev-nightly-x86_64-unknown-linux-gnu.tar.gz
Build completed unsuccessfully in 0:00:02
```
After:
```
downloading https://ci-artifacts.rust-lang.org/rustc-builds/430feb24a46993e5073c1bb1b39da190d83fa2bf/rust-dev-nightly-x86_64-unknown-linux-gnu.tar.gz
###################################################################################################################################################################### 100.0%
extracting /home/joshua/rustc/src/bootstrap/build/cache/llvm-430feb24a46993e5073c1bb1b39da190d83fa2bf-False/rust-dev-nightly-x86_64-unknown-linux-gnu.tar.gz
```
r? ```@Mark-Simulacrum```
cc ```@pnkfelix```
|
|
Previously, bootstrap.py would attempt to find the LLVM commit from
`src/llvm-project`. However, it assumed it was always being run from the
top-level directory, which isn't always the case.
Before:
```
downloading https://ci-artifacts.rust-lang.org/rustc-builds//rust-dev-nightly-x86_64-unknown-linux-gnu.tar.gz
curl: (22) The requested URL returned error: 404
failed to run: curl -# -y 30 -Y 10 --connect-timeout 30 --retry 3 -Sf -o /tmp/tmppyh4w8 https://ci-artifacts.rust-lang.org/rustc-builds//rust-dev-nightly-x86_64-unknown-linux-gnu.tar.gz
Build completed unsuccessfully in 0:00:02
```
After:
```
downloading https://ci-artifacts.rust-lang.org/rustc-builds/430feb24a46993e5073c1bb1b39da190d83fa2bf/rust-dev-nightly-x86_64-unknown-linux-gnu.tar.gz
###################################################################################################################################################################### 100.0%
extracting /home/joshua/rustc/src/bootstrap/build/cache/llvm-430feb24a46993e5073c1bb1b39da190d83fa2bf-False/rust-dev-nightly-x86_64-unknown-linux-gnu.tar.gz
```
|
|
Historically the stable tarballs were named after the version number of
the specific tool, instead of the version number of Rust. For example,
both of the following tarballs were part of the same release:
rustc-1.48.0-x86_64-unknown-linux-gnu.tar.xz
cargo-0.49.0-x86_64-unknown-linux-gnu.tar.xz
PR #77336 changed the dist code to instead use Rust's version number for
all the tarballs, regardless of the tool they contain:
rustc-1.48.0-x86_64-unknown-linux-gnu.tar.xz
cargo-1.48.0-x86_64-unknown-linux-gnu.tar.xz
Because of that there is no need anymore to have a separate `cargo`
field in src/stage0.txt, as the Cargo version will always be the same as
the rustc version. This PR removes the field, simplifying the code and
the maintenance work required while producing releases.
|
|
This fixes ongoing issues where x.py will detect the wrong host triple
between MSVC and GNU.
- Add line to changelog
|
|
Add cg_clif as optional codegen backend
Rustc_codegen_cranelift is an alternative codegen backend for rustc based on Cranelift. It has the potential to improve compilation times in debug mode. In my experience the compile time improvements over debug mode LLVM for a clean build are about 20-30% in most cases.
This PR adds cg_clif as optional codegen backend. By default it is only enabled for `./x.py check`. It can be enabled for `./x.py build` too by adding `cranelift` to the `rust.codegen-backends` array in `config.toml`.
MCP: https://github.com/rust-lang/compiler-team/issues/270
r? `@Mark-Simulacrum`
|
|
|
|
Sync LLVM submodule if it has been initialized
Since having enabled the download-ci-llvm option,
and having rebased on top of #76864,
I've noticed that I had to update the llvm-project
submodule manually if it was checked out.
Orignally, the submodule update logic was
introduced to reduce the friction for contributors
to manage the submodules, or in other words, to prevent
getting PRs that have unwanted submodule rollbacks
because the contributors didn't run git submodule update.
This commit adds logic to ensure there is no inadvertent
LLVM submodule rollback in a PR if download-ci-llvm
(or llvm-config) is enabled. It will detect whether the
llvm-project submodule is initialized, and if so, update
it in any case. If it is not initialized, behaviour is
kept to not do any update/initialization.
An alternative to the chosen implementation would
be to not pass the --init command line arg to
`git submodule update` for the src/llvm-project
submodule. This would show a confusing error message
however on all builds with an uninitialized repo.
We could pass the --silent param, but we still want
it to print something if it is initialized and has
to update something.
So we just do a manual check for whether the
submodule is initialized.
|
|
Since having enabled the download-ci-llvm option,
and having rebased on top of f05b47ccdfa63f8b4b9fb47a9aa92381801d3ff1,
I've noticed that I had to update the llvm-project
submodule manually if it was checked out.
Orignally, the submodule update logic was
introduced to reduce the friction for contributors
to manage the submodules, or in other words, to prevent
getting PRs that have unwanted submodule rollbacks
because the contributors didn't run git submodule update.
This commit adds logic to ensure there is no inadvertent
LLVM submodule rollback in a PR if download-ci-llvm
(or llvm-config) is enabled. It will detect whether the
llvm-project submodule is initialized, and if so, update
it in any case. If it is not initialized, behaviour is
kept to not do any update/initialization.
An alternative to the chosen implementation would
be to not pass the --init command line arg to
`git submodule update` for the src/llvm-project
submodule. This would show a confusing error message
however on all builds with an uninitialized repo.
We could pass the --silent param, but we still want
it to print something if it is initialized and has
to update something.
So we just do a manual check for whether the
submodule is initialized.
|
|
|
|
|
|
|
|
|
|
Don't download/sync llvm-project submodule if download-ci-llvm is set
llvm-project takes > 1GB storage space and a long time to download.
It's better to not download it unless needed.
|
|
|
|
This requires that bootstrap is run from the same worktree as the sources it'll
build, but this is basically required for the build to work anyway. You can
still run it from a different directory, just that the files it builds must be
beside it.
|
|
This moves build triple discovery for rustbuild from bootstrap.py into a build
script, meaning it will "just work" if building rustbuild via Cargo rather than
Python.
|
|
llvm-project takes > 1GB storage space and a long time to download.
It's better to not download it unless needed.
|
|
|
|
|
|
rustbuild: Do not use `rust-mingw` component when bootstrapping windows-gnu targets
Addresses https://github.com/rust-lang/rust/pull/76326#issuecomment-687273473 (ancient `x86_64-w64-mingw32-gcc` is selected as a linker wrapper, which is not usable in `use_lld=true` mode).
Perhaps the comment about incompatible mingw was true in the past, but many things changed since then.
With this change I was able to build everything successfully locally using a newer mingw toolchain, if it passes through the older toolchain on CI, then it should be good, I think.
|
|
This is generally a good idea, and will help with being able to build bootstrap
without Python over time as it means we can "just" build with cargo +beta build
rather than needing the user to set environment variables. This is a minor step,
but a necessary one on that road.
|
|
targets
|
|
|
|
|
|
The current plan is that submodule tracks the `release` branch of
rust-analyzer, which is updated once a week.
rust-analyzer is a workspace (with a virtual manifest), the actual
binary is provide by `crates/rust-analyzer` package.
Note that we intentionally don't add rust-analyzer to `Kind::Test`,
for two reasons.
*First*, at the moment rust-analyzer's test suite does a couple of
things which might not work in the context of rust repository. For
example, it shells out directly to `rustup` and `rustfmt`. So, making
this work requires non-trivial efforts.
*Second*, it seems unlikely that running tests in rust-lang/rust repo
would provide any additional guarantees. rust-analyzer builds with
stable and does not depend on the specifics of the compiler, so
changes to compiler can't break ra, unless they break stability
guarantee. Additionally, rust-analyzer itself is gated on bors, so we
are pretty confident that test suite passes.
|
|
|
|
This commit fixes a regression introduced in #73317 where an oversight
meant that `config.toml` was assumed to exist.
Signed-off-by: David Wood <david@davidtw.co>
|
|
This commit modifies bootstrap so that `config.toml` is read first from
`RUST_BOOTSTRAP_CONFIG`, then `--config` and finally `config.toml` in the
current directory.
This is a subjective change, intended to improve the ergnomics when
using "development shells" for rustc development (for example, using tools
such as Nix) which set environment variables to ensure a reproducible
environment (these development shells can then be version controlled). By
optionally reading `config.toml` from an environment variable, a `config.toml`
can be defined in the development shell and a path to it exposed in the
`RUST_BOOTSTRAP_CONFIG` environment variable - avoiding the need to manually
symlink the contents of this file to `config.toml` in the working
directory.
Signed-off-by: David Wood <david@davidtw.co>
|
|
|
|
This allows configuring the directory for build artifacts, instead of having it always be ./build. This means you can set it to a constant location, letting you reuse the same cache while working in several different directories.
The configuration lives in config.toml under build.build-dir. By default, it keeps the existing default of ./build, but it can be configured to any relative or absolute path. Additionally, it allows making outputs relative to the root of the git repository using $ROOT.
|
|
This also abstracts checking for a command into `require`.
Before:
```
Updating only changed submodules
Submodules updated in 0.01 seconds
Traceback (most recent call last):
File "./x.py", line 11, in <module>
bootstrap.main()
...
File "/home/joshua/src/rust/src/bootstrap/bootstrap.py", line 137, in run
ret = subprocess.Popen(args, **kwargs)
File "/usr/lib/python2.7/subprocess.py", line 394, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1047, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
```
After:
```
error: unable to run `curl --version`: [Errno 2] No such file or directory
Please make sure it's installed and in the path.
```
|
|
Otherwise each update is run twice and errors are printed
|
