| Age | Commit message (Collapse) | Author | Lines |
|---|
|
|
|
x.py has support for excluding some steps from the invocation, but
unfortunately that's not granular enough: some steps have the same name
in different modules, and that prevents excluding only *some* of them.
As a practical example, let's say you need to run everything in `./x.py
test` except for the standard library tests, as those tests require IPv6
and need to be executed on a separate machine. Before this commit, if
you were to just run this:
./x.py test --exclude library/std
...the execution would fail, as that would not only exclude running the
tests for the standard library, it would also exclude generating its
documentation (breaking linkchecker).
This commit adds support for an optional module annotation in --exclude
paths, allowing the user to choose which module to exclude from:
./x.py test --exclude test::library/std
This maintains backward compatibility, but also allows for more ganular
exclusion. More examples on how this works:
| `--exclude` | Docs | Tests |
| ------------------- | ------- | ------- |
| `library/std` | Skipped | Skipped |
| `doc::library/std` | Skipped | Run |
| `test::library/std` | Run | Skipped |
Note that the new behavior only works in the `--exclude` flag, and not
in other x.py arguments or flags yet.
|
|
|
|
|
|
The task of the macro is simple enough that a decl macro is almost ten
times shorter than the original proc macro. The proc macro is 159 lines
while the decl macro is just 18 lines.
This reduces the amount of dependencies of rustbuild from 45 to 37. It
also slight reduces compilation time from 47s to 44s for debug builds.
|
|
|
|
r=Mark-Simulacrum
Make new symbol mangling scheme default for compiler itself.
As suggest in https://github.com/rust-lang/rust/pull/89917#issuecomment-945888574, this PR enables the new symbol mangling scheme for the compiler itself. The standard library is still compiled using the legacy mangling scheme so that the new symbol format does not show up in user code (yet).
r? `@Mark-Simulacrum`
|
|
|
|
I'm working on some LLVM patches in concert with a Rust patch, and it's
helping me quite a bit to have this as an option. It doesn't seem that
hard, so I figured I'd formalize it in x.py and send it upstream.
|
|
r=Mark-Simulacrum
Make `llvm.download-ci-llvm="if-available"` work for tier 2 targets with host tools
`llvm.download-ci-llvm="if-available"` is used for most profiles configured via `x.py setup`. It allows downloading prebuilt LLVM tarballs from the CI artifacts for a configured list of platforms. Currently this list is restricted to tier 1 targets but it makes sense for all tier 2 targets with host tools.
|
|
|
|
Allow static linking LLVM with ThinLTO
There's no reason not to allow this if the user wants it. It works, at least in a local build on linux host.
For our use case, we're happy to spend more time building the compiler if it creates a speedup every time we run it, and we've observed speedups like this with clang.
|
|
|
|
|
|
On NixOS systems, bootstrap will patch rustc used in bootstrapping after
checking `/etc/os-release` (to confirm the current distribution is NixOS).
However, when using Nix on a non-NixOS system, it can be desirable for
bootstrap to patch rustc. In this commit, a `patch-binaries-for-nix`
option is added to `config.toml`, which allows for user opt-in to
bootstrap's Nix patching.
Signed-off-by: David Wood <david.wood@huawei.com>
|
|
This shows up to 5% less instruction counts on multiple benchmarks, and up to
19% wins on the -j1 wall times for rustc self-compilation.
We can afford to spend the extra cycles building LLVM essentially once more for
the x86_64-unknown-linux-gnu CI build today. The builder finishes in around 50
minutes on average, and this adds just 10 more minutes. Given the sizeable
improvements in compiler performance, this is definitely worth it.
|
|
The options are `overflow-checks` and `overflow-checks-std`
defaulting to false.
|
|
|
|
Don't default to `submodules = true` unless the rust repo has a .git directory
Should hopefully fix https://github.com/rust-lang/rust/pull/82653#issuecomment-885093033 - `@semarie` can you confirm?
r? `@Mark-Simulacrum`
|
|
|
|
Previously it would error out:
```
$ x check --dry-run
thread 'main' panicked at 'std::fs::read_to_string(ci_llvm.join("link-type.txt")) failed with No such file or directory (os error 2) ("CI llvm missing: /home/joshua/rustc3/build/tmp-dry-run/x86_64-unknown-linux-gnu/ci-llvm")', src/bootstrap/config.rs:795:33
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Build completed unsuccessfully in 0:00:10
```
|
|
Cleanup option parsing and config.toml.example
- Add an assertion that `link-shared = true` when `thin-lto = true`.
Previously, link-shared would be silently overwritten.
- Get rid of `Option<bool>` in bootstrap/config.rs. Set defaults
immediately instead of delaying until later in bootstrap. This makes
it easier to find what the default value is.
- Remove redundant `config.x = false` when the default was already false
- Set defaults for `bindir` in `default_opts()` instead of `parse()`
- Update `download-ci-llvm = if-supported` option to match bootstrap.py
- Remove redundant check for link_shared. Previously, it was checked twice.
- Update various options in config.toml.example to their defaults.
Previously, some options showed an example value instead of the
default value.
- Fix incorrect defaults in config.toml.example
+ `use-libcxx` defaults to false
+ Add missing `check-stage = 0`
+ Update several defaults to be conditional (e.g. `if incremental { 10 } else { 100 }`)
- Remove redundant defaults in prose
- Use the same comment for the default and target-dependent `musl-root`
- Fix typos
- Link to `cc_detect` for `cc` and `cxx`, since the logic is ... complicated.
- Update more defaults to better reflect how they actually get set
- Remove ignored `gpg-password-file` option
This stopped being used in
7704d35,
but was never removed from config.toml.
- Remove unused flags from `config.toml`
+ Disallow `infodir` and `localstatedir` in `config.toml`
+ Allow the flags in `./configure`, but give a warning that they will be
ignored.
+ Fix incorrect comment that `datadir` will be ignored.
Example output:
```
$ ./configure --set install.infodir=xxx
configure: processing command line
configure:
configure: install.infodir := xxx
configure: build.configure-args := ['--set', 'install.infodir=xxx']
warning: infodir will be ignored
configure:
configure: writing `config.toml` in current directory
configure:
configure: run `python /home/joshua/rustc3/x.py --help`
configure:
```
- Update CHANGELOG
cc https://rust-lang.zulipchat.com/#narrow/stream/131828-t-compiler/topic/bootstrap.20defaults
|
|
- Add an assertion that `link-shared = true` when `thin-lto = true`.
Previously, link-shared would be silently overwritten.
- Get rid of `Option<bool>` in bootstrap/config.rs. Set defaults
immediately instead of delaying until later in bootstrap. This makes
it easier to find what the default value is.
- Remove redundant `config.x = false` when the default was already false
- Set defaults for `bindir` in `default_opts()` instead of `parse()`
- Update `download-ci-llvm = if-supported` option to match bootstrap.py
- Remove redundant check for link_shared. Previously, it was checked twice.
- Update various options in config.toml.example to their defaults.
Previously, some options showed an example value instead of the
default value.
- Fix incorrect defaults in config.toml.example
+ `use-libcxx` defaults to false
+ Add missing `check-stage = 0`
+ Update several defaults to be conditional (e.g. `if incremental { 10 } else { 100 }`)
- Remove redundant defaults in prose
- Use the same comment for the default and target-dependent `musl-root`
- Fix typos
- Link to `cc_detect` for `cc` and `cxx`, since the logic is ... complicated.
- Update more defaults to better reflect how they actually get set
- Remove ignored `gpg-password-file` option
This stopped being used in
https://github.com/rust-lang/rust/commit/7704d35accfe1b587ce41ea09ca3bf6a47aca117,
but was never removed from config.toml.
- Remove unused flags from `config.toml`
+ Disallow `infodir` and `localstatedir` in `config.toml`
+ Allow the flags in `./configure`, but give a warning that they will be
ignored.
+ Fix incorrect comment that `datadir` will be ignored.
Example output:
```
$ ./configure --set install.infodir=xxx
configure: processing command line
configure:
configure: install.infodir := xxx
configure: build.configure-args := ['--set', 'install.infodir=xxx']
warning: infodir will be ignored
configure:
configure: writing `config.toml` in current directory
configure:
configure: run `python /home/joshua/rustc3/x.py --help`
configure:
```
- Update CHANGELOG
- Add "as an example" where appropriate
- Link to an issue instead of to ephemeral chats
|
|
Add `download-rustc = "if-unchanged"`
This allows keeping the setting to a fixed value without having to
toggle it when you want to work on the compiler instead of on tools.
This sets `BOOTSTRAP_DOWNLOAD_RUSTC` in bootstrap.py so rustbuild doesn't have to try and replicate its logic.
Helps with https://github.com/rust-lang/rust/issues/81930.
r? `@Mark-Simulacrum` cc `@camelid`
|
|
This allows keeping the setting to a fixed value without having to
toggle it when you want to work on the compiler instead of on tools.
|
|
## Motivation
This avoids having to rebuild bootstrap and tidy each time you rebase
over master. In particular, it makes rebasing and running `x.py fmt` on
each commit in a branch significantly faster. It also avoids having to
rebuild bootstrap after setting `download-rustc = true`.
## Implementation
Instead of extracting the CI artifacts directly to `stage0/`, extract
them to `ci-rustc/` instead. Continue to copy them to the proper
sysroots as necessary for all stages except stage 0.
This also requires `bootstrap.py` to download both stage0 and CI
artifacts and distinguish between the two when checking stamp files.
Note that since tools have to be built by the same compiler that built
`rustc-dev` and the standard library, the downloaded artifacts can't be
reused when building with the beta compiler. To make sure this is still
a good user experience, warn when building with the beta compiler, and
default to building with stage 2.
|
|
|
|
This way, you can debug rustdoc's JavaScript and CSS file
with normal F12 Dev Tools and you'll have useful line numbers
to work with.
|
|
Intended to piggy-back on output from existing build.print_step_timings setting.
|
|
|
|
- Use the same compiler for stage0 and stage1. This should be fixed at
some point (so bootstrap isn't constantly rebuilt).
- Make sure `x.py build` and `x.py check` work.
- Use `git merge-base` to determine the most recent commit to download.
- Copy stage0 to the various sysroots in `Sysroot`, and delegate to
Sysroot in Assemble. Leave all other code unchanged.
- Rename date -> key
This can also be a commit hash, so 'date' is no longer a good name.
- Add the commented-out option to config.toml.example
- Disable all steps by default when `download-rustc` is enabled
Most steps don't make sense when downloading a compiler, because they'll
be pre-built in the sysroot. Only enable the ones that might be useful,
in particular Rustdoc and all `check` steps.
At some point, this should probably enable other tools, but rustdoc is
enough to test out `download-rustc`.
- Don't print 'Skipping' twice in a row
Bootstrap forcibly enables a dry run if it isn't already set, so
previously it would print the message twice:
```
Skipping bootstrap::compile::Std because it is not enabled for `download-rustc`
Skipping bootstrap::compile::Std because it is not enabled for `download-rustc`
```
Now it correctly only prints once.
## Future work
- Add FIXME about supporting beta commits
- Debug logging will never work. This should be fixed.
|
|
Support non-stage0 check
Seems to work locally - a full stage 1 check succeeds, building std (because we can't get away with checking it), and then checking the compiler and other tools. This ran into the problem that a unconditional x.py check in stage 1 *both* checks and builds stage 1 std, and then has to clean up because for some reason the rmeta and rlib artifacts conflict (though I'm not actually entirely sure why, but it doesn't seem worth digging in in too much detail).
Ideally we wouldn't be building and checking like that but it's a minor worry as checking std is pretty fast and you can avoid it if you're aiming for speed by passing the compiler (e.g., compiler/rustc) explicitly.
r? ```@jyn514```
|
|
|
|
- Don't ignore packaging `llvm/lib/` for `rust-dev` when LLVM is linked
statically
- Add `link-type.txt` so bootstrap knows whether llvm was linked
statically or dynamically
- Don't assume CI LLVM is linked dynamically in `bootstrap::config`
- Fall back to dynamic linking if `link-type.txt` doesn't exist
- Fix existing bug that split the output of `llvm-config` on lines, not spaces
- Enable building LLVM tests
This works around the following llvm bug:
```
llvm-config: error: component libraries and shared library
llvm-config: error: missing: /home/joshua/rustc2/build/x86_64-unknown-linux-gnu/llvm/build/lib/libgtest.a
llvm-config: error: missing: /home/joshua/rustc2/build/x86_64-unknown-linux-gnu/llvm/build/lib/libgtest_main.a
llvm-config: error: missing: /home/joshua/rustc2/build/x86_64-unknown-linux-gnu/llvm/build/lib/libLLVMTestingSupport.a
thread 'main' panicked at 'command did not execute successfully: "/home/joshua/rustc2/build/x86_64-unknown-linux-gnu/llvm/build/bin/llvm-config" "--libfiles"
```
I'm not sure why llvm-config thinks these are required, but to avoid
the error, this builds them anyway.
- Temporarily set windows as the try builder. This should be reverted
before merging.
- Bump version of `download-ci-llvm-stamp`
`src/llvm-project` hasn't changed, but the generated tarball has.
- Only special case MacOS when dynamic linking. Static linking works fine.
- Store `link-type.txt` to the top-level of the tarball
This allows writing the link type unconditionally. Previously, bootstrap
had to keep track of whether the file IO *would* succeed (it would fail
if `lib/` didn't exist), which was prone to bugs.
- Make `link-type.txt` required
Anyone downloading this from CI should be using a version of bootstrap
that matches the version of the uploaded artifacts. So a missing
link-type indicates a bug in x.py.
|
|
remove unnecessary trailing semicolon from bootstrap
|
|
|
|
The option allows to add or remove compression formats used while
producing dist tarballs.
|
|
Utilize PGO for rustc linux dist builds
This implements support for applying PGO to the rustc compilation step (not
standard library or any tooling, including rustdoc). Expanding PGO to more tools
is not terribly difficult but will involve more work and greater CI time
commitment.
For the same reason of avoiding greater implementation time commitment,
implementing for platforms outside of x86_64-unknown-linux-gnu is skipped.
In practice it should be quite simple to extend over time to more platforms. The
initial implementation is intentionally minimal here to avoid too much work
investment before we start seeing wins for a subset of Rust users.
The choice of workloads to profile here is somewhat arbitrary, but the general
rationale was to aim for a small set that largely avoided time regressions on
perf.rust-lang.org's full suite of crates. The set chosen is libcore, cargo (and
its dependencies), and a few ad-hoc stress tests from perf.rlo. The stress tests
are arguably the most controversial, but they benefit those cases (avoiding
regressions) and do not really remove wins from other benchmarks.
The primary next step after this PR lands is to implement support for PGO in
LLVM. It is unclear whether we can afford a full LLVM rebuild in CI, though, so
the approach taken there may need to be more staggered. rustc-only PGO seems
well affordable on linux at least, giving us up to 20% wall time wins on some
crates for 15 minutes of extra CI time (1 hour with this PR, up from 45 minutes).
The PGO data is uploaded to allow others to reuse it if attempting to reproduce
the CI build or potentially, in the future, on other platforms where an
off-by-one strategy is used for dist builds at minimal performance cost.
r? `@michaelwoerister` (but tell me if you don't want to / don't feel comfortable approving and we can find others)
|
|
This implements support for applying PGO to the rustc compilation step (not
standard library or any tooling, including rustdoc). Expanding PGO to more tools
is not terribly difficult but will involve more work and greater CI time
commitment.
For the same reason of avoiding greater time commitment, this currently avoids
implementing for platforms outside of x86_64-unknown-linux-gnu, though in
practice it should be quite simple to extend over time to more platforms. The
initial implementation is intentionally minimal here to avoid too much work
investment before we start seeing wins for a subset of Rust users.
The choice of workloads to profile here is somewhat arbitrary, but the general
rationale was to aim for a small set that largely avoided time regressions on
perf.rust-lang.org's full suite of crates. The set chosen is libcore, cargo (and
its dependencies), and a few ad-hoc stress tests from perf.rlo. The stress tests
are arguably the most controversial, but they benefit those cases (avoiding
regressions) and do not really remove wins from other benchmarks.
The primary next step after this PR lands is to implement support for PGO in
LLVM. It is unclear whether we can afford a full LLVM rebuild in CI, though, so
the approach taken there may need to be more staggered. rustc-only PGO seems
well affordable on linux at least, giving us up to 20% wall time wins on some
crates for 15 minutes of extra CI time (1 hour up from 45 minutes).
The PGO data is uploaded to allow others to reuse it if attempting to reproduce
the CI build or potentially, in the future, on other platforms where an
off-by-one strategy is used for dist builds at minimal performance cost.
|
|
`dsymutil` adds time to builds on Apple platforms for no clear benefit, and also
makes it more difficult for debuggers to find debug info. The compiler currently
defaults to running `dsymutil` to preserve its historical default, but when
compiling the compiler itself, we skip it by default since we know it's safe to
do so in that case.
|
|
|
|
x.py: allow a custom string appended to the version
This adds `rust.description` to the config as a descriptive string to be
appended to `rustc --version` output, which is also used in places like
debuginfo `DW_AT_producer`. This may be useful for supplementary build
information, like distro-specific package versions.
For example, in Fedora 33, `gcc --version` outputs:
gcc (GCC) 10.2.1 20201016 (Red Hat 10.2.1-6)
With this change, we can add similar vendor info to `rustc --version`.
|
|
|
|
This adds `rust.description` to the config as a descriptive string to be
appended to `rustc --version` output, which is also used in places like
debuginfo `DW_AT_producer`. This may be useful for supplementary build
information, like distro-specific package versions.
For example, in Fedora 33, `gcc --version` outputs:
gcc (GCC) 10.2.1 20201016 (Red Hat 10.2.1-6)
With this change, we can add similar vendor info to `rustc --version`.
|
|
This allows using bootstrap with https://github.com/Canop/bacon.
|
|
Support enable/disable sanitizers/profiler per target
This PR add options under `[target.*]` of `config.toml` which can enable or disable sanitizers/profiler runtime for corresponding target.
If these options are empty, the global options under `[build]` will take effect.
Fix #78329
|
|
Enable LLVM Polly via llvm-args.
I think doing it this way is better than in #51061. Polly has other useful options and we probably don't want to create a `-Z` flag for each one of them.
[Benchmark](https://gist.github.com/JRF63/9a6268b91720958e90dbe7abffe20298)
I noticed that `-lto` seems to interfere with polly in this specific microbenchmark, as enabling it causes the perf to drop to that of non-polly builds.
Other related PRs: #75615
|
|
Prior to this, setting the rustfmt configuration was ignored:
```
% mkdir example
% cd example
% ../configure --set build.rustfmt=/usr/bin/true
% ../x.py fmt
./x.py fmt is not supported on this channel
failed to run: /Users/shep/Projects/rust/example/build/bootstrap/debug/bootstrap fmt
Build completed unsuccessfully in 0:00:01
```
And after:
```
% ../x.py fmt
Build completed successfully in 0:00:11
```
|
|
|
|
|
