| Age | Commit message (Collapse) | Author | Lines |
|---|
|
Revert "remove num_cpus dependency" in rustc and update cargo
Fixes #97549. This PR reverts #94524 and does a Cargo update to pull in rust-lang/cargo#10737.
Rust 1.61.0 has a regression in which it misidentifies the number of available CPUs in some environments, leading to enormously increased memory usage and failing builds. In between Rust 1.60 and 1.61 both rustc and cargo replaced some uses of `num_cpus` with `available_parallelism`, which eliminated support for cgroupv1, still apparently in common use. This PR switches both rustc and cargo back to using `num_cpus` in order to support environments where the available parallelism is controlled by cgroupv1. Both can use `available_parallism` again once it handles cgroupv1 (if ever).
I have confirmed that the rustc part of this PR fixes the memory usage regression in my non-Cargo environment, and others have confirmed in #97549 that the Cargo regression was at fault for the memory usage regression in their environments.
|
|
|
|
|
|
Add compiletest and bootstrap "--skip" option forwarded to libtest
With this PR, "x.py test --skip SKIP ..." will run the specified test suite, but forward "--skip SKIP" to the test tool. libtest already supports this option. The PR also adds it to compiletest which itself just forwards it to libtest.
Adds the functionality requested in https://github.com/rust-lang/rust/issues/96342. This is useful to work around tests broken upstream.
https://github.com/rust-lang/rust/issues/96362#issuecomment-1108609893 is the specific test issue my project is trying to work around.
|
|
Fix running bootstrap tests on a fresh clone
In #96303, I changed the tests not to manage submodules, with the main
goal of avoiding a clone for llvm-project. Unfortunately, there are some tests
which depend on submodules - I didn't notice locally because they were already checked out for me,
and CI doesn't use submodule handling at all. Fresh clones, however, were impacted:
```
failures:
---- builder::tests::defaults::doc_default stdout ----
thread 'main' panicked at 'fs::read_dir(builder.src.join(&relative_path).join("redirects")) failed with No such file or directory (os error 2)', src/bootstrap/doc.rs:232:21
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
---- builder::tests::dist::dist_only_cross_host stdout ----
thread 'main' panicked at 'fs::read_to_string(&toml_file_name) failed with No such file or directory (os error 2)', src/bootstrap/lib.rs:1314:20
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
```
Try and get the best of both worlds by only checking out the submodules actually used in tests.
|
|
In #96303, I changed the tests not to manage submodules, with the main
goal of avoiding a clone for llvm-project. Unfortunately, there are some tests
which depend on submodules - I didn't notice locally because they were already checked out for me,
and CI doesn't use submodule handling at all. Fresh clones, however, were impacted:
```
failures:
---- builder::tests::defaults::doc_default stdout ----
thread 'main' panicked at 'fs::read_dir(builder.src.join(&relative_path).join("redirects")) failed with No such file or directory (os error 2)', src/bootstrap/doc.rs:232:21
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
---- builder::tests::dist::dist_only_cross_host stdout ----
thread 'main' panicked at 'fs::read_to_string(&toml_file_name) failed with No such file or directory (os error 2)', src/bootstrap/lib.rs:1314:20
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
```
Try and get the best of both worlds by only checking out the submodules actually used in tests.
|
|
|
|
libtest already supports a "--skip SUBSTRING" arg which excludes any
test names matching SUBSTRING.
This adds a "--skip" argument to compiletest and bootstrap which is
forwarded to libtest.
|
|
Make "Assemble stage1 compiler" orders of magnitude faster
This used to take upwards of 5 seconds for me locally. I found that the culprit was copying the downloaded LLVM shared object:
```
[22:28:03] Install "/home/jnelson/rust-lang/rust/build/x86_64-unknown-linux-gnu/ci-llvm/lib/libLLVM-14-rust-1.62.0-nightly.so" to "/home/jnelson/rust-lang/rust/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/libLLVM-14-rust-1.62.0-nightly.so"
[22:28:09] c Sysroot { compiler: Compiler { stage: 1, host: x86_64-unknown-linux-gnu(x86_64-unknown-linux-gnu) } }
```
It turned out that `install()` used full copies unconditionally. Change it to try using a hard-link before falling back to copying.
|
|
This used to take upwards of 5 seconds for me locally. I found that the culprit was copying the downloaded LLVM shared object:
```
[22:28:03] Install "/home/jnelson/rust-lang/rust/build/x86_64-unknown-linux-gnu/ci-llvm/lib/libLLVM-14-rust-1.62.0-nightly.so" to "/home/jnelson/rust-lang/rust/build/x86_64-unknown-linux-gnu/stage1/lib/rustlib/x86_64-unknown-linux-gnu/lib/libLLVM-14-rust-1.62.0-nightly.so"
[22:28:09] c Sysroot { compiler: Compiler { stage: 1, host: x86_64-unknown-linux-gnu(x86_64-unknown-linux-gnu) } }
```
It turned out that `install()` used full copies unconditionally. Change it to try using a hard-link before falling back to copying.
|
|
Fix the `x.py clippy` command
Fixes #96826, confirmed clippy runs correctly on local
r? `@Mark-Simulacrum`
|
|
Move check-bootstrap from a makefile rule to test::Bootstrap
Fixes #96688
|
|
Improve error for missing cmake
This PR updates the error message for a missing `cmake` to be more in line with the error message for a missing installation of `ninja`.
The original issue, (#90679), suggests that both `ninja` and `cmake` are only needed for building LLVM, so I have included the suggestion from `ninja` to set `download-ci-llvm = true` if the user would rather download LLVM. If `cmake` actually is used in other areas, I can remove that part of the message.
Fixes: #90679
|
|
Initialize rust-analyzer submodule on bootstrap
Fixes #96456
|
|
|
|
Rollup of 10 pull requests
Successful merges:
- #96336 (Link to correct `as_mut` in docs for `pointer::as_ref`)
- #96586 (Add aliases for std::fs::canonicalize)
- #96667 (Add regression test)
- #96671 (Remove hard links from `env::current_exe` security example)
- #96726 (Add regression and bug tests)
- #96756 (Enable compiler-docs by default for `compiler`, `codegen`, and `tools` profiles)
- #96757 (Don't constantly rebuild clippy on `x test src/tools/clippy`.)
- #96769 (Remove `adx_target_feature` feature from active features list)
- #96777 (Make the test `check-pass` not to produce a JSON file)
- #96822 (Enforce quote rule for JS source code)
Failed merges:
r? `@ghost`
`@rustbot` modify labels: rollup
|
|
|
|
Don't constantly rebuild clippy on `x test src/tools/clippy`.
This happened because the `SYSROOT` variable was set for `x test`, but not `x build`.
Set it consistently for both to avoid unnecessary rebuilds.
This is a very small step towards https://github.com/rust-lang/rust/issues/76495.
|
|
Enable compiler-docs by default for `compiler`, `codegen`, and `tools` profiles
I had this overridden locally for a while and realized just now it should probably just be a default.
|
|
Enable cfg checking of cargo features for everything but std
This PR enable `cfg` checking of cargo features for everything but std, it also adds a `FIXME` to myself.
> Note: `std`, `alloc` and `core` imports some dependencies by #[path] (like
> backtrace, core_simd, std_float, ...), those dependencies have their own features
> but cargo isn't involved in the #[path] and so cannot pass the complete list of
> features, so for that reason we don't enable checking of features for std.
r? `@Mark-Simulacrum`
|
|
r=Mark-Simulacrum
bootstrap: bsd platform flags for split debuginfo
Addresses https://github.com/rust-lang/rust/pull/96597#issuecomment-1118905025.
Bootstrap currently provides `-Zunstable-options` for OpenBSD when using split debuginfo - this commit provides it for all BSD targets.
We should probably work out a better way of handling the stability of the split debuginfo flag - all options for the flag are unstable but one of them is the default for each platform already.
cc `@m-ou-se`
r? `@Mark-Simulacrum`
|
|
[bootstrap] Give a better error when trying to run a path with no registered step
Before:
```
thread 'main' panicked at 'error: no rules matched invalid', src/bootstrap/builder.rs:287:17
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
```
After:
```
error: no `check` rules matched 'invalid'
help: run `x.py check --help --verbose` to show a list of available paths
note: if you are adding a new Step to bootstrap itself, make sure you register it with `describe!`
```
|
|
|
|
Bootstrap currently provides `-Zunstable-options` for OpenBSD when using
split debuginfo - this commit provides it for all BSD targets.
Signed-off-by: David Wood <david.wood@huawei.com>
|
|
This happened because the `SYSROOT` variable was set for `x test`, but not `x build`.
Set it consistently for both to avoid unnecessary rebuilds.
|
|
|
|
r=davidtwco,Mark-Simulacrum
openbsd: unbreak build on native platform
after #95612, only linux and windows target are build with `-Zunstable-options`, but others platforms might use `-Csplit-debuginfo`
currently, without this PR, the build of rustc on OpenBSD fails with:
```
Building stage0 tool unstable-book-gen (x86_64-unknown-openbsd)
running: "/data/semarie/build-rust/install_dir/beta/bin/cargo" "build" "--target" "x86_64-unknown-openbsd" "-Zbinary-dep-depinfo" "-j" "4" "-v" "--release" "--frozen" "--manifest-path"
"/data/semarie/build-rust/build_dir/rustc-nightly-src/src/tools/unstable-book-gen/Cargo.toml" "--message-format" "json-render-diagnostics"
error: failed to run `rustc` to learn about target-specific information
Caused by:
process didn't exit successfully: `/data/semarie/build-rust/build_dir/build/bootstrap/debug/rustc - --crate-name ___ --print=file-names --cfg=bootstrap -Csymbol-mangling-version=v0 -Zmacro-backtrace -Clink-args=-Wl,-z,origin
'-Clink-args=-Wl,-rpath,$ORIGIN/../lib' -Csplit-debuginfo=off -Ztls-model=initial-exec --target x86_64-unknown-openbsd --crate-type bin --crate-type rlib --crate-type dylib --crate-type cdylib --crate-type staticlib --crate-type proc-macro
--print=sysroot --print=cfg` (exit status: 1)
--- stdout
Did not run successfully: exit status: 1
"/data/semarie/build-rust/install_dir/beta/bin/rustc" "-" "--crate-name" "___" "--print=file-names" "--cfg=bootstrap" "-Csymbol-mangling-version=v0" "-Zmacro-backtrace" "-Clink-args=-Wl,-z,origin" "-Clink-args=-Wl,-rpath,$ORIGIN/../lib"
"-Csplit-debuginfo=off" "-Ztls-model=initial-exec" "--target" "x86_64-unknown-openbsd" "--crate-type" "bin" "--crate-type" "rlib" "--crate-type" "dylib" "--crate-type" "cdylib" "--crate-type" "staticlib" "--crate-type" "proc-macro"
"--print=sysroot" "--print=cfg" "-Wrust_2018_idioms" "-Wunused_lifetimes" "-Wsemicolon_in_expressions_from_macros" "-Dwarnings" "--sysroot" "/data/semarie/build-rust/install_dir/beta"
-------------
--- stderr
error: `-Csplit-debuginfo` is unstable on this platform
command did not execute successfully: "/data/semarie/build-rust/install_dir/beta/bin/cargo.bin" "build" "--target" "x86_64-unknown-openbsd" "-Zbinary-dep-depinfo" "-j" "4" "-v" "--release" "--frozen" "--manifest-path"
```
I am suspecting that all unix might be affected, but I am unsure about the right conditional to use. so I only added "openbsd" target inside it as I am able to test it.
rustc nightly built correctly with this PR on openbsd.
|
|
|
|
step
Before:
```
thread 'main' panicked at 'error: no rules matched invalid', src/bootstrap/builder.rs:287:17
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
```
After:
```
error: no `check` rules matched 'invalid'
help: run `x.py check --help --verbose` to show a list of available paths
note: if you are adding a new Step to bootstrap itself, make sure you register it with `describe!`
```
|
|
Remove DebbugerScripts from bootstrap CLI
This PR implements #95992 and removes the debugger scripts from the bootstrap CLI.
I could not find a lot of documentation on the bootstrap binary so perhaps there's still some documentation to be updated.
|
|
|
|
after #95612, only linux and windows target are build with -Zunstable-options,
but others platforms might use -Csplit-debuginfo
add "openbsd" target in the list of platforms using it.
|
|
Make it possible to write doctests for bootstrap
This probably isn't super useful in practice, but it was easy to fix
and avoids confusing errors about mismatched versions between beta and the default toolchain.
|
|
|
|
This probably isn't super useful in practice, but it was easy to fix
and avoids confusing errors about mismatched versions between beta and the default toolchain.
|
|
Move `download-ci-llvm` out of bootstrap.py
This is ready for review. It has been tested on Windows, Linux, and NixOS.
The second commit ports the changes from https://github.com/rust-lang/rust/pull/95234 to Rust; I can remove it if desired.
Helps with https://github.com/rust-lang/rust/issues/94829.
As a follow-up, this makes it possible to avoid downloading llvm until it's needed for building `rustc_llvm`; it would be nice to do that, but it shouldn't go in the first draft. It might also be possible to avoid requiring python until tests run (currently there's a check in `sanity.rs`), but I haven't looked too much into that.
`@rustbot` label +A-rustbuild
|
|
Add type_name info to [TIMING] log output
Adds type_name to the [TIMING] log output:
```
[TIMING] (bootstrap::compile::Sysroot) Sysroot { compiler: Compiler { stage: 0, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } } -- 0.020
[TIMING] (bootstrap::builder::Builder::sysroot_libdir::Libdir) Libdir { compiler: Compiler { stage: 0, host: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } }, target: TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None } } -- 0.007
```
Not sure if that's the best way to format it. Thought about replacing the struct's name with the type_name output, but that feels kind of hacky?
Closes #96060
|
|
|
|
|
|
This attempts to keep the logic as close to the original python as possible.
`probably_large` has been removed, since it was always `True`, and UTF-8 paths are no longer supported when patching files for NixOS.
I can readd UTF-8 support if desired.
Note that this required making `llvm_link_shared` computed on-demand,
since we don't know whether it will be static or dynamic until we download LLVM from CI.
|
|
bootstrap: Remove dead code in rustdoc shim
The `RUSTDOC_RESOURCE_SUFFIX` variable was never actually set.
|
|
The `RUSTDOC_RESOURCE_SUFFIX` variable was never actually set.
|
|
Improve bootstrap tests
- Don't checkout submodules in bootstrap tests
This doesn't cause any tests to fail, and can greatly speed them up.
- Add a test for --exclude test::XXX
I didn't know that the `test::` syntax was valid before, and it doesn't seem to be documented anywhere. Add a test so it doesn't regress accidentally, and as executable documentation.
This also moves the `exclude` tests out of `dist`, to avoid assertion errors when the `cmd` passed to configure didn't match the `subcommand` used.
- Use run_build helper consistently across most bootstrap tests
This is not super important to do, but the consistency is nice.
I didn't change any tests that call `configure("dist")` and then override the subcommand - doing
that at all is pretty sketchy, but I don't want to mess with it while already doing a refactor.
Found while working on the "one call to Step for all paths" change mentioned in https://github.com/rust-lang/rust/pull/95503#issuecomment-1105914384, but independent of that work.
cc `@pietroalbini` for the `--exclude` test, git blame says you added support for it originally.
|
|
compiletest: combine `--*-python` args
Since these arguments are now always the same, combine them into a
singular `--python` argument.
Fixes #96011
|
|
Add `x {check,build,doc} {compiler,library}` aliases.
While working on https://github.com/rust-lang/rust/pull/95503, I realized that it will interfere with existing command lines:
Currently people run `x build library/std` expecting it to "add all library crates to the sysroot",
but after that change, it will *only* build `libstd` and its dependencies (and add them to the sysroot), not libtest or libproc_macro.
That will work for local testing in most cases, but could be confusing. Even if not, though, I think `x build library` is more clear about what actually happens than the current `x build library/std`.
The intended end goal is something like:
- For check/build/doc, we have library + compiler aliases, which correspond to basically "most possible" for that piece. This is the intended path of entry (rather than library/test or similar as today) for when you just want the thing to work -- for example, getting a compiler that is "crates.io-compatible" would be roughly `x.py build library`). #95504
- Specific crate invocations build up to that crate, which means that if you don't care about tests you probably want x.py build library/proc_macro or library/std for faster build times. #95503
Note that this is already implemented today for the `doc` command and seems to work pretty well in practice.
I plan to change the dev-guide and various instructions in the README to `build library` once this is merged.
`@rustbot` label +A-rustbuild
|
|
|
|
Ensure existance of dist directory when creating tarball
I'm not sure why this works in CI, but this is necessary to make distcheck (including the `x86_64-linux-distcheck` image) run on Fedora 35.
|
|
This is not super important to do, but the consistency is nice.
I didn't change any tests that call `configure("dist")` and then override the subcommand - doing
that at all is pretty sketchy, but I don't want to mess with it while already doing a refactor.
|
|
I didn't know that the `test::` syntax was valid before, and it doesn't
seem to be documented anywhere. Add a test so it doesn't regress accidentally,
and as executable documentation.
|
|
This doesn't cause any tests to fail, and can greatly speed them up.
|
