diff options
| author | The rustc-josh-sync Cronjob Bot <github-actions@github.com> | 2025-09-25 04:16:26 +0000 |
|---|---|---|
| committer | The rustc-josh-sync Cronjob Bot <github-actions@github.com> | 2025-09-25 04:16:26 +0000 |
| commit | ff7e7319837d5029ed4f522601debc2a7bc33274 (patch) | |
| tree | 42dbcb2d8c347a2e7c97c7e4361b1c361143da0b /src/doc/rustc-dev-guide | |
| parent | 3280c210dd6e5f26595ff4857572f8feba678992 (diff) | |
| parent | d226e7aa93425ba2090f423642341a99ab047838 (diff) | |
| download | rust-ff7e7319837d5029ed4f522601debc2a7bc33274.tar.gz rust-ff7e7319837d5029ed4f522601debc2a7bc33274.zip | |
Merge ref 'caccb4d0368b' from rust-lang/rust
Pull recent changes from https://github.com/rust-lang/rust via Josh. Upstream ref: caccb4d0368bd918ef6668af8e13834d07040417 Filtered ref: 0f345ed05d559bbfb754f1403b16199366cda2e0 Upstream diff: https://github.com/rust-lang/rust/compare/21a19c297d4f5a03501d92ca251bd7a17073c08a...caccb4d0368bd918ef6668af8e13834d07040417 This merge was created using https://github.com/rust-lang/josh-sync.
Diffstat (limited to 'src/doc/rustc-dev-guide')
74 files changed, 857 insertions, 443 deletions
diff --git a/src/doc/rustc-dev-guide/.github/workflows/ci.yml b/src/doc/rustc-dev-guide/.github/workflows/ci.yml index 6eabb999fb0..2dd695b7a47 100644 --- a/src/doc/rustc-dev-guide/.github/workflows/ci.yml +++ b/src/doc/rustc-dev-guide/.github/workflows/ci.yml @@ -14,7 +14,7 @@ jobs: if: github.repository == 'rust-lang/rustc-dev-guide' runs-on: ubuntu-latest env: - MDBOOK_VERSION: 0.4.48 + MDBOOK_VERSION: 0.4.52 MDBOOK_LINKCHECK2_VERSION: 0.9.1 MDBOOK_MERMAID_VERSION: 0.12.6 MDBOOK_OUTPUT__LINKCHECK__FOLLOW_WEB_LINKS: ${{ github.event_name != 'pull_request' }} diff --git a/src/doc/rustc-dev-guide/.github/workflows/rustc-pull.yml b/src/doc/rustc-dev-guide/.github/workflows/rustc-pull.yml index 04d6469aeaa..5ff3118960d 100644 --- a/src/doc/rustc-dev-guide/.github/workflows/rustc-pull.yml +++ b/src/doc/rustc-dev-guide/.github/workflows/rustc-pull.yml @@ -3,8 +3,8 @@ name: rustc-pull on: workflow_dispatch: schedule: - # Run at 04:00 UTC every Monday and Thursday - - cron: '0 4 * * 1,4' + # Run at 04:00 UTC every Monday + - cron: '0 4 * * 1' jobs: pull: diff --git a/src/doc/rustc-dev-guide/rust-version b/src/doc/rustc-dev-guide/rust-version index 6ec700b9b4d..0dc9ce843e9 100644 --- a/src/doc/rustc-dev-guide/rust-version +++ b/src/doc/rustc-dev-guide/rust-version @@ -1 +1 @@ -6bcdcc73bd11568fd85f5a38b58e1eda054ad1cd +9f32ccf35fb877270bc44a86a126440f04d676d0 diff --git a/src/doc/rustc-dev-guide/src/SUMMARY.md b/src/doc/rustc-dev-guide/src/SUMMARY.md index 9ded467d5cd..249140956c0 100644 --- a/src/doc/rustc-dev-guide/src/SUMMARY.md +++ b/src/doc/rustc-dev-guide/src/SUMMARY.md @@ -103,11 +103,12 @@ - [The `rustdoc-json` test suite](./rustdoc-internals/rustdoc-json-test-suite.md) - [GPU offload internals](./offload/internals.md) - [Installation](./offload/installation.md) + - [Usage](./offload/usage.md) - [Autodiff internals](./autodiff/internals.md) - [Installation](./autodiff/installation.md) - [How to debug](./autodiff/debugging.md) - [Autodiff flags](./autodiff/flags.md) - - [Current limitations](./autodiff/limitations.md) + - [Type Trees](./autodiff/type-trees.md) # Source Code Representation diff --git a/src/doc/rustc-dev-guide/src/about-this-guide.md b/src/doc/rustc-dev-guide/src/about-this-guide.md index 057e4a4ccee..4f5733ae082 100644 --- a/src/doc/rustc-dev-guide/src/about-this-guide.md +++ b/src/doc/rustc-dev-guide/src/about-this-guide.md @@ -48,9 +48,9 @@ In addition, many of the ideas discussed throughout this guide are idealized des that are not fully realized yet. All this makes keeping this guide completely up to date on everything very hard! -The Guide itself is of course open-source as well, -and the sources can be found at the [GitHub repository]. -If you find any mistakes in the guide, please file an issue about it. +The guide itself is of course open source as well, +and the sources are hosted on [a GitHub repository]. +If you find any mistakes in the guide, please file an issue. Even better, open a PR with a correction! If you do contribute to the guide, @@ -73,8 +73,7 @@ You might also find the following sites useful: - [compiler-team] -- the home-base for the Rust compiler team, with description of the team procedures, active working groups, and the team calendar. - [std-dev-guide] -- a similar guide for developing the standard library. -- [The t-compiler zulip][z] -- `#contribute` and `#wg-rustup` on [Discord](https://discord.gg/rust-lang). +- [The t-compiler Zulip][z] - The [Rust Internals forum][rif], a place to ask questions and discuss Rust's internals - The [Rust reference][rr], even though it doesn't specifically talk about @@ -106,7 +105,7 @@ You might also find the following sites useful: [cheatsheet]: https://bors.rust-lang.org/ [Miri]: https://github.com/rust-lang/miri [@bors]: https://github.com/bors -[GitHub repository]: https://github.com/rust-lang/rustc-dev-guide/ +[a GitHub repository]: https://github.com/rust-lang/rustc-dev-guide/ [rustc API docs]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_middle [Forge]: https://forge.rust-lang.org/ [compiler-team]: https://github.com/rust-lang/compiler-team/ diff --git a/src/doc/rustc-dev-guide/src/appendix/code-index.md b/src/doc/rustc-dev-guide/src/appendix/code-index.md index 65fbf752d79..4ddb58b0c39 100644 --- a/src/doc/rustc-dev-guide/src/appendix/code-index.md +++ b/src/doc/rustc-dev-guide/src/appendix/code-index.md @@ -13,7 +13,7 @@ Item | Kind | Short description | Chapter | `DefId` | struct | One of four types of HIR node identifiers | [Identifiers in the HIR] | [compiler/rustc_hir/src/def_id.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc_hir/def_id/struct.DefId.html) `Diag` | struct | A struct for a compiler diagnostic, such as an error or lint | [Emitting Diagnostics] | [compiler/rustc_errors/src/diagnostic.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc_errors/struct.Diag.html) `DocContext` | struct | A state container used by rustdoc when crawling through a crate to gather its documentation | [Rustdoc] | [src/librustdoc/core.rs](https://github.com/rust-lang/rust/blob/master/src/librustdoc/core.rs) -`HirId` | struct | One of four types of HIR node identifiers | [Identifiers in the HIR] | [compiler/rustc_hir/src/hir_id.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc_hir/hir_id/struct.HirId.html) +`HirId` | struct | One of four types of HIR node identifiers | [Identifiers in the HIR] | [compiler/rustc_hir_id/src/lib.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc_hir/struct.HirId.html) `Lexer` | struct | This is the lexer used during parsing. It consumes characters from the raw source code being compiled and produces a series of tokens for use by the rest of the parser | [The parser] | [compiler/rustc_parse/src/lexer/mod.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc_parse/lexer/struct.Lexer.html) `NodeId` | struct | One of four types of HIR node identifiers. Being phased out | [Identifiers in the HIR] | [compiler/rustc_ast/src/ast.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc_ast/node_id/struct.NodeId.html) `P` | struct | An owned immutable smart pointer. By contrast, `&T` is not owned, and `Box<T>` is not immutable. | None | [compiler/rustc_ast/src/ptr.rs](https://doc.rust-lang.org/nightly/nightly-rustc/rustc_ast/ptr/struct.P.html) diff --git a/src/doc/rustc-dev-guide/src/appendix/glossary.md b/src/doc/rustc-dev-guide/src/appendix/glossary.md index 1837b59e850..21162f8ee7d 100644 --- a/src/doc/rustc-dev-guide/src/appendix/glossary.md +++ b/src/doc/rustc-dev-guide/src/appendix/glossary.md @@ -68,6 +68,7 @@ Term | Meaning <span id="rib">rib</span> | A data structure in the name resolver that keeps track of a single scope for names. ([see more](../name-resolution.md)) <span id="rpit">RPIT</span> | A return-position `impl Trait`. ([see the reference](https://doc.rust-lang.org/reference/types/impl-trait.html#abstract-return-types)). <span id="rpitit">RPITIT</span> | A return-position `impl Trait` in trait. Unlike RPIT, this is desugared to a generic associated type (GAT). Introduced in [RFC 3425](https://rust-lang.github.io/rfcs/3425-return-position-impl-trait-in-traits.html). ([see more](../return-position-impl-trait-in-trait.md)) +<span id="rustbuild">rustbuild</span> | A deprecated term for the part of bootstrap that is written in Rust <span id="scrutinee">scrutinee</span> | A scrutinee is the expression that is matched on in `match` expressions and similar pattern matching constructs. For example, in `match x { A => 1, B => 2 }`, the expression `x` is the scrutinee. <span id="sess">`sess`</span> | The compiler _session_, which stores global data used throughout compilation <span id="side-tables">side tables</span> | Because the [AST](#ast) and HIR are immutable once created, we often carry extra information about them in the form of hashtables, indexed by the id of a particular node. diff --git a/src/doc/rustc-dev-guide/src/autodiff/internals.md b/src/doc/rustc-dev-guide/src/autodiff/internals.md index c1b31a0e4bd..c8e304f814b 100644 --- a/src/doc/rustc-dev-guide/src/autodiff/internals.md +++ b/src/doc/rustc-dev-guide/src/autodiff/internals.md @@ -17,7 +17,7 @@ fn main() { The detailed documentation for the `std::autodiff` module is available at [std::autodiff](https://doc.rust-lang.org/std/autodiff/index.html). -Differentiable programing is used in various fields like numerical computing, [solid mechanics][ratel], [computational chemistry][molpipx], [fluid dynamics][waterlily] or for Neural Network training via Backpropagation, [ODE solver][diffsol], [differentiable rendering][libigl], [quantum computing][catalyst], and climate simulations. +Differentiable programming is used in various fields like numerical computing, [solid mechanics][ratel], [computational chemistry][molpipx], [fluid dynamics][waterlily] or for Neural Network training via Backpropagation, [ODE solver][diffsol], [differentiable rendering][libigl], [quantum computing][catalyst], and climate simulations. [ratel]: https://gitlab.com/micromorph/ratel [molpipx]: https://arxiv.org/abs/2411.17011v diff --git a/src/doc/rustc-dev-guide/src/autodiff/limitations.md b/src/doc/rustc-dev-guide/src/autodiff/limitations.md deleted file mode 100644 index 90afbd51f3f..00000000000 --- a/src/doc/rustc-dev-guide/src/autodiff/limitations.md +++ /dev/null @@ -1,27 +0,0 @@ -# Current limitations - -## Safety and Soundness - -Enzyme currently assumes that the user passes shadow arguments (`dx`, `dy`, ...) of appropriate size. Under Reverse Mode, we additionally assume that shadow arguments are mutable. In Reverse Mode we adjust the outermost pointer or reference to be mutable. Therefore `&f32` will receive the shadow type `&mut f32`. However, we do not check length for other types than slices (e.g. enums, Vec). We also do not enforce mutability of inner references, but will warn if we recognize them. We do intend to add additional checks over time. - -## ABI adjustments - -In some cases, a function parameter might get lowered in a way that we currently don't handle correctly, leading to a compile time type mismatch in the `rustc_codegen_llvm` backend. Here are some [examples](https://github.com/EnzymeAD/rust/issues/105). - -## Compile Times - -Enzyme will often achieve excellent runtime performance, but might increase your compile time by a large factor. For Rust, we already have made significant improvements and have a list of further improvements planed - please reach out if you have time to help here. - -### Type Analysis - -Most of the times, Type Analysis (TA) is the reason of large (>5x) compile time increases when using Enzyme. This poster explains why we need to run Type Analysis in the bottom left part: [Poster Link](https://c.wsmoses.com/posters/Enzyme-llvmdev.pdf). - -We intend to increase the number of locations where we pass down Type information based on Rust types, which in turn will reduce the number of locations where Enzyme has to run Type Analysis, which will help compile times. - -### Duplicated Optimizations - -The key reason for Enzyme offering often excellent performance is that Enzyme differentiates already optimized LLVM-IR. However, we also (have to) run LLVM's optimization pipeline after differentiating, to make sure that the code which Enzyme generates is optimized properly. As a result you should have excellent runtime performance (please fill an issue if not), but at a compile time cost for running optimizations twice. - -### Fat-LTO - -The usage of `#[autodiff(...)]` currently requires compiling your project with Fat-LTO. We technically only need LTO if the function being differentiated calls functions in other compilation units. Therefore, other solutions are possible, but this is the most simple one to get started. diff --git a/src/doc/rustc-dev-guide/src/autodiff/type-trees.md b/src/doc/rustc-dev-guide/src/autodiff/type-trees.md new file mode 100644 index 00000000000..68cb78650b0 --- /dev/null +++ b/src/doc/rustc-dev-guide/src/autodiff/type-trees.md @@ -0,0 +1,193 @@ +# TypeTrees for Autodiff + +## What are TypeTrees? +Memory layout descriptors for Enzyme. Tell Enzyme exactly how types are structured in memory so it can compute derivatives efficiently. + +## Structure +```rust +TypeTree(Vec<Type>) + +Type { + offset: isize, // byte offset (-1 = everywhere) + size: usize, // size in bytes + kind: Kind, // Float, Integer, Pointer, etc. + child: TypeTree // nested structure +} +``` + +## Example: `fn compute(x: &f32, data: &[f32]) -> f32` + +**Input 0: `x: &f32`** +```rust +TypeTree(vec![Type { + offset: -1, size: 8, kind: Pointer, + child: TypeTree(vec![Type { + offset: 0, size: 4, kind: Float, // Single value: use offset 0 + child: TypeTree::new() + }]) +}]) +``` + +**Input 1: `data: &[f32]`** +```rust +TypeTree(vec![Type { + offset: -1, size: 8, kind: Pointer, + child: TypeTree(vec![Type { + offset: -1, size: 4, kind: Float, // -1 = all elements + child: TypeTree::new() + }]) +}]) +``` + +**Output: `f32`** +```rust +TypeTree(vec![Type { + offset: 0, size: 4, kind: Float, // Single scalar: use offset 0 + child: TypeTree::new() +}]) +``` + +## Why Needed? +- Enzyme can't deduce complex type layouts from LLVM IR +- Prevents slow memory pattern analysis +- Enables correct derivative computation for nested structures +- Tells Enzyme which bytes are differentiable vs metadata + +## What Enzyme Does With This Information: + +Without TypeTrees: +```llvm +; Enzyme sees generic LLVM IR: +define float @distance(ptr %p1, ptr %p2) { +; Has to guess what these pointers point to +; Slow analysis of all memory operations +; May miss optimization opportunities +} +``` + +With TypeTrees: +```llvm +define "enzyme_type"="{[-1]:Float@float}" float @distance( + ptr "enzyme_type"="{[-1]:Pointer, [-1,0]:Float@float}" %p1, + ptr "enzyme_type"="{[-1]:Pointer, [-1,0]:Float@float}" %p2 +) { +; Enzyme knows exact type layout +; Can generate efficient derivative code directly +} +``` + +# TypeTrees - Offset and -1 Explained + +## Type Structure + +```rust +Type { + offset: isize, // WHERE this type starts + size: usize, // HOW BIG this type is + kind: Kind, // WHAT KIND of data (Float, Int, Pointer) + child: TypeTree // WHAT'S INSIDE (for pointers/containers) +} +``` + +## Offset Values + +### Regular Offset (0, 4, 8, etc.) +**Specific byte position within a structure** + +```rust +struct Point { + x: f32, // offset 0, size 4 + y: f32, // offset 4, size 4 + id: i32, // offset 8, size 4 +} +``` + +TypeTree for `&Point` (internal representation): +```rust +TypeTree(vec![ + Type { offset: 0, size: 4, kind: Float }, // x at byte 0 + Type { offset: 4, size: 4, kind: Float }, // y at byte 4 + Type { offset: 8, size: 4, kind: Integer } // id at byte 8 +]) +``` + +Generates LLVM +```llvm +"enzyme_type"="{[-1]:Pointer, [-1,0]:Float@float, [-1,4]:Float@float, [-1,8]:Integer, [-1,9]:Integer, [-1,10]:Integer, [-1,11]:Integer}" +``` + +### Offset -1 (Special: "Everywhere") +**Means "this pattern repeats for ALL elements"** + +#### Example 1: Direct Array `[f32; 100]` (no pointer indirection) +```rust +TypeTree(vec![Type { + offset: -1, // ALL positions + size: 4, // each f32 is 4 bytes + kind: Float, // every element is float +}]) +``` + +Generates LLVM: `"enzyme_type"="{[-1]:Float@float}"` + +#### Example 1b: Array Reference `&[f32; 100]` (with pointer indirection) +```rust +TypeTree(vec![Type { + offset: -1, size: 8, kind: Pointer, + child: TypeTree(vec![Type { + offset: -1, // ALL array elements + size: 4, // each f32 is 4 bytes + kind: Float, // every element is float + }]) +}]) +``` + +Generates LLVM: `"enzyme_type"="{[-1]:Pointer, [-1,-1]:Float@float}"` + +Instead of listing 100 separate Types with offsets `0,4,8,12...396` + +#### Example 2: Slice `&[i32]` +```rust +// Pointer to slice data +TypeTree(vec![Type { + offset: -1, size: 8, kind: Pointer, + child: TypeTree(vec![Type { + offset: -1, // ALL slice elements + size: 4, // each i32 is 4 bytes + kind: Integer + }]) +}]) +``` + +Generates LLVM: `"enzyme_type"="{[-1]:Pointer, [-1,-1]:Integer}"` + +#### Example 3: Mixed Structure +```rust +struct Container { + header: i64, // offset 0 + data: [f32; 1000], // offset 8, but elements use -1 +} +``` + +```rust +TypeTree(vec![ + Type { offset: 0, size: 8, kind: Integer }, // header + Type { offset: 8, size: 4000, kind: Pointer, + child: TypeTree(vec![Type { + offset: -1, size: 4, kind: Float // ALL array elements + }]) + } +]) +``` + +## Key Distinction: Single Values vs Arrays + +**Single Values** use offset `0` for precision: +- `&f32` has exactly one f32 value at offset 0 +- More precise than using -1 ("everywhere") +- Generates: `{[-1]:Pointer, [-1,0]:Float@float}` + +**Arrays** use offset `-1` for efficiency: +- `&[f32; 100]` has the same pattern repeated 100 times +- Using -1 avoids listing 100 separate offsets +- Generates: `{[-1]:Pointer, [-1,-1]:Float@float}` \ No newline at end of file diff --git a/src/doc/rustc-dev-guide/src/backend/debugging.md b/src/doc/rustc-dev-guide/src/backend/debugging.md index 4f8712dfaf3..3dc95f25d4a 100644 --- a/src/doc/rustc-dev-guide/src/backend/debugging.md +++ b/src/doc/rustc-dev-guide/src/backend/debugging.md @@ -183,7 +183,7 @@ The quick summary is: ### Getting help and asking questions If you have some questions, head over to the [rust-lang Zulip] and -specifically the `#t-compiler/wg-llvm` stream. +specifically the `#t-compiler/wg-llvm` channel. [rust-lang Zulip]: https://rust-lang.zulipchat.com/ diff --git a/src/doc/rustc-dev-guide/src/building/bootstrapping/debugging-bootstrap.md b/src/doc/rustc-dev-guide/src/building/bootstrapping/debugging-bootstrap.md index 9c5ebbd36c4..93b11c0690a 100644 --- a/src/doc/rustc-dev-guide/src/building/bootstrapping/debugging-bootstrap.md +++ b/src/doc/rustc-dev-guide/src/building/bootstrapping/debugging-bootstrap.md @@ -1,120 +1,102 @@ # Debugging bootstrap -There are two main ways to debug bootstrap itself. The first is through println logging, and the second is through the `tracing` feature. - -> FIXME: this section should be expanded +There are two main ways of debugging (and profiling bootstrap). The first is through println logging, and the second is through the `tracing` feature. ## `println` logging Bootstrap has extensive unstructured logging. Most of it is gated behind the `--verbose` flag (pass `-vv` for even more detail). -If you want to know which `Step` ran a command, you could invoke bootstrap like so: +If you want to see verbose output of executed Cargo commands and other kinds of detailed logs, pass `-v` or `-vv` when invoking bootstrap. Note that the logs are unstructured and may be overwhelming. ``` $ ./x dist rustc --dry-run -vv learning about cargo running: RUSTC_BOOTSTRAP="1" "/home/jyn/src/rust2/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "metadata" "--format-version" "1" "--no-deps" "--manifest-path" "/home/jyn/src/rust2/Cargo.toml" (failure_mode=Exit) (created at src/bootstrap/src/core/metadata.rs:81:25, executed at src/bootstrap/src/core/metadata.rs:92:50) running: RUSTC_BOOTSTRAP="1" "/home/jyn/src/rust2/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "metadata" "--format-version" "1" "--no-deps" "--manifest-path" "/home/jyn/src/rust2/library/Cargo.toml" (failure_mode=Exit) (created at src/bootstrap/src/core/metadata.rs:81:25, executed at src/bootstrap/src/core/metadata.rs:92:50) -> Assemble { target_compiler: Compiler { stage: 1, host: x86_64-unknown-linux-gnu } } - > Libdir { compiler: Compiler { stage: 1, host: x86_64-unknown-linux-gnu }, target: x86_64-unknown-linux-gnu } - > Sysroot { compiler: Compiler { stage: 1, host: x86_64-unknown-linux-gnu }, force_recompile: false } -Removing sysroot /home/jyn/src/rust2/build/tmp-dry-run/x86_64-unknown-linux-gnu/stage1 to avoid caching bugs - < Sysroot { compiler: Compiler { stage: 1, host: x86_64-unknown-linux-gnu }, force_recompile: false } - < Libdir { compiler: Compiler { stage: 1, host: x86_64-unknown-linux-gnu }, target: x86_64-unknown-linux-gnu } -... -``` - -This will go through all the recursive dependency calculations, where `Step`s internally call `builder.ensure()`, without actually running cargo or the compiler. - -In some cases, even this may not be enough logging (if so, please add more!). In that case, you can omit `--dry-run`, which will show the normal output inline with the debug logging: - -``` - c Sysroot { compiler: Compiler { stage: 0, host: x86_64-unknown-linux-gnu }, force_recompile: false } -using sysroot /home/jyn/src/rust2/build/x86_64-unknown-linux-gnu/stage0-sysroot -Building stage0 library artifacts (x86_64-unknown-linux-gnu) -running: cd "/home/jyn/src/rust2" && env ... RUSTC_VERBOSE="2" RUSTC_WRAPPER="/home/jyn/src/rust2/build/bootstrap/debug/rustc" "/home/jyn/src/rust2/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "build" "--target" "x86_64-unknown-linux-gnu" "-Zbinary-dep-depinfo" "-Zroot-dir=/home/jyn/src/rust2" "-v" "-v" "--manifest-path" "/home/jyn/src/rust2/library/sysroot/Cargo.toml" "--message-format" "json-render-diagnostics" - 0.293440230s INFO prepare_target{force=false package_id=sysroot v0.0.0 (/home/jyn/src/rust2/library/sysroot) target="sysroot"}: cargo::core::compiler::fingerprint: fingerprint error for sysroot v0.0.0 (/home/jyn/src/rust2/library/sysroot)/Build/TargetInner { name_inferred: true, ..: lib_target("sysroot", ["lib"], "/home/jyn/src/rust2/library/sysroot/src/lib.rs", Edition2021) } ... ``` -In most cases this should not be necessary. +## `tracing` in bootstrap -TODO: we should convert all this to structured logging so it's easier to control precisely. +Bootstrap has a conditional `tracing` feature, which provides the following features: +- It enables structured logging using [`tracing`][tracing] events and spans. +- It generates a [Chrome trace file] that can be used to visualize the hierarchy and durations of executed steps and commands. + - You can open the generated `chrome-trace.json` file using Chrome, on the `chrome://tracing` tab, or e.g. using [Perfetto]. +- It generates [GraphViz] graphs that visualize the dependencies between executed steps. + - You can open the generated `step-graph-*.dot` file using e.g. [xdot] to visualize the step graph, or use e.g. `dot -Tsvg` to convert the GraphViz file to an SVG file. +- It generates a command execution summary, which shows which commands were executed, how many of their executions were cached, and what commands were the slowest to run. + - The generated `command-stats.txt` file is in a simple human-readable format. -## `tracing` in bootstrap +The structured logs will be written to standard error output (`stderr`), while the other outputs will be stored in files in the `<build-dir>/bootstrap-trace/<pid>` directory. For convenience, bootstrap will also create a symlink to the latest generated trace output directory at `<build-dir>/bootstrap-trace/latest`. -Bootstrap has conditional [`tracing`][tracing] setup to provide structured logging. +> Note that if you execute bootstrap with `--dry-run`, the tracing output directory might change. Bootstrap will always print a path where the tracing output files were stored at the end of its execution. [tracing]: https://docs.rs/tracing/0.1.41/tracing/index.html +[Chrome trace file]: https://www.chromium.org/developers/how-tos/trace-event-profiling-tool/ +[Perfetto]: https://ui.perfetto.dev/ +[GraphViz]: https://graphviz.org/doc/info/lang.html +[xdot]: https://github.com/jrfonseca/xdot.py ### Enabling `tracing` output -Bootstrap will conditionally build `tracing` support and enable `tracing` output if the `BOOTSTRAP_TRACING` env var is set. - -#### Basic usage - -Example basic usage[^just-trace]: +To enable the conditional `tracing` feature, run bootstrap with the `BOOTSTRAP_TRACING` environment variable. -[^just-trace]: It is not recommended to use *just* `BOOTSTRAP_TRACING=TRACE` because that will dump *everything* at `TRACE` level, including logs intentionally gated behind custom targets as they are too verbose even for `TRACE` level by default. +[tracing_subscriber filter]: https://docs.rs/tracing-subscriber/latest/tracing_subscriber/filter/struct.EnvFilter.html ```bash -$ BOOTSTRAP_TRACING=bootstrap=TRACE ./x build library --stage 1 +$ BOOTSTRAP_TRACING=trace ./x build library --stage 1 ``` Example output[^unstable]: ``` -$ BOOTSTRAP_TRACING=bootstrap=TRACE ./x check src/bootstrap/ +$ BOOTSTRAP_TRACING=trace ./x build library --stage 1 --dry-run Building bootstrap - Compiling bootstrap v0.0.0 (/home/joe/repos/rust/src/bootstrap) - Finished `dev` profile [unoptimized] target(s) in 2.74s - DEBUG bootstrap parsing flags - bootstrap::core::config::flags::Flags::parse args=["check", "src/bootstrap/"] - DEBUG bootstrap parsing config based on flags - DEBUG bootstrap creating new build based on config - bootstrap::Build::build - TRACE bootstrap setting up job management - TRACE bootstrap downloading rustfmt early - bootstrap::handling hardcoded subcommands (Format, Suggest, Perf) - DEBUG bootstrap not a hardcoded subcommand; returning to normal handling, cmd=Check { all_targets: false } - DEBUG bootstrap handling subcommand normally - bootstrap::executing real run - bootstrap::(1) executing dry-run sanity-check - bootstrap::(2) executing actual run -Checking stage0 library artifacts (x86_64-unknown-linux-gnu) - Finished `release` profile [optimized + debuginfo] target(s) in 0.04s -Checking stage0 compiler artifacts {rustc-main, rustc_abi, rustc_arena, rustc_ast, rustc_ast_ir, rustc_ast_lowering, rustc_ast_passes, rustc_ast_pretty, rustc_attr_data_structures, rustc_attr_parsing, rustc_baked_icu_data, rustc_borrowck, rustc_builtin_macros, rustc_codegen_llvm, rustc_codegen_ssa, rustc_const_eval, rustc_data_structures, rustc_driver, rustc_driver_impl, rustc_error_codes, rustc_error_messages, rustc_errors, rustc_expand, rustc_feature, rustc_fluent_macro, rustc_fs_util, rustc_graphviz, rustc_hir, rustc_hir_analysis, rustc_hir_pretty, rustc_hir_typeck, rustc_incremental, rustc_index, rustc_index_macros, rustc_infer, rustc_interface, rustc_lexer, rustc_lint, rustc_lint_defs, rustc_llvm, rustc_log, rustc_macros, rustc_metadata, rustc_middle, rustc_mir_build, rustc_mir_dataflow, rustc_mir_transform, rustc_monomorphize, rustc_next_trait_solver, rustc_parse, rustc_parse_format, rustc_passes, rustc_pattern_analysis, rustc_privacy, rustc_query_impl, rustc_query_system, rustc_resolve, rustc_sanitizers, rustc_serialize, rustc_session, rustc_smir, rustc_span, rustc_symbol_mangling, rustc_target, rustc_trait_selection, rustc_traits, rustc_transmute, rustc_ty_utils, rustc_type_ir, rustc_type_ir_macros, stable_mir} (x86_64-unknown-linux-gnu) - Finished `release` profile [optimized + debuginfo] target(s) in 0.23s -Checking stage0 bootstrap artifacts (x86_64-unknown-linux-gnu) - Checking bootstrap v0.0.0 (/home/joe/repos/rust/src/bootstrap) - Finished `release` profile [optimized + debuginfo] target(s) in 0.64s - DEBUG bootstrap checking for postponed test failures from `test --no-fail-fast` -Build completed successfully in 0:00:08 + Finished `dev` profile [unoptimized] target(s) in 0.05s +15:56:52.477 INFO > tool::LibcxxVersionTool {target: x86_64-unknown-linux-gnu} (builder/mod.rs:1715) +15:56:52.575 INFO > compile::Assemble {target_compiler: Compiler { stage: 0, host: x86_64-unknown-linux-gnu, forced_compiler: false }} (builder/mod.rs:1715) +15:56:52.575 INFO > tool::Compiletest {compiler: Compiler { stage: 0, host: x86_64-unknown-linux-gnu, forced_compiler: false }, target: x86_64-unknown-linux-gnu} (builder/mod.rs:1715) +15:56:52.576 INFO > tool::ToolBuild {build_compiler: Compiler { stage: 0, host: x86_64-unknown-linux-gnu, forced_compiler: false }, target: x86_64-unknown-linux-gnu, tool: "compiletest", path: "src/tools/compiletest", mode: ToolBootstrap, source_type: InTree, extra_features: [], allow_features: "internal_output_capture", cargo_args: [], artifact_kind: Binary} (builder/mod.rs:1715) +15:56:52.576 INFO > builder::Libdir {compiler: Compiler { stage: 0, host: x86_64-unknown-linux-gnu, forced_compiler: false }, target: x86_64-unknown-linux-gnu} (builder/mod.rs:1715) +15:56:52.576 INFO > compile::Sysroot {compiler: Compiler { stage: 0, host: x86_64-unknown-linux-gnu, forced_compiler: false }, force_recompile: false} (builder/mod.rs:1715) +15:56:52.578 INFO > compile::Assemble {target_compiler: Compiler { stage: 0, host: x86_64-unknown-linux-gnu, forced_compiler: false }} (builder/mod.rs:1715) +15:56:52.578 INFO > tool::Compiletest {compiler: Compiler { stage: 0, host: x86_64-unknown-linux-gnu, forced_compiler: false }, target: x86_64-unknown-linux-gnu} (builder/mod.rs:1715) +15:56:52.578 INFO > tool::ToolBuild {build_compiler: Compiler { stage: 0, host: x86_64-unknown-linux-gnu, forced_compiler: false }, target: x86_64-unknown-linux-gnu, tool: "compiletest", path: "src/tools/compiletest", mode: ToolBootstrap, source_type: InTree, extra_features: [], allow_features: "internal_output_capture", cargo_args: [], artifact_kind: Binary} (builder/mod.rs:1715) +15:56:52.578 INFO > builder::Libdir {compiler: Compiler { stage: 0, host: x86_64-unknown-linux-gnu, forced_compiler: false }, target: x86_64-unknown-linux-gnu} (builder/mod.rs:1715) +15:56:52.578 INFO > compile::Sysroot {compiler: Compiler { stage: 0, host: x86_64-unknown-linux-gnu, forced_compiler: false }, force_recompile: false} (builder/mod.rs:1715) + Finished `release` profile [optimized] target(s) in 0.11s +Tracing/profiling output has been written to <src-root>/build/bootstrap-trace/latest +Build completed successfully in 0:00:00 ``` +[^unstable]: This output is always subject to further changes. + #### Controlling tracing output -The env var `BOOTSTRAP_TRACING` accepts a [`tracing` env-filter][tracing-env-filter]. +The environment variable `BOOTSTRAP_TRACING` accepts a [`tracing_subscriber` filter][tracing-env-filter]. If you set `BOOTSTRAP_TRACING=trace`, you will enable all logs, but that can be overwhelming. You can thus use the filter to reduce the amount of data logged. There are two orthogonal ways to control which kind of tracing logs you want: -1. You can specify the log **level**, e.g. `DEBUG` or `TRACE`. -2. You can also control the log **target**, e.g. `bootstrap` or `bootstrap::core::config` vs custom targets like `CONFIG_HANDLING`. - - Custom targets are used to limit what is output when `BOOTSTRAP_TRACING=bootstrap=TRACE` is used, as they can be too verbose even for `TRACE` level by default. Currently used custom targets: - - `CONFIG_HANDLING` - -The `TRACE` filter will enable *all* `trace` level or less verbose level tracing output. +1. You can specify the log **level**, e.g. `debug` or `trace`. + - If you select a level, all events/spans with an equal or higher priority level will be shown. +2. You can also control the log **target**, e.g. `bootstrap` or `bootstrap::core::config` or a custom target like `CONFIG_HANDLING` or `STEP`. + - Custom targets are used to limit what kinds of spans you are interested in, as the `BOOTSTRAP_TRACING=trace` output can be quite verbose. Currently, you can use the following custom targets: + - `CONFIG_HANDLING`: show spans related to config handling. + - `STEP`: show all executed steps. Executed commands have `info` event level. + - `COMMAND`: show all executed commands. Executed commands have `trace` event level. + - `IO`: show performed I/O operations. Executed commands have `trace` event level. + - Note that many I/O are currently not being traced. You can of course combine them (custom target logs are typically gated behind `TRACE` log level additionally): ```bash -$ BOOTSTRAP_TRACING=CONFIG_HANDLING=TRACE ./x build library --stage 1 +$ BOOTSTRAP_TRACING=CONFIG_HANDLING=trace,STEP=info,COMMAND=trace ./x build library --stage 1 ``` -[^unstable]: This output is always subject to further changes. - [tracing-env-filter]: https://docs.rs/tracing-subscriber/0.3.19/tracing_subscriber/filter/struct.EnvFilter.html +Note that the level that you specify using `BOOTSTRAP_TRACING` also has an effect on the spans that will be recorded in the Chrome trace file. + ##### FIXME(#96176): specific tracing for `compiler()` vs `compiler_for()` The additional targets `COMPILER` and `COMPILER_FOR` are used to help trace what @@ -123,12 +105,6 @@ if [#96176][cleanup-compiler-for] is resolved. [cleanup-compiler-for]: https://github.com/rust-lang/rust/issues/96176 -### Rendering step graph - -When you run bootstrap with the `BOOTSTRAP_TRACING` environment variable configured, bootstrap will automatically output a DOT file that shows all executed steps and their dependencies. The files will have a prefix `bootstrap-steps`. You can use e.g. `xdot` to visualize the file or e.g. `dot -Tsvg` to convert the DOT file to a SVG file. - -A separate DOT file will be outputted for dry-run and non-dry-run execution. - ### Using `tracing` in bootstrap Both `tracing::*` macros and the `tracing::instrument` proc-macro attribute need to be gated behind `tracing` feature. Examples: @@ -149,15 +125,6 @@ impl Step for Foo { todo!() } - #[cfg_attr( - feature = "tracing", - instrument( - level = "trace", - name = "Foo::run", - skip_all, - fields(compiler = ?builder.compiler), - ), - )] fn run(self, builder: &Builder<'_>) -> Self::Output { trace!(?run, "entered Foo::run"); @@ -172,21 +139,6 @@ For `#[instrument]`, it's recommended to: - Explicitly pick an instrumentation name via `name = ".."` to distinguish between e.g. `run` of different steps. - Take care to not cause diverging behavior via tracing, e.g. building extra things only when tracing infra is enabled. -### Profiling bootstrap - -You can set the `BOOTSTRAP_PROFILE=1` environment variable to enable command execution profiling during bootstrap. This generates: - -* A Chrome trace file (for visualization in `chrome://tracing` or [Perfetto](https://ui.perfetto.dev)) if tracing is enabled via `BOOTSTRAP_TRACING=COMMAND=trace` -* A plain-text summary file, `bootstrap-profile-{pid}.txt`, listing all commands sorted by execution time (slowest first), along with cache hits and working directories - -Note: the `.txt` report is always generated when `BOOTSTRAP_PROFILE=1` is set — tracing is not required. - -Example usage: - -```bash -$ BOOTSTRAP_PROFILE=1 BOOTSTRAP_TRACING=COMMAND=trace ./x build library -``` - ### rust-analyzer integration? Unfortunately, because bootstrap is a `rust-analyzer.linkedProjects`, you can't ask r-a to check/build bootstrap itself with `tracing` feature enabled to get relevant completions, due to lack of support as described in <https://github.com/rust-lang/rust-analyzer/issues/8521>. diff --git a/src/doc/rustc-dev-guide/src/building/bootstrapping/what-bootstrapping-does.md b/src/doc/rustc-dev-guide/src/building/bootstrapping/what-bootstrapping-does.md index da425d8d39b..bfd75ebda40 100644 --- a/src/doc/rustc-dev-guide/src/building/bootstrapping/what-bootstrapping-does.md +++ b/src/doc/rustc-dev-guide/src/building/bootstrapping/what-bootstrapping-does.md @@ -23,7 +23,7 @@ Note that this documentation mostly covers user-facing information. See ### Overview -- Stage 0: the pre-compiled compiler +- Stage 0: the pre-compiled compiler and standard library - Stage 1: from current code, by an earlier compiler - Stage 2: the truly current compiler - Stage 3: the same-result test @@ -192,7 +192,7 @@ include, but are not limited to: artifacts'). If you're working on the standard library, this is normally the test command you want. - `./x build --stage 0` means to build with the stage0 `rustc`. -- `./x doc --stage 0` means to document using the stage0 `rustdoc`. +- `./x doc --stage 1` means to document using the stage0 `rustdoc`. #### Examples of what *not* to do @@ -211,7 +211,7 @@ include, but are not limited to: In short, _stage 0 uses the `stage0` compiler to create `stage0` artifacts which will later be uplifted to be the stage1 compiler_. -In each stage, two major steps are performed: +In each stage besides 0, two major steps are performed: 1. `std` is compiled by the stage N compiler. 2. That `std` is linked to programs built by the stage N compiler, including the diff --git a/src/doc/rustc-dev-guide/src/building/bootstrapping/writing-tools-in-bootstrap.md b/src/doc/rustc-dev-guide/src/building/bootstrapping/writing-tools-in-bootstrap.md index 41d0cf8d9fb..8ac2e6bfe28 100644 --- a/src/doc/rustc-dev-guide/src/building/bootstrapping/writing-tools-in-bootstrap.md +++ b/src/doc/rustc-dev-guide/src/building/bootstrapping/writing-tools-in-bootstrap.md @@ -3,21 +3,32 @@ There are three types of tools you can write in bootstrap: - **`Mode::ToolBootstrap`** + Use this for tools that don’t need anything from the in-tree compiler and can run with the stage0 `rustc`. - The output is placed in the "bootstrap-tools" directory. This mode is for general-purpose tools built - entirely with the stage0 compiler, including target libraries and only works for stage 0. + The output is placed in the "bootstrap-tools" directory. + This mode is for general-purpose tools built entirely with the stage0 compiler, + including target libraries, and it only works for stage 0. - **`Mode::ToolStd`** - Use this for tools that rely on the locally built std. The output goes into the "stageN-tools" directory. + + Use this for tools that rely on the locally built std. + The output goes into the "stageN-tools" directory. This mode is rarely used, mainly for `compiletest` which requires `libtest`. -- **`Mode::ToolRustc`** - Use this for tools that depend on both the locally built `rustc` and the target `std`. This is more complex than - the other modes because the tool must be built with the same compiler used for `rustc` and placed in the "stageN-tools" - directory. When you choose `Mode::ToolRustc`, `ToolBuild` implementation takes care of this automatically. - If you need to use the builder’s compiler for something specific, you can get it from `ToolBuildResult`, which is - returned by the tool's [`Step`]. +- **`Mode::ToolRustcPrivate`** + + Use this for tools that use the `rustc_private` mechanism, + and thus depend on the locally built `rustc` and its rlib artifacts. + This is more complex than the other modes, + because the tool must be built with the same compiler used for `rustc`, + and placed in the "stageN-tools" directory. + When you choose `Mode::ToolRustcPrivate`, + `ToolBuild` implementation takes care of this automatically. + If you need to use the builder’s compiler for something specific, + you can get it from `ToolBuildResult`, which is returned by the tool's [`Step`]. -Regardless of the tool type you must return `ToolBuildResult` from the tool’s [`Step`] implementation and use `ToolBuild` inside it. +Regardless of the tool type, +you must return `ToolBuildResult` from the tool’s [`Step`] implementation, +and use `ToolBuild` inside it. [`Step`]: https://doc.rust-lang.org/nightly/nightly-rustc/bootstrap/core/builder/trait.Step.html diff --git a/src/doc/rustc-dev-guide/src/building/how-to-build-and-run.md b/src/doc/rustc-dev-guide/src/building/how-to-build-and-run.md index b07d3533f59..36610f28854 100644 --- a/src/doc/rustc-dev-guide/src/building/how-to-build-and-run.md +++ b/src/doc/rustc-dev-guide/src/building/how-to-build-and-run.md @@ -149,7 +149,7 @@ On Windows, the Powershell commands may give you an error that looks like this: ``` PS C:\Users\vboxuser\rust> ./x ./x : File C:\Users\vboxuser\rust\x.ps1 cannot be loaded because running scripts is disabled on this system. For more -information, see about_Execution_Policies at https:/go.microsoft.com/fwlink/?LinkID=135170. +information, see about_Execution_Policies at https://go.microsoft.com/fwlink/?LinkID=135170. At line:1 char:1 + ./x + ~~~ diff --git a/src/doc/rustc-dev-guide/src/building/optimized-build.md b/src/doc/rustc-dev-guide/src/building/optimized-build.md index 863ed9749fb..46def66d178 100644 --- a/src/doc/rustc-dev-guide/src/building/optimized-build.md +++ b/src/doc/rustc-dev-guide/src/building/optimized-build.md @@ -118,7 +118,7 @@ Here is an example of how can `opt-dist` be used locally (outside of CI): ``` 3. Run the tool with the `local` mode and provide necessary parameters: ```bash - ./build/host/stage0-tools-bin/opt-dist local \ + ./build/host/stage1-tools-bin/opt-dist local \ --target-triple <target> \ # select target, e.g. "x86_64-unknown-linux-gnu" --checkout-dir <path> \ # path to rust checkout, e.g. "." --llvm-dir <path> \ # path to built LLVM toolchain, e.g. "/foo/bar/llvm/install" diff --git a/src/doc/rustc-dev-guide/src/compiler-debugging.md b/src/doc/rustc-dev-guide/src/compiler-debugging.md index edd2aa6c5f6..f4514470418 100644 --- a/src/doc/rustc-dev-guide/src/compiler-debugging.md +++ b/src/doc/rustc-dev-guide/src/compiler-debugging.md @@ -367,7 +367,7 @@ error: layout_of(&'a u32) = Layout { error: aborting due to previous error ``` -[`Layout`]: https://doc.rust-lang.org/nightly/nightly-rustc/stable_mir/abi/struct.Layout.html +[`Layout`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_public/abi/struct.Layout.html ## Configuring CodeLLDB for debugging `rustc` diff --git a/src/doc/rustc-dev-guide/src/compiler-src.md b/src/doc/rustc-dev-guide/src/compiler-src.md index d67bacb1b33..27b40f1fe01 100644 --- a/src/doc/rustc-dev-guide/src/compiler-src.md +++ b/src/doc/rustc-dev-guide/src/compiler-src.md @@ -153,7 +153,8 @@ The bulk of [`rustdoc`] is in [`librustdoc`]. However, the [`rustdoc`] binary itself is [`src/tools/rustdoc`], which does nothing except call [`rustdoc::main`]. There is also `JavaScript` and `CSS` for the docs in [`src/tools/rustdoc-js`] -and [`src/tools/rustdoc-themes`]. +and [`src/tools/rustdoc-themes`]. The type definitions for `--output-format=json` +are in a separate crate in [`src/rustdoc-json-types`]. You can read more about [`rustdoc`] in [this chapter][rustdoc-chapter]. @@ -162,6 +163,7 @@ You can read more about [`rustdoc`] in [this chapter][rustdoc-chapter]. [`src/tools/rustdoc-js`]: https://github.com/rust-lang/rust/tree/master/src/tools/rustdoc-js [`src/tools/rustdoc-themes`]: https://github.com/rust-lang/rust/tree/master/src/tools/rustdoc-themes [`src/tools/rustdoc`]: https://github.com/rust-lang/rust/tree/master/src/tools/rustdoc +[`src/rustdoc-json-types`]: https://github.com/rust-lang/rust/tree/master/src/rustdoc-json-types [rustdoc-chapter]: ./rustdoc.md ## Tests diff --git a/src/doc/rustc-dev-guide/src/compiler-team.md b/src/doc/rustc-dev-guide/src/compiler-team.md index 9922ee7ddf2..896d9e6f6d9 100644 --- a/src/doc/rustc-dev-guide/src/compiler-team.md +++ b/src/doc/rustc-dev-guide/src/compiler-team.md @@ -1,10 +1,14 @@ # About the compiler team +> NOTE: +> There exists much detail about the team [on Forge], making most of the following obsolete. + rustc is maintained by the [Rust compiler team][team]. The people who belong to this team collectively work to track regressions and implement new features. Members of the Rust compiler team are people who have made significant contributions to rustc and its design. +[on Forge]: https://forge.rust-lang.org/compiler [team]: https://www.rust-lang.org/governance/teams/compiler ## Discussion @@ -12,7 +16,7 @@ contributions to rustc and its design. Currently the compiler team chats in Zulip: - Team chat occurs in the [`t-compiler`][zulip-t-compiler] stream on the Zulip instance -- There are also a number of other associated Zulip streams, +- There are also a number of other associated Zulip channels, such as [`t-compiler/help`][zulip-help], where people can ask for help with rustc development, or [`t-compiler/meetings`][zulip-meetings], where the team holds their weekly triage and steering meetings. diff --git a/src/doc/rustc-dev-guide/src/contributing.md b/src/doc/rustc-dev-guide/src/contributing.md index 963bef3af8d..3d196ae2308 100644 --- a/src/doc/rustc-dev-guide/src/contributing.md +++ b/src/doc/rustc-dev-guide/src/contributing.md @@ -26,8 +26,7 @@ conditions that trigger the bug, or part of the error message if there is any. An example could be: **"impossible case reached" on lifetime inference for impl Trait in return position**. -Opening an issue is as easy as following [this -link](https://github.com/rust-lang/rust/issues/new/choose) and filling out the fields +Opening an issue is as easy as following [thi link][create an issue] and filling out the fields in the appropriate provided template. ## Bug fixes or "normal" code changes @@ -75,7 +74,7 @@ Example of things that might require MCPs include major refactorings, changes to important types, or important changes to how the compiler does something, or smaller user-facing changes. -**When in doubt, ask on [zulip]. It would be a shame to put a lot of work +**When in doubt, ask on [Zulip]. It would be a shame to put a lot of work into a PR that ends up not getting merged!** [See this document][mcpinfo] for more info on MCPs. @@ -127,7 +126,7 @@ when contributing to Rust under [the git section](./git.md). > from), and work with the compiler team to see if we can help you **break down a large potentially > unreviewable PR into a series of smaller more individually reviewable PRs**. > -> You can communicate with the compiler team by creating a [#t-compiler thread on zulip][t-compiler] +> You can communicate with the compiler team by creating a [#t-compiler thread on Zulip][t-compiler] > to discuss your proposed changes. > > Communicating with the compiler team beforehand helps in several ways: @@ -154,11 +153,14 @@ The CI in rust-lang/rust applies your patches directly against the current maste not against the commit your branch is based on. This can lead to unexpected failures if your branch is outdated, even when there are no explicit merge conflicts. -Before submitting or updating a PR, make sure to update your branch -as mentioned [here](git.md#keeping-things-up-to-date) if it's significantly -behind the master branch (e.g., more than 100 commits behind). -This fetches the latest master branch and rebases your changes on top of it, -ensuring your PR is tested against the latest code. +Update your branch only when needed: when you have merge conflicts, upstream CI is broken and blocking your green PR, or a maintainer requests it. +Avoid updating an already-green PR under review unless necessary. +During review, make incremental commits to address feedback. +Prefer to squash or rebase only at the end, or when a reviewer requests it. + +When updating, use `git push --force-with-lease` and leave a brief comment explaining what changed. +Some repos prefer merging from `upstream/master` instead of rebasing; follow the project's conventions. +See [keeping things up to date](git.md#keeping-things-up-to-date) for detailed instructions. After rebasing, it's recommended to [run the relevant tests locally](tests/intro.md) to catch any issues before CI runs. @@ -372,6 +374,11 @@ You can also use `rustdoc` directly to check small fixes. For example, `rustdoc src/doc/reference.md` will render reference to `doc/reference.html`. The CSS might be messed up, but you can verify that the HTML is right. +Please notice that we don't accept typography/spellcheck fixes to **internal documentation** +as it's usually not worth the churn or the review time. +Examples of internal documentation is code comments and rustc api docs. +However, feel free to fix those if accompanied by other improvements in the same PR. + ### Contributing to rustc-dev-guide Contributions to the [rustc-dev-guide] are always welcome, and can be made directly at @@ -434,7 +441,8 @@ Just a few things to keep in mind: #### ⚠️ Note: Where to contribute `rustc-dev-guide` changes -For detailed information about where to contribute rustc-dev-guide changes and the benefits of doing so, see [the rustc-dev-guide working group documentation](https://forge.rust-lang.org/wg-rustc-dev-guide/index.html#where-to-contribute-rustc-dev-guide-changes). +For detailed information about where to contribute rustc-dev-guide changes and the benefits of doing so, +see [the rustc-dev-guide working group documentation]. ## Issue triage @@ -451,6 +459,7 @@ Please see <https://forge.rust-lang.org/release/issue-triaging.html>. [regression-]: https://github.com/rust-lang/rust/labels?q=regression [relnotes]: https://github.com/rust-lang/rust/labels/relnotes [S-tracking-]: https://github.com/rust-lang/rust/labels?q=s-tracking +[the rustc-dev-guide working group documentation]: https://forge.rust-lang.org/wg-rustc-dev-guide/index.html#where-to-contribute-rustc-dev-guide-changes ### Rfcbot labels @@ -498,3 +507,4 @@ This section has moved to the ["About this guide"] chapter. [RFC 1574]: https://github.com/rust-lang/rfcs/blob/master/text/1574-more-api-documentation-conventions.md#appendix-a-full-conventions-text [rustc-dev-guide]: https://rustc-dev-guide.rust-lang.org/ [rdgrepo]: https://github.com/rust-lang/rustc-dev-guide +[create an issue]: https://github.com/rust-lang/rust/issues/new/choose diff --git a/src/doc/rustc-dev-guide/src/diagnostics/error-codes.md b/src/doc/rustc-dev-guide/src/diagnostics/error-codes.md index 1b6b87e4c8d..1693432b90d 100644 --- a/src/doc/rustc-dev-guide/src/diagnostics/error-codes.md +++ b/src/doc/rustc-dev-guide/src/diagnostics/error-codes.md @@ -20,7 +20,7 @@ explanations should help users understand why their code cannot be accepted by the compiler. Rust prides itself on helpful error messages and long-form explanations are no exception. However, before error explanations are overhauled[^new-explanations] it is a bit open as to how exactly they should be -written, as always: ask your reviewer or ask around on the Rust Discord or Zulip. +written, as always: ask your reviewer or ask around on the Rust Zulip. [^new-explanations]: See the draft RFC [here][new-explanations-rfc]. diff --git a/src/doc/rustc-dev-guide/src/fuzzing.md b/src/doc/rustc-dev-guide/src/fuzzing.md index 30005378617..cc98b49a97c 100644 --- a/src/doc/rustc-dev-guide/src/fuzzing.md +++ b/src/doc/rustc-dev-guide/src/fuzzing.md @@ -90,14 +90,15 @@ Here are a few things you can do to help the Rust project after filing an ICE. triggering the ICE, such as syntax errors or borrow-checking errors - Minimize the test case (see below). If successful, you can label the issue with `S-has-mcve`. Otherwise, you can apply `E-needs-mcve`. -- Add the minimal test case to the rust-lang/rust repo as a [crashes test]. +- Add the minimal test case to the rust-lang/rust repo as a [crash test]. While you're at it, consider including other "untracked" crashes in your PR. - Please don't forget to mark your issue with `S-bug-has-test` afterwards. + Please don't forget to mark all relevant issues with `S-bug-has-test` once + your PR is merged. See also [applying and removing labels][labeling]. [bisect]: https://rust-lang.github.io/cargo-bisect-rustc/ -[crashes test]: tests/compiletest.html#crashes-tests +[crash test]: tests/compiletest.html#crash-tests [labeling]: https://forge.rust-lang.org/release/issue-triaging.html#applying-and-removing-labels ## Minimization diff --git a/src/doc/rustc-dev-guide/src/getting-started.md b/src/doc/rustc-dev-guide/src/getting-started.md index 04d2e37732f..87e26d37968 100644 --- a/src/doc/rustc-dev-guide/src/getting-started.md +++ b/src/doc/rustc-dev-guide/src/getting-started.md @@ -11,7 +11,6 @@ quick guide for the most useful things. For more information, [see this chapter on how to build and run the compiler](./building/how-to-build-and-run.md). [internals]: https://internals.rust-lang.org -[rust-discord]: http://discord.gg/rust-lang [rust-zulip]: https://rust-lang.zulipchat.com [coc]: https://www.rust-lang.org/policies/code-of-conduct [walkthrough]: ./walkthrough.md @@ -20,8 +19,7 @@ chapter on how to build and run the compiler](./building/how-to-build-and-run.md ## Asking Questions If you have questions, please make a post on the [Rust Zulip server][rust-zulip] or -[internals.rust-lang.org][internals]. If you are contributing to Rustup, be aware they are not on -Zulip - you can ask questions in `#wg-rustup` [on Discord][rust-discord]. +[internals.rust-lang.org][internals]. See the [list of teams and working groups][governance] and [the Community page][community] on the official website for more resources. @@ -30,19 +28,23 @@ official website for more resources. As a reminder, all contributors are expected to follow our [Code of Conduct][coc]. -The compiler team (or `t-compiler`) usually hangs out in Zulip [in this -"stream"][z]; it will be easiest to get questions answered there. +The compiler team (or `t-compiler`) usually hangs out in Zulip in +[the #t-compiler channel][z-t-compiler]; +questions about how the compiler works can go in [#t-compiler/help][z-help]. -[z]: https://rust-lang.zulipchat.com/#narrow/stream/131828-t-compiler +[z-t-compiler]: https://rust-lang.zulipchat.com/#narrow/channel/131828-t-compiler +[z-help]: https://rust-lang.zulipchat.com/#narrow/channel/182449-t-compiler.2Fhelp **Please ask questions!** A lot of people report feeling that they are "wasting -expert time", but nobody on `t-compiler` feels this way. Contributors are +expert's time", but nobody on `t-compiler` feels this way. Contributors are important to us. Also, if you feel comfortable, prefer public topics, as this means others can see the questions and answers, and perhaps even integrate them back into this guide :) +**Tip**: If you're not a native English speaker and feel unsure about writing, try using a translator to help. But avoid using LLM tools that generate long, complex words. In daily teamwork, **simple and clear words** are best for easy understanding. Even small typos or grammar mistakes can make you seem more human, and people connect better with humans. + ### Experts Not all `t-compiler` members are experts on all parts of `rustc`; it's a @@ -162,15 +164,12 @@ incredibly helpful: - [Triaging issues][triage]: categorizing, replicating, and minimizing issues is very helpful to the Rust maintainers. - [Working groups][wg]: there are a bunch of working groups on a wide variety of rust-related things. -- Answer questions in the _Get Help!_ channels on the [Rust Discord - server][rust-discord], on [users.rust-lang.org][users], or on - [StackOverflow][so]. +- Answer questions on [users.rust-lang.org][users], or on [Stack Overflow][so]. - Participate in the [RFC process](https://github.com/rust-lang/rfcs). - Find a [requested community library][community-library], build it, and publish it to [Crates.io](http://crates.io). Easier said than done, but very, very valuable! -[rust-discord]: https://discord.gg/rust-lang [users]: https://users.rust-lang.org/ [so]: http://stackoverflow.com/questions/tagged/rust [community-library]: https://github.com/rust-lang/rfcs/labels/A-community-library diff --git a/src/doc/rustc-dev-guide/src/git.md b/src/doc/rustc-dev-guide/src/git.md index 447c6fd4546..8f0511a4548 100644 --- a/src/doc/rustc-dev-guide/src/git.md +++ b/src/doc/rustc-dev-guide/src/git.md @@ -338,13 +338,13 @@ your fork with `git push --force-with-lease`. ### Keeping things up to date -The above section on [Rebasing](#rebasing) is a specific +The [above section](#rebasing) is a specific guide on rebasing work and dealing with merge conflicts. Here is some general advice about how to keep your local repo up-to-date with upstream changes: Using `git pull upstream master` while on your local master branch regularly -will keep it up-to-date. You will also want to rebase your feature branches +will keep it up-to-date. You will also want to keep your feature branches up-to-date as well. After pulling, you can checkout the feature branches and rebase them: diff --git a/src/doc/rustc-dev-guide/src/hir.md b/src/doc/rustc-dev-guide/src/hir.md index 38ba33112f2..0b341a40f1d 100644 --- a/src/doc/rustc-dev-guide/src/hir.md +++ b/src/doc/rustc-dev-guide/src/hir.md @@ -102,7 +102,7 @@ These identifiers can be converted into one another through the `TyCtxt`. [`DefId`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_hir/def_id/struct.DefId.html [`LocalDefId`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_hir/def_id/struct.LocalDefId.html -[`HirId`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_hir/hir_id/struct.HirId.html +[`HirId`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_hir/struct.HirId.html [`BodyId`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_hir/hir/struct.BodyId.html [Node]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_hir/hir/enum.Node.html [`CrateNum`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_hir/def_id/struct.CrateNum.html diff --git a/src/doc/rustc-dev-guide/src/img/coverage-branch-counting-01.png b/src/doc/rustc-dev-guide/src/img/coverage-branch-counting-01.png index c445f3552a6..7c6c845f2cb 100644 --- a/src/doc/rustc-dev-guide/src/img/coverage-branch-counting-01.png +++ b/src/doc/rustc-dev-guide/src/img/coverage-branch-counting-01.png Binary files differdiff --git a/src/doc/rustc-dev-guide/src/img/dataflow-graphviz-example.png b/src/doc/rustc-dev-guide/src/img/dataflow-graphviz-example.png index 718411a8c42..7baa37e4323 100644 --- a/src/doc/rustc-dev-guide/src/img/dataflow-graphviz-example.png +++ b/src/doc/rustc-dev-guide/src/img/dataflow-graphviz-example.png Binary files differdiff --git a/src/doc/rustc-dev-guide/src/img/github-cli.png b/src/doc/rustc-dev-guide/src/img/github-cli.png index c3b0e7707eb..88ba95f90a8 100644 --- a/src/doc/rustc-dev-guide/src/img/github-cli.png +++ b/src/doc/rustc-dev-guide/src/img/github-cli.png Binary files differdiff --git a/src/doc/rustc-dev-guide/src/img/github-whitespace-changes.png b/src/doc/rustc-dev-guide/src/img/github-whitespace-changes.png index 9a19a10aace..e235a30b33e 100644 --- a/src/doc/rustc-dev-guide/src/img/github-whitespace-changes.png +++ b/src/doc/rustc-dev-guide/src/img/github-whitespace-changes.png Binary files differdiff --git a/src/doc/rustc-dev-guide/src/img/llvm-cov-show-01.png b/src/doc/rustc-dev-guide/src/img/llvm-cov-show-01.png index 35f04594347..ce4dec128b6 100644 --- a/src/doc/rustc-dev-guide/src/img/llvm-cov-show-01.png +++ b/src/doc/rustc-dev-guide/src/img/llvm-cov-show-01.png Binary files differdiff --git a/src/doc/rustc-dev-guide/src/img/other-peoples-commits.png b/src/doc/rustc-dev-guide/src/img/other-peoples-commits.png index e4fc2c7972e..0c949d8844d 100644 --- a/src/doc/rustc-dev-guide/src/img/other-peoples-commits.png +++ b/src/doc/rustc-dev-guide/src/img/other-peoples-commits.png Binary files differdiff --git a/src/doc/rustc-dev-guide/src/img/rustbot-submodules.png b/src/doc/rustc-dev-guide/src/img/rustbot-submodules.png index c2e6937cbeb..c099fdfcb46 100644 --- a/src/doc/rustc-dev-guide/src/img/rustbot-submodules.png +++ b/src/doc/rustc-dev-guide/src/img/rustbot-submodules.png Binary files differdiff --git a/src/doc/rustc-dev-guide/src/img/submodule-conflicts.png b/src/doc/rustc-dev-guide/src/img/submodule-conflicts.png index e90a6bbe8fd..5d4caf0b142 100644 --- a/src/doc/rustc-dev-guide/src/img/submodule-conflicts.png +++ b/src/doc/rustc-dev-guide/src/img/submodule-conflicts.png Binary files differdiff --git a/src/doc/rustc-dev-guide/src/img/wpa-initial-memory.png b/src/doc/rustc-dev-guide/src/img/wpa-initial-memory.png index b6020667ef0..177d92c794c 100644 --- a/src/doc/rustc-dev-guide/src/img/wpa-initial-memory.png +++ b/src/doc/rustc-dev-guide/src/img/wpa-initial-memory.png Binary files differdiff --git a/src/doc/rustc-dev-guide/src/img/wpa-stack.png b/src/doc/rustc-dev-guide/src/img/wpa-stack.png index 29eb5a54b5d..a4a71358ac1 100644 --- a/src/doc/rustc-dev-guide/src/img/wpa-stack.png +++ b/src/doc/rustc-dev-guide/src/img/wpa-stack.png Binary files differdiff --git a/src/doc/rustc-dev-guide/src/macro-expansion.md b/src/doc/rustc-dev-guide/src/macro-expansion.md index 54d6d2b4e81..96f12b76416 100644 --- a/src/doc/rustc-dev-guide/src/macro-expansion.md +++ b/src/doc/rustc-dev-guide/src/macro-expansion.md @@ -517,8 +517,9 @@ We use these items in macro parser: are about to ask the MBE parser to parse. We will consume the raw stream of tokens and output a binding of metavariables to corresponding token trees. The parsing session can be used to report parser errors. -- a `matcher` variable is a sequence of [`MatcherLoc`]s that we want to match - the token stream against. They're converted from token trees before matching. +- a `matcher` variable is a sequence of [`MatcherLoc`]s that we want to match the token stream + against. They're converted from the original token trees in the macro's definition before + matching. [`MatcherLoc`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_expand/mbe/macro_parser/enum.MatcherLoc.html @@ -544,41 +545,26 @@ The full interface is defined [here][code_parse_int]. The macro parser does pretty much exactly the same as a normal regex parser with one exception: in order to parse different types of metavariables, such as `ident`, `block`, `expr`, etc., the macro parser must call back to the normal -Rust parser. Both the definition and invocation of macros are parsed using -the parser in a process which is non-intuitively self-referential. - -The code to parse macro _definitions_ is in -[`compiler/rustc_expand/src/mbe/macro_rules.rs`][code_mr]. It defines the -pattern for matching a macro definition as `$( $lhs:tt => $rhs:tt );+`. In -other words, a `macro_rules` definition should have in its body at least one -occurrence of a token tree followed by `=>` followed by another token tree. -When the compiler comes to a `macro_rules` definition, it uses this pattern to -match the two token trees per the rules of the definition of the macro, _thereby -utilizing the macro parser itself_. In our example definition, the -metavariable `$lhs` would match the patterns of both arms: `(print -$mvar:ident)` and `(print twice $mvar:ident)`. And `$rhs` would match the -bodies of both arms: `{ println!("{}", $mvar); }` and `{ println!("{}", $mvar); -println!("{}", $mvar); }`. The parser keeps this knowledge around for when it -needs to expand a macro invocation. - -When the compiler comes to a macro invocation, it parses that invocation using -a NFA-based macro parser described above. However, the matcher variable -used is the first token tree (`$lhs`) extracted from the arms of the macro -_definition_. Using our example, we would try to match the token stream `print -foo` from the invocation against the matchers `print $mvar:ident` and `print -twice $mvar:ident` that we previously extracted from the definition. The -algorithm is exactly the same, but when the macro parser comes to a place in the -current matcher where it needs to match a _non-terminal_ (e.g. `$mvar:ident`), -it calls back to the normal Rust parser to get the contents of that -non-terminal. In this case, the Rust parser would look for an `ident` token, -which it finds (`foo`) and returns to the macro parser. Then, the macro parser -proceeds in parsing as normal. Also, note that exactly one of the matchers from -the various arms should match the invocation; if there is more than one match, -the parse is ambiguous, while if there are no matches at all, there is a syntax +Rust parser. + +The code to parse macro definitions is in [`compiler/rustc_expand/src/mbe/macro_rules.rs`][code_mr]. +For more information about the macro parser's implementation, see the comments in +[`compiler/rustc_expand/src/mbe/macro_parser.rs`][code_mp]. + +Using our example, we would try to match the token stream `print foo` from the invocation against +the matchers `print $mvar:ident` and `print twice $mvar:ident` that we previously extracted from the +rules in the macro definition. When the macro parser comes to a place in the current matcher where +it needs to match a _non-terminal_ (e.g. `$mvar:ident`), it calls back to the normal Rust parser to +get the contents of that non-terminal. In this case, the Rust parser would look for an `ident` +token, which it finds (`foo`) and returns to the macro parser. Then, the macro parser continues +parsing. + +Note that exactly one of the matchers from the various rules should match the invocation; if there is +more than one match, the parse is ambiguous, while if there are no matches at all, there is a syntax error. -For more information about the macro parser's implementation, see the comments -in [`compiler/rustc_expand/src/mbe/macro_parser.rs`][code_mp]. +Assuming exactly one rule matches, macro expansion will then *transcribe* the right-hand side of the +rule, substituting the values of any matches it captured when matching against the left-hand side. ## Procedural Macros diff --git a/src/doc/rustc-dev-guide/src/notification-groups/arm.md b/src/doc/rustc-dev-guide/src/notification-groups/arm.md index 3abc32c6888..5b79030d20d 100644 --- a/src/doc/rustc-dev-guide/src/notification-groups/arm.md +++ b/src/doc/rustc-dev-guide/src/notification-groups/arm.md @@ -9,7 +9,7 @@ This list will be used to ask for help both in diagnosing and testing ARM-related issues as well as suggestions on how to resolve interesting questions regarding our ARM support. -The group also has an associated Zulip stream ([`#t-compiler/arm`]) +The group also has an associated Zulip channel ([`#t-compiler/arm`]) where people can go to pose questions and discuss ARM-specific topics. diff --git a/src/doc/rustc-dev-guide/src/notification-groups/emscripten.md b/src/doc/rustc-dev-guide/src/notification-groups/emscripten.md index 100dbdf9f2b..9e4086c884e 100644 --- a/src/doc/rustc-dev-guide/src/notification-groups/emscripten.md +++ b/src/doc/rustc-dev-guide/src/notification-groups/emscripten.md @@ -9,7 +9,7 @@ This list will be used to ask for help both in diagnosing and testing Emscripten-related issues as well as suggestions on how to resolve interesting questions regarding our Emscripten support. -The group also has an associated Zulip stream ([`#t-compiler/wasm`]) +The group also has an associated Zulip channel ([`#t-compiler/wasm`]) where people can go to pose questions and discuss Emscripten-specific topics. diff --git a/src/doc/rustc-dev-guide/src/notification-groups/risc-v.md b/src/doc/rustc-dev-guide/src/notification-groups/risc-v.md index 1b31297b600..7c8a3cdf8a6 100644 --- a/src/doc/rustc-dev-guide/src/notification-groups/risc-v.md +++ b/src/doc/rustc-dev-guide/src/notification-groups/risc-v.md @@ -9,7 +9,7 @@ This list will be used to ask for help both in diagnosing and testing RISC-V-related issues as well as suggestions on how to resolve interesting questions regarding our RISC-V support. -The group also has an associated Zulip stream ([`#t-compiler/risc-v`]) +The group also has an associated Zulip channel ([`#t-compiler/risc-v`]) where people can go to pose questions and discuss RISC-V-specific topics. diff --git a/src/doc/rustc-dev-guide/src/notification-groups/rust-for-linux.md b/src/doc/rustc-dev-guide/src/notification-groups/rust-for-linux.md index 696f2038e1a..ed1de9196de 100644 --- a/src/doc/rustc-dev-guide/src/notification-groups/rust-for-linux.md +++ b/src/doc/rustc-dev-guide/src/notification-groups/rust-for-linux.md @@ -12,7 +12,7 @@ and features. The RfL maintainers should then ideally provide support for resolving the breakage or decide to temporarily accept the breakage and unblock CI by temporarily removing the RfL CI jobs. -The group also has an associated Zulip stream ([`#rust-for-linux`]) +The group also has an associated Zulip channel ([`#rust-for-linux`]) where people can go to ask questions and discuss topics related to Rust for Linux. diff --git a/src/doc/rustc-dev-guide/src/notification-groups/wasi.md b/src/doc/rustc-dev-guide/src/notification-groups/wasi.md index e438ee4bd09..88b9465be01 100644 --- a/src/doc/rustc-dev-guide/src/notification-groups/wasi.md +++ b/src/doc/rustc-dev-guide/src/notification-groups/wasi.md @@ -9,7 +9,7 @@ This list will be used to ask for help both in diagnosing and testing WASI-related issues as well as suggestions on how to resolve interesting questions regarding our WASI support. -The group also has an associated Zulip stream ([`#t-compiler/wasm`]) +The group also has an associated Zulip channel ([`#t-compiler/wasm`]) where people can go to pose questions and discuss WASI-specific topics. diff --git a/src/doc/rustc-dev-guide/src/notification-groups/wasm.md b/src/doc/rustc-dev-guide/src/notification-groups/wasm.md index c8b674cb93f..6f52b04251f 100644 --- a/src/doc/rustc-dev-guide/src/notification-groups/wasm.md +++ b/src/doc/rustc-dev-guide/src/notification-groups/wasm.md @@ -9,7 +9,7 @@ This list will be used to ask for help both in diagnosing and testing WebAssembly-related issues as well as suggestions on how to resolve interesting questions regarding our WASM support. -The group also has an associated Zulip stream ([`#t-compiler/wasm`]) +The group also has an associated Zulip channel ([`#t-compiler/wasm`]) where people can go to pose questions and discuss WASM-specific topics. diff --git a/src/doc/rustc-dev-guide/src/notification-groups/windows.md b/src/doc/rustc-dev-guide/src/notification-groups/windows.md index e615a2cbd8d..d245208e2ab 100644 --- a/src/doc/rustc-dev-guide/src/notification-groups/windows.md +++ b/src/doc/rustc-dev-guide/src/notification-groups/windows.md @@ -9,7 +9,7 @@ This list will be used to ask for help both in diagnosing and testing Windows-related issues as well as suggestions on how to resolve interesting questions regarding our Windows support. -The group also has an associated Zulip stream ([`#t-compiler/windows`]) +The group also has an associated Zulip channel ([`#t-compiler/windows`]) where people can go to pose questions and discuss Windows-specific topics. diff --git a/src/doc/rustc-dev-guide/src/offload/installation.md b/src/doc/rustc-dev-guide/src/offload/installation.md index b376e962ff6..d1ebf33ac17 100644 --- a/src/doc/rustc-dev-guide/src/offload/installation.md +++ b/src/doc/rustc-dev-guide/src/offload/installation.md @@ -1,6 +1,6 @@ # Installation -In the future, `std::offload` should become available in nightly builds for users. For now, everyone still needs to build rustc from source. +`std::offload` is partly available in nightly builds for users. For now, everyone however still needs to build rustc from source to use all features of it. ## Build instructions @@ -42,30 +42,3 @@ run ``` ./x test --stage 1 tests/codegen-llvm/gpu_offload ``` - -## Usage -It is important to use a clang compiler build on the same llvm as rustc. Just calling clang without the full path will likely use your system clang, which probably will be incompatible. -``` -/absolute/path/to/rust/build/x86_64-unknown-linux-gnu/stage1/bin/rustc --edition=2024 --crate-type cdylib src/main.rs --emit=llvm-ir -O -C lto=fat -Cpanic=abort -Zoffload=Enable -/absolute/path/to/rust/build/x86_64-unknown-linux-gnu/llvm/bin/clang++ -fopenmp --offload-arch=native -g -O3 main.ll -o main -save-temps -LIBOMPTARGET_INFO=-1 ./main -``` -The first step will generate a `main.ll` file, which has enough instructions to cause the offload runtime to move data to and from a gpu. -The second step will use clang as the compilation driver to compile our IR file down to a working binary. Only a very small Rust subset will work out of the box here, unless -you use features like build-std, which are not covered by this guide. Look at the codegen test to get a feeling for how to write a working example. -In the last step you can run your binary, if all went well you will see a data transfer being reported: -``` -omptarget device 0 info: Entering OpenMP data region with being_mapper at unknown:0:0 with 1 arguments: -omptarget device 0 info: tofrom(unknown)[1024] -omptarget device 0 info: Creating new map entry with HstPtrBase=0x00007fffffff9540, HstPtrBegin=0x00007fffffff9540, TgtAllocBegin=0x0000155547200000, TgtPtrBegin=0x0000155547200000, Size=1024, DynRefCount=1, HoldRefCount=0, Name=unknown -omptarget device 0 info: Copying data from host to device, HstPtr=0x00007fffffff9540, TgtPtr=0x0000155547200000, Size=1024, Name=unknown -omptarget device 0 info: OpenMP Host-Device pointer mappings after block at unknown:0:0: -omptarget device 0 info: Host Ptr Target Ptr Size (B) DynRefCount HoldRefCount Declaration -omptarget device 0 info: 0x00007fffffff9540 0x0000155547200000 1024 1 0 unknown at unknown:0:0 -// some other output -omptarget device 0 info: Exiting OpenMP data region with end_mapper at unknown:0:0 with 1 arguments: -omptarget device 0 info: tofrom(unknown)[1024] -omptarget device 0 info: Mapping exists with HstPtrBegin=0x00007fffffff9540, TgtPtrBegin=0x0000155547200000, Size=1024, DynRefCount=0 (decremented, delayed deletion), HoldRefCount=0 -omptarget device 0 info: Copying data from device to host, TgtPtr=0x0000155547200000, HstPtr=0x00007fffffff9540, Size=1024, Name=unknown -omptarget device 0 info: Removing map entry with HstPtrBegin=0x00007fffffff9540, TgtPtrBegin=0x0000155547200000, Size=1024, Name=unknown -``` diff --git a/src/doc/rustc-dev-guide/src/offload/usage.md b/src/doc/rustc-dev-guide/src/offload/usage.md new file mode 100644 index 00000000000..9f519984d9b --- /dev/null +++ b/src/doc/rustc-dev-guide/src/offload/usage.md @@ -0,0 +1,112 @@ +# Usage + +This feature is work-in-progress, and not ready for usage. The instructions here are for contributors, or people interested in following the latest progress. +We currently work on launching the following Rust kernel on the GPU. To follow along, copy it to a `src/lib.rs` file. + +```rust +#![feature(abi_gpu_kernel)] +#![no_std] + +#[cfg(target_os = "linux")] +extern crate libc; +#[cfg(target_os = "linux")] +use libc::c_char; + +use core::mem; + +#[panic_handler] +fn panic(_: &core::panic::PanicInfo) -> ! { + loop {} +} + +#[cfg(target_os = "linux")] +#[unsafe(no_mangle)] +#[inline(never)] +fn main() { + let array_c: *mut [f64; 256] = + unsafe { libc::calloc(256, (mem::size_of::<f64>()) as libc::size_t) as *mut [f64; 256] }; + let output = c"The first element is zero %f\n"; + let output2 = c"The first element is NOT zero %f\n"; + let output3 = c"The second element is %f\n"; + unsafe { + let val: *const c_char = if (*array_c)[0] < 0.1 { + output.as_ptr() + } else { + output2.as_ptr() + }; + libc::printf(val, (*array_c)[0]); + } + + unsafe { + kernel_1(array_c); + } + core::hint::black_box(&array_c); + unsafe { + let val: *const c_char = if (*array_c)[0] < 0.1 { + output.as_ptr() + } else { + output2.as_ptr() + }; + libc::printf(val, (*array_c)[0]); + libc::printf(output3.as_ptr(), (*array_c)[1]); + } +} + +#[cfg(target_os = "linux")] +unsafe extern "C" { + pub fn kernel_1(array_b: *mut [f64; 256]); +} +``` + +## Compile instructions +It is important to use a clang compiler build on the same llvm as rustc. Just calling clang without the full path will likely use your system clang, which probably will be incompatible. So either substitute clang/lld invocations below with absolute path, or set your `PATH` accordingly. + +First we generate the host (cpu) code. The first build is just to compile libc, take note of the hashed path. Then we call rustc directly to build our host code, while providing the libc artifact to rustc. +``` +cargo +offload build -r -v +rustc +offload --edition 2024 src/lib.rs -g --crate-type cdylib -C opt-level=3 -C panic=abort -C lto=fat -L dependency=/absolute_path_to/target/release/deps --extern libc=/absolute_path_to/target/release/deps/liblibc-<HASH>.rlib --emit=llvm-bc,llvm-ir -Zoffload=Enable -Zunstable-options +``` + +Now we generate the device code. Replace the target-cpu with the right code for your gpu. +``` +RUSTFLAGS="-Ctarget-cpu=gfx90a --emit=llvm-bc,llvm-ir" cargo +offload build -Zunstable-options -r -v --target amdgcn-amd-amdhsa -Zbuild-std=core +``` + +Now find the <libname>.ll under target/amdgcn-amd-amdhsa folder and copy it to a device.ll file (or adjust the file names below). +If you work on an NVIDIA or Intel gpu, please adjust the names acordingly and open an issue to share your results (either if you succeed or fail). +First we compile our .ll files (good for manual inspections) to .bc files and clean up leftover artifacts. The cleanup is important, otherwise caching might interfere on following runs. +``` +opt lib.ll -o lib.bc +opt device.ll -o device.bc +rm *.o +rm bare.amdgcn.gfx90a.img* +``` + +``` +clang-offload-packager" "-o" "host.out" "--image=file=device.bc,triple=amdgcn-amd-amdhsa,arch=gfx90a,kind=openmp" + +clang-21" "-cc1" "-triple" "x86_64-unknown-linux-gnu" "-S" "-save-temps=cwd" "-disable-free" "-clear-ast-before-backend" "-main-file-name" "lib.rs" "-mrelocation-model" "pic" "-pic-level" "2" "-pic-is-pie" "-mframe-pointer=all" "-fmath-errno" "-ffp-contract=on" "-fno-rounding-math" "-mconstructor-aliases" "-funwind-tables=2" "-target-cpu" "x86-64" "-tune-cpu" "generic" "-resource-dir" "/<ABSOLUTE_PATH_TO>/rust/build/x86_64-unknown-linux-gnu/llvm/lib/clang/21" "-ferror-limit" "19" "-fopenmp" "-fopenmp-offload-mandatory" "-fgnuc-version=4.2.1" "-fskip-odr-check-in-gmf" "-fembed-offload-object=host.out" "-fopenmp-targets=amdgcn-amd-amdhsa" "-faddrsig" "-D__GCC_HAVE_DWARF2_CFI_ASM=1" "-o" "host.s" "-x" "ir" "lib.bc" + +clang-21" "-cc1as" "-triple" "x86_64-unknown-linux-gnu" "-filetype" "obj" "-main-file-name" "lib.rs" "-target-cpu" "x86-64" "-mrelocation-model" "pic" "-o" "host.o" "host.s" + +clang-linker-wrapper" "--should-extract=gfx90a" "--device-compiler=amdgcn-amd-amdhsa=-g" "--device-compiler=amdgcn-amd-amdhsa=-save-temps=cwd" "--device-linker=amdgcn-amd-amdhsa=-lompdevice" "--host-triple=x86_64-unknown-linux-gnu" "--save-temps" "--linker-path=/ABSOlUTE_PATH_TO/rust/build/x86_64-unknown-linux-gnu/lld/bin/ld.lld" "--hash-style=gnu" "--eh-frame-hdr" "-m" "elf_x86_64" "-pie" "-dynamic-linker" "/lib64/ld-linux-x86-64.so.2" "-o" "bare" "/lib/../lib64/Scrt1.o" "/lib/../lib64/crti.o" "/ABSOLUTE_PATH_TO/crtbeginS.o" "-L/ABSOLUTE_PATH_TO/rust/build/x86_64-unknown-linux-gnu/llvm/bin/../lib/x86_64-unknown-linux-gnu" "-L/ABSOLUTE_PATH_TO/rust/build/x86_64-unknown-linux-gnu/llvm/lib/clang/21/lib/x86_64-unknown-linux-gnu" "-L/lib/../lib64" "-L/usr/lib64" "-L/lib" "-L/usr/lib" "host.o" "-lstdc++" "-lm" "-lomp" "-lomptarget" "-L/ABSOLUTE_PATH_TO/rust/build/x86_64-unknown-linux-gnu/llvm/lib" "-lgcc_s" "-lgcc" "-lpthread" "-lc" "-lgcc_s" "-lgcc" "/ABSOLUTE_PATH_TO/crtendS.o" "/lib/../lib64/crtn.o" +``` + +Especially for the last command I recommend to not fix the paths, but rather just re-generate them by copying a bare-mode openmp example and compiling it with your clang. By adding `-###` to your clang invocation, you can see the invidual steps. +``` +myclang++ -fuse-ld=lld -O3 -fopenmp -fopenmp-offload-mandatory --offload-arch=gfx90a omp_bare.cpp -o main -### +``` + +In the final step, you can now run your binary + +``` +./main +The first element is zero 0.000000 +The first element is NOT zero 21.000000 +The second element is 0.000000 +``` + +To receive more information about the memory transfer, you can enable info printing with +``` +LIBOMPTARGET_INFO=-1 ./main +``` diff --git a/src/doc/rustc-dev-guide/src/profiling/with_perf.md b/src/doc/rustc-dev-guide/src/profiling/with_perf.md index 0d4f23bcd9a..e452dde5226 100644 --- a/src/doc/rustc-dev-guide/src/profiling/with_perf.md +++ b/src/doc/rustc-dev-guide/src/profiling/with_perf.md @@ -4,8 +4,7 @@ This is a guide for how to profile rustc with [perf](https://perf.wiki.kernel.or ## Initial steps -- Get a clean checkout of rust-lang/master, or whatever it is you want - to profile. +- Get a clean checkout of rust-lang/rust - Set the following settings in your `bootstrap.toml`: - `rust.debuginfo-level = 1` - enables line debuginfo - `rust.jemalloc = false` - lets you do memory use profiling with valgrind diff --git a/src/doc/rustc-dev-guide/src/queries/example-0.png b/src/doc/rustc-dev-guide/src/queries/example-0.png index 14b46c44f7d..dd67d5f2ef1 100644 --- a/src/doc/rustc-dev-guide/src/queries/example-0.png +++ b/src/doc/rustc-dev-guide/src/queries/example-0.png Binary files differdiff --git a/src/doc/rustc-dev-guide/src/rustdoc-internals/rustdoc-json-test-suite.md b/src/doc/rustc-dev-guide/src/rustdoc-internals/rustdoc-json-test-suite.md index e08f7709506..10e9452eda3 100644 --- a/src/doc/rustc-dev-guide/src/rustdoc-internals/rustdoc-json-test-suite.md +++ b/src/doc/rustc-dev-guide/src/rustdoc-internals/rustdoc-json-test-suite.md @@ -1,3 +1,83 @@ # The `rustdoc-json` test suite -> **FIXME**: This section is a stub. It will be populated by [PR #2422](https://github.com/rust-lang/rustc-dev-guide/pull/2422/). +This page is specifically about the test suite named `rustdoc-json`, which tests rustdoc's [json output]. +For other test suites used for testing rustdoc, see [§Rustdoc test suites](../tests/compiletest.md#rustdoc-test-suites). + +Tests are run with compiletest, and have access to the usual set of [directives](../tests/directives.md). +Frequenly used directives here are: + +- [`//@ aux-build`][aux-build] to have dependencies. +- `//@ edition: 2021` (or some other edition). +- `//@ compile-flags: --document-hidden-items` to enable [document private items]. + +Each crate's json output is checked by 2 programs: [jsondoclint](#jsondocck) and [jsondocck](#jsondocck). + +## jsondoclint + +[jsondoclint] checks that all [`Id`]s exist in the `index` (or `paths`). +This makes sure there are no dangling [`Id`]s. + +<!-- TODO: It does some more things too? +Also, talk about how it works + --> + +## jsondocck + +[jsondocck] processes direcives given in comments, to assert that the values in the output are expected. +It's alot like [htmldocck](./rustdoc-test-suite.md) in that way. + +It uses [JSONPath] as a query language, which takes a path, and returns a *list* of values that that path is said to match to. + +### Directives + +- `//@ has <path>`: Checks `<path>` exists, i.e. matches at least 1 value. +- `//@ !has <path>`: Checks `<path>` doesn't exist, i.e. matches 0 values. +- `//@ has <path> <value>`: Check `<path>` exists, and at least 1 of the matches is equal to the given `<value>` +- `//@ !has <path> <value>`: Checks `<path>` exists, but none of the matches equal the given `<value>`. +- `//@ is <path> <value>`: Check `<path>` matches exactly one value, and it's equal to the given `<value>`. +- `//@ is <path> <value> <value>...`: Check that `<path>` matches to exactly every given `<value>`. + Ordering doesn't matter here. +- `//@ !is <path> <value>`: Check `<path>` matches exactly one value, and that value is not equal to the given `<value>`. +- `//@ count <path> <number>`: Check that `<path>` matches to `<number>` of values. +- `//@ set <name> = <path>`: Check that `<path>` matches exactly one value, and store that value to the variable called `<name>`. + +These are defined in [`directive.rs`]. + +### Values + +Values can be either JSON values, or variables. + +- JSON values are JSON literals, e.g. `true`, `"string"`, `{"key": "value"}`. + These often need to be quoted using `'`, to be processed as 1 value. See [§Argument spliting](#argument-spliting) +- Variables can be used to store the value in one path, and use it in later queries. + They are set with the `//@ set <name> = <path>` directive, and accessed with `$<name>` + + ```rust + //@ set foo = $some.path + //@ is $.some.other.path $foo + ``` + +### Argument spliting + +Arguments to directives are split using the [shlex] crate, which implements POSIX shell escaping. +This is because both `<path>` and `<value>` arguments to [directives](#directives) frequently have both +whitespace and quote marks. + +To use the `@ is` with a `<path>` of `$.index[?(@.docs == "foo")].some.field` and a value of `"bar"` [^why_quote], you'd write: + +```rust +//@ is '$.is[?(@.docs == "foo")].some.field' '"bar"' +``` + +[^why_quote]: The value needs to be `"bar"` *after* shlex splitting, because we + it needs to be a JSON string value. + +[json output]: https://doc.rust-lang.org/nightly/rustdoc/unstable-features.html#json-output +[jsondocck]: https://github.com/rust-lang/rust/tree/master/src/tools/jsondocck +[jsondoclint]: https://github.com/rust-lang/rust/tree/master/src/tools/jsondoclint +[aux-build]: ../tests/compiletest.md#building-auxiliary-crates +[`Id`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustdoc_json_types/struct.Id.html +[document private items]: https://doc.rust-lang.org/nightly/rustdoc/command-line-arguments.html#--document-private-items-show-items-that-are-not-public +[`directive.rs`]: https://github.com/rust-lang/rust/blob/master/src/tools/jsondocck/src/directive.rs +[shlex]: https://docs.rs/shlex/1.3.0/shlex/index.html +[JSONPath]: https://www.rfc-editor.org/rfc/rfc9535.html diff --git a/src/doc/rustc-dev-guide/src/rustdoc.md b/src/doc/rustc-dev-guide/src/rustdoc.md index 9290fcd3b41..6127a894a0f 100644 --- a/src/doc/rustc-dev-guide/src/rustdoc.md +++ b/src/doc/rustc-dev-guide/src/rustdoc.md @@ -107,7 +107,7 @@ This comes with several caveats: in particular, rustdoc *cannot* run any parts o require type-checking bodies; for example it cannot generate `.rlib` files or run most lints. We want to move away from this model eventually, but we need some alternative for [the people using it][async-std]; see [various][zulip stop accepting broken code] -[previous][rustdoc meeting 2024-07-08] [zulip][compiler meeting 2023-01-26] [discussion][notriddle rfc]. +[previous][rustdoc meeting 2024-07-08] [Zulip][compiler meeting 2023-01-26] [discussion][notriddle rfc]. For examples of code that breaks if this hack is removed, see [`tests/rustdoc-ui/error-in-impl-trait`]. diff --git a/src/doc/rustc-dev-guide/src/sanitizers.md b/src/doc/rustc-dev-guide/src/sanitizers.md index 29d9056c15d..34c78d4d952 100644 --- a/src/doc/rustc-dev-guide/src/sanitizers.md +++ b/src/doc/rustc-dev-guide/src/sanitizers.md @@ -45,7 +45,7 @@ implementation: [marked][sanitizer-attribute] with appropriate LLVM attribute: `SanitizeAddress`, `SanitizeHWAddress`, `SanitizeMemory`, or `SanitizeThread`. By default all functions are instrumented, but this - behaviour can be changed with `#[no_sanitize(...)]`. + behaviour can be changed with `#[sanitize(xyz = "on|off")]`. * The decision whether to perform instrumentation or not is possible only at a function granularity. In the cases were those decision differ between diff --git a/src/doc/rustc-dev-guide/src/serialization.md b/src/doc/rustc-dev-guide/src/serialization.md index 8eb37bbe20b..702d3cfa6d5 100644 --- a/src/doc/rustc-dev-guide/src/serialization.md +++ b/src/doc/rustc-dev-guide/src/serialization.md @@ -106,7 +106,7 @@ type wrapper, like [`ty::Predicate`] and manually implementing `Encodable` and [`Encodable`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_serialize/trait.Encodable.html [`Encoder`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_serialize/trait.Encoder.html [`RefDecodable`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_middle/ty/codec/trait.RefDecodable.html -[`rustc_middle`]: https://doc.rust-lang.org/nightly/nightly-rustc/src/rustc_type_ir/codec.rs.html#21 +[`rustc_middle`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_middle/index.html [`ty::Predicate`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_middle/ty/predicate/struct.Predicate.html [`TyCtxt`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_middle/ty/struct.TyCtxt.html [`TyDecodable`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_macros/derive.TyDecodable.html diff --git a/src/doc/rustc-dev-guide/src/solve/caching.md b/src/doc/rustc-dev-guide/src/solve/caching.md index e568d47ca25..cb7403de4e0 100644 --- a/src/doc/rustc-dev-guide/src/solve/caching.md +++ b/src/doc/rustc-dev-guide/src/solve/caching.md @@ -106,7 +106,7 @@ We can implement this optimization in the future. [`provisional_result`]: https://github.com/rust-lang/rust/blob/7606c13961ddc1174b70638e934df0439b7dc515/compiler/rustc_trait_selection/src/solve/search_graph.rs#L57 [initial-prov-result]: https://github.com/rust-lang/rust/blob/7606c13961ddc1174b70638e934df0439b7dc515/compiler/rustc_trait_selection/src/solve/search_graph.rs#L366-L370 [fixpoint]: https://github.com/rust-lang/rust/blob/7606c13961ddc1174b70638e934df0439b7dc515/compiler/rustc_trait_selection/src/solve/search_graph.rs#L425-L446 -[^2]: summarizing the relevant [zulip thread] +[^2]: summarizing the relevant [Zulip thread] [zulip thread]: https://rust-lang.zulipchat.com/#narrow/stream/364551-t-types.2Ftrait-system-refactor/topic/global.20cache [unstable-result-ex]: https://github.com/rust-lang/rust/blob/7606c13961ddc1174b70638e934df0439b7dc515/tests/ui/traits/next-solver/cycles/coinduction/incompleteness-unstable-result.rs#L4-L16 diff --git a/src/doc/rustc-dev-guide/src/solve/candidate-preference.md b/src/doc/rustc-dev-guide/src/solve/candidate-preference.md index 89605294735..8b28f56760a 100644 --- a/src/doc/rustc-dev-guide/src/solve/candidate-preference.md +++ b/src/doc/rustc-dev-guide/src/solve/candidate-preference.md @@ -95,7 +95,7 @@ fn overflow<T: Trait>() { ``` This preference causes a lot of issues. See [#24066]. Most of the -issues are caused by prefering where-bounds over impls even if the where-bound guides type inference: +issues are caused by preferring where-bounds over impls even if the where-bound guides type inference: ```rust trait Trait<T> { fn call_me(&self, x: T) {} diff --git a/src/doc/rustc-dev-guide/src/solve/the-solver.md b/src/doc/rustc-dev-guide/src/solve/the-solver.md index 006fb649d22..0e095b55437 100644 --- a/src/doc/rustc-dev-guide/src/solve/the-solver.md +++ b/src/doc/rustc-dev-guide/src/solve/the-solver.md @@ -71,6 +71,6 @@ fails. ## Learning more The solver should be fairly self-contained. I hope that the above information provides a -good foundation when looking at the code itself. Please reach out on zulip if you get stuck +good foundation when looking at the code itself. Please reach out on Zulip if you get stuck while doing so or there are some quirks and design decisions which were unclear and deserve better comments or should be mentioned here. diff --git a/src/doc/rustc-dev-guide/src/stability.md b/src/doc/rustc-dev-guide/src/stability.md index c26d34273d7..3c4c65fdd5a 100644 --- a/src/doc/rustc-dev-guide/src/stability.md +++ b/src/doc/rustc-dev-guide/src/stability.md @@ -182,6 +182,11 @@ of the standard library raises it to a warning with `#![warn(deprecated_in_future)]`. ## unstable_feature_bound -The `#[unstable_feature_bound(foo)]` attribute can be used together with `#[unstable]` attribute to mark an `impl` of stable type and stable trait as unstable. In std/core, an item annotated with `#[unstable_feature_bound(foo)]` can only be used by another item that is also annotated with `#[unstable_feature_bound(foo)]`. Outside of std/core, using an item with `#[unstable_feature_bound(foo)]` requires the feature to be enabled with `#![feature(foo)]` attribute on the crate. Currently, only `impl`s and free functions can be annotated with `#[unstable_feature_bound]`. +The `#[unstable_feature_bound(foo)]` attribute can be used together with `#[unstable]` attribute to mark an `impl` of stable type and stable trait as unstable. In std/core, an item annotated with `#[unstable_feature_bound(foo)]` can only be used by another item that is also annotated with `#[unstable_feature_bound(foo)]`. Outside of std/core, using an item with `#[unstable_feature_bound(foo)]` requires the feature to be enabled with `#![feature(foo)]` attribute on the crate. + +Currently, the items that can be annotated with `#[unstable_feature_bound]` are: +- `impl` +- free function +- trait [blog]: https://www.ralfj.de/blog/2018/07/19/const.html diff --git a/src/doc/rustc-dev-guide/src/stabilization_guide.md b/src/doc/rustc-dev-guide/src/stabilization_guide.md index e399930fc52..7b9e1eb5629 100644 --- a/src/doc/rustc-dev-guide/src/stabilization_guide.md +++ b/src/doc/rustc-dev-guide/src/stabilization_guide.md @@ -82,7 +82,7 @@ Most importantly, remove the code which flags an error if the feature-gate is no gate_all!(pub_restricted, "`pub(restricted)` syntax is experimental"); ``` -This `gate_feature_post!` macro prints an error if the `pub_restricted` feature is not enabled. It is not needed now that `#[pub_restricted]` is stable. +The `gate_all!` macro reports an error if the `pub_restricted` feature is not enabled. It is not needed now that `pub(restricted)` is stable. For more subtle features, you may find code like this: diff --git a/src/doc/rustc-dev-guide/src/tests/adding.md b/src/doc/rustc-dev-guide/src/tests/adding.md index e5c26bef11d..46b8a1e4cf4 100644 --- a/src/doc/rustc-dev-guide/src/tests/adding.md +++ b/src/doc/rustc-dev-guide/src/tests/adding.md @@ -29,6 +29,8 @@ guidelines: suites. - Need to inspect the resulting binary in some way? Or if all the other test suites are too limited for your purposes? Then use `run-make`. + - Use `run-make-cargo` if you need to exercise in-tree `cargo` in conjunction + with in-tree `rustc`. - Check out the [compiletest] chapter for more specialized test suites. After deciding on which kind of test to add, see [best diff --git a/src/doc/rustc-dev-guide/src/tests/best-practices.md b/src/doc/rustc-dev-guide/src/tests/best-practices.md index be00207e3fb..10372c36ac9 100644 --- a/src/doc/rustc-dev-guide/src/tests/best-practices.md +++ b/src/doc/rustc-dev-guide/src/tests/best-practices.md @@ -71,7 +71,7 @@ related tests. > //! Regression test for <https://github.com/rust-lang/rust/issues/123456>. > ``` > -> One exception to this rule is [crashes tests]: there it is canonical that +> One exception to this rule is [crash tests]: there it is canonical that > tests are named only after issue numbers because its purpose is to track > snippets from which issues no longer ICE/crash, and they would either be > removed or converted into proper ui/other tests in the fix PRs. @@ -83,10 +83,10 @@ related tests. - E.g. for an implementation of RFC 2093 specifically, we can group a collection of tests under `tests/ui/rfc-2093-infer-outlives/`. For the directory name, include what the RFC is about. -- For the [`run-make`] test suite, each `rmake.rs` must be contained within an - immediate subdirectory under `tests/run-make/`. Further nesting is not - presently supported. Avoid including issue number in the directory name too, - include that info in a comment inside `rmake.rs`. +- For the [`run-make`]/`run-make-support` test suites, each `rmake.rs` must + be contained within an immediate subdirectory under `tests/run-make/` or + `tests/run-make-cargo/` respectively. Further nesting is not presently + supported. Avoid using _only_ an issue number for the test name as well. ## Test descriptions @@ -199,4 +199,4 @@ See [LLVM FileCheck guide][FileCheck] for details. [compiletest directives]: ./directives.md [`run-make`]: ./compiletest.md#run-make-tests [FileCheck]: https://llvm.org/docs/CommandGuide/FileCheck.html -[crashes tests]: ./compiletest.md#crashes-tests +[crash tests]: ./compiletest.md#crash-tests diff --git a/src/doc/rustc-dev-guide/src/tests/ci.md b/src/doc/rustc-dev-guide/src/tests/ci.md index 750e4fa1a0f..6c0b5c2e845 100644 --- a/src/doc/rustc-dev-guide/src/tests/ci.md +++ b/src/doc/rustc-dev-guide/src/tests/ci.md @@ -1,18 +1,18 @@ # Testing with CI The primary goal of our CI system is to ensure that the `master` branch of -`rust-lang/rust` is always in a valid state and passes our test suite. +`rust-lang/rust` is always in a valid state by passing our test suite. From a high-level point of view, when you open a pull request at `rust-lang/rust`, the following will happen: - A small [subset](#pull-request-builds) of tests and checks are run after each - push to the PR. This should help catching common errors. + push to the PR. This should help catch common errors. - When the PR is approved, the [bors] bot enqueues the PR into a [merge queue]. - Once the PR gets to the front of the queue, bors will create a merge commit and run the [full test suite](#auto-builds) on it. The merge commit either contains only one specific PR or it can be a ["rollup"](#rollups) which - combines multiple PRs together, to save CI costs. + combines multiple PRs together, to reduce CI costs and merge delays. - Once the whole test suite finishes, two things can happen. Either CI fails with an error that needs to be addressed by the developer, or CI succeeds and the merge commit is then pushed to the `master` branch. @@ -38,12 +38,12 @@ input, which contains a declarative configuration of all our CI jobs. > orchestrating the scripts that drive the process. In essence, all CI jobs run `./x test`, `./x dist` or some other command with -different configurations, across various operating systems, targets and +different configurations, across various operating systems, targets, and platforms. There are two broad categories of jobs that are executed, `dist` and non-`dist` jobs. - Dist jobs build a full release of the compiler for a specific platform, - including all the tools we ship through rustup; Those builds are then uploaded + including all the tools we ship through rustup. Those builds are then uploaded to the `rust-lang-ci2` S3 bucket and are available to be locally installed with the [rustup-toolchain-install-master] tool. The same builds are also used for actual releases: our release process basically consists of copying those @@ -70,7 +70,7 @@ these execute the `x86_64-gnu-llvm-X`, `x86_64-gnu-tools`, `pr-check-1`, `pr-che and `tidy` jobs, all running on Linux. These execute a relatively short (~40 minutes) and lightweight test suite that should catch common issues. More specifically, they run a set of lints, they try to perform a cross-compile check -build to Windows mingw (without producing any artifacts) and they test the +build to Windows mingw (without producing any artifacts), and they test the compiler using a *system* version of LLVM. Unfortunately, it would take too many resources to run the full test suite for each commit on every PR. @@ -84,29 +84,27 @@ resources to run the full test suite for each commit on every PR. > Thus, it is a good idea to run `./x doc xxx` locally for any doc comment > changes to help catch these early. -PR jobs are defined in the `pr` section of [`jobs.yml`]. They run under the -`rust-lang/rust` repository, and their results can be observed directly on the -PR, in the "CI checks" section at the bottom of the PR page. +PR jobs are defined in the `pr` section of [`jobs.yml`]. Their results can be observed +directly on the PR, in the "CI checks" section at the bottom of the PR page. ### Auto builds Before a commit can be merged into the `master` branch, it needs to pass our complete test suite. We call this an `auto` build. This build runs tens of CI jobs that exercise various tests across operating systems and targets. The full -test suite is quite slow; it can take two hours or more until all the `auto` CI +test suite is quite slow; it can take several hours until all the `auto` CI jobs finish. -Most platforms only run the build steps, some run a restricted set of tests, +Most platforms only run the build steps, some run a restricted set of tests; only a subset run the full suite of tests (see Rust's [platform tiers]). Auto jobs are defined in the `auto` section of [`jobs.yml`]. They are executed -on the `auto` branch under the `rust-lang/rust` repository and -their results can be seen [here](https://github.com/rust-lang/rust/actions), -although usually you will be notified of the result by a comment made by bors on -the corresponding PR. +on the `auto` branch under the `rust-lang/rust` repository, +and the final result will be reported via a comment made by bors on the corresponding PR. +The live results can be seen on [the GitHub Actions workflows page]. At any given time, at most a single `auto` build is being executed. Find out -more [here](#merging-prs-serially-with-bors). +more in [Merging PRs serially with bors](#merging-prs-serially-with-bors). [platform tiers]: https://forge.rust-lang.org/release/platform-support.html#rust-platform-support @@ -126,7 +124,7 @@ There are several use-cases for try builds: when you start a try build). To create a try build and schedule it for a performance benchmark, you can use the `@bors try @rust-timer queue` command combination. -- Check the impact of the PR across the Rust ecosystem, using a [crater] run. +- Check the impact of the PR across the Rust ecosystem, using a [Crater](crater.md) run. Again, a working compiler build is needed for this, which can be produced by the [dist-x86_64-linux] CI job. - Run a specific CI job (e.g. Windows tests) on a PR, to quickly test if it @@ -135,18 +133,26 @@ There are several use-cases for try builds: By default, if you send a comment with `@bors try`, the jobs defined in the `try` section of [`jobs.yml`] will be executed. We call this mode a "fast try build". Such a try build will not execute any tests, and it will allow compilation warnings. It is useful when you want to -get an optimized toolchain as fast as possible, for a crater run or performance benchmarks, -even if it might not be working fully correctly. - -If you want to run a custom CI job in a try build and make sure that it passes all tests and does -not produce any compilation warnings, you can select CI jobs to be executed by adding lines -containing `try-job: <job pattern>` to the PR description. All such specified jobs will be executed -in the try build once the `@bors try` command is used on the PR. - -Each pattern can either be an exact name of a job or a glob pattern that matches multiple jobs, +get an optimized toolchain as fast as possible, for a Crater run or performance benchmarks, +even if it might not be working fully correctly. If you want to do a full build for the default try job, +specify its job name in a job pattern (explained below). + +If you want to run custom CI jobs in a try build and make sure that they pass all tests and do +not produce any compilation warnings, you can select CI jobs to be executed by specifying a *job pattern*, +which can be used in one of two ways: +- You can add a set of `try-job: <job pattern>` directives to the PR description (described below) and then + simply run `@bors try`. CI will read these directives and run the jobs that you have specified. This is + useful if you want to rerun the same set of try jobs multiple times, after incrementally modifying a PR. +- You can specify the job pattern using the `jobs` parameter of the try command: `@bors try jobs=<job pattern>`. + This is useful for one-off try builds with specific jobs. Note that the `jobs` parameter has a higher priority + than the PR description directives. + - There can also be multiple patterns specified, e.g. `@bors try jobs=job1,job2,job3`. + +Each job pattern can either be an exact name of a job or a glob pattern that matches multiple jobs, for example `*msvc*` or `*-alt`. You can start at most 20 jobs in a single try build. When using -glob patterns, you might want to wrap them in backticks (`` ` ``) to avoid GitHub rendering -the pattern as Markdown. +glob patterns in the PR description, you can optionally wrap them in backticks (`` ` ``) to avoid GitHub rendering +the pattern as Markdown if it contains e.g. an asterisk. Note that this escaping will not work when using +the `@bors jobs=` parameter. The job pattern needs to match one or more jobs defined in the `auto` or `optional` sections of [`jobs.yml`]: @@ -182,26 +188,18 @@ of [`jobs.yml`]: > However, it can be less flexible because you cannot adjust the set of tests > that are exercised this way. -Try jobs are defined in the `try` section of [`jobs.yml`]. They are executed on -the `try` branch under the `rust-lang/rust` repository and -their results can be seen [here](https://github.com/rust-lang/rust/actions), +Try builds are executed on the `try` branch under the `rust-lang/rust` repository and +their results can be seen on [the GitHub Actions workflows page], although usually you will be notified of the result by a comment made by bors on the corresponding PR. -Note that if you start the default try job using `@bors try`, it will skip building several `dist` components and running post-optimization tests, to make the build duration shorter. If you want to execute the full build as it would happen before a merge, add an explicit `try-job` pattern with the name of the default try job (currently `dist-x86_64-linux`). - -Multiple try builds can execute concurrently across different PRs. +Multiple try builds can execute concurrently across different PRs, but there can be at most +a single try build running on a single PR at any given time. -<div class="warning"> - -Bors identifies try jobs by commit hash. This means that if you have two PRs -containing the same (latest) commits, running `@bors try` will result in the -*same* try job and it really confuses `bors`. Please refrain from doing so. - -</div> +Note that try builds are handled using the [new bors] implementation. [rustc-perf]: https://github.com/rust-lang/rustc-perf -[crater]: https://github.com/rust-lang/crater +[new bors]: https://github.com/rust-lang/bors ### Modifying CI jobs @@ -211,8 +209,7 @@ If you want to modify what gets executed on our CI, you can simply modify the You can also modify what gets executed temporarily, for example to test a particular platform or configuration that is challenging to test locally (for example, if a Windows build fails, but you don't have access to a Windows -machine). Don't hesitate to use CI resources in such situations to try out a -fix! +machine). Don't hesitate to use CI resources in such situations. You can perform an arbitrary CI job in two ways: - Use the [try build](#try-builds) functionality, and specify the CI jobs that @@ -255,8 +252,8 @@ purposes. </div> Although you are welcome to use CI, just be conscious that this is a shared -resource with limited concurrency. Try not to enable too many jobs at once (one -or two should be sufficient in most cases). +resource with limited concurrency. Try not to enable too many jobs at once; +one or two should be sufficient in most cases. ## Merging PRs serially with bors @@ -280,12 +277,12 @@ by listening for either Commit Statuses or Check Runs. Since the merge commit is based on the latest `master` and only one can be tested at the same time, when the results are green, `master` is fast-forwarded to that merge commit. -Unfortunately testing a single PR at the time, combined with our long CI (~2 -hours for a full run), means we can’t merge too many PRs in a single day, and a -single failure greatly impacts our throughput for the day. The maximum number of +Unfortunately, testing a single PR at a time, combined with our long CI (~2 +hours for a full run), means we can’t merge a lot of PRs in a single day, and a +single failure greatly impacts our throughput. The maximum number of PRs we can merge in a day is around ~10. -The large CI run times and requirement for a large builder pool is largely due +The long CI run times, and requirement for a large builder pool, is largely due to the fact that full release artifacts are built in the `dist-` builders. This is worth it because these release artifacts: @@ -298,12 +295,11 @@ is worth it because these release artifacts: Some PRs don’t need the full test suite to be executed: trivial changes like typo fixes or README improvements *shouldn’t* break the build, and testing every -single one of them for 2+ hours is a big waste of time. To solve this, we +single one of them for 2+ hours would be wasteful. To solve this, we regularly create a "rollup", a PR where we merge several pending trivial PRs so they can be tested together. Rollups are created manually by a team member using the "create a rollup" button on the [merge queue]. The team member uses their -judgment to decide if a PR is risky or not, and are the best tool we have at the -moment to keep the queue in a manageable state. +judgment to decide if a PR is risky or not. ## Docker @@ -316,18 +312,22 @@ platform’s custom [Docker container]. This has a lot of advantages for us: - We can use ancient build environments to ensure maximum binary compatibility, for example [using older CentOS releases][dist-x86_64-linux] on our Linux builders. -- We can avoid reinstalling tools (like QEMU or the Android emulator) every time +- We can avoid reinstalling tools (like QEMU or the Android emulator) every time, thanks to Docker image caching. -- Users can run the same tests in the same environment locally by just running - `cargo run --manifest-path src/ci/citool/Cargo.toml run-local <job-name>`, which is awesome to debug failures. Note that there are only linux docker images available locally due to licensing and +- Users can run the same tests in the same environment locally by just running this command: + + cargo run --manifest-path src/ci/citool/Cargo.toml run-local <job-name> + + This is helpful for debugging failures. + Note that there are only Linux Docker images available locally due to licensing and other restrictions. -The docker images prefixed with `dist-` are used for building artifacts while +The Docker images prefixed with `dist-` are used for building artifacts while those without that prefix run tests and checks. We also run tests for less common architectures (mainly Tier 2 and Tier 3 -platforms) in CI. Since those platforms are not x86 we either run everything -inside QEMU or just cross-compile if we don’t want to run the tests for that +platforms) in CI. Since those platforms are not x86, we either run everything +inside QEMU, or we just cross-compile if we don’t want to run the tests for that platform. These builders are running on a special pool of builders set up and maintained @@ -364,41 +364,41 @@ invalidated if one of the following changes: [ghcr.io]: https://github.com/rust-lang/rust/pkgs/container/rust-ci [Docker registry caching]: https://docs.docker.com/build/cache/backends/registry/ -### LLVM caching with sccache +### LLVM caching with Sccache -We build some C/C++ stuff in various CI jobs, and we rely on [sccache] to cache +We build some C/C++ stuff in various CI jobs, and we rely on [Sccache] to cache the intermediate LLVM artifacts. Sccache is a distributed ccache developed by Mozilla, which can use an object storage bucket as the storage backend. -With sccache there's no need to calculate the hash key ourselves. Sccache +With Sccache there's no need to calculate the hash key ourselves. Sccache invalidates the cache automatically when it detects changes to relevant inputs, such as the source code, the version of the compiler, and important environment variables. -So we just pass the sccache wrapper on top of cargo and sccache does the rest. +So we just pass the Sccache wrapper on top of Cargo and Sccache does the rest. -We store the persistent artifacts on the S3 bucket `rust-lang-ci-sccache2`. So -when the CI runs, if sccache sees that LLVM is being compiled with the same C/C++ -compiler and the LLVM source code is the same, sccache retrieves the individual +We store the persistent artifacts on the S3 bucket, `rust-lang-ci-sccache2`. So +when the CI runs, if Sccache sees that LLVM is being compiled with the same C/C++ +compiler and the LLVM source code is the same, Sccache retrieves the individual compiled translation units from S3. [sccache]: https://github.com/mozilla/sccache ## Custom tooling around CI -During the years we developed some custom tooling to improve our CI experience. +During the years, we developed some custom tooling to improve our CI experience. ### Rust Log Analyzer to show the error message in PRs The build logs for `rust-lang/rust` are huge, and it’s not practical to find -what caused the build to fail by looking at the logs. To improve the developers’ -experience we developed a bot called [Rust Log Analyzer][rla] (RLA) that -receives the build logs on failure and extracts the error message automatically, -posting it on the PR. +what caused the build to fail by looking at the logs. +We therefore developed a bot called [Rust Log Analyzer][rla] (RLA) that +receives the build logs on failure, and extracts the error message automatically, +posting it on the PR thread. The bot is not hardcoded to look for error strings, but was trained with a bunch of build failures to recognize which lines are common between builds and which are not. While the generated snippets can be weird sometimes, the bot is pretty -good at identifying the relevant lines even if it’s an error we've never seen +good at identifying the relevant lines, even if it’s an error we've never seen before. [rla]: https://github.com/rust-lang/rust-log-analyzer @@ -430,11 +430,11 @@ More information is available in the [toolstate documentation]. ## Public CI dashboard -To monitor the Rust CI, you can have a look at the [public dashboard] maintained by the infra-team. +To monitor the Rust CI, you can have a look at the [public dashboard] maintained by the infra team. These are some useful panels from the dashboard: -- Pipeline duration: check how long the auto builds takes to run. +- Pipeline duration: check how long the auto builds take to run. - Top slowest jobs: check which jobs are taking the longest to run. - Change in median job duration: check what jobs are slowest than before. Useful to detect regressions. @@ -457,8 +457,7 @@ this: 2. Choose the job you are interested in on the left-hand side. 3. Click on the gear icon and choose "View raw logs" 4. Search for the string "Configure the build" -5. All of the build settings are listed below that starting with the - `configure:` prefix. +5. All of the build settings are listed on the line with the text, `build.configure-args` [GitHub Actions]: https://github.com/rust-lang/rust/actions [`jobs.yml`]: https://github.com/rust-lang/rust/blob/master/src/ci/github-actions/jobs.yml @@ -468,3 +467,4 @@ this: [homu]: https://github.com/rust-lang/homu [merge queue]: https://bors.rust-lang.org/queue/rust [dist-x86_64-linux]: https://github.com/rust-lang/rust/blob/master/src/ci/docker/host-x86_64/dist-x86_64-linux/Dockerfile +[the GitHub Actions workflows page]: https://github.com/rust-lang/rust/actions diff --git a/src/doc/rustc-dev-guide/src/tests/codegen-backend-tests/cg_gcc.md b/src/doc/rustc-dev-guide/src/tests/codegen-backend-tests/cg_gcc.md index 4caf4c0e0ee..74dbfb7d31b 100644 --- a/src/doc/rustc-dev-guide/src/tests/codegen-backend-tests/cg_gcc.md +++ b/src/doc/rustc-dev-guide/src/tests/codegen-backend-tests/cg_gcc.md @@ -1,3 +1,56 @@ -# GCC codegen backend tests +# GCC codegen backend -TODO: please add some more information to this page. +If you ran into an error related to tests executed with the GCC codegen backend on CI, +you can use the following command to run tests locally using the GCC backend: + +```bash +./x test tests/ui --set 'rust.codegen-backends = ["llvm", "gcc"]' --test-codegen-backend gcc +``` + +Below, you can find more information about how to configure the GCC backend in bootstrap. + +## Choosing which codegen backends are built + +The `rust.codegen-backends = [...]` bootstrap option affects which codegen backends will be built and +included in the sysroot of the produced `rustc`. To use the GCC codegen backend, `"gcc"` has to +be included in this array in `bootstrap.toml`: + +```toml +rust.codegen-backends = ["llvm", "gcc"] +``` + +If you don't want to change your `bootstrap.toml` file, you can alternatively run your `x` +commands with `--set rust.codegen-backends=["llvm", "gcc"]'`. For example: + +```bash +./x build --set 'rust.codegen-backends=["llvm", "gcc"]' +``` + +The first backend in the `codegen-backends` array will determine which backend will be used as the +*default backend* of the built `rustc`. This also determines which backend will be used to compile the +stage 1 standard library (or anything built in stage 2+). To produce `rustc` that uses the GCC backend +by default, you can thus put `"gcc"` as the first element of this array: + +```bash +./x build --set 'rust.codegen-backends=["gcc"]' library +``` + +## Choosing the codegen backend used in tests + +To run compiler tests with the GCC codegen backend being used to build the test Rust programs, you can use the +`--test-codegen-backend` flag: + +```bash +./x test tests/ui --test-codegen-backend gcc +``` + +Note that in order for this to work, the tested compiler must have the GCC codegen backend available in its sysroot +directory. You can achieve that using the [instructions above](#choosing-which-codegen-backends-are-built). + +## Downloading GCC from CI + +The `gcc.download-ci-gcc` bootstrap option controls if GCC (which is a dependency of the GCC codegen backend) +will be downloaded from CI or built locally. The default value is `true`, which will download GCC from CI +if there are no local changes to the GCC sources and the given host target is available on CI. + +Note that GCC can currently only be downloaded from CI for the `x86_64-unknown-linux-gnu` target. diff --git a/src/doc/rustc-dev-guide/src/tests/compiletest.md b/src/doc/rustc-dev-guide/src/tests/compiletest.md index 4980ed845d6..0234b394c2a 100644 --- a/src/doc/rustc-dev-guide/src/tests/compiletest.md +++ b/src/doc/rustc-dev-guide/src/tests/compiletest.md @@ -72,7 +72,7 @@ The following test suites are available, with links for more information: | [`mir-opt`](#mir-opt-tests) | Check MIR generation and optimizations | | [`coverage`](#coverage-tests) | Check coverage instrumentation | | [`coverage-run-rustdoc`](#coverage-tests) | `coverage` tests that also run instrumented doctests | -| [`crashes`](#crashes-tests) | Check that the compiler ICEs/panics/crashes on certain inputs to catch accidental fixes | +| [`crashes`](#crash-tests) | Check that the compiler ICEs/panics/crashes on certain inputs to catch accidental fixes | ### General purpose test suite @@ -397,13 +397,19 @@ your test, causing separate files to be generated for 32bit and 64bit systems. ### `run-make` tests -The tests in [`tests/run-make`] are general-purpose tests using Rust *recipes*, -which are small programs (`rmake.rs`) allowing arbitrary Rust code such as -`rustc` invocations, and is supported by a [`run_make_support`] library. Using -Rust recipes provide the ultimate in flexibility. +The tests in [`tests/run-make`] and [`tests/run-make-cargo`] are general-purpose +tests using Rust *recipes*, which are small programs (`rmake.rs`) allowing +arbitrary Rust code such as `rustc` invocations, and is supported by a +[`run_make_support`] library. Using Rust recipes provide the ultimate in +flexibility. `run-make` tests should be used if no other test suites better suit your needs. +The `run-make-cargo` test suite additionally builds an in-tree `cargo` to support +use cases that require testing in-tree `cargo` in conjunction with in-tree `rustc`. +The `run-make` test suite does not have access to in-tree `cargo` (so it can be the +faster-to-iterate test suite). + #### Using Rust recipes Each test should be in a separate directory with a `rmake.rs` Rust program, @@ -476,6 +482,7 @@ Then add a corresponding entry to `"rust-analyzer.linkedProjects"` ``` [`tests/run-make`]: https://github.com/rust-lang/rust/tree/master/tests/run-make +[`tests/run-make-cargo`]: https://github.com/rust-lang/rust/tree/master/tests/run-make-cargo [`run_make_support`]: https://github.com/rust-lang/rust/tree/master/src/tools/run-make-support ### Coverage tests @@ -550,7 +557,7 @@ only running the main `coverage` suite. [`src/tools/coverage-dump`]: https://github.com/rust-lang/rust/tree/master/src/tools/coverage-dump [`tests/coverage-run-rustdoc`]: https://github.com/rust-lang/rust/tree/master/tests/coverage-run-rustdoc -### Crashes tests +### Crash tests [`tests/crashes`] serve as a collection of tests that are expected to cause the compiler to ICE, panic or crash in some other way, so that accidental fixes are @@ -573,13 +580,13 @@ recommended to include test cases from several issues in a single PR. When you do so, each issue number should be noted in the file name (`12345.rs` should suffice) and also inside the file by means of a `//@ known-bug: #12345` directive. Please [label][labeling] the relevant issues with `S-bug-has-test` -afterwards. +once your PR is merged. If you happen to fix one of the crashes, please move it to a fitting subdirectory in `tests/ui` and give it a meaningful name. Please add a doc comment at the top of the file explaining why this test exists, even better if you can briefly explain how the example causes rustc to crash previously and -what was done to prevent rustc to ICE/panic/crash. +what was done to prevent rustc to ICE / panic / crash. Adding diff --git a/src/doc/rustc-dev-guide/src/tests/crater.md b/src/doc/rustc-dev-guide/src/tests/crater.md index 9d4ac87daf3..96bb5a4f2ae 100644 --- a/src/doc/rustc-dev-guide/src/tests/crater.md +++ b/src/doc/rustc-dev-guide/src/tests/crater.md @@ -8,30 +8,30 @@ stable compiler versions. ## When to run Crater -You should request a crater run if your PR makes large changes to the compiler +You should request a Crater run if your PR makes large changes to the compiler or could cause breakage. If you are unsure, feel free to ask your PR's reviewer. ## Requesting Crater Runs -The rust team maintains a few machines that can be used for running crater runs -on the changes introduced by a PR. If your PR needs a crater run, leave a +The Rust team maintains a few machines that can be used for Crater runs +on the changes introduced by a PR. If your PR needs a Crater run, leave a comment for the triage team in the PR thread. Please inform the team whether you -require a "check-only" crater run, a "build only" crater run, or a -"build-and-test" crater run. The difference is primarily in time; the -conservative (if you're not sure) option is to go for the build-and-test run. If +require a "check-only" Crater run, a "build only" Crater run, or a +"build-and-test" Crater run. The difference is primarily in time; +if you're not sure, go for the build-and-test run. If making changes that will only have an effect at compile-time (e.g., implementing -a new trait) then you only need a check run. +a new trait), then you only need a check run. Your PR will be enqueued by the triage team and the results will be posted when -they are ready. Check runs will take around ~3-4 days, with the other two taking +they are ready. Check runs will take around ~3-4 days, and the other two taking 5-6 days on average. -While crater is really useful, it is also important to be aware of a few +While Crater is really useful, it is also important to be aware of a few caveats: - Not all code is on crates.io! There is a lot of code in repos on GitHub and elsewhere. Also, companies may not wish to publish their code. Thus, a - successful crater run is not a magically green light that there will be no + successful Crater run does not mean there will be no breakage; you still need to be careful. - Crater only runs Linux builds on x86_64. Thus, other architectures and @@ -41,5 +41,5 @@ caveats: the crate doesn't compile any more (e.g. used old nightly features), has broken or flaky tests, requires network access, or other reasons. -- Before crater can be run, `@bors try` needs to succeed in building artifacts. - This means that if your code doesn't compile, you cannot run crater. +- Before Crater can be run, `@bors try` needs to succeed in building artifacts. + This means that if your code doesn't compile, you cannot run Crater. diff --git a/src/doc/rustc-dev-guide/src/tests/directives.md b/src/doc/rustc-dev-guide/src/tests/directives.md index f4ba9a044e6..4be78fac4ec 100644 --- a/src/doc/rustc-dev-guide/src/tests/directives.md +++ b/src/doc/rustc-dev-guide/src/tests/directives.md @@ -36,9 +36,9 @@ directive. The following is a list of compiletest directives. Directives are linked to sections that describe the command in more detail if available. This list may not be exhaustive. Directives can generally be found by browsing the -`TestProps` structure found in [`header.rs`] from the compiletest source. +`TestProps` structure found in [`directives.rs`] from the compiletest source. -[`header.rs`]: https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/header.rs +[`directives.rs`]: https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/directives.rs ### Assembly @@ -52,14 +52,14 @@ not be exhaustive. Directives can generally be found by browsing the See [Building auxiliary crates](compiletest.html#building-auxiliary-crates) -| Directive | Explanation | Supported test suites | Possible values | -|-----------------------|-------------------------------------------------------------------------------------------------------|-----------------------|-----------------------------------------------| -| `aux-bin` | Build a aux binary, made available in `auxiliary/bin` relative to test directory | All except `run-make` | Path to auxiliary `.rs` file | -| `aux-build` | Build a separate crate from the named source file | All except `run-make` | Path to auxiliary `.rs` file | -| `aux-crate` | Like `aux-build` but makes available as extern prelude | All except `run-make` | `<extern_prelude_name>=<path/to/aux/file.rs>` | -| `aux-codegen-backend` | Similar to `aux-build` but pass the compiled dylib to `-Zcodegen-backend` when building the main file | `ui-fulldeps` | Path to codegen backend file | -| `proc-macro` | Similar to `aux-build`, but for aux forces host and don't use `-Cprefer-dynamic`[^pm]. | All except `run-make` | Path to auxiliary proc-macro `.rs` file | -| `build-aux-docs` | Build docs for auxiliaries as well. Note that this only works with `aux-build`, not `aux-crate`. | All except `run-make` | N/A | +| Directive | Explanation | Supported test suites | Possible values | +|-----------------------|-------------------------------------------------------------------------------------------------------|----------------------------------------|-----------------------------------------------| +| `aux-bin` | Build a aux binary, made available in `auxiliary/bin` relative to test directory | All except `run-make`/`run-make-cargo` | Path to auxiliary `.rs` file | +| `aux-build` | Build a separate crate from the named source file | All except `run-make`/`run-make-cargo` | Path to auxiliary `.rs` file | +| `aux-crate` | Like `aux-build` but makes available as extern prelude | All except `run-make`/`run-make-cargo` | `<extern_prelude_name>=<path/to/aux/file.rs>` | +| `aux-codegen-backend` | Similar to `aux-build` but pass the compiled dylib to `-Zcodegen-backend` when building the main file | `ui-fulldeps` | Path to codegen backend file | +| `proc-macro` | Similar to `aux-build`, but for aux forces host and don't use `-Cprefer-dynamic`[^pm]. | All except `run-make`/`run-make-cargo` | Path to auxiliary proc-macro `.rs` file | +| `build-aux-docs` | Build docs for auxiliaries as well. Note that this only works with `aux-build`, not `aux-crate`. | All except `run-make`/`run-make-cargo` | N/A | [^pm]: please see the [Auxiliary proc-macro section](compiletest.html#auxiliary-proc-macro) in the compiletest chapter for specifics. @@ -111,6 +111,7 @@ for more details. | `forbid-output` | A pattern which must not appear in stderr/`cfail` output | `ui`, `incremental` | Regex pattern | | `run-flags` | Flags passed to the test executable | `ui` | Arbitrary flags | | `known-bug` | No error annotation needed due to known bug | `ui`, `crashes`, `incremental` | Issue number `#123456` | +| `compare-output-by-lines` | Compare the output by lines, rather than as a single string | All | N/A | [^check_stdout]: presently <!-- date-check: Oct 2024 --> this has a weird quirk where the test binary's stdout and stderr gets concatenated and then @@ -242,18 +243,18 @@ ignoring debuggers. ### Affecting how tests are built -| Directive | Explanation | Supported test suites | Possible values | -|---------------------|----------------------------------------------------------------------------------------------|---------------------------|--------------------------------------------------------------------------------------------| -| `compile-flags` | Flags passed to `rustc` when building the test or aux file | All except for `run-make` | Any valid `rustc` flags, e.g. `-Awarnings -Dfoo`. Cannot be `-Cincremental` or `--edition` | -| `edition` | The edition used to build the test | All except for `run-make` | Any valid `--edition` value | -| `rustc-env` | Env var to set when running `rustc` | All except for `run-make` | `<KEY>=<VALUE>` | -| `unset-rustc-env` | Env var to unset when running `rustc` | All except for `run-make` | Any env var name | -| `incremental` | Proper incremental support for tests outside of incremental test suite | `ui`, `crashes` | N/A | -| `no-prefer-dynamic` | Don't use `-C prefer-dynamic`, don't build as a dylib via a `--crate-type=dylib` preset flag | `ui`, `crashes` | N/A | +| Directive | Explanation | Supported test suites | Possible values | +|---------------------|----------------------------------------------------------------------------------------------|--------------------------------------------|--------------------------------------------------------------------------------------------| +| `compile-flags` | Flags passed to `rustc` when building the test or aux file | All except for `run-make`/`run-make-cargo` | Any valid `rustc` flags, e.g. `-Awarnings -Dfoo`. Cannot be `-Cincremental` or `--edition` | +| `edition` | The edition used to build the test | All except for `run-make`/`run-make-cargo` | Any valid `--edition` value | +| `rustc-env` | Env var to set when running `rustc` | All except for `run-make`/`run-make-cargo` | `<KEY>=<VALUE>` | +| `unset-rustc-env` | Env var to unset when running `rustc` | All except for `run-make`/`run-make-cargo` | Any env var name | +| `incremental` | Proper incremental support for tests outside of incremental test suite | `ui`, `crashes` | N/A | +| `no-prefer-dynamic` | Don't use `-C prefer-dynamic`, don't build as a dylib via a `--crate-type=dylib` preset flag | `ui`, `crashes` | N/A | <div class="warning"> -Tests (outside of `run-make`) that want to use incremental tests not in the +Tests (outside of `run-make`/`run-make-cargo`) that want to use incremental tests not in the incremental test-suite must not pass `-C incremental` via `compile-flags`, and must instead use the `//@ incremental` directive. @@ -263,9 +264,9 @@ Consider writing the test as a proper incremental test instead. ### Rustdoc -| Directive | Explanation | Supported test suites | Possible values | -|-------------|--------------------------------------------------------------|------------------------------------------|---------------------------| -| `doc-flags` | Flags passed to `rustdoc` when building the test or aux file | `rustdoc`, `rustdoc-js`, `rustdoc-json` | Any valid `rustdoc` flags | +| Directive | Explanation | Supported test suites | Possible values | +|-------------|--------------------------------------------------------------|-----------------------------------------|---------------------------| +| `doc-flags` | Flags passed to `rustdoc` when building the test or aux file | `rustdoc`, `rustdoc-js`, `rustdoc-json` | Any valid `rustdoc` flags | <!-- **FIXME(rustdoc)**: what does `check-test-line-numbers-match` do? @@ -373,7 +374,7 @@ the directive's backing store (holds the command's current value) at runtime. To add a new directive property: 1. Look for the `pub struct TestProps` declaration in - [`src/tools/compiletest/src/header.rs`] and add the new public property to + [`src/tools/compiletest/src/directives.rs`] and add the new public property to the end of the declaration. 2. Look for the `impl TestProps` implementation block immediately following the struct declaration and initialize the new property to its default value. @@ -382,7 +383,7 @@ To add a new directive property: When `compiletest` encounters a test file, it parses the file a line at a time by calling every parser defined in the `Config` struct's implementation block, -also in [`src/tools/compiletest/src/header.rs`] (note that the `Config` struct's +also in [`src/tools/compiletest/src/directives.rs`] (note that the `Config` struct's declaration block is found in [`src/tools/compiletest/src/common.rs`]). `TestProps`'s `load_from()` method will try passing the current line of text to each parser, which, in turn typically checks to see if the line begins with a @@ -405,7 +406,7 @@ and their associated parsers immediately above to see how they are used to avoid writing additional parsing code unnecessarily. As a concrete example, here is the implementation for the -`parse_failure_status()` parser, in [`src/tools/compiletest/src/header.rs`]: +`parse_failure_status()` parser, in [`src/tools/compiletest/src/directives.rs`]: ```diff @@ -232,6 +232,7 @@ pub struct TestProps { @@ -507,6 +508,6 @@ example, `//@ failure-status: 1`, `self.props.failure_status` will evaluate to 1, as `parse_failure_status()` will have overridden the `TestProps` default value, for that test specifically. -[`src/tools/compiletest/src/header.rs`]: https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/header.rs +[`src/tools/compiletest/src/directives.rs`]: https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/directives.rs [`src/tools/compiletest/src/common.rs`]: https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/common.rs [`src/tools/compiletest/src/runtest.rs`]: https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/runtest.rs diff --git a/src/doc/rustc-dev-guide/src/tests/ecosystem-test-jobs/fuchsia.md b/src/doc/rustc-dev-guide/src/tests/ecosystem-test-jobs/fuchsia.md index b19d94d6ff7..75cf782a770 100644 --- a/src/doc/rustc-dev-guide/src/tests/ecosystem-test-jobs/fuchsia.md +++ b/src/doc/rustc-dev-guide/src/tests/ecosystem-test-jobs/fuchsia.md @@ -18,12 +18,8 @@ Fuchsia builds as part of the suite of bors tests that run before a pull request is merged. If you are worried that a pull request might break the Fuchsia builder and want -to test it out before submitting it to the bors queue, simply add this line to -your PR description: - -> try-job: x86_64-fuchsia - -Then when you `@bors try` it will pick the job that builds Fuchsia. +to test it out before submitting it to the bors queue, simply ask bors to run +the try job that builds the Fuchsia integration: `@bors try jobs=x86_64-fuchsia`. ## Building Fuchsia locally diff --git a/src/doc/rustc-dev-guide/src/tests/ecosystem-test-jobs/rust-for-linux.md b/src/doc/rustc-dev-guide/src/tests/ecosystem-test-jobs/rust-for-linux.md index d549ec6fca5..a6a7374b811 100644 --- a/src/doc/rustc-dev-guide/src/tests/ecosystem-test-jobs/rust-for-linux.md +++ b/src/doc/rustc-dev-guide/src/tests/ecosystem-test-jobs/rust-for-linux.md @@ -40,12 +40,8 @@ this sysroot. RfL uses several unstable compiler/language features, therefore this workflow notifies us if a given compiler change would break it. If you are worried that a pull request might break the Rust for Linux builder -and want to test it out before submitting it to the bors queue, simply add this -line to your PR description: - -> try-job: x86_64-rust-for-linux - -Then when you `@bors try` it will pick the job that builds the Rust for Linux -integration. +and want to test it out before submitting it to the bors queue, simply ask +bors to run the try job that builds the Rust for Linux integration: +`@bors try jobs=x86_64-rust-for-linux`. [rfl-ping]: ../../notification-groups/rust-for-linux.md diff --git a/src/doc/rustc-dev-guide/src/tests/intro.md b/src/doc/rustc-dev-guide/src/tests/intro.md index b90c16d602c..4fa63b83b17 100644 --- a/src/doc/rustc-dev-guide/src/tests/intro.md +++ b/src/doc/rustc-dev-guide/src/tests/intro.md @@ -70,10 +70,12 @@ package tests: Tidy is a custom tool used for validating source code style and formatting conventions, such as rejecting long lines. There is more information in the -[section on coding conventions](../conventions.md#formatting). +[section on coding conventions](../conventions.md#formatting) or the [Tidy Readme]. > Examples: `./x test tidy` +[Tidy Readme]: https://github.com/rust-lang/rust/blob/master/src/tools/tidy/Readme.md + ### Formatting diff --git a/src/doc/rustc-dev-guide/src/tests/misc.md b/src/doc/rustc-dev-guide/src/tests/misc.md index 39f88174879..cc8f501224f 100644 --- a/src/doc/rustc-dev-guide/src/tests/misc.md +++ b/src/doc/rustc-dev-guide/src/tests/misc.md @@ -24,8 +24,8 @@ In `ui` tests and other test suites that support `//@ rustc-env`, you can specif //@ rustc-env:RUSTC_BOOTSTRAP=-1 ``` -For `run-make` tests, `//@ rustc-env` is not supported. You can do something -like the following for individual `rustc` invocations. +For `run-make`/`run-make-cargo` tests, `//@ rustc-env` is not supported. You can do +something like the following for individual `rustc` invocations. ```rust,ignore use run_make_support::rustc; diff --git a/src/doc/rustc-dev-guide/src/tests/running.md b/src/doc/rustc-dev-guide/src/tests/running.md index 317b65f98cd..482f3c42578 100644 --- a/src/doc/rustc-dev-guide/src/tests/running.md +++ b/src/doc/rustc-dev-guide/src/tests/running.md @@ -339,9 +339,34 @@ results. The Docker image is set up to launch `remote-test-server` and the build tools use `remote-test-client` to communicate with the server to coordinate running tests (see [src/bootstrap/src/core/build_steps/test.rs]). -> **TODO** -> -> - Is there any support for using an iOS emulator? +To run on the iOS/tvOS/watchOS/visionOS simulator, we can similarly treat it as +a "remote" machine. A curious detail here is that the network is shared between +the simulator instance and the host macOS, so we can use the local loopback +address `127.0.0.1`. Something like the following should work: + +```sh +# Build the test server for the iOS simulator: +./x build src/tools/remote-test-server --target aarch64-apple-ios-sim + +# If you already have a simulator instance open, copy the device UUID from: +xcrun simctl list devices booted +UDID=01234567-89AB-CDEF-0123-456789ABCDEF + +# Alternatively, create and boot a new simulator instance: +xcrun simctl list runtimes +xcrun simctl list devicetypes +UDID=$(xcrun simctl create $CHOSEN_DEVICE_TYPE $CHOSEN_RUNTIME) +xcrun simctl boot $UDID +# See https://nshipster.com/simctl/ for details. + +# Spawn the runner on port 12345: +xcrun simctl spawn $UDID ./build/host/stage2-tools/aarch64-apple-ios-sim/release/remote-test-server -v --bind 127.0.0.1:12345 + +# In a new terminal, run tests via the runner: +export TEST_DEVICE_ADDR="127.0.0.1:12345" +./x test --host='' --target aarch64-apple-ios-sim --skip tests/debuginfo +# FIXME(madsmtm): Allow debuginfo tests to work (maybe needs `.dSYM` folder to be copied to the target?). +``` [armhf-gnu]: https://github.com/rust-lang/rust/tree/master/src/ci/docker/host-x86_64/armhf-gnu/Dockerfile [QEMU]: https://www.qemu.org/ diff --git a/src/doc/rustc-dev-guide/src/tests/ui.md b/src/doc/rustc-dev-guide/src/tests/ui.md index 25dd5814cf6..d3a2c406402 100644 --- a/src/doc/rustc-dev-guide/src/tests/ui.md +++ b/src/doc/rustc-dev-guide/src/tests/ui.md @@ -95,6 +95,7 @@ will check for output files: [Normalization](#normalization)). - `dont-check-compiler-stderr` — Ignores stderr from the compiler. - `dont-check-compiler-stdout` — Ignores stdout from the compiler. +- `compare-output-by-lines` — Some tests have non-deterministic orders of output, so we need to compare by lines. UI tests run with `-Zdeduplicate-diagnostics=no` flag which disables rustc's built-in diagnostic deduplication mechanism. This means you may see some diff --git a/src/doc/rustc-dev-guide/src/tracing.md b/src/doc/rustc-dev-guide/src/tracing.md index 5e5b81fc65b..a7cdab73e79 100644 --- a/src/doc/rustc-dev-guide/src/tracing.md +++ b/src/doc/rustc-dev-guide/src/tracing.md @@ -109,7 +109,7 @@ Miri, use `MIRI_LOG` instead. You get the idea :) See the [`tracing`] crate's docs, and specifically the docs for [`debug!`] to see the full syntax you can use. (Note: unlike the compiler, the [`tracing`] -crate and its examples use the `RUST_LOG` environment variable. rustc, rustdoc, +crate and its examples use the `RUSTC_LOG` environment variable. rustc, rustdoc, and other tools set custom environment variables.) **Note that unless you use a very strict filter, the logger will emit a lot of diff --git a/src/doc/rustc-dev-guide/src/traits/chalk.md b/src/doc/rustc-dev-guide/src/traits/chalk.md index 844f42b9879..ca5ed525917 100644 --- a/src/doc/rustc-dev-guide/src/traits/chalk.md +++ b/src/doc/rustc-dev-guide/src/traits/chalk.md @@ -5,7 +5,7 @@ Its goal is to enable a lot of trait system features and bug fixes that are hard to implement (e.g. GATs or specialization). If you would like to help in hacking on the new solver, drop by on the rust-lang Zulip in the [`#t-types`] -stream and say hello! +channel and say hello! [Types team]: https://github.com/rust-lang/types-team [`#t-types`]: https://rust-lang.zulipchat.com/#narrow/stream/144729-t-types diff --git a/src/doc/rustc-dev-guide/src/type-inference.md b/src/doc/rustc-dev-guide/src/type-inference.md index 2243205f129..24982a209fd 100644 --- a/src/doc/rustc-dev-guide/src/type-inference.md +++ b/src/doc/rustc-dev-guide/src/type-inference.md @@ -239,13 +239,13 @@ differently. It uses canonical queries for trait solving which use [`take_and_reset_region_constraints`] at the end. This extracts all of the outlives constraints added during the canonical query. This is required as the NLL solver must not only know *what* regions outlive each other, -but also *where*. Finally, the NLL solver invokes [`take_region_var_origins`], +but also *where*. Finally, the NLL solver invokes [`get_region_var_infos`], providing all region variables to the solver. -[`resolve_regions_and_report_errors`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_infer/infer/struct.InferCtxt.html#method.resolve_regions_and_report_errors +[`resolve_regions_and_report_errors`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_trait_selection/traits/struct.ObligationCtxt.html#method.resolve_regions_and_report_errors [`lexical_region_resolve`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_infer/infer/lexical_region_resolve/index.html [`take_and_reset_region_constraints`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_infer/infer/struct.InferCtxt.html#method.take_and_reset_region_constraints -[`take_region_var_origins`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_infer/infer/struct.InferCtxt.html#method.take_region_var_origins +[`get_region_var_infos`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_infer/infer/struct.InferCtxt.html#method.get_region_var_infos ## Lexical region resolution diff --git a/src/doc/rustc-dev-guide/src/typing_parameter_envs.md b/src/doc/rustc-dev-guide/src/typing_parameter_envs.md index db15467a47a..45635ebfa15 100644 --- a/src/doc/rustc-dev-guide/src/typing_parameter_envs.md +++ b/src/doc/rustc-dev-guide/src/typing_parameter_envs.md @@ -2,11 +2,14 @@ ## Typing Environments -When interacting with the type system there are a few variables to consider that can affect the results of trait solving. The set of in-scope where clauses, and what phase of the compiler type system operations are being performed in (the [`ParamEnv`][penv] and [`TypingMode`][tmode] structs respectively). +When interacting with the type system there are a few variables to consider that can affect the results of trait solving. +The set of in-scope where clauses, and what phase of the compiler type system operations are being performed in (the [`ParamEnv`][penv] and [`TypingMode`][tmode] structs respectively). -When an environment to perform type system operations in has not yet been created, the [`TypingEnv`][tenv] can be used to bundle all of the external context required into a single type. +When an environment to perform type system operations in has not yet been created, +the [`TypingEnv`][tenv] can be used to bundle all of the external context required into a single type. -Once a context to perform type system operations in has been created (e.g. an [`ObligationCtxt`][ocx] or [`FnCtxt`][fnctxt]) a `TypingEnv` is typically not stored anywhere as only the `TypingMode` is a property of the whole environment, whereas different `ParamEnv`s can be used on a per-goal basis. +Once a context to perform type system operations in has been created (e.g. an [`ObligationCtxt`][ocx] or [`FnCtxt`][fnctxt]) a `TypingEnv` is typically not stored anywhere as only the `TypingMode` is a property of the whole environment, +whereas different `ParamEnv`s can be used on a per-goal basis. [ocx]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_trait_selection/traits/struct.ObligationCtxt.html [fnctxt]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_hir_typeck/fn_ctxt/struct.FnCtxt.html @@ -15,9 +18,14 @@ Once a context to perform type system operations in has been created (e.g. an [` ### What is a `ParamEnv` -The [`ParamEnv`][penv] is a list of in-scope where-clauses, it typically corresponds to a specific item's where clauses. Some clauses are not explicitly written but are instead implicitly added in the [`predicates_of`][predicates_of] query, such as `ConstArgHasType` or (some) implied bounds. +The [`ParamEnv`][penv] is a list of in-scope where-clauses, +it typically corresponds to a specific item's where clauses. +Some clauses are not explicitly written but are instead implicitly added in the [`predicates_of`][predicates_of] query, +such as `ConstArgHasType` or (some) implied bounds. -In most cases `ParamEnv`s are initially created via the [`param_env` query][query] which returns a `ParamEnv` derived from the provided item's where clauses. A `ParamEnv` can also be created with arbitrary sets of clauses that are not derived from a specific item, such as in [`compare_method_predicate_entailment`][method_pred_entailment] where we create a hybrid `ParamEnv` consisting of the impl's where clauses and the trait definition's function's where clauses. +In most cases `ParamEnv`s are initially created via the [`param_env` query][query] which returns a `ParamEnv` derived from the provided item's where clauses. +A `ParamEnv` can also be created with arbitrary sets of clauses that are not derived from a specific item, +such as in [`compare_method_predicate_entailment`][method_pred_entailment] where we create a hybrid `ParamEnv` consisting of the impl's where clauses and the trait definition's function's where clauses. --- @@ -30,7 +38,9 @@ where <T as Trait>::Assoc: Clone, {} ``` -If we were conceptually inside of `foo` (for example, type-checking or linting it) we would use this `ParamEnv` everywhere that we interact with the type system. This would allow things such as [normalization], evaluating generic constants, and proving where clauses/goals, to rely on `T` being sized, implementing `Trait`, etc. +If we were conceptually inside of `foo` (for example, type-checking or linting it) we would use this `ParamEnv` everywhere that we interact with the type system. +This would allow things such as [normalization], evaluating generic constants, +and proving where clauses/goals, to rely on `T` being sized, implementing `Trait`, etc. A more concrete example: ```rust @@ -72,9 +82,13 @@ fn foo2<T>(a: T) { ### Acquiring a `ParamEnv` -Using the wrong [`ParamEnv`][penv] when interacting with the type system can lead to ICEs, illformed programs compiling, or erroring when we shouldn't. See [#82159](https://github.com/rust-lang/rust/pull/82159) and [#82067](https://github.com/rust-lang/rust/pull/82067) as examples of PRs that modified the compiler to use the correct param env and in the process fixed ICEs. +Using the wrong [`ParamEnv`][penv] when interacting with the type system can lead to ICEs, +illformed programs compiling, or erroring when we shouldn't. +See [#82159](https://github.com/rust-lang/rust/pull/82159) and [#82067](https://github.com/rust-lang/rust/pull/82067) as examples of PRs that modified the compiler to use the correct param env and in the process fixed ICEs. -In the large majority of cases, when a `ParamEnv` is required it either already exists somewhere in scope, or above in the call stack and should be passed down. A non exhaustive list of places where you might find an existing `ParamEnv`: +In the large majority of cases, when a `ParamEnv` is required it either already exists somewhere in scope, +or above in the call stack and should be passed down. +A non exhaustive list of places where you might find an existing `ParamEnv`: - During typeck `FnCtxt` has a [`param_env` field][fnctxt_param_env] - When writing late lints the `LateContext` has a [`param_env` field][latectxt_param_env] - During well formedness checking the `WfCheckingCtxt` has a [`param_env` field][wfckctxt_param_env] @@ -82,16 +96,20 @@ In the large majority of cases, when a `ParamEnv` is required it either already - In the next-gen trait solver all `Goal`s have a [`param_env` field][goal_param_env] specifying what environment to prove the goal in - When editing an existing [`TypeRelation`][typerelation] if it implements [`PredicateEmittingRelation`][predicate_emitting_relation] then a [`param_env` method][typerelation_param_env] will be available. -If you aren't sure if there's a `ParamEnv` in scope somewhere that can be used it can be worth opening a thread in the [`#t-compiler/help`][compiler_help] zulip stream where someone may be able to point out where a `ParamEnv` can be acquired from. +If you aren't sure if there's a `ParamEnv` in scope somewhere that can be used it can be worth opening a thread in the [`#t-compiler/help`][compiler_help] Zulip channel where someone may be able to point out where a `ParamEnv` can be acquired from. -Manually constructing a `ParamEnv` is typically only needed at the start of some kind of top level analysis (e.g. hir typeck or borrow checking). In such cases there are three ways it can be done: +Manually constructing a `ParamEnv` is typically only needed at the start of some kind of top level analysis (e.g. hir typeck or borrow checking). +In such cases there are three ways it can be done: - Calling the [`tcx.param_env(def_id)` query][param_env_query] which returns the environment associated with a given definition. - Creating an empty environment with [`ParamEnv::empty`][env_empty]. -- Using [`ParamEnv::new`][param_env_new] to construct an env with an arbitrary set of where clauses. Then calling [`traits::normalize_param_env_or_error`][normalize_env_or_error] to handle normalizing and elaborating all the where clauses in the env. +- Using [`ParamEnv::new`][param_env_new] to construct an env with an arbitrary set of where clauses. + Then calling [`traits::normalize_param_env_or_error`][normalize_env_or_error] to handle normalizing and elaborating all the where clauses in the env. Using the `param_env` query is by far the most common way to construct a `ParamEnv` as most of the time the compiler is performing an analysis as part of some specific definition. -Creating an empty environment with `ParamEnv::empty` is typically only done either in codegen (indirectly via [`TypingEnv::fully_monomorphized`][tenv_mono]), or as part of some analysis that do not expect to ever encounter generic parameters (e.g. various parts of coherence/orphan check). +Creating an empty environment with `ParamEnv::empty` is typically only done either in codegen (indirectly via [`TypingEnv::fully_monomorphized`][tenv_mono]), +or as part of some analysis that do not expect to ever encounter generic parameters +(e.g. various parts of coherence/orphan check). Creating an env from an arbitrary set of where clauses is usually unnecessary and should only be done if the environment you need does not correspond to an actual item in the source code (e.g. [`compare_method_predicate_entailment`][method_pred_entailment]). @@ -113,11 +131,14 @@ Creating an env from an arbitrary set of where clauses is usually unnecessary an ### How are `ParamEnv`s constructed -Creating a [`ParamEnv`][pe] is more complicated than simply using the list of where clauses defined on an item as written by the user. We need to both elaborate supertraits into the env and fully normalize all aliases. This logic is handled by [`traits::normalize_param_env_or_error`][normalize_env_or_error] (even though it does not mention anything about elaboration). +Creating a [`ParamEnv`][pe] is more complicated than simply using the list of where clauses defined on an item as written by the user. +We need to both elaborate supertraits into the env and fully normalize all aliases. +This logic is handled by [`traits::normalize_param_env_or_error`][normalize_env_or_error] (even though it does not mention anything about elaboration). #### Elaborating supertraits -When we have a function such as `fn foo<T: Copy>()` we would like to be able to prove `T: Clone` inside of the function as the `Copy` trait has a `Clone` supertrait. Constructing a `ParamEnv` looks at all of the trait bounds in the env and explicitly adds new where clauses to the `ParamEnv` for any supertraits found on the traits. +When we have a function such as `fn foo<T: Copy>()` we would like to be able to prove `T: Clone` inside of the function as the `Copy` trait has a `Clone` supertrait. +Constructing a `ParamEnv` looks at all of the trait bounds in the env and explicitly adds new where clauses to the `ParamEnv` for any supertraits found on the traits. A concrete example would be the following function: ```rust @@ -133,13 +154,16 @@ fn bar<T: Copy + Trait>(a: T) { fn requires_impl<T: Clone + SuperSuperTrait>(a: T) {} ``` -If we did not elaborate the env then the `requires_impl` call would fail to typecheck as we would not be able to prove `T: Clone` or `T: SuperSuperTrait`. In practice we elaborate the env which means that `bar`'s `ParamEnv` is actually: +If we did not elaborate the env then the `requires_impl` call would fail to typecheck as we would not be able to prove `T: Clone` or `T: SuperSuperTrait`. +In practice we elaborate the env which means that `bar`'s `ParamEnv` is actually: `[T: Sized, T: Copy, T: Clone, T: Trait, T: SuperTrait, T: SuperSuperTrait]` This allows us to prove `T: Clone` and `T: SuperSuperTrait` when type checking `bar`. The `Clone` trait has a `Sized` supertrait however we do not end up with two `T: Sized` bounds in the env (one for the supertrait and one for the implicitly added `T: Sized` bound) as the elaboration process (implemented via [`util::elaborate`][elaborate]) deduplicates where clauses. -A side effect of this is that even if no actual elaboration of supertraits takes place, the existing where clauses in the env are _also_ deduplicated. See the following example: +A side effect of this is that even if no actual elaboration of supertraits takes place, +the existing where clauses in the env are _also_ deduplicated. +See the following example: ```rust trait Trait {} // The unelaborated `ParamEnv` would be: @@ -156,7 +180,8 @@ The [next-gen trait solver][next-gen-solver] also requires this elaboration to t #### Normalizing all bounds -In the old trait solver the where clauses stored in `ParamEnv` are required to be fully normalized as otherwise the trait solver will not function correctly. A concrete example of needing to normalize the `ParamEnv` is the following: +In the old trait solver the where clauses stored in `ParamEnv` are required to be fully normalized as otherwise the trait solver will not function correctly. +A concrete example of needing to normalize the `ParamEnv` is the following: ```rust trait Trait<T> { type Assoc; @@ -182,11 +207,14 @@ where fn requires_impl<U: Trait<u32>>(_: U) {} ``` -As humans we can tell that `<T as Other>::Bar` is equal to `u32` so the trait bound on `U` is equivalent to `U: Trait<u32>`. In practice trying to prove `U: Trait<u32>` in the old solver in this environment would fail as it is unable to determine that `<T as Other>::Bar` is equal to `u32`. +As humans we can tell that `<T as Other>::Bar` is equal to `u32` so the trait bound on `U` is equivalent to `U: Trait<u32>`. +In practice trying to prove `U: Trait<u32>` in the old solver in this environment would fail as it is unable to determine that `<T as Other>::Bar` is equal to `u32`. To work around this we normalize `ParamEnv`'s after constructing them so that `foo`'s `ParamEnv` is actually: `[T: Sized, U: Sized, U: Trait<u32>]` which means the trait solver is now able to use the `U: Trait<u32>` in the `ParamEnv` to determine that the trait bound `U: Trait<u32>` holds. -This workaround does not work in all cases as normalizing associated types requires a `ParamEnv` which introduces a bootstrapping problem. We need a normalized `ParamEnv` in order for normalization to give correct results, but we need to normalize to get that `ParamEnv`. Currently we normalize the `ParamEnv` once using the unnormalized param env and it tends to give okay results in practice even though there are some examples where this breaks ([example]). +This workaround does not work in all cases as normalizing associated types requires a `ParamEnv` which introduces a bootstrapping problem. +We need a normalized `ParamEnv` in order for normalization to give correct results, but we need to normalize to get that `ParamEnv`. +Currently we normalize the `ParamEnv` once using the unnormalized param env and it tends to give okay results in practice even though there are some examples where this breaks ([example]). In the next-gen trait solver the requirement for all where clauses in the `ParamEnv` to be fully normalized is not present and so we do not normalize when constructing `ParamEnv`s. @@ -196,9 +224,12 @@ In the next-gen trait solver the requirement for all where clauses in the `Param ## Typing Modes -Depending on what context we are performing type system operations in, different behaviour may be required. For example during coherence there are stronger requirements about when we can consider goals to not hold or when we can consider types to be unequal. +Depending on what context we are performing type system operations in, +different behaviour may be required. +For example during coherence there are stronger requirements about when we can consider goals to not hold or when we can consider types to be unequal. -Tracking which "phase" of the compiler type system operations are being performed in is done by the [`TypingMode`][tmode] enum. The documentation on the `TypingMode` enum is quite good so instead of repeating it here verbatim we would recommend reading the API documentation directly. +Tracking which "phase" of the compiler type system operations are being performed in is done by the [`TypingMode`][tmode] enum. +The documentation on the `TypingMode` enum is quite good so instead of repeating it here verbatim we would recommend reading the API documentation directly. [penv]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_middle/ty/struct.ParamEnv.html [tenv]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_middle/ty/struct.TypingEnv.html diff --git a/src/doc/rustc-dev-guide/triagebot.toml b/src/doc/rustc-dev-guide/triagebot.toml index 3ac5d57a52b..4a885239152 100644 --- a/src/doc/rustc-dev-guide/triagebot.toml +++ b/src/doc/rustc-dev-guide/triagebot.toml @@ -90,7 +90,6 @@ does not need reviews. You can request a review using `r? rustc-dev-guide` or \ [assign.adhoc_groups] rustc-dev-guide = [ "@BoxyUwU", - "@jieyouxu", "@jyn514", "@tshepang", ] |
