| Age | Commit message (Collapse) | Author | Lines |
|---|
|
If the `LocalRef` is `LocalRef::Place`, we can refer to it directly,
because the local of place is an indirect pointer.
Such a statement is `_1 = &(_2.1)`.
If the `LocalRef` is `LocalRef::Operand`,
the `OperandRef` should provide the pointer of the reference.
Such a statement is `_1 = &((*_2).1)`.
But there is a special case that hasn't been handled, scalar pairs like `(&[i32; 16], i32)`.
|
|
|
|
|
|
|
|
Turn ProjectionElem::Subtype into CastKind::Subtype
I noticed that drop elaboration can't, in general, handle `ProjectionElem::SubType`. It creates a disjoint move path that overlaps with other move paths. (`Subslice` does too, and I'm working on a different PR to make that special case less fragile.) If its skipped and treated as the same move path as its parent then `MovePath.place` has multiple possible projections. (It would probably make sense to remove all `Subtype` projections for the canonical place but it doesn't make sense to have this special case for a problem that doesn't actually occur in real MIR.)
The only reason this doesn't break is that `Subtype` is always the sole projection of the local its applied to. For the same reason, it works fine as a `CastKind` so I figured that makes more sense than documenting and validating this hidden invariant.
cc rust-lang/rust#112651, rust-lang/rust#133258
r? Icnr (bc you've been the main person dealing with `Subtype` it looks like)
|
|
Much of the compiler calls functions on Align projected from AbiAlign.
AbiAlign impls Deref to its inner Align, so we can simplify these away.
Also, it will minimize disruption when AbiAlign is removed.
For now, preserve usages that might resolve to PartialOrd or PartialEq,
as those have odd inference.
|
|
TypeTree support in autodiff
# TypeTrees for Autodiff
## What are TypeTrees?
Memory layout descriptors for Enzyme. Tell Enzyme exactly how types are structured in memory so it can compute derivatives efficiently.
## Structure
```rust
TypeTree(Vec<Type>)
Type {
offset: isize, // byte offset (-1 = everywhere)
size: usize, // size in bytes
kind: Kind, // Float, Integer, Pointer, etc.
child: TypeTree // nested structure
}
```
## Example: `fn compute(x: &f32, data: &[f32]) -> f32`
**Input 0: `x: &f32`**
```rust
TypeTree(vec![Type {
offset: -1, size: 8, kind: Pointer,
child: TypeTree(vec![Type {
offset: -1, size: 4, kind: Float,
child: TypeTree::new()
}])
}])
```
**Input 1: `data: &[f32]`**
```rust
TypeTree(vec![Type {
offset: -1, size: 8, kind: Pointer,
child: TypeTree(vec![Type {
offset: -1, size: 4, kind: Float, // -1 = all elements
child: TypeTree::new()
}])
}])
```
**Output: `f32`**
```rust
TypeTree(vec![Type {
offset: -1, size: 4, kind: Float,
child: TypeTree::new()
}])
```
## Why Needed?
- Enzyme can't deduce complex type layouts from LLVM IR
- Prevents slow memory pattern analysis
- Enables correct derivative computation for nested structures
- Tells Enzyme which bytes are differentiable vs metadata
## What Enzyme Does With This Information:
Without TypeTrees (current state):
```llvm
; Enzyme sees generic LLVM IR:
define float ``@distance(ptr*`` %p1, ptr* %p2) {
; Has to guess what these pointers point to
; Slow analysis of all memory operations
; May miss optimization opportunities
}
```
With TypeTrees (our implementation):
```llvm
define "enzyme_type"="{[]:Float@float}" float ``@distance(``
ptr "enzyme_type"="{[]:Pointer}" %p1,
ptr "enzyme_type"="{[]:Pointer}" %p2
) {
; Enzyme knows exact type layout
; Can generate efficient derivative code directly
}
```
# TypeTrees - Offset and -1 Explained
## Type Structure
```rust
Type {
offset: isize, // WHERE this type starts
size: usize, // HOW BIG this type is
kind: Kind, // WHAT KIND of data (Float, Int, Pointer)
child: TypeTree // WHAT'S INSIDE (for pointers/containers)
}
```
## Offset Values
### Regular Offset (0, 4, 8, etc.)
**Specific byte position within a structure**
```rust
struct Point {
x: f32, // offset 0, size 4
y: f32, // offset 4, size 4
id: i32, // offset 8, size 4
}
```
TypeTree for `&Point` (internal representation):
```rust
TypeTree(vec![
Type { offset: 0, size: 4, kind: Float }, // x at byte 0
Type { offset: 4, size: 4, kind: Float }, // y at byte 4
Type { offset: 8, size: 4, kind: Integer } // id at byte 8
])
```
Generates LLVM:
```llvm
"enzyme_type"="{[]:Float@float}"
```
### Offset -1 (Special: "Everywhere")
**Means "this pattern repeats for ALL elements"**
#### Example 1: Array `[f32; 100]`
```rust
TypeTree(vec![Type {
offset: -1, // ALL positions
size: 4, // each f32 is 4 bytes
kind: Float, // every element is float
}])
```
Instead of listing 100 separate Types with offsets `0,4,8,12...396`
#### Example 2: Slice `&[i32]`
```rust
// Pointer to slice data
TypeTree(vec![Type {
offset: -1, size: 8, kind: Pointer,
child: TypeTree(vec![Type {
offset: -1, // ALL slice elements
size: 4, // each i32 is 4 bytes
kind: Integer
}])
}])
```
#### Example 3: Mixed Structure
```rust
struct Container {
header: i64, // offset 0
data: [f32; 1000], // offset 8, but elements use -1
}
```
```rust
TypeTree(vec![
Type { offset: 0, size: 8, kind: Integer }, // header
Type { offset: 8, size: 4000, kind: Pointer,
child: TypeTree(vec![Type {
offset: -1, size: 4, kind: Float // ALL array elements
}])
}
])
```
|
|
|
|
naked_asm: emit a label starting with `func_end`
The `cargo asm` tool (`cargo install cargo-show-asm`) pattern matches on such labels to figure out where functions end: normal functions generated by LLVM always do have such a label. We don't guarantee that naked functions emit such a label, but having `cargo asm` work is convenient.
https://github.com/pacak/cargo-show-asm/blob/be45f67454ad8b634246a7fc69b3c6a963ee93f1/src/asm/statements.rs#L897-L901
To make the label name unique it's suffixed with the name of the current symbol.
r? ```@Amanieu```
|
|
The `cargo asm` tool pattern matches on such labels to figure out where functions end: normal functions generated by LLVM always do have such a label. We don't guarantee that naked functions emit such a label, but having `cargo asm` work is convenient
|
|
|
|
Clean up `ty::Dynamic`
1. As a follow-up to PR rust-lang/rust#143036, remove `DynKind` entirely.
2. Inside HIR ty lowering, consolidate modules `dyn_compatibility` and `lint` into `dyn_trait`
* `dyn_compatibility` wasn't about dyn compatibility itself, it's about lowering trait object types
* `lint` contained dyn-Trait-specific diagnostics+lints only
|
|
Remove Rvalue::Len again.
Now that we have `RawPtrKind::FakeForPtrMetadata`, we can reimplement `Rvalue::Len` using `PtrMetadata(&raw const (fake) place)`.
r? ``@scottmcm``
|
|
|
|
|
|
|
|
interpret: fix overlapping aggregate initialization
This fixes the problem pointed out by ````@saethlin```` in https://github.com/rust-lang/rust/issues/146383#issuecomment-3273224645.
Also clarify when exactly current de-facto MIR semantics allow overlap of the LHS and RHS in an assignment.
|
|
match clang's `va_arg` assembly on arm targets
tracking issue: https://github.com/rust-lang/rust/issues/44930
For this example
```rust
#![feature(c_variadic)]
#[unsafe(no_mangle)]
unsafe extern "C" fn variadic(a: f64, mut args: ...) -> f64 {
let b = args.arg::<f64>();
let c = args.arg::<f64>();
a + b + c
}
```
We currently generate (via llvm):
```asm
variadic:
sub sp, sp, #12
stmib sp, {r2, r3}
vmov d0, r0, r1
add r0, sp, #4
vldr d1, [sp, #4]
add r0, r0, #15
bic r0, r0, #7
vadd.f64 d0, d0, d1
add r1, r0, #8
str r1, [sp]
vldr d1, [r0]
vadd.f64 d0, d0, d1
vmov r0, r1, d0
add sp, sp, #12
bx lr
```
LLVM is not doing a good job. In fact, it's well-known that LLVM's implementation of `va_arg` is kind of bad, and we implement it ourselves (based on clang) for many targets already. For arm, our own `emit_ptr_va_arg` saves 3 instructions.
Next, it turns out it's important for LLVM to explicitly start and end the lifetime of the `va_list`. In https://github.com/rust-lang/rust/pull/146059 I already end the lifetime, but when looking at this again, I noticed that it is important to also start it, see https://godbolt.org/z/EGqvKTTsK: failing to explicitly start the lifetime uses an extra register.
So, the combination of `emit_ptr_va_arg` with starting/ending the lifetime makes rustc emit exactly the instructions that clang generates::
```asm
variadic:
sub sp, sp, #12
stmib sp, {r2, r3}
vmov d16, r0, r1
vldr d17, [sp, #4]
vadd.f64 d16, d16, d17
vldr d17, [sp, #12]
vadd.f64 d16, d16, d17
vmov r0, r1, d16
add sp, sp, #12
bx lr
```
The arguments to `emit_ptr_va_arg` are based on [the clang implementation](https://github.com/llvm/llvm-project/blob/03dc2a41f3d9a500e47b513de5c5008c06860d65/clang/lib/CodeGen/Targets/ARM.cpp#L798-L844).
r? ``@workingjubilee`` (I can re-roll if your queue is too full, but you do seem like the right person here)
try-job: armhf-gnu
|
|
and double-check that we match it in codegen
|
|
|
|
|
|
|
|
Fix tail calls to `#[track_caller]` functions
We want `#[track_caller]` to be semver independent, i.e. it should not be a breaking change to add or remove it. Since it changes ABI of a function (adding an additional argument) we have to be careful to preserve this property when adding tail calls.
The only way to achieve this that I can see is:
- we forbid tail calls in functions which are marked with `#[track_caller]` (already implemented)
- tail-calling a `#[track_caller]` marked function downgrades the tail-call to a normal call (or equivalently tail-calls the shim made by fn def to fn ptr cast) (this pr)
Ideally the downgrade would be performed by a MIR pass, but that requires post mono MIR opts (cc ```@saethlin,``` rust-lang/rust#131650). For now I've changed code in cg_ssa to accomodate this behaviour (+ added a hack to mono collector so that the shim is actually generated)
Additionally I added a lint, although I don't think it's strictly necessary.
Alternative to rust-lang/rust#144762 (and thus closes rust-lang/rust#144762)
Fixes https://github.com/rust-lang/rust/issues/144755
|
|
|
|
|
|
|
|
atomicrmw on pointers: move integer-pointer cast hacks into backend
Conceptually, we want to have atomic operations on pointers of the form `fn atomic_add(ptr: *mut T, offset: usize, ...)`. However, LLVM does not directly support such operations (https://github.com/llvm/llvm-project/issues/120837), so we have to cast the `offset` to a pointer somewhere.
This PR moves that hack into the LLVM backend, so that the standard library, intrinsic, and Miri all work with the conceptual operation we actually want. Hopefully, one day LLVM will gain a way to represent these operations without integer-pointer casts, and then the hack will disappear entirely.
Cc ```@nikic``` -- this is the best we can do right now, right?
Fixes https://github.com/rust-lang/rust/issues/134617
|
|
coverage: Remove all unstable support for MC/DC instrumentation
Preliminary support for a partial implementation of “Modified Condition/Decision Coverage” instrumentation was added behind the unstable flag `-Zcoverage-options=mcdc` in 2024. These are the most substantial PRs involved:
- rust-lang/rust#123409
- rust-lang/rust#126733
At the time, I accepted these PRs with relatively modest scrutiny, because I did not want to stand in the way of independent work on MC/DC instrumentation. My hope was that ongoing work by interested contributors would lead to the code becoming clearer and more maintainable over time.
---
However, that MC/DC code has proven itself to be a major burden on overall maintenance of coverage instrumentation, and a major obstacle to other planned improvements, such as internal changes needed for proper support of macro expansion regions.
I have also become reluctant to accept any further MC/DC-related changes that would increase this burden.
That tension has resulted in an unhappy impasse. On one hand, the present MC/DC implementation is not yet complete, and shows little sign of being complete at an acceptable level of code quality in the foreseeable future. On the other hand, the continued existence of this partial MC/DC implementation is imposing serious maintenance burdens on every other aspect of coverage instrumentation, and is preventing some of the very improvements that would make it easier to accept expanded MC/DC support in the future.
While I know this will be disappointing to some, I think the healthy way forward is accept that I made the wrong call in accepting the current implementation, and to remove it entirely from the compiler.
|
|
|
|
|
|
|
|
Implement support for `become` and explicit tail call codegen for the LLVM backend
This PR implements codegen of explicit tail calls via `become` in `rustc_codegen_ssa` and support within the LLVM backend. Completes a task on (https://github.com/rust-lang/rust/issues/112788). This PR implements all the necessary bits to make explicit tail calls usable, other backends have received stubs for now and will ICE if you use `become` on them. I suspect there is some bikeshedding to be done on how we should go about implementing this for other backends, but it should be relatively straightforward for GCC after this is merged.
During development I also put together a POC bytecode VM based on tail call dispatch to test these changes out and analyze the codegen to make sure it generates expected assembly. That is available [here](https://github.com/xacrimon/tcvm).
|
|
No longer need `alloca`s for consuming `Result<!, i32>` and similar
In optimized builds GVN gets rid of these already, but in `opt-level=0` we actually make `alloca`s for this, which particularly impacts `?`-style things that use actually-only-one-variant types like this.
While doing so, rewrite `LocalAnalyzer::process_place` to be non-recursive, solving a 6+ year old FIXME.
r? codegen
|
|
Let `codegen_transmute_operand` just handle everything
When combined with rust-lang/rust#143720, this means `rvalue_creates_operand` can just return `true` for *every* `Rvalue`. (A future PR could consider removing it, though just letting it optimize out is fine for now.)
It's nicer anyway, IMHO, because it avoids needing the layout checks to be consistent in the two places, and thus is an overall reduction in code. Plus it's a more helpful building block when used in other places this way.
(TBH, it probably would have been better to have it this way the whole time, but I clearly didn't understand `rvalue_creates_operand` when I originally wrote rust-lang/rust#109843.)
|
|
LLVM codegen backend.
|
|
Don't emit two `assume`s in transmutes when one is a subset of the other
For example, transmuting between `bool` and `Ordering` doesn't need two `assume`s because one range is a superset of the other.
Multiple are still used for things like `char` <-> `NonZero<u32>`, which overlap but where neither fully contains the other.
|
|
|
|
|
|
For example, transmuting between `bool` and `Ordering` doesn't need two `assume`s because one range is a superset of the other.
Multiple are still used for things like `char` <-> `NonZero<u32>`, which overlap but where neither fully contains the other.
|
|
Split to a separate commit to it could be reverted later if necessary, should we get new `Rvalue`s where we can't handle it this way.
|
|
involving pointers)
|
|
When combined with 143720, this means `rvalue_creates_operand` can just return `true` for *every* `Rvalue`. (A future PR could consider removing it, though just letting it optimize out is fine for now.)
It's nicer anyway, IMHO, because it avoids needing the layout checks to be consistent in the two places, and thus is an overall reduction in code. Plus it's a more helpful building block when used in other places this way.
|
|
In optimized builds GVN gets rid of these already, but in `opt-level=0` we actually make `alloca`s for this, which particularly impacts `?`-style things that use actually-only-one-variant types like this.
|
|
|
|
|
|
|
|
The conversation in 143502 made be realize how easy this is to handle, since the only possibilty is ZSTs -- everything else ends up with the destination being `LocalKind::Memory` and thus doesn't call `codegen_rvalue_operand` at all.
This gets us perilously close to a world where `rvalue_creates_operand` only ever returns true. I'll try out such a world next :)
|
|
Simplify discriminant codegen for niche-encoded variants which don't wrap across an integer boundary
Inspired by rust-lang/rust#139729, this attempts to be a much-simpler and more-localized change while still making a difference. (Specifically, this does not try to solve the problem with select-sinking, leaving that to be fixed by https://github.com/llvm/llvm-project/issues/134024 -- once it gets released -- instead of in rustc's codegen.)
What this *does* improve is checking for the variant in a 3+ variant enum when that variant is the type providing the niche. Something like `if let Foo::WithBool(_) = ...` previously compiled to `ugt(add(x, -2), 2)`, which is non-trivial to think about because it's depending on the unsigned wrapping to shift the 0/1 up above 2. With this PR it compiles to just `ult(x, 2)`, which is probably what you'd have written yourself if you were doing it by hand to look for "is this byte a bool?".
That's done by leaving most of the codegen alone, but adding a couple new special cases to the `is_niche` check. The default looks at the relative discriminant, but in the common cases where there's no wraparound involved, we can just check the original value, rather than the offsetted one.
The first commit just adds some tests, so the best way to see the effect of this change is to look at the second commit and how it updates the test expectations.
|
|
|
|
and use it for naked functions
|
