| Age | Commit message (Collapse) | Author | Lines |
|---|
|
Fix autodiff empty ret regression
closes https://github.com/rust-lang/rust/issues/147144
The two gsoc summer projects caused a bit of churn, which was to be expected, especially since we don't run autodiff in CI yet.
This adds a void return testcase that we should have had anyway, and fixes the regression.
r? `@Zalathar` (Just guessing since I've seen you in a few LLVM PRs and Oli is probably still busy. Feel free to reroll!)
|
|
|
|
|
|
Skip cleanups on unsupported targets
This commit is an update to the `AbortUnwindingCalls` MIR pass in the compiler. Specifically a new boolean is added for "can this target possibly unwind" and if that's `false` then terminators are all adjusted to be unreachable/not present. The end result is that this fixes rust-lang/rust#140293 for wasm targets.
The motivation for this PR is that currently on WebAssembly targets the usage of the `C-unwind` ABI can lead LLVM to either (a) emit exception-handling instructions or (b) hit a LLVM-ICE-style codegen error. WebAssembly as a base instruction set does not support unwinding at all, and a later proposal to WebAssembly, the exception-handling proposal, was what enabled this. This means that the current intent of WebAssembly targets is that they maintain the baseline of "don't emit exception-handling instructions unless enabled". The commit here is intended to restore this behavior by skipping these instructions even when `C-unwind` is present.
Exception-handling is a relatively tricky and also murky topic in WebAssembly, however. There are two sets of instructions LLVM can emit for WebAssembly exceptions, Rust's Emscripten target supports exceptions, WASI targets do not, the LLVM flags to enable this are not always obvious, and additionally this all touches on "changing exception-handling behavior should be a target-level concern, not a feature". Effectively WebAssembly's exception-handling integration into Rust is not finalized at this time. The best idea at this time is that a parallel set of targets will eventually be added which support exceptions, but it's not clear if/when to do this. In the meantime the goal is to keep existing targets working while still enabling experimentation with exception-handling with `-Zbuild-std` and various permutations of LLVM flags.
To that extent this commit does not blanket disable these landing pads and cleanup routines for WebAssembly but instead checks to see if panic=unwind is enabled or if `+exception-handling` is enabled. Tests are updated here as well to account for this where, by default, using a `C-unwind` ABI won't affect Rust codegen at all. If `+exception-handling` is enabled, however, then Rust codegen will look like native platforms where exceptions are caught and the program aborts. More-or-less I've done my best to keep exceptions working on wasm where it's possible to have them work, but turned them off where they're not supposed to be emitted.
Closes rust-lang/rust#140293
|
|
TypeTree support in autodiff
# TypeTrees for Autodiff
## What are TypeTrees?
Memory layout descriptors for Enzyme. Tell Enzyme exactly how types are structured in memory so it can compute derivatives efficiently.
## Structure
```rust
TypeTree(Vec<Type>)
Type {
offset: isize, // byte offset (-1 = everywhere)
size: usize, // size in bytes
kind: Kind, // Float, Integer, Pointer, etc.
child: TypeTree // nested structure
}
```
## Example: `fn compute(x: &f32, data: &[f32]) -> f32`
**Input 0: `x: &f32`**
```rust
TypeTree(vec![Type {
offset: -1, size: 8, kind: Pointer,
child: TypeTree(vec![Type {
offset: -1, size: 4, kind: Float,
child: TypeTree::new()
}])
}])
```
**Input 1: `data: &[f32]`**
```rust
TypeTree(vec![Type {
offset: -1, size: 8, kind: Pointer,
child: TypeTree(vec![Type {
offset: -1, size: 4, kind: Float, // -1 = all elements
child: TypeTree::new()
}])
}])
```
**Output: `f32`**
```rust
TypeTree(vec![Type {
offset: -1, size: 4, kind: Float,
child: TypeTree::new()
}])
```
## Why Needed?
- Enzyme can't deduce complex type layouts from LLVM IR
- Prevents slow memory pattern analysis
- Enables correct derivative computation for nested structures
- Tells Enzyme which bytes are differentiable vs metadata
## What Enzyme Does With This Information:
Without TypeTrees (current state):
```llvm
; Enzyme sees generic LLVM IR:
define float ``@distance(ptr*`` %p1, ptr* %p2) {
; Has to guess what these pointers point to
; Slow analysis of all memory operations
; May miss optimization opportunities
}
```
With TypeTrees (our implementation):
```llvm
define "enzyme_type"="{[]:Float@float}" float ``@distance(``
ptr "enzyme_type"="{[]:Pointer}" %p1,
ptr "enzyme_type"="{[]:Pointer}" %p2
) {
; Enzyme knows exact type layout
; Can generate efficient derivative code directly
}
```
# TypeTrees - Offset and -1 Explained
## Type Structure
```rust
Type {
offset: isize, // WHERE this type starts
size: usize, // HOW BIG this type is
kind: Kind, // WHAT KIND of data (Float, Int, Pointer)
child: TypeTree // WHAT'S INSIDE (for pointers/containers)
}
```
## Offset Values
### Regular Offset (0, 4, 8, etc.)
**Specific byte position within a structure**
```rust
struct Point {
x: f32, // offset 0, size 4
y: f32, // offset 4, size 4
id: i32, // offset 8, size 4
}
```
TypeTree for `&Point` (internal representation):
```rust
TypeTree(vec![
Type { offset: 0, size: 4, kind: Float }, // x at byte 0
Type { offset: 4, size: 4, kind: Float }, // y at byte 4
Type { offset: 8, size: 4, kind: Integer } // id at byte 8
])
```
Generates LLVM:
```llvm
"enzyme_type"="{[]:Float@float}"
```
### Offset -1 (Special: "Everywhere")
**Means "this pattern repeats for ALL elements"**
#### Example 1: Array `[f32; 100]`
```rust
TypeTree(vec![Type {
offset: -1, // ALL positions
size: 4, // each f32 is 4 bytes
kind: Float, // every element is float
}])
```
Instead of listing 100 separate Types with offsets `0,4,8,12...396`
#### Example 2: Slice `&[i32]`
```rust
// Pointer to slice data
TypeTree(vec![Type {
offset: -1, size: 8, kind: Pointer,
child: TypeTree(vec![Type {
offset: -1, // ALL slice elements
size: 4, // each i32 is 4 bytes
kind: Integer
}])
}])
```
#### Example 3: Mixed Structure
```rust
struct Container {
header: i64, // offset 0
data: [f32; 1000], // offset 8, but elements use -1
}
```
```rust
TypeTree(vec![
Type { offset: 0, size: 8, kind: Integer }, // header
Type { offset: 8, size: 4000, kind: Pointer,
child: TypeTree(vec![Type {
offset: -1, size: 4, kind: Float // ALL array elements
}])
}
])
```
|
|
tests: relax expectations after llvm change 902ddda120a5
LLVM 22 is able to drop assumes that seem to not help further optimizations, which actually seems to dramatically _help_ further optimizations in some of our small test cases.
I'm a little unclear how to fix the last failure, in `tests/codegen-llvm/issues/issue-122600-ptr-discriminant-update.rs`:
```
-; Function Attrs: mustprogress nofree norecurse nosync nounwind willreturn memory(argmem: readwrite, inaccessiblemem: write) uwtable
+; Function Attrs: mustprogress nofree norecurse nosync nounwind nonlazybind willreturn memory(argmem: readwrite, inaccessiblemem: write) uwtable
define void ``@update(ptr`` noundef captures(none) %s) unnamed_addr #0 {
start:
- %_3.sroa.0.0.copyload = load i8, ptr %s, align 1
- %0 = trunc nuw i8 %_3.sroa.0.0.copyload to i1
- %1 = xor i1 %0, true
- tail call void ``@llvm.assume(i1`` %1)
store i8 1, ptr %s, align 1
ret void
}
```
I'm just not conversant enough in LLVM IR to follow the changes here.
``@rustbot`` label llvm-main
r? nikic
|
|
|
|
Rollup of 8 pull requests
Successful merges:
- rust-lang/rust#116882 (rustdoc: hide `#[repr]` if it isn't part of the public ABI)
- rust-lang/rust#135771 ([rustdoc] Add support for associated items in "jump to def" feature)
- rust-lang/rust#141032 (avoid violating `slice::from_raw_parts` safety contract in `Vec::extract_if`)
- rust-lang/rust#142401 (Add proper name mangling for pattern types)
- rust-lang/rust#146293 (feat: non-panicking `Vec::try_remove`)
- rust-lang/rust#146859 (BTreeMap: Don't leak allocators when initializing nodes)
- rust-lang/rust#146924 (Add doc for `NonZero*` const creation)
- rust-lang/rust#146933 (Make `render_example_with_highlighting` return an `impl fmt::Display`)
r? `@ghost`
`@rustbot` modify labels: rollup
|
|
Add proper name mangling for pattern types
requires adding demangler support first https://github.com/rust-lang/rustc-demangle/pull/81
needed for https://github.com/rust-lang/rust/pull/136006#discussion_r2139792593 as otherwise we will have symbol collisions
|
|
Add attributes for #[global_allocator] functions
Emit `#[rustc_allocator]` etc. attributes on the functions generated by the `#[global_allocator]` macro, which will emit LLVM attributes like `"alloc-family"`. If the module with the global allocator participates in LTO, this ensures that the attributes typically emitted on the allocator declarations are not lost if the definition is imported.
There is a similar issue when the allocator shim is used, but I've opted not to fix that case in this PR, because doing that cleanly is somewhat gnarly.
Related to https://github.com/rust-lang/rust/issues/145995.
|
|
r=Urgau,davidtwco
Extends AArch64 branch protection support to include GCS
Extends existing support for AArch64 branch protection to include support for [Guarded Control Stacks](https://community.arm.com/arm-community-blogs/b/architectures-and-processors-blog/posts/arm-a-profile-architecture-2022#guarded-control-stack-gcs:~:text=Extraction%20or%20tracking.-,Guarded%20Control%20Stack%20(GCS),-With%20the%202022).
|
|
|
|
Emit `#[rustc_allocator]` etc. attributes on the functions generated
by the `#[global_allocator]` macro, which will emit LLVM attributes
like `"alloc-family"`. If the module with the global allocator
participates in LTO, this ensures that the attributes typically
emitted on the allocator declarations are not lost if the
definition is imported.
|
|
Mark float intrinsics with no preconditions as safe
Note: for ease of reviewing, the list of safe intrinsics is sorted in the first commit, and then safe intrinsics are added in the second commit.
All *recently added* float intrinsics have been correctly marked as safe to call due to the fact that they have no preconditions. This adds the remaining float intrinsics which are safe to call to the safe intrinsic list, and removes the unsafe blocks around their calls.
---
Side note: this may want a try run before being added to the queue, since I'm not sure if there's any tier-2 code that uses these intrinsics that might not be tested on the usual PR flow. We've already uncovered a few places in subtrees that do this, and it's worth double-checking before clogging up the queue.
|
|
|
|
Support ctr and lr as clobber-only registers in PowerPC inline assembly
Follow-up to rust-lang/rust#131341.
CTR and LR are marked as volatile in all ABIs, but I skipped them in rust-lang/rust#131341 due to they are currently marked as reserved.
https://github.com/rust-lang/rust/blob/dd7fda570040e8a736f7d8bc28ddd1b444aabc82/compiler/rustc_target/src/asm/powerpc.rs#L209-L212
However, they are actually only unusable as input/output of inline assembly, and should be fine to support as clobber-only registers as discussed in [#t-compiler > ppc/ppc64 inline asm support](https://rust-lang.zulipchat.com/#narrow/channel/131828-t-compiler/topic/ppc.2Fppc64.20inline.20asm.20support/with/540413845).
r? ````@Amanieu```` or ````@workingjubilee````
cc ````@programmerjake````
````@rustbot```` label +O-PowerPC +A-inline-assembly
|
|
emit attribute for readonly non-pure inline assembly
fixes https://github.com/rust-lang/rust/issues/146761
Provide a better `MemoryEffects` to LLVM when an inline assembly block specifies `readonly` but not `pure`. That means that the assembly block may not perform any writes, but that there still may be side effects from its instructions.
I haven't been able to find a case yet where this actually matters, though. So the test checks that the right attribute is applied, but the generated assembly is equivalent to not specifying `readonly` at all.
r? ````@nikic````
cc ````@Amanieu````
|
|
|
|
|
|
|
|
|
|
LLVM 22 is able to drop assumes that seem to not help further
optimizations, which actually seems to dramatically _help_ further
optimizations in some of our small test cases.
|
|
|
|
Signed-off-by: Karan Janthe <karanjanthe@gmail.com>
|
|
Prevent ABI changes affect EnzymeAD
This PR handles ABI changes for autodiff input arguments to improve Enzyme compatibility. Fundamentally this adjusts activities when a function argument is lowered as an `ScalarPair`, so there's no mismatch between diff activities and args. Also removes activities corresponding to ZSTs.
fixes: https://github.com/rust-lang/rust/issues/144025
r? `@ZuseZ4`
|
|
|
|
initial implementation of the darwin_objc unstable feature
Tracking issue: https://github.com/rust-lang/rust/issues/145496
This feature makes it possible to reference Objective-C classes and selectors using the same ABI used by native Objective-C on Apple/Darwin platforms. Without it, Rust code interacting with Objective-C must resort to loading classes and selectors using costly string-based lookups at runtime. With it, these references can be loaded efficiently at dynamic load time.
r? ```@tmandry```
try-job: `*apple*`
try-job: `x86_64-gnu-nopt`
|
|
|
|
r=Amanieu
rustc_codegen_llvm: Adjust RISC-V inline assembly's clobber list
Despite that the `fflags` register (representing floating point exception flags) is stated as a flag register [in the reference](https://doc.rust-lang.org/reference/inline-assembly.html#r-asm.rules.preserved-registers), it's not
in the default clobber list of the RISC-V inline assembly and it would be better to fix it.
|
|
r=Mark-Simulacrum
tests/codegen-llvm: Make rust-abi-arch-specific-adjustment portable
This test currently only runs on RISC-V and loongarch hosts, but assumes that the host target is the -gnu target. By using minicore, we can run this test on all host targets, regardless of architecture, as long as the LLVM components are built.
This also fixes this test on musl hosts of these architectures (though I've only tested on loongarch64-unknown-linux-musl).
|
|
tests: update new test to accept new lifetime format
Same change as rust-lang/rust@258915a55539593423c3d4c30f7b741f65c56e51, just for a newly written test.
|
|
Despite that the `fflags` register (representing floating point
exception flags) is stated as a flag register in the reference, it's not
in the default clobber list of the RISC-V inline assembly and it would
be better to fix it.
|
|
|
|
document `core::ffi::VaArgSafe`
tracking issue: https://github.com/rust-lang/rust/issues/44930
A modification of https://github.com/rust-lang/rust/pull/146454, keeping just the documentation changes, but not unsealing the trait.
Although conceptually we'd want to unseal the trait, there are many edge cases to supporting arbitrary types. We'd need to exhaustively test that all targets/calling conventions support all types that rust might generate (or generate proper error messages for unsupported cases). At present, many of the `va_arg` implementations assume that the argument is a scalar, and has an alignment of at most 8. That is totally sufficient for an MVP (accepting all of the "standard" C types), but clearly does not cover all rust types.
This PR also adds some various other tests for edge cases of c-variadic:
- the `#[inline]` attribute in its various forms. At present, LLVM is unable to inline c-variadic functions, but the attribute should still be accepted. `#[rustc_force_inline]` already rejects c-variadic functions.
- naked functions should accept and work with a C variable argument list. In the future we'd like to allow more ABIs with naked functions (basically, any ABI for which we accept defining foreign c-variadic functions), but for now only `"C"` and `"C-unwind` are supported
- guaranteed tail calls: c-variadic functions cannot be tail-called. That was already rejected, but there was not test for it.
r? `@workingjubilee`
|
|
they don't do anything, because LLVM is unable to inline c-variadic functions (on most targets, anyway)
|
|
Same change as rust-lang/rust@258915a55539593423c3d4c30f7b741f65c56e51,
just for a newly written test.
|
|
match clang's `va_arg` assembly on arm targets
tracking issue: https://github.com/rust-lang/rust/issues/44930
For this example
```rust
#![feature(c_variadic)]
#[unsafe(no_mangle)]
unsafe extern "C" fn variadic(a: f64, mut args: ...) -> f64 {
let b = args.arg::<f64>();
let c = args.arg::<f64>();
a + b + c
}
```
We currently generate (via llvm):
```asm
variadic:
sub sp, sp, #12
stmib sp, {r2, r3}
vmov d0, r0, r1
add r0, sp, #4
vldr d1, [sp, #4]
add r0, r0, #15
bic r0, r0, #7
vadd.f64 d0, d0, d1
add r1, r0, #8
str r1, [sp]
vldr d1, [r0]
vadd.f64 d0, d0, d1
vmov r0, r1, d0
add sp, sp, #12
bx lr
```
LLVM is not doing a good job. In fact, it's well-known that LLVM's implementation of `va_arg` is kind of bad, and we implement it ourselves (based on clang) for many targets already. For arm, our own `emit_ptr_va_arg` saves 3 instructions.
Next, it turns out it's important for LLVM to explicitly start and end the lifetime of the `va_list`. In https://github.com/rust-lang/rust/pull/146059 I already end the lifetime, but when looking at this again, I noticed that it is important to also start it, see https://godbolt.org/z/EGqvKTTsK: failing to explicitly start the lifetime uses an extra register.
So, the combination of `emit_ptr_va_arg` with starting/ending the lifetime makes rustc emit exactly the instructions that clang generates::
```asm
variadic:
sub sp, sp, #12
stmib sp, {r2, r3}
vmov d16, r0, r1
vldr d17, [sp, #4]
vadd.f64 d16, d16, d17
vldr d17, [sp, #12]
vadd.f64 d16, d16, d17
vmov r0, r1, d16
add sp, sp, #12
bx lr
```
The arguments to `emit_ptr_va_arg` are based on [the clang implementation](https://github.com/llvm/llvm-project/blob/03dc2a41f3d9a500e47b513de5c5008c06860d65/clang/lib/CodeGen/Targets/ARM.cpp#L798-L844).
r? ``@workingjubilee`` (I can re-roll if your queue is too full, but you do seem like the right person here)
try-job: armhf-gnu
|
|
This commit is an update to the `AbortUnwindingCalls` MIR pass in the
compiler. Specifically a new boolean is added for "can this target
possibly unwind" and if that's `false` then terminators are all adjusted
to be unreachable/not present. The end result is that this fixes 140293
for wasm targets.
The motivation for this PR is that currently on WebAssembly targets the
usage of the `C-unwind` ABI can lead LLVM to either (a) emit
exception-handling instructions or (b) hit a LLVM-ICE-style codegen
error. WebAssembly as a base instruction set does not support unwinding
at all, and a later proposal to WebAssembly, the exception-handling
proposal, was what enabled this. This means that the current intent of
WebAssembly targets is that they maintain the baseline of "don't emit
exception-handling instructions unless enabled". The commit here is
intended to restore this behavior by skipping these instructions even
when `C-unwind` is present.
Exception-handling is a relatively tricky and also murky topic in
WebAssembly, however. There are two sets of instructions LLVM can emit
for WebAssembly exceptions, Rust's Emscripten target supports
exceptions, WASI targets do not, the LLVM flags to enable this are not
always obvious, and additionally this all touches on "changing
exception-handling behavior should be a target-level concern, not a
feature". Effectively WebAssembly's exception-handling integration into
Rust is not finalized at this time. The best idea at this time is that a
parallel set of targets will eventually be added which support
exceptions, but it's not clear if/when to do this. In the meantime the
goal is to keep existing targets working while still enabling
experimentation with exception-handling with `-Zbuild-std` and various
permutations of LLVM flags.
To that extent this commit does not blanket disable these landing pads
and cleanup routines for WebAssembly but instead checks to see if
panic=unwind is enabled or if `+exception-handling` is enabled. Tests
are updated here as well to account for this where, by default, using a
`C-unwind` ABI won't affect Rust codegen at all. If
`+exception-handling` is enabled, however, then Rust codegen will look
like native platforms where exceptions are caught and the program aborts.
More-or-less I've done my best to keep exceptions working on wasm where
it's possible to have them work, but turned them off where they're not
supposed to be emitted.
|
|
r=jdonszelmann,ralfjung,traviscross
Implement `#[rustc_align_static(N)]` on `static`s
Tracking issue: https://github.com/rust-lang/rust/issues/146177
```rust
#![feature(static_align)]
#[rustc_align_static(64)]
static SO_ALIGNED: u64 = 0;
```
We need a different attribute than `rustc_align` because unstable attributes are tied to their feature (we can't have two unstable features use the same unstable attribute). Otherwise this uses all of the same infrastructure as `#[rustc_align]`.
r? `@traviscross`
|
|
We need a different attribute than `rustc_align` because unstable attributes are
tied to their feature (we can't have two unstable features use the same
unstable attribute). Otherwise this uses all of the same infrastructure
as `#[rustc_align]`.
|
|
This test currently only runs on RISC-V and loongarch hosts, but assumes
that the host target is the -gnu target. By using minicore, we can
run this test on all host targets, regardless of architecture, as long
as the LLVM components are built.
This also fixes this test on musl hosts of these architectures.
Signed-off-by: Jens Reidel <adrian@travitia.xyz>
|
|
Ignore intrinsic calls in cross-crate-inlining cost model
I noticed in a side project that a function which just compares to `[u64; 2]` for equality is not cross-crate-inlinable. That was surprising to me because I didn't think that code contained a function call, but of course our array comparisons are lowered to an intrinsic. Intrinsic calls don't make a function no longer a leaf, so it makes sense to add this as an exception to the "only leaves" cross-crate-inline heuristic.
This is the useful compare link: https://perf.rust-lang.org/compare.html?start=7cb1a81145a739c4fd858abe3c624ce8e6e5f9cd&end=c3f0a64dbf9fba4722dacf8e39d2fe00069c995e&stat=instructions%3Au because it disables CGU merging in both commits, so effects that cause changes in the sysroot to perturb partitioning downstream are excluded. Perturbations to what is and isn't cross-crate-inlinable in the sysroot has chaotic effects on what items are in which CGUs after merging. It looks like before this PR by sheer luck some of the CGUs dirtied by the patch in eza incr-unchanged happened to be merged together, and with this PR they are not.
The perf runs on this PR point to a nice runtime performance improvement.
|
|
Add amdgpu test for addrspacecasting global vars and the gpu-kernel calling convention
Add two tests that can now be added, as the amdgpu is merged.
- Global variables are casted to the default address space since rust-lang/rust#135026
- gpu-kernel calling convention, translatos to amdgpu_kernel rust-lang/rust#135047
Tracking issue: rust-lang/rust#135024
|
|
|
|
Fix LoongArch C function ABI when passing/returning structs containing floats
Similar to RISC-V, LoongArch passes structs containing only one or two floats (or a float–integer pair) in registers, as long as each element fits into a single corresponding register. Before this PR, Rust did not check the actual offset of the second float or integer; instead, it assumed the standard offset based on the default alignment. However, since the offset can be affected by `#[repr(align(N))]` and `#[repr(packed)]`, this led to miscompilations (see rust-lang/rust#145692). This PR fixes the issue by explicitly specifying the offset for the remainder of the cast.
|
|
r=rcvalle
Sanitizers target modificators
Depends on bool flag fix: https://github.com/rust-lang/rust/pull/138483.
Some sanitizers need to be target modifiers, and some do not. For now, we should mark all sanitizers as target modifiers except for these: AddressSanitizer, LeakSanitizer
For kCFI, the helper flag -Zsanitizer-cfi-normalize-integers should also be a target modifier.
Many test errors was with sanizer flags inconsistent with std deps. Tests are fixed with `-C unsafe-allow-abi-mismatch`.
|
|
Check that the gpu-kernel calling convention translates to
`amdgpu_kernel` when compiling for the amdgpu target.
|
|
Global variables are casted to the default address space works,
as the amdgpu target is now merged, a test can be added.
|
|
|
|
Adjust issue-118306.rs test after LLVM change
This updates tests/codegen-llvm/issues/issue-118306.rs to pass also after https://github.com/llvm/llvm-project/pull/155415
|
