| Age | Commit message (Collapse) | Author | Lines |
|---|
|
It is always false nowadays. ThinLTO summary writing is instead done by
llvm_optimize.
|
|
|
|
Use captures(address) instead of captures(none) for indirect args
While provenance cannot be captured through these arguments, the address / object identity can.
Fixes https://github.com/rust-lang/rust/issues/137668.
r? `@ghost`
|
|
While provenance cannot be captured through these arguments, the
address / object identity can.
|
|
The underlying implementation of `LLVMCreateConstantRangeAttribute` assumes
that each of `LowerWords` and `UpperWords` points to enough u64 values to
define an integer of the specified bit-length, and will encounter UB if that is
not the case.
Our safe wrapper function always passes pointers to `[u64; 2]` arrays,
regardless of the bit-length specified. That's fine in practice, because scalar
primitives never exceed 128 bits, but it is technically a soundness hole in a
safe function.
We can close the soundness hole by explicitly asserting `size_bits <= 128`.
This is effectively just a stricter version of the existing check that the
value must be small enough to fit in `c_uint`.
|
|
This avoids the need for `#![allow(non_upper_case_globals)]`.
|
|
|
|
`&Freeze` parameters are not only `readonly` within the function,
but any captures of the pointer can also only be used for reads.
This can now be encoded using the `captures(address, read_provenance)`
attribute.
|
|
Remove `LlvmArchiveBuilder` and supporting code/bindings
Switching over to the newer Rust-based `ArArchiveBuilder` happened in rust-lang/rust#128936, a year ago.
Per the comment in `new_archive_builder`, that seems like enough time to justify removing the older, unused `LlvmArchiveBuilder` implementation and its associated bindings.
Fixes rust-lang/rust#128955.
|
|
cg_llvm: Small cleanups to `owned_target_machine`
This PR contains a few tiny cleanups to the `owned_target_machine` code.
Each individual commit should be fairly straightforward.
|
|
cg_llvm: Use LLVM-C bindings for `LLVMSetTailCallKind`, `LLVMGetTypeKind`
This PR replaces two existing `LLVMRust` bindings with equivalent calls to the LLVM-C API.
For `LLVMGetTypeKind`, we avoid the UB hazard by declaring the foreign function to return `RawEnum<TypeKind>` (which is a wrapper around `u32`), and then perform checked conversion from `u32` to `TypeKind`.
|
|
|
|
As noted in the `ffi` module docs, passing pointer/length byte strings from
Rust to C++ is easier if we declare them as `*const c_uchar` on the Rust side,
but `const char *` (possibly signed) on the C++ side. This is allowed because
both pointer types are ABI-compatible, regardless of char signedness.
|
|
|
|
|
|
|
|
Set the dead_on_return attribute (added in LLVM 21) for arguments
that are passed indirectly, but not byval.
This indicates that the value of the argument on return does not
matter, enabling additional dead store elimination.
|
|
|
|
Implement support for `become` and explicit tail call codegen for the LLVM backend
This PR implements codegen of explicit tail calls via `become` in `rustc_codegen_ssa` and support within the LLVM backend. Completes a task on (https://github.com/rust-lang/rust/issues/112788). This PR implements all the necessary bits to make explicit tail calls usable, other backends have received stubs for now and will ICE if you use `become` on them. I suspect there is some bikeshedding to be done on how we should go about implementing this for other backends, but it should be relatively straightforward for GCC after this is merged.
During development I also put together a POC bytecode VM based on tail call dispatch to test these changes out and analyze the codegen to make sure it generates expected assembly. That is available [here](https://github.com/xacrimon/tcvm).
|
|
LLVM codegen backend.
|
|
|
|
gpu offload host code generation
r? ghost
This will generate most of the host side code to use llvm's offload feature.
The first PR will only handle automatic mem-transfers to and from the device.
So if a user calls a kernel, we will copy inputs back and forth, but we won't do the actual kernel launch.
Before merging, we will use LLVM's Info infrastructure to verify that the memcopies match what openmp offloa generates in C++. `LIBOMPTARGET_INFO=-1 ./my_rust_binary` should print that a memcpy to and later from the device is happening.
A follow-up PR will generate the actual device-side kernel which will then do computations on the GPU.
A third PR will implement manual host2device and device2host functionality, but the goal is to minimize cases where a user has to overwrite our default handling due to performance issues.
I'm trying to get a full MVP out first, so this just recognizes GPU functions based on magic names. The final frontend will obviously move this over to use proper macros, like I'm already doing it for the autodiff work.
This work will also be compatible with std::autodiff, so one can differentiate GPU kernels.
Tracking:
- https://github.com/rust-lang/rust/issues/131513
|
|
|
|
This is no longer implied by -wasm-enable-eh.
|
|
|
|
|
|
Remove support for dynamic allocas
Followup to rust-lang/rust#141811
|
|
Make some "safe" llvm ops actually sound
Noticed while doing other refactorings
it may cause some extra unnecessary allocations, but the current use sites are rare ones anyway
|
|
|
|
FFI declarations
|
|
|
|
|
|
|
|
Add PrintTAFn flag for targeted type analysis printing
## Summary
This PR adds a new `PrintTAFn` flag to the `-Z autodiff` option that allows printing type analysis information for a specific function, rather than all functions.
## Changes
### New Flag
- Added `PrintTAFn=<function_name>` option to `-Z autodiff`
- Usage: `-Z autodiff=Enable,PrintTAFn=my_function_name`
### Implementation Details
- **Rust side**: Added `PrintTAFn(String)` variant to `AutoDiff` enum
- **Parser**: Updated `parse_autodiff` to handle `PrintTAFn=<function_name>` syntax with proper error handling
- **FFI**: Added `set_print_type_fun` function to interface with Enzyme's `FunctionToAnalyze` command line option
- **Documentation**: Updated help text and documentation for the new flag
### Files Modified
- `compiler/rustc_session/src/config.rs`: Added `PrintTAFn(String)` variant
- `compiler/rustc_session/src/options.rs`: Updated parser and help text (now shows `PrintTAFn` in the list)
- `compiler/rustc_codegen_llvm/src/llvm/enzyme_ffi.rs`: Added FFI function and static variable
- `compiler/rustc_codegen_llvm/src/back/lto.rs`: Added handling for new flag
- `src/doc/rustc-dev-guide/src/autodiff/flags.md`: Updated documentation
- `src/doc/unstable-book/src/compiler-flags/autodiff.md`: Updated documentation
## Testing
The flag can be tested with:
```bash
rustc +enzyme -Z autodiff=Enable,PrintTAFn=square test.rs
```
This will print type analysis information only for the function named "square" instead of all functions.
## Error Handling
The parser includes proper error handling:
- Missing argument: `PrintTAFn` without `=<function_name>` will show an error
- Unknown options: Invalid autodiff options will be reported
r? ```@ZuseZ4```
|
|
Signed-off-by: Karan Janthe <karanjanthe@gmail.com>
|
|
intrinsics list
|
|
|
|
|
|
cg_llvm: Clean up some inline assembly bindings
This PR combines a few loosely-related cleanups to LLVM bindings related to inline assembly. These include:
- Replacing `LLVMRustInlineAsm` with LLVM-C's `LLVMGetInlineAsm`
- Adjusting FFI declarations to avoid the need for explicit `as_c_char_ptr` conversions
- Flattening control flow in `inline_asm_call`
There should be no functional changes.
|
|
As with `DIBuilderBox`, the "Box" suffix does a better job of communicating
that this is an owning pointer to some borrowable resource.
This also renames the `raw` method to `as_ref`, which is what it would have
been named originally if the `Deref` problem had been known at the time.
|
|
This avoids the need for an explicit `as_c_char_ptr` conversion.
|
|
This patch also changes the Rust-side declaration to take `*const c_uchar`
instead of `*const c_char`, to avoid the need for `AsCCharPtr`.
|
|
This LLVM-C binding replaces the existing `LLVMRustInlineAsm` function.
|
|
This module comment describes why it's OK for LLVM bindings to declare a
parameter type of `*const c_uchar` for pointer/length strings, even though the
corresponding parameter on the C/C++ side uses `const char *`.
Adding a searchable term to each such parameter should make it easier for
future maintainers to understand why `*const c_uchar` is being used instead of
`*const c_char`.
|
|
|
|
|
|
|
|
|
|
Autodiff batching
Enzyme supports batching, which is especially known from the ML side when training neural networks.
There we would normally have a training loop, where in each iteration we would pass in some data (e.g. an image), and a target vector. Based on how close we are with our prediction we compute our loss, and then use backpropagation to compute the gradients and update our weights.
That's quite inefficient, so what you normally do is passing in a batch of 8/16/.. images and targets, and compute the gradients for those all at once, allowing better optimizations.
Enzyme supports batching in two ways, the first one (which I implemented here) just accepts a Batch size,
and then each Dual/Duplicated argument has not one, but N shadow arguments. So instead of
```rs
for i in 0..100 {
df(x[i], y[i], 1234);
}
```
You can now do
```rs
for i in 0..100.step_by(4) {
df(x[i+0],x[i+1],x[i+2],x[i+3], y[i+0], y[i+1], y[i+2], y[i+3], 1234);
}
```
which will give the same results, but allows better compiler optimizations. See the testcase for details.
There is a second variant, where we can mark certain arguments and instead of having to pass in N shadow arguments, Enzyme assumes that the argument is N times longer. I.e. instead of accepting 4 slices with 12 floats each, we would accept one slice with 48 floats. I'll implement this over the next days.
I will also add more tests for both modes.
For any one preferring some more interactive explanation, here's a video of Tim's llvm dev talk, where he presents his work. https://www.youtube.com/watch?v=edvaLAL5RqU
I'll also add some other docs to the dev guide and user docs in another PR.
r? ghost
Tracking:
- https://github.com/rust-lang/rust/issues/124509
- https://github.com/rust-lang/rust/issues/135283
|
|
|
