about summary refs log tree commit diff
path: root/compiler/rustc_codegen_llvm/src/intrinsic.rs
AgeCommit message (Collapse)AuthorLines
2025-01-26Consistently use the most significant bit of vector masksJörn Horstmann-93/+104
This improves the codegen for vector `select`, `gather`, `scatter` and boolean reduction intrinsics and fixes rust-lang/portable-simd#316. The current behavior of most mask operations during llvm codegen is to truncate the mask vector to <N x i1>, telling llvm to use the least significat bit. The exception is the `simd_bitmask` intrinsics, which already used the most signifiant bit. Since sse/avx instructions are defined to use the most significant bit, truncating means that llvm has to insert a left shift to move the bit into the most significant position, before the mask can actually be used. Similarly on aarch64, mask operations like blend work bit by bit, repeating the least significant bit across the whole lane involves shifting it into the sign position and then comparing against zero. By shifting before truncating to <N x i1>, we tell llvm that we only consider the most significant bit, removing the need for additional shift instructions in the assembly.
2025-01-24Make CodegenCx and Builder genericManuel Drehwald-6/+6
Co-authored-by: Oli Scherer <github35764891676564198441@oli-obk.de>
2024-12-27Override `carrying_mul_add` in cg_llvmScott McMurray-0/+31
2024-12-07Remove unnecessary `int_type_width_signed` functionScott McMurray-91/+75
2024-11-23Add simd_relaxed_fma intrinsicCaleb Zulawski-0/+2
2024-11-18use `TypingEnv` when no `infcx` is availablelcnr-7/+6
the behavior of the type system not only depends on the current assumptions, but also the currentnphase of the compiler. This is mostly necessary as we need to decide whether and how to reveal opaque types. We track this via the `TypingMode`.
2024-11-17Likely unlikely fixJiri Bobek-2/+0
2024-11-03compiler: Directly use rustc_abi in codegenJubilee Young-5/+5
2024-10-29compiler: `rustc_abi::Abi` => `BackendRepr`Jubilee Young-6/+7
The initial naming of "Abi" was an awful mistake, conveying wrong ideas about how psABIs worked and even more about what the enum meant. It was only meant to represent the way the value would be described to a codegen backend as it was lowered to that intermediate representation. It was never meant to mean anything about the actual psABI handling! The conflation is because LLVM typically will associate a certain form with a certain ABI, but even that does not hold when the special cases that actually exist arise, plus the IR annotations that modify the ABI. Reframe `rustc_abi::Abi` as the `BackendRepr` of the type, and rename `BackendRepr::Aggregate` as `BackendRepr::Memory`. Unfortunately, due to the persistent misunderstandings, this too is now incorrect: - Scattered ABI-relevant code is entangled with BackendRepr - We do not always pre-compute a correct BackendRepr that reflects how we "actually" want this value to be handled, so we leave the backend interface to also inject various special-cases here - In some cases `BackendRepr::Memory` is a "real" aggregate, but in others it is in fact using memory, and in some cases it is a scalar! Our rustc-to-backend lowering code handles this sort of thing right now. That will eventually be addressed by lifting duplicated lowering code to either rustc_codegen_ssa or rustc_target as appropriate.
2024-10-26Use safe wrappers `get_linkage` and `set_linkage`Zalathar-7/+6
2024-10-21Auto merge of #130950 - compiler-errors:yeet-eval, r=BoxyUwUbors-10/+9
Continue to get rid of `ty::Const::{try_}eval*` This PR mostly does: * Removes all of the `try_eval_*` and `eval_*` helpers from `ty::Const`, and replace their usages with `try_to_*`. * Remove `ty::Const::eval`. * Rename `ty::Const::normalize` to `ty::Const::normalize_internal`. This function is still used in the normalization code itself. * Fix some weirdness around the `TransmuteFrom` goal. I'm happy to split it out further; for example, I could probably land the first part which removes the helpers, or the changes to codegen which are more obvious than the changes to tools. r? BoxyUwU Part of https://github.com/rust-lang/rust/issues/130704
2024-10-19Get rid of const eval_* and try_eval_* helpersMichael Goulet-10/+9
2024-10-19cg_llvm: Reuse LLVM-C Comdat supportJubilee Young-1/+3
Migrate `llvm::set_comdat` and `llvm::SetUniqueComdat` to LLVM-C FFI. Note, now we can call `llvm::set_comdat` only when the target actually supports adding comdat. As this has no convenient LLVM-C API, we implement this as `TargetOptions::supports_comdat`. Co-authored-by: Stuart Cook <Zalathar@users.noreply.github.com>
2024-10-11intrinsics.fmuladdf{16,32,64,128}: expose llvm.fmuladd.* semanticsJed Brown-0/+5
Add intrinsics `fmuladd{f16,f32,f64,f128}`. This computes `(a * b) + c`, to be fused if the code generator determines that (i) the target instruction set has support for a fused operation, and (ii) that the fused operation is more efficient than the equivalent, separate pair of `mul` and `add` instructions. https://llvm.org/docs/LangRef.html#llvm-fmuladd-intrinsic MIRI support is included for f32 and f64. The codegen_cranelift uses the `fma` function from libc, which is a correct implementation, but without the desired performance semantic. I think this requires an update to cranelift to expose a suitable instruction in its IR. I have not tested with codegen_gcc, but it should behave the same way (using `fma` from libc).
2024-10-04Use wide pointers consistenly across the compilerUrgau-2/+2
2024-09-24Auto merge of #130389 - Luv-Ray:LLVMMDNodeInContext2, r=nikicbors-3/+5
llvm: replace some deprecated functions `LLVMMDStringInContext` and `LLVMMDNodeInContext` are deprecated, replace them with `LLVMMDStringInContext2` and `LLVMMDNodeInContext2`. Also replace `Value` with `Metadata` in some function signatures for better consistency.
2024-09-22Reformat using the new identifier sorting from rustfmtMichael Goulet-213/+207
2024-09-19Reformat some comments.Nicholas Nethercote-4/+6
So they are less than 100 chars.
2024-09-19replace some deprecated functionsLuv-Ray-3/+5
2024-09-18Rollup merge of #130457 - nnethercote:cleanup-codegen-traits, r=bjorn3Matthias Krüger-1/+1
Cleanup codegen traits The traits governing codegen are quite complicated and hard to follow. This PR cleans them up a bit. r? `@bjorn3`
2024-09-17Rename `{ArgAbi,IntrinsicCall}Methods`.Nicholas Nethercote-1/+1
They both are part of `BuilderMethods`, and so should have `Builder` in their name like all the other traits in `BuilderMethods`.
2024-09-15Rollup merge of #130325 - workingjubilee:plus-minus-zero-redux, ↵Jubilee-2/+2
r=RalfJung,jieyouxu Use -0.0 in `intrinsics::simd::reduce_add_unordered` -0.0 is the actual neutral additive float, not +0.0, and this matters to codegen. try-job: aarch64-gnu
2024-09-15Use -0.0 in `intrinsics::simd::reduce_add_unordered`Jubilee Young-2/+2
-0.0 is the actual neutral additive float, not +0.0, and this matters to codegen.
2024-09-14simd_shuffle: require index argument to be a vectorRalf Jung-52/+28
2024-09-04Don't codegen `expect` in opt-level=0clubby789-7/+10
2024-08-27Rollup merge of #128731 - RalfJung:simd-shuffle-vector, r=workingjubileeTrevor Gross-7/+12
simd_shuffle intrinsic: allow argument to be passed as vector See https://github.com/rust-lang/rust/issues/128738 for context. I'd like to get rid of [this hack](https://github.com/rust-lang/rust/blob/6c0b89dfac65be9a5be12f938f23098ebc36c635/compiler/rustc_codegen_ssa/src/mir/block.rs#L922-L935). https://github.com/rust-lang/rust/pull/128537 almost lets us do that since constant SIMD vectors will then be passed as immediate arguments. However, simd_shuffle for some reason actually takes an *array* as argument, not a vector, so the hack is still required to ensure that the array becomes an immediate (which then later stages of codegen convert into a vector, as that's what LLVM needs). This PR prepares simd_shuffle to also support a vector as the `idx` argument. Once this lands, stdarch can hopefully be updated to pass `idx` as a vector, and then support for arrays can be removed, which finally lets us get rid of that hack.
2024-08-17Fix `is_val_statically_known` for floatsbeetrees-7/+15
2024-08-13simd_shuffle intrinsic: allow argument to be passed as vector (not just as ↵Ralf Jung-7/+12
array)
2024-08-11Use assert_matches around the compilerMichael Goulet-1/+2
2024-08-01Specify the integer type of the `powi` LLVM intrinsicTrevor Gross-4/+4
Since LLVM <https://reviews.llvm.org/D99439> (4c7f820b2b20, "Update @llvm.powi to handle different int sizes for the exponent"), the size of the integer can be specified for the `powi` intrinsic. Make use of this so it is more obvious that integer size is consistent across all float types. This feature is available since LLVM 13 (October 2021). Based on bootstrap we currently support >= 17.0, so there should be no support problems.
2024-07-30Auto merge of #128250 - Amanieu:select_unpredictable, r=nikicbors-1/+30
Add `select_unpredictable` to force LLVM to use CMOV Since https://reviews.llvm.org/D118118, LLVM will no longer turn CMOVs into branches if it comes from a `select` marked with an `unpredictable` metadata attribute. This PR introduces `core::intrinsics::select_unpredictable` which emits such a `select` and uses it in the implementation of `binary_search_by`.
2024-07-29Reformat `use` declarations.Nicholas Nethercote-9/+9
The previous commit updated `rustfmt.toml` appropriately. This commit is the outcome of running `x fmt --all` with the new formatting options.
2024-07-28Force LLVM to use CMOV for binary searchAmanieu d'Antras-1/+30
Since https://reviews.llvm.org/D118118, LLVM will no longer turn CMOVs into branches if it comes from a `select` marked with an `unpredictable` metadata attribute. This PR introduces `core::intrinsics::select_unpredictable` which emits such a `select` and uses it in the implementation of `binary_search_by`.
2024-07-08Add `f16` and `f128` as SIMD types in LLVMTrevor Gross-0/+2
2024-07-01fix simd_bitmask return type for non-power-of-two inputs, and add testsRalf Jung-8/+8
2024-06-10Rollup merge of #126159 - RalfJung:scalarint-size-mismatch, r=oli-obkMatthias Krüger-1/+1
ScalarInt: size mismatches are a bug, do not delay the panic Cc [Zulip](https://rust-lang.zulipchat.com/#narrow/stream/146212-t-compiler.2Fconst-eval/topic/Why.20are.20ScalarInt.20to.20iN.2FuN.20methods.20fallible.3F) r? ``@oli-obk``
2024-06-10ScalarInt: size mismatches are a bug, do not delay the panicRalf Jung-1/+1
2024-06-08simd packed types: update outdated check, extend codegen testRalf Jung-4/+6
2024-06-05Add `Ty` to `ConstKind::Value`Boxy-0/+1
2024-06-02Rollup merge of #125311 - calebzulawski:repr-packed-simd-intrinsics, ↵Jubilee-1/+53
r=workingjubilee Make repr(packed) vectors work with SIMD intrinsics In #117116 I fixed `#[repr(packed, simd)]` by doing the expected thing and removing padding from the layout. This should be the last step in providing a solution to rust-lang/portable-simd#319
2024-06-01Improve documentationCaleb Zulawski-1/+6
2024-05-23Remove `#[macro_use] extern crate tracing` from `rustc_codegen_llvm`.Nicholas Nethercote-0/+1
2024-05-22Stop using `to_hir_binop` in codegenScott McMurray-6/+7
2024-05-20Make repr(packed) vectors work with SIMD intrinsicsCaleb Zulawski-1/+48
2024-05-19clarify the second arg to llvm.ctlz and cttzJubilee Young-3/+3
2024-05-18compiler: add simd_ctpop intrinsicJubilee Young-19/+29
2024-05-17Rename Unsafe to SafetySantiago Pastorino-3/+3
2024-05-10Rollup merge of #124957 - compiler-errors:builtin-deref, r=michaelwoeristerMatthias Krüger-1/+1
Make `Ty::builtin_deref` just return a `Ty` Nowhere in the compiler are we using the mutability part of the `TyAndMut` that we used to return.
2024-05-09Make builtin_deref just return a TyMichael Goulet-1/+1
2024-05-06Refactor float `Primitive`s to a separate `Float` typebeetrees-5/+9
generated by cgit-pink 1.4.1-3-g733a5 (git 2.37.1) at 2026-03-01 02:38:00 +0000