| Age | Commit message (Collapse) | Author | Lines |
|---|
|
do not add noalias in return position
`noalias` as a return attribute in LLVM indicates that the returned pointer does not alias anything else that is reachable from the caller, *including things reachable before this function call*. This is clearly not the case with a function like `fn id(Box<T>) -> Box<T>`, so we cannot use this attribute.
Fixes https://github.com/rust-lang/unsafe-code-guidelines/issues/385 (including an actual miscompilation that `@comex` managed to produce).
|
|
Add codegen test for issue 103840
Closes #103840
|
|
|
|
Monomorphise `#[repr(transparent)]` parameterized ADTs before turning
them into an Itanium mangled String.
`#[repr(transparent)]` ADTs currently use the single field to represent
them in their CFI type ID to ensure that they are compatible. However,
if that type involves a type parameter instantiated at the ADT level, as
in `ManuallyDrop`, this will currently ICE as the `Parameter` type
cannot be mangled. Since this happens at lowering time, it should always
be concrete after substitution.
Fixes #106230
|
|
|
|
abort immediately on bad mem::zeroed/uninit
Now that we have non-unwinding panics, let's use them for these assertions. This re-establishes the property that `mem::uninitialized` and `mem::zeroed` will never unwind -- the earlier approach of causing panics here sometimes led to hard-to-debug segfaults when the surrounding code was not able to cope with the unexpected unwinding.
Cc `@bjorn3` I did not touch cranelift but I assume it needs a similar patch. However it has a `codegen_panic` abstraction that I did not want to touch since I didn't know how else it is used.
|
|
|
|
Codegen test for derived `<` on trivial newtype [TEST ONLY]
I originally wrote this for #106065, but the libcore changes there aren't necessarily a win.
So I pulled out this test to be its own PR since it's important (see https://github.com/rust-lang/rust/pull/105840#discussion_r1056030324) and well-intentioned changes to core or the derive could accidentally break it without that being obvious (other than by massive unexplained perf changes).
|
|
|
|
KCFI test: Also support LLVM 16 output
With a regex we can support the LLVM 16 output now already (and there are some third-party build bots that test Rust with LLVM head, like https://buildkite.com/llvm-project/rust-llvm-integrate-prototype/builds?branch=master).
cc ```@rcvalle```
|
|
|
|
codegen tests: adapt patterns to also work with v0 symbol mangling
No functional changes intended.
These tests were failing under `new-symbol-mangling = true`, cf. https://rust-lang.zulipchat.com/#narrow/stream/131828-t-compiler/topic/a.20few.20panic-abort.20tests.20fail.20under.20.60new-symbol-mangling.60.
This adapts the patterns to work in this case.
|
|
Add `-Zno-jump-tables`
This flag mimics GCC/Clang's `-fno-jump-tables` [1][2], which makes the codegen backend avoid generating jump tables when lowering switches.
In the case of LLVM, the `"no-jump-tables"="true"` function attribute is added to every function.
The kernel currently needs it for x86 when enabling IBT [3], as well as for Alpha (plus VDSO objects in MIPS/LoongArch).
[1] https://gcc.gnu.org/onlinedocs/gcc/Code-Gen-Options.html#index-fno-jump-tables
[2] https://clang.llvm.org/docs/ClangCommandLineReference.html#cmdoption-clang-fjump-tables
[3] https://github.com/torvalds/linux/blob/v6.1/arch/x86/Makefile#L75-L83
|
|
No functional changes intended.
These tests were failing under `new-symbol-mangling = true`.
This adapts the patterns to work in this case.
|
|
This flag mimics GCC/Clang's `-fno-jump-tables` [1][2], which makes
the codegen backend avoid generating jump tables when lowering switches.
In the case of LLVM, the `"no-jump-tables"="true"` function attribute is
added to every function.
The kernel currently needs it for x86 when enabling IBT [3], as well
as for Alpha (plus VDSO objects in MIPS/LoongArch).
[1] https://gcc.gnu.org/onlinedocs/gcc/Code-Gen-Options.html#index-fno-jump-tables
[2] https://clang.llvm.org/docs/ClangCommandLineReference.html#cmdoption-clang-fjump-tables
[3] https://github.com/torvalds/linux/blob/v6.1/arch/x86/Makefile#L75-L83
Signed-off-by: Miguel Ojeda <ojeda@kernel.org>
|
|
Add 0..=isize::MAX range metadata to size loads from vtables
This is the (much belated) size counterpart to #91569.
Inspired by https://rust-lang.zulipchat.com/#narrow/stream/187780-t-compiler.2Fwg-llvm/topic/Range.20metadata.20for.20.60size_of_val.60.20and.20other.20isize.3A.3AMAX.20limits. This could help optimize layout computations based on the size of a dyn trait. Though, admittedly, adding this to vtables wouldn't be as beneficial as adding it to slice len, which is used much more often.
Miri detects this UB already: https://github.com/rust-lang/rust/blob/b7cc99142ad0cfe47e2fe9f7a82eaf5b672c0573/compiler/rustc_const_eval/src/interpret/traits.rs#L119-L121
(In fact Miri goes further, [assuming a 48-bit address space on 64-bit platforms](https://github.com/rust-lang/rust/blob/9db224fc908059986c179fc6ec433944e9cfce50/compiler/rustc_abi/src/lib.rs#L312-L331), but I don't think we can assume that in an optimization.)
|
|
https://reviews.llvm.org/D140192 caused the LLVM variable generated
for enum discriminant checks to be named differently (%narrow vs %1).
This adjusts the test CHECK directives to match any name.
|
|
Fix transmutes between pointers in different address spaces (e.g. fn ptrs on AVR)
Currently, this causes a verifier error (https://godbolt.org/z/YYohed4bj), since it uses `bitcast`, which can't convert between address spaces.
Uncovered due to https://github.com/rust-lang/rust/pull/105545#discussion_r1045269309
r? `@bjorn3`
|
|
|
|
|
|
Don't internalize __llvm_profile_counter_bias
Currently, LLVM profiling runtime counter relocation cannot be used by rust during LTO because symbols are being internalized before all symbol information is known.
This mode makes LLVM emit a __llvm_profile_counter_bias symbol which is referenced by the profiling initialization, which itself is pulled in by the rust driver here [1].
It is enabled with -Cllvm-args=-runtime-counter-relocation for platforms which are opt-in to this mode like Linux. On these platforms there will be no link error, rather just surprising behavior for a user which request runtime counter relocation. The profiling runtime will not see that symbol go on as if it were never there. On Fuchsia, the profiling runtime must have this symbol which will cause a hard link error.
As an aside, I don't have enough context as to why rust's LTO model is how it is. AFAICT, the internalize pass is only safe to run at link time when all symbol information is actually known, this being an example as to why. I think special casing this symbol as a known one that LLVM can emit which should not have it's visbility de-escalated should be fine given how seldom this pattern of defining an undefined symbol to get initilization code pulled in is. From a quick grep, __llvm_profile_runtime is the only symbol that rustc does this for.
[1] https://github.com/rust-lang/rust/blob/0265a3e93bf1b89d97cae113ed214954d5c35e22/compiler/rustc_codegen_ssa/src/back/linker.rs#L598
|
|
Rollup of 6 pull requests
Successful merges:
- #104460 (Migrate parts of `rustc_expand` to session diagnostics)
- #105192 (Point at LHS on binop type err if relevant)
- #105234 (Remove unneeded field from `SwitchTargets`)
- #105239 (Avoid heap allocation when truncating thread names)
- #105410 (Consider `parent_count` for const param defaults)
- #105482 (Fix invalid codegen during debuginfo lowering)
Failed merges:
- #105411 (Introduce `with_forced_trimmed_paths`)
r? `@ghost`
`@rustbot` modify labels: rollup
|
|
Fix invalid codegen during debuginfo lowering
In order for LLVM to correctly generate debuginfo for msvc, we sometimes need to spill arguments to the stack and perform some direct & indirect offsets into the value. Previously, this code always performed those actions, even when not required as LLVM would clean it up during optimization.
However, when MIR inlining is enabled, this can cause problems as the operations occur prior to the spilled value being initialized. To solve this, we first calculate the necessary offsets using just the type which is side-effect free and does not alter the LLVM IR. Then, if we are in a situation which requires us to generate the LLVM IR (and this situation only occurs for arguments, not local variables) then we perform the same calculation again, this time generating the appropriate LLVM IR as we go.
r? `@tmiasko` but feel free to reassign if you want 🙂
Fixes #105386
|
|
Fix failing codegen tests on s390x
Several codegen tests are currently failing due to making assumptions that are not valid for the s390x architecture:
- catch-unwind.rs: fails due to inlining differences. Already ignored on another platform for the same reason. Solution: Ignore on s390x.
- remap_path_prefix/main.rs: fails due to different alignment requirement for string constants. Solution: Do not test for the alignment requirement.
- repr-transparent-aggregates-1.rs: many ABI assumptions. Already ignored on many platforms for the same reason. Solution: Ignore on s390x.
- repr-transparent.rs: no vector ABI by default on s390x. Already ignored on another platform for a similar reason. Solution: Ignore on s390x.
- uninit-consts.rs: hard-coded little-endian constant. Solution: Match both little- and big-endian versions.
Fixes part of https://github.com/rust-lang/rust/issues/105383.
|
|
Add LLVM KCFI support to the Rust compiler
This PR adds LLVM Kernel Control Flow Integrity (KCFI) support to the Rust compiler. It initially provides forward-edge control flow protection for operating systems kernels for Rust-compiled code only by aggregating function pointers in groups identified by their return and parameter types. (See llvm/llvm-project@cff5bef.)
Forward-edge control flow protection for C or C++ and Rust -compiled code "mixed binaries" (i.e., for when C or C++ and Rust -compiled code share the same virtual address space) will be provided in later work as part of this project by identifying C char and integer type uses at the time types are encoded (see Type metadata in the design document in the tracking issue #89653).
LLVM KCFI can be enabled with -Zsanitizer=kcfi.
Thank you again, `@bjorn3,` `@eddyb,` `@nagisa,` and `@ojeda,` for all the help!
|
|
In order for LLVM to correctly generate debuginfo for msvc, we sometimes
need to spill arguments to the stack and perform some direct & indirect
offsets into the value. Previously, this code always performed those
actions, even when not required as LLVM would clean it up during
optimization.
However, when MIR inlining is enabled, this can cause problems as the
operations occur prior to the spilled value being initialized. To solve
this, we first calculate the necessary offsets using just the type which
is side-effect free and does not alter the LLVM IR. Then, if we are in a
situation which requires us to generate the LLVM IR (and this situation
only occurs for arguments, not local variables) then we perform the same
calculation again, this time generating the appropriate LLVM IR as we
go.
|
|
|
|
This commit adds LLVM Kernel Control Flow Integrity (KCFI) support to
the Rust compiler. It initially provides forward-edge control flow
protection for operating systems kernels for Rust-compiled code only by
aggregating function pointers in groups identified by their return and
parameter types. (See llvm/llvm-project@cff5bef.)
Forward-edge control flow protection for C or C++ and Rust -compiled
code "mixed binaries" (i.e., for when C or C++ and Rust -compiled code
share the same virtual address space) will be provided in later work as
part of this project by identifying C char and integer type uses at the
time types are encoded (see Type metadata in the design document in the
tracking issue #89653).
LLVM KCFI can be enabled with -Zsanitizer=kcfi.
Co-authored-by: bjorn3 <17426603+bjorn3@users.noreply.github.com>
|
|
|
|
|
|
Currently, LLVM profiling runtime counter relocation cannot be
used by rust during LTO because symbols are being internalized
before all symbol information is known.
This mode makes LLVM emit a __llvm_profile_counter_bias symbol
which is referenced by the profiling initialization, which itself
is pulled in by the rust driver here [1].
It is enabled with -Cllvm-args=-runtime-counter-relocation for
platforms which are opt-in to this mode like Linux. On these
platforms there will be no link error, rather just surprising
behavior for a user which request runtime counter relocation.
The profiling runtime will not see that symbol go on as if it
were never there. On Fuchsia, the profiling runtime must have
this symbol which will cause a hard link error.
As an aside, I don't have enough context as to why rust's LTO
model is how it is. AFAICT, the internalize pass is only safe
to run at link time when all symbol information is actually
known, this being an example as to why. I think special casing
this symbol as a known one that LLVM can emit which should not
have it's visbility de-escalated should be fine given how
seldom this pattern of defining an undefined symbol to get
initilization code pulled in is. From a quick grep,
__llvm_profile_runtime is the only symbol that rustc does this
for.
[1] https://github.com/rust-lang/rust/blob/0265a3e93bf1b89d97cae113ed214954d5c35e22/compiler/rustc_codegen_ssa/src/back/linker.rs#L598
|
|
Several codegen tests are currently failing due to making
assumptions that are not valid for the s390x architecture:
- catch-unwind.rs: fails due to inlining differences.
Already ignored on another platform for the same reason.
Solution: Ignore on s390x.
- remap_path_prefix/main.rs: fails due to different alignment
requirement for string constants.
Solution: Do not test for the alignment requirement.
- repr-transparent-aggregates-1.rs: many ABI assumptions.
Already ignored on many platforms for the same reason.
Solution: Ignore on s390x.
- repr-transparent.rs: no vector ABI by default on s390x.
Already ignored on another platform for a similar reason.
Solution: Ignore on s390x.
- uninit-consts.rs: hard-coded little-endian constant.
Solution: Match both little- and big-endian versions.
Fixes part of https://github.com/rust-lang/rust/issues/105383.
|
|
rustc_codegen_ssa: Fix for codegen_get_discr
When doing the optimized implementation of getting the discriminant, the arithmetic needs to be done in the tag type so wrapping behavior works correctly.
Fixes #104519
|
|
|
|
Run Windows-only tests only on Windows
This removes the need to maintain an ignore-list of all other OSs.
See https://github.com/rust-lang/rust/pull/102305 for a similar change.
|
|
|
|
Add a codegen-test that verifies inter-crate linkage with the static
relocation model. We expect all symbols that are part of a rust
compilation to end up in the same DSO, thus we expect `dso_local`
annotations.
|
|
This removes the need to maintain a list of all other
OSs which ignore the tests.
|
|
Stop peeling the last iteration of the loop in `Vec::resize_with`
`resize_with` uses the `ExtendWith` code that peels the last iteration:
https://github.com/rust-lang/rust/blob/341d8b8a2c290b4535e965867e876b095461ff6e/library/alloc/src/vec/mod.rs#L2525-L2529
But that's kinda weird for `ExtendFunc` because it does the same thing on the last iteration anyway:
https://github.com/rust-lang/rust/blob/341d8b8a2c290b4535e965867e876b095461ff6e/library/alloc/src/vec/mod.rs#L2494-L2502
So this just has it use the normal `extend`-from-`TrustedLen` code instead.
r? `@ghost`
|
|
Manually implement PartialEq for Option<T> and specialize non-nullable types
This PR manually implements `PartialEq` and `StructuralPartialEq` for `Option`, which seems to produce slightly better codegen than the automatically derived implementation.
It also allows specializing on the `core::num::NonZero*` and `core::ptr::NonNull` types, taking advantage of the niche optimization by transmuting the `Option<T>` to `T` to be compared directly, which can be done in just two instructions.
A comparison of the original, new and specialized code generation is available [here](https://godbolt.org/z/dE4jxdYsa).
|
|
|
|
Previously, async constructs would be lowered to "normal" generators,
with an additional `from_generator` / `GenFuture` shim in between to
convert from `Generator` to `Future`.
The compiler will now special-case these generators internally so that
async constructs will *directly* implement `Future` without the need
to go through the `from_generator` / `GenFuture` shim.
The primary motivation for this change was hiding this implementation
detail in stack traces and debuginfo, but it can in theory also help
the optimizer as there is less abstractions to see through.
|
|
|
|
`VecDeque::resize` should re-use the buffer in the passed-in element
Today it always copies it for *every* appended element, but one of those clones is avoidable.
This adds `iter::repeat_n` (https://github.com/rust-lang/rust/issues/104434) as the primitive needed to do this. If this PR is acceptable, I'll also use this in `Vec` rather than its custom `ExtendElement` type & infrastructure that is harder to share between multiple different containers:
https://github.com/rust-lang/rust/blob/101e1822c3e54e63996c8aaa014d55716f3937eb/library/alloc/src/vec/mod.rs#L2479-L2492
|
|
`unchecked_{shl|shr}` should use `u32` as the RHS
The other shift methods, such as https://doc.rust-lang.org/nightly/std/primitive.u64.html#method.checked_shr and https://doc.rust-lang.org/nightly/std/primitive.i16.html#method.wrapping_shl, use `u32` for the shift amount. That's consistent with other things, like `count_ones`, which also always use `u32` for a bit count, regardless of the size of the type.
This PR changes `unchecked_shl` and `unchecked_shr` to also use `u32` for the shift amount (rather than Self).
cc #85122, the `unchecked_math` tracking issue
|
|
When doing the optimized implementation of getting the discriminant, the
arithmetic needs to be done in the tag type so wrapping behavior works
correctly.
Fixes #104519
|
|
|
|
Today it always copies it for *every* appended element, but one of those clones is avoidable.
|
|
In some cases we can avoid arithmetic before checking whether a niche
represents an untagged variant.
This is relevant to #101872
|
|
Use aapcs for efiapi calling convention on arm
On arm, [llvm treats the C calling convention as `aapcs` on soft-float targets and `aapcs-vfp` on hard-float targets](https://github.com/rust-lang/compiler-builtins/issues/116#issuecomment-261057422). UEFI specifies in the arm calling convention that [floating point extensions aren't used](https://uefi.org/specs/UEFI/2.10/02_Overview.html#detailed-calling-convention), so always translate `efiapi` to `aapcs` on arm.
https://github.com/rust-lang/rust/issues/65815
|
