Rollup merge of #140770 - folkertdev:custom-abi, r=tgross35

add `extern "custom"` functions

tracking issue: rust-lang/rust#140829
previous discussion: https://github.com/rust-lang/rust/issues/140566

In short, an `extern "custom"` function is a function with a custom ABI, that rust does not know about. Therefore, such functions can only be defined with `#[unsafe(naked)]` and `naked_asm!`, or via an `extern "C" { /* ... */ }` block. These functions cannot be called using normal rust syntax: calling them can only be done from inline assembly.

The motivation is low-level scenarios where a custom calling convention is used. Currently, we often pick `extern "C"`, but that is a lie because the function does not actually respect the C calling convention.

At the moment `"custom"` seems to be the name with the most support. That name is not final, but we need to pick something to actually implement this.

r? `@traviscross`
cc `@tgross35`

try-job: x86_64-apple-2

1 files changed, 5 insertions, 1 deletions

diff --git a/compiler/rustc_codegen_gcc/src/abi.rs b/compiler/rustc_codegen_gcc/src/abi.rs
index 3d0c258f576..08f3d281904 100644
--- a/compiler/rustc_codegen_gcc/src/abi.rs
+++ b/compiler/rustc_codegen_gcc/src/abi.rs
@@ -239,12 +239,16 @@ impl<'gcc, 'tcx> FnAbiGccExt<'gcc, 'tcx> for FnAbi<'tcx, Ty<'tcx>> {
 pub fn conv_to_fn_attribute<'gcc>(conv: CanonAbi, arch: &str) -> Option<FnAttribute<'gcc>> {
     let attribute = match conv {
         CanonAbi::C | CanonAbi::Rust => return None,
+        CanonAbi::RustCold => FnAttribute::Cold,
+        // Functions with this calling convention can only be called from assembly, but it is
+        // possible to declare an `extern "custom"` block, so the backend still needs a calling
+        // convention for declaring foreign functions.
+        CanonAbi::Custom => return None,
         CanonAbi::Arm(arm_call) => match arm_call {
             ArmCall::CCmseNonSecureCall => FnAttribute::ArmCmseNonsecureCall,
             ArmCall::CCmseNonSecureEntry => FnAttribute::ArmCmseNonsecureEntry,
             ArmCall::Aapcs => FnAttribute::ArmPcs("aapcs"),
         },
-        CanonAbi::RustCold => FnAttribute::Cold,
         CanonAbi::GpuKernel => {
             if arch == "amdgpu" {
                 FnAttribute::GcnAmdGpuHsaKernel