Auto merge of #36321 - uweigand:enum-abi, r=eddyb

Follow target ABI sign-/zero-extension rules for enum types

While attempting to port Rust to s390x, I ran into an ABI violation
(that caused rust_eh_personality to be miscompiled, breaking unwinding).
The problem is that this function returns an enum type, which is
supposed to be sign-extended according to the s390x ABI.  However,
common code would ignore target sign-/zero-extension rules for any
types that do not satisfy is_integral(), which includes enums.

For the general case of Rust enum types, which map to structure types
with a discriminant, that seems correct.  However, in the special case
of simple enums that map directly to C enum types (i.e. LLVM integers),
this is incorrect; we must follow the target extension rules for those.

Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>

1 files changed, 6 insertions, 0 deletions

diff --git a/src/librustc_trans/abi.rs b/src/librustc_trans/abi.rs
index c2b040c32f6..42289ec094f 100644
--- a/src/librustc_trans/abi.rs
+++ b/src/librustc_trans/abi.rs
@@ -35,6 +35,7 @@ use std::cmp;
 
 pub use syntax::abi::Abi;
 pub use rustc::ty::layout::{FAT_PTR_ADDR, FAT_PTR_EXTRA};
+use rustc::ty::layout::Layout;
 
 #[derive(Clone, Copy, PartialEq, Debug)]
 enum ArgKind {
@@ -317,6 +318,11 @@ impl FnType {
                 if ty.is_integral() {
                     arg.signedness = Some(ty.is_signed());
                 }
+                // Rust enum types that map onto C enums also need to follow
+                // the target ABI zero-/sign-extension rules.
+                if let Layout::CEnum { signed, .. } = *ccx.layout_of(ty) {
+                    arg.signedness = Some(signed);
+                }
                 if llsize_of_real(ccx, arg.ty) == 0 {
                     // For some forsaken reason, x86_64-pc-windows-gnu
                     // doesn't ignore zero-sized struct arguments.