Auto merge of #69171 - Amanieu:new-asm, r=nagisa,nikomatsakis

Implement new asm! syntax from RFC 2850

This PR implements the new `asm!` syntax proposed in https://github.com/rust-lang/rfcs/pull/2850.

# Design

A large part of this PR revolves around taking an `asm!` macro invocation and plumbing it through all of the compiler layers down to LLVM codegen. Throughout the various stages, an `InlineAsm` generally consists of 3 components:

- The template string, which is stored as an array of `InlineAsmTemplatePiece`. Each piece represents either a literal or a placeholder for an operand (just like format strings).
```rust
pub enum InlineAsmTemplatePiece {
    String(String),
    Placeholder { operand_idx: usize, modifier: Option<char>, span: Span },
}
```

- The list of operands to the `asm!` (`in`, `[late]out`, `in[late]out`, `sym`, `const`). These are represented differently at each stage of lowering, but follow a common pattern:
  - `in`, `out` and `inout` all have an associated register class (`reg`) or explicit register (`"eax"`).
  - `inout` has 2 forms: one with a single expression that is both read from and written to, and one with two separate expressions for the input and output parts.
  - `out` and `inout` have a `late` flag (`lateout` / `inlateout`) to indicate that the register allocator is allowed to reuse an input register for this output.
  - `out` and the split variant of `inout` allow `_` to be specified for an output, which means that the output is discarded. This is used to allocate scratch registers for assembly code.
  - `sym` is a bit special since it only accepts a path expression, which must point to a `static` or a `fn`.

- The options set at the end of the `asm!` macro. The only one that is particularly of interest to rustc is `NORETURN` which makes `asm!` return `!` instead of `()`.
```rust
bitflags::bitflags! {
    pub struct InlineAsmOptions: u8 {
        const PURE = 1 << 0;
        const NOMEM = 1 << 1;
        const READONLY = 1 << 2;
        const PRESERVES_FLAGS = 1 << 3;
        const NORETURN = 1 << 4;
        const NOSTACK = 1 << 5;
    }
}
```

## AST

`InlineAsm` is represented as an expression in the AST:

```rust
pub struct InlineAsm {
    pub template: Vec<InlineAsmTemplatePiece>,
    pub operands: Vec<(InlineAsmOperand, Span)>,
    pub options: InlineAsmOptions,
}

pub enum InlineAsmRegOrRegClass {
    Reg(Symbol),
    RegClass(Symbol),
}

pub enum InlineAsmOperand {
    In {
        reg: InlineAsmRegOrRegClass,
        expr: P<Expr>,
    },
    Out {
        reg: InlineAsmRegOrRegClass,
        late: bool,
        expr: Option<P<Expr>>,
    },
    InOut {
        reg: InlineAsmRegOrRegClass,
        late: bool,
        expr: P<Expr>,
    },
    SplitInOut {
        reg: InlineAsmRegOrRegClass,
        late: bool,
        in_expr: P<Expr>,
        out_expr: Option<P<Expr>>,
    },
    Const {
        expr: P<Expr>,
    },
    Sym {
        expr: P<Expr>,
    },
}
```

The `asm!` macro is implemented in librustc_builtin_macros and outputs an `InlineAsm` AST node. The template string is parsed using libfmt_macros, positional and named operands are resolved to explicit operand indicies. Since target information is not available to macro invocations, validation of the registers and register classes is deferred to AST lowering.

## HIR

`InlineAsm` is represented as an expression in the HIR:

```rust
pub struct InlineAsm<'hir> {
    pub template: &'hir [InlineAsmTemplatePiece],
    pub operands: &'hir [InlineAsmOperand<'hir>],
    pub options: InlineAsmOptions,
}

pub enum InlineAsmRegOrRegClass {
    Reg(InlineAsmReg),
    RegClass(InlineAsmRegClass),
}

pub enum InlineAsmOperand<'hir> {
    In {
        reg: InlineAsmRegOrRegClass,
        expr: Expr<'hir>,
    },
    Out {
        reg: InlineAsmRegOrRegClass,
        late: bool,
        expr: Option<Expr<'hir>>,
    },
    InOut {
        reg: InlineAsmRegOrRegClass,
        late: bool,
        expr: Expr<'hir>,
    },
    SplitInOut {
        reg: InlineAsmRegOrRegClass,
        late: bool,
        in_expr: Expr<'hir>,
        out_expr: Option<Expr<'hir>>,
    },
    Const {
        expr: Expr<'hir>,
    },
    Sym {
        expr: Expr<'hir>,
    },
}
```

AST lowering is where `InlineAsmRegOrRegClass` is converted from `Symbol`s to an actual register or register class. If any modifiers are specified for a template string placeholder, these are validated against the set allowed for that operand type. Finally, explicit registers for inputs and outputs are checked for conflicts (same register used for different operands).

## Type checking

Each register class has a whitelist of types that it may be used with. After the types of all operands have been determined, the `intrinsicck` pass will check that these types are in the whitelist. It also checks that split `inout` operands have compatible types and that `const` operands are integers or floats. Suggestions are emitted where needed if a template modifier should be used for an operand based on the type that was passed into it.

## HAIR

`InlineAsm` is represented as an expression in the HAIR:

```rust
crate enum ExprKind<'tcx> {
    // [..]
    InlineAsm {
        template: &'tcx [InlineAsmTemplatePiece],
        operands: Vec<InlineAsmOperand<'tcx>>,
        options: InlineAsmOptions,
    },
}
crate enum InlineAsmOperand<'tcx> {
    In {
        reg: InlineAsmRegOrRegClass,
        expr: ExprRef<'tcx>,
    },
    Out {
        reg: InlineAsmRegOrRegClass,
        late: bool,
        expr: Option<ExprRef<'tcx>>,
    },
    InOut {
        reg: InlineAsmRegOrRegClass,
        late: bool,
        expr: ExprRef<'tcx>,
    },
    SplitInOut {
        reg: InlineAsmRegOrRegClass,
        late: bool,
        in_expr: ExprRef<'tcx>,
        out_expr: Option<ExprRef<'tcx>>,
    },
    Const {
        expr: ExprRef<'tcx>,
    },
    SymFn {
        expr: ExprRef<'tcx>,
    },
    SymStatic {
        expr: ExprRef<'tcx>,
    },
}
```

The only significant change compared to HIR is that `Sym` has been lowered to either a `SymFn` whose `expr` is a `Literal` ZST of the `fn`, or a `SymStatic` whose `expr` is a `StaticRef`.

## MIR

`InlineAsm` is represented as a `Terminator` in the MIR:

```rust
pub enum TerminatorKind<'tcx> {
    // [..]

    /// Block ends with an inline assembly block. This is a terminator since
    /// inline assembly is allowed to diverge.
    InlineAsm {
        /// The template for the inline assembly, with placeholders.
        template: &'tcx [InlineAsmTemplatePiece],

        /// The operands for the inline assembly, as `Operand`s or `Place`s.
        operands: Vec<InlineAsmOperand<'tcx>>,

        /// Miscellaneous options for the inline assembly.
        options: InlineAsmOptions,

        /// Destination block after the inline assembly returns, unless it is
        /// diverging (InlineAsmOptions::NORETURN).
        destination: Option<BasicBlock>,
    },
}

pub enum InlineAsmOperand<'tcx> {
    In {
        reg: InlineAsmRegOrRegClass,
        value: Operand<'tcx>,
    },
    Out {
        reg: InlineAsmRegOrRegClass,
        late: bool,
        place: Option<Place<'tcx>>,
    },
    InOut {
        reg: InlineAsmRegOrRegClass,
        late: bool,
        in_value: Operand<'tcx>,
        out_place: Option<Place<'tcx>>,
    },
    Const {
        value: Operand<'tcx>,
    },
    SymFn {
        value: Box<Constant<'tcx>>,
    },
    SymStatic {
        value: Box<Constant<'tcx>>,
    },
}
```

As part of HAIR lowering, `InOut` and `SplitInOut` operands are lowered to a split form with a separate `in_value` and `out_place`.

Semantically, the `InlineAsm` terminator is similar to the `Call` terminator except that it has multiple output places where a `Call` only has a single return place output.

The constant promotion pass is used to ensure that `const` operands are actually constants (using the same logic as `#[rustc_args_required_const]`).

## Codegen

Operands are lowered one more time before being passed to LLVM codegen:

```rust
pub enum InlineAsmOperandRef<'tcx, B: BackendTypes + ?Sized> {
    In {
        reg: InlineAsmRegOrRegClass,
        value: OperandRef<'tcx, B::Value>,
    },
    Out {
        reg: InlineAsmRegOrRegClass,
        late: bool,
        place: Option<PlaceRef<'tcx, B::Value>>,
    },
    InOut {
        reg: InlineAsmRegOrRegClass,
        late: bool,
        in_value: OperandRef<'tcx, B::Value>,
        out_place: Option<PlaceRef<'tcx, B::Value>>,
    },
    Const {
        string: String,
    },
    SymFn {
        instance: Instance<'tcx>,
    },
    SymStatic {
        def_id: DefId,
    },
}
```

The operands are lowered to LLVM operands and constraint codes as follow:
- `out` and the output part of `inout` operands are added first, as required by LLVM. Late output operands have a `=` prefix added to their constraint code, non-late output operands have a `=&` prefix added to their constraint code.
- `in` operands are added normally.
- `inout` operands are tied to the matching output operand.
- `sym` operands are passed as function pointers or pointers, using the `"s"` constraint.
- `const` operands are formatted to a string and directly inserted in the template string.

The template string is converted to LLVM form:
- `$` characters are escaped as `$$`.
- `const` operands are converted to strings and inserted directly.
- Placeholders are formatted as `${X:M}` where `X` is the operand index and `M` is the modifier character. Modifiers are converted from the Rust form to the LLVM form.

The various options are converted to clobber constraints or LLVM attributes, refer to the [RFC](https://github.com/Amanieu/rfcs/blob/inline-asm/text/0000-inline-asm.md#mapping-to-llvm-ir) for more details.

Note that LLVM is sometimes rather picky about what types it accepts for certain constraint codes so we sometimes need to insert conversions to/from a supported type. See the target-specific ISelLowering.cpp files in LLVM for details.

# Adding support for new architectures

Adding inline assembly support to an architecture is mostly a matter of defining the registers and register classes for that architecture. All the definitions for register classes are located in `src/librustc_target/asm/`.

Additionally you will need to implement lowering of these register classes to LLVM constraint codes in `src/librustc_codegen_llvm/asm.rs`.

18 files changed, 227 insertions, 27 deletions

diff --git a/src/librustc_mir/borrow_check/invalidation.rs b/src/librustc_mir/borrow_check/invalidation.rs
index a8c7a959b28..178e3db17cd 100644
--- a/src/librustc_mir/borrow_check/invalidation.rs
+++ b/src/librustc_mir/borrow_check/invalidation.rs
@@ -1,8 +1,8 @@
 use rustc_data_structures::graph::dominators::Dominators;
 use rustc_middle::mir::visit::Visitor;
-use rustc_middle::mir::TerminatorKind;
 use rustc_middle::mir::{BasicBlock, Body, Location, Place, Rvalue};
 use rustc_middle::mir::{BorrowKind, Mutability, Operand};
+use rustc_middle::mir::{InlineAsmOperand, TerminatorKind};
 use rustc_middle::mir::{Statement, StatementKind};
 use rustc_middle::ty::TyCtxt;
 
@@ -183,6 +183,29 @@ impl<'cx, 'tcx> Visitor<'tcx> for InvalidationGenerator<'cx, 'tcx> {
                     }
                 }
             }
+            TerminatorKind::InlineAsm { template: _, ref operands, options: _, destination: _ } => {
+                for op in operands {
+                    match *op {
+                        InlineAsmOperand::In { reg: _, ref value }
+                        | InlineAsmOperand::Const { ref value } => {
+                            self.consume_operand(location, value);
+                        }
+                        InlineAsmOperand::Out { reg: _, late: _, place, .. } => {
+                            if let Some(place) = place {
+                                self.mutate_place(location, place, Shallow(None), JustWrite);
+                            }
+                        }
+                        InlineAsmOperand::InOut { reg: _, late: _, ref in_value, out_place } => {
+                            self.consume_operand(location, in_value);
+                            if let Some(out_place) = out_place {
+                                self.mutate_place(location, out_place, Shallow(None), JustWrite);
+                            }
+                        }
+                        InlineAsmOperand::SymFn { value: _ }
+                        | InlineAsmOperand::SymStatic { value: _ } => {}
+                    }
+                }
+            }
             TerminatorKind::Goto { target: _ }
             | TerminatorKind::Abort
             | TerminatorKind::Unreachable
diff --git a/src/librustc_mir/borrow_check/mod.rs b/src/librustc_mir/borrow_check/mod.rs
index a3ee49651ba..a0c1d96bb47 100644
--- a/src/librustc_mir/borrow_check/mod.rs
+++ b/src/librustc_mir/borrow_check/mod.rs
@@ -17,7 +17,7 @@ use rustc_middle::mir::{
 };
 use rustc_middle::mir::{AggregateKind, BasicBlock, BorrowCheckResult, BorrowKind};
 use rustc_middle::mir::{Field, ProjectionElem, Promoted, Rvalue, Statement, StatementKind};
-use rustc_middle::mir::{Terminator, TerminatorKind};
+use rustc_middle::mir::{InlineAsmOperand, Terminator, TerminatorKind};
 use rustc_middle::ty::query::Providers;
 use rustc_middle::ty::{self, RegionVid, TyCtxt};
 use rustc_session::lint::builtin::{MUTABLE_BORROW_RESERVATION_CONFLICT, UNUSED_MUT};
@@ -724,6 +724,42 @@ impl<'cx, 'tcx> dataflow::ResultsVisitor<'cx, 'tcx> for MirBorrowckCtxt<'cx, 'tc
                 self.mutate_place(loc, (resume_arg, span), Deep, JustWrite, flow_state);
             }
 
+            TerminatorKind::InlineAsm { template: _, ref operands, options: _, destination: _ } => {
+                for op in operands {
+                    match *op {
+                        InlineAsmOperand::In { reg: _, ref value }
+                        | InlineAsmOperand::Const { ref value } => {
+                            self.consume_operand(loc, (value, span), flow_state);
+                        }
+                        InlineAsmOperand::Out { reg: _, late: _, place, .. } => {
+                            if let Some(place) = place {
+                                self.mutate_place(
+                                    loc,
+                                    (place, span),
+                                    Shallow(None),
+                                    JustWrite,
+                                    flow_state,
+                                );
+                            }
+                        }
+                        InlineAsmOperand::InOut { reg: _, late: _, ref in_value, out_place } => {
+                            self.consume_operand(loc, (in_value, span), flow_state);
+                            if let Some(out_place) = out_place {
+                                self.mutate_place(
+                                    loc,
+                                    (out_place, span),
+                                    Shallow(None),
+                                    JustWrite,
+                                    flow_state,
+                                );
+                            }
+                        }
+                        InlineAsmOperand::SymFn { value: _ }
+                        | InlineAsmOperand::SymStatic { value: _ } => {}
+                    }
+                }
+            }
+
             TerminatorKind::Goto { target: _ }
             | TerminatorKind::Abort
             | TerminatorKind::Unreachable
@@ -778,7 +814,8 @@ impl<'cx, 'tcx> dataflow::ResultsVisitor<'cx, 'tcx> for MirBorrowckCtxt<'cx, 'tc
             | TerminatorKind::FalseUnwind { real_target: _, unwind: _ }
             | TerminatorKind::Goto { .. }
             | TerminatorKind::SwitchInt { .. }
-            | TerminatorKind::Unreachable => {}
+            | TerminatorKind::Unreachable
+            | TerminatorKind::InlineAsm { .. } => {}
         }
     }
 }
diff --git a/src/librustc_mir/borrow_check/type_check/mod.rs b/src/librustc_mir/borrow_check/type_check/mod.rs
index 526914721be..08aa4340107 100644
--- a/src/librustc_mir/borrow_check/type_check/mod.rs
+++ b/src/librustc_mir/borrow_check/type_check/mod.rs
@@ -1548,7 +1548,8 @@ impl<'a, 'tcx> TypeChecker<'a, 'tcx> {
             | TerminatorKind::Unreachable
             | TerminatorKind::Drop { .. }
             | TerminatorKind::FalseEdges { .. }
-            | TerminatorKind::FalseUnwind { .. } => {
+            | TerminatorKind::FalseUnwind { .. }
+            | TerminatorKind::InlineAsm { .. } => {
                 // no checks needed for these
             }
 
@@ -1855,6 +1856,11 @@ impl<'a, 'tcx> TypeChecker<'a, 'tcx> {
                     self.assert_iscleanup(body, block_data, unwind, true);
                 }
             }
+            TerminatorKind::InlineAsm { ref destination, .. } => {
+                if let &Some(target) = destination {
+                    self.assert_iscleanup(body, block_data, target, is_cleanup);
+                }
+            }
         }
     }
 
diff --git a/src/librustc_mir/dataflow/framework/direction.rs b/src/librustc_mir/dataflow/framework/direction.rs
index 76f703ec913..97b14ea771b 100644
--- a/src/librustc_mir/dataflow/framework/direction.rs
+++ b/src/librustc_mir/dataflow/framework/direction.rs
@@ -482,6 +482,12 @@ impl Direction for Forward {
                 }
             }
 
+            InlineAsm { template: _, operands: _, options: _, destination } => {
+                if let Some(target) = destination {
+                    propagate(target, exit_state);
+                }
+            }
+
             SwitchInt { ref targets, ref values, ref discr, switch_ty: _ } => {
                 let enum_ = discr
                     .place()
diff --git a/src/librustc_mir/dataflow/impls/borrowed_locals.rs b/src/librustc_mir/dataflow/impls/borrowed_locals.rs
index 6972a81cf1b..f929b2ddde0 100644
--- a/src/librustc_mir/dataflow/impls/borrowed_locals.rs
+++ b/src/librustc_mir/dataflow/impls/borrowed_locals.rs
@@ -203,6 +203,7 @@ where
             | TerminatorKind::FalseUnwind { .. }
             | TerminatorKind::GeneratorDrop
             | TerminatorKind::Goto { .. }
+            | TerminatorKind::InlineAsm { .. }
             | TerminatorKind::Resume
             | TerminatorKind::Return
             | TerminatorKind::SwitchInt { .. }
diff --git a/src/librustc_mir/dataflow/impls/borrows.rs b/src/librustc_mir/dataflow/impls/borrows.rs
index 0de8f45720e..dfca270396d 100644
--- a/src/librustc_mir/dataflow/impls/borrows.rs
+++ b/src/librustc_mir/dataflow/impls/borrows.rs
@@ -317,10 +317,19 @@ impl<'tcx> dataflow::GenKillAnalysis<'tcx> for Borrows<'_, 'tcx> {
 
     fn terminator_effect(
         &self,
-        _: &mut impl GenKill<Self::Idx>,
-        _: &mir::Terminator<'tcx>,
-        _: Location,
+        trans: &mut impl GenKill<Self::Idx>,
+        teminator: &mir::Terminator<'tcx>,
+        _location: Location,
     ) {
+        if let mir::TerminatorKind::InlineAsm { operands, .. } = &teminator.kind {
+            for op in operands {
+                if let mir::InlineAsmOperand::Out { place: Some(place), .. }
+                | mir::InlineAsmOperand::InOut { out_place: Some(place), .. } = *op
+                {
+                    self.kill_borrows_on_place(trans, place);
+                }
+            }
+        }
     }
 
     fn call_return_effect(
diff --git a/src/librustc_mir/dataflow/impls/storage_liveness.rs b/src/librustc_mir/dataflow/impls/storage_liveness.rs
index 4c784c3f1a1..bbc4942030e 100644
--- a/src/librustc_mir/dataflow/impls/storage_liveness.rs
+++ b/src/librustc_mir/dataflow/impls/storage_liveness.rs
@@ -183,6 +183,23 @@ impl<'mir, 'tcx> dataflow::GenKillAnalysis<'tcx> for MaybeRequiresStorage<'mir,
             // place to have storage *before* the yield, only after.
             TerminatorKind::Yield { .. } => {}
 
+            TerminatorKind::InlineAsm { operands, .. } => {
+                for op in operands {
+                    match op {
+                        InlineAsmOperand::Out { place, .. }
+                        | InlineAsmOperand::InOut { out_place: place, .. } => {
+                            if let Some(place) = place {
+                                trans.gen(place.local);
+                            }
+                        }
+                        InlineAsmOperand::In { .. }
+                        | InlineAsmOperand::Const { .. }
+                        | InlineAsmOperand::SymFn { .. }
+                        | InlineAsmOperand::SymStatic { .. } => {}
+                    }
+                }
+            }
+
             // Nothing to do for these. Match exhaustively so this fails to compile when new
             // variants are added.
             TerminatorKind::Call { destination: None, .. }
@@ -228,6 +245,7 @@ impl<'mir, 'tcx> dataflow::GenKillAnalysis<'tcx> for MaybeRequiresStorage<'mir,
             | TerminatorKind::FalseUnwind { .. }
             | TerminatorKind::GeneratorDrop
             | TerminatorKind::Goto { .. }
+            | TerminatorKind::InlineAsm { .. }
             | TerminatorKind::Resume
             | TerminatorKind::Return
             | TerminatorKind::SwitchInt { .. }
diff --git a/src/librustc_mir/dataflow/move_paths/builder.rs b/src/librustc_mir/dataflow/move_paths/builder.rs
index ffe84e9b655..0f2760b3f3b 100644
--- a/src/librustc_mir/dataflow/move_paths/builder.rs
+++ b/src/librustc_mir/dataflow/move_paths/builder.rs
@@ -411,6 +411,31 @@ impl<'b, 'a, 'tcx> Gatherer<'b, 'a, 'tcx> {
                     self.gather_init(destination.as_ref(), InitKind::NonPanicPathOnly);
                 }
             }
+            TerminatorKind::InlineAsm { template: _, ref operands, options: _, destination: _ } => {
+                for op in operands {
+                    match *op {
+                        InlineAsmOperand::In { reg: _, ref value }
+                        | InlineAsmOperand::Const { ref value } => {
+                            self.gather_operand(value);
+                        }
+                        InlineAsmOperand::Out { reg: _, late: _, place, .. } => {
+                            if let Some(place) = place {
+                                self.create_move_path(place);
+                                self.gather_init(place.as_ref(), InitKind::Deep);
+                            }
+                        }
+                        InlineAsmOperand::InOut { reg: _, late: _, ref in_value, out_place } => {
+                            self.gather_operand(in_value);
+                            if let Some(out_place) = out_place {
+                                self.create_move_path(out_place);
+                                self.gather_init(out_place.as_ref(), InitKind::Deep);
+                            }
+                        }
+                        InlineAsmOperand::SymFn { value: _ }
+                        | InlineAsmOperand::SymStatic { value: _ } => {}
+                    }
+                }
+            }
         }
     }
 
diff --git a/src/librustc_mir/interpret/terminator.rs b/src/librustc_mir/interpret/terminator.rs
index 9f9ec0f6a78..b048240ca8d 100644
--- a/src/librustc_mir/interpret/terminator.rs
+++ b/src/librustc_mir/interpret/terminator.rs
@@ -131,6 +131,9 @@ impl<'mir, 'tcx: 'mir, M: Machine<'mir, 'tcx>> InterpCx<'mir, 'tcx, M> {
                 "{:#?} should have been eliminated by MIR pass",
                 terminator.kind
             ),
+
+            // Inline assembly can't be interpreted.
+            InlineAsm { .. } => throw_unsup_format!("inline assembly is not supported"),
         }
 
         Ok(())
diff --git a/src/librustc_mir/monomorphize/collector.rs b/src/librustc_mir/monomorphize/collector.rs
index a8094990594..925b8d32966 100644
--- a/src/librustc_mir/monomorphize/collector.rs
+++ b/src/librustc_mir/monomorphize/collector.rs
@@ -639,7 +639,8 @@ impl<'a, 'tcx> MirVisitor<'tcx> for MirNeighborCollector<'a, 'tcx> {
             | mir::TerminatorKind::Abort
             | mir::TerminatorKind::Return
             | mir::TerminatorKind::Unreachable
-            | mir::TerminatorKind::Assert { .. } => {}
+            | mir::TerminatorKind::Assert { .. }
+            | mir::TerminatorKind::InlineAsm { .. } => {}
             mir::TerminatorKind::GeneratorDrop
             | mir::TerminatorKind::Yield { .. }
             | mir::TerminatorKind::FalseEdges { .. }
diff --git a/src/librustc_mir/transform/check_consts/validation.rs b/src/librustc_mir/transform/check_consts/validation.rs
index 8c005fdcdbf..987c9e24fc3 100644
--- a/src/librustc_mir/transform/check_consts/validation.rs
+++ b/src/librustc_mir/transform/check_consts/validation.rs
@@ -603,6 +603,10 @@ impl Visitor<'tcx> for Validator<'mir, 'tcx> {
                 }
             }
 
+            TerminatorKind::InlineAsm { .. } => {
+                self.check_op(ops::InlineAsm);
+            }
+
             // FIXME: Some of these are only caught by `min_const_fn`, but should error here
             // instead.
             TerminatorKind::Abort
diff --git a/src/librustc_mir/transform/check_unsafety.rs b/src/librustc_mir/transform/check_unsafety.rs
index f7c6396556d..9bcb45f6493 100644
--- a/src/librustc_mir/transform/check_unsafety.rs
+++ b/src/librustc_mir/transform/check_unsafety.rs
@@ -95,6 +95,12 @@ impl<'a, 'tcx> Visitor<'tcx> for UnsafetyChecker<'a, 'tcx> {
                     self.check_target_features(func_id);
                 }
             }
+
+            TerminatorKind::InlineAsm { .. } => self.require_unsafe(
+                "use of inline assembly",
+                "inline assembly is entirely unchecked and can cause undefined behavior",
+                UnsafetyViolationKind::General,
+            ),
         }
         self.super_terminator(terminator, location);
     }
diff --git a/src/librustc_mir/transform/const_prop.rs b/src/librustc_mir/transform/const_prop.rs
index e1be1daca45..f69343f4d75 100644
--- a/src/librustc_mir/transform/const_prop.rs
+++ b/src/librustc_mir/transform/const_prop.rs
@@ -1014,7 +1014,8 @@ impl<'mir, 'tcx> MutVisitor<'tcx> for ConstPropagator<'mir, 'tcx> {
             | TerminatorKind::Yield { .. }
             | TerminatorKind::GeneratorDrop
             | TerminatorKind::FalseEdges { .. }
-            | TerminatorKind::FalseUnwind { .. } => {}
+            | TerminatorKind::FalseUnwind { .. }
+            | TerminatorKind::InlineAsm { .. } => {}
             // Every argument in our function calls can be const propagated.
             TerminatorKind::Call { ref mut args, .. } => {
                 let mir_opt_level = self.tcx.sess.opts.debugging_opts.mir_opt_level;
diff --git a/src/librustc_mir/transform/generator.rs b/src/librustc_mir/transform/generator.rs
index 14faa5be02f..4bf2adcd450 100644
--- a/src/librustc_mir/transform/generator.rs
+++ b/src/librustc_mir/transform/generator.rs
@@ -981,7 +981,8 @@ fn can_unwind<'tcx>(tcx: TyCtxt<'tcx>, body: &Body<'tcx>) -> bool {
             | TerminatorKind::Unreachable
             | TerminatorKind::GeneratorDrop
             | TerminatorKind::FalseEdges { .. }
-            | TerminatorKind::FalseUnwind { .. } => {}
+            | TerminatorKind::FalseUnwind { .. }
+            | TerminatorKind::InlineAsm { .. } => {}
 
             // Resume will *continue* unwinding, but if there's no other unwinding terminator it
             // will never be reached.
diff --git a/src/librustc_mir/transform/inline.rs b/src/librustc_mir/transform/inline.rs
index 632408fde74..35d55c4cb9b 100644
--- a/src/librustc_mir/transform/inline.rs
+++ b/src/librustc_mir/transform/inline.rs
@@ -800,6 +800,11 @@ impl<'a, 'tcx> MutVisitor<'tcx> for Integrator<'a, 'tcx> {
             {
                 bug!("False unwinds should have been removed before inlining")
             }
+            TerminatorKind::InlineAsm { ref mut destination, .. } => {
+                if let Some(ref mut tgt) = *destination {
+                    *tgt = self.update_target(*tgt);
+                }
+            }
         }
     }
 
diff --git a/src/librustc_mir/transform/promote_consts.rs b/src/librustc_mir/transform/promote_consts.rs
index 6dade3c8dca..13a8b9a1000 100644
--- a/src/librustc_mir/transform/promote_consts.rs
+++ b/src/librustc_mir/transform/promote_consts.rs
@@ -113,6 +113,9 @@ pub enum Candidate {
     /// the attribute currently provides the semantic requirement that arguments
     /// must be constant.
     Argument { bb: BasicBlock, index: usize },
+
+    /// `const` operand in asm!.
+    InlineAsm { bb: BasicBlock, index: usize },
 }
 
 impl Candidate {
@@ -120,7 +123,7 @@ impl Candidate {
     fn forces_explicit_promotion(&self) -> bool {
         match self {
             Candidate::Ref(_) | Candidate::Repeat(_) => false,
-            Candidate::Argument { .. } => true,
+            Candidate::Argument { .. } | Candidate::InlineAsm { .. } => true,
         }
     }
 }
@@ -216,25 +219,39 @@ impl<'tcx> Visitor<'tcx> for Collector<'_, 'tcx> {
     fn visit_terminator_kind(&mut self, kind: &TerminatorKind<'tcx>, location: Location) {
         self.super_terminator_kind(kind, location);
 
-        if let TerminatorKind::Call { ref func, .. } = *kind {
-            if let ty::FnDef(def_id, _) = func.ty(self.ccx.body, self.ccx.tcx).kind {
-                let fn_sig = self.ccx.tcx.fn_sig(def_id);
-                if let Abi::RustIntrinsic | Abi::PlatformIntrinsic = fn_sig.abi() {
-                    let name = self.ccx.tcx.item_name(def_id);
-                    // FIXME(eddyb) use `#[rustc_args_required_const(2)]` for shuffles.
-                    if name.as_str().starts_with("simd_shuffle") {
-                        self.candidates.push(Candidate::Argument { bb: location.block, index: 2 });
+        match *kind {
+            TerminatorKind::Call { ref func, .. } => {
+                if let ty::FnDef(def_id, _) = func.ty(self.ccx.body, self.ccx.tcx).kind {
+                    let fn_sig = self.ccx.tcx.fn_sig(def_id);
+                    if let Abi::RustIntrinsic | Abi::PlatformIntrinsic = fn_sig.abi() {
+                        let name = self.ccx.tcx.item_name(def_id);
+                        // FIXME(eddyb) use `#[rustc_args_required_const(2)]` for shuffles.
+                        if name.as_str().starts_with("simd_shuffle") {
+                            self.candidates
+                                .push(Candidate::Argument { bb: location.block, index: 2 });
+
+                            return; // Don't double count `simd_shuffle` candidates
+                        }
+                    }
 
-                        return; // Don't double count `simd_shuffle` candidates
+                    if let Some(constant_args) = args_required_const(self.ccx.tcx, def_id) {
+                        for index in constant_args {
+                            self.candidates.push(Candidate::Argument { bb: location.block, index });
+                        }
                     }
                 }
-
-                if let Some(constant_args) = args_required_const(self.ccx.tcx, def_id) {
-                    for index in constant_args {
-                        self.candidates.push(Candidate::Argument { bb: location.block, index });
+            }
+            TerminatorKind::InlineAsm { ref operands, .. } => {
+                for (index, op) in operands.iter().enumerate() {
+                    match op {
+                        InlineAsmOperand::Const { .. } => {
+                            self.candidates.push(Candidate::InlineAsm { bb: location.block, index })
+                        }
+                        _ => {}
                     }
                 }
             }
+            _ => {}
         }
     }
 
@@ -402,6 +419,18 @@ impl<'tcx> Validator<'_, 'tcx> {
                     _ => bug!(),
                 }
             }
+            Candidate::InlineAsm { bb, index } => {
+                assert!(self.explicit);
+
+                let terminator = self.body[bb].terminator();
+                match &terminator.kind {
+                    TerminatorKind::InlineAsm { operands, .. } => match &operands[index] {
+                        InlineAsmOperand::Const { value } => self.validate_operand(value),
+                        _ => bug!(),
+                    },
+                    _ => bug!(),
+                }
+            }
         }
     }
 
@@ -747,7 +776,9 @@ pub fn validate_candidates(
             }
 
             match candidate {
-                Candidate::Argument { bb, index } if !is_promotable => {
+                Candidate::Argument { bb, index } | Candidate::InlineAsm { bb, index }
+                    if !is_promotable =>
+                {
                     let span = ccx.body[bb].terminator().source_info.span;
                     let msg = format!("argument {} is required to be a constant", index + 1);
                     ccx.tcx.sess.span_err(span, &msg);
@@ -1024,6 +1055,24 @@ impl<'a, 'tcx> Promoter<'a, 'tcx> {
                         _ => bug!(),
                     }
                 }
+                Candidate::InlineAsm { bb, index } => {
+                    let terminator = blocks[bb].terminator_mut();
+                    match terminator.kind {
+                        TerminatorKind::InlineAsm { ref mut operands, .. } => {
+                            match &mut operands[index] {
+                                InlineAsmOperand::Const { ref mut value } => {
+                                    let ty = value.ty(local_decls, self.tcx);
+                                    let span = terminator.source_info.span;
+
+                                    Rvalue::Use(mem::replace(value, promoted_operand(ty, span)))
+                                }
+                                _ => bug!(),
+                            }
+                        }
+
+                        _ => bug!(),
+                    }
+                }
             }
         };
 
@@ -1080,7 +1129,7 @@ pub fn promote_candidates<'tcx>(
                     }
                 }
             }
-            Candidate::Argument { .. } => {}
+            Candidate::Argument { .. } | Candidate::InlineAsm { .. } => {}
         }
 
         // Declare return place local so that `mir::Body::new` doesn't complain.
diff --git a/src/librustc_mir/transform/qualify_min_const_fn.rs b/src/librustc_mir/transform/qualify_min_const_fn.rs
index a2e2c1bf9ed..07502848893 100644
--- a/src/librustc_mir/transform/qualify_min_const_fn.rs
+++ b/src/librustc_mir/transform/qualify_min_const_fn.rs
@@ -392,5 +392,9 @@ fn check_terminator(
         TerminatorKind::Assert { cond, expected: _, msg: _, target: _, cleanup: _ } => {
             check_operand(tcx, cond, span, def_id, body)
         }
+
+        TerminatorKind::InlineAsm { .. } => {
+            Err((span, "cannot use inline assembly in const fn".into()))
+        }
     }
 }
diff --git a/src/librustc_mir/transform/remove_noop_landing_pads.rs b/src/librustc_mir/transform/remove_noop_landing_pads.rs
index a86ef7f2f52..69c0163b649 100644
--- a/src/librustc_mir/transform/remove_noop_landing_pads.rs
+++ b/src/librustc_mir/transform/remove_noop_landing_pads.rs
@@ -77,7 +77,8 @@ impl RemoveNoopLandingPads {
             | TerminatorKind::Call { .. }
             | TerminatorKind::Assert { .. }
             | TerminatorKind::DropAndReplace { .. }
-            | TerminatorKind::Drop { .. } => false,
+            | TerminatorKind::Drop { .. }
+            | TerminatorKind::InlineAsm { .. } => false,
         }
     }