Auto merge of #123948 - azhogin:azhogin/async-drop, r=oli-obk

Async drop codegen

Async drop implementation using templated coroutine for async drop glue generation.

Scopes changes to generate `async_drop_in_place()` awaits, when async droppable objects are out-of-scope in async context.

Implementation details:
https://github.com/azhogin/posts/blob/main/async-drop-impl.md

New fields in Drop terminator (drop & async_fut). Processing in codegen/miri must validate that those fields are empty (in full version async Drop terminator will be expanded at StateTransform pass or reverted to sync version). Changes in terminator visiting to consider possible new successor (drop field).

ResumedAfterDrop messages for panic when coroutine is resumed after it is started to be async drop'ed.

Lang item for generated coroutine for async function async_drop_in_place. `async fn async_drop_in_place<T>()::{{closure0}}`.

Scopes processing for generate async drop preparations. Async drop is a hidden Yield, so potentially async drops require the same dropline preparation as for Yield terminators.

Processing in StateTransform: async drops are expanded into yield-point. Generation of async drop of coroutine itself added.

Shims for AsyncDropGlueCtorShim, AsyncDropGlue and FutureDropPoll.

```rust
#[lang = "async_drop"]
pub trait AsyncDrop {
    #[allow(async_fn_in_trait)]
    async fn drop(self: Pin<&mut Self>);
}

impl Drop for Foo {
    fn drop(&mut self) {
        println!("Foo::drop({})", self.my_resource_handle);
    }
}

impl AsyncDrop for Foo {
    async fn drop(self: Pin<&mut Self>) {
        println!("Foo::async drop({})", self.my_resource_handle);
    }
}
```

First async drop glue implementation re-worked to use the same drop elaboration code as for sync drop.
`async_drop_in_place` changed to be `async fn`. So both `async_drop_in_place` ctor and produced coroutine have their lang items (`AsyncDropInPlace`/`AsyncDropInPlacePoll`) and shim instances (`AsyncDropGlueCtorShim`/`AsyncDropGlue`).
```
pub async unsafe fn async_drop_in_place<T: ?Sized>(_to_drop: *mut T) {
}
```
AsyncDropGlue shim generation uses `elaborate_drops::elaborate_drop` to produce drop ladder (in the similar way as for sync drop glue) and then `coroutine::StateTransform` to convert function into coroutine poll.

AsyncDropGlue coroutine's layout can't be calculated for generic T, it requires known final dropee type to be generated (in StateTransform). So, `templated coroutine` was introduced here (`templated_coroutine_layout(...)` etc).

Such approach overrides the first implementation using mixing language-level futures in https://github.com/rust-lang/rust/pull/121801.

14 files changed, 455 insertions, 256 deletions

diff --git a/compiler/rustc_middle/src/arena.rs b/compiler/rustc_middle/src/arena.rs
index d1bbb0598fe..2dcb0de92b7 100644
--- a/compiler/rustc_middle/src/arena.rs
+++ b/compiler/rustc_middle/src/arena.rs
@@ -9,6 +9,7 @@ macro_rules! arena_types {
     ($macro:path) => (
         $macro!([
             [] layout: rustc_abi::LayoutData<rustc_abi::FieldIdx, rustc_abi::VariantIdx>,
+            [] proxy_coroutine_layout: rustc_middle::mir::CoroutineLayout<'tcx>,
             [] fn_abi: rustc_target::callconv::FnAbi<'tcx, rustc_middle::ty::Ty<'tcx>>,
             // AdtDef are interned and compared by address
             [decode] adt_def: rustc_middle::ty::AdtDefData,
diff --git a/compiler/rustc_middle/src/middle/exported_symbols.rs b/compiler/rustc_middle/src/middle/exported_symbols.rs
index 0bfbd398797..3c62017072e 100644
--- a/compiler/rustc_middle/src/middle/exported_symbols.rs
+++ b/compiler/rustc_middle/src/middle/exported_symbols.rs
@@ -44,6 +44,7 @@ pub enum ExportedSymbol<'tcx> {
     Generic(DefId, GenericArgsRef<'tcx>),
     DropGlue(Ty<'tcx>),
     AsyncDropGlueCtorShim(Ty<'tcx>),
+    AsyncDropGlue(DefId, Ty<'tcx>),
     ThreadLocalShim(DefId),
     NoDefId(ty::SymbolName<'tcx>),
 }
@@ -63,6 +64,9 @@ impl<'tcx> ExportedSymbol<'tcx> {
             ExportedSymbol::AsyncDropGlueCtorShim(ty) => {
                 tcx.symbol_name(ty::Instance::resolve_async_drop_in_place(tcx, ty))
             }
+            ExportedSymbol::AsyncDropGlue(def_id, ty) => {
+                tcx.symbol_name(ty::Instance::resolve_async_drop_in_place_poll(tcx, def_id, ty))
+            }
             ExportedSymbol::ThreadLocalShim(def_id) => tcx.symbol_name(ty::Instance {
                 def: ty::InstanceKind::ThreadLocalShim(def_id),
                 args: ty::GenericArgs::empty(),
diff --git a/compiler/rustc_middle/src/mir/mod.rs b/compiler/rustc_middle/src/mir/mod.rs
index db19c858e7c..adc100941a3 100644
--- a/compiler/rustc_middle/src/mir/mod.rs
+++ b/compiler/rustc_middle/src/mir/mod.rs
@@ -200,7 +200,13 @@ pub struct CoroutineInfo<'tcx> {
     /// Coroutine drop glue. This field is populated after the state transform pass.
     pub coroutine_drop: Option<Body<'tcx>>,
 
-    /// The layout of a coroutine. This field is populated after the state transform pass.
+    /// Coroutine async drop glue.
+    pub coroutine_drop_async: Option<Body<'tcx>>,
+
+    /// When coroutine has sync drop, this is async proxy calling `coroutine_drop` sync impl.
+    pub coroutine_drop_proxy_async: Option<Body<'tcx>>,
+
+    /// The layout of a coroutine. Produced by the state transformation.
     pub coroutine_layout: Option<CoroutineLayout<'tcx>>,
 
     /// If this is a coroutine then record the type of source expression that caused this coroutine
@@ -220,6 +226,8 @@ impl<'tcx> CoroutineInfo<'tcx> {
             yield_ty: Some(yield_ty),
             resume_ty: Some(resume_ty),
             coroutine_drop: None,
+            coroutine_drop_async: None,
+            coroutine_drop_proxy_async: None,
             coroutine_layout: None,
         }
     }
@@ -588,6 +596,26 @@ impl<'tcx> Body<'tcx> {
     }
 
     #[inline]
+    pub fn coroutine_drop_async(&self) -> Option<&Body<'tcx>> {
+        self.coroutine.as_ref().and_then(|coroutine| coroutine.coroutine_drop_async.as_ref())
+    }
+
+    #[inline]
+    pub fn coroutine_requires_async_drop(&self) -> bool {
+        self.coroutine_drop_async().is_some()
+    }
+
+    #[inline]
+    pub fn future_drop_poll(&self) -> Option<&Body<'tcx>> {
+        self.coroutine.as_ref().and_then(|coroutine| {
+            coroutine
+                .coroutine_drop_async
+                .as_ref()
+                .or(coroutine.coroutine_drop_proxy_async.as_ref())
+        })
+    }
+
+    #[inline]
     pub fn coroutine_kind(&self) -> Option<CoroutineKind> {
         self.coroutine.as_ref().map(|coroutine| coroutine.coroutine_kind)
     }
diff --git a/compiler/rustc_middle/src/mir/mono.rs b/compiler/rustc_middle/src/mir/mono.rs
index 1ffe958dbdd..6b413a7383a 100644
--- a/compiler/rustc_middle/src/mir/mono.rs
+++ b/compiler/rustc_middle/src/mir/mono.rs
@@ -530,6 +530,8 @@ impl<'tcx> CodegenUnit<'tcx> {
                             | InstanceKind::CloneShim(..)
                             | InstanceKind::ThreadLocalShim(..)
                             | InstanceKind::FnPtrAddrShim(..)
+                            | InstanceKind::AsyncDropGlue(..)
+                            | InstanceKind::FutureDropPollShim(..)
                             | InstanceKind::AsyncDropGlueCtorShim(..) => None,
                         }
                     }
diff --git a/compiler/rustc_middle/src/mir/pretty.rs b/compiler/rustc_middle/src/mir/pretty.rs
index d57019fee0f..57ae7dc55c5 100644
--- a/compiler/rustc_middle/src/mir/pretty.rs
+++ b/compiler/rustc_middle/src/mir/pretty.rs
@@ -253,9 +253,7 @@ fn dump_path<'tcx>(
             }));
             s
         }
-        ty::InstanceKind::AsyncDropGlueCtorShim(_, Some(ty)) => {
-            // Unfortunately, pretty-printed typed are not very filename-friendly.
-            // We dome some filtering.
+        ty::InstanceKind::AsyncDropGlueCtorShim(_, ty) => {
             let mut s = ".".to_owned();
             s.extend(ty.to_string().chars().filter_map(|c| match c {
                 ' ' => None,
@@ -264,6 +262,34 @@ fn dump_path<'tcx>(
             }));
             s
         }
+        ty::InstanceKind::AsyncDropGlue(_, ty) => {
+            let ty::Coroutine(_, args) = ty.kind() else {
+                bug!();
+            };
+            let ty = args.first().unwrap().expect_ty();
+            let mut s = ".".to_owned();
+            s.extend(ty.to_string().chars().filter_map(|c| match c {
+                ' ' => None,
+                ':' | '<' | '>' => Some('_'),
+                c => Some(c),
+            }));
+            s
+        }
+        ty::InstanceKind::FutureDropPollShim(_, proxy_cor, impl_cor) => {
+            let mut s = ".".to_owned();
+            s.extend(proxy_cor.to_string().chars().filter_map(|c| match c {
+                ' ' => None,
+                ':' | '<' | '>' => Some('_'),
+                c => Some(c),
+            }));
+            s.push('.');
+            s.extend(impl_cor.to_string().chars().filter_map(|c| match c {
+                ' ' => None,
+                ':' | '<' | '>' => Some('_'),
+                c => Some(c),
+            }));
+            s
+        }
         _ => String::new(),
     };
 
@@ -1050,7 +1076,13 @@ impl<'tcx> TerminatorKind<'tcx> {
             Call { target: None, unwind: _, .. } => vec![],
             Yield { drop: Some(_), .. } => vec!["resume".into(), "drop".into()],
             Yield { drop: None, .. } => vec!["resume".into()],
-            Drop { unwind: UnwindAction::Cleanup(_), .. } => vec!["return".into(), "unwind".into()],
+            Drop { unwind: UnwindAction::Cleanup(_), drop: Some(_), .. } => {
+                vec!["return".into(), "unwind".into(), "drop".into()]
+            }
+            Drop { unwind: UnwindAction::Cleanup(_), drop: None, .. } => {
+                vec!["return".into(), "unwind".into()]
+            }
+            Drop { unwind: _, drop: Some(_), .. } => vec!["return".into(), "drop".into()],
             Drop { unwind: _, .. } => vec!["return".into()],
             Assert { unwind: UnwindAction::Cleanup(_), .. } => {
                 vec!["success".into(), "unwind".into()]
diff --git a/compiler/rustc_middle/src/mir/syntax.rs b/compiler/rustc_middle/src/mir/syntax.rs
index 304b3caa6e1..bb068f3821d 100644
--- a/compiler/rustc_middle/src/mir/syntax.rs
+++ b/compiler/rustc_middle/src/mir/syntax.rs
@@ -77,6 +77,8 @@ pub enum MirPhase {
     ///   exception is fields of packed structs. In analysis MIR, `Drop(P)` for a `P` that might be
     ///   misaligned for this reason implicitly moves `P` to a temporary before dropping. Runtime
     ///   MIR has no such rules, and dropping a misaligned place is simply UB.
+    /// - Async drops: after drop elaboration some drops may become async (`drop`, `async_fut` fields).
+    ///   StateTransform pass will expand those async drops or reset to sync.
     /// - Unwinding: in analysis MIR, unwinding from a function which may not unwind aborts. In
     ///   runtime MIR, this is UB.
     /// - Retags: If `-Zmir-emit-retag` is enabled, analysis MIR has "implicit" retags in the same
@@ -768,7 +770,34 @@ pub enum TerminatorKind<'tcx> {
     /// The `replace` flag indicates whether this terminator was created as part of an assignment.
     /// This should only be used for diagnostic purposes, and does not have any operational
     /// meaning.
-    Drop { place: Place<'tcx>, target: BasicBlock, unwind: UnwindAction, replace: bool },
+    ///
+    /// Async drop processing:
+    /// In compiler/rustc_mir_build/src/build/scope.rs we detect possible async drop:
+    ///   drop of object with `needs_async_drop`.
+    /// Async drop later, in StateTransform pass, may be expanded into additional yield-point
+    ///   for poll-loop of async drop future.
+    /// So we need prepared 'drop' target block in the similar way as for `Yield` terminator
+    ///   (see `drops.build_mir::<CoroutineDrop>` in scopes.rs).
+    /// In compiler/rustc_mir_transform/src/elaborate_drops.rs for object implementing `AsyncDrop` trait
+    ///   we need to prepare async drop feature - resolve `AsyncDrop::drop` and codegen call.
+    /// `async_fut` is set to the corresponding local.
+    /// For coroutine drop we don't need this logic because coroutine drop works with the same
+    ///   layout object as coroutine itself. So `async_fut` will be `None` for coroutine drop.
+    /// Both `drop` and `async_fut` fields are only used in compiler/rustc_mir_transform/src/coroutine.rs,
+    ///   StateTransform pass. In `expand_async_drops` async drops are expanded
+    ///   into one or two yield points with poll ready/pending switch.
+    /// When a coroutine has any internal async drop, the coroutine drop function will be async
+    ///   (generated by `create_coroutine_drop_shim_async`, not `create_coroutine_drop_shim`).
+    Drop {
+        place: Place<'tcx>,
+        target: BasicBlock,
+        unwind: UnwindAction,
+        replace: bool,
+        /// Cleanup to be done if the coroutine is dropped at this suspend point (for async drop).
+        drop: Option<BasicBlock>,
+        /// Prepared async future local (for async drop)
+        async_fut: Option<Local>,
+    },
 
     /// Roughly speaking, evaluates the `func` operand and the arguments, and starts execution of
     /// the referred to function. The operand types must match the argument types of the function.
@@ -1043,6 +1072,7 @@ pub enum AssertKind<O> {
     RemainderByZero(O),
     ResumedAfterReturn(CoroutineKind),
     ResumedAfterPanic(CoroutineKind),
+    ResumedAfterDrop(CoroutineKind),
     MisalignedPointerDereference { required: O, found: O },
     NullPointerDereference,
 }
diff --git a/compiler/rustc_middle/src/mir/terminator.rs b/compiler/rustc_middle/src/mir/terminator.rs
index 82e8422c52d..8a1ead7d19d 100644
--- a/compiler/rustc_middle/src/mir/terminator.rs
+++ b/compiler/rustc_middle/src/mir/terminator.rs
@@ -208,6 +208,16 @@ impl<O> AssertKind<O> {
                 LangItem::PanicGenFnNonePanic
             }
             NullPointerDereference => LangItem::PanicNullPointerDereference,
+            ResumedAfterDrop(CoroutineKind::Coroutine(_)) => LangItem::PanicCoroutineResumedDrop,
+            ResumedAfterDrop(CoroutineKind::Desugared(CoroutineDesugaring::Async, _)) => {
+                LangItem::PanicAsyncFnResumedDrop
+            }
+            ResumedAfterDrop(CoroutineKind::Desugared(CoroutineDesugaring::AsyncGen, _)) => {
+                LangItem::PanicAsyncGenFnResumedDrop
+            }
+            ResumedAfterDrop(CoroutineKind::Desugared(CoroutineDesugaring::Gen, _)) => {
+                LangItem::PanicGenFnNoneDrop
+            }
 
             BoundsCheck { .. } | MisalignedPointerDereference { .. } => {
                 bug!("Unexpected AssertKind")
@@ -298,6 +308,18 @@ impl<O> AssertKind<O> {
             ResumedAfterPanic(CoroutineKind::Desugared(CoroutineDesugaring::Gen, _)) => {
                 write!(f, "\"`gen fn` should just keep returning `None` after panicking\"")
             }
+            ResumedAfterDrop(CoroutineKind::Coroutine(_)) => {
+                write!(f, "\"coroutine resumed after async drop\"")
+            }
+            ResumedAfterDrop(CoroutineKind::Desugared(CoroutineDesugaring::Async, _)) => {
+                write!(f, "\"`async fn` resumed after async drop\"")
+            }
+            ResumedAfterDrop(CoroutineKind::Desugared(CoroutineDesugaring::AsyncGen, _)) => {
+                write!(f, "\"`async gen fn` resumed after async drop\"")
+            }
+            ResumedAfterDrop(CoroutineKind::Desugared(CoroutineDesugaring::Gen, _)) => {
+                write!(f, "\"`gen fn` resumed after drop\"")
+            }
         }
     }
 
@@ -345,6 +367,19 @@ impl<O> AssertKind<O> {
                 middle_assert_coroutine_resume_after_panic
             }
             NullPointerDereference => middle_assert_null_ptr_deref,
+            ResumedAfterDrop(CoroutineKind::Desugared(CoroutineDesugaring::Async, _)) => {
+                middle_assert_async_resume_after_drop
+            }
+            ResumedAfterDrop(CoroutineKind::Desugared(CoroutineDesugaring::AsyncGen, _)) => {
+                todo!()
+            }
+            ResumedAfterDrop(CoroutineKind::Desugared(CoroutineDesugaring::Gen, _)) => {
+                middle_assert_gen_resume_after_drop
+            }
+            ResumedAfterDrop(CoroutineKind::Coroutine(_)) => {
+                middle_assert_coroutine_resume_after_drop
+            }
+
             MisalignedPointerDereference { .. } => middle_assert_misaligned_ptr_deref,
         }
     }
@@ -377,7 +412,10 @@ impl<O> AssertKind<O> {
                 add!("left", format!("{left:#?}"));
                 add!("right", format!("{right:#?}"));
             }
-            ResumedAfterReturn(_) | ResumedAfterPanic(_) | NullPointerDereference => {}
+            ResumedAfterReturn(_)
+            | ResumedAfterPanic(_)
+            | NullPointerDereference
+            | ResumedAfterDrop(_) => {}
             MisalignedPointerDereference { required, found } => {
                 add!("required", format!("{required:#?}"));
                 add!("found", format!("{found:#?}"));
@@ -457,7 +495,7 @@ mod helper {
         #[define_opaque(Successors)]
         pub fn successors_for_value(&self, value: u128) -> Successors<'_> {
             let target = self.target_for_value(value);
-            (&[]).into_iter().copied().chain(Some(target))
+            (&[]).into_iter().copied().chain(Some(target).into_iter().chain(None))
         }
     }
 
@@ -467,13 +505,23 @@ mod helper {
         pub fn successors(&self) -> Successors<'_> {
             use self::TerminatorKind::*;
             match *self {
+                // 3-successors for async drop: target, unwind, dropline (parent coroutine drop)
+                Drop { target: ref t, unwind: UnwindAction::Cleanup(u), drop: Some(d), .. } => {
+                    slice::from_ref(t)
+                        .into_iter()
+                        .copied()
+                        .chain(Some(u).into_iter().chain(Some(d)))
+                }
+                // 2-successors
                 Call { target: Some(ref t), unwind: UnwindAction::Cleanup(u), .. }
                 | Yield { resume: ref t, drop: Some(u), .. }
-                | Drop { target: ref t, unwind: UnwindAction::Cleanup(u), .. }
+                | Drop { target: ref t, unwind: UnwindAction::Cleanup(u), drop: None, .. }
+                | Drop { target: ref t, unwind: _, drop: Some(u), .. }
                 | Assert { target: ref t, unwind: UnwindAction::Cleanup(u), .. }
                 | FalseUnwind { real_target: ref t, unwind: UnwindAction::Cleanup(u) } => {
-                    slice::from_ref(t).into_iter().copied().chain(Some(u))
+                    slice::from_ref(t).into_iter().copied().chain(Some(u).into_iter().chain(None))
                 }
+                // single successor
                 Goto { target: ref t }
                 | Call { target: None, unwind: UnwindAction::Cleanup(ref t), .. }
                 | Call { target: Some(ref t), unwind: _, .. }
@@ -481,23 +529,33 @@ mod helper {
                 | Drop { target: ref t, unwind: _, .. }
                 | Assert { target: ref t, unwind: _, .. }
                 | FalseUnwind { real_target: ref t, unwind: _ } => {
-                    slice::from_ref(t).into_iter().copied().chain(None)
+                    slice::from_ref(t).into_iter().copied().chain(None.into_iter().chain(None))
                 }
+                // No successors
                 UnwindResume
                 | UnwindTerminate(_)
                 | CoroutineDrop
                 | Return
                 | Unreachable
                 | TailCall { .. }
-                | Call { target: None, unwind: _, .. } => (&[]).into_iter().copied().chain(None),
+                | Call { target: None, unwind: _, .. } => {
+                    (&[]).into_iter().copied().chain(None.into_iter().chain(None))
+                }
+                // Multiple successors
                 InlineAsm { ref targets, unwind: UnwindAction::Cleanup(u), .. } => {
-                    targets.iter().copied().chain(Some(u))
+                    targets.iter().copied().chain(Some(u).into_iter().chain(None))
+                }
+                InlineAsm { ref targets, unwind: _, .. } => {
+                    targets.iter().copied().chain(None.into_iter().chain(None))
                 }
-                InlineAsm { ref targets, unwind: _, .. } => targets.iter().copied().chain(None),
-                SwitchInt { ref targets, .. } => targets.targets.iter().copied().chain(None),
-                FalseEdge { ref real_target, imaginary_target } => {
-                    slice::from_ref(real_target).into_iter().copied().chain(Some(imaginary_target))
+                SwitchInt { ref targets, .. } => {
+                    targets.targets.iter().copied().chain(None.into_iter().chain(None))
                 }
+                // FalseEdge
+                FalseEdge { ref real_target, imaginary_target } => slice::from_ref(real_target)
+                    .into_iter()
+                    .copied()
+                    .chain(Some(imaginary_target).into_iter().chain(None)),
             }
         }
 
@@ -506,16 +564,31 @@ mod helper {
         pub fn successors_mut(&mut self) -> SuccessorsMut<'_> {
             use self::TerminatorKind::*;
             match *self {
+                // 3-successors for async drop: target, unwind, dropline (parent coroutine drop)
+                Drop {
+                    target: ref mut t,
+                    unwind: UnwindAction::Cleanup(ref mut u),
+                    drop: Some(ref mut d),
+                    ..
+                } => slice::from_mut(t).into_iter().chain(Some(u).into_iter().chain(Some(d))),
+                // 2-successors
                 Call {
                     target: Some(ref mut t), unwind: UnwindAction::Cleanup(ref mut u), ..
                 }
                 | Yield { resume: ref mut t, drop: Some(ref mut u), .. }
-                | Drop { target: ref mut t, unwind: UnwindAction::Cleanup(ref mut u), .. }
+                | Drop {
+                    target: ref mut t,
+                    unwind: UnwindAction::Cleanup(ref mut u),
+                    drop: None,
+                    ..
+                }
+                | Drop { target: ref mut t, unwind: _, drop: Some(ref mut u), .. }
                 | Assert { target: ref mut t, unwind: UnwindAction::Cleanup(ref mut u), .. }
                 | FalseUnwind {
                     real_target: ref mut t,
                     unwind: UnwindAction::Cleanup(ref mut u),
-                } => slice::from_mut(t).into_iter().chain(Some(u)),
+                } => slice::from_mut(t).into_iter().chain(Some(u).into_iter().chain(None)),
+                // single successor
                 Goto { target: ref mut t }
                 | Call { target: None, unwind: UnwindAction::Cleanup(ref mut t), .. }
                 | Call { target: Some(ref mut t), unwind: _, .. }
@@ -523,22 +596,33 @@ mod helper {
                 | Drop { target: ref mut t, unwind: _, .. }
                 | Assert { target: ref mut t, unwind: _, .. }
                 | FalseUnwind { real_target: ref mut t, unwind: _ } => {
-                    slice::from_mut(t).into_iter().chain(None)
+                    slice::from_mut(t).into_iter().chain(None.into_iter().chain(None))
                 }
+                // No successors
                 UnwindResume
                 | UnwindTerminate(_)
                 | CoroutineDrop
                 | Return
                 | Unreachable
                 | TailCall { .. }
-                | Call { target: None, unwind: _, .. } => (&mut []).into_iter().chain(None),
+                | Call { target: None, unwind: _, .. } => {
+                    (&mut []).into_iter().chain(None.into_iter().chain(None))
+                }
+                // Multiple successors
                 InlineAsm { ref mut targets, unwind: UnwindAction::Cleanup(ref mut u), .. } => {
-                    targets.iter_mut().chain(Some(u))
+                    targets.iter_mut().chain(Some(u).into_iter().chain(None))
+                }
+                InlineAsm { ref mut targets, unwind: _, .. } => {
+                    targets.iter_mut().chain(None.into_iter().chain(None))
+                }
+                SwitchInt { ref mut targets, .. } => {
+                    targets.targets.iter_mut().chain(None.into_iter().chain(None))
                 }
-                InlineAsm { ref mut targets, unwind: _, .. } => targets.iter_mut().chain(None),
-                SwitchInt { ref mut targets, .. } => targets.targets.iter_mut().chain(None),
+                // FalseEdge
                 FalseEdge { ref mut real_target, ref mut imaginary_target } => {
-                    slice::from_mut(real_target).into_iter().chain(Some(imaginary_target))
+                    slice::from_mut(real_target)
+                        .into_iter()
+                        .chain(Some(imaginary_target).into_iter().chain(None))
                 }
             }
         }
@@ -671,8 +755,10 @@ impl<'tcx> TerminatorKind<'tcx> {
 
             Goto { target } => TerminatorEdges::Single(target),
 
+            // FIXME: Maybe we need also TerminatorEdges::Trio for async drop
+            // (target + unwind + dropline)
             Assert { target, unwind, expected: _, msg: _, cond: _ }
-            | Drop { target, unwind, place: _, replace: _ }
+            | Drop { target, unwind, place: _, replace: _, drop: _, async_fut: _ }
             | FalseUnwind { real_target: target, unwind } => match unwind {
                 UnwindAction::Cleanup(unwind) => TerminatorEdges::Double(target, unwind),
                 UnwindAction::Continue | UnwindAction::Terminate(_) | UnwindAction::Unreachable => {
diff --git a/compiler/rustc_middle/src/mir/visit.rs b/compiler/rustc_middle/src/mir/visit.rs
index de4d5140e85..1777756174b 100644
--- a/compiler/rustc_middle/src/mir/visit.rs
+++ b/compiler/rustc_middle/src/mir/visit.rs
@@ -353,17 +353,21 @@ macro_rules! make_mir_visitor {
                             coroutine_closure_def_id: _def_id,
                             receiver_by_ref: _,
                         }
-                        | ty::InstanceKind::AsyncDropGlueCtorShim(_def_id, None)
                         | ty::InstanceKind::DropGlue(_def_id, None) => {}
 
                         ty::InstanceKind::FnPtrShim(_def_id, ty)
                         | ty::InstanceKind::DropGlue(_def_id, Some(ty))
                         | ty::InstanceKind::CloneShim(_def_id, ty)
                         | ty::InstanceKind::FnPtrAddrShim(_def_id, ty)
-                        | ty::InstanceKind::AsyncDropGlueCtorShim(_def_id, Some(ty)) => {
+                        | ty::InstanceKind::AsyncDropGlue(_def_id, ty)
+                        | ty::InstanceKind::AsyncDropGlueCtorShim(_def_id, ty) => {
                             // FIXME(eddyb) use a better `TyContext` here.
                             self.visit_ty($(& $mutability)? *ty, TyContext::Location(location));
                         }
+                        ty::InstanceKind::FutureDropPollShim(_def_id, proxy_ty, impl_ty) => {
+                            self.visit_ty($(& $mutability)? *proxy_ty, TyContext::Location(location));
+                            self.visit_ty($(& $mutability)? *impl_ty, TyContext::Location(location));
+                        }
                     }
                     self.visit_args(callee_args, location);
                 }
@@ -521,7 +525,14 @@ macro_rules! make_mir_visitor {
                         self.visit_operand(discr, location);
                     }
 
-                    TerminatorKind::Drop { place, target: _, unwind: _, replace: _ } => {
+                    TerminatorKind::Drop {
+                        place,
+                        target: _,
+                        unwind: _,
+                        replace: _,
+                        drop: _,
+                        async_fut: _,
+                    } => {
                         self.visit_place(
                             place,
                             PlaceContext::MutatingUse(MutatingUseContext::Drop),
@@ -634,7 +645,7 @@ macro_rules! make_mir_visitor {
                     OverflowNeg(op) | DivisionByZero(op) | RemainderByZero(op) => {
                         self.visit_operand(op, location);
                     }
-                    ResumedAfterReturn(_) | ResumedAfterPanic(_) | NullPointerDereference => {
+                    ResumedAfterReturn(_) | ResumedAfterPanic(_) | NullPointerDereference | ResumedAfterDrop(_) => {
                         // Nothing to visit
                     }
                     MisalignedPointerDereference { required, found } => {
diff --git a/compiler/rustc_middle/src/query/mod.rs b/compiler/rustc_middle/src/query/mod.rs
index 3432648feab..293b1552beb 100644
--- a/compiler/rustc_middle/src/query/mod.rs
+++ b/compiler/rustc_middle/src/query/mod.rs
@@ -1359,7 +1359,11 @@ rustc_queries! {
     /// Generates a MIR body for the shim.
     query mir_shims(key: ty::InstanceKind<'tcx>) -> &'tcx mir::Body<'tcx> {
         arena_cache
-        desc { |tcx| "generating MIR shim for `{}`", tcx.def_path_str(key.def_id()) }
+        desc {
+            |tcx| "generating MIR shim for `{}`, instance={:?}",
+            tcx.def_path_str(key.def_id()),
+            key
+        }
     }
 
     /// The `symbol_name` query provides the symbol name for calling a
@@ -1590,6 +1594,10 @@ rustc_queries! {
     query is_unpin_raw(env: ty::PseudoCanonicalInput<'tcx, Ty<'tcx>>) -> bool {
         desc { "computing whether `{}` is `Unpin`", env.value }
     }
+    /// Query backing `Ty::is_async_drop`.
+    query is_async_drop_raw(env: ty::PseudoCanonicalInput<'tcx, Ty<'tcx>>) -> bool {
+        desc { "computing whether `{}` is `AsyncDrop`", env.value }
+    }
     /// Query backing `Ty::needs_drop`.
     query needs_drop_raw(env: ty::PseudoCanonicalInput<'tcx, Ty<'tcx>>) -> bool {
         desc { "computing whether `{}` needs drop", env.value }
@@ -1622,6 +1630,14 @@ rustc_queries! {
         cache_on_disk_if { true }
     }
 
+    /// A list of types where the ADT requires async drop if and only if any of
+    /// those types require async drop. If the ADT is known to always need async drop
+    /// then `Err(AlwaysRequiresDrop)` is returned.
+    query adt_async_drop_tys(def_id: DefId) -> Result<&'tcx ty::List<Ty<'tcx>>, AlwaysRequiresDrop> {
+        desc { |tcx| "computing when `{}` needs async drop", tcx.def_path_str(def_id) }
+        cache_on_disk_if { true }
+    }
+
     /// A list of types where the ADT requires drop if and only if any of those types
     /// has significant drop. A type marked with the attribute `rustc_insignificant_dtor`
     /// is considered to not be significant. A drop is significant if it is implemented
diff --git a/compiler/rustc_middle/src/ty/context.rs b/compiler/rustc_middle/src/ty/context.rs
index e8dad1e056c..23ee6514622 100644
--- a/compiler/rustc_middle/src/ty/context.rs
+++ b/compiler/rustc_middle/src/ty/context.rs
@@ -717,7 +717,6 @@ macro_rules! bidirectional_lang_item_map {
 
 bidirectional_lang_item_map! {
 // tidy-alphabetical-start
-    AsyncDestruct,
     AsyncFn,
     AsyncFnKindHelper,
     AsyncFnKindUpvars,
@@ -1715,6 +1714,10 @@ impl<'tcx> TyCtxt<'tcx> {
         self.coroutine_kind(def_id).is_some()
     }
 
+    pub fn is_async_drop_in_place_coroutine(self, def_id: DefId) -> bool {
+        self.is_lang_item(self.parent(def_id), LangItem::AsyncDropInPlace)
+    }
+
     /// Returns the movability of the coroutine of `def_id`, or panics
     /// if given a `def_id` that is not a coroutine.
     pub fn coroutine_movability(self, def_id: DefId) -> hir::Movability {
diff --git a/compiler/rustc_middle/src/ty/instance.rs b/compiler/rustc_middle/src/ty/instance.rs
index faad0a82acb..d660e7d0d60 100644
--- a/compiler/rustc_middle/src/ty/instance.rs
+++ b/compiler/rustc_middle/src/ty/instance.rs
@@ -147,6 +147,9 @@ pub enum InstanceKind<'tcx> {
     /// native support.
     ThreadLocalShim(DefId),
 
+    /// Proxy shim for async drop of future (def_id, proxy_cor_ty, impl_cor_ty)
+    FutureDropPollShim(DefId, Ty<'tcx>, Ty<'tcx>),
+
     /// `core::ptr::drop_in_place::<T>`.
     ///
     /// The `DefId` is for `core::ptr::drop_in_place`.
@@ -173,7 +176,13 @@ pub enum InstanceKind<'tcx> {
     ///
     /// The `DefId` is for `core::future::async_drop::async_drop_in_place`, the `Ty`
     /// is the type `T`.
-    AsyncDropGlueCtorShim(DefId, Option<Ty<'tcx>>),
+    AsyncDropGlueCtorShim(DefId, Ty<'tcx>),
+
+    /// `core::future::async_drop::async_drop_in_place::<'_, T>::{closure}`.
+    ///
+    /// async_drop_in_place poll function implementation (for generated coroutine).
+    /// `Ty` here is `async_drop_in_place<T>::{closure}` coroutine type, not just `T`
+    AsyncDropGlue(DefId, Ty<'tcx>),
 }
 
 impl<'tcx> Instance<'tcx> {
@@ -221,7 +230,9 @@ impl<'tcx> Instance<'tcx> {
                 .upstream_monomorphizations_for(def)
                 .and_then(|monos| monos.get(&self.args).cloned()),
             InstanceKind::DropGlue(_, Some(_)) => tcx.upstream_drop_glue_for(self.args),
-            InstanceKind::AsyncDropGlueCtorShim(_, Some(_)) => {
+            InstanceKind::AsyncDropGlue(_, _) => None,
+            InstanceKind::FutureDropPollShim(_, _, _) => None,
+            InstanceKind::AsyncDropGlueCtorShim(_, _) => {
                 tcx.upstream_async_drop_glue_for(self.args)
             }
             _ => None,
@@ -248,6 +259,8 @@ impl<'tcx> InstanceKind<'tcx> {
             | InstanceKind::DropGlue(def_id, _)
             | InstanceKind::CloneShim(def_id, _)
             | InstanceKind::FnPtrAddrShim(def_id, _)
+            | InstanceKind::FutureDropPollShim(def_id, _, _)
+            | InstanceKind::AsyncDropGlue(def_id, _)
             | InstanceKind::AsyncDropGlueCtorShim(def_id, _) => def_id,
         }
     }
@@ -257,7 +270,9 @@ impl<'tcx> InstanceKind<'tcx> {
         match self {
             ty::InstanceKind::Item(def) => Some(def),
             ty::InstanceKind::DropGlue(def_id, Some(_))
-            | InstanceKind::AsyncDropGlueCtorShim(def_id, Some(_))
+            | InstanceKind::AsyncDropGlueCtorShim(def_id, _)
+            | InstanceKind::AsyncDropGlue(def_id, _)
+            | InstanceKind::FutureDropPollShim(def_id, ..)
             | InstanceKind::ThreadLocalShim(def_id) => Some(def_id),
             InstanceKind::VTableShim(..)
             | InstanceKind::ReifyShim(..)
@@ -267,7 +282,6 @@ impl<'tcx> InstanceKind<'tcx> {
             | InstanceKind::ClosureOnceShim { .. }
             | ty::InstanceKind::ConstructCoroutineInClosureShim { .. }
             | InstanceKind::DropGlue(..)
-            | InstanceKind::AsyncDropGlueCtorShim(..)
             | InstanceKind::CloneShim(..)
             | InstanceKind::FnPtrAddrShim(..) => None,
         }
@@ -292,7 +306,9 @@ impl<'tcx> InstanceKind<'tcx> {
         let def_id = match *self {
             ty::InstanceKind::Item(def) => def,
             ty::InstanceKind::DropGlue(_, Some(_)) => return false,
-            ty::InstanceKind::AsyncDropGlueCtorShim(_, Some(_)) => return false,
+            ty::InstanceKind::AsyncDropGlueCtorShim(_, ty) => return ty.is_coroutine(),
+            ty::InstanceKind::FutureDropPollShim(_, _, _) => return false,
+            ty::InstanceKind::AsyncDropGlue(_, _) => return false,
             ty::InstanceKind::ThreadLocalShim(_) => return false,
             _ => return true,
         };
@@ -325,11 +341,12 @@ impl<'tcx> InstanceKind<'tcx> {
             | InstanceKind::FnPtrAddrShim(..)
             | InstanceKind::FnPtrShim(..)
             | InstanceKind::DropGlue(_, Some(_))
-            | InstanceKind::AsyncDropGlueCtorShim(_, Some(_)) => false,
+            | InstanceKind::FutureDropPollShim(..)
+            | InstanceKind::AsyncDropGlue(_, _) => false,
+            InstanceKind::AsyncDropGlueCtorShim(_, _) => false,
             InstanceKind::ClosureOnceShim { .. }
             | InstanceKind::ConstructCoroutineInClosureShim { .. }
             | InstanceKind::DropGlue(..)
-            | InstanceKind::AsyncDropGlueCtorShim(..)
             | InstanceKind::Item(_)
             | InstanceKind::Intrinsic(..)
             | InstanceKind::ReifyShim(..)
@@ -406,8 +423,11 @@ pub fn fmt_instance(
         InstanceKind::DropGlue(_, Some(ty)) => write!(f, " - shim(Some({ty}))"),
         InstanceKind::CloneShim(_, ty) => write!(f, " - shim({ty})"),
         InstanceKind::FnPtrAddrShim(_, ty) => write!(f, " - shim({ty})"),
-        InstanceKind::AsyncDropGlueCtorShim(_, None) => write!(f, " - shim(None)"),
-        InstanceKind::AsyncDropGlueCtorShim(_, Some(ty)) => write!(f, " - shim(Some({ty}))"),
+        InstanceKind::FutureDropPollShim(_, proxy_ty, impl_ty) => {
+            write!(f, " - dropshim({proxy_ty}-{impl_ty})")
+        }
+        InstanceKind::AsyncDropGlue(_, ty) => write!(f, " - shim({ty})"),
+        InstanceKind::AsyncDropGlueCtorShim(_, ty) => write!(f, " - shim(Some({ty}))"),
     }
 }
 
@@ -425,6 +445,51 @@ impl<'tcx> fmt::Display for Instance<'tcx> {
     }
 }
 
+// async_drop_in_place<T>::coroutine.poll, when T is a standart coroutine,
+// should be resolved to this coroutine's future_drop_poll (through FutureDropPollShim proxy).
+// async_drop_in_place<async_drop_in_place<T>::coroutine>::coroutine.poll,
+// when T is a standart coroutine, should be resolved to this coroutine's future_drop_poll.
+// async_drop_in_place<async_drop_in_place<T>::coroutine>::coroutine.poll,
+// when T is not a coroutine, should be resolved to the innermost
+// async_drop_in_place<T>::coroutine's poll function (through FutureDropPollShim proxy)
+fn resolve_async_drop_poll<'tcx>(mut cor_ty: Ty<'tcx>) -> Instance<'tcx> {
+    let first_cor = cor_ty;
+    let ty::Coroutine(poll_def_id, proxy_args) = first_cor.kind() else {
+        bug!();
+    };
+    let poll_def_id = *poll_def_id;
+    let mut child_ty = cor_ty;
+    loop {
+        if let ty::Coroutine(child_def, child_args) = child_ty.kind() {
+            cor_ty = child_ty;
+            if *child_def == poll_def_id {
+                child_ty = child_args.first().unwrap().expect_ty();
+                continue;
+            } else {
+                return Instance {
+                    def: ty::InstanceKind::FutureDropPollShim(poll_def_id, first_cor, cor_ty),
+                    args: proxy_args,
+                };
+            }
+        } else {
+            let ty::Coroutine(_, child_args) = cor_ty.kind() else {
+                bug!();
+            };
+            if first_cor != cor_ty {
+                return Instance {
+                    def: ty::InstanceKind::FutureDropPollShim(poll_def_id, first_cor, cor_ty),
+                    args: proxy_args,
+                };
+            } else {
+                return Instance {
+                    def: ty::InstanceKind::AsyncDropGlue(poll_def_id, cor_ty),
+                    args: child_args,
+                };
+            }
+        }
+    }
+}
+
 impl<'tcx> Instance<'tcx> {
     pub fn new(def_id: DefId, args: GenericArgsRef<'tcx>) -> Instance<'tcx> {
         assert!(
@@ -736,6 +801,15 @@ impl<'tcx> Instance<'tcx> {
         )
     }
 
+    pub fn resolve_async_drop_in_place_poll(
+        tcx: TyCtxt<'tcx>,
+        def_id: DefId,
+        ty: Ty<'tcx>,
+    ) -> ty::Instance<'tcx> {
+        let args = tcx.mk_args(&[ty.into()]);
+        Instance::expect_resolve(tcx, ty::TypingEnv::fully_monomorphized(), def_id, args, DUMMY_SP)
+    }
+
     #[instrument(level = "debug", skip(tcx), ret)]
     pub fn fn_once_adapter_instance(
         tcx: TyCtxt<'tcx>,
@@ -800,6 +874,9 @@ impl<'tcx> Instance<'tcx> {
         };
 
         if tcx.is_lang_item(trait_item_id, coroutine_callable_item) {
+            if tcx.is_async_drop_in_place_coroutine(coroutine_def_id) {
+                return Some(resolve_async_drop_poll(rcvr_args.type_at(0)));
+            }
             let ty::Coroutine(_, id_args) = *tcx.type_of(coroutine_def_id).skip_binder().kind()
             else {
                 bug!()
diff --git a/compiler/rustc_middle/src/ty/mod.rs b/compiler/rustc_middle/src/ty/mod.rs
index be00c0e116d..eba6d61ba7d 100644
--- a/compiler/rustc_middle/src/ty/mod.rs
+++ b/compiler/rustc_middle/src/ty/mod.rs
@@ -17,7 +17,7 @@ use std::hash::{Hash, Hasher};
 use std::marker::PhantomData;
 use std::num::NonZero;
 use std::ptr::NonNull;
-use std::{fmt, str};
+use std::{fmt, iter, str};
 
 pub use adt::*;
 pub use assoc::*;
@@ -39,6 +39,7 @@ use rustc_hir::LangItem;
 use rustc_hir::def::{CtorKind, CtorOf, DefKind, DocLinkResMap, LifetimeRes, Res};
 use rustc_hir::def_id::{CrateNum, DefId, DefIdMap, LocalDefId, LocalDefIdMap};
 use rustc_index::IndexVec;
+use rustc_index::bit_set::BitMatrix;
 use rustc_macros::{
     Decodable, Encodable, HashStable, TyDecodable, TyEncodable, TypeFoldable, TypeVisitable,
     extension,
@@ -103,7 +104,7 @@ pub use self::visit::*;
 use crate::error::{OpaqueHiddenTypeMismatch, TypeMismatchReason};
 use crate::metadata::ModChild;
 use crate::middle::privacy::EffectiveVisibilities;
-use crate::mir::{Body, CoroutineLayout};
+use crate::mir::{Body, CoroutineLayout, CoroutineSavedLocal, SourceInfo};
 use crate::query::{IntoQueryParam, Providers};
 use crate::ty;
 use crate::ty::codec::{TyDecoder, TyEncoder};
@@ -1739,11 +1740,13 @@ impl<'tcx> TyCtxt<'tcx> {
             | ty::InstanceKind::Virtual(..)
             | ty::InstanceKind::ClosureOnceShim { .. }
             | ty::InstanceKind::ConstructCoroutineInClosureShim { .. }
+            | ty::InstanceKind::FutureDropPollShim(..)
             | ty::InstanceKind::DropGlue(..)
             | ty::InstanceKind::CloneShim(..)
             | ty::InstanceKind::ThreadLocalShim(..)
             | ty::InstanceKind::FnPtrAddrShim(..)
-            | ty::InstanceKind::AsyncDropGlueCtorShim(..) => self.mir_shims(instance),
+            | ty::InstanceKind::AsyncDropGlueCtorShim(..)
+            | ty::InstanceKind::AsyncDropGlue(..) => self.mir_shims(instance),
         }
     }
 
@@ -1859,12 +1862,12 @@ impl<'tcx> TyCtxt<'tcx> {
         self.def_kind(trait_def_id) == DefKind::TraitAlias
     }
 
-    /// Returns layout of a coroutine. Layout might be unavailable if the
+    /// Returns layout of a non-async-drop coroutine. Layout might be unavailable if the
     /// coroutine is tainted by errors.
     ///
     /// Takes `coroutine_kind` which can be acquired from the `CoroutineArgs::kind_ty`,
     /// e.g. `args.as_coroutine().kind_ty()`.
-    pub fn coroutine_layout(
+    fn ordinary_coroutine_layout(
         self,
         def_id: DefId,
         coroutine_kind_ty: Ty<'tcx>,
@@ -1898,6 +1901,51 @@ impl<'tcx> TyCtxt<'tcx> {
         }
     }
 
+    /// Returns layout of a `async_drop_in_place::{closure}` coroutine
+    ///   (returned from `async fn async_drop_in_place<T>(..)`).
+    /// Layout might be unavailable if the coroutine is tainted by errors.
+    fn async_drop_coroutine_layout(
+        self,
+        def_id: DefId,
+        args: GenericArgsRef<'tcx>,
+    ) -> Option<&'tcx CoroutineLayout<'tcx>> {
+        let instance = InstanceKind::AsyncDropGlue(def_id, Ty::new_coroutine(self, def_id, args));
+        self.mir_shims(instance).coroutine_layout_raw()
+    }
+
+    /// Returns layout of a coroutine. Layout might be unavailable if the
+    /// coroutine is tainted by errors.
+    pub fn coroutine_layout(
+        self,
+        def_id: DefId,
+        args: GenericArgsRef<'tcx>,
+    ) -> Option<&'tcx CoroutineLayout<'tcx>> {
+        if self.is_async_drop_in_place_coroutine(def_id) {
+            // layout of `async_drop_in_place<T>::{closure}` in case,
+            // when T is a coroutine, contains this internal coroutine's ptr in upvars
+            // and doesn't require any locals. Here is an `empty coroutine's layout`
+            let arg_cor_ty = args.first().unwrap().expect_ty();
+            if arg_cor_ty.is_coroutine() {
+                let span = self.def_span(def_id);
+                let source_info = SourceInfo::outermost(span);
+                let variant_fields: IndexVec<VariantIdx, IndexVec<FieldIdx, CoroutineSavedLocal>> =
+                    iter::repeat(IndexVec::new()).take(CoroutineArgs::RESERVED_VARIANTS).collect();
+                let proxy_layout = CoroutineLayout {
+                    field_tys: [].into(),
+                    field_names: [].into(),
+                    variant_fields,
+                    variant_source_info: [source_info].into(),
+                    storage_conflicts: BitMatrix::new(0, 0),
+                };
+                return Some(self.arena.alloc(proxy_layout));
+            } else {
+                self.async_drop_coroutine_layout(def_id, args)
+            }
+        } else {
+            self.ordinary_coroutine_layout(def_id, args.as_coroutine().kind_ty())
+        }
+    }
+
     /// Given the `DefId` of an impl, returns the `DefId` of the trait it implements.
     /// If it implements no trait, returns `None`.
     pub fn trait_id_of_impl(self, def_id: DefId) -> Option<DefId> {
diff --git a/compiler/rustc_middle/src/ty/sty.rs b/compiler/rustc_middle/src/ty/sty.rs
index affb7b31ae1..b6043bc740f 100644
--- a/compiler/rustc_middle/src/ty/sty.rs
+++ b/compiler/rustc_middle/src/ty/sty.rs
@@ -4,7 +4,6 @@
 
 use std::assert_matches::debug_assert_matches;
 use std::borrow::Cow;
-use std::iter;
 use std::ops::{ControlFlow, Range};
 
 use hir::def::{CtorKind, DefKind};
@@ -19,7 +18,7 @@ use rustc_type_ir::TyKind::*;
 use rustc_type_ir::walk::TypeWalker;
 use rustc_type_ir::{self as ir, BoundVar, CollectAndApply, DynKind, TypeVisitableExt, elaborate};
 use tracing::instrument;
-use ty::util::{AsyncDropGlueMorphology, IntTypeExt};
+use ty::util::IntTypeExt;
 
 use super::GenericParamDefKind;
 use crate::infer::canonical::Canonical;
@@ -78,8 +77,7 @@ impl<'tcx> ty::CoroutineArgs<TyCtxt<'tcx>> {
     #[inline]
     fn variant_range(&self, def_id: DefId, tcx: TyCtxt<'tcx>) -> Range<VariantIdx> {
         // FIXME requires optimized MIR
-        FIRST_VARIANT
-            ..tcx.coroutine_layout(def_id, tcx.types.unit).unwrap().variant_fields.next_index()
+        FIRST_VARIANT..tcx.coroutine_layout(def_id, self.args).unwrap().variant_fields.next_index()
     }
 
     /// The discriminant for the given variant. Panics if the `variant_index` is
@@ -139,10 +137,14 @@ impl<'tcx> ty::CoroutineArgs<TyCtxt<'tcx>> {
         def_id: DefId,
         tcx: TyCtxt<'tcx>,
     ) -> impl Iterator<Item: Iterator<Item = Ty<'tcx>>> {
-        let layout = tcx.coroutine_layout(def_id, self.kind_ty()).unwrap();
+        let layout = tcx.coroutine_layout(def_id, self.args).unwrap();
         layout.variant_fields.iter().map(move |variant| {
             variant.iter().map(move |field| {
-                ty::EarlyBinder::bind(layout.field_tys[*field].ty).instantiate(tcx, self.args)
+                if tcx.is_async_drop_in_place_coroutine(def_id) {
+                    layout.field_tys[*field].ty
+                } else {
+                    ty::EarlyBinder::bind(layout.field_tys[*field].ty).instantiate(tcx, self.args)
+                }
             })
         })
     }
@@ -1046,10 +1048,6 @@ impl<'tcx> rustc_type_ir::inherent::Ty<TyCtxt<'tcx>> for Ty<'tcx> {
         self.discriminant_ty(interner)
     }
 
-    fn async_destructor_ty(self, interner: TyCtxt<'tcx>) -> Ty<'tcx> {
-        self.async_destructor_ty(interner)
-    }
-
     fn has_unsafe_fields(self) -> bool {
         Ty::has_unsafe_fields(self)
     }
@@ -1419,6 +1417,34 @@ impl<'tcx> Ty<'tcx> {
         cf.is_break()
     }
 
+    /// Returns the deepest `async_drop_in_place::{closure}` implementation.
+    ///
+    /// `async_drop_in_place<T>::{closure}`, when T is a coroutine, is a proxy-impl
+    /// to call async drop poll from impl coroutine.
+    pub fn find_async_drop_impl_coroutine<F: FnMut(Ty<'tcx>)>(
+        self,
+        tcx: TyCtxt<'tcx>,
+        mut f: F,
+    ) -> Ty<'tcx> {
+        assert!(self.is_coroutine());
+        let mut cor_ty = self;
+        let mut ty = cor_ty;
+        loop {
+            if let ty::Coroutine(def_id, args) = ty.kind() {
+                cor_ty = ty;
+                f(ty);
+                if tcx.is_async_drop_in_place_coroutine(*def_id) {
+                    ty = args.first().unwrap().expect_ty();
+                    continue;
+                } else {
+                    return cor_ty;
+                }
+            } else {
+                return cor_ty;
+            }
+        }
+    }
+
     /// Returns the type and mutability of `*ty`.
     ///
     /// The parameter `explicit` indicates if this is an *explicit* dereference.
@@ -1560,125 +1586,6 @@ impl<'tcx> Ty<'tcx> {
         }
     }
 
-    /// Returns the type of the async destructor of this type.
-    pub fn async_destructor_ty(self, tcx: TyCtxt<'tcx>) -> Ty<'tcx> {
-        match self.async_drop_glue_morphology(tcx) {
-            AsyncDropGlueMorphology::Noop => {
-                return Ty::async_destructor_combinator(tcx, LangItem::AsyncDropNoop)
-                    .instantiate_identity();
-            }
-            AsyncDropGlueMorphology::DeferredDropInPlace => {
-                let drop_in_place =
-                    Ty::async_destructor_combinator(tcx, LangItem::AsyncDropDeferredDropInPlace)
-                        .instantiate(tcx, &[self.into()]);
-                return Ty::async_destructor_combinator(tcx, LangItem::AsyncDropFuse)
-                    .instantiate(tcx, &[drop_in_place.into()]);
-            }
-            AsyncDropGlueMorphology::Custom => (),
-        }
-
-        match *self.kind() {
-            ty::Param(_) | ty::Alias(..) | ty::Infer(ty::TyVar(_)) => {
-                let assoc_items = tcx
-                    .associated_item_def_ids(tcx.require_lang_item(LangItem::AsyncDestruct, None));
-                Ty::new_projection(tcx, assoc_items[0], [self])
-            }
-
-            ty::Array(elem_ty, _) | ty::Slice(elem_ty) => {
-                let dtor = Ty::async_destructor_combinator(tcx, LangItem::AsyncDropSlice)
-                    .instantiate(tcx, &[elem_ty.into()]);
-                Ty::async_destructor_combinator(tcx, LangItem::AsyncDropFuse)
-                    .instantiate(tcx, &[dtor.into()])
-            }
-
-            ty::Adt(adt_def, args) if adt_def.is_enum() || adt_def.is_struct() => self
-                .adt_async_destructor_ty(
-                    tcx,
-                    adt_def.variants().iter().map(|v| v.fields.iter().map(|f| f.ty(tcx, args))),
-                ),
-            ty::Tuple(tys) => self.adt_async_destructor_ty(tcx, iter::once(tys)),
-            ty::Closure(_, args) => {
-                self.adt_async_destructor_ty(tcx, iter::once(args.as_closure().upvar_tys()))
-            }
-            ty::CoroutineClosure(_, args) => self
-                .adt_async_destructor_ty(tcx, iter::once(args.as_coroutine_closure().upvar_tys())),
-
-            ty::Adt(adt_def, _) => {
-                assert!(adt_def.is_union());
-
-                let surface_drop = self.surface_async_dropper_ty(tcx).unwrap();
-
-                Ty::async_destructor_combinator(tcx, LangItem::AsyncDropFuse)
-                    .instantiate(tcx, &[surface_drop.into()])
-            }
-
-            ty::Bound(..)
-            | ty::Foreign(_)
-            | ty::Placeholder(_)
-            | ty::Infer(ty::FreshTy(_) | ty::FreshIntTy(_) | ty::FreshFloatTy(_)) => {
-                bug!("`async_destructor_ty` applied to unexpected type: {self:?}")
-            }
-
-            _ => bug!("`async_destructor_ty` is not yet implemented for type: {self:?}"),
-        }
-    }
-
-    fn adt_async_destructor_ty<I>(self, tcx: TyCtxt<'tcx>, variants: I) -> Ty<'tcx>
-    where
-        I: Iterator + ExactSizeIterator,
-        I::Item: IntoIterator<Item = Ty<'tcx>>,
-    {
-        debug_assert_eq!(self.async_drop_glue_morphology(tcx), AsyncDropGlueMorphology::Custom);
-
-        let defer = Ty::async_destructor_combinator(tcx, LangItem::AsyncDropDefer);
-        let chain = Ty::async_destructor_combinator(tcx, LangItem::AsyncDropChain);
-
-        let noop =
-            Ty::async_destructor_combinator(tcx, LangItem::AsyncDropNoop).instantiate_identity();
-        let either = Ty::async_destructor_combinator(tcx, LangItem::AsyncDropEither);
-
-        let variants_dtor = variants
-            .into_iter()
-            .map(|variant| {
-                variant
-                    .into_iter()
-                    .map(|ty| defer.instantiate(tcx, &[ty.into()]))
-                    .reduce(|acc, next| chain.instantiate(tcx, &[acc.into(), next.into()]))
-                    .unwrap_or(noop)
-            })
-            .reduce(|other, matched| {
-                either.instantiate(tcx, &[other.into(), matched.into(), self.into()])
-            })
-            .unwrap();
-
-        let dtor = if let Some(dropper_ty) = self.surface_async_dropper_ty(tcx) {
-            Ty::async_destructor_combinator(tcx, LangItem::AsyncDropChain)
-                .instantiate(tcx, &[dropper_ty.into(), variants_dtor.into()])
-        } else {
-            variants_dtor
-        };
-
-        Ty::async_destructor_combinator(tcx, LangItem::AsyncDropFuse)
-            .instantiate(tcx, &[dtor.into()])
-    }
-
-    fn surface_async_dropper_ty(self, tcx: TyCtxt<'tcx>) -> Option<Ty<'tcx>> {
-        let adt_def = self.ty_adt_def()?;
-        let dropper = adt_def
-            .async_destructor(tcx)
-            .map(|_| LangItem::SurfaceAsyncDropInPlace)
-            .or_else(|| adt_def.destructor(tcx).map(|_| LangItem::AsyncDropSurfaceDropInPlace))?;
-        Some(Ty::async_destructor_combinator(tcx, dropper).instantiate(tcx, &[self.into()]))
-    }
-
-    fn async_destructor_combinator(
-        tcx: TyCtxt<'tcx>,
-        lang_item: LangItem,
-    ) -> ty::EarlyBinder<'tcx, Ty<'tcx>> {
-        tcx.fn_sig(tcx.require_lang_item(lang_item, None))
-            .map_bound(|fn_sig| fn_sig.output().no_bound_vars().unwrap())
-    }
-
     /// Returns the type of metadata for (potentially wide) pointers to this type,
     /// or the struct tail if the metadata type cannot be determined.
     pub fn ptr_metadata_ty_or_tail(
@@ -2020,6 +1927,13 @@ impl<'tcx> Ty<'tcx> {
         }
     }
 
+    pub fn is_async_drop_in_place_coroutine(self, tcx: TyCtxt<'_>) -> bool {
+        match self.kind() {
+            ty::Coroutine(def, ..) => tcx.is_async_drop_in_place_coroutine(*def),
+            _ => false,
+        }
+    }
+
     /// Returns `true` when the outermost type cannot be further normalized,
     /// resolved, or instantiated. This includes all primitive types, but also
     /// things like ADTs and trait objects, since even if their arguments or
diff --git a/compiler/rustc_middle/src/ty/util.rs b/compiler/rustc_middle/src/ty/util.rs
index 08cee1101d0..f46a1d4b22e 100644
--- a/compiler/rustc_middle/src/ty/util.rs
+++ b/compiler/rustc_middle/src/ty/util.rs
@@ -468,25 +468,6 @@ impl<'tcx> TyCtxt<'tcx> {
         Some(ty::AsyncDestructor { impl_did: dtor_candidate? })
     }
 
-    /// Returns async drop glue morphology for a definition. To get async drop
-    /// glue morphology for a type see [`Ty::async_drop_glue_morphology`].
-    //
-    // FIXME: consider making this a query
-    pub fn async_drop_glue_morphology(self, did: DefId) -> AsyncDropGlueMorphology {
-        let ty: Ty<'tcx> = self.type_of(did).instantiate_identity();
-
-        // Async drop glue morphology is an internal detail, so
-        // using `TypingMode::PostAnalysis` probably should be fine.
-        let typing_env = ty::TypingEnv::fully_monomorphized();
-        if ty.needs_async_drop(self, typing_env) {
-            AsyncDropGlueMorphology::Custom
-        } else if ty.needs_drop(self, typing_env) {
-            AsyncDropGlueMorphology::DeferredDropInPlace
-        } else {
-            AsyncDropGlueMorphology::Noop
-        }
-    }
-
     /// Returns the set of types that are required to be alive in
     /// order to run the destructor of `def` (see RFCs 769 and
     /// 1238).
@@ -1114,18 +1095,6 @@ impl<'tcx> TypeFolder<TyCtxt<'tcx>> for WeakAliasTypeExpander<'tcx> {
     }
 }
 
-/// Indicates the form of `AsyncDestruct::Destructor`. Used to simplify async
-/// drop glue for types not using async drop.
-#[derive(Clone, Copy, PartialEq, Eq, Debug)]
-pub enum AsyncDropGlueMorphology {
-    /// Async destructor simply does nothing
-    Noop,
-    /// Async destructor simply runs `drop_in_place`
-    DeferredDropInPlace,
-    /// Async destructor has custom logic
-    Custom,
-}
-
 impl<'tcx> Ty<'tcx> {
     /// Returns the `Size` for primitive types (bool, uint, int, char, float).
     pub fn primitive_size(self, tcx: TyCtxt<'tcx>) -> Size {
@@ -1295,16 +1264,17 @@ impl<'tcx> Ty<'tcx> {
         }
     }
 
-    /// Get morphology of the async drop glue, needed for types which do not
-    /// use async drop. To get async drop glue morphology for a definition see
-    /// [`TyCtxt::async_drop_glue_morphology`]. Used for `AsyncDestruct::Destructor`
-    /// type construction.
-    //
-    // FIXME: implement optimization to not instantiate a certain morphology of
-    // async drop glue too soon to allow per type optimizations, see array case
-    // for more info. Perhaps then remove this method and use `needs_(async_)drop`
-    // instead.
-    pub fn async_drop_glue_morphology(self, tcx: TyCtxt<'tcx>) -> AsyncDropGlueMorphology {
+    /// Checks whether values of this type `T` implement the `AsyncDrop` trait.
+    pub fn is_async_drop(self, tcx: TyCtxt<'tcx>, typing_env: ty::TypingEnv<'tcx>) -> bool {
+        !self.is_trivially_not_async_drop()
+            && tcx.is_async_drop_raw(typing_env.as_query_input(self))
+    }
+
+    /// Fast path helper for testing if a type is `AsyncDrop`.
+    ///
+    /// Returning true means the type is known to be `!AsyncDrop`. Returning
+    /// `false` means nothing -- could be `AsyncDrop`, might not be.
+    fn is_trivially_not_async_drop(self) -> bool {
         match self.kind() {
             ty::Int(_)
             | ty::Uint(_)
@@ -1316,46 +1286,26 @@ impl<'tcx> Ty<'tcx> {
             | ty::Ref(..)
             | ty::RawPtr(..)
             | ty::FnDef(..)
-            | ty::FnPtr(..)
-            | ty::Infer(ty::FreshIntTy(_))
-            | ty::Infer(ty::FreshFloatTy(_)) => AsyncDropGlueMorphology::Noop,
-
+            | ty::Error(_)
+            | ty::FnPtr(..) => true,
             // FIXME(unsafe_binders):
             ty::UnsafeBinder(_) => todo!(),
-
-            ty::Tuple(tys) if tys.is_empty() => AsyncDropGlueMorphology::Noop,
-            ty::Adt(adt_def, _) if adt_def.is_manually_drop() => AsyncDropGlueMorphology::Noop,
-
-            // Foreign types can never have destructors.
-            ty::Foreign(_) => AsyncDropGlueMorphology::Noop,
-
-            // FIXME: implement dynamic types async drops
-            ty::Error(_) | ty::Dynamic(..) => AsyncDropGlueMorphology::DeferredDropInPlace,
-
-            ty::Tuple(_) | ty::Array(_, _) | ty::Slice(_) => {
-                // Assume worst-case scenario, because we can instantiate async
-                // destructors in different orders:
-                //
-                // 1. Instantiate [T; N] with T = String and N = 0
-                // 2. Instantiate <[String; 0] as AsyncDestruct>::Destructor
-                //
-                // And viceversa, thus we cannot rely on String not using async
-                // drop or array having zero (0) elements
-                AsyncDropGlueMorphology::Custom
-            }
-            ty::Pat(ty, _) => ty.async_drop_glue_morphology(tcx),
-
-            ty::Adt(adt_def, _) => tcx.async_drop_glue_morphology(adt_def.did()),
-
-            ty::Closure(did, _)
-            | ty::CoroutineClosure(did, _)
-            | ty::Coroutine(did, _)
-            | ty::CoroutineWitness(did, _) => tcx.async_drop_glue_morphology(*did),
-
-            ty::Alias(..) | ty::Param(_) | ty::Bound(..) | ty::Placeholder(..) | ty::Infer(_) => {
-                // No specifics, but would usually mean forwarding async drop glue
-                AsyncDropGlueMorphology::Custom
+            ty::Tuple(fields) => fields.iter().all(Self::is_trivially_not_async_drop),
+            ty::Pat(elem_ty, _) | ty::Slice(elem_ty) | ty::Array(elem_ty, _) => {
+                elem_ty.is_trivially_not_async_drop()
             }
+            ty::Adt(..)
+            | ty::Bound(..)
+            | ty::Closure(..)
+            | ty::CoroutineClosure(..)
+            | ty::Dynamic(..)
+            | ty::Foreign(_)
+            | ty::Coroutine(..)
+            | ty::CoroutineWitness(..)
+            | ty::Infer(_)
+            | ty::Alias(..)
+            | ty::Param(_)
+            | ty::Placeholder(_) => false,
         }
     }
 
@@ -1401,9 +1351,6 @@ impl<'tcx> Ty<'tcx> {
     /// (Note that this implies that if `ty` has an async destructor attached,
     /// then `needs_async_drop` will definitely return `true` for `ty`.)
     ///
-    /// When constructing `AsyncDestruct::Destructor` type, use
-    /// [`Ty::async_drop_glue_morphology`] instead.
-    //
     // FIXME(zetanumbers): Note that this method is used to check eligible types
     // in unions.
     #[inline]