Auto merge of #145807 - zachs18:only-consider-auto-traits-empty, r=compiler-errors

When determining if a trait has no entries for the purposes of omitting vptrs from subtrait vtables, consider its transitive supertraits' entries, instead of just its own entries.

When determining if a non-first supertrait vptr can be omitted from a subtrait vtable, check if the supertrait or any of its (transitive) supertraits have methods, instead of only checking if the supertrait itself has methods.

This fixes the soundness issue where a vptr would be omitted for a supertrait with no methods but that itself had a supertrait with methods, while still optimizing the case where the supertrait is "truly" empty (it has no own vtable entries, and none of its (transitive) supertraits have any own vtable entries).

Fixes <https://github.com/rust-lang/rust/issues/145752>

-----

Old description:

~~Treat all non-auto traits as non-empty (possibly having methods) for purposes of determining if we need to emit a vptr for a non-direct supertrait (and for new "sibling" entries after a direct or non-direct supertrait).~~

This fixes (I believe) the soundness issue, ~~but regresses vtable sizes and possibly upcasting perf in some cases when using trait hierarchies with empty non-auto traits (see `tests/ui/traits/vtable/multiple-markers.stderr`) since we use vptrs in some cases where we could re-use the vtable.~~

Fixes <https://github.com/rust-lang/rust/issues/145752>

Re-opens (not anymore) <https://github.com/rust-lang/rust/issues/114942>

Should not affect <https://github.com/rust-lang/rust/issues/131813> (i.e. the soundness issue is still fixed, ~~though the relevant vtables in the `trait Evil` example will be larger now~~)

cc implementation history <https://github.com/rust-lang/rust/pull/131864> <https://github.com/rust-lang/rust/pull/113856>

-----

~~It should be possible to check if a trait has any methods from itself *or* supertraits (instead of just from itself), but to fix the immediate soundness issue, just assume any non-auto trait could have methods. A more optimistic check can be implemented later (or if someone does it soon it could just supercede this PR :smile:).~~ Done in latest push

`@rustbot` label A-dyn-trait F-trait_upcasting

1 files changed, 6 insertions, 1 deletions

diff --git a/compiler/rustc_trait_selection/src/traits/vtable.rs b/compiler/rustc_trait_selection/src/traits/vtable.rs
index 3565c11249a..7e8a41457d4 100644
--- a/compiler/rustc_trait_selection/src/traits/vtable.rs
+++ b/compiler/rustc_trait_selection/src/traits/vtable.rs
@@ -153,7 +153,12 @@ fn prepare_vtable_segments_inner<'tcx, T>(
 
         // emit innermost item, move to next sibling and stop there if possible, otherwise jump to outer level.
         while let Some((inner_most_trait_ref, emit_vptr, mut siblings)) = stack.pop() {
-            let has_entries = has_own_existential_vtable_entries(tcx, inner_most_trait_ref.def_id);
+            // We don't need to emit a vptr for "truly-empty" supertraits, but we *do* need to emit a
+            // vptr for supertraits that have no methods, but that themselves have supertraits
+            // with methods, so we check if any transitive supertrait has entries here (this includes
+            // the trait itself).
+            let has_entries = ty::elaborate::supertrait_def_ids(tcx, inner_most_trait_ref.def_id)
+                .any(|def_id| has_own_existential_vtable_entries(tcx, def_id));
 
             segment_visitor(VtblSegment::TraitOwnEntries {
                 trait_ref: inner_most_trait_ref,