std: Fix WASI implementation of `remove_dir_all`

This commit is a change to the WASI-specific implementation of the
`std::fs::remove_dir_all` function. Specifically it changes how
directory entries are read of a directory-being-deleted to specifically
buffer them all into a `Vec` before actually proceeding to delete
anything. This is necessary to fix an interaction with how the WASIp1
`fd_readdir` API works to have everything work out in the face of
mutations while reading a directory.

The basic problem is that `fd_readdir`, the WASIp1 API for reading
directories, is not a stateful read of a directory but instead a
"seekable" read of a directory. Its `cookie` argument enables seeking
anywhere within the directory at any time to read further entries.
Native host implementations do not have this ability, however, which
means that this seeking property must be achieved by re-reading the
directory. The problem with this is that WASIp1 has under-specified
semantics around what should happen if a directory is mutated between
two calls to `fd_readdir`. In essence there's not really any possible
implementation in hosts except to read the entire directory and support
seeking through the already-read list. This implementation is not
possible in the WASIp1-to-WASIp2 adapter that is primarily used to
create components for the `wasm32-wasip2` target where it has
constrained memory requirements and can't buffer up arbitrarily sized
directories.

The WASIp1 API definitions are effectively "dead" now at the standards
level meaning that `fd_readdir` won't be changing nor will a replacement
be coming. For the `wasm32-wasip2` target this will get fixed once
filesystem APIs are updated to use WASIp2 directly instead of WASIp1,
making this buffering unnecessary. In essence while this is a hack it's
sort of the least invasive thing that works everywhere for now. I don't
think this is viable to fix in hosts so guests compiled to wasm are
going to have to work around it by not relying on any guarantees about
what happens to a directory if it's mutated between reads.

1 files changed, 8 insertions, 1 deletions

diff --git a/library/std/src/sys/fs/wasi.rs b/library/std/src/sys/fs/wasi.rs
index b65d86de12a..0b65b9cb389 100644
--- a/library/std/src/sys/fs/wasi.rs
+++ b/library/std/src/sys/fs/wasi.rs
@@ -848,7 +848,14 @@ fn remove_dir_all_recursive(parent: &WasiFd, path: &Path) -> io::Result<()> {
 
     // Iterate over all the entries in this directory, and travel recursively if
     // necessary
-    for entry in ReadDir::new(fd, dummy_root) {
+    //
+    // Note that all directory entries for this directory are read first before
+    // any removal is done. This works around the fact that the WASIp1 API for
+    // reading directories is not well-designed for handling mutations between
+    // invocations of reading a directory. By reading all the entries at once
+    // this ensures that, at least without concurrent modifications, it should
+    // be possible to delete everything.
+    for entry in ReadDir::new(fd, dummy_root).collect::<Vec<_>>() {
         let entry = entry?;
         let path = crate::str::from_utf8(&entry.name).map_err(|_| {
             io::const_error!(io::ErrorKind::Uncategorized, "invalid utf-8 file name found")