Auto merge of #106520 - ehuss:update-mdbook, r=Mark-Simulacrum

Update mdbook

This updates mdbook from 0.4.21 to 0.4.25. The list of changes is [here](https://github.com/rust-lang/mdBook/blob/master/CHANGELOG.md#mdbook-0425). The only user-visible changes are some changes around the theme picker, and change to the copy-to-clipboard ignoring hidden lines.

Internally there were some dependency updates and small fixes.

This also updates `clap` from 4.0.15 to 4.0.32 whose changelog is [here](https://github.com/clap-rs/clap/blob/master/CHANGELOG.md#4032---2022-12-22). This impacts tools like cargo. I don't see anything particularly noteworthy there, though there are some small user-visible changes.

Unfortunately this required adding a hack for building `rustix` with a bootstrap tool. The comment explains why. I am unable to think of some other workaround (or even a cleaner way to set the rustflag). Ideas are welcome if you can think of alternatives. I'm struggling to even think of a long-term solution, other than asking projects not to do auto-nightly feature detection.

One medium-term solution is to avoid the clap dependency for the mdbook library (which is how rustix gets pulled in). That is one of my goals for the 0.5 release of mdbook, but that probably won't happen until later this year. It would also require dropping clap from `rustbook` and using some other means to parse arguments (there's only two options, so it can probably be done manually).

5 files changed, 102 insertions, 18 deletions

diff --git a/src/bootstrap/bin/rustc.rs b/src/bootstrap/bin/rustc.rs
index 1d526022812..9611c866df5 100644
--- a/src/bootstrap/bin/rustc.rs
+++ b/src/bootstrap/bin/rustc.rs
@@ -154,6 +154,41 @@ fn main() {
         cmd.arg("-Z").arg("force-unstable-if-unmarked");
     }
 
+    // allow-features is handled from within this rustc wrapper because of
+    // issues with build scripts. Some packages use build scripts to
+    // dynamically detect if certain nightly features are available.
+    // There are different ways this causes problems:
+    //
+    // * rustix runs `rustc` on a small test program to see if the feature is
+    //   available (and sets a `cfg` if it is). It does not honor
+    //   CARGO_ENCODED_RUSTFLAGS.
+    // * proc-macro2 detects if `rustc -vV` says "nighty" or "dev" and enables
+    //   nightly features. It will scan CARGO_ENCODED_RUSTFLAGS for
+    //   -Zallow-features. Unfortunately CARGO_ENCODED_RUSTFLAGS is not set
+    //   for build-dependencies when --target is used.
+    //
+    // The issues above means we can't just use RUSTFLAGS, and we can't use
+    // `cargo -Zallow-features=…`. Passing it through here ensures that it
+    // always gets set. Unfortunately that also means we need to enable more
+    // features than we really want (like those for proc-macro2), but there
+    // isn't much of a way around it.
+    //
+    // I think it is unfortunate that build scripts are doing this at all,
+    // since changes to nightly features can cause crates to break even if the
+    // user didn't want or care about the use of the nightly features. I think
+    // nightly features should be opt-in only. Unfortunately the dynamic
+    // checks are now too wide spread that we just need to deal with it.
+    //
+    // If you want to try to remove this, I suggest working with the crate
+    // authors to remove the dynamic checking. Another option is to pursue
+    // https://github.com/rust-lang/cargo/issues/11244 and
+    // https://github.com/rust-lang/cargo/issues/4423, which will likely be
+    // very difficult, but could help expose -Zallow-features into build
+    // scripts so they could try to honor them.
+    if let Ok(allow_features) = env::var("RUSTC_ALLOW_FEATURES") {
+        cmd.arg(format!("-Zallow-features={allow_features}"));
+    }
+
     if let Ok(flags) = env::var("MAGIC_EXTRA_RUSTFLAGS") {
         for flag in flags.split(' ') {
             cmd.arg(flag);
diff --git a/src/bootstrap/builder.rs b/src/bootstrap/builder.rs
index 52a83fa29c7..b4fc1d4f28d 100644
--- a/src/bootstrap/builder.rs
+++ b/src/bootstrap/builder.rs
@@ -1381,18 +1381,29 @@ impl<'a> Builder<'a> {
         // this), as well as #63012 which is the tracking issue for this
         // feature on the rustc side.
         cargo.arg("-Zbinary-dep-depinfo");
-        match mode {
-            Mode::ToolBootstrap => {
-                // Restrict the allowed features to those passed by rustbuild, so we don't depend on nightly accidentally.
-                rustflags.arg("-Zallow-features=binary-dep-depinfo");
-            }
-            Mode::ToolStd => {
-                // Right now this is just compiletest and a few other tools that build on stable.
-                // Allow them to use `feature(test)`, but nothing else.
-                rustflags.arg("-Zallow-features=binary-dep-depinfo,test,proc_macro_internals,proc_macro_diagnostic,proc_macro_span");
+        let allow_features = match mode {
+            Mode::ToolBootstrap | Mode::ToolStd => {
+                // Restrict the allowed features so we don't depend on nightly
+                // accidentally.
+                //
+                // binary-dep-depinfo is used by rustbuild itself for all
+                // compilations.
+                //
+                // Lots of tools depend on proc_macro2 and proc-macro-error.
+                // Those have build scripts which assume nightly features are
+                // available if the `rustc` version is "nighty" or "dev". See
+                // bin/rustc.rs for why that is a problem. Instead of labeling
+                // those features for each individual tool that needs them,
+                // just blanket allow them here.
+                //
+                // If this is ever removed, be sure to add something else in
+                // its place to keep the restrictions in place (or make a way
+                // to unset RUSTC_BOOTSTRAP).
+                "binary-dep-depinfo,proc_macro_span,proc_macro_span_shrink,proc_macro_diagnostic"
+                    .to_string()
             }
-            Mode::Std | Mode::Rustc | Mode::Codegen | Mode::ToolRustc => {}
-        }
+            Mode::Std | Mode::Rustc | Mode::Codegen | Mode::ToolRustc => String::new(),
+        };
 
         cargo.arg("-j").arg(self.jobs().to_string());
 
@@ -1915,7 +1926,7 @@ impl<'a> Builder<'a> {
             }
         }
 
-        Cargo { command: cargo, rustflags, rustdocflags }
+        Cargo { command: cargo, rustflags, rustdocflags, allow_features }
     }
 
     /// Ensure that a given step is built, returning its output. This will
@@ -2094,6 +2105,7 @@ pub struct Cargo {
     command: Command,
     rustflags: Rustflags,
     rustdocflags: Rustflags,
+    allow_features: String,
 }
 
 impl Cargo {
@@ -2138,6 +2150,18 @@ impl Cargo {
         self.command.current_dir(dir);
         self
     }
+
+    /// Adds nightly-only features that this invocation is allowed to use.
+    ///
+    /// By default, all nightly features are allowed. Once this is called, it
+    /// will be restricted to the given set.
+    pub fn allow_features(&mut self, features: &str) -> &mut Cargo {
+        if !self.allow_features.is_empty() {
+            self.allow_features.push(',');
+        }
+        self.allow_features.push_str(features);
+        self
+    }
 }
 
 impl From<Cargo> for Command {
@@ -2152,6 +2176,10 @@ impl From<Cargo> for Command {
             cargo.command.env("RUSTDOCFLAGS", rustdocflags);
         }
 
+        if !cargo.allow_features.is_empty() {
+            cargo.command.env("RUSTC_ALLOW_FEATURES", cargo.allow_features);
+        }
+
         cargo.command
     }
 }
diff --git a/src/bootstrap/check.rs b/src/bootstrap/check.rs
index 2771bd2264c..4b8a58e87b6 100644
--- a/src/bootstrap/check.rs
+++ b/src/bootstrap/check.rs
@@ -346,9 +346,7 @@ impl Step for RustAnalyzer {
             &["rust-analyzer/in-rust-tree".to_owned()],
         );
 
-        cargo.rustflag(
-            "-Zallow-features=proc_macro_internals,proc_macro_diagnostic,proc_macro_span",
-        );
+        cargo.allow_features(crate::tool::RustAnalyzer::ALLOW_FEATURES);
 
         // For ./x.py clippy, don't check those targets because
         // linting tests and benchmarks can produce very noisy results
diff --git a/src/bootstrap/test.rs b/src/bootstrap/test.rs
index 159f2fba671..6078e39ac9d 100644
--- a/src/bootstrap/test.rs
+++ b/src/bootstrap/test.rs
@@ -378,6 +378,7 @@ impl Step for RustAnalyzer {
             SourceType::InTree,
             &["sysroot-abi".to_owned()],
         );
+        cargo.allow_features(tool::RustAnalyzer::ALLOW_FEATURES);
 
         let dir = builder.src.join(workspace_path);
         // needed by rust-analyzer to find its own text fixtures, cf.
@@ -690,7 +691,7 @@ impl Step for CompiletestTest {
         // We need `ToolStd` for the locally-built sysroot because
         // compiletest uses unstable features of the `test` crate.
         builder.ensure(compile::Std::new(compiler, host));
-        let cargo = tool::prepare_tool_cargo(
+        let mut cargo = tool::prepare_tool_cargo(
             builder,
             compiler,
             Mode::ToolStd,
@@ -700,6 +701,7 @@ impl Step for CompiletestTest {
             SourceType::InTree,
             &[],
         );
+        cargo.allow_features("test");
 
         try_run(builder, &mut cargo.into());
     }
diff --git a/src/bootstrap/tool.rs b/src/bootstrap/tool.rs
index 24b033cc0dc..9a2100c2fb7 100644
--- a/src/bootstrap/tool.rs
+++ b/src/bootstrap/tool.rs
@@ -29,6 +29,8 @@ struct ToolBuild {
     is_optional_tool: bool,
     source_type: SourceType,
     extra_features: Vec<String>,
+    /// Nightly-only features that are allowed (comma-separated list).
+    allow_features: &'static str,
 }
 
 impl Step for ToolBuild {
@@ -59,7 +61,7 @@ impl Step for ToolBuild {
             _ => panic!("unexpected Mode for tool build"),
         }
 
-        let cargo = prepare_tool_cargo(
+        let mut cargo = prepare_tool_cargo(
             builder,
             compiler,
             self.mode,
@@ -69,6 +71,9 @@ impl Step for ToolBuild {
             self.source_type,
             &self.extra_features,
         );
+        if !self.allow_features.is_empty() {
+            cargo.allow_features(self.allow_features);
+        }
 
         builder.info(&format!("Building stage{} tool {} ({})", compiler.stage, tool, target));
         let mut duplicates = Vec::new();
@@ -292,6 +297,7 @@ macro_rules! bootstrap_tool {
         $name:ident, $path:expr, $tool_name:expr
         $(,is_external_tool = $external:expr)*
         $(,is_unstable_tool = $unstable:expr)*
+        $(,allow_features = $allow_features:expr)?
         ;
     )+) => {
         #[derive(Copy, PartialEq, Eq, Clone)]
@@ -355,6 +361,7 @@ macro_rules! bootstrap_tool {
                         SourceType::InTree
                     },
                     extra_features: vec![],
+                    allow_features: concat!($($allow_features)*),
                 }).expect("expected to build -- essential tool")
             }
         }
@@ -368,7 +375,7 @@ bootstrap_tool!(
     Tidy, "src/tools/tidy", "tidy";
     Linkchecker, "src/tools/linkchecker", "linkchecker";
     CargoTest, "src/tools/cargotest", "cargotest";
-    Compiletest, "src/tools/compiletest", "compiletest", is_unstable_tool = true;
+    Compiletest, "src/tools/compiletest", "compiletest", is_unstable_tool = true, allow_features = "test";
     BuildManifest, "src/tools/build-manifest", "build-manifest";
     RemoteTestClient, "src/tools/remote-test-client", "remote-test-client";
     RustInstaller, "src/tools/rust-installer", "rust-installer", is_external_tool = true;
@@ -435,6 +442,7 @@ impl Step for ErrorIndex {
                 is_optional_tool: false,
                 source_type: SourceType::InTree,
                 extra_features: Vec::new(),
+                allow_features: "",
             })
             .expect("expected to build -- essential tool")
     }
@@ -471,6 +479,7 @@ impl Step for RemoteTestServer {
                 is_optional_tool: false,
                 source_type: SourceType::InTree,
                 extra_features: Vec::new(),
+                allow_features: "",
             })
             .expect("expected to build -- essential tool")
     }
@@ -622,6 +631,7 @@ impl Step for Cargo {
                 is_optional_tool: false,
                 source_type: SourceType::Submodule,
                 extra_features: Vec::new(),
+                allow_features: "",
             })
             .expect("expected to build -- essential tool");
 
@@ -637,6 +647,7 @@ impl Step for Cargo {
                 is_optional_tool: true,
                 source_type: SourceType::Submodule,
                 extra_features: Vec::new(),
+                allow_features: "",
             });
         };
 
@@ -684,6 +695,7 @@ impl Step for LldWrapper {
                 is_optional_tool: false,
                 source_type: SourceType::InTree,
                 extra_features: Vec::new(),
+                allow_features: "",
             })
             .expect("expected to build -- essential tool");
 
@@ -697,6 +709,11 @@ pub struct RustAnalyzer {
     pub target: TargetSelection,
 }
 
+impl RustAnalyzer {
+    pub const ALLOW_FEATURES: &str =
+        "proc_macro_internals,proc_macro_diagnostic,proc_macro_span,proc_macro_span_shrink";
+}
+
 impl Step for RustAnalyzer {
     type Output = Option<PathBuf>;
     const DEFAULT: bool = true;
@@ -731,6 +748,7 @@ impl Step for RustAnalyzer {
             extra_features: vec!["rust-analyzer/in-rust-tree".to_owned()],
             is_optional_tool: false,
             source_type: SourceType::InTree,
+            allow_features: RustAnalyzer::ALLOW_FEATURES,
         })
     }
 }
@@ -769,6 +787,7 @@ impl Step for RustAnalyzerProcMacroSrv {
             extra_features: vec!["proc-macro-srv/sysroot-abi".to_owned()],
             is_optional_tool: false,
             source_type: SourceType::InTree,
+            allow_features: RustAnalyzer::ALLOW_FEATURES,
         })?;
 
         // Copy `rust-analyzer-proc-macro-srv` to `<sysroot>/libexec/`
@@ -788,6 +807,7 @@ macro_rules! tool_extended {
        $tool_name:expr,
        stable = $stable:expr
        $(,tool_std = $tool_std:literal)?
+       $(,allow_features = $allow_features:expr)?
        ;)+) => {
         $(
             #[derive(Debug, Clone, Hash, PartialEq, Eq)]
@@ -839,6 +859,7 @@ macro_rules! tool_extended {
                     extra_features: $sel.extra_features,
                     is_optional_tool: true,
                     source_type: SourceType::InTree,
+                    allow_features: concat!($($allow_features)*),
                 })
             }
         }