Auto merge of #82208 - jyn514:rustfmt-subtree, r=Mark-Simulacrum

Convert rustfmt from a submodule to a subtree

r? `@calebcartwright` cc `@Manishearth` `@Mark-Simulacrum`

The motivation is that submodule updates cause rustfmt to not be available on nightly a lot; most recently it was unavailable for over 10 days, causing the beta release to be delayed. Additionally this is much less work on the part of the rustfmt maintainers to keep the rustfmt compiling, since now people making breaking changes will be responsible for fixing them.

I kept the rustfmt git history so it looks like there are thousands of commits. The important commits are https://github.com/rust-lang/rust/compare/851dee3af9404bf399c3c4ffefe5105edb3debad~..pull/82208/head. This adds about 10 MB of git history, which is not terribly much compared to the 702 MB that already exist.

- Add `src/tools/rustfmt` to `x.py check`
- Fix CRLF issues with rustfmt tests (see commit for details)
- Use `rustc_private` instead of crates.io dependencies

  This was already switched upstream and would have landed in the next submodule bump anyway. This just updates Cargo.lock for rust-lang/rust.

- Add `yansi-term` to the list of allowed dependencies.

  This is a false positive - rustc doesn't actually use it, only rustfmt, but because it's activated by the cargo feature of a dependency, tidy gets confused. It's fairly innocuous in any case, it's used for color printing.
  This would have happened in the next submodule bump.

- Remove rustfmt from the list of toolstate tools.
- Give a hard error if testing or building rustfmt fails.
-  Update log to 0.4.14

   This avoids a warning about semicolons in macros; see the commit for details.

- Don't add tools to the sysroot when they finish building.

  This is the only change that could be considered a regression - this avoids a "colliding StableCrateId" error due to a bug in resolve (https://github.com/rust-lang/rust/issues/56935). The regression is that this rebuilds dependencies more often than strictly necessary. See the commit for details.

Fixes https://github.com/rust-lang/rust/issues/85226 (permanently). Closes https://github.com/rust-lang/rust/issues/82385. Helps with https://github.com/rust-lang/rust/issues/70651. Helps with https://github.com/rust-lang/rust/issues/80639.

5 files changed, 315 insertions, 0 deletions

diff --git a/src/tools/rustfmt/.github/workflows/integration.yml b/src/tools/rustfmt/.github/workflows/integration.yml
new file mode 100644
index 00000000000..b79221d0543
--- /dev/null
+++ b/src/tools/rustfmt/.github/workflows/integration.yml
@@ -0,0 +1,85 @@
+name: integration
+on:
+  push:
+    branches:
+      - master
+  pull_request:
+
+jobs:
+  integration-tests:
+    runs-on: ubuntu-latest
+    name: ${{ matrix.integration }}
+    strategy:
+      # https://help.github.com/en/actions/getting-started-with-github-actions/about-github-actions#usage-limits
+      # There's a limit of 60 concurrent jobs across all repos in the rust-lang organization.
+      # In order to prevent overusing too much of that 60 limit, we throttle the
+      # number of rustfmt jobs that will run concurrently.
+      max-parallel: 4
+      fail-fast: false
+      matrix:
+        integration: [
+          bitflags,
+          error-chain,
+          log,
+          mdbook,
+          packed_simd,
+          rust-semverver,
+          tempdir,
+          futures-rs,
+          rust-clippy,
+          failure,
+        ]
+        include:
+          # Allowed Failures
+          # Actions doesn't yet support explicitly marking matrix legs as allowed failures
+          # https://github.community/t5/GitHub-Actions/continue-on-error-allow-failure-UI-indication/td-p/37033
+          # https://github.community/t5/GitHub-Actions/Why-a-matrix-step-will-be-canceled-if-another-one-failed/td-p/30920
+          # Instead, leverage `continue-on-error`
+          # https://help.github.com/en/actions/automating-your-workflow-with-github-actions/workflow-syntax-for-github-actions#jobsjob_idstepscontinue-on-error
+          #
+          # Failing due to breaking changes in rustfmt 2.0 where empty
+          # match blocks have trailing commas removed
+          # https://github.com/rust-lang/rustfmt/pull/4226
+          - integration: chalk
+            allow-failure: true
+          - integration: crater
+            allow-failure: true
+          - integration: glob
+            allow-failure: true
+          - integration: stdsimd
+            allow-failure: true
+          # Using old rustfmt configuration option
+          - integration: rand
+            allow-failure: true
+          # Keep this as an allowed failure as it's fragile to breaking changes of rustc.
+          - integration: rust-clippy
+            allow-failure: true
+          # Using old rustfmt configuration option
+          - integration: packed_simd
+            allow-failure: true
+          # calebcartwright (2019-12-24)
+          # Keeping this as an allowed failure since it was flagged as such in the TravisCI config, even though
+          # it appears to have been passing for quite some time.
+          # Original comment was: temporal build failure due to breaking changes in the nightly compiler
+          - integration: rust-semverver
+            allow-failure: true
+          # Can be moved back to include section after https://github.com/rust-lang-nursery/failure/pull/298 is merged
+          - integration: failure
+            allow-failure: true
+
+    steps:
+    - name: checkout
+      uses: actions/checkout@v2
+
+      # Run build
+    - name: install rustup
+      run: |
+        curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs > rustup-init.sh
+        sh rustup-init.sh -y --default-toolchain none
+
+    - name: run integration tests
+      env:
+        INTEGRATION: ${{ matrix.integration }}
+        TARGET: x86_64-unknown-linux-gnu
+      run: ./ci/integration.sh
+      continue-on-error: ${{ matrix.allow-failure == true }}
diff --git a/src/tools/rustfmt/.github/workflows/linux.yml b/src/tools/rustfmt/.github/workflows/linux.yml
new file mode 100644
index 00000000000..6eaae69c708
--- /dev/null
+++ b/src/tools/rustfmt/.github/workflows/linux.yml
@@ -0,0 +1,42 @@
+name: linux
+on:
+  push:
+    branches:
+      - master
+  pull_request:
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    name: (${{ matrix.target }}, nightly)
+    strategy:
+      # https://help.github.com/en/actions/getting-started-with-github-actions/about-github-actions#usage-limits
+      # There's a limit of 60 concurrent jobs across all repos in the rust-lang organization.
+      # In order to prevent overusing too much of that 60 limit, we throttle the
+      # number of rustfmt jobs that will run concurrently.
+      max-parallel: 1
+      fail-fast: false
+      matrix:
+        target: [
+          x86_64-unknown-linux-gnu,
+        ]
+
+    steps:
+    - name: checkout
+      uses: actions/checkout@v2
+
+      # Run build
+    - name: install rustup
+      run: |
+        curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs > rustup-init.sh
+        sh rustup-init.sh -y --default-toolchain none
+        rustup target add ${{ matrix.target }}
+
+    - name: build
+      run: |
+        rustc -Vv
+        cargo -V
+        cargo build
+
+    - name: test
+      run: cargo test
diff --git a/src/tools/rustfmt/.github/workflows/mac.yml b/src/tools/rustfmt/.github/workflows/mac.yml
new file mode 100644
index 00000000000..79e4f69163e
--- /dev/null
+++ b/src/tools/rustfmt/.github/workflows/mac.yml
@@ -0,0 +1,39 @@
+name: mac
+on:
+  push:
+    branches:
+      - master
+  pull_request:
+
+jobs:
+  test:
+    # https://help.github.com/en/actions/automating-your-workflow-with-github-actions/virtual-environments-for-github-hosted-runners#supported-runners-and-hardware-resources
+    # macOS Catalina 10.15
+    runs-on: macos-latest
+    name: (${{ matrix.target }}, nightly)
+    strategy:
+      fail-fast: false
+      matrix:
+        target: [
+          x86_64-apple-darwin,
+        ]
+
+    steps:
+    - name: checkout
+      uses: actions/checkout@v2
+
+      # Run build
+    - name: install rustup
+      run: |
+        curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs > rustup-init.sh
+        sh rustup-init.sh -y --default-toolchain none
+        rustup target add ${{ matrix.target }}
+
+    - name: build
+      run: |
+        rustc -Vv
+        cargo -V
+        cargo build
+
+    - name: test
+      run: cargo test
diff --git a/src/tools/rustfmt/.github/workflows/upload-assets.yml b/src/tools/rustfmt/.github/workflows/upload-assets.yml
new file mode 100644
index 00000000000..9a5fd0dd1d3
--- /dev/null
+++ b/src/tools/rustfmt/.github/workflows/upload-assets.yml
@@ -0,0 +1,80 @@
+name: upload
+
+on:
+  release:
+    types: [created]
+
+jobs:
+  build-release:
+    name: build-release
+    strategy:
+      matrix:
+        build: [linux-x86_64, macos-x86_64, windows-x86_64-gnu, windows-x86_64-msvc]
+        include:
+          - build: linux-x86_64
+            os: ubuntu-latest
+            rust: nightly
+          - build: macos-x86_64
+            os: macos-latest
+            rust: nightly
+          - build: windows-x86_64-gnu
+            os: windows-latest
+            rust: nightly-x86_64-gnu
+          - build: windows-x86_64-msvc
+            os: windows-latest
+            rust: nightly-x86_64-msvc
+    runs-on: ${{ matrix.os }}
+    steps:
+      - uses: actions/checkout@v2
+
+      - name: Install Rust
+        uses: actions-rs/toolchain@v1
+        with:
+          profile: minimal
+          toolchain: ${{ matrix.rust }}
+          override: true
+
+      - name: Add mingw64 to path for x86_64-gnu
+        run: echo "C:\msys64\mingw64\bin" >> $GITHUB_PATH
+        if: matrix.rust == 'nightly-x86_64-gnu'
+        shell: bash
+
+      - name: Install cargo-make
+        uses: actions-rs/cargo@v1
+        with:
+          command: install
+          args: --force cargo-make
+
+      - name: Build release binaries
+        uses: actions-rs/cargo@v1
+        with:
+          command: make
+          args: release
+
+      - name: Build archive
+        shell: bash
+        run: |
+          staging="rustfmt_${{ matrix.build }}_${{ github.event.release.tag_name }}"
+          mkdir -p "$staging"
+
+          cp {README.md,Configurations.md,CHANGELOG.md,LICENSE-MIT,LICENSE-APACHE} "$staging/"
+
+          if [ "${{ matrix.os }}" = "windows-latest" ]; then
+            cp target/release/{rustfmt.exe,cargo-fmt.exe,rustfmt-format-diff.exe,git-rustfmt.exe} "$staging/"
+            7z a "$staging.zip" "$staging"
+            echo "ASSET=$staging.zip" >> $GITHUB_ENV
+          else
+            cp target/release/{rustfmt,cargo-fmt,rustfmt-format-diff,git-rustfmt} "$staging/"
+            tar czf "$staging.tar.gz" "$staging"
+            echo "ASSET=$staging.tar.gz" >> $GITHUB_ENV
+          fi
+
+      - name: Upload Release Asset
+        uses: actions/upload-release-asset@v1
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+        with:
+          upload_url: ${{ github.event.release.upload_url }}
+          asset_path: ${{ env.ASSET }}
+          asset_name: ${{ env.ASSET }}
+          asset_content_type: application/octet-stream
diff --git a/src/tools/rustfmt/.github/workflows/windows.yml b/src/tools/rustfmt/.github/workflows/windows.yml
new file mode 100644
index 00000000000..08cb52eedae
--- /dev/null
+++ b/src/tools/rustfmt/.github/workflows/windows.yml
@@ -0,0 +1,69 @@
+name: windows
+on:
+  push:
+    branches:
+      - master
+  pull_request:
+
+jobs:
+  test:
+    runs-on: windows-latest
+    name: (${{ matrix.target }}, nightly)
+    strategy:
+      # https://help.github.com/en/actions/getting-started-with-github-actions/about-github-actions#usage-limits
+      # There's a limit of 60 concurrent jobs across all repos in the rust-lang organization.
+      # In order to prevent overusing too much of that 60 limit, we throttle the
+      # number of rustfmt jobs that will run concurrently.
+      max-parallel: 2
+      fail-fast: false
+      matrix:
+        target: [
+          i686-pc-windows-gnu,
+          i686-pc-windows-msvc,
+          x86_64-pc-windows-gnu,
+          x86_64-pc-windows-msvc,
+        ]
+
+    steps:
+    # The Windows runners have autocrlf enabled by default
+    # which causes failures for some of rustfmt's line-ending sensitive tests
+    - name: disable git eol translation
+      run: git config --global core.autocrlf false
+    - name: checkout
+      uses: actions/checkout@v2
+
+      # Run build
+    - name: Install Rustup using win.rustup.rs
+      run: |
+        # Disable the download progress bar which can cause perf issues
+        $ProgressPreference = "SilentlyContinue"
+        Invoke-WebRequest https://win.rustup.rs/ -OutFile rustup-init.exe
+        .\rustup-init.exe -y --default-host=x86_64-pc-windows-msvc --default-toolchain=none
+        del rustup-init.exe
+        rustup target add ${{ matrix.target }}
+      shell: powershell
+
+    - name: Add mingw32 to path for i686-gnu
+      run: |
+        echo "C:\msys64\mingw32\bin" >> $GITHUB_PATH
+      if: matrix.target == 'i686-pc-windows-gnu' && matrix.channel == 'nightly'
+      shell: bash
+
+    - name: Add mingw64 to path for x86_64-gnu
+      run: echo "C:\msys64\mingw64\bin" >> $GITHUB_PATH
+      if: matrix.target == 'x86_64-pc-windows-gnu' && matrix.channel == 'nightly'
+      shell: bash
+
+    - name: cargo-make
+      run: cargo install --force cargo-make
+
+    - name: build
+      run: |
+        rustc -Vv
+        cargo -V
+        cargo build
+      shell: cmd
+
+    - name: test
+      run: cargo test
+      shell: cmd