ci: download awscli from our mirror

This fixes multiple network issues we had when downloading awscli from
PyPI on Azure Pipelines by vendoring awscli itself and its dependencies
in our S3 bucket. Instructions on how to update the cache are present at
the top of src/ci/install-awscli.sh

2 files changed, 35 insertions, 13 deletions

diff --git a/src/ci/awscli-requirements.txt b/src/ci/awscli-requirements.txt
deleted file mode 100644
index c1ffa525a1b..00000000000
--- a/src/ci/awscli-requirements.txt
+++ /dev/null
@@ -1,13 +0,0 @@
-awscli==1.16.201
-botocore==1.12.191
-colorama==0.3.9
-docutils==0.14
-jmespath==0.9.4
-pyasn1==0.4.5
-python-dateutil==2.8.0
-PyYAML==5.1
-rsa==3.4.2
-s3transfer==0.2.1
-six==1.12.0
-urllib3==1.25.3
-futures==3.3.0; python_version < '3.0'
diff --git a/src/ci/install-awscli.sh b/src/ci/install-awscli.sh
new file mode 100755
index 00000000000..d491b9fbcdc
--- /dev/null
+++ b/src/ci/install-awscli.sh
@@ -0,0 +1,35 @@
+#!/bin/bash
+# This script downloads and installs awscli from the packages mirrored in our
+# own S3 bucket. This follows the recommendations at:
+#
+#    https://packaging.python.org/guides/index-mirrors-and-caches/#caching-with-pip
+#
+# To create a new mirrored copy you can run the command:
+#
+#    pip wheel awscli
+#
+# Before compressing please make sure all the wheels end with `-none-any.whl`.
+# If that's not the case you'll need to remove the non-cross-platform ones and
+# replace them with the .tar.gz downloaded from https://pypi.org. Also make
+# sure it's possible to call this script with both Python 2 and Python 3.
+
+set -euo pipefail
+IFS=$'\n\t'
+
+MIRROR="https://rust-lang-ci2.s3.amazonaws.com/rust-ci-mirror/2019-07-27-awscli.tar"
+DEPS_DIR="/tmp/awscli-deps"
+
+pip="pip"
+pipflags=""
+if [[ "${AGENT_OS}" == "Linux" ]]; then
+    pip="pip3"
+    pipflags="--user"
+
+    sudo apt-get install -y python3-setuptools
+    echo "##vso[task.prependpath]$HOME/.local/bin"
+fi
+
+mkdir -p "${DEPS_DIR}"
+curl "${MIRROR}" | tar xf - -C "${DEPS_DIR}"
+"${pip}" install ${pipflags} --no-index "--find-links=${DEPS_DIR}" awscli
+rm -rf "${DEPS_DIR}"