Create and Bootstrap from a Local External Private Docker Registry¶

You can bootstrap controller-0 from a private local external Docker registry. This is useful in case you plan to perform multiple installations on your own machine and do not want to be rate-limited by public registries.

This will also speed up the bootstrap process as images will be downloaded only once. This guide assumes that you are installing this local external docker registry on a Linux system with Docker installed and configured.

Optionally, you can also create multiple registries with pull-through cache, skipping the need to pre-populate the registry and keeping the images and tags up-to-date.

Procedure

Note

In this section, you will pre-populate a local registry with multiple images. Currently, the generated images list will not contain all images, making it necessary to manually pull some images used by the kubeadm tool. Alternatively, you can follow the “pull-through cache” section, allowing docker to get the image from remote, if it doesn’t exists locally.

Create folders to store your local registry images (storage) and to place setup files that will be used later on (images):
```
mkdir -p $HOME/docker-registry/storage
mkdir -p $HOME/docker-registry/images
```

Create a configuration file that will be used by Docker’s official Registry image later on:

cat > $HOME/docker-registry/config.yml << EOF
version: 0.1
log:
  fields:
    service: registry
storage:
  cache:
    blobdescriptor: inmemory
  filesystem:
    rootdirectory: /var/lib/registry
http:
  addr: :5000
  headers:
    X-Content-Type-Options: [nosniff]
health:
  storagedriver:
    enabled: true
    interval: 10s
    threshold: 3
EOF

Run the docker container registry:

export LOCAL_REG=$HOME/docker-registry
docker run -d \
    --restart=always \
    --name registry \
    -v "$LOCAL_REG"/storage:/var/lib/registry \
    -v "$LOCAL_REG"/config.yml:/etc/docker/registry/config.yml \
    -e REGISTRY_HTTP_ADDR=0.0.0.0:5000 \
    -p 5000:5000 \
    registry:2

Note

Optional: the -p parameter configures a mapping between the host port and the container port. If you want to listen on another port on your host, say 9000, change from -p 5000:5000 \ to -p 9000:5000 \.

Create the list of images that will populate the registry.

Obtain the Kubernetes version your StarlingX uses. This can be found in the fresh_install_k8s_version value of the Kubernetes versions file. Use the branches and tags to find the value for your version.

With the Kubernetes version, you can find the corresponding folder in system images and locate the system-images.yml file. This file contains the list of images to be loaded into your registry.

To make the list of images for StarlingX 8.0, take the raw address of the corresponding system-images.yml file and set a variable with it:
```
export IMAGES_YAML_RAW_FILE="https://opendev.org/starlingx/ansible-playbooks/raw/branch/master/playbookconfig/src/playbooks/roles/common/load-images-information/vars/k8s-v1.24.4/system-images.yml"
```
Use the command to generate a list.lst file:
```
curl -s ${IMAGES_YAML_RAW_FILE} | grep -v '\-\-\-' | grep -v '^#' | cut -d ':' -f2,3 | tr -d ' ' > $HOME/docker-registry/images/list.lst
```
The expected image list will be presented in this format:
```
<REGISTRY_URL/IMAGE_NAME>:<IMAGE_TAG>
<REGISTRY_URL/IMAGE_NAME>:<IMAGE_TAG>
<REGISTRY_URL/IMAGE_NAME>:<IMAGE_TAG>...
```
Important

Due to kubeadm dynamically pulling the necessary images for creating the k8s cluster, based on the Kubernetes version used, this list doesn’t contain all necessary images. You’ll need to manually add the extra images to the list.lst file, using kubeadm. You can check the official k8s docs on how to get them.
Note

Optional: If you have a running StarlingX setup, you can run the following to create an Ansible Playbook to get the exact images you will need instead:
```
cat > list-images.yml << EOF
---
- hosts: localhost
  gather_facts: true
  tasks:
    - name: Load image info
      include_role:
        name: /usr/share/ansible/stx-ansible/playbooks/roles/common/load-images-information

    - name: Print image list
      debug:
        msg: "{{ (kubernetes_images + networking_images + static_images + storage_images + security_images) }}"
EOF
```
Then, run the following to execute the Ansible Playbook:
```
K8S_VERSION=<version>
ansible-playbook list-images.yml -e "kubernetes_version=${K8S_VERSION}"
```
You will find the Kubernetes version to assign to the K8S_VERSION variable on the aforementioned Kubernetes versions file.

Create and run a script that will populate the registry based on the list of images:

export REG_SCRIPT=$HOME/docker-registry/images/populate_registry.sh
cat > $REG_SCRIPT <<'EOF'
#!/bin/bash

if [[ -z $1 ]]; then
        echo "Please provide a file with a list of Docker images."
    exit 1
fi

TAGS_FILE=$1
LOCAL_REGISTRY=localhost:5000

while read DOCKER_IMAGE;
do
    echo ""
    echo -n "--- ${DOCKER_IMAGE}: ";

    IMAGE_ARRAY=($(echo $DOCKER_IMAGE | tr ":" " "))
    REPO=${IMAGE_ARRAY[0]}
    TAG=${IMAGE_ARRAY[1]}
    REPO_TAGS_URL="http://${LOCAL_REGISTRY}/v2/${REPO}/tags/list"
    if curl -s -X GET --insecure ${REPO_TAGS_URL} | jq | grep ${TAG} &>/dev/null; then
        echo -n "Skipping..."
        continue
    fi

    echo "Pulling..."

    set -x
    docker pull ${DOCKER_IMAGE};
    REGISTRY_IMAGE=${LOCAL_REGISTRY}/${DOCKER_IMAGE}
    docker tag ${DOCKER_IMAGE} ${REGISTRY_IMAGE};
    docker push ${REGISTRY_IMAGE};
    docker rmi ${DOCKER_IMAGE} ${REGISTRY_IMAGE};
    set +x

done < $TAGS_FILE
EOF
chmod +x $REG_SCRIPT
$REG_SCRIPT $HOME/docker-registry/images/list.lst

Note

The populate_registry.sh script checks if each image in the list is already present, which means you can update the list and re-run the script to get new images whenever necessary.

Create folders to store your local registry images (storage) and to place setup files that will be used later on (images):

mkdir -p $HOME/docker-registry/storage
mkdir -p $HOME/docker-registry/images
mkdir -p $HOME/docker-registry/config

Create a configuration file that will be used by Docker’s official registry image later on. The script will replace the PORT and REGISTRY values:

cat > $HOME/docker-registry/config.yml << EOF
version: 0.1
log:
  fields:
    service: registry
storage:
  cache:
    blobdescriptor: inmemory
  filesystem:
    rootdirectory: /var/lib/registry
http:
  addr: <PORT>
  headers:
    X-Content-Type-Options: [nosniff]
health:
  storagedriver:
    enabled: true
    interval: 10s
    threshold: 3
proxy:
  remoteurl: https://<REGISTRY>
EOF

Create and run a script that will create several private docker registries:

export REG_CACHE_SCRIPT=$HOME/docker-registry/images/create_registries.sh
cat > $REG_CACHE_SCRIPT <<'EOF'
#!/bin/bash

if [ -z "$1" ]; then
    echo "No argument supplied"
    exit 1
fi

export LOCAL_REG=$HOME/docker-registry
INSECURE_REGISTRY_LIST=()
LOCALHOST_IP=$1
[ -z "$2" ] && BASE_PORT=5000 || BASE_PORT=$2
REGISTRY_LIST=("quay.io" "gcr.io" "k8s.gcr.io" "registry-1.docker.io"
    "docker.elastic.co" "ghcr.io" "registry.k8s.io" "icr.io")
CONFIG_BASE=$(cat "$LOCAL_REG"/config.yml)

for cur_registry in "${REGISTRY_LIST[@]}"; do
    CONTAINER_NAME="registry_$cur_registry"
    NEW_CONFIG_FILE_NAME="config_$cur_registry.yml"
    NEW_CONFIG_FILE="$LOCAL_REG/configs/$NEW_CONFIG_FILE_NAME"
    NEW_CONFIG=$(echo "$CONFIG_BASE" | sed "s/<PORT>/$BASE_PORT/" | sed "s/<REGISTRY>/$cur_registry/")
    echo "$NEW_CONFIG" >"$NEW_CONFIG_FILE"

    docker run -d \
        --restart=always \
        --name "$CONTAINER_NAME" \
        -v "$LOCAL_REG"/storage:/var/lib/registry \
        -v "$NEW_CONFIG_FILE":/etc/docker/registry/config.yml \
        -e REGISTRY_HTTP_ADDR=0.0.0.0:"$BASE_PORT" \
        -p "$BASE_PORT":"$BASE_PORT" \
        registry:2.6.2

    INSECURE_REGISTRY_LIST+=("$cur_registry - $LOCALHOST_IP:$BASE_PORT")
    ((BASE_PORT++))

done

for reg in "${INSECURE_REGISTRY_LIST[@]}"; do
    echo "$reg"
done
EOF

chmod +x $REG_CACHE_SCRIPT
$REG_CACHE_SCRIPT <your ip address> <starting port (default 5000)>

Important

The create_registries.sh script create 8 registry containers. You’ll need 8 free ports, in sequence, starting from the specified starting port. Ex.: Passing port 6000 as the second argument, will create containers on 6000-6007 port range.

Note

On it’s current version, Docker CLI throws a warning when pulling images that’re still using the v1 manifest format. The official Docker image registry, on it’s latest version, does not allow images that still uses them. A few images used by the StarlingX OS are outdated, so, as of now, it’ll be necessary to use a older version of the registry image (v2.6.2). Check the official registry as a pull through cache docker guide for more info.

Note

The Docker CLI exclusively permits insecure (HTTP) registries when on the local host. When executing the provided script remotely, in addition to modifying the LOCAL_REGISTRY variable to match the IP address of the registry’s location, it is necessary to insert an entry in the insecure-registries: section within the etc/docker/daemon.json file. Following this adjustment, you must restart the Docker service.

Results

Your registry is ready! On your next StarlingX installation, update your /home/sysadmin/localhost.yml bootstrap overrides file with the following lines to use it:

docker_registries:
  quay.io:
    url: <your IP address>:5000/quay.io
  gcr.io:
    url: <your IP address>:5000/gcr.io
  k8s.gcr.io:
    url: <your IP address>:5000/k8s.gcr.io
  docker.io:
    url: <your IP address>:5000/docker.io
  docker.elastic.co:
    url: <your IP address>:5000/docker.elastic.co
  ghcr.io:
    url: <your IP address>:5000/ghcr.io
  registry.k8s.io:
    url: <your IP address>:5000/registry.k8s.io
  icr.io:
    url: <your IP address>:5000/icr.io
  defaults:
    type: docker
    secure: false

docker_registries:
  quay.io:
    url: <your IP address>:<port_1>
  gcr.io:
    url: <your IP address>:<port_2>
  k8s.gcr.io:
    url: <your IP address>:<port_3>
  docker.io:
    url: <your IP address>:<port_4>
  docker.elastic.co:
    url: <your IP address>:<port_5>
  ghcr.io:
    url: <your IP address>:<port_6>
  registry.k8s.io:
    url: <your IP address>:<port_7>
  icr.io:
    url: <your IP address>:<port_8>
  defaults:
    type: docker
    secure: false

Note

This procedure configured StarlingX to use an insecure registry via the docker_registries.defaults.secure parameter set to false in the excerpt above. Make sure you only use this on your own development environment.