Support OCI image registry with authentication turned on¶

Blueprint: support-oci-image-registry-with-authentication-turned-on

Problem Description¶

In the current openstack-helm, all charts provide an images: section in their values.yaml that have the container images references. By default, the container images are all downloaded from a registry hosted by Docker or Quay. However, the image references can be overridden by operators to download images from any OCI image registry. In the case that the OCI image registry has authentication turned on, kubelet would fail to download the images because the current Openstack-Helm does not provide a way to pass the OCI image registry credentials to kubernetes when pulling images.

Use case¶

Operators should be able to use Openstack-Helm to deploy containerized openstack services with a docker registry has authentication turned on.

Proposed Change¶

To be able to pull images from an OCI image registry which has the authentication turned on, kubernetes needs credentials. For each chart, a new endpoints: entry could be added in values.yaml to provide image credentials, a secret needs to be generated to hold the credentials and the imagePullSecrets: field should be added in each service account to specify which secret should be used to get the credentials from when pulling images by kubelet.

The detailed proposes change are described as following:

1. For each chart, add a new entry oci_image_registry: under endpoints: in values.yaml. The entry oci_image_registry: has the auth: section which provides the credentials for accessing registry images and an option enabled: to determine whether images authentication is required or not. The registry basic information would also be included for generating the registry URL by the endpoint lookup functions. Also add a new entry oci_image_registry: under secrets: to indicate the secret name. In order to create the secret that holds the provided credentials, add a new component secret_registry in manifests: section. For example:

secrets:
  oci_image_registry:
    nova: nova-oci-image-registry-key

endpoints:
  ...
  oci_image_registry:
    name: oci-image-registry
    namespace: oci-image-registry
    auth:
      enabled: false
      nova:
        username: nova
        password: password
    hosts:
      default: localhost
    host_fqdn_override:
      default: null
    port:
      registry:
        default: 5000

manifests:
  secret_registry: true

The option enabled: under auth: and the manifest secret_registry: provide the ability for operator to determine whether they would like to have secrets generated and passed to kubernetes for pulling images.

The secret would not be created with the default option enabled: false and secret_registry: true. To enable secret creation, operator should override enabled: to true. The above example shows the default credentials, operator should override the username: and password: under auth: section to provide their own credentials.

Then, add manifest secret-registry.yaml in templates/ to leverage the function that will be added in helm-toolkit to create the secret. For example:

{{- if and .Values.manifests.secret_registry .Values.endpoints.oci_image_registry.auth.enabled }}
{{ include "helm-toolkit.manifests.secret_registry" ( dict "envAll" . "registryUser" .Chart.Name ) }}
{{- end }}

Add a helm-toolkit function helm-toolkit.manifests.secret_registry to create a manifest for secret generation. For example:

{{- define "helm-toolkit.manifests.secret_registry" -}}
{{- $envAll := index . "envAll" }}
{{- $registryUser := index . "registryUser" }}
{{- $secretName := index $envAll.Values.secrets.oci_image_registry $registryUser }}
{{- $registryHost := tuple "oci_image_registry" "internal" $envAll | include "helm-toolkit.endpoints.endpoint_host_lookup" }}
{{- $registryPort := tuple "oci_image_registry" "internal" "registry" $envAll | include "helm-toolkit.endpoints.endpoint_port_lookup" }}
{{- $imageCredentials := index $envAll.Values.endpoints.oci_image_registry.auth $registryUser }}
{{- $dockerAuthToken := printf "%s:%s" $imageCredentials.username $imageCredentials.password | b64enc }}
{{- $dockerAuth := printf "{\"auths\": {\"%s:%s\": {\"auth\": \"%s\"}}}" $registryHost $registryPort $dockerAuthToken | b64enc }}
---
apiVersion: v1
kind: Secret
metadata:
  name: {{ $secretName }}
type: kubernetes.io/dockerconfigjson
data:
  .dockerconfigjson: {{ $dockerAuth }}
{{- end }}

Reference the created secret by adding the imagePullSecrets: field to ServiceAccount resource template [2] in helm-toolkit/snippets/_kubernetes_pod_rbac_serviceaccount.tpl. To handle it as optional, the field is wrapped in a conditional. For example,

---
apiVersion: v1
kind: ServiceAccount
...
{{- if $envAll.Values.endpoints.oci_image_registry.auth.enabled }}
imagePullSecrets:
  - name: {{ index $envAll.Values.secrets.oci_image_registry $envAll.Chart.Name }}
{{- end }}

If .Values.endpoints.oci_image_registry.auth.enabled will be set to true, then any containers created with the current service account will have the imagePullSecrets automatically added to their spec and the secret will be passed to kubelet to be used for pulling images.

Security Impact¶

The credentials for the registry could be exposed by running the kubectl command: kubectl get secret <secret-name> –output=”jsonpath={.data..dockerconfigjson}” | base64 –decode

Authentication should be enabled for normal users to access Kube API server via either kubectl command or kube REST API call.

Performance Impact¶

No performance impact

Alternatives¶

Before using Openstack-Helm to deploy openstack services,

Put .docker/config.json in docker/kubelet root directory on all nodes
Pre-pulling images on all nodes

But above alternatives have limitations and security impact. i.e…require root access to configure on all nodes, all pods can read any configured private registries, all pods can use any images cached on a node [1]

Implementation¶

Assignee(s)¶

Primary assignees:

Angie Wang (angiewang)

Work Items¶

Provide the credentials and add the manifest across all charts in OSH and OSH-infra
Update helm-toolkit to provide manifest to create secret for registry authentication
Update helm-toolkit serviceaccount template to pass the secret in a conditional

Support OCI image registry with authentication turned on