Kubernetes VIM Installation

This document describes the way to install Kubernetes VIM via Devstack and how to register Kubernetes VIM in Tacker.

To do that job, Tacker reuses the efforts from Kuryr-Kubernetes project in creating Kubernetes cluster and setting up native Neutron-based networking between Kubernetes and OpenStack VIMs. Features from Kuryr-Kubernetes will bring VMs and Pods (and other Kubernetes resources) on the same network.

  1. Edit local.conf file by adding the following content

    # Enable kuryr-kubernetes, docker, octavia
    KUBERNETES_VIM=True
    enable_plugin kuryr-kubernetes https://opendev.org/openstack/kuryr-kubernetes master
    enable_plugin octavia https://opendev.org/openstack/octavia master
    enable_plugin devstack-plugin-container https://opendev.org/openstack/devstack-plugin-container master
    KURYR_K8S_CLUSTER_IP_RANGE="10.0.0.0/24"
    

    The public network will be used to launch LoadBalancer for Services in Kubernetes. The example for setting public subnet is described in 1

    For more details, users also see the same examples in 2 and 3.

  2. Run stack.sh

    Command:

    $ ./stack.sh
    
  3. Get Kubernetes VIM configuration

    After successful installation, users can get authentication information.

    • Get “Bearer Token”

      First, you have to confirm Kubernetes Secret name which contains bearer token.

      Command:

      $ kubectl get secret
      

      Result:

      NAME                  TYPE                                  DATA   AGE
      default-token-cfx5m   kubernetes.io/service-account-token   3      94m
      

      Then, you can get the bearer token.

      Command:

      $ TOKEN=$(kubectl get secret default-token-cfx5m -o jsonpath="{.data.token}" | base64 --decode) && echo $TOKEN
      

      Result:

      eyJhbGciOiJSUzI1NiIsImtpZCI6ImdTeGhkUlBNRkJwemo0eXdpMmxxT2Y1aWkzYUhwRERCTWlxMzZFemFKSkUifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRlZmF1bHQtdG9rZW4tY2Z4NW0iLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGVmYXVsdCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjNhOTNiNjA0LTJjY2EtNDllZi05ODMwLWI5NDZhZjI2OTAyNyIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmRlZmF1bHQifQ.yWmZMKuCRn_9Hw07wzee2Gr072NcexuKkaG2HaBamd3BOOAaypb7a12UiKcjqQYsDq32jVGMswSroTJOJtm7xccVbU2lz6CMhTRtDbPKOQm7DLyYdpBoRAoqE8fpy4aF5agqpFYmhYHBoz2VC-sgTwWjuY5XkJ81X9rZWlTCj9p3QkanH2z77lLXo-muthDOOuNm_J05FyR_J1epYXm8JbEpTrj1upaQoKZ9hoKKQrd1crW0stqymcyiLxfPGtSW8dO6GZS4v1vTiIhAEBg3kyQsOPz_nEmDxuYXrcMJxQV8MxWvm3uLOu7wN6-MPsSdw1CQdOfjycTh0D9rG4pxUw
      

      Note

      In Kubernetes data model, values contained under .data is encoded with Base64 format, thus you must decode it with base64 --decode or base64 -d command to use it as a bearer token.

      Before using this token, users have to apply administrator role to this token.

      Command:

      $ kubectl create clusterrolebinding cluster-admin-binding \
      --clusterrole cluster-admin --serviceaccount=default:default
      

      Result:

      clusterrolebinding.rbac.authorization.k8s.io/cluster-admin-binding created
      
    • Get ssl_ca_cert:

      Users can get more information for authenticating to Kubernetes cluster.

      Command:

      $ kubectl get secrets default-token-cfx5m -o jsonpath="{.data.ca\.crt}" | base64 --decode
      

      Result:

      -----BEGIN CERTIFICATE-----
      MIIC5zCCAc+gAwIBAgIBADANBgkqhkiG9w0BAQsFADAVMRMwEQYDVQQDEwprdWJl
      cm5ldGVzMB4XDTIxMDkwOTA0MDc0NFoXDTMxMDkwNzA0MDc0NFowFTETMBEGA1UE
      AxMKa3ViZXJuZXRlczCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAN7H
      /ttxemXTrCDCvN59+g22wwWr5GWUEBxQQz04OPXz1GxBY0H2h3fToRdSs3+snD2h
      6bZ8uryxvXTAlml0IBue/nBxKVRMCRTfqEHEPeNo1yHL2thWGYDfKwEZr9Eg72F5
      mxu9wYdfQS61wg9b4kLmHCIjA58wBDv8Osccs+28BpxJaBd1oG25JWZhcCFRTQur
      URy6d1885ahvaqP9L9mhR8zVzVkAr2noNrCo4/bVMIea8n3yQPBKe3ND1UcxpoCk
      UrfWCrrFsG93RtiivLFJjG8UgUkUhzRfTFoMnTX51Qm2/q/5GZqXSd6z+nU7Bp47
      DHa0hNSPpKnRnP2WwdECAwEAAaNCMEAwDgYDVR0PAQH/BAQDAgKkMA8GA1UdEwEB
      /wQFMAMBAf8wHQYDVR0OBBYEFICl4EHfUar/PBfVTfgymIYXe/z5MA0GCSqGSIb3
      DQEBCwUAA4IBAQA8i+HhuNIJZheNfLgZ+svxmpa1AtdPv8QTrkXTn5OvBJ6l2A2e
      23fVG+8Eolmd0pwuWCMGv4UKAQ45hCMFKMkuKNs2akYueujTxTLwsIu+1EAVnzWp
      E5n+RAhgkAZ18VAGW0otrP/T2zFvci9o3pnEYnQ9Es1mFX7GkBbiI/4qYqx5ysZr
      i5We9jMO//ouJxliJAemRCHMjdqrooMb3k0QyT2lN/1O0TXj0a96pTxoAyivllwk
      LYnc2CoRegU81LeUPSNJRe5+A6kdXixL12F1182/LQgXWkdRnYwoMypyEUDEr9kf
      eGr2fBQ+2ywKH7Ho/HVRW+WcJbXt5nfMX5NK
      -----END CERTIFICATE-----
      
    • Get Kubernetes server url

      By default Kubernetes API server listens on https://127.0.0.1:6443 and https://{HOST_IP}:6443. Users can get this information through kubectl cluster-info command and try to access API server with the bearer token described in the previous step.

      First, you have to confirm the API endpoint that your Kubernetes cluster exposes.

      Command:

      $ kubectl cluster-info
      

      Result:

      Kubernetes control plane is running at https://172.30.202.39:6443
      
      To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
      

      Then, you can confirm the API endpoint and your bearer token are available.

      Command:

      $ curl -k https://172.30.202.39:6443/api/ -H "Authorization: Bearer $TOKEN"
      

      Result:

      {
         "kind": "APIVersions",
         "versions": [
         "v1"
         ],
         "serverAddressByClientCIDRs": [
            {
               "clientCIDR": "0.0.0.0/0",
               "serverAddress": "172.30.202.39:6443"
            }
         ]
      }
      

      Note

      Because SSL certificate used in Kubernetes API server is self-signed, curl returns SSL certificate problem in the response. Users can use -k or --insecure option to ignore SSL certificate warnings, or --cacert <path/to/ssl_ca_cert_file> option to use ssl_ca_cert in the verification of API server’s SSL certificate.

  4. Check Kubernetes cluster installation

    By default, after set KUBERNETES_VIM=True, Devstack creates a public network called net-k8s, and two extra ones for the Kubernetes services and pods under the project k8s:

    Command:

    $ openstack network list
    

    Result:

    +--------------------------------------+-----------------+----------------------------------------------------------------------------+
    | ID                                   | Name            | Subnets                                                                    |
    +--------------------------------------+-----------------+----------------------------------------------------------------------------+
    | 060b32dc-c720-432a-967c-e29d01c2734c | k8s-pod-net     | 792ad14d-42a6-4be0-a5f2-6cdb5395bcdc                                       |
    | 49829476-b297-4d43-bd86-9d7e81bcaebe | k8s-service-net | fdcf3012-37cf-4bbf-9035-2f9bbb99c007                                       |
    | 6a6d19a5-0ff2-4573-aa98-688b9976d3a5 | net_mgmt        | 2ae0e175-54d4-4a6d-b00c-1609bc205f5f                                       |
    | 920520a7-7235-4a20-a4c4-b6955dffa90d | public          | 2e375eca-ad17-4f36-88a5-332a5e380323, 9d83c498-ba57-4615-b81c-578afd1d5020 |
    | 9736903e-adb2-47dc-9a27-46302b4c4e56 | net1            | 843e24c1-3cc0-4d09-8e39-09a0471b6e0a                                       |
    | ad5dd7dd-eb86-49de-937a-fbbd799c5ecf | net0            | 91ed8b41-f8d6-4ddd-9927-912bf7e342e9                                       |
    | c827ecc6-0a13-415b-9954-e20984cb0a4f | lb-mgmt-net     | e33011da-bde3-4483-9e93-9e654b395be3                                       |
    | dab05a83-cf70-4b93-9fc6-9252748ae46c | private         | cc06f27c-1504-401b-b976-895702dac9fa, ffd64f3f-907d-4629-8d63-d9295650a8a1 |
    +--------------------------------------+-----------------+----------------------------------------------------------------------------+
    

    To check Kubernetes cluster works well, please see some tests in kuryr-kubernetes to get more information 4.

  5. Register Kubernetes VIM

    In vim_config.yaml, project_name is fixed as “default”, that will use to support multi tenant on Kubernetes in the future.

    Create vim_config.yaml file for Kubernetes VIM as the following examples:

    auth_url: "https://172.30.202.39:6443"
    bearer_token: "eyJhbGciOiJSUzI1NiIsImtpZCI6ImdTeGhkUlBNRkJwemo0eXdpMmxxT2Y1aWkzYUhwRERCTWlxMzZFemFKSkUifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRlZmF1bHQtdG9rZW4tY2Z4NW0iLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGVmYXVsdCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjNhOTNiNjA0LTJjY2EtNDllZi05ODMwLWI5NDZhZjI2OTAyNyIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmRlZmF1bHQifQ.yWmZMKuCRn_9Hw07wzee2Gr072NcexuKkaG2HaBamd3BOOAaypb7a12UiKcjqQYsDq32jVGMswSroTJOJtm7xccVbU2lz6CMhTRtDbPKOQm7DLyYdpBoRAoqE8fpy4aF5agqpFYmhYHBoz2VC-sgTwWjuY5XkJ81X9rZWlTCj9p3QkanH2z77lLXo-muthDOOuNm_J05FyR_J1epYXm8JbEpTrj1upaQoKZ9hoKKQrd1crW0stqymcyiLxfPGtSW8dO6GZS4v1vTiIhAEBg3kyQsOPz_nEmDxuYXrcMJxQV8MxWvm3uLOu7wN6-MPsSdw1CQdOfjycTh0D9rG4pxUw"
    ssl_ca_cert: "None"
    project_name: "default"
    type: "kubernetes"
    

    Or vim_config.yaml with ssl_ca_cert enabled:

    auth_url: "https://172.30.202.39:6443"
    bearer_token: "eyJhbGciOiJSUzI1NiIsImtpZCI6ImdTeGhkUlBNRkJwemo0eXdpMmxxT2Y1aWkzYUhwRERCTWlxMzZFemFKSkUifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRlZmF1bHQtdG9rZW4tY2Z4NW0iLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGVmYXVsdCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjNhOTNiNjA0LTJjY2EtNDllZi05ODMwLWI5NDZhZjI2OTAyNyIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmRlZmF1bHQifQ.yWmZMKuCRn_9Hw07wzee2Gr072NcexuKkaG2HaBamd3BOOAaypb7a12UiKcjqQYsDq32jVGMswSroTJOJtm7xccVbU2lz6CMhTRtDbPKOQm7DLyYdpBoRAoqE8fpy4aF5agqpFYmhYHBoz2VC-sgTwWjuY5XkJ81X9rZWlTCj9p3QkanH2z77lLXo-muthDOOuNm_J05FyR_J1epYXm8JbEpTrj1upaQoKZ9hoKKQrd1crW0stqymcyiLxfPGtSW8dO6GZS4v1vTiIhAEBg3kyQsOPz_nEmDxuYXrcMJxQV8MxWvm3uLOu7wN6-MPsSdw1CQdOfjycTh0D9rG4pxUw"
    ssl_ca_cert: "-----BEGIN CERTIFICATE-----
    MIIC5zCCAc+gAwIBAgIBADANBgkqhkiG9w0BAQsFADAVMRMwEQYDVQQDEwprdWJl
    cm5ldGVzMB4XDTIxMDkwOTA0MDc0NFoXDTMxMDkwNzA0MDc0NFowFTETMBEGA1UE
    AxMKa3ViZXJuZXRlczCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAN7H
    /ttxemXTrCDCvN59+g22wwWr5GWUEBxQQz04OPXz1GxBY0H2h3fToRdSs3+snD2h
    6bZ8uryxvXTAlml0IBue/nBxKVRMCRTfqEHEPeNo1yHL2thWGYDfKwEZr9Eg72F5
    mxu9wYdfQS61wg9b4kLmHCIjA58wBDv8Osccs+28BpxJaBd1oG25JWZhcCFRTQur
    URy6d1885ahvaqP9L9mhR8zVzVkAr2noNrCo4/bVMIea8n3yQPBKe3ND1UcxpoCk
    UrfWCrrFsG93RtiivLFJjG8UgUkUhzRfTFoMnTX51Qm2/q/5GZqXSd6z+nU7Bp47
    DHa0hNSPpKnRnP2WwdECAwEAAaNCMEAwDgYDVR0PAQH/BAQDAgKkMA8GA1UdEwEB
    /wQFMAMBAf8wHQYDVR0OBBYEFICl4EHfUar/PBfVTfgymIYXe/z5MA0GCSqGSIb3
    DQEBCwUAA4IBAQA8i+HhuNIJZheNfLgZ+svxmpa1AtdPv8QTrkXTn5OvBJ6l2A2e
    23fVG+8Eolmd0pwuWCMGv4UKAQ45hCMFKMkuKNs2akYueujTxTLwsIu+1EAVnzWp
    E5n+RAhgkAZ18VAGW0otrP/T2zFvci9o3pnEYnQ9Es1mFX7GkBbiI/4qYqx5ysZr
    i5We9jMO//ouJxliJAemRCHMjdqrooMb3k0QyT2lN/1O0TXj0a96pTxoAyivllwk
    LYnc2CoRegU81LeUPSNJRe5+A6kdXixL12F1182/LQgXWkdRnYwoMypyEUDEr9kf
    eGr2fBQ+2ywKH7Ho/HVRW+WcJbXt5nfMX5NK
    -----END CERTIFICATE-----"
    project_name: "default"
    type: "kubernetes"
    

    Run Tacker command for register VIM:

    Command:

    $ openstack vim register --config-file vim_config.yaml vim-kubernetes
    

    Result:

    +----------------+-----------------------------------------------+
    | Field          | Value                                         |
    +----------------+-----------------------------------------------+
    | auth_cred      | {                                             |
    |                |     "bearer_token": "***",                    |
    |                |     "ssl_ca_cert": "None",                    |
    |                |     "auth_url": "https://172.30.202.39:6443", |
    |                |     "username": "None",                       |
    |                |     "key_type": "barbican_key",               |
    |                |     "secret_uuid": "***",                     |
    |                |     "password": "***"                         |
    |                | }                                             |
    | auth_url       | https://172.30.202.39:6443                    |
    | created_at     | 2021-09-17 01:26:28.372552                    |
    | description    |                                               |
    | id             | 884ec305-c8ca-47ef-8cba-fafceabeda30          |
    | is_default     | False                                         |
    | name           | vim-kubernetes                                |
    | placement_attr | {                                             |
    |                |     "regions": [                              |
    |                |         "default",                            |
    |                |         "kube-node-lease",                    |
    |                |         "kube-public",                        |
    |                |         "kube-system"                         |
    |                |     ]                                         |
    |                | }                                             |
    | project_id     | 8cd3cc798ae14227a84f7b50c5ef984a              |
    | status         | PENDING                                       |
    | type           | kubernetes                                    |
    | updated_at     | None                                          |
    | vim_project    | {                                             |
    |                |     "name": "default"                         |
    |                | }                                             |
    +----------------+-----------------------------------------------+
    

    In placement_attr, there are four regions: ‘default’, ‘kube-node-lease’, ‘kube-public’ and ‘kube-system’, that map to namespace in Kubernetes environment.

    After the successful installation of VIM, you can get VIM information as follows:

    Command:

    $ openstack vim list
    

    Result:

    +--------------------------------------+----------------+----------------------------------+------------+------------+-----------+
    | ID                                   | Name           | Tenant_id                        | Type       | Is Default | Status    |
    +--------------------------------------+----------------+----------------------------------+------------+------------+-----------+
    | 884ec305-c8ca-47ef-8cba-fafceabeda30 | vim-kubernetes | 8cd3cc798ae14227a84f7b50c5ef984a | kubernetes | False      | REACHABLE |
    +--------------------------------------+----------------+----------------------------------+------------+------------+-----------+
    

    You can update those VIM information with openstack vim set:

    Command:

    $ openstack vim set --config-file path/to/updated/config 884ec305-c8ca-47ef-8cba-fafceabeda30
    

    When updating Kubernetes VIM, you can update VIM information (such as bearer_token and ssl_ca_cert) except auth_url and type of VIM.

    You can get the detail of VIM information with openstack vim show:

    Command:

    $ openstack vim show 884ec305-c8ca-47ef-8cba-fafceabeda30
    

    If you no longer use the Kubernetes VIM, you can delete it with openstack vim delete:

    Command:

    $ openstack vim delete 884ec305-c8ca-47ef-8cba-fafceabeda30
    

References

1

https://github.com/openstack-dev/devstack/blob/master/doc/source/networking.rst#shared-guest-interface

2

https://docs.openstack.org/tacker/latest/install/devstack.html

3

https://opendev.org/openstack/tacker/src/branch/master/devstack/local.conf.kubernetes

4

https://docs.openstack.org/kuryr-kubernetes/latest/installation/testing_connectivity.html