SR-IOV plugin¶

Create Network Attachment Definitions ¶

Network attachment definition specifications must be created in order to reference / request an SR-IOV interface in a container specification.

About this task

The sample network attachments shown in this procedure can be used in a container as shown in Use Network Attachment Definitions in a Container.

Prerequisites

You must have configured at least one SR-IOV interface on a host with the target datanetwork (datanet-a or datanet-b in the example below) assigned to it before creating a NetworkAttachmentDefinition referencing this data network.

Note

The configuration for this SR-IOV interface with either a netdevice or a user space network device such as a DPDK poll mode drive.

Note

Mellanox-based interfaces should be bound to the netdevice vf-driver for both kernel and user space usage.

After SR-IOV interfaces have been provisioned and the hosts labeled and unlocked, available SR-IOV VF resources are automatically advertised.

They can be referenced in subsequent StarlingX operations using the appropriate NetworkAttachmentDefinition name and the following extended resource name:

intel.com/pci_sriov_net_${DATANETWORK_NAME}

For example, with a network called datanet-a the extended resource name would be:

intel.com/pci_sriov_net_datanet_a

The extended resource name will convert all dashes (‘-’) in the data network name into underscores (’_’).
SR-IOV enabled interfaces using the netdevice VF driver must be administratively and operationally up to be advertised by the SR-IOV device plugin.
It is not possible to have multiple data networks assigned to the same interface since the VFs resources cannot be shared between pools.

Procedure

Create a simple SR-IOV network attachment definition file called net1.yaml associated with the data network datanet-a.

~(keystone_admin)]$ cat <<EOF > net1.yaml
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: net1
  annotations:
    k8s.v1.cni.cncf.io/resourceName: intel.com/pci_sriov_net_datanet_a
spec:
  config: '{
      "cniVersion": "0.3.0",
      "type": "sriov"
    }'
EOF

This NetworkAttachmentDefinition is valid for both a kernel-based and a DPDK (vfio) based device.

Create an SR-IOV network attachment with a VLAN ID or with IP Address information.

The following example creates an SR-IOV network attachment definition configured for a VLAN with an ID of 2000.

~(keystone_admin)]$ cat <<EOF > net2.yaml
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: net2
  annotations:
    k8s.v1.cni.cncf.io/resourceName: intel.com/pci_sriov_net_datanet_b
spec:
  config: '{
      "cniVersion": "0.3.0",
      "type": "sriov",
      "vlan": 2000
    }'
EOF

The following example creates an SR-IOV network attachment definition configured with IP Address information.

~(keystone_admin)]$ cat <<EOF > net3.yaml
apiVersion: crd.projectcalico.org/v1
kind: IPPool
metadata:
  name: mypool
spec:
  cidr: "10.56.219.0/24"
  ipipMode: "Never"
  natOutgoing: True
---
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: net3
  annotations:
    k8s.v1.cni.cncf.io/resourceName: intel.com/pci_sriov_net_datanet_b
spec:
  config: '{
      "cniVersion": "0.3.0",
      "type": "sriov",
      "ipam": {
        "type": "calico-ipam",
        "assign_ipv4": "true",
        "ipv4_pools": ["mypool"]
      },
      "kubernetes": {
        "kubeconfig": "/etc/cni/net.d/calico-kubeconfig"
      },
      "datastore_type": "kubernetes"
    }'
EOF

Use Network Attachment Definitions in a Container ¶

Network attachment definitions can be referenced by name when creating a container. The extended resource name of the SR-IOV pool should also be referenced in the resource requests.

About this task

The following examples use network attachment definitions net2 and net3 created in Creating Network Attachment Definitions.

As shown in these examples, it is important that you request the same number of devices as you have network annotations.

Procedure

Create a container with one additional SR-IOV interface using the net2 network attachment definition.

Populate the configuration file pod1.yaml with the following contents.

apiVersion: v1
kind: Pod
metadata:
  name: pod1
  annotations:
    k8s.v1.cni.cncf.io/networks: '[
        { "name": "net2", "interface": "sriov0" }
    ]'
spec:
  containers:
  - name: pod1
    image: centos/tools
    imagePullPolicy: IfNotPresent
    command: [ "/bin/bash", "-c", "--" ]
    args: [ "while true; do sleep 300000; done;" ]
    resources:
      requests:
        intel.com/pci_sriov_net_datanet_b: '1'
      limits:
        intel.com/pci_sriov_net_datanet_b: '1'

Apply the configuration to create the container.

~(keystone_admin)]$  kubectl create -f pod1.yaml
pod/pod1 created

After creating the container, an extra network device interface, net2, which uses one SR-IOV VF, will appear in the associated container(s).

After creating the container, the interface sriov0, which uses one SR-IOV VF will appear.

At this point you can execute commands and review links on the container. For example:

$ kubectl exec -n default -it pod1 -- ip link show

Create a container with two additional SR-IOV interfaces using the net2 network attachment definition.

Populate the configuration file pod2.yaml with the following contents.

apiVersion: v1
kind: Pod
metadata:
  name: pod2
  annotations:
    k8s.v1.cni.cncf.io/networks: '[
        { "name": "net2", "interface": "sriov0" },
        { "name": "net2", "interface": "sriov1" }
    ]'
spec:
  containers:
  - name: pod2
    image: centos/tools
    imagePullPolicy: IfNotPresent
    command: [ "/bin/bash", "-c", "--" ]
    args: [ "while true; do sleep 300000; done;" ]
    resources:
      requests:
        intel.com/pci_sriov_net_datanet_b: '2'
      limits:
        intel.com/pci_sriov_net_datanet_b: '2'

Apply the configuration to create the container.

~(keystone_admin)$ kubectl create -f pod2.yaml
pod/pod2 created

After creating the container, two net2 network device interfaces, which each use one SR-IOV VF, will appear in the associated container(s).

After creating the container, the network device interfaces sriov0 and sriov1, which uses one SR-IOV VF, will appear in the associated container(s).

At this point you can execute commands and review links on the container. For example:

$ kubectl exec -n default -it pod2 -- ip link show

Create a container with two additional SR-IOV interfaces using the net2 and net3 network attachment definitions.

Populate the configuration file pod3.yaml with the following contents.

apiVersion: v1
kind: Pod
metadata:
   name: pod3
   annotations:
      k8s.v1.cni.cncf.io/networks: '[
            { "name": "net2", "interface": "sriov0" },
            { "name": "net3", "interface": "sriov1" }
         ]'
spec:
  containers:
  - name: pod3
    image: centos/tools
    imagePullPolicy: IfNotPresent
    command: [ "/bin/bash", "-c", "--" ]
    args: [ "while true; do sleep 300000; done;" ]
    resources:
      requests:
         intel.com/pci_sriov_net_datanet_b: '2'
      limits:
         intel.com/pci_sriov_net_datanet_b: '2'

Apply the configuration to create the container.

~(keystone_admin)$  kubectl create -f pod3.yaml

net2 and net3 will each take a device from the pci_sriov_net_datanet_b pool and be configured on the container/host based on the their respective NetworkAttachmentDefinition specifications (see Creating Network Attachment Definitions).

After creating the pod, the network device interface sriov0, which uses one SR-IOV VF, will appear in the associated container(s).

After creating the container, network device interfaces net2 and net3, which each use one SR-IOV VF, will appear in the associated container(s).

At this point you can execute commands and review links on the container. For example:

$ kubectl exec -n default -it pod3 -- ip link show

Note

In the above examples, the PCI addresses allocated to the container are exported via an environment variable. For example:

~(keystone_admin)$  kubectl exec -n default -it pod3 -- printenv
                    PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
                    HOSTNAME=pod3
                    TERM=xterm
                    PCIDEVICE_INTEL_COM_PCI_SRIOV_NET_DATANET_B=0000:81:0e.4,0000:81:0e.0
                    KUBERNETES_PORT_443_TCP_PROTO=tcp
                    KUBERNETES_PORT_443_TCP_PORT=443
                    KUBERNETES_PORT_443_TCP_ADDR=10.96.0.1
                    KUBERNETES_SERVICE_HOST=10.96.0.1
                    KUBERNETES_SERVICE_PORT=443
                    KUBERNETES_SERVICE_PORT_HTTPS=443
                    KUBERNETES_PORT=tcp://10.96.0.1:443
                    KUBERNETES_PORT_443_TCP=tcp://10.96.0.1:443
                    container=docker
                    HOME=/root

Create a container with two additional SR-IOV interfaces using the net1 network attachment definition for a DPDK based application.

The configuration of the SR-IOV host interface to which the datanetwork is assigned determines whether the network attachment in the container will be kernel or DPDK-based. The SR-IOV host interface needs to be created with a vfio vf-driver for the network attachment in the container to be DPDK-based, otherwise it will be kernel-based.

Note

Mellanox based NICs should be bound to a netdevice (default) vf-driver.

Populate the configuration file pod4.yaml with the following contents.

apiVersion: v1
kind: Pod
metadata:
  name: testpmd
  annotations:
    k8s.v1.cni.cncf.io/networks: '[
            { "name": "net1" },
            { "name": "net1" }
    ]'
spec:
  restartPolicy: Never
  containers:
  - name: testpmd
    image: "amirzei/mlnx_docker_dpdk:ubuntu16.04"
    volumeMounts:
    - mountPath: /mnt/huge-1048576kB
      name: hugepage
    stdin: true
    tty: true
    securityContext:
      privileged: false
      capabilities:
        add: ["IPC_LOCK", "NET_ADMIN", "NET_RAW"]
    resources:
      requests:
        memory: 10Gi
        intel.com/pci_sriov_net_datanet_a: '2'
      limits:
        hugepages-1Gi: 4Gi
        memory: 10Gi
        intel.com/pci_sriov_net_datanet_a: '2'
  volumes:
  - name: hugepage
    emptyDir:
      medium: HugePages

Note

You must convert any dashes (-) in the datanetwork name used in the NetworkAttachmentDefinition to underscores (_).

Apply the configuration to create the container.

~(keystone_admin)$  kubectl create -f pod4.yaml
pod/testpmd created

SR-IOV plugin

SR-IOV plugin¶

Create Network Attachment Definitions¶

Use Network Attachment Definitions in a Container¶

StarlingX

Page Contents

Create Network Attachment Definitions ¶

Use Network Attachment Definitions in a Container ¶