Add customized memory and cpu for function¶

Problem description¶

• For deployers, they may want to manage system resources reasonably and safely by restricting functions’ resource occupancy so that more executions could share the fixed amount of resources and non-interfering with each other.
• For users, applying for resources according to actual needs can help them prevent the waste of resources, which will make sense when users need to pay for the resources they use.

Proposed change¶

For different types of functions, the ways of limiting resources are distinguishing.

• For image type functions, we could config the resource limitation in pod definition, and then kubernetes will help us limit the whole pod, which means the total amount of resources used by the function inside the pod could never exceed the pod’s limit.
• For package/swift type function, it turns to be the runtime server’s responsibility to set resource limits for subprocess that function runs in.

Some details are as following:

First of all, make sure the values of memory_size and cpu saved in the function database are valid. In api.controllers.v1.function.post, we do type and size check for cpu and memory_size params so that they must be integers and within the range set in config.py, and qinling will supply default values for them if users do not input anything. The default values are set to be the minimum in this range.

Besides, in api.controllers.v1.function.put, we add cpu and memory_size to UPDATE_ALLOWED. If user wants to update them, we will do type and size check for cpu and memory_size params before updating function database.

When creating execution, pass cpu and memory_size values saved in function database to qinling.engine.default_engine.create_execution. Here we will do the check for cpu and memory_size values again in case the limited scope of resources set in config.py have been reset.

Then for image type function, pass both of them to orchestrator.kubernetes.manager._create_pod, and we can get a pod with limited cpu and memory resources. For package/swift type function, pass both of them to the worker pod in the k8s deployment by using qinling.engine.utils.url_request. In the selected worker pod, cpu and memory_size values will be used to limit the total amount of resources that function process and its subprocesses can use.

We are considering using ‘cgroup’ in linux to limit cpu and memory because by ‘cgroup’ we can add pids of function process and its subprocesses to the same ‘control group’ and limit the total amount of resources. But we need to use different users to set resource files in ‘cgroup’ and run functions because function should not be granted permission to modify the resource setting files in ‘cgroup’. We use root to create a ‘control group’, and qinling can only write to the ‘tasks’ file.

• For cpu resource limitation, ‘cpu.cfs_quota_us’ and ‘cpu.cfs_period_us’ files in ‘cgroup’ will be used to convert millicpu value.
• For memory resource limitation, only ‘memory.limit_in_bytes’ file will be used to limit RAM because now in k8s source code, the kubelet does not support running with swap enabled. Although it also provides a workaround ‘–fail-swap-on=false’ to allow swap on, which may cause some performance impacts, we would better to disable swap now. For more details about swap, please see the references.

Implementation¶

Dependencies¶

None.

Testing¶

None.

References¶

• Resource model https://docs.python.org/2.7/library/resource.html#resource-limits
• Patch for image type function’s resource limit https://review.openstack.org/#/c/553947
• IRC discussions http://eavesdrop.openstack.org/irclogs/%23openstack-qinling/%23openstack-qinling.2018-03-23.log.html
• K8s source code about swap https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/cm/container_manager_linux.go#L203
• An open issue in k8s about swap https://github.com/kubernetes/kubernetes/issues/53533
• Some discussions about why disable swap https://serverfault.com/questions/881517/why-disable-swap-on-kubernetes