For a while, the CloudStack community has been working on adding support for containers. ShapeBlue successfully implemented the CloudStack Container Service and donated it to the project in 2016, but it was not completely integrated into the codebase. However, with the recent CloudStack 4.14 LTS release, the CloudStack Kubernetes Service (CKS) plugin adds full Kubernetes integration to CloudStack – allowing users to run containerized services using Kubernetes clusters through CloudStack.
CKS adds several new APIs (and updates to the UI) to provision Kubernetes clusters with minimal configuration by the user. It also provides the ability to add and manage different Kubernetes versions, meaning not only deploying clusters with chosen version, but also providing the option to upgrade an existing cluster to a new version.
The integration
CKS leverages CloudStack’s plugin framework and is disabled by default (for a fresh install or upgrade) – enabled using a global setting. It also adds global settings to set the template for a Kubernetes cluster node virtual machine for different hypervisors; to set the default network offering for a new network for a Kubernetes cluster; and to set different timeout values for the lifecycle operations of a Kubernetes cluster:
| cloud.kubernetes.service.enabled | Indicates whether the CKS plugin is enabled or not. Management server restart needed |
| cloud.kubernetes.cluster.template.name.hyperv | Name of the template to be used for creating Kubernetes cluster nodes on HyperV |
| cloud.kubernetes.cluster.template.name.kvm | Name of the template to be used for creating Kubernetes cluster nodes on KVM |
| cloud.kubernetes.cluster.template.name.vmware | Name of the template to be used for creating Kubernetes cluster nodes on VMware |
| cloud.kubernetes.cluster.template.name.xenserver | Name of the template to be used for creating Kubernetes cluster nodes on Xenserver |
| cloud.kubernetes.cluster.network.offering | Name of the network offering that will be used to create isolated network in which Kubernetes cluster VMs will be launched |
| cloud.kubernetes.cluster.start.timeout | Timeout interval (in seconds) in which start operation for a Kubernetes cluster should be completed |
| cloud.kubernetes.cluster.scale.timeout | Timeout interval (in seconds) in which scale operation for a Kubernetes cluster should be completed |
| cloud.kubernetes.cluster.upgrade.timeout | Timeout interval (in seconds) in which upgrade operation for a Kubernetes cluster should be completed* |
| cloud.kubernetes.cluster.experimental.features.enabled | Indicates whether experimental feature for Kubernetes cluster such as Docker private registry are enabled or not |
* There can be some variation while obeying cloud.kubernetes.cluster.upgrade.timeout as the upgrade on a cluster node must be finished either successfully or as failure for CloudStack to report status of the cluster upgrade.
Once the initial configuration is complete and the plugin is enabled, the UI starts showing a new tab ‘Kubernetes Service’ and different APIs become accessible:
Under the hood
Provisioning a Kubernetes cluster in itself can be a complex process based on the tool used (minikube, kubeadm, kubespray, etc.). CKS simplifies and automates the complete processusing the kubeadm tool for provisioning clusters and performing lifecycle operations. As mentioned in the kubeadm documentation:
Kubeadm performs the actions necessary to get a minimum viable cluster up and running. By design, it cares only about bootstrapping, not about provisioning machines. Likewise, installing various nice-to-have addons, like the Kubernetes Dashboard, monitoring solutions, and cloud-specific addons, is not in scope.
Therefore, all orchestration for cluster node virtual machines is taken care of by CloudStack, and it is only CloudStack that decides the host or storage for the node virtual machines. CKS uses the kubectl tool for communicating with the Kubernetes cluster to query its state, active nodes, version, etc. Kubectl is a command-line tool for controlling Kubernetes clusters.
For node virtual machines, CKS requires a CoreOS based template. CoreOS has been chosen as it provides docker installation and the networking rules needed for Kubernetes. Considering the current CoreOS situation, support for a different host OS could be added in the future.
Networking for the Kubernetes cluster is provisioned using Weave Net CNI provider plugin.
The prerequisites
To successfully provision a Kubernetes cluster using CKS there are few pre-requisites and conditions that must be met:
- The template registered for a node virtual machine must a public template.
- Currently supported Kubernetes versions are 1.11.x to 1.16.x. At present, v1.17 and above might not work due to their incompatibility with weave-net plugin.
- A multi-master, HA cluster can be created using Kubernetes versions 1.16.x only.
- While creating a multi-master, HA cluster over a shared network, an external load-balancer must be manually setup. This load-balancer should have port-forwarding rules for SSH and the Kubernetes API server access. CKS assumes SSH access to cluster nodes is available from port 2222 to (2222 + cluster node count -1). Similarly, for API access port 6443 must be forwarded to master nodes. Over the CloudStack isolated network, these rules are automatically provisioned.
- Currently only a CloudStack isolated or shared network can be used for deployment of a Kubernetes cluster. Network must have the Userdata service enabled.
- For CoreOS, a minimum of 2 CPU cores and 2GB of RAM is required for deployment of a virtual machine. Therefore, a suitable service offering must be created and used while deploying a Kubernetes cluster.
- Node virtual machines must have Internet access at the time of cluster provisioning, scale and upgrade operations, as kubeadm cannot perform certain cluster provisioning steps without it.
The flow
After completing the initial configuration and confirming the requirements, an administrator can proceed with adding the supported Kubernetes version and deploying a Kubernetes Cluster. The addition and management of Kubernetes versions can only be done by an administrator and other users only have permissions to list supported versions. Each Kubernetes version in CKS can only be added as an ISO – this will be a binaries ISO which will contain all the Kubernetes binaries and Docker images for any given Kubernetes release. Using an ISO with the required binaries allows faster installation of Kubernetes on the node virtual machines. kubeadm needs active Internet on the master nodes during the cluster provisioning. Using an ISO with binaries and docker images prevents their download from Internet. To facilitate the creation of an ISO for a given Kubernetes release a new script named create-kubernetes-binaries-iso.sh, has been added in cloudstack-common packages. More about this script can be found in the CloudStack documentation.
Add Kubernetes cluster form in CloudStack UI:
Once there is at least one enabled and ready Kubernetes version and the node VM template in place, CKS will be ready to deploy Kubernetes clusters, which can be created using either the UI or API. Several parameters such as Kubernetes version, compute offering, network, size, HA support, node VM root disk size, etc. can be configured while creating the cluster.
Different operations can be performed on a successfully created Kubernetes cluster, such as start-stop, retrieval of cluster kubeconfig, scale, upgrade or destroy. Both UI and API provide the means to do that.
Kubernetes cluster details tab in CloudStack UI:
Once a Kubernetes cluster has been successfully provisioned, CKS deploys the Kubernetes Dashboard UI. A user can download the cluster’s kubeconfig and use that to access the cluster locally, or to deploy services on the cluster. Alternatively, kubectl tool can be used along with a kubeconfig file to access the Kubernetes cluster via the command line. Instructions for both kubectl and Kubernetes dashboard access will be available in the Kubernetes cluster details page in CloudStack.
Kubernetes Dashboard UI accessible for Kubernetes clusters deployed with CKS:
The new APIs
CKS adds a number of new APIs for performing different operations on a Kubernetes supported version and Kubernetes cluster,
Kubernetes version related APIs:
| addKubernetesSupportedVersion | Available only to Admin, this API allows adding a new supported Kubernetes version |
| deleteKubernetesSupportedVersion | Available only to Admin, this API allows deletion of an existing supported Kubernetes version |
| updateKubernetesSupportedVersion | Available only to Admin, this API allows update of an existing supported Kubernetes version |
| listKubernetesSupportedVersions | Lists Kubernetes supported versions |
Kubernetes cluster related APIs:
| createKubernetesCluster | For creating a Kubernetes cluster |
| startKubernetesCluster | For starting a stopped Kubernetes cluster |
| stopKubernetesCluster | For stopping a running Kubernetes cluster |
| deleteKubernetesCluster | For deleting a Kubernetes cluster |
| getKubernetesClusterConfig | For retrieving Kubernetes cluster config |
| scaleKubernetesCluster | For scaling a created, running or stopped Kubernetes cluster |
| upgradeKubernetesCluster | For upgrading a running Kubernetes cluster |
| listKubernetesClusters | For listing Kubernetes clusters |
The CloudStack Kubernetes Service adds a new dimension to CloudStack, allowing cloud operators to provide their users with Kubernetes offerings, but this is just the beginning! There are already ideas for improvements within the community such as support for different CloudStack zone types, support for VPC network, and use of a Debian based or a user-defined host OS template for node virtual machine. If you have an improvement to suggest, please log it in the CloudStack Github project.
More details about CloudStack Kubernetes Service can be found in the CloudStack documentation.
About the author
Abhishek Kumar is a Software Engineer at ShapeBlue, the Cloud Specialists. Apart from spending most of his time implementing new features and fixing bugs in Apache CloudStack, he likes reading about technology and politics. Outside work he spends most of his time with family and tries to work out regularly.
Abhishek Kumar is a software engineer by profession. His personal interests and hobbies are technology, politics and sports. Abhishek is experienced in development and management of a variety of desktop and mobile applications. He has a particular interest in mobile application development, designing and developing highly interactive and intuitive mobile, desktop applications GUI.
Abhishek became part of ShapeBlue in 2019 and is currently an active Apache CloudStack Committer.
You can learn more about Abhishek and his background by reading his Meet The Team blog.
