Apache CloudStack stands out by supporting multiple hypervisors, offering organisations the flexibility to choose the one that best meets their specific business needs. This article analyses the three key hypervisors supported by CloudStack: KVM, VMware vSphere and XCP-ng / XenServer. We will examine their features, architecture, performance metrics, and suitability for various workloads and explore how each integrates with CloudStack to deliver cloud services. Additionally, we will analyse the impact of each hypervisor on operational complexity and cost of ownership.

Furthermore, we will present the results of a survey conducted in 2024 within the CloudStack community, which sheds light on user preferences and the rationale behind choosing specific hypervisors. This survey data provides valuable insights into these technologies’ practical considerations and real-world applications.

 

Hypervisor Architecture

KVM Architecture

Kernel-based Virtual Machine (KVM) is a Type 1 hypervisor integrated into the Linux kernel, transforming it into a robust virtualisation platform. The architecture leverages QEMU to emulate hardware for guest OS instances, each running in an isolated user space with unmodified drivers interacting through virtual network interface cards (VNICs). The KVM kernel module directly interfaces with physical hardware, utilizing the host NIC driver and a Linux bridge or Open vSwitch for network connectivity. The network tap facilitates data exchange between the virtual and physical network layers, ensuring efficient and secure virtualisation. The integration with the Linux kernel allows KVM to benefit from kernel-level performance and scalability features.

Additionally, for managing KVM hosts, Libvirt serves as a powerful management API, simplifying the creation, management, and automation of virtual machines, networks, and storage. This integration makes it easier to manage KVM environments and integrates seamlessly with cloud platforms like Apache CloudStack and other tools.

 

VMware vSphere Architecture

VMware vSphere consists of the ESXi hypervisor and vCenter Server. This combination provides a comprehensive virtualisation platform that extends ESXi’s capabilities through centralized management and advanced features. The vCenter Server is the central management point for vSphere, offering essential services and features that enhance the functionality and manageability of ESXi hosts and virtual machines (VMs).

VMware ESXi is a Type 1 hypervisor that provides a complete virtualisation platform by directly interfacing with the physical hardware. The architecture includes the VMkernel, which handles resource scheduling, device drivers, and core services like virtual networking and storage. ESXi hosts VMs through its Virtual Machine Monitor (VMM) and supports various management services, including the CIM broker for hardware management and third-party plugins. The user world API layer enables interaction with system daemons like hostd and vpxa, facilitating seamless operation and integration with VMware’s management ecosystem like vCenter.

 

XenServer / XCP-ng Architecture

XenServer and XCP-ng utilize the Xen hypervisor as their core virtualisation technology and leverage XAPI (Xen API) to manage the virtualisation stack. The Xen hypervisor is a Type 1 hypervisor that operates directly on the physical hardware, managing multiple guest operating systems (OS) in isolated environments called domains. The architecture includes:

• Domain 0 (Dom0): A privileged domain with direct access to hardware, containing the control interface, device drivers, and management tools like XenStore and xend.
• Domain U (DomU): Unprivileged domains that run user applications and interact with Dom0 through the Xen Bus.

The Xen hypervisor provides core virtualisation functions such as virtual CPU, virtual memory management (MMU), event channels, and hardware interfaces, ensuring efficient resource allocation and isolation between domains.

 

Hypervisor Capabilities Matrix

Feature	KVM	XenServer / XCP-ng	VMware
Virtualisation Type	Hardware Assisted Virtualisation	Hardware Assisted and Para-Virtualisation	Hardware Assisted Virtualisation
OS Compatibility	Linux, Windows, others	Linux, Windows	Linux, Windows, others
Scalability	High: Supports many simultaneous VMs	Moderate: Optimized for Xen-specific architectures	High: Supports many simultaneous VMs
Management Tools	Libvirt	XenCenter, Xen Orchestra	vSphere vCenter
License	Open source (GPL)	Open source (GPL) XCP-ng, mixed Open source and commercial for Citrix XenServer	Commercial
Primary use case	General enterprise, cloud services	Server virtualisation, cloud infrastructures	Enterprise solutions, fully integrated cloud management
Max RAM per physical host	48TB	6TB	24TB
Max RAM per VM (overcommitted)	16TB	1.5TB	24TB
Max vCPU per physical host	No hard limit	4096	4096
Max vCPU per VM	710	32	768
Logical CPU per physical host	1792	512	960
VM disk max size	10TB	16TB	62TB
Max VM per Host	No hard limit	1000	1024
Max 25GbE NIC per Host	No hard limit	16	16
Virtual NIC per Host	No hard limit	512	1024
Virtual NIC per VM	No hard limit	7	10

 

CloudStack Hypervisor Performance

Performance Metric	KVM	XenServer / XCP-ng	VMware
CPU Utilization	Efficient, minimal overhead	Moderate overhead	Highly optimized, low overhead
Memory Management	Uses Linux kernel features, efficient with large VMs	Good, with Dynamic Memory Control	Advanced, with Transparent Page Sharing
Disk I/O Throughput	High with Virtio drivers	High, optimized for Xen	High, with proprietary drivers
Network Throughput	High with Virtio drivers	Moderate to high	Very high, with optimized drivers
Live Migration	Supported, robust	Supported, XenMotion	Supported, vMotion with advanced features
Fault Tolerance	Basic support via third-party tools	Moderate, with some built-in options	Advanced, with high availability and fault tolerance features

 

CloudStack Hypervisors Capabilities

Feature	KVM	XenServer / XCP-ng	VMware
Instances live-migration between Hosts in the same cluster	Yes	Yes	Yes
Instances live-migration between clusters in the same Zone	Yes	No	Yes
Dynamic CPU and RAM Scaling	Yes	Yes	Yes
Hot plug vNIC to Instances	Yes	Yes	Yes
Max vNIC per Instance	No limit	7	10
Max vCPU per Instance	240	32	768
Max RAM per Instance	6TB	1.5TB	24TB
Max disk size per Instance	10TB	2TB	62TB

 

Operational Complexity

KVM integration in CloudStack is relatively simple for users familiar with Linux environments. It integrates well with CloudStack orchestration capabilities by relying on libvirt. Initial configuration may require a good understanding of the Linux network and storage. Continuous management is simplified by CloudStack’s native support, but advanced customisations can add complexity.

XCP-ng and XenServer integration with CloudStack is well supported through XAPI, making deployment smoother for those familiar with the Xen architecture. The complexity lies in the detailed configurations needed to optimize performance and ensure compatibility with CloudStack features. Managing XenServer/XCP-ng on CloudStack is usually efficient, but administrators need to be familiar with Xen-specific configurations and optimizations.

VMware vSphere seamlessly integrates with CloudStack through vCenter API, leveraging the robust support of CloudStack. Although the VMware configuration process is easy to use, managing a VMware environment on CloudStack can be more complex due to the extensive set of features. Administrators must be proficient with vSphere and CloudStack to maximize the potential using VMware vSphere.

 

Cost of Ownership

Let’s analyse the CloudStack hypervisors cost of ownership now. As a free and open-source solution, KVM significantly reduces the total cost of ownership. Lower operational costs are achieved due to the absence of licensing fees, though skilled Linux administrators might be necessary.

XCP-ng is free and open-source, while Citrix XenServer is a paid solution. It has lower operational costs than VMware but may incur costs for commercial support.

VMware has high licensing costs, increasing the total cost of ownership. Higher operational costs are offset by reduced complexity and robust support. Highly efficient resource utilization can reduce hardware costs in large deployments. Considering the recent changes in the virtualisation market, including VMware’s acquisition by Broadcom, many organizations are seeking mature alternatives like Apache CloudStack and the KVM hypervisor to reduce costs associated with price and product changes. We advise you to read this blog post where we explore CloudStack and KVM as open-source alternatives to VMware: https://www.shapeblue.com/apache-cloudstack-as-a-vmware-alternative/

 

Hypervisor Choices in the CloudStack Community

Hypervisor selection is critical in cloud computing, affecting performance, cost, compatibility, and ease of management. Recently a survey was conducted in the CloudStack community asking about the choice of hypervisor to understand the preferences and reasons behind choosing specific hypervisors. Some insights have been extracted that expose why KVM is the most appropriate preference when used with Apache CloudStack.

Below we display the results of this survey that provide us with insights into the preferences of users and their reasons:

• Current Hypervisors Used with Apache CloudStack:

 

• Primary Reasons for Choosing Hypervisors:

• Insights from Additional Responses:

  • KVM is preferred for being free, open-source, and improving performance. Users appreciate its ease of use, especially for those familiar with Linux.
  • VMware vSphere is praised for its stability and widespread usage. Its established reputation for reliability justifies its higher cost for some users.
  • XCP-ng and XenServer were mentioned for specific use cases and features.

 

Recommendations

Based on the user preferences and the comparative matrixes, KVM is the preferred hypervisor for those seeking:

• Cost-effectiveness: As it is open-source and free, it significantly reduces licensing costs.
• Performance: Known for robust performance, particularly beneficial for Linux-based environments.
• Ease of Use: Especially favoured by users with expertise in Linux.
• Flexibility and Customization: Provides lower-level access to networking needs and is highly customizable.

 

VMware vSphere is recommended for environments prioritizing:

• Stability and Reliability: Known for its mature and stable environment.
• Enterprise Features: Offers extensive and advanced features suitable for complex and large-scale deployments.
• Ease of Management: Simplifies the management and customization of installations, making it ideal for organizations with existing VMware infrastructure.

 

XCP-ng and XenServer are suitable for:

• Specific Technical Requirements: Addressing particular needs and feature sets not fully covered by other hypervisors.
• Open-Source Flexibility: Offering a balance of open-source benefits with robust performance and compatibility.

 

Conclusion

Business needs define the hypervisor choice. KVM is the most preferred among organisations using Apache CloudStack because it covers a range of use cases and workloads while offering an attractive cost of ownership due to its open-source nature. KVM is a powerful hypervisor with efficient performance and minimal overhead, enabling it to run virtual machines at scale effectively.

VMware vSphere is known for its robust feature set and high reliability, making it a top choice for enterprises that require extensive management capabilities and are willing to invest in higher licensing costs for advanced features and support. Its integration with CloudStack is seamless, providing a cohesive management experience.

XenServer / XCP-ng offers a balanced solution with superior performance and scalability, particularly for organisations familiar with Xen architecture. It provides a compelling mix of open-source flexibility with the option for commercial support, making it suitable for various infrastructure needs.

In summary, selecting the right hypervisor depends on specific organisational requirements, including budget constraints, desired features, and existing expertise. KVM’s cost-effectiveness and robust performance make it a strong contender for many CloudStack users, while VMware and XenServer / XCP-ng offer unique advantages that cater to different operational needs.