Apache CloudStack stands out by supporting multiple hypervisors, offering organisations the flexibility to choose the one that best meets their specific business needs. This article analyses the three key hypervisors supported by CloudStack: KVM, VMware vSphere and XCP-ng / XenServer. We will examine their features, architecture, performance metrics, and suitability for various workloads and explore how each integrates with CloudStack to deliver cloud services. Additionally, we will analyse the impact of each hypervisor on operational complexity and cost of ownership.
Furthermore, we will present the results of a survey conducted in 2024 within the CloudStack community, which sheds light on user preferences and the rationale behind choosing specific hypervisors. This survey data provides valuable insights into these technologies’ practical considerations and real-world applications.
Hypervisor Architecture
KVM Architecture
Kernel-based Virtual Machine (KVM) is a Type 1 hypervisor integrated into the Linux kernel, transforming it into a robust virtualisation platform. The architecture leverages QEMU to emulate hardware for guest OS instances, each running in an isolated user space with unmodified drivers interacting through virtual network interface cards (VNICs). The KVM kernel module directly interfaces with physical hardware, utilizing the host NIC driver and a Linux bridge or Open vSwitch for network connectivity. The network tap facilitates data exchange between the virtual and physical network layers, ensuring efficient and secure virtualisation. The integration with the Linux kernel allows KVM to benefit from kernel-level performance and scalability features.
Additionally, for managing KVM hosts, Libvirt serves as a powerful management API, simplifying the creation, management, and automation of virtual machines, networks, and storage. This integration makes it easier to manage KVM environments and integrates seamlessly with cloud platforms like Apache CloudStack and other tools.
VMware vSphere Architecture
VMware vSphere consists of the ESXi hypervisor and vCenter Server. This combination provides a comprehensive virtualisation platform that extends ESXi’s capabilities through centralized management and advanced features. The vCenter Server is the central management point for vSphere, offering essential services and features that enhance the functionality and manageability of ESXi hosts and virtual machines (VMs).
VMware ESXi is a Type 1 hypervisor that provides a complete virtualisation platform by directly interfacing with the physical hardware. The architecture includes the VMkernel, which handles resource scheduling, device drivers, and core services like virtual networking and storage. ESXi hosts VMs through its Virtual Machine Monitor (VMM) and supports various management services, including the CIM broker for hardware management and third-party plugins. The user world API layer enables interaction with system daemons like hostd and vpxa, facilitating seamless operation and integration with VMware’s management ecosystem like vCenter.
XenServer / XCP-ng Architecture
XenServer and XCP-ng utilize the Xen hypervisor as their core virtualisation technology and leverage XAPI (Xen API) to manage the virtualisation stack. The Xen hypervisor is a Type 1 hypervisor that operates directly on the physical hardware, managing multiple guest operating systems (OS) in isolated environments called domains. The architecture includes:
- Domain 0 (Dom0): A privileged domain with direct access to hardware, containing the control interface, device drivers, and management tools like XenStore and xend.
- Domain U (DomU): Unprivileged domains that run user applications and interact with Dom0 through the Xen Bus.
The Xen hypervisor provides core virtualisation functions such as virtual CPU, virtual memory management (MMU), event channels, and hardware interfaces, ensuring efficient resource allocation and isolation between domains.
Hypervisor Capabilities Matrix
Feature | KVM | XenServer / XCP-ng | VMware |
Virtualisation Type | Hardware Assisted Virtualisation | Hardware Assisted and Para-Virtualisation | Hardware Assisted Virtualisation |
OS Compatibility | Linux, Windows, others | Linux, Windows | Linux, Windows, others |
Scalability | High: Supports many simultaneous VMs | Moderate: Optimized for Xen-specific architectures | High: Supports many simultaneous VMs |
Management Tools | Libvirt | XenCenter, Xen Orchestra | vSphere vCenter |
License | Open source (GPL) | Open source (GPL) XCP-ng, mixed Open source and commercial for Citrix XenServer | Commercial |
Primary use case | General enterprise, cloud services | Server virtualisation, cloud infrastructures | Enterprise solutions, fully integrated cloud management |
Max RAM per physical host | 48TB | 6TB | 24TB |
Max RAM per VM (overcommitted) | 16TB | 1.5TB | 24TB |
Max vCPU per physical host | No hard limit | 4096 | 4096 |
Max vCPU per VM | 710 | 32 | 768 |
Logical CPU per physical host | 1792 | 512 | 960 |
VM disk max size | 10TB | 16TB | 62TB |
Max VM per Host | No hard limit | 1000 | 1024 |
Max 25GbE NIC per Host | No hard limit | 16 | 16 |
Virtual NIC per Host | No hard limit | 512 | 1024 |
Virtual NIC per VM | No hard limit | 7 | 10 |
CloudStack Hypervisor Performance
Performance Metric | KVM | XenServer / XCP-ng | VMware |
CPU Utilization | Efficient, minimal overhead | Moderate overhead | Highly optimized, low overhead |
Memory Management | Uses Linux kernel features, efficient with large VMs | Good, with Dynamic Memory Control | Advanced, with Transparent Page Sharing |
Disk I/O Throughput | High with Virtio drivers | High, optimized for Xen | High, with proprietary drivers |
Network Throughput | High with Virtio drivers | Moderate to high | Very high, with optimized drivers |
Live Migration | Supported, robust | Supported, XenMotion | Supported, vMotion with advanced features |
Fault Tolerance | Basic support via third-party tools | Moderate, with some built-in options | Advanced, with high availability and fault tolerance features |
CloudStack Hypervisors Capabilities
Feature | KVM | XenServer / XCP-ng | VMware |
Instances live-migration between Hosts in the same cluster | Yes | Yes | Yes |
Instances live-migration between clusters in the same Zone | Yes | No | Yes |
Dynamic CPU and RAM Scaling | Yes | Yes | Yes |
Hot plug vNIC to Instances | Yes | Yes | Yes |
Max vNIC per Instance | No limit | 7 | 10 |
Max vCPU per Instance | 240 | 32 | 768 |
Max RAM per Instance | 6TB | 1.5TB | 24TB |
Max disk size per Instance | 10TB | 2TB | 62TB |
Operational Complexity
KVM integration in CloudStack is relatively simple for users familiar with Linux environments. It integrates well with CloudStack orchestration capabilities by relying on libvirt. Initial configuration may require a good understanding of the Linux network and storage. Continuous management is simplified by CloudStack’s native support, but advanced customisations can add complexity.
XCP-ng and XenServer integration with CloudStack is well supported through XAPI, making deployment smoother for those familiar with the Xen architecture. The complexity lies in the detailed configurations needed to optimize performance and ensure compatibility with CloudStack features. Managing XenServer/XCP-ng on CloudStack is usually efficient, but administrators need to be familiar with Xen-specific configurations and optimizations.
VMware vSphere seamlessly integrates with CloudStack through vCenter API, leveraging the robust support of CloudStack. Although the VMware configuration process is easy to use, managing a VMware environment on CloudStack can be more complex due to the extensive set of features. Administrators must be proficient with vSphere and CloudStack to maximize the potential using VMware vSphere.
Cost of Ownership
Let’s analyse the CloudStack hypervisors cost of ownership now. As a free and open-source solution, KVM significantly reduces the total cost of ownership. Lower operational costs are achieved due to the absence of licensing fees, though skilled Linux administrators might be necessary.
XCP-ng is free and open-source, while Citrix XenServer is a paid solution. It has lower operational costs than VMware but may incur costs for commercial support.
VMware has high licensing costs, increasing the total cost of ownership. Higher operational costs are offset by reduced complexity and robust support. Highly efficient resource utilization can reduce hardware costs in large deployments. Considering the recent changes in the virtualisation market, including VMware’s acquisition by Broadcom, many organizations are seeking mature alternatives like Apache CloudStack and the KVM hypervisor to reduce costs associated with price and product changes. We advise you to read this blog post where we explore CloudStack and KVM as open-source alternatives to VMware: https://www.shapeblue.com/apache-cloudstack-as-a-vmware-alternative/
Hypervisor Choices in the CloudStack Community
Hypervisor selection is critical in cloud computing, affecting performance, cost, compatibility, and ease of management. Recently a survey was conducted in the CloudStack community asking about the choice of hypervisor to understand the preferences and reasons behind choosing specific hypervisors. Some insights have been extracted that expose why KVM is the most appropriate preference when used with Apache CloudStack.
Below we display the results of this survey that provide us with insights into the preferences of users and their reasons:
-
Current Hypervisors Used with Apache CloudStack:
-
Primary Reasons for Choosing Hypervisors:
-
Insights from Additional Responses:
- KVM is preferred for being free, open-source, and improving performance. Users appreciate its ease of use, especially for those familiar with Linux.
- VMware vSphere is praised for its stability and widespread usage. Its established reputation for reliability justifies its higher cost for some users.
- XCP-ng and XenServer were mentioned for specific use cases and features.
Recommendations
Based on the user preferences and the comparative matrixes, KVM is the preferred hypervisor for those seeking:
- Cost-effectiveness: As it is open-source and free, it significantly reduces licensing costs.
- Performance: Known for robust performance, particularly beneficial for Linux-based environments.
- Ease of Use: Especially favoured by users with expertise in Linux.
- Flexibility and Customization: Provides lower-level access to networking needs and is highly customizable.
VMware vSphere is recommended for environments prioritizing:
- Stability and Reliability: Known for its mature and stable environment.
- Enterprise Features: Offers extensive and advanced features suitable for complex and large-scale deployments.
- Ease of Management: Simplifies the management and customization of installations, making it ideal for organizations with existing VMware infrastructure.
XCP-ng and XenServer are suitable for:
- Specific Technical Requirements: Addressing particular needs and feature sets not fully covered by other hypervisors.
- Open-Source Flexibility: Offering a balance of open-source benefits with robust performance and compatibility.
Conclusion
Business needs define the hypervisor choice. KVM is the most preferred among organisations using Apache CloudStack because it covers a range of use cases and workloads while offering an attractive cost of ownership due to its open-source nature. KVM is a powerful hypervisor with efficient performance and minimal overhead, enabling it to run virtual machines at scale effectively.
VMware vSphere is known for its robust feature set and high reliability, making it a top choice for enterprises that require extensive management capabilities and are willing to invest in higher licensing costs for advanced features and support. Its integration with CloudStack is seamless, providing a cohesive management experience.
XenServer / XCP-ng offers a balanced solution with superior performance and scalability, particularly for organisations familiar with Xen architecture. It provides a compelling mix of open-source flexibility with the option for commercial support, making it suitable for various infrastructure needs.
In summary, selecting the right hypervisor depends on specific organisational requirements, including budget constraints, desired features, and existing expertise. KVM’s cost-effectiveness and robust performance make it a strong contender for many CloudStack users, while VMware and XenServer / XCP-ng offer unique advantages that cater to different operational needs.
Marco Sinhoreli works as a Technical Marketing Manager at ShapeBlue. Marco has a depth of experience in helping big organisations implement CloudStack. He has been consulting major companies in Brazil for their CloudStack environments. In addition, he has a strong understanding of the struggle that cloud builders and IaaS providers can experience and how open-source technologies and ACS can help them. Away from work, Marco is a lover of music (playing a mean guitar) and politics.