Overview
At the beginning of 2018 a number of vulnerabilities were discovered which allow malicious user space processes to read kernel memory and malicious code in VM guests to read hypervisor memory. These vulnerabilities affect most CPU manufacturers – Intel, AMD, ARM, MIPS, etc.
The vulnerabilities were nicknamed “Spectre” and “Meltdown” and are outlined in the following CVEs:
- Spectre variant 1 – Bounds Check Bypass: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2017-5753
- Spectre variant 2 – Branch Target Injection: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2017-5715
- Meltdown variant 3 – Rogue Data Cache Launch: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2017-5754
From a CloudStack point of view the main affected components are the system VM templates. This advisory outlines the fix provided for Meltdown only in CloudStack 4.9 (no fixes are available for Spectre). CloudStack 4.11 system VM templates were patched at release time and are therefore not affected.
Affected components summary:
| CVE | ACS 4.9 System VMs | ACS 4.11 System VMs | Hypervisors |
|---|---|---|---|
| Spectre 1 | No fix available - upgrade to 4.11 | Fix in release | Affected - consult with vendor |
| Spectre 2 | No fix available - upgrade to 4.11 | Fix in release | Affected - consult with vendor |
| Meltdown | Fixed - new system VM template | Fix in release | Affected - consult with vendor |
Effect On CloudStack
The impact on CloudStack environments is two-fold since the vulnerabilities affect both the compute hypervisor hosts and the CloudStack system VMs.
Hypervisors
As these are low level CPU call vulnerabilities all hypervisors are affected. Hypervisor vendors have been providing patches – and may continue to do so as further analysis is carried out and potential fixes are developed. The issue with the hypervisor patches is they will potentially impact performance, something which may affect hypervisor VM density figures and/or VM guest performance. ShapeBlue therefore advise users to carry out thorough testing to determine each CloudStack environment impact before rolling these out to production. ShapeBlue can not provide further information or advise on these patches and we recommend all our community users and customers to discuss with the respective hypervisor vendors.
More information can be found in the following articles:
- VMware: https://kb.vmware.com/s/article/52245
- XenServer: https://support.citrix.com/article/CTX231390
- RedHat / KVM: https://access.redhat.com/security/vulnerabilities/speculativeexecution
- CentOS / KVM: https://blog.centos.org/2018/01/meltdown-and-spectre-the-response-from-centos/
- Ubuntu / KVM: https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase/SpectreAndMeltdown
CloudStack System VMs
The CloudStack system VMs are also affected by Spectre and Meltdown. However since these vulnerabilities require local user access someone with malicious intent would first have to gain local access to the system VMs. Since these are locked down and secured in the first place the risk to CloudStack environments is considered low as long as general CloudStack security best practices are followed.
The CloudStack LTS branches system VMs are based on 64-bit Debian releases:
- CloudStack 4.11 utilises Debian 9 “Stretch” 64-bit system VMs
- CloudStack 4.9 utilises Debian 7 “Wheezy” 64-bit system VMs
CloudStack 4.11 was released in February 2018 at which point the Spectre and Meltdown fixes were already provided, and these were therefore included in the system VM templates.
However – CloudStack 4.9 utilise Debian 7 “Wheezy” system VM templates – and “Wheezy” went support end-of-life on May 31st 2018 (https://wiki.debian.org/LTS). At this point the Debian community have only provided patches for Meltdown, and there are no indications Spectre fixes will be provided. As a result ShapeBlue have made the decision to provide new CloudStack 4.9 system VM templates with only the Meltdown patch included. Our overall recommendation if full patching of the vulnerabilities is required is to upgrade to CloudStack version 4.11.
Further information on the Debian fixes can be found in https://wiki.debian.org/DebianSecurity/SpectreMeltdown.
CloudStack 4.9 system VM templates / patching procedure
Whilst system VMs may be patched in-situ they will require reboots for the patches to take effect, and the ShapeBlue recommendation is therefore to update the system VM templates to ensure the Meltdown patch is permanently applied. ShapeBlue have built new system VM templates for CloudStack 4.9 for XenServer, VMware and KVM hypervisors. These can be downloaded from http://packages.shapeblue.com/systemvmtemplate/4.6/meltdown/. The new system VM templates have gone through the full test cycle and no regressions have been found.
The procedure for updating the system VM templates is as follows:
- For each hypervisor type in the CloudStack environment upload the new system VM template with the following information:
- Name: use a descriptive name, e.g. systemvm-<hypervisor>-4.6-meltdown
- Description: add template description
- Zone: pick the correct zone(s)
- Hypervisor: pick the correct hypervisor
- Format: VHD (XenServer) / OVA (VMware) / QCOW2 (KVM)
- OS Type: Debian GNU/Linux 7.0 (64-bit) (or the highest Debian release number available in the dropdown)
- Extractable: no
- Password Enabled: no
- Public: no
- Featured: no
- Routing: yes
- Update the global settings for “router.template.<hypervisor>” to the same as the name configured during the template upload.
- Restart the management service on all management servers.
- Destroy SSVMs and CPVM instances – CloudStack management will recreate these with the new template.
- Restart all networks with the “cleanup” option, which will recreate all VRs with the new system VM template.
Further information
For ShapeBlue support customers, please contact the support team for further information.
For other CloudStack users, please use the community mailing lists.
Giles is CEO and founder of ShapeBlue and is responsible for overall company strategy, strategic relationships, finance and sales.
He is also a committer and PMC member of the Apache CloudStack project and Chairman of the European Cloudstack User Group, actively helping promote brand awareness of the technology.
Giles can regularly be heard speaking at events around the globe, delivering visionary talks on cloud computing adoption and more specifically on Cloudstack technologies.
Before ShapeBlue, Giles held C-Level technology positions for 15 years including founder and CEO of Octavia Information Systems, a leading UK Managed Service Provider.
Giles holds a BSc in Engineering Physics from Sheffield Hallam University. Outside work, Giles is married with two teenage children. He coaches children’s rugby, is a competitive masters swimmer and can regularly be seen crying when his beloved Tottenham Hotspur lose.
