UPDATE: 09-Apr-2014 – The proper upgrade command is “apt-get install openssl libssl1.0.0”. If you’ve just updated openssl, please go back and update libssl as well.

UPDATE: 10-Apr-2014 – Added detailed verification steps / Apache CloudStack 4.0 – 4.1 are not vulnerable, they use older Debian/openssl.

Thanks to all involved for helping to put together and update this information

Earlier this week, a security vulnerability was disclosed in OpenSSL, one of the software libraries that Apache CloudStack uses to encrypt data sent over network connections. As the vulnerability has existed in OpenSSL since early 2012, System VMs in Apache CloudStack versions 4.0.0-incubating-4.3 are running software using vulnerable versions of OpenSSL. This includes CloudStack’s Virtual Router VMs, Console Proxy VMs, and Secondary Storage VMs.

The CloudStack community are actively working on creating updated System VM templates for each recent version of Apache CloudStack, and for each of the hypervisor platforms which Apache CloudStack supports. Due to testing and QA processes, this will take several days. In the meantime, a temporary workaround is available for currently running System VMs.

If you are running Apache CloudStack 4.0.0-incubating through the recent 4.3 release, the following steps will help ensure the security of your cloud infrastructure until an updated version of the System VM template is available:

Logon to each Secondary Storage VM, Console Proxy VM and Virtual Router and update openssl

XenServer & KVM

1. Use the GUI to identify the Link Local IP and Host of the VM
2. Connect to the Hypervisor Host using SSH
3. From the Host, Connect to the VM using the following command, replacing n.n.n.n with the Link Local IP identified in step 1.
   • ssh -i /root/.ssh/id_rsa.cloud -p 3922 root@n.n.n.n
4. On the System VM,When updating Secondary Storage VMs, run /etc/init.d/apache2 restart
   • run apt-get update
   • then run apt-get install openssl libssl1.0.0
   • If a dialog appears asking to restart programs, accept its request
5. Log out of the System VM and host server
6. Repeat for all Secondary Storage, Console Proxy and Virtual Router VMs

 VMware

1. Use the GUI to identify the Management / Private IP of the VM
2. SSH onto a CloudStack Management Server
3. From the Management Server, connect to the VM using the following command, replacing n.n.n.n with the Managemnt/Private IP identified in step 1.
   • ​ssh -i /var/lib/cloud/management/.ssh/id_rsa -p 3922 root@n.n.n.n
4. On the System VM,When updating Secondary Storage VMs, run /etc/init.d/apache2 restart
   • run apt-get update
   • then run apt-get install openssl libssl1.0.0
   • If a dialog appears asking to restart programs, accept its request
5. Log out of the System VM and host server
6. Repeat for all Secondary Storage, Console Proxy and Virtual Router VMs

Verification

On each System VM, you can test if it has non-vulnerable openssl packages installed by listing installed packages and looking at the installed versions of openssl and libssl. As in the example below, for a system to be non-vulnerable, the packages need to be at or above version 1.0.1e-2+deb7u6:

root@v-14-VM:~# dpkg -l|grep ssl
ii  libssl1.0.0:i386                1.0.1e-2+deb7u6                  i386         SSL shared libraries
ii  openssl                              1.0.1e-2+deb7u6                  i386         Secure Socket Layer (SSL) binary and related cryptographic tools

We realise that for larger installations where System VMs are being actively created and destroyed based on customer demand, this is a very rough stop-gap. The Apache CloudStack security team is actively working on a more permanent fix and will be releasing that to the community as soon as possible.

For Apache CloudStack installations that secure the web-based user-interface with SSL, these may also be vulnerable to HeartBleed, but that is outside the scope of this blog post. We recommend testing your installation with [1] to determine if you need to patch/upgrade the SSL library used by any web servers (or other SSL-based services) you use.

Information originally posted on https://blogs.apache.org/cloudstack/entry/how_to_mitigate_openssl_heartbleed

1: http://filippo.io/Heartbleed/

About the Author

Geoff Higginbottom is CTO of ShapeBlue, the strategic cloud consultancy and an Apache CloudStack Committer. Geoff spends most of his time designing private & public cloud infrastructures for telco’s, ISP’s and enterprises based on CloudStack.

Steve Roles

Steve is ShapeBlue’s COO and is responsible for all day-to-day administrative and operational functions of the business, including the consulting, programme management, and support functions.

Involved with CloudStack since 2012, Steve has led several large customer engagements including a number of major public and private cloud deployments; co-ordinated and developed worldwide teams and helped implement and deliver an enterprise grade support product.

Prior to ShapeBlue, Steve held senior technical, project and account management roles in enterprise IT outsourcing companies where he gained domain experience in the finance, professional services and defence markets.

Away from work, Steve is a father, keen guitarist, snowboarder and singer (not necessarily in that order).

Away from work, Steve is a music lover and semi-professional musician. Although he doesn’t speak at many technology conferences, he can sometimes be heard providing the evening entertainment.