Introduction
In a previous post, ShapeBlue’s Boris Stoyanov introduced a feature for KVM which allows administrators to register templates and ISOs without needing secondary storage as an intermediate 
cache. This feature is known as Direct Download in CloudStack.
Without an intermediate cache, the templates and ISOs direct download registration process differs from the usual process, which is:
- The template and ISO registration process uses secondary storage as an intermediate cache
- The zone SSVM handles the template or ISO download and stores it on the intermediate cache in secondary storage
- On VM creation, the template is copied into primary storage from secondary storage.
With Direct Download, secondary storage is not involved in the process:
- When a user registers a template or ISO, CloudStack retains the information about the template including its URL for later download
- CloudStack selects a random host in the zone to check if the host can reach the template URL. If not, then registration fails
- On VM deployments, the template is downloaded from the URL directly into primary storage or copied from another primary storage if it has been previously downloaded
Storage provider agnostic
The Direct Download feature was initially designed and implemented for NFS primary storage and KVM hypervisor only. As of CloudStack version 4.14, the Direct Download feature is extended and is no longer restricted to only NFS, but now provides an abstraction for any storage type, as the KVM storage processor allows conversion of different storage types.
The extension of the Direct Download support was done to support the most used protocols / storage types:
- NFS – already supported before version 4.14
- Local storage
- SharedMountpoint
Support for other storage providers may be extended in the future.
Furthermore, prior to CloudStack version 4.14, it was not possible to register system VM templates using Direct Download.
Direct Downloads as of version 4.14
These are the main changes with this feature:
- A new check of available free space on the host is added before attempting to download a template. On registration, a CheckUrlCommand is sent to a random KVM host, and this host performs a HEAD command to the template URL. If the server provides the remote file size of the template, this size is passed as part of the DirectDownloadCommand and used to check against the available free space on the scratch space. If the template size cannot be retrieved from the server then the check is skipped
- The template file is downloaded to a scratch space in the storage system on the host. This location can be explicitly set by the administrator via agent.properties file on the property ‘direct.download.temporary.download.location’, and in the case where it is not set, the default location ‘/var/lib/libvirt/images’ is used
- After the template file is downloaded to the scratch space, it is extracted (or copied) to the primary storage pool selected by the CloudStack planner
- The implementation of the extract / copy method is extended to invoke the different StorageAdaptors implementations for different storage providers. These implementations include local storage, shared mount point and NFS in the scope of this feature
- After the template is extracted / copied to the destination storage pool, the downloaded file is removed from the scratch space on the host
Configuration
Changes have been made to make different timeouts for direct download functionality configurable using global settings. The following global configurations have been added:
- download.connect.timeout – Connection establishment timeout in milliseconds for direct download. Default value: 5000 milliseconds.
- download.socket.timeout – Socket timeout (SO_TIMEOUT) in milliseconds for direct download. Default value: 5000 milliseconds.
- download.connection.request.timeout – Requesting a connection from connection manager timeout in milliseconds for direct download. Default value: 5000 milliseconds. This setting is hidden and not visible in UI.
System VM templates
It is also possible to register new system VM templates using the direct download option.
Nicolas Vazquez is a Senior Software Engineer at ShapeBlue and is a PMC member of the Apache CloudStack project. He spends his time designing and implementing features in Apache CloudStack and can be seen acting as a release manager also. Nicolas is based in Uruguay and is a father of a young girl. He is a fan of sports, enjoy playing tennis and football. In his free time, he also enjoys reading and listening to economic and politics materials.
ShapeBlue
