Introduction

As part of its security capabilities, Apache CloudStack encrypts sensitive information such as user credentials and passwords. For over a decade, Apache CloudStack has used the StandardPBEStringEncryptor from the jasypt library to encrypt data. However, the encryption algorithm used by this library, “PBEWithMD5AndDes” is now considered insecure because it relies on outdated cryptographics, namely the MD5 hash function and the DES encryption cipher, which has only a 56-bit key length.

To improve the security of Apache CloudStack, it is recommended to use a more modern and secure encryption algorithm, such as AES-GCM . AES-GCM is a mode of operation for the Advanced Encryption Standard (AES) block cipher that provides both confidentiality and authenticity for data. It is a widely used and recommended encryption algorithm that offers strong security for both symmetric encryption and authentication. The use of AES-GCM with a 256-bit key size provides a high level of security against various attacks, including brute-force, where an attacker tries every possible key to decrypt the data.

Google Tink is a multi-language, cross-platform, open-source library that provides secure cryptographic APIs, which are easy to use correctly and hard to misuse. Tink has an implementation of the AES-GCM algorithm in Java, which is called AesGcmJce.

In Apache CloudStack 4.18.0, a new encryptor called AeadBase64Encryptor is introduced, based on the AesGcmJce implementation from Google Tink. This new encryptor replaces the legacy StandardPBEStringEncryptor and improves the security of Apache CloudStack by using the more modern and secure AES-GCM algorithm with a 256-bit key length.

Introducing database encryption cipher – AeadBase64Encryptor

Base64Encryptor is an interface to encrypt and decrypt values in CloudStack. It uses two abstract methods:

encrypt	encrypt a plain text and return a base64-encoded value.
decrypt	decrypt a base64-encoded value and return a plain text.

The LegacyBase64Encryptor is an older implementation of the Base64Encryptor based on the StandardPBEStringEncryptor from the jasypt library. On the other hand, the AeadBase64Encryptor is a new implementation of the Base64Encryptor, which is based on tink/AesGcmJce. This implementation uses the more modern and secure AES-GCM algorithm with a 256-bit key length. It implements the two abstract methods required for the Base64Encryptor:

encrypt	encrypt the input by AesGcmJce.encrypt method and encode the encrypted value using Base64.
decrypt	Decode the input using Base64 and decrypt the output of decoding using AesGcmJce.decrypt method.

Setting up the CloudStack database

During the CloudStack management server installation, users need to create the CloudStack databases and configuration files, create a database user and grant permission to the user, and optionally will initialize the CloudStack databases. For the whole process of CloudStack management server installation, please refer to http://docs.cloudstack.apache.org/en/latest/installguide/management-server/index.html

In this section, we will discuss the different methods for setting up the CloudStack database to utilize the modern algorithm based on AesGcmJce.

The script /usr/bin/cloudstack-setup-databases is used to set up the CloudStack Management Server and CloudStack Usage Server database (connection credentials and host information).

It has been changed to support both legacy and new encryptors.
• This new argument was added “-g ENCRYPTORVERSION, –encryptor-version=ENCRYPTORVERSION” to support the encryptor version. The options are V1(LegacyBase64Encryptor) and V2 (AeadBase64Encryptor, this is default).
• A new option “env” was added to the argument “-e ENCRYPTIONTYPE, –encrypt-type=ENCRYPTIONTYPE”. If the value is “env”, the management secret key will be loaded from the environment variable CLOUD_SECRET_KEY.

It will create some files at /etc/cloustack directory:

Directory	Description
/etc/cloudstack/management/key	the management key (default value: password)[i]
/etc/cloudstack/management/db.properties	contains the database information. Some values are encrypted by the management key, including the database passwords and a database key (default value: password).

Also, some values in the database are encrypted using the database key.

An example of setting up the CloudStack database

In a typical CloudStack environment, there are at least two management servers to ensure high availability. The CloudStack agents in System VMs and KVM hosts can connect to multiple management servers through software-based agent LB. Please refer to https://www.shapeblue.com/software-based-agent-lb-for-cloudstack/.

Since the multiple CloudStack management servers communicate with the same database, the command to set up the CloudStack database on the management servers is different.

To designate the initial management server as the preferred management server, the CloudStack database must be initialized on this specific server. Use the following command to accomplish this:

/usr/bin/cloudstack-setup-databases “cloud”:”password”@”<Database IP>” –deploy-as=”root”:”Password” -i <Management IP> -m mgmtkey -k dbkey -e file –encryptor-version v2

Executing this command will perform the following tasks:

• Create a new file at “/etc/cloudstack/management/key” and insert the value “mgmtkey” into the file. Alternatively, you can use the value you set for the “-m” argument in the previous command.

• Generate the file “/etc/cloudstack/management/db.properties” and input the necessary database information and user credentials. Please note that some values in this file, including the database key “dbkey” (or the value you set for the “-k” argument in the previous command), are crucial for encrypting sensitive data. The encryptor will use the management key “mgmtkey” (or the value set for the “-m” argument) to encrypt these values.

• Use the provided command above to initialize the CloudStack database using the root user and password credentials. Alternatively, you can replace “root”:”password” with your preferred database credentials.

It is worth noting that the management key, database key, encryption type, and encryptor version are all optional parameters. If you do not set these values, default values will be used.
• The default value for the management key (-m parameter) is “password”.
• The default value for the database key (-k parameter) is “password”.
• The default encryption type (-e parameter) is “file”. However, if you set the encryption type to “env”, the management secret key will be loaded from the environment variable CLOUD_SECRET_KEY.
• Finally, there are two available options (–encryptor-version parameter) for encryption version, V1 and V2, with V2 being the default value.

To set up the database on other management servers, please use the following command:

/usr/bin/cloudstack-setup-databases “cloud”:”Password”@”<Database IP>” -i <Management IP> -m mgmtkey -k dbkey -e file –encryptor-version v2

The parameter “–deploy-as=root”:”Password” used in the previous command is no longer necessary as the CloudStack database has already been initialized on the first management server. Therefore, you do not need to initialize the database again on subsequent management servers.

Migrating the CloudStack database

To migrate the legacy database to the new approach, which includes the generation of the management key, database key, and encryptor, use the script located at “/usr/bin/cloudstack-migrate-databases”. The following files will be updated during the migration process:

• db.properties (Some values are encrypted by the management key)
• server.properties (Some values are encrypted by the management key)
• cloudstack database (Some values are encrypted by the database key)

The script has been updated to support the following parameter:

• version: the new encryption version. Options are V1, V2. If this option is null, the same encryptor version (=db.cloud.encryptor.version in db.properties) will be used for the cloudstack database. If the encryptor version is different, the database will be migrated (unless skip-database-migration is passed). If db.cloud.encryptor.version in db.properties is not set, the default encryptor (currently V2) will be used for the cloudstack database.

Please note that db.properties and server.properties will always use the default encryptor (currently V2) during the migration process.

Parameter	Description	Additional info
–force-database-migration or -f	Force database migration even if DB Secret key and encryptor version are both not changed	This is useful for database migration if user want to use same DB secretkey and encryptor version.
— skip-database-migration or -s	Skip database migration even if DB Secret key or encryptor version is changed	With this flag, database migration will be skipped even if DB secret key or encryptor version is changed. This is useful for additional cloudstack management servers.
— load-new-management-key-from-env or -l	Load new management key from environment variable CLOUD_SECRET_KEY_NEW	Load new management key from environment variable CLOUD_SECRET_KEY_NEW.This can be used to replace the option “-e,–newDBKey <newDBKey>”

To display the usage of the migration script, simply execute the command “cloudstack-migrate-databases -h” in the terminal. This will provide you with information about the available options and how to use them.

An example of Migrating CloudStack databases

CloudStack users may need to migrate their databases for various reasons, such as upgrading from older CloudStack versions that do not support the new encryptor or having used the same management key and database key for an extended period. Using the “cloudstack-migrate-databases” script helps users to more easily migrate their databases using different management keys, database keys, or encryptor versions.

However, it is important to note that users should create backups of their properties in “/etc/cloudstack/management/” and the CloudStack database before starting the migration process.

To migrate the database on the first management server, the following command can be used:

cloudstack-migrate-databases -d <oldDBKey> -m <oldMSKey> -e <newDBKey> -n <newMSKey> -v V2

If the current encryptor version is not V2 or the newDBKey is not the same as the oldDBKey, the script will migrate the database using the new encryptor (AeadBase64Encryptor). However, if the current encryptor version is V2 and the newDBKey is the same as the oldDBKey, but the user still wants to migrate the database, they can pass “-f” or “–force-database-migration” as an option.

During the migration process, the key, db.properties, and server.properties files will be updated using the new keys and encryptor. Encrypted values in the CloudStack database will be decrypted using the old database key and re-encrypted using the new database key and encryptor version.

To migrate the database on other management servers, users can use the following command:

cloudstack-migrate-databases -d <oldDBKey> -m <oldMSKey> -e <newDBKey> -n <newMSKey> -v V2 –skip-database-migration

In this case, the new “–skip-database-migration” parameter is added to the previous command. Since the database has already been migrated on the first management server, the database should not be migrated again on other management servers.

Conclusion

With the new encryptor based on Google Tink introduced in Apache CloudStack 4.18, users can achieve greater security for their CloudStack environments and set up CloudStack properties and databases using the new encryptor. Additionally, they can migrate existing CloudStack environments using the new encryptor with the help of a script provided in the CloudStack deployment.

---

[1] https://google.github.io/tink/javadoc/tink/1.7.0/com/google/crypto/tink/subtle/AesGcmJce.html

[2] https://github.com/google/tink

[3] https://cwiki.apache.org/confluence/display/CLOUDSTACK/New+database+encryption+cipher+-+AeadBase64Encryptor and https://github.com/apache/cloudstack/pull/7003

[4] It is strongly recommended to change the default value by a hash at /etc/cloudstack/management/key file.