When resources such as Instances, Disk Volumes, Snapshots, or Templates are marked for deletion, they are typically removed from the Primary and Secondary Storage, meaning they are no longer consuming physical storage space. However, the metadata and records associated with these resources remain in the database, flagged as “removed”, consuming valuable database storage space and affecting system performance.
To address this issue, the new Auto Purging feature, introduced in Apache CloudStack 4.20, provides a solution. This feature allows administrators to efficiently manage and optimize the system by automatically removing these expunged resources from the database. By automating the purging process, CloudStack ensures that resources no longer needed are permanently deleted, thereby freeing up storage and improving database performance.
Why Purging Expunged Resources Matters
Allowing expunged resources to accumulate in the database can result in several issues:
Database Overhead: Expunged resources no longer occupy physical storage but continue to exist in the database as residual records. Over time, this increases the size of the database, leading to inefficiencies in system management, longer query times, and potential performance bottlenecks, especially in large cloud environments.
Performance Degradation: A growing database with residual expunged records can slow down system operations, increase the load on database queries, and make routine tasks like backups or restores more time-consuming. This is particularly problematic in environments with a high turnover of virtual resources, where frequent creation and deletion of instances can rapidly inflate the database size.
Data Governance and Compliance: Purging expunged resources ensures that outdated or sensitive data is permanently and securely removed from the system. This aligns with data retention policies and compliance regulations, helping to prevent unauthorized access to deleted records and ensuring that the system adheres to legal and organizational standards for data management.
New Purging Methods in CloudStack
To address these issues, CloudStack 4.20 introduces new mechanisms for automating the purging process, making it easier to manage expunged resources. Currently, the purging functionality is focused on Instances and their associated resources such as volumes, NICs, and snapshots. There are three primary methods for purging expunged resources:
1. Background Task for Automatic Purging
Administrators can configure CloudStack to automatically purge expunged resources through a background task. The following CloudStack Global Settings define how frequently and under what conditions this task will run:
| Global setting | Default values | Description |
| expunged.resources.purge.enabled | false | Enables or disables the background task for purging expunged resources. |
| expunged.resources.purge.resources | (empty) | A comma-separated list of resource types that will be considered by the background task to purge the DB records of the expunged resources. Currently only VirtualMachine is supported. An empty value will result in considering all resource types for purging. |
| expunged.resources.purge.interval | 86400 | Sets the interval (in seconds) at which the purging task runs. |
| expunged.resources.purge.delay | 300 | Defines the initial delay (in seconds) before the purging task starts. |
| expunged.resources.purge.batch.size | 50 | Sets the batch size for purging DB records. |
| expunged.resources.purge.start.time | (empty) | Specifies the start time for the purging task (in yyyy-MM-dd or yyyy-MM-dd HH:mm:ss format). |
| expunged.resources.purge.keep.past.days | 30 | Defines the number of days’ worth of expunged resources to retain before purging. Setting this to 0 purges all. |
2. On-Demand Purging via API
For immediate purging, a new admin-only API (purgeExpungedResources) has been introduced. This allows administrators to manually remove expunged resources based on a specified time range and resource type. For example, the following API command will purge all expunged virtual machines between April 15, 2024, and April 20, 2024:
| CloudMonkey |
| purge expungedresources startdate=2024-04-15 enddate=2024-04-20 resourcetype=VirtualMachine |
The response indicates the number of expunged resources removed:
| json |
| {
“purgeexpungedresourcesresponse”: { “resourcecount”: 6 } } |
3. Offering-Based Purging
Administrators can also enable purging for specific Compute Offerings by setting the purgeresources flag. This flag can be added to both new and existing offerings using the createServiceOffering or updateServiceOffering APIs. Additionally, the Global Setting expunged.resource.purge.job.delay can be configured to control the delay (in seconds) for purging resources associated with a Compute Offering after expunging them.
| Global setting | Default values | Description |
| expunged.resource.purge.job.delay | 180 | Defines the delay before purging the DB records of an expunged resource. The minimum value allowed is 180 seconds. |
Conclusion
The Auto Purging feature in Apache CloudStack 4.20 simplifies the management of expunged resources, ensuring that their associated records are efficiently removed from the database. This prevents the accumulation of unnecessary data, optimizing system performance and maintaining a leaner, more responsive environment.
By automating the purging process and offering flexible configurations, this feature helps cloud administrators keep their databases optimized while aligning with data governance requirements. With these improvements, CloudStack environments can continue to operate efficiently without manual intervention.
Abhishek Kumar is a software engineer by profession. His personal interests and hobbies are technology, politics and sports. Abhishek is experienced in development and management of a variety of desktop and mobile applications. He has a particular interest in mobile application development, designing and developing highly interactive and intuitive mobile, desktop applications GUI.
Abhishek became part of ShapeBlue in 2019 and is currently an active Apache CloudStack Committer.
You can learn more about Abhishek and his background by reading his Meet The Team blog.
ShapeBlue
