Event notifications enable real-time communication of critical events to external systems and users. They allow key actions and changes to be promptly reported, supporting immediate response and integration with other services. For example, event notifications can alert administrators for resource utilization thresholds, or system failures, enabling proactive management and automation. Likewise, users can receive alerts regarding their own resources, improving overall responsiveness.
Until now, CloudStack was offering a robust real-time events notification system with integrations via message brokers like RabbitMQ and Kafka. However, these integrations were typically limited to administrators and often require a learning curve to implement effectively. To address these challenges, a new CloudStack Webhooks Framework has been developed. It allows web-based, real-time event publishing, enabling both CloudStack Administrator and Users to create responsive, event-driven systems that can automatically react to changes and incidents.
Webhooks are HTTP-based callbacks triggered by specific events. In CloudStack, they notify web services when actions such as Instance creation, network state changes, or system alert occur. This functionality allows users to automate workflows and integrate CloudStack with other systems and services.
Key Features
The key features that make this framework a valuable tool for building responsive, event-driven systems are:
- Event-driven notifications: CloudStack sends event notifications to web-based systems, enabling users to create event-driven integrations.
- Security and authentication: CloudStack webhooks support secure communication via HTTPS protocol and can be configured with authentication headers for enhanced security.
- Scalability: The webhooks framework is designed to handle a high volume of events, ensuring reliable delivery without compromising performance.
- Role-based scope: The framework allows webhooks to be created with different scopes based, offering flexibility for diverse user needs.
Webhook Management
Managing webhooks in CloudStack is made easy through a set of new CRUD APIs, with corresponding actions also available in the UI. The webhooks subsection can be found under the Tools section in the UI.
All webhooks, based on the account access, are displayed in a list view.
The “Create webhook” form offers various options to configure and customize a webhook to meet specific needs.
The Webhooks details view displays all relevant information about the webhook, including recent deliveries.
The update action and form allow users to edit the details of an existing webhook.
Webhook Deliveries
CloudStack uses a thread pool for webhook deliveries, allowing configuration of parameters such as the number of attempts, delivery timeout, and thread pool size through Global Settings parameters:
| Name | Description | Scope | Default value |
| webhook.delivery.thread.pool.size | Size of the thread pool for webhook deliveries. | Global | 5 |
| webhook.delivery.retries | Number of attempts for a webhook delivery. | Domain | 3 |
| webhook.delivery.timeout | Timeout (in seconds) for a webhook delivery attempt. | Domain | 10 |
Webhook deliveries are sent as HTTP/HTTPS POST requests with event data as the payload, including custom headers such as:
- X-CS-Event-ID. Event ID for which the webhook delivery is made.
- X-CS-Event. Event for which the webhook delivery is made.
- User-Agent. CS-Hookshot/<ACCOUNT_ID>. Here, ACCOUNT_ID is the ID of the Account that triggered the event.
- X-CS-Signature. HMAC SHA256 signature created using the webhook secret key and the delivery payload. This header is included only when a secret key is specified for the webhook.
Recent webhook deliveries are stored in the database for monitoring and traceability. However, in a production cloud environment with numerous webhooks, continuously storing all deliveries can significantly increase the database size. To manage this, a cleanup mechanism is in place to control the number of stored deliveries and ensure regular removal of old entries. The following Global Settings parameters can be used for managing the cleanup:
| Name | Description | Default value |
| webhook.deliveries.limit | Limit for the number of deliveries to store in the database per webhook. | 10 |
| webhook.deliveries.cleanup.interval | Interval (in seconds) for cleaning up webhook deliveries. | 3600 |
| webhook.deliveries.cleanup.initial.delay | Initial delay (in seconds) for webhook deliveries cleanup task. | 180 |
To manage and test webhook deliveries, new APIs have been added, and these actions are also available through the UI. A list of recent deliveries for a webhook can be viewed in the “Recent deliveries” tab within the webhook detail view.
Deliveries can be resent or removed with using the “Redeliver” and “Delete Webhook Delivery” actions, respectively.
To test deliveries for a webhook with a custom payload, the “Test Webhook Delivery” action, available in the webhook detail view, can be used.
Working with Webhook Delivery Payload
The payload of a webhook delivery is JSON-formatted data to the corresponding action event. Below is a sample payload for the VOLUME.ATTACH action event:
{
"eventDateTime": "2024-07-10 08:05:07 +0000",
"entityuuid": "9ce31d1b-6b1d-43b2-a7e3-8b4829fb065e",
"Volume": "9ce31d1b-6b1d-43b2-a7e3-8b4829fb065e",
"description": "Successfully completed attaching volume. Volume Id: 9ce31d1b-6b1d-43b2-a7e3-8b4829fb065e VmId: b4e21db8-0fd2-4122-b604-5626bcfbd0ed",
"event": "VOLUME.ATTACH",
"user": "7f4ef0fb-0f20-4ec9-a9be-bc02a64f46d7",
"account": "7ae7c033-4cf3-49bf-ae0a-2e3b93abc3c7",
"entity": "Volume",
"status": "Completed",
"VirtualMachine": "b4e21db8-0fd2-4122-b604-5626bcfbd0ed"
}
Utilizing the payload data and HTTP request headers, webhook deliveries can be easily leveraged for automation or monitoring purposes as needed.
A sample Python script demonstrating automation with webhooks is available here: GitHub Gist. This script runs a simple HTTP server to receive webhook deliveries and uses CloudMonkey to create a snapshot policy for a volume when it is attached to an Instance.
Conclusion
The CloudStack Webhooks Framework significantly enhances CloudStack’s event-driven capabilities. It provides a flexible solution for organizations to integrate with external systems like monitoring tools and automation platforms. By enabling real-time event notifications, organizations can automate workflows, improve resource management, and respond faster to critical events.
Integrating with monitoring systems (e.g., Prometheus, Nagios) allows businesses to receive instant alerts on system issues, enhancing proactive management and reducing downtime. The secure and scalable nature of the framework, along with configurable retries and timeouts, ensures reliable performance even in large-scale environments.
By automating responses to events through platforms like Ansible or Jenkins, you can reduce manual intervention, improving operational efficiency. Additionally, role-based scope ensures that relevant users receive targeted notifications, improving security and reducing alert fatigue.
In summary, the Webhooks Framework provides you with the tools to streamline operations, integrate CloudStack with external systems, and enhance automation. With its introduction in CloudStack 4.20, companies can begin to leverage these benefits for more efficient cloud management.
Abhishek Kumar is a software engineer by profession. His personal interests and hobbies are technology, politics and sports. Abhishek is experienced in development and management of a variety of desktop and mobile applications. He has a particular interest in mobile application development, designing and developing highly interactive and intuitive mobile, desktop applications GUI.
Abhishek became part of ShapeBlue in 2019 and is currently an active Apache CloudStack Committer.
You can learn more about Abhishek and his background by reading his Meet The Team blog.
