Real-time message subscription feature of OpenEvent - DataWorks

The OpenEvent module allows you to subscribe to event messages. You can receive notifications about various change events in DataWorks and respond to the events based on your configurations at the earliest opportunity. For example, you can subscribe to table change events to receive notifications about the changes to core tables in real time. You can also subscribe to task change events to implement custom data monitoring of a dashboard that displays the status of real-time synchronization tasks. This topic describes the types of events for subscription by using OpenEvent and performing the subscription configuration procedure.

Prerequisites

DataWorks Enterprise Edition is activated in a supported region. The OpenEvent module is available for DataWorks Enterprise Edition users in the following regions: China (Beijing), China (Hangzhou), China (Shenzhen), China (Shanghai), China (Zhangjiakou), China (Chengdu), US (Silicon Valley), US (Virginia), Germany (Frankfurt), Japan (Tokyo), China (Hong Kong), and Singapore.
EventBridge is activated. The OpenEvent module uses EventBridge to subscribe to and consume messages. For more information, see What is EventBridge?

Permission description

The Open Platform administrator and tenant administrator have read and write permissions on the developer backend. Other tenant-level roles and workspace-level roles have read-only permissions on the developer backend. For more information, see Manage permissions on global-level services.
Only Alibaba Cloud accounts or RAM users to which the AliyunDataWorksFullAccess policy is attached can be used to configure settings on pages in the DataWorks console. For more information, see Manage permissions on the DataWorks services and the entities in the DataWorks console by using RAM policies.

Background information

DataWorks allows you to subscribe to messages of status change events in DataWorks key processes and can push the messages to the OpenEvent module of DataWorks Open Platform. You can use the OpenEvent module to push event messages in a workspace and use EventBridge to filter and consume the messages.

Terms

DataWorks events are classified into regular events and extension point events based on whether the operation process related to an event in DataWorks can be blocked.

Regular events: You can develop an on-premises program to receive messages of this event type and configure custom responses to the event messages based on your business requirements.
Extension point events: This type of event has the characteristics and capabilities of regular events. In addition, this type of event allows you to register your on-premises program as a DataWorks extension to process event messages generated by the operations you perform in DataWorks based on custom processing logic, such as blocking the operations.

DataWorks events are classified into workspace-level events and tenant-level events based on the effective scope of events.

Workspace-level events: Events that are generated in workspace-level services. For example, events that are generated when nodes are run in DataStudio and events that are generated when the status of instances is changed in Operation Center are workspace-level events.
Tenant-level events: Events that are generated in tenant-level services. For example, the event that is generated when a user deletes a workspace in the DataWorks console is a tenant-level event. The following tables describe the events supported by each DataWorks service, whether an event is a regular event or an extension point event, and the message format of an event.

Supported event types

The following tables list the extension point events and regular events that are supported at the workspace level and tenant level.

Workspace-level events

DataStudio

Event category	Event	Regular event	Extension point event	Example
Node-related events	Create a node			Create a node and deploy the node to the production environment.
Node-related events	Update a node			Modify a node in the development environment. Modify a node and deploy the node to the production environment.
File-related events Node Resource Function	Delete a file			Delete a node, resource, or function in DataStudio. Delete a node, resource, or function from the recycle bin in DataStudio.
	Commit a file			Commit a node, resource, or function in DataStudio.
	Deploy a file			Deploy a node, resource, or function in DataStudio.
	Run the code of a node			Run the code of a node in DataStudio.
	Review the code of a node			Review the code of a node in DataStudio to provide a comment and feedback before the node is committed.
Table-related events	Commit a table to the development environment			Commit a table on the Workspace Tables page of DataStudio.
Table-related events	Deploy a table to the production environment			Deploy a table on the Workspace Tables pane of DataStudio. Note You can deploy tables in the Workspace Tables pane only in a workspace in standard mode. Therefore, table deployment events can be generated only in workspaces that are in standard mode.

Operation Center

Event category	Event	Regular event	Extension point event	Example
Node change events	Delete a node			Deploy an undeployment operation on the Deployment Package Creation page of Deploy Center. Undeploy a node on the Auto Triggered Nodes page of Operation Center.
	Undeploy a node			Undeploy a node on the Auto Triggered Nodes page of Operation Center.
	Freeze a node			Freeze a node on the Auto Triggered Nodes page of Operation Center.
	Unfreeze a node			Unfreeze a node on the Auto Triggered Nodes page of Operation Center.
Data backfill events	Backfill data for a node			Backfill data for an auto triggered node. Backfill data for an auto triggered node and the descendant nodes of the node. Backfill data for a large number of nodes. Backfill data in advanced mode.
Scheduling task status change events	Change the status of a scheduling task			A scheduling task status change event can be generated in the following scenarios: A scheduling task is sent to Data Quality for data quality check. A scheduling task is checked based on branch conditions. A scheduling task is triggered as scheduled, manually triggered, or triggered by an event. The task status changes from Waiting for Resources to Running. A scheduling task finishes running no matter whether the running succeeds or fails. The task status changes from Running to Successful. A scheduling task is rerun after the scheduling task fails to run. The task status changes from Run Failed to Running. A scheduling task is scheduled to run. The task status changes to Waiting for Scheduling Time. The scheduling time is determined by the DueTime or cycleTime parameter.
Instance O&M events	Freeze an instance			An instance O&M event is generated when you perform the following operations on an instance in Operation Center: Unfreeze an instance. Freeze an instance. Terminate an instance. Rerun an instance. Set the instance status to Succeeded.
	Unfreeze an instance
	Terminate an instance
	Rerun an instance Rerun an instance Rerun the descendant instances of an instance
	Set the status of an instance to Succeeded
Workflow status change events	Change the status of a workflow Backfill data for a node Run a manually triggered workflow			Backfill data for a node. Run a manually triggered workflow.
Monitoring and alerting	Monitor objects and send alerts			Baseline alerts: If a node in a baseline does not finish running before the estimated completion time, an alert is triggered. Event-based alerts: If a node in a baseline fails to run, an alert is triggered. Rule-based alerts: If you create an alert rule for isolated nodes, an alert is triggered after an isolated node is identified in a workflow.

Security Center

Event category	Event	Regular event	Extension point event	Example
Request order creation and processing events	Create a request order			Create a request order for a function, table, or resource on the Permission Application tab of the Data Access Control page in Security Center.
Request order creation and processing events	Complete a request order			Process a request order for tables, resources, or functions on the Permission Application Processing tab of the Data Access Control page in Security Center.
Table permission application events	Request permissions on tables			Apply for the read, write, and modify permissions on a specific table based on the business requirements.

Data Quality

Event category

Event

Regular event

Extension point event

Example

Data quality check events

Send the feedback on a check result

Provide feedback on a check result in Data Quality.

For example, you enable a monitoring rule that is used to check whether the number of rows in a table exceeds 0 for a specific table in Data Quality. After the rule is triggered and the check is complete, users can provide their feedback on the check result. After the feedback is provided, the system sends the feedback that contains information such as the monitoring rule ID and feedback content to a subscriber.

Complete a check

Send data quality check results, including whether the check is passed and the sampling information.

For example, you enable a monitoring rule that is used to check whether the number of rows in a table exceeds 0 for a specific table in Data Quality. If the rule is triggered and the check is complete, the system sends the check result to a subscriber. The check result shows whether the data passes the quality check and contains samples that are used.

Event category	Event	Regular event	Extension point event
Appendix: Message formats
Appendix: Message formats

Tenant-level events

DataWorks console

Event category

Event

Regular event

Extension point event

Example

Workspace deletion events

Delete a workspace (pre-event)

A pre-event for workspace deletion is generated before a user deletes a workspace on the Workspaces page. The pre-event for workspace deletion can trigger the related extension to block the process.

Delete a workspace (post-event)

Data Upload and Download

Event category	Event	Regular event	Extension point event
Data download and upload events	Download data (pre-event) - file generation			A data download or upload event is generated when you download or upload data in Data Upload and Download. Before you download data, a pre-event for data download is generated to obtain information about the download operation.
	Download data (pre-event) - file download
	Upload data (pre-event)

Benefits

The OpenEvent module provides the following benefits:

Low O&M costs:
After you enable the message subscription feature in DataWorks Open Platform, you can subscribe to DataWorks events with ease.
Improved message publishing and receiving efficiency:
You need to only configure the parameters for message subscription and make preparations for SDK development. This reduces and simplifies manual operations.
Fulfillment of various needs:
You can subscribe to various types of DataWorks events. This allows you to identify important DataWorks changes and respond to the changes at the earliest opportunity.