Realtime Compute for Apache Flink allows you to view metrics to analyze a deployment. If existing metrics do not meet your business requirements, you can configure custom metrics. The metrics can be reported to other platforms. This topic describes how to configure custom metrics and report the metrics to a self-managed Prometheus service, Simple Log Service, Kafka, a self-managed platform, or multiple platforms.
Precautions
You can configure parameters to report metrics only to a self-managed Prometheus service, Simple Log Service, or Kafka. You can also configure parameters to collect and report metrics to multiple platforms.
Report metrics to multiple platforms
If you want to collect and report metrics to multiple platforms, you must modify the configurations of the metrics.reporters parameters. If metrics are reported to multiple platforms, the metric collection cost increases. For more information, see Report metrics to multiple platforms.
Report metrics to a platform other than the Realtime Compute for Apache Flink console
Application Real-Time Monitoring Service (ARMS) or CloudMonitor is no longer activated together with Realtime Compute for Apache Flink. In this case, metrics are not displayed in the Realtime Compute for Apache Flink console. You can view the metrics on the platform to which the metrics are reported.
The existing alert configurations in the Realtime Compute for Apache Flink console become invalid. For more information, see Configure alert rules. You can no longer configure alert rules in the Realtime Compute for Apache Flink console. You must configure alert rules on the platform to which the metrics are reported.
Platforms to which custom metrics are reported
Report metrics to a self-managed Prometheus service
Make sure that a network connection is established between Realtime Compute for Apache Flink and the self-managed Prometheus service. Then, log on to the Realtime Compute for Apache Flink console and perform the following operations: In the left-side navigation pane, click Deployments. On the Deployments page, click the name of the desired deployment in the Name column. On the Configuration tab, click Edit in the upper-right corner of the Parameters section, add the following code to the Other Configuration field, and then change the values of the parameters related to the self-managed Prometheus service. For more information about the configuration method, see How do I configure parameters for deployment running?
metrics.reporters: promgatewayappmgr
metrics.reporter.promgatewayappmgr.groupingKey: 'deploymentName={{deploymentName}};deploymentId={{deploymentId}};jobId={{jobId}}'
metrics.reporter.promgatewayappmgr.jobName: '{{deploymentName}}'
metrics.reporter.promgatewayappmgr.factory.class: org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporterFactory
metrics.reporter.promgatewayappmgr.host: pushgateway host
metrics.reporter.promgatewayappmgr.port: pushgateway portIn the code, you must replace pushgateway host with the hostname of your Pushgateway and replace pushgateway port with the port number of your Pushgateway. The system automatically replaces the values of the deploymentName, deploymentId, and jobId parameters.
The network connection that is established between Realtime Compute for Apache Flink and a self-managed Prometheus service must meet the following requirements:
If the self-managed Prometheus service resides in the same virtual private cloud (VPC) as the Realtime Compute for Apache Flink workspace, you must make sure that the CIDR block of Realtime Compute for Apache Flink can access the security group of the self-managed Prometheus service.
If the self-managed Prometheus service has a public endpoint and resides in a different VPC from the Realtime Compute for Apache Flink workspace, you must configure Internet access for Realtime Compute for Apache Flink. For more information, see How does Realtime Compute for Apache Flink access the Internet?
If the self-managed Prometheus service has only a VPC endpoint and resides in a different VPC from the Realtime Compute for Apache Flink workspace, you must establish a connection between the VPCs. For more information, see How does Realtime Compute for Apache Flink access a service across VPCs?
Report metrics to Simple Log Service
Make sure that a network connection is established between Realtime Compute for Apache Flink and Simple Log Service. Then, log on to the Realtime Compute for Apache Flink console and perform the following operations: In the left-side navigation pane, click Deployments. On the Deployments page, click the name of the desired deployment in the Name column. On the Configuration tab, click Edit in the upper-right corner of the Parameters section, add the following code to the Other Configuration field, and then change the values of the parameters related to Simple Log Service. For more information about the configuration method, see How do I configure parameters for deployment running?
metrics.reporters: sls
metrics.reporter.sls.factory.class: org.apache.flink.metrics.sls.SLSReporterFactory
metrics.reporter.sls.project: the project of Simple Log Service
metrics.reporter.sls.logStore: the Logstore of Simple Log Service
metrics.reporter.sls.endPoint: the endpoint of Simple Log Service
metrics.reporter.sls.accessId: the AccessKey ID that is used to connect to Simple Log Service
metrics.reporter.sls.accessKey: the AccessKey secret that is used to connect to Simple Log Service
metrics.reporter.sls.extraTags: deploymentId={{ deploymentId }};deploymentName={{ deploymentName}};namespace={{ namespace }}In the code, you must configure the parameters related to Simple Log Service. The system automatically replaces the values of the deploymentId, deploymentName, and namespace parameters. For more information about the AccessKey pair of an account, see How do I view information about the AccessKey ID and AccessKey secret of the account?
Report metrics to Kafka
Make sure that a network connection is established between Realtime Compute for Apache Flink and Kafka. Then, log on to the Realtime Compute for Apache Flink console and perform the following operations: In the left-side navigation pane, click Deployments. On the Deployments page, click the name of the desired deployment in the Name column. On the Configuration tab, click Edit in the upper-right corner of the Parameters section, add the following code to the Other Configuration field, and then change the values of the parameters related to Kafka. For more information about the configuration method, see How do I configure parameters for deployment running?
metrics.reporters: monitor
metrics.reporter.monitor.factory.class: org.apache.flink.metrics.monitor.KafkaReporterFactory
metrics.reporter.monitor.kafka.bootstrap.servers: the addresses of the Kafka brokers
metrics.reporter.monitor.topicName: the name of the Kafka topic
metrics.reporter.monitor._FLINK_CLUSTER_NAME: '{{ deploymentName }}'
metrics.reporter.monitor._JOB_NAME: '{{ deploymentName }}'
metrics.reporter.monitor._NAMESPACE_NAME: '{{ namespace }}'In the code, you must configure the parameters related to the self-managed Kafka cluster. The system automatically replaces the values of the deploymentName and namespace parameters.
Report metrics to multiple platforms
Call an ARMS API operation to report metrics to a self-managed platform
If you select Alibaba Cloud Managed Service for Prometheus when you purchase the Realtime Compute for Apache Flink workspace, you can call an ARMS API operation to obtain the metrics of Realtime Compute for Apache Flink and integrate the metrics to your self-managed platform. In this case, you can view metrics on your self-managed platform. You can also view metric curves and configure alert rules in the Realtime Compute for Apache Flink console.
For more information about ARMS API operations, see List of API operations by feature. For more information about the metrics of operators, see Operator Metric.
Report metrics to the Realtime Compute for Apache Flink console and other platforms
If you want to view metric curves and configure alert rules in the Realtime Compute for Apache Flink console and also view metrics on other platforms, you can configure parameters to report metrics to multiple platforms. This increases the metric collection cost.
For example, if you want to view metrics in the Realtime Compute for Apache Flink console and Simple Log Service, you can log on to the Realtime Compute for Apache Flink console and perform the following operations: In the left-side navigation pane, click Deployments. On the Deployments page, click the name of the desired deployment in the Name column. On the Configuration tab, click Edit in the upper-right corner of the Parameters section, add the following code to the Other Configuration field, and then change the values of the parameters related to Simple Log Service.
metrics.reporters: jmx,promappmgr,sls
metrics.reporter.sls.factory.class: org.apache.flink.metrics.sls.SLSReporterFactory
metrics.reporter.sls.project: the project of Simple Log Service
metrics.reporter.sls.logStore: the Logstore of Simple Log Service
metrics.reporter.sls.endPoint: the endpoint of Simple Log Service
metrics.reporter.sls.accessId: the AccessKey ID that is used to connect to Simple Log Service
metrics.reporter.sls.accessKey: the AccessKey secret that is used to connect to Simple Log Service
metrics.reporter.sls.extraTags: deploymentId={{ deploymentId }};deploymentName={{ deploymentName}};namespace={{ namespace }}If the metrics.reporters parameters contain jmx,promappmgr, you can view metrics in the Realtime Compute for Apache Flink console. For more information about other parameters, see Report metrics to Simple Log Service.
Report metrics to multiple platforms other than the Realtime Compute for Apache Flink console
After metrics are reported to multiple platforms other than the Realtime Compute for Apache Flink console, metrics are not displayed in the Realtime Compute for Apache Flink console. You must view the metrics on the platforms to which the metrics are reported. For more information, see Precautions.
For example, if you want to report metrics to Simple Log Service and Kafka, you can add the following code to the Other Configuration field in the Parameters section of the Configuration tab in the Realtime Compute for Apache Flink console.
metrics.reporters: sls,monitor
metrics.reporter.sls.factory.class: org.apache.flink.metrics.sls.SLSReporterFactory
metrics.reporter.sls.project: the project of Simple Log Service
metrics.reporter.sls.logStore: the Logstore of Simple Log Service
metrics.reporter.sls.endPoint: the endpoint of Simple Log Service
metrics.reporter.sls.accessId: the AccessKey ID that is used to connect to Simple Log Service
metrics.reporter.sls.accessKey: the AccessKey secret that is used to connect to Simple Log Service
metrics.reporter.sls.extraTags: deploymentId={{ deploymentId }};deploymentName={{ deploymentName}};namespace={{ namespace }}
metrics.reporter.monitor.factory.class: org.apache.flink.metrics.monitor.KafkaReporterFactory
metrics.reporter.monitor.kafka.bootstrap.servers: the addresses of the Kafka brokers
metrics.reporter.monitor.topicName: the name of the Kafka topic
metrics.reporter.monitor._FLINK_CLUSTER_NAME: '{{ deploymentName }}'
metrics.reporter.monitor._JOB_NAME: '{{ deploymentName }}'
metrics.reporter.monitor._NAMESPACE_NAME: '{{ namespace }}'For more information, see Report metrics to Simple Log Service and Report metrics to Kafka.