Realtime Compute for Apache Flink:Reference

How do I upload a resource file in the OSS console?

1. In the management console of Realtime Compute for Apache Flink, view the information about the Object Storage Service (OSS) bucket that is associated with your workspace.

   

   The following figure shows information about the OSS bucket.

2. Log on to the OSS console and upload the resource file to the /artifacts/namespaces directory of the OSS bucket.

3. In the left-side navigation pane of the development console of Realtime Compute for Apache Flink, click Artifacts to view the resource file that you uploaded in the OSS console.

   

How do I configure parameters for deployment running?

1. Log on to the Realtime Compute for Apache Flink console.

2. Find the workspace that you want to manage and click Console in the Actions column.

3. In the left-side navigation pane, click O&M > Deployments. On the Deployments page, click the name of the deployment that you want to manage.

4. In the upper-right corner of the Parameters section on the Configuration tab, click Edit.

5. In the Parameters section, add the following related configurations to the Other Configuration field.

   Make sure that a space exists after the colon (:) between each key-value pair. Sample code:

   task.cancellation.timeout: 180s

6. In the upper-right corner of the Parameters section, click Save.

How do I enable GC logs?

env.java.opts:>--XX: +PrintGCDetails-XX:+PrintGCDateStamps-Xloggc:/flink/log/gc.log-XX:+UseGCLogFileRotation-XX:NumberOfGCLogFiles=2-XX:GCLogFileSize=50M

How do I find the deployment that triggers an alert?

How do I view information about a workspace, such as the workspace ID?

How do I view information about the AccessKey ID and AccessKey secret of the account?

How do I query the engine version of Realtime Compute for Apache Flink that is used by a deployment?

How do I deactivate Prometheus Service that is automatically activated with Realtime Compute for Apache Flink?

If you no longer need to monitor Realtime Compute for Apache Flink, perform the following steps to deactivate Prometheus Service:

1. Log on to the Prometheus Service console.

2. In the left-side navigation pane, click Instances to go to the Prometheus Service page.

3. Select the ID or name of the workspace that you want to manage from the Filter by Tag drop-down list.

4. Find the instance whose Instance Type is Prometheus for Flink Serverless and click Uninstall in the Actions column.

5. In the message that appears, click OK.

How do I configure a whitelist?

What do I do if my account does not have the required permissions when I log on to the Realtime Compute for Apache Flink console?

• Problem description

  When I log on to the Realtime Compute for Apache Flink console, the error message that is shown in the following figure appears.

  

• Cause

  You do not have the permissions to view Realtime Compute for Apache Flink workspaces, or you have only the permissions on a specific resource group.

• Solution

  • If you have only permissions on a specific resource group, select the related resource group in the upper part of the console and select the region to view the desired workspace.

    

  • If you do not have the permissions to view Realtime Compute for Apache Flink workspaces, grant the related permissions to the RAM user or the Alibaba Cloud account to which the RAM role is assigned. For more information, see Grant permissions to a RAM user.

    You can select one of the following policies that include permissions to view Realtime Compute for Apache Flink workspaces:

    • System policy: AliyunStreamReadOnlyAccess (permissions to access Realtime Compute for Apache Flink in read-only mode) or AliyunStreamFullAccess (all permissions on Realtime Compute for Apache Flink). For more information, see System policies.

    • Custom policy: stream:DescribeVvpInstances (permissions to view workspaces). For more information, see Custom policies.

How does Realtime Compute for Apache Flink access the Internet?

• Background information

  By default, Realtime Compute for Apache Flink cannot access the Internet. Therefore, Alibaba Cloud provides NAT gateways to enable communications between virtual private clouds (VPCs) and the Internet. This way, users of Realtime Compute for Apache Flink can access the Internet by using user-defined functions (UDFs) or DataStream code.

• Solution

  Create a NAT gateway in the VPC. Then, create a source network address translation (SNAT) entry to bind the vSwitch that is associated with Realtime Compute for Apache Flink to an elastic IP address (EIP). This way, the service can access the Internet by using the EIP. To access the Internet by using the EIP, perform the following steps:

  1. Create a NAT gateway. For more information, see Create a NAT gateway.

  2. Create an SNAT entry and bind the vSwitch that is associated with Realtime Compute for Apache Flink to an EIP. For more information, see Create and manage SNAT entries.

How does Realtime Compute for Apache Flink access a service across VPCs?

How do I use the network detection feature in the development console of Realtime Compute for Apache Flink?

Realtime Compute for Apache Flink supports the network detection feature. To use the network detection feature, perform the following steps in the development console of Realtime Compute for Apache Flink:

1. Log on to the Realtime Compute for Apache Flink console.

2. Find the workspace that you want to manage and click Console in the Actions column.

3. In the top navigation bar, click the Network detection icon.

   

4. In the Network detection dialog box, configure the Host parameter to specify an IP address or endpoint to check whether the running environment of a Realtime Compute for Apache Flink deployment is connected to the upstream and downstream systems.

   Important

   If you specify an endpoint, remove :<port> from the end of the endpoint and enter <port> in the Port field in the Network detection dialog box.

   

How do I troubleshoot dependency conflicts of Realtime Compute for Apache Flink?

• Problem description

  • An error is reported, which is caused by an exception related to Realtime Compute for Apache Flink or Hadoop.

    java.lang.AbstractMethodError
    java.lang.ClassNotFoundException
    java.lang.IllegalAccessError
    java.lang.IllegalAccessException
    java.lang.InstantiationError
    java.lang.InstantiationException
    java.lang.InvocationTargetException
    java.lang.NoClassDefFoundError
    java.lang.NoSuchFieldError
    java.lang.NoSuchFieldException
    java.lang.NoSuchMethodError
    java.lang.NoSuchMethodException

  • No error is reported, but one of the following issues occur:

    • Logs are not generated or the Log4j configuration does not take effect.

      In most cases, this issue occurs because the dependency contains the Log4j configuration. To resolve this issue, you must check whether the dependency in the JAR file of your deployment contains the Log4j configuration. If the dependency contains the Log4j configuration, you can configure exclusions in the dependency to remove the Log4j configuration.

      Note

      If you use different versions of Log4j, you must use maven-shade-plugin to relocate Log4j-related classes.

    • The remote procedure call (RPC) fails.

      By default, errors caused by dependency conflicts during Akka RPCs of Realtime Compute for Apache Flink are not recorded in logs. To check these errors, you must enable debug logging.

      For example, a debug log records Cannot allocate the requested resources. Trying to allocate ResourceProfile{xxx}. However, the JobManager log stops at the message Registering TaskManager with ResourceID xxx and does not display any information until a resource request times out and displays the message NoResourceAvailableException. In addition, TaskManagers continuously report the error message Cannot allocate the requested resources. Trying to allocate ResourceProfile{xxx}.

      Cause: After you enable debug logging, the RPC error message InvocationTargetException appears. In this case, slots fail to be allocated for TaskManagers and the status of the TaskManagers becomes inconsistent. As a result, slots cannot be allocated and the error cannot be fixed.

• Cause

  • The JAR package of your deployment contains unnecessary dependencies, such as the dependencies for basic configurations, Realtime Compute for Apache Flink, Hadoop, and Log4j. As a result, dependency conflicts occur and cause some issues.

  • The dependency that corresponds to the connector that is required for your deployment is not included in the JAR package.

• Troubleshooting

  • Check whether the pom.xml file of your deployment contains unnecessary dependencies.

  • Run the jar tf foo.jar command to view the content of the JAR package and determine whether the package contains the content that causes dependency conflicts.

  • Run the mvn dependency:tree command to check the dependency relationship of your deployment and determine whether dependency conflicts exist.

• Solution

  • We recommend that you set scope to provided for the dependencies for basic configurations. This way, the dependencies for basic configurations are not included in the JAR package of your deployment.

    • DataStream Java

      <dependency>
        <groupId>org.apache.flink</groupId>
        <artifactId>flink-streaming-java_2.11</artifactId>
        <version>${flink.version}</version>
        <scope>provided</scope>
      </dependency>

    • DataStream Scala

      <dependency>
        <groupId>org.apache.flink</groupId>
        <artifactId>flink-streaming-scala_2.11</artifactId>
        <version>${flink.version}</version>
        <scope>provided</scope>
      </dependency>

    • DataSet Java

      <dependency>
        <groupId>org.apache.flink</groupId>
        <artifactId>flink-java</artifactId>
        <version>${flink.version}</version>
        <scope>provided</scope>
      </dependency>

    • DataSet Scala

      <dependency>
        <groupId>org.apache.flink</groupId>
        <artifactId>flink-scala_2.11</artifactId>
        <version>${flink.version}</version>
        <scope>provided</scope>
      </dependency>

  • Add the dependencies that correspond to the connectors required for the deployment, and set scope to compile. This way, the dependencies that correspond to the required connectors are included in the JAR package. The default value of scope is compile. In the following code, the Kafka connector is used as an example.

    <dependency>
        <groupId>org.apache.flink</groupId>
        <artifactId>flink-connector-kafka_2.11</artifactId>
        <version>${flink.version}</version>
    </dependency>

  • We recommend that you do not add the dependencies for Realtime Compute for Apache Flink, Hadoop, or Log4j. Take note of the following exceptions:

    • If the deployment has direct dependencies for basic configurations or connectors, we recommend that you set scope to provided. Sample code:

      <dependency>
          <groupId>org.apache.hadoop</groupId>
          <artifactId>hadoop-common</artifactId>
          <scope>provided</scope>
      </dependency>

    • If the deployment has indirect dependencies for basic configurations or connectors, we recommend that you configure exclusions to remove the dependencies. Sample code:

      <dependency>
          <groupId>foo</groupId>
            <artifactId>bar</artifactId>
            <exclusions>
              <exclusion>
              <groupId>org.apache.hadoop</groupId>
              <artifactId>hadoop-common</artifactId>
             </exclusion>
          </exclusions>
      </dependency>

How do I resolve the domain name of the service on which a Realtime Compute for Apache Flink deployment depends?