Hologres:Blink and Flink FAQ and troubleshooting

Terms

• Hologres performance

  • Write performance

    • Column-oriented tables: InsertOrIgnore > InsertOrReplace > InsertOrUpdate

    • Row-oriented tables: InsertOrReplace = InsertOrUpdate > InsertOrIgnore

    Parameter	Description
    InsertOrIgnore	If the sink table has a primary key and a duplicate primary key is encountered during a real-time write, the later record is discarded.
    InsertOrReplace	If the sink table has a primary key and a duplicate primary key is encountered during a real-time write, the record is updated based on the primary key. If the new row does not contain all columns, the missing columns are set to null.
    InsertOrUpdate	If the sink table has a primary key and a duplicate primary key is encountered during a real-time write, the record is updated based on the primary key. If the new row does not contain all columns, the existing values for the missing columns are retained.

  • Point query performance

    Row-oriented storage outperforms row-column hybrid storage, which in turn outperforms column-oriented storage.

• Support for Blink, Flink (VVR), and open source Flink

  Form factor	Data storage type					Description
  	Source table	Sink table	Dimension table	Binary logging	Hologres Catalog
  Fully managed Flink	Supports row-oriented and column-oriented storage.	Supports row-oriented and column-oriented storage.	Use row-oriented storage.	Supported	Supported	None
  Blink Dedicated	Supports row-oriented and column-oriented storage.	Supports row-oriented and column-oriented storage.	Use row-oriented storage.	Hologres V0.8 supports only row-oriented storage. Hologres V0.9 and later support row-oriented and column-oriented storage. Use row-oriented storage.	Not supported	This product is being phased out. Use fully managed Flink on Alibaba Cloud.
  Open source Flink 1.10	Supports row-oriented and column-oriented storage.	Supports row-oriented and column-oriented storage.	None	Not supported	Not supported	None
  Open source Flink 1.11 and later	Supports row-oriented and column-oriented storage.	Supports row-oriented and column-oriented storage.	Use row-oriented storage.	Not supported	Not supported	Starting from open source Flink 1.11, the Hologres code is open source. For more information, see GitHub.

• The following example shows how to use SQL to map a Flink table to a Hologres table.

  create table holo_source(
  'hg_binlog_lsn' BIGINT HEADER,
  'hg_binlog_event_type' BIGINT HEADER,
  'hg_binlog_timestamp_us' BIGINT HEADER,
  A int,
  B int,
  C timestamp )
  with (
  type = 'hologres',
  'endpoint' = 'xxx.hologres.aliyuncs.com:80',   --The Endpoint of the Hologres instance.
  'userName' = '',                               --The AccessKey ID of your Alibaba Cloud account.
  'password' = '',                               --The AccessKey secret of your Alibaba Cloud account.
  'dbName' = 'binlog',                           --The name of the database in the Hologres instance.
  'tableName' ='test'                            --The name of the table in the Hologres instance.
  'binlog' = 'true',
  );

  Blink, VVR, and Flink SQL all declare a Flink table and then map it to a specific physical table in Hologres using parameters. Therefore, mapping to a foreign table is not supported.

Troubleshoot slow real-time writes

1. Confirm the configuration

   Check the following configuration information.

   • The storage format of the destination table, including row-oriented, column-oriented, and row-column hybrid store tables.

   • The insert mode, including InsertOrIgnore, InsertOrUpdate, and InsertOrReplace.

   • The Table Group and shard count of the destination table.

2. Check the real-time write latency metric

   If the average write latency is high—in the hundreds of milliseconds or even seconds—the backend has likely reached a write bottleneck. The following issues may exist.

   • You are using InsertOrUpdate for a column-oriented table, which performs partial updates. If traffic is high, this can lead to high CPU load and write latency for the instance.

     Solution: Change the table type. Use a row-oriented table. If your instance is V1.1 or later, you can choose a row-column hybrid store table.

   • Check the CPU load of the instance in Cloud Monitor. If CPU usage is close to 100% but there are no partial updates on column-oriented tables, the cause is usually high QPS queries or a high write volume.

     Solution: Scale out the Hologres instance.

   • Check if there are continuous Insert into select from commands that trigger BulkLoad writes to the table. BulkLoad writes currently block real-time writes.

     Solution: Convert BulkLoad writes to real-time writes, or run them during off-peak hours.

3. Check for data skew

   Use the following SQL command to check for data skew.

   SELECT hg_shard_id, count(1) FROM t1 GROUP BY hg_shard_id ORDER BY hg_shard_id;

   Solution: Modify the distribution key to distribute data more evenly.

4. Check for backend pressure

   If the previous steps do not reveal any issues but write performance suddenly drops, the backend cluster is likely under high pressure and experiencing a bottleneck. Contact technical support to confirm the situation. For more information, see How do I get more online support?.

5. Check for backpressure on the Blink/Flink side

   If the previous steps show no obvious issues on the Hologres side, the client is usually slow, meaning the Blink/Flink side is slow. Check if the sink node is experiencing backpressure. If the job has only one node, you cannot see if there is backpressure. In this case, separate the sink node and observe again. For details, contact Flink technical support.

Troubleshoot data write issues

This issue is usually caused by out-of-order data. For example, data with the same primary key is distributed across different Flink tasks, and the write order cannot be guaranteed. Check the Flink SQL logic to ensure that data is shuffled by the Hologres table's primary key before being written to Hologres.

Troubleshoot dimension table query issues

• Dimension table joins and dual-stream joins

  When reading from Hologres, first confirm whether you are using a dimension table join correctly and not mistaking a dual-stream join for it. The following is an example of using Hologres as a dimension table. If the keywords proctime AS PROCTIME() and hologres_dim FOR SYSTEM_TIME AS are missing, it becomes a dual-stream join.

  CREATE TEMPORARY TABLE datagen_source (
     a INT,
     b BIGINT,
     c STRING,
     proctime AS PROCTIME()
  ) with (
     'connector' = 'datagen'
  );
  
  CREATE TEMPORARY TABLE hologres_dim (
     a INT, 
     b VARCHAR, 
     c VARCHAR
  ) with (
     'connector' = 'hologres',
     ...
  );
  
  CREATE TEMPORARY TABLE blackhole_sink (
     a INT,
     b STRING
  ) with (
     'connector' = 'blackhole'
  );
  
  insert into blackhole_sink select T.a,H.b
  FROM datagen_source AS T JOIN hologres_dim FOR SYSTEM_TIME AS OF T.proctime AS H ON T.a = H.a;

• Dimension table queries

  1. Check the dimension table storage format

     Check if the dimension table is a row-oriented, column-oriented, or row-column hybrid store table.

  2. High latency in dimension table queries

     The most common issue with dimension tables is backpressure on the join node on the Flink/Blink side, which reduces the throughput of the entire job.

     1. Check the Flink dimension table join mode

        The dimension table join feature of the Hologres Flink Connector supports synchronous and asynchronous modes. The asynchronous mode performs better than the synchronous mode. You can distinguish them by checking the Flink SQL. The following is an example of an SQL statement that enables asynchronous dimension table queries.

        CREATE TABLE hologres_dim(
         id INT,
         len INT,
         content VARCHAR
        ) with (
          'connector'='hologres',
          'dbname'='<yourDbname>',  --The name of the Hologres database.
          'tablename'='<yourTablename>',  --The name of the table in Hologres that receives data.
          'username'='<yourUsername>',  --The AccessKey ID of your Alibaba Cloud account.
          'password'='<yourPassword>',  --The AccessKey secret of your Alibaba Cloud account.
          'endpoint'='<yourEndpoint>'  --The VPC Endpoint of your Hologres instance.
          'async' = 'true'--Asynchronous mode
        );

     2. Check the backend query latency

        Check the real-time write latency metric:

        • Check if a column-oriented table is used as a dimension table. Dimension tables in column-oriented format have high overhead in high-QPS scenarios.

        • If it is a row-oriented table and the latency is high, the overall load on the instance is usually high. You need to scale out the instance.

  3. Check if the join key is the primary key of the Hologres table

     Starting from Ververica Runtime (VVR) 4.x (Flink 1.13), the Hologres Connector supports non-primary key queries on Hologres tables based on Holo Client. This usually results in poor performance and high instance load, especially if the table schema is not optimized. In this case, guide the user to optimize the table schema. The most common optimization is to set the join key as the distribution key to enable shard pruning.

  4. Check for backpressure on the Blink side

     If the previous steps show no obvious issues on the Hologres side, the client is usually slow, meaning the Blink side is slow. Check if the sink node is experiencing backpressure. If the job has only one node, you cannot see if there is backpressure. In this case, separate the sink node and observe again. You can also check if the join node is causing backpressure. For details, contact Flink technical support.

Notes on connections

Common errors