This topic describes how to troubleshoot the issue that the number of Xceiver threads on a DataNode is limited.

Error message

java.io.IOException: Xceiver count xxxx exceeds the limit of concurrent xcievers: xxxx

Cause

The dfs.datanode.max.transfer.threads parameter is used to specify the size of the thread pool for a DataNode to process read and write data streams. Default value: 4096. If the value of this parameter is small, the number of Xceiver threads on a DataNode is limited in specific scenarios.

Solution

  1. Log on to the EMR on ECS console.
  2. In the top navigation bar, select the region where your cluster resides and select a resource group based on your business requirements.
  3. Go to the Configure tab of the Hadoop Distributed File System (HDFS) service.
    1. On the EMR on ECS page, find the cluster that you want to manage and click Services in the Actions column.
    2. On the page that appears, click the Services tab. Find the HDFS service and click Configure.
  4. On the Configure tab, search for the dfs.datanode.max.transfer.threads parameter and increase the value of the parameter appropriately. In most cases, we recommend that you double the value, such as 8192 or 16384.
    Note The dfs.datanode.max.transfer.threads parameter specifies the size of the thread pool for a DataNode to process read and write data streams. Default value: 4096.