Impact of a data shard failure on the request success rate

Assume that the number of data shards in a cluster instance is N. When one of the data shards fails, a master-replica failover is initiated on the faulty data shard. The entire failover process takes anywhere from a few seconds to tens of seconds. During this process, all requests sent to the data shard fail. If the requests are evenly distributed across all data shards, the instance theoretically has a request failure rate of 1/N.

However, the actual number of failed requests may be higher than the theoretical value for the following reasons. In the example, a cluster instance that contains two data shards and runs in proxy mode is used.

• Commands that involve multiple keys are used

  • Proxy mode: The proxy server splits multi-key commands into multiple subcommands and routes them to the corresponding data shards based on the route table.

  • Direct connection mode: The client sends each subcommand to the corresponding data shard.

  When a multi-key command involves a data shard that experiences a failure, the entire request tends to fail, as shown in the following figure.

  

  Optimization solution: Minimize the number of keys in a single command to reduce the probability of multi-key command failures when a data shard fails.

• The client uses a single connection

  Some clients such as Lettuce send requests asynchronously over a single connection. For example, in the scenario where two GET requests are sent sequentially to a proxy server over one connection, Redis Serialization Protocol (RESP) requires that responses be returned in the same order as the requests were sent. If the GET key2 request fails due to a node failure, the client cannot receive the response to the request that follows GET key2 even if the request is successfully executed.

  

  Optimization solution: Use clients such as Jedis that support the connection pool model.

• Connection pool resources on the client are exhausted

  For clients that use the connection pool model, the maximum number of connections allowed can be configured. When the number of connections reaches the maximum limit and no idle connections are available, new requests fail or are blocked.

  In the following figure, the Jedis client is used. If maxTotal is set to 3, timeout is set to 2000, and three GET key2 requests are initiated within 2 seconds, the connection pool reaches its maximum capacity with all connections in use after 2 seconds. If one of the nodes fails to respond, new requests either fail or be blocked based on the blockWhenExhausted configuration, leading to the client-side requests either failing or timing out.

  

  Optimization solution: Configure the JedisPool resource pool size appropriately and set the timeout parameter to a lower value, such as 200 to 300 milliseconds (instead of the default 2,000 milliseconds). This adjustment helps achieve faster failure responses and prevents a large number of connections from being blocked when a failure occurs.