ApsaraDB for MongoDB:Troubleshoot high disk usage on an ApsaraDB for MongoDB instance

View storage usage

Replica set instance

You can use the following methods to view the disk usage on a replica set instance in the ApsaraDB for MongoDB console:

• Overview

  In the Specification Information section of the Basic Information page, view the Disk Space and Utilization information of the instance.

• Detail analysis by using monitoring charts

  In the left-side navigation pane, click Monitoring Data. On the page that appears, specify a node and view the Disk Usage (Bytes) and Disk Usage (%) values of the node.

  A replica set instance consists of a primary node that supports read and write operations, one or more high-availability secondary nodes, a hidden node, and one or more optional read-only nodes. The disk of a node is used by data and logs and the storage capacity of the disk can be calculated based on the following formula: ins_size = data_size + log_size. Parameters that you need to configure:

  • data_size indicates the disk space used by data files such as physical data files whose names start with collection, physical index files whose names start with index, and some physical metadata files such as WiredTiger.wt. The data files exclude the local database.

  • log_size indicates the disk space used by the local database, runtime logs, and some audit logs.

• Detail analysis

  You can use the following methods to view the details of disk usage:

  • Run the db.stats() and db.$collection_name.stats() commands provided by MongoDB.

    For more information, see the following references:

    • db.stats()

    • db.collection.stats()

    • db.collection.storageSize()

    • db.collection.totalIndexSize()

    • db.collection.totalSize()

  • Choose CloudDBA > Storage Analysis and view details on the Storage Analysis page.

    On the Storage Analysis page, you can view the following items:

    • Overview of the disk usage of databases and tables, average daily increment, and predicted available days of storage

    • Disk usage of abnormal databases and tables

    • Details of the disk usage of a specific business table, including the disk space used by index files and data files, compression ratio, and average row size

Sharded cluster instance

Troubleshoot large data volume caused by the compact command

Compact operation and its effects on an instance

The execution duration of the compact command is related to the data volume of a collection. If the data volume is large, the compact command runs for a long period of time. Therefore, we recommend that you run the compact command during off-peak hours.

Compact operation

Invalid compact operation

Troubleshoot high space usage caused by a large amount of log data

The gap between the spaces used by the primary and secondary nodes is large due to a large number of journal logs

2019-08-25T09:45:16.867+0800 I NETWORK [thread1] Listener: accept() returns -1 Too many open files in system 
2019-08-25T09:45:17.000+0800 I - [ftdc] Assertion: 13538:couldn't open [/proc/55692/stat] Too many open files in system src/mongo/util/processinfo_linux.cpp 74
2019-08-25T09:45:17.002+0800 W FTDC [ftdc] Uncaught exception in 'Location13538: couldn't open [/proc/55692/stat] Too many open files in system' in full-time diagnostic data capture subsystem. Shutting down the full-time diagnostic data capture subsystem.

Used log space of secondary nodes may continuously increase in size due to latency on the secondary nodes and incremental backup

If latency occurs during synchronization between primary and secondary nodes, the available space of oplogs is not limited by the fixed collection size defined in the configuration file. In theory, the available space can reach 20% of the disk space for which you apply. However, after the latency on the secondary nodes has elapsed, the physical space used by oplogs is not released.

When you perform physical backups of an instance on a hidden node, a large number of checkpoints generate large volumes of data and occupy large log space.

db.grantRolesToUser("root", [{db: "local", role: "dbAdmin"}])
use local
db.runCommand({ compact: "oplog.rs", force: true })

Troubleshoot uneven data distribution caused by unreasonable sharding

Data is unevenly distributed due to unreasonable selection of sharding key types

Data is unevenly distributed due to unreasonable selection of sharding key fields

2019-08-27T13:31:22.076+0800 W SHARDING [conn12681919] possible low cardinality key detected in superHotItemPool.haodanku_all - key is { batch: "201908260000" } 
2019-08-27T13:31:22.076+0800 W SHARDING [conn12681919] possible low cardinality key detected in superHotItemPool.haodanku_all - key is { batch: "201908260200" } 
2019-08-27T13:31:22.076+0800 W SHARDING [conn12681919] possible low cardinality key detected in superHotItemPool.haodanku_all - key is { batch: "201908260230" }

The MongoDB balancer monitors the number of chunks on each shard regardless of the data volume. In this case, the number of chunks on each shard is balanced, but the data may be severely skewed. In a chunk, sharding keys are almost the same. When the chunk size reaches 64 MB, an empty chunk is created. This way, the number of chunks increases and chunk migration is complete. However, migrated chunks are empty chunks. As a result, shards may have an equal number of chunks but have largely different data sizes. In this case, you must redesign sharding keys by using appropriate columns that have a high degree of discrimination.

Jumbo shards arise from unsharded databases

You can shard only some databases in an ApsaraDB for MongoDB instance. This way, data in an unsharded database is stored in the same shard. If large volumes of data are stored in the database, the data volume in the populated shard is much larger than that in other shards.

In another case, when data is logically imported from a source ApsaraDB for MongoDB instance to a destination ApsaraDB for MongoDB instance, the destination ApsaraDB for MongoDB instance may not be sharded.

Uneven disk usage is caused by multiple movechunk operations

The movechunk operation is used to remove the source data after data is written to destination shards. By default, the remove operation does not release space. Each table has data files and index files in an instance that runs the wiredTiger engine. If the files are not deleted, the occupied space is not released. In most cases, this issue occurs because sharding is implemented in an instance after the instance runs for a period of time.

In theory, the space fragmentation caused by the movechunk operation is similar to the one caused by the operation in which large volumes of data are deleted. Therefore, if a large number of movechunk or remove operations are required on a shard, you can perform the compact operation on the shard to release the space used by fragments.