Alibaba Cloud Linux:Usage notes of SysAK

Background information

SysAK does not add to the load overheads on the system or cause network jitters. All SysAK tools can run simultaneously at CPU utilization of up to 3%, and each SysAK tool runs at CPU utilization of up to 1%.

Install SysAK

Environment requirements

• Supported operating systems: Linux operating systems with kernel version 3.10 or later, such as Alibaba Cloud Linux 2/3, Anolis OS 8.4 ANCK, and CentOS 7.

  Note

  You can run the uname -a command to check the kernel version of the operating system of an Elastic Compute Service (ECS) instance.

• Architecture: x86_64

Installation and deployment

Use SysAK

Usage

Two modes of SysAK

SysAK supports the monitoring and diagnostics modes. In monitoring mode, SysAK monitors the system in the background and collects and tracks metrics for O&M personnel. The diagnostics mode can be enabled at any time and is used to analyze the root causes of system issues. For more information, see the System monitoring feature of SysAK and Diagnostics feature of SysAK sections of this topic.

System monitoring feature of SysAK

Enable monitoring

Monitoring metrics

• System resources

  Metric type	Category and metric		Description	Provider of enhanced features
  Computing resource	CPU	user	The user-mode CPU utilization.	None
  		sys	The system-mode CPU utilization.
  		hirq	The CPU utilization when the CPU is servicing hardware interrupts.
  		sirq	The CPU utilization when the CPU is servicing software interrupts.
  	LOAD	load*	The average load of the system in the past 1 second, 5 seconds, or 15 seconds.
  Memory resource	Memory	free	The amount of memory that is not used.
  		used	The amount of memory that is used.
  		buffer	The amount of memory that is used as buffers.
  		cache	The amount of memory that is used as cache.
  		total	The total amount of memory.
  		mem.util	The memory usage.
  	swap	swpin	The number of pages swapped in.
  		swapout	The number of pages swapped out.
  		total	The total number of swap pages.
  		swap.util	The swap usage.
  I/O resource	I/O access	rrqms	The number of merged read requests sent to the device per second.
  		wrqms	The number of merged write requests sent to the device per second.
  		rs	The number of read requests sent to the device per second.
  		ws	The number of write requests sent to the device per second.
  		rsecs	The number of sectors read from the device per second.
  		wsecs	The number of sectors written to the device per second.
  		rqsize	The average size of requests sent to the device.
  		qusize	The average queue length of requests sent to the device.
  		svctm	The average service duration of I/O requests sent to the device.
  		io.util	The percentage of CPU time during which requests are issued.
  	Disk space	bfree	The number of data blocks that are not used.
  		bused	The number of data blocks that are used.
  		btotl	The total number of data blocks.
  		patition.util	The database usage.
  		ifree	The number of available inodes.
  		itotl	The total number of inodes.
  		iutil	The inode usage.
  Network resource	Network traffic	bytin	The number of received bytes.
  		bytout	The number of sent bytes.
  		pktin	The total number of received packets.
  		pktout	The total number of sent packets.
  	TCP	active	The number of active TCP connections.
  		pasive	The number of passive TCP connections.
  		iseg	The number of received TCP packets.
  		outseg	The number of sent TCP packets.
  	UDP	idgm	The number of received UDP packets.
  		odgm	The number of sent UDP packets.

• System bottlenecks

  Metric type	Category and metric		Description	Provider of enhanced features
  I/O bottleneck	Read/write latency	await	The average waiting time of I/O operations.	None
  		rawait	The average waiting time of I/O read operations.
  		wawait	The average waiting time of I/O write operations.
  Memory bottleneck	Cache reclaim and defragmentation	kswapd	The number of times that the Kernel Swap Daemon (kswapd) reclaims pages.
  		pg_kr	The number of pages that are asynchronously reclaimed.
  		pg_dr	The number of pages that are directly reclaimed.
  		kcompd	The number of times that kcompactd compacts memory.
  		dc_all	The number of times that memory is directly compacted.
  		dc_fin	The number of times that direct memory compaction is completed.
  		oom	The number of out-of-memory (OOM) errors.
  Network bottleneck	Network transmission	pkterr	The number of error packets.
  		pktdrp	The number of packets dropped.
  		EstReset	The number of times that reset occurs when a TCP connection is in the ESTABLISHED state.
  		AtmpFail	The number of failed attempts to establish a TCP connection.
  		retran	The TCP retransmission rate.
  		noport	The number of times that UDP ports or addresses do not exist.
  		idmerr	The number of invalid UDP packets.
  CPU bottleneck	Multitask concurrency	cswch	The number of times that context switching is performed on CPU resources.
  		proc	The number of fork system calls.
  	Extended waiting period of ready queues	rqslow.dltnum	The number of times that the amount of time spent in the ready queue exceeds the threshold.	SysAK
  		rqslow.dlttm	The total latency experienced when the amount of time spent in the ready queue exceeds the threshold.	SysAK
  System software bottleneck	Access to critical resources in kernel	noschd.dltnum	The number of times that the amount of time spent by a CPU in system mode exceeds the threshold.	SysAK
  		noschd.dlttm	The total latency experienced when the amount of time spent by a CPU in system mode exceeds the threshold.	SysAK

• System interruptions

  Metric type	Category and metric		Description	Provider of enhanced features
  System interruption	Interrupt disable latency for a longer period of time	irqoff.dltnum	The number of times that the interrupt disable period of the system exceeds the threshold.	SysAK
  		irqoff.dlttm	The total latency experienced when the interrupt disable period of the system exceeds the threshold.	SysAK

• Container scenarios (metrics for each container)

  Metric type	Category and metric		Description	Provider of enhanced features
  Computing resource	CPU resource	usr/sys/hriq/sirq	The CPU utilization in user mode, in system mode, when the CPU is servicing hardware interrupts, and when the CPU is servicing software interrupts.	None
  	Load information	nrun	The number of tasks that are ready in the container.	Alibaba Cloud Linux and Anolis OS
  		nunint	The number of tasks that are in the D block state in the container.
  		load*	The average load of the container in the past 1 second, 5 seconds, or 10 seconds.
  Memory resource	Memory resource	total/free/used/cache/buffer	The total amount of memory, the amount of available memory, the amount of used memory, the amount of memory that is being used for caches, and the amount of memory that is being used for buffers in the container.	None
  	Memory bottleneck	pgfault	The number of times that page faults occur in the container.
  		pgmajfault	The number of times that page faults occur due to disk swapping or file mappings.
  		mfailcnt	The number of failed attempts to request memory in the container.
  		drgl*	The latency distribution of global memory reclaim.	Alibaba Cloud Linux and Anolis OS
  		drml*	The latency distribution of memory reclaim in the container.
  		dcl*	The latency distribution of memory compaction in the container.
  I/O resource	I/O metric	riops	The number of I/O read operations in the container.	None
  		wiops	The number of I/O write operations in the container.
  		rbps	The number of bytes read from the container.
  		wbps	The number of bytes written to the container.
  		rwait	The waiting time of read operations in the container.	Alibaba Cloud Linux and Anolis OS
  		wwait	The waiting time of write operations in the container.
  		rsrv	The read service time in the container.
  		wsrv	The write service time in the container.
  		rioq	The number of I/O read operations queued in the container.
  		wioq	The number of I/O write operations queued in the container.
  		rioqsz	The number of bytes read in I/O operations queued in the container.
  		wioqsz	The number of bytes written in I/O operations queued in the container.
  		rarqsz	The average number of bytes read in I/O operations in the container.
  		warqsz	The average number of bytes written in I/O operations in the container.
  Hardware resource	Resource bottleneck	llcref	The number of times that Last level cache (LLC) is accessed in the container.	None
  		llcmis	The number of LLC misses in the container.
  		CPI	The cycles per instruction in the container.

Service configurations

The monitoring feature of SysAK allows you to configure collected metrics by using the configuration file of the metrics. The configuration file is stored in the /usr/local/sysak/sysakmon.conf path. After the configuration file is modified, you must run the systemctl restart sysak command to restart the Mservice service.

Data viewing

Diagnostics feature of SysAK

User scenario-based diagnostics

Item-specific diagnostics under the hood

• Scheduling

  Command type	Feature	Command and description
  nosched	Diagnoses the issue that tasks cannot be scheduled in a timely manner to run on a CPU if the CPU has run in kernel mode for an extended period of time.	sysak nosched [--help] [-t THRESH(ms)] [-f LOGFILE] [-s duration(s)] -t THRESH: (optional) sets a threshold for the period of time during which the kernel does not schedule tasks. When the threshold is breached, related information is recorded. Unit: milliseconds. Default value: 10. -f LOGFILE: (optional) specifies a log file. Default value: /var/log/sysak/nosched/nosched.log. -s durations: (optional) specifies how long the program can run. Unit: seconds. If this option is not specified, the program keeps running.
  irqoff	Diagnoses the issue that interrupts are disabled for an extended period of time in the system.	sysak irqoff [--help] [-t THRESH(ms)] [-f LOGFILE] [duration(s)] -t THRESH: (optional) sets a threshold for the period of time during which interrupts are disabled. When the threshold is breached, related information is recorded. Unit: milliseconds. Default value: 10. -f LOGFILE: (optional) specifies a log file. Default value: /var/log/sysak/irqoff/irqoff.log. durations: (optional) specifies how long the program can run. Unit: seconds. If this option is not specified, the program keeps running.
  runqslower	Diagnoses the issue that the task scheduling latency is high in the system.	sysak runqslower [-s SPAN] [-t TID] [-f LOGFILE] [-P] [THRESH] -s SPAN: (optional) specifies how long the program can run. Unit: seconds. If this option is not specified, the program keeps running. THRESH: (optional) sets a threshold for the period of time during which the task is preempted. When the threshold is breached, related information is recorded. Unit: milliseconds. Default value: 20. -f LOGFILE: (optional) specifies a log file. Default value: /var/log/sysak/runqslow/runqslow.log. -t TID: (optional) specifies the ID of a monitored thread. It is a filter option. By default, all threads are monitored. -P: (optional) records the name and thread ID (TID) of the task that was previously preempted. By default, such information is not recorded.

• Memory

  Command type	Feature	Command and description
  memleak	Checks for kernel memory leaks (including slab memory leaks, vmalloc memory leaks, and buddy memory leaks) and identifies where leaks occur.	sysak memleak [-t type] [-i internal] [-c] -t: specifies the type of memory leak. The following types of memory leaks are available: slab vmalloc page -i: specifies the diagnostic period. Default value: 300. Unit: seconds. -c: specifies whether to perform a quick diagnostics. If this option is specified, a quick diagnostics is performed to determine whether the memory is leaked, but the exact locations where memory leaks occur are not identified.
  mmaptrace	Identifies where memory leaks occur and provides call stacks for requesting memory in user mode.	sysak mmaptrace [ option ] [ args ] -p <pid>: specifies the ID of a process (PID) to monitor how the process requests memory. -l: monitors the memory size requested by malloc and mmap. -s: shows the call stack for requesting memory in user mode.
  memgraph	Analyzes and shows memory usage by means of graphs.	sysak memgraph [ option ] -g: shows the memory usage chart. -f: shows the detailed information of page caches. -a: shows the detailed information of anonymous memory. -k: checks whether the memory is leaked. -l: shows the memory usage of system threads. -c: shows the memory usage of system cgroups.

• I/O

  Command type	Feature	Command and description
  iofsstat	Collects the disk I/O information with process and file granularity.	sysak iofsstat [-h] [-T TIMEOUT] [-t TOP] [-u UTIL_THRESH] [-b BW_THRESH] [-i IOPS_THRESH] [-c CYCLE] [-d DEVICE] [-p PID] [-j] [-f] -T TIMEOUT: specifies how long the command can run. Unit: seconds. -t TOP: specifies how many disks with the maximum I/O resource usage are displayed. -u UTIL_THRESH: sets a threshold for I/O utilization. Disks whose I/O utilization is lower than the threshold are ignored. -b BW_THRESH: sets a threshold for bandwidth. Disks whose bandwidth is lower than the threshold are ignored. -i IOPS_THRESH: sets a threshold for IOPS. Disks whose IOPS is lower than the threshold are ignored. -c CYCLE: specifies a refresh interval. Unit: seconds. -d DEVICE: specifies the name of the disk to be monitored. -p PID: specifies the ID of a process to be monitored. -j,--json: shows the command output in the JSON format. -f,--fs: monitors and reports the information of a specified partition.

• Network

  Command type	Feature	Command and description
  pingtrace	Detects network latency.	sysak pingtrace [ options ] -v,--version: shows the version number. -h,--help: returns help information. -s,--server: runs in server mode. -c,--client ip: runs in client mode. -C:--count UINT: shows the number of probe packets. By default, the number of packets is unlimited. -i <interval_us>: sets an interval at which packets are sent. Unit: microseconds. -t < UINT >: specifies how long the program can run. Unit: seconds. -m,--maxdelay us: sets a threshold for the ping latency. Only packets whose latency is higher than the threshold can be recorded. Default value: 0. -b <INT=556 >: specifies the size of the probe packet. The value must be greater than 144. Unit: bytes. --log TEXT=./pingtrace.log: specifies the name of the log file. --logsize INT: specifies the maximum amount of space that can be used by the log file. --logbackup INT=3: specifies the maximum number of backups that can be created for the log file. --mode auto/pingpong/compact: specifies the running mode of PingTrace. -o,--output image/json/log/imagelog: specifies the output format of PingTrace data. -n,--namespace: checks the net namespace information. --nslocal: tells the PingTrace client and server to run on the same host when net namespace information is checked to prevent obtaining redundant data. --userid UINT: specifies different user IDs for different hosts when the net namespace information is checked. This helps PingTrace identify and resolve time desynchronization issues on different hosts. --debug: shows relevant debugging information such as the libbpf information.
  skcheck	Checks for TCP and socket leaks.	sysak skcheck [ options ] [ cmdargs ] -s: enables leak detection. -i: sets a threshold to enable sockets. Default value: 2000. -l: sets a threshold to disable sockets. Default value: 500.

• Performance

  Command type	Feature	Command and description
  numa_access	Shows the information of a process with the specified PID and the non-uniform memory access (NUMA) information of a CPU.	sysak numa_access [ options ] [ cmdargs ] -p <pid>: specifies a PID. -c <cpu>: specifies a CPU. -i <time>: sets an interval for showing information.
  hw_event	Shows the information of Docker hardware events.	sysak hw_event [ options ] [ cmdargs ] -c <name>: specifies the name of a docker. If this option is not specified, the hardware events of all dockers are displayed by default. -s <time>: sets a running period. Default value: 5. Unit: seconds.

• Virtualization

  Command type	Feature	Command and description
  kvmexittime	Traces and diagnoses VM-exit events.	sysak kvmexittime [--help] [-p PID] [-t TID] [interval] -p <PID>: specifies a PID. -t <TID>: specifies a TID. interval: sets an interval for tracing and analyzing VM-exit events. --help: returns help information.

• Generic

  Command type	Feature	Command and description
  syscall_slow	Automatically analyzes the contentions for locks among application threads when the system responds slowly to calls.	sysak syscall_slow [-t THRESH(ms)] [-n sys_NR] <[-c COMM] [-p tid]> [-f LOGFILE][duration(s)] -t: (optional) sets a threshold for the system response time. If the threshold is breached, related information is recorded. Default value: 10. Unit: milliseconds. -n: (optional) checks the specified call to the system. By default, all calls to the system are checked. -c/-p: specifies a task name or a PID. You cannot specify both a task name and a PID. This option is required. -f: (optional) specifies a log file. Default value: /var/log/sysak/syscall_slow/syscall_slow.log. durations: (optional) specifies how long the program can run. Unit: seconds. If this option is not specified, the program keeps running.
  ulockcheck	Automatically analyzes the contentions for locks among application threads.	sysak ulockcheck -p <pid> | -s <thread pid>| -a | -t <0|1> | -d -p: monitors lock contention among threads of a specified process. -a: shows the current lock owner and the top five lock requesters. -s: shows the lock contention status of a monitored thread. -t: enables the output. If a thread waits for a lock for more than 100 milliseconds, the call stack of the thread in user mode is displayed. -d: disables monitoring.
  cpuirq	Shows the interrupt binding and running status of a CPU.	sysak cpuirq [-c cpu -b ] [ -t [ -i interval ] ] -c: specifies a CPU. -b: shows the interrupt binding information of the specified CPU. -t: shows the request with the most interrupts in a specific period of time. -i: sets an interval for collecting data.
  softirq	Records the running status (such as number or rate) of soft interrupts in the system.	sysak softirq [ option ] [ args ] -s: specifies the source file that contains initial data. -r: specifies a file to store the output.