Best practices for handling mining programs - Security Center

If mining viruses intrude into a server, the mining viruses consume server resources, reduce production efficiency, affect system stability, and pose a serious threat to the compromised server. You must handle mining viruses at the earliest opportunity to reinforce system security. This topic describes how to handle mining viruses.

Characteristics of mining programs

Mining programs can overclock the CPU, which consumes a large number of CPU resources and affects other applications that run on your server.
The characteristics of mining programs are similar to the characteristics of computer worms. After a mining program intrudes into your server, the mining program spreads to the servers that are deployed in the same internal network. After the servers are compromised, the mining program achieves persistence on the servers.
Mining programs spread to multiple system services and are difficult to remove from the system. Mining programs may repeatedly appear, and system commands may be replaced with malicious scripts. As a result, the system may run malicious scripts such as XOR DDoS. You must remove all trojans and persistent webshells from your server within the execution period of mining programs. This way, mining programs are prevented from appearing in the future.

Determine whether your assets contain mining programs

After you receive a mining alert or handle mining viruses, you can use the following methods to determine whether mining viruses exist on your server:

If the CPU utilization of your server significantly increases at regular intervals or for a long period of time, such as to 80% or higher, and an unknown process continues to send packets, a mining program is running on your server.
If the Security Center agent is installed on your server and mining programs intrude into your server, Security Center sends you alert notifications by text message or email. You can handle mining program alerts on the Alerts page in the Security Center console. If mining programs are associated with other alerts, such as alerts on communication with mining pools and alerts on access to malicious domain names, we recommend that you also handle the associated alerts. For more information, see View and handle alerts.

Solution 1: Use Security Center to handle mining programs

We recommend that you handle mining programs in sequence. These operations can stop the spread of mining programs, detect and remove mining programs, and scan all risks. You can also adjust the operation sequence or select specific steps based on your business requirements.

Prerequisites

Before you perform Step 1: Handle alerts to terminate malicious processes or Step 2: Perform in-depth virus detection on the server and remove the detected viruses, you must purchase the Anti-virus, Advanced, Enterprise, or Ultimate edition of Security Center. You can also use the Basic edition of Security Center to obtain its features on free trial. For more information, see Purchase Security Center and Apply for a 7-day free trial of Security Center.
Before you perform Step 3: Scan all disks, you must purchase the agentless detection feature by using the pay-as-you-go billing method. For more information, see Use the agentless detection feature.
The Security Center agent on the compromised server is online. For more information, see Install the Security Center agent and Troubleshoot why the Security Center agent is offline.

Step 1: Handle alerts to terminate malicious processes

You can use the alert feature of Security Center to terminate malicious processes, quarantine virus files, and block mining programs. For more information, see View and handle alerts.

Log on to the Security Center console. In the top navigation bar, select the region of the asset that you want to manage. You can select China or Outside China.
In the left-side navigation pane, choose Detection and Response > Alerts.
Find the mining program alert that you want to handle and click Details in the Actions column to view the details of the alert.
Important
You can identify the mining program and check whether other alerts or suspicious files exist on the server based on the alert information and event description provided in the Security Center console. You must determine whether the file that triggers the mining alert is a business file or an attack file. If the file is an attack file, we recommend that you handle other alerts and suspicious files on the server after you handle the mining program alert.
Return to the alert list and handle the mining program alert.
1. Click Handle in the Actions column.
2. In the dialog box that appears, select Detect and Remove Virus as the handling method, select Terminate Process or Terminate Process and Quarantine Source File, and then click Handle Now to prevent the mining program from restarting.
  If the file that triggers the mining program alert is not required in your workloads, we recommend that you select Terminate Process and Quarantine Source File to prevent further infection on your server.
  Security Center allows you to handle multiple alerts at a time. If you want to handle the alerts that are triggered by the same rule or rules of the same type at a time, select Handle Same Type of Alerts.
In the alert list, find an alert that is related to mining activities, such as an alert generated for communication by using mining pools, and click Handle in the Actions column. In the dialog box that appears, select Block as the handling method.
Security Center generates policies to prevent servers from communicating with the IP addresses of mining pools. This way, you have sufficient time to handle security events. You can add the IP addresses of mining pools to a security group to block the IP addresses. For more information about how to enable ports, see Add a security group rule.
In the alert list, view the alerts that are generated for suspicious processes, check whether abnormal scheduled tasks exist, and then handle the alerts.

Step 2: Perform in-depth virus detection on the server and remove the detected viruses

After you terminate malicious processes, we recommend that you use the virus detection and removal feature of Security Center to scan your assets. Detected viruses and corresponding alerts are displayed on the Alerts page. This feature can detect and remove persistent malicious files such as self-startup items and scheduled tasks. For more information, see Use the virus detection and removal feature.

Log on to the Security Center console. In the top navigation bar, select the region of the asset that you want to manage. You can select China or Outside China.
In the left-side navigation pane, choose Protection Configuration > Host Protection > Virus Detection and Removal.
On the Virus Detection and Removal page, click Immediate Scan or Scan Again.
In the Scan Settings panel, configure the Scan Mode and Scan Scope parameters and click OK.
You can set the Scan Mode parameter to Quick Scan and the Scan Scope parameter to the servers that are attacked by mining programs.
Wait until the scan is complete. Then, find the alert that you want to handle on the Virus Detection and Removal page and click Handle in the Actions column.
In the Alert Handling panel, select In-depth Cleanup and click Next.
The system starts to handle alerts. After the process is complete, you can view the handling results and the status of the alerts.

Step 3: Scan all disks

You can use the agentless detection feature provided by Security Center to check system disks and data disks of Elastic Compute Service (ECS) instances. This feature can detect mining programs but cannot fix them. You must manually handle the risks based on the risk description provided by Security Center. For more information, see Use the agentless detection feature.

Log on to the Security Center console. In the top navigation bar, select the region of the asset that you want to manage. You can select China or Outside China.
In the left-side navigation pane, choose Protection Configuration > Host Protection > Agentless Detection.
On the Agentless Detection page, click the Server Check tab and then click Create Detection Task.
In the Create Detection Task panel, select the ECS instance that you want to check and click Next.
Configure the Scan Scope and Snapshot/Image Storage Time parameters, and click Next.
After the detection task is complete, you can view the detection results of vulnerabilities, baseline risks, alerts, and sensitive files.
We recommend that you handle the risks based on the risk description provided by Security Center.

Solution 2: Manually handle mining programs

Mining programs can insert a large number of persistent webshells into a compromised server to obtain the most profits. In this case, viruses are difficult to remove or cannot be removed. If you did not purchase Security Center, you can perform the following operations to detect and handle mining programs.

Linux servers

Block malicious network communication.
After Security Center detects mining activities on your server, you may require a long period of time to thoroughly handle an entire mining event. To minimize the impact at the earliest opportunity, you must first block the network communication of mining trojans and malicious programs.
1. Run the following command to check the network connection:
```
netstat -antp
```
2. Replace c2 address in the following command with a foreign address that is not required in your workloads. Then, run the command to create a firewall rule to block all network connections between the server and the foreign address.
```
iptables -A INPUT -s c2 address -j DROP
iptables -A OUTPUT -d c2 address -j DROP
```
Clear scheduled tasks.
In most cases, mining trojans are downloaded or updated at regular intervals and started by using scheduled tasks. If you only remove mining processes and mining trojan files, mining programs cannot be completely removed and mining events may repeatedly occur.
You can run the following commands to query scheduled tasks. In the command output, if you identify download tasks and startup tasks of mining trojans in the scheduled tasks, delete the tasks.
- Run the following commands to query scheduled tasks for the current user and a specific user. Replace username with the actual username.
```
crontab -l
crontab -u username -l
```
- Check the following files used for scheduled tasks on the server:
```
/etc/crontab
/var/spool/cron/
/etc/anacrontab
/etc/cron.d/
/etc/cron.hourly/
/etc/cron.daily/
/etc/cron.weekly/
/etc/cron.monthly/
```
Delete self-startup items.
1. Run the following command to query all self-startup items in the system:
```
systemctl list-unit-files | grep enabled
```
2. Run the following commands to locate the service unit files of suspicious services and view the details of the services.
  Replace the asterisk (*) with the name of the service and <service_unit_name> with the name of the service unit file.
```
ls -al /etc/systemd/system/*.service
ls -al /usr/lib/systemd/system/*.service

# View the details of the service by querying the process file to start the service.
cat /etc/systemd/system/<service_unit_name>.service
```
3. If a malicious self-startup item is detected, run the following commands to disable the service and delete the service unit file.
  Replace <service name> with the name of the service and <service_unit_name> with the name of the service unit file.
```
systemctl disable <service name>
rm /etc/systemd/system/<service_unit_name>.service
rm /usr/lib/systemd/system/<service_unit_name>.service
```
4. Check the following service directories for self-startup items. If self-startup items are detected, follow the preceding operations to delete the items.
```
/etc/rc.local
/etc/inittab
/etc/rc.d/
/etc/init.d/
```
Delete SSH public keys.
In most cases, an attacker uses mining trojans to write the SSH public key to the ~/.ssh/authorized_keys file. Then, the attacker can log on to the compromised server over SSH without the need to enter the password and re-insert malicious files. In this case, you must check the ~/.ssh/authorized_keys file at the earliest opportunity. If a suspicious public key exists, delete the key.
Remove the .so file that is used for hijacking.
Attackers can specify a .so file in the /etc/ld.so.preload file to preload the .so file for hijacking of common system commands. This way, the attacker can hide mining programs.
1. Run the following command to check for preloaded .so files:
```
cat /etc/ld.so.preload
```
2. Run the following command to clear preloaded .so files that are used for hijacking:
```
echo > /etc/ld.so.preload
```
Delete malicious accounts.
Some mining trojan families create webshell accounts for long-term control of compromised servers. Run the following commands to check for malicious accounts. If malicious accounts are detected, delete the accounts.
- Run the following commands to check for account creation operations in logs:
```
cat /var/log/audit/audit.log | grep useradd
# or
cat /var/log/secure | grep 'new user'
```
- Run the following command to query all accounts from the /etc/passwd file and check for suspicious accounts:
```
cut -d: -f1 /etc/passwd
```
- Run the following command to check home directories, especially newly created ones, to determine the creation time or the most recent active time of a suspicious account:
```
stat /home/guest/
```
Take anti-tampering measures.
Some mining trojans write files such as files used for scheduled tasks and self-startup items to achieve persistence and configure an immutable attribute on the files to prevent the deletion of persistent tasks.
If you encounter an issue similar to the preceding figure, you can run the following commands to remove the immutable attribute. Then, delete the persistent tasks.
```
chattr -i /etc/passwd
chattr -i /etc/crontab
```
Clear mining processes.
1. Mining processes can consume a large amount of CPU resources. You can run the following commands to check for suspicious mining processes.
  - Run the following commands to check for processes that consume a large amount of CPU resources on the server:
```
top -c
```
```
ps -eo pid,ppid,cmd,%cpu --sort=-%cpu | more
```
  - Run the following command to check for suspicious network connections:
```
netstat -antp
```
  - Run the following command to query the file paths to suspicious processes:
```
ls -al /proc/$PID/exe
```
  - Run the following command to calculate the MD5 hash value of the process file. Then, search for the file information in Alibaba Cloud Threat Intelligence.
```
md5sum /proc/$PID/exe
```
2. Run the following command to terminate mining processes and delete the executable files of the mining processes:
```
kill -9 $PID
# Delete process paths that are obtained by running the ls -al /proc/$PID/exe command
rm /path/to/executable
```

Linux servers (Handling of specific mining events)

Handle a mining process that is disguised as the AliyunDuns process of Alibaba Cloud

Run the following commands to check for persistent self-startup items:

grep -rlE "\\-\\-donate\\-level|xmrig|\\/opt\\/sysetmd|A_li_yun_Duns" /etc/systemd/system/*
grep -rl "wget"  /etc/cron.hourly/*

If a result is returned, run the following commands to delete self-startup items:

rm -f /etc/systemd/system/sysetmd.service
rm -f /etc/systemd/system/monero.service
rm -f /etc/systemd/system/A_li_yun_Duns.service
chattr -i /etc/cron.hourly/0
rm -f /etc/cron.hourly/0

Identify and handle abnormal system users.
1. Run the following command to query system users and identify the abnormal username:
```
cat /etc/passwd
```
2. Run the following commands to view the information about the abnormal system user. In this example, the abnormal system user is named shaojiang99.
```
cat /etc/passwd | grep shaojiang99
cat /etc/shadow | grep shaojiang99
```
3. Run the following commands to delete the abnormal system user:
```
chattr -i /etc/passwd
chattr -i /etc/shadow
sed -i '/^shaojiang99:/d' /etc/shadow
sed -i '/^shaojiang99:/d' /etc/passwd
```

Handle Skidmap

Run the following commands to delete self-startup items:

echo "" > /lib/systemd/system/systemd-cgroup.service
echo "" > /lib/systemd/system/systemd-deltaed.service
echo "" > /usr/bin/systemd-cgroup.org
echo "" > /usr/bin/systemd-cgroup
echo "" > /usr/bin/systemd-deltaed
echo "" > /usr/bin/systemd-deltaed.org

Manually restore the authorized_keys file in which the SSH public key is stored.

Handle cleanfda

Open the following paths to scheduled tasks and check for executions of suspicious scripts such as /etc/upat.sh.
```
/var/spool/cron/
/etc/cron.d/
/var/spool/cron/crontabs
/etc/crontab
```
Check for modified commands such as ps, pstree, and top. These commands are renamed by adding the .original suffix.
```
ls -al /usr/bin | grep original
```
Run the following commands to query SSH public keys and delete suspicious keys:
```
cat ~/.ssh/authorized_keys
cat /root/.ssh/authorized_keys
```

Examples:

# Restore the command that is modified
crondir='/var/spool/cron/'"$USER"
mv /bin/ps.original /bin/ps
mv /bin/top.original /bin/top
mv /bin/pstree.original /bin/pstree

# Restore editing permissions on scheduled task files
chattr -R -ia /var/spool/cron
chattr -ia /etc/crontab
chattr -R -ia /var/spool/cron/crontabs
chattr -R -ia /etc/cron.d

# Delete suspicious scheduled tasks and payloads
sed -i '/upat.sh/d' /etc/crontab
rm -rf /etc/cron.d/httpd2
rm -rf /etc/upat.sh
rm -rf /tmp/upat.sh
rm -rf /etc/httpd2
rm -rf /tmp/httpd2

# Delete suspicious public keys
sed -i '/cKtXBjj******hVI0K7b/d' ~/.ssh/authorized_keys
sed -i '/cKtXBjj******hVI0K7b/d' /root/.ssh/authorized_keys

Handle Outlaw mining activities

Check for cron tasks and then check whether the .configrc5/ string exists.
- Check for the following paths to scheduled tasks:
```
/var/spool/cron/root
/var/spool/cron/
/etc/cron.d/
/var/spool/cron/crontabs
/etc/crontab
```
- Run the following commands to check whether the .configrc5/ string exists:
```
grep -r ".configrc5/" /var/spool/cron/*
grep -r ".configrc5/" /etc/cron*
```
If the string exists, run the following commands to delete the file that contains the string:
```
rm -rf /tmp/.X2xi-unix/.rsync
rm -rf ～/.configrc5
```

Handle kinsing mining activities

Run the following commands to check for cron tasks. Then, check whether content similar to gi.sh | bash > /dev/null exists in a line. If such content exists, delete the line.
```
grep -r "gi.sh | bash > /dev/null" /etc/cron*
grep -r "gi.sh | bash > /dev/null" /var/spool/cron/*
```
Run the following command to check whether the /lib/systemd/system/bot.service service exists:
```
cat /lib/systemd/system/bot.service 
```
If the service exists, run the following command to query the /etc/ld.so.preload file. Then, check whether libsystem.so is included in the file.
```
cat /etc/ld.so.preload
```
If the path to the libsystem.so file is found, delete this file.

Windows servers

Run the following commands in PowerShell to identify mining programs among the processes that cause high CPU utilization:
```
 ps | sort -des cpu
 While(1) {ps | sort -des cpu | select -f 15 | ft -a; sleep 1; cls}
```
Run the following command to query the executable file of the mining program and the parameters in the command that is used to start the mining program:
```
wmic process where processid=xxx get processid,executablepath,commandline,name     ///xxx indicates the process ID (PID).
```
Terminate the mining process and delete the executable file of the mining process.

Run the following command to detect suspicious ports of your server:

netstat -ano | findstr xxx            // xxx indicates the suspicious port.

Run the following command to check whether the hosts file in the server contains the address of the mining pool to which the mining program belongs:
```
type  C:\Windows\System32\drivers\etc\hosts
```
Run the following command to check whether the scheduled tasks specified by the mining program exist on your server:
```
schtasks /query
```

Other methods

If the underlying system components of your server are affected by viruses, you may fail to troubleshoot the issues or remove the viruses. After you back up important data, we recommend that you reset the operating system of the server to ensure that the mining program is completely removed. To use this method, perform the following operations:

Create a snapshot to back up data on your server. For more information, see Create a snapshot.
Initialize the operating system of the server. For more information, see Re-initialize a system disk.
Create a disk based on the snapshot. For more information, see Create a disk from a snapshot.
Attach the disk to the server after the operating system is reinstalled. For more information, see Attach a data disk.

Solution 3: Purchase the emergency response service

Alibaba Cloud provides the emergency response service that is delivered by security experts to help you resolve issues such as virus infection. The following list describes the service content:

Removes trojans, viruses, suspicious accounts, suspicious files, webshells, and hidden links from the system.
Analyzes intrusion behavior and identify the causes of intrusions.
Provides guidance on security hardening.

For more information, see Incident Response Service.

Perform security hardening

Enable the malicious host behavior defense feature

To prevent the recurrence of mining events caused by residual mining programs, Security Center provides the malicious host behavior defense feature to accurately intercept mining programs. For more information, see Use proactive defense.

To enable the feature, turn on Malicious Host Behavior Prevention on the Feature Settings page in the Security Center console, as shown in the following figure.

The following figure shows an alert details page about a mining program that is automatically intercepted by Security Center.

Improve the security of the host

Reinstall the operating system: After a mining program attacks your server, the system files are often modified or replaced due to residual security threats in the system. In this case, the system becomes untrustworthy, and we recommend that you reinstall the operating system.
Reduce attack exposure: You can use ECS security groups, Server Load Balancer (SLB) whitelists, and Cloud Firewall to restrict unnecessary service ports from being exposed to the Internet. Only necessary service ports are open for access by trusted IP addresses.
Perform security hardening on the host system: You can configure complex passwords for database systems such as MySQL, PolarDB, MaxCompute, and Redis, file systems such as network-attached storage (NAS) and Object Storage Service (OSS), background service management systems such as BT-Panel and Nacos, and operating system user passwords such as SSH passwords and remote desktop (RDP) passwords. You can also limit the number of retries for incorrect passwords to prevent password leaks.
Fix common vulnerabilities: If you use earlier application versions and a large number of application vulnerabilities are not fixed, mining programs may intrude into your server. We recommend that you pay attention to security vulnerability intelligence and fix application vulnerabilities, emergency vulnerabilities, and system vulnerabilities at the earliest opportunity.
Enhance business security: Before you publish business code, you can perform code security tests or use Web Application Firewall (WAF) to defend against common attacks that are listed by the Open Worldwide Application Security Project (OWASP), such as SQL injections, cross-site scripting (XSS), common web server plug-in vulnerabilities, Trojan uploads, and unauthorized access to core resources. This prevents business system bugs from being exploited.

Use secure credentials

If the credentials of Alibaba Cloud accounts and Resource Access Management (RAM) users are leaked, it may cause security risks to cloud resources and business. We recommend that you use your credentials in a secure manner. For more information, see Credential security solutions.

Security Center:Best practices for handling mining programs