Fix the RCE vulnerability in Apache Log4j 2 - E-MapReduce

Alibaba Cloud Computing Co., Ltd. recently discovered a remote code execution (RCE) vulnerability in Apache Log4j 2 and reported this vulnerability to Apache. This topic describes the impacts of the vulnerability and provides a solution to the vulnerability.

Impacts

The affected services in E-MapReduce (EMR) clusters include Hive, Presto, Impala, Druid, Flink, Solr, Ranger, Storm, Oozie, Spark, and Zeppelin. Spark and Zeppelin are affected because they have a dependency on Hive.

Solution

Replace the Log4j 2 JAR file of your EMR cluster with the JAR file whose Log4j 2 version is 2.17.0. In addition, set the log4j2.formatMsgNoLookups parameter for the Hive and Spark services to true to disable the JNDI lookup feature. For more information about the procedure, see Procedure.

Take note of the following points about this solution:

This solution applies to EMR V3.38.2 and earlier minor versions, EMR V5.4.2 and earlier minor versions, and EMR V4.X. In minor versions later than EMR V3.38.2 and minor versions later than V5.4.2, the vulnerability is fixed, and no repair operation is required.
After you implement this solution, you must restart the affected services in your cluster.
The script that you use to fix the vulnerability does not have an impact on online services. However, we still recommend that you run the script during off-peak hours because you must restart the related services after you run the script.

Procedure

Fix the vulnerability for an EMR cluster

Download the patch package patches-log4j.tar.gz.
Log on to the master node of your EMR cluster and place the package downloaded in Step 1 in the home directory of the emr-user or hadoop user.
Run the following commands to switch to the emr-user or hadoop user and decompress the package:
- DataLake clusters, Dataflow clusters, OLAP clusters, DataServing clusters, or custom clusters
```
su emr-user
tar zxf patches-log4j.tar.gz
```
- Other clusters
```
su hadoop
tar zxf patches-log4j.tar.gz
```
Run the following commands to open the hosts file in the patches directory. Then, add the hostnames of all the nodes in the cluster, such as emr-header-1 and emr-worker-1, to the hosts file. Enter one hostname in each line.
```
cd patches
vim hosts
```
Sample file content:
```
emr-header-1
emr-worker-1
emr-worker-2
```
Important
For clusters of EMR V3.41 or a later minor version, or clusters of EMR V5.7.0 or a later minor version, the hostnames of nodes are in a different format. Sample file content:
```
core-1-1
core-1-2
task-1-1
task-1-2
```
Run the fix.sh script to fix the vulnerability:
```
./fix.sh
```
After the script is run, the following information is returned:
```
### NOTICE: YOU CAN RESTORE THIS PATCH BY RUN RESTORE SCRIPT ABOVE
$> sh ./restore.sh 20211213001755
### DONE
```
If you want to perform a rollback, run the following command:
```
./restore.sh 20211213001755
```
Note
For jobs that are running in YARN, such as Spark Streaming or Flink jobs, you must perform a rolling restart on YARN NodeManager after the jobs are stopped.
Restart services.
Restart the affected services, including Hive, HDFS, Presto, Impala, Druid, Flink, Solr, Ranger, Storm, Oozie, Spark, and Zeppelin.
For example, choose More > Restart in the upper-right corner of the Hive service page to restart the Hive service.

Fix the vulnerability for a gateway cluster

Gateway clusters do not support password-free logon in SSH mode. Therefore, if you use a gateway cluster, you must manually upload the patch package to each node of the gateway cluster and perform the preceding fix operations on each node.

Important

You need to enter only the hostname- of the current execution node to the hosts file.
A gateway cluster does not contain services. You do not need to restart a service after you upload the patch package.

Fix the vulnerability when you create a cluster or scale out an existing cluster

When you create an EMR cluster, you can add a bootstrap action in the EMR console to fix the vulnerability. When you scale out an existing cluster, the system automatically fixes the vulnerability. When you create an EMR cluster, perform the following steps:

Download the patches-log4j.tar.gz package and the bootstrap_log4j.sh script file and upload them to an Object Storage Service (OSS) path.
In this example, the package and script file are uploaded to oss://<bucket-name>/path/to/.

Add a bootstrap action in the EMR console. For more information, see Manage bootstrap actions.

In the Add Bootstrap Actions dialog box, configure the parameters.

Parameter	Description
Name	The name of the bootstrap action that you want to add. For example, you can set this parameter to fixlog4jvulnerability.
Script Address	The OSS path where the script file is located. You must specify this parameter in the oss://*/.sh format. In this example, the path is oss://<bucket-name>/path/to/bootstrap_log4j.sh.
Parameter	The parameter of the bootstrap action script. The parameter is used to specify the value of the variable that is referenced in the script. In this example, the parameter is oss://<bucket-name>/path/to/patches-log4j.tar.gz.
Execution Scope	Select Cluster.
Execution Time	Select After Component Startup.
Execution Failure Policy	Select Proceed.

After you create the cluster, restart the HDFS, Hive, Presto, Impala, Druid, Flink, Solr, Ranger, Storm, Oozie, Spark, and Zeppelin services. If you scale out a cluster, you need to only restart the related services that are deployed on the newly added nodes.