Use SSL to encrypt Kafka data - E-MapReduce - Alibaba Cloud Documentation Center

Secure Sockets Layer (SSL) is a network protocol that provides secure connection between clients and servers. After you enable SSL for a Dataflow cluster, data transmission among producers, consumers, and brokers is encrypted. This protects all sensitive data against eavesdropping or tampering during transmission. This topic describes how to enable SSL for a Dataflow cluster on which Kafka is deployed and access Kafka over SSL.

Prerequisites

A Dataflow cluster is created in the E-MapReduce (EMR) console, and Kafka is selected when you create the cluster. For more information, see Create a Dataflow Kafka cluster.

Enable SSL

You can use one of the following methods to enable SSL for a Kafka cluster:

Use the default certificate provided by EMR to enable SSL.
Use a custom certificate and related configuration items to enable SSL.

EMR manages the policies that are used to configure SSL by using the kafka.ssl.config.type configuration item in the server.properties configuration file.

Use the default certificate to enable SSL

Note

By default, SSL is disabled for Kafka clusters. You can perform the following operations to quickly enable SSL.

Go to the Configure tab of the Kafka service page.
1. Log on to the EMR console. In the left-side navigation pane, click EMR on ECS.
2. In the top navigation bar, select the region in which your cluster resides and select a resource group based on your business requirements.
3. On the EMR on ECS page, find the desired cluster and click Services in the Actions column.
4. On the Services tab, find the Kafka service and click Configure.
Modify a configuration item.
1. On the Configure tab, click the server.properties tab.
2. Change the value of the kafka.ssl.config.type configuration item to DEFAULT.
Save the configuration.
1. Click Save.
2. In the dialog box that appears, configure the Execution Reason parameter and click Save.
Restart the Kafka service.
1. On the Configure tab of the Kafka service page, choose More > Restart in the upper-right corner.
2. In the dialog box that appears, configure the Execution Reason parameter and click OK.
3. In the Confirm message, click OK.

Use a custom certificate to enable SSL

Note

By default, SSL is disabled for Kafka clusters. You can customize configurations to enable SSL.

Go to the Configure tab of the Kafka service page.
1. Log on to the EMR console. In the left-side navigation pane, click EMR on ECS.
2. In the top navigation bar, select the region in which your cluster resides and select a resource group based on your business requirements.
3. On the EMR on ECS page, find the desired cluster and click Services in the Actions column.
4. On the Services tab, find the Kafka service and click Configure.
Modify a configuration item.
1. On the Configure tab of the Kafka service page, click the server.properties tab.
2. Change the value of the kafka.ssl.config.type configuration item to CUSTOM.
Save the configuration.
1. Click Save.
2. In the dialog box that appears, configure the Execution Reason parameter and click Save.
Modify other configuration items that are related to SSL.
You must configure SSL-related configuration items except listeners based on your business requirements. For example, you must configure the following configuration items: ssl.keystore.location, ssl.keystore.password, ssl.truststore.location, ssl.truststore.password, ssl.key.password, ssl.keystore.type, and ssl.truststore.type.
Restart the Kafka service.
1. On the Configure tab of the Kafka service page, choose More > Restart in the upper-right corner.
2. In the dialog box that appears, configure the Execution Reason parameter and click OK.
3. In the Confirm message, click OK.

Access Kafka over SSL

To access Kafka over SSL, you must configure the security.protocol, ssl.truststore.password, and ssl.truststore.location configuration items.

For example, if you use the Producer and Consumer programs of Kafka to run jobs in the Kafka cluster for which SSL is enabled, you can perform the following steps:

Log on to the master node of your cluster in SSH mode. For more information, see Log on to a cluster.
Create a configuration file.
1. Run the following command to create the ssl.properties configuration file:
```
vim ssl.properties
```
2. Add the following information to the ssl.properties configuration file.
```
security.protocol=SSL
ssl.truststore.location=/var/taihao-security/ssl/ssl/truststore
ssl.truststore.password=${password}
ssl.keystore.location=/var/taihao-security/ssl/ssl/keystore
ssl.keystore.password=${password}
ssl.endpoint.identification.algorithm=
```
  On the Configure tab of the Kafka service page in the EMR console, you can view the values of the preceding configuration items. If you run jobs outside your Kafka cluster, copy the truststore and keystore files in the specific directory for a node of the Kafka cluster to the runtime environment. Then, configure the required information.

Run the following command to create a Kafka topic:

kafka-topics.sh --partitions 10 --replication-factor 2 --bootstrap-server core-1-1:9092 --topic test --create --command-config ssl.properties

Run the following command to use the SSL configuration file to generate data:

export IP=<your_InnerIP>
kafka-producer-perf-test.sh --topic test --num-records 123456 --throughput 10000 --record-size 1024 --producer-props bootstrap.servers=${IP}:9092 --producer.config ssl.properties

Note

In the preceding command, your_InnerIP indicates the internal IP address of the master-1-1 node of your cluster.

Run the following command to use the SSL configuration file to consume data:

export IP=<your_InnerIP>
kafka-consumer-perf-test.sh --broker-list ${IP}:9092 --messages 100000000 --topic test --consumer.config ssl.properties

References

For information about how to verify the user identity that is used to log on to a Dataflow Kafka cluster, see Log on to a Kafka cluster by using SASL.