In big data processing contexts, clients outside an E-MapReduce (EMR) cluster, such as those operating on Elastic Compute Service (ECS) or Kubernetes pods, cannot submit jobs due to domain name resolution failure. You can enable the Domain Name System (DNS) hostname to connect external clients to nodes in the EMR clusters. This streamlines the job submission process.
Introduction
The DNS is a standardized protocol used on the Internet to resolve domain names into IP addresses. As a unique name that distinguishes a computer in the network, a DNS hostname comprises a hostname and a domain name. DNS servers resolve DNS hostnames to the corresponding IP addresses and allow users to access network resources by using domain names.
How it works
In a virtual private cloud (VPC), DNS hostnames are used to identify ECS instances. By enabling this feature, you can configure the built-in authoritative domain names for ECS instances in a VPC. The format of the domain name is [regionID].ecs.internal.
After the DNS hostname feature is enabled, the VPC will associate with the default DHCP options set. If the VPC does not have a default DHCP options set, the system will automatically create and bind one.
For example, the DNS hostname feature is enabled for a VPC located in the Malaysia (Kuala Lumpur) region. The built-in authoritative domain name of an ECS instance in the VPC is ap-southeast-3.ecs.internal. You can assign a hostname to the ECS instance in the format of instance ID and configure the private domain name resolution based on this format. Then the ECS instance can access the EMR through the host domain name i-******.ap-southeast-3.ecs.internal. This process is illustrated in the following chart:
Scenarios
Using DNS host domain names to access servers applies in the following scenarios:
Distributed computing clusters: In distributed environments like Hadoop, DNS host domain names enable external clients to access services in the EMR. This simplifies resource scheduling and job submissions.
Automated deployment and configuration management: By using DNS host domain names instead of IP addresses in scripts and configuration files, you can ensure that service discovery and configuration remain accurate, even if the IP addresses change.
Example
A company has a VPC in Malaysia (Kuala Lumpur) and deploys an EMR cluster in the VPC for big data analysis. An EMR cluster typically includes Master and Core nodes. Kerberos, as an authentication protocol, is deployed on the Master node to ensure cluster security and secure communication among all nodes. Job submissions are only limited to the EMR cluster.
In response to business expansion, the company needs to submit jobs from an external ECS instance client (equipped with a Gateway environment) to the Master node whose hostname is i-8ps******5py2u. The company can activate the DNS hostname feature for the VPC, after which external ECS clients can communicate with the Master node through the DNS host domain name.
Figure 1. External clients access the Master node in the EMR
Usage notes
Do not delete the default DNS server address when you enable the DNS hostname. If the server address has been deleted, the system cannot resolve the hostname.
External client ECS instances (equipped with a Gateway environment) in the same VPC must share the same security group as the Master node of the EMR cluster.
DNS hostname is only available to users who receive invitations. Contact your account manager if you need this feature.
Prerequisites
A VPC has been created in Malaysia (Kuala Lumpur).
An EMR cluster has been created in the EMR on the ECS console. The cluster uses the default settings with a Master node and two Core nodes. Kerberos is enabled for identity authentication.
An ECS instance with a Gateway environment has been created to submit EMR jobs from external sources.
Procedure
Step 1: Enable DNS hostname
Log on to the VPC console.
In the top navigation bar, select the region of your VPC. Malaysia (Kuala Lumpur) is chosen in this example.
On the VPC page, click the ID of the VPC instance. Under the Basic Information tab, enable the DNS hostname feature.
Step 2: Configure private domain name resolution for ECS instances
After the DNS hostname is enabled, you need to configure the private domain name resolution for the ECS instances of the external client and the Master node.
Log on to the ECS console.
In the left-side navigation pane, choose .
Configure the private domain name resolution for the ECS instances of the client and Master node.
In the Actions column, select
> Instance Attributes > Modify Instance Attributes. In the Modify Instance Attributes dialog box, configure the domain name resolution as follows:
Step 3: Verify results
This example uses a client with the Alibaba Cloud Linux 3.2104 64-bit operating system to verify the resolution results.
Log on to the client ECS instance and open the command line interface.
Run the ping command to test whether the ECS instance can access the Master node through the DNS domain name.
ping i-8ps******5py2u.ap-southeast-3.ecs.internalIf you receive the following response messages, this indicates that the DNS host domain name can access the Master node:
Related operations
Disable DNS hostname
After the DNS hostname has been disabled, the private domain names assigned by the VPC become invalid. In such a case, the resolution of the IP address through the private domain name will fail.
On the VPC page, click the instance ID of the VPC that you want to manage. Under the Basic Information tab, find DNS Hostname and select Disable.
References
For more information about DHCP options sets, see Work with DHCP options sets.