Managing Alibaba Cloud Resources in Ansible Using Dynamic Inventory

By He Guimin, nicknamed Xiaozhu at Alibaba.

In this blog we will introduce how you can manage Alibaba Cloud resources in Ansible using Dynamic Inventory. But, before we get ahead of ourselves, first, we'll need to introduce what is Ansible and what you can use it for.

Ansible is a mainstream automated O&M tool that can operate one or more hosts at the same time. When the number of hosts increases and machine roles become more complex and diverse in the cloud, a system of grouping management becomes essential to effectively managing your resources.

Ansible is one of the most powerful tools for this means. It can operate multiple hosts in a group at the same time. With Ansible, the relationship between groups and hosts is configured by inventory files. In Ansible, there are two types of inventory: the static inventory and dynamic inventory types.

Static inventory refers to default inventory that includes the host name, host address, connection information, and connection method in a static file. For example:

[targets]

localhost              ansible_connection=local
other1.example.com     ansible_connection=ssh        ansible_user=mpdehaan
other2.example.com     ansible_connection=ssh        ansible_user=mdehaan
10.0.2.23              ansible_connection=ssh        ansible_user=mdehaanip

In a playbook, you can directly specify the group name or the name of a specific host.

Dynamic inventory refers to the process where you can automatically obtain a list of hosts through an external script, automatically group the list of hosts according to the configured grouping method, and return the host information to the corresponding Ansible command in the format required by Ansible. The following is part of the host information obtained from the dynamic inventory of Alibaba Cloud in Ansible:

{
  "_meta": {
    "hostvars": {
      "i_bp171m264ryt9*******": {
        "ansible_ssh_host": "47.98.xx.xx", 
        "availability_zone": "cn-hangzhou-g", 
        "block_device_mapping": [
          ...
        ], 
        ...
        "host_name": "iZbp171m264ryt9******Z", 
        "id": "i-bp171m264ryt9*******", 
        "image_id": "ubuntu_16_0402_64_20G_alibase_20180409.vhd", 
        "inner_ip_address": "", 
        ...
        "tags": {
          "env": "dev"
        }      }, 
      "i_bp1i1aitghkkq*******": {
        "ansible_ssh_host": "47.96.xx.xx", 
        "availability_zone": "cn-hangzhou-g", 
        "block_device_mapping": [
          ...
        ], 
        "eip": {
          ...
          "ip_address": "47.96.xx.xx"
        }, 
        ...
        "host_name": "terraform", 
        "id": "i-bp1i1aitghkkq*******", 
        "image_id": "m-bp1243pi65bw8*****", 
        "inner_ip_address": "", 
        ...
      }, 
      ...
    }
  }
},
"alicloud": {
  "children": [
    "i_bp1i1aitghkkq*******", 
    "i_bp171m264ryt9*******"
  ]
},
"cn-hangzhou": [
  "i_bp1i1aitghkkq*******", 
  "i_bp171m264ryt9*******"
],
"cn-hangzhou-g": [
  "i_bp1i1aitghkkq*******", 
  "i_bp171m264ryt9c******"
],

The dynamic inventory responds to dynamic resource increases and decreases by calling the service API or accessing the database query to automatically query and return a list of hosts and update the host information in the inventory.

Alibaba Cloud has provided a dynamic inventory file to dynamically obtain the host information that meets the specified filtering conditions.

Preparing the Test Environment

Alibaba Cloud dynamic inventory is currently in the process of official integration. Therefore, before you use this dynamic inventory, you'll need to install the ansible_alicloud_module_utils component on which this inventory depends. This component is used for the inventory's requests to APIs in OpenAPI:

# 直接使用 Pip 完成对 Ansible Alicloud Module Utils 的安装
$ sudo pip install ansible_alicloud_module_utils

When you install ansible_alicloud_module_utils, Ansible and component dependencies, such as footmark and Alibaba Cloud Python SDK, are automatically installed. When the installation is completed, run the pip show footmark command to ensure that the footmark version is later than 1.9.0.

$ pip show footmark
Name: footmark
Version: 1.9.0
Summary: A Python interface to Aliyun Web Services
Home-page: UNKNOWN
Author: xiaozhu
Author-email: heguimin36@163.com
License: MIT
Location: /Library/Python/2.7/site-packages
Requires: aliyun-python-sdk-core, aliyun-python-sdk-ecs, aliyun-python-sdk-slb, aliyun-python-sdk-vpc, aliyun-python-sdk-rds, aliyun-python-sdk-ess, oss2, importlib
Required-by: ansible-alicloud, ansible-alicloud-module-utils

If the version number does not meet this requirement, run the sudo pip install footmark --upgrade command to upgrade the footmark version.

Specifying an Alibaba Cloud Dynamic Inventory

Like other inventories, an Alibaba Cloud dynamic inventory can be used in two ways: either you can "explicitly specify" and "specify by default". "Explicitly specify" refers to using the -i parameter to explicitly specify the dynamic inventory file when an Ansible command is run.

Downloading the Alibaba Cloud Dynamic Inventory File

Download the latest version of the Alibaba Cloud dynamic inventory file and grant it executable permissions:

$ wget https://raw.githubusercontent.com/alibaba/ansible-provider/master/contrib/inventory/alicloud.py
$ chmod +x alicloud.py

Download and obtain the alicloud.ini configuration file, and put it in the directory where alicloud.py is located:

$ wget https://raw.githubusercontent.com/alibaba/ansible-provider/master/contrib/inventory/alicloud.ini

The alicloud.ini configuration file contains a variety of configuration options, such as the cache control, destination address variable, host attributes, and region where the host is located. You can implement custom filtering of the host information by using this file.

Pre-configuration of AccessKey ID and AccessKey Secret

Before you call the inventory file, specify the AccessKey ID and the AccessKey secret to be used, which can be directly defined in the alicloud.ini configuration file:

alicloud_access_key = Abcd1234
alicloud_secret_key = Abcd2345

You can also directly specify the AccessKey ID and AccessKey secret through environment variables:

$ export ALICLOUD_ACCESS_KEY = Abcd1234
$ export ALICLOUD_SECRET_KEY = Abcd2345

Verification

To test whether the configuration is correct, run the inventory file:

$ ./alicloud.py --list

You can see the inventory information of all regions in JSON format, as shown above.

If you want to obtain only the information about some regions, edit alicloud.ini and enter the desired regions. In addition, you can filter all host information by using instance_filters in alicloud.ini.

The Alibaba Cloud dynamic inventory provides mapping from multiple groups to instances:

Global: All the instances that belong to the alicloud group. For example:

 "alicloud": {
     "children": [
       "i_bp1i1aitghkkq*******", 
       "i_bp171m264ryt9*******"
     ]
  },

Instance ID: The ID of the instance. For example:

"i-bp171m264ryt9******": [
      "i_bp171m264ryt9******"
], 
"i-bp1i1aitghkkq******": [
      "i_bp1i1aitghkkq*****"
],

Region: All the instances in an Alibaba Cloud region that form a group. For example:

"cn-hangzhou": [
    "i_bp1i1aitghkkq*******", 
    "i_bp171m264ryt9*******"
  ],

Zone: All the instances in an Alibaba Cloud zone that form a group. For example:

"cn-hangzhou-g": [
      "i_bp1i1aitghkkq*******", 
      "i_bp171m264ryt9c******"
],

Security group: An instance can belong to one or more security groups. The prefix of a group is security_group. For example:

"security_group_sg_bp1cp0behw74aa******": [
    "i_bp1i1aitghkkqp******"
], 
"security_group_sg_bp1dtemf7bv5******": [
    "i_bp171m264ryt9******"
],

Tag: Each instance has multiple different key-value pairs. These key-value pairs are called tags. You can define tag names as needed. Each key-value pair is a group. Special characters are converted to underlines, and the format is tag_KEY_VALUE. For example:

"tag_acsversion_1_0": [
    "i_t4nd1ehd9umu5******", 
    "i_t4n6v8wv6jue5******"
], 
"tag_env_dev": [
    "i_bp171m264ryt9******"
],

In addition, many other similar attributes, such as VPC ID, VSwitch ID, and image ID, which are used to divide host groups.

Scenarios

When confirming the Alibaba Cloud inventory to be executable, you can use alicloud.py as an inventory application in the specific use case of Ansible, as follows:

$ ansible -i alicloud.py alicloud -m ping
47.93.xx.xx | SUCCESS => {
    "changed": false, 
    "ping": "pong"
}
47.93.xx.xx | SUCCESS => {
    "changed": false, 
    "ping": "pong"
}

If the following problems (shown below) occur during the running process:

47.93.xx.xx| UNREACHABLE! => {
    "changed": false, 
    "msg": "Failed to connect to the host via ssh: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).\r\n", 
    "unreachable": true
}
47.93.xx.xx | UNREACHABLE! => {
    "changed": false, 
    "msg": "Failed to connect to the host via ssh: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).\r\n", 
    "unreachable": true
}

Then, you must specify the host instance username and password (shown below) during execution:

$ ansible -i alicloud.py alicloud -m ping -u root -k
SSH password: 

47.93.xx.xx | SUCCESS => {
    "changed": false, 
    "ping": "pong"
}
47.93.xx.xx | SUCCESS => {
    "changed": false, 
    "ping": "pong"
}

Refreshing Host Information

Noticeably, the Alibaba Cloud dynamic inventory file caches the result of each execution to avoid repeated API calls. You can configure the cache settings by editing cache_path in the alicloud.ini file. To explicitly clear cache, add --refresh-cache when executing the inventory:

$ ./alicloud.py --refresh-cache

Specifying an Alibaba Cloud Dynamic Inventory

In addition to explicitly specifying an Alibaba Cloud dynamic inventory file, you can also set a dynamic inventory to be the default inventory. After Ansible is installed, the default Ansible inventory file is /etc/ansible/hosts.

Download the latest version of the Alibaba Cloud dynamic inventory file, grant it executable permissions, and then use it to replace the default inventory file:

$ wget https://raw.githubusercontent.com/alibaba/ansible-provider/master/contrib/inventory/alicloud.py
$ chmod +x alicloud.py
$ sudo cp alicloud.py /etc/ansible/hosts

Download the configuration file of the Alibaba Cloud dynamic inventory and move it to /etc/ansible, where the default inventory file is located:

$ wget https://raw.githubusercontent.com/alibaba/ansible-provider/master/contrib/inventory/alicloud.ini
$ sudo cp alicloud.ini /etc/ansible

Run the following command to verify:

ansible alicloud -m ping

Thoughts

Currently, Alibaba Cloud is further integrating with Ansible, a mainstream open-source tool. Specifically, Alibaba Cloud is developing its resource modules and provisioning dynamic inventory to further improve the developer experience and the O&M efficiency for enterprises.

Alibaba Cloud has already contributed more than 20 modules that involve multiple Alibaba Cloud products and resources. We welcome you to use them.

If you have a problem while using Alibaba Cloud modules, submit it on ansible-provider on GitHub at any time. We will solve it as soon as possible.

The views expressed herein are for reference only and don't necessarily represent the official views of Alibaba Cloud.

Community

Managing Alibaba Cloud Resources in Ansible Using Dynamic Inventory

Preparing the Test Environment

Specifying an Alibaba Cloud Dynamic Inventory

Downloading the Alibaba Cloud Dynamic Inventory File

Pre-configuration of AccessKey ID and AccessKey Secret

Verification

Scenarios

Refreshing Host Information

Specifying an Alibaba Cloud Dynamic Inventory

Thoughts

Read previous post:

Read next post:

Alibaba Clouder

You may also like

Comments

Alibaba Clouder

Related Products

Simple Log Service

DevOps Solution

Alibaba Cloud Flow

ECS(Elastic Compute Service)