Alibaba Cloud DevOps Cookbook Part 1 – CLI, SDK, SSH, SFTP

By John Hanley, Alibaba Cloud Tech Share Author. Tech Share is Alibaba Cloud’s incentive program to encourage the sharing of technical knowledge and best practices within the cloud community.

This section is the beginning of applying DevOps to Alibaba Cloud. I plan to start simple and build simple tools and then following up with more complex real-world DevOps tools. The key is to start with the basics and understand the low-level features.

Now that I have a test website that is load balanced and has auto scaling, I would like to learn more about the Alibaba Cloud CLI and Python SDK. During development I often need to make changes to files that I publish on my ECS instances. Since the auto scaling group is built from an image, changing the image takes effort and time. During testing, I want to do rapid-fire edit / deploy / debug / improve. This means that I need a quick way to upload files to my ECS instances all at once.

Part 1. Working with the CLI

Using an SSH GUI based program to copy files is easy. However, with auto scaling, I don't know how many instances are running nor their IP addresses. These variables can change at any time. Today, I will start with just the CLI and then improve by combining the CLI and Python to create a program that will do the following:

Get a list of ECS instances attached my load balancer (SLB).
For each ECS instance, get the public IP address.
For each ECS instance upload a file using SFTP.

This sounds simple, but this will take some work.

To work with the SLB, we will need to install the SDK for SLB. Execute the following command while c:Python27Scripts is in your path:

# pip install aliyun-python-sdk-slb

Let's list the Server Load Balancers that are in our account. Replace the region with your region.

# aliyuncli slb DescribeLoadBalancers

The result looks like this:

{
    "LoadBalancers": {
        "LoadBalancer": [
            {
                "VpcId": "",
                "RegionIdAlias": "us-west-1",
                "Address": "47.254.99.119",
                "AddressType": "internet",
                "LoadBalancerStatus": "active",
                "LoadBalancerName": "NeoPrime-SLB",
                "PayType": "PayOnDemand",
                "SlaveZoneId": "us-west-1b",
                "ResourceGroupId": "rg-abcdefe6zn3hvwi",
                "CreateTimeStamp": 1526271303000,
                "VSwitchId": "",
                "LoadBalancerId": "lb-abcdef1a7bf644x6s1fzg",
                "InternetChargeType": "4",
                "MasterZoneId": "us-west-1a",
                "NetworkType": "classic",
                "CreateTime": "2018-05-14T12:15Z",
                "RegionId": "us-west-1"
            }
        ]
    },
    "TotalCount": 1,
    "PageNumber": 1,
    "RequestId": "3AB8086E-1111-2222-3333-B4BC43AB548E",
    "PageSize": 1
}

We are interested in the Load Balancer ID:

"LoadBalancerId": "lb-abcdef1a7bf644x6s1fzg",

Next, I want to display the ECS instances that are running under the SLB.

# aliyuncli slb DescribeLoadBalancersRelatedEcs --LoadBalancerId lb-abcdef1a7bf644x6s1fzg

The result looks like this:

{
    "LoadBalancers": {
        "LoadBalancer": [
            {
                "Count": 2,
                "MasterSlaveVServerGroups": {
                    "MasterSlaveVServerGroup": []
                },
                "VServerGroups": {
                    "VServerGroup": []
                },
                "BackendServers": {
                    "BackendServer": [
                        {
                            "NetworkType": "vpc",
                            "VmName": "i-0123451kxyhfs89rykzg",
                            "Weight": 50
                        },
                        {
                            "NetworkType": "vpc",
                            "VmName": "i-01234538slefk99bps9h",
                            "Weight": 50
                        }
                    ]
                },
                "LoadBalancerId": "lb-abcdef1a7bf644x6s1fzg"
            }
        ]
    },
    "Message": "successful",
    "RequestId": "D3FA6102-B103-4CC5-9482-2F688C38DC7D",
    "Success": true
}

We are interested in the VmName for each instance:

            "VmName": "i-0123451kxyhfs89rykzg",

            "VmName": "i-01234538slefk99bps9h",

Next, I want to display the IP Address for the first ECS instance. Note: this command is one line.

# aliyuncli ecs DescribeInstanceAttribute --InstanceId i-0123451kxyhfs89rykzg --filter PublicIpAddress.IpAddress[0]

The result looks like this:

"47.254.99.120"

Next, I want to display the IP Address for the other ECS instance. Note: this command is one line.

# aliyuncli ecs DescribeInstanceAttribute --InstanceId i-01234538slefk99bps9h --filter PublicIpAddress.IpAddress[0]

The result looks like this:

"47.254.99.121"

That was easy. Now all that is required is to use SSH and upload my files to the ECS instances using the SFTP protocol. Since I use the Bitvise SSH program, I use the Bitvise SFTP client to upload files. The following command is for Windows. You will need your Key Pair that you specified when created the auto scaling group.

set CMD="put -o c:/tmp/hello.txt /var/www/html/hello.txt" 

sftpc -keypairfile=Test-KeyPair.pem root@47.254.99.120 -cmd=%CMD% 
sftpc -keypairfile=Test-KeyPair.pem root@47.254.99.121 -cmd=%CMD%

Again, that was easy. However, tomorrow or next month, the instances running may be different and they might have different IP addresses. This means that I will have to edit my batch script that figures out the ECS instance IDs and Public IP addresses. An improved solution is to combine the CLI with Python to automate this.

Part 2. Automate the CLI with Python

You can download a zip package of the files mention in this part. Download slb_upload.zip

The python code is Python 3.x. The python version is 3.6.1.

The file slb_upload.py is the main program. This program takes either 3 or 4 arguments. If you only have one Server Load Balancer, then you do not need to specify the fourth parameter. If you do have more than one SLB, then this is the SLB ID.

Usage: slb_upload local_path remote_path [slb_id]

Example command:

python slb_upload c:\work\websites\neoprime.xyz\hello.html /var/www/html/hello.html

The file myslb.py contains routines to interface with SLB and ECS. This file contains the following functions:

ECS_UploadFIle - Uploads a file using SFTP to an ECS instance.
ECS_GetPublicIp - Takes a VmName from SLB and gets the Public IP Address for the ECS instance.
SLB_GetServerLoadBalancers - Returns an array of Sever Load Balancers
SLB_GetBackendInstances - Returns an array of ECS instances attached to an SLB

Source file: myslb.py

import json
import os
import pysftp
import subprocess
import sys
import myutils

#
# This value is the full path to the CLI
#
program_cli = 'C:\\Python27\\Scripts\\aliyuncli.exe'

#
# This function will take an IP Address and upload a file using SFTP
#
# ip - this is the Public IP address
# username - This is the username to use with the Key Pair. For Ubuntu this is "root"
# keypair - This is the SSK Key Pair (PEM file) to connect to the ECS instance
# local_path - This is the path on the Windows system of the file to be uploaded
# remote_path - This is the path on the ECS instance. This needs to be a full path.
#
# WARNING: There is a security issue with this code. This function does NOT verify the
# host fingerprint.
#

def ECS_UploadFile(ip, username, keypair, local_path, remote_path):
    cnopts = pysftp.CnOpts()
    cnopts.hostkeys = None   
    try:
        with pysftp.Connection(
                    ip,
                    username=username,
                    private_key=keypair,
                    cnopts=cnopts) as sftp:

            myutils.printf("Uploading %s to %s:%s", local_path, ip, remote_path)
            sftp.put(local_path, remote_path)

    except Exception as ex:
        print("Exception: ", ex)
        sys.exit(-1)

#
# This function will take a VmName (from the SLB json) and return the Public IP Address
#
# vmname - This is the VmName value returned from "aliyuncli slb DescribeLoadBalancersRelatedEcs"
#

def ECS_GetPublicIp(vmname):
    args =    [
            program_cli,
            'ecs',
            'DescribeInstanceAttribute',
            '--InstanceId',
            vmname,
            '--filter',
            'PublicIpAddress.IpAddress[0]'
        ]

    process = subprocess.Popen(args, stdout=subprocess.PIPE)
    (output, err) = process.communicate()
    rc = process.wait()

    data = json.loads(output)

    return data

#
# This function will return a list of Server Load Balancers
#
# This CLI command will list the SLBs
# aliyuncli slb DescribeLoadBalancers
#

def SLB_GetServerLoadBalancers():
    args =    [
            program_cli,
            'slb',
            'DescribeLoadBalancers',
            '--filter',
            'LoadBalancers.LoadBalancer[].LoadBalancerId'
        ]

    process = subprocess.Popen(args, stdout=subprocess.PIPE)
    (output, err) = process.communicate()
    rc = process.wait()

    if rc != 0:
        return []

    data = json.loads(output)

    return data

#
# List the ECS instances running under this load balancer
# aliyuncli slb DescribeLoadBalancersRelatedEcs --LoadBalancerId lb-abcdef1a7bf644x6s1012
#

def SLB_GetBackendInstances(slb_id):
    args =    [
            program_cli,
            'slb',
            'DescribeLoadBalancersRelatedEcs',
            '--LoadBalancerId',
            slb_id
        ]

    try:
        print("Describing Load Balancer ECS Instances")
        process = subprocess.Popen(args, stdout=subprocess.PIPE)
        (output, err) = process.communicate()
        rc = process.wait()
        if rc != 0:
            return []

    except Exception as ex:
        print("Exception: ", ex)
        return []

    #print(output)

    data = json.loads(output)

    slbs = data['LoadBalancers']
    slb = slbs['LoadBalancer'][0]
    bes = slb['BackendServers']['BackendServer']

    return bes

Source file: slb_upload.py

import json
import os
import pysftp
import subprocess
import sys
import myutils
import myslb

#
# This value is the full path to the CLI
#
program_cli = 'C:\\Python27\\Scripts\\aliyuncli.exe'

# This is the SSH login
username ="root"

# Key Pair for SSH to connect to the ECS instance
key_pair = "../Test-KeyPair.pem"

# Local file to upload to the ECS instance
# local_path = "c:/tmp/hello.txt"

# Remote file that will be uploaded to the ECS instance
# remote_path = "/var/www/html/hello.txt"

# This is the Alibaba Cloud SLB that we want to work with
slb_id = ''

# Array of Backend Servers (ECS instances attached to a server load balancer
bes = []

if len(sys.argv) < 3:
    print("Error: Missing command line arguments")
    print("Usage: slb_upload local_path remote_path [slb_id]")
    sys.exit(-1)

local_path = sys.argv[1]
remote_path = sys.argv[2]

# Verify that the CLI exists
if os.path.exists(sys.argv[0]) == False:
    print("Error: Missing aliyun CLI: ", args[0])
    sys.exit(-1)

# Get the list of Server Load Balancers
slbs = myslb.SLB_GetServerLoadBalancers()

if len(slbs) == 0:
    print("Error: You do not have any server load balancers")
    sys.exit(1)

# if there is only one SLB, use that one
if len(slbs) == 1:
    slb_id = slbs[0]
else:
    # if there is more than one SLB, then use the one specified on the command line
    if len(sys.argv) < 4:
        # The user did not specify a Server Load Balancer
        print("Error: You have more than one server load balancer")
        print("Please specify the server load balancer ID on the command line")
        sys.exit(1)

    slb_id = sys.argv[3]

# Make sure that we have an SLB ID (not empty test)
if slb_id == '':
    print("Error: Missing server load balancer ID")
    print("Usage: slb_upload local_path remote_path [slb_id]")
    sys.exit(1)

# Get the list of ECS instances attached to this SLB
bes = myslb.SLB_GetBackendInstances(slb_id)

if len(bes) == 0:
    print("Error: Your SLB does not have any ECS instances attached")
    sys.exit(1)

print("Total Backend ECS Servers: ", len(bes))
#print(slb['BackendServers'])

# We have everything, upload files
print('Processing:')
print("------------------------------------------------------------")
for index, vm in enumerate(bes):
    vmname = vm['VmName']
    #print("VmName: ", vmname)

    ip = myslb.ECS_GetPublicIp(vmname)
    #print("Public IP:", ip)

    myslb.ECS_UploadFile(ip, username, key_pair, local_path, remote_path)
print("------------------------------------------------------------")

sys.exit(0)

Part 3. Automating with Python and the SDKs

You can download a zip package of the files mention in this part. Download slb_upload_python.zip

The code in Part 2 demonstrates how rapidly you can use Python and the CLI to automate tasks. To start to really understand the Alibaba Cloud Tools and SDKs, the next logical step is to convert the program into 100% SDK based instead of using the CLI. This will happen next.

When using the Python SDK, the first step is to setup your credentials. The Alibaba CLI creates the directory .aliyuncli in your HOME directory. In this directory are two files:

configure - This file contains items like region, preferred output such as json, etc.
credentials - This file contains your Access Key and Access Key Secret

At the CMD Prompt execute this command:

aliyuncli configure

The CLI will prompt you for several items:

Aliyun Access Key ID
Aliyun Access Key Secret
Default Region ID
Default output format

For the first two items, these are the keys that you saved when you create the new user. For the region, either leave it blank or specify your preferred region. I used us-west-1. For the output format, I prefer json.

Next we will port the myslb.py module to replace the CLI command line calls with Python SDK calls.

Source file: myslb.py

############################################################
# Version 1.01
# Date Created: 2018-05-15
# Last Update:  2018-05-16
# https://www.neoprime.io
# Copyright (c) 2018, NeoPrime, LLC
############################################################

"""
Functions to support SLB and ECS
"""

import json
import subprocess
import sys
import myutils
import pysftp

from aliyunsdkcore.client import AcsClient
from aliyunsdkcore.acs_exception.exceptions import ClientException
from aliyunsdkcore.acs_exception.exceptions import ServerException
from aliyunsdkecs.request.v20140526 import DescribeInstanceAttributeRequest
from aliyunsdkslb.request.v20140515 import DescribeLoadBalancersRequest
from aliyunsdkslb.request.v20140515 import DescribeLoadBalancersRelatedEcsRequest

#
# This function will take an IP Address and upload a file using SFTP
#
# ip - this is the Public IP address
# username - This is the username to use with the Key Pair. For Ubuntu this is "root"
# keypair - This is the SSK Key Pair (PEM file) to connect to the ECS instance
# local_path - This is the path on the Windows system of the file to be uploaded
# remote_path - This is the path on the ECS instance. This needs to be a full path.
#
# WARNING: There is a security issue with this code. This function does NOT verify the
# host fingerprint.
#

def ECS_UploadFile(ip, username, keypair, local_path, remote_path):
    """ This function will take an IP Address and upload a file using SFTP """
    cnopts = pysftp.CnOpts()
    cnopts.hostkeys = None
    try:
        with pysftp.Connection(
                    ip,
                    username=username,
                    private_key=keypair,
                    cnopts=cnopts) as sftp:

            myutils.printf("Uploading %s to %s:%s", local_path, ip, remote_path)
            sftp.put(local_path, remote_path)

    except Exception as ex:
        print("Exception: ", ex)
        sys.exit(-1)

#
# This function will take a VmName (from the SLB json) and return the Public IP Address
#
# cred - This is the dictionary of credentials (AccessKey, AccessKeyScret, Region)
# vmname - This is the VmName value returned from "aliyuncli slb DescribeLoadBalancersRelatedEcs"
#

def ECS_GetPublicIp(cred, vmname, debugFlag=False):
    """ This function will take a VmName (from the SLB json) and return the Public IP Address """

    client = AcsClient(cred['AccessKey'], cred['AccessKeySecret'], cred['Region'])

    request = DescribeInstanceAttributeRequest.DescribeInstanceAttributeRequest()

    request.set_InstanceId(vmname)

    try:
        response = client.do_action_with_exception(request)
        data = json.loads(response)

    except Exception as ex:
        print("Exception: ", ex)
        return False;

    ip = data['PublicIpAddress']['IpAddress']

    if len(ip) > 0:
        return ip[0]

    return False;

#
# This function will return a list of Server Load Balancers
#
# This CLI command will list the SLBs
# aliyuncli slb DescribeLoadBalancers
#

def SLB_GetServerLoadBalancers(cred, debugFlag=False):
    """ This function will return a list of Server Load Balancers """

    client = AcsClient(cred['AccessKey'], cred['AccessKeySecret'], cred['Region'])

    request = DescribeLoadBalancersRequest.DescribeLoadBalancersRequest()

    try:
        response = client.do_action_with_exception(request)
        data = json.loads(response)

    except Exception as ex:
        print("Exception: ", ex)
        return [];

    slbs = data['LoadBalancers']

    if len(slbs) == 0:
        return []

    ids = []

    for slb in slbs['LoadBalancer']:
        ids.append(slb['LoadBalancerId'])

    return ids;

#
# List the ECS instances running under this load balancer
# aliyuncli slb DescribeLoadBalancersRelatedEcs --LoadBalancerId lb-abcdef1a7bf644x6s1012
#

def SLB_GetBackendInstances(cred, slb_id, debugFlag=False):
    """ List the ECS instances running under this load balancer """

    client = AcsClient(cred['AccessKey'], cred['AccessKeySecret'], cred['Region'])

    request = DescribeLoadBalancersRelatedEcsRequest.DescribeLoadBalancersRelatedEcsRequest()

    request.set_LoadBalancerId(slb_id)

    try:
        response = client.do_action_with_exception(request)
        data = json.loads(response)

    except Exception as ex:
        print("Exception: ", ex)
        return [];

    slbs = data['LoadBalancers']

    if len(slbs['LoadBalancer']) == 0:
        return []

    slb = slbs['LoadBalancer'][0]
    bes = slb['BackendServers']['BackendServer']

    return bes

The file mycred_acs.py contains very simple routines to load the credentials from the configure and credentials files mentioned above. I am a firm believer in never hard-coding your credentials in source code.

Source file: mycred_acs.py

############################################################
# Version 1.01
# Date Created: 2018-05-15
# Last Update:  2018-05-16
# https://www.neoprime.io
# Copyright (c) 2018, NeoPrime, LLC
############################################################

""" Automating with Python and the Alibaba Cloud SDKs """

import os
import sys
import platform
import configparser

def getHomeDir():
    dir = ""

    if platform.system() == "Windows":
        dir = os.environ['HOMEDRIVE'] + os.environ['HOMEPATH']
        pass
    else:
        dir = os.environ['HOME']
        pass

    return dir

def getCredentialsPath():
    dir = getHomeDir()

    dir = dir + "\\.aliyuncli/credentials"

    return dir

def getConfigurePath():
    dir = getHomeDir()

    dir = dir + "\\.aliyuncli/configure"

    return dir

def loadConfigureFile(path, cred, profileName='default'):
    if os.path.exists(path) is False:
        print("Error: Cannot load configure file from file")
        print("File: ", path)
        return False;

    config = configparser.ConfigParser()
    config.read(path)

    try:
        cred['Region'] = config[profileName]['region']

    except Exception as ex:
        print("Exception: ", ex)
        print("Error: Cannot load configure profile name: ", profileName)
        return False;

    return cred;


def loadCredentialsFile(path, profileName='default'):
    cred = {
        'AccessKey': '',
        'AccessKeySecret': '',
        'Region': ''
        }

    if os.path.exists(path) is False:
        print("Error: Cannot load credentials file from file")
        print("File: ", path)
        return False;

    config = configparser.ConfigParser()
    config.read(path)

    try:
        cred['AccessKey'] = config[profileName]['aliyun_access_key_id']
        cred['AccessKeySecret'] = config[profileName]['aliyun_access_key_secret']

    except Exception as ex:
        print("Exception: ", ex)
        print("Error: Cannot load credential profile name: ", profileName)
        return False;

    return cred;

def LoadCredentials(profileName='default'):
    cred_file = getCredentialsPath()
    conf_file = getConfigurePath()

    cred = loadCredentialsFile(cred_file, profileName)

    if cred is False:
        return False;

    loadConfigureFile(conf_file, cred, profileName)

    return cred;

And now for the main program slb_upload.py. This was modified a bit to support passing credentials to the ECS and SLB support functions.

Source file: slb_upload.py

############################################################
# Version 1.01
# Date Created: 2018-05-15
# Last Update:  2018-05-16
# https://www.neoprime.io
# Copyright (c) 2018, NeoPrime, LLC
############################################################

""" Automating with Python and the Alibaba Cloud SDKs """

import os
import sys
import platform

# Begin: Verify that we are running under Python version 3.6 or greater

if platform.sys.version_info.major < 3:
    print("This software requires Python 3.6 or greater")
    sys.exit(-1)

if platform.sys.version_info.major == 3 and platform.sys.version_info.minor < 6:
    print("This software requires Python 3.6 or greater")
    sys.exit(-1)

if sys.version_info[0] < 3:
    print("This software requires Python 3.6 or greater")
    sys.exit(-1)

# End: Verify that we are running under Python version 3.6 or greater

import myslb
import optparse
from slb_cmdline import process_cmdline, usage
# My library for processing Alibaba Cloud Services (ACS) credentials
import mycred_acs

# This is the SSH login
username = "root"

# Key Pair for SSH to connect to the ECS instance
key_pair = "../Test-KeyPair.pem"

# Local file to upload to the ECS instance
# local_path = "c:/tmp/hello.txt"

# Remote file that will be uploaded to the ECS instance
# remote_path = "/var/www/html/hello.txt"

# This is the Alibaba Cloud SLB that we want to work with
slb_id = ''

# This is the Alibaba Cloud Region that we want to work with
region_id = ''

# Array of Backend Servers (ECS instances attached to a server load balancer
bes = []

# We need a minimum of 2 command line parameters (argv >= 3)
if len(sys.argv) < 3:
    print("Error: Missing command line arguments")
    usage();
    sys.exit(-1)

# Process the command line
(options, args) = process_cmdline()

if len(args) < 2:
    print("Error: Not enough command line arguments")
    usage();
    sys.exit(-1)

if len(args) > 2:
    print("Error: Too many command line arguments")
    usage();
    sys.exit(-1)

local_path = args[0]
remote_path = args[1]

# Load the Alibaba Cloud Credentials (AccessKey)
cred = mycred_acs.LoadCredentials(profileName = options.profile)

if cred is False:
    sys.exit(-1)

if options.debug:
    print(cred)

# Verify that the CLI exists
if os.path.exists(sys.argv[0]) is False:
    print("Error: Missing aliyun CLI: ", sys.argv[0])
    sys.exit(-1)

if options.slb_id:
    slb_id = options.slb_id

if options.slb_id is False:
    # Get the list of Server Load Balancers
    slbs = myslb.SLB_GetServerLoadBalancers(cred, options.debug)

    if options.debug:
        print("Server Load Balancers")
        print(slbs)

    #if len(slbs) == 0:
    n = len(slbs)
    if n == 0:
        print("Error: You do not have any server load balancers")
        sys.exit(1)

    # if there is only one SLB, use that one
    if len(slbs) == 1:
        slb_id = slbs[0]
    else:
        # if there is more than one SLB, then use the one specified on the command line
        # The user did not specify a Server Load Balancer
        print("Error: You have more than one server load balancer")
        print("Please specify the server load balancer ID on the command line")
        sys.exit(1)
else:
    if options.debug:
        print("SLB ID: ", slb_id)

# Make sure that we have an SLB ID (not empty test)
if slb_id == '':
    print("Error: Missing server load balancer ID")
    usage();
    sys.exit(1)

# Get the list of ECS instances attached to this SLB
bes = myslb.SLB_GetBackendInstances(cred, slb_id, options.debug)

#if len(bes) == 0:
n = len(bes)
if n == 0:
    print("Error: Your SLB does not have any ECS instances attached")
    sys.exit(1)

print("Total Backend ECS Servers: ", len(bes))
#print(slb['BackendServers'])

if options.debug:
    print("Backend Servers")
    print(bes)

# We have everything, upload files
print('Processing:')
print("------------------------------------------------------------")
for index, vm in enumerate(bes):
    vmname = vm['VmName']
    #print("VmName: ", vmname)

    ip = myslb.ECS_GetPublicIp(cred, vmname, options.debug)

    if ip is False:
        print("Error: Cannnot determine IP address for instance: ", vmname)
        continue;

    print("Public IP:", ip)

    myslb.ECS_UploadFile(ip, username, key_pair, local_path, remote_path)
print("------------------------------------------------------------")

sys.exit(0)

Continue reading Alibaba Cloud DevOps Cookbook Part 2

Developer Documents

Command Line Interface (CLI):

Alibaba Cloud CLI

Server Load Balancer (SLB):

Elastic Compute Service (ECS):

Community

Alibaba Cloud DevOps Cookbook Part 1 – CLI, SDK, SSH, SFTP

Part 1. Working with the CLI

Part 2. Automate the CLI with Python

Part 3. Automating with Python and the SDKs

Developer Documents

Read previous post:

Read next post:

Alibaba Clouder

You may also like

Comments

Alibaba Clouder

Related Products

ECS(Elastic Compute Service)