By Rifandy Zulvan, Solution Architect Intern
Jenkins is tools that widely used for automation server, some use cases of Jenkins is for implementing a CI/CD pipeline for automate applications deployment pipelines like installing dependencies, build, test, and deploy your applications.
Autoscaling your Jenkins Agent can leverage your CI/CD pipeline to handle more pipelines without having to worry about high build queues. This solution can be done with Alibaba Cloud by using Alibaba Cloud ECS plugin on Jenkins, this also can be a cost-effective solution for your CI/CD pipeline by using Ephemeral ECS instance as your Jenkins Agents.
The common architecture for CI/CD in Jenkins is Master-Slave architecture. In this architecture, we have one Jenkins Master that for controlling and manage some resources, configurations on the Jenkins. And we can have lot of slaves as an agent to do some Jenkins Jobs.
This architecture can be used for a highly load build, or for a different use cases build (Android App Build, IOS App Build, Docker container build, etc). This method can speed up you DevOps implementation, because with a high load of build it can be handled with multiple of slaves, on this blog will gives you a guide to Auto Scale your Jenkins Slaves using Alibaba Cloud, so it can be more effective to handle a lot of build.
Because Jenkins is an open source tools, and it has a quite big communities in Open Source industry. Jenkins provides a lot of plugin and to support your automation needs. One of the Plugins is Alibaba Cloud ECS Plugin.
Alibaba Cloud ECS Plugin can be used to implement an auto provisioning you Jenkins Slave node. This is the mechanism of the Alibaba Cloud ECS Plugin.
So basically, the plugin can provision Alibaba Cloud ECS instances that can be used automatically as Jenkins Slave. With this plugin, you don’t need to set a manual configuration to assign each nodes IP addresses on your Jenkins Master.
This plugin can also be used to automatically scale up your slave, base on slave availability. This flowchart shows the mechanism of how this plugin can scale up your slaves.
One more important feature on this plugin is that you can use Ephemeral ECS instances and On-Demand ECS instances. Using Ephemeral ECS instances as your Jenkins Slave is a cost-effective solution because the Jenkins slaves do not need to be available at all times. Instead, they can be provisioned only when needed and deleted if the slaves are in idle state.
This is a price comparison of on-demand and ephemeral instances on Alibaba Cloud. For further information you can visit the ECS documentation Overview (alibabacloud.com).
This part will show you how to install the Alibaba Cloud ECS Plugin on your Jenkins Master. There are some prerequisites that you need to provide.
I assume you already have a Jenkins Master on your side, if you haven’t you can look at this Alibaba Cloud’s blog about Continuous Integration with Jenkins on Alibaba Cloud.
The first thing to provide is an Account Access Key, you can get it on Resource Access Management (aliyun.com) page on Alibaba Cloud Console, and just click Create AccessKey
After that you can Copy or Download the AccessKey as CSV file, we need to use this Key to set up the plugin later.
To create an SSH Key Pair, you can go to ECS console and then Navigate to Network and Security > Key Pair menu and Click Create SSH Key Pair.
After that, you can set your SSH key pair name, creation type, resource group and tag.
And then you can click OK and you will get the key file with ‘.pem’ format, keep this file we will use the key later.
After this, we can start to install the Alibaba Cloud ECS Plugin on Jenkins Master.
Go to Jenkins Master, and go to Manage Jenkins > Manage Plugins.
Next step, go to Available plugins and search Alibaba Cloud on the search box, click install on the Alibaba Cloud ECS plugin, and then click Install without restart.
Wait for the plugin to installed, and the Plugin is already installed on your Jenkins Master.
To check the plugin has been installed, you can go to Plugin Manager > Installed plugins > and check the Alibaba Cloud ECS plugin.
To configure Alibaba Cloud ECS as your Jenkins Slave, you can go back to Jenkins homepage, and go to Build Executor Status > Configure Clouds.
And on the Configure Clouds page, you can go click Add a new cloud and choose Alibaba Cloud ECS.
Set the name of your Alibaba cloud ECS. If your Jenkins Master is running on Alibaba Cloud and should be running at same VPC with your Slave, Check the Jenkins Master in Private Network options, this option will run and connect the provision ECS Instance with the master with private IP address. If you have a multi-cloud or hybrid architecture you can uncheck this option, For this Blog we uncheck this option.
Next, we need to provide Jenkins our Alibaba Cloud AccessKey. Go to Alibaba Cloud Credentials and click “Add” > “Jenkins”
Choose the credentials kind as Alibaba Cloud Credentials, add your Access Key ID and Secret Access Key, and add the ID and description also, after that you can click Add.
Next, add your credentials, region, VPC, and Security Group.
The next step is to add your ECS SSH Key, click Add > Jenkins and enter your ECS username and your private key.
After entering your SSH username and private key, click Advanced on the configuration and check Assign Public IP and No delay provision settings. You can configure this advanced configuration as your need, if your Jenkins Master running on the same VPC with your Jenkins Slave, you don’t need to Assign Public IP configurations.
After that you can do a connection test to Alibaba Cloud from your Jenkins Master, you can click Test Connection. If there is a connection OK message on the left, your configurations are already correct.
To run a slave you need to configures your Slave Images, the best practice is set an image that can support your Jenkins Jobs task. For example, if you are running a CI/CD pipeline that compile a Golang Application, you need to select an image that already have a Golang installed.
You can create your custom image, or you can choose Alibaba Cloud existing images, for this blog we only choose an existing image.
To start configuring an image, you can click Add on Images configuration.
And then, you can add your image description, and set the Image. If you are using custom Image, you can insert the ID of your custom image, If you are using an existing image, you can choose any images as your demand on this options.
Next, you can specify your Availability zone, Vswitch, Instance Charge Type,and Instance Type.
And you can configure your slave’s root directory, disk type and size.
The next step is to configure minimum and maximum instances number, agent label, instance name. After that, you can test the ECS creation by clicking the TestCreateECS, if there is a Success message, you can save the configuration and your ECS Slave configurations are ready.
To test the slave auto-scaling, we need to create Jenkins Pipeline for our Slave to execute. In this blog we just use sample pipeline like here. To do the slave auto-scale, keep in mind to assign agent pipeline label to your ECS plugin label.
pipeline {
agent {
label 'ecs'
}
stages {
stage('test') {
steps {
echo 'hello'
sleep 60
}
}
}
}
Here's how slave autoscaling works in this scenario, where there are no ECS slaves to begin with, when the build begins, the pipeline will be on the Build Queue, and the ECS slave will provision for roughly 2-5 minutes.
After the ECS slave provisioned the build is automatically assign to the existing slave.
In DevOps practices, CI/CD is a crucial tool. Alibaba Cloud ECS instance can be used as part of your CI/CD tools with Jenkins to automatically scale up your Jenkins Agents. You can also create a cost-effective CI/CD pipeline by using Ephemeral ECS instances as your Jenkins Master. This approach allows you to leverage the power and flexibility of Alibaba Cloud ECS on your Jenkins CI/CD pipeline.
Enhancing Data Privacy: Unleashing the Potential of the Alibaba Cloud Data Security Center
Empowering Businesses: Simplifying SKU Checks with Alibaba Cloud Image Search
99 posts | 15 followers
FollowAlibaba Cloud Community - June 2, 2023
Alibaba Clouder - June 24, 2019
Alibaba Container Service - July 16, 2019
Alibaba Clouder - July 27, 2020
JJ Lim - September 23, 2021
Alibaba Container Service - July 24, 2024
99 posts | 15 followers
Follow
Realtime Compute for Apache Flink
Realtime Compute for Apache Flink offers a highly integrated platform for real-time data processing, which optimizes the computing of Apache Flink.
Learn MoreMore Posts by Alibaba Cloud Indonesia