By Afzaal Ahmad Zeeshan, Alibaba Cloud Community Blog author and Alibaba Cloud MVP.
You must have realized the importance of concurrent programming and processing when working with microprocessors and their applications. Nonetheless, the concept of batch computing is an advanced topic, but the basis of the idea relates strongly to the above-mentioned example. In batch computing, you utilize several CPU cores and then have it perform a task that you require it to perform.
That being said, Alibaba Cloud's Batch Compute is one of the most powerful products and toolset which one should explore for distributed processing of a massive number of datasets. On Alibaba Cloud, Batch Compute is used extensively for several jobs; running a background job every hour, processing the incoming data as the bucket becomes full, or even high-end computing tasks such as rendering computer graphics. All these tasks can be done with Alibaba Cloud Batch Compute, and since the underlying infrastructure is strong enough to support thousands of CPU cores, you are in safe hands.
Alibaba Cloud Batch Compute consists of the toolset that offers resource management, service to automate job scheduling and post-build actions, such as, resource releasing and cleaning. Moreover, you can get all these services in a pay-as-you-use model, which is undoubtedly a cost-cutting feature. Customers do not have to manage or customize the settings at all, they just need to select the kind of worker nodes they are looking for and their plan is ready to be executed.
Fundamentally, the program or task description which you want to compute is termed as your job. In Alibaba Cloud Batch Compute, you can submit your job either by CMD or console. Also, there is a way to submit and manage your query using Alibaba Cloud's official SDK. Once you submit your job for batch computing, your program starts running on multiple VM instances which are launched by a specified image such as Windows Server machine or Ubuntu.
So far, we have explored the basic concepts of the batch computing and how Alibaba Cloud provides these services. But in this article, we will not explore how to create jobs rather we will take a look at how we can customize the underlying infrastructure as needed or required by our software packages.
One of the best features for the user traction is the image customization; an image that launches your virtual machine. As batch compute provide supports for both operating systems; Linux and Window, there could be tons of other settings and specification tweaks which need to be customized. Moreover, from the security standpoint, these virtual machines run in an entirely isolated environment which takes care of data integrity. Since Alibaba Cloud Batch Compute service uses an internal job scheduler, that is the responsibility of the infrastructure to manage the inputs to the jobs and to capture the outputs once a node has specified that it is done processing an input.
Regarding customization, we can modify VM specification and settings (I.e., network and sharing) as per our requirements, for this very fact, Alibaba Cloud provides us the facility to also import local images to launch VM instances. This can mean that we can have our own customized and proprietary image running on the cloud, with all the hardware and compute power rented from the cloud platform.
Alibaba provides three default options as an official batch computer image, which are Windows, Ubuntu, or Centos. Therefore, you must need to specify the batch compute image ID at the time of either creating a cluster or during job submission. In turn, whatever the image ID you will ponder, the virtual machine will be started with that image.
If the above-listed default image options are not meeting your requirements then, then you should create your custom image on which you will be installing the software which you require as per your need. You can always find a bunch of officially supported and uploaded images on the portal,
In case you decide to go with the custom image creation then, firstly, you need to purchase a basic batch compute image and launch ECS instance on it. The values from that ECS instance would be used to create the custom image of yours, as you can see in the following image,
While creating the ECS instance, you can also select the size of the virtual machine resources that you might need. They must be officially supported, and currently you cannot modify the size of the virtual machines.
Secondly, you would be allowed to install whatever the software you need to use on your VM, then you after the creation of ECS image, you will be able to register your Batch Compute image to further go along with it, and at the end, you will be supposed to release the ECS instance.
Now let us discuss the most important features that come with your custom batch computing platform. Alibaba Cloud is the largest cloud services provider in Asia, and among the top cloud vendors across the globe, so the hardware power that it provides is huge. For a custom environment, let us see how that can enable us to achieve more from the vendor's standpoint.
Although you have configured everything as per the best possible options available, you may need to face some down time in the start due to some busy resource occasionally. In that case, VMs must need to be launched with an average time margin before you start your program for the sake of the smooth running and execution of your program. For avoiding this delay and glitch in the performance, you can create a cluster before running jobs. Which drastically improves the operational capabilities; the batch compute will allocate your required resources and they will be in an active state once you have created your cluster, that said, you can freely submit your job anytime without having any kind of fear or busy resources or low performance.
To start your batch computing program, you must need to have a job description written in JSON format describing the program details, the number of required hosts VMs, and other specification and usage details. This is the graphical view that supports the configuration of the worker node,
And here is the JSON editor, that you can use, yourself to write the job script or you can download the script written for the job that you created in the graphical window.
Moreover, you can also configure some other settings such as where log files should be created and placed, etc. Once you have a job description JSON file, you can submit your job to Batch Compute to run on the pre-configured VM with the pre-specified image instance—these settings can be controlled by double-clicking the task icon in the graphical window as shown in the image above.
Job management is centralized and handy; you can use a console based or command line tool to manage the overall activities for your jobs; stop, start, delete job or view all the conditions and operational log files for the respective jobs as well. Also, your jobs are controlled by Batch Compute, so it can easily provide a redeployment policy for your jobs.
For the fast and quality rendering of your video and audio, Alibaba batch computing is highly recommended. It breaks the task, the schedule rendering speed, and required settings and makes your process a lot easier.
Most of the times the rendering of the images is done on several hundreds (if not thousands) of CPU cores that can finish the overall job in a matter of hours with a couple hundred or thousand dollars, if it would take weeks on an on-premises setup with millions of dollars of investment.
The field of Data Science and Machine Learning highly require data munging and then driving out possible outcomes as predictive analysis. For this, data-savvy works on huge data sets and require powerful machines to process the information on them. Therefore, batch computing eventually becomes a viable option for these sorts of actions and operations in the field of business or data analysis.
Alibaba Catch Compute expeditious and massive processing on a huge number of cores to support currency model. Furthermore, it automates the processes, such as data delivery, task scheduling etc. For scaling purposes, we get dynamic resource allocation when needed.
I/O access and transparent handling become possible with the direct ease of OSS mounting for the batch computing. Moreover, at any given time, data caching optimizes the performance and query time quickly. Also, batch compute provides high security by letting you access integration APIs.
At last, you do not need to much concern about cost and resource consumption, as this is a pay-as-you-use model, which means you will be charged only as per your business usage and requirements.
Fault Tolerance with Application High Availability or Batch Compute
2,599 posts | 764 followers
FollowJJ Lim - September 17, 2021
amap_tech - March 16, 2021
Alibaba Cloud Community - March 21, 2022
Alibaba Cloud Native Community - December 29, 2023
Alibaba Cloud Native Community - March 11, 2024
Alibaba Cloud Native Community - November 22, 2023
2,599 posts | 764 followers
Follow
Batch Compute
Resource management and task scheduling for large-scale batch processing
Learn More
ECS(Elastic Compute Service)
Elastic and secure virtual cloud servers to cater all your cloud hosting needs.
Learn MoreLearn More
More Posts by Alibaba Clouder
