By Alexandru Andrei, Alibaba Cloud Community Blog author.
Among system administrators, Debian is well known for a lot of its qualities, like stability, high number of software packages included, extremely low number of bugs, ease of use. But this operating system does another thing really well -- it's very easy to upgrade and the process is extremely reliable, as long as the system is in a good working state. It's true that other Linux distributions have this feature as well, some of them adding it in recent years, but Debian did this long before most of the others and perfected the procedure. Other distros usually recommended a clean install of the new release, rather than an upgrade, since upgrades were often unreliable.
Most of this article will consist of preparatory steps like backing up or testing the upgrade on a cloned server. This is because, while it's easy to upgrade the operating system itself, some of the things you run on your Elastic Compute Service (ECS) instance, the way you customized and set them up, the websites that you may host, PHP scripts, and so on, may occasionally encounter issues on the new release. This rarely happens, but it's good to take precautionary measures if the ECS instance serves a very important purpose and you can't afford to have it fail and keep services offline while you fix it.
The most common scenario is when you have third-party software installed (packages that weren't created by Debian developers). If you ever downloaded .deb packages, installed them with dpkg -i or added third-party repositories to your package manager's sources.list file, chances for a failed upgrade increase.
Other problems that can appear are not related to the upgrade process itself, but rather to old applications having trouble on the new release. For example, let's say you host a PHP website, served by Nginx or Apache, on your ECS instance. Your current Debian version uses PHP 7.0 and the website works without a hiccup. You upgrade to a newer release and this includes a newer version of PHP, 8.0. Your PHP application is programmed with version 7.0 in mind. Some of the functions it uses may have been deprecated in 8.0. Thus, after you upgrade the operating system, your website may begin to malfunction in certain areas. This shouldn't usually happen with a well written PHP application because the developers take into account functions that will be deprecated, and other aspects relating to users being able to upgrade their PHP interpreter. But, it can happen with some applications, either because the developers didn't think ahead, or forgot to update something in the code to prepare for the new PHP release.
The bottom line is that if you didn't break Debian in some of the ways described here, the upgrade process should be pain-free.
Let's say you're currently using Debian 9.7. The first number, 9, represents the major release number (or simply, Debian version) and the second number, 7, represents that this is the 7th point release. The next point release will be Debian 9.8. So, in this example, when a new major version of the distribution is launched, it will be Debian 10.0. You should never upgrade to any ".0" version. Always wait for the first point release (e.g., Debian 10.1) to be launched. That's because, while the developers are careful, they can't possibly test for tens of thousands of different scenarios, hardware, system states, etc. Some bugs will still exist. As more and more people upgrade, those hidden/missed bugs are discovered. It's ok to upgrade to, say, Debian 10.0, if you're an enthusiast and want the new stuff, on your desktop, or even on your new ECS instance, as long as you can afford the extra risk. But don't do it if your instance serves an important purpose, or you might hit bugs that impact its service(s) in unexpected ways.
When the first point release is launched, a lot of initial bugs are fixed and you will have a much higher chance for a smoother upgrade path and a more stable system.
You should always periodically backup a server, but even more so before you make important changes, as is the case here. Alibaba makes it very easy to execute this operation, with the help of its snapshot service. Here, you can learn more about how snapshots work. You don't have to worry about data that may change while the backup is being made.
Let's say that you have files 1, 2, 3 on the disk. You start the backup at 12:00. While this carries out, at 12:01, file 4 is added, and file 3 is modified. The backup will finish at 12:02 and will contain files 1, 2, 3 (but not 4). File 3 will have its original contents, not the ones modified at 12:01. When you issue a backup command, the system freezes data in place and then begins to copy it to another location. This ensures consistency and it also allows the server to continue to operate normally -- read, write, modify files, and serve requests to users/customers, while the backup is being made.
Since it works as mentioned in the previous section, you can snapshot an ECS instance whenever you want to. However, note that the read/write performance of the disk(s) will be slightly impacted while this is running. The impact is relatively small, but can be an issue if your server makes heavy use of disk operations, constantly reading and/or writing data. If you can spare about 10-20% of your disk performance, backup at any time. If you can't spare that amount, try to backup when the system is used less, for example, at 04:00 AM in the morning.
Ideally, if your specific situation allows it, you might also temporarily power off your ECS instance and take a snapshot while it's stopped. This guarantees that the backup will be in a perfect state.
To exemplify why it's preferred to power off the instance before a snapshot, imagine this scenario: some process is in the middle of receving a 100MB file, through the network (say, a user uploads something). It writes 50MB and during that time the snapshot operation begins, which means that only those 50MB may be backed up and the rest might be missing if you ever decide to later restore the backup.
Most of the programs that write files on servers, though, are made in such a way that these issues are rare enough. For example, database servers, such as MySQL/MariaDB, write atomically). This ensures that you can never have such states as a "half done job."
Bottom line is, that if you can, power off the machine before you back it up. Otherwise, just backup at any time you can, while it's running.
You have a chance to test if the backup is in a good state, if you follow the optional step in the next section and clone your instance. You can also try to upgrade Debian on the cloned system, which makes it possible to test the whole scenario, without any risk of damaging your original instance.
At a time convenient for you, follow the steps here to create a snapshot of your system disk (and optionally, data disks, if you have them). The relevant steps are included in the section titled "Create a snapshot in the ECS console" (at the time of writing, title may change in the future).
Although this step is optional it is highly recommended, since it makes the upgrade process risk-free!
If the instance you want to upgrade doesn't have to be online all the time, you can skip this step. In case you encounter any problems, you can just roll back the disk (restore backup), and try again, avoiding whatever made the upgrade process fail. However, this means you will have to temporarily power off your instance for a few minutes or hours, depending on how large the disk is.
If you can't afford to potentially have your server offline for that time, follow the steps in this section. In fact, if your infrastructure allows it, you can even do this to take an entirely different route. Instead of upgrading Debian on your main ECS instance, you can clone it, upgrade Debian on the cloned system, test if everything works fine and once you're satisfied you can just insert this clone into your infrastructure, to replace the old ECS instance that runs the older Debian distribution. In most cases, it's just a matter of replacing an old IP address with the new IP address. For example, if this is a web server included in a high-availability setup, you just need to inform load balancers about the change: remove the IP of the original ECS instance and insert the IP of the cloned and upgraded instance.
Follow the steps here to create a custom image from a snapshot. After this is done, in your Alibaba ECS console, navigate in the left menu to Instances & Images -> Instances. Here's an example of how this might look like, in the picture below.
Keep in mind that the interface might change in the future and you may find this elsewhere.
On this page that lists current instances, on the instance row that you want to clone, click on More and then Buy Same Type. You will find these options on the right side of the page, like in this picture:
On the page that opens, scroll down until you can find the text Select all configurations, Click here. Click on that, as you'll need to expand ECS options to change the operating system image that the instance will use.
Next, click on the small icon, highlighted in the next picture, next to Basic Configurations.
Scroll down to the Image section, change to the Custom Image tab and select the custom image you previously created from your snapshot.
Go to the final step and launch your instance. It's important to not change any of the other settings for the ECS instance. For example, if you would give it less RAM, it might not be able to launch the same programs since it may run out of free memory.
If, in the future, these interface elements will be changed, just try to follow the same principle: buy same type of instance, with the same specifications, the only difference being that you use your custom image for the system disk, instead of a public image.
Log in as root, either to your cloned ECS instance, if you created one, or to your original instance, if you didn't clone it.
For a smooth upgrade from one Debian release to another, it's very important that all packages are up to date on the current release:
apt update && apt upgrade
Reboot the system, if you can. You shouldn't encounter any issue if you continue without rebooting though.
systemctl reboot
If you know you never added third party repositories, you can skip the steps in this section.
Clean up your package manager's sources.list file of any software sources outside of Debian's official repositories. On an ECS instance your software will be pulled in from Alibaba mirror servers that are an exact replica of Debian's official servers. Bottom line is, anything that doesn't come from Debian and its mirrors should be removed, both repositories and packages.
Here's an example of how a clean sources.list file looks like on an Alibaba server:
Lines such as deb http://nginx.org/packages/debian should be removed. Only the lines containing strings like deb.debian.org or aliyuncs.com should be kept.
Edit the sources.list file.
apt edit-sources
If you get prompted to choose an editor after this command, choose Nano. Remove any lines that don't point to Alibaba mirrors or Debian servers. Press CTRL+X, then press y, and then ENTER to save the file.
Check to see if you have any repositories added in optional files in the /etc/apt/sources.list.d/ directory.
ls /etc/apt/sources.list.d/
If there are no results, you're done. If you notice any files, edit each one with
nano /etc/apt/sources.list.d/name-of-file.list
where you replace name-of-file.list with the actual name of each file. If these files only refer to third-party repositories, you can delete all of them, with
rm -i /etc/apt/sources.list.d/*
-i command line switch has been added to avoid accidentally removing files. This way, the command will ask you to confirm each one you want to delete by typing "y" and then ENTER. Type "n" if you don't want to delete a file.
Refresh your package manager to pick up on changes made to software sources files:
apt update
Install apt-forktracer.
apt install apt-forktracer
Run it to find any packages that come from outside of Debian's official repositories.
apt-forktracer
Here is some example output:
root@debian-clone:~# apt-forktracer
aliyun-assist (1.0.1.368)
nginx (1.16.0-1~stretch) [Debian Backports: 1.14.1-1~bpo9+1] [Debian: 1.10.3-1+deb9u2 1.10.3-1+deb9u2]In this case, aliyun-assist and nginx "taint" the system. However, aliyun-assist is a utility provided by Alibaba and you should keep it. Same is true with any other utilities that are prefixed by the "aliyun" string. They may provide certain security features, help you control the instance through the web page interface, and so on. Which means, that in this case only nginx should be removed, with a command like apt autoremove nginx.
If you have multiple packages, just list them like this:
apt autoremove package1 package2 package3
If you need these packages, you will have to reinstall them after you upgrade your system. But since the new Debian release will include newer versions of software, hopefully you won't need to install them from third-party sources again and can instead pull them in from Debian's official repositories or mirrors.
The upgrade will take a while and in that time there's a small chance you may get disconnected. If that happens, the process may get interrupted and may result in errors you have to fix manually. To avoid this, install a terminal multiplexer.
apt install tmux
This will allow you to later reconnect to the session, if necessary, and continue where you left off. Processes running in the session don't get interrupted. Begin your new tmux session.
tmux
Now let's say that in the middle of the upgrade, your computer unexpectedly reboots. You would log in as root, again, and then enter the following command to reattach to the running tmux session.
Don't run this now, only use it if you got disconnected from your SSH session.
tmux attach
Make sure the lsb_release utility is installed on your system. Normally, this should be available, by default, in which case the next command will do nothing, just tell you that it's already installed.
apt install lsb-release
Check the version number and codename of your current release.
lsb_release -a
Remember, you can only upgrade your operating system to the next version, Debian 9 to Debian 10, Debian 10 to Debian 11, but you can't upgrade directly from 9 to 11. Also, only upgrade to a stable release, not a development release.
If you don't already know the codename of the next stable release that you want to upgrade to, you can find it on Debian's release page. Scroll down until you find a list like this:
IMPORTANT: In the next command you will have to replace release codenames to whatever currently applies to you. The example command shows how to upgrade from Debian 9, codenamed Stretch, to Debian 10, codenamed Buster. Copy this command to a text editor (e.g., Notepad), replace the strings stretch and buster, if necessary, and only afterwards copy it again, from the text editor and paste it to the terminal. Or better yet, type the command in your SSH session. DO NOT COPY AND PASTE this in your terminal and edit it there. If you do so, you may accidentally paste a newline, which is like pressing ENTER, and will make the command execute before you get a chance to edit it, possibly messing up your sources.list with wrong codename values.
sed -i 's/stretch/buster/g' /etc/apt/sources.list
The first codename should be the name of your current release and the second codename should be the name of the next version of Debian. For example, if you're reading this after the year 2021 and Debian 11, codenamed Bullseye is already released, you may have to change the command to sed -i 's/buster/bullseye/g' /etc/apt/sources.list. Otherwise put, the command is: sed -i 's/codename_of_your_current_release/codename_of_next_release/g' /etc/apt/sources.list
Refresh your package manager.
apt update
Do a safe upgrade first. This upgrades only those packages that can be upgraded without removing anything.
apt upgrade
A changelog may appear, notifying you of important changes in some packages. Scroll through it with arrow keys or PAGE UP/PAGE DOWN. When you want to close it, press q.
Next, you might get a question like this one:
You can select whichever answer you prefer.
The trickier questions will be the ones regarding package configuration files.
Some files have been changed by Alibaba. For example, in the case of the openssh-server package, Alibaba may have modified /etc/ssh/sshd_config to disallow password logins and only allow key-based authentication, for increased security. Installing the new config files may negate these enhancements. So, at least in the case of /etc/ssh/sshd_config and /etc/sysctl.conf, type N to keep the old files around. In /etc/sysctl.conf Alibaba may improve/fix some networking/system settings.
If, in the case of other configuration files you want to install the new, "factory default" version, type Y. For example, in the case of /etc/security/limits.conf, you can type Y.
For some of the other packages, you may get the same question, but in a prettier text user interface (TUI).
Use the UP and DOWN arrow keys and then ENTER to select your desired answer. In the case of chrony.conf, you may keep that file too, though it is not vital that you do so. Alibaba configures its own NTP servers there. If you get Debian's version, you will use other servers for time synchronization.
Pay attention when you're asked about the SSH server config file and select "keep the local version currently installed."
After you're done with all the questions, execute the final upgrade command, which is less conservative and will remove some packages if they are obsolete, replaced by newer packages, or just incompatible/conflict with whatever has to be installed. This is the part where you have to take a good look at the list of software being added/removed and decide if it causes any issues for your specific use-case.
apt full-upgrade
Optionally, remove packages that are no longer necessary. Also check here if it removes anything you might need and cancel the action if you need to.
apt autoremove
If, for example, a package like xml-core is recommended for removal, and you need it, you can run a command like apt install xml-core. Now, run the autoremove command again and it won't show up in that list anymore.
Finally, reboot your system, so that the new kernel, init system and other software gets loaded into memory.
systemctl reboot
If you encounter problems with the upgrade, that you can't seem to solve on your own, search the web for "debian [codename] release notes." For example, "debian buster release notes" will show up a result leading to this page. In the comprehensive release notes you will also find a link titled Upgrades from Debian 9 (stretch). The release notes and upgrade notes should help you solve any issue you may have.
How to Manage Systemd Services and Units on ECS with the Systemctl Command
2,599 posts | 764 followers
Followfrancisndungu - December 12, 2018
Alibaba Clouder - August 28, 2020
Alibaba Clouder - December 19, 2017
Alibaba Clouder - November 15, 2018
Alibaba Clouder - May 18, 2018
Alibaba Clouder - June 19, 2018
2,599 posts | 764 followers
Follow
ECS(Elastic Compute Service)
Elastic and secure virtual cloud servers to cater all your cloud hosting needs.
Learn MoreLearn More
Learn More
More Posts by Alibaba Clouder
246639228145529041 January 10, 2022 at 5:25 am
I have upgrade the Ubuntu 18.04 to 20.04 , inside the terminal, when I check, it is Ubuntu 20.04, but the Alibaba User Interface still showing 18.04 Instead of 20.04