Elastic GPU Service:Manually install a Tesla driver on a GPU-accelerated compute-optimized Linux instance

(Optional) Enable the persistence mode (Persistence-M) by using the NVIDIA Persistence Daemon.

After the Tesla driver is installed, Persistence-M is in the disabled (off) state by default. A Tesla driver can achieve more stable performance when Persistence-M is enabled. To ensure business continuity, we recommend that you enable Persistence-M by using the NVIDIA Persistence Daemon. For more information, see Persistence Daemon on the official NVIDIA website.

1. Run the following command to run the NVIDIA Persistence Daemon:

   sudo nvidia-persistenced --user username 
   # Replace username with your username.

2. Run the following command to view the status of Persistence-M:

   nvidia-smi

   If the following command output is displayed, Persistence-M is in the enabled (On) state.

   

(Optional) Enable Persistence-M after you restart the system.

If you restart the system, the enabled (on) state of Persistence-M becomes invalid. You can perform the following operations to enable Persistence-M:

Install the installation scripts provided by NVIDIA, such as the sample script and the installer script, to the /usr/share/doc/NVIDIA_GLX-1.0/samples/nvidia-persistenced-init.tar.bz2 path by installing the Tesla driver installation package.

1. Run the following commands to decompress and install the installation script provided by NVIDIA:

   cd  /usr/share/doc/NVIDIA_GLX-1.0/samples/
   sudo tar xf nvidia-persistenced-init.tar.bz2
   cd  nvidia-persistenced-init
   sudo sh install.sh

2. Run the following command to check whether the NVIDIA Persistence Daemon runs as expected:

   sudo systemctl status nvidia-persistenced

   If the following command output is displayed, the NVIDIA Persistence Daemon runs as expected.

   

   Note

   You can adapt the NVIDIA Persistence Daemon installation script based on your operating system to ensure that the NVIDIA Persistence Daemon works as expected.

3. Run the following command to verify that Persistence-M is in the enabled (on) state:

   nvidia-smi

4. (Optional) Run the following commands to disable the NVIDIA Persistence Daemon.

   You can disable the NVIDIA Persistence Daemon based on your business requirements.

   sudo systemctl stop nvidia-persistenced
   sudo systemctl disable nvidia-persistenced

(Optional) Enable the persistence mode (Persistence-M) by using the NVIDIA Persistence Daemon.

After the Tesla driver is installed, Persistence-M is in the disabled (off) state by default. A Tesla driver can achieve more stable performance when Persistence-M is enabled. To ensure business continuity, we recommend that you enable Persistence-M by using the NVIDIA Persistence Daemon. For more information, see Persistence Daemon on the official NVIDIA website.

1. Run the following command to run the NVIDIA Persistence Daemon:

   sudo nvidia-persistenced --user username 
   # Replace username with your username.

2. Run the following command to view the status of Persistence-M:

   nvidia-smi

   If the following command output is displayed, Persistence-M is in the enabled (On) state.

   

(Optional) Enable Persistence-M after you restart the system.

If you restart the system, the enabled (on) state of Persistence-M becomes invalid. You can perform the following operations to enable Persistence-M:

Install the installation scripts provided by NVIDIA, such as the sample script and the installer script, to the /usr/share/doc/NVIDIA_GLX-1.0/samples/nvidia-persistenced-init.tar.bz2 path by installing the Tesla driver installation package.

1. Run the following commands to decompress and install the installation script provided by NVIDIA:

   cd  /usr/share/doc/NVIDIA_GLX-1.0/samples/
   sudo tar xf nvidia-persistenced-init.tar.bz2
   cd  nvidia-persistenced-init
   sudo sh install.sh

2. Run the following command to check whether the NVIDIA Persistence Daemon runs as expected:

   sudo systemctl status nvidia-persistenced

   If the following command output is displayed, the NVIDIA Persistence Daemon runs as expected.

   

   Note

   You can adapt the NVIDIA Persistence Daemon installation script based on your operating system to ensure that the NVIDIA Persistence Daemon works as expected.

3. Run the following command to verify that Persistence-M is in the enabled (on) state:

   nvidia-smi

4. (Optional) Run the following commands to disable the NVIDIA Persistence Daemon.

   You can disable the NVIDIA Persistence Daemon based on your business requirements.

   sudo systemctl stop nvidia-persistenced
   sudo systemctl disable nvidia-persistenced

(Conditionally required) Install NVIDIA Fabric Manager that matches the driver version. This operation is required when your GPU-accelerated instance belongs to the ebmgn7 or ebmgn7e instance family.

1. Install NVIDIA Fabric Manager.

   You can install NVIDIA Fabric Manager by using the source code or the installation package. The commands that are required to install NVIDIA Fabric Manager vary based on the operating system. In the following examples, the driver versions are 460.91.03 and 535.154.05, and Ubuntu 16.04, Ubuntu 18.04, Ubuntu 20.04, and Ubuntu 22.04 are used. Replace driver_version with the version of the driver that you downloaded in Step 1: Download a Tesla driver.

   Important

   When you install NVIDIA Fabric Manager on Ubuntu 22.04, the version of the Tesla driver must be later than 515.48.07. In the following sample commands for Ubuntu 22.04, the driver version is 535.154.05.

   • Source code

     Ubuntu 16.04, Ubuntu 18.04, or Ubuntu 20.04

     driver_version=460.91.03
     driver_version_main=$(echo $driver_version | awk -F '.' '{print $1}')
     distribution=$(. /etc/os-release;echo $ID$VERSION_ID | sed -e 's/\.//g')
     sudo wget https://developer.download.nvidia.com/compute/cuda/repos/$distribution/x86_64/cuda-$distribution.pin
     sudo mv cuda-$distribution.pin /etc/apt/preferences.d/cuda-repository-pin-600
     sudo wget https://developer.download.nvidia.com/compute/cuda/repos/$distribution/x86_64/3bf863cc.pub
     sudo apt-key add 3bf863cc.pub
     sudo rm 3bf863cc.pub
     sudo echo "deb https://developer.download.nvidia.com/compute/cuda/repos/$distribution/x86_64 /" | tee /etc/apt/sources.list.d/cuda.list
     sudo apt-get update
     sudo apt-get -y install nvidia-fabricmanager-${driver_version_main}=${driver_version}-*

     Ubuntu 22.04

     driver_version=535.154.05
     driver_version_main=$(echo $driver_version | awk -F '.' '{print $1}')
     distribution=$(. /etc/os-release;echo $ID$VERSION_ID | sed -e 's/\.//g')
     sudo wget https://developer.download.nvidia.com/compute/cuda/repos/$distribution/x86_64/cuda-$distribution.pin
     sudo mv cuda-$distribution.pin /etc/apt/preferences.d/cuda-repository-pin-600
     sudo wget https://developer.download.nvidia.com/compute/cuda/repos/$distribution/x86_64/3bf863cc.pub
     sudo apt-key add 3bf863cc.pub
     sudo rm 3bf863cc.pub
     sudo echo "deb https://developer.download.nvidia.com/compute/cuda/repos/$distribution/x86_64 /" | tee /etc/apt/sources.list.d/cuda.list
     sudo apt-get update
     sudo apt-get -y install nvidia-fabricmanager-${driver_version_main}=${driver_version}-*

   • Installation package

     • Ubuntu 16.04

       driver_version=460.91.03
       driver_version_main=$(echo $driver_version | awk -F '.' '{print $1}')
       sudo wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/nvidia-fabricmanager-${driver_version_main}_${driver_version}-1_amd64.deb
       sudo dpkg -i nvidia-fabricmanager-${driver_version_main}_${driver_version}-1_amd64.deb

     • Ubuntu 18.04

       driver_version=460.91.03
       driver_version_main=$(echo $driver_version | awk -F '.' '{print $1}')
       sudo wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/nvidia-fabricmanager-${driver_version_main}_${driver_version}-1_amd64.deb
       sudo dpkg -i nvidia-fabricmanager-${driver_version_main}_${driver_version}-1_amd64.deb

     • Ubuntu 20.04

       driver_version=460.91.03
       driver_version_main=$(echo $driver_version | awk -F '.' '{print $1}')
       sudo wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/nvidia-fabricmanager-${driver_version_main}_${driver_version}-1_amd64.deb
       sudo dpkg -i nvidia-fabricmanager-${driver_version_main}_${driver_version}-1_amd64.deb

     • Ubuntu 22.04

       driver_version=535.154.05 
       driver_version_main=$(echo $driver_version | awk -F '.' '{print $1}')
       sudo wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/nvidia-fabricmanager-${driver_version_main}_${driver_version}-1_amd64.deb
       sudo dpkg -i nvidia-fabricmanager-${driver_version_main}_${driver_version}-1_amd64.deb

2. Run the following commands to start NVIDIA Fabric Manager:

   sudo systemctl enable nvidia-fabricmanager
   sudo systemctl start nvidia-fabricmanager

3. Run the following command to check whether NVIDIA Fabric Manager is installed:

   systemctl status nvidia-fabricmanager

   If the following command output is displayed, NVIDIA Fabric Manager is installed.

   

   Note

   The GPU can work as expected only if the version of NVIDIA Fabric Manager is consistent with the Tesla driver version. For GPU-accelerated compute-optimized instances that run Ubuntu, the apt-daily service may automatically update NVIDIA Fabric Manager if you install NVIDIA Fabric Manager by using an installation package. This results in version inconsistency between NVIDIA Fabric Manager and the Tesla driver. As a result, NVIDIA Fabric Manager fails to start and the GPU fails to work as expected. For information about how to resolve this issue, see What do I do if the GPU fails to work because the nvidia-fabricmanager version is inconsistent with the Tesla driver version?