Use Intel oneAPI to compile and run LAMMPS - Elastic High Performance Computing

Elastic High Performance Computing (E-HPC) clusters integrate the Intel oneAPI toolkits. The toolkits can be used together with high-performance computing (HPC) software to quickly construct applications that are used across different architectures. This topic describes how to use oneAPI to compile and run Large-scale Atomic/Molecular Massively Parallel Simulator (LAMMPS) in an E-HPC cluster.

Background information

oneAPI is an open, standard, and unified programming model. The oneAPI toolkits provide a unified toolset to deploy applications and solutions across CPU and field-programmable gate array (FPGA) architectures. The toolkits include optimized compilers, libraries, frameworks, and analysis tools that are used for high-performance heterogeneous computing. The toolkits help developers simplify programming and improve productivity.

LAMMPS is a classical molecular dynamics (MD) program that models ensembles of particles in the liquid, solid, or gaseous state. LAMMPS can concurrently model atoms and molecules with high efficiency. It is widely used in materials science, physics, and chemistry.

If you use oneAPI to compile and run LAMMPS, you can quickly construct applications and improve application performance.

Preparations

Create an E-HPC cluster. For more information, see Create a cluster by using the wizard.

The following parameter configurations are used in this example.

Parameter	Description
Hardware settings	Deploy a standard cluster that consists of two management nodes, one compute node, and one logon node. All nodes must be ecs.c7.large Elastic Compute Service (ECS) instances. Each instance must have 2 vCPUs, 4 GiB of memory, and one 2.7 GHz third-generation Intel Xeon Scalable (Ice Lake) processor.
Software settings	Deploy a CentOS 7.6 public image and the PBS scheduler.

Create a cluster user. For more information, see Create a user.
The user is used to log on to the cluster, compile LAMMPS, and submit jobs. Grant the sudo permissions to the user.
Install the oneAPI toolkits. For more information, see Install software.
Install the following software:
- intel-oneapi-mkl (version: 2022.1.2)
- intel-oneapi-mpi (version: 2022.1.2)
- intel-oneapi-hpckit (version: 2022.1.2)

Step 1: Connect to the cluster

Use one of the following methods to connect to the cluster.

Use an E-HPC client to log on to a cluster
The scheduler of the cluster must be PBS. Make sure that you have downloaded and installed an E-HPC client and deployed the environment required for the client. For more information, see Deploy an environment for an E-HPC client.
1. Start and log on to your E-HPC client.
2. In the left-side navigation pane, click Session Management.
3. In the upper-right corner of the Session Management page, click terminal to open the Terminal window.
Use the E-HPC console to log on to a cluster
1. Log on to the E-HPC console.
2. In the upper-left corner of the top navigation bar, select a region.
3. In the left-side navigation pane, click Cluster.
4. On the Cluster page, find the cluster and click Connect.
5. In the Connect panel, enter a username and a password, and click Connect via SSH.

Step 2: Compile LAMMPS

Download the latest LAMMPS source code.
1. Run the following command to download the LAMMPS source code from GitHub:
  Note
  If git is not installed in the cluster, run the sudo yum install -y git to install git.
```
git clone -b release https://github.com/lammps/lammps.git mylammps
```
2. Run the following command to view the file of the LAMMPS source code:
```
ls -al
```
  The following command output is returned:
```
...
drwxr-xr-x 15 test users 4096 May 31 16:39 mylammps
...
```

Load the oneAPI module.

Run the following command to write the environment variables to the $HOME/.bashrc file:

vim $HOME/.bashrc

Add the following information:

source /opt/intel-oneapi-mpi/oneapi/setvars.sh --force 
source /opt/intel-oneapi-mkl/oneapi/setvars.sh --force 
source /opt/intel-hpckit/oneapi/setvars.sh --force

Run the following command to update the $HOME/.bashrc file:
```
source $HOME/.bashrc
```

Compile LAMMPS.
1. Run the following command to use two processes for compilation:
```
cd $HOME/mylammps/src
make -j 2 intel_cpu_intelmpi
```
2. Run the following command to view the generated LAMMPS executable file in the current file path:
```
ll lmp_intel_cpu_intelmpi
```
  A response similar to the following command output is returned:
```
-rwxr-xr-x 1 test users 9041824 May 31 16:48 lmp_intel_cpu_intelmpi
```

Configure the LAMMPS executable file as a shared command:

mkdir -p $HOME/bin
mv $HOME/mylammps/src/lmp_intel_cpu_intelmpi  $HOME/bin

Step 3: Run LAMMPS

Run the following command to go to the bin directory where the lmp_intel_cpu_intelmpi file is located:
```
cd $HOME/bin
```

Run the following command to create an example file named in.intel.lj:

vim in.intel.lj

Example:

# 3d Lennard-Jones melt

variable        x index 1
variable        y index 1
variable        z index 1

variable        xx equal 20*$x
variable        yy equal 20*$y
variable        zz equal 20*$z

units           lj
atom_style      atomic

lattice         fcc 0.8442
region          box block 0 ${xx} 0 ${yy} 0 ${zz}
create_box      1 box
create_atoms    1 box
mass            1 1.0

velocity        all create 1.44 87287 loop geom

pair_style      lj/cut 2.5
pair_coeff      1 1 1.0 1.0 2.5

neighbor        0.3 bin
neigh_modify    delay 0 every 20 check no

fix             1 all nve
dump 1 all xyz 100 sample.xyz
run             10000

Run the following command to write a test script named test.pbs:

vim test.pbs

Test script:

#!/bin/bash
# PBS -N testLmp             # Enter the name of the job.
# PBS -l nodes=2:ppn=2       # Apply for two compute nodes from the scheduler. Each compute node uses two processes to run the job. 
export I_MPI_HYDRA_BOOTSTRAP=ssh
cd $PBS_O_WORKDIR 
mpirun ./lmp_intel_cpu_intelmpi -in ./in.intel.lj

Run the following command to submit the job:
```
qsub test.pbs
```
The following command output is returned, which indicates that the generated job ID is 0.scheduler:
```
0.scheduler
```

Step 4: View the job results

Run the following command to view the job status:

qstat -x 0.scheduler

The following command output is returned. If the value of S is F, the job is completed.

Job id            Name             User              Time Use S Queue
----------------  ---------------- ----------------  -------- - -----
0.scheduler       test.pbs         test              00:00:00 F workq

Run the following command to query the logs of the job:

cat log.lammps

A response similar to the following command output is returned.

...
Per MPI rank memory allocation (min/avg/max) = 11.75 | 11.75 | 11.75 Mbytes
   Step          Temp          E_pair         E_mol          TotEng         Press     
         0   1.44          -6.7733681      0             -4.6134356     -5.0197073    
     10000   0.69579461    -5.6648333      0             -4.621174       0.7601771    
Loop time of 108.622 on 4 procs for 10000 steps with 32000 atoms

Performance: 39770.920 tau/day, 92.062 timesteps/s
97.0% CPU use with 2 MPI tasks x 2 OpenMP threads

MPI task timing breakdown:
Section |  min time  |  avg time  |  max time  |%varavg| %total
---------------------------------------------------------------
Pair    | 85.42      | 85.632     | 85.844     |   2.3 | 78.83
Neigh   | 13.523     | 13.564     | 13.604     |   1.1 | 12.49
Comm    | 4.4182     | 4.5452     | 4.6722     |   6.0 |  4.18
Output  | 2.1572     | 2.1683     | 2.1793     |   0.7 |  2.00
Modify  | 2.1047     | 2.1398     | 2.175      |   2.4 |  1.97
Other   |            | 0.5734     |            |       |  0.53

Nlocal:          16000 ave       16007 max       15993 min
Histogram: 1 0 0 0 0 0 0 0 0 1
Nghost:          13030 ave       13047 max       13013 min
Histogram: 1 0 0 0 0 0 0 0 0 1
Neighs:         600054 ave      604542 max      595567 min
Histogram: 1 0 0 0 0 0 0 0 0 1

Total # of neighbors = 1200109
Ave neighs/atom = 37.503406
Neighbor list builds = 500
Dangerous builds not checked
Total wall time: 0:01:48