Container Service for Kubernetes:Use CNFS to automatically collect the heap dumps of a JVM

Considerations

• Set the Java argument Xmx to a value smaller than the memory limit of your application pod. This prevents the situations where OOM errors occur in the pod but not in the JVM.

• To collect the heap dumps of a JVM, we recommend that you mount a new CNFS volume. This way, your application data and the heap dumps are stored in separate CNFS volumes. This prevents the .hprof file from occupying excessive storage, which may cause a negative impact on your application.

Procedure

1. You can use the registry.cn-hangzhou.aliyuncs.com/acs1/java-oom-test:v1.0 image to deploy a Java program that is used to trigger OOM errors in the JVM.

   For more information about how to build an image, see Use a Container Registry Enterprise Edition instance to build an image.

2. Use the following template to create a Deployment named java-application.

   When you launch the Mycode program, the heap size is set to 80 MB, and the heap dumps are written to the /mnt/oom/logs directory. If the heap cannot meet the requirement of the JVM, a HeapDumpOnOutOfMemoryError error is returned.

   cat << EOF | kubectl apply -f -
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: java-application
   spec:
     selector:
       matchLabels:
         app: java-application
     template:
       metadata:
         labels:
           app: java-application
       spec:
         containers:
         - name: java-application
           image: registry.cn-hangzhou.aliyuncs.com/acs1/java-oom-test:v1.0  # The image address of the sample Java application. 
           imagePullPolicy: Always
           env:                               # Specify two environment variables. Set the key of one variable to POD_NAME and the value to metadata.name. Set the key of the other variable to POD_NAMESPACE and the value to metadata.namespace. 
           - name: POD_NAME
             valueFrom:
               fieldRef:
                 apiVersion: v1
                 fieldPath: metadata.name
           - name: POD_NAMESPACE
             valueFrom:
               fieldRef:
                 apiVersion: v1
                 fieldPath: metadata.namespace
           args:
           - java                            # Run the Java command. 
           - -Xms80m                         # The minimum heap size. 
           - -Xmx80m                         # The maximum heap size. 
           - -XX:HeapDumpPath=/mnt/oom/logs  # The path in which heap dumps are stored when OOM errors occur. 
           - -XX:+HeapDumpOnOutOfMemoryError # Generate heap dumps when OOM errors occur. 
           - Mycode                          # Run the Mycode program. 
           volumeMounts:
           - name: java-oom-pv
             mountPath: "/mnt/oom/logs"      # Mount the CNFS volume to the /mnt/oom/logs directory. 
             subPathExpr: $(POD_NAMESPACE).$(POD_NAME)   # Create a subdirectory named $(POD_NAMESPACE).$(POD_NAME). The subdirectory is used to store heap dumps that are generated due to OOM errors. 
         volumes:
         - name: java-oom-pv
           persistentVolumeClaim:
             claimName: cnfs-nas-pvc         # The persistent volume claim (PVC) that is used to mount the CNFS volume. The PVC name is cnfs-nas-pvc. 
   EOF

3. Go to the Event Center module of the Container Service for Kubernetes (ACK) console. If a Back-off restarting warning event appears on the page, an OOM error has occurred in the java-application application.

   1. Log on to the ACK console.

   2. In the left-side navigation pane of the ACK console, click Clusters.

   3. On the Clusters page, find the cluster that you want to manage and click the name of the cluster or click Details in the Actions column. The details page of the cluster appears.

   4. In the left-side navigation pane of the cluster details page, choose Operations > Event Center.

   

4. To view, upload, and download files in NAS file systems, you can deploy a File Browser application. This allows you to perform these operations on a web page. Mount the NAS file system to the rootDir path of the File Browser application. Then, run the kubectl port-forward command to map the container port of the File Browser application to your on-premises machine. This way, you can use your browser to access files in the NAS file system.

   1. Use the following template to create a ConfigMap that is used by File Browser and the File Browser Deployment. By default, port 80 is opened.

      cat << EOF | kubectl apply -f -
      apiVersion: v1
      data:
        .filebrowser.json: |
          {
            "port": 80
          }
      kind: ConfigMap
      metadata:
        labels:
          app.kubernetes.io/instance: filebrowser
          app.kubernetes.io/name: filebrowser
        name: filebrowser
        namespace: default
      ---
      apiVersion: apps/v1
      kind: Deployment
      metadata:
        labels:
          app.kubernetes.io/instance: filebrowser
          app.kubernetes.io/name: filebrowser
        name: filebrowser
        namespace: default
      spec:
        progressDeadlineSeconds: 600
        replicas: 1
        revisionHistoryLimit: 10
        selector:
          matchLabels:
            app.kubernetes.io/instance: filebrowser
            app.kubernetes.io/name: filebrowser
        template:
          metadata:
            labels:
              app.kubernetes.io/instance: filebrowser
              app.kubernetes.io/name: filebrowser
          spec:
            containers:
            - image: docker.io/filebrowser/filebrowser:v2.18.0
              imagePullPolicy: IfNotPresent
              name: filebrowser
              ports:
              - containerPort: 80
                name: http
                protocol: TCP
              resources: {}
              securityContext: {}
              terminationMessagePath: /dev/termination-log
              terminationMessagePolicy: File
              volumeMounts:
              - mountPath: /.filebrowser.json
                name: config
                subPath: .filebrowser.json
              - mountPath: /db
                name: rootdir
              - mountPath: /rootdir
                name: rootdir
            dnsPolicy: ClusterFirst
            restartPolicy: Always
            schedulerName: default-scheduler
            securityContext: {}
            terminationGracePeriodSeconds: 30
            volumes:
            - configMap:
                defaultMode: 420
                name: filebrowser
              name: config
            - name: rootdir
              persistentVolumeClaim:
                claimName: cnfs-nas-pvc
      EOF

      Expected output:

      configmap/filebrowser unchanged
      deployment.apps/filebrowser configured

   2. Map port 80 of File Browser to your on-premises machine.

      kubectl port-forward deployment/filebrowser 8080:80

      Expected output:

      Forwarding from 127.0.0.1:8080 -> 80
      Forwarding from [::1]:8080 -> 80

   3. Open your browser, enter 127.0.0.1:8080 in the address bar, and then press Enter. The File Browser logon page appears. Enter the default username (admin) and password (admin). Then, click Login.

      

   4. The cnfs-nas-pvc PVC is mounted to the rootDir directory. Double-click rootDir to open the NAS file system.