Platform For AI:AIMaster: Elastic fault tolerance engine

If you want to receive notifications about fault tolerance events, such as a job restart, you can configure a rule for such notifications by performing the following steps: Go to the Workspace Details page, click the Events tab, and then click Create Event Rule. In the panel that appears, select DLC Jobs > Automatic Fault Tolerance from the Event Type drop-down list. For more information, see Workspace notification.

You can also use AIMaster SDK to configure custom notifications when an error occurs, such as when the loss function returns NaN. Sample code:

from aimaster import job_monitor as jm

job_monitor_client = jm.Monitor(config=jm.PyTorchConfig())

...

if loss == Nan and rank == 0:
  st = job_monitor_client.send_custom_message(content="The loss function returns NaN")
  if not st.ok():
      print('failed to send message, error %s' % st.to_string())

The fault tolerance monitoring feature provides a built-in module that can automatically detect common retriable errors. If you want to trigger fault tolerance when error logs carry information that matches specific keywords, you can use the following methods to configure custom keywords. After you configure custom keywords, the module scans the tail logs of failed instances for key information that matches your custom keywords.

• Sample configuration for PyTorch jobs:

  from aimaster import job_monitor as jm
  
  jm_config_params = {}
  jm_config = jm.PyTorchConfig(**jm_config_params)
  monitor = jm.Monitor(config=jm_config)
  monitor.set_retryable_errors(["connect timeout", "error_yyy", "error_zzz"])

  In the preceding code, the custom keywords are configured by using the monitor.set_retryable_errors function.

• Sample configuration for TensorFlow jobs:

  from aimaster import job_monitor as jm
  
  jm_config_params = {}
  jm_config = jm.TFConfig(**jm_config_params)
  monitor = jm.Monitor(config=jm_config)
  monitor.set_retryable_errors(["connect timeout", "error_yyy", "error_zzz"])

By default, hang detection is configured for the entire job runtime. However, you may need different configurations at different runtime stages. For example, when a large-scale job is initializing, it takes a long time to establish the connections between instances, whereas the execution of the job is faster. DLC allows you to configure the hang detection policy based on the current status of the job by using the following methods:

monitor.reset_config(jm_config_params)

# Example:
#     monitor.reset_config(job_hang_interval=10)
#     or
#     config_params = {"job_hang_interval": 10, }
#     monitor.reset_config(**config_params)

Sample configuration for a PyTorch job:

import torch
import torch.distributed as dist
from aimaster import job_monitor as jm

jm_config_params = {
    "job_hang_interval": 1800 # Configure a global hang detection policy. This value specifies that the maximum duration during which the job can be non-responsive is 30 minutes.
}
jm_config = jm.PyTorchConfig(**jm_config_params)
monitor = jm.Monitor(config=jm_config)

dist.init_process_group('nccl')

...

# impl these two funcs in aimaster sdk
# user just need to add annotations to their func
def reset_hang_detect(hang_seconds):
    jm_config_params = {
        "job_hang_interval": hang_seconds
    }
    monitor.reset_config(**jm_config_params)

def hang_detect(interval):
    reset_hang_detect(interval)
    ...

@hang_detect(180) # reset hang detect to 3 min, only for func scope
def train():
    ...

@hang_detect(-1) # disable hang detect temperally, only for func scope
def test():
    ...

for epoch in range(0, 100):
    train(epoch)
    test(epoch)
    self.scheduler.step()