Job Training Examples

Introduction and Architecture

The YanFlow cluster job system aims to provide flexible cluster resource scheduling capabilities, supporting various job scheduling modes such as single-machine single-GPU, single-machine multi-GPU, and multi-machine multi-GPU. In the platform, a job is a user-defined process with a lifecycle, characterized by automatic process termination and resource release upon job completion.

The job system includes Single-Machine Jobs, Multi-Machine Jobs, and Slurm Jobs.

• Single-Machine Jobs: Run on a single compute node in the cluster, scheduled using Shell commands.
• Multi-Machine Jobs: Divided into Kubernetes Multi-Machine Jobs and Determined-Scheduled Multi-Machine Jobs. Can utilize frameworks such as PyTorch DDP, Accelerate, DeepSpeed, Horovod, etc., as distributed underlying schedulers to execute deep learning AI training tasks.
• Slurm Jobs: A special job mode provided for users familiar with Slurm clusters.

The overall scheduling logic of jobs is shown in the figure:

Workflow and Advantages

Workflow

Users only need to prepare job code projects and datasets (which can be uploaded via the file management system) and select an appropriate distributed job image.

• Start Container: The platform will start the corresponding job container on the host machine.
• Data Mounting: Datasets are mapped into the job container through mounting.
• Resource Reclamation: After job completion, containers are automatically reclaimed and resources are automatically released.
• Result Viewing: Users can download model weights and view training logs on the web page.

Platform Advantages

Using the platform's job system provides a flexible resource scheduling method, achieving dynamic allocation and release of related resources, and adapting to different user groups.

Main platform features include:

• Simplified Environment Configuration: Integrates state-of-the-art multi-machine multi-GPU training technology, provides a large number of distributed job /images/zh/help, reducing the time users spend configuring deep learning environments.
• Template Projects: Provides job template projects. Users only need to implement corresponding functions without configuring local_rank, rank, communication libraries, and other distributed framework settings to perform multi-machine jobs.
• Platform Compatibility: Supports multiple deep learning platforms such as PyTorch and Tensorflow, with built-in MMDetection object detection framework.
• Improved R&D Efficiency: Provides advanced functions such as training process visualization (TensorBoard), log management, weight management, hyperparameter search, etc.
• Slurm Adaptation: Provides Slurm-based resource scheduling, based on hierarchical management and plugin design, capable of flexibly adapting to different computing requirements and scheduling strategies, suitable for HPC environments.

Detailed Type Descriptions

The structure of single-machine and multi-machine jobs in YanFlow cluster is shown in the figure:

Single-Machine Jobs

Single-machine jobs are provided by the Determined framework.

• Basic Users: For users unfamiliar with the Determined framework, simply input entrypoints (commands to be executed) and select appropriate resources (data mounting, /images/zh/help, GPUs, etc.).
• Advanced Users: For users familiar with Determined, custom code can be written by implementing corresponding inherited classes, then run the following command:

  python3 -m determined.launch.torch_distributed --trial model_def:MyTrial

Multi-Machine Jobs

Multi-machine jobs are divided into Kubernetes-based Multi-Machine Jobs and Determined-based Multi-Machine Jobs.

• Scheduling Principle: Multi-machine jobs use entire machines as scheduling units, meaning no other types of resource occupation (containers or jobs) can exist on nodes running jobs.
• Kubernetes Multi-Machine Jobs: Users need to understand at least one multi-machine multi-GPU startup method, such as torchrun, deepspeed, accelerate, etc.
• Determined Multi-Machine Jobs: Please refer to the relevant documentation provided by the platform.

Slurm Jobs

Slurm jobs are based on the open-source Slurm cluster resource management and job scheduling system. Users need to have some understanding of Slurm.

Quick Start

Job Type	Target Audience/Features	Documentation Reference
Single Node Job	Simple and easy to use, recommended for beginners.	Here
Kubernetes Multi Node Job	Suitable for users familiar with distributed startup commands (such as torchrun).	Here
Determined Multi Node Job	Advanced encapsulation of Kubernetes jobs, providing hyperparameter search, checkpoint training, parameter visualization, and other advanced functions, requires modifying code to inherit base classes, has a certain learning curve.	Here
Slurm Job	Suitable for users familiar with the Slurm cluster resource scheduling system.	Here