iTranslated by AI

The content below is an AI-generated translation. This is an experimental feature, and may contain errors. View original article
👻

Setting Up the SGLang Environment on a Supercomputer

に公開

SGLang is a library that enables fast inference with LLMs. I encountered some difficulties when trying to run SGLang on the supercomputer "Genkai" (H100 x 4 units), so I'm leaving this as a memo.

For the Python virtual environment, I chose uv because it allowed the smoothest environment setup. I initially tried using pip on a venv, but dependency resolution was suboptimal, leading to several errors that required manual resolution.

pyproject.toml is as follows:

[project]
name = "deepseek"
version = "0.1.0"
description = "Add your description here"
readme = "README.md"
requires-python = ">=3.11"
dependencies = [
    "sglang[all]>=0.4.8.post1",
]

Below is the job script. It is said that cudnn version 9 series must be used.

#!/bin/bash
#PJM -L rscgrp=b-batch
#PJM -L node=1
#PJM -L elapse=0:05:00
#PJM -j
#PJM -N check_env
#PJM -o logs/check_env.log

module load cuda/12.2.2 cudnn/9.2.0 nccl/2.22.3
uv run python -m sglang.check_env

By the way, I had been struggling for a long time with the issue that cudnn was not visible even when using module avail, but I recently solved it.
It turns out that after loading cuda, the corresponding cudnn and nccl versions for that cuda version become visible. I wish I had known this sooner.

$ module avail
----------------------------------------------------------------------------------- /home/modules/modulefiles/LN/core ------------------------------------------------------------------------------------
cuda/11.8.0           cuda/12.6.1  cuda/12.9.1              gcc-toolset/13  intel/2023.2(default)  intel/2025.1.3        nvidia/24.11  python/3.12.11(default)
cuda/12.2.2(default)  cuda/12.8.1  gcc-toolset/12(default)  gcc/8(default)  intel/2024.1           nvidia/23.9(default)  nvidia/25.5

----------------------------------------------------------------------------------- /home/modules/modulefiles/LN/util ------------------------------------------------------------------------------------
avs/express85(default)         fieldview/2023(default)     julia/1.10.3(default)            mathematica/14.0(default)               molpro/2024.1.0_mpipr    ocicli/3.42.0(default)
aws_pcluster/3.9.1(default)    gaussian/16.C.01(default)   jupyter_notebook/7.2.1(default)  matlab/R2024a(default)                  molpro/2024.1.0_sockets  papi/7.1.0(default)
awscli/2.16.8(default)         gcloudcli/447.0.0(default)  llvm/7.1.0                       matlab_parallel_server/R2024a(default)  nastran/2024.1(default)  singularity-ce/4.1.3(default)
azure_cyclecli/8.6.2(default)  gcloudcli/502.0.0           marc/2024.1(default)             mesa/20.3.5                             node/12.22.12            tensorflow-cpu/2.17.0(default)

---------------------------------------------------------------------------------------- /home/center/modulefiles ----------------------------------------------------------------------------------------
alphafold/2.3.2(default)  alphafold/3.0.1  mvapich/3.0-gcc8.5.0  mvapich/3.0-intel2023.2

----------------------------------------------------------------------------------------- /home/rist/modulefiles -----------------------------------------------------------------------------------------
abinit-mp/v1r22     akaikkr/cpa2021v02.GPU  ffvhc-ace/0.1  genesis/2.1.4.double      genesis/2.1.4.GPU.single  hphi/3.5.2     ntchem/24.10.mpi     phase0/2024.01  salmon/2.2.1.GPU
abinit-mp/v2r8      alamode/1.5.0           ffx/03.01.01   genesis/2.1.4.GPU.double  genesis/2.1.4.mixed       modylas/1.1.0  ntchem/24.10.mpiomp  phonopy/2.27.0  smash/3.0.2
akaikkr/cpa2021v02  ffb/9.0                 frontistr/5.6  genesis/2.1.4.GPU.mixed   genesis/2.1.4.single      mvmc/1.3.0     openmx/3.9.9         salmon/2.
$ module load cuda/12.2.2
$ module avail
--------------------------------------------------------------------------- /home/modules/modulefiles/LN/compiler/cuda/12.2.2 ----------------------------------------------------------------------------
cudnn/8.8.0  cudnn/8.9.7  cudnn/9.1.1           nccl/2.18.5  nccl/2.20.5  nccl/2.22.3(default)       tensorflow-cuda/2.61.1-12.2.2
cudnn/8.9.4  cudnn/9.0.0  cudnn/9.2.0(default)  nccl/2.19.3  nccl/2.21.5  pytorch-cuda/2.3.1-12.2.2

----------------------------------------------------------------------------------- /home/modules/modulefiles/LN/core ------------------------------------------------------------------------------------
cuda/11.8.0           cuda/12.6.1  cuda/12.9.1              gcc-toolset/13  intel/2023.2(default)  intel/2025.1.3        nvidia/24.11  python/3.12.11(default)
cuda/12.2.2(default)  cuda/12.8.1  gcc-toolset/12(default)  gcc/8(default)  intel/2024.1           nvidia/23.9(default)  nvidia/25.5

----------------------------------------------------------------------------------- /home/modules/modulefiles/LN/util ------------------------------------------------------------------------------------
avs/express85(default)         fieldview/2023(default)     julia/1.10.3(default)            mathematica/14.0(default)               molpro/2024.1.0_mpipr    ocicli/3.42.0(default)
aws_pcluster/3.9.1(default)    gaussian/16.C.01(default)   jupyter_notebook/7.2.1(default)  matlab/R2024a(default)                  molpro/2024.1.0_sockets  papi/7.1.0(default)
awscli/2.16.8(default)         gcloudcli/447.0.0(default)  llvm/7.1.0                       matlab_parallel_server/R2024a(default)  nastran/2024.1(default)  singularity-ce/4.1.3(default)
azure_cyclecli/8.6.2(default)  gcloudcli/502.0.0           marc/2024.1(default)             mesa/20.3.5                             node/12.22.12            tensorflow-cpu/2.17.0(default)

---------------------------------------------------------------------------------------- /home/center/modulefiles ----------------------------------------------------------------------------------------
alphafold/2.3.2(default)  alphafold/3.0.1  mvapich/3.0-gcc8.5.0  mvapich/3.0-intel2023.2

----------------------------------------------------------------------------------------- /home/rist/modulefiles -----------------------------------------------------------------------------------------
abinit-mp/v1r22     akaikkr/cpa2021v02.GPU  ffvhc-ace/0.1  genesis/2.1.4.double      genesis/2.1.4.GPU.single  hphi/3.5.2     ntchem/24.10.mpi     phase0/2024.01  salmon/2.2.1.GPU
abinit-mp/v2r8      alamode/1.5.0           ffx/03.01.01   genesis/2.1.4.GPU.double  genesis/2.1.4.mixed       modylas/1.1.0  ntchem/24.10.mpiomp  phonopy/2.27.0  smash/3.0.2
akaikkr/cpa2021v02  ffb/9.0                 frontistr/5.6  genesis/2.1.4.GPU.mixed   genesis/2.1.4.single      mvmc/1.3.0     openmx/3.9.9         salmon/2.2.1

Discussion