ApiTestProbeTmp: Difference between revisions

From HPCWIKI
Jump to navigation Jump to search
(A real-content existing-page)
(neutralize probe (admin-only delete))
Tag: New redirect
 
Line 1: Line 1:
== DLS Modules ==
#REDIRECT [[Main Page]]
 
{| class="wikitable"
|+
!Version
!Change log
!Applicable
|-
|1.0
|Initial version
|DLS systems (Rocky 9 DLS2 generation)
|}
 
HPCMATE DLS (Deep Learning Station) systems are built from a set of installable modules. Each module provisions one software layer of the deep learning stack, so a complete, validated development environment can be stood up in a single automated pass. This page documents the DLS modules, the two DLS product configurations they form, and where the DLS module family ends and the UCM (cluster) family begins.
 
'''The DLS module family is separate from UCM (Universal Cluster Management).''' UCM manages multi-node HPC clusters and has its own module set. See [[Manual:UCM]] for UCM documentation.
 
== Purpose ==
 
DLS modules let HPCMATE deliver a consistent, pre-validated GPU development environment without per-package manual configuration. Each module handles one component; the setup configuration selects which modules apply to a given node.
 
* What: A set of installable software components that compose the DLS deep learning stack
* Why: A consistent, validated, upgradable GPU/DL environment with controlled dependency ordering
* When: Applied during factory setup, and re-applied for upgrades or module changes
 
== Key Concepts ==
 
{| class="wikitable"
! Concept
! Description
! Related
|-
| Module
| One installable software component with a defined configuration surface
| Module Reference
|-
| DLS2 (Rocky 9)
| Current DLS platform: Rocky 9, NVIDIA Blackwell RTX PRO 5000, [[CUDA]] 13, [[Docker]]
| DLS2 Configuration
|-
| Full DLS
| Extended DLS set that adds the interactive ML development stack
| Full DLS Configuration
|-
| UCM module family
| The separate multi-node cluster management set (UCM, GlusterFS, NIS, PXE, ...)
| [[Manual:UCM]]
|}
 
== Module Reference ==
 
The union of the two DLS configurations defines 26 distinct modules. The table marks which configuration each belongs to: '''DLS2''' (Rocky 9), '''Full''' (extended), or both.
 
{| class="wikitable"
|+ DLS module reference (26 modules)
! Module
! Role
! What it does
! DLS2
! Full
|-
| updatesystem
| Base
| HPCMATE package updates, sysctl tuning, service [[optimization]]
| Yes
| Yes
|-
| cleanup
| Base
| Removes unneeded packages and tidies first-boot state
| Yes
| Yes
|-
| validation
| Base
| Pre-ship hardware and software validation suite
| Yes
| Yes
|-
| hmsplash
| Base
| HPCMATE login splash screen
| Yes
| Yes
|-
| hmmotd
| Base
| HPCMATE message-of-the-day banner
| Yes
| Yes
|-
| hostnames
| Base
| Sets node hostnames from the configuration
| Yes
| Yes
|-
| ipmi
| Base
| Configures [[IPMI]] / BMC management user
| Yes
| Yes
|-
| zfs
| Storage
| ZFS file system for data volumes
| Yes
| Yes
|-
| envmodule
| Environment
| Environment Modules for managing toolkit and [[library]] versions
| Yes
| Yes
|-
| [[nouveau]]
| GPU
| Blacklists the nouveau driver (required before the [[NVIDIA driver]] installs)
| Yes
| Yes
|-
| nvidiadriver
| GPU
| Installs the NVIDIA proprietary driver with [[DKMS]]
| Yes
| Yes
|-
| cuda_cudnn
| GPU
| Installs the CUDA toolkit and [[cuDNN]] library
| Yes
| Yes
|-
| dockerenv
| Container
| Installs Docker and the NVIDIA Container Toolkit ( --gpus [[support]] )
| Yes
| Yes
|-
| registrycompose
| Container
| Private Docker image registry plus docker-compose
| Yes
| Yes
|-
| nicbond
| [[Network]]
| NIC bonding for redundancy or aggregated throughput
| Yes
|
|-
| [[samba]]
| Storage
| Samba share exposing /home for network access
| Yes
|
|-
| system_optimization
| Base
| [[Kernel]], CPU governor, and I/O scheduler tuning
| Yes
|
|-
| gcc
| Compiler
| Multiple GCC / G++ versions
|
| Yes
|-
| openmpi
| HPC
| OpenMPI for multi-node communication
|
| Yes
|-
| nccl
| GPU
| NVIDIA Collective Communications Library
|
| Yes
|-
| tensorrt
| GPU
| NVIDIA [[TensorRT]] inference optimization
|
| Yes
|-
| anaconda
| Python
| Installs the Miniforge base distribution
|
| Yes
|-
| condaenv
| Python
| Creates and configures conda virtual environments
|
| Yes
|-
| pythonlib
| Python
| System-level Python scientific libraries
|
| Yes
|-
| jupyterhub
| Dev
| JupyterHub multi-user web notebook server
|
| Yes
|-
| displaymanager
| Display
| GPU display mode management (headless vs display)
|
| Yes
|}
 
== DLS2 Configuration ==
 
The DLS2 platform (Rocky 9.8, NVIDIA RTX PRO 5000 Blackwell 48GB) ships with 17 modules. This is a lean GPU development stack plus the container runtime and core system services.
 
{| class="wikitable"
|+ DLS2 default set (17 modules)
! #
! Module
|-
| 1
| updatesystem
|-
| 2
| cleanup
|-
| 3
| validation
|-
| 4
| hmsplash
|-
| 5
| hmmotd
|-
| 6
| hostnames
|-
| 7
| ipmi
|-
| 8
| zfs
|-
| 9
| envmodule
|-
| 10
| nouveau
|-
| 11
| nvidiadriver
|-
| 12
| cuda_cudnn
|-
| 13
| dockerenv
|-
| 14
| registrycompose
|-
| 15
| nicbond
|-
| 16
| samba
|-
| 17
| system_optimization
|}
 
Modules not in the DLS2 set: gcc, openmpi, nccl, tensorrt, anaconda, condaenv, pythonlib, jupyterhub, displaymanager. These can be enabled in the setup configuration when the customer needs the fuller development environment.
 
== Full DLS Configuration ==
 
The Full DLS set (23 modules) is the extended deep learning configuration. Relative to DLS2 it drops nicbond, samba and system_optimization and adds the interactive ML development stack: multiple compilers, MPI, [[NCCL]], TensorRT, the conda base distribution, conda environments, Python libraries, JupyterHub, and GPU display management.
 
{| class="wikitable"
|+ Full DLS set (23 modules)
! #
! Module
|-
| 1
| updatesystem
|-
| 2
| cleanup
|-
| 3
| validation
|-
| 4
| hmsplash
|-
| 5
| hmmotd
|-
| 6
| hostnames
|-
| 7
| ipmi
|-
| 8
| zfs
|-
| 9
| envmodule
|-
| 10
| nouveau
|-
| 11
| nvidiadriver
|-
| 12
| cuda_cudnn
|-
| 13
| dockerenv
|-
| 14
| registrycompose
|-
| 15
| gcc
|-
| 16
| openmpi
|-
| 17
| nccl
|-
| 18
| tensorrt
|-
| 19
| anaconda
|-
| 20
| condaenv
|-
| 21
| pythonlib
|-
| 22
| jupyterhub
|-
| 23
| displaymanager
|}
 
== DLS vs UCM Modules ==
 
The HPCMATE setup system defines two independent module families. They are configured separately and do not depend on each other.
 
{| class="wikitable"
|+ DLS vs UCM module families
! Family
! Scope
! Example modules
! Documentation
|-
| DLS
| Single-node deep learning workstation
| nvidiadriver, cuda_cudnn, dockerenv, jupyterhub, anaconda
| This page
|-
| UCM
| Multi-node HPC cluster management
| UCM, UCMmongo, ucmglusterfs, PXE, NIS, noderegistry
| [[Manual:UCM]]
|}
 
Nine modules are shared by both families because a cluster node and a DLS node both need the same base and GPU layers:
 
{| class="wikitable"
|+ Modules shared by DLS and UCM
! Module
! Shared role
|-
| updatesystem
| Base updates and tuning
|-
| cleanup
| First-boot cleanup
|-
| hmsplash
| Login splash
|-
| envmodule
| Environment Modules
|-
| nouveau
| Disable nouveau
|-
| nvidiadriver
| NVIDIA driver
|-
| dockerenv
| Docker and GPU runtime
|-
| zfs
| ZFS storage
|-
| cuxfan
| Fan control
|}
 
A note on the GPU modules: the shared set is the union of each family. In practice a UCM '''master''' node typically does not carry the NVIDIA driver, while UCM '''worker''' nodes do — the GPU is present where the compute is.
 
UCM modules that are not part of DLS handle the cluster-specific services: the UCM daemon and its database (UCM, UCMmongo), shared storage (ucmglusterfs), authentication (NIS, ucmldap), network boot (PXE), node registration (noderegistry), time and naming (ntpmaster, ntpclient, DNS, CUXIP), logging (rsyslog), and access control (ucmautofs, hostnginx). These do not apply to standalone DLS systems.
 
== Configuration ==
 
Modules are selected in the HPCMATE setup YAML configuration under the modules section. A module listed with no sub-keys installs with its defaults.
 
<syntaxhighlight lang="yaml">
modules:
    updatesystem :
    nvidiadriver :
    cuda_cudnn :
    dockerenv :
    # A module with options:
    gcc:
        install : True
</syntaxhighlight>
 
The setup system resolves dependency ordering automatically (for example nouveau runs before nvidiadriver). To add a module to a running system, add it to the configuration and re-run the setup.
 
== Verification ==
 
After setup, confirm the installed modules are functioning:
 
<syntaxhighlight lang="bash">
# NVIDIA driver
$ nvidia-smi
 
# CUDA toolkit
$ nvcc --version
 
# Docker with GPU passthrough
$ docker run --rm --gpus all nvidia/cuda:12.2.0-base-ubuntu22.04 nvidia-smi
 
# Environment modules
$ module avail
 
# Samba share (DLS2)
$ smbclient -L localhost -U%
</syntaxhighlight>
 
The validation module runs a pre-ship validation suite over the enabled modules. See [[Manual:DLS validation]] for the full process.
 
== Best Practices ==
 
* Start from the DLS2 set (17 modules) for Blackwell / Rocky 9 systems
* Add the ML development modules (anaconda, condaenv, jupyterhub) only when the customer needs interactive development
* Let the setup system order nouveau before nvidiadriver; do not hand-sequence the two
* After a driver or CUDA upgrade, re-run nvidiadriver and cuda_cudnn to refresh the installed packages
 
== Limitations ==
 
* Module installation is one-way; there is no automatic uninstall. Removing a module means manually removing its packages.
* The validation module reports findings but does not fix them; remediation needs an operator.
* Module versions track the HPCMATE release, so moving to a newer release can change the versions of the underlying packages.
 
== References ==
 
* [[Manual:DLSSystem]] — DLS framework overview and usage guide
* [[Manual:UCM]] — UCM cluster management (separate module family)
* [[Manual:DLS validation]] — Pre-ship validation process
 
== Related Pages ==
 
* [[Manual:DLSSystem]]
* [[Manual:UCM]]
* [[Manual:DLS validation]]
 
[[Category:DLS]]
[[Category:Reference]]

Latest revision as of 15:36, 23 September 2026

Redirect to:

Retrieved from "http://wiki.hpcmate.com/index.php?title=ApiTestProbeTmp&oldid=3647"