Ansible AI Supercomputing — Deploy GPU Clusters and HPC

Introduction

AI supercomputing platforms — purpose-built GPU clusters for training foundation models — are a defining 2026 infrastructure trend. Deloitte and Gartner both highlight the explosion in GPU cluster deployments. Ansible automates the full stack: bare-metal GPU provisioning, SLURM job scheduling, InfiniBand fabric configuration, distributed training orchestration, and high-performance parallel storage.

AI Supercomputing Stack

Layer	Component	Purpose
Compute	NVIDIA H100/H200, AMD MI300X	GPU acceleration
Scheduler	SLURM, PBS Pro	Job management
Network	InfiniBand NDR, RoCE	GPU-to-GPU communication
Storage	Lustre, GPFS, BeeGFS	Parallel I/O
Software	CUDA, NCCL, PyTorch	Training frameworks
Monitoring	DCGM, Prometheus, Grafana	GPU health + utilization

Provision GPU Compute Nodes

---
- name: Provision GPU compute nodes
  hosts: gpu_nodes
  become: true
  vars:
    cuda_version: "12.4"
    nvidia_driver: "550"

  tasks:
    - name: Install NVIDIA driver
      ansible.builtin.package:
        name:
          - "nvidia-driver-{{ nvidia_driver }}"
          - "nvidia-driver-{{ nvidia_driver }}-server"
          - nvidia-utils-{{ nvidia_driver }}
        state: present
      notify: Reboot for driver

    - name: Install CUDA toolkit
      ansible.builtin.apt:
        deb: "https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb"
      when: ansible_os_family == "Debian"

    - name: Install CUDA packages
      ansible.builtin.package:
        name:
          - "cuda-toolkit-{{ cuda_version | replace('.', '-') }}"
          - libcudnn9-cuda-12
          - libnccl2
          - libnccl-dev
        state: present

    - name: Set CUDA environment
      ansible.builtin.copy:
        content: |
          export PATH=/usr/local/cuda-{{ cuda_version }}/bin:$PATH
          export LD_LIBRARY_PATH=/usr/local/cuda-{{ cuda_version }}/lib64:$LD_LIBRARY_PATH
          export NCCL_DEBUG=INFO
        dest: /etc/profile.d/cuda.sh
        mode: "0644"

    - name: Configure GPU persistence mode
      ansible.builtin.command: nvidia-smi -pm 1
      changed_when: true

    - name: Set GPU clock speeds for training
      ansible.builtin.command: >
        nvidia-smi -lgc {{ gpu_min_clock }},{{ gpu_max_clock }}
      vars:
        gpu_min_clock: 1410
        gpu_max_clock: 1410
      changed_when: true
      when: lock_gpu_clocks | default(false)

    - name: Verify GPU availability
      ansible.builtin.command: nvidia-smi --query-gpu=gpu_name,memory.total --format=csv,noheader
      register: gpu_info
      changed_when: false

    - name: Display GPU inventory
      ansible.builtin.debug:
        msg: "{{ inventory_hostname }}: {{ gpu_info.stdout_lines }}"

  handlers:
    - name: Reboot for driver
      ansible.builtin.reboot:
        reboot_timeout: 600

Deploy SLURM Cluster

- name: Deploy SLURM controller
  hosts: slurm_controller
  become: true
  tasks:
    - name: Install SLURM controller packages
      ansible.builtin.package:
        name:
          - slurm-wlm
          - slurmctld
          - slurmdbd
          - munge
        state: present

    - name: Configure SLURM
      ansible.builtin.template:
        src: slurm.conf.j2
        dest: /etc/slurm/slurm.conf
      notify: Restart slurmctld

    - name: Configure GPU resources (gres.conf)
      ansible.builtin.template:
        src: gres.conf.j2
        dest: /etc/slurm/gres.conf
      notify: Restart slurmctld

    - name: Configure accounting database
      ansible.builtin.template:
        src: slurmdbd.conf.j2
        dest: /etc/slurm/slurmdbd.conf
        mode: "0600"
      notify: Restart slurmdbd

    - name: Start SLURM services
      ansible.builtin.service:
        name: "{{ item }}"
        state: started
        enabled: true
      loop:
        - munge
        - slurmdbd
        - slurmctld

  handlers:
    - name: Restart slurmctld
      ansible.builtin.service:
        name: slurmctld
        state: restarted
    - name: Restart slurmdbd
      ansible.builtin.service:
        name: slurmdbd
        state: restarted

- name: Deploy SLURM compute nodes
  hosts: gpu_nodes
  become: true
  tasks:
    - name: Install SLURM compute packages
      ansible.builtin.package:
        name:
          - slurmd
          - munge
        state: present

    - name: Distribute SLURM configuration
      ansible.builtin.template:
        src: slurm.conf.j2
        dest: /etc/slurm/slurm.conf

    - name: Distribute munge key
      ansible.builtin.copy:
        src: munge.key
        dest: /etc/munge/munge.key
        owner: munge
        mode: "0400"
      no_log: true

    - name: Start compute services
      ansible.builtin.service:
        name: "{{ item }}"
        state: started
        enabled: true
      loop: [munge, slurmd]

{# templates/slurm.conf.j2 #}
ClusterName={{ cluster_name }}
SlurmctldHost={{ slurm_controller_hostname }}

# Scheduling
SchedulerType=sched/backfill
SelectType=select/cons_tres
SelectTypeParameters=CR_Core_Memory

# GPU Support
GresTypes=gpu
TaskPlugin=task/affinity,task/cgroup

# Accounting
AccountingStorageType=accounting_storage/slurmdbd
AccountingStorageHost={{ slurm_controller_hostname }}
JobAcctGatherType=jobacct_gather/cgroup

# Nodes
{% for host in groups['gpu_nodes'] %}
NodeName={{ host }} CPUs={{ hostvars[host]['ansible_processor_vcpus'] }} RealMemory={{ hostvars[host]['ansible_memtotal_mb'] }} Gres=gpu:{{ hostvars[host].get('gpu_count', 8) }} State=UNKNOWN
{% endfor %}

# Partitions
PartitionName=gpu Nodes={{ groups['gpu_nodes'] | join(',') }} Default=YES MaxTime=168:00:00 State=UP
PartitionName=interactive Nodes={{ groups['gpu_nodes'][:2] | join(',') }} MaxTime=4:00:00 State=UP Priority=100

InfiniBand Fabric Configuration

- name: Configure InfiniBand networking
  hosts: gpu_nodes
  become: true
  tasks:
    - name: Install InfiniBand packages
      ansible.builtin.package:
        name:
          - ibverbs-utils
          - infiniband-diags
          - rdma-core
          - opensm
          - perftest
        state: present

    - name: Enable RDMA service
      ansible.builtin.service:
        name: rdma
        state: started
        enabled: true

    - name: Configure IPoIB interface
      ansible.builtin.template:
        src: ib0.j2
        dest: /etc/netplan/60-infiniband.yaml
      notify: Apply netplan

    - name: Verify InfiniBand link
      ansible.builtin.command: ibstat
      register: ib_status
      changed_when: false
      failed_when: "'Active' not in ib_status.stdout"

    - name: Run NCCL bandwidth test
      ansible.builtin.command: >
        /opt/nccl-tests/build/all_reduce_perf
        -b 8 -e 4G -f 2 -g {{ gpu_count }}
      register: nccl_test
      changed_when: false
      when: run_nccl_test | default(false)

  handlers:
    - name: Apply netplan
      ansible.builtin.command: netplan apply

Parallel Storage (BeeGFS)

- name: Deploy BeeGFS parallel filesystem
  hosts: storage_nodes
  become: true
  tasks:
    - name: Install BeeGFS packages
      ansible.builtin.package:
        name:
          - beegfs-mgmtd
          - beegfs-meta
          - beegfs-storage
          - beegfs-utils
        state: present

    - name: Configure storage targets
      ansible.builtin.template:
        src: beegfs-storage.conf.j2
        dest: /etc/beegfs/beegfs-storage.conf
      notify: Restart BeeGFS storage

    - name: Format and mount NVMe for storage targets
      ansible.posix.mount:
        path: "/data/beegfs/target{{ item }}"
        src: "/dev/nvme{{ item }}n1"
        fstype: xfs
        opts: noatime,nodiscard
        state: mounted
      loop: "{{ range(0, nvme_count) | list }}"

- name: Mount BeeGFS on compute nodes
  hosts: gpu_nodes
  become: true
  tasks:
    - name: Install BeeGFS client
      ansible.builtin.package:
        name: beegfs-client
        state: present

    - name: Configure BeeGFS mount
      ansible.builtin.template:
        src: beegfs-mounts.conf.j2
        dest: /etc/beegfs/beegfs-mounts.conf

    - name: Mount parallel filesystem
      ansible.posix.mount:
        path: /scratch
        src: beegfs
        fstype: beegfs
        opts: "cfgFile=/etc/beegfs/beegfs-client.conf"
        state: mounted

GPU Monitoring (DCGM + Prometheus)

- name: Deploy GPU monitoring
  hosts: gpu_nodes
  become: true
  tasks:
    - name: Install DCGM
      ansible.builtin.package:
        name: datacenter-gpu-manager
        state: present

    - name: Deploy DCGM exporter
      community.docker.docker_container:
        name: dcgm-exporter
        image: nvcr.io/nvidia/k8s/dcgm-exporter:latest
        state: started
        restart_policy: unless-stopped
        ports: ["9400:9400"]
        device_requests:
          - driver: nvidia
            count: -1
            capabilities: [[gpu]]

    - name: Configure GPU health checks
      ansible.builtin.cron:
        name: "GPU health check"
        minute: "*/5"
        job: >
          nvidia-smi --query-gpu=temperature.gpu,utilization.gpu,memory.used,ecc.errors.uncorrected.aggregate.total
          --format=csv,noheader >> /var/log/gpu-health.csv 2>&1

Conclusion

AI supercomputing in 2026 demands infrastructure automation at scale. Ansible provisions GPU compute nodes with NVIDIA drivers and CUDA, deploys SLURM for job scheduling with GPU-aware resource management (gres.conf), configures InfiniBand fabrics for GPU-to-GPU RDMA communication, sets up parallel filesystems like BeeGFS for training I/O, and monitors GPU health with DCGM. The pattern: treat each HPC layer as an Ansible role, compose them into a cluster playbook, and scale from a single node to thousands with inventory changes alone.