# 30×4060 集群 · 软件栈完全部署手册

> 从裸机到 30 人同时上课，每层都有命令，每步都能回滚
> 2026-06-15

---

## 总体架构（先看全貌）

```
┌─────────────────────────────────────────────────────┐
│                  学生终端 (浏览器/SSH)                │
├─────────────────────────────────────────────────────┤
│  Layer 5: Web 接入    OpenClaw / llama.cpp server    │
├─────────────────────────────────────────────────────┤
│  Layer 4: 调度        Slurm + GPU 资源管理           │
├─────────────────────────────────────────────────────┤
│  Layer 3: 推理服务    llama.cpp server ×30 (systemd) │
├─────────────────────────────────────────────────────┤
│  Layer 2: 引擎        llama.cpp + 量化模型           │
├─────────────────────────────────────────────────────┤
│  Layer 1: OS          Ubuntu 24.04 + NVIDIA Driver   │
├─────────────────────────────────────────────────────┤
│              4 节点 × 8×4060 (最后一个节点 6 张)      │
└─────────────────────────────────────────────────────┘
```

### 四节点规划

| 节点 | 角色 | GPU 数 | 额外服务 |
|------|------|--------|---------|
| node1 (master) | Slurm 主控 + 计算 | 8 | Slurmctld, NFS Server, Prometheus |
| node2 | 纯计算 | 8 | — |
| node3 | 纯计算 | 8 | — |
| node4 | 纯计算 | 6 | — |

---

# Layer 1：OS + 驱动（每节点执行）

## 1.1 系统准备

```bash
#!/bin/bash
# 每台节点以 root 执行
# 文件: /opt/setup/01-os-prep.sh

set -e

echo "=== 更新系统 ==="
apt update && apt upgrade -y

echo "=== 基础依赖 ==="
apt install -y \
  build-essential cmake git curl wget \
  htop iotop iftop net-tools \
  nfs-common nfs-kernel-server \
  python3 python3-pip python3-venv \
  jq prometheus-node-exporter

echo "=== 禁用 Nouveau（开源驱动会冲突）==="
cat > /etc/modprobe.d/blacklist-nouveau.conf << 'EOF'
blacklist nouveau
options nouveau modeset=0
EOF
update-initramfs -u

echo "=== 安装 NVIDIA 驱动 550 ==="
apt install -y nvidia-driver-550 nvidia-cuda-toolkit
# 或者用 NVIDIA 官方 runfile（版本更可控）：
# wget https://us.download.nvidia.com/XFree86/Linux-x86_64/550.120/NVIDIA-Linux-x86_64-550.120.run
# sh NVIDIA-Linux-x86_64-550.120.run --no-questions --ui=none

echo "=== 重启生效 ==="
echo "请执行: reboot"
```

重启后验证：

```bash
nvidia-smi
# 期望：看到所有 8 张卡，Driver Version 550.x
# 如果有卡 missing：重新插拔、检查供电线

nvidia-smi topo -m
# 期望输出中 GPU 之间是 PXB（PCIe Bridge），不是 SOC
# 如果看到 SOC 或者 NODE，说明有 NVLink（4060 不可能，可能是误识别）
```

## 1.2 PCIe 带宽验证（关键！）

```bash
#!/bin/bash
# 文件: /opt/setup/02-check-pcie.sh

echo "=== PCIe 拓扑检查 ==="
nvidia-smi topo -m

echo ""
echo "=== 每张卡的 PCIe 链路速度 ==="
for gpu in $(seq 0 7); do
    speed=$(nvidia-smi -i $gpu -q -d PCIE | grep -A1 "Link Current" | tail -1 | awk '{print $NF}')
    echo "GPU $gpu: PCIe $speed"
done
# 期望：全部 Gen4×8 或 Gen4×16
# Gen3×4 以下需要检查：卡没插牢 / BIOS 没配 Gen4 / 延长线质量问题
```

## 1.3 网络配置

```bash
#!/bin/bash
# 文件: /opt/setup/03-network.sh
# 在 master 节点执行

# /etc/hosts 加节点解析（每节点都要）
cat >> /etc/hosts << 'EOF'
192.168.100.1   node1 master
192.168.100.2   node2
192.168.100.3   node3
192.168.100.4   node4
EOF

# NFS 共享模型目录（master 导出，所有节点挂载）
# master 上：
mkdir -p /nfs/models /nfs/student-home
cat >> /etc/exports << 'EOF'
/nfs/models       *(rw,sync,no_subtree_check,no_root_squash)
/nfs/student-home *(rw,sync,no_subtree_check,no_root_squash)
EOF
exportfs -a
systemctl restart nfs-kernel-server

# 各计算节点上：
# mount -t nfs node1:/nfs/models /nfs/models
# mount -t nfs node1:/nfs/student-home /nfs/student-home
# echo "node1:/nfs/models /nfs/models nfs defaults 0 0" >> /etc/fstab
```

---

# Layer 2：llama.cpp 引擎

## 2.1 编译（每节点）

```bash
#!/bin/bash
# 文件: /opt/setup/04-build-llamacpp.sh

cd /opt
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp

mkdir build && cd build

# Ada Lovelace (RTX 4060) 的计算能力是 8.9
cmake .. \
  -DGGML_CUDA=ON \
  -DCMAKE_CUDA_ARCHITECTURES="89" \
  -DGGML_CUDA_FORCE_MMQ=OFF \
  -DGGML_CUDA_F16=ON

make -j$(nproc) llama-server llama-bench llama-cli llama-quantize

# 验证
./llama-bench --version
./llama-server --version

# 软链到 PATH
ln -sf /opt/llama.cpp/build/llama-server /usr/local/bin/llama-server
ln -sf /opt/llama.cpp/build/llama-bench   /usr/local/bin/llama-bench
ln -sf /opt/llama.cpp/build/llama-cli     /usr/local/bin/llama-cli
```

## 2.2 模型下载与量化流水线

```bash
#!/bin/bash
# 文件: /opt/setup/05-model-pipeline.sh
# 模型统一放在 /nfs/models/

MODEL_DIR=/nfs/models
mkdir -p $MODEL_DIR

# === 方案 A：直接下载 GGUF（最快） ===
# 从 ModelScope 国内源下载预量化好的 GGUF

# Qwen3-7B Q4_K_M（教学主力，~4.5GB）
wget -P $MODEL_DIR \
  "https://modelscope.cn/models/Qwen/Qwen3-7B-GGUF/resolve/master/qwen3-7b-q4_k_m.gguf"

# DeepSeek-R1-Distill-Qwen-7B Q4_K_M（编程课用，~4.2GB）
wget -P $MODEL_DIR \
  "https://modelscope.cn/models/deepseek-ai/DeepSeek-R1-Distill-Qwen-7B-GGUF/resolve/master/DeepSeek-R1-Distill-Qwen-7B-Q4_K_M.gguf"

# Qwen3-1.5B Q8_0（轻量实训用，~1.6GB）
wget -P $MODEL_DIR \
  "https://modelscope.cn/models/Qwen/Qwen3-1.5B-GGUF/resolve/master/qwen3-1.5b-q8_0.gguf"

# === 方案 B：自己量化（如果需要特定量化级别） ===

# 步骤 1：从 HuggingFace 下载原始模型（用 modelscope 代理更快）
# pip install modelscope
# python3 -c "
# from modelscope import snapshot_download
# snapshot_download('Qwen/Qwen3-7B', cache_dir='/tmp/model-dl')
# "

# 步骤 2：转 GGUF
# python3 /opt/llama.cpp/convert_hf_to_gguf.py /tmp/model-dl/Qwen3-7B \
#   --outfile $MODEL_DIR/Qwen3-7B-F16.gguf --outtype f16

# 步骤 3：量化
# /opt/llama.cpp/build/llama-quantize \
#   $MODEL_DIR/Qwen3-7B-F16.gguf \
#   $MODEL_DIR/Qwen3-7B-Q4_K_M.gguf \
#   Q4_K_M
```

## 2.3 基准测试矩阵

```bash
#!/bin/bash
# 文件: /opt/setup/06-benchmark.sh
# 产出的数据直接用于论文

BENCH_DIR=/opt/results/benchmarks
mkdir -p $BENCH_DIR

declare -a MODELS=(
  "qwen3-7b-q4_k_m"
  "qwen3-1.5b-q8_0"
  "deepseek-r1-distill-qwen-7b-q4_k_m"
)

echo "model,quant,gpu_id,pp512,tg128,batch_size,ctx_len" > $BENCH_DIR/single_gpu.csv

for model in "${MODELS[@]}"; do
  for gpu in 0 1 2 3 4 5 6 7; do
    CUDA_VISIBLE_DEVICES=$gpu llama-bench \
      -m /nfs/models/${model}.gguf \
      -ngl 99 \
      -b 512 \
      -t 4 \
      --json-output 2>/dev/null | \
      jq -r --arg m "$model" --arg g "$gpu" \
        '.[] | "\($m),Q4,\($g),\(.pp512),\(.tg128),512,4096"' \
      >> $BENCH_DIR/single_gpu.csv
  done
done

echo "单卡基准完成，结果在 $BENCH_DIR/single_gpu.csv"

# 多卡并发测试
echo "model,cards,concurrent_users,throughput_tps,avg_latency_ms,p95_latency_ms" \
  > $BENCH_DIR/multi_gpu.csv

# 启动 8 个 server 实例（每个 GPU 一个）
for gpu in 0 1 2 3 4 5 6 7; do
  CUDA_VISIBLE_DEVICES=$gpu llama-server \
    -m /nfs/models/qwen3-7b-q4_k_m.gguf \
    --host 0.0.0.0 --port $((8080 + gpu)) \
    -ngl 99 -c 4096 -b 512 &
done
sleep 10  # 等所有实例启动

# 用 wrk2 做并发压测
# 先装 wrk2: apt install -y wrk
# 然后编写压测脚本（略，用 Python requests 也能做）

echo "多卡并发测试脚本框架已就绪，具体压测命令见 /opt/tests/concurrent_bench.py"
```

---

# Layer 3：推理服务层

## 3.1 systemd 模板服务

```bash
#!/bin/bash
# 文件: /opt/setup/07-llama-services.sh
# 每张卡一个 systemd 服务，自动隔离 GPU

for gpu in 0 1 2 3 4 5 6 7; do
  cat > /etc/systemd/system/llama-server@${gpu}.service << UNITEOF
[Unit]
Description=llama.cpp inference server on GPU %i
After=network.target nfs-client.target
Wants=nfs-client.target

[Service]
Type=simple
Environment=CUDA_VISIBLE_DEVICES=%i
ExecStartPre=/usr/bin/nvidia-smi -i %i > /dev/null
ExecStart=/opt/llama.cpp/build/llama-server \
  -m /nfs/models/qwen3-7b-q4_k_m.gguf \
  --host 0.0.0.0 \
  --port ${gpu}080 \
  -ngl 99 \
  -c 4096 \
  -b 512 \
  --threads 4 \
  --log-disable
ExecStop=/bin/kill -TERM \$MAINPID
Restart=on-failure
RestartSec=5
TimeoutStopSec=10

# 安全限制
NoNewPrivileges=yes
PrivateTmp=yes
ProtectSystem=strict
ProtectHome=yes
ReadWritePaths=/nfs/models

[Install]
WantedBy=multi-user.target
UNITEOF

  # 模型可通过模板参数切换，用 drop-in：
  # systemctl edit llama-server@0
  # [Service]
  # ExecStart=
  # ExecStart=/opt/llama.cpp/build/llama-server -m /nfs/models/other-model.gguf ...

done

systemctl daemon-reload
# 启用全部 8 个 GPU 服务
for gpu in 0 1 2 3 4 5 6 7; do
  systemctl enable llama-server@${gpu}
done
```

## 3.2 健康检查 + 自动恢复

```bash
#!/bin/bash
# 文件: /opt/setup/08-health-check.sh
# 放到 cron：*/5 * * * * /opt/scripts/health-check.sh

#!/bin/bash
# 文件: /opt/scripts/health-check.sh
# 每 5 分钟运行一次

LOG=/var/log/gpu-health.log

for gpu in 0 1 2 3 4 5 6 7; do
  port=$((gpu * 1000 + 8080))  # GPU0→8080, GPU1→9080, ...
  
  # 检查 1：进程在不在
  if ! systemctl is-active --quiet llama-server@${gpu}; then
    echo "[$(date)] GPU$gpu service dead, restarting" >> $LOG
    systemctl restart llama-server@${gpu}
    continue
  fi

  # 检查 2：端口有没在监听
  if ! ss -tlnp | grep -q ":${port} "; then
    echo "[$(date)] GPU$gpu port $port not listening, restarting" >> $LOG
    systemctl restart llama-server@${gpu}
    continue
  fi

  # 检查 3：API 健康端点
  if ! curl -s -o /dev/null -w "%{http_code}" --max-time 5 \
    http://localhost:${port}/health 2>/dev/null | grep -q 200; then
    echo "[$(date)] GPU$gpu health check failed, restarting" >> $LOG
    systemctl restart llama-server@${gpu}
    continue
  fi

  # 检查 4：GPU 还在不在（防掉卡）
  if ! nvidia-smi -i $gpu > /dev/null 2>&1; then
    echo "[$(date)] GPU$gpu missing from nvidia-smi!" >> $LOG
    # 严重问题，发告警（后续接 Prometheus Alertmanager）
    continue
  fi
done

# 汇总
active=$(systemctl list-units --state=active llama-server@* 2>/dev/null | grep -c llama)
total=8
echo "[$(date)] Status: $active/$total GPUs healthy" >> $LOG
```

**安装到 cron：**

```bash
mkdir -p /opt/scripts
cp health-check.sh /opt/scripts/
chmod +x /opt/scripts/health-check.sh
echo "*/5 * * * * root /opt/scripts/health-check.sh" > /etc/cron.d/gpu-health
```

## 3.3 GPU 按模型切换脚本

```bash
#!/bin/bash
# 文件: /opt/scripts/switch-model.sh
# 用法: switch-model.sh <gpu_id> <model_name>
# 示例: switch-model.sh 3 deepseek-coder-7b

GPU=$1
MODEL=$2
MODEL_DIR=/nfs/models

declare -A MODEL_PATHS=(
  ["qwen3-7b"]="${MODEL_DIR}/qwen3-7b-q4_k_m.gguf"
  ["qwen3-1.5b"]="${MODEL_DIR}/qwen3-1.5b-q8_0.gguf"
  ["deepseek-coder-7b"]="${MODEL_DIR}/deepseek-r1-distill-qwen-7b-q4_k_m.gguf"
)

MODEL_PATH="${MODEL_PATHS[$MODEL]}"
if [ -z "$MODEL_PATH" ]; then
  echo "Unknown model: $MODEL. Available: ${!MODEL_PATHS[@]}"
  exit 1
fi

systemctl stop llama-server@${GPU}

# 用 drop-in 覆盖模型路径
mkdir -p /etc/systemd/system/llama-server@${GPU}.service.d
cat > /etc/systemd/system/llama-server@${GPU}.service.d/model-override.conf << EOF
[Service]
ExecStart=
ExecStart=/opt/llama.cpp/build/llama-server \
  -m ${MODEL_PATH} \
  --host 0.0.0.0 \
  --port ${GPU}080 \
  -ngl 99 -c 4096 -b 512 --threads 4 --log-disable
EOF

systemctl daemon-reload
systemctl start llama-server@${GPU}
echo "GPU $GPU switched to $MODEL ($MODEL_PATH)"
```

---

# Layer 4：Slurm 调度层

## 4.1 安装（master 节点）

```bash
#!/bin/bash
# 文件: /opt/setup/09-slurm-install.sh

# 控制节点 (node1)
apt install -y slurm-wlm slurmdbd munge
# 计算节点
apt install -y slurmd munge

# Munge 密钥（所有节点必须相同）
/usr/sbin/create-munge-key
scp /etc/munge/munge.key node2:/etc/munge/
scp /etc/munge/munge.key node3:/etc/munge/
scp /etc/munge/munge.key node4:/etc/munge/
```

## 4.2 slurm.conf（完整配置）

```bash
# 文件: /etc/slurm/slurm.conf（master 节点）
cat > /etc/slurm/slurm.conf << 'CONFEOF'
# 30×4060 教学集群 Slurm 配置

ClusterName=4060-cluster
ControlMachine=node1
SlurmUser=slurm
SlurmctldPort=6817
SlurmdPort=6818
AuthType=auth/munge
StateSaveLocation=/var/spool/slurm/ctld
SlurmdSpoolDir=/var/spool/slurm/d
SwitchType=switch/none
MpiDefault=none
ProctrackType=proctrack/cgroup
ReturnToService=2
SlurmctldTimeout=300
SlurmdTimeout=300
InactiveLimit=0
MinJobAge=300
KillWait=30
Waittime=0
SchedulerType=sched/backfill
SelectType=select/cons_tres

# 日志
SlurmctldDebug=info
SlurmctldLogFile=/var/log/slurmctld.log
SlurmdDebug=info
SlurmdLogFile=/var/log/slurmd.log

# GPU 资源定义
GresTypes=gpu
NodeName=node1 CPUs=32 Gres=gpu:4060:8 State=UNKNOWN
NodeName=node2 CPUs=32 Gres=gpu:4060:8 State=UNKNOWN
NodeName=node3 CPUs=32 Gres=gpu:4060:8 State=UNKNOWN
NodeName=node4 CPUs=32 Gres=gpu:4060:6 State=UNKNOWN  # 最后节点 6 张卡

# 分区
# teaching: 默认分区，最大 4 小时（一节课时间）
# research: 教师专用，无时长限制，但只有指定账号能用
PartitionName=teaching Nodes=node[1-4] Default=YES MaxTime=04:00:00
PartitionName=research Nodes=node[1-4] MaxTime=INFINITE AllowAccounts=faculty

# 任务限制：每人最多 1 个 GPU（防止霸占）
AssociationParams=MaxTRESPerJob=gres:gpu:4060:1
CONFEOF
```

## 4.3 gres.conf（GPU 资源映射）

```bash
# 文件: /etc/slurm/gres.conf（每节点）

# node1 (8 张 4060)
cat > /etc/slurm/gres.conf << 'EOF'
Name=gpu Type=4060 File=/dev/nvidia0
Name=gpu Type=4060 File=/dev/nvidia1
Name=gpu Type=4060 File=/dev/nvidia2
Name=gpu Type=4060 File=/dev/nvidia3
Name=gpu Type=4060 File=/dev/nvidia4
Name=gpu Type=4060 File=/dev/nvidia5
Name=gpu Type=4060 File=/dev/nvidia6
Name=gpu Type=4060 File=/dev/nvidia7
EOF

# node4 只有 6 张卡，对应文件删掉后两条
```

## 4.4 学生用户批量创建

```bash
#!/bin/bash
# 文件: /opt/setup/10-create-students.sh

CLASS="drone-2026"  # 班级标识

for i in $(seq -w 1 30); do
  USER="stu${i}"
  
  # 系统账户
  useradd -m -s /bin/bash -G students $USER
  echo "${USER}:ChangeMe123!" | chpasswd
  
  # Slurm 账户
  sacctmgr -i add account $CLASS \
    description="无人机应用技术 2026" \
    organization=internet-info-school
  
  sacctmgr -i add user $USER \
    account=$CLASS \
    DefaultAccount=$CLASS \
    Partitions=teaching
  
  # 创建作业脚本模板
  mkdir -p /home/$USER/slurm
  cp /opt/templates/student-job.sh /home/$USER/slurm/
  chown -R $USER:$USER /home/$USER/slurm
  
  echo "Created: $USER"
done

echo "Done. All passwords: ChangeMe123! (force change on first login)"
```

## 4.5 学生作业脚本模板

```bash
#!/bin/bash
# 文件: /opt/templates/student-job.sh
# 学生用这个脚本申请 GPU 并启动自己的模型实例

#SBATCH --job-name=my-llm-lab
#SBATCH --partition=teaching
#SBATCH --gres=gpu:4060:1
#SBATCH --time=02:00:00
#SBATCH --output=/nfs/student-home/%u/lab-%j.log
#SBATCH --error=/nfs/student-home/%u/lab-%j.err

echo "=== 你的 GPU 信息 ==="
echo "Job ID:    $SLURM_JOB_ID"
echo "Node:      $SLURMD_NODENAME"
echo "GPU ID:    $SLURM_JOB_GPUS"

# Slurm 已经通过 CUDA_VISIBLE_DEVICES 隔离
# 你的模型只看到分配给你的那张卡

MODEL="/nfs/models/qwen3-7b-q4_k_m.gguf"

# 启动专属模型实例
echo "=== 启动 llama.cpp server ==="
CUDA_VISIBLE_DEVICES=$SLURM_JOB_GPUS /opt/llama.cpp/build/llama-server \
  -m $MODEL \
  --host 0.0.0.0 \
  --port 0 \
  -ngl 99 \
  -c 4096 \
  -b 512 &
SERVER_PID=$!

# 等待启动
sleep 5

# 找出实际端口
ACTUAL_PORT=$(ss -tlnp | grep $SERVER_PID | awk '{print $4}' | rev | cut -d: -f1 | rev)
echo "=== 你的模型已就绪 ==="
echo "Address: http://${SLURMD_NODENAME}:${ACTUAL_PORT}"
echo "API:     http://${SLURMD_NODENAME}:${ACTUAL_PORT}/v1/chat/completions"
echo ""
echo "在浏览器打开上面的地址，或 curl 测试："
echo "curl http://${SLURMD_NODENAME}:${ACTUAL_PORT}/v1/chat/completions \\"
echo "  -H 'Content-Type: application/json' \\"
echo "  -d '{\"messages\":[{\"role\":\"user\",\"content\":\"你好\"}]}'"
echo ""

# 保持运行直到作业结束
wait $SERVER_PID
```

---

# Layer 5：监控层

## 5.1 GPU 聚合监控脚本

```bash
#!/bin/bash
# 文件: /opt/scripts/gpu-aggregate.sh
# 用 cron 每 60 秒跑一次，生成 JSON 供 Grafana 拉取

OUTPUT=/var/www/html/gpu-status.json  # nginx/apache 暴露出去

while true; do
  echo "{" > $OUTPUT.tmp
  echo "  \"timestamp\": \"$(date -Iseconds)\"," >> $OUTPUT.tmp
  echo "  \"nodes\": {" >> $OUTPUT.tmp

  first_node=true
  for node in node1 node2 node3 node4; do
    $first_node || echo "," >> $OUTPUT.tmp
    first_node=false

    echo "    \"$node\": $(ssh $node 'nvidia-smi --query-gpu=index,name,temperature.gpu,utilization.gpu,memory.used,memory.total,power.draw --format=csv,noheader,nounits 2>/dev/null | \
      jq -R -s "split(\"\n\") | map(select(length > 0) | split(\",\") | {gpu: .[0]|tonumber, name: .[1], temp: .[2]|tonumber, util: .[3]|tonumber, mem_used: .[4]|tonumber, mem_total: .[5]|tonumber, power: .[6]|tonumber})")' 2>/dev/null || echo '[]')" >> $OUTPUT.tmp
  done

  echo "  }" >> $OUTPUT.tmp
  echo "}" >> $OUTPUT.tmp

  mv $OUTPUT.tmp $OUTPUT
  sleep 60
done
```

## 5.2 一键状态面板

```bash
#!/bin/bash
# 文件: /opt/scripts/cluster-status.sh
# 随时跑，看集群全貌

echo "╔══════════════════════════════════════════╗"
echo "║     30×4060 集群状态 · $(date +%H:%M:%S)   ║"
echo "╠══════════════════════════════════════════╣"

echo "║ GPU 健康度："
for node in node1 node2 node3 node4; do
  total=$(ssh $node "nvidia-smi -L 2>/dev/null | wc -l")
  used=$(ssh $node "nvidia-smi --query-gpu=utilization.gpu --format=csv,noheader,nounits 2>/dev/null | awk '\$1>0' | wc -l")
  printf "║   %-8s  %2d/%2d 活跃\n" "$node:" "$used" "$total"
done

echo "╠══════════════════════════════════════════╣"
echo "║ Slurm 作业队列："
squeue --format="%.8i %.10u %.8T %.6D %.10M %R" 2>/dev/null | head -20 | while read line; do
  echo "║   $line"
done

echo "╠══════════════════════════════════════════╣"
echo "║ 推理服务状态："
for node in node1 node2 node3 node4; do
  running=$(ssh $node "systemctl list-units --state=active llama-server@* 2>/dev/null | grep -c llama" 2>/dev/null || echo "?")
  echo "║   $node: $running llama-server 实例运行中"
done

echo "╚══════════════════════════════════════════╝"
```

---

# Layer 6：运维工具箱

## 6.1 一键部署脚本

```bash
#!/bin/bash
# 文件: /opt/setup/deploy-all.sh
# 在新节点上执行，从裸机到就绪

echo "=== 30×4060 集群一键部署脚本 ==="
echo "目标节点: $(hostname)"
echo ""

./01-os-prep.sh     && echo "✅ OS 准备完成"     || { echo "❌ OS 准备失败"; exit 1; }
./02-check-pcie.sh  && echo "✅ PCIe 检查通过"    || echo "⚠️  PCIe 检查有警告"
./03-network.sh     && echo "✅ 网络配置完成"    || { echo "❌ 网络配置失败"; exit 1; }
./04-build-llamacpp.sh && echo "✅ llama.cpp 编译完成" || { echo "❌ llama.cpp 编译失败"; exit 1; }
./05-model-pipeline.sh  && echo "✅ 模型就绪"    || { echo "❌ 模型下载失败"; exit 1; }
./07-llama-services.sh  && echo "✅ 推理服务创建" || { echo "❌ 服务创建失败"; exit 1; }

echo ""
echo "=== 部署完成 ==="
echo "检查: systemctl status llama-server@0"
echo "检查: curl http://localhost:8080/health"
```

## 6.2 紧急重置脚本

```bash
#!/bin/bash
# 文件: /opt/scripts/emergency-reset.sh
# 教学出问题时用——清理所有学生作业，恢复干净状态

echo "⚠️  紧急重置：将终止所有学生作业，你确定？(yes/no)"
read confirm
[ "$confirm" != "yes" ] && echo "取消" && exit 0

echo "=== 终止所有学生 Slurm 作业 ==="
scancel -u stu00,stu01,stu02,stu03,stu04,stu05,stu06,stu07,stu08,stu09 || true
scancel -u stu10,stu11,stu12,stu13,stu14,stu15,stu16,stu17,stu18,stu19 || true
scancel -u stu20,stu21,stu22,stu23,stu24,stu25,stu26,stu27,stu28,stu29,stu30 || true

echo "=== 重启所有推理服务 ==="
for node in node1 node2 node3 node4; do
  ssh $node "for gpu in 0 1 2 3 4 5 6 7; do systemctl restart llama-server@\${gpu}; done" &
done
wait

echo "=== 验证 ==="
sleep 10
/opt/scripts/cluster-status.sh
echo "✅ 集群已重置到干净状态"
```

## 6.3 诊断脚本

```bash
#!/bin/bash
# 文件: /opt/scripts/diagnose.sh
# 出问题时跑，收集全部诊断信息

REPORT="/tmp/cluster-diag-$(date +%Y%m%d-%H%M%S).txt"

{
  echo "========== HOST =========="
  hostname && uname -a

  echo "========== GPU =========="
  nvidia-smi

  echo "========== PCIe =========="
  nvidia-smi topo -m

  echo "========== DOCKER/VM =========="
  systemctl list-units --state=running | grep -E 'llama|slurm|nvidia|cuda'

  echo "========== DISK =========="
  df -h /nfs/models

  echo "========== NETWORK =========="
  ip addr show | grep "inet "

  echo "========== LOGS =========="
  journalctl -u 'llama-server@*' --since "1 hour ago" --no-pager | tail -50

  echo "========== SLURM =========="
  sinfo && squeue

  echo "========== ERRORS =========="
  dmesg | grep -iE 'error|fail|nvidia|gpu' | tail -30
} > $REPORT

echo "诊断报告: $REPORT"
```

---

# 附录 A：常见故障排查

| 现象 | 可能原因 | 解决 |
|------|---------|------|
| `nvidia-smi` 看不到某张卡 | 供电不足/PCIe 松动 | 重新插拔、换 8-pin 线 |
| llama-server OOM | 模型太大或 KV cache 太大 | 降量化(Q5→Q4)、缩 ctx(4096→2048) |
| 多卡并发时 latency 飙升 | PCIe 带宽共享 | 每卡只跑一个 server，不做张量并行 |
| Slurm 作业一直 PENDING | gres 配置错误 | `scontrol show node` 检查 GPU 资源 |
| NFS 挂载慢 | 网络拥堵 | 模型放本地 SSD，NFS 只放学生家目录 |
| llama-server 端口被占 | 上次没杀干净 | `fuser -k 8080/tcp` |

---

# 附录 B：文件清单

```
/opt/
├── setup/                          # 部署脚本
│   ├── 01-os-prep.sh
│   ├── 02-check-pcie.sh
│   ├── 03-network.sh
│   ├── 04-build-llamacpp.sh
│   ├── 05-model-pipeline.sh
│   ├── 06-benchmark.sh
│   ├── 07-llama-services.sh
│   ├── 08-health-check.sh
│   ├── 09-slurm-install.sh
│   ├── 10-create-students.sh
│   └── deploy-all.sh
├── scripts/                        # 运维脚本
│   ├── health-check.sh
│   ├── switch-model.sh
│   ├── cluster-status.sh
│   ├── emergency-reset.sh
│   ├── diagnose.sh
│   └── gpu-aggregate.sh
├── templates/                      # 学生模板
│   └── student-job.sh
├── llama.cpp/                      # llama.cpp 源码+编译
│   └── build/
│       ├── llama-server
│       ├── llama-bench
│       ├── llama-cli
│       └── llama-quantize
├── tests/                          # 压测脚本
│   └── concurrent_bench.py
└── results/                        # 测试数据
    └── benchmarks/
        ├── single_gpu.csv
        └── multi_gpu.csv

/etc/
├── slurm/
│   ├── slurm.conf
│   └── gres.conf
├── systemd/system/
│   └── llama-server@.service       # 模板服务
├── cron.d/
│   └── gpu-health
└── fstab                           # NFS 挂载

/nfs/
├── models/                         # 共享模型目录
│   ├── qwen3-7b-q4_k_m.gguf
│   ├── qwen3-1.5b-q8_0.gguf
│   └── deepseek-r1-distill-qwen-7b-q4_k_m.gguf
└── student-home/                   # 学生家目录
    ├── stu01/
    ├── stu02/
    └── ...
```

---

*部署时从 01 按顺序执行到 10，遇到问题查附录 A。*