RKE2 HA 集群 + Rancher v2.14.3 气隙部署操作手册

日期: 2026-08-15 集群: 3 节点 RKE2 HA | szb122032/33/34 | Rocky Linux 9.8 域名: gpu.ai-ear.cn

1. 环境信息

集群拓扑

                    ┌──────────────────────────────────────────────┐
                    │          Hypervisor (192.168.122.1)          │
                    │  libvirt dnsmasq: gpu.ai-ear.cn → .32/.33/34 │
                    │  HTTP file server :31001 (临时)               │
                    │  ┌─────────────────────────────────────────┐ │
                    │  │       Ingress: nginx-ingress            │ │
                    │  │  ┌──────────┐ ┌──────────┐ ┌──────────┐ │ │
                    │  │  │ .32      │ │ .33      │ │ .34      │ │ │
                    │  │  │ szb122032│ │ szb122033│ │ szb122034│ │ │
                    │  │  │ etcd+CP  │ │ etcd+CP  │ │ etcd+CP  │ │ │
                    │  │  │ 8C/32G   │ │ 8C/32G   │ │ 8C/32G   │ │ │
                    │  │  │ 40GB     │ │ 40GB     │ │ 40GB     │ │ │
                    │  │  └──────────┘ └──────────┘ └──────────┘ │ │
                    │  └─────────────────────────────────────────┘ │
                    │  Harbor: 192.168.122.156:30000               │
                    └──────────────────────────────────────────────┘

基础参数

参数	值
集群节点	192.168.122.32 / .33 / .34
RKE2 版本	v1.35.6+rke2r1 (Kubernetes v1.35.6)
Rancher 版本	v2.14.3
cert-manager	v1.20.3
域名	gpu.ai-ear.cn
Pod CIDR	10.12.0.0/16
Service CIDR	10.13.0.0/16
镜像仓库	192.168.122.156:30000 (Harbor, HTTP, 自签名)
kube-proxy	IPVS 模式
CNI	Calico v3.32.0
容器运行时	containerd 2.2.5-k3s2
OS	Rocky Linux 9.8 (Blue Onyx), kernel 5.14.0-687
磁盘	40GB XFS (从 20GB 扩容)
Bootstrap 密码	`xxx`
离线包位置	`/u02/repo/rke2/rke2-1.35.6/` (宿主机)

离线包清单

文件	大小	说明
`rke2.linux-amd64.tar.gz`	~700MB	RKE2 二进制包
`rke2-images.linux-amd64.tar.zst`	768MB	核心镜像 (apiserver, etcd, coredns 等)
`rke2-images-calico.linux-amd64.tar.zst`	574MB	Calico CNI 镜像
`install.sh`	~10KB	RKE2 安装脚本
`cert-manager-v1.20.3.tgz`	~1MB	cert-manager Helm chart
`rancher-2.14.3.tgz`	~20MB	Rancher Helm chart

2. 部署脚本

2.1 临时 HTTP 文件服务器

在宿主机上启动 Python HTTP 服务器，供 VM 节点下载离线包：

bash

# 在 hypervisor (192.168.122.1) 上执行
cd /u02/repo
nohup python3 -m http.server 31001 --bind 0.0.0.0 > /tmp/repo-server.log 2>&1 &
disown

# 验证
curl -sfL http://192.168.122.1:31001/rke2/rke2-1.35.6/rke2-images.linux-amd64.tar.zst -o /dev/null -w "%{http_code}\n"

# 部署完成后停止
pkill -f 'http.server 31001'

2.2 下载 airgap 镜像脚本

文件: /tmp/download_images.sh

bash

#!/bin/bash
set -ex
REPO="http://192.168.122.1:31001/rke2/rke2-1.35.6"
IMAGES_DIR="/var/lib/rancher/rke2/agent/images"

mkdir -p $IMAGES_DIR

echo "Downloading core images..."
curl -sfL -o ${IMAGES_DIR}/rke2-images.linux-amd64.tar.zst ${REPO}/rke2-images.linux-amd64.tar.zst
echo "Core images done ($(ls -lh ${IMAGES_DIR}/rke2-images.linux-amd64.tar.zst | awk '{print $5}'))"

echo "Downloading calico images..."
curl -sfL -o ${IMAGES_DIR}/rke2-images-calico.linux-amd64.tar.zst ${REPO}/rke2-images-calico.linux-amd64.tar.zst
echo "Calico images done ($(ls -lh ${IMAGES_DIR}/rke2-images-calico.linux-amd64.tar.zst | awk '{print $5}'))"

echo "ALL DONE"

2.3 节点初始化脚本

文件: /tmp/rke2_init_node.sh

在每台 VM 上执行，完成系统调优、内核模块、swap 禁用、RKE2 二进制安装和 registries 配置。

bash

#!/bin/bash
set -ex

NODE_NUM=$1
NODE_NAME="szb1220${NODE_NUM}"
HARBOR="192.168.122.156:30000"
REPO="http://192.168.122.1:31001"

echo "=== Initializing $NODE_NAME ==="

# 1. Kernel tuning
cat > /etc/sysctl.d/99-k8s.conf << 'EOF'
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
net.ipv4.conf.all.forwarding = 1
vm.max_map_count = 262144
fs.inotify.max_user_instances = 8192
fs.inotify.max_user_watches = 524288
net.core.somaxconn = 32768
net.ipv4.tcp_max_syn_backlog = 32768
net.netfilter.nf_conntrack_max = 524288
EOF
sysctl --system 2>/dev/null || true

# 2. Load required kernel modules
cat > /etc/modules-load.d/k8s.conf << 'EOF'
overlay
br_netfilter
nf_conntrack
EOF
modprobe overlay 2>/dev/null || true
modprobe br_netfilter 2>/dev/null || true
modprobe nf_conntrack 2>/dev/null || true

# 3. Disable swap
swapoff -a 2>/dev/null || true
sed -i '/swap/d' /etc/fstab 2>/dev/null || true

# 4. Set hostname
hostnamectl set-hostname $NODE_NAME

# 5. Set SELinux to permissive
setenforce 0 2>/dev/null || true
sed -i 's/^SELINUX=enforcing/SELINUX=permissive/' /etc/selinux/config 2>/dev/null || true

# 6. Download and extract RKE2 binary
mkdir -p /opt/rke2
cd /opt/rke2
curl -sfL "${REPO}/rke2/1.35.6/rke2.linux-amd64.tar.gz" -o rke2.linux-amd64.tar.gz
tar -xzf rke2.linux-amd64.tar.gz -C /usr/local/

# 7. Run install.sh to create systemd unit
curl -sfL "${REPO}/rke2/1.35.6/install.sh" -o /opt/rke2/install.sh
chmod +x /opt/rke2/install.sh
cd /opt/rke2
INSTALL_RKE2_SKIP_DOWNLOAD=true INSTALL_RKE2_TYPE=server ./install.sh

# 8. Configure containerd registries for Harbor
mkdir -p /etc/rancher/rke2
cat > /etc/rancher/rke2/registries.yaml << REGEOF
mirrors:
  docker.io:
    endpoint:
      - "http://${HARBOR}"
  quay.io:
    endpoint:
      - "http://${HARBOR}"
  registry.k8s.io:
    endpoint:
      - "http://${HARBOR}"
  "${HARBOR}":
    endpoint:
      - "http://${HARBOR}"
configs:
  "${HARBOR}":
    tls:
      insecure_skip_verify: true
    auth:
      username: xxx
      password: xxx
REGEOF

# 9. Create crictl config
cat > /etc/crictl.yaml << 'EOF'
runtime-endpoint: unix:///run/k3s/containerd/containerd.sock
image-endpoint: unix:///run/k3s/containerd/containerd.sock
timeout: 10
EOF

echo "=== Node $NODE_NAME initialized ==="

2.4 节点环境配置脚本

文件: /tmp/setup_rke2_nodes.sh

重新创建 systemd unit 文件、内核调优、Harbor registries 配置。

bash

#!/bin/bash
set -ex

NODE_NUM=$1
NODE_NAME="szb1220${NODE_NUM}"
HARBOR="192.168.122.156:30000"

if [ -f /usr/local/bin/rke2 ]; then
  cp -f /usr/local/bin/rke2 /usr/bin/rke2 2>/dev/null || true
fi
/usr/local/bin/rke2 --version

cat > /etc/systemd/system/rke2-server.service << 'EOF'
[Unit]
Description=Rancher Kubernetes Engine v2 (server)
Documentation=https://github.com/rancher/rke2#readme
Wants=network-online.target
After=network-online.target

[Install]
WantedBy=multi-user.target

[Service]
Type=notify
EnvironmentFile=-/etc/default/%N
EnvironmentFile=-/etc/sysconfig/%N
EnvironmentFile=-/usr/lib/systemd/system/%N.env
KillMode=process
Delegate=yes
LimitNOFILE=1048576
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
TimeoutStartSec=0
Restart=always
RestartSec=5s
ExecCondition=/bin/sh -c 'if systemctl is-active --quiet rke2-agent.service; then echo "Error: rke2-agent is running!"; exit 1; fi'
ExecStartPre=-/sbin/modprobe br_netfilter
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/rke2 server
ExecStopPost=-/bin/sh -c "systemd-cgls /system.slice/%n | grep -Eo '[0-9]+ (containerd|kubelet)' | awk '{print $1}' | xargs -r kill"
EOF

mkdir -p /etc/sysconfig
cat > /etc/sysconfig/rke2-server << 'EOF'
RKE2_DATA_DIR=/var/lib/rancher/rke2
EOF

cat > /etc/sysctl.d/99-k8s.conf << 'EOF'
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
net.ipv4.conf.all.forwarding = 1
vm.max_map_count = 262144
fs.inotify.max_user_instances = 8192
fs.inotify.max_user_watches = 524288
net.core.somaxconn = 32768
net.ipv4.tcp_max_syn_backlog = 32768
net.netfilter.nf_conntrack_max = 524288
EOF
sysctl --system 2>/dev/null || true

cat > /etc/modules-load.d/k8s.conf << 'EOF'
overlay
br_netfilter
nf_conntrack
EOF
modprobe overlay || true
modprobe br_netfilter || true
modprobe nf_conntrack || true

swapoff -a || true

mkdir -p /etc/rancher/rke2
cat > /etc/rancher/rke2/registries.yaml << REGEOF
mirrors:
  docker.io:
    endpoint:
      - "http://${HARBOR}"
  quay.io:
    endpoint:
      - "http://${HARBOR}"
  registry.k8s.io:
    endpoint:
      - "http://${HARBOR}"
  "${HARBOR}":
    endpoint:
      - "http://${HARBOR}"
configs:
  "${HARBOR}":
    tls:
      insecure_skip_verify: true
    auth:
      username: xxx
      password: xxx
REGEOF

cat > /etc/crictl.yaml << 'EOF'
runtime-endpoint: unix:///run/k3s/containerd/containerd.sock
image-endpoint: unix:///run/k3s/containerd/containerd.sock
timeout: 10
EOF

systemctl daemon-reload
echo "=== Node ${NODE_NAME} setup complete ==="

2.5 加入集群脚本

文件: /tmp/join_server.sh

将节点以 server 角色加入 RKE2 HA 集群，含 IPVS 模块加载、RKE2 config.yaml 创建。

bash

#!/bin/bash
set -ex

NODE_NUM=$1
NODE_NAME="szb1220${NODE_NUM}"
FIRST_NODE="192.168.122.32"
TOKEN="xxx"
HARBOR="192.168.122.156:30000"

echo "=== Joining $NODE_NAME to cluster ==="

# 1. Load IPVS modules
for mod in ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh ip_vs_sed ip_vs_nq ip_vs_wlc \
           ip_vs_dh ip_vs_lblc ip_vs_lblcr ip_vs_lc ip_vs_fo ip_vs_ovf \
           nf_conntrack br_netfilter overlay; do
  modprobe $mod 2>/dev/null || true
done

# 2. Persistent modules config
cat > /etc/modules-load.d/k8s.conf << 'EOF'
overlay
br_netfilter
nf_conntrack
ip_vs
ip_vs_rr
ip_vs_wrr
ip_vs_sh
ip_vs_sed
ip_vs_nq
ip_vs_wlc
ip_vs_dh
ip_vs_lblc
ip_vs_lblcr
ip_vs_lc
ip_vs_wlc
ip_vs_fo
ip_vs_ovf
EOF

# 3. Create RKE2 server config (join mode)
mkdir -p /etc/rancher/rke2
cat > /etc/rancher/rke2/config.yaml << CONFEOF
server: https://${FIRST_NODE}:9345
token: ${TOKEN}
write-kubeconfig-mode: "0644"
tls-san:
  - szb122032
  - szb122033
  - szb122034
  - gpu.ai-ear.cn
  - 192.168.122.32
  - 192.168.122.33
  - 192.168.122.34
cni:
  - calico
kube-proxy-arg:
  - proxy-mode=ipvs
disable-scheduler-node-taints: true
etcd-expose-metrics: true
cluster-cidr: 10.12.0.0/16
service-cidr: 10.13.0.0/16
service-node-port-range: 30000-60000
CONFEOF

# 4. Registries config
cat > /etc/rancher/rke2/registries.yaml << REGEOF
mirrors:
  docker.io:
    endpoint:
      - "http://${HARBOR}"
  quay.io:
    endpoint:
      - "http://${HARBOR}"
  registry.k8s.io:
    endpoint:
      - "http://${HARBOR}"
  "${HARBOR}":
    endpoint:
      - "http://${HARBOR}"
configs:
  "${HARBOR}":
    tls:
      insecure_skip_verify: true
    auth:
      username: xxx
      password: xxx
REGEOF

# 5. crictl config
cat > /etc/crictl.yaml << 'EOF'
runtime-endpoint: unix:///run/k3s/containerd/containerd.sock
image-endpoint: unix:///run/k3s/containerd/containerd.sock
timeout: 10
EOF

# 6. Verify airgap images exist
ls -lh /var/lib/rancher/rke2/agent/images/ 2>/dev/null || echo "WARNING: No airgap images found!"

# 7. Enable and start rke2-server
systemctl enable rke2-server
systemctl start --no-block rke2-server
echo "=== $NODE_NAME joining cluster ==="

首节点 config.yaml (无 server/token 字段):

yaml

write-kubeconfig-mode: "0644"
tls-san:
  - szb122032
  - szb122033
  - szb122034
  - gpu.ai-ear.cn
  - 192.168.122.32
  - 192.168.122.33
  - 192.168.122.34
cni:
  - calico
kube-proxy-arg:
  - proxy-mode=ipvs
disable-scheduler-node-taints: true
etcd-expose-metrics: true
cluster-cidr: 10.12.0.0/16
service-cidr: 10.13.0.0/16
service-node-port-range: 30000-60000

2.6 VIP 管理器脚本

文件: /tmp/vip-manager.sh — 轻量级 VIP failover，无需 keepalived。

bash

#!/bin/bash
CONF="/etc/vip-manager.conf"
[ -f "$CONF" ] && source "$CONF"

VIP="${VIP:-192.168.122.240}"
IFACE="${IFACE:-eth0}"
MY_PRIO="${MY_PRIO:-100}"
PEERS="${PEERS:-}"
CHECK_INTERVAL="${CHECK_INTERVAL:-3}"

has_vip() { ip addr show "$IFACE" | grep -q "${VIP}/"; }
add_vip() {
    ip addr add "${VIP}/24" dev "$IFACE" 2>/dev/null
    arping -U -c 3 -I "$IFACE" "$VIP" 2>/dev/null
    logger -t vip-manager "VIP ${VIP} claimed (priority ${MY_PRIO})"
}
remove_vip() {
    ip addr del "${VIP}/24" dev "$IFACE" 2>/dev/null
    logger -t vip-manager "VIP ${VIP} released"
}
peer_alive() { ping -c 1 -W 1 "$1" &>/dev/null; }
rke2_healthy() { systemctl is-active rke2-server &>/dev/null; }

cleanup() { has_vip && remove_vip; exit 0; }
trap cleanup SIGTERM SIGINT

logger -t vip-manager "Starting (prio=${MY_PRIO}, vip=${VIP}, peers=${PEERS})"

while true; do
    if ! rke2_healthy; then
        has_vip && remove_vip
        sleep "$CHECK_INTERVAL"
        continue
    fi

    higher_prio_alive=false
    for peer in $PEERS; do
        peer_prio=$(echo "$peer" | cut -d: -f2)
        peer_ip=$(echo "$peer" | cut -d: -f1)
        if [ "$peer_prio" -gt "$MY_PRIO" ] 2>/dev/null && peer_alive "$peer_ip"; then
            higher_prio_alive=true
            break
        fi
    done

    if $higher_prio_alive; then
        has_vip && remove_vip
    else
        if ! has_vip; then
            sleep 1
            still_no_higher=true
            for peer in $PEERS; do
                peer_prio=$(echo "$peer" | cut -d: -f2)
                peer_ip=$(echo "$peer" | cut -d: -f1)
                if [ "$peer_prio" -gt "$MY_PRIO" ] 2>/dev/null && peer_alive "$peer_ip"; then
                    still_no_higher=false
                    break
                fi
            done
            $still_no_higher && add_vip
        else
            [ $((RANDOM % 10)) -eq 0 ] && arping -U -c 1 -I "$IFACE" "$VIP" 2>/dev/null
        fi
    fi
    sleep "$CHECK_INTERVAL"
done

各节点配置文件 /etc/vip-manager.conf:

bash

# szb122032 (最高优先级)
VIP=192.168.122.240
IFACE=eth0
MY_PRIO=100
PEERS="192.168.122.33:90 192.168.122.34:80"

# szb122033
VIP=192.168.122.240
IFACE=eth0
MY_PRIO=90
PEERS="192.168.122.32:100 192.168.122.34:80"

# szb122034
VIP=192.168.122.240
IFACE=eth0
MY_PRIO=80
PEERS="192.168.122.32:100 192.168.122.33:90"

systemd unit /etc/systemd/system/vip-manager.service:

ini

[Unit]
Description=Lightweight VIP Failover Manager
After=network-online.target rke2-server.service

[Service]
Type=simple
ExecStart=/usr/local/bin/vip-manager.sh
Restart=always
RestartSec=3

[Install]
WantedBy=multi-user.target

3. 操作步骤记录

3.1 首节点启动 (szb122032)

bash

# === 在 hypervisor 上启动 HTTP 文件服务器 ===
cd /u02/repo
nohup python3 -m http.server 31001 --bind 0.0.0.0 > /tmp/repo-server.log 2>&1 &

# === SSH 到 szb122032 ===
ssh rocky@192.168.122.32

# 1. 执行节点初始化
sudo bash /tmp/rke2_init_node.sh 32

# 2. 下载 airgap 镜像
sudo bash /tmp/download_images.sh

# 3. 写入首节点 config.yaml (无 server/token)
sudo mkdir -p /etc/rancher/rke2
sudo tee /etc/rancher/rke2/config.yaml << 'EOF'
write-kubeconfig-mode: "0644"
tls-san:
  - szb122032
  - szb122033
  - szb122034
  - gpu.ai-ear.cn
  - 192.168.122.32
  - 192.168.122.33
  - 192.168.122.34
cni:
  - calico
kube-proxy-arg:
  - proxy-mode=ipvs
disable-scheduler-node-taints: true
etcd-expose-metrics: true
cluster-cidr: 10.12.0.0/16
service-cidr: 10.13.0.0/16
service-node-port-range: 30000-60000
EOF

# 4. 启动 RKE2
sudo systemctl enable rke2-server
sudo systemctl start rke2-server

# 5. 等待就绪 (约 2-3 分钟，镜像导入需要时间)
for i in $(seq 1 30); do
  sleep 15
  STATUS=$(systemctl is-active rke2-server)
  echo "$(date): rke2=$STATUS"
  if [ "$STATUS" = "active" ]; then
    echo "RKE2 is active!"
    break
  fi
done

# 6. 获取 join token (供后续节点使用)
sudo cat /var/lib/rancher/rke2/server/node-token
# 输出: fd12102bac7f6d0f7e47b89623a0f4866c7e064e5ee75d4e8bedd7d1c413c3dc

# 7. 验证节点
sudo /var/lib/rancher/rke2/bin/kubectl \
  --kubeconfig /etc/rancher/rke2/rke2.yaml get nodes

3.2 加入第二/三节点

bash

# === 在 szb122033 和 szb122034 上分别执行 ===

# 1. 节点初始化 + 下载镜像
ssh rocky@192.168.122.33
sudo bash /tmp/rke2_init_node.sh 33
sudo bash /tmp/download_images.sh

ssh rocky@192.168.122.34
sudo bash /tmp/rke2_init_node.sh 34
sudo bash /tmp/download_images.sh

# 2. 加入集群
sudo bash /tmp/join_server.sh 33   # szb122033
sudo bash /tmp/join_server.sh 34   # szb122034

# 3. 等待所有节点 Ready (监控脚本)
for i in $(seq 1 30); do
  sleep 15
  READY=$(ssh -o StrictHostKeyChecking=no rocky@192.168.122.32 \
    'sudo /var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml get nodes --no-headers --request-timeout=5s 2>/dev/null | grep -c Ready')
  echo "$(date): Ready=$READY Total=3"
  if [ "$READY" = "3" ]; then
    echo "ALL 3 NODES READY!"
    break
  fi
done

3.3 磁盘扩容

VM 原始磁盘 20GB 不够用（RKE2 + 镜像约 15GB），导致 disk-pressure taint。通过 virsh blockresize 扩容到 40GB：

bash

# === 在 hypervisor 上执行 ===

# 1. 查找 VM 磁盘路径
virsh domblklist szb122032
# 输出: vda  /var/lib/libvirt/images/szb122032.qcow2

# 2. 扩容 (在线，无需关机)
for vm in szb122032 szb122033 szb122034; do
  virsh blockresize $vm vda 40G
done

# 3. 在 VM 内部扩展文件系统
for ip in 192.168.122.32 192.168.122.33 192.168.122.34; do
  ssh rocky@$ip 'sudo bash -c "
    growpart /dev/vda 2 2>/dev/null || true
    xfs_growfs / 2>/dev/null || resize2fs /dev/vda2 2>/dev/null || true
    df -h /
  "'
done

3.4 cert-manager 安装

bash

# 传输 chart 到 szb122032
scp cert-manager-v1.20.3.tgz rocky@192.168.122.32:/tmp/

# 安装
ssh rocky@192.168.122.32

sudo /var/lib/rancher/rke2/bin/helm upgrade --install cert-manager \
  /tmp/cert-manager-v1.20.3.tgz \
  --namespace cert-manager --create-namespace \
  --kubeconfig /etc/rancher/rke2/rke2.yaml \
  --set installCRDs=true \
  --wait --timeout 5m

# 验证
sudo /var/lib/rancher/rke2/bin/kubectl \
  --kubeconfig /etc/rancher/rke2/rke2.yaml \
  get pods -n cert-manager

# 创建自签名 ClusterIssuer
sudo /var/lib/rancher/rke2/bin/kubectl \
  --kubeconfig /etc/rancher/rke2/rke2.yaml \
  apply -f - << 'EOF'
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: selfsigned-issuer
spec:
  selfSigned: {}
EOF

3.5 Rancher Helm 安装

bash

# 传输 chart 到 szb122032
scp rancher-2.14.3.tgz rocky@192.168.122.32:/tmp/

# 安装 Rancher
ssh rocky@192.168.122.32

sudo /var/lib/rancher/rke2/bin/helm upgrade --install rancher \
  /tmp/rancher-2.14.3.tgz \
  --namespace cattle-system --create-namespace \
  --kubeconfig /etc/rancher/rke2/rke2.yaml \
  --set hostname=gpu.ai-ear.cn \
  --set bootstrapPassword='xxx' \
  --set ingress.tls.source=secret \
  --set replicas=1 \
  --set global.cattle.psp.enabled=false \
  --set systemDefaultRegistry=192.168.122.156:30000 \
  --wait --timeout 10m

# 验证
sudo /var/lib/rancher/rke2/bin/kubectl \
  --kubeconfig /etc/rancher/rke2/rke2.yaml \
  get pods -n cattle-system

sudo /var/lib/rancher/rke2/bin/kubectl \
  --kubeconfig /etc/rancher/rke2/rke2.yaml \
  get ingress -n cattle-system

3.6 libvirt dnsmasq DNS

bash

# === 在 hypervisor (192.168.122.1) 上执行 ===

# 编辑 libvirt 网络 DNS
virsh net-edit default

# 在 <dns> 块中添加:
#   <host ip='192.168.122.32'>
#     <hostname>gpu.ai-ear.cn</hostname>
#   </host>
#   <host ip='192.168.122.33'>
#     <hostname>gpu.ai-ear.cn</hostname>
#   </host>
#   <host ip='192.168.122.34'>
#     <hostname>gpu.ai-ear.cn</hostname>
#   </host>

# 重启 dnsmasq
virsh net-destroy default
virsh net-start default

# 验证 DNS 解析
dig gpu.ai-ear.cn @192.168.122.1 +short
# 期望: 192.168.122.32 192.168.122.33 192.168.122.34 (round-robin)

4. 故障排查与修复记录

4.1 IPVS 内核模块加载失败

现象: kube-proxy Pod CrashLoopBackOff，日志报 can't open ip_vs_rr: module not found

根因: /etc/modules-load.d/k8s.conf 只写了 overlay/br_netfilter，未包含 IPVS 模块。

修复: 在 join_server.sh 中显式加载所有 IPVS 模块并写入持久配置:

bash

for mod in ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh ip_vs_sed ip_vs_nq ip_vs_wlc \
           ip_vs_dh ip_vs_lblc ip_vs_lblcr ip_vs_lc ip_vs_fo ip_vs_ovf \
           nf_conntrack br_netfilter overlay; do
  modprobe $mod 2>/dev/null || true
done

4.2 节点磁盘 100% 满

现象: 节点报 DiskPressure taint，Pod 被 Evicted，df -h 显示根分区 100%。

根因: VM 原始磁盘 20GB，RKE2 安装 + airgap 镜像 (~1.4GB) + 容器层 (~5GB) 耗尽空间。

修复:

bash

# 在 hypervisor 上在线扩容 (无需关机)
virsh blockresize szb122032 vda 40G
virsh blockresize szb122033 vda 40G
virsh blockresize szb122034 vda 40G

# 在 VM 内部扩展 XFS
ssh rocky@192.168.122.32 'sudo growpart /dev/vda 2 && sudo xfs_growfs /'
ssh rocky@192.168.122.33 'sudo growpart /dev/vda 2 && sudo xfs_growfs /'
ssh rocky@192.168.122.34 'sudo growpart /dev/vda 2 && sudo xfs_growfs /'

4.3 containerd 快照损坏

现象: calico-node Pod 在 szb122034 上持续 FailedCreatePodSandBox，错误信息:

failed to stat parent: stat /var/lib/rancher/rke2/agent/containerd/
io.containerd.snapshotter.v1.overlayfs/snapshots/1/fs: no such file or directory

根因: containerd overlayfs 快照元数据与实际文件系统不一致（可能由于磁盘满导致写入中断）。

修复:

bash

ssh rocky@192.168.122.34 'sudo bash -c "
  systemctl stop rke2-server
  sleep 5
  pkill -9 -f containerd 2>/dev/null
  sleep 2
  rm -rf /var/lib/rancher/rke2/agent/containerd
  rm -rf /run/k3s/containerd
  rm -rf /var/lib/rancher/rke2/agent/etc/containerd

  # 重新下载 airgap 镜像
  mkdir -p /var/lib/rancher/rke2/agent/images
  curl -sfL -o /var/lib/rancher/rke2/agent/images/rke2-images.linux-amd64.tar.zst \\
    http://192.168.122.1:31001/rke2/rke2-1.35.6/rke2-images.linux-amd64.tar.zst
  curl -sfL -o /var/lib/rancher/rke2/agent/images/rke2-images-calico.linux-amd64.tar.zst \\
    http://192.168.122.1:31001/rke2/rke2-1.35.6/rke2-images-calico.linux-amd64.tar.zst

  systemctl start --no-block rke2-server
"'

4.4 僵尸进程占用端口（关键问题）

现象: systemctl restart rke2-server 后，kube-apiserver 和 cloud-controller-manager 在 szb122034 上 CrashLoopBackOff。

日志:

kube-apiserver: failed to listen on 0.0.0.0:6443: bind: address already in use
cloud-controller-manager: failed to listen on 127.0.0.1:10258: bind: address already in use

根因: systemctl restart rke2-server 未能完全终止旧的 containerd-shim 管理的进程。旧的 kube-apiserver (PID 90941)、etcd (PID 90726)、cloud-controller-manager (PID 90978)、kube-scheduler (PID 90514/94476)、kube-controller-manager (PID 91015/95159) 仍以孤儿进程存在，占用端口 6443、2379、10258、10259、10257。

修复:

bash

# 1. 查找僵尸进程
ssh rocky@192.168.122.34 \
  'sudo ps -eo pid,ppid,start,cmd | grep -E "kube-|cloud-|etcd|scheduler" | grep -v grep | grep -v containerd-shim'

# 2. 杀掉所有僵尸进程
ssh rocky@192.168.122.34 \
  'sudo kill -9 90941 90726 90648 90978 90514 91015 94476 95159'

# 3. 或使用 fuser 释放端口
sudo fuser -k 6443/tcp
sudo fuser -k 10258/tcp

# 4. 重启 rke2-server
sudo systemctl restart rke2-server

# 5. 删除 CrashLoopBackOff 的 Pod 强制重新调度
sudo /var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml \
  delete pod -n kube-system cloud-controller-manager-szb122034

排查命令汇总:

bash

# 查看端口占用
sudo ss -tlnp | grep -E "6443|2379|10257|10258|10259"

# 查看 containerd 容器状态
sudo /var/lib/rancher/rke2/bin/crictl \
  --runtime-endpoint unix:///run/k3s/containerd/containerd.sock \
  ps -a --name kube-apiserver

# 查看容器日志
sudo /var/lib/rancher/rke2/bin/crictl \
  --runtime-endpoint unix:///run/k3s/containerd/containerd.sock \
  logs --tail 50 <container-id>

# 查看所有 RKE2 相关进程
sudo ps -eo pid,ppid,start,cmd | grep -E "kube-|cloud-|etcd|scheduler" | grep -v grep

4.5 Calico Pod 无法启动 (node 34)

现象: 删除旧的 calico-node Pod 后，新 Pod 仍然无法启动，持续报 FailedCreatePodSandBox。

根因: containerd overlayfs 快照层损坏（见 §4.3），仅删除 Pod 不够，需要完全重置 containerd 状态目录。

修复: 执行 §4.3 的完整修复流程。修复后 calico-node Pod 正常启动：

calico-node-8dfdz   1/1  Running  192.168.122.32  szb122032
calico-node-qcvsx   1/1  Running  192.168.122.34  szb122034
calico-node-rn49d   1/1  Running  192.168.122.33  szb122033

4.6 Rancher Settings 气隙配置 (2026-08-15)

问题: Rancher 默认会每 1440 分钟从外网 (releases.rancher.com) 拉取 KDM (Kontainer Driver Metadata) 更新版本列表，默认创建集群版本为 v1.35.7+rke2r1，但 Harbor 中只有 v1.35.6+rke2r1 的完整镜像。

修改的 Settings:

Setting	旧值	新值	说明
`rke-metadata-config`	`{"refresh-interval-minutes":"1440","url":"https://releases.rancher.com/..."}`	`{"branch":"dev-v2.14","url":"","refresh-interval-minutes":"0"}`	禁用 KDM 自动更新
`rke2-default-version`	`v1.35.7+rke2r1`	`v1.35.6+rke2r1`	锁定为 Harbor 已有的版本
`k3s-default-version`	`v1.35.7+k3s1`	`v1.35.6+k3s1`	锁定为 Harbor 已有的版本
`ui-offline-preferred`	`dynamic` (空值)	`true`	强制使用内嵌 UI，不访问外网
`system-catalog`	`external` (空值)	`bundled`	使用内嵌 Helm chart catalog

修改命令:

bash

# 从远程使用 kubectl + kubeconfig 修改 Settings
export KUBECONFIG=/tmp/kubeconfig-rke2.yaml

# 1. 关闭 KDM 自动更新
kubectl patch settings.management.cattle.io rke-metadata-config \
  --type merge -p '{"value":"{\"branch\":\"dev-v2.14\",\"url\":\"\",\"refresh-interval-minutes\":\"0\"}"}'

# 2. 锁定 RKE2 默认版本
kubectl patch settings.management.cattle.io rke2-default-version \
  --type merge -p '{"value":"v1.35.6+rke2r1"}'

# 3. 锁定 K3s 默认版本
kubectl patch settings.management.cattle.io k3s-default-version \
  --type merge -p '{"value":"v1.35.6+k3s1"}'

# 4. UI 离线模式
kubectl patch settings.management.cattle.io ui-offline-preferred \
  --type merge -p '{"value":"true"}'

# 5. System Catalog 内嵌
kubectl patch settings.management.cattle.io system-catalog \
  --type merge -p '{"value":"bundled"}'

# 重启 Rancher 使其加载新配置
kubectl rollout restart deploy/rancher -n cattle-system
kubectl rollout status deploy/rancher -n cattle-system --timeout=120s

验证:

bash

# 检查 Settings
kubectl get settings.management.cattle.io rke-metadata-config -o jsonpath='{.value}'
kubectl get settings.management.cattle.io rke2-default-version -o jsonpath='{.value}'

# 检查 Rancher Pod 状态
kubectl get pods -n cattle-system -l app=rancher

Harbor 中 v1.35.6+rke2r1 相关镜像:

镜像	Tags
`rancher/system-agent-installer-rke2`	`v1.35.6-rke2r1`, `v1.35.6-rke2r1-linux-amd64`
`rancher/rke2-runtime`	`v1.35.6-rke2r1`, `v1.35.6-rke2r1-linux-amd64`
`rancher/rke2-upgrade`	`v1.35.6-rke2r1`, `v1.35.6-rke2r1-linux-amd64`
`rancher/rke2-cloud-provider`	`v1.35.6-0.20260610221957-158346759a70-build20260710`

4.7 Rancher cacerts 为空导致 system-agent 安装失败 (2026-08-15)

问题: 在新节点 (如 192.168.122.37) 上执行 Rancher 生成的 system-agent 安装命令时报错：

[FATAL]  Aborting system-agent installation due to requested strict CA verification with no CA checksum provided

根因: Rancher 使用 cert-manager 自签名证书 (ingress.tls.source=secret) 安装时，cacerts Setting 为空。而 agent-tls-mode 默认为 strict，安装脚本需要从 /v3/settings/cacerts 获取 CA 证书并计算 SHA256 checksum，但 cacerts 为空导致无法获取 checksum，安装中止。

修复: 从 tls-rancher-ingress Secret 中提取 ca.crt，写入 cacerts Setting：

bash

# 1. 获取 CA 证书
kubectl get secret tls-rancher-ingress -n cattle-system \
  -o jsonpath='{.data.ca\.crt}' | base64 -d > /tmp/rancher-ca.crt

# 2. 将 CA 证书写入 cacerts Setting
CA_CERT=$(cat /tmp/rancher-ca.crt)
kubectl patch settings.management.cattle.io cacerts \
  --type merge -p "{\"value\":\"$CA_CERT\"}"

# 3. 验证 (安装脚本中应包含 CA checksum)
curl -sk https://gpu.ai-ear.cn/system-agent-install.sh | grep CATTLE_CA_CHECKSUM
# 期望: CATTLE_CA_CHECKSUM="cd9078b7f2b7e605ab8d6fd8629d8de8dda9efbd5fd6fa5455f634c769bec19a"

# 4. 重新执行安装命令 (无需额外参数，脚本已自动包含 checksum)
curl --insecure -fL https://gpu.ai-ear.cn/system-agent-install.sh | sudo sh -s - \
  --server https://gpu.ai-ear.cn \
  --label 'cattle.io/os=linux' \
  --token <token> \
  --etcd --controlplane --worker

验证: 安装脚本现在自动包含 CA checksum，节点可正常加入集群。

5. 管理命令速查

bash

# === kubectl 命令 (在任意节点上) ===
export KUBECONFIG=/etc/rancher/rke2/rke2.yaml
alias k='sudo /var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml'

# 集群状态
k get nodes -o wide
k get pods -A
k get pods -n kube-system -l tier=control-plane
k get pods -n calico-system
k get pods -n cattle-system
k get pods -n cert-manager

# 节点详情
k describe node szb122034

# Pod 日志
k logs -n kube-system kube-apiserver-szb122034
k logs -n cattle-system rancher-76f495475c-px2qv

# crictl 命令 (底层容器管理)
sudo /var/lib/rancher/rke2/bin/crictl \
  --runtime-endpoint unix:///run/k3s/containerd/containerd.sock ps -a

# RKE2 服务
sudo systemctl status rke2-server
sudo systemctl restart rke2-server
sudo journalctl -u rke2-server --no-pager -n 50

# etcd 健康检查
sudo /var/lib/rancher/rke2/bin/etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt \
  --cert=/var/lib/rancher/rke2/server/tls/etcd/client.crt \
  --key=/var/lib/rancher/rke2/server/tls/etcd/client.key \
  endpoint health

# Rancher bootstrap 密码
k get secret --namespace cattle-system bootstrap-secret \
  -o jsonpath='{.data.bootstrapPassword}' | base64 -d

# Ingress 信息
k get ingress -n cattle-system
k get svc -n kube-system rke2-ingress-nginx-controller

# 清理 Evicted/Failed pods
k get pods -A --field-selector=status.phase=Failed -o name | xargs k delete --force
k get pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded \
  -o name | xargs k delete --force --grace-period=0

6. 最终集群状态

节点

NAME        STATUS   ROLES                AGE   VERSION          INTERNAL-IP      OS-IMAGE                    KERNEL                    CONTAINER-RUNTIME
szb122032   Ready    control-plane,etcd   66m   v1.35.6+rke2r1   192.168.122.32   Rocky Linux 9.8 (Blue Onyx) 5.14.0-687.10.1.el9_8   containerd://2.2.5-k3s2
szb122033   Ready    control-plane,etcd   61m   v1.35.6+rke2r1   192.168.122.33   Rocky Linux 9.8 (Blue Onyx) 5.14.0-687.10.1.el9_8   containerd://2.2.5-k3s2
szb122034   Ready    control-plane,etcd   60m   v1.35.6+rke2r1   192.168.122.34   Rocky Linux 9.8 (Blue Onyx) 5.14.0-687.10.1.el9_8   containerd://2.2.5-k3s2

关键 Pod (全部 Running)

kube-system/
  ├── kube-apiserver-szb122032/33/34          ✅ Running
  ├── etcd-szb122032/33/34                   ✅ Running
  ├── kube-scheduler-szb122032/33/34         ✅ Running
  ├── kube-controller-manager-szb122032/33/34 ✅ Running
  ├── cloud-controller-manager-szb122032/33/34 ✅ Running
  ├── kube-proxy-szb122032/33/34             ✅ Running
  ├── rke2-coredns-*                          ✅ Running (2 replicas + autoscaler)
  ├── rke2-ingress-nginx-controller-*         ✅ Running (3 replicas)
  └── rke2-metrics-server-*                   ✅ Running

calico-system/
  ├── calico-node-*                           ✅ Running (3 nodes)
  ├── calico-kube-controllers-*               ✅ Running
  └── calico-typha-*                          ✅ Running (2 replicas)

cert-manager/
  ├── cert-manager-*                          ✅ Running
  ├── cert-manager-cainjector-*               ✅ Running
  └── cert-manager-webhook-*                  ✅ Running

cattle-system/
  ├── rancher-*                               ✅ Running (v2.14.3)
  ├── rancher-webhook-*                       ✅ Running
  └── system-upgrade-controller-*             ✅ Running

tigera-operator/
  └── tigera-operator-*                       ✅ Running

Rancher 访问

项目	值
URL	`https://gpu.ai-ear.cn`
Ingress Hosts	`gpu.ai-ear.cn`
Ingress Address	192.168.122.32, 192.168.122.33
Bootstrap Password	`xxx`
TLS	self-signed (cert-manager ClusterIssuer)

7. kubeconfig

文件: /tmp/kubeconfig-rke2.yaml

从远程机器使用 kubectl 访问集群：

bash

# 复制 kubeconfig
cp /tmp/kubeconfig-rke2.yaml ~/.kube/config-rke2
export KUBECONFIG=~/.kube/config-rke2

# 验证
kubectl get nodes

完整 kubeconfig 内容:

yaml

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: <base64-encoded-CA-cert>
    server: https://192.168.122.32:6443
  name: default
contexts:
- context:
    cluster: default
    user: default
  name: default
current-context: default
kind: Config
users:
- name: default
  user:
    client-certificate-data: <base64-encoded-client-cert>
    client-key-data: <base64-encoded-client-key>

注意: 完整内容见 /tmp/kubeconfig-rke2.yaml，包含实际的 base64 编码证书和密钥。

8. 经验教训

systemctl restart 不干净: RKE2 使用 containerd 管理容器，systemctl restart rke2-server 可能不会完全终止旧的 containerd-shim 及其子进程。重启前必须手动 kill 残留进程，或使用 systemctl stop + 等待 + systemctl start。
IPVS 模块必须显式配置: kube-proxy IPVS 模式要求所有 ip_vs_* 内核模块预加载。仅在 /etc/modules-load.d/k8s.conf 中写 ip_vs 不够，需逐个列出所有调度算法模块。
磁盘空间监控: 20GB 磁盘对于 RKE2 + airgap 镜像不够用。部署前确认至少 40GB。df -h / 应在每个关键步骤后检查。
containerd 快照损坏修复: 当 overlayfs 快照元数据损坏时，仅删除 Pod 无效。需完全清除 /var/lib/rancher/rke2/agent/containerd/ 目录并重新导入 airgap 镜像。
airgap 镜像导入耗时: RKE2 启动时自动从 tar.zst 导入镜像到 containerd，768MB + 574MB 的导入约需 2-3 分钟，期间 rke2-server 状态为 activating。不要误认为启动失败。
etcd 对 I/O 敏感: etcd wal_fsync 和 backend_commit 延迟超过阈值会触发 leader election。在资源紧张环境中，避免在 etcd 节点上执行大量 I/O 操作。
端口冲突排查顺序: 当组件 CrashLoopBackOff 时，首先检查端口占用 (ss -tlnp)，然后查看容器日志 (crictl logs)，最后查看系统日志 (journalctl)。
HTTP 文件服务器: 气隙环境中，Python http.server 是分发离线包的最简方案。部署完成后务必停止，避免安全风险。

9. 文件清单

文件	路径	节点	说明
RKE2 配置 (首节点)	`/etc/rancher/rke2/config.yaml`	.32	无 server/token 字段
RKE2 配置 (其他节点)	`/etc/rancher/rke2/config.yaml`	.33, .34	含 server/token
镜像仓库配置	`/etc/rancher/rke2/registries.yaml`	全部	Harbor mirror 配置
Airgap 镜像 (核心)	`/var/lib/rancher/rke2/agent/images/rke2-images.linux-amd64.tar.zst`	全部	768MB
Airgap 镜像 (Calico)	`/var/lib/rancher/rke2/agent/images/rke2-images-calico.linux-amd64.tar.zst`	全部	574MB
crictl 配置	`/etc/crictl.yaml`	全部	containerd socket
内核模块配置	`/etc/modules-load.d/k8s.conf`	全部	overlay + IPVS 模块
内核参数配置	`/etc/sysctl.d/99-k8s.conf`	全部	网络转发 + inotify
VIP 管理器脚本	`/usr/local/bin/vip-manager.sh`	全部	零依赖 VIP failover
VIP 管理器配置	`/etc/vip-manager.conf`	全部	各节点优先级不同
VIP systemd unit	`/etc/systemd/system/vip-manager.service`	全部	systemd service
kubeconfig	`/etc/rancher/rke2/rke2.yaml`	全部	集群访问凭证
kubeconfig (远程)	`/tmp/kubeconfig-rke2.yaml`	宿主机	远程 kubectl 使用
部署脚本	`/tmp/download_images.sh`	宿主机	下载 airgap 镜像
部署脚本	`/tmp/rke2_init_node.sh`	宿主机	节点初始化
部署脚本	`/tmp/setup_rke2_nodes.sh`	宿主机	环境配置
部署脚本	`/tmp/join_server.sh`	宿主机	加入集群
部署脚本	`/tmp/vip-manager.sh`	宿主机	VIP 管理器
部署文档	本文件	-	操作记录与恢复手册

10. 公网访问 (FRP + Traefik)

架构

用户浏览器 → gpu.ai-ear.cn (DNS→8.153.84.140)
  443 端口:  → Traefik websecure (TCP/SNI passthrough) ──┐
  16444 端口: → Traefik ep16444 (TCP/SNI passthrough) ──┤
                                                        ▼
                          → 127.0.0.1:16445 (frps 隧道端口)
                            → FRP tunnel → frpc
                              → 192.168.122.32:443 (RKE2 nginx-ingress)
                                → Rancher Pod

配置变更

10.1 本地 frpc 独立配置

文件: /home/xxx/frpc-gpu.toml

toml

serverAddr = "8.153.84.140"
serverPort = 7000
transport.tcpMux = true
auth.method = "token"
auth.token = "xxx"

[[proxies]]
name = "gpu-rancher-web-16445"
type = "tcp"
localIP = "192.168.122.32"
localPort = 443
remotePort = 16445

注意: 同时也在 /etc/frp/frpc.toml (主配置) 中添加了相同的 proxy 块，下次主 frpc 服务重启后会自动加载。当前由独立 frpc 实例运行。

10.2 公网 Traefik 配置

文件 (公网服务器 8.153.84.140): /etc/traefik/traefik.yaml

新增 entryPoint:

yaml

  ep16444:
    address: ":16444"

文件 (公网服务器 8.153.84.140): /etc/traefik/dynamic.yaml

新增 TCP 路由 (TLS passthrough):

yaml

tcp:
  routers:
    gpu-rancher:
      entryPoints: [websecure, ep16444]
      rule: "HostSNI(\`gpu.ai-ear.cn\`)"
      service: gpu-rancher-svc
      tls:
        passthrough: true

  services:
    gpu-rancher-svc:
      loadBalancer:
        servers:
          - address: "127.0.0.1:16445"

持久化

FRP: crontab @reboot 自动启动

@reboot sleep 10 && nohup /usr/local/bin/frpc -c /home/xxx/frpc-gpu.toml > /home/xxx/frpc-gpu.log 2>&1 &

Traefik: systemd service (公网服务器，已 enable)

访问地址

项目	值
Rancher URL	`https://gpu.ai-ear.cn` (443, 也可用 `:16444`)
Bootstrap 密码	`xxx`
证书	self-signed (CN=gpu.ai-ear.cn, 有效期至 2027-08-15)
DNS	`gpu.ai-ear.cn` → `8.153.84.140` (公网)

RKE2 HA 集群 + Rancher v2.14.3 气隙部署操作手册 ​

目录 ​

1. 环境信息 ​

集群拓扑 ​

基础参数 ​

离线包清单 ​

2. 部署脚本 ​

2.1 临时 HTTP 文件服务器 ​

2.2 下载 airgap 镜像脚本 ​

2.3 节点初始化脚本 ​

2.4 节点环境配置脚本 ​

2.5 加入集群脚本 ​

2.6 VIP 管理器脚本 ​

3. 操作步骤记录 ​

3.1 首节点启动 (szb122032) ​

3.2 加入第二/三节点 ​

3.3 磁盘扩容 ​

3.4 cert-manager 安装 ​

3.5 Rancher Helm 安装 ​

3.6 libvirt dnsmasq DNS ​

4. 故障排查与修复记录 ​

4.1 IPVS 内核模块加载失败 ​

4.2 节点磁盘 100% 满 ​

4.3 containerd 快照损坏 ​

4.4 僵尸进程占用端口（关键问题） ​

4.5 Calico Pod 无法启动 (node 34) ​

4.6 Rancher Settings 气隙配置 (2026-08-15) ​

4.7 Rancher cacerts 为空导致 system-agent 安装失败 (2026-08-15) ​

5. 管理命令速查 ​

6. 最终集群状态 ​

节点 ​

关键 Pod (全部 Running) ​

Rancher 访问 ​

7. kubeconfig ​

8. 经验教训 ​

9. 文件清单 ​

10. 公网访问 (FRP + Traefik) ​

架构 ​

配置变更 ​

10.1 本地 frpc 独立配置 ​

10.2 公网 Traefik 配置 ​

持久化 ​

访问地址 ​

RKE2 HA 集群 + Rancher v2.14.3 气隙部署操作手册

目录

1. 环境信息

集群拓扑

基础参数

离线包清单

2. 部署脚本

2.1 临时 HTTP 文件服务器

2.2 下载 airgap 镜像脚本

2.3 节点初始化脚本

2.4 节点环境配置脚本

2.5 加入集群脚本

2.6 VIP 管理器脚本

3. 操作步骤记录

3.1 首节点启动 (szb122032)

3.2 加入第二/三节点

3.3 磁盘扩容

3.4 cert-manager 安装

3.5 Rancher Helm 安装

3.6 libvirt dnsmasq DNS

4. 故障排查与修复记录

4.1 IPVS 内核模块加载失败

4.2 节点磁盘 100% 满

4.3 containerd 快照损坏

4.4 僵尸进程占用端口（关键问题）

4.5 Calico Pod 无法启动 (node 34)

4.6 Rancher Settings 气隙配置 (2026-08-15)

4.7 Rancher cacerts 为空导致 system-agent 安装失败 (2026-08-15)

5. 管理命令速查

6. 最终集群状态

节点

关键 Pod (全部 Running)

Rancher 访问

7. kubeconfig

8. 经验教训

9. 文件清单

10. 公网访问 (FRP + Traefik)

架构

配置变更

10.1 本地 frpc 独立配置

10.2 公网 Traefik 配置

持久化

访问地址