Rancher v2.14.3 气隙环境部署操作手册

最后更新: 2026-08-15 集群: 3 节点 RKE2 GPU 集群 | Rocky Linux 9.8

环境信息

集群拓扑

                          ┌─────────────────────────────────────────────┐
                          │           Hypervisor (192.168.122.1)        │
                          │  libvirt dnsmasq: rancher.ai-ear.cn         │
                          │  ┌───────────────────────────────────────┐  │
  Internet                │  │          VIP: 192.168.122.240         │  │
    │                     │  │                                        │  │
    ▼                     │  │  ┌──────────┐ ┌──────────┐ ┌──────────┐│  │
 Traefik (公网)           │  │  │ .32      │ │ .33      │ │ .34      ││  │
 SNI passthrough          │  │  │ etcd+CP  │ │ etcd+CP  │ │ etcd+CP  ││  │
    │                     │  │  │ prio=100 │ │ prio=90  │ │ prio=80  ││  │
    ▼                     │  │  │ rke2     │ │ rke2     │ │ rke2     ││  │
 FRP Server               │  │  │          │ │          │ │          ││  │
 :10443                   │  │  └──────────┘ └──────────┘ └──────────┘│  │
    │                     │  │                                        │  │
    ▼                     │  └───────────────────────────────────────┘  │
 FRP Client               │                                             │
 (内网侧)                 │  Harbor: 192.168.122.156:30000              │
                          └─────────────────────────────────────────────┘

基础参数

参数	值
集群节点	192.168.122.32 / .33 / .34
RKE2 版本	v1.35.6+rke2r1 (Kubernetes v1.35.6)
Rancher 版本	v2.14.3
cert-manager	v1.20.3
VIP	192.168.122.240 (自定义 bash 脚本替代 keepalived)
域名	rancher.ai-ear.cn
Pod CIDR	10.12.0.0/16
Service CIDR	10.13.0.0/16
镜像仓库	192.168.122.156:30000 (Harbor, HTTP, 自签名)
kube-proxy	IPVS 模式
CNI	Canal (Calico + Flannel)
OS	Rocky Linux 9.8 (Blue Onyx), kernel 5.14.0-687

部署状态

步骤	组件	状态	备注
§1	RKE2 集群	✅ 已完成	3 节点 control-plane+etcd
§2	VIP 管理器	✅ 已完成	自定义脚本, 无外部依赖
§3	镜像同步	✅ 已完成	Harbor → 3 节点
§4	cert-manager	✅ 已安装	v1.20.3, CRD included
§5	Rancher	✅ Helm deployed	v2.14.3, 3 replicas
§6	dnsmasq DNS	⏳ 待完成	hypervisor round-robin
§7	FRP 配置	⏳ 待完成	localIP → VIP .240
§8	Traefik 路由	⏳ 待完成	SNI passthrough
§9	登录验证	⏳ 待完成	bootstrap password

§0 预检脚本

在每一步操作前后执行以下检查，确保环境正常。

0.1 全局健康检查脚本

将以下脚本保存为 /usr/local/bin/cluster-check.sh，在每个关键步骤后执行：

bash

#!/bin/bash
# cluster-check.sh — RKE2 + Rancher 全局健康检查
set -uo pipefail
RED='\033[0;31m'; GREEN='\033[0;32m'; YELLOW='\033[0;33m'; NC='\033[0m'

NODES="192.168.122.32 192.168.122.33 192.168.122.34"
VIP="192.168.122.240"
DOMAIN="rancher.ai-ear.cn"
HARBOR="192.168.122.156:30000"

pass() { echo -e "  ${GREEN}✓${NC} $1"; }
fail() { echo -e "  ${RED}✗${NC} $1"; }
warn() { echo -e "  ${YELLOW}!${NC} $1"; }

echo "=============================="
echo " 集群健康检查 $(date +%H:%M:%S)"
echo "=============================="

# 1. 网络连通性
echo -e "\n[1/7] 网络连通性"
for node in $NODES; do
    if ping -c 1 -W 1 "$node" &>/dev/null; then pass "$node reachable"
    else fail "$node unreachable"; fi
done
if ping -c 1 -W 1 "$VIP" &>/dev/null; then pass "VIP $VIP reachable"
else warn "VIP $VIP not responding (vip-manager may be stopped)"; fi

# 2. RKE2 服务状态
echo -e "\n[2/7] RKE2 服务状态"
for node in $NODES; do
    status=$(ssh -o ConnectTimeout=5 -o StrictHostKeyChecking=no root@$node \
        'systemctl is-active rke2-server' 2>/dev/null || echo "ssh-failed")
    if [ "$status" = "active" ]; then pass "$node rke2-server $status"
    else fail "$node rke2-server $status"; fi
done

# 3. kube-apiserver 可达
echo -e "\n[3/7] kube-apiserver"
api_result=$(ssh -o ConnectTimeout=5 root@192.168.122.32 \
    'kubectl get nodes --no-headers --request-timeout=5s 2>&1' 2>/dev/null)
if echo "$api_result" | grep -q "control-plane"; then
    node_count=$(echo "$api_result" | wc -l)
    pass "API server responding ($node_count nodes)"
else
    fail "API server: $(echo "$api_result" | head -1)"
fi

# 4. etcd 健康
echo -e "\n[4/7] etcd 健康状态"
etcd_result=$(ssh -o ConnectTimeout=5 root@192.168.122.32 \
    'ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
    --cacert=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt \
    --cert=/var/lib/rancher/rke2/server/tls/etcd/client.crt \
    --key=/var/lib/rancher/rke2/server/tls/etcd/client.key \
    endpoint health 2>&1' 2>/dev/null)
if echo "$etcd_result" | grep -q "healthy"; then pass "etcd endpoint healthy"
else warn "etcd: $(echo "$etcd_result" | head -1)"; fi

# 5. VIP 归属
echo -e "\n[5/7] VIP 归属"
for node in $NODES; do
    has=$(ssh -o ConnectTimeout=5 root@$node \
        'ip -4 addr show eth0 2>/dev/null | grep -c 192.168.122.240' 2>/dev/null)
    if [ "$has" = "1" ]; then pass "VIP held by $node"; fi
done

# 6. Harbor 可达
echo -e "\n[6/7] Harbor 镜像仓库"
if curl -sk --connect-timeout 5 "http://$HARBOR/v2/" &>/dev/null; then
    pass "Harbor $HARBOR reachable"
else
    fail "Harbor $HARBOR unreachable"
fi

# 7. 关键 Pod 状态
echo -e "\n[7/7] 关键 Pod 状态"
for ns in kube-system cert-manager cattle-system; do
    echo "  --- namespace: $ns ---"
    ssh -o ConnectTimeout=5 root@192.168.122.32 \
        "kubectl get pods -n $ns --no-headers --request-timeout=5s 2>/dev/null" | \
        head -5 | while read line; do
        if echo "$line" | grep -qE "Running|Completed"; then
            echo -e "    ${GREEN}✓${NC} $line"
        else
            echo -e "    ${RED}✗${NC} $line"
        fi
    done
done

echo -e "\n=============================="
echo " 检查完成 $(date +%H:%M:%S)"
echo "=============================="

使用方式：

bash

chmod +x /usr/local/bin/cluster-check.sh
cluster-check.sh

0.2 快速单节点检查（一行命令）

bash

# 在任意节点上快速检查核心状态
systemctl is-active rke2-server && \
  kubectl get nodes --no-headers --request-timeout=5s && \
  kubectl get pods -A --no-headers --request-timeout=5s | \
    grep -cE 'Running|Completed' && echo "pods healthy" || echo "ISSUES DETECTED"

§1 RKE2 集群配置

1.1 首节点配置（192.168.122.32）

文件: /etc/rancher/rke2/config.yaml

yaml

node-name: szb122032
node-ip: 192.168.122.32
advertise-address: 192.168.122.32
tls-san:
  - 192.168.122.32
  - 192.168.122.33
  - 192.168.122.34
cluster-cidr: 10.12.0.0/16
service-cidr: 10.13.0.0/16
kube-proxy-arg:
  - "proxy-mode=ipvs"
system-default-registry: 192.168.122.156:30000
etcd-snapshot-retention: 5
etcd-snapshot-schedule-cron: "0 */6 * * *"

启动命令:

bash

mkdir -p /etc/rancher/rke2
# 写入上面的 config.yaml
curl -sfL https://get.rke2.io | INSTALL_RKE2_TYPE="server" sh -
systemctl enable --now rke2-server

气隙环境注意: 需提前下载 RKE2 离线安装包和镜像到本地。

验证首节点:

bash

# 检查服务状态
systemctl is-active rke2-server
# 期望: active

# 检查端口
ss -tlnp | grep -E '6443|9345|2379|10250'
# 期望: 6443 (kube-apiserver), 9345 (rke2), 2379 (etcd), 10250 (kubelet)

# 检查节点
export PATH=$PATH:/var/lib/rancher/rke2/bin
export KUBECONFIG=/etc/rancher/rke2/rke2.yaml
kubectl get nodes
# 期望: szb122032 Ready control-plane,etcd

# 检查系统 Pod
kubectl get pods -n kube-system
# 期望: canal, coredns, metrics-server, ingress-nginx 均 Running

# 获取 join token（供后续节点使用）
cat /var/lib/rancher/rke2/server/node-token

1.2 第二节点配置（192.168.122.33）

文件: /etc/rancher/rke2/config.yaml

yaml

server: https://192.168.122.32:9345
token: <从首节点获取的 token>
node-name: szb122033
node-ip: 192.168.122.33
advertise-address: 192.168.122.33
tls-san:
  - 192.168.122.32
  - 192.168.122.33
  - 192.168.122.34
cluster-cidr: 10.12.0.0/16
service-cidr: 10.13.0.0/16
kube-proxy-arg:
  - "proxy-mode=ipvs"
system-default-registry: 192.168.122.156:30000
etcd-snapshot-retention: 5
etcd-snapshot-schedule-cron: "0 */6 * * *"

启动命令:

bash

mkdir -p /etc/rancher/rke2
# 写入 config.yaml（替换 token）
curl -sfL https://get.rke2.io | INSTALL_RKE2_TYPE="server" sh -
systemctl enable --now rke2-server

1.3 第三节点配置（192.168.122.34）

文件: /etc/rancher/rke2/config.yaml

yaml

server: https://192.168.122.32:9345
token: <从首节点获取的 token>
node-name: szb122034
node-ip: 192.168.122.34
advertise-address: 192.168.122.34
tls-san:
  - 192.168.122.32
  - 192.168.122.33
  - 192.168.122.34
cluster-cidr: 10.12.0.0/16
service-cidr: 10.13.0.0/16
kube-proxy-arg:
  - "proxy-mode=ipvs"
system-default-registry: 192.168.122.156:30000
etcd-snapshot-retention: 5
etcd-snapshot-schedule-cron: "0 */6 * * *"

启动命令: 同上。

1.4 集群验证排查脚本

bash

#!/bin/bash
# rke2-check.sh — RKE2 集群健康排查
echo "=== RKE2 集群排查 ==="

echo -e "\n[1] 服务状态:"
for node in 32 33 34; do
  status=$(ssh -o ConnectTimeout=5 root@192.168.122.$node \
    'systemctl is-active rke2-server' 2>/dev/null)
  echo "  .${node}: $status"
done

echo -e "\n[2] 节点列表:"
kubectl get nodes -o wide

echo -e "\n[3] etcd 成员:"
ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt \
  --cert=/var/lib/rancher/rke2/server/tls/etcd/client.crt \
  --key=/var/lib/rancher/rke2/server/tls/etcd/client.key \
  member list -w table

echo -e "\n[4] 系统 Pod:"
kubectl get pods -n kube-system -o wide

echo -e "\n[5] 事件 (最近 10 条 Warning):"
kubectl get events -A --field-selector type=Warning \
  --sort-by='.lastTimestamp' 2>/dev/null | tail -10

常见问题排查:

症状	排查命令	原因	解决
节点 NotReady	`kubectl describe node <name>`	kubelet 未就绪	`systemctl restart rke2-server`
Pod Pending	`kubectl describe pod <name>`	资源不足/调度约束	检查节点资源
API Server 超时	`ss -tlnp \| grep 6443`	端口被占用或 etcd 慢	等待或重启
etcd 选举失败	`journalctl -u rke2-server -f`	I/O 延迟/网络分区	检查磁盘 I/O 和网络

§2 VIP 管理器

背景

keepalived 在 Rocky 9 气隙环境依赖链过深（libnfnetlink, libnl3, libipset, iptables-nft 等十余个 RPM），无法离线安装。本方案使用纯 bash + systemd 实现零依赖 VIP 故障转移。

依赖: 仅需 ip（iproute2）、arping（iputils）、ping（iputils）— 均为系统自带。

2.1 VIP 管理器脚本

文件: /usr/local/bin/vip-manager.sh（三节点相同）

bash

#!/bin/bash
# Lightweight VIP failover for RKE2 clusters
# No dependencies beyond: ip, arping, ping, systemd

CONF="/etc/vip-manager.conf"
[ -f "$CONF" ] && source "$CONF"

VIP="${VIP:-192.168.122.240}"
IFACE="${IFACE:-eth0}"
MY_PRIO="${MY_PRIO:-100}"
PEERS="${PEERS:-}"
CHECK_INTERVAL="${CHECK_INTERVAL:-3}"
FAIL_COUNT="${FAIL_COUNT:-2}"

has_vip() { ip addr show "$IFACE" | grep -q "${VIP}/"; }
add_vip() {
    ip addr add "${VIP}/24" dev "$IFACE" 2>/dev/null
    arping -U -c 3 -I "$IFACE" "$VIP" 2>/dev/null
    logger -t vip-manager "VIP ${VIP} claimed (priority ${MY_PRIO})"
}
remove_vip() {
    ip addr del "${VIP}/24" dev "$IFACE" 2>/dev/null
    logger -t vip-manager "VIP ${VIP} released"
}

peer_alive() {
    ping -c 1 -W 1 "$1" &>/dev/null
}

rke2_healthy() {
    systemctl is-active rke2-server &>/dev/null
}

cleanup() {
    has_vip && remove_vip
    exit 0
}
trap cleanup SIGTERM SIGINT

logger -t vip-manager "Starting (prio=${MY_PRIO}, vip=${VIP}, peers=${PEERS})"

consecutive_fail=0

while true; do
    if ! rke2_healthy; then
        has_vip && remove_vip
        consecutive_fail=0
        sleep "$CHECK_INTERVAL"
        continue
    fi

    higher_prio_alive=false
    for peer in $PEERS; do
        peer_prio=$(echo "$peer" | cut -d: -f2)
        peer_ip=$(echo "$peer" | cut -d: -f1)
        if [ "$peer_prio" -gt "$MY_PRIO" ] 2>/dev/null && peer_alive "$peer_ip"; then
            higher_prio_alive=true
            break
        fi
    done

    if $higher_prio_alive; then
        has_vip && remove_vip
        consecutive_fail=0
    else
        consecutive_fail=0
        if ! has_vip; then
            sleep 1  # brief delay to avoid VIP race
            still_no_higher=true
            for peer in $PEERS; do
                peer_prio=$(echo "$peer" | cut -d: -f2)
                peer_ip=$(echo "$peer" | cut -d: -f1)
                if [ "$peer_prio" -gt "$MY_PRIO" ] 2>/dev/null && peer_alive "$peer_ip"; then
                    still_no_higher=false
                    break
                fi
            done
            $still_no_higher && add_vip
        else
            # Re-send gratuitous ARP periodically (every ~30s)
            if [ $((RANDOM % 10)) -eq 0 ]; then
                arping -U -c 1 -I "$IFACE" "$VIP" 2>/dev/null
            fi
        fi
    fi

    sleep "$CHECK_INTERVAL"
done

2.2 各节点配置文件

每个节点的优先级不同，配置各异：

192.168.122.32 — /etc/vip-manager.conf（最高优先级，默认 master）:

ini

VIP=192.168.122.240
IFACE=eth0
MY_PRIO=100
PEERS="192.168.122.33:90 192.168.122.34:80"
CHECK_INTERVAL=3

192.168.122.33 — /etc/vip-manager.conf:

ini

VIP=192.168.122.240
IFACE=eth0
MY_PRIO=90
PEERS="192.168.122.32:100 192.168.122.34:80"
CHECK_INTERVAL=3

192.168.122.34 — /etc/vip-manager.conf:

ini

VIP=192.168.122.240
IFACE=eth0
MY_PRIO=80
PEERS="192.168.122.32:100 192.168.122.33:90"
CHECK_INTERVAL=3

2.3 systemd unit 文件

文件: /etc/systemd/system/vip-manager.service（三节点相同）

ini

[Unit]
Description=VIP Failover Manager for RKE2
After=network-online.target rke2-server.service
Wants=network-online.target

[Service]
Type=simple
ExecStart=/usr/local/bin/vip-manager.sh
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

2.4 部署步骤

bash

# 在每个节点上执行（.32, .33, .34）
for node in 32 33 34; do
  echo "=== Deploying to .${node} ==="
  
  # 1. 写入脚本
  ssh root@192.168.122.$node 'cat > /usr/local/bin/vip-manager.sh' < vip-manager.sh
  ssh root@192.168.122.$node 'chmod +x /usr/local/bin/vip-manager.sh'
  
  # 2. 写入配置（根据节点调整 MY_PRIO 和 PEERS）
  # 上面已给出各节点的 /etc/vip-manager.conf 内容
  
  # 3. 写入 systemd unit
  ssh root@192.168.122.$node 'cat > /etc/systemd/system/vip-manager.service' < vip-manager.service
  
  # 4. 启用并启动
  ssh root@192.168.122.$node 'systemctl daemon-reload && systemctl enable --now vip-manager'
done

2.5 VIP 排查脚本

bash

#!/bin/bash
# vip-check.sh — VIP 故障转移排查
VIP="192.168.122.240"
NODES="192.168.122.32 192.168.122.33 192.168.122.34"

echo "=== VIP 排查 ==="

echo -e "\n[1] VIP 连通性:"
if ping -c 2 -W 2 "$VIP" &>/dev/null; then
  echo "  ✓ VIP $VIP reachable"
else
  echo "  ✗ VIP $VIP unreachable"
fi

echo -e "\n[2] VIP 归属节点:"
for node in $NODES; do
  has=$(ssh -o ConnectTimeout=5 root@$node \
    "ip -4 addr show eth0 | grep -c '${VIP}/'" 2>/dev/null || echo 0)
  prio=$(ssh -o ConnectTimeout=5 root@$node \
    "grep MY_PRIO /etc/vip-manager.conf" 2>/dev/null)
  svc=$(ssh -o ConnectTimeout=5 root@$node \
    "systemctl is-active vip-manager" 2>/dev/null)
  if [ "$has" = "1" ]; then
    echo "  ★ $node — HOLDS VIP ($prio, svc=$svc)"
  else
    echo "    $node — no VIP ($prio, svc=$svc)"
  fi
done

echo -e "\n[3] ARP 缓存（本机）:"
ip neigh show "$VIP" 2>/dev/null || echo "  VIP not in ARP table"

echo -e "\n[4] vip-manager 日志（最近 10 行）:"
journalctl -t vip-manager -n 10 --no-pager 2>/dev/null

故障转移测试:

bash

# 在 .32 上停止 vip-manager，观察 VIP 是否转移到 .33
systemctl stop vip-manager
sleep 5
# 在 .33 上检查
ssh root@192.168.122.33 'ip -4 addr show eth0 | grep 192.168.122.240'
# 期望: inet 192.168.122.240/24

# 恢复
systemctl start vip-manager
sleep 5
# .32 应重新获得 VIP（优先级最高）

§3 Harbor 镜像仓库

3.1 Harbor 服务信息

参数	值
地址	`192.168.122.156:30000`
协议	HTTP（非标准端口）
TLS	自签名（需 `insecure_skip_verify`）
认证	admin / xxx（默认）

3.2 RKE2 registries.yaml 配置

文件: /etc/rancher/rke2/registries.yaml（三节点相同）

yaml

mirrors:
  "192.168.122.156:30000":
    endpoint:
      - "http://192.168.122.156:30000"
  docker.io:
    endpoint:
      - "http://192.168.122.156:30000"
configs:
  "192.168.122.156:30000":
    tls:
      insecure_skip_verify: true

docker.io 的 mirror 指向 Harbor，确保 RKE2 系统组件也走本地仓库。

配置生效:

bash

# 修改后需重启 containerd 或 rke2-server
systemctl restart rke2-server

3.3 镜像同步操作

在能访问公网的机器上拉取，推送到 Harbor：

bash

# === Rancher 核心镜像 ===
# rancher/rancher:v2.14.3 (~700MB)
ctr -n k8s.io pull docker.io/rancher/rancher:v2.14.3
ctr -n k8s.io tag docker.io/rancher/rancher:v2.14.3 \
  192.168.122.156:30000/rancher/rancher:v2.14.3
ctr -n k8s.io push --plain-http 192.168.122.156:30000/rancher/rancher:v2.14.3

# rancher/shell:v0.3.0 (Rancher job 镜像)
ctr -n k8s.io pull docker.io/rancher/shell:v0.3.0
ctr -n k8s.io tag docker.io/rancher/shell:v0.3.0 \
  192.168.122.156:30000/rancher/shell:v0.3.0
ctr -n k8s.io push --plain-http 192.168.122.156:30000/rancher/shell:v0.3.0

# === cert-manager 镜像 ===
# jetstack/cert-manager-controller:v1.20.3
# jetstack/cert-manager-cainjector:v1.20.3
# jetstack/cert-manager-webhook:v1.20.3
for img in controller cainjector webhook; do
  ctr -n k8s.io pull docker.io/jetstack/cert-manager-${img}:v1.20.3
  ctr -n k8s.io tag docker.io/jetstack/cert-manager-${img}:v1.20.3 \
    192.168.122.156:30000/jetstack/cert-manager-${img}:v1.20.3
  ctr -n k8s.io push --plain-http 192.168.122.156:30000/jetstack/cert-manager-${img}:v1.20.3
done

重要: 逐节点拉取镜像，避免三节点同时拉取导致 etcd 资源耗尽。

3.4 镜像同步排查脚本

bash

#!/bin/bash
# registry-check.sh — Harbor 镜像仓库排查
HARBOR="192.168.122.156:30000"

echo "=== Harbor 镜像仓库排查 ==="

echo -e "\n[1] Harbor 连通性:"
if curl -sk --connect-timeout 5 "http://${HARBOR}/v2/" &>/dev/null; then
  echo "  ✓ Harbor reachable"
else
  echo "  ✗ Harbor unreachable"
  echo "  → 检查 Harbor 服务是否运行: ssh root@192.168.122.156 'docker ps'"
  exit 1
fi

echo -e "\n[2] Harbor 仓库列表:"
curl -sk "http://${HARBOR}/api/v2.0/projects?page_size=10" 2>/dev/null | \
  python3 -m json.tool 2>/dev/null | grep '"name"' | head -10

echo -e "\n[3] Rancher 镜像检查:"
for tag in v2.14.3; do
  result=$(curl -sk "http://${HARBOR}/v2/rancher/rancher/tags/list" 2>/dev/null)
  if echo "$result" | grep -q "$tag"; then
    echo "  ✓ rancher/rancher:${tag} exists"
  else
    echo "  ✗ rancher/rancher:${tag} missing"
  fi
done

echo -e "\n[4] 各节点已缓存镜像:"
for node in 32 33 34; do
  count=$(ssh -o ConnectTimeout=5 root@192.168.122.$node \
    'crictl images -o json 2>/dev/null | python3 -c "import json,sys; print(len(json.load(sys.stdin)[\"images\"]))" 2>/dev/null' || echo "?")
  echo "  .${node}: ${count} images cached"
done

echo -e "\n[5] registries.yaml 配置:"
cat /etc/rancher/rke2/registries.yaml 2>/dev/null || echo "  NOT FOUND"

常见问题排查:

症状	排查命令	原因	解决
`ImagePullBackOff`	`kubectl describe pod <name>`	镜像不在 Harbor	重新推送镜像
`x509: certificate signed by unknown`	`curl -k http://HARBOR/v2/`	registries.yaml 未配置 `insecure_skip_verify`	修改配置并重启 rke2-server
Harbor 502/503	`ssh root@HARBOR 'docker ps'`	Harbor 容器崩溃	`docker-compose restart`
拉取超时	`crictl pull <image>`	网络/磁盘 I/O	逐节点串行拉取

§4 cert-manager 安装

4.1 添加 Helm Repo

bash

# 在有网络的机器上下载 chart 离线包
helm repo add jetstack https://charts.jetstack.io
helm repo update
helm pull jetstack/cert-manager --version v1.20.3

# 传输到集群节点
scp cert-manager-v1.20.3.tgz root@192.168.122.32:/root/

4.2 安装 cert-manager

bash

# 在 .32 上执行
helm install cert-manager /root/cert-manager-v1.20.3.tgz \
  --namespace cert-manager --create-namespace \
  --set crds.enabled=true

注意: crds.enabled=true 自动安装 CRD，无需手动 kubectl apply。

4.3 cert-manager 排查脚本

bash

#!/bin/bash
# cert-manager-check.sh — cert-manager 安装排查

echo "=== cert-manager 排查 ==="

echo -e "\n[1] Helm Release:"
helm list -n cert-manager 2>&1

echo -e "\n[2] Pod 状态:"
kubectl get pods -n cert-manager -o wide

echo -e "\n[3] CRD 安装:"
kubectl get crd | grep cert-manager
# 期望: certificaterequests, certificates, challenges, 
#       clusterissuers, issuers, orders

echo -e "\n[4] Pod 日志 (controller):"
kubectl logs -n cert-manager \
  $(kubectl get pods -n cert-manager -l app=cert-manager -o jsonpath='{.items[0].metadata.name}') \
  --tail 10

echo -e "\n[5] Webhook 测试:"
kubectl create -f - <<'EOF'
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: test-cert
  namespace: default
spec:
  secretName: test-tls
  issuerRef:
    name: selfsigned-issuer
    kind: ClusterIssuer
  commonName: test.example.com
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: selfsigned-issuer
spec:
  selfSigned: {}
EOF
sleep 5
kubectl get certificate test-cert -n default
kubectl delete -f /dev/stdin <<'EOF'
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: test-cert
  namespace: default
spec:
  secretName: test-tls
  issuerRef:
    name: selfsigned-issuer
    kind: ClusterIssuer
  commonName: test.example.com
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: selfsigned-issuer
spec:
  selfSigned: {}
EOF
kubectl delete secret test-tls -n default 2>/dev/null

常见问题排查:

症状	排查命令	原因	解决
cainjector Unknown	`kubectl describe pod`	节点重启/Pod 驱逐	`kubectl delete pod -n cert-manager -l app=cainjector`
webhook timeout	`kubectl logs -n cert-manager -l app=webhook`	证书未就绪	等待或重启 Pod
CRD 缺失	`kubectl get crd \| grep cert`	未启用 crds.enabled	`helm upgrade --set crds.enabled=true`

§5 Rancher Helm 安装

5.1 生成 TLS 证书

Rancher 需要 TLS 证书。气隙环境使用自签名证书：

bash

# 生成私钥
openssl genrsa -out rancher.ai-ear.cn.key 2048

# 生成自签名证书（10年有效期）
openssl req -x509 -new -nodes \
  -key rancher.ai-ear.cn.key \
  -sha256 -days 3650 \
  -out rancher.ai-ear.cn.crt \
  -subj "/CN=rancher.ai-ear.cn" \
  -addext "subjectAltName=DNS:rancher.ai-ear.cn"

# 验证证书
openssl x509 -in rancher.ai-ear.cn.crt -noout -subject -dates -ext subjectAltName

5.2 创建 TLS Secret

bash

kubectl create namespace cattle-system

kubectl create secret tls tls-rancher-ingress \
  --namespace cattle-system \
  --cert=rancher.ai-ear.cn.crt \
  --key=rancher.ai-ear.cn.key

# 验证 Secret
kubectl get secret tls-rancher-ingress -n cattle-system
# 期望: TYPE kubernetes.io/tls, 包含 tls.crt 和 tls.key

5.3 添加 Helm Repo 并安装

bash

# 离线下载 chart
helm repo add rancher-stable https://releases.rancher.com/server-charts/stable
helm repo update
helm pull rancher-stable/rancher --version 2.14.3

# 安装
helm install rancher rancher-2.14.3.tgz \
  --namespace cattle-system \
  --set hostname=rancher.ai-ear.cn \
  --set bootstrapPassword=xxx \
  --set replicas=3 \
  --set global.cattle.pki.enabled=false \
  --set ingress.tls.source=secret \
  --set privateCA=false

参数说明:

参数	值	说明
`hostname`	`rancher.ai-ear.cn`	Ingress 域名
`bootstrapPassword`	`admin`	首次登录密码
`replicas`	`3`	3 副本分布到 3 节点
`global.cattle.pki.enabled`	`false`	禁用内部 PKI（使用外部证书）
`ingress.tls.source`	`secret`	使用手动创建的 TLS Secret
`privateCA`	`false`	非私有 CA（自签名即可）

5.4 Rancher 排查脚本

bash

#!/bin/bash
# rancher-check.sh — Rancher 安装排查

echo "=== Rancher 排查 ==="

echo -e "\n[1] Helm Release:"
helm list -n cattle-system 2>&1

echo -e "\n[2] Pod 状态:"
kubectl get pods -n cattle-system -o wide
echo ""
kubectl get deploy/rancher -n cattle-system -o wide

echo -e "\n[3] Ingress:"
kubectl get ingress -n cattle-system -o wide
echo ""
kubectl get ingress -n cattle-system -o yaml | grep -A5 'rules:'

echo -e "\n[4] TLS Secret:"
kubectl get secret tls-rancher-ingress -n cattle-system
echo ""
# 验证证书内容
kubectl get secret tls-rancher-ingress -n cattle-system \
  -o jsonpath='{.data.tls\.crt}' | base64 -d | \
  openssl x509 -noout -subject -dates 2>/dev/null

echo -e "\n[5] Rancher Pod 日志 (第一个 Running Pod):"
POD=$(kubectl get pods -n cattle-system -l app=rancher \
  --field-selector=status.phase=Running \
  -o jsonpath='{.items[0].metadata.name}' 2>/dev/null)
if [ -n "$POD" ]; then
  kubectl logs "$POD" -n cattle-system --tail 15
else
  echo "  No Running Rancher pod found"
  kubectl get pods -n cattle-system
fi

echo -e "\n[6] ServiceAccount Token:"
kubectl get secret -n cattle-system | grep rancher-token
echo ""
# 检查 token 是否已填充
kubectl get secret -n cattle-system -l 'cattle.io/creator=norman' \
  -o jsonpath='{range .items[*]}{.metadata.name}: {.data.token}{"\n"}{end}' 2>/dev/null | head -5

echo -e "\n[7] Events (cattle-system):"
kubectl get events -n cattle-system --sort-by='.lastTimestamp' 2>/dev/null | tail -10

echo -e "\n[8] 直接访问测试 (从 VIP):"
curl -sk --resolve rancher.ai-ear.cn:443:192.168.122.240 \
  https://rancher.ai-ear.cn/ping 2>&1 | head -5

常见问题排查:

症状	排查命令	原因	解决
Pod CrashLoopBackOff	`kubectl logs <pod> --previous`	SA token 未填充/证书错误	删除 Pod 让 Deployment 重建，或检查 TLS Secret
`Waiting for server to become available`	`kubectl logs <pod>`	Rancher 内部等待自身就绪	等待 2-3 分钟
`ensure secret ... for service account`	`kubectl get sa rancher -n cattle-system`	SA token Secret 未自动创建	K8s 1.24+ 需要手动创建 token Secret
ImagePullBackOff	`kubectl describe pod <pod>`	镜像未同步到 Harbor	见 §3 镜像同步
Ingress 无 ADDRESS	`kubectl get ingress -n cattle-system`	ingress-nginx 未就绪	`kubectl get pods -n kube-system -l app.kubernetes.io/name=ingress-nginx`

5.5 手动修复 ServiceAccount Token（K8s 1.24+）

K8s 1.24+ 不再自动为 ServiceAccount 创建 Secret，需手动创建：

bash

# 检查 rancher SA 是否有 token
kubectl get sa rancher -n cattle-system -o yaml

# 手动创建 token Secret
kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: rancher-token-manual
  namespace: cattle-system
  annotations:
    kubernetes.io/service-account.name: rancher
type: kubernetes.io/service-account-token
EOF

# 验证 token 已填充
sleep 3
kubectl get secret rancher-token-manual -n cattle-system \
  -o jsonpath='{.data.token}' | base64 -d
echo

§6 libvirt dnsmasq DNS

6.1 配置目标

在 hypervisor (192.168.122.1) 上配置 dnsmasq，将 rancher.ai-ear.cn 解析到集群节点（round-robin）。

6.2 配置步骤

bash

# 在 hypervisor (192.168.122.1) 上执行
cat > /etc/dnsmasq.d/rancher.conf << 'EOF'
address=/rancher.ai-ear.cn/192.168.122.32
address=/rancher.ai-ear.cn/192.168.122.33
address=/rancher.ai-ear.cn/192.168.122.34
EOF

systemctl restart dnsmasq

替代方案: 如果使用 VIP，可以直接解析到 VIP:
bash
cat > /etc/dnsmasq.d/rancher.conf << 'EOF'
address=/rancher.ai-ear.cn/192.168.122.240
EOF

6.3 DNS 排查脚本

bash

#!/bin/bash
# dns-check.sh — DNS 解析排查
DOMAIN="rancher.ai-ear.cn"
HYPERVISOR="192.168.122.1"
VIP="192.168.122.240"

echo "=== DNS 排查 ==="

echo -e "\n[1] dnsmasq 配置:"
ssh -o ConnectTimeout=5 root@${HYPERVISOR} \
  'cat /etc/dnsmasq.d/rancher.conf 2>/dev/null || echo "NOT FOUND"'

echo -e "\n[2] dnsmasq 服务:"
ssh -o ConnectTimeout=5 root@${HYPERVISOR} \
  'systemctl is-active dnsmasq 2>/dev/null || echo "NOT RUNNING"'

echo -e "\n[3] DNS 解析（从 hypervisor）:"
ssh -o ConnectTimeout=5 root@${HYPERVISOR} \
  "dig +short ${DOMAIN} @127.0.0.1" 2>/dev/null

echo -e "\n[4] DNS 解析（从集群节点 .32）:"
ssh -o ConnectTimeout=5 root@192.168.122.32 \
  "dig +short ${DOMAIN} @${HYPERVISOR}" 2>/dev/null

echo -e "\n[5] curl 测试:"
curl -sk --connect-timeout 5 \
  --resolve ${DOMAIN}:443:${VIP} \
  "https://${DOMAIN}/ping" 2>&1 | head -3

echo -e "\n[6] 多次解析验证 round-robin:"
for i in 1 2 3 4 5 6; do
  ip=$(dig +short ${DOMAIN} @${HYPERVISOR} | head -1)
  echo "  attempt $i: $ip"
  sleep 1
done

§7 FRP 内网穿透

7.1 架构

Internet → Traefik (公网) → FRP Server (:10443) → FRP Client (内网) → VIP:443

7.2 FRP Client 配置

文件: /etc/frp/frpc.toml（或 frpc.ini）

toml

serverAddr = "<公网服务器IP>"
serverPort = 7000

[[proxies]]
name = "rancher-https"
type = "tcp"
localIP = "192.168.122.240"
localPort = 443
remotePort = 10443

关键: localIP 必须设置为 VIP 192.168.122.240，确保请求通过持有 VIP 的节点路由。

7.3 部署步骤

bash

# 写入配置
cat > /etc/frp/frpc.toml << 'EOF'
serverAddr = "<公网服务器IP>"
serverPort = 7000

[[proxies]]
name = "rancher-https"
type = "tcp"
localIP = "192.168.122.240"
localPort = 443
remotePort = 10443
EOF

systemctl restart frpc
systemctl enable frpc

7.4 FRP 排查脚本

bash

#!/bin/bash
# frp-check.sh — FRP 内网穿透排查
VIP="192.168.122.240"

echo "=== FRP 排查 ==="

echo -e "\n[1] frpc 服务状态:"
systemctl is-active frpc
systemctl status frpc --no-pager | head -10

echo -e "\n[2] frpc 配置:"
cat /etc/frp/frpc.toml 2>/dev/null || cat /etc/frp/frpc.ini 2>/dev/null

echo -e "\n[3] frpc 日志:"
journalctl -u frpc -n 20 --no-pager

echo -e "\n[4] 本地 VIP 连通性:"
curl -sk --connect-timeout 5 --resolve rancher.ai-ear.cn:443:${VIP} \
  https://rancher.ai-ear.cn/ping 2>&1 | head -3

echo -e "\n[5] 远程端口检查 (从公网服务器):"
# 需在公网服务器上执行:
# nc -zv localhost 10443
echo "  → 在公网服务器上执行: nc -zv localhost 10443"

§8 Traefik SNI Passthrough

8.1 架构

Traefik 不终止 TLS，直接将 TCP 流量透传到 FRP Server：

Client (:443) → Traefik Router → FRP Server (:10443) → FRP Client → VIP:443

8.2 Traefik 配置

动态配置 (dynamic/rancher.yaml):

yaml

tcp:
  routers:
    rancher:
      rule: "HostSNI(`rancher.ai-ear.cn`)"
      entryPoints:
        - websecure
      service: rancher-svc
      tls:
        passthrough: true

  services:
    rancher-svc:
      loadBalancer:
        servers:
          - address: "127.0.0.1:10443"  # FRP Server 的 remotePort

8.3 Traefik 排查

bash

# 在公网服务器上检查 Traefik 状态
docker exec traefik wget -qO- http://localhost:8080/api/tcp/routers 2>/dev/null | \
  python3 -m json.tool 2>/dev/null | head -30

# 检查端口监听
ss -tlnp | grep -E '443|10443|8080'

# 直接测试 FRP 端口
curl -sk --connect-timeout 5 \
  --resolve rancher.ai-ear.cn:443:127.0.0.1 \
  https://rancher.ai-ear.cn/ping

§9 登录验证

9.1 获取 Bootstrap 密码

bash

kubectl get secret --namespace cattle-system bootstrap-secret \
  -o go-template='{{.data.bootstrapPassword|base64decode}}{{"\n"}}'

9.2 生成登录 URL

bash

BOOTSTRAP=$(kubectl get secret --namespace cattle-system bootstrap-secret \
  -o go-template='{{.data.bootstrapPassword|base64decode}}')
echo "https://rancher.ai-ear.cn/dashboard/?setup=${BOOTSTRAP}"

9.3 端到端访问验证脚本

bash

#!/bin/bash
# e2e-check.sh — 端到端访问验证
DOMAIN="rancher.ai-ear.cn"
VIP="192.168.122.240"

echo "=== 端到端访问验证 ==="

echo -e "\n[1] 内网直接访问 (VIP):"
HTTP_CODE=$(curl -sk -o /dev/null -w "%{http_code}" --connect-timeout 5 \
  --resolve ${DOMAIN}:443:${VIP} "https://${DOMAIN}/ping")
echo "  HTTP $HTTP_CODE — https://${DOMAIN}/ping (via VIP)"

HTTP_CODE=$(curl -sk -o /dev/null -w "%{http_code}" --connect-timeout 5 \
  --resolve ${DOMAIN}:443:${VIP} "https://${DOMAIN}/dashboard/")
echo "  HTTP $HTTP_CODE — https://${DOMAIN}/dashboard/ (via VIP)"

echo -e "\n[2] TLS 证书验证:"
echo | openssl s_client -connect ${VIP}:443 \
  -servername ${DOMAIN} 2>/dev/null | \
  openssl x509 -noout -subject -dates -issuer 2>/dev/null

echo -e "\n[3] 登录页面检测:"
BODY=$(curl -sk --connect-timeout 5 \
  --resolve ${DOMAIN}:443:${VIP} "https://${DOMAIN}/dashboard/" 2>&1)
if echo "$BODY" | grep -qi "rancher"; then
  echo "  ✓ Rancher dashboard page detected"
else
  echo "  ✗ Unexpected response"
  echo "$BODY" | head -5
fi

echo -e "\n[4] API Server 健康:"
API_CODE=$(curl -sk -o /dev/null -w "%{http_code}" --connect-timeout 5 \
  --resolve ${DOMAIN}:443:${VIP} "https://${DOMAIN}/version")
echo "  HTTP $API_CODE — /version endpoint"

echo -e "\n[5] Bootstrap 密码:"
BOOTSTRAP=$(kubectl get secret --namespace cattle-system bootstrap-secret \
  -o go-template='{{.data.bootstrapPassword|base64decode}}' 2>/dev/null)
if [ -n "$BOOTSTRAP" ]; then
  echo "  ✓ Bootstrap password available"
  echo "  Login URL: https://${DOMAIN}/dashboard/?setup=${BOOTSTRAP}"
else
  echo "  ✗ Bootstrap secret not found"
  kubectl get secrets -n cattle-system | grep bootstrap
fi

故障恢复手册

A. etcd Quorum 丢失恢复

症状:

所有节点 kubectl 命令返回 context deadline exceeded 或 connection refused
etcd 日志出现 leader election timeout
节点状态从 Ready 变为 NotReady

根因: 并发拉取大量镜像（~700MB x 3 节点）导致资源耗尽，etcd heartbeat 超时，quorum 丢失。

恢复步骤

bash

#!/bin/bash
# etcd-recovery.sh — etcd Quorum 恢复脚本
# 在能 SSH 到所有节点的管理机上执行

set -e
NODE1="192.168.122.32"
NODE2="192.168.122.33"
NODE3="192.168.122.34"

echo "=== etcd Quorum Recovery ==="

# Step 1: 停止所有节点 rke2-server
echo -e "\n[Step 1] 停止所有节点 rke2-server..."
for node in $NODE1 $NODE2 $NODE3; do
  echo "  Stopping rke2-server on $node..."
  ssh -o ConnectTimeout=10 root@$node 'systemctl stop rke2-server' 2>/dev/null || true
  sleep 2
done

# Step 2: 确认所有进程已停止
echo -e "\n[Step 2] 确认进程已停止..."
for node in $NODE1 $NODE2 $NODE3; do
  count=$(ssh root@$node 'pgrep -c "rke2 server" 2>/dev/null || echo 0')
  etcd_count=$(ssh root@$node 'pgrep -c "etcd" 2>/dev/null || echo 0')
  echo "  $node: rke2=$count, etcd=$etcd_count"
done

# Step 3: 在 NODE1 执行 cluster-reset
echo -e "\n[Step 3] 在 $NODE1 执行 cluster-reset..."
ssh root@$NODE1 'nohup rke2 server --cluster-reset > /tmp/cluster-reset.log 2>&1 &'

# Step 4: 等待 reset 完成
echo -e "\n[Step 4] 等待 reset 完成（最长 120s）..."
for i in $(seq 1 24); do
  sleep 5
  if ssh root@$NODE1 'grep -q "Managed etcd cluster membership has been reset" /tmp/cluster-reset.log 2>/dev/null'; then
    echo "  ✓ Cluster reset completed at attempt $i"
    break
  fi
  if ssh root@$NODE1 'grep -q "error" /tmp/cluster-reset.log 2>/dev/null | tail -1'; then
    echo "  Checking for errors..."
  fi
  echo "  Waiting... ($((i*5))s)"
done

# 显示最后几行日志
echo -e "\n  Reset 日志 (最后 10 行):"
ssh root@$NODE1 'tail -10 /tmp/cluster-reset.log'

# Step 5: 清理残留 etcd 进程并启动 NODE1
echo -e "\n[Step 5] 启动 $NODE1 rke2-server..."
ssh root@$NODE1 'pkill -9 etcd 2>/dev/null; sleep 2; systemctl start rke2-server'
sleep 20

# Step 6: 验证 NODE1 API Server
echo -e "\n[Step 6] 验证 $NODE1 API Server..."
ssh root@$NODE1 'kubectl get nodes --request-timeout=10s 2>&1'

# Step 7: 清理 NODE2/NODE3 的 etcd 数据并重启
echo -e "\n[Step 7] 清理并重启 $NODE2 和 $NODE3..."
for node in $NODE2 $NODE3; do
  echo "  Cleaning etcd data on $node..."
  ssh root@$node 'rm -rf /var/lib/rancher/rke2/server/db'
  echo "  Starting rke2-server on $node..."
  ssh root@$node 'nohup systemctl start rke2-server &'
  sleep 5
done

# Step 8: 等待所有节点恢复
echo -e "\n[Step 8] 等待所有节点恢复（最长 120s）..."
for i in $(seq 1 24); do
  sleep 5
  ready=$(ssh root@$NODE1 'kubectl get nodes --no-headers --request-timeout=5s 2>/dev/null | grep -c Ready' || echo 0)
  echo "  Ready nodes: $ready/3 ($((i*5))s elapsed)"
  if [ "$ready" = "3" ]; then
    echo "  ✓ All 3 nodes Ready!"
    break
  fi
done

# Step 9: 最终验证
echo -e "\n[Step 9] 最终状态:"
ssh root@$NODE1 'kubectl get nodes -o wide --request-timeout=10s'

echo -e "\n=== Recovery Complete ==="

恢复后检查清单

bash

# 1. 所有节点 Ready
kubectl get nodes -o wide

# 2. etcd 集群健康
ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt \
  --cert=/var/lib/rancher/rke2/server/tls/etcd/client.crt \
  --key=/var/lib/rancher/rke2/server/tls/etcd/client.key \
  endpoint health -w table
# 期望: 127.0.0.1:2379 is healthy

# 3. 系统 Pod 全部 Running
kubectl get pods -n kube-system

# 4. Rancher Pod 恢复
kubectl get pods -n cattle-system
# 如果 Rancher Pod 卡住:
kubectl rollout restart deploy/rancher -n cattle-system

# 5. VIP 管理器恢复
for node in 32 33 34; do
  ssh root@192.168.122.$node 'systemctl start vip-manager'
done
ping -c 2 192.168.122.240

B. Rancher Pod CrashLoopBackOff

症状: Rancher Pod 反复重启，日志显示 EnsureSecretForServiceAccount: waiting for secret

排查步骤:

bash

# 1. 查看 Pod 状态
kubectl get pods -n cattle-system -o wide

# 2. 查看上一次崩溃日志
POD=$(kubectl get pods -n cattle-system -l app=rancher \
  -o jsonpath='{.items[0].metadata.name}')
kubectl logs "$POD" -n cattle-system --previous --tail 30

# 3. 检查 ServiceAccount Token
kubectl get sa rancher -n cattle-system -o yaml
kubectl get secrets -n cattle-system | grep rancher-token

# 4. 如果 token 缺失，手动创建
kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: rancher-token-manual
  namespace: cattle-system
  annotations:
    kubernetes.io/service-account.name: rancher
type: kubernetes.io/service-account-token
EOF

# 5. 重启 Rancher Pod
kubectl rollout restart deploy/rancher -n cattle-system

# 6. 等待 Pod Ready
kubectl wait --for=condition=Ready pods -l app=rancher \
  -n cattle-system --timeout=300s

C. Harbor 崩溃恢复

症状: Harbor 容器停止运行，镜像拉取失败

bash

# 1. SSH 到 Harbor 服务器
ssh root@192.168.122.156

# 2. 检查 Docker 服务
systemctl is-active docker
docker ps -a | grep harbor

# 3. 重启 Harbor
cd /opt/harbor
docker-compose restart
# 或
docker-compose up -d

# 4. 等待健康检查
sleep 30
curl -sk http://192.168.122.156:30000/v2/ 
# 期望: {} (空 JSON 对象，HTTP 200)

# 5. 验证镜像存在
curl -sk "http://192.168.122.156:30000/v2/rancher/rancher/tags/list"
# 期望: 包含 "v2.14.3"

D. VIP 不响应排查

bash

#!/bin/bash
# vip-troubleshoot.sh — VIP 深度排查
VIP="192.168.122.240"
NODES="192.168.122.32 192.168.122.33 192.168.122.34"

echo "=== VIP 深度排查 ==="

echo -e "\n[1] 各节点 vip-manager 服务状态:"
for node in $NODES; do
  status=$(ssh -o ConnectTimeout=5 root@$node 'systemctl is-active vip-manager' 2>/dev/null)
  journal=$(ssh -o ConnectTimeout=5 root@$node \
    'journalctl -t vip-manager -n 3 --no-pager 2>/dev/null')
  echo "  $node: $status"
  echo "$journal" | sed 's/^/    /'
done

echo -e "\n[2] 各节点网络接口:"
for node in $NODES; do
  ip_info=$(ssh -o ConnectTimeout=5 root@$node \
    "ip -4 addr show eth0 | grep -E 'inet|192.168.122.240'" 2>/dev/null)
  echo "  $node:"
  echo "$ip_info" | sed 's/^/    /'
done

echo -e "\n[3] VIP ARP:"
ip neigh show "$VIP" 2>/dev/null || echo "  VIP not in ARP cache"

echo -e "\n[4] VIP ping:"
ping -c 3 -W 1 "$VIP" 2>&1

echo -e "\n[5] RKE2 服务状态（vip-manager 依赖 rke2 运行）:"
for node in $NODES; do
  rke2=$(ssh -o ConnectTimeout=5 root@$node 'systemctl is-active rke2-server' 2>/dev/null)
  echo "  $node rke2-server: $rke2"
done

经验教训

气隙环境依赖管理: keepalived 在 Rocky 9 气隙环境依赖链过深（libnfnetlink, libnl3, libipset 等 10+ RPM），直接用自定义 bash 脚本更可靠。只需 ip, arping, ping 三个系统自带工具。
并发镜像拉取风险: 在资源受限的集群中，3 节点同时拉取 ~700MB 镜像导致 I/O 争抢，etcd heartbeat 超时（默认 100ms/1s），触发 leader election 失败，最终 quorum 丢失。必须逐节点串行拉取。
etcd 稳定性: etcd 对磁盘 I/O 延迟极其敏感。wal_fsync 和 backend_commit 延迟超过阈值即触发 leader election。在资源紧张的环境中，任何额外的 I/O 负载（如镜像拉取、大量 Pod 启动）都可能影响 etcd。
RKE2 cluster-reset 流程: --cluster-reset 只重置当前节点的 etcd 成员关系，其他节点的 /var/lib/rancher/rke2/server/db 必须手动清理后才能重新加入集群。
K8s 1.24+ ServiceAccount Token: 不再自动为 ServiceAccount 创建 Secret，Rancher 依赖 SA token 来认证内部通信，需手动创建 Secret 并绑定。
VIP 与 etcd 恢复顺序: etcd 恢复期间应停止 vip-manager，避免 VIP 频繁漂移加剧 API Server 压力。恢复完成后重启。

文件清单

文件	路径	节点	说明
RKE2 配置 (首节点)	`/etc/rancher/rke2/config.yaml`	.32	无 server/token 字段
RKE2 配置 (其他节点)	`/etc/rancher/rke2/config.yaml`	.33, .34	含 server/token
镜像仓库配置	`/etc/rancher/rke2/registries.yaml`	全部	Harbor mirror 配置
VIP 管理器脚本	`/usr/local/bin/vip-manager.sh`	全部	零依赖 VIP failover
VIP 管理器配置	`/etc/vip-manager.conf`	全部	各节点优先级不同
VIP systemd unit	`/etc/systemd/system/vip-manager.service`	全部	systemd service
TLS 证书	`/root/rancher.ai-ear.cn.{crt,key}`	.32	Rancher Ingress 证书
Helm Chart	`rancher-2.14.3.tgz`	.32	Rancher chart 离线包
全局检查脚本	`/usr/local/bin/cluster-check.sh`	.32	一键健康检查
部署文档	本文件	-	操作记录与恢复手册

Rancher v2.14.3 气隙环境部署操作手册 ​

目录 ​

环境信息 ​

集群拓扑 ​

基础参数 ​

部署状态 ​

§0 预检脚本 ​

0.1 全局健康检查脚本 ​

0.2 快速单节点检查（一行命令） ​

§1 RKE2 集群配置 ​

1.1 首节点配置（192.168.122.32） ​

1.2 第二节点配置（192.168.122.33） ​

1.3 第三节点配置（192.168.122.34） ​

1.4 集群验证排查脚本 ​

§2 VIP 管理器 ​

背景 ​

2.1 VIP 管理器脚本 ​

2.2 各节点配置文件 ​

2.3 systemd unit 文件 ​

2.4 部署步骤 ​

2.5 VIP 排查脚本 ​

§3 Harbor 镜像仓库 ​

3.1 Harbor 服务信息 ​

3.2 RKE2 registries.yaml 配置 ​

3.3 镜像同步操作 ​

3.4 镜像同步排查脚本 ​

§4 cert-manager 安装 ​

4.1 添加 Helm Repo ​

4.2 安装 cert-manager ​

4.3 cert-manager 排查脚本 ​

§5 Rancher Helm 安装 ​

5.1 生成 TLS 证书 ​

5.2 创建 TLS Secret ​

5.3 添加 Helm Repo 并安装 ​

5.4 Rancher 排查脚本 ​

5.5 手动修复 ServiceAccount Token（K8s 1.24+） ​

§6 libvirt dnsmasq DNS ​

6.1 配置目标 ​

6.2 配置步骤 ​

6.3 DNS 排查脚本 ​

§7 FRP 内网穿透 ​

7.1 架构 ​

7.2 FRP Client 配置 ​

7.3 部署步骤 ​

7.4 FRP 排查脚本 ​

§8 Traefik SNI Passthrough ​

8.1 架构 ​

8.2 Traefik 配置 ​

8.3 Traefik 排查 ​

§9 登录验证 ​

9.1 获取 Bootstrap 密码 ​

9.2 生成登录 URL ​

9.3 端到端访问验证脚本 ​

故障恢复手册 ​

A. etcd Quorum 丢失恢复 ​

恢复步骤 ​

恢复后检查清单 ​

B. Rancher Pod CrashLoopBackOff ​

C. Harbor 崩溃恢复 ​

D. VIP 不响应排查 ​

经验教训 ​

文件清单 ​

Rancher v2.14.3 气隙环境部署操作手册

目录

环境信息

集群拓扑

基础参数

部署状态

§0 预检脚本

0.1 全局健康检查脚本

0.2 快速单节点检查（一行命令）

§1 RKE2 集群配置

1.1 首节点配置（192.168.122.32）

1.2 第二节点配置（192.168.122.33）

1.3 第三节点配置（192.168.122.34）

1.4 集群验证排查脚本

§2 VIP 管理器

背景

2.1 VIP 管理器脚本

2.2 各节点配置文件

2.3 systemd unit 文件

2.4 部署步骤

2.5 VIP 排查脚本

§3 Harbor 镜像仓库

3.1 Harbor 服务信息

3.2 RKE2 registries.yaml 配置

3.3 镜像同步操作

3.4 镜像同步排查脚本

§4 cert-manager 安装

4.1 添加 Helm Repo

4.2 安装 cert-manager

4.3 cert-manager 排查脚本

§5 Rancher Helm 安装

5.1 生成 TLS 证书

5.2 创建 TLS Secret

5.3 添加 Helm Repo 并安装

5.4 Rancher 排查脚本

5.5 手动修复 ServiceAccount Token（K8s 1.24+）

§6 libvirt dnsmasq DNS

6.1 配置目标

6.2 配置步骤

6.3 DNS 排查脚本

§7 FRP 内网穿透

7.1 架构

7.2 FRP Client 配置

7.3 部署步骤

7.4 FRP 排查脚本

§8 Traefik SNI Passthrough

8.1 架构

8.2 Traefik 配置

8.3 Traefik 排查

§9 登录验证

9.1 获取 Bootstrap 密码

9.2 生成登录 URL

9.3 端到端访问验证脚本

故障恢复手册

A. etcd Quorum 丢失恢复

恢复步骤

恢复后检查清单

B. Rancher Pod CrashLoopBackOff

C. Harbor 崩溃恢复

D. VIP 不响应排查

经验教训

文件清单