节点 192.168.122.37 集群初始化排查修复记录

日期: 2026-08-15
目标节点: 192.168.122.37 (hostname: szb122037.local)
集群: GPU AI-EAR RKE2 v1.35.6
结果: ✅ 节点 Ready，所有组件 Running

一、集群环境概览

项目	值
集群版本	RKE2 v1.35.6+rke2r1
CNI	Calico v3.32.0
Harbor 仓库	192.168.122.156:30000 (HTTP)
Rancher 管理面	gpu.ai-ear.cn (192.168.122.32/33/34:443)
目标节点 OS	Rocky Linux 9.8 (Blue Onyx)
目标节点 IP	192.168.122.37
节点角色	control-plane, etcd, worker

二、排查思路总览

┌─────────────────────────────────────────────────────┐
│                 排查决策树                            │
├─────────────────────────────────────────────────────┤
│                                                     │
│  1. SSH 连通性                                       │
│     ├── 失败 → 配置免密 / 检查 sshd                  │
│     └── 成功 ↓                                      │
│                                                     │
│  2. 节点基础环境                                     │
│     ├── 内核模块 (overlay, br_netfilter, nf_conntrack)│
│     ├── sysctl 参数 (ip_forward, bridge-nf-call)     │
│     ├── swap 状态                                    │
│     └── 网络连通性 (→ Rancher, → Harbor)              │
│                                                     │
│  3. RKE2 服务状态                                    │
│     ├── activating → 查 journal 日志                │
│     │   ├── 镜像拉取失败 → 检查 registries/hosts.toml│
│     │   ├── etcd 连接拒绝 → 镜像缺失导致 pod 未启动  │
│     │   └── 其他错误 → 查具体日志                     │
│     └── active ↓                                    │
│                                                     │
│  4. 节点就绪状态                                     │
│     ├── NotReady → 检查 CNI pod (calico)             │
│     │   └── ErrImagePull → containerd HTTPS/HTTP 问题│
│     └── Ready ↓                                     │
│                                                     │
│  5. 业务组件状态                                     │
│     ├── cattle-cluster-agent Error → CA checksum     │
│     └── 全部 Running → 完成                          │
│                                                     │
└─────────────────────────────────────────────────────┘

三、具体排查与修复步骤

步骤 1：SSH 免密连通性检查

问题现象：

xxx@192.168.122.37: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password)

排查命令：

bash

# 测试 SSH 连接（verbose 模式查看详细认证过程）
ssh -v -o ConnectTimeout=5 root@192.168.122.37 'hostname'

# 检查本地密钥是否存在
ls -la ~/.ssh/id_rsa*
cat ~/.ssh/id_rsa.pub

修复方法：

bash

ssh-copy-id root@192.168.122.37

验证：

bash

ssh root@192.168.122.37 'hostname && echo SSH_OK'
### 步骤 2：节点基础环境检查

**一键检查脚本**：
```bash
ssh root@192.168.122.37 << 'CHECK_EOF'
echo "=== OS ==="
cat /etc/os-release | grep -E "^(NAME|VERSION)="

echo "=== Kernel Modules ==="
lsmod | grep -E "overlay|br_netfilter|nf_conntrack" || echo "MISSING modules"

echo "=== Sysctl ==="
sysctl net.bridge.bridge-nf-call-iptables net.ipv4.ip_forward 2>/dev/null \
  || echo "sysctl not set"

echo "=== Swap ==="
swapon --show
echo "(empty = no swap, good)"

echo "=== Network ==="
ip addr show | grep "inet " | grep -v 127.0.0.1

echo "=== Connectivity to Rancher ==="
curl -k --connect-timeout 5 -o /dev/null -w "HTTP %{http_code}" https://gpu.ai-ear.cn
echo

echo "=== Connectivity to Harbor ==="
curl -s -o /dev/null -w "HTTP %{http_code}" http://192.168.122.156:30000/v2/
echo

echo "=== Disk ==="
df -h /

echo "=== RKE2 Binary ==="
which rke2 2>/dev/null && rke2 --version 2>/dev/null || echo "rke2 not installed"

echo "=== Services ==="
systemctl is-active rke2-server 2>/dev/null || echo "no rke2-server"
systemctl is-active rancher-system-agent 2>/dev/null || echo "no rancher-system-agent"
CHECK_EOF

本次检查结果：

检查项	期望值	实际值	状态
OS	Rocky Linux 9.x	Rocky Linux 9.8	✅
overlay 模块	loaded	loaded	✅
br_netfilter	loaded	loaded	✅
nf_conntrack	loaded	loaded	✅
bridge-nf-call-iptables	1	1	✅
ip_forward	1	1	✅
swap	无	无	✅
IP 地址	192.168.122.37	192.168.122.37	✅
Rancher 连通	HTTP 200	HTTP 200	✅
Harbor 连通	HTTP 401	HTTP 401	✅
磁盘空间	>10% free	18% used	✅
RKE2 版本	v1.35.6	v1.35.6+rke2r1	✅
rke2-server	active	activating	❌
rancher-system-agent	active	active	✅

结论: 基础环境正常，rke2-server 卡在 activating 状态，需进一步排查日志。

步骤 3：RKE2 服务状态排查

查看服务状态和日志：

bash

# 查看服务状态
ssh root@192.168.122.37 'systemctl status rke2-server --no-pager 2>&1 | head -30'

# 查看最近日志
ssh root@192.168.122.37 'journalctl -u rke2-server --no-pager -n 50 2>&1 | tail -50'

# 过滤关键错误
ssh root@192.168.122.37 'journalctl -u rke2-server --no-pager -n 100 2>&1 \
  | grep -iE "error|fail|warn|image|pull|etcd" | head -30'

发现的错误日志：

level=info msg="Pod for etcd not synced (pod sandbox not found), retrying"
level=error msg="Failed to check local etcd status for learner management:
  rpc error: code = Unavailable desc = connection error:
  desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""

进一步检查镜像和配置：

bash

# 查看 registries 配置
ssh root@192.168.122.37 'cat /etc/rancher/rke2/registries.yaml'

# 查看容器内镜像
ssh root@192.168.122.37 '/var/lib/rancher/rke2/bin/crictl \
  --runtime-endpoint unix:///run/k3s/containerd/containerd.sock images 2>&1'

# 查看 containerd certs.d 配置
ssh root@192.168.122.37 'for d in /var/lib/rancher/rke2/agent/etc/containerd/certs.d/*/; do
  echo "=== $d ==="
  for f in $d*; do echo "--- $f ---"; cat $f; done
done'

根因确认：

failed to pull and unpack image "192.168.122.156:30000/rancher/hardened-etcd:v3.6.12-k3s1-build20260603":
failed to do request: Head "https://192.168.122.156:30000/v2/...":
http: server gave HTTP response to HTTPS client

┌──────────────────┐     HTTPS (默认)     ┌────────────────────┐
│   containerd     │ ──────────────────> │  Harbor :30000     │
│  (kubelet/rke2)  │                      │  (仅支持 HTTP)     │
│                  │ <─ HTTP Response ── │                    │
└──────────────────┘                      └────────────────────┘
          │
          ✗ "http: server gave HTTP response to HTTPS client"
          ✗ 所有镜像拉取失败
          ✗ etcd 镜像缺失 → etcd 容器未启动
          ✗ 127.0.0.1:2379 连接拒绝
          ✗ rke2-server 卡在 activating

步骤 4：修复 Registries 配置（核心修复）

4.1 修复 registries.yaml

修复前的错误配置（JSON 格式，由 Rancher agent 生成，缺少完整凭证）：

json

{
  "configs": {
    "192.168.122.156:30000": {
      "auth": null,
      "tls": {"insecure_skip_verify": true}
    }
  },
  "mirrors": {"*": {"endpoint": ["http://192.168.122.156:30000"]}}
}

修复操作：

bash

ssh root@192.168.122.37 'cat > /etc/rancher/rke2/registries.yaml << "EOF"
mirrors:
  docker.io:
    endpoint:
      - "http://192.168.122.156:30000"
  quay.io:
    endpoint:
      - "http://192.168.122.156:30000"
  registry.k8s.io:
    endpoint:
      - "http://192.168.122.156:30000"
  "192.168.122.156:30000":
    endpoint:
      - "http://192.168.122.156:30000"
configs:
  "192.168.122.156:30000":
    tls:
      insecure_skip_verify: true
    auth:
      username: xxx
      password: xxx
EOF'

4.2 修复 containerd certs.d hosts.toml

修复前的错误配置：

toml

# File generated by rke2. DO NOT EDIT.
server = "https://192.168.122.156:30000/v2"    # ← HTTPS!
capabilities = ["pull", "resolve", "push"]
skip_verify = true

[host]

修复操作：

bash

ssh root@192.168.122.37 'cat > /var/lib/rancher/rke2/agent/etc/containerd/certs.d/192.168.122.156:30000/hosts.toml << "EOF"
# Fixed: use http instead of https
server = "http://192.168.122.156:30000/v2"
capabilities = ["pull", "resolve", "push"]
skip_verify = true

[host."http://192.168.122.156:30000/v2"]
  capabilities = ["pull", "resolve"]
  skip_verify = true
EOF'

4.3 重启 RKE2 并验证

bash

# 重启 rke2-server（后台执行，避免阻塞）
ssh root@192.168.122.37 'systemctl restart rke2-server' &

# 等待启动（约 30-60 秒）
sleep 60

# 验证服务状态
ssh root@192.168.122.37 'systemctl is-active rke2-server'
# 期望: active

# 验证镜像已拉取
ssh root@192.168.122.37 '/var/lib/rancher/rke2/bin/crictl \
  --runtime-endpoint unix:///run/k3s/containerd/containerd.sock images'

验证成功输出：

IMAGE                                                TAG                                SIZE
192.168.122.156:30000/rancher/hardened-etcd          v3.6.12-k3s1-build20260603         18.2MB
192.168.122.156:30000/rancher/hardened-kubernetes    v1.35.6-rke2r1-build20260612       153MB
192.168.122.156:30000/rancher/klipper-helm           v0.11.1-build20260615              63.7MB
192.168.122.156:30000/rancher/mirrored-pause         3.6                                298kB
192.168.122.156:30000/rancher/rke2-cloud-provider    v1.35.4-...                        20.9MB
192.168.122.156:30000/rancher/rke2-runtime           v1.35.6-rke2r1                     102MB

步骤 5：节点就绪状态验证

bash

# 查看节点状态
ssh root@192.168.122.37 'export KUBECONFIG=/etc/rancher/rke2/rke2.yaml
/var/lib/rancher/rke2/bin/kubectl get nodes -o wide'

输出：

NAME              STATUS   ROLES                AGE    VERSION          INTERNAL-IP
szb122037.local   Ready    control-plane,etcd   4m     v1.35.6+rke2r1   192.168.122.37

步骤 6：修复 cattle-cluster-agent CA Checksum

问题现象：

bash

kubectl get pods -n cattle-system
# cattle-cluster-agent 反复 Error/CrashLoopBackOff

查看错误日志：

bash

ssh root@192.168.122.37 'export KUBECONFIG=/etc/rancher/rke2/rke2.yaml
/var/lib/rancher/rke2/bin/kubectl logs -n cattle-system deployment/cattle-cluster-agent --tail=50'

错误输出：

ERROR: Configured cacerts checksum
  (6903fb1eaadc30860dfe49b374cc4dedd7f45fb45d2f578f995a53b9ebda4424)
does not match given --ca-checksum
  (cd9078b7f2b7e605ab8d6fd8629d8de8dda9efbd5fd6fa5455f634c769bec19a)

原因：Rancher 服务端 TLS 证书已更换，init.sh 中的 CA checksum 已过期。

修复操作：

bash

# 获取当前正确的 CA checksum（从日志中获取，即服务器实际返回的值）
ACTUAL_CHECKSUM="6903fb1eaadc30860dfe49b374cc4dedd7f45fb45d2f578f995a53b9ebda4424"

# 更新 deployment 环境变量
ssh root@192.168.122.37 "export KUBECONFIG=/etc/rancher/rke2/rke2.yaml
/var/lib/rancher/rke2/bin/kubectl set env deployment/cattle-cluster-agent \
  -n cattle-system CATTLE_CA_CHECKSUM=${ACTUAL_CHECKSUM}"

# 验证
sleep 15
ssh root@192.168.122.37 'export KUBECONFIG=/etc/rancher/rke2/rke2.yaml
/var/lib/rancher/rke2/bin/kubectl get pods -n cattle-system'
# 期望: cattle-cluster-agent-xxx  1/1  Running  0

四、完整修复脚本

bash

#!/bin/bash
# fix_node_122037.sh - 节点 192.168.122.37 集群初始化修复脚本
# 用法: bash fix_node_122037.sh
set -ex

NODE="root@192.168.122.37"
HARBOR="192.168.122.156:30000"
HARBOR_USER="admin"
HARBOR_PASS="xxx"
CA_CHECKSUM="6903fb1eaadc30860dfe49b374cc4dedd7f45fb45d2f578f995a53b9ebda4424"

echo "=== Step 1: 验证 SSH 连通性 ==="
ssh -o ConnectTimeout=5 ${NODE} 'hostname && echo SSH_OK'

echo "=== Step 2: 修复 registries.yaml ==="
ssh ${NODE} "cat > /etc/rancher/rke2/registries.yaml << 'EOF'
mirrors:
  docker.io:
    endpoint:
      - \"http://${HARBOR}\"
  quay.io:
    endpoint:
      - \"http://${HARBOR}\"
  registry.k8s.io:
    endpoint:
      - \"http://${HARBOR}\"
  \"${HARBOR}\":
    endpoint:
      - \"http://${HARBOR}\"
configs:
  \"${HARBOR}\":
    tls:
      insecure_skip_verify: true
    auth:
      username: ${HARBOR_USER}
      password: ${HARBOR_PASS}
EOF"

echo "=== Step 3: 修复 containerd certs.d hosts.toml ==="
ssh ${NODE} "cat > /var/lib/rancher/rke2/agent/etc/containerd/certs.d/${HARBOR}/hosts.toml << 'EOF'
# Fixed: use http instead of https
server = \"http://${HARBOR}/v2\"
capabilities = [\"pull\", \"resolve\", \"push\"]
skip_verify = true

[host.\"http://${HARBOR}/v2\"]
  capabilities = [\"pull\", \"resolve\"]
  skip_verify = true
EOF"

echo "=== Step 4: 重启 rke2-server ==="
ssh ${NODE} 'systemctl restart rke2-server' &

echo "等待 rke2-server 启动 (60s)..."
sleep 60

echo "=== Step 5: 验证 rke2-server ==="
ssh ${NODE} 'systemctl is-active rke2-server'

echo "=== Step 6: 验证镜像拉取 ==="
ssh ${NODE} '/var/lib/rancher/rke2/bin/crictl \
  --runtime-endpoint unix:///run/k3s/containerd/containerd.sock images'

echo "=== Step 7: 验证节点状态 ==="
ssh ${NODE} 'export KUBECONFIG=/etc/rancher/rke2/rke2.yaml
/var/lib/rancher/rke2/bin/kubectl get nodes -o wide'

echo "=== Step 8: 修复 cattle-cluster-agent CA checksum ==="
ssh ${NODE} "export KUBECONFIG=/etc/rancher/rke2/rke2.yaml
/var/lib/rancher/rke2/bin/kubectl set env deployment/cattle-cluster-agent \
  -n cattle-system CATTLE_CA_CHECKSUM=${CA_CHECKSUM}"

echo "等待 cattle-cluster-agent 重启..."
sleep 15

echo "=== Step 9: 最终验证 ==="
ssh ${NODE} 'export KUBECONFIG=/etc/rancher/rke2/rke2.yaml
echo "=== Nodes ==="
/var/lib/rancher/rke2/bin/kubectl get nodes -o wide
echo
echo "=== All Pods ==="
/var/lib/rancher/rke2/bin/kubectl get pods -A
echo
echo "=== Services ==="
systemctl is-active rke2-server
systemctl is-active rancher-system-agent'

echo "=== 修复完成 ==="

五、更新后的 init.sh

CA checksum 已更新为最新值：

bash

curl --insecure -fL https://gpu.ai-ear.cn/system-agent-install.sh \
  | sudo sh -s - \
    --server https://gpu.ai-ear.cn \
    --label 'cattle.io/os=linux' \
    --token jtk9h7hxkrzzc4vrjcvx4sqmt88s4x4k7f9jpdvs8bfs5r74ssdxqp \
    --ca-checksum 6903fb1eaadc30860dfe49b374cc4dedd7f45fb45d2f578f995a53b9ebda4424 \
    --etcd --controlplane --worker

六、问题根因总结

核心问题：Harbor HTTP vs Containerd HTTPS 协议不匹配

RKE2/containerd 对 registry 连接默认使用 HTTPS，即使 registries.yaml 中写了 http://，containerd 的 certs.d 仍可能生成 https:// 的 hosts.toml，需要同时检查两处配置。

修复路径

registries.yaml  →  添加 auth 凭证 + 标准 YAML 格式
hosts.toml       →  https:// 改为 http://     ← 关键修复
rke2 restart     →  重新加载配置并拉取镜像
CA checksum      →  更新为 Rancher 当前证书哈希

经验教训

RKE2 默认 HTTPS: RKE2/containerd 对 registry 连接默认使用 HTTPS，即使 registries.yaml 中写了 http://，containerd 的 certs.d 仍可能生成 https:// 的 hosts.toml，需要同时检查两处配置。
Rancher Agent 覆盖: Rancher system-agent 下发的 registries 配置可能覆盖手动修改的内容（JSON 格式 vs YAML 格式），修复后需确认 RKE2 是否正确加载了新配置。
CA Checksum 时效性: Rancher 服务端 TLS 证书更换后，旧的 --ca-checksum 会失效，需要从 Rancher UI 或日志中获取新的 checksum。

七、最终验证结果

=== Nodes ===
NAME              STATUS   ROLES                AGE    VERSION          INTERNAL-IP
szb122037.local   Ready    control-plane,etcd   5m     v1.35.6+rke2r1   192.168.122.37

=== All Pods (全部 Running/Completed) ===
NAMESPACE         NAME                                                   READY   STATUS
calico-system     calico-kube-controllers-b8cc449fd-tn2lb                1/1     Running
calico-system     calico-node-jxtzx                                      1/1     Running
calico-system     calico-typha-7f5d67f7bc-gftk4                          1/1     Running
cattle-system     cattle-cluster-agent-86d59bbc9c-gx6mj                  1/1     Running
kube-system       cloud-controller-manager-szb122037.local               1/1     Running
kube-system       etcd-szb122037.local                                   1/1     Running
kube-system       kube-apiserver-szb122037.local                         1/1     Running
kube-system       kube-controller-manager-szb122037.local                1/1     Running
kube-system       kube-proxy-szb122037.local                             1/1     Running
kube-system       kube-scheduler-szb122037.local                         1/1     Running
kube-system       rke2-coredns-rke2-coredns-84c8f55475-g2dvk             1/1     Running
kube-system       rke2-metrics-server-645859dfd8-vd2jw                   1/1     Running
kube-system       rke2-snapshot-controller-5c5ffd8c45-gb8zv              1/1     Running
kube-system       rke2-traefik-2sz7f                                     1/1     Running
tigera-operator   tigera-operator-5577bd4677-txj9x                       1/1     Running

所有 25 个 Pod 正常运行，0 个异常。

节点 192.168.122.37 集群初始化排查修复记录 ​

一、集群环境概览 ​

二、排查思路总览 ​

三、具体排查与修复步骤 ​

步骤 1：SSH 免密连通性检查 ​

步骤 3：RKE2 服务状态排查 ​

步骤 4：修复 Registries 配置（核心修复） ​

4.1 修复 registries.yaml ​

4.2 修复 containerd certs.d hosts.toml ​

4.3 重启 RKE2 并验证 ​

步骤 5：节点就绪状态验证 ​

步骤 6：修复 cattle-cluster-agent CA Checksum ​

四、完整修复脚本 ​

五、更新后的 init.sh ​

六、问题根因总结 ​

核心问题：Harbor HTTP vs Containerd HTTPS 协议不匹配 ​

修复路径 ​

经验教训 ​

七、最终验证结果 ​

节点 192.168.122.37 集群初始化排查修复记录

一、集群环境概览

二、排查思路总览

三、具体排查与修复步骤

步骤 1：SSH 免密连通性检查

步骤 3：RKE2 服务状态排查

步骤 4：修复 Registries 配置（核心修复）

4.1 修复 registries.yaml

4.2 修复 containerd certs.d hosts.toml

4.3 重启 RKE2 并验证

步骤 5：节点就绪状态验证

步骤 6：修复 cattle-cluster-agent CA Checksum

四、完整修复脚本

五、更新后的 init.sh

六、问题根因总结

核心问题：Harbor HTTP vs Containerd HTTPS 协议不匹配

修复路径

经验教训

七、最终验证结果