主题
运维管理体系
1. 监控告警体系
1.1 监控架构
┌──────────────────────────────────────────────────────┐
│ 监控体系 │
│ │
│ ┌──────────┐ ┌──────────┐ ┌──────────────────┐ │
│ │Prometheus│ │ Grafana │ │ AlertManager │ │
│ │ (采集) │ │ (展示) │ │ (告警) │ │
│ └────┬─────┘ └────┬─────┘ └────────┬─────────┘ │
│ │ │ │ │
│ ┌────┴─────────────┴──────────────────┴──────────┐ │
│ │ 监控数据源 │ │
│ │ • OpenStack API metrics │ │
│ │ • Node exporter (每个物理节点) │ │
│ │ • Ceph exporter (Ceph mgr prometheus) │ │
│ │ • OVS/OVN exporter │ │
│ │ • MySQL exporter │ │
│ │ • RabbitMQ exporter │ │
│ │ • HAProxy exporter │ │
│ │ • Container exporter (cAdvisor) │ │
│ └─────────────────────────────────────────────────┘ │
└──────────────────────────────────────────────────────┘1.2 Prometheus 配置
yaml
# /etc/prometheus/prometheus.yml
global:
scrape_interval: 15s
evaluation_interval: 15s
rule_files:
- /etc/prometheus/rules/*.yml
alerting:
alertmanagers:
- static_configs:
- targets: ['mgmt-01:9093']
scrape_configs:
# OpenStack 服务
- job_name: 'openstack-nova'
static_configs:
- targets: ['ctrl-01:8774', 'ctrl-02:8774', 'ctrl-03:8774']
- job_name: 'openstack-neutron'
static_configs:
- targets: ['ctrl-01:9696', 'ctrl-02:9696']
# 节点监控 (千节点分批采集)
- job_name: 'node-exporter'
file_sd_configs:
- files: ['/etc/prometheus/targets/compute-*.json']
relabel_configs:
- source_labels: [__address__]
target_label: instance
# Ceph
- job_name: 'ceph'
static_configs:
- targets: ['ceph-mon-01:9283', 'ceph-mon-02:9283', 'ceph-mon-03:9283']
# 基础设施
- job_name: 'mysql'
static_configs:
- targets: ['ctrl-01:9104', 'ctrl-02:9104', 'ctrl-03:9104']
- job_name: 'rabbitmq'
static_configs:
- targets: ['ctrl-01:15692', 'ctrl-02:15692']1.3 核心告警规则
yaml
# /etc/prometheus/rules/openstack.yml
groups:
- name: openstack_critical
rules:
# Nova 服务异常
- alert: NovaServiceDown
expr: openstack_nova_agent_state{state="down"} > 0
for: 5m
labels:
severity: critical
annotations:
summary: "Nova {{ $labels.service }} 在 {{ $labels.hostname }} 宕机"
# Neutron Agent 异常
- alert: NeutronAgentDown
expr: openstack_neutron_agent_state{adminState="down"} > 0
for: 3m
labels:
severity: critical
annotations:
summary: "Neutron agent {{ $labels.service }} 在 {{ $labels.hostname }} 异常"
# API 响应超时
- alert: APIResponseTimeHigh
expr: histogram_quantile(0.99, openstack_api_request_duration_seconds) > 2
for: 5m
labels:
severity: warning
annotations:
summary: "API P99 响应超过 2s"
- name: ceph_alerts
rules:
- alert: CephHealthWarn
expr: ceph_health_status == 1
for: 10m
labels:
severity: warning
- alert: CephHealthCrit
expr: ceph_health_status == 2
for: 5m
labels:
severity: critical
- alert: CephOSDDown
expr: ceph_osd_up == 0
for: 5m
labels:
severity: critical
- alert: CephPoolFull
expr: (ceph_pool_bytes_used / ceph_pool_max_bytes) > 0.8
for: 30m
labels:
severity: warning
- name: compute_node_alerts
rules:
- alert: ComputeNodeHighLoad
expr: node_load15 / node_cpu_count > 4
for: 10m
labels:
severity: warning
- alert: ComputeNodeDiskFull
expr: (node_filesystem_avail_bytes{mountpoint="/"} /
node_filesystem_size_bytes{mountpoint="/"}) < 0.1
for: 5m
labels:
severity: critical
- alert: ComputeNodeHighMemory
expr: (node_memory_MemAvailable_bytes /
node_memory_MemTotal_bytes) < 0.05
for: 5m
labels:
severity: critical2. 日志管理
2.1 集中日志架构
┌──────────┐ ┌──────────┐ ┌──────────┐
│ 各服务 │────►│Filebeat │────►│Elastic- │
│ 容器日志 │ │ (采集) │ │ search │
└──────────┘ └──────────┘ │ (存储) │
└────┬─────┘
│
┌────┴─────┐
│ Kibana │
│ (查询) │
└──────────┘2.2 日志保留策略
| 日志类型 | 保留期 | Hot | Warm | Cold |
|---|---|---|---|---|
| API 访问日志 | 90天 | 7天 | 30天 | 53天 |
| 审计日志 | 1年 | 30天 | 90天 | 245天 |
| 系统日志 | 30天 | 7天 | 23天 | - |
| Ceph 日志 | 60天 | 7天 | 53天 | - |
3. 日常运维操作
3.1 计算节点管理
bash
# 添加新计算节点
kolla-ansible -i inventory deploy --limit compute-1001
# 禁用计算节点 (维护模式)
openstack compute service set --disable \
compute-001 nova-compute --disable-reason "maintenance"
# 迁移节点上所有 VM
nova-manage cell_v2 discover_hosts
for vm in $(openstack server list --host compute-001 -f value -c ID); do
openstack server migrate $vm --live-migration --wait
done
# 重新启用
openstack compute service set --enable compute-001 nova-compute3.2 虚拟机热迁移
bash
# 指定目标迁移
openstack server migrate <vm-id> --host compute-002 --wait
# 自动调度迁移
openstack server migrate <vm-id> --live-migration --wait
# 批量迁移 (维护窗口)
for vm in $(openstack server list --host $HOST -f value -c ID); do
openstack server migrate $vm --live-migration &
done
wait3.3 故障处理流程
故障发生
│
├── 自动告警 (Prometheus → AlertManager → 飞书/钉钉)
│
├── 影响评估
│ ├── 单 VM 故障 → 自动重建 (Nova HA)
│ ├── 单节点故障 → VM 自动迁移
│ ├── 控制节点故障 → VIP 自动切换
│ └── 存储故障 → Ceph 自动恢复
│
├── 排查定位
│ ├── 检查日志: docker logs <container>
│ ├── 检查服务: docker ps | grep nova
│ ├── 检查资源: df -h / free -m
│ └── 检查网络: ovs-vsctl show / ip addr
│
└── 恢复确认
├── 服务状态: openstack compute service list
├── Ceph 状态: ceph status
└── VM 状态: openstack server list --status ACTIVE4. 备份与恢复
4.1 关键数据备份
bash
#!/bin/bash
# backup-openstack.sh - 每日执行
BACKUP_DIR="/backup/openstack/$(date +%Y%m%d)"
mkdir -p $BACKUP_DIR
# 1. 数据库备份
for db in nova keystone neutron glance cinder heat; do
docker exec mariadb mysqldump \
--single-transaction --routines --triggers \
$db > $BACKUP_DIR/${db}.sql
done
# 2. Keystone Fernet 密钥备份
tar czf $BACKUP_DIR/fernet-keys.tar.gz \
/etc/kolla/keystone/fernet-keys/
# 3. Kolla 配置备份
tar czf $BACKUP_DIR/kolla-config.tar.gz \
/etc/kolla/
# 4. 保留最近 30 天备份
find /backup/openstack/ -maxdepth 1 -mtime +30 -exec rm -rf {} \;4.2 数据库恢复
bash
# 恢复 Nova 数据库
docker exec -i mariadb mysql nova < /backup/openstack/20250101/nova.sql
# 重启相关服务
docker restart nova_api nova_conductor nova_scheduler5. 容量规划与扩容
5.1 容量监控指标
| 指标 | 告警阈值 | 扩容触发 |
|---|---|---|
| CPU 使用率 | > 70% | 持续 2 周 > 60% |
| 内存使用率 | > 80% | 持续 2 周 > 70% |
| 存储使用率 | > 75% | > 65% 时规划 |
| 网络带宽 | > 80% | 持续峰值 > 60% |
5.2 扩容流程
bash
# 计算节点扩容
# 1. 准备新节点 (硬件上架、OS安装、网络配置)
# 2. 加入 Kolla-Ansible inventory
# 3. 执行部署
kolla-ansible -i inventory bootstrap-servers --limit new-compute
kolla-ansible -i inventory deploy --limit new-compute
# 4. 验证
openstack compute service list | grep new-compute
# 5. 自动发现
nova-manage cell_v2 discover_hosts
# Ceph 扩容: 参见 storage-design.md 第 5 节