fix(k3s): 持久化香港控制面磁盘调度缓解
原因：香港虚拟磁盘在低吞吐下仍出现高同步写延迟，触发 etcd slow apply、Handler timeout 和 PostgreSQL API Server 连通性异常。\n\n影响：新增仅针对香港根块设备的 mq-deadline 配置脚本，应用时不重启 K3s，并提供显式 rollback 恢复 none；运行手册明确该项仅为缓解，不能替代高性能云盘。\n\n验证：新香港节点同步写探针从 13.28ms 降至 6.25ms；bash -n、ShellCheck、manual-release-test.sh 通过。
This commit is contained in:
@@ -96,6 +96,30 @@ K3s containerd 同时配置 Docker Hub、GHCR 与 Quay 的镜像端点；已有
|
||||
server 都保持 etcd quorum。宁波停机可能令香港数据库副本晋升；首次切换前如需恢复宁波
|
||||
初始主库，显式执行 `promote-cnpg-primary.sh`，故障恢复流程不得自动回切。
|
||||
|
||||
#### 香港控制面云盘延迟缓解
|
||||
|
||||
生产同构验收前必须用 etcd metrics 的增量窗口检查 backend commit 与 WAL fsync，不能只看
|
||||
节点 Ready。香港虚拟磁盘若使用 `none` 调度器并出现持续 `Handler timeout`、
|
||||
`etcdserver: request timed out` 或 PostgreSQL `API server connectivity issue`，可以先应用已验证、
|
||||
可回滚的 `mq-deadline` 缓解：
|
||||
|
||||
```bash
|
||||
./scripts/cluster/configure-control-plane-io.sh status
|
||||
./scripts/cluster/configure-control-plane-io.sh apply
|
||||
```
|
||||
|
||||
该脚本只修改香港根块设备调度器，写入一个早于 `k3s.service` 执行的 systemd oneshot；应用时
|
||||
不重启 K3s。回退命令会删除该 unit 并恢复 `none`：
|
||||
|
||||
```bash
|
||||
./scripts/cluster/configure-control-plane-io.sh rollback
|
||||
```
|
||||
|
||||
这只是尾延迟缓解，不是慢云盘的容量认证。任何 60 秒窗口中 backend commit 平均值达到
|
||||
50 ms、WAL fsync 达到 15 ms，或仍出现上述控制面硬错误，都必须停止验收。最终修复应升级
|
||||
为具有稳定同步写延迟的高性能云盘，或按 etcd 成员替换流程迁移香港控制节点；禁止以
|
||||
`unsafe-no-fsync`、双成员 etcd 或放宽硬门禁代替存储修复。
|
||||
|
||||
### 3. 平台和入口预置
|
||||
|
||||
```bash
|
||||
|
||||
+84
@@ -0,0 +1,84 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
script_dir=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
|
||||
# shellcheck source=scripts/cluster/common.sh
|
||||
source "$script_dir/common.sh"
|
||||
|
||||
action=${1:-status}
|
||||
[[ $action == status || $action == apply || $action == rollback ]] || {
|
||||
echo 'usage: configure-control-plane-io.sh {status|apply|rollback}' >&2
|
||||
exit 64
|
||||
}
|
||||
|
||||
load_cluster_env
|
||||
unit_name=easyai-k3s-io-scheduler.service
|
||||
unit_path=/etc/systemd/system/$unit_name
|
||||
|
||||
cluster_ssh "$CLUSTER_HONGKONG_HOST" bash -s -- "$action" "$unit_name" "$unit_path" <<'REMOTE'
|
||||
set -euo pipefail
|
||||
action=$1
|
||||
unit_name=$2
|
||||
unit_path=$3
|
||||
|
||||
root_source=$(findmnt -n -o SOURCE /)
|
||||
root_partition=${root_source#/dev/}
|
||||
device=$(lsblk -no PKNAME "/dev/$root_partition" | head -n 1)
|
||||
[[ $device =~ ^[a-zA-Z0-9._-]+$ && -f /sys/block/$device/queue/scheduler ]]
|
||||
|
||||
current_scheduler() {
|
||||
sed -n 's/.*\[\([^]]*\)\].*/\1/p' "/sys/block/$device/queue/scheduler"
|
||||
}
|
||||
|
||||
case $action in
|
||||
status)
|
||||
configured=false
|
||||
systemctl is-enabled --quiet "$unit_name" 2>/dev/null && configured=true
|
||||
printf 'control_plane_io=STATUS node=hongkong device=%s scheduler=%s configured=%s\n' \
|
||||
"$device" "$(current_scheduler)" "$configured"
|
||||
;;
|
||||
apply)
|
||||
grep -qw mq-deadline "/sys/block/$device/queue/scheduler" || {
|
||||
echo 'mq-deadline is not supported by the Hong Kong root block device' >&2
|
||||
exit 1
|
||||
}
|
||||
temporary=$(mktemp /etc/systemd/system/.easyai-k3s-io-scheduler.XXXXXX)
|
||||
cleanup() {
|
||||
[[ -f ${temporary:-} ]] && rm -f -- "$temporary" || true
|
||||
}
|
||||
trap cleanup EXIT
|
||||
cat >"$temporary" <<EOF
|
||||
[Unit]
|
||||
Description=EasyAI K3s control-plane disk scheduler
|
||||
After=local-fs.target
|
||||
Before=k3s.service
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=/bin/sh -c 'printf "%s\\n" mq-deadline > /sys/block/$device/queue/scheduler'
|
||||
ExecStop=/bin/sh -c 'printf "%s\\n" none > /sys/block/$device/queue/scheduler'
|
||||
RemainAfterExit=yes
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
EOF
|
||||
chmod 0644 "$temporary"
|
||||
install -o root -g root -m 0644 "$temporary" "$unit_path"
|
||||
systemctl daemon-reload
|
||||
systemctl enable --now "$unit_name" >/dev/null
|
||||
[[ $(current_scheduler) == mq-deadline ]]
|
||||
printf 'control_plane_io=APPLIED node=hongkong device=%s scheduler=mq-deadline restart=false\n' "$device"
|
||||
;;
|
||||
rollback)
|
||||
if [[ -f $unit_path ]]; then
|
||||
systemctl disable --now "$unit_name" >/dev/null 2>&1 || true
|
||||
rm -f -- "$unit_path"
|
||||
systemctl daemon-reload
|
||||
else
|
||||
printf '%s\n' none >"/sys/block/$device/queue/scheduler"
|
||||
fi
|
||||
[[ $(current_scheduler) == none ]]
|
||||
printf 'control_plane_io=ROLLED_BACK node=hongkong device=%s scheduler=none\n' "$device"
|
||||
;;
|
||||
esac
|
||||
REMOTE
|
||||
@@ -307,6 +307,10 @@ if grep -Fq 'archive_timeout: 60s' "$root/deploy/kubernetes/production/database.
|
||||
fi
|
||||
grep -Fq 'wal_sender_timeout: 30s' "$root/deploy/kubernetes/production/database.yaml"
|
||||
grep -Fq 'wal_receiver_timeout: 30s' "$root/deploy/kubernetes/production/database.yaml"
|
||||
bash -n "$root/scripts/cluster/configure-control-plane-io.sh"
|
||||
grep -Fq 'Before=k3s.service' "$root/scripts/cluster/configure-control-plane-io.sh"
|
||||
grep -Fq 'restart=false' "$root/scripts/cluster/configure-control-plane-io.sh"
|
||||
grep -Fq 'configure-control-plane-io.sh apply' "$root/docs/operations/k3s-ha-runbook.md"
|
||||
|
||||
"$root/tests/release/cluster-release-helper-test.sh" >/dev/null
|
||||
"$root/tests/release/production-acceptance-script-test.sh" >/dev/null
|
||||
|
||||
Reference in New Issue
Block a user