首批通过分布式安全可靠测评，为关键业务系统打造
划线反馈
节点频繁启停
更新时间：2026-07-02 15:41
分享
适用版本：V3.1.x、V3.2.x内容类型：Troubleshoot
问题现象
集群中某一台节点 5:04 -> 6:05 期间频繁启停，查看 __all_rootservice_history 结果中有 start server 和 stop server 的命令下发。
关键信息
存在 server report status change 相关日志，且 new report status=1，认为磁盘存在问题。
➜ 119_rslog grep "server report status change" rootservice*
rootservice.log.20251102051301:[2025-11-02 05:04:38.717591] INFO [RS] ob_server_manager.cpp:1088 [1720068][0][xxxxx-xxxxx-xxxxx-xxxxx] [lt=12] [dc=0] server report status change(old report status=0, new report status=1)
rootservice.log.20251102051301:[2025-11-02 05:05:36.820174] INFO [RS] ob_server_manager.cpp:1088 [1720068][0][xxxxx-xxxxx-xxxxx-xxxxx] [lt=6] [dc=0] server report status change(old report status=0, new report status=1)
rootservice.log.20251102051301:[2025-11-02 05:07:02.972988] INFO [RS] ob_server_manager.cpp:1088 [1720066][0][xxxxx-xxxxx-xxxxx-xxxxx] [lt=8] [dc=0] server report status change(old report status=0, new report status=1)
rootservice.log.20251102051301:[2025-11-02 05:07:12.990784] INFO [RS] ob_server_manager.cpp:1088 [1720066][0][xxxxx-xxxxx-xxxxx-xxxxx] [lt=7] [dc=0] server report status change(old report status=1, new report status=0)
rootservice.log.20251102051301:[2025-11-02 05:08:41.147852] INFO [RS] ob_server_manager.cpp:1088 [1720067][0][xxxxx-xxxxx-xxxxx-xxxxx] [lt=15] [dc=0] server report status change(old report status=0, new report status=1)
rootservice.log.20251102051301:[2025-11-02 05:11:59.503102] INFO [RS] ob_server_manager.cpp:1088 [1720068][0][xxxxx-xxxxx-xxxxx-xxxxx] [lt=9] [dc=0] server report status change(old report status=0, new report status=1)
rootservice.log.20251102054253:[2025-11-02 05:13:31.666018] INFO [RS] ob_server_manager.cpp:1088 [1720066][0][xxxxx-xxxxx-xxxxx-xxxxx] [lt=11] [dc=0] server report status change(old report status=0, new report status=1)
rootservice.log.20251102054253:[2025-11-02 05:13:35.673101] INFO [RS] ob_server_manager.cpp:1088 [1720066][0][xxxxx-xxxxx-xxxxx-xxxxx] [lt=9] [dc=0] server report status change(old report status=1, new report status=0)
rootservice.log.20251102054253:[2025-11-02 05:14:11.737293] INFO [RS] ob_server_manager.cpp:1088 [1720066][0][xxxxx-xxxxx-xxxxx-xxxxx] [lt=10] [dc=0] server report status change(old report status=0, new report status=1)
rootservice.log.20251102054253:[2025-11-02 05:15:11.843691] INFO [RS] ob_server_manager.cpp:1088 [1720068][0][xxxxx-xxxxx-xxxxx-xxxxx] [lt=11] [dc=0] server report status change(old report status=1, new report status=0)
rootservice.log.20251102054253:[2025-11-02 05:15:41.896744] INFO [RS] ob_server_manager.cpp:1088 [1720067][0][xxxxx-xxxxx-xxxxx-xxxxx] [lt=10] [dc=0] server report status change(old report status=0, new report status=1)
rootservice.log.20251102054253:[2025-11-02 05:16:29.981688] INFO [RS] ob_server_manager.cpp:1088 [1720066][0][xxxxx-xxxxx-xxxxx-xxxxx] [lt=12] [dc=0] server report status change(old report status=0, new report status=1)
rootservice.log.20251102054253:[2025-11-02 05:18:46.222445] INFO [RS] ob_server_manager.cpp:1088 [1720067][0][xxxxx-xxxxx-xxxxx-xxxxx] [lt=12] [dc=0] server report status change(old report status=1, new report status=0)
rootservice.log.20251102054253:[2025-11-02 05:19:16.275557] INFO [RS] ob_server_manager.cpp:1088 [1720067][0][xxxxx-xxxxx-xxxxx-xxxxx] [lt=9] [dc=0] server report status change(old report status=0, new report status=1)
rootservice.log.20251102054253:[2025-11-02 05:19:46.328820] INFO [RS] ob_server_manager.cpp:1088 [1720067][0][xxxxx-xxxxx-xxxxx-xxxxx] [lt=11] [dc=0] server report status change(old report status=1, new report status=0)
rootservice.log.20251102054253:[2025-11-02 05:21:04.466549] INFO [RS] ob_server_manager.cpp:1088 [1720066][0][xxxxx-xxxxx-xxxxx-xxxxx] [lt=9] [dc=0] server report status change(old report status=0, new report status=1)
rootservice.log.20251102054253:[2025-11-02 05:21:52.551333] INFO [RS] ob_server_manager.cpp:1088 [1720066][0][xxxxx-xxxxx-xxxxx-xxxxx] [lt=13] [dc=0] server report status change(old report status=1, new report status=0)
rootservice.log.20251102054253:[2025-11-02 05:22:22.604452] INFO [RS] ob_server_manager.cpp:1088 [1720066][0][xxxxx-xxxxx-xxxxx-xxxxx] [lt=12] [dc=0] server report status change(old report status=0, new report status=1)
rootservice.log.20251102054253:[2025-11-02 05:27:37.161001] INFO [RS] ob_server_manager.cpp:1088 [1720066][0][xxxxx-xxxxx-xxxxx-xxxxx] [lt=13] [dc=0] server report status change(old report status=1, new report status=0)
rootservice.log.20251102054253:[2025-11-02 05:28:07.214117] INFO [RS] ob_server_manager.cpp:1088 [1720067][0][xxxxx-xxxxx-xxxxx-xxxxx] [lt=11] [dc=0] server report status change(old report status=0, new report status=1)
rootservice.log.20251102054253:[2025-11-02 05:29:05.317182] INFO [RS] ob_server_manager.cpp:1088 [1720066][0][xxxxx-xxxxx-xxxxx-xxxxx] [lt=15] [dc=0] server report status change(old report status=1, new report status=0)
rootservice.log.20251102054253:[2025-11-02 05:29:35.370229] INFO [RS] ob_server_manager.cpp:1088 [1720067][0][xxxxx-xxxxx-xxxxx-xxxxx] [lt=7] [dc=0] server report status change(old report status=0, new report status=1)
rootservice.log.20251102054253:[2025-11-02 05:29:47.391585] INFO [RS] ob_server_manager.cpp:1088 [1720067][0][xxxxx-xxxxx-xxxxx-xxxxx] [lt=12] [dc=0] server report status change(old report status=1, new report status=0)
rootservice.log.20251102054253:[2025-11-02 05:31:01.523221] INFO [RS] ob_server_manager.cpp:1088 [1720067][0][xxxxx-xxxxx-xxxxx-xxxxx] [lt=8] [dc=0] server report status change(old report status=1, new report status=0)
rootservice.log.20251102054253:[2025-11-02 05:31:41.593920] INFO [RS] ob_server_manager.cpp:1088 [1720066][0][xxxxx-xxxxx-xxxxx-xxxxx] [lt=10] [dc=0] server report status change(old report status=0, new report status=1)
rootservice.log.20251102054253:[2025-11-02 05:35:23.985972] INFO [RS] ob_server_manager.cpp:1088 [1720068][0][xxxxx-xxxxx-xxxxx-xxxxx] [lt=9] [dc=0] server report status change(old report status=0, new report status=1)
rootservice.log.20251102061921:[2025-11-02 05:45:17.033899] INFO [RS] ob_server_manager.cpp:1088 [1720067][0][xxxxx-xxxxx-xxxxx-xxxxx] [lt=10] [dc=0] server report status change(old report status=1, new report status=0)
rootservice.log.20251102061921:[2025-11-02 05:46:17.139898] INFO [RS] ob_server_manager.cpp:1088 [1720066][0][xxxxx-xxxxx-xxxxx-xxxxx] [lt=9] [dc=0] server report status change(old report status=0, new report status=1)
rootservice.log.20251102061921:[2025-11-02 05:47:01.218075] INFO [RS] ob_server_manager.cpp:1088 [1720066][0][xxxxx-xxxxx-xxxxx-xxxxx] [lt=10] [dc=0] server report status change(old report status=1, new report status=0)
rootservice.log.20251102061921:[2025-11-02 05:48:27.370648] INFO [RS] ob_server_manager.cpp:1088 [1720066][0][xxxxx-xxxxx-xxxxx-xxxxx] [lt=10] [dc=0] server report status change(old report status=0, new report status=1)
rootservice.log.20251102061921:[2025-11-02 05:49:29.480804] INFO [RS] ob_server_manager.cpp:1088 [1720068][0][xxxxx-xxxxx-xxxxx-xxxxx] [lt=9] [dc=0] server report status change(old report status=0, new report status=1)
rootservice.log.20251102061921:[2025-11-02 05:52:15.778137] INFO [RS] ob_server_manager.cpp:1088 [1720068][0][xxxxx-xxxxx-xxxxx-xxxxx] [lt=9] [dc=0] server report status change(old report status=1, new report status=0)
rootservice.log.20251102061921:[2025-11-02 05:52:57.852571] INFO [RS] ob_server_manager.cpp:1088 [1720066][0][xxxxx-xxxxx-xxxxx-xxxxx] [lt=11] [dc=0] server report status change(old report status=0, new report status=1)
rootservice.log.20251102061921:[2025-11-02 06:00:20.634605] INFO [RS] ob_server_manager.cpp:1088 [1720067][0][xxxxx-xxxxx-xxxxx-xxxxx] [lt=9] [dc=0] server report status change(old report status=1, new report status=0)
rootservice.log.20251102061921:[2025-11-02 06:01:40.776906] INFO [RS] ob_server_manager.cpp:1088 [1720066][0][xxxxx-xxxxx-xxxxx-xxxxx] [lt=11] [dc=0] server report status change(old report status=0, new report status=1)
rootservice.log.20251102061921:[2025-11-02 06:05:13.152523] INFO [RS] ob_server_manager.cpp:1088 [1720067][0][xxxxx-xxxxx-xxxxx-xxxxx] [lt=9] [dc=0] server report status change(old report status=1, new report status=0)
➜ 119_rslog
此外可以查看节点 stop 开始时间点的 OBServer 日志中有没有 set_disk_error: attention!!! 的 error 级别日志，有的话可以证明节点认为磁盘故障。
问题原因
磁盘 IO 有问题导致 RS 主将节点标记为异常节点，下发 stop server 命令。
问题的风险及影响
节点被自动 stop 规避 IO 问题带来的抖动，短时间内部分副本不可服务。
适用版本
OceanBase 数据库 V3.x 版本。
解决方法
不用修复，OceanBase 对于磁盘抖动的避险行为。
规避方式
可以观察一下 sdb 和 sdc 盘是否有比较多写入响应时间和平时比很高很多的情况，如果环境的磁盘不稳定，超时比较敏感，可以适当放大两个配置项的设置，容忍一些 IO 超时：data_storage_warning_tolerance_time_，data_storage_error_tolerance_time_，详情参考 OceanBase 数据库 V3.X 版本的磁盘故障检测原理。
