首批通过分布式安全可靠测评，为关键业务系统打造
划线反馈
Read Superblock Error 导致 OBServer 启动失败
更新时间：2023-12-18 10:01
分享
适用版本：V2.1.x、V2.2.x、V3.1.x、V3.2.x、V4.0.x、V4.1.x、V4.2.x内容类型：Troubleshoot
适用版本
OceanBase 数据库所有版本。
问题现象
通过 OCP 发现集群合并异常，于是重启问题节点，但是卡在 wait observer active 这一步，发现进程不在，于是又黑屏启动进程，进程启动后在极短时间内日志报 observer is exit，但是进程还在。以下为日志内容，有报错 read_super_block。
[2023-06-20 22:48:18.916643] WARN [STORAGE.TRANS] getClock (ob_clock_generator.h:63) [234211][406][Y0-0000000000000000] [lt=8] [dc=0] clock generator not inited
[2023-06-20 22:48:18.916921] INFO [COMMON] ob_io_disk.cpp:182 [233996][0][Y0-0000000000000000] [lt=4] [dc=0] succeed to start disk channel thread(chanel_count=16)
[2023-06-20 22:48:18.916933] INFO [COMMON] ob_io_disk.cpp:189 [233996][0][Y0-0000000000000000] [lt=11] [dc=0] Success to init ObDisk
[2023-06-20 22:48:18.916941] INFO [COMMON] ob_io_disk.cpp:760 [233996][0][Y0-0000000000000000] [lt=5] [dc=0] succeed to add disk(fd={fd:875, disk_id:{disk_idx:0, install_seq:0}}, disk_count_=1)
[2023-06-20 22:48:18.916960] INFO [STORAGE] ob_local_file_system.cpp:123 [233996][0][Y0-0000000000000000] [lt=17] [dc=0] finish open store file(ret=0, fd_={fd:875, disk_id:{disk_idx:0, install_seq:0}}, store_path_="/home/admin/oceanbase/store/ob003_test_life/sstable/block_file", data_file_size_=0)
[2023-06-20 22:48:18.917042] WARN [STORAGE] read_super_block (ob_local_file_system.cpp:363) [233996][0][Y0-0000000000000000] [lt=7] [dc=0] read superblock error.(ret=-4009, offset=0, read_size=-1, errno=5, super_block_buf_holder={buf:0x7f62bdfdd000, len:65536}, fd={fd:875, disk_id:{disk_idx:0, install_seq:0}}, errmsg="Input/output error")
[2023-06-20 22:48:18.917070] INFO [STORAGE] ob_local_file_system.cpp:368 [233996][0][Y0-0000000000000000] [lt=27] [dc=0] finish read_super_block(ret=-4009, offset=0, super_block={header:{super_block_size:0, version:0, magic:0, attr:0}, content:{create_timestamp:0, modify_timestamp:0, macro_block_size:0, total_macro_block_count:0, free_macro_block_count:0, total_file_size:0, replay_start_point:ObLogCursor{file_id=0, log_id=0, offset=0}, macro_block_meta:{block_index:-1, log_seq:0, file_id:0, file_size:0}, partition_meta:{block_index:-1, log_seq:0, file_id:0, file_size:0}, table_mgr_meta:{block_index:-1, log_seq:0, file_id:0, file_size:0}, tenant_config_meta:{block_index:-1, log_seq:0, file_id:0, file_size:0}}})
[2023-06-20 22:48:18.917092] WARN [STORAGE] read_super_block (ob_local_file_system.cpp:272) [233996][0][Y0-0000000000000000] [lt=21] [dc=0] load master super block error.(ret=-4009)
[2023-06-20 22:48:18.917105] WARN [STORAGE] read_super_block (ob_local_file_system.cpp:363) [233996][0][Y0-0000000000000000] [lt=3] [dc=0] read superblock error.(ret=-4009, offset=2097152, read_size=-1, errno=5, super_block_buf_holder={buf:0x7f62bdfdd000, len:65536}, fd={fd:875, disk_id:{disk_idx:0, install_seq:0}}, errmsg="Input/output error")
[2023-06-20 22:48:18.917112] INFO [STORAGE] ob_local_file_system.cpp:368 [233996][0][Y0-0000000000000000] [lt=7] [dc=0] finish read_super_block(ret=-4009, offset=2097152, super_block={header:{super_block_size:0, version:0, magic:0, attr:0}, content:{create_timestamp:0, modify_timestamp:0, macro_block_size:0, total_macro_block_count:0, free_macro_block_count:0, total_file_size:0, replay_start_point:ObLogCursor{file_id=0, log_id=0, offset=0}, macro_block_meta:{block_index:-1, log_seq:0, file_id:0, file_size:0}, partition_meta:{block_index:-1, log_seq:0, file_id:0, file_size:0}, table_mgr_meta:{block_index:-1, log_seq:0, file_id:0, file_size:0}, tenant_config_meta:{block_index:-1, log_seq:0, file_id:0, file_size:0}}})
[2023-06-20 22:48:18.917122] WARN [STORAGE] read_super_block (ob_local_file_system.cpp:276) [233996][0][Y0-0000000000000000] [lt=10] [dc=0] load backup super block error.(ret=-4009)
[2023-06-20 22:48:18.917125] WARN [STORAGE] init (ob_store_file.cpp:461) [233996][0][Y0-0000000000000000] [lt=3] [dc=0] Fail to load super block, (ret=-4009)
问题原因
observer 进程重启时会判断日志与数据的完整性，日志中的报错显示 observer 在读取磁盘数据（Superblock）时发生错误，一般与磁盘文件损坏或者硬件故障有关。
解决方法
对问题节点进行如下操作来恢复。
- stop server
- delete server
- add server
对此类磁盘损坏导致 OBServer 起不来的，不适用于永久下线再上线。永久下线再上线的场景，只适用于下线的主机可正常启动的情况（数据文件没有损坏），不适用于无法启动的情况（比如数据文件损害启动异常）。
