问题现象
启动 OBServer 失败，进程会退出，查看对应的 OBServer 节点上的 observer.log 里出现 Structure needs cleaning。
[2024-09-24 10:31:43.950117] WDIAG [SHARE] pread_impl (ob_local_device.cpp:1578) [15124][observer][T0][Y0-0000000000000000-0-0] [lt=3][errcode=-4009] failed to pread(ret=-4009, fd=749, read_sz=65536, read_offset=0, errno=117, errmsg="Structure needs cleaning")
[2024-09-24 10:31:43.950163] WDIAG [SHARE] pread (ob_local_device.cpp:882) [15124][observer][T0][Y0-0000000000000000-0-0] [lt=46][errcode=-4009] read main super block fail(ret=-4009, offset=0, size=65536, buf=0xfffc3b7ed000, block_fd_=749, errmsg="Structure needs cleaning")
[2024-09-24 10:31:43.950173] WDIAG [STORAGE.BLKMGR] read_super_block (ob_block_manager.cpp:390) [15124][observer][T0][Y0-0000000000000000-0-0] [lt=8][errcode=-4009] fail to write super block(ret=-4009, super_block_fd={first_id:0, second_id:0, device_handle:null}, super_block_buf_holder={buf:0xfffc3b7ed000, len:65536}, read_size=0)
[2024-09-24 10:31:43.950183] WDIAG [STORAGE.BLKMGR] start (ob_block_manager.cpp:206) [15124][observer][T0][Y0-0000000000000000-0-0] [lt=9][errcode=-4009] fail to read server super block(ret=-4009)
[2024-09-24 10:31:43.950191] ERROR issue_dba_error (ob_log.cpp:1866) [15124][observer][T0][Y0-0000000000000000-0-0] [lt=6][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4009, file="ob_server.cpp", line_no=857, info="start block manager fail")
[2024-09-24 10:31:43.950198] EDIAG [SERVER] start (ob_server.cpp:857) [15124][observer][T0][Y0-0000000000000000-0-0] [lt=7][errcode=-4009] start block manager fail(ret=-4009, ret="OB_IO_ERROR") BACKTRACE:0xf820e88 0x54a553c 0x54a50d0 0x54a4d74 0x5486140 0x8302ed8 0x8301020 0x5478fc0 0xfffcc8e2485c 0x3cb9878
问题原因
文件系统损坏。
适用版本
OceanBase 数据库 V4.x 版本。
解决方法
通过 dmesg -T 在输出结果中看到如下内容：
[Tue Sep 24 10:05:10 2024] XFS (dm-4): Corruption warning: Metadata has LSN (3193:1737048) ahead of current LSN (3193:1669624). Please unmount and run xfs_repair (>= v4.3) to resolve.卸载已挂载的对应的文件系统。
umount file-system使用 xfs_repair 工具来修复卸载的文件系统。
xfs_repair block-device注意
此处强烈不推荐加上 -L 选项。
重新挂载对应的文件系统。
mount file-system如果重新挂载对应文件系统失败，建议替换该 observer 节点或者 delete observer 后修复硬件问题后再重新加回该集群。
