问题现象
OceanBase 数据库 V4.x 业务租户卡合并,observer.log 日志存在报错 read invalid data, need clean cache, maybe storage device cann't guarantee linear consistency reading(ret=-4070。
关键日志信息。
[2024-03-06 08:07:20.663449] WDIAG [PALF] next (log_iterator_impl.h:593) [3900563][T1014_ReplaySrv][T1014][Y0-0000000000000000-0-0] [lt=0][errcode=-4070] read invalid data, need clean cache, maybe storage device cann't guarantee linear consistency reading(ret=-4070, this={buf_:0x7f413a805000, next_round_pread_size:2121728, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:2121728, log_storage_:{IteratorStorage:{start_lsn:{lsn:46394078050}, end_lsn:{lsn:46396199778}, read_buf:{buf_len_:2125824, buf_:0x7f413a805000}, block_size:67104768, log_storage_:0x7f41d6ff55b0}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1709363079392798465}, curr_entry:{LogEntryHeader:{magic:19528, version:1, log_size:13447, scn_:{val:1709363079392798465}, data_checksum:2751407106, flag:0}}, init_mode_version:0, accumlate_checksum:3699105139})
[2024-03-06 08:07:20.663495] WDIAG [PALF] next (log_iterator_impl.h:600) [3900563][T1014_ReplaySrv][T1014][Y0-0000000000000000-0-0] [lt=47][errcode=-4070] get_next_entry_ failed(ret=-4070, this={buf_:0x7f413a805000, next_round_pread_size:2121728, curr_read_pos:0, curr_read_buf_start_pos:0, curr_read_buf_end_pos:0, log_storage_:{IteratorStorage:{start_lsn:{lsn:46394078050}, end_lsn:{lsn:46394078050}, read_buf:{buf_len_:2125824, buf_:0x7f413a805000}, block_size:67104768, log_storage_:0x7f41d6ff55b0}, IteratorStorageType::"DiskIteratorStorage"}, curr_entry_is_raw_write:false, curr_entry_size:0, prev_entry_scn:{val:1709363079392798465}, curr_entry:{LogEntryHeader:{magic:19528, version:1, log_size:13447, scn_:{val:1709363079392798465}, data_checksum:2751407106, flag:0}}, init_mode_version:0, accumlate_checksum:3699105139})
问题原因
由于 clog 磁盘数据文件损坏,导致租户 Replay 卡住,节点备机读无法推进,阻塞租户合并。
问题的风险及影响
阻塞租户合并进行。
影响租户
影响 OceanBase 数据库中的 SYS 租户和 Oracle 租户以及 MySQL 租户。
适用版本
OceanBase 数据库 V4.x 版本。
解决方法
locality 变更删除租户日志流副本,然后重新添加回来。
规避方式
增强系统硬件巡检,及时发现并进行技术处理。