问题现象
OCP 上告警 invalid data(ret=-4070、checksum is not consistent。
OBServer 日志中存在大量 ERROR 日志。
observer.log.20240611083651:[2024-06-11 08:31:25.317598] ERROR [ARCHIVE] check_data_checksum_ (ob_archive_block.cpp:104) [1594612][3411][Y0-0000000000000000] [lt=5] [dc=0] checksum is not consistent(buf=0xffa2910f1072, data_len=117, crc_check_sum=147568363, data_checksum_=4162716401) BACKTRACE:0x97b527f 0x6a48f13 0x6a69963 0x27cb70f 0x5329e7b 0x532a1d7 0x532e5a3 0x4e1beb7 0x4e1e76b 0x4e1ec4f 0x4e1ef4f 0x51874cb 0x5187cdf 0x5187eb7 0x518811b 0x5188707 0x5188a23 0x5188d3f 0x5163c7b 0x5164617 0x5165b87 0x5165c07 0x40de3d3 0x2cdd683 0x96060c7 0x960352f 0x95ff23f 0x95ff23e
observer.log.20240611083651:[2024-06-11 08:31:25.317616] WARN [ARCHIVE] check_integrity (ob_archive_block.cpp:81) [1594612][3411][Y0-0000000000000000] [lt=17] [dc=0] check data checksum fails
observer.log.20240611083651:[2024-06-11 08:31:25.317627] ERROR [ARCHIVE] extract_block_meta_ (ob_archive_entry_iterator.cpp:539) [1594612][3411][Y0-0000000000000000] [lt=3] [dc=0] invalid data(ret=-4070, ret="OB_INVALID_DATA", pg_key_={tid:1102810162711313, partition_id:0, part_cnt:0}, file_id_=624, pos=104, meta={magic_:16706, total_len_:221, timestamp_:1717679643631954, clog_epoch_id_:1717650303607630, accum_checksum_:361449248, min_log_id_in_file_:46446, min_log_ts_in_file_:1717650303613920, max_log_id_:46463, max_checkpoint_ts_:1717679598436408, max_log_submit_ts_:1717679598480283, input_bytes_:0, output_bytes_:0, data_checksum_:4162716401, meta_checksum_:3751868946}) BACKTRACE:0x97b527f 0x6a48f13 0x28070e3 0x2807777 0x4e19577 0x4e1c0f7 0x4e1e76b 0x4e1ec4f 0x4e1ef4f 0x51874cb 0x5187cdf 0x5187eb7 0x518811b 0x5188707 0x5188a23 0x5188d3f 0x5163c7b 0x5164617 0x5165b87 0x5165c07 0x40de3d3 0x2cdd683 0x96060c7 0x960352f 0x95ff23f 0x95ff23e
observer.log.20240611083651:[2024-06-11 08:31:25.317640] WARN [ARCHIVE] try_construct_log_buf_ (ob_archive_entry_iterator.cpp:492) [1594612][3411][Y0-0000000000000000] [lt=10] [dc=0] extract_block_meta_ fail(ret=-4070, ret="OB_INVALID_DATA", pg_key_={tid:1102810162711313, partition_id:0, part_cnt:0})
其中关键日志信息 checksum is not consistent。
[2024-06-11 08:31:25.317598] ERROR [ARCHIVE] check_data_checksum_ (ob_archive_block.cpp:104) [1594612][3411][Y0-0000000000000000] [lt=5] [dc=0] checksum is not consistent(buf=0xffa2910f1072, data_len=117, crc_check_sum=147568363, data_checksum_=4162716401) BACKTRACE:0x97b527f 0x6a48f13 0x6a69963 0x27cb70f 0x5329e7b 0x532a1d7 0x532e5a3 0x4e1beb7 0x4e1e76b 0x4e1ec4f 0x4e1ef4f 0x51874cb 0x5187cdf 0x5187eb7 0x518811b 0x5188707 0x5188a23 0x5188d3f 0x5163c7b 0x5164617 0x5165b87 0x5165c07 0x40de3d3 0x2cdd683 0x96060c7 0x960352f 0x95ff23f 0x95ff23e
问题原因
告警原因为检测到归档文件的 checksum 异常，归档文件损坏在中间位置。
问题的风险及影响
会导致日志归档任务卡住；会导致无法恢复到此文件对应的时间节点；会导致 OCP 产生告警。
影响租户
影响 OceanBase 数据库中的 SYS 租户和 Oracle 租户以及 MySQL 租户。
适用版本
OceanBase 数据库 V3.x 版本。
解决方法及规避方式
重启归档，NFS（网络文件系统）需要严格按照官网文档所述进行标准化挂载。
