首批通过分布式安全可靠测评,为关键业务系统打造
OceanBase 数据库 V3.2.4 版本合并超时 OBServer 日志报错 -4024 advance buffer failed
更新时间:2026-06-12 08:51
问题现象
OBServer 合并超时,OBServer 日志报错 -4024 advance buffer failed。
关键诊断信息
三节点合并超时。
MySQL [oceanbase]> select * from __all_zone where name='merge_status';
+----------------------------+----------------------------+-------+--------------+-------+---------+
| gmt_create | gmt_modified | zone | name | value | info |
+----------------------------+----------------------------+-------+--------------+-------+---------+
| 2024-06-07 01:08:01.834756 | 2024-12-27 04:50:54.524330 | | merge_status | 2 | TIMEOUT |
| 2024-06-07 01:08:01.835808 | 2024-12-27 04:50:54.523278 | zone1 | merge_status | 2 | TIMEOUT |
| 2024-06-07 01:08:01.837913 | 2024-12-27 04:50:54.527489 | zone2 | merge_status | 2 | TIMEOUT |
| 2024-06-07 01:08:01.838938 | 2024-12-27 04:50:54.531630 | zone3 | merge_status | 2 | TIMEOUT |
+----------------------------+----------------------------+-------+--------------+-------+---------+
4 rows in set (0.011 sec)
三节点都卡在一个分区上。
MySQL [oceanbase]> select * from oceanbase.__all_virtual_partition_table where data_version != '229';
+-----------+------------------+--------------+---------------+----------+----------+---------+---------------+-------+------+-----------+-----------+--------------+---------------+--------------+-----------------+--------------------+------------------+-------------+---------+--------------+-----------------------+--------------------+---------------+------------+--------+-----------+--------------------+------------------+
| tenant_id | table_id | partition_id | svr_ip | svr_port | sql_port | unit_id | partition_cnt | zone | role | row_count | data_size | data_version | data_checksum | row_checksum | column_checksum | is_original_leader | to_leader_time | create_time | rebuild | replica_type | status | partition_checksum | required_size | is_restore | quorum | fail_list | recovery_timestamp | memstore_percent |
+-----------+------------------+--------------+---------------+----------+----------+---------+---------------+-------+------+------------------------------------------------------------------------------------------------------------+-----------+-----------+--------------+---------------+--------------+-----------------+--------------------+------------------+-------------+---------+--------------+-----------------------+--------------------+---------------+------------+--------+-----------+--------------------+------------------+
| 1001 | 1100611139702520 | 189 | xx | 2882 | 2881 | 1007 | 0| zone2 | 2 | 387057 | 18353056 | 228 | 1110043723 | 0 | | 0 | 0 | 0 | 0 | 0 | REPLICA_STATUS_NORMAL | 0 | 33554432 | 0 | 3 | (null) | 0 | 100 |
| 1001 | 1100611139702520 | 189 | xx | 2882 | 2881 | 1004 | 0| zone3 | 2 | 387057 | 18353056 | 228 | 1110043723 | 0 | | 0 | 0 | 0 | 0 | 0 | REPLICA_STATUS_NORMAL | 0 | 33554432 | 0 | 3 | (null) | 0 | 100 |
| 1001 | 1100611139702520 | 189 | xx | 2882 | 2881 | 1001 | 0| zone1 | 1 | 387057 | 18353056 | 228 | 1110043723 | 0 | | 0 | 1732460537561386 | 0 | 0 | 0 | REPLICA_STATUS_NORMAL | 0 | 33554432 | 0 | 3 | (null) | 0 | 100 |
+-----------+------------------+--------------+---------------+----------+----------+---------+---------------+-------+------+------------------------------------------------------------------------------------------------------------+-----------+-----------+--------------+---------------+--------------+-----------------+--------------------+------------------+-------------+---------+--------------+-----------------------+--------------------+---------------+------------+--------+-----------+--------------------+------------------+
3 rows in set (12.992 sec)
RS 日志报错 -4201,这段日志不是根本原因,可以忽略。
[2024-12-27 11:11:19.690275] WDIAG [RS] ob_daily_merge_scheduler.cpp:2377 [4110528][0][xxxxx-xxxxx-xxxxx-xxxxx] [lt=18] [dc=0][errcode=-4201] fail to check column checksum(ret=-4201)
[2024-12-27 11:11:19.690300] WDIAG [RS] ob_daily_merge_scheduler.cpp:2226 [4110528][0][xxxxx-xxxxx-xxxxx-xxxxx] [lt=12] [dc=0][errcode=-4201] fail to check global index column checksum(tmp_ret=-4201)
[2024-12-27 11:11:53.691892] WDIAG [RS] ob_daily_merge_scheduler.cpp:2377 [4110528][0][xxxxx-xxxxx-xxxxx-xxxxx] [lt=17] [dc=0][errcode=-4201] fail to check column checksum(ret=-4201)
[2024-12-27 11:11:53.691910] WDIAG [RS] ob_daily_merge_scheduler.cpp:2226 [4110528][0][xxxxx-xxxxx-xxxxx-xxxxx] [lt=8] [dc=0][errcode=-4201] fail to check global index column checksum(tmp_ret=-4201)
[2024-12-27 11:12:26.928301] WDIAG [RS] ob_daily_merge_scheduler.cpp:2377 [4110528][0][xxxxx-xxxxx-xxxxx-xxxxx] [lt=24] [dc=0][errcode=-4201] fail to check column checksum(ret=-4201)
在卡副本的节点上过滤 OBServer 日志,此处为关键日志,由此可以判断是 encoding 缺陷。
grep "STORAGE.*merge.*1100611139702520" 40.*/* | grep 'WD'
40.208/observer.log:[2024-12-27 10:15:48.772555] WDIAG [STORAGE.COMPACTION] ob_partition_merge_task.cpp:189 [8284][0][xxxxx-xxxxx-xxxxx-xxxxx] [lt=11] [dc=0][errcode=-4024] Fail to merge, (ret=-4024, ctx_->param_={merge_type:0, merge_version:"229-0-0", pkey:{tid:1100611139702520, partition_id:189, part_cnt:0}, index_id:1100611139702520, schedule_merge_type:0, pg_key:{tid:1100611139702520, partition_id:189, part_cnt:0}}, idx_=0)
40.208/observer.log.20241227101440621:[2024-12-27 10:12:42.864272] WDIAG [STORAGE.COMPACTION] ob_partition_merge_task.cpp:189 [8213][0][xxxxx-xxxxx-xxxxx-xxxxx] [lt=10] [dc=0][errcode=-4024] Fail to merge, (ret=-4024, ctx_->param_={merge_type:0, merge_version:"229-0-0", pkey:{tid:1100611139702520, partition_id:189, part_cnt:0}, index_id:1100611139702520, schedule_merge_type:0, pg_key:{tid:1100611139702520, partition_id:189, part_cnt:0}}, idx_=0)
40.208/observer.log.20241227101440621:[2024-12-27 10:14:15.785388] WDIAG [STORAGE.COMPACTION] ob_partition_merge_task.cpp:189 [316670][0][xxxxx-xxxxx-xxxxx-xxxxx] [lt=11] [dc=0][errcode=-4024] Fail to merge, (ret=-4024, ctx_->param_={merge_type:0, merge_version:"229-0-0", pkey:{tid:1100611139702520, partition_id:189, part_cnt:0}, index_id:1100611139702520, schedule_merge_type:0, pg_key:{tid:1100611139702520, partition_id:189, part_cnt:0}}, idx_=0)
问题原因
合并宏块 encoding 估算 size 不准,实际大小超过 2MB,写入失败。
问题的风险及影响
合并超时。
影响租户
影响 OceanBase 数据库中的 SYS 租户和 Oracle 租户以及 MySQL 租户。
适用版本
OceanBase 数据库 V3.2.x 版本。
解决方法
暂无。
规避方式
可通过临时关闭分区合并的 encoding 功能绕过问题,具体步骤如下。
登录 SYS 租户,下述操作均在 SYS 租户中进行。
查询当前的版本号信息。
select zone,name,value from oceanbase.__all_zone where name in ('frozen_version','last_merged_version');找出上次合并存在问题的表和分区,其中 xxx 为上次合并的版本。
select table_id,partition_id from oceanbase.__all_virtual_meta_table where data_version!=xxx;通过应急配置项
_force_skip_encoding_partition_id临时关闭这个分区这次合并的 encoding 功能。-- 如当前卡住的是 1100611139702520 的 189 分区 alter system set _force_skip_encoding_partition_id = '189@1100611139702520';重新发起合并。
等版本号成功推高后(合并完成)将配置项修改回默认值。
alter system set _force_skip_encoding_partition_id = '';