本文主要介绍 OceanBase 数据库中合并问题的排查方法。
适用版本
OceanBase 数据库所有版本
排查步骤
OceanBase 数据库 V4.X 之前版本
确认合并配置项。
OceanBase 数据库中包含以下合并相关的配置项。
enable_manual_merge：是否开启手动合并，取值为TRUE表示需要手动合并，默认为FALSE。zone_merge_concurrency：用于设置在合并时，支持多少个 Zone 并发。当值为0时，由系统根据部署情况自动选择最佳并发度，一般不需要设置。默认为1，表示单次只有一个 Zone 进行合并。zone_merge_order：用于设置 Zone 的轮转合并顺序。不指定时，由系统自动决定，一般不需要设置。enable_merge_by_turn：用于设置是否开启轮转合并策略，设置为TRUE时表示开启轮转合并，默认为FALSE。major_freeze_duty_time：每日合并发起时间。enable_auto_leader_switch：用于设置是否开启自动切主，默认为TRUE。
有关以上系统配置项的详细信息，请参见 OceanBase 数据库《参考指南》中的 配置项参考 章节。
确认合并状态。
本例中，当前的
frozen_version为25，表示集群需要合并到25版本，但cn-shanghai-e这个 Zone 副本只合并到了24版本。obclient> select * from __all_zone where name = "frozen_version" or name = "last_merged_version"; +----------------------------+----------------------------+---------------+---------------------+-------+------+ | gmt_create | gmt_modified | zone | name | value | info | +----------------------------+----------------------------+---------------+---------------------+-------+------+ | 2020-12-07 16:01:30.793286 | 2020-12-30 02:00:00.445594 | | frozen_version | 25 | | | 2020-12-07 16:01:30.793490 | 2020-12-29 02:01:12.449651 | | last_merged_version | 24 | | | 2020-12-07 16:01:30.794375 | 2020-12-29 02:01:11.880114 | cn-shanghai-e | last_merged_version | 24 | | | 2020-12-07 16:01:30.795109 | 2020-12-30 02:01:15.563291 | cn-shanghai-f | last_merged_version | 25 | | | 2020-12-07 16:01:30.795842 | 2020-12-30 02:01:22.694016 | cn-shanghai-g | last_merged_version | 25 | | +----------------------------+----------------------------+---------------+---------------------+-------+------+ 5 rows in set (0.00 sec)继续查询
__all_zone表，查看cn-shanghai-e的合并状态，发现状态为MERGING，表示正在合并。obclient> select * from __all_zone where name = "merge_status"; +----------------------------+----------------------------+---------------+---------------------+---------+------+ | gmt_create | gmt_modified | zone | name | value | info | +----------------------------+----------------------------+---------------+---------------------+---------+------+ | 2020-12-07 16:01:30.793286 | 2020-12-30 02:00:00.445594 | | merge_status | MERGING | | | 2020-12-07 16:01:30.794375 | 2020-12-29 02:01:11.880114 | cn-shanghai-e | merge_status | MERGING | | | 2020-12-07 16:01:30.795109 | 2020-12-30 02:01:15.563291 | cn-shanghai-f | merge_status | IDLE | | | 2020-12-07 16:01:30.795842 | 2020-12-30 02:01:22.694016 | cn-shanghai-g | merge_status | IDLE | | +----------------------------+----------------------------+---------------+---------------------+---------+------+ 5 rows in set (0.00 sec)排查切主相关问题。
根据上面的信息，发现正在合并，此时查询该 Zone 的
global_broadcast_version与broadcast_version。 如果broadcast_version等于last_merged_version，且last_merged_version落后于global_broadcast_version，说明 RootService 没有发起相关 Zone 的合并。obclient> select * from __all_zone where name = "global_broadcast_version" or name = "broadcast_version"; +----------------------------+----------------------------+---------------+--------------------------+---------+------+ | gmt_create | gmt_modified | zone | name | value | info | +----------------------------+----------------------------+---------------+--------------------------+---------+------+ | 2020-12-07 16:01:30.793286 | 2020-12-30 02:00:00.445594 | | global_broadcast_version | 25 | | | 2020-12-07 16:01:30.794375 | 2020-12-29 02:01:11.880114 | cn-shanghai-e | broadcast_version | 24 | | | 2020-12-07 16:01:30.795109 | 2020-12-30 02:01:15.563291 | cn-shanghai-f | broadcast_version | 25 | | | 2020-12-07 16:01:30.795842 | 2020-12-30 02:01:22.694016 | cn-shanghai-g | broadcast_version | 25 | | +----------------------------+----------------------------+---------------+--------------------------+---------+------+ 5 rows in set (0.00 sec)对于 Root Service 未发起相关 Zone 合并的情况，按以下方法排查。
首先排查 RootService 合并调度线程是否有报错。
如果有返回值，则说明合并调度线程存在错误。
grep "daily.*merge.*ret=-" rootservice.log然后检查集群是否正在补副本。
如果以下查询有返回值，则说明正在进行副本的负载均衡。
obclient> select count(*) from __all_virtual_replica_task;
确认未合并副本信息。
本例继续假设当前要合并的目标版本是
25，查询 meta 表查看data_version != 25的副本以缩小排查范围。对于 OceanBase 数据库 V1.X 版本，查询
__all_virtual_core_meta_table、__all_virtual_core_root_table、__all_root_table与__all_meta_table表。对于 OceanBase 数据库 V2.X 及后续版本，查询
__all_virtual_core_meta_table、__all_virtual_core_root_table、__all_root_table和__all_virtual_meta_table表。
obclient> select * from __all_virtual_meta_table where data_version != 25 limit 10; +-----------+------------------+--------------+---------------+----------+----------------------------+----------------------------+----------+---------+---------------+---------------+------+-----------------------------------------------------------------------------------------------------------+-----------+-----------+--------------+---------------+--------------+-----------------+--------------------+--------------------+-------------+---------+--------------+---------------+-----------------------+------------+--------------------+--------+-----------+--------------------+------------------+ | tenant_id | table_id | partition_id | svr_ip | svr_port | gmt_create | gmt_modified | sql_port | unit_id | partition_cnt | zone | role | member_list | row_count | data_size | data_version | data_checksum | row_checksum | column_checksum | is_original_leader | is_previous_leader | create_time | rebuild | replica_type | required_size | status | is_restore | partition_checksum | quorum | fail_list | recovery_timestamp | memstore_percent | +-----------+------------------+--------------+---------------+----------+----------------------------+----------------------------+----------+---------+---------------+---------------+------+-----------------------------------------------------------------------------------------------------------+-----------+-----------+--------------+---------------+--------------+-----------------+--------------------+--------------------+-------------+---------+--------------+---------------+-----------------------+------------+--------------------+--------+-----------+--------------------+------------------+ | 1001 | 1100611139463766 | 0 | xxx.xxx.x.xa | xxxx | 2020-12-29 10:34:15.176561 | 2020-12-29 10:34:15.205753 | 2881 | 1001 | 0 | cn-shanghai-e | 1 | xxx.xxx.x.xx:2882:1609209255175464,xxx.xxx.x.xx:2882:1609209255175464,xxx.xxx.x.xx:2882:1609209255175464 | 0 | 0 | 24 | 0 | 0 | | 0 | 1609209255204831 | 0 | 0 | 0 | 0 | REPLICA_STATUS_NORMAL | 0 | 0 | 3 | | 0 | 100 | +-----------+------------------+--------------+---------------+----------+----------------------------+----------------------------+----------+---------+---------------+---------------+------+-----------------------------------------------------------------------------------------------------------+-----------+-----------+--------------+---------------+--------------+-----------------+--------------------+--------------------+-------------+---------+--------------+---------------+-----------------------+------------+--------------------+--------+-----------+--------------------+------------------+ 1 row in set (0.07 sec)根据上面的信息，发现是 pkey 为
{tid:1100611139463766, partition_id:0}的表在xxx.xxx.x.xa:xxxx机器上未合并。也可以通过搜索
rootservice.log定位是哪个表阻塞了合并。
OceanBase 数据库 V4.X 版本
找到未合并完成的租户 ID。
select* from CDB_OB_MAJOR_COMPACTION where STATUS != "IDLE";查询 server 级别进度表，看有多少分区没有合并完成。
select * from __all_virtual_server_compaction_progress where tenant_id = xxx;
如果 comments 字段显示
TABLET_COMPACTION_FINISHED，表示存储层所有 Tablet 都合并完成。看分区级别进度表，可以找到没有合并完成的分区及 TRACE_ID。
select * from __all_virtual_tablet_compaction_progress where tenant_id = xxx;
通过 SSTable 信息表可以看到 SSTable 的数据量和数据分布。
select * from GV$OB_SSTABLES where tenant_id = xxx and tablet_id = xxx and svr_ip = "xxx";以下图为例，数据都集中在 MINI/MINOR SSTable中，MAJOR SSTable 中没有数据。撞上了并行合并的 BAD_CASE，导致合并比较慢。
