SSTable 连续性检查丢失

问题现象

内核的一个 BUG，没有检查 SSTable 的连续性，一些并发操作将 SSTable 放到 table store 的时候可能会出现不连续的 SSTable；transfer 的操作可能会加剧出现不连续的 SSTable 的场景，会造成 transfer 在 DOING 阶段卡住，日志中会有 ERROR 日志报错 -4016 。

关键诊断信息

触发条件

1. 在执行 Unit 变更或者更改 Primary Zone 个数的变化，这个时候会进行 tablet 的 transfer 。

2. transfer 期间源端的 SSTable 可能会进行回收（多版本可读位点推高），transfer 正在做源端数据的回填；回填的 SSTable 正好是被回收掉的 SSTable 。

3. 回收操作先完成，回填的时候再将对应的 SSTable 放到 tablet 中。

事前巡检

无 。

事后诊断

1. 检查 transfer 是否卡在了 DOING 状态。

   select * from oceanbase.__all_virtual_transfer_task where tenant_id = xxx and status = 'DOING';
   

   select * from __all_virtual_transfer_task where tenant_id = xxx and status = 'DOING';
   

   内部表有记录说明 transfer 卡在了 DOING 状态。

2. 检查对应的 transfer tablet 是否存在不连续的情况。

   select * from oceanbase.__all_virtual_table_mgr where tenant_id = xxx and tablet_id = xxx and svr_ip = 'xxx' and svr_port = xxx;
   

   其中对应的记录中有 start_log_scn 和 end_log_scn，minor SSTable 中连续的两个 minor SSTable end_log_scn < start_log_scn 说明 SSTable range 不连续了。

3. 检查 OBServer 的报错日志。

   WDIAG [STORAGE] cut_ha_sstable_scn_range_ (ob_tablet_table_store.cpp:2153) [3127085][T1004_TRANSFER_][T1004][xxxxx-xxxxx-xxxxx-xxxxx] [lt=18][errcode=-4016] minor sstable log ts is not continue or scn has overlap(ret=-4016, orig_minor_sstables=[{ObITable:{this:0x1001a23c7068, key:{tablet_id:{id:206201}, column_group_idx:0, table_type:"MINOR", scn_range:{start_scn:{val:1715229880079118008, v:0}, end_scn:{val:1715230715493514021, v:0}}}, ref_cnt:0, upper_trans_version:1715233304680158018, timestamp:0}, this:0x1001a23c7068, addr:[1526716-169606-0-2305843327043368077-0](offset=2094221,size=296,type=2,seq=0), upper_trans_version:1715233304680158018, max_merged_trans_version:1715233304680158018, data_macro_block_count:0, nested_size:2097152, nested_offset:0, contain_uncommitted_row:false, filled_tx_scn:{val:1715234132641648003, v:0}, meta:{basic_meta:{version:1, length:86, row_count_:0, occupy_size_:0, original_size_:0, data_checksum_:0, index_type_:0, rowkey_column_count_:4, column_cnt_:9, data_macro_block_count_:0, data_micro_block_count_:0, use_old_macro_block_count_:0, index_macro_block_count_:0, sstable_format_version_:1, 
   schema_version_:1715140251362992, create_snapshot_version_:0, progressive_merge_round_:0, progressive_merge_step_:0, data_index_tree_height_:0, table_mode_:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, upper_trans_version_:1715233304680158018, max_merged_trans_version_:1715233304680158018, recycle_version:0, ddl_scn_:{val:0, v:0}, filled_tx_scn_:{val:1715234132641648003, v:0}, contain_uncommitted_row_:false, status_:4, root_row_store_type:0, compressor_type:2, encrypt_id:0, master_key_id:0, sstable_logic_seq:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", latest_row_store_type:1}, column_checksums:0x1001a23c7318, column_checksum_count:0, data_root_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, macro_info:{macro_meta_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, data_block_count:0, other_block_count:0, linked_block_count:0, data_block_ids:null, other_block_ids:null, linked_block_ids:null, entry_id:[-1](ver=0,mode=0,seq=0), is_meta_root:false, nested_offset:0, nested_size:2097152}}, valid_for_reading:true}, {ObITable:{this:0x1001a23c7318, key:{tablet_id:{id:206201}, column_group_idx:0, table_type:"MINI", scn_range:{start_scn:{val:1715231622239266032, v:0}, end_scn:{val:1715233304680158045, v:0}}}, ref_cnt:0, upper_trans_version:1715233189159216002, timestamp:0}, this:0x1001a23c7318, addr:[1526716-169606-0-2305843925117564341-0](offset=2094517,size=853,type=2,seq=0), upper_trans_version:1715233189159216002, max_merged_trans_version:1715233189159216002, data_macro_block_count:1, nested_size:4096, nested_offset:544768, contain_uncommitted_row:false, filled_tx_scn:{val:1715233304680158045, v:0}, meta:{basic_meta:{version:1, length:92, row_count_:1, occupy_size_:646, original_size_:346, data_checksum_:1554084397, index_type_:0, 
   rowkey_column_count_:4, column_cnt_:9, data_macro_block_count_:1, data_micro_block_count_:1, use_old_macro_block_count_:0, index_macro_block_count_:0, sstable_format_version_:1, schema_version_:1715140251362992, create_snapshot_version_:0, progressive_merge_round_:0, progressive_merge_step_:0, data_index_tree_height_:2, table_mode_:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, upper_trans_version_:1715233189159216002, max_merged_trans_version_:1715233189159216002, recycle_version:0, ddl_scn_:{val:0, v:0}, filled_tx_scn_:{val:1715233304680158045, v:0}, contain_uncommitted_row_:false, status_:4, root_row_store_type:0, compressor_type:2, encrypt_id:0, master_key_id:0, sstable_logic_seq:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", latest_row_store_type:1}, column_checksums:0x1001a23c75c8, column_checksum_count:0, data_root_info:{addr:[0-0-0-3458764782255996928-0](offset=0,size=250,type=3,seq=0), block_data:{buf:0x1001a23c75c8, size:250, extra_buf:0x1001a23c76c2, extra_size:368, type:1}}, macro_info:{macro_meta_info:{addr:[0-0-0-3458764790845931520-0](offset=0,size=258,type=3,seq=0), block_data:{buf:0x1001a23c7832, size:258, extra_buf:null, extra_size:0, type:1}}, data_block_count:1, other_block_count:0, linked_block_count:0, data_block_ids:0x1001a23c7934, other_block_ids:null, linked_block_ids:null, entry_id:[-1](ver=0,mode=0,seq=0), is_meta_root:true, nested_offset:544768, nested_size:4096}}, 
   valid_for_reading:true}, {ObITable:{this:0x10031302e060, key:{tablet_id:{id:206201}, column_group_idx:0, table_type:"MINOR", scn_range:{start_scn:{val:1715233304680158045, v:0}, end_scn:{val:1715234132641648003, v:0}}}, ref_cnt:1, upper_trans_version:0, timestamp:0}, this:0x10031302e060, addr:[0-0-0-3458764685619232768-0](offset=0,size=160,type=3,seq=0), upper_trans_version:0, max_merged_trans_version:0, data_macro_block_count:0, nested_size:2097152, nested_offset:0, contain_uncommitted_row:false, filled_tx_scn:{val:0, v:0}, meta:{basic_meta:{version:1, length:71, row_count_:0, occupy_size_:0, original_size_:0, data_checksum_:0, index_type_:0, rowkey_column_count_:4, column_cnt_:9, data_macro_block_count_:0, data_micro_block_count_:0, use_old_macro_block_count_:0, index_macro_block_count_:0, sstable_format_version_:1, schema_version_:1715140251362992, create_snapshot_version_:0, progressive_merge_round_:1, progressive_merge_step_:0, data_index_tree_height_:0, table_mode_:{table_mode_flag:0, pk_mode:0, table_state_flag:0, view_created_method_flag:0, table_organization_mode:0, auto_increment_mode:0, rowid_mode:0, view_column_filled_flag:0}, upper_trans_version_:0, max_merged_trans_version_:0, recycle_version:0, ddl_scn_:{val:0, v:0}, filled_tx_scn_:{val:0, v:0}, contain_uncommitted_row_:false, status_:4, root_row_store_type:0, compressor_type:1, encrypt_id:0, master_key_id:-1, sstable_logic_seq:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", latest_row_store_type:0}, column_checksums:null, column_checksum_count:0, data_root_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, macro_info:{macro_meta_info:{addr:[0-0-0-0-0](offset=0,size=0,type=0,seq=0), block_data:{buf:null, size:0, extra_buf:null, extra_size:0, type:1}}, data_block_count:0, other_block_count:0, linked_block_count:0, data_block_ids:null, other_block_ids:null, linked_block_ids:null, entry_id:[-1](ver=0,mode=0,seq=0), is_meta_root:false, nested_offset:0, nested_size:2097152}}, valid_for_reading:true}])
   

问题原因

SSTable 放到 tablet 中的时候没有检查 SSTable 的连续性，会将一个不连续的 SSTable 放到 table_store 中；后续会影响这个 tablet 的 transfer 。

问题的风险及影响

SSTable 不连续的情况下不影响数据的正确性；全部副本在 transfer 过程中遇到了 SSTable 不连续的情况无法恢复。

影响租户

影响 OceanBase 数据库中的 SYS 租户和 Oracle 租户以及 MySQL 租户。

影响版本

OceanBase 数据库企业版 V4.2.1 GA（oceanbase-4.2.1.0-100000182023092722）及之后版本、V4.2.2 GA（oceanbase-4.2.2.0-100000082024011317）及之后版本、V4.3.0（oceanbase-4.3.0.0-100000072024020200）及之后版本。

解决方法及规避方式

• 解决方法：

  • 升级至问题已修复版本。目前已修复的版本包括 OceanBase 数据库企业版 V4.2.1 BP7（oceanbase-4.2.1.7-107000112024052920）及之后版本、V4.2.3 BP1（oceanbase-4.2.3.1-101000012024061216）及之后版本、V4.3.1（100000212024051522）及之后版本。

  • transfer 流程进入到 DOING 阶段无法取消，如果 DOING 状态是因为 SSTable 不连续导致的，请联系 OceanBase 技术支持团队处理。

• 规避方式：

  无 。