问题现象
物理恢复卡在 MODIFY_SCHEMA 阶段。
关键日志信息。
[2024-07-25 15:09:50.811312] WARN [RS.RESTORE] modify_schema (ob_restore_scheduler.cpp:1823) [84325][0][xxxxx-xxxxx-xxxxx-xxxxx] [lt=3] [dc=0] fail to filter schema(ret=-6224, ret="OB_TRANS_NEED_ROLLBACK", job_info={job_id:93, tenant_id:1040, restore_data_version:816, status:4, restore_start_ts:1721889212567831, restore_schema_version:1721872735049520, rebuild_index_schema_version:-1, restore_job_id:-1, restore_timestamp:1721872800000000, cluster_id:1655634774, restore_option:"backup_cluster_name=ob_bjrs_prod&backup_cluster_id=1655634774&pool_list=restore_pool_2024071", backup_dest:"file:///obbackup/", tenant_name:"RES_BJSI_JYCY", backup_tenant_name:"bjsimis", backup_cluster_name:"ob_bjrs_prod", pool_list:"restore_pool_2024071", locality:"", primary_zone:"", backup_locality:"FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3", backup_primary_zone:"zone1,zone2,zone3", info:""compat_mode:1, backup_tenant_id:1003, incarnation:1, full_backup_set_id:745, inc_backup_set_id:746, log_archive_round:11, snapshot_version:1721851204326172, schema_version:1721848553885896, frozen_data_version:816, frozen_snapshot_version:1721851204326172, frozen_schema_version:1721848553885896, passwd_array:"", kms_encrypt:false, kms_info:"", source_cluster_version:12885033731, pre_cluster_version:12885033731, post_cluster_version:12885033731, compatible:4, white_list:{table_items:[{database_name:"BJSI_JYCY", table_name:"JZB_USERINFO"}]}, multi_restore_path_list:{backup_set_path_list:[{backup_set_id:745, copy_id:0, backup_dest:"file:///obbackup/test/xxxxxx/incarnation_1/1003/data/backup_xxxxxx", file_status:0}, {backup_set_id:746, copy_id:0, backup_dest:"file:///obbackup/test/xxxxxx/incarnation_1/1003/data/backup_xxxxxx", file_status:0}], backup_piece_path_list:[{round_id:11, backup_piece_id:210, copy_id:0, backup_dest:"file:///obbackup/test/xxxxxx/incarnation_1/1003/data/backup_xxxxxx", file_status:0}]}})
[2024-07-25 15:09:50.811330] INFO [RS.RESTORE] ob_restore_scheduler.cpp:1849 [84325][0][xxxxx-xxxxx-xxxxx-xxxxx] [lt=5] [dc=0] [RESTORE] modify schema(ret=-6224, job_info={job_id:93, tenant_id:1040, restore_data_version:816, status:4, restore_start_ts:1721889212567831, restore_schema_version:1721872735049520, rebuild_index_schema_version:-1, restore_job_id:-1
触发条件
执行表级物理恢复，且 __all_histogram_stat_v2_history 表中有很多的记录。
问题原因
低版本表级恢复需要过滤租户 schema，清理与白名单中的表无关 schema。删表时执行的 DELETE from __all_histogram_v2_history; 表（内部逻辑）的语句耗时较久，整个删表操作超时，导致恢复任务在 MODIFY_SCHEMA 阶段不断重试。恢复调用删表的时的超时时间为 max(10s, rpc_timeout)。
问题的风险及影响
物理恢复任务卡住。
影响租户
影响 OceanBase 数据库中的 SYS 租户和 Oracle 租户以及 MySQL 租户。
适用版本
OceanBase 数据库 V2.X、V3.x 版本。
解决方法
恢复时适当调大 rpc_timeout 参数，如调大到 30s。
