问题现象
系统租户处于 prepare 状态，普通租户处于 schedule 状态，当发起 do_cancel 时，备份的状态卡在 cancel，无法退出，无法发起下一个备份任务。
备份过程中用户执行
alter system cancel all backup force命令，使用select value from oceanbase.__all_tenant_backup_info where name = 'backup_status'可以看到集群备份状态一直卡在 cancel 状态，长时间不能推进。通过 trace_id 查找相关日志看到 cancel 的报错信息
grep "YBxxxxx-xxxxxx-xxxxxx-xxxxxx" observer.log.2* observer.log | vi -若看到报错文件、错误码以及报错信息完全符合，即可确认该错误。
[2023-11-07 16:46:16.519279] WDIAG [SHARE] ob_backup_path.cpp:1922 [2697475][0][YBxxxxx-xxxxxx-xxxxxx-xxxxxx] [lt=10] [dc=0][errcode=-4002] invalid argument!(ret=-4002, param={tenant_id:1, incarnation:1, c opy_id:0, full_set_id:397, inc_set_id:397, backup_date:20231030, compatible:0, backup_dest_ptr:0x7fa0a53636f0}, buffer_length=2048, buffer=0x7fa0a53621b0) [2023-11-07 16:46:16.519310] WDIAG [SERVER] ob_backup_operator.cpp:2784 [2697475][0][YBxxxxx-xxxxxx-xxxxxx-xxxxxx] [lt=14] [dc=0][errcode=-4002] fail to get backup set relative path(ret=-4002, param={tenan t_id:1, incarnation:1, copy_id:0, full_set_id:397, inc_set_id:397, backup_date:20231030, compatible:0, backup_dest_ptr:0x7fa0a53636f0}, item={tenant_id:1, backup_set_id:397, incarnation:1, copy_id:0, snapshot_v ersion:1698644686857754, prev_full_backup_set_id:0, prev_inc_backup_set_id:0, prev_backup_data_version:0, pg_count:0, macro_block_count:0, finish_pg_count:0, finish_macro_block_count:0, input_bytes:0, output_by tes:0, start_time:1698644686857754, end_time:0, compatible:0, cluster_version:0, backup_type:{type:1}, status:0, result:-4072, cluster_id:1641437681, backup_dest:"oss://xxx", backup_data_version:0, backup_schema_version:1697688796461568, partition_count:0, finish_partition_co unt:0, encryption_mode:0, passwd:"", file_status:1, start_replay_log_ts:0, date:20231030, backup_level:2}) [2023-11-07 16:46:16.519334] WDIAG [SERVER] ob_backup_operator.cpp:3264 [2697475][0][YBxxxxx-xxxxxx-xxxxxx-xxxxxx] [lt=12] [dc=0][errcode=-4002] failed to fill on item(ret=-4002, backup_set_file_info={tena nt_id:1, backup_set_id:397, incarnation:1, copy_id:0, snapshot_version:1698644686857754, prev_full_backup_set_id:0, prev_inc_backup_set_id:0, prev_backup_data_version:0, pg_count:0, macro_block_count:0, finish_ pg_count:0, finish_macro_block_count:0, input_bytes:0, output_bytes:0, start_time:1698644686857754, end_time:0, compatible:0, cluster_version:0, backup_type:{type:1}, status:0, result:-4072, cluster_id:16414376 81, backup_dest:"oss://xxx", backup_data_version:0, backup_schema_ version:1697688796461568, partition_count:0, finish_partition_count:0, encryption_mode:0, passwd:"", file_status:1, start_replay_log_ts:0, date:20231030, backup_level:2}) [2023-11-07 16:46:16.519395] WDIAG [RS] ob_root_backup.cpp:3177 [2697475][0][YBxxxxx-xxxxxx-xxxxxx-xxxxxx] [lt=49] [dc=0][errcode=-4002] failed to insert tenant backup set file info(ret=-4002, backup_set_f ile_info={tenant_id:1, backup_set_id:397, incarnation:1, copy_id:0, snapshot_version:1698644686857754, prev_full_backup_set_id:0, prev_inc_backup_set_id:0, prev_backup_data_version:0, pg_count:0, macro_block_co unt:0, finish_pg_count:0, finish_macro_block_count:0, input_bytes:0, output_bytes:0, start_time:1698644686857754, end_time:0, compatible:0, cluster_version:0, backup_type:{type:1}, status:0, result:-4072, clust er_id:1641437681, backup_dest:"oss://xxx", backup_data_version:0, backup_schema_version:1697688796461568, partition_count:0, finish_partition_count:0, encryption_mode:0, passwd:"", file_status:1, start_replay_log_ts:0, date:20231030, backup_level:2}) [2023-11-07 16:46:16.519420] WDIAG [RS] ob_root_backup.cpp:2142 [2697475][0][YBxxxxx-xxxxxx-xxxxxx-xxxxxx] [lt=11] [dc=0][errcode=-4002] failed to insert tenant backup set file failed(ret=-4002, sys_backup _task={tenant_id:1, backup_set_id:397, incarnation:1, snapshot_version:1698644686857754, prev_full_backup_set_id:0, prev_inc_backup_set_id:0, prev_backup_data_version:0, pg_count:0, macro_block_count:0, finish_ pg_count:0, finish_macro_block_count:0, input_bytes:0, output_bytes:0, start_time:1698644686857754, end_time:0, compatible:0, cluster_version:0, backup_type:{type:1}, status:1, device_type:0, result:-4072, clus ter_id:1641437681, backup_dest:{device_type:0, root_path:"oss://xxx", type:"OSS"}, backup_data_version:0, backup_schema_version:1697688796461568, partition_count:0, finish_partition_count:0, encrypt ion_mode:0, passwd:"", is_mark_deleted:false, start_replay_log_ts:0, date:20231030, copy_id:0, backup_level:2}) [2023-11-07 16:46:16.519665] WDIAG [RS] ob_root_backup.cpp:1056 [2697475][0][YBxxxxx-xxxxxx-xxxxxx-xxxxxx] [lt=11] [dc=0][errcode=-4002] failed to do sys tenant cancel(ret=-4002, info={tenant_id:1, backup_ set_id:397, incarnation:1, backup_dest:"oss://xxx", backup_backup_ dest:"", backup_snapshot_version:1698644686857754, backup_schema_version:1697688796461568, backup_data_version:1169, detected_backup_region:"", backup_type:{type:1}, backup_status:{status:4, status_str:"CANCEL" }, backup_task_id:0, copy_id:0, encryption_mode:0}) [2023-11-07 16:46:16.519686] WDIAG [RS] ob_root_backup.cpp:554 [2697475][0][YBxxxxx-xxxxxx-xxxxxx-xxxxxx] [lt=13] [dc=0][errcode=-4002] failed to do cancel(ret=-4002, info={tenant_id:1, backup_set_id:397, incarnation:1, backup_dest:"oss://xxx", backup_backup_dest:"", bac kup_snapshot_version:1698644686857754, backup_schema_version:1697688796461568, backup_data_version:1169, detected_backup_region:"", backup_type:{type:1}, backup_status:{status:4, status_str:"CANCEL"}, backup_ta sk_id:0, copy_id:0, encryption_mode:0})
关键诊断信息
触发条件
系统租户处于 prepare 状态，普通租户处于 schedule 状态，发起 do_cancel，备份的状态卡在 cancel。
事后诊断
备份过程中用户执行
alter system cancel all backup force命令，使用select value from oceanbase.__all_tenant_backup_info where name = 'backup_status'可以看到集群备份状态一直卡在 cancel 状态，长时间不能推进。通过 trace_id 查找相关日志看到 cancel 的报错信息
grep "YBxxxxx-xxxxxx-xxxxxx-xxxxxx" observer.log.2* observer.log | vi -若看到报错文件、错误码以及报错信息完全符合，即可确认该错误。
[2023-11-07 16:46:16.519279] WDIAG [SHARE] ob_backup_path.cpp:1922 [2697475][0][YBxxxxx-xxxxxx-xxxxxx-xxxxxx] [lt=10] [dc=0][errcode=-4002] invalid argument!(ret=-4002, param={tenant_id:1, incarnation:1, c opy_id:0, full_set_id:397, inc_set_id:397, backup_date:20231030, compatible:0, backup_dest_ptr:0x7fa0a53636f0}, buffer_length=2048, buffer=0x7fa0a53621b0) [2023-11-07 16:46:16.519310] WDIAG [SERVER] ob_backup_operator.cpp:2784 [2697475][0][YBxxxxx-xxxxxx-xxxxxx-xxxxxx] [lt=14] [dc=0][errcode=-4002] fail to get backup set relative path(ret=-4002, param={tenan t_id:1, incarnation:1, copy_id:0, full_set_id:397, inc_set_id:397, backup_date:20231030, compatible:0, backup_dest_ptr:0x7fa0a53636f0}, item={tenant_id:1, backup_set_id:397, incarnation:1, copy_id:0, snapshot_v ersion:1698644686857754, prev_full_backup_set_id:0, prev_inc_backup_set_id:0, prev_backup_data_version:0, pg_count:0, macro_block_count:0, finish_pg_count:0, finish_macro_block_count:0, input_bytes:0, output_by tes:0, start_time:1698644686857754, end_time:0, compatible:0, cluster_version:0, backup_type:{type:1}, status:0, result:-4072, cluster_id:1641437681, backup_dest:"oss://xxx", backup_data_version:0, backup_schema_version:1697688796461568, partition_count:0, finish_partition_co unt:0, encryption_mode:0, passwd:"", file_status:1, start_replay_log_ts:0, date:20231030, backup_level:2}) [2023-11-07 16:46:16.519334] WDIAG [SERVER] ob_backup_operator.cpp:3264 [2697475][0][YBxxxxx-xxxxxx-xxxxxx-xxxxxx] [lt=12] [dc=0][errcode=-4002] failed to fill on item(ret=-4002, backup_set_file_info={tena nt_id:1, backup_set_id:397, incarnation:1, copy_id:0, snapshot_version:1698644686857754, prev_full_backup_set_id:0, prev_inc_backup_set_id:0, prev_backup_data_version:0, pg_count:0, macro_block_count:0, finish_ pg_count:0, finish_macro_block_count:0, input_bytes:0, output_bytes:0, start_time:1698644686857754, end_time:0, compatible:0, cluster_version:0, backup_type:{type:1}, status:0, result:-4072, cluster_id:16414376 81, backup_dest:"oss://xxx", backup_data_version:0, backup_schema_ version:1697688796461568, partition_count:0, finish_partition_count:0, encryption_mode:0, passwd:"", file_status:1, start_replay_log_ts:0, date:20231030, backup_level:2}) [2023-11-07 16:46:16.519395] WDIAG [RS] ob_root_backup.cpp:3177 [2697475][0][YBxxxxx-xxxxxx-xxxxxx-xxxxxx] [lt=49] [dc=0][errcode=-4002] failed to insert tenant backup set file info(ret=-4002, backup_set_f ile_info={tenant_id:1, backup_set_id:397, incarnation:1, copy_id:0, snapshot_version:1698644686857754, prev_full_backup_set_id:0, prev_inc_backup_set_id:0, prev_backup_data_version:0, pg_count:0, macro_block_co unt:0, finish_pg_count:0, finish_macro_block_count:0, input_bytes:0, output_bytes:0, start_time:1698644686857754, end_time:0, compatible:0, cluster_version:0, backup_type:{type:1}, status:0, result:-4072, clust er_id:1641437681, backup_dest:"oss://xxx", backup_data_version:0, backup_schema_version:1697688796461568, partition_count:0, finish_partition_count:0, encryption_mode:0, passwd:"", file_status:1, start_replay_log_ts:0, date:20231030, backup_level:2}) [2023-11-07 16:46:16.519420] WDIAG [RS] ob_root_backup.cpp:2142 [2697475][0][YBxxxxx-xxxxxx-xxxxxx-xxxxxx] [lt=11] [dc=0][errcode=-4002] failed to insert tenant backup set file failed(ret=-4002, sys_backup _task={tenant_id:1, backup_set_id:397, incarnation:1, snapshot_version:1698644686857754, prev_full_backup_set_id:0, prev_inc_backup_set_id:0, prev_backup_data_version:0, pg_count:0, macro_block_count:0, finish_ pg_count:0, finish_macro_block_count:0, input_bytes:0, output_bytes:0, start_time:1698644686857754, end_time:0, compatible:0, cluster_version:0, backup_type:{type:1}, status:1, device_type:0, result:-4072, clus ter_id:1641437681, backup_dest:{device_type:0, root_path:"oss://xxx", type:"OSS"}, backup_data_version:0, backup_schema_version:1697688796461568, partition_count:0, finish_partition_count:0, encrypt ion_mode:0, passwd:"", is_mark_deleted:false, start_replay_log_ts:0, date:20231030, copy_id:0, backup_level:2}) [2023-11-07 16:46:16.519665] WDIAG [RS] ob_root_backup.cpp:1056 [2697475][0][YBxxxxx-xxxxxx-xxxxxx-xxxxxx] [lt=11] [dc=0][errcode=-4002] failed to do sys tenant cancel(ret=-4002, info={tenant_id:1, backup_ set_id:397, incarnation:1, backup_dest:"oss://xxx", backup_backup_ dest:"", backup_snapshot_version:1698644686857754, backup_schema_version:1697688796461568, backup_data_version:1169, detected_backup_region:"", backup_type:{type:1}, backup_status:{status:4, status_str:"CANCEL" }, backup_task_id:0, copy_id:0, encryption_mode:0}) [2023-11-07 16:46:16.519686] WDIAG [RS] ob_root_backup.cpp:554 [2697475][0][YBxxxxx-xxxxxx-xxxxxx-xxxxxx] [lt=13] [dc=0][errcode=-4002] failed to do cancel(ret=-4002, info={tenant_id:1, backup_set_id:397, incarnation:1, backup_dest:"oss://xxx", backup_backup_dest:"", bac kup_snapshot_version:1698644686857754, backup_schema_version:1697688796461568, backup_data_version:1169, detected_backup_region:"", backup_type:{type:1}, backup_status:{status:4, status_str:"CANCEL"}, backup_ta sk_id:0, copy_id:0, encryption_mode:0})
问题原因
发起备份时，系统租户处于 prepare 状态，普通租户处于 schedule 状态，当 do_scheduler 出错时，导致构造的备份 task 没有插入内部表。当发起 force_cancel 时，因没有在内部表内查到任务，会对任务进行构造，构造新任务 task struct 时，没有将 compatible 赋值，在后续的流程中进行了对 task 的有效性进行了校检，compatible 需要大于 0。导致出现了 4002 错误。从而使得备份卡住。
问题的风险及影响
会导致 force_cancel 备份任务一直无法退出，下一轮新的备份不能发起。
影响租户
影响 OceanBase 数据库中的 SYS 租户和 Oracle 租户以及 MySQL 租户。
影响版本
OceanBase 数据库企业版 V3.1.2 GA（oceanbase-3.1.2-20210618150922）及之后版本、V3.2.3 GA（oceanbase-3.2.3.0-20220418212020）及之后版本、V3.2.4 GA（oceanbase-3.2.4.0-100000072022102819）及之后版本。
解决方法
升级至问题已修复版本。目前已修复的版本包括 OceanBase 数据库企业版 V3.2.3 BP11（oceanbase-3.2.3.3-111000032024070822）、V3.2.4 BP7（oceanbase-3.2.4.7-107000012023113010）。
规避方式
调整发起备份语句级别事务超时时间。
线上应急手段（人工介入恢复）。
警告
更新内部表 __all_tennat_backup_info 存在风险，请勿自行操作。如果需要修改，请咨询 OceanBase 技术支持。
用 root 权限链接到 1002 租户，执行 SQL。
update __all_tenant_backup_info set value = 'STOP' where name = 'backup_status';用 root 权限链接到 SYS 租户，执行 SQL。
update __all_tenant_backup_info set value = 'CLEANUP' where name = 'backup_status';等待 SYS 租户的
__all_tennat_backup_info中的backup_status变成 STOP 就恢复了。update __all_tenant_backup_info set value = '' where name = 'backup_data_version'; update __all_tenant_backup_info set value = '' where name = 'backup_schema_version'; update __all_tenant_backup_info set value = '' where name = 'backup_schema_snapshot_version'; update __all_tenant_backup_info set value = 'STOP' where name = 'backup_status'; update __all_tenant_backup_info set value = '' where name = 'backup_type';
