OceanBase 数据库 V3.2.x 数据备份 force_cancel 会导致备份流程卡住，退不出来

问题现象

系统租户处于 prepare 状态，普通租户处于 schedule 状态，当发起 do_cancel 时，备份的状态卡在 cancel，无法退出，无法发起下一个备份任务。

1. 备份过程中用户执行 alter system cancel all backup force 命令，使用 select value from oceanbase.__all_tenant_backup_info where name = 'backup_status' 可以看到集群备份状态一直卡在 cancel 状态，长时间不能推进。

2. 通过 trace_id 查找相关日志看到 cancel 的报错信息

   grep "YBxxxxx-xxxxxx-xxxxxx-xxxxxx" observer.log.2* observer.log | vi -
   

   若看到报错文件、错误码以及报错信息完全符合，即可确认该错误。

   [2023-11-07 16:46:16.519279] WDIAG [SHARE] ob_backup_path.cpp:1922 [2697475][0][YBxxxxx-xxxxxx-xxxxxx-xxxxxx] [lt=10] [dc=0][errcode=-4002] invalid argument!(ret=-4002, param={tenant_id:1, incarnation:1, c
   opy_id:0, full_set_id:397, inc_set_id:397, backup_date:20231030, compatible:0, backup_dest_ptr:0x7fa0a53636f0}, buffer_length=2048, buffer=0x7fa0a53621b0)
   [2023-11-07 16:46:16.519310] WDIAG [SERVER] ob_backup_operator.cpp:2784 [2697475][0][YBxxxxx-xxxxxx-xxxxxx-xxxxxx] [lt=14] [dc=0][errcode=-4002] fail to get backup set relative path(ret=-4002, param={tenan
   t_id:1, incarnation:1, copy_id:0, full_set_id:397, inc_set_id:397, backup_date:20231030, compatible:0, backup_dest_ptr:0x7fa0a53636f0}, item={tenant_id:1, backup_set_id:397, incarnation:1, copy_id:0, snapshot_v
   ersion:1698644686857754, prev_full_backup_set_id:0, prev_inc_backup_set_id:0, prev_backup_data_version:0, pg_count:0, macro_block_count:0, finish_pg_count:0, finish_macro_block_count:0, input_bytes:0, output_by
   tes:0, start_time:1698644686857754, end_time:0, compatible:0, cluster_version:0, backup_type:{type:1}, status:0, result:-4072, cluster_id:1641437681, backup_dest:"oss://xxx", backup_data_version:0, backup_schema_version:1697688796461568, partition_count:0, finish_partition_co
   unt:0, encryption_mode:0, passwd:"", file_status:1, start_replay_log_ts:0, date:20231030, backup_level:2})
   [2023-11-07 16:46:16.519334] WDIAG [SERVER] ob_backup_operator.cpp:3264 [2697475][0][YBxxxxx-xxxxxx-xxxxxx-xxxxxx] [lt=12] [dc=0][errcode=-4002] failed to fill on item(ret=-4002, backup_set_file_info={tena
   nt_id:1, backup_set_id:397, incarnation:1, copy_id:0, snapshot_version:1698644686857754, prev_full_backup_set_id:0, prev_inc_backup_set_id:0, prev_backup_data_version:0, pg_count:0, macro_block_count:0, finish_
   pg_count:0, finish_macro_block_count:0, input_bytes:0, output_bytes:0, start_time:1698644686857754, end_time:0, compatible:0, cluster_version:0, backup_type:{type:1}, status:0, result:-4072, cluster_id:16414376
   81, backup_dest:"oss://xxx", backup_data_version:0, backup_schema_
   version:1697688796461568, partition_count:0, finish_partition_count:0, encryption_mode:0, passwd:"", file_status:1, start_replay_log_ts:0, date:20231030, backup_level:2})
   [2023-11-07 16:46:16.519395] WDIAG [RS] ob_root_backup.cpp:3177 [2697475][0][YBxxxxx-xxxxxx-xxxxxx-xxxxxx] [lt=49] [dc=0][errcode=-4002] failed to insert tenant backup set file info(ret=-4002, backup_set_f
   ile_info={tenant_id:1, backup_set_id:397, incarnation:1, copy_id:0, snapshot_version:1698644686857754, prev_full_backup_set_id:0, prev_inc_backup_set_id:0, prev_backup_data_version:0, pg_count:0, macro_block_co
   unt:0, finish_pg_count:0, finish_macro_block_count:0, input_bytes:0, output_bytes:0, start_time:1698644686857754, end_time:0, compatible:0, cluster_version:0, backup_type:{type:1}, status:0, result:-4072, clust
   er_id:1641437681, backup_dest:"oss://xxx", backup_data_version:0,
   backup_schema_version:1697688796461568, partition_count:0, finish_partition_count:0, encryption_mode:0, passwd:"", file_status:1, start_replay_log_ts:0, date:20231030, backup_level:2})
   [2023-11-07 16:46:16.519420] WDIAG [RS] ob_root_backup.cpp:2142 [2697475][0][YBxxxxx-xxxxxx-xxxxxx-xxxxxx] [lt=11] [dc=0][errcode=-4002] failed to insert tenant backup set file failed(ret=-4002, sys_backup
   _task={tenant_id:1, backup_set_id:397, incarnation:1, snapshot_version:1698644686857754, prev_full_backup_set_id:0, prev_inc_backup_set_id:0, prev_backup_data_version:0, pg_count:0, macro_block_count:0, finish_
   pg_count:0, finish_macro_block_count:0, input_bytes:0, output_bytes:0, start_time:1698644686857754, end_time:0, compatible:0, cluster_version:0, backup_type:{type:1}, status:1, device_type:0, result:-4072, clus
   ter_id:1641437681, backup_dest:{device_type:0, root_path:"oss://xxx", type:"OSS"}, backup_data_version:0, backup_schema_version:1697688796461568, partition_count:0, finish_partition_count:0, encrypt
   ion_mode:0, passwd:"", is_mark_deleted:false, start_replay_log_ts:0, date:20231030, copy_id:0, backup_level:2})
   [2023-11-07 16:46:16.519665] WDIAG [RS] ob_root_backup.cpp:1056 [2697475][0][YBxxxxx-xxxxxx-xxxxxx-xxxxxx] [lt=11] [dc=0][errcode=-4002] failed to do sys tenant cancel(ret=-4002, info={tenant_id:1, backup_
   set_id:397, incarnation:1, backup_dest:"oss://xxx", backup_backup_
   dest:"", backup_snapshot_version:1698644686857754, backup_schema_version:1697688796461568, backup_data_version:1169, detected_backup_region:"", backup_type:{type:1}, backup_status:{status:4, status_str:"CANCEL"
   }, backup_task_id:0, copy_id:0, encryption_mode:0})
   [2023-11-07 16:46:16.519686] WDIAG [RS] ob_root_backup.cpp:554 [2697475][0][YBxxxxx-xxxxxx-xxxxxx-xxxxxx] [lt=13] [dc=0][errcode=-4002] failed to do cancel(ret=-4002, info={tenant_id:1, backup_set_id:397,
   incarnation:1, backup_dest:"oss://xxx", backup_backup_dest:"", bac
   kup_snapshot_version:1698644686857754, backup_schema_version:1697688796461568, backup_data_version:1169, detected_backup_region:"", backup_type:{type:1}, backup_status:{status:4, status_str:"CANCEL"}, backup_ta
   sk_id:0, copy_id:0, encryption_mode:0})
   

关键诊断信息

触发条件

系统租户处于 prepare 状态，普通租户处于 schedule 状态，发起 do_cancel，备份的状态卡在 cancel。

事后诊断

1. 备份过程中用户执行 alter system cancel all backup force 命令，使用 select value from oceanbase.__all_tenant_backup_info where name = 'backup_status' 可以看到集群备份状态一直卡在 cancel 状态，长时间不能推进。

2. 通过 trace_id 查找相关日志看到 cancel 的报错信息

   grep "YBxxxxx-xxxxxx-xxxxxx-xxxxxx" observer.log.2* observer.log | vi -
   

   若看到报错文件、错误码以及报错信息完全符合，即可确认该错误。

   [2023-11-07 16:46:16.519279] WDIAG [SHARE] ob_backup_path.cpp:1922 [2697475][0][YBxxxxx-xxxxxx-xxxxxx-xxxxxx] [lt=10] [dc=0][errcode=-4002] invalid argument!(ret=-4002, param={tenant_id:1, incarnation:1, c
   opy_id:0, full_set_id:397, inc_set_id:397, backup_date:20231030, compatible:0, backup_dest_ptr:0x7fa0a53636f0}, buffer_length=2048, buffer=0x7fa0a53621b0)
   [2023-11-07 16:46:16.519310] WDIAG [SERVER] ob_backup_operator.cpp:2784 [2697475][0][YBxxxxx-xxxxxx-xxxxxx-xxxxxx] [lt=14] [dc=0][errcode=-4002] fail to get backup set relative path(ret=-4002, param={tenan
   t_id:1, incarnation:1, copy_id:0, full_set_id:397, inc_set_id:397, backup_date:20231030, compatible:0, backup_dest_ptr:0x7fa0a53636f0}, item={tenant_id:1, backup_set_id:397, incarnation:1, copy_id:0, snapshot_v
   ersion:1698644686857754, prev_full_backup_set_id:0, prev_inc_backup_set_id:0, prev_backup_data_version:0, pg_count:0, macro_block_count:0, finish_pg_count:0, finish_macro_block_count:0, input_bytes:0, output_by
   tes:0, start_time:1698644686857754, end_time:0, compatible:0, cluster_version:0, backup_type:{type:1}, status:0, result:-4072, cluster_id:1641437681, backup_dest:"oss://xxx", backup_data_version:0, backup_schema_version:1697688796461568, partition_count:0, finish_partition_co
   unt:0, encryption_mode:0, passwd:"", file_status:1, start_replay_log_ts:0, date:20231030, backup_level:2})
   [2023-11-07 16:46:16.519334] WDIAG [SERVER] ob_backup_operator.cpp:3264 [2697475][0][YBxxxxx-xxxxxx-xxxxxx-xxxxxx] [lt=12] [dc=0][errcode=-4002] failed to fill on item(ret=-4002, backup_set_file_info={tena
   nt_id:1, backup_set_id:397, incarnation:1, copy_id:0, snapshot_version:1698644686857754, prev_full_backup_set_id:0, prev_inc_backup_set_id:0, prev_backup_data_version:0, pg_count:0, macro_block_count:0, finish_
   pg_count:0, finish_macro_block_count:0, input_bytes:0, output_bytes:0, start_time:1698644686857754, end_time:0, compatible:0, cluster_version:0, backup_type:{type:1}, status:0, result:-4072, cluster_id:16414376
   81, backup_dest:"oss://xxx", backup_data_version:0, backup_schema_
   version:1697688796461568, partition_count:0, finish_partition_count:0, encryption_mode:0, passwd:"", file_status:1, start_replay_log_ts:0, date:20231030, backup_level:2})
   [2023-11-07 16:46:16.519395] WDIAG [RS] ob_root_backup.cpp:3177 [2697475][0][YBxxxxx-xxxxxx-xxxxxx-xxxxxx] [lt=49] [dc=0][errcode=-4002] failed to insert tenant backup set file info(ret=-4002, backup_set_f
   ile_info={tenant_id:1, backup_set_id:397, incarnation:1, copy_id:0, snapshot_version:1698644686857754, prev_full_backup_set_id:0, prev_inc_backup_set_id:0, prev_backup_data_version:0, pg_count:0, macro_block_co
   unt:0, finish_pg_count:0, finish_macro_block_count:0, input_bytes:0, output_bytes:0, start_time:1698644686857754, end_time:0, compatible:0, cluster_version:0, backup_type:{type:1}, status:0, result:-4072, clust
   er_id:1641437681, backup_dest:"oss://xxx", backup_data_version:0,
   backup_schema_version:1697688796461568, partition_count:0, finish_partition_count:0, encryption_mode:0, passwd:"", file_status:1, start_replay_log_ts:0, date:20231030, backup_level:2})
   [2023-11-07 16:46:16.519420] WDIAG [RS] ob_root_backup.cpp:2142 [2697475][0][YBxxxxx-xxxxxx-xxxxxx-xxxxxx] [lt=11] [dc=0][errcode=-4002] failed to insert tenant backup set file failed(ret=-4002, sys_backup
   _task={tenant_id:1, backup_set_id:397, incarnation:1, snapshot_version:1698644686857754, prev_full_backup_set_id:0, prev_inc_backup_set_id:0, prev_backup_data_version:0, pg_count:0, macro_block_count:0, finish_
   pg_count:0, finish_macro_block_count:0, input_bytes:0, output_bytes:0, start_time:1698644686857754, end_time:0, compatible:0, cluster_version:0, backup_type:{type:1}, status:1, device_type:0, result:-4072, clus
   ter_id:1641437681, backup_dest:{device_type:0, root_path:"oss://xxx", type:"OSS"}, backup_data_version:0, backup_schema_version:1697688796461568, partition_count:0, finish_partition_count:0, encrypt
   ion_mode:0, passwd:"", is_mark_deleted:false, start_replay_log_ts:0, date:20231030, copy_id:0, backup_level:2})
   [2023-11-07 16:46:16.519665] WDIAG [RS] ob_root_backup.cpp:1056 [2697475][0][YBxxxxx-xxxxxx-xxxxxx-xxxxxx] [lt=11] [dc=0][errcode=-4002] failed to do sys tenant cancel(ret=-4002, info={tenant_id:1, backup_
   set_id:397, incarnation:1, backup_dest:"oss://xxx", backup_backup_
   dest:"", backup_snapshot_version:1698644686857754, backup_schema_version:1697688796461568, backup_data_version:1169, detected_backup_region:"", backup_type:{type:1}, backup_status:{status:4, status_str:"CANCEL"
   }, backup_task_id:0, copy_id:0, encryption_mode:0})
   [2023-11-07 16:46:16.519686] WDIAG [RS] ob_root_backup.cpp:554 [2697475][0][YBxxxxx-xxxxxx-xxxxxx-xxxxxx] [lt=13] [dc=0][errcode=-4002] failed to do cancel(ret=-4002, info={tenant_id:1, backup_set_id:397,
   incarnation:1, backup_dest:"oss://xxx", backup_backup_dest:"", bac
   kup_snapshot_version:1698644686857754, backup_schema_version:1697688796461568, backup_data_version:1169, detected_backup_region:"", backup_type:{type:1}, backup_status:{status:4, status_str:"CANCEL"}, backup_ta
   sk_id:0, copy_id:0, encryption_mode:0})
   

问题原因

发起备份时，系统租户处于 prepare 状态，普通租户处于 schedule 状态，当 do_scheduler 出错时，导致构造的备份 task 没有插入内部表。当发起 force_cancel 时，因没有在内部表内查到任务，会对任务进行构造，构造新任务 task struct 时，没有将 compatible 赋值，在后续的流程中进行了对 task 的有效性进行了校检，compatible 需要大于 0。导致出现了 4002 错误。从而使得备份卡住。

问题的风险及影响

会导致 force_cancel 备份任务一直无法退出，下一轮新的备份不能发起。

影响租户

影响 OceanBase 数据库中的 SYS 租户和 Oracle 租户以及 MySQL 租户。

影响版本

OceanBase 数据库企业版 V3.1.2 GA（oceanbase-3.1.2-20210618150922）及之后版本、V3.2.3 GA（oceanbase-3.2.3.0-20220418212020）及之后版本、V3.2.4 GA（oceanbase-3.2.4.0-100000072022102819）及之后版本。

解决方法

升级至问题已修复版本。目前已修复的版本包括 OceanBase 数据库企业版 V3.2.3 BP11（oceanbase-3.2.3.3-111000032024070822）、V3.2.4 BP7（oceanbase-3.2.4.7-107000012023113010）。

规避方式

调整发起备份语句级别事务超时时间。

线上应急手段（人工介入恢复）。

1. 用 root 权限链接到 1002 租户，执行 SQL。

   update __all_tenant_backup_info set value = 'STOP' where name = 'backup_status';
   

2. 用 root 权限链接到 SYS 租户，执行 SQL。

   update __all_tenant_backup_info set value = 'CLEANUP' where name = 'backup_status';
   

3. 等待 SYS 租户的 __all_tennat_backup_info 中的 backup_status 变成 STOP 就恢复了。

   update __all_tenant_backup_info set value = '' where name = 'backup_data_version';
   update __all_tenant_backup_info set value = '' where name = 'backup_schema_version';
   update __all_tenant_backup_info set value = '' where name = 'backup_schema_snapshot_version';
   update __all_tenant_backup_info set value = 'STOP' where name = 'backup_status';
   update __all_tenant_backup_info set value = '' where name = 'backup_type';