OceanBase 数据库执行备份过程中出现 format file does not exist 异常的原因和解决方法

2024-06-28 13:26:47.104  INFO 111 --- [pool-manual-subtask-executor16,340412badd2f467f,2d666ca3a2d5] c.o.ocp.obsdk.connector.ConnectTemplate  : [obsdk] sql: SELECT * FROM (SELECT * FROM (SELECT job_id, incarnation, job.tenant_id as tenant_id, job.backup_set_id as backup_set_id, backup_type, path AS backup_dest, start_timestamp as start_time, end_timestamp as end_time, now(6) as check_time, status, comment, description, 'TENANT' as backup_level, file.MIN_RESTORE_SCN_DISPLAY as snapshot_version_time FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) AS job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id AS sid, min_restore_scn_display FROM CDB_OB_BACKUP_SET_FILES) AS file ON job.tenant_id = file.tid AND job.backup_set_id = file.sid) WHERE tenant_id != 1 AND description = ? ORDER BY start_time DESC)  where start_time >= '2024-06-28 13:26:21', args: [ALTER BACKUP TENANT FULL_BACKUP BY BACKUP_NOW AT 2024-06-28 13:26:21.0]
2024-06-28 13:26:47.182  INFO 111 --- [pool-manual-subtask-executor16,340412badd2f467f,2d666ca3a2d5] c.o.o.b.i.t.s.WaitDataBackupSuccessTask  : check backup task result: [DataBackupHistoryWrapperTask(super=DataBackupHistoryTask(taskId=75, backupType=FULL, taskType=null, taskCount=null, clusterName=null, clusterId=null, obClusterId=null, obTenantId=1002, clusterType=null, tenantId=1002, tenantName=null, snapshotVersionTime=null, startTime=2024-06-28T05:26:22.097556Z, finishTime=2024-06-28T05:26:46.105445Z, progress=null, status=FAILED, errorMsg=(ROOTSERVICE)addr: 10.55.172.80:2882, result: -9080(the format file does not exist under the destination), trace_id: YBxxxxx-xxxxx-xxxxx-xxxxx, dataVersion=null, backupUri=file:///obbackup/oceanbase_xxxxx/xxxxx/tenant_incarnation_1/xxxxx/data, duration=24, backupSize=null, dataType=null, schemaVersion=null, lastSuccessTime=null, backupSetId=75, backupLevel=null), backupMode=PHYSICAL_BACKUP, dataBackupTaskType=FULL_BACKUP, backupDimension=TENANT, backupSizeUnit=null, backupSetFile=null, backupSetFilePath=null, dataSecondaryBackupTaskIds=null)]
2024-06-28 13:26:47.223 ERROR 111 --- [pool-manual-subtask-executor16,340412badd2f467f,2d666ca3a2d5] c.o.o.c.t.e.c.w.subtask.SubtaskExecutor  : null

java.lang.NullPointerException: null
    at com.oceanbase.ocp.backup.internal.data.meta.PhysicalBackupFileMetaManager.updateClusterDataSetCheckpoint(PhysicalBackupFileMetaManager.java:111)
    at com.oceanbase.ocp.backup.internal.task.schedule.WaitDataBackupSuccessTask.lambda$checkDataBackupTaskResult$3(WaitDataBackupSuccessTask.java:140)
    at java.util.ArrayList.forEach(ArrayList.java:1259)
    at com.oceanbase.ocp.backup.internal.task.schedule.WaitDataBackupSuccessTask.checkDataBackupTaskResult(WaitDataBackupSuccessTask.java:134)
    at com.oceanbase.ocp.backup.internal.task.schedule.WaitDataBackupSuccessTask.run(WaitDataBackupSuccessTask.java:205)
    at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.execute(JavaSubtaskRunner.java:59)
    at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:31)
    at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:25)
    at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:76)
    at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.doRun(SubtaskExecutor.java:193)
    at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.redirectConsoleOutput(SubtaskExecutor.java:187)
    at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.lambda$submit$2(SubtaskExecutor.java:124)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

Set state for subtask: 13044117, operation:EXECUTE, state: FAILED

rootservice.log.20240628134017776:[2024-06-28 13:26:46.105391] WDIAG [SHARE] check_dest_validity (ob_backup_store.cpp:443) [1497617][T1001_BackupDat][T1001][YBxxxxx-xxxxx-xxxxx-xxxxx] [lt=9][errcode=-9080] format file does not exist(ret=-9080, backup_dest={root_path:"file:///obbackup/oceanbase_xxxxx/xxxxx/tenant_incarnation_1/xxxxx/data", storage_info:{endpoint:"", access_id:"", extension:"", type:"FILE"}})

2024-06-28 13:26:47.104  INFO 111 --- [pool-manual-subtask-executor16,340412badd2f467f,2d666ca3a2d5] c.o.ocp.obsdk.connector.ConnectTemplate  : [obsdk] sql: SELECT * FROM (SELECT * FROM (SELECT job_id, incarnation, job.tenant_id as tenant_id, job.backup_set_id as backup_set_id, backup_type, path AS backup_dest, start_timestamp as start_time, end_timestamp as end_time, now(6) as check_time, status, comment, description, 'TENANT' as backup_level, file.MIN_RESTORE_SCN_DISPLAY as snapshot_version_time FROM ( SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, NULL AS end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOBS UNION SELECT job_id, incarnation, tenant_id, backup_set_id, backup_type, path, start_timestamp, end_timestamp, status, comment, description FROM CDB_OB_BACKUP_JOB_HISTORY ) AS job LEFT JOIN (SELECT tenant_id AS tid, backup_set_id AS sid, min_restore_scn_display FROM CDB_OB_BACKUP_SET_FILES) AS file ON job.tenant_id = file.tid AND job.backup_set_id = file.sid) WHERE tenant_id != 1 AND description = ? ORDER BY start_time DESC)  where start_time >= '2024-06-28 13:26:21', args: [ALTER BACKUP TENANT FULL_BACKUP BY BACKUP_NOW AT 2024-06-28 13:26:21.0]