问题现象
备租户数据备份失败,CDB_OB_BACKUP_JOB_HISTORY 视图报错 (ROOTSERVICE)addr: xxxx, result: -4012(Timeout), trace_id: xxx。
查询 SQL 如下。
select * from oceanbase.CDB_OB_BACKUP_JOB_HISTORY where status = 'FAILED' and result = '-4012';
关键诊断信息
触发条件
备库积攒日志较多的时候发起数据备份。
事前巡检
检查备租户的可读位点和可回放位点相差是否较大。
select timestampdiff(minute, scn_to_timestamp(readable_scn), scn_to_timestamp(replayable_scn)) >= 10 as time_diff from oceanbase.__all_virtual_tenant_info where tenant_id = xxx;
若结果为 true,则可认为可读位点和可回放位点相差较大,存在发起备份失败的风险。
事后诊断
关键日志信息。
rootservice.log.20240414182348114:[2024-04-14 18:22:25.874392] INFO [SHARE] get_backup_scn (ob_backup_struct.cpp:2105) [1418765][T1007_BackupDat][T1007][xxxxx-xxxxx-xxxxx-xxxxx] [lt=8] get tenant gts(ret=0, ret="OB_SUCCESS", tenant_id=1008, scn={val:1713086125166411949, v:0})
rootservice.log.20240414182348114:[2024-04-14 18:22:25.879557] WDIAG [RS] persist_ls_attr_info_ (ob_backup_data_set_task_mgr.cpp:251) [1418765][T1007_BackupDat][T1007][xxxxx-xxxxx-xxxxx-xxxxx] [lt=9][errcode=-4012] failed to calc backup user ls scn(ret=-4012)
rootservice.log.20240414182348114:[2024-04-14 18:22:25.879568] WDIAG [RS] backup_sys_meta_ (ob_backup_data_set_task_mgr.cpp:445) [1418765][T1007_BackupDat][T1007][xxxxx-xxxxx-xxxxx-xxxxx] [lt=9][errcode=-4012] fail to do persist ls task(ret=-4012)
rootservice.log.20240414182348114:[2024-04-14 18:22:25.879573] WDIAG [RS] process (ob_backup_data_set_task_mgr.cpp:143) [1418765][T1007_BackupDat][T1007][xxxxx-xxxxx-xxxxx-xxxxx] [lt=3][errcode=-4012] fail to backup sys meta(ret=-4012, set_task_attr_={task_id:72, tenant_id:1008, incarnation_id:1, job_id:1157, backup_set_id:8, start_ts:1713089521669309, end_ts:0, start_scn:{val:1713084812609502439, v:0}, end_scn:{val:0, v:0}, user_ls_start_scn:{val:0, v:0}, data_turn_id:0, meta_turn_id:1, minor_turn_id:1, major_turn_id:1, status:{status:6}, encryption_mode:0, passwd:"", stats:{input_bytes:0, output_bytes:0, tablet_count:0, finish_tablet_count:0, macro_block_count:0, finish_macro_block_count:0, extra_bytes:0, finish_file_count:0}, backup_path:"", retry_cnt:0, result:0, comment:""})
rootservice.log.20240414182348114:[2024-04-14 18:22:25.879594] WDIAG [RS] do_set_task_ (ob_backup_data_scheduler.cpp:1377) [1418765][T1007_BackupDat][T1007][xxxxx-xxxxx-xxxxx-xxxxx] [lt=20][errcode=-4012] [DATA_BACKUP]failed to backup ls task(ret=-4012, set_task_mgr={meta_tenant_id:1007, set_task_attr:{task_id:72, tenant_id:1008, incarnation_id:1, job_id:1157, backup_set_id:8, start_ts:1713089521669309, end_ts:0, start_scn:{val:1713084812609502439, v:0}, end_scn:{val:0, v:0}, user_ls_start_scn:{val:0, v:0}, data_turn_id:0, meta_turn_id:1, minor_turn_id:1, major_turn_id:1, status:{status:6}, encryption_mode:0, passwd:"", stats:{input_bytes:0, output_bytes:0, tablet_count:0, finish_tablet_count:0, macro_block_count:0, finish_macro_block_count:0, extra_bytes:0, finish_file_count:0}, backup_path:"", retry_cnt:0, result:0, comment:""}})
rootservice.log.20240414182348114:[2024-04-14 18:22:25.879608] WDIAG [RS] process (ob_backup_data_scheduler.cpp:1135) [1418765][T1007_BackupDat][T1007][xxxxx-xxxxx-xxxxx-xxxxx] [lt=12][errcode=-4012] [DATA_BACKUP]failed to backup data(ret=-4012, job_attr_={job_id:1157, tenant_id:1008, incarnation_id:1, backup_set_id:8, initiator_tenant_id:1, initiator_job_id:9, executor_tenant_id:[1008], plus_archivelog:false, backup_level:{level:2}, backup_type:{type:1}, encryption_mode:0, passwd:"", backup_path:"", description:"ALTER BACKUP TENANT FULL_BACKUP BY BACKUP_NOW AT 2024-04-14 18:12:01.0", start_ts:1713089521669309, end_ts:0, status:{status:1}, result:0, can_retry:true, retry_count:0, comment:""})
问题原因
主租户跑批使机器压力比较大,备租户积攒了较多的未回放日志,导致备租户的可读位点推得很慢,与可回放位点相差较大。此时发起备份,备份为保证看到完整的 ls 列表,要求可读位点必须在 10 分钟内推过 1 号日志流的转储位点。因为可读位点推得很慢,所以一直检查失败。导致备份流程重试 4 次失败退出。
问题的风险及影响
备租户数据备份失败。
影响租户
影响 OceanBase 数据库中的 SYS 租户和 Oracle 租户以及 MySQL 租户。
影响版本
OceanBase 数据库企业版 V4.2.0 GA(oceanbase-4.2.0.0-100010082023083014)及之后版本、V4.2.1 GA(oceanbase-4.2.1.0-100000182023092722)及之后版本、V4.2.2 GA(oceanbase-4.2.2.0-100000082024011317)及之后版本、V4.3.0(oceanbase-4.3.0.0-100000072024020200)及之后版本。
解决方法及规避方式
解决方法:
重新发起备份。
规避方式:
无法规避。