OceanBase 数据库 V4.2.1 版本因为备份卡住导致合并卡住报错 Too Many sstables, cannot add another sstable any 的原因和解决方法

MySQL [oceanbase]> select * from __all_virtual_dag_warning_history where tenant_id=1002;

+---------------+----------+-----------+-----------------------------------+------------+----------------------+---------------------------------+---------+----------------------------+----------------------------+-----------+----------------------------------------------------------------------------------------------------------------------------------+
| svr_ip        | svr_port | tenant_id | task_id                           | module     | type                 | ret                             | status  | gmt_create                 | gmt_modified               | retry_cnt | warning_info                                                                                                                     |
+---------------+----------+-----------+-----------------------------------+------------+----------------------+---------------------------------+---------+----------------------------+----------------------------+-----------+----------------------------------------------------------------------------------------------------------------------------------+
| xxx.xxx.xxx.160 |     2882 |      1002 | xxxxxxxxxxxx-xxxxxxxxxxxxxxxx-x-x | BACKUP     | PREFETCH_BACKUP_INFO | OB_OSS_ERROR                    | WARNING | 2024-05-13 13:32:28.483716 | 2024-05-13 13:32:28.483716 |         0 | info="DAG_PREFETCH_BACKUP_INFO";tenant_id=1002;backup_set_id=244;backup_data_type=2;ls_id=1001;turn_id=1;retry_id=14;task_id=2;  |
| xxx.xxx.xxx.160 |     2882 |      1002 | xxxxxxxxxxxx-xxxxxxxxxxxxxxxx-x-x | BACKUP     | BACKUP_DATA          | OB_OSS_ERROR                    | WARNING | 2024-05-13 15:32:45.892451 | 2024-05-13 15:32:45.892451 |         0 | info="DAG_BACKUP_DATA";tenant_id=1002;backup_set_id=244;backup_data_type=2;ls_id=1001;turn_id=1;retry_id=18;task_id=5;           |
| xxx.xxx.xxx.161 |     2882 |      1002 | xxxxxxxxxxxx-xxxxxxxxxxxxxxxx-x-x | BACKUP     | PREFETCH_BACKUP_INFO | OB_OSS_ERROR                    | WARNING | 2024-05-13 11:31:21.155562 | 2024-05-13 11:31:21.155562 |         0 | info="DAG_PREFETCH_BACKUP_INFO";tenant_id=1002;backup_set_id=244;backup_data_type=2;ls_id=1001;turn_id=1;retry_id=0;task_id=116; |
| xxx.xxx.xxx.161 |     2882 |      1002 | xxxxxxxxxxxx-xxxxxxxxxxxxxxxx-x-x | BACKUP     | PREFETCH_BACKUP_INFO | OB_OSS_ERROR                    | WARNING | 2024-05-13 11:51:52.892445 | 2024-05-13 11:51:52.892445 |         0 | info="DAG_PREFETCH_BACKUP_INFO";tenant_id=1002;backup_set_id=244;backup_data_type=2;ls_id=1001;turn_id=1;retry_id=3;task_id=0;   |
| xxx.xxx.xxx.161 |     2882 |      1002 | xxxxxxxxxxxx-xxxxxxxxxxxxxxxx-x-x | BACKUP     | PREFETCH_BACKUP_INFO | OB_BACKUP_MAJOR_NOT_COVER_MINOR | WARNING | 2024-09-19 15:13:26.035467 | 2024-09-19 15:13:26.035467 |         0 | info="DAG_PREFETCH_BACKUP_INFO";tenant_id=1002;backup_set_id=304;backup_data_type=2;ls_id=1001;turn_id=1;retry_id=0;task_id=114; |
| xxx.xxx.xxx.161 |     2882 |      1002 | xxxxxxxxxxxx-xxxxxxxxxxxxxxxx-x-x | COMPACTION | MAJOR_MERGE          | OB_SIZE_OVERFLOW                | RETRYED | 2024-09-19 02:05:43.665832 | 2024-09-20 14:32:39.611969 |      1092 | info="DAG_MAJOR_MERGE";ls_id=1001;tablet_id=207447;compaction_scn=1726682401689713864;extra_info="merge_type="MEDIUM_MERGE"";    |
| xxx.xxx.xxx.161 |     2882 |      1002 | xxxxxxxxxxxx-xxxxxxxxxxxxxxxx-x-x | COMPACTION | MAJOR_MERGE          | OB_SIZE_OVERFLOW                | RETRYED | 2024-09-19 02:06:20.253585 | 2024-09-20 14:32:39.801456 |      1092 | info="DAG_MAJOR_MERGE";ls_id=1001;tablet_id=207445;compaction_scn=1726682401689713864;extra_info="merge_type="MEDIUM_MERGE"";    |
| xxx.xxx.xxx.161 |     2882 |      1002 | xxxxxxxxxxxx-xxxxxxxxxxxxxxxx-x-x | COMPACTION | MAJOR_MERGE          | OB_SIZE_OVERFLOW                | RETRYED | 2024-09-19 02:07:43.587289 | 2024-09-20 14:32:40.533494 |      1092 | info="DAG_MAJOR_MERGE";ls_id=1001;tablet_id=207450;compaction_scn=1726682401689713864;extra_info="merge_type="MEDIUM_MERGE"";    |
| xxx.xxx.xxx.161 |     2882 |      1002 | xxxxxxxxxxxx-xxxxxxxxxxxxxxxx-x-x | COMPACTION | MAJOR_MERGE          | OB_SIZE_OVERFLOW                | RETRYED | 2024-09-19 02:07:49.163709 | 2024-09-20 14:32:41.415234 |      1092 | info="DAG_MAJOR_MERGE";ls_id=1001;tablet_id=207451;compaction_scn=1726682401689713864;extra_info="merge_type="MEDIUM_MERGE"";    |
| xxx.xxx.xxx.161 |     2882 |      1002 | xxxxxxxxxxxx-xxxxxxxxxxxxxxxx-x-x | COMPACTION | MAJOR_MERGE          | OB_SIZE_OVERFLOW                | RETRYED | 2024-09-19 02:07:37.161357 | 2024-09-20 14:32:41.594973 |      1090 | info="DAG_MAJOR_MERGE";ls_id=1001;tablet_id=207448;compaction_scn=1726682401689713864;extra_info="merge_type="MEDIUM_MERGE"";    |
| xxx.xxx.xxx.161 |     2882 |      1002 | xxxxxxxxxxxx-xxxxxxxxxxxxxxxx-x-x | COMPACTION | MAJOR_MERGE          | OB_SIZE_OVERFLOW                | RETRYED | 2024-09-19 02:07:40.274592 | 2024-09-20 14:32:42.065449 |      1090 | info="DAG_MAJOR_MERGE";ls_id=1001;tablet_id=207446;compaction_scn=1726682401689713864;extra_info="merge_type="MEDIUM_MERGE"";    |
| xxx.xxx.xxx.161 |     2882 |      1002 | xxxxxxxxxxxx-xxxxxxxxxxxxxxxx-x-x | COMPACTION | MAJOR_MERGE          | OB_SIZE_OVERFLOW                | RETRYED | 2024-09-19 02:10:18.698726 | 2024-09-20 14:32:42.068798 |      1089 | info="DAG_MAJOR_MERGE";ls_id=1001;tablet_id=207449;compaction_scn=1726682401689713864;extra_info="merge_type="MEDIUM_MERGE"";    |
| xxx.xxx.xxx.161 |     2882 |      1002 | xxxxxxxxxxxx-xxxxxxxxxxxxxxxx-x-x | COMPACTION | MAJOR_MERGE          | OB_SIZE_OVERFLOW                | RETRYED | 2024-09-19 02:10:35.532628 | 2024-09-20 14:32:42.289718 |      1089 | info="DAG_MAJOR_MERGE";ls_id=1001;tablet_id=207444;compaction_scn=1726682401689713864;extra_info="merge_type="MEDIUM_MERGE"";    |
+---------------+----------+-----------+-----------------------------------+------------+----------------------+---------------------------------+---------+----------------------------+----------------------------+-----------+----------------------------------------------------------------------------------------------------------------------------------+
13 rows in set (0.01 sec)

observer.log.20240920120623366:[2024-09-20 12:00:40.261715] WDIAG [STORAGE] check_ready_for_read (ob_tablet_table_store.cpp:1599) [57654][T1002_MAJOR_MER][T1002][xxxxx-xxxxx-xxxxx-xxxxx] [lt=10][errcode=-4019] Too Many sstables, cannot add another sstable any more(ret=-4019, this={this:0x7efa34c266e0, version:257, major_tables:{cnt:129, is_inited:true}, minor_tables:{cnt:0, is_inited:false}, ddl_sstables:{cnt:0, is_inited:false}, meta_major_tables:{cnt:0, is_inited:false}, memtables:{count:0}, is_ready_for_read:false, sstable_arrays:{major_tables:{{i:0, addr:388350493-1567367-0-2305855460324012032-0, type:"MAJOR", tablet_id:{id:207444}