OMS Store 报错 errcode=-4018，DML parser thread exits 问题的原因和解决方法-OceanBase数据库使用指南

问题现象

OMS 数据迁移同步链路异常，导致链路延迟无法推进，重启 Store 无法解决。同步链路报错信息 DML parser thread exits, thread_index=4, err=-4018。

重启 Store 报错 errcode=-4018，OB_ENTRY_NOT_EXIST，Store 进程自动退出，OBCDC 日志报错如下。

[2024-08-10 14:43:11.616202] EDIAG [TLOG.PARSER] get_table_info_of_tablet_ (ob_log_part_trans_parser.cpp:515) [7236][][T0][xxxxx-xxxxx-xxxxx-xxxxx] [lt=68][errcode=-4018] get_table_info_of_tablet for dml_tx failed(ret=-4018, ret="OB_ENTRY_NOT_EXIST", tablet_id={id:269631}, tenant_ls_id={tenant_id:1056, ls_id:{id:1003}}, tenant={tenant_id:1056, tenant_name:"s_6010_dps_r", state:"NORMAL", active_ls_count:4, sys_ls_progress:"[2024-08-10 14:28:51.525076]", drop_tenant_tstamp:"[INVALID]", ddl_log_lsn:{lsn:18446744073709551615}, global_seq:0, global_schema_version:1723270596000000, start_schema_version:1723270596000000, committer_trans_commit_version:1723270595999999999, committer_global_heartbeat:1723270595999999999, committer_cur_schema_version:1723270596000000, committer_next_trans_schema_version:1723270596000000, task_queue:{log_entry_task_count:0, next_task_seq:0}}) BACKTRACE:0x17e6fee5 0xbd6c770 0xbd6c331 0xbd6bfb4 0xbd6bda4 0xbf8fe47 0xbf8c027 0xbf8a337 0xbf89350 0xbdb7a86 0xbea8109 0x7f9aaf90bea5 0x7f9aaef25b0d

版本： OMS V4.2.2、libobcdc V4.2.1.7、OceanBase 数据库 V4.2.1 BP8。

排查过程

上述报错发现一个 tablet 的映射关系找不到。

排除 OceanBase 数据库已知的 BUG：CDC 报错进程退出 errno=-4018, errmsg=DML parser thread exits。

可能出现被删除的tablet仍然有事务写入对应集群下需要查询下 select * from oceanbase.__all_virtual_tablet_to_table_history where tablet_id = xxxx; 发现这个 tablet 没有被删除。

MySQL [oceanbase]> select * from __all_virtual_tablet_to_table_history where tablet_id in (267851,269631);
+-----------+-----------+------------------+----------------------------+----------------------------+----------+------------+
| tenant_id | tablet_id | schema_version   | gmt_create                 | gmt_modified               | table_id | is_deleted |
+-----------+-----------+------------------+----------------------------+----------------------------+----------+------------+
|      1056 |    267851 | 1720254127553504 | 2024-07-06 16:22:07.600503 | 2024-07-06 16:22:07.600503 |   618309 |          0 |
|      1056 |    269631 | 1720957003267936 | 2024-07-14 19:36:43.364500 | 2024-07-14 19:36:43.364500 |   620136 |          0 |
+-----------+-----------+------------------+----------------------------+----------------------------+----------+------------+
2 rows in set (1.324 sec)

MySQL [oceanbase]> select /*+ query_timeout(100000000) parallel(16) */ * from __all_virtual_table_history where tenant_id=1056 and table_id in (618309,620136) and (gmt_create>'2024-08-10' or gmt_modified>'2024-08-10');
Empty set (0.016 sec)

MySQL [oceanbase]> select /*+ query_timeout(100000000) parallel(16) */ * from __all_virtual_ddl_operation where tenant_id=1056 and table_id in (618309,620136) and (gmt_create>'2024-08-10' or gmt_modified>'2024-08-10');
Empty set (17.096 sec)

获取租户的启动位点，看字典位置信息，发现这个租户 2 小时 DUMP 一次字典，但小于等于启动位点的最近的字典 DUMP 记录距离上一次字典 DUMP 不足2小时，怀疑存在切主；同时怀疑该字典内容不完整。

MySQL [oceanbase]> select * from cdb_ob_data_dictionary_in_log where tenant_id=1056 and report_time like '2024-08-10%' order by report_time;
+-----------+---------------------+----------------------------+-------------+-------------+
| TENANT_ID | SNAPSHOT_SCN        | REPORT_TIME                | START_LSN   | END_LSN     |
+-----------+---------------------+----------------------------+-------------+-------------+
|      1056 | 1723222548141847405 | 2024-08-10 00:55:50.041311 | 98565483648 | 98629303840 |
|      1056 | 1723229750556239820 | 2024-08-10 02:55:52.459552 | 98691924939 | 98755124638 |
|      1056 | 1723236952967386132 | 2024-08-10 04:55:54.835007 | 98766848815 | 98830618399 |
|      1056 | 1723244155343688225 | 2024-08-10 06:55:57.250106 | 98845323320 | 98908203296 |
|      1056 | 1723251357755076318 | 2024-08-10 08:55:59.619237 | 98919979228 | 98983721551 |
|      1056 | 1723258560129462554 | 2024-08-10 10:56:02.029073 | 98995753278 | 99059209539 |
|      1056 | 1723265762536957850 | 2024-08-10 12:56:04.411407 | 99073606127 | 99136792593 |
|      1056 | 1723268481901533653 | 2024-08-10 13:41:18.249053 | 99142620808 | 99159393430 |
|      1056 | 1723272964918528107 | 2024-08-10 14:56:06.782833 | 99182932122 | 99245813096 |
|      1056 | 1723280167287802343 | 2024-08-10 16:56:09.210860 | 99257845155 | 99321339640 |
|      1056 | 1723287369716006392 | 2024-08-10 18:56:12.034496 | 99351502860 | 99415704725 |
|      1056 | 1723294572538628236 | 2024-08-10 20:56:14.421463 | 99428079955 | 99491192213 |
|      1056 | 1723301774926844132 | 2024-08-10 22:56:16.817293 | 99503141291 | 99566684724 |
+-----------+---------------------+----------------------------+-------------+-------------+
13 rows in set (0.016 sec)

获取异常字典和正常字典的 clog 日志，发现异常字典中查不到对应的 tabelt_id 的数据，而正常字典包含。

从内部表获取切主记录，结合切主场景以及代码分析定位问题。

 SELECT value4,svr_ip,svr_port,event,name3,value3 FROM oceanbase.__all_server_event_history WHERE module="ELECTION" AND value1=${tenant_id} AND value2=${ls_id} ORDER BY value4;

 MySQL [oceanbase]> source sql/servers.sql
 Query OK, 0 rows affected (0.001 sec)

 +---------------------+-------+---------------+----------+-----------------+--------+----------------------------+-----------+----------------------------+
 | now()               | zone  | svr_ip        | svr_port | with_rootserver | status | start_service_time         | stop_time | build_version              |
 +---------------------+-------+---------------+----------+-----------------+--------+----------------------------+-----------+----------------------------+
 | 2024-08-17 13:08:22 | zone1 | xx.xx.xxx.81  |     2882 | NO              | ACTIVE | 2024-08-01 12:43:44.224229 | NULL      | 4.2.1.8_108000052024072217 |
 | 2024-08-17 13:08:22 | zone1 | xx.xx.xxx.82  |     2882 | NO              | ACTIVE | 2024-08-01 12:42:52.174352 | NULL      | 4.2.1.8_108000052024072217 |
 | 2024-08-17 13:08:22 | zone1 | xx.xx.xxx.83  |     2882 | NO              | ACTIVE | 2024-08-01 12:43:56.428796 | NULL      | 4.2.1.8_108000052024072217 |
 | 2024-08-17 13:08:22 | zone2 | xx.xx.xxx.92  |     2882 | NO              | ACTIVE | 2024-08-01 12:33:27.341245 | NULL      | 4.2.1.8_108000052024072217 |
 | 2024-08-17 13:08:22 | zone2 | xx.xx.xxx.93  |     2882 | YES             | ACTIVE | 2024-08-01 12:32:30.224258 | NULL      | 4.2.1.8_108000052024072217 |
 | 2024-08-17 13:08:22 | zone2 | xx.xx.xxx.94  |     2882 | NO              | ACTIVE | 2024-08-01 12:33:36.343530 | NULL      | 4.2.1.8_108000052024072217 |
 | 2024-08-17 13:08:22 | zone3 | xx.xx.xxx.103 |     2882 | NO              | ACTIVE | 2024-08-01 12:54:15.355386 | NULL      | 4.2.1.8_108000052024072217 |
 | 2024-08-17 13:08:22 | zone3 | xx.xx.xxx.104 |     2882 | NO              | ACTIVE | 2024-08-01 12:53:04.639901 | NULL      | 4.2.1.8_108000052024072217 |
 | 2024-08-17 13:08:22 | zone3 | xx.xx.xxx.105 |     2882 | NO              | ACTIVE | 2024-08-01 12:53:45.917248 | NULL      | 4.2.1.8_108000052024072217 |
 +---------------------+-------+---------------+----------+-----------------+--------+----------------------------+-----------+----------------------------+
 9 rows in set (0.004 sec)

 MySQL [oceanbase]> select tenant_id,ls_id,svr_ip,svr_port,role from gv$ob_log_stat where tenant_id=1056 order by 1,2,5;
 +-----------+-------+---------------+----------+----------+
 | tenant_id | ls_id | svr_ip        | svr_port | role     |
 +-----------+-------+---------------+----------+----------+
 |      1056 |     1 | xx.xx.xxx.104 |     2882 | FOLLOWER |
 |      1056 |     1 | xx.xx.xxx.92  |     2882 | FOLLOWER |
 |      1056 |     1 | xx.xx.xxx.81  |     2882 | LEADER   |
 |      1056 |  1001 | xx.xx.xxx.104 |     2882 | FOLLOWER |
 |      1056 |  1001 | xx.xx.xxx.92  |     2882 | FOLLOWER |
 |      1056 |  1001 | xx.xx.xxx.81  |     2882 | LEADER   |
 |      1056 |  1002 | xx.xx.xxx.104 |     2882 | FOLLOWER |
 |      1056 |  1002 | xx.xx.xxx.81  |     2882 | FOLLOWER |
 |      1056 |  1002 | xx.xx.xxx.92  |     2882 | LEADER   |
 |      1056 |  1003 | xx.xx.xxx.81  |     2882 | FOLLOWER |
 |      1056 |  1003 | xx.xx.xxx.92  |     2882 | FOLLOWER |
 |      1056 |  1003 | xx.xx.xxx.104 |     2882 | LEADER   |
 +-----------+-------+---------------+----------+----------+
 12 rows in set (0.027 sec)

 MySQL [oceanbase]> select value4,svr_ip,svr_port,event,name3,value3 
       from __all_server_event_history 
       where module='ELECTION' and value1=1056 and value2=1 
       order by value4;
 +----------------------------+---------------+----------+----------------------------+-------+---------------------------------------------------------------------------------------------------------+
 | value4                     | svr_ip        | svr_port | event                      | name3 | value3                                                                                                  |
 +----------------------------+---------------+----------+----------------------------+-------+---------------------------------------------------------------------------------------------------------+
 | 2024-08-10 13:15:06.554995 | xx.xx.xxx.104 |     2882 | lease expired              | INFO  | owner:"xx.xx.xxx.81:2882", expired time:13:15:06.539417, ballot number:26                               |
 | 2024-08-10 13:15:06.555127 | xx.xx.xxx.104 |     2882 | vote                       | INFO  | vote for "xx.xx.xxx.104:2882", reason:the only request                                                  |
 | 2024-08-10 13:15:06.557335 | xx.xx.xxx.92  |     2882 | lease expired              | INFO  | owner:"xx.xx.xxx.81:2882", expired time:13:15:06.539602, ballot number:26                               |
 | 2024-08-10 13:15:06.557451 | xx.xx.xxx.92  |     2882 | vote                       | INFO  | vote for "xx.xx.xxx.92:2882", reason:the only request                                                   |
 | 2024-08-10 13:15:07.36419  | xx.xx.xxx.81  |     2882 | lease expired to revoke    | INFO  | leader lease expired, server|lease:[13:15:10.533|13:15:06.539|13:15:06.539]                             |
 | 2024-08-10 13:15:10.557425 | xx.xx.xxx.81  |     2882 | lease expired              | INFO  | owner:"xx.xx.xxx.81:2882", expired time:13:15:10.533525, ballot number:26                               |
 | 2024-08-10 13:15:10.557562 | xx.xx.xxx.81  |     2882 | vote                       | INFO  | vote for "xx.xx.xxx.81:2882", reason:the only request                                                   |
 | 2024-08-10 13:15:17.552569 | xx.xx.xxx.92  |     2882 | vote                       | INFO  | vote for "xx.xx.xxx.81:2882", reason:LOG TS                                                             |
 | 2024-08-10 13:15:17.556527 | xx.xx.xxx.81  |     2882 | vote                       | INFO  | vote for "xx.xx.xxx.81:2882", reason:LOG TS                                                             |
 | 2024-08-10 13:15:17.557296 | xx.xx.xxx.81  |     2882 | decentralized to be leader | INFO  | server|lease:["xx.xx.xxx.81:2882"|2024-08-10 13:15:21.556,"xx.xx.xxx.92:2882"|2024-08-10 13:15:21.556]  |
 | 2024-08-10 13:41:12.536573 | xx.xx.xxx.81  |     2882 | lease expired to revoke    | INFO  | leader lease expired, server|lease:[13:41:16.332|13:41:12.390|13:41:12.390]                             |
 | 2024-08-10 13:41:12.56677  | xx.xx.xxx.104 |     2882 | lease expired              | INFO  | owner:"xx.xx.xxx.81:2882", expired time:13:41:12.39313, ballot number:28                                |
 | 2024-08-10 13:41:12.56807  | xx.xx.xxx.104 |     2882 | vote                       | INFO  | vote for "xx.xx.xxx.92:2882", reason:IP-PORT(priority equal)                                            |
 | 2024-08-10 13:41:12.58175  | xx.xx.xxx.92  |     2882 | lease expired              | INFO  | owner:"xx.xx.xxx.81:2882", expired time:13:41:12.39292, ballot number:28                                |
 | 2024-08-10 13:41:12.58277  | xx.xx.xxx.92  |     2882 | vote                       | INFO  | vote for "xx.xx.xxx.92:2882", reason:IP-PORT(priority equal)                                            |
 | 2024-08-10 13:41:12.58872  | xx.xx.xxx.92  |     2882 | decentralized to be leader | INFO  | server|lease:["xx.xx.xxx.92:2882"|2024-08-10 13:41:16.583,"xx.xx.xxx.104:2882"|2024-08-10 13:41:16.583] |
 | 2024-08-10 13:41:16.58566  | xx.xx.xxx.81  |     2882 | witness change leader      | INFO  | witness change leader : "xx.xx.xxx.81:2882" -> "xx.xx.xxx.92:2882"                                      |
 | 2024-08-10 13:41:17.202462 | xx.xx.xxx.92  |     2882 | prepare change leader      | INFO  | leader prepare to change leader : "xx.xx.xxx.92:2882" -> "xx.xx.xxx.81:2882", reason : ZONE PRIORITY    |
 | 2024-08-10 13:41:17.203543 | xx.xx.xxx.92  |     2882 | change leader to revoke    | INFO  | old leader revoke : "xx.xx.xxx.92:2882" -> "xx.xx.xxx.81:2882"                                          |
 | 2024-08-10 13:41:17.204620 | xx.xx.xxx.81  |     2882 | change leader to takeover  | INFO  | new leader takeover : "xx.xx.xxx.92:2882" -> "xx.xx.xxx.81:2882"                                        |
 | 2024-08-10 13:41:17.302390 | xx.xx.xxx.92  |     2882 | witness change leader      | INFO  | witness change leader : "xx.xx.xxx.92:2882" -> "xx.xx.xxx.81:2882"                                      |
 | 2024-08-10 13:41:17.306285 | xx.xx.xxx.81  |     2882 | witness change leader      | INFO  | witness change leader : "xx.xx.xxx.92:2882" -> "xx.xx.xxx.81:2882"                                      |
 | 2024-08-10 13:41:18.223468 | xx.xx.xxx.104 |     2882 | lease expired              | INFO  | owner:"xx.xx.xxx.92:2882", expired time:13:41:18.202143, ballot number:29                               |
 | 2024-08-10 13:41:18.223693 | xx.xx.xxx.104 |     2882 | vote                       | INFO  | vote for "xx.xx.xxx.104:2882", reason:the only request                                                  |
 | 2024-08-10 13:42:14.34434  | xx.xx.xxx.81  |     2882 | lease expired to revoke    | INFO  | leader lease expired, server|lease:[13:42:17.531|13:42:14.300|13:42:14.300]                             |
 | 2024-08-10 13:42:14.54862  | xx.xx.xxx.104 |     2882 | lease expired              | INFO  | owner:"xx.xx.xxx.81:2882", expired time:13:42:14.30307, ballot number:30                                |
 | 2024-08-10 13:42:14.54964  | xx.xx.xxx.104 |     2882 | vote                       | INFO  | vote for "xx.xx.xxx.104:2882", reason:the only request                                                  |
 | 2024-08-10 13:42:14.65612  | xx.xx.xxx.92  |     2882 | lease expired              | INFO  | owner:"xx.xx.xxx.81:2882", expired time:13:42:14.30297, ballot number:30                                |
 | 2024-08-10 13:42:14.65812  | xx.xx.xxx.92  |     2882 | vote                       | INFO  | vote for "xx.xx.xxx.92:2882", reason:the only request                                                   |
 | 2024-08-10 13:42:17.566071 | xx.xx.xxx.81  |     2882 | lease expired              | INFO  | owner:"xx.xx.xxx.81:2882", expired time:13:42:17.531586, ballot number:30                               |
 | 2024-08-10 13:42:17.566165 | xx.xx.xxx.81  |     2882 | vote                       | INFO  | vote for "xx.xx.xxx.81:2882", reason:the only request                                                   |
 | 2024-08-10 13:42:25.52073  | xx.xx.xxx.92  |     2882 | vote                       | INFO  | vote for "xx.xx.xxx.81:2882", reason:ZONE PRIORITY                                                      |
 | 2024-08-10 13:42:25.56706  | xx.xx.xxx.81  |     2882 | vote                       | INFO  | vote for "xx.xx.xxx.81:2882", reason:ZONE PRIORITY                                                      |
 | 2024-08-10 13:42:25.57757  | xx.xx.xxx.81  |     2882 | decentralized to be leader | INFO  | server|lease:["xx.xx.xxx.81:2882"|2024-08-10 13:42:29.570,"xx.xx.xxx.92:2882"|2024-08-10 13:42:29.570]  |
 | 2024-08-10 13:46:17.536782 | xx.xx.xxx.81  |     2882 | lease expired to revoke    | INFO  | leader lease expired, server|lease:[13:46:21.338|13:46:17.532|13:46:17.532]                             |
 | 2024-08-10 13:46:17.550585 | xx.xx.xxx.104 |     2882 | lease expired              | INFO  | owner:"xx.xx.xxx.81:2882", expired time:13:46:17.532494, ballot number:32                               |
 | 2024-08-10 13:46:17.550846 | xx.xx.xxx.104 |     2882 | vote                       | INFO  | vote for "xx.xx.xxx.92:2882", reason:IP-PORT(priority equal)                                            |
 | 2024-08-10 13:46:17.558890 | xx.xx.xxx.92  |     2882 | lease expired              | INFO  | owner:"xx.xx.xxx.81:2882", expired time:13:46:17.532476, ballot number:32                               |
 | 2024-08-10 13:46:17.558988 | xx.xx.xxx.92  |     2882 | vote                       | INFO  | vote for "xx.xx.xxx.92:2882", reason:IP-PORT(priority equal)                                            |
 | 2024-08-10 13:46:17.559655 | xx.xx.xxx.92  |     2882 | decentralized to be leader | INFO  | server|lease:["xx.xx.xxx.92:2882"|2024-08-10 13:46:21.559,"xx.xx.xxx.104:2882"|2024-08-10 13:46:21.559] |
 | 2024-08-10 13:46:21.57073  | xx.xx.xxx.81  |     2882 | lease expired              | INFO  | owner:"xx.xx.xxx.81:2882", expired time:13:46:21.34094, ballot number:32                                |
 | 2024-08-10 13:46:21.57206  | xx.xx.xxx.81  |     2882 | vote                       | INFO  | vote for "xx.xx.xxx.81:2882", reason:the only request                                                   |
 | 2024-08-10 13:46:22.701886 | xx.xx.xxx.92  |     2882 | prepare change leader      | INFO  | leader prepare to change leader : "xx.xx.xxx.92:2882" -> "xx.xx.xxx.81:2882", reason : ZONE PRIORITY    |
 | 2024-08-10 13:46:22.702944 | xx.xx.xxx.92  |     2882 | change leader to revoke    | INFO  | old leader revoke : "xx.xx.xxx.92:2882" -> "xx.xx.xxx.81:2882"                                          |
 | 2024-08-10 13:46:22.704108 | xx.xx.xxx.81  |     2882 | change leader to takeover  | INFO  | new leader takeover : "xx.xx.xxx.92:2882" -> "xx.xx.xxx.81:2882"                                        |
 | 2024-08-10 13:46:22.801924 | xx.xx.xxx.92  |     2882 | witness change leader      | INFO  | witness change leader : "xx.xx.xxx.92:2882" -> "xx.xx.xxx.81:2882"                                      |
 | 2024-08-10 13:46:22.808154 | xx.xx.xxx.81  |     2882 | witness change leader      | INFO  | witness change leader : "xx.xx.xxx.92:2882" -> "xx.xx.xxx.81:2882"                                      |
 | 2024-08-10 13:46:23.726573 | xx.xx.xxx.104 |     2882 | lease expired              | INFO  | owner:"xx.xx.xxx.92:2882", expired time:13:46:23.702679, ballot number:33                               |
 | 2024-08-10 13:46:23.726800 | xx.xx.xxx.104 |     2882 | vote                       | INFO  | vote for "xx.xx.xxx.104:2882", reason:the only request                                                  |
 | 2024-08-10 13:47:18.34703  | xx.xx.xxx.81  |     2882 | lease expired to revoke    | INFO  | leader lease expired, server|lease:[13:47:21.531|13:47:18.305|13:47:18.305]                             |
 | 2024-08-10 13:47:18.55303  | xx.xx.xxx.92  |     2882 | lease expired              | INFO  | owner:"xx.xx.xxx.81:2882", expired time:13:47:18.30714, ballot number:34                                |
 | 2024-08-10 13:47:18.55444  | xx.xx.xxx.92  |     2882 | vote                       | INFO  | vote for "xx.xx.xxx.92:2882", reason:the only request                                                   |
 | 2024-08-10 13:47:18.59981  | xx.xx.xxx.104 |     2882 | lease expired              | INFO  | owner:"xx.xx.xxx.81:2882", expired time:13:47:18.30752, ballot number:34                                |
 | 2024-08-10 13:47:18.60147  | xx.xx.xxx.104 |     2882 | vote                       | INFO  | vote for "xx.xx.xxx.104:2882", reason:the only request                                                  |
 | 2024-08-10 13:47:21.554849 | xx.xx.xxx.81  |     2882 | lease expired              | INFO  | owner:"xx.xx.xxx.81:2882", expired time:13:47:21.531522, ballot number:34                               |
 | 2024-08-10 13:47:21.554941 | xx.xx.xxx.81  |     2882 | vote                       | INFO  | vote for "xx.xx.xxx.81:2882", reason:the only request                                                   |
 | 2024-08-10 13:47:29.50801  | xx.xx.xxx.92  |     2882 | vote                       | INFO  | vote for "xx.xx.xxx.92:2882", reason:the only request                                                   |
 | 2024-08-10 13:47:29.54189  | xx.xx.xxx.104 |     2882 | vote                       | INFO  | vote for "xx.xx.xxx.104:2882", reason:the only request                                                  |
 | 2024-08-10 13:47:32.557405 | xx.xx.xxx.81  |     2882 | vote                       | INFO  | vote for "xx.xx.xxx.81:2882", reason:the only request                                                   |
 | 2024-08-10 13:47:39.53716  | xx.xx.xxx.104 |     2882 | vote                       | INFO  | vote for "xx.xx.xxx.81:2882", reason:LOG TS                                                             |
 | 2024-08-10 13:47:39.54514  | xx.xx.xxx.81  |     2882 | vote                       | INFO  | vote for "xx.xx.xxx.81:2882", reason:LOG TS                                                             |
 | 2024-08-10 13:47:39.55319  | xx.xx.xxx.81  |     2882 | decentralized to be leader | INFO  | server|lease:["xx.xx.xxx.81:2882"|2024-08-10 13:47:43.547,"xx.xx.xxx.92:2882"|2024-08-10 13:47:43.547]  |
 +----------------------------+---------------+----------+----------------------------+-------+---------------------------------------------------------------------------------------------------------+
 62 rows in set (0.103 sec)

触发条件

CDC 使用数据字典模式获取集群元信息（CDC 配置项）。
租户在生成数据字典时有过租户一号日志流的切主操作。
且上述字典的位置信息被汇报到了内部表中。

综合来看，可以从报错的 CDC 的启动位点向前找最新一个基线字典（SELECT SNAPSHOT_SCN, REPORT_TIME FROM oceanbase.CDB_OB_DATA_DICTIONARY_IN_LOG WHERE TENANT_ID = xxx WHERE SNAPSHOT_SCN <= ${CDC 启动位点} ORDER BY SNAPSHOT_SCN DESC LIMIT 1）。

如果查询到的 REPORT_TIME 附近刚好在做租户一号日志流的切主，则可能（但不一定）触发该问题。这里不一定触发的原因如下：

可能不完整的字典本身没有被汇报到内部表（因可能及时感知到了日志流切主而没有汇报），但切主后的新主在切主完成后马上完成了一轮数据字典的生成和汇报（即字典中的记录是新主产生的，此时字典中内容完整，不影响 CDC 消费）。
有问题的字典位置信息汇报到内部表条件比较苛刻：
- 字典每 1.875M 数据提交日志到 clog，字典刚好提交了此次 clog 日志，新的日志 BUF 还没填充前，感知到了切主操作从而停止生成字典。
- 或者高可用框架通知字典服务停止生成字典后，到 CDC 感知到 CDC 准备汇报字典位置信息之前，该租户的1号日志流再一次切主并成为了新主（即当前租户的 1 号日志流段时间内实施了至少两次切主（Leader -> Follower -> Leader））。

问题原因

OceanBase 数据库切主时可能导致正在生成中的数据字典生成过程中断（符合预期），但随后把这个不完整的数据字典的位置信息汇报到了内部表（__all_data_dictionary_in_log）中。CDC 消费到这个不完整的数据字典，在后续处理到这个不完整字典中没有记录的表的数据时，就会在 get_table_info_of_tablet_ 时报错 OB_ENTRY_NOT_EXIST(-4018)。

注意

这个问题容易和 get_table_info_of_tablet 报错 ENTRY_NOT_EXIST（CDC 报错进程退出 errno=-4018, errmsg=DML parser thread exits）混淆，两者进程退出的直接报错相同，排查过程中 SQL 结果不同，原因不同，应急方案不同，最终解决方案都需要升级 OceanBase 数据库集群；如何分辨两个问题：报错日志中有 tablet_id，从 OceanBase 数据库系统租户查询 select * from oceanbase.__all_virtual_tablet_to_table_history where tablet_id = ${ CDC 报错中的tablet_id}，如果看到了 is_deleleted=1 的记录，是之前的问题，如果看不到 is_deleted=1 的记录，可能是遇到当前文档的问题。

问题的风险及影响

OMS 数据同步链路中断，链路延迟。

适用版本

OceanBase 数据库 V4.1 及之后版本。

解决方法

解决方法： 升级至问题已修复版本。目前已修复的版本包括 OceanBase 数据库 V4.2.1 BP9（oceanbase-4.2.1.9-109000092024091919）版本。
应急方式：
1. 改用 online schema 模式同步数据。
```
meta_data_refresh_mode=online
skip_ob_version_compat_check=1
```
  OMS Store 白屏更新参数时需要给每个参数加 liboblog. 的前缀。
2. 向前回拨位点直到启动位点早于这个异常的字典的 SNAPSHOT_SCN。

规避方式

暂无。