适用版本
OceanBase 数据库 V2.x、V3.x 版本。
问题现象
执行批量业务处理时，以下监控语句发现 clog 同步延迟超过5秒、副本数不足 3 的问题。
obclient> SELECT /*+ READ_CONSISTENCY(WEAK) QUERY_TIMEOUT(10000000)*/
svr_ip,table_id,round(max(next_replay_ts_delta) / 1000) as max_clog_delay_ms
from oceanbase.__all_virtual_clog_stat
group by svr_ip,table_id
having max_clog_delay_ms > 5000
order by max_clog_delay_ms;
+---------------+------------------+-------------------+
| svr_ip | table_id | max_clog_delay_ms |
+---------------+------------------+-------------------+
| xxx.xx.xxx.93 | 1102810162727990 | 13122 |
| xxx.xx.xxx.90 | 1102810162727990 | 13123 |
| xxx.xx.xxx.91 | 1102810162727990 | 13126 |
| xxx.xx.xxx.94 | 1102810162727990 | 13126 |
| xxx.xx.xxx.92 | 1102810162727990 | 13157 |
| xxx.xx.xxx.95 | 1102810162727990 | 13159 |
+---------------+------------------+-------------------+
obclient> SELECT /*+ READ_CONSISTENCY(WEAK) QUERY_TIMEOUT(10000000)*/
table_id, part_id, count(1) as replicas, member_list
from oceanbase.__all_virtual_meta_table
group by table_id, partition_id, member_list
having cx < 3;
+------------------+---------+----------+--------------------------------------------------------------------------------------------------------------+
| table_id | part_id | replicas | member_list |
+------------------+---------+----------+--------------------------------------------------------------------------------------------------------------+
| 1102810162727990 | 0 | 2 | xxx.xx.xxx.90:2882:1678485637021113,xxx.xx.xxx.93:2882:1678485637021113,xxx.xx.xxx.134:2882:1678485637021113 |
| 1102810162727990 | 1 | 2 | xxx.xx.xxx.91:2882:1678485637021185,xxx.xx.xxx.94:2882:1678485637021185,xxx.xx.xxx.134:2882:1678485637021185 |
| 1102810162727990 | 2 | 2 | xxx.xx.xxx.92:2882:1678485637021242,xxx.xx.xxx.95:2882:1678485637021242,xxx.xx.xxx.134:2882:1678485637021242 |
| 1102810162727990 | 3 | 2 | xxx.xx.xxx.90:2882:1678485637021411,xxx.xx.xxx.93:2882:1678485637021411,xxx.xx.xxx.134:2882:1678485637021411 |
| 1102810162727990 | 4 | 2 | xxx.xx.xxx.91:2882:1678485637021450,xxx.xx.xxx.94:2882:1678485637021450,xxx.xx.xxx.134:2882:1678485637021450 |
| 1102810162727990 | 5 | 2 | xxx.xx.xxx.92:2882:1678485637021491,xxx.xx.xxx.95:2882:1678485637021491,xxx.xx.xxx.134:2882:1678485637021491 |
| 1102810162727990 | 6 | 2 | xxx.xx.xxx.90:2882:1678485637021543,xxx.xx.xxx.93:2882:1678485637021543,xxx.xx.xxx.134:2882:1678485637021543 |
问题原因
业务流程中有 Truncate Table 的操作，Truncate 是一个 Drop + Create 的过程，发生告警的 table_id 是 Drop 的旧 table_id。
obclient [oceanbase]> select * from __all_virtual_ddl_operation order by gmt_create desc;
+-----------+------------------+----------------------------+----------------------------+---------+------------------+---------------+------------------+------------------+------------+----------------+-------------------------+----------------+
| tenant_id | schema_version | gmt_create | gmt_modified | user_id | database_id | database_name | tablegroup_id | table_id | table_name | operation_type | ddl_stmt_str | exec_tenant_id |
+-----------+------------------+----------------------------+----------------------------+---------+------------------+---------------+------------------+------------------+------------+----------------+-------------------------+----------------+
| 1003 | 1678695584746168 | 2023-03-13 16:19:44.746217 | 2023-03-13 16:19:44.746217 | 0 | 0 | | 0 | 0 | | 1503 | | 1003 |
| 1003 | 1678695584738248 | 2023-03-13 16:19:44.746078 | 2023-03-13 16:19:44.746078 | 0 | 1102810162660380 | | 1103359918474218 | 1102810162727990 | | 2 | | 1003 |
| 1003 | 1678695584624568 | 2023-03-13 16:19:44.624615 | 2023-03-13 16:19:44.624615 | 0 | 0 | | 0 | 0 | | 1503 | | 1003 |
| 1003 | 1678695584622040 | 2023-03-13 16:19:44.624498 | 2023-03-13 16:19:44.624498 | 0 | 0 | | 0 | 1102810162727990 | | 14 | | 1003 |
| 1003 | 1678695584619920 | 2023-03-13 16:19:44.621930 | 2023-03-13 16:19:44.621930 | 0 | 1102810162660380 | | -1 | 1102810162727991 | | 2 | | 1003 |
| 1003 | 1678695014170832 | 2023-03-13 16:10:14.170895 | 2023-03-13 16:10:14.170895 | 0 | 0 | | 0 | 0 | | 1503 | | 1003 |
| 1003 | 1678695013782280 | 2023-03-13 16:10:13.784487 | 2023-03-13 16:10:13.784487 | 0 | 0 | | 0 | 1102810162728085 | | 14 | | 1003 |
| 1003 | 1678695013777840 | 2023-03-13 16:10:13.782173 | 2023-03-13 16:10:13.782173 | 0 | 1102810162660380 | | -1 | 1102810162728086 | | 20 | | 1003 |
| 1003 | 1678695013753920 | 2023-03-13 16:10:13.777702 | 2023-03-13 16:10:13.777702 | 0 | 1102810162660380 | | 1103359918474218 | 1102810162728085 | | 23 | truncate table xxx | 1003 |
| 1003 | 1678695013749736 | 2023-03-13 16:10:13.753836 | 2023-03-13 16:10:13.753836 | 0 | 1102810162660380 | | 1103359918474218 | 1102810162727990 | | 48 | truncate table xxx | 1003 |
关于系统表 __all_virtual_ddl_operation 中 operation_type 字段的含义，参见 DDL 操作记录表中 operation_type 描述。本示例中，相关的代码含义如下。
- 2：OB_DDL_DROP_TABLE
- 14：OB_DDL_MODIFY_TABLE_SCHEMA_VERSION
- 20：OB_DDL_CREATE_INDEX
- 23：OB_DDL_TRUNCATE_TABLE_CREATE
- 48：OB_DDL_DELAY_DELETE_TABLE
- 1503：OB_DDL_END_SIGN
在开启归档或者接入备库的 OceanBase 集群，OceanBase 数据库都会开启 schema 延迟删除，通过后台 GC 任务异步执行，一般在合并后 30分钟之内会处理完成。在 GC 完成前，被 Drop 的表、分区出现 clog 延迟、缺失副本的现象是符合预期的，并不会对业务本身产生任何影响。
解决方法
对业务没有影响，可以忽略此类告警。
