刷新 location cache 的应急方法

问题描述

OceanBase 数据库集群存在 location cache 刷新异常, 日志信息如下:

WARN [SHARE.PT] init_batch_rpc_renew_struct (ob_partition_location_cache.cpp:1488) [xxxxx][xxxxx][xxxxx-xxxxx] [lt=3] [dc=0] fail to check is local server(ret=-4018, addr="xx.xx.xx.xx:xxxx")

适用版本

OceanBase 数据库 V2.x 和 V3.x 版本。

问题原因

集群中的租户曾经通过 Locality 变更下掉 R 副本，而且后续删除了 R 副本所在的 OBServer/zone，在切主（运维触发或者故障触发）等场景下，可能会触发批量 location_cache 刷新的逻辑，但由于 OceanBase 产品 BUG，批量刷新 location_cache 时会误向已下线的R副本持续发 RPC，且在报错后不触发其他刷新方式，进而导致无法成功获取 location 信息。

解决方法

应急方法如下：

1. 直连集群所有 OBServer 2881 端口，执行刷新 location cache 命令。

   obclient> ALTER SYSTEM flush location cache;
   

2. 如果低版本 OBServer 不支持 flush location cache 命令，使用手动刷新方式。

   SELECT concat("SELECT * FROM  __all_virtual_proxy_schema WHERE tenant_name='", t.tenant_name,"' and database_name='", d.database_name,"' AND table_name='", tb.table_name,"' AND partition_id=", l.partition_id,";")  FROM __all_tenant as t JOIN __all_virtual_table as tb ON t.tenant_id = tb.tenant_id JOIN __all_virtual_database as d ON tb.tenant_id = d.tenant_id AND tb.database_id = d.database_id JOIN (SELECT tenant_id, table_id, partition_id FROM  __all_virtual_meta_table GROUP BY tenant_id, table_id, partition_id) as l ON t.tenant_id = l.tenant_id AND tb.table_id = l.table_id;
   

   SELECT concat("SELECT * FROM __all_virtual_proxy_schema WHERE tenant_name='", t.tenant_name,"' AND database_name='oceanbase' AND table_name='", tb.table_name,"' AND partition_id=", l.partition_id,";")  FROM  __all_tenant as t JOIN (SELECT table_id, table_name FROM __all_virtual_table WHERE tenant_id = 1 AND table_id & 0xffffffffff < 10000 UNION SELECT table_id, table_name FROM __all_virtual_core_all_table) as tb JOIN (SELECT tenant_id, table_id, partition_id FROM  __all_root_table GROUP BY tenant_id, table_id, partition_id) as l ON t.tenant_id = l.tenant_id AND (tb.table_id & 0xffffffffff = l.table_id & 0xffffffffff);
   

   执行上述两条语句生成待执行的 SQL 集合，在上述 SQL 集合上加上以下两条 SQL。

   SELECT * FROM __all_virtual_proxy_schema WHERE tenant_name = "sys" and database_name = "oceanbase" AND table_name = "__all_root_table"  AND partition_id = 0;
   

   SELECT * FROM __all_virtual_proxy_schema WHERE tenant_name = "sys" AND database_name = "oceanbase" AND table_name = "__all_core_table" and partition_id = 0;
   

   用系统租户登录，直连集群所有 OBServer 2881 端口执行上述 SQL 集合。

3. 重启 OBServer 整个集群。