OceanBase 数据库集群修改 locality 后任务卡住，__all_rootservice_job 显示任务一直在 inprogress

问题现象

在 OceanBase 数据库集群执行添加添加副本的任务后，任务无法结束，具体表现如下。

1. __all_rootservice_job 显示任务一直在 inprogress。

   gmt_create    gmt_modified    job_id    job_type    job_status    return_code    progress    tenant_id    tenant_name    database_id    database_name    table_id    table_name    partition_id    svr_ip    svr_port    unit_id    rs_svr_ip    rs_svr_port    sql_text    extra_info    resource_pool_id    tablegroup_id    tablegroup_name
   2024-06-04 11:30:22.170631    2024-06-04 11:30:22.170631    77    ALTER_TENANT_LOCALITY    INPROGRESS    null    0    1031    zh_test_fxq    null    null    null    null    null    null    null    null    16.55.0.27    2882    ALTER TENANT `zh_test_fxq` LOCALITY = 'FULL{1}@zone1, FULL{1}@zone2,FULL@zone3'    FULL{1}@zone1, FULL{1}@zone2    null    null    null
   2024-06-03 15:56:01.996227    2024-06-03 15:56:01.996227    76    ROLLBACK_ALTER_TENANT_LOCALITY    INPROGRESS    null    0    1031    zh_test_fxq    null    null    null    null    null    null    null    null    16.55.0.27    2882    alter tenant zh_test_fxq set locality='FULL{1}@zone1, FULL{1}@zone2' force    FULL{1}@zone1, FULL{1}@zone2    null    null    null
   2024-05-31 21:46:21.766573    2024-05-31 21:49:07.917395    75    ALTER_TENANT_LOCALITY    SUCCESS    0    100    1030    zh_poc_htgl    null    null    null    null    null    null    null    null    16.55.0.27    2882    ALTER TENANT `zh_poc_htgl` LOCALITY = 'FULL{1}@zone1, FULL{1}@zone2,FULL@zone3'    FULL{1}@zone1, FULL{1}@zone2    null    null    null
   

2. __all_virtual_replica_task 内有补副本的任务。

   tenant_id    table_id    partition_id    src_ip    src_port    src_replica_type    zone    region    dst_ip    dst_port    dst_replica_type    cmd_type    comment
   1031    1133596488289648    0    0.0.0.0    0    262    zone3    \E    0.0.0.0    0    0    ADD_REPLICA    replicate enough replica
   1031    1133596488289649    0    0.0.0.0    0    262    zone3    \E    0.0.0.0    0    0    ADD_REPLICA    replicate enough replica
   1031    1133596488289650    0    0.0.0.0    0    262    zone3    \E    0.0.0.0    0    0    ADD_REPLICA    replicate enough replica
   1031    1133596488289652    0    0.0.0.0    0    262    zone3    \E    0.0.0.0    0    0    ADD_REPLICA    replicate enough replica
   1031    1133596488289653    0    0.0.0.0    0    262    zone3    \E    0.0.0.0    0    0    ADD_REPLICA    replicate enough replica
   1031    1133596488289654    0    0.0.0.0    0    262    zone3    \E    0.0.0.0    0    0    ADD_REPLICA    replicate enough replica
   1031    1133596488289925    0    0.0.0.0    0    262    zone3    \E    0.0.0.0    0    0    ADD_REPLICA    replicate enough replica
   1031    1133596488290181    0    0.0.0.0    0    262    zone3    \E    0.0.0.0    0    0    ADD_REPLICA    replicate enough replica
   1031    1133596488290182    0    0.0.0.0    0    262    zone3    \E    0.0.0.0    0    0    ADD_REPLICA    replicate enough replica
   1031    1133596488290183    0    0.0.0.0    0    262    zone3    \E    0.0.0.0    0    0    ADD_REPLICA    replicate enough replica
   1031    1133596488290184    0    0.0.0.0    0    262    zone3    \E    0.0.0.0    0    0    ADD_REPLICA    replicate enough replica
   1031    1133596488290764    0    0.0.0.0    0    262    zone3    \E    0.0.0.0    0    0    ADD_REPLICA    replicate enough replica
   

3. rootservice 日志出现如下异常信息。

   [2024-06-06 15:00:35.629593] WARN  [RS] replicate_enough_paxos_replica (ob_rereplication.cpp:704) [13014][0][xxxxx-xxxxx-xxxxx-xxxxx] [lt=9] [dc=0] partition can't do replicate(partition={table_id:{value:1133596488289648, first:1031, second:52592}, tablegroup_id:{value:18446744073709551615, first:16777215, second:1099511627775}, partition_id:0, partition_idx:0, schema_partition_cnt:1, schema_replica_cnt:3, schema_full_replica_cnt:3, valid_member_cnt:2, begin:2862, end:2864, can_rereplicate:false, primary:false, quorum:2, has_leader:true, can_balance:true, filter_logonly_count:0, all_pg_idx:806, has_flag_replica:false})
   

关键信息

关键日志信息如下。

[2024-06-06 15:00:35.629593] WARN  [RS] replicate_enough_paxos_replica (ob_rereplication.cpp:704) [13014][0][xxxxx-xxxxx-xxxxx-xxxxx] [lt=9] [dc=0] partition can't do replicate(partition={table_id:{value:1133596488289648, first:1031, second:52592}, tablegroup_id:{value:18446744073709551615, first:16777215, second:1099511627775}, partition_id:0, partition_idx:0, schema_partition_cnt:1, schema_replica_cnt:3, schema_full_replica_cnt:3, valid_member_cnt:2, begin:2862, end:2864, can_rereplicate:false, primary:false, quorum:2, has_leader:true, can_balance:true, filter_logonly_count:0, all_pg_idx:806, has_flag_replica:false})

问题原因

OceanBase 数据库集群内部在进行 locality 变更操作时，会借助内部组件进行副本的创建，副本创建过程卡住无法完成时，会导致 locality 任务无法结束，在本案例中，副本创建异常主要是因为如下两点导致：

1. 这些副本都是索引。

2. 这些索引的状态不正常。

问题的风险及影响

影响 locality 的变更操作。

影响租户

影响 OceanBase 数据库中的 SYS 租户和 Oracle 租户以及 MySQL 租户。

适用版本

OceanBase 数据库 V2.x、V3.x 版本。

解决方法及规避方式

将异常的索引删除，重新创建即可。