低版本变换副本类型带来的一些问题

问题描述

1. add_member 新增副本时报 -4109, 告警日志信息如下。

   WARN  [CLOG] pre_change_member (ob_partition_log_service.cpp:6197) [91114][1468][xxxxx-xxxxx] [lt=6] [dc=0] can not change member(ret=-4109, partition_key_={tid:xxxxx, partition_id:0, part_cnt:0}, role=2, state=4, is_changing_leader=false)
   WARN  [CLOG] pre_batch_change_member_(ob_clog_mgr.cpp:1242) [91114][1468][xxxxx-xxxxx] [lt=8] [dc=0] pre_change_member failed(ret=-4109, partition_key={tid:xxxxx, partition_id:0, part_cnt:0}, quorum=4, is_add_member=true, mc_timestamp=-1, member_list=0, proposal_id={time_to_usec:-1, server:"x.x.x.x"})
   WARN  [CLOG] pre_batch_change_member_ (ob_clog_mgr.cpp:1248) [91114][1468][xxxxx-xxxxx] [lt=6] [dc=0] have partition failed(partition_key={tid:xxxxx, partition_id:0, part_cnt:0}, ret=-4109)
   WARN  [CLOG] batch_change_member_(ob_clog_mgr.cpp:1162) [91114][1468][xxxxx-xxxxx] [lt=3] [dc=0] pre_batch_change_member_ failed(ret=-4109)
   INFO  [CLOG] ob_clog_mgr.cpp:1048 [91114][1468][xxxxx-xxxxx] [lt=3] [dc=0] batch_add_member finished(ret=-4109, partition_array=[{tid:xxxxxx, partition_id:0, part_cnt:0}, {tid:xxxxx, partition_id:0, part_cnt:0}, {tid:xxxxx, partition_id:0, part_cnt:0}, {tid:xxxxx, partition_id:0, part_cnt:0}, {tid:xxxxx, partition_id:0, part_cnt:0}, {tid:xxxxx, partition_id:0, part_cnt:0}, {tid:xxxxx, partition_id:0, part_cnt:0}, {tid:xxxxx, partition_id:0, part_cnt:0}, {tid:xxxxx, partition_id:0, part_cnt:0}, {tid:xxxxx, partition_id:0, part_cnt:0}, {tid:xxxxx, partition_id:0, part_cnt:0}, {tid:1120402348753764, partition_id:0, part_cnt:0}, {tid:xxxxx, partition_id:0, part_cnt:0}, {tid:xxxxx, partition_id:0, part_cnt:0}, {tid:1120402348753767, partition_id:0, part_cnt:0}, {tid:xxxxx, partition_id:0, part_cnt:0}, {tid:xxxxx, partition_id:0, part_cnt:0}], member_array=[{server:"xx.xx.xx.xx:xxxx", timestamp:xxxxx, flag:0, need_encrypt:false}, {server:"xx.xx.xx.xx:xxxx", timestamp:xxxxx, flag:0, need_encrypt:false}, {server:"1xx.xx.xx.xx:xxxx", timestamp:xxxxx, flag:0, need_encrypt:false}, {server:"1xx.xx.xx.xx:xxxx", timestamp:xxxxx, flag:0, need_encrypt:false}, {server:"xx.xx.xx.xx:xxxx", timestamp:xxxxx, flag:0, need_encrypt:false}, {server:"xx.xx.xx.xx:xxxx", timestamp:xxxxx, flag:0, need_encrypt:false}, {server:"xx.xx.xx.xx:xxxx", timestamp:xxxxx, flag:0, need_encrypt:false}, {server:"xx.xx.xx.xx:xxxx", timestamp:xxxxx, flag:0, need_encrypt:false}, {server:"xx.xx.xx.xx:xxxx", timestamp:xxxxx, flag:0, need_encrypt:false}, {server:"xx.xx.xx.xx:xxxx", timestamp:xxxxx, flag:0, need_encrypt:false}, {server:"xx.xx.xx.xx:xxxx", timestamp:xxxxx, flag:0, need_encrypt:false}, {server:"xx.xx.xx.xx:xxxx", timestamp:xxxxx, flag:0, need_encrypt:false}, {server:"xx.xx.xx.xx:xxxx", timestamp:xxxxx, flag:0, need_encrypt:false}, {server:"xx.xx.xx.xx:xxxx", timestamp:xxxxx, flag:0, need_encrypt:false}, {server:"xx.xx.xx.xx:xxxx", timestamp:xxxxx, flag:0, need_encrypt:false}, {server:"xx.xx.xx.xx:xxxx", timestamp:xxxxx, flag:0, need_encrypt:false}, {server:"xx.xx.xx.xx:xxxx", timestamp:xxxxx, flag:0, need_encrypt:false}], quorum_array=[x,x,x,x,x,x,x,x,x,x], is_add_member=true, ret_array=[], log_info_array=[])
   WARN  [STORAGE] process (ob_partition_service_rpc.cpp:1599) [91114][1468][xxxxx-xxxxx] [lt=9] [dc=0] failed to batch_add_member(result.result_code_=-4109, arg=[{partition_key:{tid:xxxxx, partition_id:0, part_cnt:0}, member:
   

2. 任务执行时 pg_storage 报错 -5081，原因是 replica_type 不是 FULL, 无法写入行。

   INFO [SERVER] obmp_disconnect.ccp:79 [316327][1894][xxxxx-xxxxx] [lt=12] [dc=0] free session successfully(sessid=xxxxx version=0)
   ERROR [STORAGE] update_rows (ob_pg_storage.ccp:3727) [920403][10783][xxxxx-xxxxx] [lt=4][dc=0] replica is not writable(ret=-5081,this={this:0x7ecc96994a40, pkey:{tid:1120402348703970, partition_id=0, part_cnt:0}, meta:0x7f14aeaad8e0,log_seq_num:0,last_freeze_ts:1667239229382541}) BACKTRACE:0x9e781da 0x9ca65e0 0x9ca1c3f
   

适用版本

OceanBase 数据库 V2.x 和 V3.x 版本。

问题原因

OceanBase 数据库 V3.1 版本的已知 bug，副本类型转换时 pg_storage 里的 cached_replica_type_ 不会变。

解决方法

• 升级至问题已修复版本，目前已修复的版本 OceanBase 数据库 V3.1.2 BP11 (oceanbase-3.1.2-111000052023010412) 及之后版本。

• 临时方法：依次重启 OBServer。

  有关重启 OBServer 的详细操作，参见 重启 OBServer。

  说明

  1. svr_ip 表示待停止的节点所在的 IP。
  2. svr_port 表示待停止的节点的 RPC 端口，默认为 2882。