适用版本
OceanBase 数据库 V2.x 和 V3.x 版本。
问题现象
具体报错日志为 -4019,deliver request failed。
[2023-02-13 18:34:20.842080] EDIAG [SERVER] deliver_mysql_request (ob_srv_deliver.cpp:425) [2151][T1002_LockWaitM][T1002][Y0-0000000000000000-0-0] [lt=13][errcode=-4019] deliver request fail(req={packet:{header:{length:101, sequence:0}, can_reroute:false, weak_read:false}, type:1, group:0, sql_req_level:0, connection_phase:2, recv_timestamp_:1676284417982783, enqueue_timestamp_:1676284460842052, request_arrival_time_:0, trace_id_:YAC910BA2DABA-0005F48B80570880-0-0}, *tenant={id:1002, tenant_meta:{unit:{tenant_id:1002, unit_id:1003, has_memstore:true, unit_status:"NORMAL", config:{unit_config_id:1001, name:"mysql_unit", resource:{min_cpu:3, max_cpu:3, memory_size:"21.6GB", log_disk_size:"64.8GB", min_iops:30000, max_iops:30000, iops_weight:0}}, mode:0, create_timestamp:1676254545424591, is_removed:false}, super_block:{tenant_id:1002, replay_start_point:ObLogCursor{file_id=151, log_id=4242545, offset=56499779}, ls_meta_entry:[200185](ver=0,mode=0,seq=4260536), tablet_meta_entry:[172555](ver=0,mode=0,seq=4261073), is_hidden:false}, create_status:1}, unit_min_cpu:"3.000000000000000000e+00", unit_max_cpu:"3.000000000000000000e+00", slice:"0.000000000000000000e+00", slice_remain:"0.000000000000000000e+00", token_cnt:12, sug_token_cnt:12, ass_token_cnt:12, lq_tokens:3, used_lq_tokens:0, stopped:false, idle_us:13033113, recv_hp_rpc_cnt:1963162, recv_np_rpc_cnt:2597171, recv_lp_rpc_cnt:0, recv_mysql_cnt:15505415, recv_task_cnt:3571, recv_large_req_cnt:9747, tt_large_quries:513045, pop_normal_cnt:36008060, actives:12, workers:12, nesting workers:8, lq waiting workers:0, req_queue:total_size=8193 queue[0]=0 queue[1]=0 queue[2]=7958 queue[3]=0 queue[4]=235 queue[5]=0 , large queued:0, multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0 , recv_level_rpc_cnt:cnt[0]=0 cnt[1]=0 cnt[2]=8120 cnt[3]=0 cnt[4]=0 cnt[5]=69513 cnt[6]=16 cnt[7]=0 cnt[8]=0 cnt[9]=0 , group_map:group_id = 2,queue_size = 0,recv_req_cnt = 721376,pop_req_cnt = 721376,token_cnt = 3,min_token_cnt = 3,max_token_cnt = 3,ass_token_cnt = 3 group_id = 5,queue_size = 0,recv_req_cnt = 153203,pop_req_cnt = 153203,token_cnt = 3,min_token_cnt = 3,max_token_cnt = 3,ass_token_cnt = 3 group_id = 1,queue_size = 0,recv_req_cnt = 30972566,pop_req_cnt = 30972566,token_cnt = 3,min_token_cnt = 3,max_token_cnt = 3,ass_token_cnt = 3 , rpc_stat_info: pcode=0x1501:cnt=273925 pcode=0x1516:cnt=163317 pcode=0x1502:cnt=51990 pcode=0x9f0:cnt=24151 pcode=0x750:cnt=10857}) BACKTRACE:0xe13382b 0xdda5916 0x4b2a155 0x4b29e0a 0x4b29c0a 0x4b0a69b 0x4e7da25 0x48be7b6 0x48be1de 0xb681fba 0x497540e 0xe4a7a94 0xe4ab837 0xe4a4a2a 0x7f52f643be25 0x7f52f5efaf1d
问题原因
租户队列已经用满,需要释放。
req_queue:total_size=8193 queue[0]=0 queue[1]=0 queue[2]=7958 queue[3]=0 queue[4]=235 queue[5]=0 , large queued:0, multi_level_queue:total_size=0 queue[0]=0 queue[1]=0 queue[2]=0 queue[3]=0 queue[4]=0 queue[5]=0 queue[6]=0 queue[7]=0 queue[8]=0 queue[9]=0
解决方法
重启这台 OBServer。
将这个对应租户的资源进行扩容(具体到 CPU 和内存)。