虚拟表查询用 hint 指定索引会导致 core

问题现象

在 OBServer 问题版本中查询虚拟表时，如果使用 hint 指定索引，可能会导致 OBServer core。

-- 问题 SQL 举例
select /*+INDEX(@SEL$1 oceanbase.__all_virtual_plan_cache_stat all_virtual_plan_cache_stat_i1)*/ * from oceanbase.__all_virtual_plan_cache_stat where tenant_id =1001 and svr_ip='xx.xx.xx.xx' and svr_port=2882;

关键诊断信息

触发条件

查询虚拟表时使用 hint 指定索引。

事后诊断

1. 根据日志/gdb 确认导致 core 的 SQL 特征为：虚拟表查询、hint 指定索引。

2. core 堆栈顶层如下，且 create_access_path 帧中 index_info_entry = 0x1。

   oceanbase::sql::ObJoinOrder::create_normal_access_path
   oceanbase::sql::ObJoinOrder::create_access_path
   ceanbase::sql::ObJoinOrder::add_table_by_heuristics
   

   问题 core 堆栈举例如下。

    #0  0x00007f31480adfcb in raise () from /lib64/libpthread.so.0
    #1  0x000000000d3c9df9 in oceanbase::common::coredump_cb (sig=11, si=0x7f2e47268bf0) at ./deps/oblib/src/lib/signal/ob_signal_handlers.cpp:140
    #2  <signal handler called>
    #3  0x000000000be8b70b in oceanbase::sql::ObJoinOrder::create_normal_access_path (this=this@entry=0x7ec7888b3270, table_id=1099511638807, ref_id=1099511638807, index_id=1099511638807, index_info_entry=...) at ./src/sql/optimizer/ob_join_order.cpp:1179
    #4  0x000000000be8bc77 in oceanbase::sql::ObJoinOrder::create_access_path (this=this@entry=0x7ec7888b3270, table_id=1099511638807, ref_id=1099511638807, index_id=1099511638807, index_info_cache=...) at ./src/sql/optimizer/ob_join_order.cpp:1130
    #5  0x000000000be8d058 in oceanbase::sql::ObJoinOrder::add_table_by_heuristics (this=this@entry=0x7ec7888b3270, table_id=1099511638807, ref_table_id=1099511638807, index_info_cache=..., valid_index_ids=..., added=@0x7f2e4634817a: false) at ./src/sql/optimizer/ob_join_order.cpp:772
    #6  0x000000000be8d4c3 in oceanbase::sql::ObJoinOrder::add_table (this=this@entry=0x7ec7888b3270, table_id=1099511638807, ref_table_id=1099511638807) at ./src/sql/optimizer/ob_join_order.cpp:2065
    #7  0x000000000be99a0f in oceanbase::sql::ObJoinOrder::generate_base_paths (this=0x7ec7888b3270) at ./src/sql/optimizer/ob_join_order.cpp:4618
    #8  0x000000000bdbe025 in oceanbase::sql::ObLogPlan::generate_join_order (this=this@entry=0x7ec7888aa680, table_items=..., semi_infos=..., quals=..., join_order=...) at ./src/sql/optimizer/ob_log_plan.cpp:1708
    #9  0x000000000bdbea32 in oceanbase::sql::ObLogPlan::generate_join_orders (this=this@entry=0x7ec7888aa680) at ./src/sql/optimizer/ob_log_plan.cpp:410
    #10 0x000000000bdbec85 in oceanbase::sql::ObLogPlan::generate_plan_tree (this=0x7ec7888aa680) at ./src/sql/optimizer/ob_log_plan.cpp:8124
    #11 0x000000000c1127ec in oceanbase::sql::ObSelectLogPlan::generate_plan_for_plain_select (this=this@entry=0x7ec7888aa680) at ./src/sql/optimizer/ob_select_log_plan.cpp:1164
    #12 0x000000000c112f48 in oceanbase::sql::ObSelectLogPlan::generate_raw_plan (this=0x7ec7888aa680) at ./src/sql/optimizer/ob_select_log_plan.cpp:1146
    #13 0x000000000c103c4b in oceanbase::sql::ObSelectLogPlan::generate_plan (this=0x7ec7888aa680) at ./src/sql/optimizer/ob_select_log_plan.cpp:1362
    #14 0x000000000c037be2 in oceanbase::sql::ObOptimizer::optimize (this=0x7f2e46349130, stmt=..., logical_plan=@0x7f2e46349118: 0x0) at ./src/sql/optimizer/ob_optimizer.cpp:36
    #15 0x000000000bb3a6a8 in oceanbase::sql::ObSql::optimize_stmt (optimizer=..., session_info=..., stmt=..., logical_plan=@0x7f2e46349118: 0x0) at ./src/sql/ob_sql.cpp:2322
    #16 0x00000000033a0cad in oceanbase::sql::ObSql::generate_physical_plan (this=this@entry=0xee47a40 <oceanbase::observer::ObServer::get_instance()::THE_ONE+1112064>, parse_result=..., pc_ctx=pc_ctx@entry=0x7ec7887dd7b0, sql_ctx=..., result=..., is_ps_mode=is_ps_mode@entry=false) at./src/sql/ob_sql.cpp:2011
    #17 0x00000000032d28c0 in oceanbase::sql::ObSql::handle_physical_plan (this=this@entry=0xee47a40 <oceanbase::observer::ObServer::get_instance()::THE_ONE+1112064>, trimed_stmt=..., context=..., result=..., pc_ctx=..., get_plan_err=-5138, is_psmode=false) at ./src/sql/ob_sql.cpp:392
    #18 0x000000000326fa40 in oceanbase::sql::ObSql::handle_text_query (this=0xee47a40 <oceanbase::observer::ObServer::get_instance()::THE_ONE+1112064>, stmt=..., context=..., result=...) at ./src/sql/ob_sql.cpp:1532
    #19 0x000000000326f160 in oceanbase::sql::ObSql::stmt_query (this=0xee47a40 <oceanbase::observer::ObServer::get_instance()::THE_ONE+1112064>, stmt=..., context=..., result=...) at ./src/sql/ob_sql.cpp:187
    #20 0x0000000003224a51 in oceanbase::observer::ObMPQuery::do_process (need_disconnect=@0x7f2e4634c042: true, async_resp_used=<synthetic pointer>: <optimized out>, force_sync_resp=false, has_more_result=false, session=..., this=0x7ed2893ed4d0) at ./src/observer/mysql/obmp_query.cpp:644
    #21 oceanbase::observer::ObMPQuery::process_single_stmt (need_disconnect=@0x7f2e4634c042: true, async_resp_used=<synthetic pointer>: <optimized out>, force_sync_resp=false, has_more_result=false, session=..., multi_stmt_item=..., this=0x7ed2893ed4d0) at ./src/observer/mysql/obmp_query.cpp:476
    #22 oceanbase::observer::ObMPQuery::process (this=0x7ed2893ed4d0) at ./src/observer/mysql/obmp_query.cpp:287
    #23 0x0000000003221dba in oceanbase::rpc::frame::ObReqProcessor::run (this=0x7ed2893ed4d0) at ./deps/oblib/src/rpc/frame/ob_req_processor.cpp:47
    #24 0x000000000321f792 in oceanbase::omt::ObWorkerProcessor::process_one (process_ret=<synthetic pointer>: <optimized out>, req=..., this=0xf5c0200 <oceanbase::observer::ObServer::get_instance()::THE_ONE+8945600>) at ./src/observer/omt/ob_worker_processor.cpp:63
    #25 oceanbase::omt::ObWorkerProcessor::process (this=0xf5c0200 <oceanbase::observer::ObServer::get_instance()::THE_ONE+8945600>, req=...) at ./src/observer/omt/ob_worker_processor.cpp:122
    #26 0x0000000003219fad in oceanbase::omt::ObThWorker::process_request (req=..., this=0x7f2e9877a040) at ./src/observer/omt/ob_th_worker.cpp:260
    #27 oceanbase::omt::ObThWorker::worker (this=0x7f2e9877a040, tenant_id=@0x7f2e4634dee0: 1, tenant_id@entry=@0x7f2e4634dee0: -1, req_recv_timestamp=@0x7f2e4634dee8: 1679647720254663, req_recv_timestamp@entry=@0x7f2e4634dee8: -1, worker_level=@0x7f2e4634dedc: 0, worker_level@entry=@
    0x7f2e4634dedc: -1) at ./src/observer/omt/ob_th_worker.cpp:413
    #28 0x000000000a64fb9b in oceanbase::omt::ObThWorker::run (this=<optimized out>, idx=<optimized out>) at ./src/observer/omt/ob_th_worker.cpp:449
    #29 0x0000000003ba0030 in oceanbase::lib::CoKThreadTemp<oceanbase::lib::CoUserThreadTemp<oceanbase::lib::CoSetSched> >::start()::{lambda()#1}::operator()() const (__closure=<optimized out>, __closure=<optimized out>) at ./deps/oblib/src/lib/coro/co_user_thread.h:274
    #30 std::_Function_handler<void (), oceanbase::lib::CoKThreadTemp<oceanbase::lib::CoUserThreadTemp<oceanbase::lib::CoSetSched> >::start()::{lambda()#1}>::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/local/gcc-5.2.0/include/c++/5.2.0/functional:1871
    #31 0x000000000d167797 in std::function<void ()>::operator()() const (this=0x7f2e97cd0560) at /usr/local/gcc-5.2.0/include/c++/5.2.0/functional:2271
    #32 oceanbase::lib::CoSetSched::Worker::run (this=0x7f2e97ccc2c0) at ./deps/oblib/src/lib/coro/co_set_sched.cpp:95
    #33 0x000000000d1654f0 in oceanbase::lib::CoRoutine::__start (from=...) at ./deps/oblib/src/lib/coro/co_routine.cpp:130
    #34 0x000000000d1632af in make_fcontext () at /home/admin/77_20211116202822728_237671098_code/rpm_workspace/deps/oblib/src/lib/coro/context/asm/make_x86_64_sysv_elf_gas.S:71
    #35 0x0000000000000000 in ?? ()
   

问题原因

使用 hint 指定索引，只会为指定的索引抽取 range 创建 index info 并存储在 cache 中，但是虚拟表采用“基于规则的索引选择策略”添加路径时，可能会重新选择主表拿 primary key，此时再去拿 index info 就会失败（指定索引 与 primary key 对应的 index info 不匹配）。在这种情况下，问题版本 OBServer 中 create_access_path 里对应的指针（entry）没有初始化为 NULL，导致后续 index info 相应指针（index_info_entry）判空防御检查没有拦住，进而导致 OBServer core 掉。

问题的风险及影响

影响集群可用性。

影响租户

影响 OceanBase 数据库中的 SYS 租户和 Oracle 租户以及 MySQL 租户。

影响的版本

OceanBase 数据库企业版 V2.2.77 GA（oceanbase-2.2.77-20210508211731）及之后版本、V3.1.2 GA（oceanbase-3.1.2-20210618150922）及之后版本、V3.2.3 GA（oceanbase-3.2.3.0-20220418212020）及之后版本。

解决方法

升级至问题已修复版本。目前已修复的版本包括 OceanBase 数据库企业版 V2.2.77 BP15 (oceanbase-2.2.77-115000012023010607)、V3.1.2 BP10 (oceanbase-3.1.2-110000022022092714)、V3.2.3.3 BP6 (oceanbase-3.2.3.3-106000102022111521) 及之后版本。

规避方式

查询虚拟表时，请勿使用 hint 指定索引。