首批通过分布式安全可靠测评,为关键业务系统打造
OBServer 单节点在大 in 场景,core 掉,爆栈
更新时间:2026-06-04 09:56
问题现象
导致 OBServer 单节点 core 的 SQL。
SELECT /*idb_check_count*/ COUNT(1) cnt FROM ( SELECT order_id,a.serial_number
from xxx_xxx_xxx a
where a.serial_number in('4xxxxxx','6xxxxxx','6xxxxxx','6xxxxxx','6xxxxxx','6xxxxxx','6xxxxxx','6xxxxxx','6xxxxxx','6xxxxxx','6xxxxxx','6xxxxxx','6xxxxxx','6xxxxxx','6xxxxxx','6xxxxxx',
'6xxxxxx','6xxxxxx','6xxxxxx', ...) );
上述 SQL 中 xxx_xxx_xxx 表的 serial_number 列类型为 text。 in 表达式中探测的列(即 serial_numer)为 text 类型,in 列表中元素为111333 个。
执行该 SQL,OBServer core 掉。
## log 中 crash 信息
CRASH ERROR!!! IP=ffffffffffffffff, RBP=ffffffffffffffff, sig=11, sig_code=2, sig_addr=fffc1ea05000, RLIMIT_CORE=unlimited, timestamp=1750333438467682, tid=1070, tname=T1002_L0_G0, trace_id
=12378274872200-1737206412398334-0-0, extra_info=((null)), lbt=, SQL=SELECT /*idb_check_count*/ COUNT(1) cnt FROM ( SELECT order_id,a.serial_number
from xxx_xxx_xxx a
where a.serial_number in('4xxxxxx','6xxxxxx','6xxxxxx','6xxxxxx','6xxxxxx','6xxxxxx','6xxxxxx','6xxxxxx','6xxxxxx','6xxxxxx','6xxxxxx','6xxxxxx','6xxxxxx','6xxxxxx','6xxxxxx','6xxxxxx',
'6xxxxxx','6xxxxxx','6xxxxxx','
具体 core 在函数 eval_batch_in_without_row_fallback。
#0 0x0000ffff9aea2aec in raise () from /lib64/libpthread.so.0
#1 0x0000aaaae047a838 in oceanbase::common::coredump_cb(int, siginfo_t*, void*) ()
#2 <signal handler called>
#3 0x0000aaaad8fcd224 in oceanbase::sql::ObExprInOrNotIn::eval_batch_in_without_row_fallback(oceanbase::sql::ObExpr const&, oceanbase::sql::ObEvalCtx&, oceanbase::sql::ObBitVector const&, long) ()
#4 0x0000aaaadb36eca4 in oceanbase::sql::ObExpr::do_eval_batch(oceanbase::sql::ObEvalCtx&, oceanbase::sql::ObBitVector const&, long) const ()
#5 0x0000aaaad988d0d0 in oceanbase::sql::ObBlackFilterExecutor::eval_exprs_batch(oceanbase::sql::ObBitVector&, long) ()
#6 0x0000aaaad988d918 in oceanbase::sql::ObBlackFilterExecutor::filter_batch(oceanbase::sql::ObPushdownFilterExecutor*, long, long, oceanbase::common::ObBitmap&) ()
#7 0x0000aaaadbafbd5c in oceanbase::storage::ObBlockBatchedRowStore::filter_micro_block_batch(oceanbase::blocksstable::ObMicroBlockDecoder&, oceanbase::sql::ObPushdownFilterExecutor*, oceanbase::sql::ObBlackFilterExecutor&, oceanbase::common::ObBitmap&) ()
#8 0x0000aaaadc699d1c in oceanbase::blocksstable::ObIMicroBlockRowScanner::filter_pushdown_filter(oceanbase::sql::ObPushdownFilterExecutor*, oceanbase::sql::ObPushdownFilterExecutor*, oceanbase::storage::PushdownFilterInfo&, oceanbase::common::ObBitmap&) ()
#9 0x0000aaaadbafe880 in oceanbase::storage::ObBlockRowStore::filter_micro_block(long, oceanbase::blocksstable::ObIMicroBlockRowScanner&, oceanbase::sql::ObPushdownFilterExecutor*, oceanbase::sql::ObPushdownFilterExecutor*) ()
#10 0x0000aaaadbafe91c in oceanbase::storage::ObBlockRowStore::filter_micro_block(long, oceanbase::blocksstable::ObIMicroBlockRowScanner&, oceanbase::sql::ObPushdownFilterExecutor*, oceanbase::sql::ObPushdownFilterExecutor*) ()
#11 0x0000aaaadbafdda0 in oceanbase::storage::ObBlockRowStore::apply_blockscan(oceanbase::blocksstable::ObIMicroBlockRowScanner&, long, bool, oceanbase::storage::ObTableStoreStat&) ()
#12 0x0000aaaadc698774 in oceanbase::blocksstable::ObIMicroBlockRowScanner::apply_blockscan(oceanbase::storage::ObBlockRowStore*, oceanbase::storage::ObTableStoreStat&) ()
#13 0x0000aaaad9d3de68 in oceanbase::storage::ObSSTableRowScanner<oceanbase::storage::ObIndexTreeMultiPassPrefetcher<32, 3> >::open_cur_data_block(oceanbase::storage::ObSSTableReadHandle&) ()
#14 0x0000aaaad9c9e454 in oceanbase::storage::ObSSTableRowScanner<oceanbase::storage::ObIndexTreeMultiPassPrefetcher<32, 3> >::fetch_row(oceanbase::storage::ObSSTableReadHandle&, oceanbase::blocksstable::ObDatumRow const*&) ()
#15 0x0000aaaad9c966f0 in oceanbase::storage::ObStoreRowIterator::get_next_row_ext(oceanbase::blocksstable::ObDatumRow const*&, unsigned char&) ()
#16 0x0000aaaad9c95538 in oceanbase::storage::ObMultipleScanMerge::supply_consume() ()
#17 0x0000aaaad9c932ec in oceanbase::storage::ObMultipleScanMerge::inner_get_next_row(oceanbase::blocksstable::ObDatumRow&) ()
#18 0x0000aaaad9c90530 in oceanbase::storage::ObMultipleMerge::get_next_normal_rows(long&, long) ()
#19 0x0000aaaad9c8e394 in oceanbase::sql::ObTableScanOp::get_next_batch_with_das(long&, long) ()
#20 0x0000aaaad9c8091c in oceanbase::sql::ObTableScanOp::inner_get_next_batch_for_tsc(long) ()
#21 0x0000aaaad9c64088 in oceanbase::sql::ObOperator::get_next_batch(long, oceanbase::sql::ObBatchRows const*&) ()
#22 0x0000aaaad9b6e3f4 in oceanbase::sql::ObSubPlanScanOp::inner_get_next_batch(long) ()
#23 0x0000aaaad9c6431c in oceanbase::sql::ObOperator::get_next_batch(long, oceanbase::sql::ObBatchRows const*&) ()
#24 0x0000aaaad9db8b3c in oceanbase::sql::ObScalarAggregateOp::inner_get_next_batch(long) ()
#25 0x0000aaaad9c6431c in oceanbase::sql::ObOperator::get_next_batch(long, oceanbase::sql::ObBatchRows const*&) ()
#26 0x0000aaaad9c5f4ac in oceanbase::sql::ObResultSet::get_next_row(oceanbase::common::ObNewRow const*&) ()
#27 0x0000aaaad9c5a878 in oceanbase::observer::ObQueryDriver::response_query_result(oceanbase::sql::ObResultSet&, bool, bool, bool&, long) ()
#28 0x0000aaaad9c599d8 in oceanbase::observer::ObSyncPlanDriver::response_result(oceanbase::observer::ObMySQLResultSet&) ()
#29 0x0000aaaad9c21558 in oceanbase::observer::ObMPQuery::process_single_stmt(oceanbase::sql::ObMultiStmtItem const&, oceanbase::sql::ObSQLSessionInfo&, bool, bool, bool&, bool&) ()
#30 0x0000aaaad9c1a568 in oceanbase::observer::ObMPQuery::process() ()
#31 0x0000aaaad9c0ee00 in oceanbase::omt::ObWorkerProcessor::process(oceanbase::rpc::ObRequest&) ()
#32 0x0000aaaad9bfe1d8 in oceanbase::omt::ObThWorker::worker(long&, long&, int&) ()
#33 0x0000aaaad74673d4 in non-virtual thunk to oceanbase::omt::ObThWorker::run(long) ()
#34 0x0000aaaae04888a0 in oceanbase::lib::Thread::__th_start(void*) ()
#35 0x0000ffff9ae978b8 in start_thread () from /lib64/libpthread.so.0
#36 0x0000ffff9ad33fcc in thread_start () from /lib64/libc.so.6
关键诊断信息
触发条件
server 支持 filter 下压(系统配置项 _pushdown_storage_level 的值大于等于 2),保证表达式涉及的表的微块数量大于 1,且数据落盘合并过。 大 in 场景,in 表达式探测列为 text 类型或者 urowid,in 列表中元素超过 6.6w 个。
//now only support c1 in (1,2,3,4...) to be vectorized
if (is_param_can_vectorized()) {
//目前认为右边参数 <= 2 时, nest_loop 算法的效果一定比 hash 更好
if (rt_expr.inner_func_cnt_ <= 2 || is_string_text_cmp ||
ob_is_urowid(left_type) || ob_is_urowid(right_type)) {
rt_expr.eval_batch_func_ = &ObExprInOrNotIn::eval_batch_in_without_row_fallback;
} else {
rt_expr.eval_batch_func_ = &ObExprInOrNotIn::eval_batch_in_without_row;
}
}
OceanBase 数据库 V4.2.1 版本开启
_rowsets_enabled。OceanBase 数据库 V4.2.5 版本使用
values table改写优化。OceanBase 数据库 V4.3.x 版本需要关闭向量化接口,走到 in 表达式的批处理接口
eval_batch_in_without_row_fallback。
事前巡检
检查大 in 个数与栈大小。
事后诊断
core 在函数 eval_batch_in_without_row_fallback,sig_code=2。
问题原因
在 eval_batch_in_without_row_fallback 函数中声明了大小为 in 列表元素个数的数组。 该数组使用栈上内存,OBServer 默认大小为 512k,理论上 6.6w 的数组就会爆栈 core 掉,导致在极限大 in 场景下 OBServer 出现 core。
问题的风险及影响
该问题会导致 OBServer core 掉。
影响租户
影响 OceanBase 数据库中的 SYS 租户和 Oracle 租户以及 MySQL 租户。
影响版本
OceanBase 数据库企业版 V4.1.0 GA(oceanbase-4.1.0.0-100001122023040322)及之后版本、V4.2.0 GA(oceanbase-4.2.0.0-100010082023083014)及之后版本、V4.2.1 GA(oceanbase-4.2.1.0-100000182023092722)及之后版本、V4.2.2 GA(oceanbase-4.2.2.0-100000082024011317)及之后版本、V4.2.5 GA(oceanbase-4.2.5.0-100000082024102022)及之后版本、V4.3.0(oceanbase-4.3.0.0-100000072024020200)及之后版本、V4.3.5 GA(oceanbase-4.3.5.0-100000122024123020)及之后版本。
解决方法
升级至问题已修复版本。目前已修复的版本包括 OceanBase 数据库企业版 V4.2.5 BP5(oceanbase-4.2.5.5-105000052025070822)及之后版本、V4.3.5 BP3(oceanbase-4.3.5.3-103000102025071821)及之后版本。
调整
_inlist_rewrite_threshold,使用大 in 改写values table。对于不能使用
values table改写优化的版本,可以尝试减少 in 的个数或者临时把栈调大规避。
规避方式
对于 OceanBase 数据库 V4.3.x 版本,默认不会走到
eval_batch_in_without_row_fallback。对于 OceanBase 数据库 V4.2.x 版本,对于不能使用
values table改写优化的版本,可以尝试减少 in 的个数或者临时把栈调大规避。