划线反馈

OceanBase 数据库 V3.x 大事务回滚卡住的原因

更新时间：2026-06-10 09:51

适用版本：V1.4.x、V2.1.x、V2.2.x、V3.1.x、V3.2.x内容类型：Troubleshoot

问题现象

事务回滚一直超时，通过 kill session_id 的方法杀不掉，会导致 kill 动作卡住；多次抓 obstack（事务协调者节点）都卡在 transaction::ObScheTransCtx::wait_end_stmt上。

Thread 1346403 (TNT_L0_1006)
#0    0x00007fd20333ecf6 in ??? from /usr/lib64/libpthread-2.28.so
#1    0x0000000009b305c9 in bool obutil::Cond::timed_wait_impl<obutil::Mutex>(obutil::Mutex const&, obutil::ObSysTime const&) const from /home/admin/oceanbase/bin/observer
#2    0x000000000b0fc7f8 in oceanbase::transaction::ObTransCond::wait(long, int&) from /home/admin/oceanbase/bin/observer
#3    0x000000000b1175b3 in oceanbase::transaction::ObScheTransCtx::wait_end_stmt(long, long) from /home/admin/oceanbase/bin/observer
#4    0x000000000b19e55d in oceanbase::transaction::ObTransService::rollback_stmt_(oceanbase::transaction::ObTransDesc&) from /home/admin/oceanbase/bin/observer
#5    0x0000000004f37140 in oceanbase::transaction::ObTransService::end_stmt_v2(oceanbase::transaction::ObStmtParam const&, oceanbase::transaction::ObTransDesc&, oceanbase::transaction::ObTransExecResult const&, bool) from /home/admin/oceanbase/bin/observer
#6    0x0000000004f34e66 in oceanbase::storage::ObPartitionService::end_stmt_v2(oceanbase::transaction::ObStmtParam const&, oceanbase::transaction::ObTransDesc&, oceanbase::transaction::ObTransExecResult const&, bool) from /home/admin/oceanbase/bin/observer
#7    0x0000000004f0de69 in oceanbase::sql::ObSqlTransControl::end_stmt(oceanbase::sql::ObExecContext&, bool) from /home/admin/oceanbase/bin/observer
#8    0x0000000004f0c1cf in oceanbase::sql::ObResultSet::end_stmt(bool) from /home/admin/oceanbase/bin/observer
#9    0x0000000004f0a6a3 in oceanbase::sql::ObResultSet::close(bool) from /home/admin/oceanbase/bin/observer
#10   0x000000000b915318 in oceanbase::observer::ObInnerSQLResult::inner_close(bool) from /home/admin/oceanbase/bin/observer
#11   0x000000000b9030eb in oceanbase::observer::ObInnerSQLResult::force_close(bool) from /home/admin/oceanbase/bin/observer
#12   0x000000000b906a46 in oceanbase::observer::ObInnerSQLConnection::execute(oceanbase::common::Ob2DArray<oceanbase::common::ObObjParam, 2096128, oceanbase::common::ObWrapperAllocator, false, oceanbase::common::ObSEArray<oceanbase::common::ObObjParam*, 1l, oceanbase::common::ObWrapperAllocator, false> >&, oceanbase::observer::ObInnerSQLResult&, oceanbase::observer::ObVirtualTableIteratorFactory*, bool, bool) from /home/admin/oceanbase/bin/observer
#13   0x000000000b90be91 in oceanbase::observer::ObInnerSQLConnection::execute(unsigned long, unsigned long, oceanbase::sql::stmt::StmtType, oceanbase::common::Ob2DArray<oceanbase::common::ObObjParam, 2096128, oceanbase::common::ObWrapperAllocator, false, oceanbase::common::ObSEArray<oceanbase::common::ObObjParam*, 1l, oceanbase::common::ObWrapperAlloc
ator, false> >&, oceanbase::common::ObISQLClient::ReadResult&, bool, bool) from /home/admin/oceanbase/bin/observer
#14   0x000000000e2ac67b in oceanbase::sql::ObSPIService::inner_open(oceanbase::pl::ObPLExecCtx*, char const*, unsigne
d long, long, oceanbase::common::Ob2DArray<oceanbase::common::ObObjParam, 2096128, oceanbase::common::ObWrapperAllocat
or, false, oceanbase::common::ObSEArray<oceanbase::common::ObObjParam*, 1l, oceanbase::common::ObWrapperAllocator, fal
se> >&, oceanbase::common::ObISQLClient::ReadResult&, oceanbase::sql::ObSPIOutParams&) from /home/admin/oceanbase/bin/
observer
#15   0x000000000e2a9c3a in oceanbase::sql::ObSPIService::inner_open(oceanbase::pl::ObPLExecCtx*, oceanbase::common::O
bIAllocator&, char const*, unsigned long, long, oceanbase::sql::ObSqlExpression const**, long, oceanbase::sql::ObSqlEx
pression const**, long, oceanbase::common::ObISQLClient::ReadResult&, oceanbase::sql::ObSPIOutParams&) from /home/admi
n/oceanbase/bin/observer

日志中一直在重试回滚，并且一直失败，如下图所示。

日志中有大量的此事务写日志超时信息 fill mutator log cost too much time，如下图所示。

并且写日志时间超过 _ob_trans_rpc_timeout 参数的默认值 3s；并且存在非常大量的callback（千万级别以上）。

如下图所示日志需要在事务实际执行节点过滤 mt_ctx_.get_callback_count():185867544，此事务产生了 1 亿多条 callback。

关键信息

obstack 中可以看到堆栈卡在 transaction::ObScheTransCtx::wait_end_stmt 长时间不推进。
日志中过滤 mt_ctx_.get_callback_count 大小超过 7000000。
日志中 fill mutator log cost too much time 的 used 大小超过 3000000。

问题原因

事务过大（处于 freeze_trigger_percentage 阈值比较大，机器内存较大的环境中），事务产生的回调 callback 过多，有 1 亿多条 callback 在内存中（内存规格很大未及时转储），而事务在回滚时，rollback to 会转到已经提交过日志的地方，这个时候会把 cursor 放到最头部，重新遍历这 1 亿条 callback，导致事务回滚时遍历时间过久。而且每次 rollback to 都要写日志，写日志超时后会失败重试，所以此时线程会一直重试，导致回滚超时，事务状态不推进。而 kill session 需要等待 session 所属线程释放锁，所以需要等待事务回滚完成，若事务回滚卡住，session 就无法被 kill。

总结：大事务 callback list 较多时，语句回滚存在极端场景，需要遍历整个链表，使得回滚耗时较久，最终回滚消息超时，事务回滚语句动作一直挂住无法完成。