划线反馈

并发创建租户和修改配置项导致 SYS 租户线程死锁的原因及解决方法

更新时间：2026-06-10 01:56

适用版本：V2.1.x、V2.2.x、V3.1.x、V3.2.x内容类型：Troubleshoot

问题现象

OceanBase 数据库 V2.x、V3.x 版本中在并发创建租户和配置项时候（创建租户时候匹配租户参数模版），存在 sys 租户线程死锁情况，导致 sys 租户队列积压，DDL 执行 Hang，sys 租户登录卡等问题通过查询 __all_virtual_dump_tenant_info 查看积压情况。

MySQL [oceanbase]> select svr_ip,tenant_id ,req_queue_total_size,queue_2,queue_3,queue_4,queue_0 from __all_virtual_dump_tenant_info where req_queue_total_size > 0 and tenant_id=1;

输出结果如下：

+---------------+-----------+----------------------+---------+---------+---------+---------+
| svr_ip        | tenant_id | req_queue_total_size | queue_2 | queue_3 | queue_4 | queue_0 |
+---------------+-----------+----------------------+---------+---------+---------+---------+
| xx.xxx.xxx.1  |         1 |                13212 |   12584 |     306 |     322 |       0 |
| xx.xxx.xxx.2  |         1 |                   43 |       0 |       0 |      43 |       0 |
| xx.xxx.xxx.3  |         1 |                   25 |       0 |       1 |      24 |       0 |
+---------------+-----------+----------------------+---------+---------+---------+---------+
3 rows in set (0.02 sec)

问题原因

该问题是 OceanBase 数据库 V2.x、V3.x 版本中的缺陷，并发创建租户和配置项场景下多线程对共享临界区资源重复加锁导致。当出现该问题时候，及时通过 obstack -ao 收集信息，同时重启有问题的 OBServer 节点。查阅 obstack 堆栈发现如下信息，以 5C 规格 sys 租户为例可以命中该问题。

SYS 20 个线程，1 在处理 px，1 个预留高优先级，剩余 18 个，如下：

11 个 sys 线程执行到 admin_set_config 等待 --> sys 租户线程 A 执行到 got_version（持有 config_lock 锁）等待 ---> ConfigMgr 线程（持有 ObTenantConfig::update_task_.task_lock），如下信息。

Threads (79953-TNT_L0_1, 79954-TNT_L0_1, 79956-TNT_L0_1, 79957-TNT_L0_1, 79958-TNT_L0_1, 79960-TNT_L0_1, 79962-TNT_L0_1, 79963-TNT_L0_1, 79964-TNT_L0_1, 79969-TNT_L0_1, 79970-TNT_L0_1)
#0    0x00007fec66ce6239 in syscall from /usr/lib64/libc-2.17.so
#1    0x00000000050dbeed in int oceanbase::common::ObLatchWaitQueue::wait<oceanbase::common::ObLatch::LowTryWRLock>(oceanbase::common::ObWaitProc&, unsigned int, unsigned int, oceanbase::common::ObLatch::LowTryWRLock&, oceanbase::common::ObLatch::LowTryWRLock&, long) from /home/admin/oceanbase/bin/observer
#2    0x0000000004f855c0 in oceanbase::common::ObLatch::wrlock(unsigned int, long, unsigned int const*) from /home/admin/oceanbase/bin/observer
#3    0x0000000008c6b914 in oceanbase::rootserver::ObRootService::admin_set_config(oceanbase::obrpc::ObAdminSetConfigArg&) from /home/admin/oceanbase/bin/observer
#4    0x000000000bc84a4e in oceanbase::rootserver::ObRootServerRPCProcessorBase::process_(oceanbase::obrpc::ObRpcPacketCode) from /home/admin/oceanbase/bin/observer
#5    0x0000000004f44648 in oceanbase::obrpc::ObRpcProcessorBase::run() from /home/admin/oceanbase/bin/observer
#6    0x000000000b8c71f1 in oceanbase::omt::ObWorkerProcessor::process_one(oceanbase::rpc::ObRequest&) from /home/admin/oceanbase/bin/observer
#7    0x0000000004ec9c90 in oceanbase::omt::ObWorkerProcessor::process(oceanbase::rpc::ObRequest&) from /home/admin/oceanbase/bin/observer
#8    0x000000000b8c4d31 in oceanbase::omt::ObThWorker::process_request(oceanbase::rpc::ObRequest&) from /home/admin/oceanbase/bin/observer
#9    0x0000000004ec58f6 in oceanbase::omt::ObThWorker::worker(long&, long&, int&) from /home/admin/oceanbase/bin/observer
#10   0x000000000b8c52a7 in non-virtual thunk to oceanbase::omt::ObThWorker::run(long) from /home/admin/oceanbase/bin/observer
#11   0x000000000f3f17f3 in std::_Function_handler<void (), oceanbase::lib::UserThread::start()::$_14::operator()(oblib::Thread*) const::{lambda()#1}>::_M_invoke(std::_Any_data const&) from /home/admin/oceanbase/bin/observer
#12   0x000000000f3f164f in eeentry(boost::context::detail::transfer_t) from /home/admin/oceanbase/bin/observer
#13   0x000000000f6901df in make_fcontext from /home/admin/oceanbase/bin/observer

5 个 sys 租户线程执行 create_new_tenant 等待 --> sys 租户线程 B 执行到 add_tenant_config（持有 get_serialize_lock 读锁）请求 config_lock 的写锁，如下信息。

Threads (79951-TNT_L0_1, 79959-TNT_L0_1, 79965-TNT_L0_1, 79966-TNT_L0_1, 79967-TNT_L0_1)
#0    0x00007fec66ce6239 in syscall from /usr/lib64/libc-2.17.so
#1    0x00000000050dbeed in int oceanbase::common::ObLatchWaitQueue::wait<oceanbase::common::ObLatch::LowTryWRLock>(oceanbase::common::ObWaitProc&, unsigned int, unsigned int, oceanbase::common::ObLatch::LowTryWRLock&, oceanbase::common::ObLatch::LowTryWRLock&, long) from /home/admin/oceanbase/bin/observer
#2    0x0000000004f855c0 in oceanbase::common::ObLatch::wrlock(unsigned int, long, unsigned int const*) from /home/admin/oceanbase/bin/observer
#3    0x00000000050a16f1 in oceanbase::common::TCRWLock::wrlock(long) from /home/admin/oceanbase/bin/observer
#4    0x000000000b8b9988 in oceanbase::omt::ObTenantNodeBalancer::create_new_tenant(oceanbase::share::ObUnitInfoGetter::ObTenantConfig&) from /home/admin/oceanbase/bin/observer
#5    0x000000000b8b9386 in oceanbase::omt::ObTenantNodeBalancer::notify_create_tenant(oceanbase::obrpc::TenantServerUnitConfig const&) from /home/admin/oceanbase/bin/observer
#6    0x000000000b951dcc in oceanbase::observer::ObRpcNotifyTenantServerUnitResourceP::process() from /home/admin/oceanbase/bin/observer
#7    0x0000000004f44648 in oceanbase::obrpc::ObRpcProcessorBase::run() from /home/admin/oceanbase/bin/observer
#8    0x000000000b8c71f1 in oceanbase::omt::ObWorkerProcessor::process_one(oceanbase::rpc::ObRequest&) from /home/admin/oceanbase/bin/observer
#9    0x0000000004ec9c90 in oceanbase::omt::ObWorkerProcessor::process(oceanbase::rpc::ObRequest&) from /home/admin/oceanbase/bin/observer
#10   0x000000000b8c4d31 in oceanbase::omt::ObThWorker::process_request(oceanbase::rpc::ObRequest&) from /home/admin/oceanbase/bin/observer
#11   0x0000000004ec58f6 in oceanbase::omt::ObThWorker::worker(long&, long&, int&) from /home/admin/oceanbase/bin/observer
#12   0x000000000b8c52a7 in non-virtual thunk to oceanbase::omt::ObThWorker::run(long) from /home/admin/oceanbase/bin/observer
#13   0x000000000f3f17f3 in std::_Function_handler<void (), oceanbase::lib::UserThread::start()::$_14::operator()(oblib::Thread*) const::{lambda()#1}>::_M_invoke(std::_Any_data const&) from /home/admin/oceanbase/bin/observer
#14   0x000000000f3f164f in eeentry(boost::context::detail::transfer_t) from /home/admin/oceanbase/bin/observer
#15   0x000000000f6901df in make_fcontext from /home/admin/oceanbase/bin/observer

ConfigMgr 执行到 dump2file（持有 ObTenantConfig::update_task_.task_lock_）等待 get_serialize_lock() 写锁，如下信息。

Threads (79003-ConfigMgr)
#0    0x00007fec66ce6239 in syscall from /usr/lib64/libc-2.17.so
#1    0x00000000050dbeed in int oceanbase::common::ObLatchWaitQueue::wait<oceanbase::common::ObLatch::LowTryWRLock>(oceanbase::common::ObWaitProc&, unsigned int, unsigned int, oceanbase::common::ObLatch::LowTryWRLock&, oceanbase::common::ObLatch::LowTryWRLock&, long) from /home/admin/oceanbase/bin/observer
#2    0x0000000004f855c0 in oceanbase::common::ObLatch::wrlock(unsigned int, long, unsigned int const*) from /home/admin/oceanbase/bin/observer
#3    0x0000000005d58c05 in oceanbase::common::DRWLock::wrlock() from /home/admin/oceanbase/bin/observer
#4    0x000000000697b232 in oceanbase::common::ObConfigManager::dump2file(char const*) const from /home/admin/oceanbase/bin/observer
#5    0x000000000697c6a3 in oceanbase::common::ObConfigManager::config_backup() from /home/admin/oceanbase/bin/observer
#6    0x000000000b8ac45a in oceanbase::omt::ObTenantConfig::update_local(long, oceanbase::common::ObISQLClient::ReadResult&, bool) from /home/admin/oceanbase/bin/observer
#7    0x000000000b8aaec0 in oceanbase::omt::ObTenantConfigMgr::update_local(unsigned long, long) from /home/admin/oceanbase/bin/observer
#8    0x000000000b8aa65f in oceanbase::omt::ObTenantConfig::TenantConfigUpdateTask::runTimerTask() from /home/admin/oceanbase/bin/observer
#9    0x000000000507a359 in oceanbase::common::ObTimer::run1() from /home/admin/oceanbase/bin/observer
#10   0x00000000065b657e in oceanbase::lib::ThreadPool::run0() from /home/admin/oceanbase/bin/observer
#11   0x000000000f3f17f3 in std::_Function_handler<void (), oceanbase::lib::UserThread::start()::$_14::operator()(oblib::Thread*) const::{lambda()#1}>::_M_invoke(std::_Any_data const&) from /home/admin/oceanbase/bin/observer
#12   0x000000000f3f164f in eeentry(boost::context::detail::transfer_t) from /home/admin/oceanbase/bin/observer
#13   0x000000000f6901df in make_fcontext from /home/admin/oceanbase/bin/observer

线程 OmtNodeBalancer（持有 get_serialize_lock 读锁）请求 config_lock 写锁等待 ---〉 sys 线程 A 持有 config_lock 锁，如下信息。

Threads (81789-OmtNodeBalancer)
#0    0x00007fec66ce6239 in syscall from /usr/lib64/libc-2.17.so
#1    0x00000000050dbeed in int oceanbase::common::ObLatchWaitQueue::wait<oceanbase::common::ObLatch::LowTryWRLock>(oceanbase::common::ObWaitProc&, unsigned int, unsigned int, oceanbase::common::ObLatch::LowTryWRLock&, oceanbase::common::ObLatch::LowTryWRLock&, long) from /home/admin/oceanbase/bin/observer
#2    0x0000000004f855c0 in oceanbase::common::ObLatch::wrlock(unsigned int, long, unsigned int const*) from /home/admin/oceanbase/bin/observer
#3    0x000000000b897883 in oceanbase::omt::ObTenantConfigMgr::add_tenant_config(unsigned long) from /home/admin/oceanbase/bin/observer
#4    0x000000000b8b07b3 in oceanbase::omt::ObTenantConfigMgr::refresh_tenants(oceanbase::common::ObIArray<unsigned long> const&) from /home/admin/oceanbase/bin/observer
#5    0x000000000b8b72d4 in oceanbase::omt::ObTenantNodeBalancer::run1() from /home/admin/oceanbase/bin/observer
#6    0x0000000005235ba4 in oceanbase::share::ObThreadPool::run0() from /home/admin/oceanbase/bin/observer
#7    0x000000000f3f17f3 in std::_Function_handler<void (), oceanbase::lib::UserThread::start()::$_14::operator()(oblib::Thread*) const::{lambda()#1}>::_M_invoke(std::_Any_data const&) from /home/admin/oceanbase/bin/observer
#8    0x000000000f3f164f in eeentry(boost::context::detail::transfer_t) from /home/admin/oceanbase/bin/observer
#9    0x000000000f6901df in make_fcontext from /home/admin/oceanbase/bin/observer

上述持锁堆栈和链路形成了死锁，触发场景：多租户创建 + 修改配置项（OCP 多租户创建时候同时应用租户参数模版）。