--- title: 并发创建租户和修改配置项导致 SYS 租户线程死锁的原因及解决方法-OceanBase数据库使用指南 description: 了解OceanBase数据库在实际应用中关于并发创建租户和修改配置项导致 SYS 租户线程死锁的原因及解决方法相关的常见问题和使用技巧，帮助您快速解决并发创建租户和修改配置项导致 SYS 租户线程死锁的原因及解决方法的难题。 --- 切换语言 - 中文站 - 简体中文 - International - English - 日本站 - 日本語 划线反馈 # 并发创建租户和修改配置项导致 SYS 租户线程死锁的原因及解决方法 更新时间：2026-06-10 01:56 适用版本： V2.1.x、V2.2.x、V3.1.x、V3.2.x 内容类型：Troubleshoot ## 问题现象 OceanBase 数据库 V2.x、V3.x 版本中在并发创建租户和配置项时候（创建租户时候匹配租户参数模版），存在 sys 租户线程死锁情况，导致 sys 租户队列积压，DDL 执行 Hang，sys 租户登录卡等问题通过查询 `__all_virtual_dump_tenant_info` 查看积压情况。 ```shell MySQL [oceanbase]> select svr_ip,tenant_id ,req_queue_total_size,queue_2,queue_3,queue_4,queue_0 from __all_virtual_dump_tenant_info where req_queue_total_size > 0 and tenant_id=1; ``` 输出结果如下： ```shell +---------------+-----------+----------------------+---------+---------+---------+---------+ | svr_ip | tenant_id | req_queue_total_size | queue_2 | queue_3 | queue_4 | queue_0 | +---------------+-----------+----------------------+---------+---------+---------+---------+ | xx.xxx.xxx.1 | 1 | 13212 | 12584 | 306 | 322 | 0 | | xx.xxx.xxx.2 | 1 | 43 | 0 | 0 | 43 | 0 | | xx.xxx.xxx.3 | 1 | 25 | 0 | 1 | 24 | 0 | +---------------+-----------+----------------------+---------+---------+---------+---------+ 3 rows in set (0.02 sec) ``` ## 问题原因 该问题是 OceanBase 数据库 V2.x、V3.x 版本中的缺陷，并发创建租户和配置项场景下多线程对共享临界区资源重复加锁导致。当出现该问题时候，及时通过 `obstack -ao` 收集信息，同时重启有问题的 OBServer 节点。 查阅 obstack 堆栈发现如下信息，以 5C 规格 sys 租户为例可以命中该问题。 SYS 20 个线程，1 在处理 px，1 个预留高优先级，剩余 18 个，如下： 11 个 sys 线程执行到 `admin_set_config` 等待 --> sys 租户线程 A 执行到 `got_version`（持有 `config_lock` 锁） 等待 ---> `ConfigMgr` 线程（持有 `ObTenantConfig::update_task_.task_lock`），如下信息。 ```shell Threads (79953-TNT_L0_1, 79954-TNT_L0_1, 79956-TNT_L0_1, 79957-TNT_L0_1, 79958-TNT_L0_1, 79960-TNT_L0_1, 79962-TNT_L0_1, 79963-TNT_L0_1, 79964-TNT_L0_1, 79969-TNT_L0_1, 79970-TNT_L0_1) #0 0x00007fec66ce6239 in syscall from /usr/lib64/libc-2.17.so #1 0x00000000050dbeed in int oceanbase::common::ObLatchWaitQueue::wait(oceanbase::common::ObWaitProc&, unsigned int, unsigned int, oceanbase::common::ObLatch::LowTryWRLock&, oceanbase::common::ObLatch::LowTryWRLock&, long) from /home/admin/oceanbase/bin/observer #2 0x0000000004f855c0 in oceanbase::common::ObLatch::wrlock(unsigned int, long, unsigned int const*) from /home/admin/oceanbase/bin/observer #3 0x0000000008c6b914 in oceanbase::rootserver::ObRootService::admin_set_config(oceanbase::obrpc::ObAdminSetConfigArg&) from /home/admin/oceanbase/bin/observer #4 0x000000000bc84a4e in oceanbase::rootserver::ObRootServerRPCProcessorBase::process_(oceanbase::obrpc::ObRpcPacketCode) from /home/admin/oceanbase/bin/observer #5 0x0000000004f44648 in oceanbase::obrpc::ObRpcProcessorBase::run() from /home/admin/oceanbase/bin/observer #6 0x000000000b8c71f1 in oceanbase::omt::ObWorkerProcessor::process_one(oceanbase::rpc::ObRequest&) from /home/admin/oceanbase/bin/observer #7 0x0000000004ec9c90 in oceanbase::omt::ObWorkerProcessor::process(oceanbase::rpc::ObRequest&) from /home/admin/oceanbase/bin/observer #8 0x000000000b8c4d31 in oceanbase::omt::ObThWorker::process_request(oceanbase::rpc::ObRequest&) from /home/admin/oceanbase/bin/observer #9 0x0000000004ec58f6 in oceanbase::omt::ObThWorker::worker(long&, long&, int&) from /home/admin/oceanbase/bin/observer #10 0x000000000b8c52a7 in non-virtual thunk to oceanbase::omt::ObThWorker::run(long) from /home/admin/oceanbase/bin/observer #11 0x000000000f3f17f3 in std::_Function_handler::_M_invoke(std::_Any_data const&) from /home/admin/oceanbase/bin/observer #12 0x000000000f3f164f in eeentry(boost::context::detail::transfer_t) from /home/admin/oceanbase/bin/observer #13 0x000000000f6901df in make_fcontext from /home/admin/oceanbase/bin/observer ``` 5 个 sys 租户线程执行 `create_new_tenant` 等待 --> sys 租户线程 B 执行到 `add_tenant_config`（持有 `get_serialize_lock` 读锁）请求 `config_lock` 的写锁，如下信息。 ```shell Threads (79951-TNT_L0_1, 79959-TNT_L0_1, 79965-TNT_L0_1, 79966-TNT_L0_1, 79967-TNT_L0_1) #0 0x00007fec66ce6239 in syscall from /usr/lib64/libc-2.17.so #1 0x00000000050dbeed in int oceanbase::common::ObLatchWaitQueue::wait(oceanbase::common::ObWaitProc&, unsigned int, unsigned int, oceanbase::common::ObLatch::LowTryWRLock&, oceanbase::common::ObLatch::LowTryWRLock&, long) from /home/admin/oceanbase/bin/observer #2 0x0000000004f855c0 in oceanbase::common::ObLatch::wrlock(unsigned int, long, unsigned int const*) from /home/admin/oceanbase/bin/observer #3 0x00000000050a16f1 in oceanbase::common::TCRWLock::wrlock(long) from /home/admin/oceanbase/bin/observer #4 0x000000000b8b9988 in oceanbase::omt::ObTenantNodeBalancer::create_new_tenant(oceanbase::share::ObUnitInfoGetter::ObTenantConfig&) from /home/admin/oceanbase/bin/observer #5 0x000000000b8b9386 in oceanbase::omt::ObTenantNodeBalancer::notify_create_tenant(oceanbase::obrpc::TenantServerUnitConfig const&) from /home/admin/oceanbase/bin/observer #6 0x000000000b951dcc in oceanbase::observer::ObRpcNotifyTenantServerUnitResourceP::process() from /home/admin/oceanbase/bin/observer #7 0x0000000004f44648 in oceanbase::obrpc::ObRpcProcessorBase::run() from /home/admin/oceanbase/bin/observer #8 0x000000000b8c71f1 in oceanbase::omt::ObWorkerProcessor::process_one(oceanbase::rpc::ObRequest&) from /home/admin/oceanbase/bin/observer #9 0x0000000004ec9c90 in oceanbase::omt::ObWorkerProcessor::process(oceanbase::rpc::ObRequest&) from /home/admin/oceanbase/bin/observer #10 0x000000000b8c4d31 in oceanbase::omt::ObThWorker::process_request(oceanbase::rpc::ObRequest&) from /home/admin/oceanbase/bin/observer #11 0x0000000004ec58f6 in oceanbase::omt::ObThWorker::worker(long&, long&, int&) from /home/admin/oceanbase/bin/observer #12 0x000000000b8c52a7 in non-virtual thunk to oceanbase::omt::ObThWorker::run(long) from /home/admin/oceanbase/bin/observer #13 0x000000000f3f17f3 in std::_Function_handler::_M_invoke(std::_Any_data const&) from /home/admin/oceanbase/bin/observer #14 0x000000000f3f164f in eeentry(boost::context::detail::transfer_t) from /home/admin/oceanbase/bin/observer #15 0x000000000f6901df in make_fcontext from /home/admin/oceanbase/bin/observer ``` `ConfigMgr` 执行到 `dump2file`（持有 `ObTenantConfig::update_task_.task_lock_`）等待 `get_serialize_lock()` 写锁，如下信息。 ```shell Threads (79003-ConfigMgr) #0 0x00007fec66ce6239 in syscall from /usr/lib64/libc-2.17.so #1 0x00000000050dbeed in int oceanbase::common::ObLatchWaitQueue::wait(oceanbase::common::ObWaitProc&, unsigned int, unsigned int, oceanbase::common::ObLatch::LowTryWRLock&, oceanbase::common::ObLatch::LowTryWRLock&, long) from /home/admin/oceanbase/bin/observer #2 0x0000000004f855c0 in oceanbase::common::ObLatch::wrlock(unsigned int, long, unsigned int const*) from /home/admin/oceanbase/bin/observer #3 0x0000000005d58c05 in oceanbase::common::DRWLock::wrlock() from /home/admin/oceanbase/bin/observer #4 0x000000000697b232 in oceanbase::common::ObConfigManager::dump2file(char const*) const from /home/admin/oceanbase/bin/observer #5 0x000000000697c6a3 in oceanbase::common::ObConfigManager::config_backup() from /home/admin/oceanbase/bin/observer #6 0x000000000b8ac45a in oceanbase::omt::ObTenantConfig::update_local(long, oceanbase::common::ObISQLClient::ReadResult&, bool) from /home/admin/oceanbase/bin/observer #7 0x000000000b8aaec0 in oceanbase::omt::ObTenantConfigMgr::update_local(unsigned long, long) from /home/admin/oceanbase/bin/observer #8 0x000000000b8aa65f in oceanbase::omt::ObTenantConfig::TenantConfigUpdateTask::runTimerTask() from /home/admin/oceanbase/bin/observer #9 0x000000000507a359 in oceanbase::common::ObTimer::run1() from /home/admin/oceanbase/bin/observer #10 0x00000000065b657e in oceanbase::lib::ThreadPool::run0() from /home/admin/oceanbase/bin/observer #11 0x000000000f3f17f3 in std::_Function_handler::_M_invoke(std::_Any_data const&) from /home/admin/oceanbase/bin/observer #12 0x000000000f3f164f in eeentry(boost::context::detail::transfer_t) from /home/admin/oceanbase/bin/observer #13 0x000000000f6901df in make_fcontext from /home/admin/oceanbase/bin/observer ``` 线程 `OmtNodeBalancer`（持有 `get_serialize_lock` 读锁）请求 `config_lock` 写锁等待 ---〉 sys 线程 A 持有 `config_lock` 锁，如下信息。 ```shell Threads (81789-OmtNodeBalancer) #0 0x00007fec66ce6239 in syscall from /usr/lib64/libc-2.17.so #1 0x00000000050dbeed in int oceanbase::common::ObLatchWaitQueue::wait(oceanbase::common::ObWaitProc&, unsigned int, unsigned int, oceanbase::common::ObLatch::LowTryWRLock&, oceanbase::common::ObLatch::LowTryWRLock&, long) from /home/admin/oceanbase/bin/observer #2 0x0000000004f855c0 in oceanbase::common::ObLatch::wrlock(unsigned int, long, unsigned int const*) from /home/admin/oceanbase/bin/observer #3 0x000000000b897883 in oceanbase::omt::ObTenantConfigMgr::add_tenant_config(unsigned long) from /home/admin/oceanbase/bin/observer #4 0x000000000b8b07b3 in oceanbase::omt::ObTenantConfigMgr::refresh_tenants(oceanbase::common::ObIArray const&) from /home/admin/oceanbase/bin/observer #5 0x000000000b8b72d4 in oceanbase::omt::ObTenantNodeBalancer::run1() from /home/admin/oceanbase/bin/observer #6 0x0000000005235ba4 in oceanbase::share::ObThreadPool::run0() from /home/admin/oceanbase/bin/observer #7 0x000000000f3f17f3 in std::_Function_handler::_M_invoke(std::_Any_data const&) from /home/admin/oceanbase/bin/observer #8 0x000000000f3f164f in eeentry(boost::context::detail::transfer_t) from /home/admin/oceanbase/bin/observer #9 0x000000000f6901df in make_fcontext from /home/admin/oceanbase/bin/observer ``` 上述持锁堆栈和链路形成了死锁，触发场景：多租户创建 + 修改配置项（OCP 多租户创建时候同时应用租户参数模版）。 ## 关键信息 通过 `sys@diag` 诊断用户登录查 `__all_virtual_dump_tenant_info` 查看 sys 租户积压情况，通过执行 `obstack -ao` 分析 `TNT_L0_L1` 线程持锁堆栈情况是否与问题原因章节中堆栈匹配。 ## 问题的风险及影响 SYS 租户不可用，DDL 执行超时。 ## 影响租户 影响 OceanBase 数据库中的 SYS 租户和 Oracle 租户以及 MySQL 租户。 ## 适用版本 OceanBase 数据库 V2.x、V3.x 版本。 ## 解决方法及规避方式 - 解决方法： 调整 SYS 租户的 CPU 规格及重启 OBServer。 - 规避方式： 避免并发创建租户和更新租户配置项。 上一篇 [如何对租户缩容时删除指定的 unit](https://www.oceanbase.com/knowledge-base/oceanbase-database-1000000000207642) 下一篇 [减少 Unit 配置错误，错误代码 1235](https://www.oceanbase.com/knowledge-base/oceanbase-database-20000000096) ![有帮助](https://gw.alipayobjects.com/mdn/ob_asset/afts/img/A*y6ocSqN8cqsAAAAAAAAAAAAAARQnAQ)![无帮助](https://gw.alipayobjects.com/mdn/ob_asset/afts/img/A*BG9IQJyLHF8AAAAAAAAAAAAAARQnAQ)![反馈](https://gw.alipayobjects.com/mdn/ob_asset/afts/img/A*eTWdQKCRKHwAAAAAAAAAAAAAARQnAQ)[AI](https://www.oceanbase.com/obi) 咨询热线