问题现象
PL 执行时出现卡住,日志不断出现如下重试日志信息且无好转。
observer.log.20240618170825983:[2024-06-18 17:07:31.215282] INFO [SQL.CG] allocate_mapped_memory (ob_jit_allocator.cpp:125) [178042][T1002_L0_G0][T1002][xxxxx-xxxxx-xxxxx-xxxxx] [lt=67] aarch64 memory allocated not safe, try again(addr=0xfff2ffff0000, start=281423436972031, page_size=65536, num_pages=1)
抓取 obstack 发现卡在 oceanbase::jit::core::ObJitMemory::allocate_mapped_memory 信息如下。
Threads (178042-T1002_L0_G0)
#0 0x0000fffbf6487598 in ??? from /usr/lib64/libc-2.28.so
#1 0x0000fffbf64af6f8 in ??? from /usr/lib64/libc-2.28.so
#2 0x0000aaac871e2ae8 in oceanbase::jit::core::ObJitMemory::allocate_mapped_memory(long, long) from /home/admin/oceanbase/bin/observer
#3 0x0000aaac871e3b78 in oceanbase::jit::core::ObJitMemoryGroup::reserve(long, long, long) from /home/admin/oceanbase/bin/observer
#4 0x0000aaac7b80a044 in llvm::RuntimeDyldImpl::loadObjectImpl(llvm::object::ObjectFile const&) from /home/admin/oceanbase/bin/observer
#5 0x0000aaac7b81bff4 in llvm::RuntimeDyldELF::loadObject(llvm::object::ObjectFile const&) from /home/admin/oceanbase/bin/observer
#6 0x0000aaac7b804f54 in llvm::RuntimeDyld::loadObject(llvm::object::ObjectFile const&) from /home/admin/oceanbase/bin/observer
#7 0x0000aaac871e19fc in llvm::orc::LegacyRTDyldObjectLinkingLayer::ConcreteLinkedObject<std::shared_ptr<llvm::RuntimeDyld::MemoryManager> >::finalize() from /home/admin/oceanbase/bin/observer
#8 0x0000aaac871e2848 in llvm::Expected<unsigned long> llvm::detail::UniqueFunctionBase<llvm::Expected<unsigned long>>::CallImpl<llvm::orc::LegacyRTDyldObjectLinkingLayer::ConcreteLinkedObject<std::shared_ptr<llvm::RuntimeDyld::MemoryManager> >::getSymbolMaterializer(std::string)::{lambda()#1}>(void*) from /home/admin/oceanbase/bin/observer
#9 0x0000aaac871df674 in llvm::JITSymbol::getAddress() from /home/admin/oceanbase/bin/observer
#10 0x0000aaac871df3c4 in oceanbase::jit::core::ObOrcJit::get_function_address(std::string) from /home/admin/oceanbase/bin/observer
#11 0x0000aaac871c240c in oceanbase::jit::ObLLVMHelper::get_function_address(oceanbase::common::ObString const&) from /home/admin/oceanbase/bin/observer
#12 0x0000aaac7e0c0848 in oceanbase::pl::ObPLCodeGenerator::generate_normal(oceanbase::pl::ObPLFunction&) from /home/admin/oceanbase/bin/observer
#13 0x0000aaac7e0bf0e4 in oceanbase::pl::ObPLCodeGenerator::generate(oceanbase::pl::ObPLFunction&) from /home/admin/oceanbase/bin/observer
#14 0x0000aaac7df81ad0 in oceanbase::pl::ObPLCompiler::compile(_ParseNode const*, unsigned long, oceanbase::pl::ObPLFunction&, oceanbase::common::Ob2DArray<oceanbase::common::ObObjParam, 2079744, oceanbase::common::ObWrapperAllocator, false, oceanbase::common::ObSEArray<oceanbase::common::ObObjParam*, 1l, oceanbase::common::ObWrapperAllocator, false> >*, bool) from /home/admin/oceanbase/bin/observer
#15 0x0000aaac7df7dfa4 in oceanbase::pl::ObPL::get_pl_function(oceanbase::sql::ObExecContext&, oceanbase::common::Ob2DArray<oceanbase::common::ObObjParam, 2079744, oceanbase::common::ObWrapperAllocator, false, oceanbase::common::ObSEArray<oceanbase::common::ObObjParam*, 1l, oceanbase::common::ObWrapperAllocator, false> >&, unsigned long, oceanbase::common::ObString const&, oceanbase::sql::ObCacheObjGuard&) from /home/admin/oceanbase/bin/observer
#16 0x0000aaac7df84440 in oceanbase::pl::ObPL::execute(oceanbase::sql::ObExecContext&, oceanbase::common::Ob2DArray<oceanbase::common::ObObjParam, 2079744, oceanbase::common::ObWrapperAllocator, false, oceanbase::common::ObSEArray<oceanbase::common::ObObjParam*, 1l, oceanbase::common::ObWrapperAllocator, false> >&, unsigned long, oceanbase::common::ObString const&, oceanbase::common::ObBitSet<256l, oceanbase::common::ModulePageAllocator, false>&) from /home/admin/oceanbase/bin/observer
#17 0x0000aaac80a76b3c in oceanbase::sql::ObAnonymousBlockExecutor::execute(oceanbase::sql::ObExecContext&, oceanbase::sql::ObAnonymousBlockStmt&) from /home/admin/oceanbase/bin/observer
#18 0x0000aaac8084f7c0 in oceanbase::sql::ObCmdExecutor::execute(oceanbase::sql::ObExecContext&, oceanbase::sql::ObICmd&)::$_115::operator()() const from /home/admin/oceanbase/bin/observer
#19 0x0000aaac816a50e0 in oceanbase::sql::ObCmdExecutor::execute(oceanbase::sql::ObExecContext&, oceanbase::sql::ObICmd&) from /home/admin/oceanbase/bin/observer
#20 0x0000aaac815ae194 in oceanbase::sql::ObResultSet::execute() from /home/admin/oceanbase/bin/observer
#21 0x0000aaac815abff4 in oceanbase::sql::ObResultSet::open() from /home/admin/oceanbase/bin/observer
#22 0x0000aaac8254b048 in oceanbase::sql::ObSql::handle_pl_execute(oceanbase::common::ObString const&, oceanbase::sql::ObSQLSessionInfo&, oceanbase::common::Ob2DArray<oceanbase::common::ObObjParam, 2079744, oceanbase::common::ObWrapperAllocator, false, oceanbase::common::ObSEArray<oceanbase::common::ObObjParam*, 1l, oceanbase::common::ObWrapperAllocator, false> >&, oceanbase::sql::ObResultSet&, oceanbase::sql::ObSqlCtx&, bool, bool) from /home/admin/oceanbase/bin/observer
#23 0x0000aaac825464a8 in oceanbase::sql::ObSPIService::spi_execute_immediate(oceanbase::pl::ObPLExecCtx*, oceanbase::sql::ObSqlExpression const*, oceanbase::common::ObObjParam**, long const*, long, oceanbase::sql::ObSqlExpression const**, long, oceanbase::common::ObDataType const*, long, bool const*, long const*, bool, bool, bool) from /home/admin/oceanbase/bin/observer
#24 0x0000fffaa6208188 in ??? from ???
#25 0x0000fffaa6208188 in ??? from ???
关键信息
需要不断出现重试日志信息且无好转才可证明存在 hang 的问题,关键日志信息如下。
observer.log.20240618170825983:[2024-06-18 17:07:31.215282] INFO [SQL.CG] allocate_mapped_memory (ob_jit_allocator.cpp:125) [178042][T1002_L0_G0][T1002][xxxxx-xxxxx-xxxxx-xxxxx] [lt=67] aarch64 memory allocated not safe, try again(addr=0xfff2ffff0000, start=281423436972031, page_size=65536, num_pages=1)
问题原因
PL 执行期间需要分配内存进行编译,编译时内存从 500 租户分配,如果内存分配出现问题,OceanBase 数据库内部会进行不断重试而不出错,从而导致 PL 执行 hang 住,内存分配出现问题的原因有如下二个。
OceanBase 数据库内存或操作系统内存不足,导致 500 租户内存分配时分配不出内存导致。
OceanBase 数据库 V4.2.1 之前依赖的 LLVM 版本上存在 BUG,分配的内存区域跨越 4G 地址边界的场景下会导致执行时出现问题,因此这里分配内存的时候需要检查内存区域是不是跨越了 4G 边界,如果跨越了 4G 边界就需要重新分配内存,直到分配到不跨越 4G 边界的内存。由于 OceanBase 缺陷可能会出现重新分配出来的内存一直是同一个地址,因此会不断的重新分配地址导致无法终止循环。
问题的风险及影响
PL 执行时间变长,可能会造成业务影响。
影响租户
影响 OceanBase 数据库中的 SYS 租户和 Oracle 租户以及 MySQL 租户。
适用版本
OceanBase 数据库 V3.x、V4.x 版本。
解决方法及规避方式
重启异常 OBServer 节点。