首批通过分布式安全可靠测评，为关键业务系统打造
基于常态化 Memleak 的内存泄露诊断机制
更新时间：2024-03-06 16:06:44
编辑
分享
OceanBase 数据库通过 Memleak 工具来诊断内存泄漏，其工作原理是记录跟踪模块的内存分配堆栈，如果某一堆栈的累积次数在一定时段里持续增加，则该堆栈对应的内存分配上下文可能发生了内存泄漏。然而，开启 Memleak 的时机一般是内存发生泄漏之后，如果 Memleak 开启后内存不再泄漏，则无法跟踪到发生泄漏的模块，只能等待复现。因此，我们需要使用常态化的 Memleak，实时保存内存分配信息的采样，根据采样的结果分析可能发生内存泄漏的模块。
常态化 Memleak 通过配置项 _min_malloc_sample_interval 和 _max_malloc_sample_interval 来控制对内存分配信息的采样率。
| 配置项 | 描述 | 取值范围 | 默认值 |
|---|---|---|---|
| _min_malloc_sample_interval | 相邻两次采样之间 ob_malloc 的最小次数 | [1,10000] | 16 |
| _max_malloc_sample_interval | 相邻两次采样之间 ob_malloc 的最大次数 | [1,10000] | 256 |
通过设置配置项来控制对内存分配信息的采样率可分为三种模式：
零采样模式：该模式下，系统会停止记录内存分配信息。
obclient> ALTER SYSTEM SET _min_malloc_sample_interval=10000; Query OK, 0 rows affected obclient> ALTER SYSTEM SET _max_malloc_sample_interval=10000; Query OK, 0 rows affected全采样模式：该模式下，系统会记录全部的内存分配信息。在此模式下，OceanBase 数据库会出现很大程度的性能回退。
obclient> ALTER SYSTEM SET _min_malloc_sample_interval=1; Query OK, 0 rows affected obclient> ALTER SYSTEM SET _max_malloc_sample_interval=1; Query OK, 0 rows affected常态化模式：该模式下，系统会根据设置的采样间隔，记录内存分配信息的采样。
obclient> ALTER SYSTEM SET _min_malloc_sample_interval=16; Query OK, 0 rows affected obclient> ALTER SYSTEM SET _max_malloc_sample_interval=256; Query OK, 0 rows affected
注意
- 在 OceanBase 数据库中，名称以 "_" 开头的配置项称为隐藏配置项，仅供开发人员在故障排查或紧急运维时使用。
- 在设置配置项的值时，需要满足如下要求：
_min_malloc_sample_interval <= _max_malloc_sample_interval。 - 配置项
_min_malloc_sample_interval和_max_malloc_sample_interval不建议用户随意修改，如果修改不当会导致集群性能变差。
使用常态化 Memleak 进行内存泄漏诊断
使用
root用户登录到集群的sys租户。通过虚拟表
__all_virtual_malloc_sample_info查询怀疑发生内存泄漏的模块的内存分配信息。下面以模块
glibc_malloc为例，查询其内存分配信息：obclient [oceanbase]> SELECT * FROM __all_virtual_malloc_sample_info WHERE mod_name='glibc_malloc' ORDER BY alloc_bytes DESC limit 10; +-------------+----------+-----------+--------+--------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+-------------+-------------+ | svr_ip | svr_port | tenant_id | ctx_id | mod_name | back_trace | ctx_name | alloc_count | alloc_bytes | +-------------+----------+-----------+--------+--------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+-------------+-------------+ | xx.xx.xx.xx | 25224 | 500 | 22 | glibc_malloc | 0x5f8bf23 0x5c5a5bd 0x5c52ec3 0x198ed1c6 0x5d788ba 0x1d8e28a5 0x1d8e2919 0x187f31a6 0x188a867a 0x188aa126 0x15c53d3f 0x16b06ae2 0x16b369ef 0x16b32d7e 0x15c8bcf8 0x15c8606a | GLIBC | 3 | 9437232 | | xx.xx.xx.xx | 25225 | 500 | 22 | glibc_malloc | 0x5f8bf23 0x5c5a5bd 0x5c52ec3 0x198ed1c6 0x5d788ba 0x1d8e28a5 0x1d8e2919 0x187f31a6 0x188a867a 0x188aa126 0x15c53d3f 0x16b06ae2 0x16b369ef 0x16b32d7e 0x15c8bcf8 0x15c8606a | GLIBC | 3 | 9437232 | | xx.xx.xx.xx | 25226 | 500 | 22 | glibc_malloc | 0x5f8bf23 0x5c5a5bd 0x5c52ec3 0x198ed1c6 0x5d788ba 0x1d8e28a5 0x1d8e2919 0x187f31a6 0x188a867a 0x188aa126 0x15c53d3f 0x16b06ae2 0x16b369ef 0x16b32d7e 0x15c8bcf8 0x15c8606a | GLIBC | 3 | 9437232 | | xx.xx.xx.xx | 25224 | 500 | 22 | glibc_malloc | 0x5f8bf23 0x5c5a5bd 0x5c52ec3 0x198ed1c6 0x5d788ba 0x19dee05c 0x19e00077 0xb49139a 0x5f6d17f 0x7fc3a6c85445 0x5f6abe1 0x0 0x0 0x0 0x0 0x0 | GLIBC | 1 | 4194320 | | xx.xx.xx.xx | 25225 | 500 | 22 | glibc_malloc | 0x5f8bf23 0x5c5a5bd 0x5c52ec3 0x198ed1c6 0x5d788ba 0x19dee05c 0x19e00077 0xb49139a 0x5f6d17f 0x7ff8c02b3445 0x5f6abe1 0x0 0x0 0x0 0x0 0x0 | GLIBC | 1 | 4194320 | | xx.xx.xx.xx | 25226 | 500 | 22 | glibc_malloc | 0x5f8bf23 0x5c5a5bd 0x5c52ec3 0x198ed1c6 0x5d788ba 0x19dee05c 0x19e00077 0xb49139a 0x5f6d17f 0x7f06f1b63445 0x5f6abe1 0x0 0x0 0x0 0x0 0x0 | GLIBC | 1 | 4194320 | | xx.xx.xx.xx | 25224 | 500 | 22 | glibc_malloc | 0x5f8bf23 0x5c5a5bd 0x5c52ec3 0x198ed1c6 0x5d788ba 0x1d8e28a5 0x1d8e2919 0x187f31a6 0x15c76adf 0xb49e963 0xb4915b6 0x5f6d17f 0x7fc3a6c85445 0x5f6abe1 0x0 0x0 | GLIBC | 1 | 3145744 | | xx.xx.xx.xx | 25225 | 500 | 22 | glibc_malloc | 0x5f8bf23 0x5c5a5bd 0x5c52ec3 0x198ed1c6 0x5d788ba 0x1d8e28a5 0x1d8e2919 0x187f31a6 0x15c76adf 0xb49e963 0xb4915b6 0x5f6d17f 0x7ff8c02b3445 0x5f6abe1 0x0 0x0 | GLIBC | 1 | 3145744 | | xx.xx.xx.xx | 25226 | 500 | 22 | glibc_malloc | 0x5f8bf23 0x5c5a5bd 0x5c52ec3 0x198ed1c6 0x5d788ba 0x1d8e28a5 0x1d8e2919 0x187f31a6 0x15c76adf 0xb49e963 0xb4915b6 0x5f6d17f 0x7f06f1b63445 0x5f6abe1 0x0 0x0 | GLIBC | 1 | 3145744 | | xx.xx.xx.xx | 25224 | 500 | 22 | glibc_malloc | 0x5f8bf23 0x5c5a5bd 0x5c52ec3 0x198ed1c6 0x5d788ba 0x1d8e28a5 0x1d8e28d9 0x161a7a6b 0x161a7148 0x161aa227 0x64f4472 0x64f003c 0x64b9e68 0x64b1d45 0x6520720 0x651e8db | GLIBC | 2 | 131120 | +-------------+----------+-----------+--------+--------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+-------------+-------------+ 10 rows in set (0.012 sec)通过查询虚拟表，分析得知，
0x5f8bf23 0x5c5a5bd 0x5c52ec3 0x198ed1c6 0x5d788ba 0x1d8e28a5 0x1d8e2919 0x187f31a6 0x188a867a 0x188aa126 0x15c53d3f 0x16b06ae2 0x16b369ef 0x16b32d7e 0x15c8bcf8 0x15c8606a这个堆栈的alloc_bytes最大，怀疑其可能发生了内存泄漏。字段说明：
列名 类型 描述 svr_ip varchar:MAX_IP_ADDR_LENGTH IP 地址 svr_port bigint(20) 端口号 tenant_id bigint(20) 租户 ID ctx_id bigint(20) CTX ID mod_name varchar:OB_MAX_CHAR_LENGTH 模块名称 back_trace varchar:DEFAULT_BUF_LENGTH 内存分配的堆栈 ctx_name varchar:OB_MAX_CHAR_LENGTH CTX 名称 alloc_count bigint(20) 内存分配次数 alloc_bytes bigint(20) 内存分配的总大小 根据虚拟表查询的结果，分析
alloc_bytes较大的堆栈back_trace:$addr2line -pCfe bin/observer 0x5f8bf23 0x5c5a5bd 0x5c52ec3 0x198ed1c6 0x5d788ba 0x1d8e28a5 0x1d8e2919 0x187f31a6 0x188a867a 0x188aa126 0x15c53d3f 0x16b06ae2 0x16b369ef 0x16b32d7e 0x15c8bcf8 0x15c8606a void* oceanbase::lib::ObTenantCtxAllocator::common_alloc<oceanbase::lib::ObjectMgr>(long, oceanbase::lib::ObMemAttr const&, oceanbase::lib::ObTenantCtxAllocator&, oceanbase::lib::ObjectMgr&) at /home/distcc/tmp/./deps/oblib/src/lib/alloc/ob_tenant_ctx_allocator.cpp:414 (discriminator 2) oceanbase::lib::ObTenantCtxAllocator::alloc(long, oceanbase::lib::ObMemAttr const&) at /home/distcc/tmp/./deps/oblib/src/lib/alloc/ob_tenant_ctx_allocator.cpp:35 oceanbase::lib::ObMallocAllocator::alloc(long, oceanbase::lib::ObMemAttr const&) at ??:? oceanbase::common::ob_malloc(long, oceanbase::lib::ObMemAttr const&) at ./build_sanity/deps/oblib/src/lib/../../../../../deps/oblib/src/lib/allocator/ob_malloc.h:39 ob_malloc_hook(unsigned long, void const*) at ??:? operator new(unsigned long) at ??:? operator new[](unsigned long, std::nothrow_t const&) at ??:? oceanbase::share::schema::ObSchemaMgrCache::init(long, oceanbase::share::schema::ObSchemaMgrCache::Mode) at ./build_sanity/src/share/./src/share/schema/ob_schema_mgr_cache.cpp:153 oceanbase::share::schema::ObSchemaStore::init(unsigned long, long, long) at ./build_sanity/src/share/./src/share/schema/ob_schema_store.cpp:29 oceanbase::share::schema::ObSchemaStoreMap::create(unsigned long, long, long) at ./build_sanity/src/share/./src/share/schema/ob_schema_store.cpp:131 (discriminator 2) oceanbase::share::schema::ObMultiVersionSchemaService::init_multi_version_schema_struct(unsigned long) at ./build_sanity/src/share/./src/share/schema/ob_multi_version_schema_service.cpp:218 (discriminator 52) oceanbase::share::schema::ObServerSchemaService::update_schema_mgr(oceanbase::common::ObISQLClient&, oceanbase::share::schema::ObRefreshSchemaStatus const&, oceanbase::share::schema::ObSchemaMgr&, long, oceanbase::share::schema::ObServerSchemaService::AllSchemaKeys&) at ./build_sanity/src/share/./src/share/schema/ob_server_schema_service.cpp:4232 (discriminator 54) oceanbase::share::schema::ObServerSchemaService::refresh_increment_schema(oceanbase::share::schema::ObRefreshSchemaStatus const&) at ./build_sanity/src/share/./src/share/schema/ob_server_schema_service.cpp:5663 (discriminator 4) oceanbase::share::schema::ObServerSchemaService::refresh_schema(oceanbase::share::schema::ObRefreshSchemaStatus const&) at ./build_sanity/src/share/./src/share/schema/ob_server_schema_service.cpp:5288 oceanbase::share::schema::ObMultiVersionSchemaService::refresh_tenant_schema(unsigned long) at ??:? operator() at ./build_sanity/src/share/./src/share/schema/ob_multi_version_schema_service.cpp:2328 (discriminator 4)用
addr2line命令解析alloc_bytes最大的堆栈，联系技术支持人员协助分析该堆栈是否发生内存泄漏。
