--- title: "基于常态化 Memleak 的内存泄露诊断机制 - OceanBase 数据库 V4.2.0 | OceanBase 文档中心" description: 基于常态化 Memleak 的内存泄露诊断机制 OceanBase 数据库通过 Memleak 工具来诊断内存泄漏，其工作原理是记录跟踪模块的内存分配堆栈，如果某一堆栈的累积次数在一定时段里持续增加，则该堆栈对应的内存分配上下文可能发生了内存泄漏。然而，开启 Memleak 的时机一般是内存发生泄漏之后，如果 Mem… --- 切换语言 - 中文站 - 简体中文 - International - English - 日本站 - 日本語 文档反馈![](https://mdn.alipayobjects.com/huamei_22khvb/afts/img/A*P8CuR4UJ_FkAAAAAAAAAAAAADiGDAQ/original) OceanBase 数据库分布式版 - V 4.2.0 # 基于常态化 Memleak 的内存泄露诊断机制 更新时间：2024-03-06 16:06:44 [编辑](https://github.com/oceanbase/oceanbase-doc/edit/V4.2.0/zh-CN/700.reference/200.system-management/700.memory-management/700.use-memleak.md) OceanBase 数据库通过 Memleak 工具来诊断内存泄漏，其工作原理是记录跟踪模块的内存分配堆栈，如果某一堆栈的累积次数在一定时段里持续增加，则该堆栈对应的内存分配上下文可能发生了内存泄漏。然而，开启 Memleak 的时机一般是内存发生泄漏之后，如果 Memleak 开启后内存不再泄漏，则无法跟踪到发生泄漏的模块，只能等待复现。因此，我们需要使用常态化的 Memleak，实时保存内存分配信息的采样，根据采样的结果分析可能发生内存泄漏的模块。 常态化 Memleak 通过配置项 `_min_malloc_sample_interval` 和 `_max_malloc_sample_interval` 来控制对内存分配信息的采样率。 | 配置项 | 描述 | 取值范围 | 默认值 | | --- | --- | --- | --- | | _min_malloc_sample_interval | 相邻两次采样之间 ob_malloc 的最小次数 | [1,10000] | 16 | | _max_malloc_sample_interval | 相邻两次采样之间 ob_malloc 的最大次数 | [1,10000] | 256 | 通过设置配置项来控制对内存分配信息的采样率可分为三种模式： - 零采样模式：该模式下，系统会停止记录内存分配信息。 ```shell obclient> ALTER SYSTEM SET _min_malloc_sample_interval=10000; Query OK, 0 rows affected obclient> ALTER SYSTEM SET _max_malloc_sample_interval=10000; Query OK, 0 rows affected ``` - 全采样模式：该模式下，系统会记录全部的内存分配信息。在此模式下，OceanBase 数据库会出现很大程度的性能回退。 ```shell obclient> ALTER SYSTEM SET _min_malloc_sample_interval=1; Query OK, 0 rows affected obclient> ALTER SYSTEM SET _max_malloc_sample_interval=1; Query OK, 0 rows affected ``` - 常态化模式：该模式下，系统会根据设置的采样间隔，记录内存分配信息的采样。 ```shell obclient> ALTER SYSTEM SET _min_malloc_sample_interval=16; Query OK, 0 rows affected obclient> ALTER SYSTEM SET _max_malloc_sample_interval=256; Query OK, 0 rows affected ``` #### 注意 - 在 OceanBase 数据库中，名称以 "_" 开头的配置项称为隐藏配置项，仅供开发人员在故障排查或紧急运维时使用。 - 在设置配置项的值时，需要满足如下要求：`_min_malloc_sample_interval <= _max_malloc_sample_interval`。 - 配置项 `_min_malloc_sample_interval` 和 `_max_malloc_sample_interval` 不建议用户随意修改，如果修改不当会导致集群性能变差。 ## 使用常态化 Memleak 进行内存泄漏诊断 1. 使用 `root` 用户登录到集群的 `sys` 租户。 2. 通过虚拟表 `__all_virtual_malloc_sample_info` 查询怀疑发生内存泄漏的模块的内存分配信息。 下面以模块 `glibc_malloc` 为例，查询其内存分配信息： ```shell obclient [oceanbase]> SELECT * FROM __all_virtual_malloc_sample_info WHERE mod_name='glibc_malloc' ORDER BY alloc_bytes DESC limit 10; +-------------+----------+-----------+--------+--------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+-------------+-------------+ | svr_ip | svr_port | tenant_id | ctx_id | mod_name | back_trace | ctx_name | alloc_count | alloc_bytes | +-------------+----------+-----------+--------+--------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+-------------+-------------+ | xx.xx.xx.xx | 25224 | 500 | 22 | glibc_malloc | 0x5f8bf23 0x5c5a5bd 0x5c52ec3 0x198ed1c6 0x5d788ba 0x1d8e28a5 0x1d8e2919 0x187f31a6 0x188a867a 0x188aa126 0x15c53d3f 0x16b06ae2 0x16b369ef 0x16b32d7e 0x15c8bcf8 0x15c8606a | GLIBC | 3 | 9437232 | | xx.xx.xx.xx | 25225 | 500 | 22 | glibc_malloc | 0x5f8bf23 0x5c5a5bd 0x5c52ec3 0x198ed1c6 0x5d788ba 0x1d8e28a5 0x1d8e2919 0x187f31a6 0x188a867a 0x188aa126 0x15c53d3f 0x16b06ae2 0x16b369ef 0x16b32d7e 0x15c8bcf8 0x15c8606a | GLIBC | 3 | 9437232 | | xx.xx.xx.xx | 25226 | 500 | 22 | glibc_malloc | 0x5f8bf23 0x5c5a5bd 0x5c52ec3 0x198ed1c6 0x5d788ba 0x1d8e28a5 0x1d8e2919 0x187f31a6 0x188a867a 0x188aa126 0x15c53d3f 0x16b06ae2 0x16b369ef 0x16b32d7e 0x15c8bcf8 0x15c8606a | GLIBC | 3 | 9437232 | | xx.xx.xx.xx | 25224 | 500 | 22 | glibc_malloc | 0x5f8bf23 0x5c5a5bd 0x5c52ec3 0x198ed1c6 0x5d788ba 0x19dee05c 0x19e00077 0xb49139a 0x5f6d17f 0x7fc3a6c85445 0x5f6abe1 0x0 0x0 0x0 0x0 0x0 | GLIBC | 1 | 4194320 | | xx.xx.xx.xx | 25225 | 500 | 22 | glibc_malloc | 0x5f8bf23 0x5c5a5bd 0x5c52ec3 0x198ed1c6 0x5d788ba 0x19dee05c 0x19e00077 0xb49139a 0x5f6d17f 0x7ff8c02b3445 0x5f6abe1 0x0 0x0 0x0 0x0 0x0 | GLIBC | 1 | 4194320 | | xx.xx.xx.xx | 25226 | 500 | 22 | glibc_malloc | 0x5f8bf23 0x5c5a5bd 0x5c52ec3 0x198ed1c6 0x5d788ba 0x19dee05c 0x19e00077 0xb49139a 0x5f6d17f 0x7f06f1b63445 0x5f6abe1 0x0 0x0 0x0 0x0 0x0 | GLIBC | 1 | 4194320 | | xx.xx.xx.xx | 25224 | 500 | 22 | glibc_malloc | 0x5f8bf23 0x5c5a5bd 0x5c52ec3 0x198ed1c6 0x5d788ba 0x1d8e28a5 0x1d8e2919 0x187f31a6 0x15c76adf 0xb49e963 0xb4915b6 0x5f6d17f 0x7fc3a6c85445 0x5f6abe1 0x0 0x0 | GLIBC | 1 | 3145744 | | xx.xx.xx.xx | 25225 | 500 | 22 | glibc_malloc | 0x5f8bf23 0x5c5a5bd 0x5c52ec3 0x198ed1c6 0x5d788ba 0x1d8e28a5 0x1d8e2919 0x187f31a6 0x15c76adf 0xb49e963 0xb4915b6 0x5f6d17f 0x7ff8c02b3445 0x5f6abe1 0x0 0x0 | GLIBC | 1 | 3145744 | | xx.xx.xx.xx | 25226 | 500 | 22 | glibc_malloc | 0x5f8bf23 0x5c5a5bd 0x5c52ec3 0x198ed1c6 0x5d788ba 0x1d8e28a5 0x1d8e2919 0x187f31a6 0x15c76adf 0xb49e963 0xb4915b6 0x5f6d17f 0x7f06f1b63445 0x5f6abe1 0x0 0x0 | GLIBC | 1 | 3145744 | | xx.xx.xx.xx | 25224 | 500 | 22 | glibc_malloc | 0x5f8bf23 0x5c5a5bd 0x5c52ec3 0x198ed1c6 0x5d788ba 0x1d8e28a5 0x1d8e28d9 0x161a7a6b 0x161a7148 0x161aa227 0x64f4472 0x64f003c 0x64b9e68 0x64b1d45 0x6520720 0x651e8db | GLIBC | 2 | 131120 | +-------------+----------+-----------+--------+--------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+-------------+-------------+ 10 rows in set (0.012 sec) ``` 通过查询虚拟表，分析得知，`0x5f8bf23 0x5c5a5bd 0x5c52ec3 0x198ed1c6 0x5d788ba 0x1d8e28a5 0x1d8e2919 0x187f31a6 0x188a867a 0x188aa126 0x15c53d3f 0x16b06ae2 0x16b369ef 0x16b32d7e 0x15c8bcf8 0x15c8606a` 这个堆栈的 `alloc_bytes` 最大，怀疑其可能发生了内存泄漏。 字段说明： | 列名 | 类型 | 描述 | | --- | --- | --- | | svr_ip | varchar:MAX_IP_ADDR_LENGTH | IP 地址 | | svr_port | bigint(20) | 端口号 | | tenant_id | bigint(20) | 租户 ID | | ctx_id | bigint(20) | CTX ID | | mod_name | varchar:OB_MAX_CHAR_LENGTH | 模块名称 | | back_trace | varchar:DEFAULT_BUF_LENGTH | 内存分配的堆栈 | | ctx_name | varchar:OB_MAX_CHAR_LENGTH | CTX 名称 | | alloc_count | bigint(20) | 内存分配次数 | | alloc_bytes | bigint(20) | 内存分配的总大小 | 3. 根据虚拟表查询的结果，分析 `alloc_bytes` 较大的堆栈 `back_trace`: ```shell $addr2line -pCfe bin/observer 0x5f8bf23 0x5c5a5bd 0x5c52ec3 0x198ed1c6 0x5d788ba 0x1d8e28a5 0x1d8e2919 0x187f31a6 0x188a867a 0x188aa126 0x15c53d3f 0x16b06ae2 0x16b369ef 0x16b32d7e 0x15c8bcf8 0x15c8606a void* oceanbase::lib::ObTenantCtxAllocator::common_alloc(long, oceanbase::lib::ObMemAttr const&, oceanbase::lib::ObTenantCtxAllocator&, oceanbase::lib::ObjectMgr&) at /home/distcc/tmp/./deps/oblib/src/lib/alloc/ob_tenant_ctx_allocator.cpp:414 (discriminator 2) oceanbase::lib::ObTenantCtxAllocator::alloc(long, oceanbase::lib::ObMemAttr const&) at /home/distcc/tmp/./deps/oblib/src/lib/alloc/ob_tenant_ctx_allocator.cpp:35 oceanbase::lib::ObMallocAllocator::alloc(long, oceanbase::lib::ObMemAttr const&) at ??:? oceanbase::common::ob_malloc(long, oceanbase::lib::ObMemAttr const&) at ./build_sanity/deps/oblib/src/lib/../../../../../deps/oblib/src/lib/allocator/ob_malloc.h:39 ob_malloc_hook(unsigned long, void const*) at ??:? operator new(unsigned long) at ??:? operator new[](unsigned long, std::nothrow_t const&) at ??:? oceanbase::share::schema::ObSchemaMgrCache::init(long, oceanbase::share::schema::ObSchemaMgrCache::Mode) at ./build_sanity/src/share/./src/share/schema/ob_schema_mgr_cache.cpp:153 oceanbase::share::schema::ObSchemaStore::init(unsigned long, long, long) at ./build_sanity/src/share/./src/share/schema/ob_schema_store.cpp:29 oceanbase::share::schema::ObSchemaStoreMap::create(unsigned long, long, long) at ./build_sanity/src/share/./src/share/schema/ob_schema_store.cpp:131 (discriminator 2) oceanbase::share::schema::ObMultiVersionSchemaService::init_multi_version_schema_struct(unsigned long) at ./build_sanity/src/share/./src/share/schema/ob_multi_version_schema_service.cpp:218 (discriminator 52) oceanbase::share::schema::ObServerSchemaService::update_schema_mgr(oceanbase::common::ObISQLClient&, oceanbase::share::schema::ObRefreshSchemaStatus const&, oceanbase::share::schema::ObSchemaMgr&, long, oceanbase::share::schema::ObServerSchemaService::AllSchemaKeys&) at ./build_sanity/src/share/./src/share/schema/ob_server_schema_service.cpp:4232 (discriminator 54) oceanbase::share::schema::ObServerSchemaService::refresh_increment_schema(oceanbase::share::schema::ObRefreshSchemaStatus const&) at ./build_sanity/src/share/./src/share/schema/ob_server_schema_service.cpp:5663 (discriminator 4) oceanbase::share::schema::ObServerSchemaService::refresh_schema(oceanbase::share::schema::ObRefreshSchemaStatus const&) at ./build_sanity/src/share/./src/share/schema/ob_server_schema_service.cpp:5288 oceanbase::share::schema::ObMultiVersionSchemaService::refresh_tenant_schema(unsigned long) at ??:? operator() at ./build_sanity/src/share/./src/share/schema/ob_multi_version_schema_service.cpp:2328 (discriminator 4) ``` 用 `addr2line` 命令解析 `alloc_bytes` 最大的堆栈，联系技术支持人员协助分析该堆栈是否发生内存泄漏。 上一篇 下一篇 ![有帮助](https://gw.alipayobjects.com/mdn/ob_asset/afts/img/A*y6ocSqN8cqsAAAAAAAAAAAAAARQnAQ)![无帮助](https://gw.alipayobjects.com/mdn/ob_asset/afts/img/A*BG9IQJyLHF8AAAAAAAAAAAAAARQnAQ)![反馈](https://gw.alipayobjects.com/mdn/ob_asset/afts/img/A*eTWdQKCRKHwAAAAAAAAAAAAAARQnAQ)[AI](https://www.oceanbase.com/obi) 咨询热线