问题现象
读临时文件时所使用的 IO buffer 来自 500 租户的一个固定大小内存区，标签为 TmpPageCache，每个 OBServer 单独设限，上限为 4G。正常情况下，当 SQL 生命周期结束后会释放。
在大查询、建索引等使用临时文件场景中，读取文件遇到 IO 失败或线程异常结束退出，callback 未能成功执行时，出现 READ IO 内存泄露。当泄漏一段时间后，可用的 READ IO 内存会逐渐减少，达到 4G 限制后，执行中间结果集较大的 SQL 或创建索引任务就可能出现读临时文件报申请内存失败的报错。
您可以通过如下两种方式确认内存模块占用情况。
通过日志确认。
grep '\[MEMORY]' observer.log | grep 'TmpPageCache'通过虚拟表确认。
obclient> select * from __all_virtual_memory_info where label like '%TmpPageCache%';
详细报错堆栈如下。
[2023-10-10 00:15:19.650506] ERROR [STORAGE] alloc_io_buf (ob_tmp_file_cache.cpp:263) [115827][0][YB420A5701C2-0005E70197BDFDF6-0-0] [lt=6] [dc=0] Fail to allocate memory, (ret=-4013) BACKTRACE:0xf5f6bd5 0xf49023a 0x52e63ce 0x5326a03 0x52e236e 0x5326877 0x5326497 0xab0b4ee 0xab0b393 0x528698e 0x5286892 0x5295dba 0x5294b45 0x5294151 0x66f66fd 0xf370132 0xf36ff8e 0xf60ab3e
[2023-10-10 00:15:19.650517] ERROR [STORAGE] alloc_io_buf (ob_tmp_file_cache.cpp:263) [115859][0][YB420A5701C2-0005E70197BDFDF6-0-0] [lt=8] [dc=0] Fail to allocate memory, (ret=-4013) BACKTRACE:0xf5f6bd5 0xf49023a 0x52e63ce 0x5326a03 0x52e236e 0x5326877 0x5326497 0xab0b4ee 0xab0b393 0x528698e 0x5286892 0x5295dba 0x5294b45 0x5294151 0x66f66fd 0xf370132 0xf36ff8e 0xf60ab3e
[2023-10-10 00:15:19.650560] WARN [COMMON] alloc_aligned_io_buf (ob_io_struct.cpp:622) [115827][0][YB420A5701C2-0005E70197BDFDF6-0-0] [lt=45] [dc=0] callback allocate memory failed(ret=-4013, io_info_={fd_:{first_id:0, second_id:1805051}, offset_:2023424, size_:8192, flag_:{mode_:0, category_:1, wait_event_id_:13, reserved_:0}, label_:TMP_FILE_READ, write_buf_:null, callback_:0x7f122266e2a8})
[2023-10-10 00:15:19.650569] WARN [COMMON] alloc_aligned_io_buf (ob_io_struct.cpp:622) [115859][0][YB420A5701C2-0005E70197BDFDF6-0-0] [lt=44] [dc=0] callback allocate memory failed(ret=-4013, io_info_={fd_:{first_id:0, second_id:1805051}, offset_:1835008, size_:57344, flag_:{mode_:0, category_:1, wait_event_id_:13, reserved_:0}, label_:TMP_FILE_READ, write_buf_:null, callback_:0x7f122266e2c0})
[2023-10-10 00:15:19.650586] WARN [COMMON] prepare (ob_io_struct.cpp:650) [115827][0][YB420A5701C2-0005E70197BDFDF6-0-0] [lt=7] [dc=0] alloc io buffer for read failed(ret=-4013, *this={is_inited_:true, is_finished_:false, is_canceled_:false, has_estimated_:false, io_info_:{fd_:{first_id:0, second_id:1805051}, offset_:2023424, size_:8192, flag_:{mode_:0, category_:1, wait_event_id_:13, reserved_:0}, label_:TMP_FILE_READ, write_buf_:null, callback_:0x7f122266e2a8}, deadline_ts_:1696868119650493, control_block_:0x7f10701ae400, raw_buf_:null, io_buf_:null, io_offset_:2023424, io_size_:8192, complete_size_:0, time_log_:{begin_ts_:1696868119650492, enqueue_ts_:1696868119650493, dequeue_ts_:1696868119650501, submit_ts_:0, return_ts_:0, callback_enqueue_ts_:0, callback_dequeue_ts_:0, callback_finish_ts_:0, end_ts_:0}, channel_:0x7f1554dad010, ref_cnt_:3, out_ref_cnt_:1, trace_id_:YB420A5701C2-0005E70197BDFDF6-0-0, ret_code_:{io_ret_:0, fs_errno_:0}, retry_count_:0, copied_callback_:0x7ee0ea26cc50})
[2023-10-10 00:15:19.650591] WARN [COMMON] prepare (ob_io_struct.cpp:650) [115859][0][YB420A5701C2-0005E70197BDFDF6-0-0] [lt=6] [dc=0] alloc io buffer for read failed(ret=-4013, *this={is_inited_:true, is_finished_:false, is_canceled_:false, has_estimated_:false, io_info_:{fd_:{first_id:0, second_id:1805051}, offset_:1835008, size_:57344, flag_:{mode_:0, category_:1, wait_event_id_:13, reserved_:0}, label_:TMP_FILE_READ, write_buf_:null, callback_:0x7f122266e2c0}, deadline_ts_:1696868119650505, control_block_:0x7f10701ae460, raw_buf_:null, io_buf_:null, io_offset_:1835008, io_size_:57344, complete_size_:0, time_log_:{begin_ts_:1696868119650504, enqueue_ts_:1696868119650505, dequeue_ts_:1696868119650512, submit_ts_:0, return_ts_:0, callback_enqueue_ts_:0, callback_dequeue_ts_:0, callback_finish_ts_:0, end_ts_:0}, channel_:0x7f1554dae710, ref_cnt_:3, out_ref_cnt_:1, trace_id_:YB420A5701C2-0005E70197BDFDF6-0-0, ret_code_:{io_ret_:0, fs_errno_:0}, retry_count_:0, copied_callback_:0x7ee0ea26d1f0})
[2023-10-10 00:15:19.650605] WARN [COMMON] pop_and_submit (ob_io_struct.cpp:2421) [115827][0][YB420A5701C2-0005E70197BDFDF6-0-0] [lt=8] [dc=0] prepare io request failed(ret=-4013, *req={is_inited_:true, is_finished_:false, is_canceled_:false, has_estimated_:false, io_info_:{fd_:{first_id:0, second_id:1805051}, offset_:2023424, size_:8192, flag_:{mode_:0, category_:1, wait_event_id_:13, reserved_:0}, label_:TMP_FILE_READ, write_buf_:null, callback_:0x7f122266e2a8}, deadline_ts_:1696868119650493, control_block_:0x7f10701ae400, raw_buf_:null, io_buf_:null, io_offset_:2023424, io_size_:8192, complete_size_:0, time_log_:{begin_ts_:1696868119650492, enqueue_ts_:1696868119650493, dequeue_ts_:1696868119650501, submit_ts_:0, return_ts_:0, callback_enqueue_ts_:0, callback_dequeue_ts_:0, callback_finish_ts_:0, end_ts_:0}, channel_:0x7f1554dad010, ref_cnt_:3, out_ref_cnt_:1, trace_id_:YB420A5701C2-0005E70197BDFDF6-0-0, ret_code_:{io_ret_:0, fs_errno_:0}, retry_count_:0, copied_callback_:0x7ee0ea26cc50})
[2023-10-10 00:15:19.650607] WARN [COMMON] pop_and_submit (ob_io_struct.cpp:2421) [115859][0][YB420A5701C2-0005E70197BDFDF6-0-0] [lt=5] [dc=0] prepare io request failed(ret=-4013, *req={is_inited_:true, is_finished_:false, is_canceled_:false, has_estimated_:false, io_info_:{fd_:{first_id:0, second_id:1805051}, offset_:1835008, size_:57344, flag_:{mode_:0, category_:1, wait_event_id_:13, reserved_:0}, label_:TMP_FILE_READ, write_buf_:null, callback_:0x7f122266e2c0}, deadline_ts_:1696868119650505, control_block_:0x7f10701ae460, raw_buf_:null, io_buf_:null, io_offset_:1835008, io_size_:57344, complete_size_:0, time_log_:{begin_ts_:1696868119650504, enqueue_ts_:1696868119650505, dequeue_ts_:1696868119650512, submit_ts_:0, return_ts_:0, callback_enqueue_ts_:0, callback_dequeue_ts_:0, callback_finish_ts_:0, end_ts_:0}, channel_:0x7f1554dae710, ref_cnt_:3, out_ref_cnt_:1, trace_id_:YB420A5701C2-0005E70197BDFDF6-0-0, ret_code_:{io_ret_:0, fs_errno_:0}, retry_count_:0, copied_callback_:0x7ee0ea26d1f0})
[2023-10-10 00:15:19.650686] WARN [COMMON] wait (ob_io_struct.cpp:258) [115228][0][YB420A5701C2-0005E70197BDFDF6-0-0] [lt=9] [dc=0] IO error, (ret=-4013, *req_={is_inited_:true, is_finished_:true, is_canceled_:false, has_estimated_:false, io_info_:{fd_:{first_id:0, second_id:1805051}, offset_:2023424, size_:8192, flag_:{mode_:0, category_:1, wait_event_id_:13, reserved_:0}, label_:TMP_FILE_READ, write_buf_:null, callback_:0x7f122266e2a8}, deadline_ts_:1696868119650493, control_block_:0x7f10701ae400, raw_buf_:null, io_buf_:null, io_offset_:2023424, io_size_:8192, complete_size_:0, time_log_:{begin_ts_:1696868119650492, enqueue_ts_:1696868119650493, dequeue_ts_:1696868119650501, submit_ts_:0, return_ts_:0, callback_enqueue_ts_:0, callback_dequeue_ts_:0, callback_finish_ts_:0, end_ts_:1696868119650611}, channel_:0x7f1554dad010, ref_cnt_:1, out_ref_cnt_:1, trace_id_:YB420A5701C2-0005E70197BDFDF6-0-0, ret_code_:{io_ret_:-4013, fs_errno_:0}, retry_count_:0, copied_callback_:0x7ee0ea26cc50})
[2023-10-10 00:15:19.650701] WARN [STORAGE] wait (ob_storage_file.cpp:314) [115228][0][YB420A5701C2-0005E70197BDFDF6-0-0] [lt=12] [dc=0] fail to wait block io, may be retry(macro_id_=[0-1805051](ver=1,mode=0,seq=43165422), ret=-4013, timeout_ms=175496)
[2023-10-10 00:15:19.650708] WARN [STORAGE] wait (ob_storage_file.cpp:316) [115228][0][YB420A5701C2-0005E70197BDFDF6-0-0] [lt=5] [dc=0] fail to handle aio error(ret=-4013, macro_id_=[0-1805051](ver=1,mode=0,seq=43165422), timeout_ms=175496)
[2023-10-10 00:15:19.650716] WARN [STORAGE] wait (ob_tmp_file.cpp:124) [115228][0][YB420A5701C2-0005E70197BDFDF6-0-0] [lt=7] [dc=0] fail to wait tmp read io(ret=-4013)
问题原因
在异常场景下，如 IO 失败或线程异常结束 callback 未执行的时，该线程所申请的用于读临时文件的 buffer 会发生泄漏，无法释放。当泄漏一段时间后，可用的内存逐渐减少，4G 内存耗尽后，执行中间结果集较大的 SQL 或创建索引任务就可能出现读临时文件报申请内存失败的报错。
问题的风险及影响
导致该 OBServer 后续的大查询和建索引任务遇到 -4013 报错。
影响版本
OceanBase 数据库 V3.1.2 BP11（oceanbase-3.1.2-111000052023010412）、V3.2.3 BP10（oceanbase-3.2.3.3-110000092023091219）、V3.2.4 BP6（oceanbase-3.2.4.6-106000062023110109）及之前的 BP 版本。
解决方法及规避方式
- 触发该问题后，需要重启 TmpPageCache 内存满的 OBServer 进程。
- 升级到问题已修复版本，目前已修复的版本包含 OceanBase 数据库 V3.2.4 BP7（oceanbase-3.2.4.7-107000012023113010）及之后版本。
