本文介绍租户 ssstore_read IO 响应异常 Callback、Enqueue 性能优化。
详细说明
问题现象
租户 ssstore_read IO 吞吐量为 1GB/s～2GB/s 的负载下，ssstore_read IO 耗时为 7ms～10ms，而同期 OCP 主机监控显示 vg_obdata 读响应时延为 100μs～200μs，两者存在显著差异。
OCP 主机监控中 vg_obdata read 响应时延（对应采集项 node_disk_read_time_seconds_total）与 oswatch 抓取的 r_await 指标基本持平，表明磁盘设备层的响应时间无异常，可排除文件系统层 IO 等待因素的影响。
租户 ssstore 读 IO 耗时为全链路耗时，涵盖 IO 磁盘设备响应耗时、OS 文件系统耗时以及 OB 内核处理耗时。
分析优化过程
在集群 sys 租户（oceanbase）中生成 ASH Report 报告。
CALL DBMS_WORKLOAD_REPOSITORY.ASH Report( '2026-03-10 14:00:00', '2026-03-10 14:10:00', NULL, NULL, NULL, 'TEXT', NULL, NULL, 1002 );注意
在 OceanBase 数据库 V4.4.2 版本，需指定业务租户 ID 生成 ASH Report 报告。
查看 ASH Report 中的 Top IO Events 章节内容。
+---------------------+------------------------------------------------------------------------------------------------+--------------------+--------------------+--------------------+-----------------+--------------------+----------+--------------------+--------------------+--------------------+ | Node| Program Module Action / SQL ID| Plan Hash| IO Event Samples| %IO Event Samples| Top Event | IO Type| % EVENT| Enqueue Time(S)| Device Time(S)| Callback Time(S)| +---------------------+------------------------------------------------------------------------------------------------+--------------------+--------------------+--------------------+-----------------+--------------------+----------+--------------------+--------------------+--------------------+ | xx.xx.xx.xx:2882 | 19E114A39C246CCDA551221B5F50567E|15934910045500840308| 3920| 1.93%|db file data read| USER_IO| 1.93%| 1921.95| 170.22| 8268.85| | xx.xx.xx.xx:2882 | A6805371BE7F830F46D58A4B568F5CEA| 3290318591324336132| 2780| 1.37%|db file data read| USER_IO| 1.37%| 715.99| 61.77| 2123.47| | xx.xx.xx.xx:2882 | 8AF3EF71F1F89C68EF7719BD0D216D11| 2316111163021930497| 2670| 1.31%|db file data read| USER_IO| 1.31%| 1265.45| 62.88| 1619.73| | xx.xx.xx.xx:2882 | 15FAE97AF1F1A7FF8E9940601D5F38D4|16342883034819352967| 2640| 1.30%|db file data read| USER_IO| 1.30%| 845.49| 53.78| 1985.51| | xx.xx.xx.xx:2882 | 14BF728CCDB4FF670CF01AC5CEB6F6C7| 9865145744221640246| 2460| 1.21%|db file data read| USER_IO| 1.20%| 898.36| 54.54| 2267.72| | xx.xx.xx.xx:2882 | 19E114A39C246CCDA551221B5F50567E| 5823850713607507024| 2230| 1.10%|db file data read| USER_IO| 1.10%| 1564.23| 94.44| 1986.13| | xx.xx.xx.xx:2882 | 10EBBDF1153A9DB07F4A7B51AC0BB9DA|17012946199879217909| 2160| 1.06%|db file data read| USER_IO| 1.06%| 1420.10| 88.35| 3608.03| | xx.xx.xx.xx:2882 | 8AF3EF71F1F89C68EF7719BD0D216D11| 4156311742363319116| 980| 0.48%|db file data read| USER_IO| 0.48%| 11.34| 19.39| 1002.88| | xx.xx.xx.xx:2882 | 915E36D86EEA184E706F3EDC5F732716| 2329978838165517734| 930| 0.46%|db file data read| USER_IO| 0.46%| 298.05| 19.55| 817.54| | xx.xx.xx.xx:2882 | 8AF3EF71F1F89C68EF7719BD0D216D11| 2786336680839048234| 800| 0.39%|db file data read| USER_IO| 0.39%| 262.43| 23.17| 624.69| | xx.xx.xx.xx:2882 | 27004B460C9295AC7BDA20555E286226|12286895946054136810| 790| 0.39%|db file data read| USER_IO| 0.39%| 314.34| 18.34| 672.51| | xx.xx.xx.xx:2882 | 4C4EC743414E9FAC4F87037FD4368955|13547021336636461829| 750| 0.37%|db file data read| USER_IO| 0.37%| 172.46| 11.56| 580.93| | xx.xx.xx.xx:2882 | A7BD8F14738FE3B80D428F0939E4602B| 9346729163145940399| 670| 0.33%|db file data read| USER_IO| 0.33%| 460.19| 35.87| 514.90| | xx.xx.xx.xx:2882 | E1365E63299E3C03FE6E07F7605FE48F|16136479997846618721| 640| 0.31%|db file data read| USER_IO| 0.31%| 377.90| 19.77| 825.53| | xx.xx.xx.xx:2882 | BD8B76717898D01ACC02C574FC66DE1E|16009975959627027919| 620| 0.30%|db file data read| USER_IO| 0.30%| 193.03| 30.65| 601.35| | xx.xx.xx.xx:2882 | 31A8BBEB080FC658A4C4F7C5FB791264|14742276189199990932| 620| 0.30%|db file data read| USER_IO| 0.30%| 278.91| 8.76| 343.95| | xx.xx.xx.xx:2882 | 90475422BF57A1A11965C8F089A432F2| 2263864110369198877| 510| 0.25%|db file data read| USER_IO| 0.25%| 133.47| 9.38| 453.61| | xx.xx.xx.xx:2882 | 294343C801A31D1DD3CDEFC497225F84|16540031703057389164| 510| 0.25%|db file data read| USER_IO| 0.25%| 113.42| 16.44| 490.75| | xx.xx.xx.xx:2882 | 91B449762F346D317F1EE9F0807E1F36| 5738600337598561356| 440| 0.22%|db file data read| USER_IO| 0.22%| 153.29| 28.73| 425.49| | xx.xx.xx.xx:2882 | 8AF3EF71F1F89C68EF7719BD0D216D11|15334867199645859086| 430| 0.21%|db file data read| USER_IO| 0.21%| 39.65| 8.27| 424.40| | xx.xx.xx.xx:2882 | 46DA531745F05713D9ADC90F6F392273|13547021336636461829| 400| 0.20%|db file data read| USER_IO| 0.20%| 197.65| 5.63| 199.96| | xx.xx.xx.xx:2882 | D7B1EAE4C248580600B0BA14FA2BC827|11478612769567053043| 390| 0.19%|db file data read| USER_IO| 0.19%| 90.86| 8.65| 357.96| | xx.xx.xx.xx:2882 | C31849FF50D8E2299E4DEAB36C721531| 1928653344943955482| 360| 0.18%|db file data read| USER_IO| 0.18%| 147.02| 11.56| 329.82| | xx.xx.xx.xx:2882 | 3AE80CE344F28E47124C28BF5FF1D798| 2837811648834355979| 340| 0.17%|db file data read| USER_IO| 0.17%| 106.58| 6.03| 306.91| | xx.xx.xx.xx:2882 | BEC7389E7883A136C72DF18392CA5B29| 9397252551866160720| 300| 0.15%|db file data read| USER_IO| 0.15%| 327.75| 19.06| 188.73| | xx.xx.xx.xx:2882 | 79F27B0257CD6849902B59B3C754B865|14240837807115279646| 300| 0.15%|db file data read| USER_IO| 0.15%| 232.88| 10.94| 423.39| | xx.xx.xx.xx:2882 | 897D3F28579306D6F8D73AE13EEC3CBD| 3290318591324336132| 280| 0.14%|db file data read| USER_IO| 0.14%| 112.81| 7.88| 223.41| | xx.xx.xx.xx:2882 | C310136D61058DFBF479F7151D2731BF| 5627809337787546800| 280| 0.14%|db file data read| USER_IO| 0.14%| 114.18| 8.64| 170.24| | xx.xx.xx.xx:2882 | ED37DFA0EF25ABF83A18C06EA505A8C6| 4744763601351590283| 250| 0.12%|db file data read| USER_IO| 0.12%| 103.19| 8.19| 325.06| +---------------------+------------------------------------------------------------------------------------------------+--------------------+--------------------+--------------------+-----------------+--------------------+----------+--------------------+--------------------+--------------------+根据 ASH Report 中的 Top IO Events 章节内容分析可知 Callback Time 占比最高（部分条目达 8000+秒），主要事件为
db file data read，类型为 USER_IO。优化 Callback Time 占比高问题。
-- 未优化 Callback Time 占比高问题时 IO Callback 线程数 _io_callback_thread_count 参数信息 +-----------+-------+----------+----------------+----------+---------------------------+-----------+-------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+--------+---------+-------------------+ | tenant_id | zone | svr_type | svr_ip | svr_port | name | data_type | value | info | section | scope | source | edit_level | +-----------+-------+----------+----------------+----------+---------------------------+-----------+-------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+--------+---------+-------------------+ | 1002 | zone1 | observer | xx.xx.xx.xx | 2882 | _io_callback_thread_count | INT | 0 | The number of io callback threads. The default value is 0. Range: [0,128] in integer. If not specified, The number of threads is dynamically configured according to the memory size | OBSERVER | TENANT | DEFAULT | DYNAMIC_EFFECTIVE | | 1002 | zone1 | observer | xx.xx.xx.xx | 2882 | _io_callback_thread_count | INT | 0 | The number of io callback threads. The default value is 0. Range: [0,128] in integer. If not specified, The number of threads is dynamically configured according to the memory size | OBSERVER | TENANT | DEFAULT | DYNAMIC_EFFECTIVE | | 1002 | zone2 | observer | xx.xx.xx.xx | 2882 | _io_callback_thread_count | INT | 0 | The number of io callback threads. The default value is 0. Range: [0,128] in integer. If not specified, The number of threads is dynamically configured according to the memory size | OBSERVER | TENANT | DEFAULT | DYNAMIC_EFFECTIVE | | 1002 | zone2 | observer | xx.xx.xx.xx | 2882 | _io_callback_thread_count | INT | 0 | The number of io callback threads. The default value is 0. Range: [0,128] in integer. If not specified, The number of threads is dynamically configured according to the memory size | OBSERVER | TENANT | DEFAULT | DYNAMIC_EFFECTIVE | +-----------+-------+----------+----------------+----------+---------------------------+-----------+-------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+--------+---------+-------------------+ -- 优化 Callback Time 占比高问题将调大租户 IO Callback 线程数 _io_callback_thread_count 参数值 ALTER SYSTEM SET _io_callback_thread_count=64 TENANT='xxx';调大租户 IO Callback 线程数之后，进行业务跑批场景下的复测，查看 ASH Report Top IO Events 章节内容。
+---------------------+------------------------------------------------------------------------------------------------+--------------------+--------------------+--------------------+-----------------+--------------------+----------+--------------------+--------------------+--------------------+ | Node| Program Module Action / SQL ID| Plan Hash| IO Event Samples| %IO Event Samples| Top Event | IO Type| % EVENT| Enqueue Time(S)| Device Time(S)| Callback Time(S)| +---------------------+------------------------------------------------------------------------------------------------+--------------------+--------------------+--------------------+-----------------+--------------------+----------+--------------------+--------------------+--------------------+ | xx.xx.xx.xx:2882 | 9305012D582265DD70879E974F18BA1D| 3290318591324336132| 9740| 4.82%|db file data read| USER_IO| 4.82%| 9985.14| 79.70| 112.99| | xx.xx.xx.xx:2882 | C46434386C23972D24D602C68C7EDB1E| 4058911751418824560| 9660| 4.78%|db file data read| USER_IO| 4.78%| 16651.51| 114.78| 156.10| | xx.xx.xx.xx:2882 | A177BC1E462AD62E9716549BCE6851E4| 4058911751418824560| 9380| 4.64%|db file data read| USER_IO| 4.64%| 18840.13| 130.81| 159.03| | xx.xx.xx.xx:2882 | B8A4CB712652C81C823A8067D6300C3B|11357197997738832342| 7350| 3.63%|db file data read| USER_IO| 3.63%| 12817.65| 95.58| 116.96| | xx.xx.xx.xx:2882 | 2C89C66BC7D3F95C2ACDBDE3D6D81C54|11357197997738832342| 6970| 3.45%|db file data read| USER_IO| 3.45%| 11320.78| 133.47| 103.47| | xx.xx.xx.xx:2882 | 57BEB7FF336884C71314A4000146F671|11357197997738832342| 6340| 3.13%|db file data read| USER_IO| 3.13%| 10432.48| 92.61| 101.07| | xx.xx.xx.xx:2882 | D756B3DA54596EFBCBA5CB1ABB34F0E8| 3290318591324336132| 3600| 1.78%|db file data read| USER_IO| 1.78%| 3832.49| 35.30| 43.46| | xx.xx.xx.xx:2882 | 88B09EB48AB82F3A388E030B99E34675|11357197997738832342| 3300| 1.63%|db file data read| USER_IO| 1.63%| 6609.48| 85.33| 94.64| | xx.xx.xx.xx:2882 | CA6ECFEA017983902A9C9103D010EDFC| 3290318591324336132| 3240| 1.60%|db file data read| USER_IO| 1.60%| 3507.49| 25.48| 35.13| | xx.xx.xx.xx:2882 | C9143D461EA0A0DF7FD22431F153EBE1|10153848727941741313| 1900| 0.94%|db file data read| USER_IO| 0.94%| 2271.78| 17.40| 17.77| | xx.xx.xx.xx:2882 | 05789F787084A2EF7AE77D805171DEA8|13609123790938669898| 1210| 0.60%|db file data read| USER_IO| 0.60%| 1332.46| 12.42| 7.20| | xx.xx.xx.xx:2882 | 950B7B7251D560424696C2D4CAACEF8E|11939112550181076438| 1050| 0.52%|db file data read| USER_IO| 0.52%| 1167.85| 10.59| 7.10| | xx.xx.xx.xx:2882 | 9B0B048D1A1E0350E66649C641B96ED0| 6269093822223643688| 920| 0.46%|db file data read| USER_IO| 0.46%| 2199.53| 18.38| 8.57| | xx.xx.xx.xx:2882 | 9B9093C5FE70C360911D0A27D54738C6| 3290318591324336132| 480| 0.24%|db file data read| USER_IO| 0.24%| 496.83| 3.66| 2.60| | xx.xx.xx.xx:2882 | T1002_LogService/LogIOWorker/aggregate io task| 0| 420| 0.21%|palf write | SYSTEM_IO| 0.21%| 34.30| 296.85| 0.00| | xx.xx.xx.xx:2882 | T1002_LogService/LogIOWorker/aggregate io task| 0| 410| 0.20%|palf write | SYSTEM_IO| 0.20%| 70.25| 277.68| 0.00| | xx.xx.xx.xx:2882 | T1002_LogService/LogIOWorker/aggregate io task| 0| 330| 0.16%|palf write | SYSTEM_IO| 0.16%| 25.52| 246.83| 0.00| | xx.xx.xx.xx:2882 | CE65565351C31BB5622B770D176054EE| 7259655605471307834| 320| 0.16%|db file data read| USER_IO| 0.16%| 269.81| 34.24| 2.82| | xx.xx.xx.xx:2882 | 68D84F3DDF7742BD6D8405B647DFB913| 2388368257584294686| 310| 0.15%|db file data read| USER_IO| 0.15%| 346.19| 3.08| 4.96| | xx.xx.xx.xx:2882 | 743B176087C0050B04BB12E5EEF6D73D| 6296486517421659276| 250| 0.12%|db file data read| USER_IO| 0.12%| 352.29| 2.33| 2.35| | xx.xx.xx.xx:2882 | 08C5EC666A60D4E471EA2DFFA55A52F4| 131154037633687114| 240| 0.12%|db file data read| USER_IO| 0.12%| 532.15| 9.73| 9.74| | xx.xx.xx.xx:2882 | 8C3AB41DB4CBEAF8CDF34F389A683D60|14395013788055075194| 230| 0.11%|db file data read| USER_IO| 0.11%| 1370.01| 2.12| 2.93| | xx.xx.xx.xx:2882 | BB1DFA3323E8B8225CF8CAAAF227DE99| 1929772058942612463| 230| 0.11%|db file data read| USER_IO| 0.11%| 237.59| 1.62| 1.05| | xx.xx.xx.xx:2882 | T1002_LogService/LogIOWorker/aggregate io task| 0| 220| 0.11%|palf write | SYSTEM_IO| 0.11%| 26.03| 166.74| 0.00| | xx.xx.xx.xx:2882 | 31A8BBEB080FC658A4C4F7C5FB791264|14742276189199990932| 220| 0.11%|db file data read| USER_IO| 0.11%| 291.13| 2.13| 2.95| | xx.xx.xx.xx:2882 | 387CA1BFD696A5B7AF78868BE2604B12| 335559101225077327| 220| 0.11%|db file data read| USER_IO| 0.11%| 230.57| 1.71| 0.00| | xx.xx.xx.xx:2882 | 34CF9F520D0AAFA44463B16F9A661CC6|10283100810656172606| 170| 0.08%|db file data read| USER_IO| 0.08%| 237.91| 2.17| 2.41| | xx.xx.xx.xx:2882 | C6D3FC5DF85855955049E2E895E6CDEA| 335559101225077327| 170| 0.08%|db file data read| USER_IO| 0.08%| 179.60| 1.73| 0.00| | xx.xx.xx.xx:2882 | 03D25A62F526AB73C73370B747FBC6B8| 7259655605471307834| 160| 0.08%|db file data read| USER_IO| 0.08%| 144.03| 12.09| 0.00| | xx.xx.xx.xx:2882 | EE0EEB880C587BFB18F4328890FDE282| 7259655605471307834| 160| 0.08%|db file data read| USER_IO| 0.08%| 143.45| 13.11| 0.00| +---------------------+------------------------------------------------------------------------------------------------+--------------------+--------------------+--------------------+-----------------+--------------------+----------+--------------------+--------------------+--------------------+根据 ASH Report 中的 Top IO Events 章节内容分析可知 ssstore 读 IO 耗时未见明显改善，Callback Time 显著降低，但 Enqueue Time 成为新瓶颈，Enqueue Time 占比最高（部分条目达 10000+秒），主要事件仍为
db file data read，类型为 USER_IO。优化 Enqueue Time 占比高问题。
-- 未优化 Enqueue Time 占比高问题时 IO Scheduler 线程数 io_scheduler_thread_count 参数信息 +-------+----------+----------------+----------+---------------------------+-----------+-------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+---------+---------+------------------+---------------+-----------+ | zone | svr_type | svr_ip | svr_port | name | data_type | value | info | section | scope | source | edit_level | default_value | isdefault | +-------+----------+----------------+----------+---------------------------+-----------+-------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+---------+---------+------------------+---------------+-----------+ | zone1 | observer | xx.xx.xx.xx | 2882 | io_scheduler_thread_count | INT | 4 | Range: [1, 16] in integer. This parameter is used to adjust the number of threads in the I/O scheduler. The default value is 4. You can set it within the range of 1 to 16 to control the concurrency level of the I/O scheduler. | OBSERVER | CLUSTER | DEFAULT | STATIC_EFFECTIVE | 4 | 1 | | zone2 | observer | xx.xx.xx.xx | 2882 | io_scheduler_thread_count | INT | 4 | Range: [1, 16] in integer. This parameter is used to adjust the number of threads in the I/O scheduler. The default value is 4. You can set it within the range of 1 to 16 to control the concurrency level of the I/O scheduler. | OBSERVER | CLUSTER | DEFAULT | STATIC_EFFECTIVE | 4 | 1 | | zone1 | observer | xx.xx.xx.xx | 2882 | io_scheduler_thread_count | INT | 4 | Range: [1, 16] in integer. This parameter is used to adjust the number of threads in the I/O scheduler. The default value is 4. You can set it within the range of 1 to 16 to control the concurrency level of the I/O scheduler. | OBSERVER | CLUSTER | DEFAULT | STATIC_EFFECTIVE | 4 | 1 | | zone2 | observer | xx.xx.xx.xx | 2882 | io_scheduler_thread_count | INT | 4 | Range: [1, 16] in integer. This parameter is used to adjust the number of threads in the I/O scheduler. The default value is 4. You can set it within the range of 1 to 16 to control the concurrency level of the I/O scheduler. | OBSERVER | CLUSTER | DEFAULT | STATIC_EFFECTIVE | 4 | 1 | +-------+----------+----------------+----------+---------------------------+-----------+-------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+---------+---------+------------------+---------------+-----------+ -- 优化 Enqueue Time 占比高问题将调整 IO Scheduler 线程数（默认值为 4，属于 STATIC_EFFECTIVE 静态参数，需重启生效）io_scheduler_thread_count 参数值 ALTER SYSTEM SET io_scheduler_thread_count=16;
适用版本
OceanBase 数据库 V4.x 版本。
