根因分析场景：clog 盘满 clog_disk_full

更新时间：2026-04-13 17:11:14

适用场景

本文适用于 clog 盘满的场景，依赖内部表数据进行分析。

目前支持 OceanBase 数据库 V4.0.0.0 及之后的版本。使用 obdiag 时需根据选择在 ~/.obdiag/config.yml 文件中配置集群信息，或在命令中通过 --config 选项配置集群信息。

注意

由于内部数据存在及时性，所以仅支持对当前时的数据进行分析。

可导入环境变量

变量名	是否必选	数据类型	默认值	说明
since	否	string	"30m"	指定待分析日志的时间段，默认为 `30m` 表示分析过去 30 分钟内的日志。

使用示例

配置分析场景，默认执行示例如下：

obdiag rca run --scene=clog_disk_full

结果示例

根据如下输出可发现节点中的 1005 日志流存在回放问题，需要进一步追踪。

+-------------------------------------------------------------------------------------------------------------+
|                                                    record                                                   |
+------+------------------------------------------------------------------------------------------------------+
| step | info                                                                                                 |
+------+------------------------------------------------------------------------------------------------------+
|  1   | check error tenant_ls_data is {'tenant_id': 1xxx, 'ls_id': 1005, 'ip': 'xxx.xxx.xxx.xxx', 'port':      |
|      | 2882}                                                                                                |
|  2   | __check_checkpoint                                                                                   |
|  3   | find log_disk_full about checkpoint in                                                               |
|      | ./rca//obdiag_clog_disk_full_20240719104714/tenant_id_1xxx/ls_id_1005/checkpoint/                    |
|  4   | is_clog_checkpoint_stuck is True                                                                     |
|  5   | the log is [2024-07-19 10:45:08.043682] INFO  [STORAGE] update_clog_checkpoint                       |
|      | (ob_checkpoint_executor.cpp:158) [52704][T1xxx_CKClogDis][T1xxx][Y0-0000000000000000-0-0] [lt=0]     |
|      | [CHECKPOINT] clog checkpoint no change(checkpoint_scn={val:1720543850356918166, v:0},                |
|      | checkpoint_scn_in_ls_meta={val:1720543850356918166, v:0}, ls_id={id:1005},                           |
|      | service_type="TRANS_SERVICE")                                                                        |
|  6   | stuck_service_type is TRANS_SERVICE                                                                  |
|  7   | __check_checkpoint end                                                                               |
|  8   | __get_min_ckpt_type start                                                                            |
|  9   | get min ckpt type is on [2024-07-19 10:45:59.748107] INFO  [STORAGE.TRANS] get_rec_scn               |
|      | (ob_ls_tx_service.cpp:507) [52704][T1xxx_CKClogDis][T1xxx][Y0-0000000000000000-0-0] [lt=0]           |
|      | [CHECKPOINT] ObLSTxService::get_rec_scn(common_checkpoint_type="MDS_TABLE_TYPE",                     |
|      | common_checkpoints_[min_rec_scn_common_checkpoint_type_index]={this:0x7f4a69ac8510, is_inited:true,  |
|      | freezing_scn:{val:1720543850382118208, v:0}, ls:{ls_meta:{tenant_id:1xxx, ls_id:{id:1005},           |
|      | ls_create_status:1, clog_checkpoint_scn:{val:1720543850356918166, v:0},                              |
|      | clog_base_lsn:{lsn:2367456215040}, rebuild_seq:1, migration_status:0, gc_state_:1,                   |
|      | offline_scn_:{val:18446744073709551615, v:3}, restore_status:{status:8},                             |
|      | replayable_point:{val:1720662756177146368, v:0},                                                     |
|      | tablet_change_checkpoint_scn:{val:1720543765175444272, v:0}, all_id_meta:{id_meta:[{limited_id:1,    |
|      | latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1,                                       |
|      | latest_log_ts:{val:18446744073709551615, v:3}}, {limited_id:1,                                       |
|      | latest_log_ts:{val:18446744073709551615, v:3}}]}, transfer_scn:{val:1720536600653472428, v:0},       |
|      | rebuild_info:{status:{status:0}, type:{type:0}}}, switch_epoch:1, log_handler:{role:2,               |
|      | proposal_id:17, palf_env_:0x7f1de657c030, is_in_stop_state_:false, is_inited_:true, id_:1005},       |
|      | restore_handler:{is_inited:true, is_in_stop_state:false, id:1005, proposal_id:9223372036854775807,   |
|      | role:2, parent:null, context:{issue_task_num:0, issue_version:-1, last_fetch_ts:-1,                  |
|      | max_submit_lsn:{lsn:18446744073709551615}, max_fetch_lsn:{lsn:18446744073709551615},                 |
|      | max_fetch_scn:{val:18446744073709551615, v:3}, error_context:{ret_code:0,                            |
|      | trace_id:Y0-0000000000000000-0-0, error_type:1048580, err_lsn:{lsn:18446744073709551615}},           |
|      | task_count:0}, restore_context:{seek_done:false, lsn:{lsn:18446744073709551615}}}, is_inited:true,   |
|      | tablet_gc_handler:{tablet_persist_trigger:2, is_inited:true}, startup_transfer_info:{ls_id:{id:0},   |
|      | transfer_start_scn:{val:18446744073709551615, v:3}}}}, min_rec_scn={val:1720543850356918166, v:0},   |
|      | ls_id_={id:1005})                                                                                    |
|  10  | min_checkpoint_tx_log_type is MDS_TABLE_TYPE                                                         |
|  11  | min_checkpoint_scn is 1720543850356918166                                                            |
|  12  | check_min_ckpt_type is True                                                                          |
|  13  | __get_min_ckpt_type end                                                                              |
|  14  | __check_replay_stuck start                                                                           |
|  15  | check_replay_stuck is True. the line: [2024-07-19 10:47:33.367691] INFO  [CLOG]                      |
|      | get_min_unreplayed_log_info (ob_replay_status.cpp:984)                                               |
|      | [4600][T1xxx_TenantWea][T1xxx][Y0-0000000000000000-0-0] [lt=14]                                      |
|      | get_min_unreplayed_log_info(lsn={lsn:2367477760943}, scn={val:1720543850382118209, v:0},             |
|      | this={ls_id_:{id:1005}, is_enabled_:true, is_submit_blocked_:false, role_:2,                         |
|      | err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false,    |
|      | err_ts_:0, err_ret_:0}, ref_cnt_:4, post_barrier_lsn_:{lsn:2367477760943}, pending_task_count_:1,    |
|      | submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1721357253367659,                  |
|      | err_info_:{has_fatal_error_:false, fail_ts_:1721213879495661, fail_cost_:176528190,                  |
|      | ret_code_:-4023}}, next_to_submit_lsn_:{lsn:2367477763353},                                          |
|      | next_to_submit_scn_:{val:1720543850382118210, v:0}, base_lsn_:{lsn:2332293316608},                   |
|      | base_scn_:{val:1720468881546872775, v:0}, iterator_:{iterator_impl:{buf_:0x7f00f0805000,             |
|      | next_round_pread_size:2121728, curr_read_pos:387999, curr_read_buf_start_pos:0,                      |
|      | curr_read_buf_end_pos:2121728, log_storage_:{IteratorStorage:{start_lsn:{lsn:2367477375410},         |
|      | end_lsn:{lsn:2367479497138}, read_buf:{buf_len_:2125824, buf_:0x7f00f0805000}, block_size:67104768,  |
|      | log_storage_:0x7f2b1a5bf2f0}, IteratorStorageType::"DiskIteratorStorage"},                           |
|      | curr_entry_is_raw_write:true, curr_entry_size:99, prev_entry_scn:{val:1720543850382118210, v:0},     |
|      | curr_entry:{LogEntryHeader:{magic:19528, version:1, log_size:67, scn_:{val:1720543850382118210,      |
|      | v:0}, data_checksum:927461962, flag:1}}, init_mode_version:14, accumulate_checksum:1210730574,       |
|      | curr_entry_is_padding:0, padding_entry_size:99, padding_entry_scn:{val:1720543850382118210,          |
|      | v:0}}}}})                                                                                            |
|  16  | get min unreplayed log info is [2024-07-19 10:47:33.367691] INFO  [CLOG] get_min_unreplayed_log_info |
|      | (ob_replay_status.cpp:984) [4600][T1xxx_TenantWea][T1xxx][Y0-0000000000000000-0-0] [lt=14]           |
|      | get_min_unreplayed_log_info(lsn={lsn:2367477760943}, scn={val:1720543850382118209, v:0},             |
|      | this={ls_id_:{id:1005}, is_enabled_:true, is_submit_blocked_:false, role_:2,                         |
|      | err_info_:{lsn_:{lsn:18446744073709551615}, scn_:{val:0, v:0}, log_type_:0, is_submit_err_:false,    |
|      | err_ts_:0, err_ret_:0}, ref_cnt_:4, post_barrier_lsn_:{lsn:2367477760943}, pending_task_count_:1,    |
|      | submit_log_task_:{ObReplayServiceSubmitTask:{type_:1, enqueue_ts_:1721357253367659,                  |
|      | err_info_:{has_fatal_error_:false, fail_ts_:1721213879495661, fail_cost_:176528190,                  |
|      | ret_code_:-4023}}, next_to_submit_lsn_:{lsn:2367477763353},                                          |
|      | next_to_submit_scn_:{val:1720543850382118210, v:0}, base_lsn_:{lsn:2332293316608},                   |
|      | base_scn_:{val:1720468881546872775, v:0}, iterator_:{iterator_impl:{buf_:0x7f00f0805000,             |
|      | next_round_pread_size:2121728, curr_read_pos:387999, curr_read_buf_start_pos:0,                      |
|      | curr_read_buf_end_pos:2121728, log_storage_:{IteratorStorage:{start_lsn:{lsn:2367477375410},         |
|      | end_lsn:{lsn:2367479497138}, read_buf:{buf_len_:2125824, buf_:0x7f00f0805000}, block_size:67104768,  |
|      | log_storage_:0x7f2b1a5bf2f0}, IteratorStorageType::"DiskIteratorStorage"},                           |
|      | curr_entry_is_raw_write:true, curr_entry_size:99, prev_entry_scn:{val:1720543850382118210, v:0},     |
|      | curr_entry:{LogEntryHeader:{magic:19528, version:1, log_size:67, scn_:{val:1720543850382118210,      |
|      | v:0}, data_checksum:927461962, flag:1}}, init_mode_version:14, accumulate_checksum:1210730574,       |
|      | curr_entry_is_padding:0, padding_entry_size:99, padding_entry_scn:{val:1720543850382118210,          |
|      | v:0}}}}})                                                                                            |
|  17  | log_time - replay_scn_time : 2024-07-19 10:47:33.367691 - 2024-07-10 00:50:50.382118                 |
|  18  | datetime.timedelta(minutes=0.5): 0:00:30                                                             |
|  19  | log_time - replay_scn_time > datetime.timedelta(minutes=0.5) is True                                 |
|  20  | log_time:{0}, replay_scn_time:{1}                                                                    |
|  21  | check_replay_stuck is True                                                                           |
|  22  | check_replay_stuck is True. Please check replay status                                               |
|  23  | __check_replay_stuck end                                                                             |
|  24  | __check_dump_stuck start                                                                             |
|  25  | check_dump_stuck is False                                                                            |
|  26  | __check_dump_stuck end                                                                               |
|  27  | __check_data_disk_full start                                                                         |
|  28  | check_data_disk_full is False                                                                        |
|  29  | __check_data_disk_full end                                                                           |
|  30  | __check_too_many_sstable start                                                                       |
|  31  | check_too_many_sstable is False                                                                      |
|  32  | __check_too_many_sstable end                                                                         |
|  33  | check end                                                                                            |
+------+------------------------------------------------------------------------------------------------------+
The suggest: min_checkpoint_tx_log_type is MDS_TABLE_TYPE. please check it.

参考博客

OceanBase诊断调优 (十二) ——clog盘满问题排查