日志报错-4009 io error 的原因和解决方法

[2024-08-30 22:30:58.321133] ERROR detect_palf_hang_failure_ (ob_failure_detector.cpp:356) [143509][T1_Occam][T1][Y0-0000000000000000-0-0] [lt=12][errcode=-4392] disk is hung(msg="clog disk may be hung, add failure event", clog_disk_hang_event={type:PROCESS HANG, module:LOG, info:clog disk hang, sen: 0, level:FATAL})
[2024-08-30 22:30:58.349092] ERROR issue_dba_error (ob_log.cpp:1875) [143746][T1_L0_G0][T1][xxxxx-xxxxx-xxxxx-xxxxx] [lt=50][errcode=-4388] Unexpected internal error happen, please checkout the internal errcode(errcode=-4009, file="ob_tx_data_functor.cpp", line_no=391, info="unexpected io error")

[2024-08-30 22:30:55.103188] INFO  [COMMON] print_io_status (ob_io_manager.cpp:1407) [142910][IO_TUNING0][T0][Y0-0000000000000000-0-0] [lt=29] [IO STATUS](tenant_id=500, io_status_log=group_id: 0, group_name: OTHER_GROUPS, mode:  read, size:       1.00, iops:  8192.00, rt: 1857279.00; )

[2024-08-30 22:30:58.435426] WDIAG [COMMON] estimate (ob_io_define.cpp:1021) [143396][IO_HEALTH0][T0][Y0-0000000000000000-0-0] [lt=47][errcode=-4016] io request wait too long(req_={is_inited_:true, is_finished_:true, is_canceled_:false, has_estimated_:true, io_info_:{tenant_id_:500, fd_:{first_id:295, second_id:291, device_handle:0xfffc9502a080}, offset_:1004069, size_:4096, flag_:{mode:"READ", group_id_:0, wait_event_id_:1, is_sync_:false, is_unlimited_:true, reserved_:0, is_detect_:true, is_time_detect_:true}, buf_:null, callback_:null}, deadline_ts_:0, sender_index_:0, control_block_:0xfffc94645678, raw_buf_:0xfffb917f9880, io_buf_:0xfffb917fa000, io_offset_:1003520, io_size_:8192, complete_size_:8192, time_log_:{begin_ts_:1725028254959265, enqueue_ts_:0, dequeue_ts_:0, submit_ts_:1725028254959269, return_ts_:1725028258435400, callback_enqueue_ts_:0, callback_dequeue_ts_:0, callback_finish_ts_:0, end_ts_:1725028258435405}, channel_:0xfffc94f912e0, ref_cnt_:1, out_ref_cnt_:1, trace_id_:Y0-0000000000000000-0-0, ret_code_:{io_ret_:0, fs_errno_:0}, retry_count_:0, callback_buf_size_:0, copied_callback_:null, tenant_io_mgr_:{ptr:0xfffc947a6250}}, prepare_delay=0, schedule_queue_delay=0, submit_delay=600000000, device_delay=3476131, start_callback_delay=0, callback_queue_delay=0, callback_process_delay=0, finish_notify_delay=5, request_delay=3476140)