问题描述
1004 租户归档情况（11 月 6 号查询，start_scn_display 还卡在 11 月 3 号）。
MySQL [oceanbase]> select USED_PIECE_ID,* from oceanbase.cdb_ob_archivelog
where tenant_id=1004 \G
*************************** 1. row ***************************
USED_PIECE_ID: 77
TENANT_ID: 1004
DEST_ID: 1009
ROUND_ID: 6
INCARNATION: 1
DEST_NO: 0
STATUS: DOING
START_SCN: 1698993729762606386
START_SCN_DISPLAY: 2023-11-03 14:42:09.762606
CHECKPOINT_SCN: 1698993729762606386
CHECKPOINT_SCN_DISPLAY: 2023-11-03 14:42:09.762606
COMPATIBLE: 1
BASE_PIECE_ID: 74
USED_PIECE_ID: 77
PIECE_SWITCH_INTERVAL: 86400000000
UNIT_SIZE: 1
COMPRESSION: none
INPUT_BYTES: 25357733034
INPUT_BYTES_DISPLAY: 23.62GB
OUTPUT_BYTES: 25357733034
OUTPUT_BYTES_DISPLAY: 23.62GB
COMPRESSION_RATIO: 1.00
DELETED_INPUT_BYTES: 0
DELETED_INPUT_BYTES_DISPLAY: 0.00MB
DELETED_OUTPUT_BYTES: 0
DELETED_OUTPUT_BYTES_DISPLAY: 0.00MB
COMMENT:
PATH: file:///arch/nfs/xxx/xxx/archive
1 row in set (0.03 sec)
问题版本
OceanBase 数据库 V4.x 版本。
问题原因
OBServer 节点 NFS 未挂载或者权限有问题。
解决方法
确认租户日志流归档情况。
MySQL [oceanbase]> select *, scn_to_timestamp(checkpoint_scn) from oceanbase.__all_virtual_ls_log_archive_progress where tenant_id = 1004 and piece_id = 77;输出结果如下：
+-----------+---------+----------+----------+-------+----------------------------+----------------------------+-------------+---------------------+-------------+-------------+---------------------+--------+---------+-------------+-------------+--------------+----------------------------------+ | tenant_id | dest_id | round_id | piece_id | ls_id | gmt_create | gmt_modified | incarnation | start_scn | min_lsn | max_lsn | checkpoint_scn | status | file_id | file_offset | input_bytes | output_bytes | scn_to_timestamp(checkpoint_scn) | +-----------+---------+----------+----------+-------+----------------------------+----------------------------+-------------+---------------------+-------------+-------------+---------------------+--------+---------+-------------+-------------+--------------+----------------------------------+ | 1004 | 1009 | 6 | 77 | 1 | 2023-11-06 14:42:11.608909 | 2023-11-06 21:58:38.224490 | 1 | 1699252929762606000 | 16012563081 | 16093678274 | 1699279118131157674 | DOING | 240 | 55638722 | 81115193 | 81115193 | 2023-11-06 21:58:38.131157 | | 1004 | 1009 | 6 | 77 | 1001 | 2023-11-06 14:42:11.611017 | 2023-11-06 21:58:38.225544 | 1 | 1699252929762606000 | 7115535700 | 7145075019 | 1699279118082764862 | DOING | 107 | 29539319 | 29539319 | 29539319 | 2023-11-06 21:58:38.082764 | +-----------+---------+----------+----------+-------+----------------------------+----------------------------+-------------+---------------------+-------------+-------------+---------------------+--------+---------+-------------+-------------+--------------+----------------------------------+ 2 rows in set (0.03 sec)确认租户日志流状态。
MySQL [oceanbase]> select * from oceanbase.__all_virtual_ls_status where tenant_id=1004;输出结果如下：
+-----------+-------+----------------------------+----------------------------+------------------+--------------------+--------+-------------+---------------+--------------+ | tenant_id | ls_id | gmt_create | gmt_modified | init_member_list | b_init_member_list | status | ls_group_id | unit_group_id | primary_zone | +-----------+-------+----------------------------+----------------------------+------------------+--------------------+--------+-------------+---------------+--------------+ | 1004 | 1 | 2023-08-25 21:05:55.646509 | 2023-08-25 21:05:55.688068 | | | NORMAL | 0 | 0 | zone3 | | 1004 | 1001 | 2023-08-25 21:06:04.771557 | 2023-08-25 21:06:04.808467 | | | NORMAL | 1001 | 1003 | zone3 | | 1004 | 1002 | 2023-08-25 21:06:04.812671 | 2023-08-25 21:06:05.843835 | | | NORMAL | 1002 | 1004 | zone3 | +-----------+-------+----------------------------+----------------------------+------------------+--------------------+--------+-------------+---------------+--------------+ 3 rows in set (0.02 sec)确认日志流 leader 节点。
对比第 1 和第 2 步，可以看出 1004 租户的少了一个 1002 号日志流归档的记录，归档日志由日志流 leader 负责的。
MySQL [oceanbase]> select * from oceanbase.gv$ob_log_stat where tenant_id=1004 and ls_id=1002;输出结果如下：
+-----------+-------+--------------+----------+----------+-------------+---------------------------------+-------------+-------------------------------------------------------------+-------------------+---------+------------+-----------+---------------------+------------+---------------------+------------+---------------------+--------------------+---------------+ | TENANT_ID | LS_ID | SVR_IP | SVR_PORT | ROLE | PROPOSAL_ID | CONFIG_VERSION | ACCESS_MODE | PAXOS_MEMBER_LIST | PAXOS_REPLICA_NUM | IN_SYNC | BASE_LSN | BEGIN_LSN | BEGIN_SCN | END_LSN | END_SCN | MAX_LSN | MAX_SCN | ARBITRATION_MEMBER | DEGRADED_LIST | +-----------+-------+--------------+----------+----------+-------------+---------------------------------+-------------+-------------------------------------------------------------+-------------------+---------+------------+-----------+---------------------+------------+---------------------+------------+---------------------+--------------------+---------------+ | 1004 | 1002 | 10.xxx.xx.15 | 2882 | LEADER | 10 | {proposal_id:10, config_seq:11} | APPEND | 10.xxx.xx.11:2882:1,10.xxx.xx.13:2882:1,10.xxx.xx.15:2882:1 | 3 | YES | 7918362624 | 0 | 1692968764811129415 | 8000469638 | 1699320789901413260 | 8000469638 | 1699320789901413260 | | | | 1004 | 1002 | 10.xxx.xx.11 | 2882 | FOLLOWER | 10 | {proposal_id:10, config_seq:11} | APPEND | 10.xxx.xx.11:2882:1,10.xxx.xx.13:2882:1,10.xxx.xx.15:2882:1 | 3 | YES | 7918362624 | 0 | 1692968764811129415 | 8000469638 | 1699320789901413260 | 8000469638 | 1699320789901413260 | | | | 1004 | 1002 | 10.xxx.xx.13 | 2882 | FOLLOWER | 10 | {proposal_id:10, config_seq:11} | APPEND | 10.xxx.xx.11:2882:1,10.xxx.xx.13:2882:1,10.xxx.xx.15:2882:1 | 3 | YES | 7918362624 | 0 | 1692968764811129415 | 8000469638 | 1699320789901413260 | 8000469638 | 1699320789901413260 | | | +-----------+-------+--------------+----------+----------+-------------+---------------------------------+-------------+-------------------------------------------------------------+-------------------+---------+------------+-----------+---------------------+------------+---------------------+------------+---------------------+--------------------+---------------+ 3 rows in set (0.05 sec确认日志。
在 10.xxx.xx.15 节点最新的 observer.log 日志里可以看到 目录权限有问题，进一步
df -h命令确认发现没有挂载的 NFS（网络文件系统）。
挂载 NFS（网络文件系统）后，1004 租户的日志归档进度恢复正常。
