--- title: OceanBase 数据库 V4.0 版本迁移 & 复制 & Rebuild 排查指南-OceanBase数据库使用指南 description: 了解OceanBase数据库在实际应用中关于 OceanBase 数据库 V4.0 版本迁移 & 复制 & Rebuild 排查指南相关的常见问题和使用技巧，帮助您快速解决 OceanBase 数据库 V4.0 版本迁移 & 复制 & Rebuild 排查指南的难题。 --- 切换语言 - 中文站 - 简体中文 - International - English - 日本站 - 日本語 划线反馈 # OceanBase 数据库 V4.0 版本迁移 & 复制 & Rebuild 排查指南 更新时间：2024-08-19 07:51 适用版本： V4.0.x 内容类型：Troubleshoot 本文汇总了迁移/复制/Rebuild 的主要流程，以及相关的内部表/日志信息，便于快速定位问题。 ## 相关内部表/视图 - 视图 `DBA_OB_ROOTSERVICE_EVENT_HISTORY` 。 关注 `event`，`source` 和 `destination`，如果表中有对应条目，说明 RS 已经调度的相应任务。其中 `value6` 可以看到任务的执行结果，`ret` 表示该次任务成功与否，`elapsed` 表示该次任务的执行时间，如果失败，可能会有 `ret_comment` 字段直接反应失败在哪一步。 ```shell OceanBase(root@oceanbase)>select * from oceanbase.DBA_OB_ROOTSERVICE_EVENT_HISTORY where module like "%disaster%" and value1 = 1003 and value2 = 1; ``` 输出结果如下： ```shell +----------------------------+-------------------+--------------------------------+-----------+--------+-------+--------+---------+------------------------------------+--------+----------------------+---------------+-------------------------------------------------------------------------------------------------------------------+----------------+------------------------------------------------------+------------------------------------------------------+--------------+-------------+ | TIMESTAMP | MODULE | EVENT | NAME1 | VALUE1 | NAME2 | VALUE2 | NAME3 | VALUE3 | NAME4 | VALUE4 | NAME5 | VALUE5 | NAME6 | VALUE6 | EXTRA_INFO | RS_SVR_IP | RS_SVR_PORT | +----------------------------+-------------------+--------------------------------+-----------+--------+-------+--------+---------+------------------------------------+--------+----------------------+---------------+-------------------------------------------------------------------------------------------------------------------+----------------+------------------------------------------------------+------------------------------------------------------+--------------+-------------+ | 2022-09-26 19:37:11.661502 | disaster_recovery | start_remove_ls_paxos_replica | tenant_id | 1003 | ls_id | 1 | task_id | xxxxxxxxxxxxx-xxxxxxxxxxxxxxxx-x-x | leader | "xx.xxx.xx.xx:xxxxx" | remove_server | {server:"xx.xxx.xx.xx:xxxxx", timestamp:1, flag:0, replica_type:0, region:"default_region", memstore_percent:100} | comment | remove redundant paxos replica according to locality | | xx.xxx.xx.xx | xxxxx | | 2022-09-26 19:37:11.733052 | disaster_recovery | finish_remove_ls_paxos_replica | tenant_id | 1003 | ls_id | 1 | task_id | xxxxxxxxxxxxx-xxxxxxxxxxxxxxxx-x-x | leader | "xx.xxx.xx.xx:xxxxx" | remove_server | {server:"xx.xxx.xx.xx:xxxxx", timestamp:1, flag:0, replica_type:0, region:"default_region", memstore_percent:100} | execute_result | ret:0; elapsed:71571; | remove redundant paxos replica according to locality | xx.xxx.xx.xx | xxxxx | +----------------------------+-------------------+--------------------------------+-----------+--------+-------+--------+---------+------------------------------------+--------+----------------------+---------------+-------------------------------------------------------------------------------------------------------------------+----------------+------------------------------------------------------+------------------------------------------------------+--------------+-------------+ 2 rows in set (0.02 sec) ``` - 视图 `DBA_OB_SERVER_EVENT_HISTORY` 记录了server 上的所有事件，包括 迁移、复制、rebuild 等事件。关注调度记录和 `is_failed` 。 以迁移为例，unit 从 xxx.xxx.xxx.xxx:12348 迁移到 1xxx.xxx.xxx.xxx:12347，在视图 `DBA_OB_SERVER_EVENT_HISTORY` 中观察以下事件： 1. 迁移任务在目的端是否已经被调度。 2. 迁移过程中各任务的执行情况。 如果迁移任务卡住，可以通过该表观察目前迁移任务的执行阶段排查。 ```shell # 排查 SQL 。 # 获取已经被 rs 调度但还没有完成的任务列表（tenant_id, ls_id）。 # 第一个 event 关键字根据任务类型替换。 (select value1, value2 from oceanbase.DBA_OB_ROOTSERVICE_EVENT_HISTORY where event like '%add_ls%') except (select value1, value2 from oceanbase.DBA_OB_SERVER_EVENT_HISTORY where module like 'storage_ha' and event like '%finish_complete%'); ``` ## 获取报错的迁移/复制任务 1. 通过视图 `DBA_OB_SERVER_EVENT_HISTORY` 获取事件历史记录信息。 ```shell obclient> select * from oceanbase.DBA_OB_SERVER_EVENT_HISTORY where event like '%migrat%' and name6 like '%fail%' and value6=1; Empty set (0.044 sec) ``` 2. 通过视图 `DBA_OB_SERVER_EVENT_HISTORY` 获取事件历史记录所属的模块的信息。 ```shell obclient> select * from oceanbase.DBA_OB_SERVER_EVENT_HISTORY where module like 'storage_ha' and event like '%schedule%'; ``` 输出结果如下： ```shell +----------------------------+-------------+----------+------------+-----------------------------+-------+-----------+----------+---------------------+-------+---------------------+-------+--------+-------+--------+-------+--------+------------+ | gmt_create | svr_ip | svr_port | module | event | name1 | value1 | name2 | value2 | name3 | value3 | name4 | value4 | name5 | value5 | name6 | value6 | extra_info | +----------------------------+-------------+----------+------------+-----------------------------+-------+-----------+----------+---------------------+-------+---------------------+-------+--------+-------+--------+-------+--------+------------+ | 2022-06-17 14:31:31.110597 | xx.xxx.x.xx | 12347 | storage_ha | schedule_ls_migration start | ls_id | {id:1001} | data_src | "xx.xxx.x.xx:xxxxx" | dest | "xx.xxx.x.xx:xxxxx" | | | | | | | | +----------------------------+-------------+----------+------------+-----------------------------+-------+-----------+----------+---------------------+-------+---------------------+-------+--------+-------+--------+-------+--------+------------+ ``` 3. 通过视图 `DBA_OB_SERVER_EVENT_HISTORY` 获取事件历史记录所属的模块的第一个，第二个事件属性值的信息。 ```shell obclient> select * from oceanbase.DBA_OB_SERVER_EVENT_HISTORY where module like 'storage_ha' and value1 = 1002 and value2=1001; ``` 输出结果如下： ```shell +----------------------------+-------------+----------+------------+---------------------------------+-----------+--------+-------+--------+-------+---------------------+-------+---------------------+-----------+------------------------------------+-------------------+--------+---------------+ | gmt_create | svr_ip | svr_port | module | event | name1 | value1 | name2 | value2 | name3 | value3 | name4 | value4 | name5 | value5 | name6 | value6 | extra_info | +----------------------------+-------------+----------+------------+---------------------------------+-----------+--------+-------+--------+-------+---------------------+-------+---------------------+-----------+------------------------------------+-------------------+--------+---------------+ | 2022-06-17 14:31:34.831234 | xx.xxx.x.xx | xxxxx | storage_ha | initial_migration_task | tenant_id | 1002 | ls_id | 1001 | src | "xx.xxx.x.xx:xxxxx" | dst | "xx.xxx.x.xx:xxxxx" | task_id | xxxxxxxxxxxxx-xxxxxxxxxxxxxxxx-x-x | is_failed | 0 | MIGRATE_LS_OP | | 2022-06-17 14:31:34.833923 | xx.xxx.x.xx | xxxxx | storage_ha | start_migration_task | tenant_id | 1002 | ls_id | 1001 | src | "xx.xxx.x.xx:xxxxx" | dst | "xx.xxx.x.xx:xxxxx" | task_id | xxxxxxxxxxxxx-xxxxxxxxxxxxxxxx-x-x | data_tablet_count | 1 | MIGRATE_LS_OP | | 2022-06-17 14:31:40.843227 | xx.xxx.x.xx | xxxxx | storage_ha | sys_tablets_migration_task | tenant_id | 1002 | ls_id | 1001 | src | "xx.xxx.x.xx:xxxxx" | dst | "xx.xxx.x.xx:xxxxx" | task_id | xxxxxxxxxxxxx-xxxxxxxxxxxxxxxx-x-x | is_failed | 0 | MIGRATE_LS_OP | | 2022-06-17 14:31:45.845290 | xx.xxx.x.xx | xxxxx | storage_ha | tablet_migration_task | tenant_id | 1002 | ls_id | 1001 | src | "xx.xxx.x.xx:xxxxx" | dst | "xx.xxx.x.xx:xxxxx" | tablet_id | 49492 | cost_us | 1030 | MIGRATE_LS_OP | | 2022-06-17 14:31:50.850248 | xx.xxx.x.xx | xxxxx | storage_ha | tablet_migration_task | tenant_id | 1002 | ls_id | 1001 | src | "xx.xxx.x.xx:xxxxx" | dst | "xx.xxx.x.xx:xxxxx" | tablet_id | 49491 | cost_us | 1125 | MIGRATE_LS_OP | | 2022-06-17 14:31:55.855032 | xx.xxx.x.xx | xxxxx | storage_ha | tablet_migration_task | tenant_id | 1002 | ls_id | 1001 | src | "xx.xxx.x.xx:xxxxx" | dst | "xx.xxx.x.xx:xxxxx" | tablet_id | 49490 | cost_us | 1174 | MIGRATE_LS_OP | | 2022-06-17 14:32:00.860459 | xx.xxx.x.xx | xxxxx | storage_ha | tablet_migration_task | tenant_id | 1002 | ls_id | 1001 | src | "xx.xxx.x.xx:xxxxx" | dst | "xx.xxx.x.xx:xxxxx" | tablet_id | 49493 | cost_us | 1160 | MIGRATE_LS_OP | | 2022-06-17 14:32:05.867814 | xx.xxx.x.xx | xxxxx | storage_ha | data_tablets_migration_task | tenant_id | 1002 | ls_id | 1001 | src | "xx.xxx.x.xx:xxxxx" | dst | "xx.xxx.x.xx:xxxxx" | task_id | xxxxxxxxxxxxx-xxxxxxxxxxxxxxxx-x-x | is_failed | 0 | MIGRATE_LS_OP | | 2022-06-17 14:32:10.870567 | xx.xxx.x.xx | xxxxx | storage_ha | tablet_group_migration_task | tenant_id | 1002 | ls_id | 1001 | src | "xx.xxx.x.xx:xxxxx" | dst | "xx.xxx.x.xx:xxxxx" | task_id | xxxxxxxxxxxxx-xxxxxxxxxxxxxxxx-x-x | tablet_count | 1 | MIGRATE_LS_OP | | 2022-06-17 14:32:15.872683 | xx.xxx.x.xx | xxxxx | storage_ha | tablet_migration_task | tenant_id | 1002 | ls_id | 1001 | src | "xx.xxx.x.xx:xxxxx" | dst | "xx.xxx.x.xx:xxxxx" | tablet_id | 50001 | cost_us | 1068 | MIGRATE_LS_OP | | 2022-06-17 14:32:20.876816 | xx.xxx.x.xx | xxxxx | storage_ha | migration_finish_task | tenant_id | 1002 | ls_id | 1001 | src | "xx.xxx.x.xx:xxxxx" | dst | "xx.xxx.x.xx:xxxxx" | task_id | xxxxxxxxxxxxx-xxxxxxxxxxxxxxxx-x-x | is_failed | 0 | MIGRATE_LS_OP | | 2022-06-17 14:32:24.878956 | xx.xxx.x.xx | xxxxx | storage_ha | initial_complete_migration_task | tenant_id | 1002 | ls_id | 1001 | src | "xx.xxx.x.xx:xxxxx" | dst | "xx.xxx.x.xx:xxxxx" | task_id | xxxxxxxxxxxxx-xxxxxxxxxxxxxxxx-x-x | is_failed | 0 | MIGRATE_LS_OP | | 2022-06-17 14:32:27.733626 | xx.xxx.x.xx | xxxxx | storage_ha | start_complete_migration_task | tenant_id | 1002 | ls_id | 1001 | src | "xx.xxx.x.xx:xxxxx" | dst | "xx.xxx.x.xx:xxxxx" | task_id | xxxxxxxxxxxxx-xxxxxxxxxxxxxxxx-x-x | is_failed | 0 | MIGRATE_LS_OP | | 2022-06-17 14:32:31.734910 | xx.xxx.x.xx | xxxxx | storage_ha | finish_complete_migration_task | tenant_id | 1002 | ls_id | 1001 | src | "xx.xxx.x.xx:xxxxx" | dst | "xx.xxx.x.xx:xxxxx" | task_id | xxxxxxxxxxxxx-xxxxxxxxxxxxxxxx-x-x | is_failed | 0 | MIGRATE_LS_OP | +----------------------------+-------------+----------+------------+---------------------------------+-----------+--------+-------+--------+-------+---------------------+-------+---------------------+-----------+------------------------------------+-------------------+--------+---------------+ ``` - 视图 `CDB_OB_LS_REPLICA_TASKS` 。 该视图展示所有正在运行的 迁移/复制/rebuild 任务，如果怀疑某任务有问题，可用 `task_id` 搜索对应时间段的日志。 下例展示正在运行的补副本任务： ```shell OceanBase(root@oceanbase)>select * from oceanbase.CDB_OB_LS_REPLICA_TASKS; ``` 输出结果如下： ```shell +-----------+-------+-------------+------------------------------------+-------------+----------+-----------------------+-------------------------+-----------------------------+---------------------+-----------------------+-------------------------+-----------------------------+---------------------+------------------+--------------------+---------------------+---------------------+---------------------+-----------------------------------------+ | TENANT_ID | LS_ID | TASK_TYPE | TASK_ID | TASK_STATUS | PRIORITY | TARGET_REPLICA_SVR_IP | TARGET_REPLICA_SVR_PORT | TARGET_PAXOS_REPLICA_NUMBER | TARGET_REPLICA_TYPE | SOURCE_REPLICA_SVR_IP | SOURCE_REPLICA_SVR_PORT | SOURCE_PAXOS_REPLICA_NUMBER | SOURCE_REPLICA_TYPE | TASK_EXEC_SVR_IP | TASK_EXEC_SVR_PORT | CREATE_TIME | START_TIME | MODIFY_TIME | COMMENT | +-----------+-------+-------------+------------------------------------+-------------+----------+-----------------------+-------------------------+-----------------------------+---------------------+-----------------------+-------------------------+-----------------------------+---------------------+------------------+--------------------+---------------------+---------------------+---------------------+-----------------------------------------+ | 1003 | 1 | ADD REPLICA | xxxxxxxxxxxxx-xxxxxxxxxxxxxxxx-x-x | INPROGRESS | HIGH | xx.xxx.xx.xx | 52616 | 3 | FULL | xx.xxx.xx.xx | 52614 | 2 | FULL | xx.xxx.xx.xx | 52616 | 2022-09-19 10:32:18 | 2022-09-19 10:32:18 | 2022-09-19 10:32:18 | add paxos replica according to locality | | 1004 | 1 | ADD REPLICA | xxxxxxxxxxxxx-xxxxxxxxxxxxxxxx-x-x | INPROGRESS | HIGH | xx.xxx.xx.xx | 52616 | 3 | FULL | xx.xxx.xx.xx | 52614 | 2 | FULL | xx.xxx.xx.xx | 52616 | 2022-09-19 10:32:18 | 2022-09-19 10:32:18 | 2022-09-19 10:32:18 | add paxos replica according to locality | | 1004 | 1001 | ADD REPLICA | xxxxxxxxxxxxx-xxxxxxxxxxxxxxxx-x-x | INPROGRESS | HIGH | xx.xxx.xx.xx | 52616 | 3 | FULL | xx.xxx.xx.xx | 52614 | 2 | FULL | xx.xxx.xx.xx | 52616 | 2022-09-19 10:32:18 | 2022-09-19 10:32:18 | 2022-09-19 10:32:18 | add paxos replica according to locality | +-----------+-------+-------------+------------------------------------+-------------+----------+-----------------------+-------------------------+-----------------------------+---------------------+-----------------------+-------------------------+-----------------------------+---------------------+------------------+--------------------+---------------------+---------------------+---------------------+-----------------------------------------+ 3 rows in set (0.01 sec) ``` - 动态视图 `V$OB_LS_REPLICA_TASK_PLAN`。 该视图展示当前应该生成的 迁移/复制/rebuild 任务，如果当前预期生成的任务这张表查询不到，可以联系RD处理。 下面展示需要生成的补副本任务： ```shell OceanBase(root@oceanbase)>select * from oceanbase.V$OB_LS_REPLICA_TASK_PLAN; ``` 输出结果如下： ```shell +-----------+-------+-------------+----------+-----------------------+-------------------------+-----------------------------+---------------------+-----------------------+-------------------------+-----------------------------+---------------------+------------------+--------------------+-----------------------------------------+ | TENANT_ID | LS_ID | TASK_TYPE | PRIORITY | TARGET_REPLICA_SVR_IP | TARGET_REPLICA_SVR_PORT | TARGET_PAXOS_REPLICA_NUMBER | TARGET_REPLICA_TYPE | SOURCE_REPLICA_SVR_IP | SOURCE_REPLICA_SVR_PORT | SOURCE_PAXOS_REPLICA_NUMBER | SOURCE_REPLICA_TYPE | TASK_EXEC_SVR_IP | TASK_EXEC_SVR_PORT | COMMENT | +-----------+-------+-------------+----------+-----------------------+-------------------------+-----------------------------+---------------------+-----------------------+-------------------------+-----------------------------+---------------------+------------------+--------------------+-----------------------------------------+ | 1003 | 1 | ADD REPLICA | HIGH | xx.xxx.xx.xx | 52616 | 3 | FULL | xx.xxx.xx.xx | 52614 | 2 | FULL | xx.xxx.xx.xx | 52616 | add paxos replica according to locality | | 1004 | 1 | ADD REPLICA | HIGH | xx.xxx.xx.xx | 52616 | 3 | FULL | xx.xxx.xx.xx | 52614 | 2 | FULL | xx.xxx.xx.xx | 52616 | add paxos replica according to locality | | 1004 | 1001 | ADD REPLICA | HIGH | xx.xxx.xx.xx | 52616 | 3 | FULL | xx.xxx.xx.xx | 52614 | 2 | FULL | xx.xxx.xx.xx | 52616 | add paxos replica according to locality | +-----------+-------+-------------+----------+-----------------------+-------------------------+-----------------------------+---------------------+-----------------------+-------------------------+-----------------------------+---------------------+------------------+--------------------+-----------------------------------------+ 3 rows in set (0.09 sec) ``` ## 迁移复制流程解析 **如果是首次阅读此排查文档，建议阅读此节后再进行排查，便于迅速准确定位问题。** 以下 SQL展示了一次迁移任务的大致过程。 1. 运行命令 `alter system migrate unit 1003 destination 'xx.xxx.x.xx:12347'` 开始迁移。 2. RS 发起迁移日志流副本任务，1002 租户 的 1001 号日志流副本从 `xx.xxx.x.xx:12348` 迁移到 `xx.xxx.x.xx:12347` 。 3. RS 将迁移命令发送到迁移目的端 `xx.xxx.x.xx:12347` 。 4. 迁移目的端收到迁移命令 RPC 。 这一阶段会在迁移目的端新建日志流副本，创建日志流副本迁移任务，并开始调度。 5. 迁移目的端开始运行迁移任务，迁移任务主要包括以下几个阶段： 1. initial_migration_task 。 这一阶段主要做一些前置检查(参数校验、数据盘空间是否足够等)，构造并调度迁移任务 DAG。 #### 注意 同一个 `OBServer` 同时运行的数据复制任务不能超过配置项 `server_data_copy_in_concurrency`（默认 2 个），否则会报错。 2. start_migration_task 。 这一阶段做一些迁移前的准备工作，包括但不限于： - 关闭 `Clog` 日志同步和回放。 - 选择源端，同步 `RPC` 获取数据源端的 `meta` 信息（包括日志同步 meta 信息 `PalfBaseInfo` 和日志流元信息 `LSMeta` ）。 - 用从源端获取到的 `meta` 重置日志同步起始位点（第 4 步创建出的日志流副本日志同步起始位点是 0，但迁移目的端可能）。 - 构造并调度 `sys_tablets_migration_task` 和 `data_tablets_migration_task` 。 3. sys_tablets_migration_task 。 - 创建 `sys tablet` 。 - 逐个迁移 `sys tablet` 。 4. data_tablets_migration_task 。 - 创建并加载 `tablet` 。 - 创建并调度 `tablet_group_migration_task` 。 5. tablet_group_migration_task 。 `tablet_group_migration_task` 中包括一个 `tablet` 列表，是所有需要迁移的 `tablet` 。此任务会构造多个 `tablet` 迁移任务，将源端的 `tablet` 复制到目的端。 6. migration_finish_task 。 此阶段做一些简单的内部状态切换。 7. initial_complete_migration_task 。 构造并调度 `start_complete_migration_task` 和 `finish_complete_migration_task` 。 8. start_complete_migration_task 。 - 开启日志同步和日志回放，并等待目的端日志同步进度追上日志流 `Leader` 。 - 等待日志回放进度。 - `Paxos` 成员变更；对于迁移，先将迁移目的端加入成员列表，再移除迁移源端。对于补副本，将目的端加入成员列表。 9. finish_complete_migration_task 。 检查是否需要重试，结束任务 下面的内部表展示了一次完整迁移任务的记录。 1. 通过视图 `DBA_OB_SERVER_EVENT_HISTORY` 获取历史事件所属的模块的第一个，第二个事件属性值的信息。 ```shell obclient> select * from oceanbase.DBA_OB_ROOTSERVICE_EVENT_HISTORY where module like '%dis%' and value1=1002 and value2=1001; ``` 输出结果如下： ```shell +----------------------------+-------------------+---------------------------+-----------+--------+-------+--------+--------+---------------------+-------------+---------------------+----------------+---------------------+---------+---------------------------------------------+---------------------------------------------+-------------+-------------+ | gmt_create | module | event | name1 | value1 | name2 | value2 | name3 | value3 | name4 | value4 | name5 | value5 | name6 | value6 | extra_info | rs_svr_ip | rs_svr_port | +----------------------------+-------------------+---------------------------+-----------+--------+-------+--------+--------+---------------------+-------------+---------------------+----------------+---------------------+---------+---------------------------------------------+---------------------------------------------+-------------+-------------+ | 2022-06-17 14:31:31.110178 | disaster_recovery | start_migrate_ls_replica | tenant_id | 1002 | ls_id | 1001 | source | "xx.xxx.x.xx:12348" | destination | "xx.xxx.x.xx:12347" | data_source | "xx.xxx.x.xx:12348" | comment | migrate replica due to unit group not match | | xx.xxx.x.xx | 12345 | | 2022-06-17 14:32:31.743073 | disaster_recovery | finish_migrate_ls_replica | tenant_id | 1002 | ls_id | 1001 | source | "xx.xxx.x.xx:12348" | destination | "xx.xxx.x.xx:12347" | execute_result | 0 | elapsed | 60641409 | migrate replica due to unit group not match | xx.xxx.x.xx | 12345 | +----------------------------+-------------------+---------------------------+-----------+--------+-------+--------+--------+---------------------+-------------+---------------------+----------------+---------------------+---------+---------------------------------------------+---------------------------------------------+-------------+-------------+ ```shell obclient> select * from oceanbase.__all_server_event_history where module like 'storage_ha' and event like '%schedule%'; ``` 输出结果如下： ```shell +----------------------------+-------------+----------+------------+-----------------------------+-------+-----------+----------+---------------------+-------+---------------------+-------+--------+-------+--------+-------+--------+------------+ | gmt_create | svr_ip | svr_port | module | event | name1 | value1 | name2 | value2 | name3 | value3 | name4 | value4 | name5 | value5 | name6 | value6 | extra_info | +----------------------------+-------------+----------+------------+-----------------------------+-------+-----------+----------+---------------------+-------+---------------------+-------+--------+-------+--------+-------+--------+------------+ | 2022-06-17 14:31:31.110597 | 11.124.5.18 | 12347 | storage_ha | schedule_ls_migration start | ls_id | {id:1001} | data_src | "xx.xxx.x.xx:12348" | dest | "xx.xxx.x.xx:12347" | | | | | | | | +----------------------------+-------------+----------+------------+-----------------------------+-------+-----------+----------+---------------------+-------+---------------------+-------+--------+-------+--------+-------+--------+------------+ ``` 3. 再次通过视图 `DBA_OB_SERVER_EVENT_HISTORY` 获取历史事件所属的模块的第一个，第二个事件属性值的信息。 ```shell obclient> select * from oceanbase.__all_server_event_history where module like 'storage_ha' and value1 = 1002 and value2=1001; ``` 输出结果如下： ```shell +----------------------------+-------------+----------+------------+---------------------------------+-----------+--------+-------+--------+-------+---------------------+-------+---------------------+-----------+------------------------------------+-------------------+--------+---------------+ | gmt_create | svr_ip | svr_port | module | event | name1 | value1 | name2 | value2 | name3 | value3 | name4 | value4 | name5 | value5 | name6 | value6 | extra_info | +----------------------------+-------------+----------+------------+---------------------------------+-----------+--------+-------+--------+-------+---------------------+-------+---------------------+-----------+------------------------------------+-------------------+--------+---------------+ | 2022-06-17 14:31:34.831234 | xx.xxx.x.xx | 12347 | storage_ha | initial_migration_task | tenant_id | 1002 | ls_id | 1001 | src | "xx.xxx.x.xx:xxxxx" | dst | "xx.xxx.x.xx:xxxxx" | task_id | xxxxxxxxxxxxx-xxxxxxxxxxxxxxxx-x-x | is_failed | 0 | MIGRATE_LS_OP | | 2022-06-17 14:31:34.833923 | xx.xxx.x.xx | 12347 | storage_ha | start_migration_task | tenant_id | 1002 | ls_id | 1001 | src | "xx.xxx.x.xx:xxxxx" | dst | "xx.xxx.x.xx:xxxxx" | task_id | xxxxxxxxxxxxx-xxxxxxxxxxxxxxxx-x-x | data_tablet_count | 1 | MIGRATE_LS_OP | | 2022-06-17 14:31:40.843227 | xx.xxx.x.xx | 12347 | storage_ha | sys_tablets_migration_task | tenant_id | 1002 | ls_id | 1001 | src | "xx.xxx.x.xx:xxxxx" | dst | "xx.xxx.x.xx:xxxxx" | task_id | xxxxxxxxxxxxx-xxxxxxxxxxxxxxxx-x-x | is_failed | 0 | MIGRATE_LS_OP | | 2022-06-17 14:31:45.845290 | xx.xxx.x.xx | 12347 | storage_ha | tablet_migration_task | tenant_id | 1002 | ls_id | 1001 | src | "xx.xxx.x.xx:xxxxx" | dst | "xx.xxx.x.xx:xxxxx" | tablet_id | 49492 | cost_us | 1030 | MIGRATE_LS_OP | | 2022-06-17 14:31:50.850248 | xx.xxx.x.xx | 12347 | storage_ha | tablet_migration_task | tenant_id | 1002 | ls_id | 1001 | src | "xx.xxx.x.xx:xxxxx" | dst | "xx.xxx.x.xx:xxxxx" | tablet_id | 49491 | cost_us | 1125 | MIGRATE_LS_OP | | 2022-06-17 14:31:55.855032 | xx.xxx.x.xx | 12347 | storage_ha | tablet_migration_task | tenant_id | 1002 | ls_id | 1001 | src | "xx.xxx.x.xx:xxxxx" | dst | "xx.xxx.x.xx:xxxxx" | tablet_id | 49490 | cost_us | 1174 | MIGRATE_LS_OP | | 2022-06-17 14:32:00.860459 | xx.xxx.x.xx | 12347 | storage_ha | tablet_migration_task | tenant_id | 1002 | ls_id | 1001 | src | "xx.xxx.x.xx:xxxxx" | dst | "xx.xxx.x.xx:xxxxx" | tablet_id | 49493 | cost_us | 1160 | MIGRATE_LS_OP | | 2022-06-17 14:32:05.867814 | xx.xxx.x.xx | 12347 | storage_ha | data_tablets_migration_task | tenant_id | 1002 | ls_id | 1001 | src | "xx.xxx.x.xx:xxxxx" | dst | "xx.xxx.x.xx:xxxxx" | task_id | xxxxxxxxxxxxx-xxxxxxxxxxxxxxxx-x-x | is_failed | 0 | MIGRATE_LS_OP | | 2022-06-17 14:32:10.870567 | xx.xxx.x.xx | 12347 | storage_ha | tablet_group_migration_task | tenant_id | 1002 | ls_id | 1001 | src | "xx.xxx.x.xx:xxxxx" | dst | "xx.xxx.x.xx:xxxxx" | task_id | xxxxxxxxxxxxx-xxxxxxxxxxxxxxxx-x-x | tablet_count | 1 | MIGRATE_LS_OP | | 2022-06-17 14:32:15.872683 | xx.xxx.x.xx | 12347 | storage_ha | tablet_migration_task | tenant_id | 1002 | ls_id | 1001 | src | "xx.xxx.x.xx:xxxxx" | dst | "xx.xxx.x.xx:xxxxx" | tablet_id | 50001 | cost_us | 1068 | MIGRATE_LS_OP | | 2022-06-17 14:32:20.876816 | xx.xxx.x.xx | 12347 | storage_ha | migration_finish_task | tenant_id | 1002 | ls_id | 1001 | src | "xx.xxx.x.xx:xxxxx" | dst | "xx.xxx.x.xx:xxxxx" | task_id | xxxxxxxxxxxxx-xxxxxxxxxxxxxxxx-x-x | is_failed | 0 | MIGRATE_LS_OP | | 2022-06-17 14:32:24.878956 | xx.xxx.x.xx | 12347 | storage_ha | initial_complete_migration_task | tenant_id | 1002 | ls_id | 1001 | src | "xx.xxx.x.xx:xxxxx" | dst | "xx.xxx.x.xx:xxxxx" | task_id | xxxxxxxxxxxxx-xxxxxxxxxxxxxxxx-x-x | is_failed | 0 | MIGRATE_LS_OP | | 2022-06-17 14:32:27.733626 | xx.xxx.x.xx | 12347 | storage_ha | start_complete_migration_task | tenant_id | 1002 | ls_id | 1001 | src | "xx.xxx.x.xx:xxxxx" | dst | "xx.xxx.x.xx:xxxxx" | task_id | xxxxxxxxxxxxx-xxxxxxxxxxxxxxxx-x-x | is_failed | 0 | MIGRATE_LS_OP | | 2022-06-17 14:32:31.734910 | xx.xxx.x.xx | 12347 | storage_ha | finish_complete_migration_task | tenant_id | 1002 | ls_id | 1001 | src | "xx.xxx.x.xx:xxxxx" | dst | "xx.xxx.x.xx:xxxxx" | task_id | xxxxxxxxxxxxx-xxxxxxxxxxxxxxxx-x-x | is_failed | 0 | MIGRATE_LS_OP | +----------------------------+-------------+----------+------------+---------------------------------+-----------+--------+-------+--------+-------+---------------------+-------+---------------------+-----------+------------------------------------+-------------------+--------+---------------+ ``` ## 排查步骤 1. 查询 `DBA_OB_ROOTSERVICE_EVENT_HISTORY` 记录，如果可以看到迁移开始记录，说明 `RS` 已经调度了迁移任务。查看 `value6` 中记录的任务执行结果，如果任务失败，大概率会记录一个ret_comment告诉我们具体是哪里报错了。如果没有 `ret_comment`，可以参考该文档后续的步骤来定位原因。需要排查 `RS` 部分的报错原因，或者任务没有调度的原因，详细内容可参见：[OceanBase 数据库 V4.0 版本容灾问题排查（RS 相关）](https://www.oceanbase.com/knowledge-base/oceanbase-database-1000000001229179?back=kb)。 ```shell select * from oceanbase.DBA_OB_ROOTSERVICE_EVENT_HISTORY where module like '%disas%' order by TIMESTAMP desc limit 10; ``` 2. 查询 `__all_server_event_history`表，确认： 1. 迁移/复制任务是否已经在目的端 server 上被调度。 2. 任务执行到哪个阶段，是否有明显报错。 一般到此阶段即可明确迁移任务的状态，如果有报错或者卡住。用内部表中的 `task_id` 在迁移目的端搜索对应时间的日志，并找对应的 `RD` 即可。 #### 注意 - 如果服务可用，优先查询内部表排查。 - 如果服务不可用，先排查服务不可用的原因，再排查迁移/复制是否结束。 ## 日志信息 `Migration` 选择迁移源端并获取 `PalfBaseInfo`，用于目的端新建日志流 可在迁移源端搜索以下日志，如有结果，说明 `PALF` 返回了 `PalfBaseInfo` 。 ```shell # $tenant_id like T1001 grep $tenant_id observer.log | grep 'get_palf_base_info success' ``` 获取其余日志流meta信息日志待补充。 迁移目的端用获取到的 `PalfBaseInfo` 新建日志流。 ```shell # ObLogService 新建 PALF 成功 grep $tenant_id observer.log | grep 'ObLogService create_ls success' ``` ### 新建迁移目的端日志流 ```shell grep $tenant_id observer.log | grep 'create_ls for ha finish' ``` 目的端开启回放和日志同步。 ```shell # 迁移模块enable log成功 grep $tenant_id observer.log | grep 'succeed enable log' ``` ### 迁移模块 enable log 中包括 2 步 1. enable replayservice 。 ```shell grep $tenant_id observer.log | grep 'enable replay success' ``` 2. enable palf log sync 。 ```shell grep $tenant_id observer.log | grep 'enable_sync succes' ``` enable 日志流成功后会创建 `tablet group migrate dag` 。 ```shell grep $tenant_id observer.log | grep 'succeed generate tablet group dag' ``` 后续 `DAG` 任务调度不同阶段日志、拷贝 `tablet` 成功日志待补充。 迁移模块会在超时时间内循环等待目的端日志同步追上 `Leader` 。 ```shell # 如果有以下日志，说明目的端日志已经同步 grep $tenant_id observer.log | grep 'log is sync, stop wait_log_syn' ``` ```shell # 如果没有以上日志，则搜索 palf 日志 grep $tenant_id observer.log | grep PALF | grep 'id=$ls_id' | grep is_in_sync ``` ### 输出样例 #### 日志已经同步 is_log_sync=true 日志同步后，迁移模块会发起 `replace_member` 成员变更，将迁移目的端加入 `Paxos` 成员列表，并将源端从成员列表移除。 - 发起成员变更前会有以下日志。 ```shell grep $tenant_id observer.log | grep 'start clear dag net ctx' ``` - 在迁移目的段，成员变更开始和结束分别对应以下 2 条日志。 ```shell grep $tenant_id observer.log | grep 'config_change start' grep $tenant_id observer.log | grep 'membership_change finish' ``` - 迁移结束。 1. 查询是否有迁移结束的记录 ```shell select * from oceanbase.DBA_OB_ROOTSERVICE_EVENT_HISTORY where module like '%disas%' order by gmt_create desc limit 10; ``` 2. 查询日志流成员列表是否改变。 ```shell select * from oceanbase.__all_virtual_log_stat where tenant_id=1001 and id=1; ``` #### 注意 此节仅用作备忘，排查一般迁移问题的不需要用到此章的内容，查内部表。 ## 适用版本 OceanBase 数据库 V4.x 版本。 上一篇 [V4.2.1.5 之前的版本升级 V4.2.1.5、V4.2.1.6 版本，触发迁移、复制、rebuild 失败，补副本流程，报错 -4735](https://www.oceanbase.com/knowledge-base/oceanbase-database-1000000000824039) 下一篇 [磁盘扩缩容后重启拉起归档失败](https://www.oceanbase.com/knowledge-base/oceanbase-database-1000000000838541) ![有帮助](https://gw.alipayobjects.com/mdn/ob_asset/afts/img/A*y6ocSqN8cqsAAAAAAAAAAAAAARQnAQ)![无帮助](https://gw.alipayobjects.com/mdn/ob_asset/afts/img/A*BG9IQJyLHF8AAAAAAAAAAAAAARQnAQ)![反馈](https://gw.alipayobjects.com/mdn/ob_asset/afts/img/A*eTWdQKCRKHwAAAAAAAAAAAAAARQnAQ)[AI](https://www.oceanbase.com/obi) 咨询热线