OceanBase 数据库 V4.0 版本迁移 & 复制 & Rebuild 排查指南

本文汇总了迁移/复制/Rebuild 的主要流程，以及相关的内部表/日志信息，便于快速定位问题。

相关内部表/视图

• 视图 DBA_OB_ROOTSERVICE_EVENT_HISTORY 。

  关注 event，source 和 destination，如果表中有对应条目，说明 RS 已经调度的相应任务。其中 value6 可以看到任务的执行结果，ret 表示该次任务成功与否，elapsed 表示该次任务的执行时间，如果失败，可能会有 ret_comment 字段直接反应失败在哪一步。

  OceanBase(root@oceanbase)>select * from oceanbase.DBA_OB_ROOTSERVICE_EVENT_HISTORY where module like "%disaster%" and value1 = 1003 and value2 = 1;
  

  输出结果如下：

  +----------------------------+-------------------+--------------------------------+-----------+--------+-------+--------+---------+------------------------------------+--------+----------------------+---------------+-------------------------------------------------------------------------------------------------------------------+----------------+------------------------------------------------------+------------------------------------------------------+--------------+-------------+
  | TIMESTAMP                  | MODULE            | EVENT                          | NAME1     | VALUE1 | NAME2 | VALUE2 | NAME3   | VALUE3                             | NAME4  | VALUE4               | NAME5         | VALUE5                                                                                                            | NAME6          | VALUE6                                               | EXTRA_INFO                                           | RS_SVR_IP    | RS_SVR_PORT |
  +----------------------------+-------------------+--------------------------------+-----------+--------+-------+--------+---------+------------------------------------+--------+----------------------+---------------+-------------------------------------------------------------------------------------------------------------------+----------------+------------------------------------------------------+------------------------------------------------------+--------------+-------------+
  | 2022-09-26 19:37:11.661502 | disaster_recovery | start_remove_ls_paxos_replica  | tenant_id | 1003   | ls_id | 1      | task_id | xxxxxxxxxxxxx-xxxxxxxxxxxxxxxx-x-x | leader | "xx.xxx.xx.xx:xxxxx" | remove_server | {server:"xx.xxx.xx.xx:xxxxx", timestamp:1, flag:0, replica_type:0, region:"default_region", memstore_percent:100} | comment        | remove redundant paxos replica according to locality |                                                      | xx.xxx.xx.xx |       xxxxx |
  | 2022-09-26 19:37:11.733052 | disaster_recovery | finish_remove_ls_paxos_replica | tenant_id | 1003   | ls_id | 1      | task_id | xxxxxxxxxxxxx-xxxxxxxxxxxxxxxx-x-x | leader | "xx.xxx.xx.xx:xxxxx" | remove_server | {server:"xx.xxx.xx.xx:xxxxx", timestamp:1, flag:0, replica_type:0, region:"default_region", memstore_percent:100} | execute_result | ret:0; elapsed:71571;                                | remove redundant paxos replica according to locality | xx.xxx.xx.xx |       xxxxx |
  +----------------------------+-------------------+--------------------------------+-----------+--------+-------+--------+---------+------------------------------------+--------+----------------------+---------------+-------------------------------------------------------------------------------------------------------------------+----------------+------------------------------------------------------+------------------------------------------------------+--------------+-------------+
  2 rows in set (0.02 sec)
  

• 视图 DBA_OB_SERVER_EVENT_HISTORY

  记录了server 上的所有事件，包括 迁移、复制、rebuild 等事件。关注调度记录和 is_failed 。

  以迁移为例，unit 从 xxx.xxx.xxx.xxx:12348 迁移到 1xxx.xxx.xxx.xxx:12347，在视图 DBA_OB_SERVER_EVENT_HISTORY 中观察以下事件：

  1. 迁移任务在目的端是否已经被调度。

  2. 迁移过程中各任务的执行情况。

  如果迁移任务卡住，可以通过该表观察目前迁移任务的执行阶段排查。

  # 排查 SQL 。
  # 获取已经被 rs 调度但还没有完成的任务列表（tenant_id, ls_id）。
  # 第一个 event 关键字根据任务类型替换。
  (select value1, value2 from oceanbase.DBA_OB_ROOTSERVICE_EVENT_HISTORY where event like '%add_ls%') except (select value1, value2 from oceanbase.DBA_OB_SERVER_EVENT_HISTORY where module like 'storage_ha' and event like '%finish_complete%');
  

获取报错的迁移/复制任务

1. 通过视图 DBA_OB_SERVER_EVENT_HISTORY 获取事件历史记录信息。

   obclient> select * from oceanbase.DBA_OB_SERVER_EVENT_HISTORY where event like '%migrat%' and name6 like '%fail%' and value6=1;
   Empty set (0.044 sec)
   

2. 通过视图 DBA_OB_SERVER_EVENT_HISTORY 获取事件历史记录所属的模块的信息。

   obclient> select * from oceanbase.DBA_OB_SERVER_EVENT_HISTORY where module like 'storage_ha' and event like '%schedule%';
   

   输出结果如下：

   +----------------------------+-------------+----------+------------+-----------------------------+-------+-----------+----------+---------------------+-------+---------------------+-------+--------+-------+--------+-------+--------+------------+
   | gmt_create                 | svr_ip      | svr_port | module     | event                       | name1 | value1    | name2    | value2              | name3 | value3              | name4 | value4 | name5 | value5 | name6 | value6 | extra_info |
   +----------------------------+-------------+----------+------------+-----------------------------+-------+-----------+----------+---------------------+-------+---------------------+-------+--------+-------+--------+-------+--------+------------+
   | 2022-06-17 14:31:31.110597 | xx.xxx.x.xx |    12347 | storage_ha | schedule_ls_migration start | ls_id | {id:1001} | data_src | "xx.xxx.x.xx:xxxxx" | dest  | "xx.xxx.x.xx:xxxxx" |       |        |       |        |       |        |            |
   +----------------------------+-------------+----------+------------+-----------------------------+-------+-----------+----------+---------------------+-------+---------------------+-------+--------+-------+--------+-------+--------+------------+
   

3. 通过视图 DBA_OB_SERVER_EVENT_HISTORY 获取事件历史记录所属的模块的第一个，第二个事件属性值的信息。

   obclient> select * from oceanbase.DBA_OB_SERVER_EVENT_HISTORY where module like 'storage_ha' and value1 = 1002 and value2=1001;
   

   输出结果如下：

   +----------------------------+-------------+----------+------------+---------------------------------+-----------+--------+-------+--------+-------+---------------------+-------+---------------------+-----------+------------------------------------+-------------------+--------+---------------+
   | gmt_create                 | svr_ip      | svr_port | module     | event                           | name1     | value1 | name2 | value2 | name3 | value3              | name4 | value4              | name5     | value5                             | name6             | value6 | extra_info    |
   +----------------------------+-------------+----------+------------+---------------------------------+-----------+--------+-------+--------+-------+---------------------+-------+---------------------+-----------+------------------------------------+-------------------+--------+---------------+
   | 2022-06-17 14:31:34.831234 | xx.xxx.x.xx |    xxxxx | storage_ha | initial_migration_task          | tenant_id | 1002   | ls_id | 1001   | src   | "xx.xxx.x.xx:xxxxx" | dst   | "xx.xxx.x.xx:xxxxx" | task_id   | xxxxxxxxxxxxx-xxxxxxxxxxxxxxxx-x-x | is_failed         | 0      | MIGRATE_LS_OP |
   | 2022-06-17 14:31:34.833923 | xx.xxx.x.xx |    xxxxx | storage_ha | start_migration_task            | tenant_id | 1002   | ls_id | 1001   | src   | "xx.xxx.x.xx:xxxxx" | dst   | "xx.xxx.x.xx:xxxxx" | task_id   | xxxxxxxxxxxxx-xxxxxxxxxxxxxxxx-x-x | data_tablet_count | 1      | MIGRATE_LS_OP |
   | 2022-06-17 14:31:40.843227 | xx.xxx.x.xx |    xxxxx | storage_ha | sys_tablets_migration_task      | tenant_id | 1002   | ls_id | 1001   | src   | "xx.xxx.x.xx:xxxxx" | dst   | "xx.xxx.x.xx:xxxxx" | task_id   | xxxxxxxxxxxxx-xxxxxxxxxxxxxxxx-x-x | is_failed         | 0      | MIGRATE_LS_OP |
   | 2022-06-17 14:31:45.845290 | xx.xxx.x.xx |    xxxxx | storage_ha | tablet_migration_task           | tenant_id | 1002   | ls_id | 1001   | src   | "xx.xxx.x.xx:xxxxx" | dst   | "xx.xxx.x.xx:xxxxx" | tablet_id | 49492                              | cost_us           | 1030   | MIGRATE_LS_OP |
   | 2022-06-17 14:31:50.850248 | xx.xxx.x.xx |    xxxxx | storage_ha | tablet_migration_task           | tenant_id | 1002   | ls_id | 1001   | src   | "xx.xxx.x.xx:xxxxx" | dst   | "xx.xxx.x.xx:xxxxx" | tablet_id | 49491                              | cost_us           | 1125   | MIGRATE_LS_OP |
   | 2022-06-17 14:31:55.855032 | xx.xxx.x.xx |    xxxxx | storage_ha | tablet_migration_task           | tenant_id | 1002   | ls_id | 1001   | src   | "xx.xxx.x.xx:xxxxx" | dst   | "xx.xxx.x.xx:xxxxx" | tablet_id | 49490                              | cost_us           | 1174   | MIGRATE_LS_OP |
   | 2022-06-17 14:32:00.860459 | xx.xxx.x.xx |    xxxxx | storage_ha | tablet_migration_task           | tenant_id | 1002   | ls_id | 1001   | src   | "xx.xxx.x.xx:xxxxx" | dst   | "xx.xxx.x.xx:xxxxx" | tablet_id | 49493                              | cost_us           | 1160   | MIGRATE_LS_OP |
   | 2022-06-17 14:32:05.867814 | xx.xxx.x.xx |    xxxxx | storage_ha | data_tablets_migration_task     | tenant_id | 1002   | ls_id | 1001   | src   | "xx.xxx.x.xx:xxxxx" | dst   | "xx.xxx.x.xx:xxxxx" | task_id   | xxxxxxxxxxxxx-xxxxxxxxxxxxxxxx-x-x | is_failed         | 0      | MIGRATE_LS_OP |
   | 2022-06-17 14:32:10.870567 | xx.xxx.x.xx |    xxxxx | storage_ha | tablet_group_migration_task     | tenant_id | 1002   | ls_id | 1001   | src   | "xx.xxx.x.xx:xxxxx" | dst   | "xx.xxx.x.xx:xxxxx" | task_id   | xxxxxxxxxxxxx-xxxxxxxxxxxxxxxx-x-x | tablet_count      | 1      | MIGRATE_LS_OP |
   | 2022-06-17 14:32:15.872683 | xx.xxx.x.xx |    xxxxx | storage_ha | tablet_migration_task           | tenant_id | 1002   | ls_id | 1001   | src   | "xx.xxx.x.xx:xxxxx" | dst   | "xx.xxx.x.xx:xxxxx" | tablet_id | 50001                              | cost_us           | 1068   | MIGRATE_LS_OP |
   | 2022-06-17 14:32:20.876816 | xx.xxx.x.xx |    xxxxx | storage_ha | migration_finish_task           | tenant_id | 1002   | ls_id | 1001   | src   | "xx.xxx.x.xx:xxxxx" | dst   | "xx.xxx.x.xx:xxxxx" | task_id   | xxxxxxxxxxxxx-xxxxxxxxxxxxxxxx-x-x | is_failed         | 0      | MIGRATE_LS_OP |
   | 2022-06-17 14:32:24.878956 | xx.xxx.x.xx |    xxxxx | storage_ha | initial_complete_migration_task | tenant_id | 1002   | ls_id | 1001   | src   | "xx.xxx.x.xx:xxxxx" | dst   | "xx.xxx.x.xx:xxxxx" | task_id   | xxxxxxxxxxxxx-xxxxxxxxxxxxxxxx-x-x | is_failed         | 0      | MIGRATE_LS_OP |
   | 2022-06-17 14:32:27.733626 | xx.xxx.x.xx |    xxxxx | storage_ha | start_complete_migration_task   | tenant_id | 1002   | ls_id | 1001   | src   | "xx.xxx.x.xx:xxxxx" | dst   | "xx.xxx.x.xx:xxxxx" | task_id   | xxxxxxxxxxxxx-xxxxxxxxxxxxxxxx-x-x | is_failed         | 0      | MIGRATE_LS_OP |
   | 2022-06-17 14:32:31.734910 | xx.xxx.x.xx |    xxxxx | storage_ha | finish_complete_migration_task  | tenant_id | 1002   | ls_id | 1001   | src   | "xx.xxx.x.xx:xxxxx" | dst   | "xx.xxx.x.xx:xxxxx" | task_id   | xxxxxxxxxxxxx-xxxxxxxxxxxxxxxx-x-x | is_failed         | 0      | MIGRATE_LS_OP |
   +----------------------------+-------------+----------+------------+---------------------------------+-----------+--------+-------+--------+-------+---------------------+-------+---------------------+-----------+------------------------------------+-------------------+--------+---------------+
   

• 视图 CDB_OB_LS_REPLICA_TASKS 。

  该视图展示所有正在运行的 迁移/复制/rebuild 任务，如果怀疑某任务有问题，可用 task_id 搜索对应时间段的日志。

  下例展示正在运行的补副本任务：

  OceanBase(root@oceanbase)>select * from oceanbase.CDB_OB_LS_REPLICA_TASKS;
  

  输出结果如下：

  +-----------+-------+-------------+------------------------------------+-------------+----------+-----------------------+-------------------------+-----------------------------+---------------------+-----------------------+-------------------------+-----------------------------+---------------------+------------------+--------------------+---------------------+---------------------+---------------------+-----------------------------------------+
  | TENANT_ID | LS_ID | TASK_TYPE   | TASK_ID                            | TASK_STATUS | PRIORITY | TARGET_REPLICA_SVR_IP | TARGET_REPLICA_SVR_PORT | TARGET_PAXOS_REPLICA_NUMBER | TARGET_REPLICA_TYPE | SOURCE_REPLICA_SVR_IP | SOURCE_REPLICA_SVR_PORT | SOURCE_PAXOS_REPLICA_NUMBER | SOURCE_REPLICA_TYPE | TASK_EXEC_SVR_IP | TASK_EXEC_SVR_PORT | CREATE_TIME         | START_TIME          | MODIFY_TIME         | COMMENT                                 |
  +-----------+-------+-------------+------------------------------------+-------------+----------+-----------------------+-------------------------+-----------------------------+---------------------+-----------------------+-------------------------+-----------------------------+---------------------+------------------+--------------------+---------------------+---------------------+---------------------+-----------------------------------------+
  |      1003 |     1 | ADD REPLICA | xxxxxxxxxxxxx-xxxxxxxxxxxxxxxx-x-x | INPROGRESS  | HIGH     | xx.xxx.xx.xx          |                   52616 |                           3 | FULL                | xx.xxx.xx.xx          |                   52614 |                           2 | FULL                | xx.xxx.xx.xx     |              52616 | 2022-09-19 10:32:18 | 2022-09-19 10:32:18 | 2022-09-19 10:32:18 | add paxos replica according to locality |
  |      1004 |     1 | ADD REPLICA | xxxxxxxxxxxxx-xxxxxxxxxxxxxxxx-x-x | INPROGRESS  | HIGH     | xx.xxx.xx.xx          |                   52616 |                           3 | FULL                | xx.xxx.xx.xx          |                   52614 |                           2 | FULL                | xx.xxx.xx.xx     |              52616 | 2022-09-19 10:32:18 | 2022-09-19 10:32:18 | 2022-09-19 10:32:18 | add paxos replica according to locality |
  |      1004 |  1001 | ADD REPLICA | xxxxxxxxxxxxx-xxxxxxxxxxxxxxxx-x-x | INPROGRESS  | HIGH     | xx.xxx.xx.xx          |                   52616 |                           3 | FULL                | xx.xxx.xx.xx          |                   52614 |                           2 | FULL                | xx.xxx.xx.xx     |              52616 | 2022-09-19 10:32:18 | 2022-09-19 10:32:18 | 2022-09-19 10:32:18 | add paxos replica according to locality |
  +-----------+-------+-------------+------------------------------------+-------------+----------+-----------------------+-------------------------+-----------------------------+---------------------+-----------------------+-------------------------+-----------------------------+---------------------+------------------+--------------------+---------------------+---------------------+---------------------+-----------------------------------------+
  3 rows in set (0.01 sec)
  

• 动态视图 V$OB_LS_REPLICA_TASK_PLAN。

  该视图展示当前应该生成的 迁移/复制/rebuild 任务，如果当前预期生成的任务这张表查询不到，可以联系RD处理。

  下面展示需要生成的补副本任务：

  OceanBase(root@oceanbase)>select * from oceanbase.V$OB_LS_REPLICA_TASK_PLAN;
  

  输出结果如下：

  +-----------+-------+-------------+----------+-----------------------+-------------------------+-----------------------------+---------------------+-----------------------+-------------------------+-----------------------------+---------------------+------------------+--------------------+-----------------------------------------+
  | TENANT_ID | LS_ID | TASK_TYPE   | PRIORITY | TARGET_REPLICA_SVR_IP | TARGET_REPLICA_SVR_PORT | TARGET_PAXOS_REPLICA_NUMBER | TARGET_REPLICA_TYPE | SOURCE_REPLICA_SVR_IP | SOURCE_REPLICA_SVR_PORT | SOURCE_PAXOS_REPLICA_NUMBER | SOURCE_REPLICA_TYPE | TASK_EXEC_SVR_IP | TASK_EXEC_SVR_PORT | COMMENT                                 |
  +-----------+-------+-------------+----------+-----------------------+-------------------------+-----------------------------+---------------------+-----------------------+-------------------------+-----------------------------+---------------------+------------------+--------------------+-----------------------------------------+
  |      1003 |     1 | ADD REPLICA | HIGH     | xx.xxx.xx.xx          |                   52616 |                           3 | FULL                | xx.xxx.xx.xx          |                   52614 |                           2 | FULL                | xx.xxx.xx.xx     |              52616 | add paxos replica according to locality |
  |      1004 |     1 | ADD REPLICA | HIGH     | xx.xxx.xx.xx          |                   52616 |                           3 | FULL                | xx.xxx.xx.xx          |                   52614 |                           2 | FULL                | xx.xxx.xx.xx     |              52616 | add paxos replica according to locality |
  |      1004 |  1001 | ADD REPLICA | HIGH     | xx.xxx.xx.xx          |                   52616 |                           3 | FULL                | xx.xxx.xx.xx          |                   52614 |                           2 | FULL                | xx.xxx.xx.xx     |              52616 | add paxos replica according to locality |
  +-----------+-------+-------------+----------+-----------------------+-------------------------+-----------------------------+---------------------+-----------------------+-------------------------+-----------------------------+---------------------+------------------+--------------------+-----------------------------------------+
  3 rows in set (0.09 sec)
  

迁移复制流程解析

如果是首次阅读此排查文档，建议阅读此节后再进行排查，便于迅速准确定位问题。

以下 SQL展示了一次迁移任务的大致过程。

1. 运行命令 alter system migrate unit 1003 destination 'xx.xxx.x.xx:12347' 开始迁移。

2. RS 发起迁移日志流副本任务，1002 租户 的 1001 号日志流副本从 xx.xxx.x.xx:12348 迁移到 xx.xxx.x.xx:12347 。

3. RS 将迁移命令发送到迁移目的端 xx.xxx.x.xx:12347 。

4. 迁移目的端收到迁移命令 RPC 。

   这一阶段会在迁移目的端新建日志流副本，创建日志流副本迁移任务，并开始调度。

5. 迁移目的端开始运行迁移任务，迁移任务主要包括以下几个阶段：

   1. initial_migration_task 。

      这一阶段主要做一些前置检查(参数校验、数据盘空间是否足够等)，构造并调度迁移任务 DAG。

      注意

      同一个 OBServer 同时运行的数据复制任务不能超过配置项 server_data_copy_in_concurrency（默认 2 个），否则会报错。

   2. start_migration_task 。

      这一阶段做一些迁移前的准备工作，包括但不限于：

      • 关闭 Clog 日志同步和回放。

        • 选择源端，同步 RPC 获取数据源端的 meta 信息（包括日志同步 meta 信息 PalfBaseInfo 和日志流元信息 LSMeta ）。

      • 用从源端获取到的 meta 重置日志同步起始位点（第 4 步创建出的日志流副本日志同步起始位点是 0，但迁移目的端可能）。

      • 构造并调度 sys_tablets_migration_task 和 data_tablets_migration_task 。

   3. sys_tablets_migration_task 。

      • 创建 sys tablet 。

        • 逐个迁移 sys tablet 。

   4. data_tablets_migration_task 。

      • 创建并加载 tablet 。

      • 创建并调度 tablet_group_migration_task 。

   5. tablet_group_migration_task 。

      tablet_group_migration_task 中包括一个 tablet 列表，是所有需要迁移的 tablet 。此任务会构造多个 tablet 迁移任务，将源端的 tablet 复制到目的端。

   6. migration_finish_task 。

      此阶段做一些简单的内部状态切换。

   7. initial_complete_migration_task 。

      构造并调度 start_complete_migration_task 和 finish_complete_migration_task 。

   8. start_complete_migration_task 。

      • 开启日志同步和日志回放，并等待目的端日志同步进度追上日志流 Leader 。

      • 等待日志回放进度。

      • Paxos 成员变更；对于迁移，先将迁移目的端加入成员列表，再移除迁移源端。对于补副本，将目的端加入成员列表。

   9. finish_complete_migration_task 。

      检查是否需要重试，结束任务

下面的内部表展示了一次完整迁移任务的记录。

1. 通过视图 DBA_OB_SERVER_EVENT_HISTORY 获取历史事件所属的模块的第一个，第二个事件属性值的信息。

   obclient> select * from oceanbase.DBA_OB_ROOTSERVICE_EVENT_HISTORY where module like '%dis%' and value1=1002 and value2=1001;
   

   输出结果如下：

   +----------------------------+-------------------+---------------------------+-----------+--------+-------+--------+--------+---------------------+-------------+---------------------+----------------+---------------------+---------+---------------------------------------------+---------------------------------------------+-------------+-------------+
   | gmt_create                 | module            | event                     | name1     | value1 | name2 | value2 | name3  | value3              | name4       | value4              | name5          | value5              | name6   | value6                                      | extra_info                                  | rs_svr_ip   | rs_svr_port |
   +----------------------------+-------------------+---------------------------+-----------+--------+-------+--------+--------+---------------------+-------------+---------------------+----------------+---------------------+---------+---------------------------------------------+---------------------------------------------+-------------+-------------+
   | 2022-06-17 14:31:31.110178 | disaster_recovery | start_migrate_ls_replica  | tenant_id | 1002   | ls_id | 1001   | source | "xx.xxx.x.xx:12348" | destination | "xx.xxx.x.xx:12347" | data_source    | "xx.xxx.x.xx:12348" | comment | migrate replica due to unit group not match |                                             | xx.xxx.x.xx |       12345 |
   | 2022-06-17 14:32:31.743073 | disaster_recovery | finish_migrate_ls_replica | tenant_id | 1002   | ls_id | 1001   | source | "xx.xxx.x.xx:12348" | destination | "xx.xxx.x.xx:12347" | execute_result | 0                   | elapsed | 60641409                                    | migrate replica due to unit group not match | xx.xxx.x.xx |       12345 |
   +----------------------------+-------------------+---------------------------+-----------+--------+-------+--------+--------+---------------------+-------------+---------------------+----------------+---------------------+---------+---------------------------------------------+---------------------------------------------+-------------+-------------+
   

2. 通过视图 DBA_OB_SERVER_EVENT_HISTORY 获取事件历史记录所属的模块的信息。

   obclient> select * from oceanbase.__all_server_event_history where module like 'storage_ha' and event like '%schedule%';
   

   输出结果如下：

   +----------------------------+-------------+----------+------------+-----------------------------+-------+-----------+----------+---------------------+-------+---------------------+-------+--------+-------+--------+-------+--------+------------+
   | gmt_create                 | svr_ip      | svr_port | module     | event                       | name1 | value1    | name2    | value2              | name3 | value3              | name4 | value4 | name5 | value5 | name6 | value6 | extra_info |
   +----------------------------+-------------+----------+------------+-----------------------------+-------+-----------+----------+---------------------+-------+---------------------+-------+--------+-------+--------+-------+--------+------------+
   | 2022-06-17 14:31:31.110597 | 11.124.5.18 |    12347 | storage_ha | schedule_ls_migration start | ls_id | {id:1001} | data_src | "xx.xxx.x.xx:12348" | dest  | "xx.xxx.x.xx:12347" |       |        |       |        |       |        |            |
   +----------------------------+-------------+----------+------------+-----------------------------+-------+-----------+----------+---------------------+-------+---------------------+-------+--------+-------+--------+-------+--------+------------+
   

3. 再次通过视图 DBA_OB_SERVER_EVENT_HISTORY 获取历史事件所属的模块的第一个，第二个事件属性值的信息。

   obclient> select * from oceanbase.__all_server_event_history where module like 'storage_ha' and value1 = 1002 and value2=1001;
   

   输出结果如下：

   +----------------------------+-------------+----------+------------+---------------------------------+-----------+--------+-------+--------+-------+---------------------+-------+---------------------+-----------+------------------------------------+-------------------+--------+---------------+
   | gmt_create                 | svr_ip      | svr_port | module     | event                           | name1     | value1 | name2 | value2 | name3 | value3              | name4 | value4              | name5     | value5                             | name6             | value6 | extra_info    |
   +----------------------------+-------------+----------+------------+---------------------------------+-----------+--------+-------+--------+-------+---------------------+-------+---------------------+-----------+------------------------------------+-------------------+--------+---------------+
   | 2022-06-17 14:31:34.831234 | xx.xxx.x.xx |    12347 | storage_ha | initial_migration_task          | tenant_id | 1002   | ls_id | 1001   | src   | "xx.xxx.x.xx:xxxxx" | dst   | "xx.xxx.x.xx:xxxxx" | task_id   | xxxxxxxxxxxxx-xxxxxxxxxxxxxxxx-x-x | is_failed         | 0      | MIGRATE_LS_OP |
   | 2022-06-17 14:31:34.833923 | xx.xxx.x.xx |    12347 | storage_ha | start_migration_task            | tenant_id | 1002   | ls_id | 1001   | src   | "xx.xxx.x.xx:xxxxx" | dst   | "xx.xxx.x.xx:xxxxx" | task_id   | xxxxxxxxxxxxx-xxxxxxxxxxxxxxxx-x-x | data_tablet_count | 1      | MIGRATE_LS_OP |
   | 2022-06-17 14:31:40.843227 | xx.xxx.x.xx |    12347 | storage_ha | sys_tablets_migration_task      | tenant_id | 1002   | ls_id | 1001   | src   | "xx.xxx.x.xx:xxxxx" | dst   | "xx.xxx.x.xx:xxxxx" | task_id   | xxxxxxxxxxxxx-xxxxxxxxxxxxxxxx-x-x | is_failed         | 0      | MIGRATE_LS_OP |
   | 2022-06-17 14:31:45.845290 | xx.xxx.x.xx |    12347 | storage_ha | tablet_migration_task           | tenant_id | 1002   | ls_id | 1001   | src   | "xx.xxx.x.xx:xxxxx" | dst   | "xx.xxx.x.xx:xxxxx" | tablet_id | 49492                              | cost_us           | 1030   | MIGRATE_LS_OP |
   | 2022-06-17 14:31:50.850248 | xx.xxx.x.xx |    12347 | storage_ha | tablet_migration_task           | tenant_id | 1002   | ls_id | 1001   | src   | "xx.xxx.x.xx:xxxxx" | dst   | "xx.xxx.x.xx:xxxxx" | tablet_id | 49491                              | cost_us           | 1125   | MIGRATE_LS_OP |
   | 2022-06-17 14:31:55.855032 | xx.xxx.x.xx |    12347 | storage_ha | tablet_migration_task           | tenant_id | 1002   | ls_id | 1001   | src   | "xx.xxx.x.xx:xxxxx" | dst   | "xx.xxx.x.xx:xxxxx" | tablet_id | 49490                              | cost_us           | 1174   | MIGRATE_LS_OP |
   | 2022-06-17 14:32:00.860459 | xx.xxx.x.xx |    12347 | storage_ha | tablet_migration_task           | tenant_id | 1002   | ls_id | 1001   | src   | "xx.xxx.x.xx:xxxxx" | dst   | "xx.xxx.x.xx:xxxxx" | tablet_id | 49493                              | cost_us           | 1160   | MIGRATE_LS_OP |
   | 2022-06-17 14:32:05.867814 | xx.xxx.x.xx |    12347 | storage_ha | data_tablets_migration_task     | tenant_id | 1002   | ls_id | 1001   | src   | "xx.xxx.x.xx:xxxxx" | dst   | "xx.xxx.x.xx:xxxxx" | task_id   | xxxxxxxxxxxxx-xxxxxxxxxxxxxxxx-x-x | is_failed         | 0      | MIGRATE_LS_OP |
   | 2022-06-17 14:32:10.870567 | xx.xxx.x.xx |    12347 | storage_ha | tablet_group_migration_task     | tenant_id | 1002   | ls_id | 1001   | src   | "xx.xxx.x.xx:xxxxx" | dst   | "xx.xxx.x.xx:xxxxx" | task_id   | xxxxxxxxxxxxx-xxxxxxxxxxxxxxxx-x-x | tablet_count      | 1      | MIGRATE_LS_OP |
   | 2022-06-17 14:32:15.872683 | xx.xxx.x.xx |    12347 | storage_ha | tablet_migration_task           | tenant_id | 1002   | ls_id | 1001   | src   | "xx.xxx.x.xx:xxxxx" | dst   | "xx.xxx.x.xx:xxxxx" | tablet_id | 50001                              | cost_us           | 1068   | MIGRATE_LS_OP |
   | 2022-06-17 14:32:20.876816 | xx.xxx.x.xx |    12347 | storage_ha | migration_finish_task           | tenant_id | 1002   | ls_id | 1001   | src   | "xx.xxx.x.xx:xxxxx" | dst   | "xx.xxx.x.xx:xxxxx" | task_id   | xxxxxxxxxxxxx-xxxxxxxxxxxxxxxx-x-x | is_failed         | 0      | MIGRATE_LS_OP |
   | 2022-06-17 14:32:24.878956 | xx.xxx.x.xx |    12347 | storage_ha | initial_complete_migration_task | tenant_id | 1002   | ls_id | 1001   | src   | "xx.xxx.x.xx:xxxxx" | dst   | "xx.xxx.x.xx:xxxxx" | task_id   | xxxxxxxxxxxxx-xxxxxxxxxxxxxxxx-x-x | is_failed         | 0      | MIGRATE_LS_OP |
   | 2022-06-17 14:32:27.733626 | xx.xxx.x.xx |    12347 | storage_ha | start_complete_migration_task   | tenant_id | 1002   | ls_id | 1001   | src   | "xx.xxx.x.xx:xxxxx" | dst   | "xx.xxx.x.xx:xxxxx" | task_id   | xxxxxxxxxxxxx-xxxxxxxxxxxxxxxx-x-x | is_failed         | 0      | MIGRATE_LS_OP |
   | 2022-06-17 14:32:31.734910 | xx.xxx.x.xx |    12347 | storage_ha | finish_complete_migration_task  | tenant_id | 1002   | ls_id | 1001   | src   | "xx.xxx.x.xx:xxxxx" | dst   | "xx.xxx.x.xx:xxxxx" | task_id   | xxxxxxxxxxxxx-xxxxxxxxxxxxxxxx-x-x | is_failed         | 0      | MIGRATE_LS_OP |
   +----------------------------+-------------+----------+------------+---------------------------------+-----------+--------+-------+--------+-------+---------------------+-------+---------------------+-----------+------------------------------------+-------------------+--------+---------------+
   

排查步骤

1. 查询 DBA_OB_ROOTSERVICE_EVENT_HISTORY 记录，如果可以看到迁移开始记录，说明 RS 已经调度了迁移任务。查看 value6 中记录的任务执行结果，如果任务失败，大概率会记录一个ret_comment告诉我们具体是哪里报错了。如果没有 ret_comment，可以参考该文档后续的步骤来定位原因。需要排查 RS 部分的报错原因，或者任务没有调度的原因，详细内容可参见：OceanBase 数据库 V4.0 版本容灾问题排查（RS 相关）。

   select * from oceanbase.DBA_OB_ROOTSERVICE_EVENT_HISTORY where module like '%disas%' order by TIMESTAMP desc limit 10;
   

2. 查询 __all_server_event_history表，确认：

   1. 迁移/复制任务是否已经在目的端 server 上被调度。

   2. 任务执行到哪个阶段，是否有明显报错。

一般到此阶段即可明确迁移任务的状态，如果有报错或者卡住。用内部表中的 task_id 在迁移目的端搜索对应时间的日志，并找对应的 RD 即可。

日志信息

Migration 选择迁移源端并获取 PalfBaseInfo，用于目的端新建日志流 可在迁移源端搜索以下日志，如有结果，说明 PALF 返回了 PalfBaseInfo 。

# $tenant_id like T1001
grep $tenant_id observer.log | grep 'get_palf_base_info success'

获取其余日志流meta信息日志待补充。

迁移目的端用获取到的 PalfBaseInfo 新建日志流。

# ObLogService 新建 PALF 成功
grep $tenant_id observer.log | grep 'ObLogService create_ls success'

新建迁移目的端日志流

grep $tenant_id observer.log | grep 'create_ls for ha finish'

目的端开启回放和日志同步。

# 迁移模块enable log成功
grep $tenant_id observer.log | grep 'succeed enable log'

迁移模块 enable log 中包括 2 步

1. enable replayservice 。

   grep $tenant_id observer.log | grep 'enable replay success'
   

2. enable palf log sync 。

   grep $tenant_id observer.log | grep 'enable_sync succes'
   

   enable 日志流成功后会创建 tablet group migrate dag 。

   grep $tenant_id observer.log | grep 'succeed generate tablet group dag'
   

   后续 DAG 任务调度不同阶段日志、拷贝 tablet 成功日志待补充。

   迁移模块会在超时时间内循环等待目的端日志同步追上 Leader 。

   # 如果有以下日志，说明目的端日志已经同步
   grep $tenant_id observer.log | grep 'log is sync, stop wait_log_syn'
   

   # 如果没有以上日志，则搜索 palf 日志
   grep $tenant_id observer.log | grep PALF | grep 'id=$ls_id' | grep is_in_sync
   

输出样例

日志已经同步 is_log_sync=true

日志同步后，迁移模块会发起 replace_member 成员变更，将迁移目的端加入 Paxos 成员列表，并将源端从成员列表移除。

• 发起成员变更前会有以下日志。

  grep $tenant_id observer.log | grep 'start clear dag net ctx'
  

• 在迁移目的段，成员变更开始和结束分别对应以下 2 条日志。

  grep $tenant_id observer.log | grep 'config_change start'
  grep $tenant_id observer.log | grep 'membership_change finish'
  

• 迁移结束。

  1. 查询是否有迁移结束的记录

     select * from oceanbase.DBA_OB_ROOTSERVICE_EVENT_HISTORY where module like '%disas%' order by gmt_create desc limit 10;
     

  2. 查询日志流成员列表是否改变。

     select * from oceanbase.__all_virtual_log_stat where tenant_id=1001 and id=1;
     

适用版本

OceanBase 数据库 V4.x 版本。