适用版本
OceanBase 数据库 V2.x、V3.x 版本。
问题描述
对主备库集群进行容灾演练，执行 Failover 操作，耗时长达 10+ 分钟，影响 RTO。
测试经过如下。
应用发起压力测试，TPS 为 1w 左右。
查看主备库同步延时在 1~2s。
预先设置备集群优化参数。
obclient> alter system set _mini_merge_concurrency=16; obclient> alter system set _ob_minor_merge_schedule_interval = '3S'; obclient> alter system set data_copy_concurrency = 600; obclient> alter system set data_copy_concurrency = 300; obclient> alter system set server_data_copy_in_concurrency = 300; obclient> alter system set migrate_concurrency=64; //增加 cutdata并发通过
kill主集群 OBServer 模拟主集群宕机。在备库执行一次转储。
在备库执行 Failover。
查询
v$ob_cluster_event_history查看切换时间，Failover 过程花费 10 分钟进行分区的闪回操作。obclient> select facility,timestamp from v$ob_cluster_event_history order by timestamp desc;返回结果如下。
+-------------------------------------------+----------------------------+ | facility | timestamp | +-------------------------------------------+----------------------------+ | failover_to primary | 2022-04-08 15:05:10.351119 | | switch c1uster | 2022-04-08 15:05:10.345675 | | switch c1uster | 2022-04-08 15:05:10.286845 | | switch cluster | 2022-04-08 15:05:09.115988 | | flashback end | 2022-04-08 15:05:09.114110 | | do flashback partitionsend | 2022-04-08 15:05:09.114058 | | do flashback partitions | 2022-04-08 15:04:59.668079 | | do flashback partitions | 2022-04-08 15:04:29.651350 | | do flashback partition | 2022-04-08 15:03:59.634803 | | do flashback partitions | 2022-04-08 15:03:29.618243 | | do flashback partitions | 2022-04-08 15:02:59.600997 | | do flashback partitions | 2022-04-08 15:02:29.578329 | | dof1ash back partitions | 2022-04-08 15:01:59.560984 | | do flashback partitipartitions | 2022-04-08 15:01:29.544056 | | do flashback partiti opartitions | 2022-04-08 15:00:59.527327 | | dof1ash back partitions | 2022-04-08 15:00:29.509835 | | do flashback partitions | 2022-04-08 14:59:59.492694 | | do flashback partiti opartitions | 2022-04-08 14:59:29.476264 | | do flashback partitipartitions | 2022-04-08 14:58:59.459317 | | do flashback partitipartitions | 2022-04-08 14:58:29.426171 | | do flashback partitipartitions | 2022-04-08 14:57:59.409291 | | do flashback partitions | 2022-04-08 14:57:29.392145 | | do flashback partitions | 2022-04-08 14:56:59.375697 | | do flashback partiti | 2022-04-08 14:56:29.358629 | | do flashback partitions | 2022-04-08 14:55:59.340919 | | switch cluster | 2022-04-08 14:55:59.051772 | | waitpartitions ready forf1ash backend | 2022-04-08 14:55:59.049907 | | waitflashback.info dump | 2022-04-08 14:55:58.908481 | | prepare partitionsforf1ash backend | 2022-04-08 14:55:07.122053 | | prepare partitionsforflashback | 2022-04-08 14:55:05.679962 | | switch cluster | 2022-04-08 14:55:05.643877 | | switch cluster | 2022-04-08 14:55:05.270201 | | flashback partition send | 2022-04-08 14:55:05.268395 | | do flashback partitions | 2022-04-08 14:55:04.140999 | | do flashback partitions | 2022-04-08 14:54:34.052234 | | switch cluster | 2022-04-08 14:54:33.858247 | | wait partitions ready for flashback end | 2022-04-08 14:54:33.856296 | | waitf1ash back.info_dump | 2022-04-08 14:54:33.712179 | | prepare partitionsforf1ash backend | 2022-04-08 14:54:25.927469 | | prepare partitionsfor f1ash back | 2022-04-08 14:54:24.827807 | | switch cluster | 2022-04-08 14:54:22.609675 | | failover to primary | 2022-04-08 14:53:52.340052 | +-------------------------------------------+----------------------------+
问题原因
在 OceanBase 数据库 V4.0 版本之前，主备切换耗时的最大瓶颈在于 cutdata，即把备库的数据还原到一个全局一致的点。分区数据的还原需要扫描整个分区，无法通过日志裁剪和日志向前向后回滚进行数据补齐，因此耗时受分区大小以及切换期间的交易写入并发影响。V4.0 版本后改为单日志流后可以达到秒级。
解决方法
Failover 切换耗时与主库的分区数、数据量以及交易量有关，在 OceanBase 数据库 V2.x、V3.x 版本能调优的地方不多。除了设置前述的备库参数外，可以通过以下操作来减少 Failover 切换耗时。
- 切换前对主库做一次合并（真实场景不适用）。
- 对日志流水表做 truncate 来清理数据（真实场景不适用）。
- 大表尽可能分区，包括全局索引分区。
