本文讲述 OceanBase 数据库如何从系统表查看切主记录以及从 RS 日志里找到切主原因。
适用版本
OceanBase 数据库 V2.x 和 V3.x 版本。
操作步骤
查看切主记录的 SQL 语句如下。
MySQL [oceanbase]> select * from __all_virtual_election_event_history
where gmt_create between '2022-05-31 15:30:00' and '2022-05-31 15:35:00'
and table_id=1106108697592658 order by gmt_create;
以下为结果演示。
+----------------------------+-------------+----------+------------------+---------------+-----------------+--------------------+--------------------+
| gmt_create | svr_ip | svr_port | table_id | partition_idx | event | leader | info |
+----------------------------+-------------+----------+------------------+---------------+-----------------+--------------------+--------------------+
| 2022-05-31 15:32:19.849628 | 11.xx.xx.xx | 2882 | 1106108697592658 | 12 | leader revoke | "10.xx.xx.xx:2882" | lease expire |
| 2022-05-31 15:32:19.849849 | 12.xx.xx.xx | 2882 | 1106108697592658 | 12 | leader revoke | "10.xx.xx.xx:2882" | lease expire |
| 2022-05-31 15:32:19.869719 | 11.xx.xx.xx | 2882 | 1106108697592658 | 15 | leader revoke | "10.xx.xx.xx:2882" | lease expire |
| 2022-05-31 15:32:19.869845 | 12.xx.xx.xx | 2882 | 1106108697592658 | 15 | leader revoke | "10.xx.xx.xx:2882" | lease expire |
| 2022-05-31 15:32:19.920361 | 11.xx.xx.xx | 2882 | 1106108697592658 | 9 | leader revoke | "10.xx.xx.xx:2882" | lease expire |
| 2022-05-31 15:32:19.920464 | 12.xx.xx.xx | 2882 | 1106108697592658 | 9 | leader revoke | "10.xx.xx.xx:2882" | lease expire |
| 2022-05-31 15:32:20.050355 | 12.xx.xx.xx | 2882 | 1106108697592658 | 18 | leader revoke | "10.xx.xx.xx:2882" | lease expire |
| 2022-05-31 15:32:20.050413 | 11.xx.xx.xx | 2882 | 1106108697592658 | 18 | leader revoke | "10.xx.xx.xx:2882" | lease expire |
| 2022-05-31 15:32:20.209185 | 12.xx.xx.xx | 2882 | 1106108697592658 | 2 | leader revoke | "10.xx.xx.xx:2882" | lease expire |
| 2022-05-31 15:32:20.210027 | 11.xx.xx.xx | 2882 | 1106108697592658 | 2 | leader revoke | "10.xx.xx.xx:2882" | lease expire |
| 2022-05-31 15:32:20.329250 | 12.xx.xx.xx | 2882 | 1106108697592658 | 6 | leader revoke | "10.xx.xx.xx:2882" | lease expire |
| 2022-05-31 15:32:20.330478 | 11.xx.xx.xx | 2882 | 1106108697592658 | 6 | leader revoke | "10.xx.xx.xx:2882" | lease expire |
| 2022-05-31 15:32:24.851006 | 12.xx.xx.xx | 2882 | 1106108697592658 | 12 | leader takeover | "12.xx.xx.xx:2882" | handle devote succ |
| 2022-05-31 15:32:24.851434 | 11.xx.xx.xx | 2882 | 1106108697592658 | 12 | leader takeover | "12.xx.xx.xx:2882" | handle devote succ |
| 2022-05-31 15:32:24.852048 | 10.xx.xx.xx | 2882 | 1106108697592658 | 12 | leader takeover | "12.xx.xx.xx:2882" | handle devote succ |
| 2022-05-31 15:32:24.869953 | 12.xx.xx.xx | 2882 | 1106108697592658 | 15 | leader takeover | "12.xx.xx.xx:2882" | handle devote succ |
| 2022-05-31 15:32:24.872091 | 10.xx.xx.xx | 2882 | 1106108697592658 | 15 | leader takeover | "12.xx.xx.xx:2882" | handle devote succ |
| 2022-05-31 15:32:24.872556 | 11.xx.xx.xx | 2882 | 1106108697592658 | 15 | leader takeover | "12.xx.xx.xx:2882" | handle devote succ |
| 2022-05-31 15:32:24.920534 | 12.xx.xx.xx | 2882 | 1106108697592658 | 9 | leader takeover | "12.xx.xx.xx:2882" | handle devote succ |
| 2022-05-31 15:32:24.921057 | 11.xx.xx.xx | 2882 | 1106108697592658 | 9 | leader takeover | "12.xx.xx.xx:2882" | handle devote succ |
| 2022-05-31 15:32:24.921685 | 10.xx.xx.xx | 2882 | 1106108697592658 | 9 | leader takeover | "12.xx.xx.xx:2882" | handle devote succ |
| 2022-05-31 15:32:25.051052 | 12.xx.xx.xx | 2882 | 1106108697592658 | 18 | leader takeover | "12.xx.xx.xx:2882" | handle devote succ |
| 2022-05-31 15:32:25.053383 | 11.xx.xx.xx | 2882 | 1106108697592658 | 18 | leader takeover | "12.xx.xx.xx:2882" | handle devote succ |
| 2022-05-31 15:32:25.053749 | 10.xx.xx.xx | 2882 | 1106108697592658 | 18 | leader takeover | "12.xx.xx.xx:2882" | handle devote succ |
| 2022-05-31 15:32:25.209891 | 12.xx.xx.xx | 2882 | 1106108697592658 | 2 | leader takeover | "12.xx.xx.xx:2882" | handle devote succ |
| 2022-05-31 15:32:25.211098 | 10.xx.xx.xx | 2882 | 1106108697592658 | 2 | leader takeover | "12.xx.xx.xx:2882" | handle devote succ |
| 2022-05-31 15:32:25.211619 | 11.xx.xx.xx | 2882 | 1106108697592658 | 2 | leader takeover | "12.xx.xx.xx:2882" | handle devote succ |
| 2022-05-31 15:32:25.330874 | 12.xx.xx.xx | 2882 | 1106108697592658 | 6 | leader takeover | "12.xx.xx.xx:2882" | handle devote succ |
| 2022-05-31 15:32:25.332071 | 11.xx.xx.xx | 2882 | 1106108697592658 | 6 | leader takeover | "12.xx.xx.xx:2882" | handle devote succ |
| 2022-05-31 15:32:25.332664 | 10.xx.xx.xx | 2882 | 1106108697592658 | 6 | leader takeover | "12.xx.xx.xx:2882" | handle devote succ |
| 2022-05-31 15:32:31.001707 | 12.xx.xx.xx | 2882 | 1106108697592658 | 12 | leader revoke | "12.xx.xx.xx:2882" | old leader revoke |
| 2022-05-31 15:32:31.001707 | 12.xx.xx.xx | 2882 | 1106108697592658 | 12 | leader takeover | "11.xx.xx.xx:2882" | renew lease succ |
| 2022-05-31 15:32:31.001707 | 12.xx.xx.xx | 2882 | 1106108697592658 | 2 | leader revoke | "12.xx.xx.xx:2882" | old leader revoke |
| 2022-05-31 15:32:31.001707 | 12.xx.xx.xx | 2882 | 1106108697592658 | 2 | leader takeover | "11.xx.xx.xx:2882" | renew lease succ |
| 2022-05-31 15:32:31.003312 | 10.xx.xx.xx | 2882 | 1106108697592658 | 12 | leader takeover | "11.xx.xx.xx:2882" | change leader succ |
| 2022-05-31 15:32:31.003312 | 10.xx.xx.xx | 2882 | 1106108697592658 | 2 | leader takeover | "11.xx.xx.xx:2882" | change leader succ |
| 2022-05-31 15:32:31.003557 | 11.xx.xx.xx | 2882 | 1106108697592658 | 2 | leader takeover | "11.xx.xx.xx:2882" | change leader succ |
| 2022-05-31 15:32:31.003557 | 11.xx.xx.xx | 2882 | 1106108697592658 | 12 | leader takeover | "11.xx.xx.xx:2882" | change leader succ |
| 2022-05-31 15:32:31.003822 | 12.xx.xx.xx | 2882 | 1106108697592658 | 18 | leader takeover | "11.xx.xx.xx:2882" | renew lease succ |
| 2022-05-31 15:32:31.003822 | 12.xx.xx.xx | 2882 | 1106108697592658 | 18 | leader revoke | "12.xx.xx.xx:2882" | old leader revoke |
| 2022-05-31 15:32:31.005426 | 10.xx.xx.xx | 2882 | 1106108697592658 | 18 | leader takeover | "11.xx.xx.xx:2882" | change leader succ |
| 2022-05-31 15:32:31.005672 | 11.xx.xx.xx | 2882 | 1106108697592658 | 18 | leader takeover | "11.xx.xx.xx:2882" | change leader succ |
+----------------------------+-------------+----------+------------------+---------------+-----------------+--------------------+--------------------+
其中 info 列的解释如下。
assign leader 建分区初始 leader 上任
handle devote succ 无主选举的 leader 上任
renew lease succ leader 续约成功
change leader succ 主动改选场景下新主上任
query leader succ 询问其他副本谁是 leader，收到了回应
lease expire leader lease 过期，卸任，leader/follower 都会记录
not candidate anymore 自己无法当主，记录一个 event
old leader revoke 主动改选场景下旧主卸任
示例中的选举过程解释如下。
10.xx.xx.xx 续约失败，导致 follower 租约过期，产生了一次无主选举，紧接着进行了一次主动改选。
那如何知道为什么第一次 10.xx.xx.xx 会续约失败，需要观察这台机器的 election.log，并查看 ERROR 日志，可以看到原因为 revoke reason="disk error"。
[2022-05-31 15:32:10.405269] ERROR [ELECT] leader_revoke (ob_election.cpp:1986) [44778][Y0-0000000000000000] [log=20] leader_revoke, please attention!(revoke reason="disk error", election={partition:{tid:1107208209170686, partition_id:0, part_cnt:0}, is_running:true, is_changing_leader:false, self:"10.xx.xx.xx:2882", proposal_leader:"0.0.0.0", cur_leader:"10.xx.xx.xx:2882", curr_candidates:3{server:"11.xx.xx.xx:2882", timestamp:1652078072614852, flag:0, need_encrypt:false}{server:"12.xx.xx.xx:2882", timestamp:1652078072614852, flag:0, need_encrypt:false}{server:"10.xx.xx.xx:2882", timestamp:1652078072614852, flag:0, need_encrypt:false}, curr_membership_version:0, leader_lease:[1652078078000000, 1652078088200000], election_time_offset:90000, active_timestamp:1652078072724378, T1_timestamp:1652078078000000, leader_epoch:1652078072400000, state:1, role:1, stage:1, type:-1, replica_num:3, unconfirmed_leader:"10.xx.xx.xx:2882", unconfirmed_leader_lease:[1652078078000000, 1652078088800000], takeover_t1_timestamp:1652078078000000, is_need_query:false, valid_candidates:2{server:"11.xx.xx.xx:2882", timestamp:-1, flag:0, need_encrypt:false}{server:"10.xx.xx.xx:2882", timestamp:-1, flag:0, need_encrypt:false}, change_leader_timestamp:0, ignore_log:false, leader_revoke_timestamp:1652078072724378, vote_period:4, lease_time:9800000, move_out_timestamp:0, eg_part_array_idx:96, eg_id_hash:10254784234596374943})
