1、redis哨兵(Sentinel) 1.1、redis集群介紹 前面文章講的主從複製集群是無法實現master和slave角色的自動切換的,如果master節點出現現redis服務異常、主機斷電、磁碟損壞等問題導致master無法使用,而redis主從複製無法實現自動的故障轉移(將slave 自 ...
1、redis哨兵(Sentinel)
1.1、redis集群介紹
前面文章講的主從複製集群是無法實現master和slave角色的自動切換的,如果master節點出現現redis服務異常、主機斷電、磁碟損壞等問題導致master無法使用,而redis主從複製無法實現自動的故障轉移(將slave 自動提升為新master),需要手動修改環境配置,才能切換到slave redis伺服器,另外當單台Redis伺服器性能無法滿足業務寫入需求的時候,也無法橫向擴展Redis服務的並行寫入性能。
需要解決以上的兩個核心問題:
- master和slave角色的無縫切換,讓業務無感知從而不影響業務使用;
- 可橫向動態擴展Redis伺服器,從而實現多台伺服器並行寫入以實現更高併發的目的。
Redis集群實現的方式:
- 客戶端分片: 由應用決定將不同的KEY發送到不同的Redis伺服器
- 代理分片: 由代理決定將不同的KEY發送到不同的Redis伺服器,代理程式如:codis,twemproxy等
- Redis Cluster
1.2、redis哨兵(Sentinel)的工作原理
Sentinel可以管理多個redis主從集群
Sentinel 進程是用於監控redis集群中Master主伺服器工作的狀態,在Master主伺服器發生故障的時候,可以實現Master和Slave伺服器的切換,保證系統的高可用,此功能在redis2.6+的版本已引用,Redis的哨兵模式到了2.8版本之後就穩定了下來。一般在生產環境也建議使用Redis的2.8版本的以後版本
哨兵(Sentinel) 是一個分散式系統,可以在一個架構中運行多個哨兵(sentinel) 進程,這些進程使用流言協議(gossip protocols)來接收關於Master主伺服器是否下線的信息,並使用投票協議(Agreement Protocols)來決定是否執行自動故障遷移,以及選擇哪個Slave作為新的Master
每個哨兵(Sentinel)進程會向其它哨兵(Sentinel)、Master、Slave定時發送消息,以確認對方是否”活”著,如果發現對方在指定配置時間(此項可配置)內未得到回應,則暫時認為對方已離線,也就是所謂的”主觀認為宕機” (主觀:是每個成員都具有的獨自的而且可能相同也可能不同的意識),英文名稱:Subjective Down,簡稱SDOWN
有主觀宕機,對應的有客觀宕機。當“哨兵群”中的多數Sentinel進程在對Master主伺服器做出SDOWN 的判斷,並且通過 SENTINEL is-master-down-by-addr 命令互相交流之後,得出的Master Server下線判斷,這種方式就是“客觀宕機”(客觀:是不依賴於某種意識而已經實際存在的一切事物),英文名稱是:Objectively Down, 簡稱 ODOWN
通過一定的vote演算法,從剩下的slave從伺服器節點中,選一臺提升為Master伺服器節點,然後自動修改相關配置,並開啟故障轉移(failover)
Sentinel 機制可以解決master和slave角色的自動切換問題,但單個 Master 的性能瓶頸問題無法解決,類似於MySQL中的MHA功能
Redis Sentinel中的Sentinel節點個數應該為大於等於3且最好為奇數
客戶端初始化時連接的是Sentinel節點集合,不再是具體的Redis節點,但Sentinel只是配置中心不是代理。
Redis Sentinel節點與普通redis沒有區別,要實現讀寫分離依賴於客戶端程式
redis 3.0之前版本中,生產環境一般使用哨兵模式,3.0後推出redis cluster功能,可以支持更大規模的生產環境sentinel中的三個定時任務:
1.2.1 sentinel中的三個定時任務:
- 每10秒每個sentinel對master和slave執行info,發現slave節點,確認主從關係。
- 每2秒每個sentinel通過master節點的channel交換信息(pub/sub)通過sentinel__:hello頻道交互,交互對節點的“看法”和自身信息
- 每1秒每個sentinel對其他sentinel和redis執行ping
1.3、實現哨兵
環境準備:
準備三台主機搭建主從集群,再在每個機器上搭建sentinel
IP | redis版本 | 主機名 |
10.0.0.101 | redis-5.0.14 | node1 |
10.0.0.102 | redis-5.0.14 | node2 |
10.0.0.103 | redis-5.0.14 | node3 |
1.3.1、實現哨兵需要先實現一下主從複製的架構
哨兵的前提是已經實現了一個redis的主從複製的運行環境,從而實現一個一主兩從基於哨兵的高可用redis架構
註意: master的配置文件中masterauth和slave都必須相同
主從複製上篇文章已經實現了,這裡繼續用上一篇的環境
#主節點 [root@master etc]#redis-cli -a 123456 Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. 127.0.0.1:6379> info replication # Replication role:master connected_slaves:2 slave0:ip=10.0.0.102,port=6379,state=online,offset=14,lag=1 slave1:ip=10.0.0.103,port=6379,state=online,offset=14,lag=0 master_replid:e9310fdb8dd9f91d265d4c9a8621a6879e0262ff master_replid2:0000000000000000000000000000000000000000 master_repl_offset:14 second_repl_offset:-1 repl_backlog_active:1 repl_backlog_size:1048576 repl_backlog_first_byte_offset:1 repl_backlog_histlen:14 127.0.0.1:6379> #slave1節點 [root@slave1 ~]#redis-cli -a 123456 Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. 127.0.0.1:6379> info replication # Replication role:slave master_host:10.0.0.101 master_port:6379 master_link_status:up master_last_io_seconds_ago:8 master_sync_in_progress:0 slave_repl_offset:224 slave_priority:100 slave_read_only:1 connected_slaves:0 master_replid:e9310fdb8dd9f91d265d4c9a8621a6879e0262ff master_replid2:0000000000000000000000000000000000000000 master_repl_offset:224 second_repl_offset:-1 repl_backlog_active:1 repl_backlog_size:1048576 repl_backlog_first_byte_offset:1 repl_backlog_histlen:224 #slave2節點 [root@slave2 ~]#redis-cli -a 123456 Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. 127.0.0.1:6379> info replication # Replication role:slave master_host:10.0.0.101 master_port:6379 master_link_status:up master_last_io_seconds_ago:7 master_sync_in_progress:0 slave_repl_offset:2016 slave_priority:100 slave_read_only:1 connected_slaves:0 master_replid:ee9d24063f9d79d698e875634517923d6a9c2a10 master_replid2:0000000000000000000000000000000000000000 master_repl_offset:2016 second_repl_offset:-1 repl_backlog_active:1 repl_backlog_size:1048576 repl_backlog_first_byte_offset:1 repl_backlog_histlen:2016
1.3.2、編輯哨兵的配置文件
Sentinel實際上是一個特殊的redis伺服器,有些redis指令支持,但很多指令並不支持.預設監聽在26379/tcp埠.所有redis節點使用相同的以下的配置文件.
#這裡因為我是源碼編譯安裝的,所以要人源碼的路徑下找到sentinel.conf將其複製到redis安裝的目錄下 [root@master]# cd /usr/local/src/redis-5.0.14[root@master /usr/local/src/redis-5.0.14# ls 00-RELEASENOTES CONTRIBUTING INSTALL README.md runtest-cluster sentinel.conf TLS.md BUGS COPYING Makefile redis.conf runtest-moduleapi src utils CONDUCT deps MANIFESTO runtest runtest-sentinel tests
[root@master etc]#sed -i "/^#/d" redis-sentinel.conf
[root@master etc]#sed -i "/^$/d" redis-sentinel.conf
[root@master etc]#vim redis-sentinel.con
bind 0.0.0.0 port 26379 daemonize no pidfile /apps/redis/run/redis-sentinel.pid logfile /apps/redis/log/sentinel.log dir /tmp sentinel monitor mymaster 10.0.0.100 6379 2 #mymaster是集群的名稱,此行指定當前mymaster集群中master伺服器的地址和埠 #2為法定人數限制(quorum),即有幾個sentinel認為master down了就進行故障轉移,一般此值是所有sentinel節點(一般總數是>=3的 奇數,如:3,5,7等)的一半以上的整數值,比如,總數是3,即3/2=1.5,取整為2,是master的ODOWN客觀下線的依據 sentinel auth-pass mymaster wm521314 #mymaster集群中master的密碼,註意此行要在上面行的下麵 sentinel down-after-milliseconds mymaster 3000 #(SDOWN)判斷mymaster集群中所有節點的主觀下線的時間,單位:毫秒,建議3000 sentinel parallel-syncs mymaster 1 #發生故障轉移後,可以同時向新master同步數據的slave的數量,數字越小總同步時間越長,但可以減輕新master的負載壓力 sentinel failover-timeout mymaster 180000 #所有slaves指向新的master所需的超時時間,單位:毫秒 sentinel deny-scripts-reconfig yes #禁止修改腳本 [root@master etc]# scp /apps/redis/etc/redis-sentinel.conf 10.0.0.102:/apps/redis/etc/[root@master etc]# scp /apps/redis/etc/redis-sentinel.conf 10.0.0.103:/apps/redis/etc/
三個哨兵伺服器的配置都如下:
[root@slave2 etc]#cat redis-sentinel.conf port 26379 daemonize no pidfile /apps/redis/run/redis-sentinel.pid logfile /apps/redis/log/sentinel.log dir /tmp sentinel monitor mymaster 10.0.0.101 6379 2 sentinel down-after-milliseconds mymaster 30000 sentinel parallel-syncs mymaster 1 sentinel failover-timeout mymaster 180000 sentinel deny-scripts-reconfig yes
1.3.2、啟動哨兵
#我這裡是編譯安裝,所以在所有節點生成新的service文件 [root@master ~]#vim /lib/systemd/system/redis-sentinel.service [Unit] Description=Redis Sentinel After=network.target [Service] ExecStart=/apps/redis/bin/redis-sentinel /apps/redis/etc/redis-sentinel.conf --supervised systemd ExecStop=/bin/kill -s QUIT $MAINPID User=redis Group=redis RuntimeDirectory=redis RuntimeDirectoryMode=0755 [Install] WantedBy=multi-user.target [root@master ~]#scp /lib/systemd/system/redis-sentinel.service 10.0.0.102:/lib/systemd/system/ [root@master ~]#scp /lib/systemd/system/redis-sentinel.service 10.0.0.103:/lib/systemd/system/
#在所有節點修改一下文件的所有者所屬組,不然無法啟動
chown -R redis.redis /apps/redis/
#在所有節點啟動哨兵
[root@master ~]#systemctl daemon-reload
[root@master ~]#systemctl enable --now redis-sentinel.service
1.3.3、驗證哨兵埠
[root@master ~]#ss -nlt State Recv-Q Send-Q Local Address:Port Peer Address:Port Process LISTEN 0 511 0.0.0.0:26379 0.0.0.0:* #已經啟動,另外兩個節點同樣查看 LISTEN 0 511 0.0.0.0:6379 0.0.0.0:* LISTEN 0 1024 127.0.0.53%lo:53 0.0.0.0:* LISTEN 0 128 0.0.0.0:22 0.0.0.0:* LISTEN 0 128 127.0.0.1:6010 0.0.0.0:* LISTEN 0 511 [::]:26379 [::]:* LISTEN 0 128 [::]:22 [::]:* LISTEN 0 128 [::1]:6010 [::]:*
1.3.4、查看哨兵日誌
[root@master ~]#tail -f /apps/redis/log/sentinel.log #查看主節點日誌
37184:X 04 Nov 2022 23:32:07.023 # Configuration loaded
37184:X 04 Nov 2022 23:32:07.023 # systemd supervision requested, but NOTIFY_SOCKET not found
37184:X 04 Nov 2022 23:32:07.024 * Increased maximum number of open files to 10032 (it was originally set to 1024).
37184:X 04 Nov 2022 23:32:07.024 * Running mode=sentinel, port=26379.
37184:X 04 Nov 2022 23:32:07.032 # Sentinel ID is 2f9b5e2cb38fc5b3db19417b9626f0ebcc3a7120
37184:X 04 Nov 2022 23:32:07.032 # +monitor master mymaster 10.0.0.101 6379 quorum 2
37184:X 04 Nov 2022 23:32:07.033 * +slave slave 10.0.0.102:6379 10.0.0.102 6379 @ mymaster 10.0.0.101 6379
37184:X 04 Nov 2022 23:32:07.036 * +slave slave 10.0.0.103:6379 10.0.0.103 6379 @ mymaster 10.0.0.101 6379
37184:X 04 Nov 2022 23:32:07.595 * +sentinel sentinel f505346a3640b4942963b26829e5d99a1fa6691e 10.0.0.102 26379 @ mymaster 10.0.0.101 6379
37184:X 04 Nov 2022 23:32:12.779 * +sentinel sentinel b6aebc495fe5d6e81e57ffd5d4f5bef795b3819a 10.0.0.103 26379 @ mymaster 10.0.0.101 6379
#查看從節點日誌
[root@slave1 ~]#tail -f /apps/redis/log/sentinel.log
6376:X 04 Nov 2022 23:31:59.446 # Configuration loaded
6376:X 04 Nov 2022 23:31:59.446 # systemd supervision requested, but NOTIFY_SOCKET not found
6376:X 04 Nov 2022 23:31:59.446 * Increased maximum number of open files to 10032 (it was originally set to 1024).
6376:X 04 Nov 2022 23:31:59.447 * Running mode=sentinel, port=26379.
6376:X 04 Nov 2022 23:31:59.455 # Sentinel ID is f505346a3640b4942963b26829e5d99a1fa6691e
6376:X 04 Nov 2022 23:31:59.455 # +monitor master mymaster 10.0.0.101 6379 quorum 2
6376:X 04 Nov 2022 23:31:59.457 * +slave slave 10.0.0.102:6379 10.0.0.102 6379 @ mymaster 10.0.0.101 6379
6376:X 04 Nov 2022 23:31:59.458 * +slave slave 10.0.0.103:6379 10.0.0.103 6379 @ mymaster 10.0.0.101 6379
6376:X 04 Nov 2022 23:32:08.989 * +sentinel sentinel 2f9b5e2cb38fc5b3db19417b9626f0ebcc3a7120 10.0.0.101 26379 @ mymaster 10.0.0.101 6379
6376:X 04 Nov 2022 23:32:12.716 * +sentinel sentinel b6aebc495fe5d6e81e57ffd5d4f5bef795b3819a 10.0.0.103 26379 @ mymaster 10.0.0.101 6379
1.3.5、當前sentinel狀態
[root@slave1 ~]#redis-cli -p 26379 127.0.0.1:26379> INFO sentinel # Sentinel sentinel_masters:1 sentinel_tilt:0 sentinel_running_scripts:0 sentinel_scripts_queue_length:0 sentinel_simulate_failure_flags:0 master0:name=mymaster,status=ok,address=10.0.0.101:6379,slaves=2,sentinels=3
1.3.6、驗證數據是否同步
[root@master ~]#redis-cli -a 123456 Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. 127.0.0.1:6379> set k1 v1 OK 127.0.0.1:6379> set k2 v2 OK [root@slave1 ~]#redis-cli -a 123456 Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. 127.0.0.1:6379> get k1 "v1" [root@slave2 ~]#redis-cli -a 123456 Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. 127.0.0.1:6379> get k1 "v1"
1.3.7、製造Redis Master節點故障故障轉移並查看其過程
#停掉master節點的redis服務來模擬節點故障 [root@master ~]#killall redis-server #這時候查看日誌發現主節點已經變成了10.0.0.102機器 [root@master ~]#tail -f /apps/redis/log/sentinel.log 800:X 06 Nov 2022 14:36:38.502 # -sdown sentinel b6aebc495fe5d6e81e57ffd5d4f5bef795b3819a 10.0.0.103 26379 @ mymaster 10.0.0.101 6379 800:X 06 Nov 2022 14:36:38.604 # -sdown slave 10.0.0.103:6379 10.0.0.103 6379 @ mymaster 10.0.0.101 6379 800:X 06 Nov 2022 15:21:26.435 # +sdown master mymaster 10.0.0.101 6379 800:X 06 Nov 2022 15:21:26.500 # +new-epoch 1 800:X 06 Nov 2022 15:21:26.501 # +vote-for-leader f505346a3640b4942963b26829e5d99a1fa6691e 1 800:X 06 Nov 2022 15:21:27.019 # +config-update-from sentinel f505346a3640b4942963b26829e5d99a1fa6691e 10.0.0.102 26379 @ mymaster 10.0.0.101 6379 800:X 06 Nov 2022 15:21:27.019 # +switch-master mymaster 10.0.0.101 6379 10.0.0.102 6379 800:X 06 Nov 2022 15:21:27.019 * +slave slave 10.0.0.103:6379 10.0.0.103 6379 @ mymaster 10.0.0.102 6379 800:X 06 Nov 2022 15:21:27.019 * +slave slave 10.0.0.101:6379 10.0.0.101 6379 @ mymaster 10.0.0.102 6379 800:X 06 Nov 2022 15:21:30.032 # +sdown slave 10.0.0.101:6379 10.0.0.101 6379 @ mymaster 10.0.0.102 6379
#查看各節點sentinel信息
[root@master ~]#redis-cli -p 26379
127.0.0.1:26379> INFO sentinel
# Sentinel
sentinel_masters:1
sentinel_tilt:0
sentinel_running_scripts:0
sentinel_scripts_queue_length:0
sentinel_simulate_failure_flags:0
master0:name=mymaster,status=ok,address=10.0.0.102:6379,slaves=2,sentinels=3
#故障轉移後的sentinel配置文件會被自動修改
[root@slave1 ~]#grep "^[a-Z]" /apps/redis/etc/redis-sentinel.conf
bind 0.0.0.0
port 26379
daemonize no
pidfile "/apps/redis/run/redis-sentinel.pid"
logfile "/apps/redis/log/sentinel.log"
dir "/tmp"
sentinel myid f505346a3640b4942963b26829e5d99a1fa6691e
sentinel deny-scripts-reconfig yes
sentinel monitor mymaster 10.0.0.102 6379 2 #這裡發現主節點已經變成10.0.0.102
sentinel down-after-milliseconds mymaster 3000
sentinel auth-pass mymaster 123456
sentinel config-epoch mymaster 1
protected-mode no
supervised systemd
sentinel leader-epoch mymaster 1
sentinel known-replica mymaster 10.0.0.101 6379
sentinel known-replica mymaster 10.0.0.103 6379
sentinel known-sentinel mymaster 10.0.0.103 26379 b6aebc495fe5d6e81e57ffd5d4f5bef795b3819a
sentinel known-sentinel mymaster 10.0.0.101 26379 2f9b5e2cb38fc5b3db19417b9626f0ebcc3a7120
sentinel current-epoch 1
#進入redis查看現在info replication信息
[root@slave1 ~]#redis-cli -a 123456
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
127.0.0.1:6379> info replication
# Replication
role:master
connected_slaves:1
slave0:ip=10.0.0.103,port=6379,state=online,offset=905513,lag=0
master_replid:2824cc99085826d751e848cd7b04c9f4abc398cb
master_replid2:488fa34f2c093be4886741a38323dd1a0ecf9e03
master_repl_offset:905648
second_repl_offset:537903
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:905648
[root@slave2 ~]#redis-cli -a 123456
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe .
127.0.0.1:6379> INFO replication
# Replication
role:slave
master_host:10.0.0.102
master_port:6379
master_link_status:up
master_last_io_seconds_ago:0
master_sync_in_progress:0
slave_repl_offset:901300
slave_priority:100
slave_read_only:1
connected_slaves:0
master_replid:2824cc99085826d751e848cd7b04c9f4abc398cb
master_replid2:488fa34f2c093be4886741a38323dd1a0ecf9e03
master_repl_offset:901300
second_repl_offset:537903
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:564
repl_backlog_histlen:900737
1.3.8、查看選舉新主的過程日誌監測
[root@slave2 ~]#tail -f /apps/redis/log/sentinel.log 798:X 06 Nov 2022 15:21:26.509 # +sdown master mymaster 10.0.0.101 6379 798:X 06 Nov 2022 15:21:26.534 # +new-epoch 1 798:X 06 Nov 2022 15:21:26.535 # +vote-for-leader f505346a3640b4942963b26829e5d99a1fa6691e 1 798:X 06 Nov 2022 15:21:26.609 # +odown master mymaster 10.0.0.101 6379 #quorum 3/2 798:X 06 Nov 2022 15:21:26.609 # Next failover delay: I will not start a failover before Sun Nov 6 15:27:27 2022 798:X 06 Nov 2022 15:21:27.053 # +config-update-from sentinel f505346a3640b4942963b26829e5d99a1fa6691e 10.0.0.102 26379 @ mymaster 10.0.0.101 6379 798:X 06 Nov 2022 15:21:27.053 # +switch-master mymaster 10.0.0.101 6379 10.0.0.102 6379 798:X 06 Nov 2022 15:21:27.053 * +slave slave 10.0.0.103:6379 10.0.0.103 6379 @ mymaster 10.0.0.102 6379 798:X 06 Nov 2022 15:21:27.053 * +slave slave 10.0.0.101:6379 10.0.0.101 6379 @ mymaster 10.0.0.102 6379 798:X 06 Nov 2022 15:21:30.094 # +sdown slave 10.0.0.101:6379 10.0.0.101 6379 @ mymaster 10.0.0.102 6379
1.3.9、恢復故障的原master重新加入redis集群
#重新其他原master的redis服務,會發現會自動指向新的master [root@master ~]#systemctl restart redis #在原master上觀察狀態 [root@master ~]#redis-cli -a 123456 Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. 127.0.0.1:6379> info replication # Replication role:slave master_host:10.0.0.102 master_port:6379 master_link_status:up master_last_io_seconds_ago:1 master_sync_in_progress:0 slave_repl_offset:942522 slave_priority:100 slave_read_only:1 connected_slaves:0 master_replid:2824cc99085826d751e848cd7b04c9f4abc398cb master_replid2:0000000000000000000000000000000000000000 master_repl_offset:942522 second_repl_offset:-1 repl_backlog_active:1 repl_backlog_size:1048576 repl_backlog_first_byte_offset:936383 repl_backlog_histlen:6140 [root@master ~]#redis-cli -p 26379 127.0.0.1:26379> INFO sentinel # Sentinel sentinel_masters:1 sentinel_tilt:0 sentinel_running_scripts:0 sentinel_scripts_queue_length:0 sentinel_simulate_failure_flags:0 master0:name=mymaster,status=ok,address=10.0.0.102:6379,slaves=2,sentinels=3 #再回到新master節點觀察狀態 [root@slave1 ~]#redis-cli -a 123456 Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. 127.0.0.1:6379> info replication # Replication role:master connected_slaves:2 slave0:ip=10.0.0.103,port=6379,state=online,offset=976915,lag=0 slave1:ip=10.0.0.101,port=6379,state=online,offset=977185,lag=0 master_replid:2824cc99085826d751e848cd7b04c9f4abc398cb master_replid2:488fa34f2c093be4886741a38323dd1a0ecf9e03 master_repl_offset:977185 second_repl_offset:537903 repl_backlog_active:1 repl_backlog_size:1048576 repl_backlog_first_byte_offset:1 repl_backlog_histlen:977185 #在新主查看一下日誌 [root@slave1 ~]#tail -f /apps/redis/log/sentinel.log 786:X 06 Nov 2022 15:21:27.030 * +slave-reconf-sent slave 10.0.0.103:6379 10.0.0.103 6379 @ mymaster 10.0.0.101 6379 786:X 06 Nov 2022 15:21:27.580 * +slave-reconf-inprog slave 10.0.0.103:6379 10.0.0.103 6379 @ mymaster 10.0.0.101 6379 786:X 06 Nov 2022 15:21:27.639 # -odown master mymaster 10.0.0.101 6379 786:X 06 Nov 2022 15:21:28.621 * +slave-reconf-done slave 10.0.0.103:6379 10.0.0.103 6379 @ mymaster 10.0.0.101 6379 786:X 06 Nov 2022 15:21:28.672 # +failover-end master mymaster 10.0.0.101 6379 786:X 06 Nov 2022 15:21:28.672 # +switch-master mymaster 10.0.0.101 6379 10.0.0.102 6379 786:X 06 Nov 2022 15:21:28.672 * +slave slave 10.0.0.103:6379 10.0.0.103 6379 @ mymaster 10.0.0.102 6379 786:X 06 Nov 2022 15:21:28.672 * +slave slave 10.0.0.101:6379 10.0.0.101 6379 @ mymaster 10.0.0.102 6379 786:X 06 Nov 2022 15:21:31.738 # +sdown slave 10.0.0.101:6379 10.0.0.101 6379 @ mymaster 10.0.0.102 6379 786:X 06 Nov 2022 15:54:28.253 # -sdown slave 10.0.0.101:6379 10.0.0.101 6379 @ mymaster 10.0.0.102 6379