Redis主從架構持久化存在一個問題,即前次測試的結論,持久化需要配置在主實例上才能跨越實例保證數據不丟失,這樣以來主實例在持久化數據到硬 盤的過程中,勢必會造成磁碟的I/O等待,經過實際測試,這個持久化寫硬碟的過程給應用程式帶來的影響無法忍受;因而在大多數場景下,會考慮把持久化配置 在從實例上,當 ...
Redis主從架構持久化存在一個問題,即前次測試的結論,持久化需要配置在主實例上才能跨越實例保證數據不丟失,這樣以來主實例在持久化數據到硬 盤的過程中,勢必會造成磁碟的I/O等待,經過實際測試,這個持久化寫硬碟的過程給應用程式帶來的影響無法忍受;因而在大多數場景下,會考慮把持久化配置 在從實例上,當主實例宕機後,通過手動或者自動的方式將從實例提升為主實例,繼續提供服務!當主實例恢復後,先從原從實例上同步數據,同步完成後再恢復到 原始的主從狀態!要實現這種的要求,需要有keepalive的配合,一方面keepalive提供了VIP,可以避免修改應用程式連接,同時redis 實例的配置文件監聽部分也需要修改為全網監聽;另一方面keepalive定時調度腳本來監控主從實例的狀態,根據具體情況進行切換!本文將重點介紹下使 用keepalive實現redis主從自動failover!
環境介紹
操作系統版本均為:rhel5.4 64bit
redis版本:2.6.4
redis實例埠均為:6379
redis實例密碼均為:123
VIP:192.168.1.120
主實例為server11(192.168.1.112)
從實例為server12(192.168.1.113,開啟快照持久化)
一:安裝keepalive軟體,server11安裝完成後直接scp至server12上即可
[root@server11 ~]# wget http://keepalived.org/software/keepalived-1.1.19.tar.gz
[root@server11 ~]# tar -zxvf ../tarbag/keepalived-1.1.19.tar.gz
[root@server11 ~]# cd keepalived-1.1.19/
[root@server11 ~]# ./configure --prefix=/usr/local/keepalived && make && make install
二:配置主節點server11配置文件
[root@server11 ~]# cat /usr/local/keepalived/etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
router_id LVS_DEVEL
}
vrrp_script Monitor_redis {
script "/usr/local/scripts/redis_monitor.sh"
interval 2
weight 2
}
vrrp_instance VI_1{
state MASTER
interface eth0
virtual_router_id 51
mcast_src_ip 192.168.1.112
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass password_123
}
track_script {
Monitor_redis
}
virtual_ipaddress {
192.168.1.120
}
notify_fault /usr/local/scripts/redis_fault.sh
notify_stop /usr/local/scripts/redis_stop.sh
}
三:配置從節點server12配置文件
[root@server12 ~]# cat /usr/local/keepalived/etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
router_id LVS_DEVEL
}
vrrp_script Monitor_redis {
script "/usr/local/scripts/redis_monitor.sh"
interval 2
weight 2
}
vrrp_instance VI_1{
state BACKUP
interface eth0
virtual_router_id 51
mcast_src_ip 192.168.1.113
priority 99
advert_int 1
authentication {
auth_type PASS
auth_pass password_123
}
track_script {
Monitor_redis
}
virtual_ipaddress {
192.168.1.120
}
notify_master /usr/local/scripts/redis_master.sh
notify_backup /usr/local/scripts/redis_backup.sh
notify_fault /usr/local/scripts/redis_fault.sh
notify_stop /usr/local/scripts/redis_stop.sh
}
四:準備相關的腳本,主從實例上都需要存在這些腳本,同時註意腳本需要由可執行許可權
[root@server11 ~]# cat /usr/local/scripts/redis_monitor.sh
#!/bin/bash
ALIVE=$(/usr/local/redis2/bin/redis-cli -h 192.168.1.112 -p 6379 -a 123 PING)
if [ "$ALIVE" == "PONG" ]; then
echo $ALIVE
exit 0
else
echo $ALIVE
killall -9 keepalived
service network restart
exit 1
fi
[root@server11 ~]# sh /usr/local/scripts/redis_monitor.sh
PONG
[root@server11 ~]# cat /usr/local/scripts/redis_master.sh
#!/bin/bash
REDISCLI="/usr/local/redis2/bin/redis-cli -h 192.168.1.112 -p 6379 -a 123"
LOGFILE="/usr/local/redis2/var/keepalived-redis-state.log"
echo "[master]" >> $LOGFILE
date >> $LOGFILE
echo "Being master...." >> $LOGFILE 2>&1
echo "Run SLAVEOF cmd ..." >> $LOGFILE
$REDISCLI SLAVEOF 192.168.1.113 6379 >> $LOGFILE 2>&1
sleep 10
echo "Run SLAVEOF NO ONE cmd ..." >> $LOGFILE
$REDISCLI SLAVEOF NO ONE >> $LOGFILE 2>&1
[root@server11 ~]# cat /usr/local/scripts/redis_backup.sh
#!/bin/bash
REDISCLI="/usr/local/redis2/bin/redis-cli -h 192.168.1.112 -p 6379 -a 123"
LOGFILE="/usr/local/redis2/var/keepalived-redis-state.log"
echo "[backup]" >> $LOGFILE
date >> $LOGFILE
echo "Being slave...." >> $LOGFILE 2>&1
sleep 15
echo "Run SLAVEOF cmd ..." >> $LOGFILE
$REDISCLI SLAVEOF 192.168.1.113 6379 >> $LOGFILE 2>&1
[root@server11 ~]# cat /usr/local/scripts/redis_stop.sh
#!/bin/bash
LOGFILE="/usr/local/redis2/var/keepalived-redis-state.log"
echo "[stop]" >> $LOGFILE
date >> $LOGFILE
[root@server11 ~]# cat /usr/local/scripts/redis_fault.sh
#!/bin/bash
LOGFILE="/usr/local/redis2/var/keepalived-redis-state.log"
echo "[fault]" >> $LOGFILE
date >> $LOGFILE
五:主從實例分別啟動keepalive進程,測試VIP是否正常(這裡就要修改redis配置文件的監聽地址為0.0.0.0)
[root@server11 ~]# /usr/local/keepalived/sbin/keepalived -D -f /usr/local/keepalived/etc/keepalived/keepalived.conf
[root@server11 ~]# tail -f /var/log/messages
Dec 12 09:25:49 server11 Keepalived_healthcheckers[7710]: Configuration is using : 5499 Bytes
Dec 12 09:25:49 server11 Keepalived_healthcheckers[7710]: Using LinkWatch kernel netlink reflector...
Dec 12 09:25:49 server11 Keepalived_vrrp[7712]: VRRP sockpool: [ifindex(2), proto(112), fd(12,13)]
Dec 12 09:25:49 server11 Keepalived_vrrp[7712]: VRRP_Script(Monitor_redis) succeeded
Dec 12 09:25:50 server11 Keepalived_vrrp[7712]: VRRP_Instance(VI_1{) Transition to MASTER STATE
Dec 12 09:25:51 server11 Keepalived_vrrp[7712]: VRRP_Instance(VI_1{) Entering MASTER STATE
Dec 12 09:25:51 server11 Keepalived_vrrp[7712]: VRRP_Instance(VI_1{) setting protocol VIPs.
Dec 12 09:25:51 server11 Keepalived_vrrp[7712]: VRRP_Instance(VI_1{) Sending gratuitous ARPs on eth0 for 192.168.1.120
Dec 12 09:25:51 server11 avahi-daemon[4519]: Registering new address record for 192.168.1.120 on eth0.
Dec 12 09:25:51 server11 Keepalived_healthcheckers[7710]: Netlink reflector reports IP 192.168.1.120 added
Dec 12 09:25:51 server11 Keepalived_vrrp[7712]: Netlink reflector reports IP 192.168.1.120 added
Dec 12 09:25:56 server11 Keepalived_vrrp[7712]: VRRP_Instance(VI_1{) Sending gratuitous ARPs on eth0 for 192.168.1.120
[root@server11 ~]# ip a |grep 192
inet 192.168.1.112/24 brd 192.168.1.255 scope global eth0
inet 192.168.1.120/32 scope global eth0
[root@server12 ~]# /usr/local/keepalived/sbin/keepalived -D -f /usr/local/keepalived/etc/keepalived/keepalived.conf
[root@server12 ~]# tail -f /var/log/messages
Dec 12 09:26:55 server12 Keepalived_healthcheckers[3106]: Configuration is using : 5595 Bytes
Dec 12 09:26:55 server12 Keepalived_vrrp[3108]: VRRP_Instance(VI_1{) Entering BACKUP STATE
Dec 12 09:26:55 server12 Keepalived_healthcheckers[3106]: Using LinkWatch kernel netlink reflector...
Dec 12 09:26:55 server12 Keepalived_vrrp[3108]: VRRP sockpool: [ifindex(2), proto(112), fd(12,13)]
Dec 12 09:26:55 server12 Keepalived_vrrp[3108]: VRRP_Script(Monitor_redis) succeeded
[root@server11 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.120 -a 123 info |grep -A 3 'Replication'
# Replication
role:master
connected_slaves:1
slave0:192.168.1.113,6379,online
六:主實例寫入測試數據,該腳本原則上會寫入25條測試數據,不過由於未優化redis預設併發數,會導致一些寫入請求失敗,最終功寫入231839條測 試數據,占記憶體總大小為25M左右,寫入過程中可以觀察主從實例的持久化文件變化情況,主實例的持久化文件維持在30k,從實例的則不斷的擴展!
[root@server11 ~]# cat test.sh
#!/bin/bash
REDISCLI="/usr/local/redis2/bin/redis-cli -h 192.168.1.120 -a 123 -n 1 SET"
ID=1
while(($ID<50001))
do
INSTANCE_NAME="i-2-$ID-VM"
UUID=`cat /proc/sys/kernel/random/uuid`
PRIVATE_IP_ADDRESS=10.`echo "$RANDOM % 255 + 1" | bc`.`echo "$RANDOM % 255 + 1" | bc`.`echo
"$RANDOM % 255 + 1" | bc`\
CREATED=`date "+%Y-%m-%d %H:%M:%S"`
$REDISCLI vm_instance:$ID:instance_name "$INSTANCE_NAME"
$REDISCLI vm_instance:$ID:uuid "$UUID"
$REDISCLI vm_instance:$ID:private_ip_address "$PRIVATE_IP_ADDRESS"
$REDISCLI vm_instance:$ID:created "$CREATED"
$REDISCLI vm_instance:$INSTANCE_NAME:id "$ID"
ID=$(($ID+1))
done
[root@server11 ~]# sh test.sh
[root@server11 redis2]# /usr/local/redis2/bin/redis-cli -h 192.168.1.120 -a 123 info |egrep
'used_memory_peak_human|db1:keys'
used_memory_peak_human:24.98M
db1:keys=231839,expires=0
七:模擬主實例故障,觀察日誌輸出,驗證從實例是否能成功接管VIP,同時將實例變成讀寫模式
[root@server11 ~]# killall -9 redis-server
[root@server11 ~]# ip a |grep 192
inet 192.168.1.112/24 brd 192.168.1.255 scope global eth0
[root@server11 ~]# ps -ef |grep redis
root 15886 6458 0 09:49 pts/0 00:00:00 grep redis
[root@server11 ~]# ps -ef |grep keep
root 16029 6458 0 09:49 pts/0 00:00:00 grep keep
[root@server12 ~]# tail -f /usr/local/redis2/var/keepalived-redis-state.log
[master]
Wed Dec 12 09:48:52 CST 2012
Being master....
Run SLAVEOF cmd ...
OK Already connected to specified master
Run SLAVEOF NO ONE cmd ...
OK
[root@server12 ~]# tail -f /var/log/messages
Dec 12 09:48:51 server12 Keepalived_vrrp[3108]: VRRP_Instance(VI_1{) Transition to MASTER STATE
Dec 12 09:48:52 server12 Keepalived_vrrp[3108]: VRRP_Instance(VI_1{) Entering MASTER STATE
Dec 12 09:48:52 server12 Keepalived_vrrp[3108]: VRRP_Instance(VI_1{) setting protocol VIPs.
Dec 12 09:48:52 server12 Keepalived_vrrp[3108]: VRRP_Instance(VI_1{) Sending gratuitous ARPs on eth0 for 192.168.1.120
Dec 12 09:48:52 server12 Keepalived_vrrp[3108]: Netlink reflector reports IP 192.168.1.120 added
Dec 12 09:48:52 server12 avahi-daemon[2921]: Registering new address record for 192.168.1.120 on eth0.
Dec 12 09:48:52 server12 Keepalived_healthcheckers[3106]: Netlink reflector reports IP 192.168.1.120 added
Dec 12 09:48:57 server12 Keepalived_vrrp[3108]: VRRP_Instance(VI_1{) Sending gratuitous ARPs on eth0 for 192.168.1.120
[root@server12 ~]# ip a |grep 192
inet 192.168.1.113/24 brd 192.168.1.255 scope global eth0
inet 192.168.1.120/32 scope global eth0
[root@server12 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.120 -a 123 info |grep -A 3 'Replication'
# Replication
role:master
connected_slaves:0
[root@server12 ~]# sh test.sh
[root@server12 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.120 -a 123 info |egrep
'used_memory_peak_human|db1:keys'
used_memory_peak_human:26.78M
db1:keys=249925,expires=0
九:主實例角色的恢復過程,使用shell腳本自動恢復
[root@server11 ~]# ssh-keygen
[root@server11 ~]# cd .ssh/
[root@server11 .ssh]# ssh-copy-id -i id_rsa.pub [email protected]
[root@server11 ~]# cat /usr/local/scripts/recover_mastart.sh
#!/bin/sh
ALIVE=$(/usr/local/redis2/bin/redis-cli -h 192.168.1.113 -p 6379 -a 123 PING)
MDB=/usr/local/redis2/master_dump.rdb
SDB=/usr/local/redis2/slave_dump.rdb
if [ "$ALIVE" == "PONG" ]; then
echo $ALIVE
scp [email protected]:$SDB $MDB
else
echo $ALIVE
exit 1
fi
/usr/local/redis2/bin/redis-server /usr/local/redis2/etc/redis.conf
/usr/local/keepalived/sbin/keepalived -D -f
/usr/local/keepalived/etc/keepalived/keepalived.conf
[root@server11 ~]# chmod +x /usr/local/scripts/recover_mastart.sh
[root@server11 ~]# sh /usr/local/scripts/recover_mastart.sh
十:驗證數據完整性和主從角色恢復情況
[root@server11 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.120 -a 123 info |egrep 'used_memory_peak_human|db1:keys'
used_memory_peak_human:26.78M
db1:keys=249925,expires=0
[root@server11 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.120 -a 123 info |grep -A 3 'Replication'
# Replication
role:master
connected_slaves:1
slave0:192.168.1.113,6379,online
[root@server12 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.113 -a 123 info |grep -A 3 'Replication'
# Replication
role:slave
master_host:192.168.1.112
master_port:6379
[root@server12 ~]# /usr/local/redis2/bin/redis-cli -h 192.168.1.120 -a 123 info |egrep 'used_memory_peak_human|db1:keys'
used_memory_peak_human:26.78M
db1:keys=249925,expires=0
主實例keepalive日誌:
[root@server11 ~]# tail -f /var/log/messages
Dec 12 10:08:13 server11 Keepalived_vrrp[20231]: VRRP sockpool: [ifindex(2), proto(112), fd(11,12)]
Dec 12 10:08:13 server11 Keepalived_vrrp[20231]: VRRP_Script(Monitor_redis) succeeded
Dec 12 10:08:13 server11 Keepalived_vrrp[20231]: VRRP_Instance(VI_1{) Transition to MASTER STATE
Dec 12 10:08:13 server11 Keepalived_vrrp[20231]: VRRP_Instance(VI_1{) Received higher prio advert
Dec 12 10:08:13 server11 Keepalived_vrrp[20231]: VRRP_Instance(VI_1{) Entering BACKUP STATE
Dec 12 10:08:15 server11 Keepalived_vrrp[20231]: VRRP_Instance(VI_1{) forcing a new MASTER election
Dec 12 10:08:16 server11 Keepalived_vrrp[20231]: VRRP_Instance(VI_1{) Transition to MASTER STATE
Dec 12 10:08:17 server11 Keepalived_vrrp[20231]: VRRP_Instance(VI_1{) Entering MASTER STATE
Dec 12 10:08:17 server11 Keepalived_vrrp[20231]: VRRP_Instance(VI_1{) setting protocol VIPs.
Dec 12 10:08:17 server11 Keepalived_healthcheckers[20230]: Netlink reflector reports IP 192.168.1.120 added
Dec 12 10:08:17 server11 Keepalived_vrrp[20231]: VRRP_Instance(VI_1{) Sending gratuitous ARPs on eth0 for 192.168.1.120
Dec 12 10:08:17 server11 Keepalived_vrrp[20231]: Netlink reflector reports IP 192.168.1.120 added
Dec 12 10:08:17 server11 avahi-daemon[4519]: Registering new address record for 192.168.1.120 on eth0.
[root@server11 ~]# ip a |grep 192
inet 192.168.1.112/24 brd 192.168.1.255 scope global eth0
inet 192.168.1.120/32 scope global eth0
從實例keepalive日誌:
[root@server12 ~]# tail -f /var/log/messages
Dec 12 09:56:01 server12 last message repeated 4 times
Dec 12 10:08:13 server12 Keepalived_vrrp[3108]: VRRP_Instance(VI_1{) Received lower prio advert, forcing new election
Dec 12 10:08:13 server12 Keepalived_vrrp[3108]: VRRP_Instance(VI_1{) Sending gratuitous ARPs on eth0 for 192.168.1.120
Dec 12 10:08:15 server12 Keepalived_vrrp[3108]: VRRP_Instance(VI_1{) Received higher prio advert
Dec 12 10:08:15 server12 Keepalived_vrrp[3108]: VRRP_Instance(VI_1{) Entering BACKUP STATE
Dec 12 10:08:15 server12 Keepalived_vrrp[3108]: VRRP_Instance(VI_1{) removing protocol VIPs.
Dec 12 10:08:15 server12 Keepalived_healthcheckers[3106]: Netlink reflector reports IP 192.168.1.120 removed
Dec 12 10:08:15 server12 Keepalived_vrrp[3108]: Netlink reflector reports IP 192.168.1.120 removed
Dec 12 10:08:15 server12 avahi-daemon[2921]: Withdrawing address record for 192.168.1.120 on eth0.
從實例角色轉換日誌:
[root@server12 ~]# tail -f /usr/local/redis2/var/keepalived-redis-state.log
[backup]
Wed Dec 12 10:08:15 CST 2012
Being slave....
Run SLAVEOF cmd ...
OK
為系統而生,為框架而死,為debug奮鬥一輩子; 吃符號的虧,上大小寫的當,最後死在需求上。