1 概述 1.1 業務需求 公司共有兩個業務,網上圖書館和一個電商網站。現要求運維設計一個安全架構,本著高可用、廉價的原則。 具體情況如下: 網上圖書館是基於jsp開發; 電商系統是基於php開發; 兩個網站都是使用的mysql資料庫; 要求你提供一個高可用且廉價的架構方案,部署分發業務到外網; 請 ...
1 概述
1.1 業務需求
公司共有兩個業務,網上圖書館和一個電商網站。現要求運維設計一個安全架構,本著高可用、廉價的原則。
具體情況如下:
網上圖書館是基於jsp開發;
電商系統是基於php開發;
兩個網站都是使用的mysql資料庫;
要求你提供一個高可用且廉價的架構方案,部署分發業務到外網;
請畫出拓撲圖及通過虛擬機搭建測試環境。
2 方案設計
2.1 拓撲圖
整體方案拓撲如下:
防火牆及負載均衡各使用兩台伺服器,採用雙主的架構;防火牆主要用於發佈內網資源到公網,負載均衡用於調度;兩台伺服器充當web集群,同時安裝Nginx+Php+Tomcat,WEB集群中的靜態資源存儲在NFS伺服器中,動態資源從MYSQL資料庫伺服器中調取。NFS伺服器中的業務數據實時備份到backup伺服器中,全網的系統及業務關鍵配置定時同步到backup伺服器上。運維管理區的伺服器暫用於批量管理;
2.2 業務規劃
整體業務軟體及IP規劃如下表:
伺服器名 |
物理IP地址 |
虛擬IP地址 |
系統及業務軟體 |
FW01 |
公網10.0.0.100 內網172.16.1.100 |
公網10.0.0.200(電商) 內網172.16.1.210 |
Centos 6.7 系統自帶的iptables |
FW02 |
公網10.0.0.101 內網172.16.1.101 |
公網10.0.0.201(圖書館) 內網172.16.1.211 |
|
Lb01 |
172.16.1.5 |
172.16.1.200(電商) |
Centos 6.7 nginx-1.16.1-1.el6 |
Lb02 |
172.16.1.6 |
172.16.1.201(圖書館) |
Centos 6.7 nginx-1.16.1-1.el6 |
Web01 |
172.16.1.7 |
\ |
Centos 6.7 nginx-1.16.1-1.el6 PHP7.1 tomcat-9.0.24 |
Web02 |
172.16.1.8 |
Centos 6.7 nginx-1.16.1-1.el6 PHP7.1 tomcat-9.0.24 |
|
nfs |
172.16.1.31 |
Centos 6.7 rpcbind-0.2.0-16 nfs-utils-1.2.3-78 sersync |
|
backup |
172.16.1.41 |
Centos 6.7 rsync-3.0.6-12 |
|
mysql |
172.16.1.51 |
Centos 6.7 mysql-5.7-community |
|
manage |
172.16.1.61 |
Centos 6.7 ansible-2.6.17-1 |
3 系統實施
3.1 基礎環境部署
3.1.1 在管理伺服器上配置通過密鑰管理所有伺服器並完成所有伺服器的基礎配置
1.安裝ansible和sshpass軟體:
yum install ansible sshpass -y
2.編寫自動生成密鑰,並把公鑰傳給所有伺服器的腳本:
[root@manage ~]vim /server/scripts/ssh.sh
#!/bin/bash
ssh-keygen -f /root/.ssh/id_rsa -P '' -q
for ip in 5 6 7 8 31 41 51 100 101
do
sshpass -pAdmin@1234 ssh-copy-id -i /root/.ssh/id_rsa.pub "-o StrictHostKeyChecking=no
172.16.1.$ip"
done
3.編寫測試的腳本:
[root@manage ~]#vim /server/scripts/test_ssh.sh
#!/bin/bash
if [ $# -ne 1 ];
then
echo "please input one args"
exit 1
fi
for ip in 5 6 7 8 31 41 51 100 101
do
echo "----------this is 172.16.1.$ip--------------"
ssh [email protected].$ip $1
done
4.執行腳本並測試:
[root@manage ~]sh /server/scripts/ssh.sh
[root@manage ~]#sh /server/scripts/test_ssh.sh w
----------this is 172.16.1.5--------------
09:48:31 up 21 min, 1 user, load average: 0.00, 0.00, 0.00
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
root pts/0 10.0.0.1 09:38 9:49 0.00s 0.00s -bash
----------this is 172.16.1.6--------------
09:48:30 up 21 min, 1 user, load average: 0.00, 0.00, 0.00
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
root pts/0 10.0.0.1 09:38 9:48 0.00s 0.00s -bash
----------this is 172.16.1.7--------------
09:48:30 up 21 min, 1 user, load average: 0.00, 0.00, 0.00
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
root pts/0 10.0.0.1 09:38 9:48 0.00s 0.00s -bash
----------this is 172.16.1.8--------------
09:48:31 up 21 min, 1 user, load average: 0.00, 0.00, 0.00
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
root pts/0 10.0.0.1 09:38 9:48 0.00s 0.00s -bash
----------this is 172.16.1.31--------------
09:48:31 up 21 min, 1 user, load average: 0.00, 0.00, 0.00
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
root pts/0 10.0.0.1 09:38 9:51 0.00s 0.00s -bash
----------this is 172.16.1.41--------------
09:48:31 up 21 min, 1 user, load average: 0.00, 0.00, 0.00
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
root pts/0 10.0.0.1 09:38 9:50 0.00s 0.00s -bash
----------this is 172.16.1.51--------------
09:48:31 up 21 min, 1 user, load average: 0.00, 0.00, 0.00
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
root pts/0 10.0.0.1 09:38 9:50 0.00s 0.00s -bash
----------this is 172.16.1.100--------------
09:48:31 up 20 min, 2 users, load average: 0.00, 0.00, 0.00
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
root tty1 - 09:36 7:49 0.02s 0.01s bash
root pts/0 10.0.0.1 09:40 8:25 0.00s 0.00s -bash
----------this is 172.16.1.101--------------
09:48:32 up 20 min, 2 users, load average: 0.00, 0.00, 0.00
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
root tty1 - 09:36 7:57 0.05s 0.01s bash
root pts/0 10.0.0.1 09:41 7:10 0.00s 0.00s -bash
5.配置ansible主機清單
[root@manage ~]#vim /etc/ansible/hosts
[lb]
172.16.1.5
172.16.1.6
[nfs]
172.16.1.31
[backup]
172.16.1.41
[web]
172.16.1.7
172.16.1.8
[nginx]
172.16.1.5
172.16.1.6
172.16.1.7
172.16.1.8
[fw]
172.16.1.100
172.16.1.101
[keepalievd]
172.16.1.5
172.16.1.6
172.16.1.100
172.16.1.101
6.配置管理伺服器目錄
mkdir -p /ansible/{ backup,lb,nfs,nginx,playbook,web,fw,mysql}
[root@manage ~]#tree /ansible/
/ansible/
├── backup #存放backup伺服器的相關腳本及軟體
├── fw #存放fw伺服器的相關腳本及軟體
├── lb #存放lb伺服器的相關腳本及軟體
├── mysql #存放mysql伺服器的相關腳本及軟體
├── nfs #存放nfs伺服器的相關腳本及軟體
├── nginx #存放nginx軟體
├── playbook #存放劇本
└── web #存放web伺服器的相關腳本及軟體
7.編寫ansible-playbook部署基礎環境
[root@manage /ansible/playbook]#vim jichu.yaml
- hosts: all
tasks:
- name: Add user_www #所有伺服器上增加www用戶
user: name=www uid=1000 create_home=no shell=/sbin/nologin
- name: Mkdir backup #所有伺服器上增加/backup目錄
file: dest=/backup state=directory owner=www group=www recurse=yes
- name: Mkdir data #所有伺服器上增加/data目錄
file: dest=/data state=directory owner=www group=www recurse=yes
- name: Mkdir /server/scripts #所有伺服器上增加/server/scripts目錄
file: dest=/server/scripts state=directory
8.執行jichu.yaml劇本:
[root@manage /ansible/playbook]#ansible-playbook jichu.yaml
PLAY RECAP *****************************************************************************
172.16.1.100 : ok=5 changed=4 unreachable=0 failed=0
172.16.1.101 : ok=5 changed=4 unreachable=0 failed=0
172.16.1.31 : ok=5 changed=4 unreachable=0 failed=0
172.16.1.41 : ok=5 changed=4 unreachable=0 failed=0
172.16.1.5 : ok=5 changed=4 unreachable=0 failed=0
172.16.1.6 : ok=5 changed=4 unreachable=0 failed=0
172.16.1.7 : ok=5 changed=4 unreachable=0 failed=0
172.16.1.8 : ok=5 changed=4 unreachable=0 failed=0
3.2 搭建backup伺服器
1.編寫ansible-playbook部署backup伺服器
[root@manage /ansible/playbook]#vim rsync.yaml
- hosts: backup
tasks:
- name: Copy Rsyncd.conf To Nfs-Server #把rsync配置文件拷貝到backup伺服器
copy: src=/ansible/backup/rsyncd.conf dest=/etc/rsyncd.conf
- name: Touch File /etc/rsync.passwd #在backup伺服器上增加密碼文件
file: dest=/etc/rsync.passwd state=touch mode=600
- name: Add Passwd to /etc/rsync.passwd #在backup伺服器上配置密碼
copy: content="rsync_backup:Admin@1234" dest=/etc/rsync.passwd
- name: Start Rsync.service #在backup伺服器上開啟rsync守護進程
shell: rsync --daemon
- name: Enable Rsync.service ##把開機自啟動腳本拷貝到backup伺服器
copy: src=/ansible/backup/rc.local dest=/etc/rc.d/rc.local
2.rsync的配置文件如下:
[root@manage /ansible/playbook]#cat /ansible/backup/rsyncd.conf
uid = www
gid = www
port = 873
fake super = yes
use chroot = no
max connections = 200
timeout = 600
ignore errors
read only = false
list = false
#hosts allow = 10.0.0.0/24
auth users = rsync_backup
secrets file = /etc/rsync.passwd
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
log file = /var/run/rsync.lock
#################################
[backup]
path = /backup
[data]
path = /data
3.開啟自啟動配置文件如下:
[root@manage /ansible/playbook]#cat /ansible/backup/rc.local
touch /var/lock/subsys/local
>/etc/udev/rules.d/70-persistent-net.rules
rsync --daemon
4.執行rsync.yaml劇本,結果如下:
[root@manage /ansible/playbook]#ansible-playbook rsync.yaml
PLAY RECAP ***************************************************************************
172.16.1.41 : ok=6 changed=5 unreachable=0 failed=0
3.3 搭建NFS伺服器,並把web伺服器的/data目錄掛載到nfs伺服器上用於測試
1.編寫ansible-playbook部署nfs及web伺服器
[root@manage /ansible/playbook]#vim nfs.yaml
- hosts: nfs
tasks:
- name: Install Rpcbind #安裝Rpcbind
yum: name=rpcbind state=installed
- name: Install nfs-utls #安裝nfs-utls
yum: name=nfs-utils state=installed
- name: Copy Config-file to nfs #把nfs配置文件拷貝到nfs伺服器上
copy: src=/ansible/nfs/exports dest=/etc/exports
notify: Reload nfs #更改配置後,刷新NFS服務
- name: Start rpcbind and nfs-utils #開啟rpcbind and nfs-utils服務
shell: /etc/init.d/rpcbind start && /etc/init.d/nfs start
- name: Enable Rpcbind and Nfs-utils #把開機自啟動文件拷貝到NFS伺服器上
copy: src=/ansible/nfs/rc.local dest=/etc/rc.d/rc.local
handlers:
- name: Reload nfs
shell: exportfs -rv
- hosts: web
tasks:
- name: Install Rpcbind #安裝Rpcbind
yum: name=rpcbind state=installed
- name: Install nfs-utls #安裝nfs-utls
yum: name=nfs-utils state=installed
- name: Start rpcbind and nfs-utils #開啟rpcbind and nfs-utils服務
shell: /etc/init.d/rpcbind start && /etc/init.d/nfs start
- name: Mount Dir_data to Nfs #掛載目錄
mount: src=172.16.1.31:/data path=/data fstype=nfs state=mounted
- name: Enable Rpcbind and Nfs-utils #把開機自啟動文件拷貝到web伺服器上
copy: src=/ansible/web/rc.local dest=/etc/rc.d/rc.local
2.NFS伺服器配置文件如下
[root@manage /ansible/playbook]#cat /ansible/nfs/exports
/data 172.16.1.0/24(rw,sync,all_squash,anonuid=1000,anongid=1000)
3.NFS伺服器開機自啟動文件如下:
[root@manage /ansible/playbook]#cat /ansible/nfs/rc.local
touch /var/lock/subsys/local
>/etc/udev/rules.d/70-persistent-net.rules
/etc/init.d/rpcbind start
/etc/init.d/nfs start
4.Web伺服器開機自啟動文件如下:
[root@manage /ansible/playbook]#cat /ansible/web/rc.local
touch /var/lock/subsys/local
>/etc/udev/rules.d/70-persistent-net.rules
/etc/init.d/rpcbind start
/etc/init.d/nfs start
mount -a
5.執行劇本:
[root@manage /ansible/playbook]#ansible-playbook nfs.yaml
PLAY RECAP **************************************************************************
172.16.1.31 : ok=7 changed=6 unreachable=0 failed=0
172.16.1.7 : ok=6 changed=5 unreachable=0 failed=0
172.16.1.8 : ok=6 changed=5 unreachable=0 failed=0
6.驗證web伺服器是否掛載成功
[root@manage /ansible/playbook]#ansible web -m shell -a "df -h"
172.16.1.8 | SUCCESS | rc=0 >>
172.16.1.31:/data 19G 1.5G 16G 9% /data
172.16.1.7 | SUCCESS | rc=0 >>
172.16.1.31:/data 19G 1.5G 16G 9% /data
3.4 在NFS伺服器上部署sersync軟體
1.編寫ansible-playbook,部署sersync:
[root@manage /ansible/playbook]#vim sersync.yaml
- hosts: nfs
tasks:
- name: Mkdir /usr/local/sersync #增加/usr/local/sersync目錄
file: dest=/usr/local/sersync state=directory
- name: Copy Sersync File To Nfs #把sersync文件拷貝到NFS伺服器上
copy: src=/ansible/nfs/GNU-Linux-x86/ dest=/usr/local/sersync/ mode=755
- name: Touch Rsync_Client_Pass #在NFS伺服器上創建密碼文件
file: dest=/etc/rsync.pass state=touch mode=600
- name: Config Rsync_Client_Pass #在NFS伺服器上配置密碼
copy: content="Admin@1234" dest=/etc/rsync.pass
- name: Start Sersync #在NFS伺服器上啟動sersync
shell: /usr/local/sersync/sersync2 -dro /usr/local/sersync/confxml.xml
- name: Enable Sersync #加入開機自啟動
copy: src=/ansible/nfs/rc.local dest=/etc/rc.d/rc.local
2.編輯sersync配置文件
[root@manage /ansible/playbook]#cat /ansible/nfs/GNU-Linux-x86/confxml.xml
[root@manage /ansible/playbook]#cat /ansible/nfs/GNU-Linux-x86/confxml.xml
<?xml version="1.0" encoding="ISO-8859-1"?>
<head version="2.5">
<host hostip="localhost" port="8008"></host>
<debug start="false"/>
<fileSystem xfs="false"/>
<filter start="false">
<exclude expression="(.*)\.svn"></exclude>
<exclude expression="(.*)\.gz"></exclude>
<exclude expression="^info/*"></exclude>
<exclude expression="^static/*"></exclude>
</filter>
<inotify>
<delete start="true"/>
<createFolder start="true"/>
<createFile start="true"/>
<closeWrite start="true"/>
<moveFrom start="true"/>
<moveTo start="true"/>
<attrib start="true"/>
<modify start="true"/>
</inotify>
<sersync>
<localpath watch="/data">
<remote ip="172.16.1.41" name="data"/>
<!--<remote ip="192.168.8.39" name="tongbu"/>-->
<!--<remote ip="192.168.8.40" name="tongbu"/>-->
</localpath>
<rsync>
<commonParams params="-az"/>
<auth start="true" users="rsync_backup" passwordfile="/etc/rsync.pass"/>
<userDefinedPort start="false" port="874"/><!-- port=874 -->
<timeout start="true" time="100"/><!-- timeout=100 -->
<ssh start="false"/>
</rsync>
<failLog path="/tmp/rsync_fail_log.sh" timeToExecute="60"/><!--default every 60mins execute once-->
<crontab start="false" schedule="600"><!--600mins-->
<crontabfilter start="false">
<exclude expression="*.php"></exclude>
<exclude expression="info/*"></exclude>
</crontabfilter>
</crontab>
<plugin start="false" name="command"/>
</sersync>
<plugin name="command">
<param prefix="/bin/sh" suffix="" ignoreError="true"/> <!--prefix /opt/tongbu/mmm.sh
suffix-->
<filter start="false">
<include expression="(.*)\.php"/>
<include expression="(.*)\.sh"/>
</filter>
</plugin>
<plugin name="socket">
<localpath watch="/opt/tongbu">
<deshost ip="192.168.138.20" port="8009"/>
</localpath>
</plugin>
<plugin name="refreshCDN">
<localpath watch="/data0/htdocs/cms.xoyo.com/site/">
<cdninfo domainname="ccms.chinacache.com" port="80" username="xxxx"
passwd="xxxx"/>
<sendurl base="http://pic.xoyo.com/cms"/>
<regexurl regex="false"
match="cms.xoyo.com/site([/a-zA-Z0-9]*).xoyo.com/images"/>
</localpath>
</plugin>
</head>
3.編輯NFS伺服器的自啟動腳本
[root@manage /ansible/playbook]#cat /ansible/nfs/rc.local
touch /var/lock/subsys/local
>/etc/udev/rules.d/70-persistent-net.rules
/etc/init.d/rpcbind start
/etc/init.d/nfs start
/usr/local/sersync/sersync2 -dro /usr/local/sersync/confxml.xml
4.執行劇本:
[root@manage /ansible/playbook]#ansible-playbook sersync.yaml
5.驗證:在web01的/data目錄下創建文件,檢查是否同步到backup伺服器的/data目錄中。
[root@manage /ansible/playbook]#ansible 172.16.1.7 -m shell -a "touch /data/web01"
172.16.1.7 | SUCCESS | rc=0 >>
[root@manage /ansible/playbook]#ansible backup -m shell -a "ls /data/"
172.16.1.41 | SUCCESS | rc=0 >>
web01
3.5 在lb及web伺服器上安裝nginx
1.編寫ansible-playbook,部署nginx:
[root@manage /ansible/playbook]#vim nginx.yaml
- hosts: nginx
tasks:
- name: Copy Nginx.repo To web_lb #把yum文件拷貝到各伺服器上
copy: src=/etc/yum.repos.d/nginx.repo dest=/etc/yum.repos.d/
- name: Install Nginx #各伺服器上安裝nginx
yum: name=nginx state=installed
- name: Copy Nginx_conf To Nginx_server #更改nginx基礎配置後,重啟nginx服務
copy: src=/ansible/nginx/nginx.conf dest=/etc/nginx/
notify: Restart Nginx_server
- name: Start Nginx #啟動nginx服務
shell: /etc/init.d/nginx start
handlers: #重啟nginx服務
- name: Restart Nginx_server
shell: /etc/init.d/nginx restart
- hosts: web
tasks:
- name: Enable Web_Nginx #nginx加入web伺服器開機自啟動
copy: src=/ansible/web/rc.local dest=/etc/rc.d/rc.local
- hosts: lb
tasks:
- name: Enable Lb_Nginx #nginx加入lb伺服器開機自啟動
copy: src=/ansible/lb/rc.local dest=/etc/rc.d/rc.local
2.yum源配置文件如下:
[nginx-stable]
name=nginx stable repo
baseurl=http://nginx.org/packages/centos/$releasever/$basearch/
gpgcheck=1
enabled=1
gpgkey=https://nginx.org/keys/nginx_signing.key
3.nginx基礎配置如下,更改nginx進程的用戶為www:
[root@manage /ansible/playbook]#cat /ansible/nginx/nginx.conf
user www;
···略
4.Web伺服器的開機自啟動文件如下:
[root@manage /ansible/playbook]#cat /ansible/web/rc.local
touch /var/lock/subsys/local
>/etc/udev/rules.d/70-persistent-net.rules
/etc/init.d/rpcbind start
/etc/init.d/nfs start
mount -a
/etc/init.d/nginx start
5.Lb伺服器的開機自啟動文件如下:
[root@manage /ansible/playbook]#cat /ansible/lb/rc.local
touch /var/lock/subsys/local
>/etc/udev/rules.d/70-persistent-net.rules
/etc/init.d/nginx start
6.執行劇本:
[root@manage /ansible/playbook]#ansible-playbook nginx.yaml
PLAY RECAP ******************************************************************************
172.16.1.5 : ok=8 changed=6 unreachable=0 failed=0
172.16.1.6 : ok=8 changed=6 unreachable=0 failed=0
172.16.1.7 : ok=8 changed=5 unreachable=0 failed=0
172.16.1.8 : ok=8 changed=5 unreachable=0 failed=0
7.驗證是否安裝成功:
[root@manage /ansible/playbook]#ansible nginx -m shell -a "rpm -qa nginx "
172.16.1.7 | SUCCESS | rc=0 >>
nginx-1.16.1-1.el6.ngx.x86_64
172.16.1.8 | SUCCESS | rc=0 >>
nginx-1.16.1-1.el6.ngx.x86_64
172.16.1.5 | SUCCESS | rc=0 >>
nginx-1.16.1-1.el6.ngx.x86_64
172.16.1.6 | SUCCESS | rc=0 >>
nginx-1.16.1-1.el6.ngx.x86_64
8.驗證nginx進程的用戶
[root@manage /ansible/playbook]#ansible nginx -m shell -a "ps -aux | grep nginx"
172.16.1.8 | SUCCESS | rc=0 >>
www 28424 0.0 0.1 47752 1812 ? S 11:01 0:00 nginx: worker process
172.16.1.6 | SUCCESS | rc=0 >>
www 27507 0.0 0.3 47752 1808 ? S 11:01 0:00 nginx: worker process
172.16.1.7 | SUCCESS | rc=0 >>
www 28490 0.0 0.1 47752 1804 ? S 11:01 0:00 nginx: worker process
172.16.1.5 | SUCCESS | rc=0 >>
www 27506 0.0 0.3 47752 1808 ? S 11:01 0:00 nginx: worker process
3.6 在WEB伺服器上安裝PHP
兩台伺服器配置一致:
1.安裝yum源
[root@web01 ~]#yum localinstall http://mirror.webtatic.com/yum/el6/latest.rpm -y
2.安裝PHP
[root@