Installating and Initializing a Greenplum Database System...

1 安裝說明

1.1 環境說明

1.2 集群說明

2 準備工作

2.1 Linux用戶

在所有節點上創建greenplum管理員用戶。

groupadd -g 530 gpadmin
useradd -g 530 -u 530 -m -d /home/gpadmin -s /bin/bash gpadmin
chown -R gpadmin:gpadmin /home/gpadmin
echo "gpadmin" | passwd --stdin gpadmin

2.2 主機名和hosts配置

相同的配置先在一個節點上配置，配置完成後在2.6小節中複製到其它節點上。

vi /etc/hosts

192.168.56.10 gp-master
192.168.56.12 gp-sdw1
192.168.56.14 gp-sdw2
192.168.56.16 gp-sdw3

分別對應每一臺主機修改主機名；

vi /etc/sysconfig/network

2.3 防火牆

禁用防火牆；

vi /etc/selinux/config

SELINUX=disabled

service iptables stop
chkconfig iptables off

查看防火牆狀態 service iptables status

2.4 系統資源配置

vi /etc/sysctl.conf

kernel.shmmni = 4096
kernel.shmall = 4000000000
kernel.sem = 250 512000 100 2048
kernel.sysrq = 1
kernel.core_uses_pid = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.msgmni = 2048
net.ipv4.tcp_syncookies = 1
net.ipv4.ip_forward = 0
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_max_syn_backlog = 4096
net.ipv4.conf.defalut.arp_filter = 1
net.ipv4.ip_local_port_range = 1025 65535
net.core.netdev_max_backlog = 10000
net.core.rmem_max = 2097152
net.core.wmem_max = 2097152
#vm.overcommit_memory = 2     ### 測試環境要取消這個，否則oracle啟不來 ### 值為1

使資源文件生效；

sysctl -p

進程數配置；

vi /etc/security/limits.d/90-nproc.conf

*          soft    nproc     131072
root       soft    nproc     unlimited

2.5 暫時啟用gpadmin sudo

因為後面的集群節點上安裝greenplum時會涉及到創建目錄和文件操作，在此臨時啟用sudo，安裝成功後撤銷。

visudo

gpadmin    ALL=(ALL)       ALL
gpadmin    ALL=(ALL)       NOPASSWD:ALL

2.6 複製配置文件到所有節點上

scp /etc/hosts gp-sdw1:/etc
scp /etc/sysctl.conf gp-sdw1:/etc
scp /etc/security/limits.d/90-nproc.conf gp-sdw1:/etc/security/limits.d
scp /etc/selinux/config gp-sdw1:/etc/selinux

重啟操作系統。

3 安裝Greenplum DB

3.1 在Master節點上安裝Greenplum DB

首先在master節點上安裝，設置安裝路徑為/opt/greenplum/greenplum-db-5.0.0-alpha.5；

cd /tmp
unzip greenplum-db-5.0.0-alpha.5-rhel6-x86_64.zip
/tmp/greenplum-db-5.0.0-alpha.5-rhel6-x86_64.bin 

********************************************************************************
Do you accept the Pivotal Database license agreement? [yes|no]
********************************************************************************

yes

********************************************************************************
Provide the installation path for Greenplum Database or press ENTER to 
accept the default installation path: /usr/local/greenplum-db-5.0.0-alpha.5
********************************************************************************

/opt/greenplum/greenplum-db-5.0.0-alpha.5

********************************************************************************
Install Greenplum Database into /opt/greenplum/greenplum-db-5.0.0-alpha.5? [yes|no]
********************************************************************************

yes

********************************************************************************
/opt/greenplum/greenplum-db-5.0.0-alpha.5 does not exist.
Create /opt/greenplum/greenplum-db-5.0.0-alpha.5 ? [yes|no]
(Selecting no will exit the installer)
********************************************************************************

yes

Extracting product to /opt/greenplum/greenplum-db-5.0.0-alpha.5

********************************************************************************
Installation complete.
Greenplum Database is installed in /opt/greenplum/greenplum-db-5.0.0-alpha.5

Pivotal Greenplum documentation is available
for download at http://gpdb.docs.pivotal.io
********************************************************************************

安裝過程中系統會預設創建一個指向greenplum-db-5.0.0-alpha.5的軟鏈接(greenplum-db)；

ls -ltr /opt/greenplum/
total 8
lrwxrwxrwx  1 gpadmin gpadmin   28 May 30 12:14 greenplum-db -> ./greenplum-db-5.0.0-alpha.5
drwxr-xr-x 11 gpadmin gpadmin 4096 May 30 12:19 greenplum-db-5.0.0-alpha.5

修改目錄許可權和所有者為gpadmin；

chown -R gpadmin:gpadmin /opt/greenplum/
chown -R gpadmin:gpadmin /opt/greenplum/greenplum-db

3.2 在Master節點上配置集群host

su - gpadmin
mkdir -p /opt/greenplum/greenplum-db/conf
vi /opt/greenplum/greenplum-db/conf/hostlist

gp-master
gp-sdw1
gp-sdw2
gp-sdw3

創建一個 seg_hosts ，包含所有的Segment Host的主機名；

vi /opt/greenplum/greenplum-db/conf/seg_hosts

gp-sdw1
gp-sdw2
gp-sdw3

3.3 配置SSH免密連接

su - gpadmin
source /opt/greenplum/greenplum-db/greenplum_path.sh   # 不設置報錯Error: unable to import module: No module named gppylib.commands
/opt/greenplum/greenplum-db/bin/gpssh-exkeys -f /opt/greenplum/greenplum-db/conf/hostlist

[STEP 1 of 5] create local ID and authorize on local host
  ... /home/gpadmin/.ssh/id_rsa file exists ... key generation skipped

[STEP 2 of 5] keyscan all hosts and update known_hosts file

[STEP 3 of 5] authorize current user on remote hosts
  ... send to gp-sdw1
  ... send to gp-sdw2
  ... send to gp-sdw3

[STEP 4 of 5] determine common authentication file content

[STEP 5 of 5] copy authentication files to all remote hosts
  ... finished key exchange with gp-sdw1
  ... finished key exchange with gp-sdw2
  ... finished key exchange with gp-sdw3

[INFO] completed successfully

測試ssh gp-sdw1，不需要密碼即可登錄。

3.4 Segment節點上安裝Greenplum DB

在Master節點上遠程創建Segment節點所需的目錄，並更改目錄許可權和所有者為gpadmin；

su - gpadmin
source /opt/greenplum/greenplum-db/greenplum_path.sh 
/opt/greenplum/greenplum-db/bin/gpssh -f /opt/greenplum/greenplum-db/conf/seg_hosts -e -v "sudo mkdir -p /opt/greenplum && sudo chown gpadmin:gpadmin -R /opt/greenplum"

[INFO] login gp-master
[INFO] login gp-sdw1
[INFO] login gp-sdw2
[INFO] login gp-sdw3
[  gp-sdw1] sudo mkdir -p /opt/greenplum && sudo chown gpadmin:gpadmin -R /opt/greenplum
[  gp-sdw2] sudo mkdir -p /opt/greenplum && sudo chown gpadmin:gpadmin -R /opt/greenplum
[  gp-sdw3] sudo mkdir -p /opt/greenplum && sudo chown gpadmin:gpadmin -R /opt/greenplum
[INFO] completed successfully

[Cleanup...]

將Master節點上安裝的Greenplum db文件複製到所有Segment節點上安裝；

su - gpadmin
source /opt/greenplum/greenplum-db/greenplum_path.sh 
/opt/greenplum/greenplum-db/bin/gpseginstall -f /opt/greenplum/greenplum-db/conf/hostlist -u gpadmin -p gpadmin

20170530:12:26:50:004409 gpseginstall:gp-master:gpadmin-[INFO]:-Installation Info:
link_name greenplum-db
binary_path /opt/greenplum/greenplum-db-5.0.0-alpha.5
binary_dir_location /opt/greenplum
binary_dir_name greenplum-db-5.0.0-alpha.5
20170530:12:26:50:004409 gpseginstall:gp-master:gpadmin-[INFO]:-check cluster password access
20170530:12:26:50:004409 gpseginstall:gp-master:gpadmin-[INFO]:-de-duplicate hostnames
20170530:12:26:50:004409 gpseginstall:gp-master:gpadmin-[INFO]:-master hostname: gp-master
20170530:12:26:51:004409 gpseginstall:gp-master:gpadmin-[INFO]:-rm -f /opt/greenplum/greenplum-db-5.0.0-alpha.5.tar; rm -f /opt/greenplum/greenplum-db-5.0.0-alpha.5.tar.gz
20170530:12:26:51:004409 gpseginstall:gp-master:gpadmin-[INFO]:-cd /opt/greenplum; tar cf greenplum-db-5.0.0-alpha.5.tar greenplum-db-5.0.0-alpha.5
20170530:12:26:54:004409 gpseginstall:gp-master:gpadmin-[INFO]:-gzip /opt/greenplum/greenplum-db-5.0.0-alpha.5.tar
20170530:12:27:22:004409 gpseginstall:gp-master:gpadmin-[INFO]:-remote command: mkdir -p /opt/greenplum
20170530:12:27:23:004409 gpseginstall:gp-master:gpadmin-[INFO]:-remote command: rm -rf /opt/greenplum/greenplum-db-5.0.0-alpha.5
20170530:12:27:24:004409 gpseginstall:gp-master:gpadmin-[INFO]:-scp software to remote location
20170530:12:27:40:004409 gpseginstall:gp-master:gpadmin-[INFO]:-remote command: gzip -f -d /opt/greenplum/greenplum-db-5.0.0-alpha.5.tar.gz
20170530:12:28:00:004409 gpseginstall:gp-master:gpadmin-[INFO]:-md5 check on remote location
20170530:12:28:05:004409 gpseginstall:gp-master:gpadmin-[INFO]:-remote command: cd /opt/greenplum; tar xf greenplum-db-5.0.0-alpha.5.tar
20170530:12:28:38:004409 gpseginstall:gp-master:gpadmin-[INFO]:-remote command: rm -f /opt/greenplum/greenplum-db-5.0.0-alpha.5.tar
20170530:12:28:39:004409 gpseginstall:gp-master:gpadmin-[INFO]:-remote command: cd /opt/greenplum; rm -f greenplum-db; ln -fs greenplum-db-5.0.0-alpha.5 greenplum-db
20170530:12:28:40:004409 gpseginstall:gp-master:gpadmin-[INFO]:-rm -f /opt/greenplum/greenplum-db-5.0.0-alpha.5.tar.gz
20170530:12:28:41:004409 gpseginstall:gp-master:gpadmin-[INFO]:-version string on master: gpssh version 5.0.0 alpha.5 build commit:2e87c5aa435c779b2f3837fa8c7273876497f6ba
20170530:12:28:41:004409 gpseginstall:gp-master:gpadmin-[INFO]:-remote command: . /opt/greenplum/greenplum-db/./greenplum_path.sh; /opt/greenplum/greenplum-db/./bin/gpssh --version
20170530:12:28:48:004409 gpseginstall:gp-master:gpadmin-[INFO]:-remote command: . /opt/greenplum/greenplum-db-5.0.0-alpha.5/greenplum_path.sh; /opt/greenplum/greenplum-db-5.0.0-alpha.5/bin/gpssh --version
20170530:12:28:49:004409 gpseginstall:gp-master:gpadmin-[INFO]:-SUCCESS -- Requested commands completed

檢查每個節點安裝和目錄情況；

su - gpadmin
source /opt/greenplum/greenplum-db/greenplum_path.sh 
/opt/greenplum/greenplum-db/bin/gpssh -f /opt/greenplum/greenplum-db/conf/hostlist -e ls -l $GPHOME
```xml
[gp-master] ls -l /opt/greenplum/greenplum-db/.
[gp-master] total 40
[gp-master] drwxr-xr-x 7 gpadmin gpadmin 4096 May 20 02:29 bin
[gp-master] drwxrwxr-x 2 gpadmin gpadmin 4096 May 30 12:21 conf
[gp-master] drwxr-xr-x 4 gpadmin gpadmin 4096 May 20 02:20 docs
[gp-master] drwxr-xr-x 2 gpadmin gpadmin 4096 May 20 02:20 etc
[gp-master] drwxr-xr-x 3 gpadmin gpadmin 4096 May 20 02:20 ext
[gp-master] -rw-r--r-- 1 gpadmin gpadmin  745 May 30 12:14 greenplum_path.sh
[gp-master] drwxr-xr-x 6 gpadmin gpadmin 4096 May 20 02:20 include
[gp-master] drwxr-xr-x 7 gpadmin gpadmin 4096 May 20 02:20 lib
[gp-master] drwxr-xr-x 2 gpadmin gpadmin 4096 May 20 02:27 sbin
[gp-master] drwxr-xr-x 4 gpadmin gpadmin 4096 May 20 02:16 share
[  gp-sdw2] ls -l /opt/greenplum/greenplum-db/.
[  gp-sdw2] total 40
[  gp-sdw2] drwxr-xr-x 7 gpadmin gpadmin 4096 May 20 02:29 bin
[  gp-sdw2] drwxrwxr-x 2 gpadmin gpadmin 4096 May 30 12:21 conf
[  gp-sdw2] drwxr-xr-x 4 gpadmin gpadmin 4096 May 20 02:20 docs
[  gp-sdw2] drwxr-xr-x 2 gpadmin gpadmin 4096 May 20 02:20 etc
[  gp-sdw2] drwxr-xr-x 3 gpadmin gpadmin 4096 May 20 02:20 ext
[  gp-sdw2] -rw-r--r-- 1 gpadmin gpadmin  745 May 30 12:14 greenplum_path.sh
[  gp-sdw2] drwxr-xr-x 6 gpadmin gpadmin 4096 May 20 02:20 include
[  gp-sdw2] drwxr-xr-x 7 gpadmin gpadmin 4096 May 20 02:20 lib
[  gp-sdw2] drwxr-xr-x 2 gpadmin gpadmin 4096 May 20 02:27 sbin
[  gp-sdw2] drwxr-xr-x 4 gpadmin gpadmin 4096 May 20 02:16 share
[  gp-sdw1] ls -l /opt/greenplum/greenplum-db/.
[  gp-sdw1] total 40
[  gp-sdw1] drwxr-xr-x 7 gpadmin gpadmin 4096 May 20 02:29 bin
[  gp-sdw1] drwxrwxr-x 2 gpadmin gpadmin 4096 May 30 12:21 conf
[  gp-sdw1] drwxr-xr-x 4 gpadmin gpadmin 4096 May 20 02:20 docs
[  gp-sdw1] drwxr-xr-x 2 gpadmin gpadmin 4096 May 20 02:20 etc
[  gp-sdw1] drwxr-xr-x 3 gpadmin gpadmin 4096 May 20 02:20 ext
[  gp-sdw1] -rw-r--r-- 1 gpadmin gpadmin  745 May 30 12:14 greenplum_path.sh
[  gp-sdw1] drwxr-xr-x 6 gpadmin gpadmin 4096 May 20 02:20 include
[  gp-sdw1] drwxr-xr-x 7 gpadmin gpadmin 4096 May 20 02:20 lib
[  gp-sdw1] drwxr-xr-x 2 gpadmin gpadmin 4096 May 20 02:27 sbin
[  gp-sdw1] drwxr-xr-x 4 gpadmin gpadmin 4096 May 20 02:16 share
[  gp-sdw3] ls -l /opt/greenplum/greenplum-db/.
[  gp-sdw3] total 40
[  gp-sdw3] drwxr-xr-x 7 gpadmin gpadmin 4096 May 20 02:29 bin
[  gp-sdw3] drwxrwxr-x 2 gpadmin gpadmin 4096 May 30 12:21 conf
[  gp-sdw3] drwxr-xr-x 4 gpadmin gpadmin 4096 May 20 02:20 docs
[  gp-sdw3] drwxr-xr-x 2 gpadmin gpadmin 4096 May 20 02:20 etc
[  gp-sdw3] drwxr-xr-x 3 gpadmin gpadmin 4096 May 20 02:20 ext
[  gp-sdw3] -rw-r--r-- 1 gpadmin gpadmin  745 May 30 12:14 greenplum_path.sh
[  gp-sdw3] drwxr-xr-x 6 gpadmin gpadmin 4096 May 20 02:20 include
[  gp-sdw3] drwxr-xr-x 7 gpadmin gpadmin 4096 May 20 02:20 lib
[  gp-sdw3] drwxr-xr-x 2 gpadmin gpadmin 4096 May 20 02:27 sbin
[  gp-sdw3] drwxr-xr-x 4 gpadmin gpadmin 4096 May 20 02:16 share

創建數據存儲區域目錄；

su - gpadmin
source /opt/greenplum/greenplum-db/greenplum_path.sh
/opt/greenplum/greenplum-db/bin/gpssh -f /opt/greenplum/greenplum-db/conf/hostlist -e 'mkdir -p /opt/greenplum/data'

在master上創建master數據存儲區域；

su - gpadmin
source /opt/greenplum/greenplum-db/greenplum_path.sh
/opt/greenplum/greenplum-db/bin/gpssh -h gp-master -e 'mkdir -p /opt/greenplum/data/master'

在Segment節點上創建數據存儲區域

su - gpadmin
source /opt/greenplum/greenplum-db/greenplum_path.sh
/opt/greenplum/greenplum-db/bin/gpssh -f /opt/greenplum/greenplum-db/conf/seg_hosts -e 'mkdir -p /opt/greenplum/data/primary && mkdir -p /opt/greenplum/data/mirror'

3.5 環境變數配置

gpssh -f /opt/greenplum/greenplum-db/conf/hostlist -e -v "cat >> /home/gpadmin/.bash_profile <<EOF

source /opt/greenplum/greenplum-db/greenplum_path.sh
export MASTER_DATA_DIRECTORY=/opt/greenplum/data/master
export GPPORT=5432
export PGDATABASE=gp_sydb
EOF"

3.6 NTP 配置

啟用master節點上的ntp，併在Segment節點上配置和啟用NTP；

echo "server gp-master perfer" >>/etc/ntp.conf
/opt/greenplum/greenplum-db/bin/gpssh -f /opt/greenplum/greenplum-db/conf/hostlist -v -e 'sudo ntpd'
/opt/greenplum/greenplum-db/bin/gpssh -f /opt/greenplum/greenplum-db/conf/hostlist -v -e 'sudo /etc/init.d/ntpd start && sudo chkconfig --level 35 ntpd on'

4 初始化Greenplum DB

4.1 初始化前檢查

檢查主機名配置；

su gpadmin
source /opt/greenplum/greenplum-db/greenplum_path.sh
gpssh -f /opt/greenplum/greenplum-db/conf/hostlist -e hostname

[  gp-sdw3] hostname
[  gp-sdw3] gp-sdw3
[  gp-sdw1] hostname
[  gp-sdw1] gp-sdw1
[gp-master] hostname
[gp-master] gp-master
[  gp-sdw2] hostname
[  gp-sdw2] gp-sdw2

檢查節點與節點之間文件讀取；

gpcheckperf -h gp-sdw1 -h gp-sdw2 -d /tmp -r d -D -v
gpcheckperf -f /opt/greenplum/greenplum-db/conf/hostlist -d /tmp -r d -D -v
$ gpcheckperf -f /opt/greenplum/greenplum-db/conf/hostlist -r N -d /tmp
/opt/greenplum/greenplum-db/./bin/gpcheckperf -f /opt/greenplum/greenplum-db/conf/hostlist -r N -d /tmp

-------------------
--  NETPERF TEST
-------------------

====================
==  RESULT
====================
Netperf bisection bandwidth test
gp-master -> gp-sdw1 = 72.220000
gp-sdw2 -> gp-sdw3 = 21.470000
gp-sdw1 -> gp-master = 43.510000
gp-sdw3 -> gp-sdw2 = 44.200000

Summary:
sum = 181.40 MB/sec
min = 21.47 MB/sec
max = 72.22 MB/sec
avg = 45.35 MB/sec
median = 44.20 MB/sec

[Warning] connection between gp-sdw2 and gp-sdw3 is no good
[Warning] connection between gp-sdw1 and gp-master is no good
[Warning] connection between gp-sdw3 and gp-sdw2 is no good

4.2 初始化

初始化 Greenplum 配置文件模板都在/opt/greenplum/greenplum-db/docs/cli_help/gpconfigs目錄下,gpinitsystem_config是初始化 Greenplum 的模板，此模板中 Mirror Segment的配置都被註釋；創建一個副本，對其修改；

cd /opt/greenplum/greenplum-db/docs/cli_help/gpconfigs
cp gpinitsystem_config initgp_config
vi initgp_config        

declare -a DATA_DIRECTORY=(/opt/greenplum/data/primary /opt/greenplum/data/primary /opt/greenplum/data/primary)
MASTER_HOSTNAME=gp-master
MASTER_DIRECTORY=/opt/greenplum/data/master
declare -a MIRROR_DATA_DIRECTORY=(/opt/greenplum/data/mirror /opt/greenplum/data/mirror /opt/greenplum/data/mirror)
DATABASE_NAME=gp_sydb
MACHINE_LIST_FILE=/opt/greenplum/greenplum-db/conf/seg_hosts

執行初始化；

gpinitsystem -c initgp_config -S

若初始化失敗，需要刪除數據目錄重新初始化；

5 後續操作

5.1 停止和啟動集群

gpstop -a
gpstart -a

5.2 登錄資料庫

$ psql -d postgres

postgres=# \l # 查詢資料庫
                 List of databases
   Name    |  Owner  | Encoding |  Access privileges  
-----------+---------+----------+---------------------
 gp_sydb   | gpadmin | UTF8     | 
 postgres  | gpadmin | UTF8     | 
 template0 | gpadmin | UTF8     | =c/gpadmin          
                                : gpadmin=CTc/gpadmin
 template1 | gpadmin | UTF8     | =c/gpadmin          
                                : gpadmin=CTc/gpadmin
(4 rows)

postgres=# \l # 查詢資料庫表

5.3 集群狀態

gpstate -e #查看mirror的狀態
gpstate -f #查看standby master的狀態
gpstate -s #查看整個GP群集的狀態
gpstate -i #查看GP的版本
gpstate --help #幫助文檔，可以查看gpstate更多用法