kafka2.3.1+zookeeper3.5.6+kafka-manager2.0.0.2集群部署（centos7.7） ...
一、準備三台伺服器，配置好主機名和ip地址
二、伺服器初始化：包括安裝常用命令工具，修改系統時區，校對系統時間，關閉selinux，關閉firewalld，修改主機名，修改系統文件描述符，優化內核參數，優化數據盤掛載參數
1、安裝常用命令工具
yum install vim net-tools bash-completion wget unzip ntp bzip2 epel-release -y
2、修改系統時區，校對系統時間
timedatectl set-timezone Asia/Shanghai
ntpdate pool.ntp.org
3、關閉selinux
vim /etc/selinux/config
SELINUX=disabled
4、關閉防火牆
systemctl stop firewalld.service
systemctl disable firewalld.service
5、修改主機名
vim /etc/hostname
kafka-broker1
6、修改系統文件描述符大小
vim /etc/security/limits.conf
最後添加：
* soft nofile 655360
* hard nofile 655360
* soft nproc 655360
* hard nproc 655360
* soft memlock unlimited
* hard memlock unlimited
7.優化內核參數
vim /etc/sysctl.conf
最後添加：
vm.max_map_count = 655360
net.core.rmem_max = 2097152
net.core.wmem_max = 2097152
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_max_syn_backlog = 8192
net.core.netdev_max_backlog = 10000
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
8.優化數據盤掛載參數，我的數據盤掛載在opt下，對應自己數據盤作相應調整
vim /etc/fstab
/dev/mapper/centos-opt /opt xfs defaults,noatime,largeio 0 0
9.重啟系統使配置生效
init 6
三、安裝zookeeper集群
1.因zookeeper和kafka需要java啟動
首先安裝jdk1.8環境
yum install java-1.8.0-openjdk-devel.x86_64 -y
2.Kakfa集群需要依賴ZooKeeper存儲Broker、Topic等信息，所以我們先安裝zookeeper集群
cd /opt
wget https://mirrors.tuna.tsinghua.edu.cn/apache/zookeeper/zookeeper-3.5.6/apache-zookeeper-3.5.6-bin.tar.gz
tar -zxvf apache-zookeeper-3.5.6-bin.tar.gz
mv apache-zookeeper-3.5.6 zookeeper-3.5.6
3.修改zookeeper配置文件
cd zookeeper-3.5.6/conf/
cp zoo_sample.cfg zoo.cfg
vim zoo.cfg
修改：
dataDir=/opt/zookeeper-3.5.6/data
末尾添加集群其他節點信息
server.1=192.168.0.13:2888:3888
server.2=192.168.0.14:2888:3888
server.3=192.168.0.15:2888:3888
4.添加zookeeper數據目錄
創建/opt/zookeeper-3.5.6/data目錄
mkdir /opt/zookeeper-3.5.6/data
在data目錄里創建myid文件，寫上該節點id
然後將opt下zookeeper-3.5.6目錄上傳到其他兩個節點上
scp -r zookeeper-3.5.6 192.168.0.14:/opt
scp -r zookeeper-3.5.6 192.168.0.15:/opt
修改其他兩個節點data下myid文件內容分別為2和3
5.啟動zookeeper服務
三台節點分別啟動zookeeper服務
/opt/zookeeper-3.5.6/bin/zkServer.sh start
6.查看節點zookeeper狀態
/opt/zookeeper-3.5.6/bin/zkServer.sh status
四、安裝kafka集群
1.官網下載kafka
cd /opt
wget http://mirrors.tuna.tsinghua.edu.cn/apache/kafka/2.3.1/kafka_2.12-2.3.1.tgz
tar -zxvf kafka_2.12-2.3.1.tgz
2.修改kafka配置文件
cd kafka_2.12-2.3.1/config
vim server.properties
修改以下參數
broker.id=1
host.name=192.168.0.13
listeners=PLAINTEXT://192.168.0.13:9092
delete.topic.enable=true
log.cleanup.policy=delete
message.max.byte=5242880
default.replication.factor=2
replica.fetch.max.bytes=5242880
auto.create.topics.enable=true
num.network.threads=24
num.io.threads=48
log.dirs=/opt/kafka_2.12-2.3.1/kafka-logs
num.partitions=3
zookeeper.connect=192.168.0.13:2181,192.168.0.14:2181,192.168.0.15:2181
3.優化調整kafka jvm堆記憶體大小
vim /opt/kafka_2.12-2.3.1/bin/kafka-server-start.sh
export KAFKA_HEAP_OPTS="-Xmx5G -Xms5G"
4.開啟kafka JMX監控
vim /opt/kafka_2.12-2.3.1/bin/kafka-server-start.sh
export JMX_PORT="9999"
4.其他兩台節點按照上面步驟同步安裝kafka
5.三台節點分別啟動kafka
/opt/kafka_2.12-2.3.1/bin/kafka-server-start.sh -daemon /opt/kafka_2.12-2.3.1/config/server.properties
五、安裝kafka-manager，方便管理kafka
1.下載kafka-manager源碼
cd /opt
wget https://github.com/yahoo/kafka-manager/archive/2.0.0.2.zip
mv 2.0.0.2.zip kafka-manager-2.0.0.2.zip
2.解壓zip包
unzip kafka-manager-2.0.0.2.zip
cd kafka-manager-2.0.0.2
3.yum安裝sbt(因為kafka-manager需要sbt編譯)
curl https://bintray.com/sbt/rpm/rpm > /etc/yum.repos.d/bintray-sbt-rpm.repo
yum install sbt -y
4.編譯kafka-manager
./sbt clean dist
可能要等好幾個小時。
看到Your package is ready in /opt/kafka-manager-2.0.0.2/target/universal/kafka-manager-2.0.0.2.zip代表編譯成功了。
5.然後將編譯好的zip包拷貝到/opt下，並刪除原來kafka-manager-2.0.0.2文件夾
cp /opt/kafka-manager-2.0.0.2/target/universal/kafka-manager-2.0.0.2.zip /opt
rm -rf kafka-manager-2.0.0.2
unzip kafka-manager-2.0.0.2.zip
cd kafka-manager-2.0.0.2
6. 修改kafka-manager配置
vim conf/application.conf
kafka-manager.zkhosts="192.168.0.13:2181,192.168.0.14:2181,192.168.0.15:2181"
7.啟動kafka-manager
nohup /opt/kafka-manager-2.0.0.2/bin/kafka-manager &>>/dev/null &
8.用瀏覽器訪問http://192.168.0.13:9000/，添加集群
9.添加成功
