工作案例: 大數據開發用系統腳本自動在kafka建topic,檢查後才發現副本數只有1個,存在數據丟失的風險。需要立刻把副本數改為3個。 開始幹活,首先想到的是下麵的命令: ${BIN_PATH}/kafka-topics.sh --zookeeper ${ZOOKEEPER_ADDR} --top ...
工作案例:
大數據開發用系統腳本自動在kafka建topic,檢查後才發現副本數只有1個,存在數據丟失的風險。需要立刻把副本數改為3個。
開始幹活,首先想到的是下麵的命令:
${BIN_PATH}/kafka-topics.sh --zookeeper ${ZOOKEEPER_ADDR} --topic $1 --partitions $2 --alter
${BIN_PATH}/kafka-topics.sh --zookeeper ${ZOOKEEPER_ADDR} --topic $1 --replication-factor $2 --alter
運行時修改partition成功,修改副本數失敗,錯誤如下:
:2181 --topic test --replication-factor 2 --alter
Option "[replication-factor]" can't be used with option "[alter]"
馬上度娘,有很多方法都是用json文件的方式修改,試驗了下都不成功,直接去官網查權威
https://kafka.apache.org/21/documentation.html#basic_ops_increase_replication_factor
思路沒錯,原來是網上的json文件寫不對。解決方法:
1/ 建一個測試topic
data1/kafka/bin/kafka-topics.sh --zookeeper ${ZOOKEEPER_ADDR} --create --partitions 5 --replication-factor 3 --topic test
2/ 察看test的詳細信息
Topic:test PartitionCount:5 ReplicationFactor:3 Configs:
Topic: test Partition: 0 Leader: 21 Replicas: 21,22,31 Isr: 21,22,31
Topic: test Partition: 1 Leader: 22 Replicas: 21,22,31 Isr: 22,31,21
Topic: test Partition: 2 Leader: 31 Replicas: 21,22,31 Isr: 31,21,22
Topic: test Partition: 3 Leader: 21 Replicas: 21,22,31 Isr: 21,31,22
Topic: test Partition: 4 Leader: 21 Replicas: 21,22,31 Isr: 31,22,21
3/ 準備修改副本的json文件,格式可以找個json生成器查看。[]中是集群的broker-list
[root@zk-kafka-1 ~]# cat increase-replication-factor.json
{"version":1,
"partitions":[
{"topic":"test","partition":0,"replicas":[21,22,31,11]},
{"topic":"test","partition":1,"replicas":[21,22,31,11]},
{"topic":"test","partition":2,"replicas":[21,22,31,11]},
{"topic":"test","partition":3,"replicas":[21,22,31,11]},
{"topic":"test","partition":4,"replicas":[21,22,31,11]}
]
}
4/ 執行修改命令,把副本數修改為4
[root@zk-kafka-1 ~]# /data1/kafka/bin/kafka-reassign-partitions.sh --zookeeper ${ZOOKEEPER_ADDR} --reassignment-json-file increase-replication-factor.json --execute
Current partition replica assignment
{"version":1,"partitions":[{"topic":"test","partition":0,"replicas":[21,22,31],"log_dirs":["any","any","any"]},{"topic":"test","partition":1,"replicas":[21,22,31],"log_dirs":["any","any","any"]},{"topic":"test","partition":4,"replicas":[21,22,31],"log_dirs":["any","any","any"]},{"topic":"test","partition":3,"replicas":[21,22,31],"log_dirs":["any","any","any"]},{"topic":"test","partition":2,"replicas":[21,22,31],"log_dirs":["any","any","any"]}]}
Save this to use as the --reassignment-json-file option during rollback
Successfully started reassignment of partitions.
5/ 看到successfully,查看現在test的副本數已經修改為4
Topic:test PartitionCount:5 ReplicationFactor:4 Configs:
Topic: test Partition: 0 Leader: 21 Replicas: 21,22,31,11 Isr: 21,22,31,11
Topic: test Partition: 1 Leader: 22 Replicas: 21,22,31,11 Isr: 22,31,21,11
Topic: test Partition: 2 Leader: 31 Replicas: 21,22,31,11 Isr: 31,21,22,11
Topic: test Partition: 3 Leader: 21 Replicas: 21,22,31,11 Isr: 21,31,22,11
Topic: test Partition: 4 Leader: 21 Replicas: 21,22,31,11 Isr: 31,22,21,11