這次向大家分享發表在SOSP 2013上的另一篇經典圖計算框架論文X-Stream,構建了單機上基於外存的Scatter-Gather圖處理框架。 ...
1 ambari + bigtop 構建大數據基礎平臺
1.1 參考:
- https://blog.csdn.net/qq_42896460/article/details/132717278
- Ambari+Metrics+Bigtop 全家桶編譯部署攻略——Ambari系列(ambari 2.8.0+centos 7.9 +bigtop 3.3.0) https://blog.csdn.net/qq_18730505/article/details/136968967
- 對ambari做二次開發:https://www.zhihu.com/question/62343277/answer/2211340398
- Ambari自定義服務集成實戰(全):
- ambari二次開發知識庫:https://www.yuque.com/create17/ambari
- ambari:
- bigtop:https://gitee.com/piaolingzxh/bigtop.git
- QQ群:ambari+bigtop精華文章
- 系列文章1:
- BigTop3.2.0 大數據組件編譯--基礎環境準備:https://blog.51cto.com/u_15670647/6179237
- ambari2.8.0+bigtop3.2.0發行版大數據平臺編譯指南:https://blog.51cto.com/u_15670647/6179238
- Ambari+Bigtop大數據平臺安裝部署指南(Centos7)一:https://blog.51cto.com/u_15670647/6181593
- 系列文章2
- CentOS內網yum源構建&使用最佳實踐:https://blog.csdn.net/m0_48319997/article/details/127972654
- Ambari3.0.0+BigTop3.2.0 SNAPSHOT初體驗:https://blog.csdn.net/m0_48319997/article/details/127998559
- ambari2.8.0+bigtop3.2.0部署環境準備教程::https://blog.csdn.net/m0_48319997/article/details/130069050
- ambari2.8.0+bigtop3.2.0發行版大數據平臺編譯指南:https://blog.csdn.net/m0_48319997/article/details/130046296
- BigTop3.2.0 大數據組件編譯--組件編譯:https://blog.csdn.net/m0_48319997/article/details/128101667
- Ambari+Bigtop大數據平臺安裝部署指南(Centos7)一:https://blog.csdn.net/m0_48319997/article/details/130069050
- Ambari+Bigtop大數據平臺安裝部署指南(Centos7)二:https://blog.csdn.net/m0_48319997/article/details/130233526
- 資源:
- ambari2.8.0+bigtop3.2.0編譯好的組件:https://www.aliyundrive.com/s/VrNpPyDqthd
- Ambari2.8.0+Metrics3.0.0+Bigtop3.2.0編譯包,所有組件均驗證可正常部署,自行下載。鏈接: https://pan.baidu.com/s/1_SmkYn7aTKNt1w15irLDSQ 提取碼: ep12 永久有效
- ambari2.8.0-麒麟v10-rpm包,只有ambari無bigtop。:鏈接:https://pan.baidu.com/s/1V2zGngqUwcoWroB_NL71zg 提取碼:nqgq
- 系列文章1:
1.2 參考
amabri
bigtop
- Apache Bigtop再論與賣書求生存:https://blog.51cto.com/slaytanic/1887685
打包部署
- Ambari+Bigtop大數據平臺安裝部署指南(Centos7)一:https://blog.csdn.net/m0_48319997/article/details/130069050
- Ambari+Bigtop大數據平臺安裝部署指南(Centos7)二:https://blog.csdn.net/m0_48319997/article/details/130233526
2 ambari+bigtop編譯、打包、部署
操作步驟
- 時間:2024-07-18
- 環境準備:
- 系統centos7
- yum源
- 系統基礎組件
- 防火牆、selinux設置、句柄數,時區
- java、scala、maven、gradle、ant、nodejs、環境配置
- 編譯步驟:
- 編譯ambari:最新分支branch-2.8
- 編譯ambari-metrics:最新分支branch-3.0
- 編譯bigtop:最新分支barnch-3.3,此處採用branch-3.2
- 打包:將以下文件copy到目標目錄
- ambari-server(ambari)
- ambari-agent(ambari)
- ambari-metrics-collector(ambari-metrics)
- ambari-metrics-grafana(ambari-metrics)
- ambari-metrics-hadoop-sink(ambari-metrics)
- ambari-metrics-monitor(ambari-metrics)
- bigtop打包出來的大數據組件(bigtop)
- ambari+bigtop部署
- 伺服器準備:4c-16G-60G三台
- 環境檢查:操作系統、預設語言、時區、機器名、功能變數名稱解析、網路、防火牆關閉、selinux關閉、limits句柄數、禁用交換分區、unmask設置、磁碟掛載
- 基礎設置:免密、基礎軟體、ntp服務、離線鏡像源+httpd服務,
- 資料庫安裝、ambari元資料庫配置
- 創建並配置bigtop的yum源
- 安裝啟動ambari-server,初始化(生成表)
2.0 基礎環境準備
2.1 ambari編譯
2.2 ambari-metrics編譯
2.3 bigtop編譯
2.4 製作發版鏡像
#創建bdp3.2文件夾-所有rpm包將都拷貝到這個文件夾
mkdir -p bdp3.2
#將ambari包拷貝
mkdir -p bdp3.2/ambari # 存放ambari項目打包出來的rpm包
cp ambari/ambari-server/target/rpm/ambari-server/RPMS/x86_64/ambari-server-2.8.0.0-0.x86_64.rpm bdp3.2/ambari/
cp ambari/ambari-agent/target/rpm/ambari-agent/RPMS/x86_64/ambari-agent-2.8.0.0-0.x86_64.rpm bdp3.2/ambari/
#將ambari-metrics包拷貝,# 存放ambari-metrics項目打包出來的rpm包
mkdir -p bdp3.2/ambari-metrics
cp ambari-metrics/ambari-metrics-assembly/target/rpm/ambari-metrics-collector/RPMS/x86_64/ambari-metrics-collector-3.0.1-1.x86_64.rpm bdp3.2/ambari-metrics/
cp ambari-metrics/ambari-metrics-assembly/target/rpm/ambari-metrics-grafana/RPMS/x86_64/ambari-metrics-grafana-3.0.1-1.x86_64.rpm bdp3.2/ambari-metrics/
cp ambari-metrics/ambari-metrics-assembly/target/rpm/ambari-metrics-hadoop-sink/RPMS/x86_64/ambari-metrics-hadoop-sink-3.0.1-1.x86_64.rpm bdp3.2/ambari-metrics/
cp ambari-metrics/ambari-metrics-assembly/target/rpm/ambari-metrics-monitor/RPMS/x86_64/ambari-metrics-monitor-3.0.1-1.x86_64.rpm bdp3.2/ambari-metrics/
#將bigtop包拷貝
mkdir -p bdp3.2/bigtop-3.2.1 # 存放bigtop項目打包出來的rpm包
cp -r bigtop/output/* bdp3.2/bigtop-3.2.1
# 製作鏡像源,在bdp3.2 目錄下生成repodata(也可以將文件拷貝到安裝位置後,再製作鏡像源)
createrepo bdp3.2/
# bdp3.2.tar.gz,將是最終發版的包
tar zcvf bdp3.2.tar.gz bdp3.2
# yum源模板
vim ambari.repo #必須是此文件名,安裝ambari-agent時,界面有檢查項
# BIGTOP-3.2.1 必須是固定字元串,大寫。ambari安裝大數據組件時,有檢查項,否則找不到數據源,3.2.1 是bigtop.bom 中的base_version
# http://172.16.76.107/chdp3.2 替換成自己的地址,確保訪問地址後,能看到上邊ambari,ambari-metrics,bigtop-3.2.1 這三個目錄
[BIGTOP-3.2.1]
name=BIGTOP-3.2.1
baseurl=http://172.16.76.107/bdp3.2
failovermethod=priority
enabled=1
gpgcheck=0
2.5 使用鏡像|鏡像測試
搭建yum源伺服器
#將bdp3.2.tar.gz 複製到目標伺服器
tar zxfv bdp3.2.tar.gz
ln -s /to/your/path/bdp3.2 /var/wwww/html
systemctl restart httpd # 重啟httpd服務
systemctl enable httpd.service # 開機自啟
訪問:http://${yum_hosts}/bdp3.2, 查看根目錄下是否有ambari,ambari-metrics,bigtop-3.2.1,repodata四個文件夾
客戶機訪問bdp3.2的yum源
1)客戶機添加yum源
vim /etc/yum.repos.d/ambari.repo # 內容參考上邊的yum源模板
[BIGTOP-3.2.1]
name=BIGTOP-3.2.1
baseurl=http://172.16.76.107/bdp3.2 ## 請替換此處IP
failovermethod=priority
enabled=1
gpgcheck=0
2)客戶機刷新yum緩存(務必執行此操作,否則安裝時不會報錯,也安裝不上)
yum clean all
yum makecache
3)查看是否能夠訪問bdp3.2上的安裝包,
yum search ambari-server
4)查看bigtop源下的包有哪些(註意,若是只有兩三個,可以去掉wc -l,看一下是具體那幾個)
/usr/bin/yum list available --showduplicates --disablerepo=* --enablerepo=BIGTOP-3.2.0 |wc -l
/usr/bin/yum list available --showduplicates --disablerepo=* --enablerepo=BIGTOP-3.2.1 |wc -l
bdp3.2 壓縮包文件結構預覽
bdp3.2/
├── ambari
│ └── x86_64
│ ├── ambari-agent-2.8.0.0-0.x86_64.rpm
│ └── ambari-server-2.8.0.0-0.x86_64.rpm
├── ambari-metrics
│ └── x86_64
│ ├── ambari-metrics-collector-3.0.1-1.x86_64.rpm
│ ├── ambari-metrics-grafana-3.0.1-1.x86_64.rpm
│ ├── ambari-metrics-hadoop-sink-3.0.1-1.x86_64.rpm
│ └── ambari-metrics-monitor-3.0.1-1.x86_64.rpm
├── bigtop3.2.1
│ ├── bigtop-ambari-mpack
│ │ ├── bigtop-ambari-mpack-2.7.5.0-1.el7.src.rpm
│ │ └── noarch
│ │ └── bigtop-ambari-mpack-2.7.5.0-1.el7.noarch.rpm
│ ├── bigtop-groovy
│ │ ├── bigtop-groovy-2.5.4-1.el7.src.rpm
│ │ └── noarch
│ │ └── bigtop-groovy-2.5.4-1.el7.noarch.rpm
│ ├── bigtop-jsvc
│ │ ├── bigtop-jsvc-1.2.4-1.el7.src.rpm
│ │ └── x86_64
│ │ ├── bigtop-jsvc-1.2.4-1.el7.x86_64.rpm
│ │ └── bigtop-jsvc-debuginfo-1.2.4-1.el7.x86_64.rpm
│ ├── bigtop-select
│ │ ├── bigtop-select-3.2.1-1.el7.src.rpm
│ │ └── noarch
│ │ └── bigtop-select-3.2.1-1.el7.noarch.rpm
│ ├── bigtop-utils
│ │ ├── bigtop-utils-3.2.1-1.el7.src.rpm
│ │ └── noarch
│ │ └── bigtop-utils-3.2.1-1.el7.noarch.rpm
│ ├── flink
│ │ ├── flink_3_2_1-1.15.3-1.el7.src.rpm
│ │ └── noarch
│ │ ├── flink_3_2_1-1.15.3-1.el7.noarch.rpm
│ │ ├── flink_3_2_1-jobmanager-1.15.3-1.el7.noarch.rpm
│ │ └── flink_3_2_1-taskmanager-1.15.3-1.el7.noarch.rpm
│ ├── hadoop
│ │ ├── hadoop_3_2_1-3.3.6-1.el7.src.rpm
│ │ └── x86_64
│ │ ├── hadoop_3_2_1-3.3.6-1.el7.x86_64.rpm
│ │ ├── hadoop_3_2_1-client-3.3.6-1.el7.x86_64.rpm
│ │ ├── hadoop_3_2_1-conf-pseudo-3.3.6-1.el7.x86_64.rpm
│ │ ├── hadoop_3_2_1-debuginfo-3.3.6-1.el7.x86_64.rpm
│ │ ├── hadoop_3_2_1-doc-3.3.6-1.el7.x86_64.rpm
│ │ ├── hadoop_3_2_1-hdfs-3.3.6-1.el7.x86_64.rpm
│ │ ├── hadoop_3_2_1-hdfs-datanode-3.3.6-1.el7.x86_64.rpm
│ │ ├── hadoop_3_2_1-hdfs-dfsrouter-3.3.6-1.el7.x86_64.rpm
│ │ ├── hadoop_3_2_1-hdfs-fuse-3.3.6-1.el7.x86_64.rpm
│ │ ├── hadoop_3_2_1-hdfs-journalnode-3.3.6-1.el7.x86_64.rpm
│ │ ├── hadoop_3_2_1-hdfs-namenode-3.3.6-1.el7.x86_64.rpm
│ │ ├── hadoop_3_2_1-hdfs-secondarynamenode-3.3.6-1.el7.x86_64.rpm
│ │ ├── hadoop_3_2_1-hdfs-zkfc-3.3.6-1.el7.x86_64.rpm
│ │ ├── hadoop_3_2_1-httpfs-3.3.6-1.el7.x86_64.rpm
│ │ ├── hadoop_3_2_1-kms-3.3.6-1.el7.x86_64.rpm
│ │ ├── hadoop_3_2_1-libhdfs-3.3.6-1.el7.x86_64.rpm
│ │ ├── hadoop_3_2_1-libhdfs-devel-3.3.6-1.el7.x86_64.rpm
│ │ ├── hadoop_3_2_1-libhdfspp-3.3.6-1.el7.x86_64.rpm
│ │ ├── hadoop_3_2_1-libhdfspp-devel-3.3.6-1.el7.x86_64.rpm
│ │ ├── hadoop_3_2_1-mapreduce-3.3.6-1.el7.x86_64.rpm
│ │ ├── hadoop_3_2_1-mapreduce-historyserver-3.3.6-1.el7.x86_64.rpm
│ │ ├── hadoop_3_2_1-yarn-3.3.6-1.el7.x86_64.rpm
│ │ ├── hadoop_3_2_1-yarn-nodemanager-3.3.6-1.el7.x86_64.rpm
│ │ ├── hadoop_3_2_1-yarn-proxyserver-3.3.6-1.el7.x86_64.rpm
│ │ ├── hadoop_3_2_1-yarn-resourcemanager-3.3.6-1.el7.x86_64.rpm
│ │ ├── hadoop_3_2_1-yarn-router-3.3.6-1.el7.x86_64.rpm
│ │ └── hadoop_3_2_1-yarn-timelineserver-3.3.6-1.el7.x86_64.rpm
│ ├── hbase
│ │ ├── hbase_3_2_1-2.4.17-1.el7.src.rpm
│ │ ├── noarch
│ │ │ └── hbase_3_2_1-doc-2.4.17-1.el7.noarch.rpm
│ │ └── x86_64
│ │ ├── hbase_3_2_1-2.4.17-1.el7.x86_64.rpm
│ │ ├── hbase_3_2_1-master-2.4.17-1.el7.x86_64.rpm
│ │ ├── hbase_3_2_1-regionserver-2.4.17-1.el7.x86_64.rpm
│ │ ├── hbase_3_2_1-rest-2.4.17-1.el7.x86_64.rpm
│ │ ├── hbase_3_2_1-thrift2-2.4.17-1.el7.x86_64.rpm
│ │ └── hbase_3_2_1-thrift-2.4.17-1.el7.x86_64.rpm
│ ├── hive
│ │ ├── hive_3_2_1-3.1.3-1.el7.src.rpm
│ │ └── noarch
│ │ ├── hive_3_2_1-3.1.3-1.el7.noarch.rpm
│ │ ├── hive_3_2_1-hbase-3.1.3-1.el7.noarch.rpm
│ │ ├── hive_3_2_1-hcatalog-3.1.3-1.el7.noarch.rpm
│ │ ├── hive_3_2_1-hcatalog-server-3.1.3-1.el7.noarch.rpm
│ │ ├── hive_3_2_1-jdbc-3.1.3-1.el7.noarch.rpm
│ │ ├── hive_3_2_1-metastore-3.1.3-1.el7.noarch.rpm
│ │ ├── hive_3_2_1-server2-3.1.3-1.el7.noarch.rpm
│ │ ├── hive_3_2_1-webhcat-3.1.3-1.el7.noarch.rpm
│ │ └── hive_3_2_1-webhcat-server-3.1.3-1.el7.noarch.rpm
│ ├── kafka
│ │ ├── kafka_3_2_1-2.8.2-1.el7.src.rpm
│ │ └── noarch
│ │ ├── kafka_3_2_1-2.8.2-1.el7.noarch.rpm
│ │ └── kafka_3_2_1-server-2.8.2-1.el7.noarch.rpm
│ ├── phoenix
│ │ ├── noarch
│ │ │ └── phoenix-5.1.3-1.el7.noarch.rpm
│ │ └── phoenix-5.1.3-1.el7.src.rpm
│ ├── solr
│ │ ├── noarch
│ │ │ ├── solr_3_2_1-8.11.2-1.el7.noarch.rpm
│ │ │ ├── solr_3_2_1-doc-8.11.2-1.el7.noarch.rpm
│ │ │ └── solr_3_2_1-server-8.11.2-1.el7.noarch.rpm
│ │ └── solr_3_2_1-8.11.2-1.el7.src.rpm
│ ├── spark
│ │ ├── noarch
│ │ │ ├── spark_3_2_1-3.2.3-1.el7.noarch.rpm
│ │ │ ├── spark_3_2_1-core-3.2.3-1.el7.noarch.rpm
│ │ │ ├── spark_3_2_1-datanucleus-3.2.3-1.el7.noarch.rpm
│ │ │ ├── spark_3_2_1-external-3.2.3-1.el7.noarch.rpm
│ │ │ ├── spark_3_2_1-history-server-3.2.3-1.el7.noarch.rpm
│ │ │ ├── spark_3_2_1-master-3.2.3-1.el7.noarch.rpm
│ │ │ ├── spark_3_2_1-python-3.2.3-1.el7.noarch.rpm
│ │ │ ├── spark_3_2_1-sparkr-3.2.3-1.el7.noarch.rpm
│ │ │ ├── spark_3_2_1-thriftserver-3.2.3-1.el7.noarch.rpm
│ │ │ ├── spark_3_2_1-worker-3.2.3-1.el7.noarch.rpm
│ │ │ └── spark_3_2_1-yarn-shuffle-3.2.3-1.el7.noarch.rpm
│ │ └── spark_3_2_1-3.2.3-1.el7.src.rpm
│ ├── tez
│ │ ├── noarch
│ │ │ └── tez_3_2_1-0.10.2-1.el7.noarch.rpm
│ │ └── tez_3_2_1-0.10.2-1.el7.src.rpm
│ └── zookeeper
│ ├── x86_64
│ │ ├── zookeeper_3_2_1-3.6.4-1.el7.x86_64.rpm
│ │ ├── zookeeper_3_2_1-debuginfo-3.6.4-1.el7.x86_64.rpm
│ │ ├── zookeeper_3_2_1-native-3.6.4-1.el7.x86_64.rpm
│ │ ├── zookeeper_3_2_1-rest-3.6.4-1.el7.x86_64.rpm
│ │ └── zookeeper_3_2_1-server-3.6.4-1.el7.x86_64.rpm
│ └── zookeeper_3_2_1-3.6.4-1.el7.src.rpm
└── repodata
├── 15cc6a5320a9f6e38cda7b8ec54e7e28ed527df25ee6f56c1bf20cfa421dfe73-other.sqlite.bz2
├── 1a5c0f03d6edae3170255a282d113afe9b22f654f246ea308f162e14234cf5f8-filelists.sqlite.bz2
├── 28cb1ff79328b8ebeaa55021078984e8da908081b0b389e4037d9b1e5854ae22-primary.xml.gz
├── 5066ad167311131c46e983c0d66c4901b92a4e862b11aff3492f02f90ec2eb36-other.xml.gz
├── 617177ba2b017e6e6b4a62b157fe3441ea97b74455d60792e78471f938b86cb8-filelists.xml.gz
├── 7ea377d3a0b59ffaaef9661cb7dd3f2144bd7bc27da1a11070b6d78d5116dddd-primary.sqlite.bz2
└── repomd.xml
37 directories, 103 files
2.6 ambari 安裝
-
- 找一臺機子搭建yum源,參考搭建yum源伺服器
-
- 找一臺機子配置yum源,安裝ambari-server,參考客戶機訪問bdp3.2的yum源。
- 安裝過程參考大神文檔:
- Ambari+Bigtop大數據平臺安裝部署指南(Centos7)一:https://blog.csdn.net/m0_48319997/article/details/130069050
- Ambari+Bigtop大數據平臺安裝部署指南(Centos7)二:https://blog.csdn.net/m0_48319997/article/details/130233526
安裝中可能出現的問題及解決方案
- 問題1:yum install ambari-server後出現以下日誌,軟體沒有安裝成功,也沒有任何反應
- 解決方案:yum clean all && yum makecache
Loaded plugins:aliases,changelog,fastestmirror, kabi, langpacks, tmprepo, verify, versionlock
Loading support for RedHat kernel ABI
- 問題2:yum源配置
/etc/yum.repos.d/ambari.repo (必須為此名稱,安裝過程有校驗)
[BIGTOP-3.2.1] # 必須為此名稱,和ambari-server中的配置一致,包含大小寫,下一行同理
name=BIGTOP-3.2.1
baseurl=http://192.168.76.107/chdp3.2
failovermethod=priority
enabled=1
gpgcheck=0