主要是常用的hbase shell命令,包括表的創建與刪除,表數據的增刪查【hbase沒有修改】;以及hbase的導出與導入。 參考教程:HBase教程 參考博客:hbase shell基礎和常用命令詳解 參考博客:hbase shell常用命令和filter 參考博客:hbase導入導出數據 1. ...
主要是常用的hbase shell命令,包括表的創建與刪除,表數據的增刪查【hbase沒有修改】;以及hbase的導出與導入。
參考教程:HBase教程
參考博客:hbase shell基礎和常用命令詳解
參考博客:hbase導入導出數據
1. hbase命令
1.1. 命令的登錄與退出
1 1.1. 命令的登錄與退出 2 [yun@mini03 ~]$ hbase shell # 登錄hbase 3 SLF4J: Class path contains multiple SLF4J bindings. 4 ……………… 5 Version 2.0.0, r7483b111e4da77adbfc8062b3b22cbe7c2cb91c1, Sun Apr 22 20:26:55 PDT 2018 6 Took 0.0126 seconds 7 hbase(main):001:0> quit # 退出hbase 8 [yun@mini03 ~]$
1.2. 常用命令
名稱 |
命令表達式 |
創建表 |
create '表名', '列族名1','列族名2','列族名N' |
查看所有表 |
list |
描述表 |
describe ‘表名’ |
判斷表存在 |
exists '表名' |
判斷是否禁用啟用表 |
is_enabled '表名' is_disabled ‘表名’ |
添加記錄 |
put ‘表名’, ‘rowKey’, ‘列族 : 列‘ , '值' |
查看記錄rowkey下的所有數據 |
get '表名' , 'rowKey' |
查看表中的記錄總數 |
count '表名' |
獲取某個列族 |
get '表名','rowkey','列族' |
獲取某個列族的某個列 |
get '表名','rowkey','列族:列’ |
刪除記錄 |
delete ‘表名’ ,‘行名’ , ‘列族:列' |
刪除整行 |
deleteall '表名','rowkey' |
刪除一張表 |
先要屏蔽該表,才能對該表進行刪除 第一步 disable ‘表名’ ,第二步 drop '表名' |
清空表 |
truncate '表名' |
查看所有記錄 |
scan "表名" |
查看某個表某個列中所有數據 |
scan "表名" , {COLUMNS=>'列族名:列名'} |
更新記錄 |
就是重寫一遍,進行覆蓋,hbase沒有修改,都是追加 |
說明:
1、列族裡邊可以自由添加子列很方便。如果列族下沒有子列,加不加冒號都是可以的。
hbase> put 't1', 'r1', 'c1', 'value', ts1
2、t1指表名,r1指行鍵名,c1指列名,value指單元格值。ts1指時間戳,一般都省略掉了。
1.3. 示例1-建表,查看表結構信息,刪表
hbase(main):006:0* create 'user','info1','info2','info3' # 創建表 Created table user Took 2.9927 seconds => Hbase::Table - user hbase(main):007:0> list # 查看所有表 TABLE user 1 row(s) Took 0.0468 seconds => ["user"] hbase(main):008:0> describe 'user' # 描述表 Table user is ENABLED ##### 當前表可用 user COLUMN FAMILIES DESCRIPTION {NAME => 'info1', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} {NAME => 'info2', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} {NAME => 'info3', VERSIONS => '1', EVICT_BLOCKS_ON_CLOSE => 'false', NEW_VERSION_BEHAVIOR => 'false', KEEP_DELETED_CELLS => 'FALSE', CACHE_DATA_ON_WRITE => 'false', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', CACHE_INDEX_ON_WRITE => 'false', IN_MEMORY => 'false', CACHE_BLOOMS_ON_WRITE => 'false', PREFETCH_BLOCKS_ON_OPEN => 'false', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'} 3 row(s) Took 0.1933 seconds hbase(main):011:0* exists 'user' # 判斷表是否存在 Table user does exist Took 0.0398 seconds => true hbase(main):026:0> is_enabled 'user' # 判斷表是否可用 true Took 0.0149 seconds => true hbase(main):027:0> is_disabled 'user' # 判斷表是否不可用 false Took 0.0148 seconds => 1 hbase(main):015:0> drop 'user' # 刪除表【必須先將表置為不可用】 ERROR: Table user is enabled. Disable it first. Drop the named table. Table must first be disabled: hbase> drop 't1' hbase> drop 'ns1:t1' Took 0.0342 seconds hbase(main):018:0* disable 'user' # 將表置為不可用 Took 0.5544 seconds hbase(main):019:0> drop 'user' # 之後刪除表 Took 0.5143 seconds hbase(main):020:0> list TABLE 0 row(s) Took 0.0687 seconds => []
1.4. 示例2-添加表數據和查詢表數據
hbase(main):024:0> create 'user','info1','info2','info3' # 創建表 Created table user Took 1.2717 seconds => Hbase::Table - user hbase(main):025:0> list # 查看所有表信息 TABLE user 1 row(s) Took 0.0466 seconds => ["user"] hbase(main):042:0* put 'user','1234','info1:name','zhang' # 添加數據 Took 0.0634 seconds hbase(main):043:0> scan 'user' # 查看所有記錄 ROW COLUMN+CELL 1234 column=info1:name, timestamp=1533740907054, value=zhang 1 row(s) Took 0.0776 seconds hbase(main):044:0> put 'user','1234','info1:name','zhangsan' # 再次添加【實際類似修改】 Took 0.0055 seconds hbase(main):045:0> scan 'user' ROW COLUMN+CELL 1234 column=info1:name, timestamp=1533740935956, value=zhangsan 1 row(s) Took 0.0160 seconds hbase(main):046:0> put 'user','1234','info2:name','zhang' # 在另一個列族添加數據 Took 0.0160 seconds hbase(main):047:0> put 'user','1234','info2:age','23' # 在另一個列族的另外一個列添加數據 Took 0.0133 seconds hbase(main):048:0> scan 'user' ROW COLUMN+CELL 1234 column=info1:name, timestamp=1533740935956, value=zhangsan 1234 column=info2:age, timestamp=1533741066465, value=23 1234 column=info2:name, timestamp=1533741052169, value=zhang 1 row(s) Took 0.0171 seconds hbase(main):050:0* put 'user','12345','info1:name','lisi' Took 0.0125 seconds hbase(main):051:0> put 'user','12345','info2:age','25' Took 0.0143 seconds hbase(main):052:0> scan 'user' ROW COLUMN+CELL 1234 column=info1:name, timestamp=1533740935956, value=zhangsan 1234 column=info2:age, timestamp=1533741066465, value=23 1234 column=info2:name, timestamp=1533741052169, value=zhang 12345 column=info1:name, timestamp=1533741585906, value=lisi 12345 column=info2:age, timestamp=1533741595725, value=25 2 row(s) Took 0.0179 seconds hbase(main):058:0* count 'user' # 查看表中的記錄總數【根據 row keys 判斷】 2 row(s) Took 0.1065 seconds => 2 hbase(main):053:0> get 'user','1234' # 查看記錄rowkey下的所有數據 COLUMN CELL info1:name timestamp=1533740935956, value=zhangsan info2:age timestamp=1533741066465, value=23 info2:name timestamp=1533741052169, value=zhang 1 row(s) Took 0.0371 seconds hbase(main):067:0* get 'user','1234','info2' # 獲取某個列族 COLUMN CELL info2:age timestamp=1533741066465, value=23 info2:name timestamp=1533741052169, value=zhang 1 row(s) Took 0.0305 seconds hbase(main):068:0> get 'user','1234','info2:name' # 獲取某個列族的某個列 COLUMN CELL info2:name timestamp=1533741052169, value=zhang 1 row(s) Took 0.0182 seconds
1.5. 示例3-刪除行數據
hbase(main):072:0> get 'user','1234' COLUMN CELL info1:name timestamp=1533740935956, value=zhangsan info2:address timestamp=1533742368985, value=China info2:age timestamp=1533741066465, value=23 info2:name timestamp=1533741052169, value=zhang 1 row(s) Took 0.0146 seconds hbase(main):073:0> delete 'user','1234','info2:age' # 刪除指定行,指定列記錄 #### 註意其中: delete 'user','1234','info2' 是無效的 ★★★ Took 0.0288 seconds hbase(main):074:0> get 'user','1234' COLUMN CELL info1:name timestamp=1533740935956, value=zhangsan info2:address timestamp=1533742368985, value=China info2:name timestamp=1533741052169, value=zhang 1 row(s) Took 0.0140 seconds hbase(main):100:0* deleteall 'user','1234' # 刪除整行 Took 0.0119 seconds hbase(main):101:0> get 'user','1234' COLUMN CELL 0 row(s) Took 0.0145 seconds
1.6. 示例4-條件scan和truncate
hbase(main):122:0* scan 'user' ROW COLUMN+CELL 1234 column=info2:address, timestamp=1533743416815, value=CN 1234 column=info2:age, timestamp=1533743407616, value=20 1234 column=info2:name, timestamp=1533743396872, value=wangwu 12345 column=info1:name, timestamp=1533741585906, value=lisi 12345 column=info2:age, timestamp=1533741595725, value=25 2 row(s) Took 0.0241 seconds hbase(main):123:0> scan 'user',{COLUMNS => 'info2'} ROW COLUMN+CELL 1234 column=info2:address, timestamp=1533743416815, value=CN 1234 column=info2:age, timestamp=1533743407616, value=20 1234 column=info2:name, timestamp=1533743396872, value=wangwu 12345 column=info2:age, timestamp=1533741595725, value=25 2 row(s) Took 0.0161 seconds hbase(main):124:0> scan 'user',{COLUMNS => 'info2:age'} ROW COLUMN+CELL 1234 column=info2:age, timestamp=1533743407616, value=20 12345 column=info2:age, timestamp=1533741595725, value=25 2 row(s) Took 0.0158 seconds hbase(main):128:0* truncate truncate truncate_preserve hbase(main):128:0* truncate 'user' # 截斷表 Truncating 'user' table (it may take a while): Disabling table... Truncating table... Took 2.6156 seconds hbase(main):129:0> scan 'user' ROW COLUMN+CELL 0 row(s) Took 0.1305 seconds
2. hbase導入導出數據
在實際應用HBase過程中,經常需要將生產環境中的數據備份,或者需要在開發環境中利用生產環境的數據(更加符合實際情況),因此HBase存儲的數據的導入導出必不可少。
2.1. 表信息準備
hbase(main):002:0> create 'zhang','userinfo','baseinfo','eduinfo','workinfo' Created table zhang Took 1.4311 seconds => Hbase::Table - zhang hbase(main):005:0* put 'zhang','12345','userinfo:username','zhangsan' Took 0.3751 seconds hbase(main):006:0> put 'zhang','12345','userinfo:password','111111' Took 0.0220 seconds hbase(main):007:0> put 'zhang','12345','baseinfo:name','zhangsan' Took 0.0136 seconds hbase(main):008:0> put 'zhang','12345','baseinfo:age','22' Took 0.0136 seconds hbase(main):009:0> put 'zhang','12345','baseinfo:name','zhangnew' Took 0.0106 seconds hbase(main):010:0> put 'zhang','12345','baseinfo:age','25' Took 0.0138 seconds hbase(main):013:0> put 'zhang','12345','eduinfo:pri_school','star school' Took 0.0106 seconds hbase(main):014:0> scan 'zhang' ROW COLUMN+CELL 12345 column=baseinfo:age, timestamp=1533884261796, value=25 12345 column=baseinfo:name, timestamp=1533884258020, value=zhangnew 12345 column=eduinfo:pri_school, timestamp=1533884297216, value=star school 12345 column=userinfo:password, timestamp=1533884246132, value=111111 12345 column=userinfo:username, timestamp=1533884241334, value=zhangsan 1 row(s) Took 0.0179 seconds
2.2. hbase表導出到HDFS
[yun@mini02 ~]$ hbase org.apache.hadoop.hbase.mapreduce.Export zhang /zhang/hbase/zhang_tab …………………… 2018-08-10 15:01:27,354 INFO [main] mapreduce.Job: The url to track the job: http://mini02:8088/proxy/application_1533865678790_0001/ 2018-08-10 15:01:27,355 INFO [main] mapreduce.Job: Running job: job_1533865678790_0001 2018-08-10 15:01:39,564 INFO [main] mapreduce.Job: Job job_1533865678790_0001 running in uber mode : false 2018-08-10 15:01:39,566 INFO [main] mapreduce.Job: map 0% reduce 0% 2018-08-10 15:01:52,384 INFO [main] mapreduce.Job: map 100% reduce 0% 2018-08-10 15:01:53,416 INFO [main] mapreduce.Job: Job job_1533865678790_0001 completed successfully 2018-08-10 15:01:53,554 INFO [main] mapreduce.Job: ps in occupied slots (ms)=9661 Total time spent by all reduces in occupied slots (ms)=0 Total time spent by all map tasks (ms)=9661 Total vcore-milliseconds taken by all map tasks=9661 Total megabyte-milliseconds taken by all map tasks=9892864 Map-Reduce Framework Map input records=1 Map output records=1 Input split bytes=124 Spilled Records=0 Failed Shuffles=0 Merged Map outputs=0 GC time elapsed (ms)=493 CPU time spent (ms)=7670 Physical memory (bytes) snapshot=259244032 Virtual memory (bytes) snapshot=2178433024 Total committed heap usage (bytes)=89653248 HBase Counters BYTES_IN_REMOTE_RESULTS=252 BYTES_IN_RESULTS=252 MILLIS_BETWEEN_NEXTS=1258 NOT_SERVING_REGION_EXCEPTION=0 NUM_SCANNER_RESTARTS=0 NUM_SCAN_RESULTS_STALE=0 REGIONS_SCANNED=1 REMOTE_RPC_CALLS=1 REMOTE_RPC_RETRIES=0 ROWS_FILTERED=0 ROWS_SCANNED=1 RPC_CALLS=1 RPC_RETRIES=0 File Input Format Counters Bytes Read=0 File Output Format Counters Bytes Written=364
HDFS查看
1 [yun@mini03 ~]$ hadoop fs -ls /zhang/hbase/zhang_tab 2 Found 2 items 3 -rw-r--r-- 3 yun supergroup 0 2018-08-10 15:01 /zhang/hbase/zhang_tab/_SUCCESS 4 -rw-r--r-- 3 yun supergroup 364 2018-08-10 15:01 /zhang/hbase/zhang_tab/part-m-00000
2.3. hbase表導出到本地集群系統
具體導出到哪台機器,需要自己去查找
[yun@mini02 hbase_data]$ hbase org.apache.hadoop.hbase.mapreduce.Export zhang file:///app/software/hbase_data/zhang_tab …………………… 2018-08-10 16:14:17,619 INFO [main] mapreduce.JobSubmitter: Submitting tokens for job: job_1533865678790_0003 2018-08-10 16:14:18,249 INFO [main] impl.YarnClientImpl: Submitted application application_1533865678790_0003 2018-08-10 16:14:18,290 INFO [main] mapreduce.Job: The url to track the job: http://mini02:8088/proxy/application_1533865678790_0003/ 2018-08-10 16:14:18,290 INFO [main] mapreduce.Job: Running job: job_1533865678790_0003 2018-08-10 16:14:29,932 INFO [main] mapreduce.Job: Job job_1533865678790_0003 running in uber mode : false 2018-08-10 16:14:29,935 INFO [main] mapreduce.Job: map 0% reduce 0% 2018-08-10 16:14:39,958 INFO [main] mapreduce.Job: map 100% reduce 0% 2018-08-10 16:14:40,984 INFO [main] mapreduce.Job: Job job_1533865678790_0003 completed successfully 2018-08-10 16:14:41,131 INFO [main] mapreduce.Job: aps in occupied slots (ms)=7246 Total time spent by all reduces in occupied slots (ms)=0 Total time spent by all map tasks (ms)=7246 Total vcore-milliseconds taken by all map tasks=7246 Total megabyte-milliseconds taken by all map tasks=7419904 Map-Reduce Framework Map input records=1 Map output records=1 Input split bytes=124 Spilled Records=0 Failed Shuffles=0 Merged Map outputs=0 GC time elapsed (ms)=242 CPU time spent (ms)=5580 Physical memory (bytes) snapshot=263946240 Virtual memory (bytes) snapshot=2176024576 Total committed heap usage (bytes)=94896128 HBase Counters BYTES_IN_REMOTE_RESULTS=252 BYTES_IN_RESULTS=252 MILLIS_BETWEEN_NEXTS=959 NOT_SERVING_REGION_EXCEPTION=0 NUM_SCANNER_RESTARTS=0 NUM_SCAN_RESULTS_STALE=0 REGIONS_SCANNED=1 REMOTE_RPC_CALLS=1 REMOTE_RPC_RETRIES=0 ROWS_FILTERED=0 ROWS_SCANNED=1 RPC_CALLS=1 RPC_RETRIES=0 File Input Format Counters Bytes Read=0 File Output Format Counters Bytes Written=376
導出到本地的文件
1 [yun@mini04 task_1533865678790_0003_m_000000]$ pwd 2 /app/software/hbase_data/zhang_tb/_temporary/1/task_1533865678790_0003_m_000000 3 [yun@mini04 task_1533865678790_0003_m_000000]$ ll 4 total 4 5 -rw-r--r-- 1 yun yun 364 Aug 10 16:14 part-m-00000
2.4. 導入數據到hbase
把剛纔導出的數據,導入到hbase。
方式1【該方式導入失敗】
1 # 把 hbase 中表 zhang 的數據給刪掉,然後導入 2 hbase(main):058:0* deleteall 'zhang','12345' 3 Took 0.0541 seconds \ 4 hbase(main):059:0> scan 'zhang' 5 ROW COLUMN+CELL 6 0 row(s) 7 Took 0.0247 seconds
結果導入時死活沒數據。。。
方式2【導入正常】
1 # 對錶 zhang 進行truncate,然後導入數據 或者 刪除表 zhang,再重建該表 2 hbase(main):018:0* truncate 'zhang' 3 Truncating 'zhang' table (it may take a while): 4 Disabling table... 5 Truncating table... 6 Took 1.8954 seconds 7 hbase(main):020:0> scan 'zhang' 8 ROW COLUMN+CELL 9 0 row(s) 10 Took 1.4118 seconds
導入數據命令
[yun@mini04 task_1533865678790_0003_m_000000]$ hbase org.apache.hadoop.hbase.mapreduce.Import zhang file:///app/software/hbase_data/zhang_tb/_temporary/1/task_1533865678790_0003_m_000000/part-m-00000 ……………………
導入後查看
hbase(main):023:0* scan 'zhang' ROW COLUMN+CELL 12345 column=baseinfo:age, timestamp=1533884261796, value=25 12345 column=baseinfo:name, timestamp=1533884258020, value=zhangnew 12345 column=eduinfo:pri_school, timestamp=1533884297216, value=star school 12345 column=userinfo:password, timestamp=1533884246132, value=111111 12345 column=userinfo:username, timestamp=1533884241334, value=zhangsan 1 row(s) Took 0.0439 seconds
方式3【導入正常】
1 # 新建表導入 要求:表結構一樣 2 hbase(main):070:0* create 'zhang_test','userinfo','baseinfo','eduinfo','workinfo' 3 Created table zhang_test 4 Took 0.7815 seconds \ 5 => Hbase::Table - zhang_test 6 hbase(main):071:0> list # 查看所有表 7 TABLE 8 scores 9 user 10 zhang 11 zhang_test 12 4 row(s) 13 Took 0.0280 seconds 14 => ["scores", "user", "zhang", "zhang_test"]
導入數據命令
[yun@mini04 task_1533865678790_0003_m_000000]$ hbase org.apache.hadoop.hbase.mapreduce.Import zhang_test file:///app/software/hbase_data/zhang_tb/_temporary/1/task_1533865678790_0003_m_000000/part-m-00000 ………………
導入後查看
1 hbase(main):074:0* scan 'zhang_test' 2 ROW COLUMN+CELL 3 12345 column=baseinfo:age, timestamp=1533884261796, value=25 4 12345 column=baseinfo:name, timestamp=1533884258020, value=zhangnew 5 12345 column=eduinfo:pri_school, timestamp=1533884297216, value=star school 6 12345 column=userinfo:password, timestamp=1533884246132, value=111111 7 12345 column=userinfo:username, timestamp=1533884241334, value=zhangsan 8 1 row(s) 9 Took 0.0544 seconds