存儲類存儲類（storage class）是kubernetes資源類型，它是由管理員為管理PV之便而按需創建的類別存儲類好處是支持 PV 的動態創建，系統按PVC的需求標準動態創建適配的PV會為存儲管理帶來極大的靈活性。 PV的動態供給，其重點是在存儲類的定義，其分類大概是對存儲的性能進行分類 ...

存儲類

存儲類（storage class）是kubernetes資源類型，它是由管理員為管理PV之便而按需創建的類別
存儲類好處是支持 PV 的動態創建，系統按PVC的需求標準動態創建適配的PV會為存儲管理帶來極大的靈活性。
PV的動態供給，其重點是在存儲類的定義，其分類大概是對存儲的性能進行分類的，如圖1：金存儲類、銀存儲類、銅存儲類等。

圖1 基於綜合服務質量的存儲系統分類

一、Provisioner（存儲分配器）

Storage class 有一個分配器，用來決定使用哪個捲插件分配 PV。圖2 中可以看到，目前 Ceph 只有 RBD 介面支持內部分配器。

圖2 各存儲插件對動態供給方式的支持狀況

二、實驗環境搭建 - 動態供給

圖3 實驗環境架構圖

環境設置

1、搭建環境

操作系統版本： SLES15 SP1，無需安裝 swap
內核版本：4.12.14-197.18-default
Kubernetes版本：v1.15.2
VMware Workstation 14

2、虛擬化環境搭建和系統安裝參考：

安裝部署

1、所有 CaaS Platform 節點安裝

# zypper install ceph-common

複製 ceph.conf 到 worker 節點上

# scp admin:/etc/ceph/ceph.conf /etc/ceph/

2、創建池,並將應用名稱與存儲池關聯

# ceph osd pool create caasp4-dynamic 64
# ceph osd pool application enable caasp4-dynamic rbd

3、創建 CaaSP4 client user

# cd /etc/ceph
# ceph auth get-or-create client.caasp4-dynamic mon 'allow r' \
    osd 'allow class-read object_prefix rbd_children, allow rwx pool=caasp4-dynamic' \
    -o ceph.client.dynamic.keyring

4、獲取 client.admin 用戶 key 信息，並生成基於 base64 編碼 key

# ceph auth get client.admin
exported keyring for client.admin
[client.admin]
        key = AQA9w4VdAAAAABAAHZr5bVwkALYo6aLVryt7YA==
        caps mds = "allow *"
        caps mgr = "allow *"
        caps mon = "allow *"
        caps osd = "allow *

# echo AQA9w4VdAAAAABAAHZr5bVwkALYo6aLVryt7YA== | base64
QVFBOXc0VmRBQUFBQUJBQUhacjViVndrQUxZbzZhTFZyeXQ3WUE9PQo=

5、在Master節點上，為 client.admin，創建 secret 資源

# vim ceph-secret-admin.yaml
apiVersion: v1
kind: Secret
metadata:
  name: ceph-secret-admin
  namespace: kube-system
data:
  key: QVFBOXc0VmRBQUFBQUJBQUhacjViVndrQUxZbzZhTFZyeXQ3WUE9PQo=
type: kubernetes.io/rbd

# kubectl get secrets -n kube-system
NAME                                       TYPE                                 DATA   AGE
....
ceph-secret-admin                          kubernetes.io/rbd                     1      24s
....

6、獲取 client.caasp4-dynamic 用戶 key 信息，並生成基於 base64編碼的key

# ceph auth get client.caasp4-dynamic
exported keyring for client.caasp4-dynamic
[client.caasp4-dynamic]
        key = AQA29ppdTDmzHhAAET2mSbvovrS67kspPlqmLA==
        caps mon = "allow r"
        caps osd = "allow class-read object_prefix rbd_children, allow rwx pool=caasp4-dynamic"

# echo AQA29ppdTDmzHhAAET2mSbvovrS67kspPlqmLA== | base64
QVFBMjlwcGRURG16SGhBQUVUMm1TYnZvdnJTNjdrc3BQbHFtTEE9PQo=

7、在Master節點上，為 client.caasp4-dynamic 創建 secret

# vim ceph-secret-user.yaml
apiVersion: v1
kind: Secret
metadata:
  name: ceph-user-secret
  namespace: default
data:
  key: QVFBMjlwcGRURG16SGhBQUVUMm1TYnZvdnJTNjdrc3BQbHFtTEE9PQo=
type: kubernetes.io/rbd

# kubectl create -f ceph-secret-user.yaml
secret "ceph-user-secret" created

# kubectl get secrets      
NAME                  TYPE                                  DATA   AGE
ceph-secret-test      Opaque                                1      20h
ceph-user-secret      kubernetes.io/rbd                     1      4s
default-token-4hslq   kubernetes.io/service-account-token   3      24h

8、創建 storage class 存儲類

# vim ceph-storageclass.yaml
apiVersion: storage.k8s.io/v1beta1
kind: StorageClass
metadata:
  name: dynamic
  annotations:
     storageclass.beta.kubernetes.io/is-default-class: "true"
provisioner: kubernetes.io/rbd
parameters:
  monitors: 192.168.2.40:6789,192.168.2.41:6789,192.168.2.42:6789
  adminId: admin
  adminSecretName: ceph-secret-admin
  adminSecretNamespace: kube-system
  pool: caasp4-dynamic
  userId: caasp4-dynamic
  userSecretName: ceph-user-secret

# kubectl create -f ceph-storageclass.yaml
storageclass "dynamic" created

# kubectl get storageclasses
NAME                PROVISIONER         AGE
dynamic (default)   kubernetes.io/rbd   10m

StorageClass SPEC

（1）provisioner(供給方)：即提供存儲資源的存儲系統，供給方名字都以“kubernetes.io”為首碼
（2）parameters（參數）：使用參數描述要關聯到的存儲捲，不同的provisioner有不同的參數

kubernetes 官方參數說明

9、創建 PVC

# vim ceph-pvc.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: ceph-claim-dynamic
spec:
  accessModes:  
    - ReadWriteOnce
  resources:
    requests:
      storage: 3Gi

# kubectl create -f ceph-pvc.yaml
persistentvolumeclaim "ceph-claim-dynamic" created

# kubectl get pvc -o wide
NAME                 STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE   VOLUMEMODE
ceph-claim-dynamic   Bound    pvc-70b3b3ca-5267-4417-a898-c1f5524de008   3Gi        RWO            dynamic        10m   Filesystem

10、創建 Pod

vim ceph-pod-dynamic.yaml
apiVersion: v1
kind: Pod
metadata:
  name: ceph-pod1-dynamic
spec:
  containers:
  - name: ceph-busybox
    image: busybox
    command: ["sleep", "60000"]
    volumeMounts:
    - name: ceph-vol1-dynamic
      mountPath: /usr/share/busybox
      readOnly: false
  volumes:
  - name: ceph-vol1-dynamic
    persistentVolumeClaim:
      claimName: ceph-claim-dynamic

# kubectl create -f ceph-pod-dynamic.yaml
pod "ceph-pod1-dynamic" created

# kubectl get pods
NAME                READY   STATUS    RESTARTS   AGE
ceph-pod1-dynamic   1/1     Running   0          24m

# kubectl get pods -o wide
NAME                READY   STATUS    RESTARTS   AGE   IP             NODE       NOMINATED NODE   READINESS GATES
ceph-pod1-dynamic   1/1     Running   0          48m   10.244.2.194   worker02   <none>           <none>

# kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                        STORAGECLASS   REASON   AGE
pvc-70b3b3ca-5267-4417-a898-c1f5524de008   3Gi        RWO            Delete           Bound    default/ceph-claim-dynamic   dynamic                 169m

11、storage6 分散式存儲上，查看是否創建了鏡像

admin:/etc/ceph # rbd ls -p caasp4-dynamic
kubernetes-dynamic-pvc-e6d98bbf-50e1-4488-a9fc-867d1db810c8

admin:/etc/ceph # rbd info kubernetes-dynamic-pvc-e6d98bbf-50e1-4488-a9fc-867d1db810c8 -p caasp4-dynamic
rbd image 'kubernetes-dynamic-pvc-e6d98bbf-50e1-4488-a9fc-867d1db810c8':
        size 3 GiB in 768 objects
        order 22 (4 MiB objects)
        snapshot_count: 0
        id: cd53f75a03b15
        block_name_prefix: rbd_data.cd53f75a03b15
        format: 2
        features:
        op_features:
        flags:
        create_timestamp: Mon Oct  7 17:14:23 2019
        access_timestamp: Mon Oct  7 17:14:23 2019
        modify_timestamp: Mon Oct  7 17:14:23 2019

12、worker02節點上，查看RBD映射

# rbd showmapped
id pool           namespace image                                                       snap device    
0  caasp4-dynamic           kubernetes-dynamic-pvc-e6d98bbf-50e1-4488-a9fc-867d1db810c8 -    /dev/rbd0

# df -h | grep dev/rbd0  
/dev/rbd0                2.9G  9.0M  2.9G   1% /var/lib/kubelet/pods/e7c75785-4533-4fac-b4ab-368c75e16421/volumes/kubernetes.io~rbd/pvc-70b3b3ca-5267-4417-a898-c1f5524de008

排錯

1、搭建的時候，發現創建PVC的時候 pending 狀態。

# kubectl get pvc
NAME                 STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
ceph-claim-dynamic   Pending                                      dynamic        44s

2、通過 event 事件查看，由於忘記創建 admin secret 導致，重新創建 admin secret 即可。

# kubectl get events
LAST SEEN   TYPE      REASON               OBJECT                                     MESSAGE
60s         Warning   ProvisioningFailed   persistentvolumeclaim/ceph-claim-dynamic   Failed to provision volume with StorageClass "dynamic": failed to get admin secret from ["kube-system"/"ceph-secret-admin"]: failed to get secret from ["kube-system"/"ceph-secret-admin"]: secrets "ceph-secret-admin" not found
35s         Warning   FailedScheduling     pod/ceph-pod1-dynamic                      pod has unbound immediate PersistentVolumeClaims (repeated 2 times)

SUSE CaaS Platform 4 - 使用 Ceph RBD 作為持久存儲（動態）

存儲類

一、Provisioner（存儲分配器）

二、實驗環境搭建 - 動態供給

環境設置

安裝部署

排錯