一 CSI存儲機制 1.1 CSI簡介 Kubernetes從1.9版本開始引入容器存儲介面Container Storage Interface(CSI)機制,用於在Kubernetes和外部存儲系統之間建立一套標準的存儲管理介面,通過該介面為容器提供存儲服務。 1.2 CSI的設計背景 Kube ...
一 CSI存儲機制
1.1 CSI簡介
Kubernetes從1.9版本開始引入容器存儲介面Container Storage Interface(CSI)機制,用於在Kubernetes和外部存儲系統之間建立一套標準的存儲管理介面,通過該介面為容器提供存儲服務。1.2 CSI的設計背景
Kubernetes通過PV、PVC、Storageclass已經提供了一種強大的基於插件的存儲管理機制,但是各種存儲插件提供的存儲服務都是基於一種被稱為“in-true”(樹內)的方式提供的,這要求存儲插件的代碼必須被放進Kubernetes的主幹代碼庫中才能被Kubernetes調用,屬於緊耦合的開發模式。這種“in-tree”方式會帶來一些問題:- 存儲插件的代碼需要與Kubernetes的代碼放在同一代碼庫中,並與Kubernetes的二進位文件共同發佈;
- 存儲插件代碼的開發者必須遵循Kubernetes的代碼開發規範;
- 存儲插件代碼的開發者必須遵循Kubernetes的發佈流程,包括添加對Kubernetes存儲系統的支持和錯誤修複;
- Kubernetes社區需要對存儲插件的代碼進行維護,包括審核、測試等工作;
- 存儲插件代碼中的問題可能會影響Kubernetes組件的運行,並且很難排查問題;
- 存儲插件代碼與Kubernetes的核心組件(kubelet和kubecontroller-manager)享有相同的系統特權許可權,可能存在可靠性和安全性問題。
Kubernetes已有的FlexVolume插件機制試圖通過為外部存儲暴露一個基於可執行程式(exec)的API來解決這些問題。儘管它允許第三方存儲提供商在Kubernetes核心代碼之外開發存儲驅動,但仍然有兩個問題沒有得到很好的解決:
- 部署第三方驅動的可執行文件仍然需要宿主機的root許可權,存在安全隱患;
- 存儲插件在執行mount、attach這些操作時,通常需要在宿主機上安裝一些第三方工具包和依賴庫,使得部署過程更加複雜,例如部署Ceph時需要安裝rbd庫,部署GlusterFS時需要安裝mount.glusterfs庫,等等。
基於以上這些問題和考慮,Kubernetes逐步推出與容器對接的存儲介面標準,存儲提供方只需要基於標準介面進行存儲插件的實現,就能使用Kubernetes的原生存儲機製為容器提供存儲服務。這套標準被稱為CSI(容器存儲介面)。 在CSI成為Kubernetes的存儲供應標準之後,存儲提供方的代碼就能和Kubernetes代碼徹底解耦,部署也與Kubernetes核心組件分離,顯然,存儲插件的開發由提供方自行維護,就能為Kubernetes用戶提供更多的存儲功能,也更加安全可靠。 基於CSI的存儲插件機制也被稱為“out-of-tree”(樹外)的服務提供方式,是未來Kubernetes第三方存儲插件的標準方案。
二 CSI架構
2.1 CSI存儲組件/部署架構
KubernetesCSI存儲插件的關鍵組件和推薦的容器化部署架構:
2.2 CSI Controller
CSI Controller的主要功能是提供存儲服務視角對存儲資源和存儲捲進行管理和操作。在Kubernetes中建議將其部署為單實例Pod,可以使用StatefulSet或Deployment控制器進行部署,設置副本數量為1,保證為一種存儲插件只運行一個控制器實例。 在這個Pod內部署兩個容器:- 與Master(kube-controller-manager)通信的輔助sidecar容器。在sidecar容器內又可以包含external-attacher和external-provisioner兩個容器,它們的功能分別如下。
- external-attacher:監控VolumeAttachment資源對象的變更,觸髮針對CSI端點的ControllerPublish和ControllerUnpublish操作。
- external-provisioner:監控PersistentVolumeClaim資源對象的變更,觸髮針對CSI端點的CreateVolume和DeleteVolume操作。
- CSI Driver存儲驅動容器,由第三方存儲提供商提供,需要實現上述介面。
2.3 CSI Node
CSI Node的主要功能是對主機(Node)上的Volume進行管理和操作。在Kubernetes中建議將其部署為DaemonSet,在每個Node上都運行一個Pod。 在這個Pod中部署以下兩個容器:- 與kubelet通信的輔助sidecar容器node-driver-registrar,主要功能是將存儲驅動註冊到kubelet中;
- CSI Driver存儲驅動容器,由第三方存儲提供商提供,主要功能是接收kubelet的調用,需要實現一系列與Node相關的CSI介面,例如NodePublishVolume介面(用於將Volume掛載到容器內的目標路徑)、NodeUnpublishVolume介面(用於從容器中卸載Volume),等等。
三 CSI插件使用實踐
3.1 實驗說明
以csi-hostpath插件為例,演示部署CSI插件、用戶使用CSI插件提供的存儲資源。3.2 開啟特性
設置Kubernetes服務啟動參數,為kube-apiserver、kubecontroller-manager和kubelet服務的啟動參數添加。 [root@k8smaster01 ~]# vi /etc/kubernetes/manifests/kube-apiserver.yaml…… - --allow-privileged=true - --feature-gates=CSIPersistentVolume=true - --runtime-config=storage.k8s.io/v1alpha1=true ……[root@k8smaster01 ~]# vi /etc/kubernetes/manifests/kube-controller-manager.yaml
…… - --feature-gates=CSIPersistentVolume=true ……[root@k8smaster01 ~]# vi /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf
# Note: This dropin only works with kubeadm and kubelet v1.11+ [Service] Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --feature-gates=CSIPersistentVolume=true" ……[root@k8smaster01 ~]# systemctl daemon-reload [root@k8smaster01 ~]# systemctl restart kubelet.service
3.3 創建CRD資源對象
創建CSINodeInfo和CSIDriverRegistry CRD資源對象: [root@k8smaster01 ~]# vi csidriver.yamlapiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: name: csidrivers.csi.storage.k8s.io labels: addonmanager.kubernetes.io/mode: Reconcile spec: group: csi.storage.k8s.io names: kind: CSIDriver plural: csidrivers scope: Cluster validation: openAPIV3Schema: properties: spec: description: Specification of the CSI Driver. properties: attachRequired: description: Indicates this CSI volume driver requires an attach operation,and that Kubernetes should call attach and wait for any attach operationto complete before proceeding to mount. type: boolean podInfoOnMountVersion: description: Indicates this CSI volume driver requires additional pod information (like podName, podUID, etc.) during mount operations. type: string version: v1alpha1[root@k8smaster01 ~]# vi csinodeinfo.yaml
apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: name: csinodeinfos.csi.storage.k8s.io labels: addonmanager.kubernetes.io/mode: Reconcile spec: group: csi.storage.k8s.io names: kind: CSINodeInfo plural: csinodeinfos scope: Cluster validation: openAPIV3Schema: properties: spec: description: Specification of CSINodeInfo properties: drivers: description: List of CSI drivers running on the node and their specs. type: array items: properties: name: description: The CSI driver that this object refers to. type: string nodeID: description: The node from the driver point of view. type: string topologyKeys: description: List of keys supported by the driver. items: type: string type: array status: description: Status of CSINodeInfo properties: drivers: description: List of CSI drivers running on the node and their statuses. type: array items: properties: name: description: The CSI driver that this object refers to. type: string available: description: Whether the CSI driver is installed. type: boolean volumePluginMechanism: description: Indicates to external components the required mechanism to use for any in-tree plugins replaced by this driver. pattern: in-tree|csi type: string version: v1alpha1[root@k8smaster01 ~]# kubectl apply -f csidriver.yaml [root@k8smaster01 ~]# kubectl apply -f csinodeinfo.yaml
3.4 創建相應RBAC
[root@k8smaster01 ~]# git clone https://github.com/kubernetes-csi/drivers [root@k8smaster01 ~]# cd drivers/deploy/hostpath/ [root@k8smaster01 hostpath]# vi csi-hostpath-attacher-rbac.yaml--- apiVersion: v1 kind: ServiceAccount metadata: name: csi-attacher # replace with non-default namespace name namespace: default --- kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: external-attacher-runner rules: - apiGroups: [""] resources: ["persistentvolumes"] verbs: ["get", "list", "watch", "update"] - apiGroups: [""] resources: ["nodes"] verbs: ["get", "list", "watch"] - apiGroups: ["csi.storage.k8s.io"] resources: ["csinodeinfos"] verbs: ["get", "list", "watch"] - apiGroups: ["storage.k8s.io"] resources: ["volumeattachments"] verbs: ["get", "list", "watch", "update"] --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: csi-attacher-role subjects: - kind: ServiceAccount name: csi-attacher # replace with non-default namespace name namespace: default roleRef: kind: ClusterRole name: external-attacher-runner apiGroup: rbac.authorization.k8s.io --- kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: # replace with non-default namespace name namespace: default name: external-attacher-cfg rules: - apiGroups: [""] resources: ["configmaps"] verbs: ["get", "watch", "list", "delete", "update", "create"] --- kind: RoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: csi-attacher-role-cfg # replace with non-default namespace name namespace: default subjects: - kind: ServiceAccount name: csi-attacher # replace with non-default namespace name namespace: default roleRef: kind: Role name: external-attacher-cfg apiGroup: rbac.authorization.k8s.io[root@k8smaster01 hostpath]# vi csi-hostpath-provisioner-rbac.yaml
--- apiVersion: v1 kind: ServiceAccount metadata: name: csi-provisioner # replace with non-default namespace name namespace: default --- kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: external-provisioner-runner rules: - apiGroups: [""] resources: ["secrets"] verbs: ["get", "list"] - apiGroups: [""] resources: ["persistentvolumes"] verbs: ["get", "list", "watch", "create", "delete"] - apiGroups: [""] resources: ["persistentvolumeclaims"] verbs: ["get", "list", "watch", "update"] - apiGroups: ["storage.k8s.io"] resources: ["storageclasses"] verbs: ["get", "list", "watch"] - apiGroups: [""] resources: ["events"] verbs: ["list", "watch", "create", "update", "patch"] - apiGroups: ["snapshot.storage.k8s.io"] resources: ["volumesnapshots"] verbs: ["get", "list"] - apiGroups: ["snapshot.storage.k8s.io"] resources: ["volumesnapshotcontents"] verbs: ["get", "list"] - apiGroups: ["csi.storage.k8s.io"] resources: ["csinodeinfos"] verbs: ["get", "list", "watch"] - apiGroups: [""] resources: ["nodes"] verbs: ["get", "list", "watch"] --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: csi-provisioner-role subjects: - kind: ServiceAccount name: csi-provisioner # replace with non-default namespace name namespace: default roleRef: kind: ClusterRole name: external-provisioner-runner apiGroup: rbac.authorization.k8s.io --- kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: # replace with non-default namespace name namespace: default name: external-provisioner-cfg rules: - apiGroups: [""] resources: ["endpoints"] verbs: ["get", "watch", "list", "delete", "update", "create"] --- kind: RoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: csi-provisioner-role-cfg # replace with non-default namespace name namespace: default subjects: - kind: ServiceAccount name: csi-provisioner # replace with non-default namespace name namespace: default roleRef: kind: Role name: external-provisioner-cfg apiGroup: rbac.authorization.k8s.io[root@k8smaster01 hostpath]# vi csi-hostpathplugin-rbac.yaml
--- apiVersion: v1 kind: ServiceAccount metadata: name: csi-driver-registrar # replace with non-default namespace name namespace: default --- kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: driver-registrar-runner rules: - apiGroups: [""] resources: ["events"] verbs: ["get", "list", "watch", "create", "update", "patch"] # The following permissions are only needed when running # driver-registrar without the --kubelet-registration-path # parameter, i.e. when using driver-registrar instead of # kubelet to update the csi.volume.kubernetes.io/nodeid # annotation. That mode of operation is going to be deprecated # and should not be used anymore, but is needed on older # Kubernetes versions. # - apiGroups: [""] # resources: ["nodes"] # verbs: ["get", "update", "patch"] --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: csi-driver-registrar-role subjects: - kind: ServiceAccount name: csi-driver-registrar # replace with non-default namespace name namespace: default roleRef: kind: ClusterRole name: driver-registrar-runner apiGroup: rbac.authorization.k8s.io[root@k8smaster01 hostpath]# kubectl create -f csi-hostpath-attacher-rbac.yaml [root@k8smaster01 hostpath]# kubectl create -f csi-hostpath-provisioner-rbac.yaml [root@k8smaster01 hostpath]# kubectl create -f csi-hostpathplugin-rbac.yaml
3.5 正式部署
[root@k8smaster01 ~]# cd drivers/deploy/hostpath/ [root@k8smaster01 hostpath]# kubectl create -f csi-hostpath-attacher.yaml [root@k8smaster01 hostpath]# kubectl create -f csi-hostpath-provisioner.yaml [root@k8smaster01 hostpath]# kubectl create -f csi-hostpathplugin.yaml 提示:如上相應yaml建議修改鏡像源為國內: gcr.io ----> gcr.azk8s.cn (國內) quay.io ----> quay.azk8s.cn (國內)四 測試使用
4.1 確認驗證
[root@k8smaster01 ~]# kubectl get pods
4.2 創建StorageClass
[root@k8smaster01 ~]# vi drivers/examples/hostpath/csi-storageclass.yamlapiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: csi-hostpath-sc
provisioner: csi-hostpath
reclaimPolicy: Delete
volumeBindingMode: Immediate
[root@k8smaster01 ~]# kubectl create -f drivers/examples/hostpath/csi-storageclass.yaml
4.3 創建PVC
[root@k8smaster01 ~]# vi drivers/examples/hostpath/csi-pvc.yamlapiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: csi-pvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
storageClassName: csi-hostpath-sc
[root@k8smaster01 ~]# kubectl create -f drivers/examples/hostpath/csi-pvc.yaml
[root@k8smaster01 ~]# kubectl get pvc
[root@k8smaster01 ~]# kubectl get pv

4.4 創建應用
[root@k8smaster01 ~]# vi drivers/examples/hostpath/csi-app.yamlkind: Pod apiVersion: v1 metadata: name: my-csi-app spec: containers: - name: my-frontend image: busybox volumeMounts: - mountPath: "/data" name: my-csi-volume command: [ "sleep", "1000000" ] volumes: - name: my-csi-volume persistentVolumeClaim: claimName: csi-pvc[root@k8smaster01 ~]# kubectl create -f drivers/examples/hostpath/csi-app.yaml [root@k8smaster01 ~]# kubectl get pods

提示:更多CSI插件示例參考:https://feisky.gitbooks.io/kubernetes/plugins/csi.html。 CSI官方文檔:https://kubernetes-csi.github.io/docs/