forked from intel/pmem-csi
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Implemented persistency models - Cache and Persistent
With this change, PMEM volumes are created in CreateVolume() call, irrespective of persistency model. This means there is no 'delayed volume' creation in ControllerPublishVolume. In normal persistent volume case the driver creates a pmem volume on a free node in the order of 'CreateVolumeRequest.TopologyRequirement.Preferred'. And locks the newly created volume to that node by using 'Volume.Topology'. The container orchestrator has to make sure that the application which claims this volume shall run on this node. Unlike normal persistent volume, cache volume creates set of pmem volumes on different number of nodes, each with its own local data, in the order of 'CreateVolumeRequest.TopologyRequirement.Preferred'. Applications are started on those nodes and then get to use the volume on their node. Data persists across application restarts. This is useful when the data is only cached information that can be discarded and reconstructed at any time and the application can reuse existing local data when restarting. Introduced new volume/StorageClass parameters: - "persistencyModel", which shall set to "cache" for cache volumes and - "cacheSize" to request no of nodes the created volume can be used. Updated README with different possible persistency models and Kubernetes usage for cache and persistent volumes. Removed node locking for volume tests, as we implemented full topology and provisioning in cluster supposed to work.
- Loading branch information
Showing
16 changed files
with
554 additions
and
215 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,42 @@ | ||
| apiVersion: apps/v1beta2 | ||
| kind: ReplicaSet | ||
| metadata: | ||
| name: my-csi-app | ||
| spec: | ||
| selector: | ||
| matchLabels: | ||
| app: my-csi-app | ||
| replicas: 2 | ||
| template: | ||
| metadata: | ||
| labels: | ||
| app: my-csi-app | ||
| spec: | ||
| # make sure that no two Pods run on same node | ||
| affinity: | ||
| podAntiAffinity: | ||
| requiredDuringSchedulingIgnoredDuringExecution: | ||
| - labelSelector: | ||
| matchExpressions: | ||
| - key: app | ||
| operator: In | ||
| values: [ my-csi-app ] | ||
| topologyKey: "kubernetes.io/hostname" | ||
| containers: | ||
| - name: my-frontend | ||
| image: busybox | ||
| command: [ "/bin/sh" ] | ||
| args: [ "-c", "touch /data/$(POD_NAME); sleep 100000" ] | ||
| env: | ||
| - name: POD_NAME | ||
| valueFrom: | ||
| fieldRef: | ||
| apiVersion: v1 | ||
| fieldPath: metadata.name | ||
| volumeMounts: | ||
| - mountPath: "/data" | ||
| name: my-csi-volume | ||
| volumes: | ||
| - name: my-csi-volume | ||
| persistentVolumeClaim: | ||
| claimName: pmem-csi-pvc-cache |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| apiVersion: v1 | ||
| kind: PersistentVolumeClaim | ||
| metadata: | ||
| name: pmem-csi-pvc-cache | ||
| spec: | ||
| accessModes: | ||
| - ReadWriteMany # cache volumes are multi-node volumes | ||
| resources: | ||
| requests: | ||
| storage: 8Gi | ||
| storageClassName: pmem-csi-sc-cache # defined in pmem-storageclass-cache.yaml |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,10 @@ | ||
| apiVersion: storage.k8s.io/v1 | ||
| kind: StorageClass | ||
| metadata: | ||
| name: pmem-csi-sc-cache | ||
| provisioner: pmem-csi | ||
| reclaimPolicy: Delete | ||
| volumeBindingMode: Immediate | ||
| parameters: | ||
| persistencyModel: cache | ||
| cacheSize: "2" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,170 @@ | ||
| @startuml "pmem-csi-cache-sequence-diagram" | ||
|
|
||
| title \nDynamic provisioning of pmem-csi "cache" volume\n | ||
|
|
||
| skinparam BoxPadding 40 | ||
|
|
||
| actor Admin as admin #red | ||
| actor User as user | ||
| entity Kubernetes as k8s | ||
| box "Master node" | ||
| entity kubelet as masterkubelet | ||
| participant "external-provisioner" as provisioner | ||
| participant "external-attacher" as attacher | ||
| participant "pmem-csi-driver" as masterdriver | ||
| endbox | ||
|
|
||
| box "Compute node X" | ||
| entity kubelet as nodekubeletX | ||
| participant "pmem-csi-driver" as nodedriverX | ||
| endbox | ||
|
|
||
| box "Compute node Y" | ||
| entity kubelet as nodekubeletY | ||
| participant "pmem-csi-driver" as nodedriverY | ||
| endbox | ||
|
|
||
| == Driver setup == | ||
| admin->k8s:Label nvdimm nodes: <b>storage=nvdimm</b> | ||
| k8s->admin | ||
|
|
||
| ' deploy driver | ||
| admin->k8s:deploy driver\nkubectl create -f pmem-csi.yaml | ||
| k8s->admin | ||
| k8s->masterkubelet:start driver pod | ||
| masterkubelet-->provisioner:start container | ||
| masterkubelet-->attacher:start container | ||
| masterkubelet-->masterdriver:start container | ||
| note right of masterdriver | ||
| listen on tcp port 10000 | ||
| end note | ||
| k8s-->nodekubeletX:start driver pod | ||
| nodekubeletX-->nodedriverX:start container | ||
| note left of nodedriverX | ||
| * prepare logical volume groups | ||
| * listen on port 10001 | ||
| * listen on unix socket: | ||
| /var/lib/kubelet/plugins/pmem-csi/csi.sock | ||
| end note | ||
| nodedriverX->masterdriver:RegistryServer.RegisterNodeController(\n{nodeId:"node-x", endpoint:"http://ip:10001"}) | ||
|
|
||
| k8s-->nodekubeletY:start driver pod | ||
| nodekubeletY-->nodedriverY:start container | ||
| note left of nodedriverY | ||
| * prepare logical volume groups | ||
| * listen on port 10001 | ||
| * listen on unix socket: | ||
| /var/lib/kubelet/plugins/pmem-csi/csi.sock | ||
| end note | ||
| nodedriverY->masterdriver:RegistryServer.RegisterNodeController(\n{nodeId:"node-y", endpoint:"http://ip:10001"}) | ||
|
|
||
| ' install a storage class | ||
| admin->k8s:create StorageClass\nkubectl create -f pmem-storageclass-cache.yaml | ||
| note left of k8s | ||
| metadata: | ||
| name: pmem-csi-sc-cache | ||
| volumeBindingMode: <b>Immediate | ||
| paramters: | ||
| persistencyModel: cache | ||
| cacheSize: "2" | ||
| end note | ||
| k8s->admin | ||
|
|
||
| ' provision a cache volume | ||
| == Volume provisioning == | ||
| admin->k8s:create PVC object\nkubectl create -f pmem-pvc-cache.yaml | ||
| note left of k8s | ||
| metatdata: | ||
| name: pmem-csi-pvc-cache | ||
| spec: | ||
| storageClassName: pmem-csi-sc-cache | ||
| end note | ||
| k8s->admin | ||
| k8s-->provisioner:<<Event>>\nPersistentVolumeClaim created | ||
| activate provisioner | ||
| provisioner->masterdriver:CSI.Controller.CreateVolume() | ||
| masterdriver->nodedriverX:csi.Controller.CreateVolume() | ||
| nodedriverX->nodedriverX:create pmem volume | ||
| nodedriverX->masterdriver:success | ||
| masterdriver->nodedriverY:csi.Controller.CreateVolume() | ||
| nodedriverY->nodedriverY:create pmem volume | ||
| nodedriverY->masterdriver:success | ||
| masterdriver->provisioner:success | ||
| note left of masterdriver | ||
| prepare Topology information: | ||
| Volume{ | ||
| accessible_topology: [ | ||
| segments:{ "pmem-csi.intel.com/node":"node-x"}, | ||
| segments:{ "pmem-csi.intel.com/node":"node-y"} ] | ||
| } | ||
| end note | ||
| provisioner->k8s:Create PV object | ||
| deactivate provisioner | ||
|
|
||
| == Volume usage == | ||
| ' Start an application | ||
| user->k8s:Create application pod | ||
| note left of k8s | ||
| volumes: | ||
| - name: my-csi-volume | ||
| persistentVolumeClaim: | ||
| claimName: pmem-csi-pvc-cache | ||
| end note | ||
|
|
||
| k8s->user:success | ||
|
|
||
| k8s->nodekubeletX:schedules pod on node-x | ||
| note right of k8s | ||
| Kubernetes is might choose <b>node-x</b> or <b>node-y</b>. | ||
| end note | ||
|
|
||
| k8s-->nodekubeletX:make available volume to pod | ||
| nodekubeletX->nodedriverX:csi.Node.StageVolume() | ||
| activate nodedriverX | ||
| nodedriverX->nodedriverX:mount pmem device | ||
| nodedriverX->nodekubeletX:success | ||
| deactivate nodedriverX | ||
|
|
||
| nodekubeletX->nodedriverX:csi.Node.PublishVolume() | ||
| activate nodedriverX | ||
| nodedriverX->nodedriverX:bind mount pmem device | ||
| nodedriverX->nodekubeletX:success | ||
| deactivate nodedriverX | ||
|
|
||
| ' deprovision a cache volume | ||
| == Volume Deletion == | ||
| ' stop pod | ||
| user->k8s:stop applicaiton pod | ||
| k8s->user:success | ||
| k8s->nodekubeletX:stop pod containers | ||
|
|
||
| nodekubeletX->nodedriverX:csi.Node.UnPublishVolume() | ||
| activate nodedriverX | ||
| nodedriverX->nodedriverX:unmout pod's bind mount | ||
| nodedriverX->nodekubeletX:success | ||
| deactivate nodedriverX | ||
|
|
||
| nodekubeletX->nodedriverX:csi.Node.UnstageVolume() | ||
| activate nodedriverX | ||
| nodedriverX->nodedriverX:unmount pmem device | ||
| nodedriverX->nodekubeletX:success | ||
| deactivate nodedriverX | ||
|
|
||
| ''''''''''''''''''''''''''' | ||
| admin->k8s:Delete PVC object\nkubectl delete pvc pmem-pvc-cache | ||
| k8s->admin | ||
| k8s-->provisioner:<<Event>>\nPersistentVolumeClaim deleted | ||
| activate provisioner | ||
| provisioner->masterdriver:CSI.Controller.DeleteVolume() | ||
| masterdriver->nodedriverX:csi.Controller.DeleteVolume() | ||
| nodedriverX->nodedriverX:delete pmem volume | ||
| nodedriverX->masterdriver:success | ||
| masterdriver->nodedriverY:csi.Controller.DeleteVolume() | ||
| nodedriverY->nodedriverY:delete pmem volume | ||
| nodedriverY->masterdriver:success | ||
| masterdriver->provisioner:success | ||
| provisioner->k8s:Delete PV object | ||
| deactivate provisioner | ||
|
|
||
|
|
||
| @enduml |
Oops, something went wrong.