diff --git a/README.md b/README.md index 6335edcb..6219f990 100644 --- a/README.md +++ b/README.md @@ -18,6 +18,7 @@ - [Creating a PersistentVolumeClaim](docs/usage.md#creating-a-persistentvolumeclaim) - [Encrypted Drives using LUKS](docs/encrypted-drives.md) - [Adding Tags to Created Volumes](docs/volume-tags.md) + - [Topology-Aware Provisioning](docs/topology-aware-provisioning.md) - [Development Setup](docs/development-setup.md) - [Prerequisites](docs/development-setup.md#-prerequisites) - [Setting Up the Local Development Environment](docs/development-setup.md#-setting-up-the-local-development-environment) diff --git a/deploy/kubernetes/base/csi-storageclass.yaml b/deploy/kubernetes/base/csi-storageclass.yaml index b40ca161..40338fdb 100644 --- a/deploy/kubernetes/base/csi-storageclass.yaml +++ b/deploy/kubernetes/base/csi-storageclass.yaml @@ -16,3 +16,21 @@ metadata: provisioner: linodebs.csi.linode.com reclaimPolicy: Retain allowVolumeExpansion: true +--- +kind: StorageClass +apiVersion: storage.k8s.io/v1 +metadata: + name: linode-block-storage-wait-for-consumer + namespace: kube-system +provisioner: linodebs.csi.linode.com +reclaimPolicy: Delete +volumeBindingMode: WaitForFirstConsumer +--- +kind: StorageClass +apiVersion: storage.k8s.io/v1 +metadata: + name: linode-block-storage-wait-for-consumer-retain + namespace: kube-system +provisioner: linodebs.csi.linode.com +reclaimPolicy: Retain +volumeBindingMode: WaitForFirstConsumer diff --git a/deploy/kubernetes/base/ss-csi-linode-controller.yaml b/deploy/kubernetes/base/ss-csi-linode-controller.yaml index bda99931..b85a12e1 100644 --- a/deploy/kubernetes/base/ss-csi-linode-controller.yaml +++ b/deploy/kubernetes/base/ss-csi-linode-controller.yaml @@ -43,6 +43,7 @@ spec: - "--volume-name-prefix=pvc" - "--volume-name-uuid-length=16" - "--csi-address=$(ADDRESS)" + - "--feature-gates=Topology=true" - "--v=2" env: - name: ADDRESS diff --git a/docs/topology-aware-provisioning.md b/docs/topology-aware-provisioning.md new file mode 100644 index 00000000..84679b74 --- /dev/null +++ b/docs/topology-aware-provisioning.md @@ -0,0 +1,88 @@ +## 🌐 Topology-Aware Provisioning + +This CSI driver supports topology-aware provisioning, optimizing volume placement based on the physical infrastructure layout. + +**Notes:** + +1. **Volume Cloning**: Cloning only works within the same region, not across regions. +2. **Volume Migration**: We can't move volumes across regions. +3. **Remote Provisioning**: Volume provisioning is supported in remote regions (nodes or clusters outside of the region where the controller server is deployed). + +> [!IMPORTANT] +> Make sure you are using the latest release v0.8.6+ to utilize the remote provisioning feature. + +#### 📝 Example StorageClass and PVC + +```yaml +allowVolumeExpansion: true +apiVersion: storage.k8s.io/v1 +kind: StorageClass +metadata: + name: linode-block-storage-wait-for-consumer +provisioner: linodebs.csi.linode.com +reclaimPolicy: Delete +volumeBindingMode: WaitForFirstConsumer +--- +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: pvc-filesystem +spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 10Gi + storageClassName: linode-block-storage-wait-for-consumer +``` + +> **Important**: The `volumeBindingMode: WaitForFirstConsumer` setting is crucial for topology-aware provisioning. It delays volume binding and creation until a pod using the PVC is created. This allows the system to consider the pod's scheduling requirements and node assignment when selecting the most appropriate storage location, ensuring optimal data locality and performance. + +#### 🖥️ Example Pod + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: e2e-pod +spec: + nodeSelector: + topology.linode.com/region: us-ord + tolerations: + - key: "node-role.kubernetes.io/control-plane" + operator: "Exists" + effect: "NoSchedule" + containers: + - name: e2e-pod + image: ubuntu + command: + - sleep + - "1000000" + volumeMounts: + - mountPath: /data + name: csi-volume + volumes: + - name: csi-volume + persistentVolumeClaim: + claimName: pvc-filesystem +``` + +This example demonstrates how to set up topology-aware provisioning using the Linode Block Storage CSI Driver. The StorageClass defines the provisioner and reclaim policy, while the PersistentVolumeClaim requests storage from this class. The Pod specification shows how to use the PVC and includes a node selector for region-specific deployment. + +> [!IMPORTANT] +> To enable topology-aware provisioning, make sure to pass the following argument to the csi-provisioner sidecar: +> ``` +> --feature-gates=CSINodeInfo=true +> ``` +> This enables the CSINodeInfo feature gate, which is required for topology-aware provisioning to function correctly. +> +> Note: This feature is enabled by default in release v0.8.6 and later versions. + +#### Provisioning Process + +1. CO (Kubernetes) determines required topology based on application needs (pod scheduled region) and cluster layout. +2. external-provisioner gathers topology requirements from CO and includes `TopologyRequirement` in `CreateVolume` call. +3. CSI driver creates volume satisfying topology requirements. +4. Driver returns actual topology of created volume. + +By leveraging topology-aware provisioning, CSI drivers ensure optimal volume placement within the infrastructure, improving performance, availability, and data locality. diff --git a/helm-chart/csi-driver/templates/csi-linode-controller.yaml b/helm-chart/csi-driver/templates/csi-linode-controller.yaml index b69f5dc6..6e4c2ead 100644 --- a/helm-chart/csi-driver/templates/csi-linode-controller.yaml +++ b/helm-chart/csi-driver/templates/csi-linode-controller.yaml @@ -23,6 +23,7 @@ spec: - --volume-name-prefix=pvc - --volume-name-uuid-length=16 - --csi-address=$(ADDRESS) + - --feature-gates=Topology=true - --v=2 {{- if .Values.enable_metrics}} - --metrics-address={{ .Values.csiProvisioner.metrics.address }} diff --git a/helm-chart/csi-driver/templates/linode-block-storage-topology-aware-retain.yaml b/helm-chart/csi-driver/templates/linode-block-storage-topology-aware-retain.yaml new file mode 100644 index 00000000..365a4630 --- /dev/null +++ b/helm-chart/csi-driver/templates/linode-block-storage-topology-aware-retain.yaml @@ -0,0 +1,17 @@ +apiVersion: storage.k8s.io/v1 +kind: StorageClass +metadata: + name: linode-block-storage-wait-for-consumer-retain + namespace: {{ required ".Values.namespace required" .Values.namespace }} +{{- if eq .Values.defaultStorageClass "linode-block-storage-wait-for-consumer-retain" }} + annotations: + storageclass.kubernetes.io/is-default-class: "true" +{{- end }} +{{- if .Values.volumeTags }} +parameters: + linodebs.csi.linode.com/volumeTags: {{ join "," .Values.volumeTags }} +{{- end}} +allowVolumeExpansion: true +provisioner: linodebs.csi.linode.com +reclaimPolicy: Retain +volumeBindingMode: WaitForFirstConsumer diff --git a/helm-chart/csi-driver/templates/linode-block-storage-topology-aware.yaml b/helm-chart/csi-driver/templates/linode-block-storage-topology-aware.yaml new file mode 100644 index 00000000..cf9cecc7 --- /dev/null +++ b/helm-chart/csi-driver/templates/linode-block-storage-topology-aware.yaml @@ -0,0 +1,17 @@ +apiVersion: storage.k8s.io/v1 +kind: StorageClass +metadata: + name: linode-block-storage-wait-for-consumer + namespace: {{ required ".Values.namespace required" .Values.namespace }} +{{- if eq .Values.defaultStorageClass "linode-block-storage-wait-for-consumer" }} + annotations: + storageclass.kubernetes.io/is-default-class: "true" +{{- end }} +{{- if .Values.volumeTags }} +parameters: + linodebs.csi.linode.com/volumeTags: {{ join "," .Values.volumeTags }} +{{- end}} +allowVolumeExpansion: true +provisioner: linodebs.csi.linode.com +reclaimPolicy: Delete +volumeBindingMode: WaitForFirstConsumer diff --git a/internal/driver/controllerserver.go b/internal/driver/controllerserver.go index 1f05d8aa..dfe0e6a3 100644 --- a/internal/driver/controllerserver.go +++ b/internal/driver/controllerserver.go @@ -76,24 +76,27 @@ func (cs *ControllerServer) CreateVolume(ctx context.Context, req *csi.CreateVol return &csi.CreateVolumeResponse{}, err } - // Create volume context - volContext := cs.createVolumeContext(ctx, req) + contentSource := req.GetVolumeContentSource() + accessibilityRequirements := req.GetAccessibilityRequirements() // Attempt to retrieve information about a source volume if the request includes a content source. // This is important for scenarios where the volume is being cloned from an existing one. - sourceVolInfo, err := cs.getContentSourceVolume(ctx, req.GetVolumeContentSource()) + sourceVolInfo, err := cs.getContentSourceVolume(ctx, contentSource, accessibilityRequirements) if err != nil { return &csi.CreateVolumeResponse{}, err } // Create the volume - vol, err := cs.createAndWaitForVolume(ctx, volName, sizeGB, req.GetParameters()[VolumeTags], sourceVolInfo) + vol, err := cs.createAndWaitForVolume(ctx, volName, sizeGB, req.GetParameters()[VolumeTags], sourceVolInfo, accessibilityRequirements) if err != nil { return &csi.CreateVolumeResponse{}, err } + // Create volume context + volContext := cs.createVolumeContext(ctx, req, vol) + // Prepare and return response - resp := cs.prepareCreateVolumeResponse(ctx, vol, size, volContext, sourceVolInfo, req.GetVolumeContentSource()) + resp := cs.prepareCreateVolumeResponse(ctx, vol, size, volContext, sourceVolInfo, contentSource) log.V(2).Info("CreateVolume response", "response", resp) return resp, nil @@ -154,9 +157,15 @@ func (cs *ControllerServer) ControllerPublishVolume(ctx context.Context, req *cs return resp, err } + // Retrieve and validate the instance associated with the Linode ID + instance, err := cs.getInstance(ctx, linodeID) + if err != nil { + return resp, err + } + // Check if the volume exists and is valid. // If the volume is already attached to the specified instance, it returns its device path. - devicePath, err := cs.getAndValidateVolume(ctx, volumeID, linodeID) + devicePath, err := cs.getAndValidateVolume(ctx, volumeID, instance, req.GetVolumeContext()) if err != nil { return resp, err } @@ -169,12 +178,6 @@ func (cs *ControllerServer) ControllerPublishVolume(ctx context.Context, req *cs }, nil } - // Retrieve and validate the instance associated with the Linode ID - instance, err := cs.getInstance(ctx, linodeID) - if err != nil { - return resp, err - } - // Check if the instance can accommodate the volume attachment if capErr := cs.checkAttachmentCapacity(ctx, instance); capErr != nil { return resp, capErr diff --git a/internal/driver/controllerserver_helper.go b/internal/driver/controllerserver_helper.go index 88cdd2d1..4c0189c1 100644 --- a/internal/driver/controllerserver_helper.go +++ b/internal/driver/controllerserver_helper.go @@ -131,7 +131,7 @@ func (cs *ControllerServer) maxAllowedVolumeAttachments(ctx context.Context, ins // getContentSourceVolume retrieves information about the Linode volume to clone from. // It returns a LinodeVolumeKey if a valid source volume is found, or an error if the source is invalid. -func (cs *ControllerServer) getContentSourceVolume(ctx context.Context, contentSource *csi.VolumeContentSource) (volKey *linodevolumes.LinodeVolumeKey, err error) { +func (cs *ControllerServer) getContentSourceVolume(ctx context.Context, contentSource *csi.VolumeContentSource, accessibilityRequirements *csi.TopologyRequirement) (volKey *linodevolumes.LinodeVolumeKey, err error) { log := logger.GetLogger(ctx) log.V(4).Info("Attempting to get content source volume") @@ -167,9 +167,16 @@ func (cs *ControllerServer) getContentSourceVolume(ctx context.Context, contentS return nil, errInternal("source volume *linodego.Volume is nil") // Throw an internal error if the processed linodego.Volume is nil } - // Check if the volume's region matches the server's metadata region - if volumeData.Region != cs.metadata.Region { - return nil, errRegionMismatch(volumeData.Region, cs.metadata.Region) + // Check if the source volume's region matches the required region + requiredRegion := cs.metadata.Region + if accessibilityRequirements != nil { + if topologyRegion := getRegionFromTopology(accessibilityRequirements); topologyRegion != "" { + requiredRegion = topologyRegion + } + } + + if volumeData.Region != requiredRegion { + return nil, errRegionMismatch(volumeData.Region, requiredRegion) } log.V(4).Info("Content source volume", "volumeData", volumeData) @@ -179,7 +186,7 @@ func (cs *ControllerServer) getContentSourceVolume(ctx context.Context, contentS // attemptCreateLinodeVolume creates a Linode volume while ensuring idempotency. // It checks for existing volumes with the same label and either returns the existing // volume or creates a new one, optionally cloning from a source volume. -func (cs *ControllerServer) attemptCreateLinodeVolume(ctx context.Context, label string, sizeGB int, tags string, sourceVolume *linodevolumes.LinodeVolumeKey) (*linodego.Volume, error) { +func (cs *ControllerServer) attemptCreateLinodeVolume(ctx context.Context, label string, sizeGB int, tags string, sourceVolume *linodevolumes.LinodeVolumeKey, accessibilityRequirements *csi.TopologyRequirement) (*linodego.Volume, error) { log := logger.GetLogger(ctx) log.V(4).Info("Attempting to create Linode volume", "label", label, "sizeGB", sizeGB, "tags", tags) @@ -209,18 +216,43 @@ func (cs *ControllerServer) attemptCreateLinodeVolume(ctx context.Context, label return cs.cloneLinodeVolume(ctx, label, sourceVolume.VolumeID) } - return cs.createLinodeVolume(ctx, label, sizeGB, tags) + return cs.createLinodeVolume(ctx, label, sizeGB, tags, accessibilityRequirements) +} + +// Helper function to extract region from topology +func getRegionFromTopology(requirements *csi.TopologyRequirement) string { + topologies := requirements.GetPreferred() + if len(topologies) == 0 { + topologies = requirements.GetRequisite() + } + + if len(topologies) > 0 { + if value, ok := topologies[0].GetSegments()[VolumeTopologyRegion]; ok { + return value + } + } + + return "" } // createLinodeVolume creates a new Linode volume with the specified label, size, and tags. // It returns the created volume or an error if the creation fails. -func (cs *ControllerServer) createLinodeVolume(ctx context.Context, label string, sizeGB int, tags string) (*linodego.Volume, error) { +func (cs *ControllerServer) createLinodeVolume(ctx context.Context, label string, sizeGB int, tags string, accessibilityRequirements *csi.TopologyRequirement) (*linodego.Volume, error) { log := logger.GetLogger(ctx) log.V(4).Info("Creating Linode volume", "label", label, "sizeGB", sizeGB, "tags", tags) + // Get the region from req.AccessibilityRequirements if it exists. Fall back to the controller's metadata region if not specified. + region := cs.metadata.Region + if accessibilityRequirements != nil { + if topologyRegion := getRegionFromTopology(accessibilityRequirements); topologyRegion != "" { + log.V(4).Info("Using region from topology", "region", topologyRegion) + region = topologyRegion + } + } + // Prepare the volume creation request with region, label, and size. volumeReq := linodego.VolumeCreateOptions{ - Region: cs.metadata.Region, + Region: region, Label: label, Size: sizeGB, } @@ -394,7 +426,7 @@ func (cs *ControllerServer) prepareVolumeParams(ctx context.Context, req *csi.Cr // createVolumeContext creates a context map for the volume based on the request parameters. // If the volume is encrypted, it adds relevant encryption attributes to the context. -func (cs *ControllerServer) createVolumeContext(ctx context.Context, req *csi.CreateVolumeRequest) map[string]string { +func (cs *ControllerServer) createVolumeContext(ctx context.Context, req *csi.CreateVolumeRequest, vol *linodego.Volume) map[string]string { log := logger.GetLogger(ctx) log.V(4).Info("Entering createVolumeContext()", "req", req) defer log.V(4).Info("Exiting createVolumeContext()") @@ -408,18 +440,20 @@ func (cs *ControllerServer) createVolumeContext(ctx context.Context, req *csi.Cr volumeContext[LuksKeySizeAttribute] = req.GetParameters()[LuksKeySizeAttribute] } + volumeContext[VolumeTopologyRegion] = vol.Region + log.V(4).Info("Volume context created", "volumeContext", volumeContext) return volumeContext } // createAndWaitForVolume attempts to create a new volume and waits for it to become active. // It logs the process and handles any errors that occur during creation or waiting. -func (cs *ControllerServer) createAndWaitForVolume(ctx context.Context, name string, sizeGB int, tags string, sourceInfo *linodevolumes.LinodeVolumeKey) (*linodego.Volume, error) { +func (cs *ControllerServer) createAndWaitForVolume(ctx context.Context, name string, sizeGB int, tags string, sourceInfo *linodevolumes.LinodeVolumeKey, accessibilityRequirements *csi.TopologyRequirement) (*linodego.Volume, error) { log := logger.GetLogger(ctx) log.V(4).Info("Entering createAndWaitForVolume()", "name", name, "sizeGB", sizeGB, "tags", tags) defer log.V(4).Info("Exiting createAndWaitForVolume()") - vol, err := cs.attemptCreateLinodeVolume(ctx, name, sizeGB, tags, sourceInfo) + vol, err := cs.attemptCreateLinodeVolume(ctx, name, sizeGB, tags, sourceInfo, accessibilityRequirements) if err != nil { return nil, err } @@ -518,14 +552,20 @@ func (cs *ControllerServer) validateControllerPublishVolumeRequest(ctx context.C return linodeID, volumeID, nil } -// getAndValidateVolume retrieves the volume by its ID and checks if it is -// attached to the specified Linode instance. If the volume is found and -// already attached to the instance, it returns its device path. -// If the volume is not found or attached to a different instance, it -// returns an appropriate error. -func (cs *ControllerServer) getAndValidateVolume(ctx context.Context, volumeID, linodeID int) (string, error) { +// getAndValidateVolume retrieves the volume by its ID and run checks. +// +// It performs the following checks: +// 1. If the volume is found and already attached to the specified Linode instance, +// it returns the device path of the volume. +// 2. If the volume is not found, it returns an error indicating that the volume does not exist. +// 3. If the volume is attached to a different instance, it returns an error indicating +// that the volume is already attached elsewhere. +// +// Additionally, it checks if the volume and instance are in the same region based on +// the provided volume context. If they are not in the same region, it returns an internal error. +func (cs *ControllerServer) getAndValidateVolume(ctx context.Context, volumeID int, instance *linodego.Instance, volContext map[string]string) (string, error) { log := logger.GetLogger(ctx) - log.V(4).Info("Entering getAndValidateVolume()", "volumeID", volumeID, "linodeID", linodeID) + log.V(4).Info("Entering getAndValidateVolume()", "volumeID", volumeID, "linodeID", instance.ID) defer log.V(4).Info("Exiting getAndValidateVolume()") volume, err := cs.client.GetVolume(ctx, volumeID) @@ -536,14 +576,19 @@ func (cs *ControllerServer) getAndValidateVolume(ctx context.Context, volumeID, } if volume.LinodeID != nil { - if *volume.LinodeID == linodeID { + if *volume.LinodeID == instance.ID { log.V(4).Info("Volume already attached to instance", "volume_id", volume.ID, "node_id", *volume.LinodeID, "device_path", volume.FilesystemPath) return volume.FilesystemPath, nil } - return "", errVolumeAttached(volumeID, linodeID) + return "", errVolumeAttached(volumeID, instance.ID) + } + + // check if the volume and instance are in the same region + if instance.Region != volContext[VolumeTopologyRegion] { + return "", errRegionMismatch(volContext[VolumeTopologyRegion], instance.Region) } - log.V(4).Info("Volume validated and is not attached to instance", "volume_id", volume.ID, "node_id", linodeID) + log.V(4).Info("Volume validated and is not attached to instance", "volume_id", volume.ID, "node_id", instance.ID) return "", nil } diff --git a/internal/driver/controllerserver_helper_test.go b/internal/driver/controllerserver_helper_test.go index 04e54129..89cad9de 100644 --- a/internal/driver/controllerserver_helper_test.go +++ b/internal/driver/controllerserver_helper_test.go @@ -135,6 +135,9 @@ func TestPrepareCreateVolumeResponse(t *testing.T) { } func TestCreateVolumeContext(t *testing.T) { + vol := &linodego.Volume{ + Region: "us-east", + } tests := []struct { name string req *csi.CreateVolumeRequest @@ -146,7 +149,9 @@ func TestCreateVolumeContext(t *testing.T) { Name: "test-volume", Parameters: map[string]string{}, }, - expectedResult: map[string]string{}, + expectedResult: map[string]string{ + VolumeTopologyRegion: "us-east", + }, }, { name: "Encrypted volume with all parameters", @@ -163,6 +168,7 @@ func TestCreateVolumeContext(t *testing.T) { PublishInfoVolumeName: "encrypted-volume", LuksCipherAttribute: "aes-xts-plain64", LuksKeySizeAttribute: "512", + VolumeTopologyRegion: "us-east", }, }, // IMPORTANT:Now sure if we want this behavior, but it's what the code currently does. @@ -179,6 +185,7 @@ func TestCreateVolumeContext(t *testing.T) { PublishInfoVolumeName: "partial-encrypted-volume", LuksCipherAttribute: "", LuksKeySizeAttribute: "", + VolumeTopologyRegion: "us-east", }, }, { @@ -191,7 +198,9 @@ func TestCreateVolumeContext(t *testing.T) { LuksKeySizeAttribute: "512", }, }, - expectedResult: map[string]string{}, + expectedResult: map[string]string{ + VolumeTopologyRegion: "us-east", + }, }, } @@ -199,7 +208,7 @@ func TestCreateVolumeContext(t *testing.T) { t.Run(tt.name, func(t *testing.T) { cs := &ControllerServer{} ctx := context.Background() - result := cs.createVolumeContext(ctx, tt.req) + result := cs.createVolumeContext(ctx, tt.req, vol) if !reflect.DeepEqual(result, tt.expectedResult) { t.Errorf("createVolumeContext() = %v, want %v", result, tt.expectedResult) @@ -217,6 +226,16 @@ func TestCreateAndWaitForVolume(t *testing.T) { client: mockClient, } + topology := &csi.TopologyRequirement{ + Preferred: []*csi.Topology{ + { + Segments: map[string]string{ + VolumeTopologyRegion: "us-east", + }, + }, + }, + } + testCases := []struct { name string volumeName string @@ -302,7 +321,7 @@ func TestCreateAndWaitForVolume(t *testing.T) { t.Run(tc.name, func(t *testing.T) { tc.setupMocks() - volume, err := cs.createAndWaitForVolume(context.Background(), tc.volumeName, tc.sizeGB, tc.tags, tc.sourceInfo) + volume, err := cs.createAndWaitForVolume(context.Background(), tc.volumeName, tc.sizeGB, tc.tags, tc.sourceInfo, topology) if err != nil && !reflect.DeepEqual(tc.expectedError, err) { if tc.expectedError != nil { @@ -550,6 +569,9 @@ func TestValidateControllerPublishVolumeRequest(t *testing.T) { Mode: csi.VolumeCapability_AccessMode_SINGLE_NODE_WRITER, }, }, + VolumeContext: map[string]string{ + VolumeTopologyRegion: "us-east", + }, }, expectedNodeID: 12345, expectedVolID: 67890, @@ -654,10 +676,14 @@ func TestGetAndValidateVolume(t *testing.T) { client: mockClient, } + volContext := map[string]string{ + VolumeTopologyRegion: "us-east", + } + testCases := []struct { name string volumeID int - linodeID int + linode *linodego.Instance setupMocks func() expectedResult string expectedError error @@ -665,7 +691,9 @@ func TestGetAndValidateVolume(t *testing.T) { { name: "Volume found and attached to correct instance", volumeID: 123, - linodeID: 456, + linode: &linodego.Instance{ + ID: 456, + }, setupMocks: func() { mockClient.EXPECT().GetVolume(gomock.Any(), 123).Return(&linodego.Volume{ ID: 123, @@ -679,7 +707,10 @@ func TestGetAndValidateVolume(t *testing.T) { { name: "Volume found but not attached", volumeID: 123, - linodeID: 456, + linode: &linodego.Instance{ + ID: 456, + Region: "us-east", + }, setupMocks: func() { mockClient.EXPECT().GetVolume(gomock.Any(), 123).Return(&linodego.Volume{ ID: 123, @@ -692,7 +723,9 @@ func TestGetAndValidateVolume(t *testing.T) { { name: "Volume found but attached to different instance", volumeID: 123, - linodeID: 456, + linode: &linodego.Instance{ + ID: 456, + }, setupMocks: func() { mockClient.EXPECT().GetVolume(gomock.Any(), 123).Return(&linodego.Volume{ ID: 123, @@ -705,7 +738,9 @@ func TestGetAndValidateVolume(t *testing.T) { { name: "Volume not found", volumeID: 123, - linodeID: 456, + linode: &linodego.Instance{ + ID: 456, + }, setupMocks: func() { mockClient.EXPECT().GetVolume(gomock.Any(), 123).Return(nil, &linodego.Error{ Code: http.StatusNotFound, @@ -718,7 +753,9 @@ func TestGetAndValidateVolume(t *testing.T) { { name: "API error", volumeID: 123, - linodeID: 456, + linode: &linodego.Instance{ + ID: 456, + }, setupMocks: func() { mockClient.EXPECT().GetVolume(gomock.Any(), 123).Return(nil, errors.New("API error")) }, @@ -731,7 +768,7 @@ func TestGetAndValidateVolume(t *testing.T) { t.Run(tc.name, func(t *testing.T) { tc.setupMocks() - result, err := cs.getAndValidateVolume(context.Background(), tc.volumeID, tc.linodeID) + result, err := cs.getAndValidateVolume(context.Background(), tc.volumeID, tc.linode, volContext) if err != nil && !reflect.DeepEqual(tc.expectedError, err) { t.Errorf("expected error %v, got %v", tc.expectedError, err) @@ -802,7 +839,7 @@ func TestCheckAttachmentCapacity(t *testing.T) { } } -func TestAttemptGetContentSourceVolume(t *testing.T) { +func TestGetContentSourceVolume(t *testing.T) { ctrl := gomock.NewController(t) defer ctrl.Finish() @@ -816,22 +853,26 @@ func TestAttemptGetContentSourceVolume(t *testing.T) { testCases := []struct { name string - contentSource *csi.VolumeContentSource + req *csi.CreateVolumeRequest setupMocks func() expectedResult *linodevolumes.LinodeVolumeKey expectedError error }{ { - name: "Nil content source", - contentSource: nil, + name: "Nil content source", + req: &csi.CreateVolumeRequest{ + VolumeContentSource: nil, + }, setupMocks: func() {}, expectedResult: nil, expectedError: errNilSource, }, { name: "Invalid content source type", - contentSource: &csi.VolumeContentSource{ - Type: &csi.VolumeContentSource_Snapshot{}, + req: &csi.CreateVolumeRequest{ + VolumeContentSource: &csi.VolumeContentSource{ + Type: &csi.VolumeContentSource_Snapshot{}, + }, }, setupMocks: func() {}, expectedResult: nil, @@ -839,9 +880,11 @@ func TestAttemptGetContentSourceVolume(t *testing.T) { }, { name: "Nil volume", - contentSource: &csi.VolumeContentSource{ - Type: &csi.VolumeContentSource_Volume{ - Volume: nil, + req: &csi.CreateVolumeRequest{ + VolumeContentSource: &csi.VolumeContentSource{ + Type: &csi.VolumeContentSource_Volume{ + Volume: nil, + }, }, }, setupMocks: func() {}, @@ -850,10 +893,12 @@ func TestAttemptGetContentSourceVolume(t *testing.T) { }, { name: "Invalid volume ID", - contentSource: &csi.VolumeContentSource{ - Type: &csi.VolumeContentSource_Volume{ - Volume: &csi.VolumeContentSource_VolumeSource{ - VolumeId: "test-volume", + req: &csi.CreateVolumeRequest{ + VolumeContentSource: &csi.VolumeContentSource{ + Type: &csi.VolumeContentSource_Volume{ + Volume: &csi.VolumeContentSource_VolumeSource{ + VolumeId: "test-volume", + }, }, }, }, @@ -863,10 +908,12 @@ func TestAttemptGetContentSourceVolume(t *testing.T) { }, { name: "Valid content source, matching region", - contentSource: &csi.VolumeContentSource{ - Type: &csi.VolumeContentSource_Volume{ - Volume: &csi.VolumeContentSource_VolumeSource{ - VolumeId: "123-testvolume", + req: &csi.CreateVolumeRequest{ + VolumeContentSource: &csi.VolumeContentSource{ + Type: &csi.VolumeContentSource_Volume{ + Volume: &csi.VolumeContentSource_VolumeSource{ + VolumeId: "123-testvolume", + }, }, }, }, @@ -884,10 +931,12 @@ func TestAttemptGetContentSourceVolume(t *testing.T) { }, { name: "Valid content source, mismatched region", - contentSource: &csi.VolumeContentSource{ - Type: &csi.VolumeContentSource_Volume{ - Volume: &csi.VolumeContentSource_VolumeSource{ - VolumeId: "456-othervolume", + req: &csi.CreateVolumeRequest{ + VolumeContentSource: &csi.VolumeContentSource{ + Type: &csi.VolumeContentSource_Volume{ + Volume: &csi.VolumeContentSource_VolumeSource{ + VolumeId: "456-othervolume", + }, }, }, }, @@ -902,10 +951,12 @@ func TestAttemptGetContentSourceVolume(t *testing.T) { }, { name: "API error", - contentSource: &csi.VolumeContentSource{ - Type: &csi.VolumeContentSource_Volume{ - Volume: &csi.VolumeContentSource_VolumeSource{ - VolumeId: "789-errorvolume", + req: &csi.CreateVolumeRequest{ + VolumeContentSource: &csi.VolumeContentSource{ + Type: &csi.VolumeContentSource_Volume{ + Volume: &csi.VolumeContentSource_VolumeSource{ + VolumeId: "789-errorvolume", + }, }, }, }, @@ -921,7 +972,7 @@ func TestAttemptGetContentSourceVolume(t *testing.T) { t.Run(tc.name, func(t *testing.T) { tc.setupMocks() - result, err := cs.getContentSourceVolume(context.Background(), tc.contentSource) + result, err := cs.getContentSourceVolume(context.Background(), tc.req.GetVolumeContentSource(), tc.req.GetAccessibilityRequirements()) if err != nil && !reflect.DeepEqual(tc.expectedError, err) { t.Errorf("expected error %v, got %v", tc.expectedError, err) @@ -1071,3 +1122,91 @@ func TestGetInstance(t *testing.T) { }) } } + +func Test_getRegionFromTopology(t *testing.T) { + tests := []struct { + name string + requirements *csi.TopologyRequirement + want string + }{ + { + name: "Nil requirements", + requirements: nil, + want: "", + }, + { + name: "Empty preferred", + requirements: &csi.TopologyRequirement{ + Preferred: []*csi.Topology{}, + }, + want: "", + }, + { + name: "Single preferred topology with region", + requirements: &csi.TopologyRequirement{ + Preferred: []*csi.Topology{ + { + Segments: map[string]string{ + VolumeTopologyRegion: "us-east", + }, + }, + }, + }, + want: "us-east", + }, + { + name: "Multiple preferred topologies", + requirements: &csi.TopologyRequirement{ + Preferred: []*csi.Topology{ + { + Segments: map[string]string{ + VolumeTopologyRegion: "us-east", + }, + }, + { + Segments: map[string]string{ + VolumeTopologyRegion: "us-west", + }, + }, + }, + }, + want: "us-east", + }, + { + name: "Preferred topology without region", + requirements: &csi.TopologyRequirement{ + Preferred: []*csi.Topology{ + { + Segments: map[string]string{ + "some-key": "some-value", + }, + }, + }, + }, + want: "", + }, + { + name: "Empty preferred, non-empty requisite", + requirements: &csi.TopologyRequirement{ + Preferred: []*csi.Topology{}, + Requisite: []*csi.Topology{ + { + Segments: map[string]string{ + VolumeTopologyRegion: "eu-west", + }, + }, + }, + }, + want: "eu-west", + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + got := getRegionFromTopology(tt.requirements) + if got != tt.want { + t.Errorf("getRegionFromTopology() = %v, want %v", got, tt.want) + } + }) + } +} diff --git a/internal/driver/controllerserver_test.go b/internal/driver/controllerserver_test.go index e19e2fa5..100654d9 100644 --- a/internal/driver/controllerserver_test.go +++ b/internal/driver/controllerserver_test.go @@ -200,6 +200,9 @@ func TestControllerPublishVolume(t *testing.T) { Mode: csi.VolumeCapability_AccessMode_SINGLE_NODE_WRITER, }, }, + VolumeContext: map[string]string{ + VolumeTopologyRegion: "us-east", + }, Readonly: false, }, resp: &csi.ControllerPublishVolumeResponse{ @@ -208,11 +211,11 @@ func TestControllerPublishVolume(t *testing.T) { }, }, expectLinodeClientCalls: func(m *mocks.MockLinodeClient) { + m.EXPECT().GetInstance(gomock.Any(), gomock.Any()).Return(&linodego.Instance{ID: 1003, Specs: &linodego.InstanceSpec{Memory: 16 << 10}}, nil) m.EXPECT().WaitForVolumeLinodeID(gomock.Any(), 630706045, gomock.Any(), gomock.Any()).Return(&linodego.Volume{ID: 1001, LinodeID: createLinodeID(1003), Size: 10, Status: linodego.VolumeActive}, nil) m.EXPECT().AttachVolume(gomock.Any(), 630706045, gomock.Any()).Return(&linodego.Volume{ID: 1001, LinodeID: createLinodeID(1003), Size: 10, Status: linodego.VolumeActive}, nil) - m.EXPECT().ListInstanceVolumes(gomock.Any(), 1001, gomock.Any()).Return([]linodego.Volume{{ID: 1001, LinodeID: createLinodeID(1003), Size: 10, Status: linodego.VolumeActive}}, nil) - m.EXPECT().ListInstanceDisks(gomock.Any(), 1001, gomock.Any()).Return([]linodego.InstanceDisk{}, nil) - m.EXPECT().GetInstance(gomock.Any(), gomock.Any()).Return(&linodego.Instance{ID: 1001, Specs: &linodego.InstanceSpec{Memory: 16 << 10}}, nil) + m.EXPECT().ListInstanceVolumes(gomock.Any(), 1003, gomock.Any()).Return([]linodego.Volume{{ID: 1001, LinodeID: createLinodeID(1003), Size: 10, Status: linodego.VolumeActive}}, nil) + m.EXPECT().ListInstanceDisks(gomock.Any(), 1003, gomock.Any()).Return([]linodego.InstanceDisk{}, nil) m.EXPECT().GetVolume(gomock.Any(), gomock.Any()).Return(&linodego.Volume{ID: 1001, LinodeID: createLinodeID(1003), Size: 10, Status: linodego.VolumeActive}, nil) }, expectedError: nil, diff --git a/internal/driver/deploy/releases/linode-blockstorage-csi-driver.yaml b/internal/driver/deploy/releases/linode-blockstorage-csi-driver.yaml index 0f05a324..ee871a38 100644 --- a/internal/driver/deploy/releases/linode-blockstorage-csi-driver.yaml +++ b/internal/driver/deploy/releases/linode-blockstorage-csi-driver.yaml @@ -17,6 +17,24 @@ metadata: provisioner: linodebs.csi.linode.com reclaimPolicy: Retain --- +apiVersion: storage.k8s.io/v1 +kind: StorageClass +metadata: + name: linode-block-storage-topology-aware-retain + namespace: kube-system +provisioner: linodebs.csi.linode.com +reclaimPolicy: Retain +volumeBindingMode: WaitForFirstConsumer +--- +apiVersion: storage.k8s.io/v1 +kind: StorageClass +metadata: + name: linode-block-storage-topology-aware + namespace: kube-system +provisioner: linodebs.csi.linode.com +reclaimPolicy: Delete +volumeBindingMode: WaitForFirstConsumer +--- apiVersion: v1 kind: ServiceAccount metadata: @@ -313,6 +331,7 @@ spec: - --volume-name-prefix=pvc - --volume-name-uuid-length=16 - --csi-address=$(ADDRESS) + - --feature-gates=Topology=true - --v=2 env: - name: ADDRESS diff --git a/internal/driver/examples/kubernetes/topology-aware.yaml b/internal/driver/examples/kubernetes/topology-aware.yaml new file mode 100644 index 00000000..b8eed2a5 --- /dev/null +++ b/internal/driver/examples/kubernetes/topology-aware.yaml @@ -0,0 +1,46 @@ +# This StorageClass is for topology-aware provisioning based on the pod region +allowVolumeExpansion: true +apiVersion: storage.k8s.io/v1 +kind: StorageClass +metadata: + name: linode-block-storage-wait-for-consumer +provisioner: linodebs.csi.linode.com +reclaimPolicy: Delete +volumeBindingMode: WaitForFirstConsumer +--- +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: pvc-filesystem +spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 10Gi + storageClassName: linode-block-storage-wait-for-consumer +--- +apiVersion: v1 +kind: Pod +metadata: + name: pod-topology-aware +spec: + nodeSelector: + topology.linode.com/region: us-ord + tolerations: + - key: "node-role.kubernetes.io/control-plane" + operator: "Exists" + effect: "NoSchedule" + containers: + - name: pod-topology-aware + image: ubuntu + command: + - sleep + - "1000000" + volumeMounts: + - mountPath: /data + name: csi-volume + volumes: + - name: csi-volume + persistentVolumeClaim: + claimName: pvc-filesystem