Volume Placement Strategies
When creating PersistentVolumeClaims (PVCs) in the cluster, Portworx allows you to specify placement strategies for the volume. Below are some of the use cases that can be satisfied by placement strategies:
- Distribute volumes across failure domains (This is the default Portworx behavior).
- Specify failure domains to include (affinity) or exclude (anti-affinity) when placing volume replicas.
- Specify storage pools to include (affinity) or exclude (anti-affinity) based on
- Pool Cos (Class of Service): Select between high, medium or low pools
- Pool media type: Use pools with SSD drives
- Pool labels: Select pools based on custom user specified labels
- Spread different replicas of volumes across different pools. E.g place first replica is SSD pools, rest of them in other pools
- Specify affinity or anti-affinity between different volumes matched by labels. The labels here are labels on the PVC metadata.
- Specify affinity or anti-affinity between multiple replicas of the same volume
The VolumePlacementStrategy CRD
Portworx provides a CustomResouceDefinition (CRD) called VolumePlacementStrategy
. The specification for this CRD is composed of 4 main sections:
replicaAffinity
The replicaAffinity
section allows you to specify a rule which creates an affinity for replicas within a volume. You should use this to place one or more replicas of a volume on the same failure domain, on multiple failure domains, nodes, or storage pools that match the given labels.
Schema
Field | Description | Optional? | Default |
---|---|---|---|
enforcement | Specifies if the given rule is required (hard) or preferred (soft) | Yes | required |
affectedReplicas | Number indicating the number of volume replicas that are affected by this rule. | Yes | 0 (Interpreted as all volume replicas) |
topologyKey | Key for the node label that the system uses to denote a topology domain. The key can be for any node label that is present on the Kubernetes node. Using topologyKey requires nodes to be consistently labelled, i.e. every node in the cluster must have an appropriate label matching topologyKey. If some or all nodes are missing the specified topologyKey label, it can lead to unintended behavior. |
Yes* * required if matchExpressions is empty |
empty |
matchExpressions | matchExpressions is a list of label selector requirements. The requirements are ANDed. The labels provided here are matched against the following:
|
Yes* * required if topologyKey is empty |
empty |
Example uses cases
By default, Portworx automatically adds the following labels to each of its storage pools. These can be used for replica affinity anti-affinity rules to target replicas on a specific storage pool type:
iopriority
medium
replicaAntiAffinity
The replicaAntiAffinity
section allows you to specify a rule that creates an anti-affinity for replicas within a volume. You should use this to exclude failure domains, nodes or storage pools that match the given labels for one or more replicas of a volume.
Schema
Field | Description | Optional? | Default |
---|---|---|---|
enforcement | Specifies if the given rule is required (hard) or preferred (soft) | Yes | required |
affectedReplicas | Number indicating the number of volume replicas that are affected by this rule. | Yes | 0 (Interpreted as all volume replicas) |
topologyKey | Key for the node label that the system uses to denote a topology domain. The key can be for any node label that is present on the Kubernetes node. Using topologyKey requires nodes to be consistently labelled, i.e. every node in the cluster must have an appropriate label matching topologyKey. If some or all nodes are missing the specified topologyKey label, it can lead to unintended behavior. | Yes* * required if matchExpressions is empty | empty |
matchExpressions | matchExpressions is a list of label selector requirements. The requirements are ANDed. The labels provided here are matched against the following:
|
Yes* * required if topologyKey is empty |
empty |
Example uses cases
Anti-affinity to not use SATA pools
volumeAffinity
The volumeAffinity
section allows specifying rules that create affinity between 2 or more volumes that match the given labels. This should be used when you want to establish an affinity relationship between different volumes.
Schema
Field | Description | Optional? | Default |
---|---|---|---|
enforcement | Specifies if the given rule is required (hard) or preferred (soft) | Yes | required |
topologyKey | Key for the node label that the system uses to denote a topology domain. The key can be for any node label that is present on the Kubernetes node. Using topologyKey requires nodes to be consistently labelled, i.e. every node in the cluster must have an appropriate label matching topologyKey. If some or all nodes are missing the specified topologyKey label, it can lead to unintended behavior. |
Yes | empty |
matchExpressions | matchExpressions is a list of label selector requirements. The requirements are ANDed. The labels provided here are matched against the following:
|
No | empty |
Example uses cases
volumeAntiAffinity
The volumeAntiAffinity
section allows you to specify rules that create anti affinity between 2 or more volumes that match the given labels. This should be used when you want to establish an anti affinity (repel) relationship between different volumes
Schema
Same as Schema for volumeAffinity.
Example uses cases
Do not collocate with other cassandra volumes
How to use
Pre-requisities
- Portworx version: 2.1.2 and above
CRD version: Make sure the VolumePlacementStrategy CRD (CustomResourceDefinition) is at the right version:
- Unregister existing CustomResourceDefinition
kubectl delete crd volumeplacementstrategies.portworx.io
- Register the following CustomResourceDefinition
apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: name: volumeplacementstrategies.portworx.io spec: group: portworx.io version: v1beta2 scope: Cluster names: plural: volumeplacementstrategies singular: volumeplacementstrategy kind: VolumePlacementStrategy shortNames: - vps - vp
End user workflow
- User applies one or more VolumePlacementStrategy specs as per their requirements
- User creates a StorageClass that references the VolumePlacementStrategy. (Example)
- Users creates a PVC from the StorageClass (Example)