Volume Placement Strategies


When creating PersistentVolumeClaims (PVCs) in the cluster, Portworx allows you to specify placement strategies for the volume. Below are some of the use cases that can be satisfied by placement strategies:

  • Distribute volumes across failure domains (This is the default Portworx behavior).
  • Specify failure domains to include (affinity) or exclude (anti-affinity) when placing volume replicas.
  • Specify storage pools to include (affinity) or exclude (anti-affinity) based on
    • Pool Cos (Class of Service): Select between high, medium or low pools
    • Pool media type: Use pools with SSD drives
    • Pool labels: Select pools based on custom user specified labels
  • Spread different replicas of volumes across different pools. E.g place first replica is SSD pools, rest of them in other pools
  • Specify affinity or anti-affinity between different volumes matched by labels. The labels here are labels on the PVC metadata.
  • Specify affinity or anti-affinity between multiple replicas of the same volume

The VolumePlacementStrategy CRD

Portworx provides a CustomResouceDefinition (CRD) called VolumePlacementStrategy. The specification for this CRD is composed of 4 main sections:

replicaAffinity

The replicaAffinity section allows you to specify a rule which creates an affinity for replicas within a volume. You should use this to place one or more replicas of a volume on the same failure domain, on multiple failure domains, nodes, or storage pools that match the given labels.

Schema
Field Description Optional? Default
enforcement Specifies if the given rule is required (hard) or preferred (soft) Yes required
affectedReplicas Number indicating the number of volume replicas that are affected by this rule. Yes 0 (Interpreted as all volume replicas)
topologyKey Key for the node label that the system uses to denote a topology domain. The key can be for any node label that is present on the Kubernetes node.

Using topologyKey requires nodes to be consistently labelled, i.e. every node in the cluster must have an appropriate label matching topologyKey. If some or all nodes are missing the specified topologyKey label, it can lead to unintended behavior.
Yes*

* required if matchExpressions is empty
empty
matchExpressions matchExpressions is a list of label selector requirements. The requirements are ANDed.

The labels provided here are matched against the following:
  • Kubernetes node labels
  • PVC labels
  • Portworx storage pool labels
Yes*

* required if topologyKey is empty
empty
Example uses cases

By default, Portworx automatically adds the following labels to each of its storage pools. These can be used for replica affinity anti-affinity rules to target replicas on a specific storage pool type:

  • iopriority
  • medium

replicaAntiAffinity

The replicaAntiAffinity section allows you to specify a rule that creates an anti-affinity for replicas within a volume. You should use this to exclude failure domains, nodes or storage pools that match the given labels for one or more replicas of a volume.

Schema
Field Description Optional? Default
enforcement Specifies if the given rule is required (hard) or preferred (soft) Yes required
affectedReplicas Number indicating the number of volume replicas that are affected by this rule. Yes 0 (Interpreted as all volume replicas)
topologyKey Key for the node label that the system uses to denote a topology domain. The key can be for any node label that is present on the Kubernetes node. Using topologyKey requires nodes to be consistently labelled, i.e. every node in the cluster must have an appropriate label matching topologyKey. If some or all nodes are missing the specified topologyKey label, it can lead to unintended behavior. Yes* * required if matchExpressions is empty empty
matchExpressions matchExpressions is a list of label selector requirements. The requirements are ANDed.

The labels provided here are matched against the following:
  • Kubernetes node labels
  • PVC labels
  • Portworx storage pool labels
Yes*

* required if topologyKey is empty
empty
Example uses cases

Anti-affinity to not use SATA pools

volumeAffinity

The volumeAffinity section allows specifying rules that create affinity between 2 or more volumes that match the given labels. This should be used when you want to establish an affinity relationship between different volumes.

Schema
Field Description Optional? Default
enforcement Specifies if the given rule is required (hard) or preferred (soft) Yes required
topologyKey Key for the node label that the system uses to denote a topology domain. The key can be for any node label that is present on the Kubernetes node.
Using topologyKey requires nodes to be consistently labelled, i.e. every node in the cluster must have an appropriate label matching topologyKey. If some or all nodes are missing the specified topologyKey label, it can lead to unintended behavior.
Yes empty
matchExpressions matchExpressions is a list of label selector requirements. The requirements are ANDed.

The labels provided here are matched against the following:
  • Kubernetes node labels
  • PVC labels
  • Portworx
  • storage pool labels
No empty
Example uses cases

volumeAntiAffinity

The volumeAntiAffinity section allows you to specify rules that create anti affinity between 2 or more volumes that match the given labels. This should be used when you want to establish an anti affinity (repel) relationship between different volumes

Schema

Same as Schema for volumeAffinity.

Example uses cases

Do not collocate with other cassandra volumes

How to use

Pre-requisities

  1. Portworx version: 2.1.2 and above
  2. CRD version: Make sure the VolumePlacementStrategy CRD (CustomResourceDefinition) is at the right version:

    • Unregister existing CustomResourceDefinition
    kubectl delete crd volumeplacementstrategies.portworx.io
    • Register the following CustomResourceDefinition
    apiVersion: apiextensions.k8s.io/v1beta1
    kind: CustomResourceDefinition
    metadata:
      name: volumeplacementstrategies.portworx.io
    spec:
      group: portworx.io
      version: v1beta2
      scope: Cluster
      names:
        plural: volumeplacementstrategies
        singular: volumeplacementstrategy
        kind: VolumePlacementStrategy
        shortNames:
        - vps
        - vp

End user workflow

  1. User applies one or more VolumePlacementStrategy specs as per their requirements
  2. User creates a StorageClass that references the VolumePlacementStrategy. (Example)
  3. Users creates a PVC from the StorageClass (Example)


Last edited: Wednesday, Oct 2, 2019