Portworx vSphere storage operations guide

Installation

During installation, user provides following parameters which are used to create disks (VMDKs) for the Portworx storage pools. The sum total of disks will be the starting storage capacity of your Portworx cluster.

Prefix for names of ESXi Datastore(s) or Datastore cluster(s)
Specification of type and size of disks to create.
Access details for vCenter server

The Portworx VMware installation covers this in detail.

Monitoring

Listing disks created by Portworx

Portworx ships with the pxctl CLI out of the box that users can use to perform management operations.

Where are Portworx VMDKs located?

Portworx creates disks in a folder called osd-provisioned-disks in the ESXi datastore. The names of the VMDK created by Portworx will have a prefix PX-DO-NOT-DELETE-.

The Cloud Drives (ASG) using pxctl CLI command is particularly useful in getting more insight into the disks provisioned by Portworx in a vSphere environment. Follow command gives details on all VMware disks (VMDKs) created by Portworx in your cluster.

pxctl clouddrive list

Cloud Drives Summary
        Number of nodes in the cluster:  3
        Number of drive sets in use:  3
        List of storage nodes:  [4267f9b3-5785-463e-8358-62ab1195b839 731ec7dc-e581-4dd2-b5dd-ad3ef62255f5 cb9762d6-29e8-4bd1-8228-3cb9c9a42ff8]
        List of storage less nodes:  []

Drive Set List
        NodeIndex       NodeID                                  InstanceID                              Zone    Drive IDs
        0               4267f9b3-5785-463e-8358-62ab1195b839    422387fd-00c7-58b0-f275-63bad5b6e6d8    AZ1     [datastore-589] 260f0d5d-207e-2372-3d57-ac1f6b204d08/PX-DO-NOT-DELETE-ca2a2eed-78cb-4ebd-a173-e094ba6210ad.vmdk(data)
        1               731ec7dc-e581-4dd2-b5dd-ad3ef62255f5    4223719d-c9c7-c8a9-0fb9-25f4f2a66fd4    AZ1     [datastore-589] 260f0d5d-207e-2372-3d57-ac1f6b204d08/PX-DO-NOT-DELETE-b54e705b-d14a-4069-8277-3c82ece5131b.vmdk(data)
        2               cb9762d6-29e8-4bd1-8228-3cb9c9a42ff8    4223fa3d-5eca-00a1-0744-8295a8c69374    AZ1     [datastore-589] 260f0d5d-207e-2372-3d57-ac1f6b204d08/PX-DO-NOT-DELETE-2eb48140-d5e1-4a4c-8e6a-99c7a8d697fd.vmdk(data)

Where do I run pxctl?

You can run pxctl by ssh’ing to any worker node in cluster or by kubectl exec on any Portworx pod.

Prometheus and Grafana

Portworx exposes many useful metrics out of the box. These metrics are listed on this page.

Portworx also alerts users on a predefined set of alerts that are listed on the Portworx Alerts page.

The Prometheus and Grafana on Kubernetes page describes how to use Prometheus and Grafana to monitor Portworx in a Kubernetes cluster and extract the metrics and alerts.

Capacity Management

As storage usage increases, you will need to increase the backing storage capacity for Portworx. Increasing capacity falls into 2 primary buckets:

a. Increase storage capacity for a single node

To expand the storage capacity of a single node, you have 2 options

1. Add new disk to a node

Identify the Portworx node where you want to add the disk.
Drain all apps using Portworx volumes from this node. You can either use kubectl drain or kubectl cordon and kubectl delete pod for this.
SSH to the node.
Enter Portworx maintenance mode.
```
pxctl service maintenance --enter
```
Wait for about 2 minutes until pxctl status says PX is in maintenance Mode.
List Portworx pools and save the output.
```
pxctl service pool show
```
Use pxctl to add a new disk. This command will provision a new VMDK as per the given spec. Below example creates a 20GB VMDK of type eagerzeroedthick. Change the size as per your needs. It is recommended to use a size that’s same as the current disks in your storage pools. This will allow us to expand the existing storage pool.
```
pxctl service drive add --spec size=20,type=eagerzeroedthick
```
```
Drive add done: Storage rebalance is in progress
```
List Portworx pools again and you will see one of the pools now has a new drive added.
```
pxctl service pool show
```
Resize the pool to account for this newly added disk. <pool-id> here is the ID of the pool where you see the newly added disk in the previous step.
```
pxctl service pool update --resize <pool-id>
```
Exit maintenance mode.
```
pxctl service maintenance --exit
```
Wait for about 2 minutes until pxctl status says PX is operational.
List Portworx pools again and you will now see the pool has the expanded size.
```
pxctl service pool show
```
Uncordon the node from Kubernetes.

2. Resize existing disk on a node

Portworx 2.2 will have an improved resize workflow that will automate below steps.

Use pxctl to list disks created by Portworx as shown previously on this page
Identify the VMDK to resize from this list.
Identify the Portworx node where this VMDK is being used. This is shown in the NodeID column.
Inspect the Portworx node where the VMDK is being used.
```
pxctl cluster inspect <node-id>
```
Using vSphere web client, expand the VMDK on the VM for the above node.
Drain all apps using Portworx volumes from this node. You can either use kubectl drain or kubectl cordon and kubectl delete pod for this.
SSH to the Portworx node identified in step 3.
Enter Portworx maintenance mode.
```
pxctl service maintenance --enter
```
Wait for about 2 minutes until pxctl status says PX is in maintenance Mode.
List Portworx pools and find the pool that’s affected by the resized vol
```
pxctl service pool show
```
Resize the pool to account for the expanded VMDK.
```
pxctl service pool update --resize <pool-id>
```
Exit maintenance mode.
```
pxctl service maintenance --exit
```
Wait for about 2 minutes until pxctl status says PX is operational.
List Portworx pools again and you will now see the pool has the expanded size.
```
pxctl service pool show
```
Uncordon the node from Kubernetes.

b. Increase storage of the backing ESXi datastore(s)/datastore cluster(s)

When do I need this? You would need this when the existing datastores being used by Portworx are filling up and there is no space left to increase storage capacity of individual nodes.

This can be done in one of 2 ways. Once the datastore(s) or datastore cluster(s) are expanded, Portworx can use the increased capacity to provision new VMDKs or resize existing VMDKs.

1. Resize existing ESXi datatores

You can expand the drives backing the datastore in your storage array and then increase datastore capacity from the vSphere web client.

2. Add new datastore(s) to the Datastore cluster

If you provided the prefix of a datastore cluster names to Portworx during installation, you can dynamically add new datastores to the datastore cluster. This will increase the capacity of the datastore cluster.

Kubernetes secret for VMware

Portworx VMware shared architecture