How we brought a universal, cluster-wide storage solution to MicroK8s.

• Why is storage in Kubernetes hard?
• Introducing Mayastor
• Sounds interesting! How do I try this out?
  • Deploy Mayastor
  • Deploy a test workload
• Conclusion and future steps

Why is storage in Kubernetes hard?

Kubernetes is well-known as an open-source system for automating deployment, scaling and management of containerised applications. As such, it uses concepts such as Pods and Deployments to abstract away details regarding the underlying compute, networking and storage infrastructure.

Storage is abstracted as PersistentVolumes (a volume that is provisioned in the underlying infrastructure) and PersistentVolumeClaims (a claim for a pod that uses a persistent volume) resources. Typically, running a Kubernetes cluster also involves running a CSI provisioner, which watches for PersistentVolumeClaims and automatically provisions the requested volumes.

There are a large number of CSI drivers available. For example, if running Kubernetes on top of AWS, one would most likely use the aws-ebs-csi-driver. If running Kubernetes on-premises or in an OpenStack cloud, cinder-csi-driver would be more appropriate. If running on-premises and a Ceph cluster is available, ceph-csi could also be used.

While this solves the storage problems for Kubernetes users, it moves the hurdle of managing everything to the administrators of the Kubernetes cluster, as well the Kubernetes distribution.

So, the question is, how can storage be made simpler for the administrator of the Kubernetes cluster?

Introducing Mayastor

Mayastor is currently under-development as a sub project of the Open Source CNCF project OpenEBS.

Mayastor can be deployed on any Kubernetes cluster and provide Kubernetes-native storage. Mayastor can use existing block devices on the Kubernetes cluster nodes (if available), as well as image files.

Further, the Mayastor data plane supports replication. You can replicate your volume data across multiple pools, to ensure redundancy in case of a single node failure.

In general, Mayastor is a very interesting solution for Kubernetes-native, fast, redundant storage that works universally across any Kubernetes cluster.

Sounds interesting! How do I try this out?

Deploy Mayastor

NOTE: For the latest instructions, refer to the mayastor addon page.

The mayastor addon will be available with MicroK8s 1.24. You can already try it out by installing MicroK8s from latest/edge:

1. Enable nvme_tcp and HugePages:

   sudo apt-get install linux-modules-extra-$(uname -r)
   sudo modprobe nvme_tcp
   sudo sysctl vm.nr_hugepages=1024
   

2. Install MicroK8s from latest/edge:

   sudo snap install microk8s --classic --channel latest/edge
   

3. Enable Mayastor:

   sudo microk8s enable core/mayastor
   

Deploy a test workload

Now that mayastor is deployed, it is time to test it using an example pod:

# pod-with-pvc.yaml
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pvc
spec:
  storageClassName: mayastor
  accessModes: [ReadWriteOnce]
  resources: { requests: { storage: 5Gi } }
---
apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  volumes:
    - name: pvc
      persistentVolumeClaim:
        claimName: my-pvc
  containers:
    - name: nginx
      image: nginx
      ports:
        - containerPort: 80
      volumeMounts:
        - name: pvc
          mountPath: /usr/share/nginx/html

Create the pod with:

sudo microk8s.kubectl create -f pod-with-pvc.yaml

Then, we can check to see that Mayastor has created our persistent volume, and the pod has come up:

sudo microk8s.kubectl get pod,pvc

The output should look like this:

NAME        READY   STATUS    RESTARTS   AGE
pod/nginx   1/1     Running   0          4m

NAME                           STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
persistentvolumeclaim/my-pvc   Bound    pvc-e280b734-3224-4af3-af0b-e7ad3c4e6d79   5Gi        RWO            mayastor       4m

Conclusion and future steps

The new mayastor addon in MicroK8s is a very simple way to provision fast, replicated and redundant (though not highly-available) persistent volumes in Kubernetes.

The mayastor addon is considered beta as of MicroK8s 1.24 release. You can already try it by installing from the latest/edge track.

Future steps include:

• Scale and stress testing.
• Graduate addon from beta to stable status.
• Refine the mayastor addon. Right now it automatically provisions image-file backed pools. but we want to make the addon easier to configure for MicroK8s clusters with designated block devices for storage.
• Compare Mayastor with Rook (Ceph) and other Kubernetes native storage solutions.
• Document and test backup and restore of mayastor volumes.
• Support arm64 architectures. This is pending support in mayastor