OpenShift Disaster Recovery using Stretch Cluster

Prior to installing ODF the nodes used for storage and the node used to host the arbiter function must be specifically labeled to define their function. In our case we will use the label datacenter1 for one location with storage nodes and datacenter2 for the other location. The zone that will have the arbiter function in the case of a site outage will use the arbiter label. These labels are arbitrary but they do need to be unique for the 3 locations.

For example, you can label the nodes and in Figure 1 as follows (including master nodes not shown in diagram):

To apply the labels to the node using oc CLI do the following:

To validate the labels run the following commands using the example labels for the 3 zones:

The OpenShift DR stretch cluster topology zone labels are now applied to the appropriate OCP nodes to define the three locations. Next step is installing the storage operators from OCP OperatorHub.

Once you are logged in, navigate to the Operators → OperatorHub menu.

Now type local storage in the Filter by keyword…​ box.

Select Local Storage and then select Install.

On the next screen make sure the settings are as shown in this figure.

Click Install.

Verify the Local Storage Operator deployment is successful.

Start with creating the openshift-storage namespace.

You must add the monitoring label to this namespace. This is required to get prometheus metrics and alerts for the OCP storage dashboards. To label the openshift-storage namespace use the following command:

Navigate to the Operators → OperatorHub menu again.

Now type openshift container storage in the Filter by keyword…​ box.

Select OpenShift Data Foundation Operator and then select Install.

On the next screen make sure the settings are as shown in this figure.

Click Install.

Now you can go back to your terminal window to check the progress of the installation. Verify the operator is deployed successfully.

Navigate to the Operators → Installed Operators menu.

Click on Storage Cluster as indicated in the graphic above.

Click on Create Storage Cluster on the far right side.

Select the Internal - Attached Devices deployment option.

Provide storage cluster details.

Click Next at the bottom of the screen.

Enter the desired configuration for your Local Storage Operator and click Next.

Click Yes when asked to confirm the storage class creation.

Next check the Enable arbiter checkbox. Select the correct topology zone that is to receive the Arbiter Monitor. The zone label is arbiter in this case.

Select the LSO storage class you created as illustrated in the screen capture. Then click Next.

When asked if you want to enable encryption just click Next again.

Review parameters and create the cluster.

Click Create at the bottom of the Review storage cluster window.

Currently the file-upoader Deployment is not zone aware and could schedule all of the Pods in the same zone. If this happened and there was a site outage then the application would be unavailable.

Search for containers and repeat the search a few times until your output is similar. There is currently no pod placement rules in the default Deployment file-uploader.

Currently the deployment is not configured to be zone aware. The Deployment needs to be modified to use the topology zone labels as shown below. Edit the deployment and add the new lines below between the start and end point.

Now scale the deployment to zero Pods. and then back to 4 Pods. This is needed because the deployment changed in terms of Pod placement.

And then back to 4 Pods.

Validate now that the 4 Pods are spread across the 4 nodes in datacenter1 and datacenter2 zones.

Now let’s use the file uploader web application using your browser to upload new files.

First, find the Route that has been created:

This will return a route similar to this one.

Point your browser to the web application using your route above. Your route will be different.

The web app simply lists all uploaded files and offers the ability to upload new ones as well as download the existing data. Right now there is nothing.

Select an arbitrary file from your local machine and upload it to the app.

Once done click List uploaded files to see the list of all currently uploaded files.

For purposes of this section we will consider a zone failure to be a failure where all OCP nodes, masters and workers, in a zone are no longer communicating with the resources in the second data zone (e.g., nodes powered down). If communication between data zones is partially still working (intermittent down/up), steps should be taken by cluster, storage, network admins to sever the communication path between the data zones for disaster recovery to succeed.

Applications that are deployed with topologyKey: topology.kubernetes.io/zone, have one or more replicas scheduled in each data zone, and are using shared storage (i.e., RWX cephfs volume) will recover on the active zone within 30-60 seconds for new connections. The short pause is for HAProxy to refresh connections if a router pod is now offline in the failed data zone. An example of this type of application is detailed in the Install Zone Aware Sample Application section.

Applications that are using topologyKey: kubernetes.io/hostname or no topology configuration whatsoever, have no protection against all of the application replicas being in the same zone.

If this happens, all replicas in the same zone, and there is a zone failure the application using RWX storage will take 6-8 minutes to recover on the active zone. This pause is for the OCP nodes in the failed zone to become NotReady (60 seconds) and then for the default pod eviction timeout to expire (300 seconds).

Applications that use RWO storage (ReadWriteOnce) have a know behavior described in this Kubernetes issue. Because of this issue, if there is a data zone failure any application Pods in that zone mounting RWO volumes (i.e., cephrbd) are stuck in Terminating after 6-8 minutes and will not be recreated on the active zone without manual intervention.

To get the Terminating Pods to recreate on the active zone do one of these two actions:

Once one of these two actions are done the application Pod should recreate on the active zone and mount its RWO storage.

Pods that are part of a stateful set have a similar issue as Pods mounting RWO volumes. Reference Kubernetes resource StatefulSet considerations for more information. To get the Pods part of a StatefulSet to recreate on the active zone after 6-8 minutes, the Pod needs to be force deleted with the same requirements (i.e., OCP node powered off or communication severed) as Pods with RWO volumes.

As was described for our sample application the OCP image-registry can use topologySpreadConstraints in the Deployment to spread Pods between zones equally and then between nodes in a zone.

The configs.imageregistry.operator.openshift.io must be edited to use shared storage (i.e., RWX cephfs volume) to scale the number of image-registry Pods.

Now the configs.imageregistry.operator.openshift.io must be configured to be Unmanaged so the Deployment changes will not reconcile immediately back to default configuration.

Once this is done, delete everything under affinity: between spec and containers in the image-registry deployment and replace with the following:

To schedule the Pods equally between zones you will need to scale the replicas for the image-registry Deployment to 1 and then back to the desired number of Pods. Verify the Pods are equally scheduled between zones.

The prometheus and alertmanager Pods are both created with a StatefulSet.

Because of the reconciling back to default configuration (shown for prometheus) that uses host-based placement, this configuration cannot be deleted from the StatefulSet and replaced with topologySpreadConstraints as was done for the image-registry Deployment. The same podAntiAffinity is used in the alertmanager Statefulset which does not guarantee that the Pods will be scheduled in both zones.

Therefore a different solution is needed to make monitoring resources highly available across zones. The method is to use node labeling and create a new (or modify existing) ConfigMap named cluster-monitoring-config for placement via a nodeSelector configuration.

First add the labels to the nodes in the different zones.

Now create and then modify the ConfigMap cluster-monitoring-config.

Patch the ConfigMap with the labels for prometheus and alertmanager Pod placement.

Scale the replicas for both alertmanager and prometheus StatefulSet to 1 and then back to 3 and 2 respectively. Verify the Pods scheduled on the OCP nodes with the new labels.

The OCP routers have a similar issue as the monitoring StatefulSets for prometheus and alertmanager. In the case of the routers it is the Deployment that reconciles back to default host-based placement.

Given the Deployment can not be edited (and have the edits stay) the same method can be used for Pod placement as was done for prometheus and alertmanager Pods. Label the nodes in each zone and then patch the IngressController default to use the label as a nodeSelector.

First add the labels to the nodes in the different zones.

Now patch the IngressController default with the labels for the router Pod placement.

Scale the replicas for the router-default Deployment to 1 and then back to 2. Verify the router Pods are scheduled on the OCP nodes with the new labels.