OpenShift Regional Disaster Recovery with Advanced Cluster Management

Find ACM in OperatorHub on the Hub cluster and follow instructions to install this operator.

Verify that the operator was successfully installed and that the MultiClusterHub is ready to be installed.

Select MultiClusterHub and use either Form view or YAML view to configure the deployment and select Create.

Once the deployment is complete you can logon to the ACM console using your OpenShift credentials.

First, find the Route that has been created for the ACM console:

This will return a route similar to this one.

After logging in you should see your local cluster imported.

Now that ACM is installed on the Hub cluster it is time to either create or import the Primary managed cluster and the Secondary managed cluster. You should see selections (as in above diagram) for Create cluster and Import cluster. Chose the selection appropriate for your environment. After the managed clusters are successfully created or imported you should see something similar to below.

In order to connect the OpenShift cluster and service networks using the Submariner add-ons, it is necessary to validate the two clusters have non-overlapping networks. This can be done by running the following command for each of the managed clusters.

These outputs show that the two example managed clusters have non-overlapping clusterNetwork and serviceNetwork ranges so it is safe to proceed.

Now that we know the cluster and service networks have non-overlapping ranges, it is time to move on to installing the Submariner add-ons for each managed cluster. This is done by using the ACM console and Cluster sets.

Navigate to selection shown below and at the bottom of the same page, select Create cluster set.

Once the new Cluster set is created select Manage resource assignments.

Follow the instructions and add the two managed clusters to the new Cluster set. Select Save and then navigate to Submariner add-ons.

Select Install Submariner add-ons at the bottom of the page and add the two managed clusters. Click through the wizard selections and make changes as needed. After Review of your selections select Install.

A successful deployment will show Connection status and Agent status as Healthy.

Execute the following steps on the Primary managed cluster and the Secondary managed cluster to enable the OMAP and Volume-Replication CSI sidecar containers in the csi-rbdplugin-provisioner Pods.

Edit the rook-ceph-operator-config ConfigMap and add CSI_ENABLE_OMAP_GENERATOR set to true.

Edit the rook-ceph-operator-config ConfigMap and add CSI_ENABLE_VOLUME_REPLICATION set to true.

Validate that the there are the two new CSI sidecar containers per csi-rbdplugin-provisioner Pod.

This is a new controller that will be installed from OCP OperatorHub on the Hub cluster. The job of this Multicluster Orchestrator controller, along with the MirrorPeer Custom Resource, is to create a bootstrap token and exchanges this token between the managed clusters.

Navigate to OperatorHub on the Hub cluster and filter for odf multicluster orchestrator.

Keep all default settings and Install this operator.

Validate successful installation by having the ability to select View Operator. This means the installation has completed.

Mirror Peer is a cluster-scoped resource to hold information about the managed clusters that will have a peering relationship.

Requirements:

After selecting View Operator in prior step you should see the Mirror Peer API. Select Create instance and then select YAML view.

Save the following YAML (below) to filename mirror-peer.yaml after replacing <cluster1> and <cluster2> with the correct names of your managed clusters in ACM.

Now create the Mirror Peer resource by copying the contents of your unique mirror-peer.yaml file into the YAML view (completely replacing original content). Select Create at the bottom of the YAML view screen.

You can also create this resource using CLI.

You can validate the secret (created from token) has been exchanged with this validation command.

Currently to enable mirroring the StorageCluster for each managed cluster will need to have the mirroring setting changed to enabled. This is a manual step using CLI and the oc patch command.

Validate mirroring is enabled on default CephBlockPool.

Validate rbd-mirror Pod is up and running.

Validate the status of the daemon health.

These steps are necessary so that metadata can be stored on the alternate cluster in a MCG object bucket using a secure transport protocol and in addition the Hub cluster needs to verify access to the object buckets.

Extract the ingress certificate for the Primary managed cluster and save the output to primary.crt.

Extract the ingress certificate for the Secondary managed cluster and save the output to secondary.crt.

Create a new YAML file cm-clusters-crt.yaml to hold the certificate bundle for both the Primary managed cluster and the Secondary managed cluster.

This ConfigMap needs to be created on the Primary managed cluster, Secondary managed cluster, and the Hub cluster.

After all the user-ca-bundle ConfigMaps are created, the default Proxy cluster resource needs to be modified.

Patch the default Proxy resource on the Primary managed cluster, Secondary managed cluster, and the Hub cluster.

Before starting this section check that MCG is installed on Primary managed cluster and the Secondary managed cluster and that the Phase is Ready.

The first step is to create an MCG object bucket or OBC (Object Bucket Claim) to be used to store persistent volume metadata on the Primary managed cluster and the Secondary managed cluster.

Copy the following YAML file to filename odrbucket.yaml

Extract the odrbucket OBC access key and secret key for each managed cluster as their base-64 encoded values. This can be done using these commands:

Now that the necessary MCG information has been extracted for the object buckets there must be new Secrets created on the Primary managed cluster and the Secondary managed cluster. These new Secrets will store the MCG object bucket access key and secret key for both managed clusters.

The S3 secret YAML format for the Primary managed cluster is similar to the following:

Create this secret on the Primary managed cluster and the Secondary managed cluster.

The S3 secret YAML format for the Secondary managed cluster is similar to the following:

Create this secret on the Primary managed cluster and the Secondary managed cluster.

On each managed cluster the ConfigMap ramen-dr-cluster-operator-config will be edited and new content added.

To find the s3CompatibleEndpoint or route for MCG on each managed cluster execute the following command:

To find the s3Bucket for the odrbucket OBC exact bucket name execute the following command:

Edit the ConfigMap to add the new content starting at s3StoreProfiles on the Primary managed cluster and the Secondary managed cluster after replacing the variables with correct values for your environment.

The values for the access and secret key must be base-64 encoded.The encoded values for the keys were retrieved in the prior section and the resulting Secrets are exactly the same as those created already on the managed clusters.

The S3 secret YAML format for the Primary managed cluster is similar to the following:

Create this secret on the Hub cluster.

The S3 secret YAML format for the Secondary managed cluster is similar to the following:

Create this secret on the Hub cluster.

After the operator is successfully created there will be a new ConfigMap called ramen-hub-operator-config.

The following new content starting at s3StoreProfiles needs to be added to the ConfigMap on the Hub cluster.

DRPlacementControl is an API available after the ODR Hub Operator is installed on the Hub cluster. It is broadly an ACM PlacementRule reconciler that orchestrates placement decisions based on data availability across clusters that are part of a DRPolicy.

On the Hub cluster navigate to Installed Operators in the busybox-sample project and select ODR Hub Operator. You should see two available APIs, DRPolicy and DRPlacementControl.

Create instance for DRPlacementControl and then go to YAML view. Make sure the busybox-sample namespace is selected at the top.

Save the following YAML (below) to filename busybox-drpc.yaml after replacing <cluster1> with the correct name of your managed cluster in ACM.

Now create the DRPlacementControl resource by copying the contents of your unique busybox-drpc.yaml file into the YAML view (completely replacing original content). Select Create at the bottom of the YAML view screen.

You can also create this resource using CLI.

Placement rules define the target clusters where resource templates can be deployed. Use placement rules to help you facilitate the multicluster deployment of your applications.

Save the following YAML (below) to filename busybox-placementrule.yaml.

Now create the PlacementRule resource for the busybox-sample application.

Start by loggin into the ACM console using your OpenShift credentials if not already logged in.

This will return a route similar to this one.

After logging in select Create application in the top right and choose Subscription.

Fill out the top of the Create an application form as shown below and select repository type Git.

The next section to fill out is below the Git box and is the repository URL for the sample application, the github branch and path to resources that will be created, the busybox Pod and PVC.

Scroll down in the form until you see Select an existing placement configuration and then put your cursor in the box below. You should see the PlacementRule created in prior section. Select this rule.

After selecting available rule then select Save in the upper right hand corner.

On the follow-on screen scroll to the bottom. You should see that there are all Green checkmarks on the application topology.

Now that the busybox application has been deployed to your preferredCluster (specified in the DRPlacementControl) the deployment can be validated.

Logon to your managed cluster where busybox was deployed by ACM. This is most likely your Primary managed cluster.

To validate that the replication resources are also created for the busybox PVC do the following:

To validate that the busybox volume has been replicated to the alternate cluster run this command on both the Primary managed cluster and the Secondary managed cluster.

Deleting the busybox application can be done using the ACM console. Navigate to Applications and then find the application to be deleted (busybox in this case).

When Delete application is selected a new screen will appear asking if the application related resources should also be deleted. Make sure to check the box to delete the Subscription and PlacementRule.

Select Delete in this screen. This will delete the busybox application on the Primary managed cluster (or whatever cluster the application was running on).

In addition to the resources deleted using the ACM console, the DRPlacementControl must also be deleted immediately after deleting the busybox application. Logon to the OpenShift Web console for the Hub cluster. Navigate to Installed Operators for the project busybox-sample. Choose OpenShift DR Hub Operator and the DRPlacementControl.

Select Delete DRPlacementControl.