Enable use of the RGW on an OCS internal deployment

Depending on the infrastructure where you deploy OCS, the RGW may or may not be deployed. It is not deployed where MCG has the ability to provision directly an ObjectStore to use as a backing store. For example, RGW is not deployed when OCP/OCS runs on AWS, as in this case MCG will just provision directly an AWS S3 bucket.

But there may be cases when you want to specifically use the RGW, like to make use of the Ceph bucket notifications feature.
The following steps will show you how to do this deployment manually.

1. Status verification

First, make sure that the RGW is not already deployed. You can do this with the following command:

oc get -n openshift-storage CephObjectStore

This should return nothing. Otherwise you already have an ObjectStore, and therefore an active RGW. In this case you can directly go to the Service and Route steps to gain access to it.

2. Creating the CephObjectStore

The CephObjectStore can be deployed with this YAML file (oc apply -f cephobjectstore.yaml):

cephobjectstore.yaml

---
apiVersion: ceph.rook.io/v1
kind: CephObjectStore
metadata:
  name: ocs-storagecluster-cephobjectstore
  namespace: openshift-storage
spec:
  dataPool:
    crushRoot: ""
    deviceClass: ""
    erasureCoded:
      algorithm: ""
      codingChunks: 0
      dataChunks: 0
    failureDomain: host
    replicated:
      size: 3
  gateway:
    allNodes: false
    instances: 1
    placement:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
          - matchExpressions:
            - key: cluster.ocs.openshift.io/openshift-storage
              operator: Exists
      podAntiAffinity:
        preferredDuringSchedulingIgnoredDuringExecution:
        - podAffinityTerm:
            labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - rook-ceph-rgw
            topologyKey: kubernetes.io/hostname
          weight: 100
      tolerations:
      - effect: NoSchedule
        key: node.ocs.openshift.io/storage
        operator: Equal
        value: "true"
    port: 80
    resources:
      limits:
        cpu: "2"
        memory: 4Gi
      requests:
        cpu: "1"
        memory: 4Gi
    securePort: 0
    sslCertificateRef: ""
  metadataPool:
    crushRoot: ""
    deviceClass: ""
    erasureCoded:
      algorithm: ""
      codingChunks: 0
      dataChunks: 0
    failureDomain: host
    replicated:
      size: 3

Note: the parameters you may want to change are:

name: you can change it but make sure to adapt the other files that follow.
failureDomain: default is zone for AWS. You may want to adapt for other infrastructures.
instances: if you want more than one RGW. In this case, make sure to put some load-balancing in front.

3. Service and Route

To access the RGW internally, you’ll need a Service which has already been created automatically with the ObjectStore, and a Route if you want to access it from anywhere. So you can apply this file to create the Route:

route.yaml

---
kind: Route
apiVersion: route.openshift.io/v1
metadata:
  name: s3-rgw
  namespace: openshift-storage
  labels:
    app: rook-ceph-rgw
    ceph_daemon_id: ocs-storagecluster-cephobjectstore
    ceph_daemon_type: rgw
    rgw: ocs-storagecluster-cephobjectstore
    rook_cluster: openshift-storage
    rook_object_store: ocs-storagecluster-cephobjectstore
spec:
  to:
    kind: Service
    name: rook-ceph-rgw-ocs-storagecluster-cephobjectstore
    weight: 100
  port:
    targetPort: http
  tls:
    termination: edge
    insecureEdgeTerminationPolicy: Allow
  wildcardPolicy: None

The Service or the Route you have created are the endpoints that you can use in your application or code that connects to Object Storage.

4. StorageClass

To create ObjectBucketClaims against the RGW (and not against the MCG which is default), you can create the following StorageClass:

storageclass.yaml

---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: ocs-storagecluster-ceph-rgw
  annotations:
    description: Provides Object Bucket Claims (OBCs) using the RGW
provisioner: openshift-storage.ceph.rook.io/bucket
parameters:
  objectStoreName: ocs-storagecluster-cephobjectstore
  objectStoreNamespace: openshift-storage
  region: us-east-1
reclaimPolicy: Delete
volumeBindingMode: Immediate

If you want to use the RGW when creating an ObjectBucketClaim, you can now select ocs-storagecluster-ceph-rgw as the storage class.

4.1. Using the Rook-Ceph toolbox to check on the Ceph backing storage

Since the Rook-Ceph toolbox is not shipped with OCS, we need to deploy it manually.

You can patch the OCSInitialization ocsinit using the following command line:

oc patch OCSInitialization ocsinit -n openshift-storage --type json --patch  '[{ "op": "replace", "path": "/spec/enableCephTools", "value": true }]'

5. Create a S3 user

5.1. Method 1

To create a new S3 user interactively, log into the Ceph toolbox using the command below:

oc rsh -n openshift-storage $(oc get pod -n openshift-storage -l app=rook-ceph-tools -o jsonpath='{.items[0].metadata.name}')

Create a S3 user using the following command:

radosgw-admin user create --display-name="Your user" --uid=your-user

The output of the command will give you all the details for the newly create user, especially this part:

{
  "user": "your-user",
  "access_key": "XXXXXXXXXXXXXXXX",
  "secret_key": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
}

5.2. Method 2

To be honest, it’s the same as the previous one, but in one line…

oc exec -n openshift-storage $(oc get pod -n openshift-storage -l app=rook-ceph-tools -o jsonpath='{.items[0].metadata.name}') -- radosgw-admin user create --uid="<user-name>" --display-name="<Display Name>"