Mixed OSD device type configuration

Starting with ODF 4.7 you can customize the deployment of your OSDs to consume different device types. This feature can be combined with the BlueStore placement customization and it is illustrated below.

1. Hardware Configuration (HDD+SSD)

Your hardware configuration allows you to create partitions on an NVMe device for the OSD metadata and you want to assign a CRUSH Device Class as hdd for these OSDs. To achieve this configuration you would create a storageDeviceSet configuration illustrated below.

[...]
spec:
[...]
  storageDeviceSets:
  - count: 1
    dataPVCTemplate:
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: {size}
        storageClassName: {storageclass-hdd}
        volumeMode: Block
    name: ocs-deviceset-hdd
    portable: false
    replica: 3
    deviceType: hdd (1)
    metadataPVCTemplate:
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: {size}
        storageClassName: {storageclass-ssd}
        volumeMode: Block

1	The authorized values for the device types are `hdd`, `ssd` and `nvme`. Those device types will be used to assign a CRUSH device class within your underlying cluster. See the document on data segregation for more information on this.

2. Hardware Configuration (HDD+SSD+NVMe)

Your hardware configuration allows you to create partitions on an NVMe device for the OSD WAL, create partitions on an SSD for OSD metadata and you will also want to assign a CRUSH Device Class as ssd for these OSDs. To achieve this configuration you would create a storageDeviceSet configuration illustrated below.

[...]
spec:
[...]
  storageDeviceSets:
  - count: 1
    dataPVCTemplate:
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: {size}
        storageClassName: {storageclass-hdd}
        volumeMode: Block
    name: ocs-deviceset-mix
    portable: false
    replica: 3
    deviceType: hdd (1)
    metadataPVCTemplate:
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: {size}
        storageClassName: {storageclass-ssd}
        volumeMode: Block
    walPVCTemplate: (2)
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: {size}
        storageClassName: {storageclass-nvme}
        volumeMode: Block

1	The authorized values for the device types are `hdd`, `ssd` and `nvme`. Those device types will be used to assign a CRUSH device class within your underlying cluster. See the document on data segregation for more information on this.
2	Following the recommendations used for Red Hat Ceph Storage the only interest in separating the WAL placement is if the WAL can reside on a faster device than the device where the metadata lives.

If you do not specify a deviceType value the OSD will auto-detect the type of device using the value contained in /sys/block/{devicename}/queue/rotational. If the file contains a 0 the device type assigned will be ssd but if the value is 1 the device type assigned will be hdd. Another easy way to check if to use lsblk -d -o name,rota.

There is no built-in mechanism in the OSD to detect a difference between a SSD and a NVMe. Both will be assigend a type ssd without your customization.

3. CRUSH Tree Example

Here is an example of the CRUSH tree being generated in the underlying cluster with the specific CRUSH device class value assigned in our lab. This is an example to illustrate the CRUSH device class that gets assigned to each OSD and does not reflect the actual yaml example in this chapter.

CRUSH tree

ID  CLASS WEIGHT   TYPE NAME                        STATUS REWEIGHT PRI-AFF
 -1       39.75000 root default
 -7       39.75000     region us-east-2
-18       13.25000         zone us-east-2a
-33        8.50000             host ip-10-0-149-187
  0   hdd  8.50000                 osd.0                up  1.00000 1.00000
-17        4.75000             host ip-10-0-152-149
  3  nvme  0.50000                 osd.3                up  1.00000 1.00000
  5   ssd  4.25000                 osd.5                up  1.00000 1.00000
 -6       13.25000         zone us-east-2b
-41        8.50000             host ip-10-0-161-186
  8   hdd  8.50000                 osd.8                up  1.00000 1.00000
 -5        4.75000             host ip-10-0-179-156
  1  nvme  0.50000                 osd.1                up  1.00000 1.00000
  2   ssd  4.25000                 osd.2                up  1.00000 1.00000
-26       13.25000         zone us-east-2c
-25        4.75000             host ip-10-0-196-12
  4  nvme  0.50000                 osd.4                up  1.00000 1.00000
  7   ssd  4.25000                 osd.7                up  1.00000 1.00000
-37        8.50000             host ip-10-0-211-21
  6   hdd  8.50000                 osd.6                up  1.00000 1.00000

You can see that different CRUSH device class values are assigned to your OSDs based on the deviceType field value in your yaml configuration.

You can leverage the CRUSH device class assigned to the OSDs to write specific CRUSH rules that will target specific OSDs in a Ceph cluster.