Setting up multi-site replication

Overview

Here we discuss setting up RADOS Gateway multi-site replication in a charmed Ceph deployment. Native replication between ceph-radosgw applications is supported via juju relations. By default, both primary and secondary RGW applications accept write operations (i.e. active-active replication is configured).

Prerequisites

In order to setup multi-site, we need two Ceph clusters (one primary and one secondary) both backing separate ceph-radosgw applications. For simplicity we will refer to these as primary-ceph-radosgw and secondary-ceph-radosgw.

Multi-site configuration

Multi-site configuration is done using charm configs for the respective primary and secondary ceph-radosgw application.

As will be shown in the below sections, multi-site can either be deployed directly or by having an existing cluster migrated to multi-site by means of a new secondary cluster.

Configuring multi-site as initial deployment

To configure multi-site in an initial deployment, ensure that the following charm configurations are set for both the primary and secondary ceph-radosgw applications in the bundle file.

    # Primary Ceph rgw config
    primary-ceph-radosgw:
        realm: replicated
        zonegroup: us
        zone: us-east
    # Secondary Ceph rgw config
    secondary-ceph-radosgw:
        realm: replicated
        zonegroup: us
        zone: us-west

Multi-site migration for an existing Ceph cluster

To configure multi-site for an existing Ceph cluster, configuration can be done directly via Juju commands:

# Primary Ceph configuration
juju config primary-ceph-radosgw realm=replicated
juju config primary-ceph-radosgw zonegroup=us
juju config primary-ceph-radosgw zone=us-east

# Secondary Ceph configuration
juju config primary-ceph-radosgw realm=replicated
juju config primary-ceph-radosgw zonegroup=us
juju config primary-ceph-radosgw zone=us-west

Relating the primary and secondary RGW applications

Once the ceph-radosgw applications are configured, they will go into a blocked state until the primary/secondary relation is added.

primary-ceph-radosgw/0*    blocked  idle  0  10.5.2.103  80/tcp  multi-site configuration but primary/secondary relation missing
secondary-ceph-radosgw/0*  blocked  idle  1  10.5.2.197  80/tcp  multi-site configuration but primary/secondary relation missing

The multi-site relation can be added as:

juju integrate primary-ceph-radosgw:primary secondary-ceph-radosgw:secondary

Once the integration has been added the realm, zonegroup and zone configuration will be created in the primary deployment and then synced to the secondary deployment.

The current sync status can be validated from either rgw app using the radosgw-admin sync status command:

sudo radosgw-admin --id=rgw.$(hostname -s) sync status
          realm 3a7acb0c-c7a3-4b74-8d52-43a330c7a9f0 (replicated)
          zonegroup 170a884a-6610-40f8-aec8-24117aa7f088 (us)
          zone 77218517-8bf7-4c29-af0e-bb5312861cf8 (us-east)
    metadata sync no sync (zone is master)
    data sync source: aac90d5a-f4ac-4dd7-acbe-61879b7ea4b8 (us-west)
                        syncing
                        full sync: 0/128 shards
                        incremental sync: 128/128 shards
                        data is caught up with source

Cross model relations

When the Ceph clusters are managed through separate Juju models, we would need a cross model relationship to establish the relation between the primary and secondary RGW applications.

One ceph-radosgw application acts as the initial primary for the deployment - setup the primary relation endpoint as the provider of the offer for the cross-model relation (here for simplicity, pmodel and smodel are taken to be the respective primary and secondary model names):

juju offer -m pmodel primary-ceph-radosgw:primary

The cross-model relation offer can then be consumed in the other model and related to the secondary ceph-radosgw application:

juju consume -m smodel admin/pmodel.primary-ceph-radosgw
juju integrate -m smodel secondary-ceph-radosgw:secondary primary-ceph-radosgw:primary

Once the deployment is complete, the default zone and zonegroup can optionally be tidied using the ‘tidydefaults’ action:

juju run -m smodel --wait secondary-ceph-radosgw/0 tidydefaults

Read/write vs read-only

By default all zones within a deployment will be read/write capable but this can be configured according to need using the readonly and readwrite actions.

Non-primary zones can be made as read-only by using the ‘readonly’ action:

juju run -m smodel --wait secondary-ceph-radosgw/0 readonly

A zone that is currently read-only can be switched to read/write mode by using the ‘readwrite’ action:

juju run -m smodel --wait secondary-ceph-radosgw/0 readwrite