OpenStack Ansible Ceph

Deploy: OpenStack-Ansible
2026-04-17T07:12:20+00:00
Configure OpenStack Ansible to use external Ceph storage for Glance, Cinder, and Nova by setting up pools, keyrings, and deployment variables.

OpenStack Ansible Ceph

OpenStack-Ansible connects to external Ceph clusters by configuring specific pools, keyrings, and deployment variables in user_variables.yml.

Prerequisites

  • Ceph Version: Reef (18.x) or later.
  • OSA Version: 2024.2 (Dalmatian) or later.
  • Network: OpenStack nodes must reach Ceph monitors (TCP 6789, 3300, 6800).
  • Pools: volumes, images, vms must exist and be RBD-enabled.
  • Keyrings: client.cinder, client.glance, client.nova must exist with appropriate permissions.

Step 1: Prepare Ceph Pools and Keys

On a Ceph monitor node, create pools and enable RBD applications:

ceph osd pool create volumes 128
ceph osd pool create images 64
ceph osd pool create vms 128

ceph osd pool application enable volumes rbd
ceph osd pool application enable images rbd
ceph osd pool application enable vms rbd

Create service keyrings with minimal required permissions:

ceph auth get-or-create client.cinder \
  mon 'profile rbd' \
  osd 'profile rbd pool=volumes, profile rbd pool=vms, profile rbd pool=images'

ceph auth get-or-create client.glance \
  mon 'profile rbd' \
  osd 'profile rbd pool=images'

ceph auth get-or-create client.nova \
  mon 'profile rbd' \
  osd 'profile rbd pool=vms'

Step 2: Collect Ceph Details

Gather the following values for OSA configuration:

# Cluster UUID
ceph fsid

# Monitor addresses (format: host:port)
ceph mon dump | grep "public_addr"

# Export keyrings
ceph auth get client.cinder > /tmp/ceph.client.cinder.keyring
ceph auth get client.glance > /tmp/ceph.client.glance.keyring
ceph auth get client.nova > /tmp/ceph.client.nova.keyring

Step 3: Configure OSA Variables

Edit /etc/openstack_deploy/user_variables.yml:

# Glance Configuration
glance_default_store: rbd
glance_rbd_store_pool: images
glance_rbd_store_user: glance
glance_ceph_client: glance

# Cinder Configuration
cinder_backends:
  ceph:
    volume_driver: cinder.volume.drivers.rbd.RBDDriver
    rbd_pool: volumes
    rbd_ceph_conf: /etc/ceph/ceph.conf
    rbd_user: cinder
    rbd_secret_uuid: "{{ cinder_ceph_rbd_secret_uuid }}"
    volume_backend_name: ceph
    report_discard_supported: true

# Define the secret UUID (must match libvirt secret)
cinder_ceph_rbd_secret_uuid: "457eb676-33da-42ec-9a8c-9293d545c337"

# Nova Configuration (Ephemeral disks)
nova_libvirt_images_rbd_pool: vms
nova_ceph_client: nova
nova_libvirt_rbd_secret_uuid: "{{ cinder_ceph_rbd_secret_uuid }}"

Step 4: Deploy Ceph Configuration Files

Create /etc/openstack_deploy/user_secrets.yml to define monitor addresses and config paths:

ceph_mons:
  - 10.0.0.11
  - 10.0.0.12
  - 10.0.0.13

ceph_extra_confs:
  - src: /etc/openstack_deploy/ceph.conf
    dest: /etc/ceph/ceph.conf

Create /etc/openstack_deploy/ceph.conf on the deployment host:

[global]
fsid = <your-cluster-fsid>
mon_host = 10.0.0.11:6789,10.0.0.12:6789,10.0.0.13:6789
auth_supported = cephx
auth_client_required = cephx

Place the exported keyring files in /etc/openstack_deploy/ (ceph.client.cinder.keyring, ceph.client.glance.keyring, ceph.client.nova.keyring) with permissions 600 owned by root.

Step 5: Create Component Overrides

Create /etc/openstack_deploy/env.d/ceph.yml to ensure ceph-common is installed in required containers:

component_skel:
  ceph_client:
    belongs_to:
      - cinder_all
      - glance_all
      - nova_compute

Step 6: Run the Playbooks

Navigate to the OSA directory:

cd /opt/openstack-ansible

Re-run specific service playbooks for existing deployments:

openstack-ansible playbooks/os-glance-install.yml
openstack-ansible playbooks/os-cinder-install.yml
openstack-ansible playbooks/os-nova-install.yml

Run the full setup for fresh deployments:

openstack-ansible playbooks/setup-openstack.yml

Step 7: Verify Integration

Test Glance (Images):

openstack image create --disk-format raw --container-format bare \
  --file cirros.img cirros-ceph
rbd -p images ls

Test Cinder (Volumes):

openstack volume create --size 5 test-ceph-vol
rbd -p volumes ls

Test Nova (Boot from Volume):

openstack server create --flavor m1.small \
  --block-device source=image,id=<image-id>,dest=volume,size=10,bootindex=0 \
  --network demo-net test-bfv-vm

Troubleshooting

  • ceph-common missing: Ensure env.d/ceph.yml exists and re-run setup-hosts.yml or the specific service playbook.
  • Permission denied on RBD: Verify keyring files exist in /etc/ceph/ inside the container and match ceph.conf auth_client_required.
  • Glance upload timeout: Check network connectivity from the Glance container to Ceph monitors (nc -zv <mon-ip> 6789).
  • Cinder volume creation fails: Verify rbd_secret_uuid in user_variables.yml matches the secret created in the container (virsh secret-uuid).
  • Nova ephemeral disk errors: Ensure nova_libvirt_rbd_secret_uuid matches the Cinder secret and the vms pool is accessible.