Ceph Setup Ubuntu Three Node

Ceph Storage
2026-04-17T07:12:20+00:00
Step-by-step guide to deploying a three-node Ceph cluster on Ubuntu 22.04 using cephadm for high-availability storage.

Ceph Setup Ubuntu Three Node

Deploy a three-node Ceph cluster on Ubuntu 22.04 using cephadm with triple replication.

Prerequisites

  • OS: Ubuntu 22.04 LTS (minimal) on all 3 nodes
  • RAM: 4 GB minimum per node
  • Disks: 1 OS disk + 1 dedicated, unpartitioned OSD disk per node
  • Network: All nodes on the same L2 subnet with static IPs
  • Access: Root or sudo on all nodes; passwordless SSH between nodes
Host IP Hostname
Node 1 10.0.0.11 ceph01
Node 2 10.0.0.12 ceph02
Node 3 10.0.0.13 ceph03

Step 1: Prepare All Nodes

Update packages, install dependencies, and synchronize time on all nodes.

sudo apt update && sudo apt upgrade -y
sudo apt install -y chrony lvm2 cephadm
sudo timedatectl set-ntp true

Set the hostname and update /etc/hosts on each node.

sudo hostnamectl set-hostname ceph01

cat <<EOF | sudo tee -a /etc/hosts
10.0.0.11 ceph01
10.0.0.12 ceph02
10.0.0.13 ceph03
EOF

Step 2: Bootstrap the First Node

Initialize the cluster on ceph01 and record the dashboard credentials.

sudo cephadm bootstrap \
  --mon-ip 10.0.0.11 \
  --cluster-network 10.0.0.0/24

Install CLI tools and verify the initial state.

sudo cephadm install ceph-common
sudo ceph -s

Step 3: Add Remaining Hosts

Configure passwordless SSH from ceph01 to the other nodes using the cluster key.

sudo ssh-copy-id -i /etc/ceph/ceph.pub root@ceph02
sudo ssh-copy-id -i /etc/ceph/ceph.pub root@ceph03

Register the new hosts with the orchestrator.

sudo ceph orch host add ceph02 10.0.0.12
sudo ceph orch host add ceph03 10.0.0.13
sudo ceph orch host ls

Step 4: Deploy Monitors and Managers

Deploy monitors and managers across all three nodes for high availability.

sudo ceph orch apply mon --placement="ceph01,ceph02,ceph03"
sudo ceph orch apply mgr --placement="ceph01,ceph02,ceph03"

Step 5: Add OSDs

Automatically claim all available unpartitioned disks on the nodes.

sudo ceph orch apply osd --all-available-devices
sudo ceph osd tree

Step 6: Create a Storage Pool

Create a replicated pool named rbd-pool with 64 PGs and a size of 3.

sudo ceph osd pool create rbd-pool 64 64 replicated
sudo ceph osd pool set rbd-pool size 3
sudo ceph osd pool set rbd-pool min_size 2
sudo ceph osd pool application enable rbd-pool rbd

Step 7: Verify Cluster Health

Check that the cluster reports HEALTH_OK with 3 monitors, 3 managers, and 3 OSDs.

sudo ceph -s

Step 8: Enable the Dashboard

Access the dashboard at https://<bootstrap-node-ip>:8443 or reset the admin password.

sudo ceph dashboard ac-user-set-password admin NewSecurePass123

Step 9: Optional — Add an S3 Gateway

Deploy an S3-compatible object storage gateway on ceph01 on port 8080.

sudo ceph orch apply rgw my-rgw --placement="ceph01" --port=8080

Troubleshooting

  • HEALTH_WARN clock skew: Verify chrony is running: chronyc sources -v
  • OSD not appearing: Check available devices: sudo ceph orch device ls
  • Dashboard unreachable: Allow TCP 8443: sudo ufw allow 8443/tcp
  • Mon election flapping: Check network latency and packet loss between nodes
  • cephadm fails to install: Ensure cephadm package is installed on the bootstrap node