OpenStack Stacking vs Spreading

The Nova scheduler decides which compute host runs each new instance. Two fundamental strategies govern this decision: stacking (fill hosts before moving to the next) and spreading (distribute instances evenly across all hosts). Choosing the right strategy directly impacts resource utilization, performance, and power consumption.

Stacking vs Spreading at a Glance

How the Nova Scheduler Works

The Nova scheduler uses a two-phase process:

1. Filtering — eliminates hosts that cannot run the instance (not enough RAM, wrong AZ, affinity rules, etc.)
2. Weighing — scores remaining hosts to pick the best one

The ram_weight_multiplier setting controls stacking vs spreading:

Positive value (1.0) = prefer hosts with MORE free RAM = spreading
Negative value (-1.0) = prefer hosts with LESS free RAM = stacking

Configuring Spreading (Default)

Edit /etc/nova/nova.conf on the controller:

[filter_scheduler]
available_filters = nova.scheduler.filters.all_filters
enabled_filters = AvailabilityZoneFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter,NUMATopologyFilter

[DEFAULT]
scheduler_weight_classes = nova.scheduler.weights.ram.RAMWeigher

[filter_scheduler]
ram_weight_multiplier = 1.0

With ram_weight_multiplier = 1.0, the scheduler prefers the host with the most free RAM, distributing instances evenly.

Configuring Stacking

[filter_scheduler]
ram_weight_multiplier = -1.0

With a negative value, the scheduler prefers the host with the least free RAM (but still enough to fit the instance), packing hosts tightly.

Adding CPU and Disk Weighers

For more nuanced scheduling, combine multiple weighers:

[DEFAULT]
scheduler_weight_classes = nova.scheduler.weights.ram.RAMWeigher,nova.scheduler.weights.cpu.CPUWeigher,nova.scheduler.weights.disk.DiskWeigher

[filter_scheduler]
ram_weight_multiplier = 1.0
cpu_weight_multiplier = 1.0
disk_weight_multiplier = 1.0

All positive values = spreading across RAM, CPU, and disk simultaneously.

Overcommit Ratios

Overcommit ratios interact with stacking/spreading by defining how much more than physical capacity you allow:

[DEFAULT]
cpu_allocation_ratio = 4.0    # 4:1 vCPU to physical CPU
ram_allocation_ratio = 1.5    # 1.5:1 virtual to physical RAM
disk_allocation_ratio = 1.0   # no disk overcommit

Real-World Strategies

Dev/Test Cloud (Stacking + High Overcommit)

[filter_scheduler]
ram_weight_multiplier = -1.0

[DEFAULT]
cpu_allocation_ratio = 16.0
ram_allocation_ratio = 2.0

Maximizes density. Acceptable for non-production workloads.

Production Cloud (Spreading + Low Overcommit)

[filter_scheduler]
ram_weight_multiplier = 1.0

[DEFAULT]
cpu_allocation_ratio = 2.0
ram_allocation_ratio = 1.0

Ensures consistent performance and limits blast radius.

Mixed Strategy with Host Aggregates

Use host aggregates to apply different strategies to different host groups:

# Create aggregates
openstack aggregate create --zone dev-zone dev-aggregate
openstack aggregate create --zone prod-zone prod-aggregate

# Add hosts
openstack aggregate add host dev-aggregate compute-dev-01
openstack aggregate add host prod-aggregate compute-prod-01
openstack aggregate add host prod-aggregate compute-prod-02

# Set metadata
openstack aggregate set --property cpu_allocation_ratio=16.0 dev-aggregate
openstack aggregate set --property cpu_allocation_ratio=2.0 prod-aggregate

Monitoring Placement

Verify where instances land:

openstack server list --long -c Name -c Host -c Status
openstack hypervisor stats show
openstack hypervisor list --long

Check per-host resource usage:

openstack hypervisor show compute-01 -c vcpus_used -c memory_mb_used -c running_vms

Troubleshooting

Summary

Stacking maximizes per-host density and reduces power costs; spreading minimizes noisy-neighbor effects and failure blast radius. Most production clouds use spreading with moderate overcommit ratios, while dev/test environments benefit from stacking.