By default, when instances are created, they are spread in the cluster. So the first VM will be in the first compute node, the second one in the second compute node and so on. This will ensure that all nodes are properly used up and filled up.
Suppose, C11 has 8 CPUs and C12 has 8 CPUs each.
Suppose you have the following flavors:
In default OpenStack setup (spreading)
|Add compute c11||8||8||8|
|Add compute c12||8||16||16|
|add new instance: my-4cpu-flavor||c11||4||12|
|add new instance: my-4cpu-flavor||c12||4||8|
|add new instance: my-6cpu-flavor||host not found|
As you see, the dashboard will show that you have 8 CPU remaining, but due to spreading, you cannot spawn a new VM.
I have seen similar problem even when you have 50+ nodes in the cluster, shows 200 vcpu free, but you are unable to launch an instance because they are evenly spread, so combined it has 200 vcpu free, but no node has enough cpu to handle 6 vcpus.
This also makes your capacity planning tricky, because do not know how much you are growing.
To change this behaviour, you need to edit your nova.conf and enable this setting:
What this does is now the nova scheduler will start filling up on the first available node as much as possible before moving on to the next node. So in our above example, the first 2 4cpu flavor will end up in c11 and the next one 6cpu will be on c12.
For customers, you can point them to this document where they can setup anti-affinity rules to ensure that their instances do not land on the same node.