Lets create another network in the same tenant, and create 3 more VMS.

+--------------------------------------+---------+----------------------------------+--------+------------+-------------+---------------------------------+
| ID                                   | Name    | Tenant ID                        | Status | Task State | Power State | Networks                        |
+--------------------------------------+---------+----------------------------------+--------+------------+-------------+---------------------------------+
| ee98e356-e017-4cf1-93c7-7483370d4fe2 | vm201-1 | 3b86f724614540bc902d3570b0bf86f1 | ACTIVE | -          | Running     | dhcp-192-168-201=192.168.201.15 |
| 756558c6-2064-4494-b432-26ac1c408f15 | vm201-2 | 3b86f724614540bc902d3570b0bf86f1 | ACTIVE | -          | Running     | dhcp-192-168-201=192.168.201.17 |
| cecc9d0f-5301-4d5f-bcfd-63fb1f357fb5 | vm201-3 | 3b86f724614540bc902d3570b0bf86f1 | ACTIVE | -          | Running     | dhcp-192-168-201=192.168.201.16 |
| f3d39ca9-3291-4aa3-b1e2-73279d1409bf | vx-1    | 3b86f724614540bc902d3570b0bf86f1 | ACTIVE | -          | Running     | officeNet=10.0.0.7              |
| 9f0fb1af-192b-4a30-a41b-8c8694ac9f05 | vx-2    | 3b86f724614540bc902d3570b0bf86f1 | ACTIVE | -          | Running     | officeNet=10.0.0.8              |
| a0b423d0-a017-408e-9988-bb86abf1943f | vx-3    | 3b86f724614540bc902d3570b0bf86f1 | ACTIVE | -          | Running     | officeNet=10.0.0.9              |
| 82fcbddc-f10a-41d9-b628-c558994d5e8c | vx2-1   | 3b86f724614540bc902d3570b0bf86f1 | ACTIVE | -          | Running     | officeNet-2=10.0.0.8            |
| e71a3d7b-cc66-4ddd-82b3-c26554e43bcf | vx2-2   | 3b86f724614540bc902d3570b0bf86f1 | ACTIVE | -          | Running     | officeNet-2=10.0.0.10           |
| 25745ee9-35f5-48e4-9c51-f35ac6988d95 | vx2-3   | 3b86f724614540bc902d3570b0bf86f1 | ACTIVE | -          | Running     | officeNet-2=10.0.0.9            |
+--------------------------------------+---------+----------------------------------+--------+------------+-------------+---------------------------------+

Lets check where these VMs are running [you can also do nova show $uuid  and get the host where its running currently]

MariaDB [nova]> select hostname,host,uuid from instances where vm_state='active';
+----------+------+--------------------------------------+
| hostname | host | uuid                                 |
+----------+------+--------------------------------------+
| vm201-1  | c25  | ee98e356-e017-4cf1-93c7-7483370d4fe2 |
| vm201-2  | c24  | 756558c6-2064-4494-b432-26ac1c408f15 |
| vm201-3  | c25  | cecc9d0f-5301-4d5f-bcfd-63fb1f357fb5 |
| vx-1     | c25  | f3d39ca9-3291-4aa3-b1e2-73279d1409bf |
| vx-2     | c24  | 9f0fb1af-192b-4a30-a41b-8c8694ac9f05 |
| vx-3     | c25  | a0b423d0-a017-408e-9988-bb86abf1943f |
| vx2-1    | c24  | 82fcbddc-f10a-41d9-b628-c558994d5e8c |
| vx2-2    | c25  | e71a3d7b-cc66-4ddd-82b3-c26554e43bcf |
| vx2-3    | c24  | 25745ee9-35f5-48e4-9c51-f35ac6988d95 |
+----------+------+--------------------------------------+

 

neutron net-list

+--------------------------------------+----------------------------------------------------+-------------------------------------------------------+
| id                                   | name                                               | subnets                                               |
+--------------------------------------+----------------------------------------------------+-------------------------------------------------------+
| 720995e0-bc7c-4f1e-91b0-3a61f9254776 | HA network tenant adf998ea387e4fb684246431de401047 | 3d4c8a7f-cb08-45b7-8a6d-e5e4b177da81 169.254.192.0/18 |
| c4520434-aea7-4263-b714-ac0b6fcdc57a | officeNet                                          | 070ef3e3-5087-4d57-a907-2821af869575 10.0.0.0/24      |
| 6bfcadcd-6251-4480-8b4e-c22a9151bf00 | HA network tenant 3b86f724614540bc902d3570b0bf86f1 | 8306c1e7-6d0f-46ec-bbc5-1c02362c1f44 169.254.192.0/18 |
| f42a0fb8-0111-466a-b17a-9ff4b95d8c14 | floating-192-168-101                               | 4ef2012a-f526-4e05-bce6-7f8d202d903c 192.168.101.0/24 |
| a8ccf499-4ff3-44b3-9541-1d35693ecf80 | dhcp-192-168-201                                   | 7fe7c672-613e-440a-9780-547a94b38805 192.168.201.0/24 |
| 52fcc1ec-86da-4ade-befa-fc81d868b44b | officeNet-2                                        | 0cd4831b-65a6-4be7-a74b-9efd3f74d826 10.0.0.0/24      |
+--------------------------------------+----------------------------------------------------+-------------------------------------------------------+

neutron net-show c4520434-aea7-4263-b714-ac0b6fcdc57a

+---------------------------+--------------------------------------+
| Field                     | Value                                |
+---------------------------+--------------------------------------+
| admin_state_up            | True                                 |
| id                        | c4520434-aea7-4263-b714-ac0b6fcdc57a |
| mtu                       | 0                                    |
| name                      | officeNet                            |
| port_security_enabled     | True                                 |
| provider:network_type     | vxlan                                |
| provider:physical_network |                                      |
| provider:segmentation_id  | 87                                   |
| router:external           | False                                |
| shared                    | False                                |
| status                    | ACTIVE                               |
| subnets                   | 070ef3e3-5087-4d57-a907-2821af869575 |
| tenant_id                 | 3b86f724614540bc902d3570b0bf86f1     |
+---------------------------+--------------------------------------+

check segmentation_id — its 87

neutron net-show officeNet-2  [using name also works ]

+---------------------------+--------------------------------------+
| Field                     | Value                                |
+---------------------------+--------------------------------------+
| admin_state_up            | True                                 |
| id                        | 52fcc1ec-86da-4ade-befa-fc81d868b44b |
| mtu                       | 0                                    |
| name                      | officeNet-2                          |
| port_security_enabled     | True                                 |
| provider:network_type     | vxlan                                |
| provider:physical_network |                                      |
| provider:segmentation_id  | 15                                   |
| router:external           | False                                |
| shared                    | False                                |
| status                    | ACTIVE                               |
| subnets                   | 0cd4831b-65a6-4be7-a74b-9efd3f74d826 |
| tenant_id                 | 3b86f724614540bc902d3570b0bf86f1     |
+---------------------------+--------------------------------------+

check segmentation_id — its 15

lets take a closer look at these:

+--------------------------------------+---------+----------------------------------+--------+------------+-------------+---------------------------------+
| ID                                   | Name    | Tenant ID                        | Status | Task State | Power State | Networks                        |
+--------------------------------------+---------+----------------------------------+--------+------------+-------------+---------------------------------+
| 9f0fb1af-192b-4a30-a41b-8c8694ac9f05 | vx-2    | 3b86f724614540bc902d3570b0bf86f1 | ACTIVE | -          | Running     | officeNet=10.0.0.8              |
| 82fcbddc-f10a-41d9-b628-c558994d5e8c | vx2-1   | 3b86f724614540bc902d3570b0bf86f1 | ACTIVE | -          | Running     | officeNet-2=10.0.0.8            |    |
+--------------------------------------+---------+----------------------------------+--------+------------+-------------+---------------------------------+

So I have vx-2 and vx-2-1 with the same IP address in the same host. (c24)

openstackfaq_admin0_vxlan

root@c14_utility_container-cc97b930:~# openstack server show 9f0fb1af-192b-4a30-a41b-8c8694ac9f05

+--------------------------------------+----------------------------------------------------------+
| Field                                | Value                                                    |
+--------------------------------------+----------------------------------------------------------+
| OS-DCF:diskConfig                    | AUTO                                                     |
| OS-EXT-AZ:availability_zone          | nova                                                     |
| OS-EXT-SRV-ATTR:host                 | c24                                                      |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | c24                                                      |
| OS-EXT-SRV-ATTR:instance_name        | instance-0000002a                                        |
| OS-EXT-STS:power_state               | 1                                                        |
| OS-EXT-STS:task_state                | None                                                     |
| OS-EXT-STS:vm_state                  | active                                                   |
| OS-SRV-USG:launched_at               | 2016-04-04T21:53:09.000000                               |
| OS-SRV-USG:terminated_at             | None                                                     |
| accessIPv4                           |                                                          |
| accessIPv6                           |                                                          |
| addresses                            | officeNet=10.0.0.8                                       |
| config_drive                         |                                                          |
| created                              | 2016-04-04T21:52:55Z                                     |
| flavor                               | m1.tiny (7dcb6a3f-6299-43d5-8634-660bab9f61cf)           |
| hostId                               | 120b05fb25dc760cd00e26062cd3b678f39c37b82a57b19741350abc |
| id                                   | 9f0fb1af-192b-4a30-a41b-8c8694ac9f05                     |
| image                                | cirros (166eb283-efc8-4a48-bbca-a9797c6b6136)            |
| key_name                             | sda                                                      |
| name                                 | vx-2                                                     |
| os-extended-volumes:volumes_attached | []                                                       |
| progress                             | 0                                                        |
| project_id                           | 3b86f724614540bc902d3570b0bf86f1                         |
| properties                           |                                                          |
| security_groups                      | [{u'name': u'default'}]                                  |
| status                               | ACTIVE                                                   |
| updated                              | 2016-04-04T21:53:10Z                                     |
| user_id                              | 40089a72571c48ab9817f209c262b38a                         |
+--------------------------------------+----------------------------------------------------------+

root@c14_utility_container-cc97b930:~# openstack server show 82fcbddc-f10a-41d9-b628-c558994d5e8c

+--------------------------------------+----------------------------------------------------------+
| Field                                | Value                                                    |
+--------------------------------------+----------------------------------------------------------+
| OS-DCF:diskConfig                    | AUTO                                                     |
| OS-EXT-AZ:availability_zone          | nova                                                     |
| OS-EXT-SRV-ATTR:host                 | c24                                                      |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | c24                                                      |
| OS-EXT-SRV-ATTR:instance_name        | instance-00000033                                        |
| OS-EXT-STS:power_state               | 1                                                        |
| OS-EXT-STS:task_state                | None                                                     |
| OS-EXT-STS:vm_state                  | active                                                   |
| OS-SRV-USG:launched_at               | 2016-04-05T10:04:49.000000                               |
| OS-SRV-USG:terminated_at             | None                                                     |
| accessIPv4                           |                                                          |
| accessIPv6                           |                                                          |
| addresses                            | officeNet-2=10.0.0.8                                     |
| config_drive                         |                                                          |
| created                              | 2016-04-05T10:04:34Z                                     |
| flavor                               | m1.tiny (7dcb6a3f-6299-43d5-8634-660bab9f61cf)           |
| hostId                               | 120b05fb25dc760cd00e26062cd3b678f39c37b82a57b19741350abc |
| id                                   | 82fcbddc-f10a-41d9-b628-c558994d5e8c                     |
| image                                | cirros (166eb283-efc8-4a48-bbca-a9797c6b6136)            |
| key_name                             | sda                                                      |
| name                                 | vx2-1                                                    |
| os-extended-volumes:volumes_attached | []                                                       |
| progress                             | 0                                                        |
| project_id                           | 3b86f724614540bc902d3570b0bf86f1                         |
| properties                           |                                                          |
| security_groups                      | [{u'name': u'default'}]                                  |
| status                               | ACTIVE                                                   |
| updated                              | 2016-04-05T10:04:50Z                                     |
| user_id                              | 40089a72571c48ab9817f209c262b38a                         |
+--------------------------------------+----------------------------------------------------------+

root@c14_utility_container-cc97b930:~# ssh c24 virsh dumpxml instance-0000002a

    <interface type='bridge'>
      <mac address='fa:16:3e:cc:cc:a5'/>
      <source bridge='brqc4520434-ae'/>
      <target dev='tap3a901e7d-bb'/>
      <model type='virtio'/>
      <driver name='qemu'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>

 

root@c14_utility_container-cc97b930:~# ssh c24 virsh dumpxml instance-00000033

    <interface type='bridge'>
      <mac address='fa:16:3e:54:14:68'/>
      <source bridge='brq52fcc1ec-86'/>
      <target dev='tap5f2653c4-19'/>
      <model type='virtio'/>
      <driver name='qemu'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>

 

root@c14_utility_container-cc97b930:~# ssh c24 brctl show

br-vxlan                8000.525400314cee       no      eth4
brqc4520434-ae          8000.3a6872e704d4       no      tap3a901e7d-bb
                                                        vxlan-87
brq52fcc1ec-86          8000.1aa8a59e8510       no      tap5f2653c4-19
                                                        vxlan-15

As you can see, vx-2 is vlan-87 and  vx2-1 is on vlan-15, so they are separated and treated differently.

If there are 2 instances of the same network (same vxlan), they would appear in the same brqXXX bridge.

root@c14_utility_container-cc97b930:~# ssh c25 brctl show

brq52fcc1ec-86          8000.aa6e9cf8e558       no      tap7bc323f8-8e
                                                        vxlan-15
brqc4520434-ae          8000.6a859e839df6       no      tap790aea7e-c0
                                                        tape2b6748f-8c
                                                        vxlan-87

So there is also something on c25 that is on vxlan-15

ifconfig vxlan-15 or ip link show vxlan-15 is not going to show anything special 🙂

you need to do

ip -d link show vxlan-15

XX: vxlan-15: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master brq52fcc1ec-86 state UNKNOWN mode DEFAULT group default
    link/ether 1a:a8:a5:9e:85:10 brd ff:ff:ff:ff:ff:ff promiscuity 1
    vxlan id 15 group 239.1.1.1 dev br-vxlan port 32768 61000 ageing 300

and its going to show the vxlan id – 15 , that its a part of the multicast group and uses dev-vxlan.

So as you see, there are a lot of interfaces 🙂

eth0 (inside the VM) <=> tapXXXX (outside the VM, on compute node) <=> brqXXX(bridge) <=> vxlan-XX (vtep) <=> br-vxlan <=>eth4 (physical interface) <=> switch <=>  [ the other side]

So when I ping between instances on different compute hosts and run tcpdump
I can  see the packets directly on tapXXX and on brqXXX  and also on vxlan-xx.

tcpdump on br-vxlan and eth4 shows the layer-3 br-vxlan ips on top.

 

When i ping 10.0.0.9(c25)  from  10.0.0.8 (on c24),

inside the 10.0.0.8 instance

$ arp -n

(10.0.0.9)  at fa:16:3e:da:3e:57  [ether] on eth0

From outside the instance:

All learned mac addresses are in the bridge  (vxlan-87)

root@c24:~# brctl showmacs brqc4520434-ae

port no mac addr                is local?       ageing timer
  2     3a:68:72:e7:04:d4       yes                0.00
  1     fa:16:3e:cc:cc:a5       no                 0.08
  2     fa:16:3e:da:3e:57       no                 0.08
  1     fe:16:3e:cc:cc:a5       yes                0.00

and

root@c24:~# bridge fdb show dev vxlan-87

3a:68:72:e7:04:d4 vlan 0 permanent
fa:16:3e:da:3e:57 vlan 0
00:00:00:00:00:00 dst 239.1.1.1 via br-vxlan self permanent
fa:16:3e:da:3e:57 dst 172.29.240.25 self

see that dst is 172.29.240.25 which is the vxlan ip of c25.

 

 

 

 

About The Author