neutron

quantum router cannot ping default gateway

Asked by Shixiong Shang

Hi, guys:

I created quantum network node in the lab with 3 bridges: br-int, br-eth1, br-eth2. The latter two are mapped to eth1 and eth2 respectively.

dmd@os-net1:~$ sudo ovs-vsctl show
a57d694d-17b0-4609-8cd5-f56f2092dc3e
    Bridge "br-eth1"
        Port "phy-br-eth1"
            Interface "phy-br-eth1"
        Port "br-eth1"
            Interface "br-eth1"
                type: internal
        Port "eth1"
            Interface "eth1"
    Bridge "br-eth2"
        Port "qg-3a81b07f-5b"
            Interface "qg-3a81b07f-5b"
                type: internal
        Port "br-eth2"
            Interface "br-eth2"
                type: internal
        Port "eth2"
            Interface "eth2"
    Bridge br-int
        Port br-int
            Interface br-int
                type: internal
        Port "qr-6e478915-3e"
            tag: 1
            Interface "qr-6e478915-3e"
                type: internal
        Port "tap645a1a4e-ba"
            tag: 1
            Interface "tap645a1a4e-ba"
                type: internal
        Port "int-br-eth1"
            Interface "int-br-eth1"
    ovs_version: "1.4.0+build0"

dmd@os-net1:~$ ifconfig -a
br-eth1 Link encap:Ethernet HWaddr 00:50:56:95:04:49
          inet6 addr: fe80::250:56ff:fe95:449/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:2 errors:0 dropped:0 overruns:0 frame:0
          TX packets:39 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:120 (120.0 B) TX bytes:6012 (6.0 KB)

br-eth2 Link encap:Ethernet HWaddr 00:50:56:95:04:4b
          inet6 addr: fe80::250:56ff:fe95:44b/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:123520 errors:0 dropped:14 overruns:0 frame:0
          TX packets:39 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:16935735 (16.9 MB) TX bytes:6012 (6.0 KB)

br-int Link encap:Ethernet HWaddr 1a:e5:05:c1:69:41
          inet6 addr: fe80::18e5:5ff:fec1:6941/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:12 errors:0 dropped:0 overruns:0 frame:0
          TX packets:40 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:936 (936.0 B) TX bytes:6082 (6.0 KB)

eth0 Link encap:Ethernet HWaddr 00:50:56:95:24:40
          inet addr:7.10.177.143 Bcast:7.10.177.255 Mask:255.255.255.0
          inet6 addr: fe80::250:56ff:fe95:2440/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:154078 errors:0 dropped:0 overruns:0 frame:0
          TX packets:77439 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:39418654 (39.4 MB) TX bytes:19629489 (19.6 MB)

eth1 Link encap:Ethernet HWaddr 00:50:56:95:04:49
          inet6 addr: fe80::250:56ff:fe95:449/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:3 errors:0 dropped:0 overruns:0 frame:0
          TX packets:11046 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:180 (180.0 B) TX bytes:2400695 (2.4 MB)

eth2 Link encap:Ethernet HWaddr 00:50:56:95:04:4b
          inet6 addr: fe80::250:56ff:fe95:44b/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:123601 errors:0 dropped:0 overruns:0 frame:0
          TX packets:241 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:16776242 (16.7 MB) TX bytes:26610 (26.6 KB)

eth2.280 Link encap:Ethernet HWaddr 00:50:56:95:04:4b
          inet addr:7.10.180.143 Bcast:7.10.180.255 Mask:255.255.255.0
          inet6 addr: fe80::250:56ff:fe95:44b/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:85 errors:0 dropped:0 overruns:0 frame:0
          TX packets:152 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:4763 (4.7 KB) TX bytes:14388 (14.3 KB)

int-br-eth1 Link encap:Ethernet HWaddr 9e:78:39:f7:af:9a
          inet6 addr: fe80::9c78:39ff:fef7:af9a/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:54 errors:0 dropped:0 overruns:0 frame:0
          TX packets:54 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:7440 (7.4 KB) TX bytes:7440 (7.4 KB)

lo Link encap:Local Loopback
          inet addr:127.0.0.1 Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING MTU:16436 Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

phy-br-eth1 Link encap:Ethernet HWaddr 6a:d4:dd:51:bd:a7
          inet6 addr: fe80::68d4:ddff:fe51:bda7/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:54 errors:0 dropped:0 overruns:0 frame:0
          TX packets:54 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:7440 (7.4 KB) TX bytes:7440 (7.4 KB)

I also created network ext_net and subnet ext_sub with ip 7.10.180.1 - 10/24. After associating it with router, the gateway gets IP address of 7.10.180.1. In addition, the quantum network node created the following namespace:

dmd@os-net1:~$ sudo ip netns list
qdhcp-0d4f531b-5d83-483e-8865-946afa766404
qrouter-9498ec31-14ef-4e68-b068-54d6db3bbbdb

Now, I tried to ping the default gateway, but ping failed:

dmd@os-net1:~$ sudo ip netns exec qrouter-9498ec31-14ef-4e68-b068-54d6db3bbbdb ping 7.10.180.254
PING 7.10.180.254 (7.10.180.254) 56(84) bytes of data.
From 7.10.180.1 icmp_seq=1 Destination Host Unreachable
From 7.10.180.1 icmp_seq=2 Destination Host Unreachable
From 7.10.180.1 icmp_seq=3 Destination Host Unreachable

Then I realized the ARP table was not populated with default gateway's MAC address:

dmd@os-net1:~$ sudo ip netns exec qrouter-9498ec31-14ef-4e68-b068-54d6db3bbbdb arp -a
? (7.10.180.254) at <incomplete> on qg-3a81b07f-5b

Would you please let me know what I can take a look to root cause the problem?

Thanks!

Shixiong

Question information

Language:
English Edit question
Status:
Answered
For:
neutron Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
Shixiong Shang (sparkofwisdom-cloud) said : 		#1

Here are more details.....A couple of things I noticed:

1) In "sudo ovs-ofctl show br-eth2" command output, the port for external gateway is DOWN/DOWN.....Any potential reasons this port can be stuck in this status?

 4(qg-3a81b07f-5b): addr:67:02:00:00:00:00
     config: PORT_DOWN
     state: LINK_DOWN

2) In "sudo ovs-dpctl dump-flows br-eth2" command output, the ARP is not VLAN tagged. I expected it to have vlan 280. Otherwise, there is no way ARP can work since the eth2 link is VLAN tagged.

3) In "sudo ovs-dpctl show br-eth2" command output, there is an error, but I have no idea what it refers to...

   Jan 23 10:04:34|00001|netdev_linux|WARN|/sys/class/net/qg-3a81b07f-5b/carrier: open failed: No such file or directory

Thanks again!

Shixiong

!!!!! While ping from inside the namespace is going on......!!!!!
dmd@os-net1:~$ sudo ovs-ofctl dump-flows br-eth2
NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=480.006s, table=0, n_packets=37, n_bytes=5593, priority=2,in_port=3 actions=drop
 cookie=0x0, duration=480.617s, table=0, n_packets=1878, n_bytes=247840, priority=1 actions=NORMAL

dmd@os-net1:~$ sudo ovs-ofctl dump-flows br-eth2
NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=482.687s, table=0, n_packets=37, n_bytes=5593, priority=2,in_port=3 actions=drop
 cookie=0x0, duration=483.298s, table=0, n_packets=1884, n_bytes=248580, priority=1 actions=NORMAL

dmd@os-net1:~$ sudo ovs-ofctl dump-flows br-eth2
NXST_FLOW reply (xid=0x4):
 cookie=0x0, duration=487.772s, table=0, n_packets=37, n_bytes=5593, priority=2,in_port=3 actions=drop
 cookie=0x0, duration=488.383s, table=0, n_packets=1907, n_bytes=250726, priority=1 actions=NORMAL

dmd@os-net1:~$ sudo ovs-ofctl show br-eth2
OFPT_FEATURES_REPLY (xid=0x1): ver:0x1, dpid:000000505695044b
n_tables:255, n_buffers:256
features: capabilities:0xc7, actions:0xfff
 2(eth2): addr:00:50:56:95:04:4b
     config: 0
     state: 0
     current: 10GB-FD COPPER
     advertised: COPPER
     supported: 1GB-FD 10GB-FD COPPER
 3(phy-br-eth2): addr:2e:f6:7c:63:13:6a
     config: 0
     state: 0
     current: 10GB-FD COPPER
 4(qg-3a81b07f-5b): addr:67:02:00:00:00:00
     config: PORT_DOWN
     state: LINK_DOWN
 LOCAL(br-eth2): addr:00:50:56:95:04:4b
     config: 0
     state: 0
OFPT_GET_CONFIG_REPLY (xid=0x3): frags=normal miss_send_len=0

dmd@os-net1:~$ sudo ovs-dpctl dump-flows br-eth2
in_port(4),eth(src=fa:16:3e:a1:66:4e,dst=ff:ff:ff:ff:ff:ff),eth_type(0x0806),arp(sip=7.10.180.1,tip=7.10.180.254,op=1,sha=fa:16:3e:a1:66:4e,tha=00:00:00:00:00:00), packets:456, bytes:19152, used:0.796s, actions:3,0,2

dmd@os-net1:~$ sudo ovs-dpctl show br-eth2
system@br-eth2:
        lookups: hit:95276 missed:37845 lost:0
        flows: 7
        port 0: br-eth2 (internal)
        port 2: eth2
        port 3: phy-br-eth2
Jan 23 10:04:34|00001|netdev_linux|WARN|/sys/class/net/qg-3a81b07f-5b/carrier: open failed: No such file or directory
        port 4: qg-3a81b07f-5b (internal)

dmd@os-cli:~$ quantum net-show ext_net
+---------------------------+--------------------------------------+
| Field | Value |
 +---------------------------+--------------------------------------+
| admin_state_up | True |
| id | 1a92289c-6422-4bcd-a6be-80022f8e30ef |
| name | ext_net |
| provider:network_type | vlan |
| provider:physical_network | physnet2 |
| provider:segmentation_id | 280 |
| router:external | True |
| shared | False |
| status | ACTIVE |
| subnets | c0b963e0-15bf-4a76-818a-34b6d8753cde |
| tenant_id | 328049adf27c4bbbbe5ac1974d89aed9 |
 +---------------------------+--------------------------------------+
dmd@os-cli:~$

Revision history for this message
Shixiong Shang (sparkofwisdom-cloud) said : 		#2

I just noticed this error message in syslog...Seems like kernel cannot find driver for the port....This is on Ubuntu 12.04:

Jan 24 10:23:08 os-net1 ovs-vsctl: 00001|vsctl|INFO|Called as /usr/bin/ovs-vsctl -- --may-exist add-port br-eth2 qg-bfb769f2-81 -- set Interface qg-bfb769f2-81 type=internal -- set Interface qg-bfb769f2-81 external-ids:iface-id=bfb769f2-81f7-48d4-9550-9c6d2adb39c2 -- set Interface qg-bfb769f2-81 external-ids:iface-status=active -- set Interface qg-bfb769f2-81 external-ids:attached-mac=fa:16:3e:97:d5:5d
Jan 24 10:23:08 os-net1 kernel: [ 1430.074688] device qg-bfb769f2-81 entered promiscuous mode
Jan 24 10:23:08 os-net1 NetworkManager[1125]: SCPlugin-Ifupdown: devices added (path: /sys/devices/virtual/net/qg-bfb769f2-81, iface: qg-bfb769f2-81)
Jan 24 10:23:08 os-net1 NetworkManager[1125]: SCPlugin-Ifupdown: device added (path: /sys/devices/virtual/net/qg-bfb769f2-81, iface: qg-bfb769f2-81): no ifupdown configuration found.
Jan 24 10:23:08 os-net1 NetworkManager[1125]: <warn> /sys/devices/virtual/net/qg-bfb769f2-81: couldn't determine device driver; ignoring...

Revision history for this message
dan wendlandt (danwent) said : 		#3

my guess would be that the OVS kernel module is not loaded.

easiest thing to do would be to reboot your box and make sure that ovs-dpctl show does not return an error.

Can you help with this problem?

Provide an answer of your own, or ask Shixiong Shang for more information if necessary.

To post a message you must log in.