VM can't get IP address when running a multinodes deployment with GRE tunneling

On the network node, try to 'ifconfig tapcd468438-56 up'. If the the VM gets an IP after that, you're running into the same problem I am. I'm still looking for a permanent solution to my problem. I rebuilt my three node cluster mentioned in a recent launchpad question, but came up with the same issue.

-Joshua

Thanks Joshua,
Now my VM can get a IP address. However, it sucked in this line:

Sending discover...
Sending select for 50.50.1.4...
Lease of 50.50.1.4 obtained, lease time 120
deleting routers
route: SIOCDELRT: No such process
adding dns 50.50.1.2
cloud-setup: checking http://169.254.169.254/2009-04-04/meta-data/instance-id

Seems it cannot found the metadata server? The VM just sucked there and no error info.

After some minutes wait, it said timed out:

Starting logging: OK
Initializing random number generator... done.
Starting network...
udhcpc (v1.18.5) started
Sending discover...
Sending select for 50.50.1.4...
Lease of 50.50.1.4 obtained, lease time 120
deleting routers
route: SIOCDELRT: No such process
adding dns 50.50.1.2
cloud-setup: checking http://169.254.169.254/2009-04-04/meta-data/instance-id
wget: can't connect to remote host (169.254.169.254): Connection timed out
cloud-setup: failed 1/30: up 5.68. request failed

Hi Joshua,
The problem has been solved.
I changed network node setting and switch to USE namespace in both dhcp-agent and l3-agent. After this, I clean all routers and networks in quantum and do these steps:
1. create a tenant private network with private IP address
2. create a corresponding subnet for the private net
3. create a public network
4. create a corresponding subnet for that public network. Remember to set the start and end ip pool and disable dhcp on it
5. create a router
6. add a interface with private network to that router
7. set gateway for that router to the public net
8. run
quntum port-list
on the CONTROLLER node and look for the router's public IP address. You should able to ping it.
9. add a route on CONTROLLER node for the private net you added.
route add -net PRIVATE_NET/MASK gw ROUTER_PUBLIC_IP_ADDR
10. enjoy :-)

If you need to troubleshooting something, you have to run:
ip netns list
this command will return a list of existing namespace. The one you are looking for should start with 'qroute'
Then run commands in that namespace like this:
ip netns exec NAMESPACE ifconfig
ip netns exec NAMESPACE ping x.x.x.1
ip netns exec NAMESPACE ping x.x.x.2

Hope this helpful.

BTW: namespace = false is hell :-(

Hi,

Thanks very much for the help! I've completed all of your steps, but I'm curious about which node to add the route. My L3 and DHCP agents run on the network node, which has a public interface. Is the Control Node route being added to facilitate the control node's metadata's path back to the L3 external router on the network node?

Is the following link correct in stating that Security Groups and Nova Metadata, now that we've enable Namespaces and thereby, overlapping IP's?

http://docs.openstack.org/trunk/openstack-network/admin/content/ch_limitations.html

Joshua

I created the route on the control node and it worked. Wow. This is crazy..lol...

Thanks again,
Joshua

Hi Joshua,
I'm very sorry that I'm deploying openstack to production environment these days... too busy.

For your question, why we create a route on controller node to private network:

http://docs.openstack.org/trunk/openstack-network/admin/content/adv_cfg_l3_agent_metadata.html
OpenStack does not manage this routing for you, so you need to make sure that your host running the metadata service always has a route to reach each private network's subnet via the external network IP of that subnet's quantum router. To do this, you can either run quantum without namespaces, and run the quantum-l3-agent on the same host as nova-api. Otherwise, you can identify an IP prefix that includes all private network subnet's (e.g., 10.0.0.0/8) and then make sure that your metadata server has a route for that prefix with the quantum router's external IP address as the next hop.

Is the router must be created on the control node?

I create the router on the network node, but the vms can not access the external network....

Hi marvel,
It will be no difference creating a router on controller node or other node or even a PC running Windows. The "qunatum" command you invoked is just a 'client' which connect to quantum server and send your commands there. So wherever you invoke quantum or nova or keystone or other openstack commands, they always run on corresponding nodes :-)
If your VM cannot access external network, try to see if it can obtain a correct private IP address. If so, run these commands:
1. run "ip netns list" to show your namespaces
2. you should see a namespace start with qrouter. copy it
3. run "ipsh netns exec THE_NAMESPACE_YOU_JUST_COPIED exec ifconfig"
"ipsh netns exec THE_NAMESPACE_YOU_JUST_COPIED exec router"
"ipsh netns exec THE_NAMESPACE_YOU_JUST_COPIED exec ping YOUR_PRIVATE_NET_GATEWAY" // usually x.x.x.1, like 50.50.50.1
"ipsh netns exec THE_NAMESPACE_YOU_JUST_COPIED exec ping YOUR_VM_IP_ADDRESS_IN_PRIVATE_NET_GATEWAY"
and try to diagnostic your problems.

Thanks Lingfeng

I can ping vm and the internal gateway, but I can not ping the external network gateway.

Mine seems to be a l3agent problem, I list it in the following thread
https://bugs.launchpad.net/quantum/+bug/1092763

Can you tak a look at it, thanks.