Instances has no fixed ip allocated and stay in pending state

Asked by Gary

Hi all,
I have configured a two-node environment with one contrllor node (OpenstackServer) runs all the nova services including nova-compute and one compute node (openstack2) that runs only nova-compute. Both nodes are running Diablo based on Ubuntu 11.10 server. For now, I can successfully run an instance on the controller node, and the compute node is in :-) state when I run "sudo nova-manage service list" on the controller node. But when I spawn an instance on the compute node, the instance stay in pending state and has no ip allcated.

On controller node, I have network interfaces as below:
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
  address 10.201.194.101
  netmask 255.255.255.0
  broadcast 10.201.194.127
  gateway 10.201.194.126
  dns-nameservers 8.8.8.8

auto eth1
iface eth1 inet static
  address 192.168.3.1
  netmask 255.255.255.0
  network 192.168.3.0
  broadcast 192.168.3.255

br100 Link encap:Ethernet HWaddr 00:26:b9:8d:a5:37
          inet addr:172.16.0.3 Bcast:172.16.0.255 Mask:255.255.255.0
          inet6 addr: fe80::226:b9ff:fe8d:a537/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:528469 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1502 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:25478029 (25.4 MB) TX bytes:136029 (136.0 KB)

eth0 Link encap:Ethernet HWaddr 00:26:b9:8d:a5:36
          inet addr:10.201.194.101 Bcast:10.201.194.127 Mask:255.255.255.0
          inet6 addr: fe80::226:b9ff:fe8d:a536/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:3710355 errors:0 dropped:0 overruns:0 frame:0
          TX packets:20689912 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:443067374 (443.0 MB) TX bytes:6690527683 (6.6 GB)
          Interrupt:16 Memory:da000000-da012800

eth1 Link encap:Ethernet HWaddr 00:26:b9:8d:a5:37
          inet6 addr: fe80::226:b9ff:fe8d:a537/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:556611 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3310 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:36944751 (36.9 MB) TX bytes:376678 (376.6 KB)
          Interrupt:17 Memory:dc000000-dc012800

lo Link encap:Local Loopback
          inet addr:127.0.0.1 Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING MTU:16436 Metric:1
          RX packets:1444405 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1444405 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1125911682 (1.1 GB) TX bytes:1125911682 (1.1 GB)

virbr0 Link encap:Ethernet HWaddr 42:9e:88:46:ac:01
          inet addr:192.168.122.1 Bcast:192.168.122.255 Mask:255.255.255.0
          UP BROADCAST MULTICAST MTU:1500 Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

vnet0 Link encap:Ethernet HWaddr fe:16:3e:1c:2b:0e
          inet6 addr: fe80::fc16:3eff:fe1c:2b0e/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:498 errors:0 dropped:0 overruns:0 frame:0
          TX packets:623 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:151152 (151.1 KB) TX bytes:184020 (184.0 KB)

And the network interfaces of the compute node:
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
  address 10.201.194.102
  netmask 255.255.255.0
  broadcast 10.201.194.127
  gateway 10.201.194.126
  dns-nameservers 8.8.8.8

auto eth1
iface eth1 inet static
  address 192.168.3.4
  netmask 255.255.255.0
  network 192.168.3.0
  broadcast 192.168.3.255

eth0 Link encap:Ethernet HWaddr 00:26:b9:8d:7b:46
          inet addr:10.201.194.102 Bcast:10.201.194.127 Mask:255.255.255.0
          inet6 addr: fe80::226:b9ff:fe8d:7b46/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:721192 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3299126 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:160267210 (160.2 MB) TX bytes:1107300583 (1.1 GB)
          Interrupt:16 Memory:da000000-da012800

eth1 Link encap:Ethernet HWaddr 00:26:b9:8d:7b:47
          inet addr:192.168.3.4 Bcast:192.168.3.255 Mask:255.255.255.0
          inet6 addr: fe80::226:b9ff:fe8d:7b47/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:76258 errors:0 dropped:0 overruns:0 frame:0
          TX packets:203 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:5011394 (5.0 MB) TX bytes:18222 (18.2 KB)
          Interrupt:17 Memory:dc000000-dc012800

lo Link encap:Local Loopback
          inet addr:127.0.0.1 Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING MTU:16436 Metric:1
          RX packets:301 errors:0 dropped:0 overruns:0 frame:0
          TX packets:301 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:27872 (27.8 KB) TX bytes:27872 (27.8 KB)

virbr0 Link encap:Ethernet HWaddr 96:58:5f:ef:60:ce
          inet addr:192.168.122.1 Bcast:192.168.122.255 Mask:255.255.255.0
          UP BROADCAST MULTICAST MTU:1500 Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

The nova.conf of both controller and compute node:

--verbose=true
--allow_admin_api=true
--use_deprecated_auth=false

#logging and other administrative
--dhcpbridge_flagfile=/etc/nova/nova.conf
--dhcpbridge=/usr/bin/nova-dhcpbridge
--logdir=/var/log/nova
--state_path=/var/lib/nova
--lock_path=/var/lock/nova

#network
--network_manager=nova.network.manager.FlatDHCPManager
--public_interface=eth0
--flat_interface=eth1
--flat_network_bridge=br100
--force_dhcp_release=True
--fixed_range=172.16.0.0/24
--flat_network_dhcp_start=172.16.0.1
--routing_source_ip = 10.201.194.101
--multi_host=T
--network_size=255
--flat_injected=false
--network_host=OpenstackServer
--my_ip=10.201.194.101
--api_paste_config=/etc/nova/api-paste.ini

#vnc
--vncproxy_url=http://10.201.194.101:6080
--vncproxy_wwwroot=/var/lib/nova/noVNC/

#scheduler
--scheduler_driver=nova.scheduler.simple.SimpleScheduler

#glance
--image_service=nova.image.glance.GlanceImageService
--glance_api_servers=10.201.194.101:9292
#--glance_api_servers=127.0.0.1:9292
#rabbit
--rabbit_host=10.201.194.101

#nova database
--sql_connection=mysql://nova:novirus@10.201.194.101/nova
#volumes
--volume_group=nova-volumes
--aoe_eth_dev=eth0
--iscsi_ip_prefix=10.201.194

The setting of network seem to be fine:
id IPv4 IPv6 start address DNS1 DNS2 VlanID project uuid
1 172.16.0.0/24 None 172.16.0.2 8.8.8.8 None None None None

And following are the log files:
nova-network.log:
2012-04-05 16:00:15,501 DEBUG nova.rpc [-] received {u'_context_roles': [u'Admin', u'Admin'], u'_msg_id': u'3093b09f9bdd4b96a5de54061b1de2bd', u'_context_read_deleted': False, u'_context_request_id': u'dbf5ff2f-d677-4c81-9e9c-6d3e9e3c406e', u'args': {u'project_id': u'1', u'requested_networks': None, u'instance_type_id': 2, u'instance_id': 39, u'host': u'openstack3', u'vpn': False}, u'_context_auth_token': u'999888777666', u'_context_strategy': u'keystone', u'_context_is_admin': True, u'_context_project_id': u'1', u'_context_timestamp': u'2012-04-05T08:00:12.808550', u'_context_user_id': u'adminUser', u'method': u'allocate_for_instance', u'_context_remote_address': u'10.201.194.101'} from (pid=1042) __call__ /usr/lib/pymodules/python2.7/nova/rpc/impl_kombu.py:600
2012-04-05 16:00:15,502 DEBUG nova.rpc [-] unpacked context: {'user_id': u'adminUser', 'roles': [u'Admin', u'Admin'], 'timestamp': u'2012-04-05T08:00:12.808550', 'auth_token': u'999888777666', 'msg_id': u'3093b09f9bdd4b96a5de54061b1de2bd', 'remote_address': u'10.201.194.101', 'strategy': u'keystone', 'is_admin': True, 'request_id': u'dbf5ff2f-d677-4c81-9e9c-6d3e9e3c406e', 'project_id': u'1', 'read_deleted': False} from (pid=1042) _unpack_context /usr/lib/pymodules/python2.7/nova/rpc/impl_kombu.py:646
2012-04-05 16:00:15,502 DEBUG nova.network.manager [dbf5ff2f-d677-4c81-9e9c-6d3e9e3c406e adminUser 1] floating IP allocation for instance |39| from (pid=1042) allocate_for_instance /usr/lib/pymodules/python2.7/nova/network/manager.py:217
2012-04-05 16:00:15,503 DEBUG nova.network.manager [dbf5ff2f-d677-4c81-9e9c-6d3e9e3c406e adminUser 1] network allocations for instance 39 from (pid=1042) allocate_for_instance /usr/lib/pymodules/python2.7/nova/network/manager.py:437
2012-04-05 16:00:15,554 DEBUG nova.rpc [-] Making asynchronous call on network.openstack3 ... from (pid=1042) multicall /usr/lib/pymodules/python2.7/nova/rpc/impl_kombu.py:728
2012-04-05 16:00:15,555 DEBUG nova.rpc [-] MSG_ID is 686f963f85db48749a2921610d01002a from (pid=1042) multicall /usr/lib/pymodules/python2.7/nova/rpc/impl_kombu.py:731

nova-compute.log of compute node:
2012-04-05 16:00:14,418 DEBUG nova.rpc [-] received {u'_context_roles': [u'Admin', u'Admin'], u'_context_request_id': u'dbf5ff2f-d677-4c81-9e9c-6d3e9e3c406e', u'_context_read_deleted': False, u'args': {u'request_spec': {u'num_instances': 1, u'image': {u'status': u'active', u'name': u'tty0-ami', u'deleted': False, u'container_format': u'ami', u'created_at': u'2012-04-05 07:59:38', u'disk_format': u'ami', u'updated_at': u'2012-04-05 07:59:39', u'properties': {u'kernel_id': u'26', u'owner': u'1', u'min_ram': u'0', u'ramdisk_id': u'27', u'min_disk': u'0'}, u'location': u'swift+http://system:root:testpass@10.201.194.101:8080/auth/v1.0//CONTAINER/28', u'checksum': u'10047a119149e08fb206eea89832eee0', u'is_public': True, u'deleted_at': None, u'id': 28, u'size': 25165824}, u'filter': None, u'instance_type': {u'rxtx_quota': 0, u'flavorid': 1, u'name': u'm1.tiny', u'deleted': False, u'created_at': None, u'updated_at': None, u'memory_mb': 512, u'vcpus': 1, u'rxtx_cap': 0, u'extra_specs': {}, u'swap': 0, u'deleted_at': None, u'id': 2, u'local_gb': 0}, u'blob': None, u'instance_properties': {u'vm_state': u'building', u'availability_zone': None, u'ramdisk_id': u'27', u'instance_type_id': 2, u'user_data': u'', u'vm_mode': None, u'reservation_id': u'r-bs0t3fth', u'user_id': u'adminUser', u'display_description': u'rerw', u'key_data': None, u'power_state': 0, u'project_id': u'1', u'metadata': {}, u'access_ip_v6': None, u'access_ip_v4': None, u'kernel_id': u'26', u'key_name': None, u'display_name': u'rerw', u'config_drive_id': u'', u'local_gb': 0, u'architecture': None, u'locked': False, u'launch_time': u'2012-04-05T08:00:13Z', u'memory_mb': 512, u'vcpus': 1, u'image_ref': 28, u'root_device_name': None, u'os_type': None, u'config_drive': u''}}, u'requested_networks': None, u'availability_zone': None, u'instance_id': 39, u'admin_password': u'FcXvjsWoKxYC58SB', u'injected_files': []}, u'_context_auth_token': u'999888777666', u'_context_strategy': u'keystone', u'_context_is_admin': True, u'_context_project_id': u'1', u'_context_timestamp': u'2012-04-05T08:00:12.808550', u'_context_user_id': u'adminUser', u'method': u'run_instance', u'_context_remote_address': u'10.201.194.101'} from (pid=1905) __call__ /usr/lib/pymodules/python2.7/nova/rpc/impl_kombu.py:600
2012-04-05 16:00:14,419 DEBUG nova.rpc [-] unpacked context: {'user_id': u'adminUser', 'roles': [u'Admin', u'Admin'], 'timestamp': u'2012-04-05T08:00:12.808550', 'auth_token': u'999888777666', 'msg_id': None, 'remote_address': u'10.201.194.101', 'strategy': u'keystone', 'is_admin': True, 'request_id': u'dbf5ff2f-d677-4c81-9e9c-6d3e9e3c406e', 'project_id': u'1', 'read_deleted': False} from (pid=1905) _unpack_context /usr/lib/pymodules/python2.7/nova/rpc/impl_kombu.py:646
2012-04-05 16:00:15,350 AUDIT nova.compute.manager [dbf5ff2f-d677-4c81-9e9c-6d3e9e3c406e adminUser 1] instance 39: starting...
2012-04-05 16:00:15,483 DEBUG nova.rpc [-] Making asynchronous call on network ... from (pid=1905) multicall /usr/lib/pymodules/python2.7/nova/rpc/impl_kombu.py:728
2012-04-05 16:00:15,483 DEBUG nova.rpc [-] MSG content: {'args': {'vpn': False, 'requested_networks': None, 'instance_type_id': 2L, 'instance_id': 39L, 'host': 'openstack3', 'project_id': '1'}, 'method': 'allocate_for_instance'} <nova.context.RequestContext object at 0x561da50> from (pid=1905) multicall /usr/lib/pymodules/python2.7/nova/rpc/impl_kombu.py:729
2012-04-05 16:00:15,484 DEBUG nova.rpc [-] MSG_ID is 3093b09f9bdd4b96a5de54061b1de2bd from (pid=1905) multicall /usr/lib/pymodules/python2.7/nova/rpc/impl_kombu.py:732
2012-04-05 16:00:15,484 DEBUG nova.rpc [-] Pool creating new connection from (pid=1905) create /usr/lib/pymodules/python2.7/nova/rpc/impl_kombu.py:504
2012-04-05 16:00:15,489 INFO nova.rpc [-] Connected to AMQP server on 10.201.194.101:5672
2012-04-05 16:00:40,318 INFO nova.compute.manager [-] Found 1 in the database and 0 on the hypervisor.
2012-04-05 16:01:40,328 INFO nova.compute.manager [-] Updating host status

It seems like nova-network cannot get to the network_info step because the fixed ip does not be allocated. Can anyone help or do you need any further information? Thanks in advance.

Gary

Question information

Language:
English Edit question
Status:
Solved
For:
OpenStack Compute (nova) Edit question
Assignee:
No assignee Edit question
Solved by:
yong sheng gong
Solved:
Last query:
Last reply:
Revision history for this message
Best yong sheng gong (gongysh) said :
#1

on compute node, I cannot see your br100 bridge. It seems there is no nova-network running on compute node. since you have enabled multi_host=T, there should be nova-network on compute node as far as I know.

Revision history for this message
Gary (g9762645) said :
#2

Thanks yong sheng gong, that solved my question.