OpenStack Compute (nova)

computeNode don‘work，IOError: Socket closed

Asked by WangPeng

Hi，everyone：
      I add a compute node(install ntp&nova-compute only,copy config from controller),use nova-manage service list is ok.
     But,when I create a instance in compute node.
     It show instance error.
     I find error from /var/log/nova/nova/nova-scheduler.log,as follow:
2012-08-08 10:57:07 ERROR nova.rpc.common [-] Failed to consume message from queue: Socket closed
2012-08-08 10:57:07 TRACE nova.rpc.common Traceback (most recent call last):
2012-08-08 10:57:07 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 490, in ensure
2012-08-08 10:57:07 TRACE nova.rpc.common return method(*args, **kwargs)
2012-08-08 10:57:07 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 567, in _consume
2012-08-08 10:57:07 TRACE nova.rpc.common return self.connection.drain_events(timeout=timeout)
2012-08-08 10:57:07 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 175, in drain_events
2012-08-08 10:57:07 TRACE nova.rpc.common return self.transport.drain_events(self.connection, **kwargs)
2012-08-08 10:57:07 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 238, in drain_events
2012-08-08 10:57:07 TRACE nova.rpc.common return connection.drain_events(**kwargs)
2012-08-08 10:57:07 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 57, in drain_events
2012-08-08 10:57:07 TRACE nova.rpc.common return self.wait_multi(self.channels.values(), timeout=timeout)
2012-08-08 10:57:07 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 63, in wait_multi
2012-08-08 10:57:07 TRACE nova.rpc.common chanmap.keys(), allowed_methods, timeout=timeout)
2012-08-08 10:57:07 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 120, in _wait_multiple
2012-08-08 10:57:07 TRACE nova.rpc.common channel, method_sig, args, content = read_timeout(timeout)
2012-08-08 10:57:07 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 88, in read_timeout
2012-08-08 10:57:07 TRACE nova.rpc.common return self.method_reader.read_method()
2012-08-08 10:57:07 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/method_framing.py", line 221, in read_method
2012-08-08 10:57:07 TRACE nova.rpc.common raise m
2012-08-08 10:57:07 TRACE nova.rpc.common IOError: Socket closed
2012-08-08 10:57:07 TRACE nova.rpc.common
2012-08-08 10:57:07 INFO nova.rpc.common [-] Reconnecting to AMQP server on 172.18.32.7:5672
2012-08-08 10:57:07 ERROR nova.rpc.common [-] AMQP server on 172.18.32.7:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.
2012-08-08 10:57:07 TRACE nova.rpc.common Traceback (most recent call last):
2012-08-08 10:57:07 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 446, in reconnect
2012-08-08 10:57:07 TRACE nova.rpc.common self._connect()
2012-08-08 10:57:07 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 423, in _connect
2012-08-08 10:57:07 TRACE nova.rpc.common self.connection.connect()
2012-08-08 10:57:07 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 154, in connect
2012-08-08 10:57:07 TRACE nova.rpc.common return self.connection
2012-08-08 10:57:07 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 560, in connection
2012-08-08 10:57:07 TRACE nova.rpc.common self._connection = self._establish_connection()
2012-08-08 10:57:07 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 521, in _establish_connection
2012-08-08 10:57:07 TRACE nova.rpc.common conn = self.transport.establish_connection()
2012-08-08 10:57:07 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 255, in establish_connection
2012-08-08 10:57:07 TRACE nova.rpc.common connect_timeout=conninfo.connect_timeout)
2012-08-08 10:57:07 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 52, in __init__
2012-08-08 10:57:07 TRACE nova.rpc.common super(Connection, self).__init__(*args, **kwargs)
2012-08-08 10:57:07 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/connection.py", line 129, in __init__
2012-08-08 10:57:07 TRACE nova.rpc.common self.transport = create_transport(host, connect_timeout, ssl)
2012-08-08 10:57:07 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/transport.py", line 281, in create_transport
2012-08-08 10:57:07 TRACE nova.rpc.common return TCPTransport(host, connect_timeout)
2012-08-08 10:57:07 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/transport.py", line 85, in __init__
2012-08-08 10:57:07 TRACE nova.rpc.common raise socket.error, msg
2012-08-08 10:57:07 TRACE nova.rpc.common error: [Errno 111] ECONNREFUSED
2012-08-08 10:57:07 TRACE nova.rpc.common
2012-08-08 10:57:10 AUDIT nova.service [-] Starting scheduler node (version 2012.1-LOCALBRANCH:LOCALREVISION)
2012-08-08 10:57:10 INFO nova.rpc.common [req-5ae8f2f4-6e4d-49d7-968c-e0a2f4547152 None None] Connected to AMQP server on 172.18.32.7:5672
2012-08-08 11:00:11 INFO nova.rpc.common [req-7ef3bced-fc4e-4812-a96c-4c32a37ea047 6ed9496e07724c969e1e470e9ea4621e f4c42d279f124477a487c57f2a96d2df] Connected to AMQP server on 172.18.32.7:5672
2012-08-08 11:16:43 ERROR nova.rpc.common [-] Failed to consume message from queue: Socket closed
2012-08-08 11:16:43 TRACE nova.rpc.common Traceback (most recent call last):
2012-08-08 11:16:43 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 490, in ensure
2012-08-08 11:16:43 TRACE nova.rpc.common return method(*args, **kwargs)
2012-08-08 11:16:43 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 567, in _consume
:$
2012-08-08 12:43:21 TRACE nova.rpc.common super(Connection, self).__init__(*args, **kwargs)
2012-08-08 12:43:21 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/connection.py", line 129, in __init__
2012-08-08 12:43:21 TRACE nova.rpc.common self.transport = create_transport(host, connect_timeout, ssl)
2012-08-08 12:43:21 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/transport.py", line 281, in create_transport
2012-08-08 12:43:21 TRACE nova.rpc.common return TCPTransport(host, connect_timeout)
2012-08-08 12:43:21 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/transport.py", line 85, in __init__
2012-08-08 12:43:21 TRACE nova.rpc.common raise socket.error, msg
2012-08-08 12:43:21 TRACE nova.rpc.common error: [Errno 111] ECONNREFUSED
2012-08-08 12:43:21 TRACE nova.rpc.common
2012-08-08 12:43:24 INFO nova.rpc.common [-] Reconnecting to AMQP server on 172.18.32.7:5672
2012-08-08 12:43:24 INFO nova.rpc.common [-] Connected to AMQP server on 172.18.32.7:5672
2012-08-08 12:44:25 ERROR nova.rpc.common [req-8e9ddc98-d9a3-43b0-9fe8-d376243fc79b 6ed9496e07724c969e1e470e9ea4621e f4c42d279f124477a487c57f2a96d2df] Failed to publish message to topic 'compute.keystone': Socket closed
2012-08-08 12:44:25 TRACE nova.rpc.common Traceback (most recent call last):
2012-08-08 12:44:25 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 490, in ensure
2012-08-08 12:44:25 TRACE nova.rpc.common return method(*args, **kwargs)
2012-08-08 12:44:25 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 593, in _publish
2012-08-08 12:44:25 TRACE nova.rpc.common publisher = cls(self.channel, topic, **kwargs)
2012-08-08 12:44:25 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 287, in __init__
2012-08-08 12:44:25 TRACE nova.rpc.common **options)
2012-08-08 12:44:25 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 239, in __init__
2012-08-08 12:44:25 TRACE nova.rpc.common self.reconnect(channel)
2012-08-08 12:44:25 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 246, in reconnect
2012-08-08 12:44:25 TRACE nova.rpc.common channel=channel, routing_key=self.routing_key)
2012-08-08 12:44:25 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 85, in __init__
2012-08-08 12:44:25 TRACE nova.rpc.common self.declare()
2012-08-08 12:44:25 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 98, in declare
2012-08-08 12:44:25 TRACE nova.rpc.common self.exchange.declare()
2012-08-08 12:44:25 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/entity.py", line 154, in declare
2012-08-08 12:44:25 TRACE nova.rpc.common nowait=nowait)
2012-08-08 12:44:25 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/syn.py", line 23, in blocking
2012-08-08 12:44:25 TRACE nova.rpc.common return __sync_current(fun, *args, **kwargs)
2012-08-08 12:44:25 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/syn.py", line 49, in __eblocking__
2012-08-08 12:44:25 TRACE nova.rpc.common return spawn(fun, *args, **kwargs).wait()
2012-08-08 12:44:25 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 166, in wait
2012-08-08 12:44:25 TRACE nova.rpc.common return self._exit_event.wait()
2012-08-08 12:44:25 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 116, in wait
2012-08-08 12:44:25 TRACE nova.rpc.common return hubs.get_hub().switch()
2012-08-08 12:44:25 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 177, in switch
2012-08-08 12:44:25 TRACE nova.rpc.common return self.greenlet.switch()
2012-08-08 12:44:25 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 192, in main
2012-08-08 12:44:25 TRACE nova.rpc.common result = function(*args, **kwargs)
2012-08-08 12:44:25 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/channel.py", line 843, in exchange_declare
2012-08-08 12:44:25 TRACE nova.rpc.common (40, 11), # Channel.exchange_declare_ok
2012-08-08 12:44:25 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/abstract_channel.py", line 95, in wait
2012-08-08 12:44:25 TRACE nova.rpc.common self.channel_id, allowed_methods)
2012-08-08 12:44:25 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/connection.py", line 202, in _wait_method
2012-08-08 12:44:25 TRACE nova.rpc.common self.method_reader.read_method()
2012-08-08 12:44:25 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/method_framing.py", line 221, in read_method
2012-08-08 12:44:25 TRACE nova.rpc.common raise m
2012-08-08 12:44:25 TRACE nova.rpc.common IOError: Socket closed

please help me!
thank you

Question information

Language:
English Edit question
Status:
Solved
For:
OpenStack Compute (nova) Edit question
Assignee:
No assignee Edit question
Solved by:
John Garbutt
Solved:
Last query:
Last reply:
Revision history for this message
WangPeng (breakwindwp) said : 		#1

error instance get stuck in networking

so /var/log/nova/nova-network.log
2012-08-08 11:16:43 ERROR nova.rpc.common [-] Failed to consume message from queue: Socket closed
2012-08-08 11:16:43 TRACE nova.rpc.common Traceback (most recent call last):
2012-08-08 11:16:43 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 490, in ensure
2012-08-08 11:16:43 TRACE nova.rpc.common return method(*args, **kwargs)
2012-08-08 11:16:43 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 567, in _consume
2012-08-08 11:16:43 TRACE nova.rpc.common return self.connection.drain_events(timeout=timeout)
2012-08-08 11:16:43 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 175, in drain_events
2012-08-08 11:16:43 TRACE nova.rpc.common return self.transport.drain_events(self.connection, **kwargs)
2012-08-08 11:16:43 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 238, in drain_events
2012-08-08 11:16:43 TRACE nova.rpc.common return connection.drain_events(**kwargs)
2012-08-08 11:16:43 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 57, in drain_events
2012-08-08 11:16:43 TRACE nova.rpc.common return self.wait_multi(self.channels.values(), timeout=timeout)
2012-08-08 11:16:43 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 63, in wait_multi
2012-08-08 11:16:43 TRACE nova.rpc.common chanmap.keys(), allowed_methods, timeout=timeout)
2012-08-08 11:16:43 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 120, in _wait_multiple
2012-08-08 11:16:43 TRACE nova.rpc.common channel, method_sig, args, content = read_timeout(timeout)
2012-08-08 11:16:43 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 88, in read_timeout
2012-08-08 11:16:43 TRACE nova.rpc.common return self.method_reader.read_method()
2012-08-08 11:16:43 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/method_framing.py", line 221, in read_method
2012-08-08 11:16:43 TRACE nova.rpc.common raise m
2012-08-08 11:16:43 TRACE nova.rpc.common IOError: Socket closed
2012-08-08 11:16:43 TRACE nova.rpc.common
2012-08-08 11:16:43 INFO nova.rpc.common [-] Reconnecting to AMQP server on 172.18.32.7:5672
2012-08-08 11:16:43 ERROR nova.rpc.common [-] AMQP server on 172.18.32.7:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.
2012-08-08 11:16:43 TRACE nova.rpc.common Traceback (most recent call last):
2012-08-08 11:16:43 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 446, in reconnect
2012-08-08 11:16:43 TRACE nova.rpc.common self._connect()
2012-08-08 11:16:43 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 423, in _connect
2012-08-08 11:16:43 TRACE nova.rpc.common self.connection.connect()
2012-08-08 11:16:43 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 154, in connect
2012-08-08 11:16:43 TRACE nova.rpc.common return self.connection
2012-08-08 11:16:43 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 560, in connection
2012-08-08 11:16:43 TRACE nova.rpc.common self._connection = self._establish_connection()
2012-08-08 11:16:43 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 521, in _establish_connection
2012-08-08 11:16:43 TRACE nova.rpc.common conn = self.transport.establish_connection()
2012-08-08 11:16:43 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 255, in establish_connection
2012-08-08 11:16:43 TRACE nova.rpc.common connect_timeout=conninfo.connect_timeout)
2012-08-08 11:16:43 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 52, in __init__
2012-08-08 11:16:43 TRACE nova.rpc.common super(Connection, self).__init__(*args, **kwargs)
2012-08-08 11:16:43 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/connection.py", line 129, in __init__
2012-08-08 11:16:43 TRACE nova.rpc.common self.transport = create_transport(host, connect_timeout, ssl)
2012-08-08 11:16:43 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/transport.py", line 281, in create_transport
2012-08-08 11:16:43 TRACE nova.rpc.common return TCPTransport(host, connect_timeout)
2012-08-08 11:16:43 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/transport.py", line 85, in __init__
2012-08-08 11:16:43 TRACE nova.rpc.common raise socket.error, msg
2012-08-08 11:16:43 TRACE nova.rpc.common error: [Errno 111] ECONNREFUSED
2012-08-08 11:16:43 TRACE nova.rpc.common
2012-08-08 11:16:46 AUDIT nova.service [-] Starting network node (version 2012.1-LOCALBRANCH:LOCALREVISION)
2012-08-08 11:16:48 INFO nova.rpc.common [-] Connected to AMQP server on 172.18.32.7:5672
2012-08-08 11:17:25 INFO nova.rpc.common [req-4a044830-2d44-4d24-9322-4e8211545c3c None None] Connected to AMQP server on 172.18.32.7:5672
2012-08-08 11:20:37 ERROR nova.rpc.common [-] Failed to consume message from queue: Socket closed
2012-08-08 11:20:37 TRACE nova.rpc.common Traceback (most recent call last):
:$
2012-08-08 13:22:58 TRACE nova.rpc.common return self.method_reader.read_method()
2012-08-08 13:22:58 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/method_framing.py", line 221, in read_method
2012-08-08 13:22:58 TRACE nova.rpc.common raise m
2012-08-08 13:22:58 TRACE nova.rpc.common IOError: Socket closed
2012-08-08 13:22:58 TRACE nova.rpc.common
2012-08-08 13:22:58 INFO nova.rpc.common [-] Reconnecting to AMQP server on 172.18.32.7:5672
2012-08-08 13:22:58 ERROR nova.rpc.common [-] AMQP server on 172.18.32.7:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.
2012-08-08 13:22:58 TRACE nova.rpc.common Traceback (most recent call last):
2012-08-08 13:22:58 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 446, in reconnect
2012-08-08 13:22:58 TRACE nova.rpc.common self._connect()
2012-08-08 13:22:58 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 423, in _connect
2012-08-08 13:22:58 TRACE nova.rpc.common self.connection.connect()
2012-08-08 13:22:58 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 154, in connect
2012-08-08 13:22:58 TRACE nova.rpc.common return self.connection
2012-08-08 13:22:58 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 560, in connection
2012-08-08 13:22:58 TRACE nova.rpc.common self._connection = self._establish_connection()
2012-08-08 13:22:58 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 521, in _establish_connection
2012-08-08 13:22:58 TRACE nova.rpc.common conn = self.transport.establish_connection()
2012-08-08 13:22:58 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 255, in establish_connection
2012-08-08 13:22:58 TRACE nova.rpc.common connect_timeout=conninfo.connect_timeout)
2012-08-08 13:22:58 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 52, in __init__
2012-08-08 13:22:58 TRACE nova.rpc.common super(Connection, self).__init__(*args, **kwargs)
2012-08-08 13:22:58 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/connection.py", line 129, in __init__
2012-08-08 13:22:58 TRACE nova.rpc.common self.transport = create_transport(host, connect_timeout, ssl)
2012-08-08 13:22:58 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/transport.py", line 281, in create_transport
2012-08-08 13:22:58 TRACE nova.rpc.common return TCPTransport(host, connect_timeout)
2012-08-08 13:22:58 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/transport.py", line 85, in __init__
2012-08-08 13:22:58 TRACE nova.rpc.common raise socket.error, msg
2012-08-08 13:22:58 TRACE nova.rpc.common error: [Errno 111] ECONNREFUSED
2012-08-08 13:22:58 TRACE nova.rpc.common
2012-08-08 13:23:01 AUDIT nova.service [-] Starting network node (version 2012.1-LOCALBRANCH:LOCALREVISION)
2012-08-08 13:23:03 INFO nova.rpc.common [-] Connected to AMQP server on 172.18.32.7:5672
2012-08-08 13:23:04 INFO nova.rpc.common [req-549a4990-f9bd-4210-ad54-66bc02301369 None None] Connected to AMQP server on 172.18.32.7:5672
2012-08-08 13:24:43 INFO nova.rpc.common [req-c258f321-a53c-4e95-a33f-b8fefa56f567 None None] Connected to AMQP server on 172.18.32.7:5672
2012-08-08 13:24:56 ERROR nova.rpc.common [-] Timed out waiting for RPC response: timed out
2012-08-08 13:24:56 TRACE nova.rpc.common Traceback (most recent call last):
2012-08-08 13:24:56 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 490, in ensure
2012-08-08 13:24:56 TRACE nova.rpc.common return method(*args, **kwargs)
2012-08-08 13:24:56 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 567, in _consume
2012-08-08 13:24:56 TRACE nova.rpc.common return self.connection.drain_events(timeout=timeout)
2012-08-08 13:24:56 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 175, in drain_events
2012-08-08 13:24:56 TRACE nova.rpc.common return self.transport.drain_events(self.connection, **kwargs)
2012-08-08 13:24:56 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 238, in drain_events
2012-08-08 13:24:56 TRACE nova.rpc.common return connection.drain_events(**kwargs)
2012-08-08 13:24:56 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 57, in drain_events
2012-08-08 13:24:56 TRACE nova.rpc.common return self.wait_multi(self.channels.values(), timeout=timeout)
2012-08-08 13:24:56 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 63, in wait_multi
2012-08-08 13:24:56 TRACE nova.rpc.common chanmap.keys(), allowed_methods, timeout=timeout)
2012-08-08 13:24:56 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 120, in _wait_multiple
2012-08-08 13:24:56 TRACE nova.rpc.common channel, method_sig, args, content = read_timeout(timeout)
2012-08-08 13:24:56 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 94, in read_timeout
2012-08-08 13:24:56 TRACE nova.rpc.common return self.method_reader.read_method()
2012-08-08 13:24:56 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/method_framing.py", line 221, in read_method
2012-08-08 13:24:56 TRACE nova.rpc.common raise m
2012-08-08 13:24:56 TRACE nova.rpc.common timeout: timed out
2012-08-08 13:24:56 TRACE nova.rpc.common

Revision history for this message
John Garbutt (johngarbutt) said : 		#2

It looks like your nova services can't connect to your RabbitMQ server:

2012-08-08 10:57:07 ERROR nova.rpc.common [-] AMQP server on 172.18.32.7:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.
And sometimes this error:
Connected to AMQP server on 172.18.32.7:5672
2012-08-08 11:17:25 INFO nova.rpc.common [req-4a044830-2d44-4d24-9322-4e8211545c3c None None] Connected to AMQP server on 172.18.32.7:5672
2012-08-08 11:20:37 ERROR nova.rpc.common [-] Failed to consume message from queue: Socket closed

What host is your RabbitMQ server on? Is the RabbitMQ service running? Does it have firewall rules blocking the AMPQ traffic?

Do you have the correct nova.conf settings for these:
rabbit_host="localhost"
rabbit_password="guest"
rabbit_port=5672

Once you are happy you have the settings right, you are best restarting all your services. They will attempt to create the queues when they first start.

I hope that helps.

Revision history for this message
Best John Garbutt (johngarbutt) said : 		#3

I forgot to mention, these settings might need checking:
rabbit_use_ssl=false
rabbit_userid="guest"
rabbit_virtual_host="/"

Revision history for this message
WangPeng (breakwindwp) said : 		#4

Thanks John Garbutt, that solved my question.