nova evacuate command failed

Asked by gareen

hi, i run command nova evacuate failed.
steps:
1.run two compute host A and B
2.run instance vm01 on host A, shutdown host A
3.run command nova evacuate --on-shared-storage vm01 B

B compute log:
nova-nova.virt.libvirt.driver WARNING: Timeout waiting for vif plugging callback for instance 091cde07-297b-437c-b21f-32bc53c24701
nova-nova.compute.manager INFO: Lifecycle event 1 on VM 091cde07-297b-437c-b21f-32bc53c24701
nova-nova.compute.manager INFO: During sync_power_state the instance has a pending task. Skip.
nova-nova.virt.libvirt.driver INFO: Deleting instance files /instance_path/091cde07-297b-437c-b21f-32bc53c24701
nova-nova.virt.libvirt.driver INFO: Deletion of /instance_path/091cde07-297b-437c-b21f-32bc53c24701 complete
nova-nova.compute.manager ERROR: Setting instance vm_state to ERROR
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 5529, in _error_out_instance_on_exception
    yield
  File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 2524, in rebuild_instance
    self._rebuild_default_impl(**kwargs)
  File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 2377, in _rebuild_default_impl
    block_device_info=new_block_device_info)
  File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 2265, in spawn
    block_device_info)
  File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 3675, in _create_domain_and_network
    raise exception.VirtualInterfaceCreateException()
VirtualInterfaceCreateException: Virtual Interface creation failed
nova-oslo.messaging.rpc.dispatcher ERROR: Exception during message handling: Virtual Interface creation failed
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
    incoming.message))
  File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
    return self._do_dispatch(endpoint, method, ctxt, args)
  File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
    result = getattr(endpoint, method)(ctxt, **new_args)
  File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 393, in decorated_function
    return function(self, context, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/server.py", line 139, in inner
    return func(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/nova/exception.py", line 88, in wrapped
    payload)
  File "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
    six.reraise(self.type_, self.value, self.tb)
  File "/usr/lib/python2.6/site-packages/nova/exception.py", line 71, in wrapped
    return f(self, context, *args, **kw)
  File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 274, in decorated_function
    pass
  File "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
    six.reraise(self.type_, self.value, self.tb)
  File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 260, in decorated_function
    return function(self, context, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 327, in decorated_function
    function(self, context, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 303, in decorated_function
    e, sys.exc_info())
  File "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
    six.reraise(self.type_, self.value, self.tb)
  File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 290, in decorated_function
    return function(self, context, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 2524, in rebuild_instance
    self._rebuild_default_impl(**kwargs)
  File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 2377, in _rebuild_default_impl
    block_device_info=new_block_device_info)
  File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 2265, in spawn
    block_device_info)
  File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 3675, in _create_domain_and_network
    raise exception.VirtualInterfaceCreateException()
VirtualInterfaceCreateException: Virtual Interface creation failed

it is timeout for waiting neutron network-vif-plugged rpc message.
no any error in neutron server log during evacuation.
but the network-vif-plugged message showed up in neutron server log.

i see that it has some notifier with port api create/delete/updateand port db insert/update/set.
i think that the port data is not changed during evacuation, so no any message to notified.
how about it? need help, thanks.

Question information

Language:
English Edit question
Status:
Expired
For:
Ubuntu nova Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
gareen (thesea1982-e) said :
#1

need change one line:
but the network-vif-plugged message showed up in neutron server log during running instance.

In additional, vif_plugging_is_fatal=true and vif_plugging_timeout=600

Revision history for this message
gareen (thesea1982-e) said :
#2

i solved it.
add three lines code into function nova.compute.manager._rebuild_default_impl
.......
        instance.save(
            expected_task_state=[task_states.REBUILD_BLOCK_DEVICE_MAPPING])

+ if recreate:
+ migration = {'source_compute': instance['host'],
+ 'dest_compute': self.host, }
+ self.conductor_api.network_migrate_instance_finish(context,
+ instance,
+ migration)
+
        self.driver.spawn(context, instance, image_meta, injected_files,
                          admin_password, network_info=network_info,
                          block_device_info=new_block_device_info)
......
i think it is a bug of evacuation.
need migrate port from failed host to target host as binding:host_id is host name where the port is plugged.
but evacuation does not sent port update api to neutron.
port binding:host_id can not be migrated.

my installed rpm version is *-2014.1.1-2.el6.noarch.
whether fix the problem or not so far in today?

Revision history for this message
Launchpad Janitor (janitor) said :
#3

This question was expired because it remained in the 'Open' state without activity for the last 15 days.

Revision history for this message
Chris Friesen (cbf123) said :
#4

Looks like a real issue, I've linked it to the relevent nova bug. An upstream fix is under way.

Revision history for this message
Chris Friesen (cbf123) said :
#5

The bug was tracked in upstream OpenStack under https://bugs.launchpad.net/nova/+bug/1430042

A fix has been merged into upstream OpenStack. You can find it linked from the above bug report.