multi node instance failure

Hi,

  I've 6 machines I want to run OpenStack on. I'm planning on one controller, and the rest nodes.

I've configured the controller to run instances locally, using ubuntu1010-UEC-localuser-image.tar.gz, and I'm able to ping and SSH to them, etc. Yay :D The joy.

I then add a compute node. I'm manually installing parts to get a feel for what's happening. Here's what I do:-

$ sudo apt-get install python-software-properties
$ sudo add-apt-repository ppa:nova-core/trunk
$ sudo apt-get update
$ sudo apt-get install -y nova-common nova-compute python-mysqldb python-suds vlan

I'll then scp nova.conf from the controller to the node, and restart all services on all machines, after nova-manage db sync on the controller. I don't edit network/interfaces for br100 as I'm running VlanManager, this could be a mistake. The node will then appear in the services table.

When I attempt to run instances, and when they're scheduled for on the additional nodes the instances always fail. Here are 2 log files, from 2 nodes I've tried to configure:-

CLOUD NODE 1

2011-03-30 17:07:34,326 WARNING nova.virt.libvirt_conn [-] instance instance-00000014: ignoring error injecting data into image 808928202 (Unexpected error whi
le running command.
Command: sudo tune2fs -c 0 -i 0 /dev/nbd15
Exit code: 1
Stdout: 'tune2fs 1.41.12 (17-May-2010)\n'
Stderr: "tune2fs: Invalid argument while trying to open /dev/nbd15\nCouldn't find valid filesystem superblock.\n")
2011-03-30 17:08:05,516 ERROR nova.exception [-] Uncaught exception
(nova.exception): TRACE: Traceback (most recent call last):
(nova.exception): TRACE: File "/usr/lib/pymodules/python2.6/nova/exception.py", line 120, in _wrap
(nova.exception): TRACE: return f(*args, **kw)
(nova.exception): TRACE: File "/usr/lib/pymodules/python2.6/nova/virt/libvirt_conn.py", line 569, in spawn
(nova.exception): TRACE: self._conn.createXML(xml, 0)
(nova.exception): TRACE: File "/usr/lib/python2.6/dist-packages/libvirt.py", line 1358, in createXML
(nova.exception): TRACE: if ret is None:raise libvirtError('virDomainCreateXML() failed', conn=self)
(nova.exception): TRACE: libvirtError: internal error process exited while connecting to monitor: char device redirected to /dev/pts/1
(nova.exception): TRACE: qemu: could not load kernel '/var/lib/nova/instances/instance-00000014/kernel': Inappropriate ioctl for device
(nova.exception): TRACE:
(nova.exception): TRACE:
2011-03-30 17:08:05,518 ERROR nova.compute.manager [E44FOOY3XA5BTH21IXRG admin cloud] Instance '20' failed to spawn. Is virtualization enabled in the B
IOS?
(nova.compute.manager): TRACE: Traceback (most recent call last):
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/compute/manager.py", line 234, in run_instance
(nova.compute.manager): TRACE: self.driver.spawn(instance_ref)
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/exception.py", line 126, in _wrap
(nova.compute.manager): TRACE: raise Error(str(e))
(nova.compute.manager): TRACE: Error: internal error process exited while connecting to monitor: char device redirected to /dev/pts/1
(nova.compute.manager): TRACE: qemu: could not load kernel '/var/lib/nova/instances/instance-00000014/kernel': Inappropriate ioctl for device
(nova.compute.manager): TRACE:
(nova.compute.manager): TRACE:

CLOUD NODE 2

2011-03-30 17:29:12,937 WARNING nova.virt.libvirt_conn [-] instance instance-00000017: ignoring error injecting data into image 808928202 (Unexpected error whi
le running command.
Command: sudo tune2fs -c 0 -i 0 /dev/nbd15
Exit code: 1
Stdout: 'tune2fs 1.41.12 (17-May-2010)\n'
Stderr: "tune2fs: Invalid argument while trying to open /dev/nbd15\nCouldn't find valid filesystem superblock.\n")
2011-03-30 17:29:15,123 ERROR nova.exception [-] Uncaught exception
(nova.exception): TRACE: Traceback (most recent call last):
(nova.exception): TRACE: File "/usr/lib/pymodules/python2.6/nova/exception.py", line 120, in _wrap
(nova.exception): TRACE: return f(*args, **kw)
(nova.exception): TRACE: File "/usr/lib/pymodules/python2.6/nova/virt/libvirt_conn.py", line 569, in spawn
(nova.exception): TRACE: self._conn.createXML(xml, 0)
(nova.exception): TRACE: File "/usr/lib/python2.6/dist-packages/libvirt.py", line 1358, in createXML
(nova.exception): TRACE: if ret is None:raise libvirtError('virDomainCreateXML() failed', conn=self)
(nova.exception): TRACE: libvirtError: operation failed: failed to retrieve chardev info in qemu with 'info chardev'
(nova.exception): TRACE:
2011-03-30 17:29:15,125 ERROR nova.compute.manager [KO1VZT8AJ3DDN-UYB8F9 admin cloud] Instance '23' failed to spawn. Is virtualization enabled in the B
IOS?
(nova.compute.manager): TRACE: Traceback (most recent call last):
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/compute/manager.py", line 234, in run_instance
(nova.compute.manager): TRACE: self.driver.spawn(instance_ref)
(nova.compute.manager): TRACE: File "/usr/lib/pymodules/python2.6/nova/exception.py", line 126, in _wrap
(nova.compute.manager): TRACE: raise Error(str(e))
(nova.compute.manager): TRACE: Error: operation failed: failed to retrieve chardev info in qemu with 'info chardev'
(nova.compute.manager): TRACE:
2011-03-30 17:29:19,186 INFO nova.compute.manager [-] Found instance 'instance-00000017' in DB but no VM. State=8, so setting state to shutoff.
2011-03-30 17:29:19,187 INFO nova.compute.manager [-] DB/VM state mismatch. Changing state from '8' to '5'

There are error messages about virtualization not being enabled in the BIOS, but this is not the case, it is enabled.

Any ideas?