Two Node Network Configuration with Private Subnet

On Apr 25, 2011, at 8:02 AM, Graham Hemingway wrote:

> New question #154185 on OpenStack Compute (nova):
> https://answers.launchpad.net/nova/+question/154185
>
> Hello OpenStack,
>
> I am setting up a two machine cloud with the Cactus release on Ubuntu. One node will acts as the cloud controller and runs rabbit, mysql, nova-api, nova-objectstore, and nova-scheduler. The other node hosts all of the VM instances and runs nova-compute and nova-network. So far so good, I think.
>
> My question regards the network configuration. I want to use VLan if possible. Let's say my publicly accessible IPs are all 99.99.99.x. I want the compute node to sit on a private subnet (ideally 192.168.x.x) and the cloud controller to sit on 99.99.99.x and have a second interface to 192.168.x.x. Is this possible?

This is definitely possible

> If so, what would the two nova.conf files need to include?

Assuming that eth0 is 192.168.x.x and eth1 is 99.99.99.x

--public_interface=eth1
--vlan_interface=eth0

>
> I assume that VM instances can be given either a 192.168.x.x address or could even be on a 10.x.x.x type of address, correct?

Yes, this is correct, but generally in a situation like this you want to have 3 networks
99.99.99.x for public
192.168.x for management network (all of the hosts have an ip on this network on eth0)
10.x for vms

This allows the rabbit /mysql/etc. communication to be going over the private addressses, but keeps the addressing for the vms separate.

> "Elastic IPs" should have the 99.99.99.x address and be assigned to the VMs. Which machine (and which nova component) takes care of mapping network traffic from 99.99.99.x to the VM address?

the machine that runs nova-network (the 'cloud controller' node in a two machine config like this). The public addresses are assigned to the nova-network host and natted to the proper vm.
>
> I hope that this question is clear. Please let me know if any additional information is needed.

> Cheers,
> Graham
>
>
> --
> You received this question notification because you are a member of Nova
> Core, which is an answer contact for OpenStack Compute (nova).

Vish,

Thank you for getting back to me so quickly. Three further questions just to verify.

1) I put --public_interface=eth1 in the nova.conf for the cloud controller only, correct? Do I set my_ip to the cloud controller's eth0 address? Do I do anything with --routing_source?

2) Do I need nova-network running at all on the node controller (i.e. the machine with the VM instances)?

3) You said I needed three networks. Your suggestion for the 10.x network needs to be configured using the --fixed_range and --network_size flags on both the cloud controller and the node controller, correct?

Thanks,
   Graham

On Apr 25, 2011, at 10:43 AM, Graham Hemingway wrote:

> Question #154185 on OpenStack Compute (nova) changed:
> https://answers.launchpad.net/nova/+question/154185
>
> Status: Answered => Open
>
> Graham Hemingway is still having a problem:
> Vish,
>
> Thank you for getting back to me so quickly. Three further questions
> just to verify.
>
> 1) I put --public_interface=eth1 in the nova.conf for the cloud
> controller only, correct? Do I set my_ip to the cloud controller's eth0
> address?

yes, to both

> Do I do anything with --routing_source?

yes --routing_source_ip=99.x address
>
> 2) Do I need nova-network running at all on the node controller (i.e.
> the machine with the VM instances)?

only on cloud controller

>
> 3) You said I needed three networks. Your suggestion for the 10.x
> network needs to be configured using the --fixed_range and
> --network_size flags on both the cloud controller and the node
> controller, correct?

Yes, although the defaults here are usually fine.

>
> Thanks,
> Graham
>
> --
> You received this question notification because you are a member of Nova
> Core, which is an answer contact for OpenStack Compute (nova).

Vish,

I really appreciate all of your help. If you would rather I split this into multiple questions please let me know. So, using your feedback here are the interfaces and nova.conf for both the cloud controller and the node controller.

### Cloud Controller /etc/network/interfaces ###
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
 address 192.168.50.1
 netmask 255.255.255.0

auto eth1
iface eth1 inet static
 address 99.99.99.177
 netmask 255.255.255.0
 gateway 99.99.99.1

### Cloud Controller /etc/nova/nova.conf ###
--daemonize=1
--dhcpbridge_flagfile=/etc/nova/nova.conf
--dhcpbridge=/usr/bin/nova-dhcpbridge
--logdir=/var/log/nova
--state_path=/var/lib/nova
--lock_path=/var/lock/nova
--verbose
--public_interface=eth1
--vlan_interface=eth0
--s3_host=192.168.50.1
--rabbit_host=192.168.50.1
--cc_host=192.168.50.1
--ec2_url=http://192.168.50.1:8443/services/Cloud
--sql_connection=mysql://nova:notnova@192.168.50.1/nova
--my_ip=192.168.50.1
--routing_source_ip=99.99.99.177
--fixed_range=192.168.51.0/24
--network_size=32

### Node Controller /etc/network/interfaces ###
# The loopback network interface
auto lo
iface lo inet loopback

auto br100
iface br100 inet static
 address 192.168.50.2
 netmask 255.255.255.0
 bridge_ports eth0
 bridge_stp off
 bridge_maxwait 0
 bridge_fd 0

### Node Controller /etc/nova/nova.conf ###
--daemonize=1
--dhcpbridge_flagfile=/etc/nova/nova.conf
--dhcpbridge=/usr/bin/nova-dhcpbridge
--logdir=/var/log/nova
--state_path=/var/lib/nova
--lock_path=/var/lock/nova
--verbose
--s3_host=192.168.50.1
--rabbit_host=192.168.50.1
--cc_host=192.168.50.1
--ec2_url=http://192.168.50.1:8443/services/Cloud
--sql_connection=mysql://nova:notnova@192.168.50.1/nova
--my_ip=192.168.50.1
--vlan_interface=eth0

My questions (hopefully the last of them):

1) Give my (many) questions, does this configuration look appropriate? Am I missing any important flags? I assume that the node controller does not need --public_interface or --routing_source, correct?

2) Does the cloud controller need a br100 bridge defined in its interfaces?

3) What configuration do I need to do on my switch or in the nova DB do support the Vlans

Thank you again for all of the help.
Cheers,
   Graham

On Apr 25, 2011, at 12:25 PM, Graham Hemingway wrote:

> Question #154185 on OpenStack Compute (nova) changed:
> https://answers.launchpad.net/nova/+question/154185
>
> Status: Answered => Open
>
> Graham Hemingway is still having a problem:
> Vish,
>
> I really appreciate all of your help. If you would rather I split this
> into multiple questions please let me know. So, using your feedback
> here are the interfaces and nova.conf for both the cloud controller and
> the node controller.
>
> ### Cloud Controller /etc/network/interfaces ###
> auto lo
> iface lo inet loopback
>
> auto eth0
> iface eth0 inet static
> address 192.168.50.1
> netmask 255.255.255.0
>
> auto eth1
> iface eth1 inet static
> address 99.99.99.177
> netmask 255.255.255.0
> gateway 99.99.99.1
>
>
> ### Cloud Controller /etc/nova/nova.conf ###
> --daemonize=1
> --dhcpbridge_flagfile=/etc/nova/nova.conf
> --dhcpbridge=/usr/bin/nova-dhcpbridge
> --logdir=/var/log/nova
> --state_path=/var/lib/nova
> --lock_path=/var/lock/nova
> --verbose
> --public_interface=eth1
> --vlan_interface=eth0
> --s3_host=192.168.50.1
> --rabbit_host=192.168.50.1
> --cc_host=192.168.50.1
> --ec2_url=http://192.168.50.1:8443/services/Cloud
> --sql_connection=mysql://nova:notnova@192.168.50.1/nova
> --my_ip=192.168.50.1
> --routing_source_ip=99.99.99.177
> --fixed_range=192.168.51.0/24
> --network_size=32
>
>
> ### Node Controller /etc/network/interfaces ###
> # The loopback network interface
> auto lo
> iface lo inet loopback
>
> auto br100
> iface br100 inet static
> address 192.168.50.2
> netmask 255.255.255.0
> bridge_ports eth0
> bridge_stp off
> bridge_maxwait 0
> bridge_fd 0
>
>
> ### Node Controller /etc/nova/nova.conf ###
> --daemonize=1
> --dhcpbridge_flagfile=/etc/nova/nova.conf
> --dhcpbridge=/usr/bin/nova-dhcpbridge
> --logdir=/var/log/nova
> --state_path=/var/lib/nova
> --lock_path=/var/lock/nova
> --verbose
> --s3_host=192.168.50.1
> --rabbit_host=192.168.50.1
> --cc_host=192.168.50.1
> --ec2_url=http://192.168.50.1:8443/services/Cloud
> --sql_connection=mysql://nova:notnova@192.168.50.1/nova
> --my_ip=192.168.50.1
> --vlan_interface=eth0
>
>
> My questions (hopefully the last of them):
>
> 1) Give my (many) questions, does this configuration look appropriate?
> Am I missing any important flags? I assume that the node controller
> does not need --public_interface or --routing_source, correct?

This looks good (hopefully i didn't forget anything)
>
> 2) Does the cloud controller need a br100 bridge defined in its
> interfaces?

no, nova will create vlans and bridges

>
> 3) What configuration do I need to do on my switch or in the nova DB do
> support the Vlans

so you will need to turn on vlan trunking for vlans 100 - 100 + X where X is the number of projects you want to support, and you should set --num_networks=X

then (from a clean db) nova-manage network create 10.0.0.0/8 X 256
(this will take a while)

>
> Thank you again for all of the help.
> Cheers,
> Graham
>
> --
> You received this question notification because you are a member of Nova
> Core, which is an answer contact for OpenStack Compute (nova).

Great. Thanks Vish. Your help has been awesome.
Cheers,
   Graham