One of the difficulties with setting up a container system can be how the network is configured. In most cases if communication is required between the host and the container the advice is to use linux bridges on your host system. This works well in most cases, except with some network drivers configured for interface bonding with bridges and sub-interfaces on other vlans.
To summarize what didn't work with the tigon3 driver on Ubuntu 10.04.2, with a PPA 2.6.38-8-server kernel. This may or may not be the case with other drivers and distros.
bond0
slaves eth0 eth1
br0
bond0
vethLXC0
vethLXC1
bond0.100
So it turned out the tigon3 driver didn't support vlans when the base port was part of a bridge. This meant bond0.100 was behaving erratically, even with various bridge options configured, such as bridge_fd 0, etc.
The result was that using veth mode for the lxc containers was out of the question. So I turned to macvlan. Unfortunately the containers can't communicate with the host system when using the macvlan interface type, so another hurdle here.
Enter the newer macvlan bridge mode. The lxc.conf man page indicates that that when using macvlan bridge mode the guests can communicate with each other. Unfortunately they still can't communicate with the host. Easily solved - give the host a macvlan interface in bridge mode too. Well - not so easily solved since the Ubuntu 10.04 version of iproute2 doesn't support this.
So hack around this and install the Debian squeeze version of iproute into Ubuntu Lucid, then do the config and it works. Note that I'm using a PPA 2.6.38 kernel which has a more recent implementation of the macvlan code.
Once you have the right kernel and the right iproute2 version you need the following config in your /etc/network/interfaces to make this work:
auto bond0
iface bond0 inet static
bond-slaves eth0 eth1
bond-mode 1
bond-miimon 100
# address & netmask are required to satisfy the startup scripts
address 0.0.0.0
netmask 0.0.0.0
auto mv0
iface mv0 inet static
pre-up ip link add link bond0 name mv0 type macvlan mode bridge
address 10.0.0.3
netmask 255.255.255.0
auto bond0.100
iface bond0.100 inet static
address 192.168.0.3
netmask 255.255.255.0
Now the host and guests can all communicate with each other and the guests will appear on the network with their own MAC address.
This approach also works on Debian Squeeze 6.0 with the FAI kernels.
