I spent too much time figuring out why traffic was not routing with an iocage/jail configured to run on a VLAN to not write something about it. Here's the desired outcome:

A BSD iocage/jail running on a TruNas core system using vnet with it's own IP address that's part of a VLAN (not the untagged network of the NAS system itself).

The questions that arise are:

  1. What bridges to use
  2. Where to assign IP addresses for the NAS side
  3. How to configure iocage to attach to the plumbing correctly
  4. When it doesn't work, what to look for



Starting with the base interface, bge0, when setting up (or before doing this work) the host IP address is assigned here. It can stay assigned directly to the interface, it doesn't seem to have any impact.

Moving on to the VLAN (let's say the VLAN Tag is 52) you want to put the jail on, create a new interface in Network->Interfaces:

  1. Add
  2. Type: VLAN
  3. Name: VLAN52
  4. Parent Interface: bge0
  5. Vlan tag: 52
  6. IP address: use an available IP on your 52 vlan that will be for the NAS system itself. This ,just be different from the IP you want to assign to the jail. Be sure to set the mask (/24 usually) to be correct for the IP segment.
  7. Apply

Now create a bridge for this interface. A bridge is necessary for iocage to attach the vnet interfaces, think of it as a wiring point where the VLAN52 interface will connect with the jail vnet interface(s), there can be more than one jail/vnet attached to this bridge if you want them to all be on VLAN52. Create the bridge:

  1. Type: bridge
  2. Name: bridge52 (named after the VLAN it connects)
  3. Bridge members: vlan52
  4. IP address: none, since vlan52 has an assigned IP and because assigning an IP for the NAS to the bridge didn't seem to work.
  5. Apply

Now you'll need to go through "Network Test", the IP you assigned to the vlan52 interface should be pingable now. You'll then need to "Save" the network changes. Running ifconfig after this process should show something that looks like this:

bge0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=c019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
	ether aa:aa:aa:aa:aa:aa
	inet 10.10.10.11 netmask 0xffffff00 broadcast 10.10.10.255
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	nd6 options=9<PERFORMNUD,IFDISABLED>
vlan52: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=80001<RXCSUM,LINKSTATE>
	ether aa:aa:aa:aa:aa:aa
	inet 10.10.52.11 netmask 0xffffff00 broadcast 10.10.52.255
	groups: vlan
	vlan: 52 vlanpcp: 0 parent interface: bge0
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
	nd6 options=9<PERFORMNUD,IFDISABLED>
bridge52: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	ether 02:42:03:1c:d1:34
	id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
	maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200
	root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
	member: vlan52 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
	        ifmaxaddr 0 port 4 priority 128 path cost 55
	groups: bridge
	nd6 options=9<PERFORMNUD,IFDISABLED>

iocage configuration

The important point about the iocage configuration to understand, that is not obvious, is that without a few specific and simple steps, iocage will create a new bridge (bridge0) and attach the vnet's to that. If you do only have of the steps, it'll look sort of ok, but won't work.

Create the jail, the first page of options is routine, name, type (clone), release (based on your system). Page 2 (Configure Networking):

  1. VNET: check
  2. vnet_default_interface: none (important!)
  3. ipv4 interface: vnet0
  4. ipv4 address: set to the IP you've chosen for the jail
  5. ipv4 default router: set if you what the jail to have a route off the vlan

You can also use DHCP. At this point all you can do is review and submit. Once you've done that, go back an edit the settings. This could be done using 'Advanced' during creation also. Go to the network section and observe the setting for interfaces, it'll default to vnet0:bridge0. This needs to be changed to vnet0:bridge52, referring to the bridge we set up above.

The important part here is that setting vnet_default_interface: none and interfaces: vnet0:bridge52 are the critical pair of settings that will properly put the jail onto bridge52 and create the right connectivity

Problems

So what happens if these get skipped.

  1. without `interfaces: `vnet0:bridge52` the jail will be put on the default bridge0 attached to bge0
  2. without `vnet_default_interface: none` the vnet0 will be attached to bridge52, but iocage will also attach bge0 to bridge52 creating a senseless and non-functioning network. If this happens you'll see bge0 as a member of bridge52.