Mosh, NAT & LXC containers

I've been using mosh for a few days now and find it very promising.  There are a few aspects of it that are not the most convenient, but these are balanced out by other features.  In particular the following points need patience or workarounds:

1. Lack of SSH agent forwarding.  This is a pretty big deal and a source of much frustration.
2. No IPv6 support yet.
3. No configuration file similar to SSH's ssh_config or ~/.ssh/config
4. No graceful handling of NAT'd servers.

The first point about SSH agent forwarding I'm hoping will just need some patience.  There appear to be enough people feeling the pain so I suspect this will be resolved soon.

The lack of IPv6 support is frustrating since there is the lack of SSH agent forwarding and the inability for mosh to easily traverse NAT's.  I expect this too should be an easy win and I hope it will be coming sooner than later.

Regarding point three, I find myself wanting a configuration file much like SSH for various reasons.  For example to create shortcut entries for hosts, or to handle some sort of ProxyCommand, etc.  However my motivation for this post relates more to the need to configure a port for a particular host, which leads me to the NAT related point.

I have LXC containers running on bare metal hardware.  Frequently these LXC's are behind a NAT like this:

                                     +--- (eth0) lxc-guest-1
                                     |    10.1.1.1
WAN --- (eth0) lxc-master-1 (br0) ---+
       1.2.3.4            10.1.1.254 |
                                     +--- (eth0) lxc-guest-2
                                          10.1.1.2

In this case you wouldn't normally be able to establish a mosh connection to either of the guest servers, however with some ugly setup it can be done.  This requires iptables rules and some client-side magic.

With SSH there are a couple of ways to handle this scenario.  My preferred approach is to make use of the ProxyCommand through the NAT'ing master, however this doesn't work with mosh:

Host lxc-guest-1
    ProxyCommand ssh lxc-master-1 /bin/nc -q 8 -w 3 lxc-guest-1 22

For mosh you need to get far more involved, setting up port forwarding through the NAT and finally having custom command lines to connect.

First, to get this NAT working the standard easiest approach is to:

iptables -A POSTROUTING -o eth0 -s 10.1.1.0/24 -j MASQUERADE 

Then you need to set up port forwarding for ssh:

iptables -A PREROUTING -i eth0 -p tcp --dport 22001 -j DNAT --to-destination 10.1.1.1:22
iptables -A PREROUTING -i eth0 -p tcp --dport 22002 -j DNAT --to-destination 10.1.1.2:22

And finally port forwarding for mosh.  It seems that by default mosh will start listening at the low end of the ports, so each LXC container will listen on port 60001 by default first.  This will quickly overlap between host and guests.

iptables -A PREROUTING -i eth0 -p udp --dport 60010:60019 -j DNAT --to-destination 10.1.1.1
iptables -A PREROUTING -i eth0 -p udp --dport 60020:60029 -j DNAT --to-destination 10.1.1.2

With all the relevant ports being forwarded you can now set up your SSH config in ~/.ssh/config

Host lxc-guest-1
    Hostname lxc-guest-1
    Port 22001

Host lxc-guest-2
    Hostname lxc-guest-2
    Port 22002

And finally you should be able to establish a mosh connection with a defined port:

mosh -p 60010 lxc-guest-1

With this config you can establish up to 5 total connections to the each guest.  This is because mosh uses the requested port for one direction of the communication, and the port+1 for the return communication.  Thus for each additional connection you'll need to increment your port number by 2.

Since you've made it this far hopefully this is working for you too now.

Since there could easily be many ports with many hosts I've made a very simple script to handle the port config for the first connection.  Put this in one of your PATH folders.  I've saved this script at /usr/local/bin/moshi

#!/bin/bash

source ~/.ssh/mosh_config
eval opts=\$${1//-/_}
mosh $opts $1

And as you can see the script expects a rudimentary mosh config file at ~/.ssh/mosh_config.  For the above server setup the config file contains the following:

lxc_guest_1="-p 60010"
lxc_guest_2="-p 60020"

Now to connect to either guest I use:

moshi lxc-guest-1

As you see this is horribly messy, but for now working with several dozen servers at the other side of the world it's making my life slightly easier.  If I find it useful enough I might even create a chef recipe for it, but I sincerely hope this won't be necessary and that both SSH agent forwarding and IPv6 support will be implemented soon.