TCP Load Balancing en masse

[jhunt]

I would like to be able to load-balance lots of TCP ports to lots of backend hosts (where each host can respond in kind to any of the port range, without needing to be persistent beyond the established connection lifetime).

I think we can do this with a combination of haproxy and iptables, where a 3rd process watches the stats socket for haproxy (like hatop does) and when the set of healthy backend servers changes, re-configures iptables.

Here's an example iptables ruleset, for three backing hosts and a 1000 port range:

iptables -A PREROUTING -t nat -p tcp -m multiport -d 192.168.1.1 --dports 8192:9191 \
         -m statistic --mode random --probability 0.33            \
         -j DNAT --to-destination 10.20.30.1
iptables -A PREROUTING -t nat -p tcp -m multiport -d 192.168.1.1 --dports 8192:9191 \
         -m statistic --mode random --probability 0.5            \
         -j DNAT --to-destination 10.20.30.2
iptables -A PREROUTING -t nat -p tcp -m multiport -d 192.168.1.1 --dports 8192:9191 \
         -j DNAT --to-destination 10.20.30.3

if we assume that 192.168.1.1 is our VIP (thanks keepalived!) and we need 1/3 of the new (SYN sent) connections to go to 10.20.30.1, 1/3 (1/2 of the remaining 2/3) to go to 10.20.30.2, and 1/3 (all of the remaining 1/3) to go to 10.20.30.3.

The probabilities will have to be reconfigured whenever the number of healthy IPs in the pool (determined by haproxy or something like it) changes.