The first problem is to make the traffic that is sourced from a specific set of IP addresses use a different routing scheme. In this example, all of this traffic is going out to the internet, so with a default configuration, it would use the default route regardless of what the source address was set to.
We have a router machine with a bunch of interfaces - 2 onboard interfaces and a Silicom PXG-6 (6-port gigabit card). Lets say we have two DSL lines with a small set of static IP's, 184.108.40.206/28 and 220.127.116.11/28. We had 18.104.22.168 delivered first so it's set up as the default route to the internet over eth7, while eth6 is hooked up to the router for our internal LAN.
The routing table looks like this:
The new link has been configured on eth0.
22.214.171.124 0.0.0.0 255.255.255.240 U 0 0 0 eth7
126.96.36.199 0.0.0.0 255.255.255.240 U 0 0 0 eth0
192.168.0.0 192.168.1.13 255.255.0.0 UG 0 0 0 eth6
10.0.0.0 192.168.1.13 255.0.0.0 UG 0 0 0 eth6
0.0.0.0 188.8.131.52 0.0.0.0 UG 0 0 0 eth7
The first thing we'll need to configure this is the "iproute" tool, on debian, as easy as "apt-get install iproute".
First thing to be done with this tool is to create a parallel routing table. We will call the new table "pipe2". First, we need to edit /etc/iproute2/rt_tables and add a new line:
/etc/iproute2/rt_tables: (New entry added in bold)
# reserved values(The number 200 is arbitrary but must fall between local and unspec)
Now add routes to these tables. I ended up putting these commands as "post-up" rules in the debian networking scripts for this interface (/etc/network/interfaces)
ip route add 10.0.0.0/8 via 192.168.1.13 table pipe2This sets up what the routing table should look like for traffic sourced from the second set of public addresses. Note that the rules to send office LAN traffic internally have to be duplicated in this table.
ip route add 192.168.0.0/16 via 192.168.1.13 table pipe2
ip route add 172.16.0.0/12 via 192.168.1.13 table pipe2
ip route add default via 184.108.40.206 table pipe2
Next, we must insert a policy route that tells the kernel when to apply this routing table to the traffic:
ip rule add from 220.127.116.11/28 table pipe2
This gets traffic that is sourced from an IP on 18.104.22.168/28 to use the correct default router. However, there are still a few more steps. By default, linux will answer arps for any IP addresses it owns over any interface. This means that, in the above example, eth7(22.214.171.124 net) could claim to be the owner of an IP on the 126.96.36.199 network.
This is solved with the arp_filter control in /proc/sys/net/ipv4/(interface)/arp_filter. We eliminated this with a:
for i in `echo /proc/sys/net/ipv4/conf/*/arp_filter`; do echo 1 > $i; done
Here is a great discussion on what arp_filter does.
An excellent discussion of ARP as implemented on linux is Here. (This is where I found the solution to this problem, under "Arp Flux")
In retropect we might have wanted to do that part first, to prevent the arp caches of various equipment from getting the MAC of the wrong interface. If stuff upstream gets the wrong MAC in their table, you can reset the hardware (DSL modem) or ask your ISP to flush their arp cache. We also may or may not have had some luck with the "ip neigh flush".
Next, I will publish the patches to apache2 that allow for this proxy multisourcing.