Using multiple internet providers
This article tries to show you the "proof of concept" solution for the problem: We have a
multihomed host - the host with multiple up-links (I use "link" to mention the connection to the internet to avoid confusing with TCP connections), how to use these links efficiently? This is the basic article so some issues could be missed in exchange for simplicity. I have an advanced linux multihoming article which covers missing issues.
There are few
outbound load balancing techniques, the technique discussed here is connection teaming, it
doesn't require you to patch the kernel it's a big advantage for manageability and stability of your system. Connection teaming doesn't require any support from ISPs, thus it will work with independent (may be provided by different ISPs) links of all kinds, from dialup, satellite to DSL, cable et al. In contrast, Border Gateway Protocol and Link Aggregation (Bonding) are only possible if your ISPs support them, which is very rare, so connection teaming is likely to be your only choice.
Network topology
Let's assume that we have 2 DSLs, each comes with its own modem that allows us to connect to the internet by PPPOE. Setting up PPPOE connections is easy, by using rp-pppoe or pppoe kernel module, if you want to know more about PPPOE, go to PPPOE with linux to consult my PPPOE experiences.
After setting up PPPOE, we have 2 ppp connections: ppp0 with IP address of 11.1.1.1 and ppp1 with IP address of 22.2.2.2 but we have to configure PPPOE to
not setup the default route. We will load balance 2 links using special routes so it's not desired if PPPOE setup a default gateway automatically.
/
+-----------------+ +---------+ DSL link X / +------
| 11.1.1.1 ppp0 +---+ Modem X +--------------+ ISP
| | +---------+ | +------
| Linux host | | Internet
| | +---------+ | +------
| 22.2.2.2 ppp1 +---+ Modem Y +--------------+ ISP
+-----------------+ +---------+ DSL link Y \ +------
\
Incoming connections
A friend from the IP address of X.X.X.X wants to connect to 11.1.1.1 (link X), another friend, from Y.Y.Y.Y wants to connect to 22.2.2.2 (link Y), now we want to work with both of them simultaneously. it's clear that ISPs have the responsibility to route packets coming from them to our host via the correct link, so our job is just to route outgoing packets from us to them via the corresponding interface.
Note: We have to route to corresponding because most ISPs will DROP ip spoofing packets.
Outgoing packets belong to the connection from X.X.X.X has the source of 11.1.1.1, and outgoing packets of the connection from Y.Y.Y.Y has the source of 22.2.2.2. Thus we need to create 2 tables - one for each of the sources:
# ip rule add prio 1 from 11.1.1.1 lookup 1
# ip rule add prio 2 from 22.2.2.2 lookup 2
Then set default routes for packets from each source
# ip route add table 1 to default dev ppp0
# ip route add table 2 to default dev ppp1
Now we can handle connections from both interfaces, if we want to run a web server on the host, we can setup a round robin DNS to load balance requests effectively now.
Load balancing outgoing connections
Now we try to ping gnu.org to see what we have:
# ping 199.232.76.164
connect: Network is unreachable
another try:
# ping -I 11.1.1.1 199.232.76.164
PING 199.232.76.164 from 11.1.1.1 : 56(84) bytes of data.
64 bytes from 199.232.76.164: icmp_seq=1 ttl=47 time=151 ms
We realize that we can ping only if we bind the ICMP packet to a proper IP address! It's because we have only created routes for packets having source addresses of 11.1.1.1 and 22.2.2.2, while packets initializing new outgoing connections don't have any source address yet.
We can solve this by creating a generic route for all packets that don't match any of our 2 source addresses. In this case, we will spread these packets equally to our links to balance the loads, this technique is called multipath route:
# ip route add to default nexthop dev ppp0 nexthop dev ppp1
Now we have finished load balancing our host to the internet using 2 different links.
Note: this connection teaming technique will not be perfect, as it is connections based and route based, some connections could use more bandwidth than the others, and routes are cached. Caching may makes a route to an often used site always be over the same link. But it's not too bad because in the long run, we will have an acceptable balance between links. Anyway, I have a solution for those who don't want this caching behaviour, the solution is called Routing Cache Inverse Multiplexing
Acting as an internet gateway
Usually, only 1 broadband link is enough for most single hosts. Here we have a multihomed host, usually, it's because that host will serve as the internet gateway for a big LAN. In this section, we will discuss about MASQUERADE/SNAT in a multihomed server.
Firstly - IP forwarding is required:
# echo 1 > /proc/sys/net/ipv4/ip_forward
Secondly - binding each connection to just one link. It's important because usually, ISPs avoid IP address spoofing. Thus no matter how many links we have, each connection can only choose just one link and stick with that link until it's closed.
The first packet of each outgoing connection is free to choose it's link (the decision is made by the multipath route), then all following packets of that connection must respect that decision. With techniques for stateless firewalls, we have no clue to recognize which link is chosen for the first packet. But a stateful firewall of liunux can do it. CONNMARK can store a connection based value in ip_conntrack so we can use it to mark the link chosen for the first packet. Based on that value, the routing system will route following packets to the correct link.
Note: From version 2.6.10, CONNMARK is included in the vanilla kernel, prior, you have need Patch O Matic to get this feature.
Now, mark first packets based on the interface selectected by the multipath route
# iptables -A POSTROUTING -t mangle -j MARK --set-mark 1 \
-m state --state NEW -o ppp0
# iptables -A POSTROUTING -t mangle -j MARK --set-mark 2 \
-m state --state NEW -o ppp1
# iptables -A POSTROUTING -t mangle -j CONNMARK --save-mark \
-m state --state NEW
For following packets, we will restore the saved mark before reaching the routing system
# iptables -A PREROUTING -t mangle -j CONNMARK --restore-mark
Route following packets based on the restored mark
# ip rule add fwmark 1 lookup 1
# ip rule add fwmark 2 lookup 2
The last step - Source-NAT them to the public IP addresses before send to the internet so the returning packets know where to go back to us
# iptables -A POSTROUTING -t nat -m mark --mark 1 \
-j SNAT --to-source 11.1.1.1
# iptables -A POSTROUTING -t nat -m mark --mark 2 \
-j SNAT --to-source 22.2.2.2
Note: Some people know SNAT under the name of MASQUERADE but actually, MASQUERADE is just a special case of SNAT, don't try to use MASQUERADE with multipath routes, you may get undesired results such as Route sent us somewhere else problem
Now you we'll see that the computers in LAN can access to the internet via both 2 links perfectly, congratulations!
Dealing with DHCP links and routers
Sometimes, you may want to use existing routers instead of PPPOE. For example, when you have internet gateways available in your LAN already, and you want to route packets there to practice multihoming before you are confident to change the topo of your network. If you have 1 NIC for each router, everything is too simple, just replace ppp interfaces (pppX) by ethernet interfaces (ethX). If your gateways are in the same subnet, the problem is a more complicated, please read the multiple gateways on one subnet.
The other example is that some cable internet links assign IP addresses to your ethernet interfaces dynamically, by DHCP. The DHCP server will try to automatically add a traditional default route to your routing table. It's unacceptable because we have to add a
multipath route (the route that point to multiple gateways) by ourself. This problem is discussed in DHCP Conflict.
A real world application
You known the concept, but it's very hard to create an usable software that can solve the real world problems. For example, sometimes the links get disconnected, or PPPOE links have dynamic IP addresses, and changed quite often, making SNAT rules to SNAT the packets to wrong values. So you need a software to watch over everything and tune the network settings dynamically. Routeskeeper project is here to bring you that software, fulfilling the need of all multihomed hosts, it includes all techniques discussed here as well as countless advanced features. Look at
Routeskeeper routing daemon for more info.
Good luck
I hope that you enjoyed the article. If you find it helpful, you may want to read advanced linux multihoming. Please recommend it to your friends, and link to it from your homepage.
I'm an
independent consultant so I'm willing to offer the service, helping you to build top class multihomed networks.
