Dual-Homing Servers on AT&T U-verse

Phil Karn, KA9Q
karn@ka9q.net
April 2013

The quick and dirty how-to

If you have AT&T U-verse with their public static IPv4 address option, no doubt you've run into trouble if you've tried to dual-home a server, i.e., configure it with both a public and a private (RFC1918) IP address. Here's one way to do it on Linux that works for me.

I assume the following:

  1. Your Linux machine has a single Ethernet interface, 'eth0'.
  2. Your AT&T U-verse Residential Gateway (RG) has the usual default configuration:
  3. AT&T has assigned you the public IPv4 address block 192.0.2.0/24 and properly configured it into your RG with 192.0.2.254 as the public address of the RG itself.
  4. You've assigned the public address 192.0.2.1 to your Linux server.
  5. You want to use 192.168.1.4 as the private address of your Linux server. Note that this is outside the DHCP range handled by the RG.

First ensure that you have the arptables command. In Debian Linux, install the arptables package and make sure the appropriate modules are enabled in your kernel.

Execute the following four commands after your network interface is normally configured with its public address:

arptables --flush # assuming you have no other entries

# ignore RG ARPs for private addresses
arptables --append INPUT --source-ip 192.168.1.254 --jump DROP

# statically configure your private address "hidden" from the RG
ifconfig eth0:0 192.168.1.4 netmask 255.255.255.0

# so we can still reach the RG's private address for management
ip -4 route add 192.168.1.254 via 192.0.2.254 src 192.0.2.1

One way to execute these commands automatically during network setup is to put them in your /etc/network/interfaces file like this:

auto eth0
iface eth0 inet static
address 192.0.2.1
netmask 255.255.255.0
broadcast 192.0.2.255
gateway 192.0.2.254
post-up arptables --flush || exit 0
post-up arptables --append INPUT --source-ip 192.168.1.254 --jump DROP || exit 0
post-up ifconfig eth0:0 192.168.1.4 netmask 255.255.255.0 || exit 0
post-up ip -4 route add 192.168.1.254 via 192.0.2.254 src 192.0.2.1 || exit 0

Note the '|| exit 0' appended to each post-up line. This keeps the script from aborting should one of the commands return a non-zero status code.

The gory details

AT&T's U-verse is one of the few residential broadband ISPs to make public IPv4 address blocks available at a reasonable price ($15/mo for a block of 8 addresses, 5 usable). But bugs in the U-verse Residential Gateway (RG) make them unnecessarily difficult to use.

One big problem appears when you "dual home" a local server with both a private and a public IPv4 address. There's no way to do this on the RG's management web page, and big problems appear even if you just statically configure them into your server.

Why would you need this?

Why would a local server need both private and public addresses? To optimize intra-LAN connectivity. If your server has only a public address, your local clients with private addresses won't realize that it's on the same LAN. They'll think it's just another external address and route traffic to it via the RG. Similarly, your servers with public addresses will route traffic to the local private address block through the RG as well.

On a Linux server this half of the problem can be fixed with a static route like this:

ip -4 route add 192.168.1.0/24 dev eth0

but most "dumb" clients lack such a feature for their traffic to the server.

The RG correctly "hairpins" all this traffic so you will still have connectivity between the public and private subnets but it will be inefficient. The RG has only a 100 Mb/s Ethernet port so you won't get the benefit of your own gigabit switch and host ports. This can be a real problem for backups and other intensive transfers.

Giving your servers both private and public addresses solves this problem. Local clients can use the servers' private addresses and keep their intra-LAN traffic out of the RG.

What's the problem?

The RG tracks LAN address usage by broadcasting an Address Resolution Protocol (ARP) query every few seconds for a few of the IP addresses it manages. (ARP was never meant to be used in this way. This is an abuse of the protocol.) Whoever wrote its software never anticipated more than one IP address per host, so every ARP response apparently displaces any other IP address association it may have had for that MAC address.

It also marks any previous IP address as unused. As a misguided "security" feature, the RG firewalls (drops) all incoming traffic for what it thinks are unused IP addresses. (Incoming traffic to an unused address can at most trigger an ARP query that goes unanswered. Big deal, especially since the RG is already flooding the LAN with ARP broadcasts.)

So when a local host innocently answers an ARP query for its private address it immediately loses external connectivity on its public address. The RG also sets the "firewall" flag for that host, as it always does for private addresses except those configured for port forwarding. And even after the RG later queries (and gets a response) for the host's public address, another RG software bug causes the "firewall" flag to remain on for that host even when the "auto open firewall" option is set.

So trying to dual-home a local host causes it to permanently lose external connectivity until the problem is manually fixed. This entails reconfiguring the host with only a single IP address (at least among those managed by the RG) and often manually clearing the RG's list of local hosts.

I had no idea what was going on the first few times it happened. It was a real mess.

Workarounds

Workaround #0

When I first diagnosed this problem shortly after getting U-verse in November 2009, I devised several workarounds. One of the simplest would assign a distinct MAC address to each ethernet interface for each IP address. But I haven't found a way to do this in Linux so I haven't been able to try it.

Workaround #1

For a while I simply used a separate private address block not managed by the RG: 192.168.2.0/24. (Only the TV set-top box remained on the RG-managed private block.) This worked well though I had to provide my own DHCP server and NAT (running on one of my servers with a public address) for local clients on this subnet to reach the outside world.

Workaround #2

But I prefer to rely on as few boxes as possible for my basic networking connectivity. I especially don't want to throw my wife off the network whenever I take my Linux server down just because it's providing her NAT functionality. So as an experiment, a while ago I reconfigured my network to eliminate the separate private network block; all local hosts use either public or private addresses managed by the RG. To allow my Linux servers to have one of each, I configured them with two separate physical connections each, one for each address.

I also had to change the following ARP configuration settings in the kernel with sysctl so that ARP queries were answered only through the interface that owned the target IP address; otherwise the problem described above could still happen.


# in ARP requests, use best local IP address for target IP address
net.ipv4.conf.all.arp_announce = 2

# don't answer ARPs except on interface we'd use to reach the sending IP address
net.ipv4.conf.all.arp_filter = 1

# reply only if target IP address is configured on the local address
net.ipv4.conf.all.arp_ignore = 1

I haven't played with these options to find the minimum necessary set, but of these three, arp_ignore=1 seems to be the most important as it suppresses responses to the RG from any interface that doesn't own the target IP address. That's what confuses the RG.

Workaround #3

This worked well, but its kludgy nature continued to bug me so recently I came up with a third workaround that also seems to work well.

Linux has a very rich set of packet filters that can operate on the Ethernet frame header, the Internet protocol headers (e.g., IPv4, IPv6, TCP and UDP), and even on ARP packets (which are not actually IPv4 packets).

Blocking the RG's ARP queries for our private address

Since the problem appears when one of my dual-homed interfaces responds to an RG ARP query for its private address, we need to keep the RG from knowing we're using it. That is, we want to ignore those ARP requests. We can't just block every ARP request for our private address because we need to answer them from local clients. So I ignore only the RG's ARPs for it with the following command:

arptables --append INPUT --source-ip 192.168.1.254 --jump DROP
192.168.1.254 is the RG's own address in the 192.168.1.0/24 private IPv4 address block that it manages. This command causes Linux to ignore only those ARP queries with the RG's own private address in the sender's field.

The RG also issues ARP queries for addresses in the public address block (192.0.2.0/24 in our example) using its own public IPv4 address in the ARP source field (192.0.2.254 in our example). We do not want to block them because the RG would not know where to deliver our inbound traffic from the outside. We'd lose our external connectivity.

Directly reaching the private network (except the RG)

I then assigned a static private address to each Linux server from the 192.168.1.0/24 subnet but outside the range dynamically assigned by the RG's DHCP server, e.g.:

ifconfig eth0:0 192.168.1.4 netmask 255.255.255.0
This also implicitly creates the required routing entry. Now the servers and clients can directly communicate with private addresses while my servers ignore those odious ARP queries from the RG.
Reaching the RG's management webserver

There was one last nit to fix. The RG runs a management webserver that will only accept connections on its private IP address 192.168.1.254. So I added the following command so the Linux server could still connect to the RG's management server on its private address via its public address:

ip -4 route add 192.168.1.254 via 192.0.2.254 src 192.0.2.1
where 192.0.2.254 is the public IP address of the RG. The src 192.0.2.1 option ensures that when we originate connections to the RG we'll use our public address as the source, not the private address that we don't want the RG to know about. (If we did that, the RG wouldn't be able to answer us.)

This works because the RG answers with the same MAC address ARP requests for either its public or private IP address. Our IP packets to the RG will have the RG's private address in the IP destination field so the RG will accept them, and our public address will be in the IP source field so the RG will know how to answer them. It already knows that both the private and public networks are on the LAN, and it doesn't seem to object to a connection to its private address from a local public address.

When this idea first occurred to me I was concerned about the ARP queries sent by my Linux servers to discover other hosts on the local private subnet. ARP queries contain the IP address/MAC address association of the sender, and since they're Ethernet broadcasts there was no easy way to keep them from being seen by the RG.

Fortunately, it turns out that the RG only seems to care about the IP/MAC address relationships asserted in the responses to its own queries. This is another protocol violation by the RG, but here it's a useful one. So the Linux ARP queries aren't a problem, and this scheme has been working perfectly on two separate Linux servers.