The big problem: you just can't get the RG's stateful firewall out of the way!
Most "static IP" services provide a simple bridged Ethernet interface to a router somewhere at the other end of your phone or cable line, with no filtering or firewalling whatsoever. Not U-verse. Even if you turn off every "protection" option and check the "Auto Firewall Open" option under "Advanced Home Network Settings", it still blocks some traffic such as the ICMP messages that make traceroute work. (Ironically, traceroute works fine through the RG's built-in NAT.) And every stateful firewall is potentially subject to memory exhaustion from large numbers of TCP connections or UDP associations that even individual users may legitimately create (think of Bit Torrent, Skype, and even loading CNN's main page).
Problem number two: the RG abuses ARP to limit each Ethernet MAC address (i.e., physical computer) to just one IP address, either public or private -- not both. This precludes attaching a "real" router to the RG to manage your address block. It can also complicate life for servers with clients on both the local network and the outside Internet, but this problem can be worked around by using private addresses from another local block block not managed by the RG.
When I first wrote this, the U-verse 3800HGV-B Residential Gateway (RG) was running software version 5.29.135.47. It has since been changed to 6.1.9.24-enh.tm. It has a very different 'look' to its configuration pages, but I have seen no change in functionally; the problems I describe here seem to persist. I sincerely hope another update will soon moot everything I say here.
But the real problem is that the software in the RG to support static IP addresses is badly broken. Although static (public) IP addresses bypass the RG's NAT, they are still processed by a badly configured stateful packet-filtering firewall, and there doesn't seem to be any way to just turn the damn thing off.
AT&T did one thing right: when you order a static IP address block, it is provisioned in addition to the single IP address already configured into the residential gateway and used by its internal NAT to support the built-in Voice Over IP adapter, WiFi base station and video control subsystem. I had assumed that my static allocation would replace whatever single address I had been assigned, and I might have to give up some to the set top box and VoIP adapter. But the RG still assigns addresses to the set top boxes from the private IP address block 192.168.1.0/24.
The RG still wastes three public IP addresses for other reasons, but at least it doesn't need any for those other devices.
Although the U-verse RG has some firewall configuration screens accessible to the user, they are poorly documented and limited in functionality. In particular, there is no way for the user to completely bypass the firewall. Even checking the helpful-sounding "Auto Firewall Open" flag on the "Edit Advanced Home Network Settings" page doesn't completely do the trick.
Nor is replacing the RG with something less broken an option. Its use is mandatory because AT&T uses 802.1x authentication on the VDSL2 link and the RG contains the necessary cryptographic credentials. [1a]
Even with the firewall supposedly turned off with respect to a block of static IP addresses, at least two TCP ports (445 and 139?) are still blocked. Sure, those two aren't a great loss, but this is at least partly a matter of principle; my computers are perfectly capable of dropping those packets all by themselves, thankyouverymuch. (Note: I don't run Windows.) Since attacks on port 445 are especially common (every few seconds), I want to use them in some host-based attack detection and mitigation experiments. I would also like to study various other kinds of attacks.
Certain types of ICMP traffic, particularly the Destination Unreachable - TTL Exceeded messages used by the traceroute command, are also completely blocked. This is a significant impediment to network debugging.
The RG's firewall is of the "connection tracking" or "stateful" type, so
any loss of connection state will cause even valid traffic to be discarded.
Several things can cause this to happen:
This last cause is especially common on UDP associations because a NAT or firewall cannot detect the end of a transaction by looking at the UDP header. Since UDP is heavily used for short transactions such as DNS look ups, short idle timeouts are necessary to keep large numbers of old UDP associations from piling up in the table. A stateful firewall can use a much longer idle timeout for TCP because it can easily detect the normal end of a TCP connection by observing the FIN bits in the TCP headers.
Yes, it is possible to keep a firewall from dropping live connections by sending periodic "keep alive" messages. But this has two problems: it creates useless Internet traffic, and it doesn't solve the problem for sleeping hosts. I see no reason I shouldn't be able to put my laptop or desktop to sleep, wake it up hours later, and still have all of my Internet connections. [2]
So I had to find a way for it to coexist with my existing ISC DHCP server on my local network. I did this by connecting my LAN to the RG through a small Linux system acting as an Ethernet bridge that, among other things, blocks DHCP packets. DHCP clients internal to the RG, or connected directly to it (or through its WiFi access point), receive addresses on the 192.168.1.0/24 network from the RG's DHCP server. At the moment, its only client is the U-verse set top box. Clients on the rest of the local network, on the other side of the Linux bridge, receive one of the public addresses in the AT&T-assigned static IP block and/or a private address in the 192.168.2.0/24 block that the RG does not manage.
Computers, once configured with IP addresses, can connect to either side of the bridge and operate properly. The bridge does not have to drop ARP because the RG ignores ARP activity involving addresses it doesn't control. However, my plan is to keep everything but the set top box on the other side of the Linux bridge from the RG so I can use the traffic control features in Linux to shape the traffic going to the Internet. This is important in keeping latency low for interactive traffic even when the link is operating near capacity with "scavenger" traffic like Bit Torrent.
Why would I want a host to have both a private IP address and a public one? Efficiency. Consider a large Linux server running lots of services. They're used from both the LAN and from the outside Internet, so clearly this system needs a public IP address. The server also handles a collection of Squeezeboxes, "dumb" network audio players on the local LAN. Obviously it would be a waste to assign precious public IP addresses to these devices. But the Squeezebox does not have a sophisticated IP router; it only knows the prefix of its local network and the IP address of the default router to be used to reach everything else. So if their server has only the single public IP address, then all traffic from the Squeezeboxes would be relayed through the default router -- the RG -- even though they're all on the same physical LAN. This puts unnecessary load on the RG, creates extra points of failure (such as the RG), and adds delay.
The simple and elegant solution to the problem is to give the Linux server two IP addresses: one public for external use and a private one for talking to the players.
But the RG has other plans. It detects statically configured hosts by broadcasting a near-continuous stream of ARP requests for every IP address it has ever seen, one every few seconds. [2a] Yet, inexplicably, it can take minutes for a new statically configured host to appear in the RG's network summary page.
If an ARP for a private IP address managed by the RG (i.e., in the 192.168.1.0/24 block) is answered by a MAC address that had previously answered an ARP for a public IP address, the private IP address replaces the public IP address in the entry in the RG's local device list, the RG slams its firewall shut on the public address and no longer routes packets to or from that address.
The fix for the Squeezebox (local client) problem is to give them addresses from another private IP address block, e.g., 192.168.2.0/24. Their server also needs a secondary address in this block. The RG seems to ignore all ARP exchanges involving these addresses since it does not manage them. You will have to either configure your clients manually (i.e., statically) with these addresses or set up a separate DHCP server to manage them. In the latter case you must isolate your DHCP server from the one in the RG by dropping DHCP packets in the bridge between it and the LAN with the DHCP server, Squeezebox clients and server. Also, if a client with an address on this second private block has to talk to the outside world you will have to provide your own NAT for it as the NAT in the RG only serves the 192.168.1.0/24 subnet. (Linux can easily be configured to provide both NAT and DHCP services.)
But when I try to restore maggie's public IP address 75.60.237.92 like this, watch what happens. The error message "For the public routed subinterface only WAN IP mapping is allowed" is clearly spurious. I'm doing exactly what it's telling me to do - I'm specifying an IP address on the WAN (wide area network). I find that I can only recover from this situation by clearing the RG's local host table and rebuilding it from scratch, host by host. This is quite simply unacceptable.
Why am I trying to use a private address in the 192.168.1.0/24 from a machine that already has a public IP address? To work around the traceroute bug mentioned earlier; only a traceroute from a private address through the RG's NAT will work. So working around one bug has revealed another...
You must be very careful even when you use separate Ethernet interfaces in the same machine for the different addresses. By default, Linux (and probably other operating systems) will answer an ARP for any of its IP addresses on any of its interfaces. Normally this is just fine -- but not in the U-verse universe. For example, if you configure eth0 with 75.60.237.89 and eth1 with 192.168.1.89 and then connect both interfaces to the RG, Linux might answer an ARP for 192.168.1.89 through eth0 - and this would cause the same problem as configuring both addresses on eth0. Fortunately, Linux has configuration flags to stop this; see the kernel documentation for
/proc/sys/net/ipv4/conf/*/arp_announce
/proc/sys/net/ipv4/conf/*/arp_ignore
/proc/sys/net/ipv4/conf/*/arp_filter
Among other things, this nasty RG misfeature keeps you from connecting a real IP router (e.g., a Cisco or a Linux system acting as a router) to the RG and having it proxy ARP for the entire public IP address block. I suppose this might still be possible if the router's Ethernet interface could be given a set of alternate MAC addresses so that the router can answer the RG's ARPs with a distinct MAC address for each public IP address. But I found it easier to configure the Linux system as a bridge and have it pass the actual MAC addresses of the hosts on the other side. Linux can still perform all the functions I need while bridging, such as packet filtering (specifically to block DHCP as discussed earlier) and traffic shaping.
At least the RG doesn't seem to care about Ethernet source addresses when it receives an IP packet to forward over the Internet. This means that a local host can loop its outbound traffic through another machine on its way to the U-verse RG. I've done this to set IP DSCP values, and it works. But Linux can optionally invoke iptables on bridged Ethernet traffic so it isn't actually necessary to explicitly route traffic in this manner.
Firewalls and NAT have greatly complicated life for every network administrator and seriously hampered experimentation, development and deployment of new Internet protocols and applications.
Because firewalls are usually controlled by network administrators, not users, they frequently block traffic that the intended recipients actually want. That's why they're so evil. Likewise, the NAT in the U-verse RG is especially evil because you can't replace it with one of your own, or do without one by connecting just a single computer and having it inherit the single IP public address assigned to you by default.
[1a]
It's hard to understand what makes AT&T so paranoid that they require authentication on a residential phone line.
Ordinary POTS telephones never had to authenticate themselves to the phone company.
Nearly every telephone company also uses PPPoE on regular DSL so that the user has to "log in" to the network before he can use it.
What's the point? If the user doesn't pay his bill they can always just disconnect his line and then it doesn't matter what he puts on his end of it.
[2]
What's so sad about all this is that simplicity, statelessness and end-to-end transparency
were once among the Internet's most powerful features.
They made it fast, flexible and reliable.
It was easy to
experiment at the edges of the network without having to beg
permission from anyone.
NAT and stateful firewalls reintroduce all of the
problems with alternative "connection-oriented" networks that the
Internet was designed to solve.