Temporary U-verse Connectivity Loss During Wired/Wireless Connection Switching

We have both wireless (802.11n WiFi) and wired (802.3 Gigabit Ethernet) networks in our house and it's very convenient to switch between them as we move around. But this often results in a loss of connectivity with the outside world for some minutes, and I've traced the cause to protocol violations (i.e., bugs) in the current firmware (version 6.1.9.24-enh.tm) in the 2Wire 3800 HGV-B U-verse Residential Gateway (RG) between our home LAN and AT&T's network.

Our WiFi base stations (separate from the one inside the RG) operate in "bridging" mode so that our wired and wireless networks form a single logical LAN. My laptop is a MacBook Pro with both wired Ethernet (en0) and 802.11n WiFi (en1) interfaces, each with its own 48-bit Ethernet MAC address, and I statically configure both en0 and en1 with the same IP address (75.60.237.92) from the block of 8 static IP addresses assigned by AT&T. Using the same IP address for both interfaces is perfectly legal by the applicable Internet protocol standards, and this lets me keep my network connections open when I unplug my laptop from the Ethernet cable next to the living room couch and switch to the WiFi network. Any connections my MacBook might have with the other computers at home do in fact stay up through the transfer, but it loses connectivity with the outside world for some minutes. It can't even ping the RG! Eventually (after 10-15 minutes) the RG finally responds and connectivity returns.

This packet trace demonstrates the problem. I took it on the MacBook's Ethernet interface with 'tcpdump'; non-pertinent traffic was removed for clarity. The WiFi interface was turned off. As the trace begins, the MacBook (75.60.237.92) is pinging the RG (75.60.237.94) every second with ICMP echo request packets that are going unanswered:

20:54:37.468608 00:23:32:9c:52:7e > 00:25:3c:72:68:29, ethertype IPv4 (0x0800), length 98: 75.60.237.92 > 75.60.237.94: ICMP echo request, id 55815, seq 315, length 64
20:54:38.468728 00:23:32:9c:52:7e > 00:25:3c:72:68:29, ethertype IPv4 (0x0800), length 98: 75.60.237.92 > 75.60.237.94: ICMP echo request, id 55815, seq 316, length 64
20:54:39.468851 00:23:32:9c:52:7e > 00:25:3c:72:68:29, ethertype IPv4 (0x0800), length 98: 75.60.237.92 > 75.60.237.94: ICMP echo request, id 55815, seq 317, length 64

As usual, the RG continually polls various local IP addresses with the Address Resolution Protocol (ARP) in violation of the spirit, if not the exact specifications of that protocol. The RG's second ARP targets us and we dutifully respond:

20:54:39.860895 00:25:3c:72:68:29 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 75.60.237.89 (ff:ff:ff:ff:ff:ff) tell 75.60.237.94, length 46
20:54:39.867043 00:25:3c:72:68:29 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 75.60.237.92 (ff:ff:ff:ff:ff:ff) tell 75.60.237.94, length 46
20:54:39.867093 00:23:32:9c:52:7e > 00:25:3c:72:68:29, ethertype ARP (0x0806), length 42: Reply 75.60.237.92 is-at 00:23:32:9c:52:7e, length 28
20:54:39.873462 00:25:3c:72:68:29 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.1.65 (ff:ff:ff:ff:ff:ff) tell 192.168.1.254, length 46
20:54:39.898581 00:25:3c:72:68:29 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 75.60.237.91 (ff:ff:ff:ff:ff:ff) tell 75.60.237.94, length 46
20:54:39.904301 00:25:3c:72:68:29 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.1.66 (ff:ff:ff:ff:ff:ff) tell 192.168.1.254, length 46

Yet the RG still doesn't answer our pings!

20:54:40.468974 00:23:32:9c:52:7e > 00:25:3c:72:68:29, ethertype IPv4 (0x0800), length 98: 75.60.237.92 > 75.60.237.94: ICMP echo request, id 55815, seq 318, length 64

This continues for some time. (Note that we don't see any ARP responses from other targets because of unicast filtering in my Ethernet switch. This is normal.)

20:54:40.905637 00:25:3c:72:68:29 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.1.66 (ff:ff:ff:ff:ff:ff) tell 192.168.1.254, length 46
20:54:40.906187 00:25:3c:72:68:29 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 75.60.237.90 (ff:ff:ff:ff:ff:ff) tell 75.60.237.94, length 46
20:54:41.469094 00:23:32:9c:52:7e > 00:25:3c:72:68:29, ethertype IPv4 (0x0800), length 98: 75.60.237.92 > 75.60.237.94: ICMP echo request, id 55815, seq 319, length 64
20:54:41.906881 00:25:3c:72:68:29 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.1.67 (ff:ff:ff:ff:ff:ff) tell 192.168.1.254, length 46
20:54:41.907642 00:25:3c:72:68:29 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 75.60.237.90 (ff:ff:ff:ff:ff:ff) tell 75.60.237.94, length 46
20:54:42.469163 00:23:32:9c:52:7e > 00:25:3c:72:68:29, ethertype IPv4 (0x0800), length 98: 75.60.237.92 > 75.60.237.94: ICMP echo request, id 55815, seq 320, length 64
20:54:42.907876 00:25:3c:72:68:29 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 99.71.137.120 (ff:ff:ff:ff:ff:ff) tell 99.71.136.1, length 46
20:54:42.908617 00:25:3c:72:68:29 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.1.67 (ff:ff:ff:ff:ff:ff) tell 192.168.1.254, length 46
20:54:43.469274 00:23:32:9c:52:7e > 00:25:3c:72:68:29, ethertype IPv4 (0x0800), length 98: 75.60.237.92 > 75.60.237.94: ICMP echo request, id 55815, seq 321, length 64
20:54:43.910153 00:25:3c:72:68:29 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 99.71.137.120 (ff:ff:ff:ff:ff:ff) tell 99.71.136.1, length 46

The RG again ARPs for the MacBook, and we reply:

20:54:43.910688 00:25:3c:72:68:29 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 75.60.237.92 (ff:ff:ff:ff:ff:ff) tell 75.60.237.94, length 46
20:54:43.910718 00:23:32:9c:52:7e > 00:25:3c:72:68:29, ethertype ARP (0x0806), length 42: Reply 75.60.237.92 is-at 00:23:32:9c:52:7e, length 28

Yet the RG still ignores our pings:

20:54:44.469387 00:23:32:9c:52:7e > 00:25:3c:72:68:29, ethertype IPv4 (0x0800), length 98: 75.60.237.92 > 75.60.237.94: ICMP echo request, id 55815, seq 322, length 64
20:54:45.469495 00:23:32:9c:52:7e > 00:25:3c:72:68:29, ethertype IPv4 (0x0800), length 98: 75.60.237.92 > 75.60.237.94: ICMP echo request, id 55815, seq 323, length 64
20:54:46.469617 00:23:32:9c:52:7e > 00:25:3c:72:68:29, ethertype IPv4 (0x0800), length 98: 75.60.237.92 > 75.60.237.94: ICMP echo request, id 55815, seq 324, length 64
20:54:47.469741 00:23:32:9c:52:7e > 00:25:3c:72:68:29, ethertype IPv4 (0x0800), length 98: 75.60.237.92 > 75.60.237.94: ICMP echo request, id 55815, seq 325, length 64
20:54:48.469847 00:23:32:9c:52:7e > 00:25:3c:72:68:29, ethertype IPv4 (0x0800), length 98: 75.60.237.92 > 75.60.237.94: ICMP echo request, id 55815, seq 326, length 64
20:54:49.470018 00:23:32:9c:52:7e > 00:25:3c:72:68:29, ethertype IPv4 (0x0800), length 98: 75.60.237.92 > 75.60.237.94: ICMP echo request, id 55815, seq 327, length 64
20:54:50.470134 00:23:32:9c:52:7e > 00:25:3c:72:68:29, ethertype IPv4 (0x0800), length 98: 75.60.237.92 > 75.60.237.94: ICMP echo request, id 55815, seq 328, length 64
20:54:51.470248 00:23:32:9c:52:7e > 00:25:3c:72:68:29, ethertype IPv4 (0x0800), length 98: 75.60.237.92 > 75.60.237.94: ICMP echo request, id 55815, seq 329, length 64
20:54:52.470378 00:23:32:9c:52:7e > 00:25:3c:72:68:29, ethertype IPv4 (0x0800), length 98: 75.60.237.92 > 75.60.237.94: ICMP echo request, id 55815, seq 330, length 64
20:54:53.470450 00:23:32:9c:52:7e > 00:25:3c:72:68:29, ethertype IPv4 (0x0800), length 98: 75.60.237.92 > 75.60.237.94: ICMP echo request, id 55815, seq 331, length 64
20:54:54.470556 00:23:32:9c:52:7e > 00:25:3c:72:68:29, ethertype IPv4 (0x0800), length 98: 75.60.237.92 > 75.60.237.94: ICMP echo request, id 55815, seq 332, length 64
20:54:55.127955 00:25:3c:72:68:29 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 192.168.1.65 (ff:ff:ff:ff:ff:ff) tell 192.168.1.254, length 46
20:54:55.470667 00:23:32:9c:52:7e > 00:25:3c:72:68:29, ethertype IPv4 (0x0800), length 98: 75.60.237.92 > 75.60.237.94: ICMP echo request, id 55815, seq 333, length 64

The RG ARPs for the MacBook yet again...

20:54:55.471426 00:25:3c:72:68:29 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 75.60.237.92 (ff:ff:ff:ff:ff:ff) tell 75.60.237.94, length 46
20:54:55.471483 00:23:32:9c:52:7e > 00:25:3c:72:68:29, ethertype ARP (0x0806), length 42: Reply 75.60.237.92 is-at 00:23:32:9c:52:7e, length 28

Hallelujah! The RG finally answers!

20:54:55.472270 00:25:3c:72:68:29 > 00:23:32:9c:52:7e, ethertype IPv4 (0x0800), length 98: 75.60.237.94 > 75.60.237.92: ICMP echo reply, id 55815, seq 333, length 64
20:54:56.040593 00:25:3c:72:68:29 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has 75.60.237.89 (ff:ff:ff:ff:ff:ff) tell 75.60.237.94, length 46
20:54:56.470774 00:23:32:9c:52:7e > 00:25:3c:72:68:29, ethertype IPv4 (0x0800), length 98: 75.60.237.92 > 75.60.237.94: ICMP echo request, id 55815, seq 334, length 64
20:54:56.471290 00:25:3c:72:68:29 > 00:23:32:9c:52:7e, ethertype IPv4 (0x0800), length 98: 75.60.237.94 > 75.60.237.92: ICMP echo reply, id 55815, seq 334, length 64
20:54:57.470885 00:23:32:9c:52:7e > 00:25:3c:72:68:29, ethertype IPv4 (0x0800), length 98: 75.60.237.92 > 75.60.237.94: ICMP echo request, id 55815, seq 335, length 64
20:54:57.471395 00:25:3c:72:68:29 > 00:23:32:9c:52:7e, ethertype IPv4 (0x0800), length 98: 75.60.237.94 > 75.60.237.92: ICMP echo reply, id 55815, seq 335, length 64
20:54:58.470996 00:23:32:9c:52:7e > 00:25:3c:72:68:29, ethertype IPv4 (0x0800), length 98: 75.60.237.92 > 75.60.237.94: ICMP echo request, id 55815, seq 336, length 64
20:54:58.471582 00:25:3c:72:68:29 > 00:23:32:9c:52:7e, ethertype IPv4 (0x0800), length 98: 75.60.237.94 > 75.60.237.92: ICMP echo reply, id 55815, seq 336, length 64
20:54:59.471110 00:23:32:9c:52:7e > 00:25:3c:72:68:29, ethertype IPv4 (0x0800), length 98: 75.60.237.92 > 75.60.237.94: ICMP echo request, id 55815, seq 337, length 64
20:54:59.471619 00:25:3c:72:68:29 > 00:23:32:9c:52:7e, ethertype IPv4 (0x0800), length 98: 75.60.237.94 > 75.60.237.92: ICMP echo reply, id 55815, seq 337, length 64

ARP Discussion

In this section I explain the Address Resolution Protocol (ARP) so I can show how and why it's legal to configure my laptop as I do, how this keeps my connections to local computers going when I switch between wired Ethernet and WiFi, and why it should also do so for remote connections through the RG.

The Address Resolution Protocol (ARP) defined in RFC-826 is almost universally used in the Internet wherever it is necessary to map a 32-bit Internet Protocol (IP) address to the addresses used in a specific local area network (LAN), provided that network supports broadcasting. Ethernet, which uses 48-bit "MAC" (Medium Access Controller) addresses, is by far the most common LAN but there are others.

An ARP request is sent as an Ethernet broadcast that specifies the IP address whose Ethernet MAC address is being sought. Because broadcasts have to be processed by everyone on the network, it is highly desirable to minimize them and ARP includes several provisions to this end.

Hosts must cache IP/Ethernet address pairs for a minimum of 15 minutes.

Nearly all Internet traffic is bidirectional, so an ARP request includes the sender's own IP and Ethernet addresses so the target can learn them without having to send a request of its own. This cuts ARP traffic essentially in half.

Everyone, not just the listed target, must examine every ARP request. If the sender's IP and Ethernet addresses are already in the local ARP cache, then the entry is refreshed for another 15 minutes. (If the local cache has no existing entry for the sender, it is not required to create one because it is probably not communicating with this host anyway.) Should the ARP request specify a different Ethernet address than that in the local cache, the entry must be updated and refreshed for 15 minutes.

That last provision dealing with a change in Ethernet MAC address is directly relevant to the RG bug discussed in this article. ARP explicitly allows an Internet host to change the Ethernet address associated with an IP address, and that's what switching between Ethernet and WiFi does. So my laptop need broadcast only a single ARP request (for any target) with this new Ethernet address for every node on the network to update its cache. That, along with the auto-learning property of Ethernet switches, is why my connections (should) stay up when I move between wired Ethernet and WiFi. But the packet trace discussed here clearly shows that the RG, for some reason, does not adhere to the ARP specification, making it unable to stay connected with my laptop through the switchover.

[email protected]

Last modified: Sat Oct 2 00:19:17 PDT 2010