As with the previous two scenarios, in this scenario, a user complains of no Internet access from his workstation. This user has narrowed the issue down to a single website, http://www.google.com/. Upon further investigation, we find that this issue is affecting everyone in the organization—no one can access Google domains.
The network is configured as in the two prior scenarios, with a few simple switches and a single router connecting the network to the Internet.
Tapping into the Wire
In order to troubleshoot this issue, we first browse to http://www.google.com/ to generate traffic. Because this issue is network wide—meaning it’s also affecting your computer, and it could be the result of a massive malware infection—you shouldn’t sniff directly from your device. When you find yourself in a situation like this, a tap is the best solution, because it allows you to be completely passive after a brief interruption of service. The file resulting from the capture via a tap is nowebaccess3.pcap.
Analysis
This packet capture begins with DNS traffic instead of the ARP traffic we are used to seeing. Because the first packet in the capture is to an external address, and packet 2 contains a reply from that address, we can assume that the ARP process has already happened and the MAC-to-IP address mapping for our gateway router already exists in the host’s ARP cache at 172.16.0.8.
As shown in Figure 8-25, the first packet in the capture is from the host 172.16.0.8 to address 4.2.2.1 , and it’s a DNS packet . Examining the contents of the packet, we see that it is a query for the A record for www.google.com .
Figure 8-25: DNS query for www.google.com A record
The response to the query from 4.2.2.1 is the second packet in the capture file, as shown in Figure 8-26. Examining the Packet Details pane, we see that the name server that responded to this request provided multiple answers to the query . At this point, all looks well, and communication is occurring as it should.
Figure 8-26: DNS reply with multiple A records
Now that the user’s computer has determined the web server’s IP address, it can attempt to communicate with the server. As shown in Figure 8-27, this process is initiated in packet 3, with a TCP packet sent from 172.16.0.8 to
74.125.95.105 . This destination address comes from the first A record provided in the DNS query response seen in packet 2. The TCP packet has the SYN flag set , and it’s attempting to communicate with the remote server on port 80 .
Figure 8-27: SYN packet attempting to initiate a connection on port 80
Because this is a TCP handshake process, we know that we should see a TCP SYN/ACK packet sent in response, but instead, after a short time, another SYN packet is sent from the source to the destination. This process occurs once more after approximately a second, as shown in Figure 8-28, at which point communication stops and the browser reports that the website could not be found.
Figure 8-28: The TCP SYN packet is attempted three times with no response received.
As we troubleshoot this scenario, we consider that we know that the workstation within our network can connect to the outside world because the DNS query to our external DNS server at 4.2.2.1 is successful. The DNS server responds with what appears to be a valid address, and our hosts attempt to connect to one of those addresses. Also, the local workstation we are attempting to connect from appears to be functioning.
The problem is that the remote server simply isn’t responding to our connection requests; a TCP RST packet is not sent. This might occur for several reasons: a misconfigured web server, a corrupted protocol stack on the web server, or a packet-filtering device on the remote network (such as a firewall). Assuming there is no local packet filtering device in place, all of the other potential solutions are on the remote network and beyond our control. In this case, the web server was not functioning correctly, and no attempt to access it succeeded. Once the problem was corrected on Google’s end, communication was able to proceed.
Lessons Learned
In this scenario, the problem was not one that we could correct. Our analysis determined that the issue was not with the hosts on our network, our router, or the external DNS server providing us with name resolution services. The issue lay outside our network infrastructure.
Sometimes discovering that a problem isn’t really ours can not only relieve stress, but also save face when management comes knocking. I have fought with many ISPs, vendors, and software companies who claim that an issue is
not their fault, but as you’ve just seen, packets don’t lie.