Troubleshooting TCP Connection Problem with Wireshark

TCP Connection Problem

Previously, we discussed the fundamentals of TCP. In this section, we’ll focus on troubleshooting TCP connection problem using packet capture examples. When TCP communication occurs, a connection is opened, data is transferred, and the connection is closed. However, issues can arise during the connection setup or teardown, causing network disruptions. This article explores how to identify and resolve these TCP connection problems using Wireshark network packet captures.

More Information

Common Manifestations of TCP Connection Problem:

The problem may manifest itself in several ways:

• Tried to run an application but it didn’t work. Tried to browse the web but it didn’t respond.
• Tried to send mail but could not connect to the mail server.
• The problem could be caused by something as simple as the server being down, the application not being run on the server, or a network disconnect somewhere between the client and the server.
• The problem may also be caused by complex reasons, such as DNS problems, insufficient server memory and inability to connect (for example, an application occupies a high memory space), duplicate IP, and other reasons.

Methods for Troubleshooting TCP Connection Problem:

The following sections describe clues to solving the problem and how to diagnose TCP connection problems by capturing packets. Usually, these problems will result in no results when running the application.

When you are running an application, such as a database client, mail client, watching a video, etc., and cannot get output, follow these steps to diagnose:

1. Verify that the server and application are running.
2. Verify that the client is running, has an IP address configured (either manually or via DHCP), and is connected to the network.
3. Ping the server and confirm the connection is working.
4. In some cases, the server cannot be pinged but the connection is normal. This is because the firewall intercepts ICMP information, so if it cannot be pinged, it does not necessarily mean that there is a problem with the connection. The firewall may be a dedicated device in the network or a firewall installed on the Windows/Linux/UNIX terminal device.

5. In the captured file , look for the following pattern:

• Triple SYN message with no response (see screenshot below)
• SYN message with a reset (RST) response

In both cases it is possible that the firewall is blocking the specific application or that the application is not running.

The following screenshot is a simple case: the client cannot connect to the web server 81.218.31.171 (packets 61, 62 and 63). It may be that it is not allowed by the firewall, or the server is faulty. You can see that the connection to another site 108.160.163.43 (packets 65, 66 and 67) is normal, so the connection problem is limited to 81.218.31.171.

The following example is a relatively complex case. In this case, the client wants to log in to the camera server to access the camera at a remote site. The IP address of the camera server is 135.82.12.1. The problem is that the client can see the login window on the server homepage, but cannot log in to the system. In the screenshot below, you can see that a connection is open to the IP address 135.82.12.1. The TCP connection to the HTTP server is open, and there seems to be no connection problems at first:

When we filter out the data stream with the destination IP address 135.82.12.1, which is the camera server, we can see that when trying to connect to TCP port 6036, we get a RST/ACK response. There are the following possibilities:

• Firewall blocks port 6036
• If Port Address Translation (PAT) is configured , only port 80 is translated, not 6036
• Username and password verification is done on TCP port 6036. The firewall only allows port 80, so the verification is blocked and the application does not work.

In short, when you cannot connect to the server normally, check whether all TCP/UDP ports of the server and client can be forwarded through the network, and whether there are any unknown ports.

Working process:

When a TCP connection is started, the following three steps occur:

1. The client TCP process sends a SYN message. The SYN flag in this message is set to 1. In this message, the client:

• Specify your own initial sequence number. This is the first byte the client sends to the server.
• Indicates its own window size. This is the size of the cache (located in the client’s RAM) that the client allocates to the process.
• Set the options you want to use: MSS, Selective ACK, etc.

2. When the server receives the connection request, the server:

• Send SYN/ACK to the client to confirm receipt of the SYN request.
• Indicates the initial sequence number of the server. This is the first byte sent by the server to the client.
• Indicates the server’s window size. This is the size of the cache (in server RAM) that the server allocates to the process.
• Reply request options and set server side options.

3. Upon receiving the server’s SYN/ACK, the client:

• Send an ACK message to the server to confirm that the SYN/ACK is received from the server.
• Indicates the client window size. Although this parameter is defined in the first message, the server will still refer to this value because it is the latest window size.

In the option field of the TCP header, there are the following main options:

• Maximum Segment Size (MSS): The maximum number of bytes in a TCP datagram, that is, the number of bytes from the TCP header to the end of the message.
• Windows Scale Option (WSopt): This factor is multiplied by the Window Size field in the TCP header to notify the receiver to expand the buffer. Since the maximum window size in the header is 64KB, multiplying it by a factor of 4 results in a window size of 256KB.
• SACK: Selective ACK, this option enables both parties to confirm only the specified message. When a single message is lost, only this message will be retransmitted. When the connection is established, both parties need to agree to SACK.
• Timestamps Option (TSopt): This parameter refers to the delay between the client and the server.

During this phase, both parties:

• Agree to establish connection
• Know the other party’s initial sequence number
• Knowing the other party’s window size
  When establishing a connection, except for the three-way handshake signal, all other signals indicate problems, including SYN without response, SYN/ACK after SYN but no ACK at the end, SYN response with RST, etc.

Summarize:

• If the SYN message receives a reply RST, check the firewall that blocked the port number.
• Three SYNs without any reply is either due to the application not responding or due to a firewall blocking requests on a specific port.
• Always remember to check for NAT , port forwarding , and other mechanisms involving TCP and UDP ports that may disrupt normal TCP operation.