TCP stack sometimes loses ACKs ... or something
From: Neil Brown
Date: Tue Nov 07 2006 - 01:37:04 EST
I upgraded my notebook from 2.6.16 to 2.6.18 recently and noticed that
I couldn't talk to my VOIP device (which has a WEB interface).
Watching traffic I see the three-way-handshake working perfectly, and
then the first data packet is sent (a partial HTTP request:
GET / HTTP/1.1 ....) and an ACK comes back from the device.
Then the next data packet (remainder of the HTTP request) is sent, but
tcpdump never sees the ACK, nor does the TCP stack. So the data gets
recent repeatedly. No ack. Ever.
With 2.6.16, The ack comes back just fine and the connection proceeds
as you would expect.
As it was a very reproducible problem I decided to try "git bisect"
bad: [7b4f4b5ebceab67ce440a61081a69f0265e17c2a] [TCP]: Set default max buffers from memory pool size
I double checked as this seemed a fairly unlikely patch to cause the
problem, but this definitely is it.
The net effect of this patch is to change the last of the three
from well below 2^20 to well above. 2^20 seems to be a significant
number. I set tcp_wmem to that and the ACK was lost. I set it to
one less and the first ACK (at least) was accepted.
I ended up setting both r and w to 100000 and everything is fine.
Exploring more deeply, and comparing:
- a failing connection (to VIOP box, [rw]mem large)
- a working connection to VOIP box ([rw]mem small)
- a working connection to another machine ([rw]mem irrelevant).
The VIOP returns MSS=1360 in the SYN/ACK packet. Other machine
The ack that is getting lost contains data as well as the
ACK. i.e. the same packet that ACKs at the TCP level includes the
HTTP level reply.
The matching ACK from the other machine (some Linux 2.6.8 I think)
is a data-less ACK followed very quickly by the HTTP reply in
a separate packet.
The 'Timestamps' option coming back from the VOIP box is a little
odd. The Timestamp in the SYN/ACK is the same as the timestamp in
the next ACK (the ack for the first partial HTTP request).
The Timestamp in the next packet which is the one that gets lost has
exactly the same TSval as previous packets, and TSecr is one more
than in the previous packet.
I assume that one (or more) of these differences combined with the
large tcp_[rw]mem value cause the packet loss, but I have no idea
I can make the tcp traces available if needed, but these are really
the only non-trivial differences.
I'm willing to test patches.
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/