Re: question about 3sec timeouts with tcp

[H. Willstrand]

On Tue, Apr 1, 2008 at 8:43 PM, Gabriel Barazer <gabriel@xxxxxxxx> wrote:
> On 04/01/2008 8:28:14 PM +0200, "H. Willstrand" <h.willstrand@xxxxxxxxx>
>
>
> wrote:
>  > On Tue, Apr 1, 2008 at 7:59 PM, Gabriel Barazer <gabriel@xxxxxxxx> wrote:
>  >> On 04/01/2008 7:17:31 PM +0200, Leo <neleo@xxxxxxx> wrote:
>  >>  > H. Willstrand wrote:
>  >>  >> On Tue, Apr 1, 2008 at 5:43 PM, Gabriel Barazer <gabriel@xxxxxxxx> wrote:
>  >>  >>
>  >>  >>> On 04/01/2008 4:43:20 PM +0200, Brett Paden <paden@xxxxxxxxxxxx> wrote:
>  >>  >>>  >> If I'm right Brett's problem relays in the test client (provided in
>  >>  >>>  >> the first mail). This has probably to do with the number of ports
>  >>  >>>  >> opened and closed during a short time period.
>  >>  >>>  >
>  >>  >>>  > My test client is designed to simulate the sort of load our
>  >>  >>> production
>  >>  >>>  > databases and web servers see.  We're talking on the order of 100-400
>  >>  >>>  > connections per second.  On an unloaded server the 3000ms occur right
>  >>  >>>  > around 400 connections a second but we have seen them a lower
>  >>  >>> connection
>  >>  >>>  > rates.  Are you suggesting that we could do something simple (like
>  >>  >>> reap
>  >>  >>>  > TIME_WAIT connections) to allevaite the problem?
>  >>  >>>
>  >>  >>>  Using tcp_tw_recycle / tcp_tw_reuse doesn't solve the problem either on
>  >>  >>>  the client nor on the server. I tested with and without these options
>  >>  >>>  enabled, disabled netfilter's connection tracking and none solved this
>  >>  >>>  delay. If even the "lo" interface is concerned, there is definitely
>  >>  >>>  something into the network stack and not the device drivers.
>  >>  >>>
>  >>  >>>  Here is a thread I started on LKML about this very same bug.
>  >>  >>>  http://lkml.org/lkml/2008/3/14/353
>  >>  >>>  There is a forum thread with french hosting providers talking about it.
>  >>  >>>  (if some of you read french:
>  >>  >>>  http://www.webmasterclub.fr/forum/topic,59486,0.html)
>  >>  >>>
>  >>  >>>  We are far from being alone!
>  >>  >>>
>  >>  > Welcome to the club, Gabriel!
>  >>  >>>  Gabriel
>  >>
>  >>  How lucky I am!
>  >>  I suspect there are many other people having this problem out there,
>  >>  they just don't notice these delays on small infrastructures and because
>  >>  this bug doesn't actually cause a connection error, but "only" an
>  >>  unacceptable delay for moderate to high busy servers.
>  >>
>  >>
>  >>  >> Ok, seams to be the same issue that Leo has (has nothing to do with
>  >>  >> the Brett / Marlon issue, only common dominator is the 3000ms).
>  >>  >>
>  >>  > But Gabriel is also talking about 3 second timeouts on the client as
>  >>  > Brett and I did. I have read Gabriel's  description on the provided link
>  >>  > and it seems to be exactly the same problem. I think Brett can confirm
>  >>  > this ...
>  >>  >> This issue is probably caused by server delivering as miscalculated
>  >>  >> SYN/ACK (the acked number is miscalculated, see my second mail).
>  >>  >>
>  >>  > When you look at my first tcpdump with two machines as server and client
>  >>  > then you can see that there are no miscalculated SYN/ACK packets from
>  >>  > the server (and therefore no RST packet from the client). All packets
>  >>  > have the right number but the client never receives the SYN/ACK packet
>  >>  > from the server. Only at the lo test there are RST packets and wrong
>  >>  > packet numbers. But as I told you in my last email I think this is a
>  >>  > different problem and not important for us. We should ignore the lo test
>  >>  > and concentrate on the "real" problem of Brett, Gabriel and myself  (and
>  >>  > even a lot of other people out there).
>  >>
>  >>  I confirm that there is no problem is the sequence numbers. Attached is
>  >>  the pcap compatible capture of the relevant packets (608 bytes, 6
>  >>  packets total: 2 for the failed handshake, 3 for the successful one and
>  >>  1 for the first mysql data packet). This capture has been filtered to
>  >>  show only the relevant packets and done in promiscuous mode.
>  >>
>  >
>
> > I'm missing the tcpdump...
>
>  Sorry, I forgot to include it when reformatting my e-mail. Here it is!
>
>  Gabriel
>

The packages are OK.
Still, how did you produce this situation? Let me guess, you used one
client to mass produce connections to your mysql-server, right?

//HW
--
To unsubscribe from this list: send the line "unsubscribe linux-net" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html