Re: question about 3sec timeouts with tcp

From: H. Willstrand
Date: Mon Mar 31 2008 - 07:10:10 EST

On Mon, Mar 31, 2008 at 12:50 PM, Leo <neleo@xxxxxxx> wrote:
> Here comes the tcpdump (the server has IP 192.168.1.1 listening on port
> 3306 and the client has IP 192.168.1.99).
>
> tcpdump on the server:
>
> 11:24:44.226373 IP 192.168.1.99.39028 > 192.168.1.1.3306: S
> 3068401272:3068401272(0) win 5840 <mss 1460,sackOK,timestamp 10592352
> 0,nop,wscal e 6>
> 11:24:44.226389 IP 192.168.1.1.3306 > 192.168.1.99.39028: S
> 3052139931:3052139931(0) ack 3068401273 win 5792 <mss
> 1460,sackOK,timestamp 10478 87 10592352,nop,wscale 7>
> 11:24:47.226673 IP 192.168.1.99.39028 > 192.168.1.1.3306: S
> 3068401272:3068401272(0) win 5840 <mss 1460,sackOK,timestamp 10595352
> 0,nop,wscale 6>
> 11:24:47.226715 IP 192.168.1.1.3306 > 192.168.1.99.39028: S
> 3052139931:3052139931(0) ack 3068401273 win 5792 <mss
> 1460,sackOK,timestamp 1048187 10592352,nop,wscale 7>
> 11:24:47.226957 IP 192.168.1.99.39028 > 192.168.1.1.3306: . ack 1 win 92
> <nop,nop,timestamp 10595352 1048187>
> 11:24:47.227488 IP 192.168.1.99.39028 > 192.168.1.1.3306: F 1:1(0) ack 1
> win 92 <nop,nop,timestamp 10595352 1048187>
> 11:24:47.227580 IP 192.168.1.1.3306 > 192.168.1.99.39028: F 1:1(0) ack 2
> win 46 <nop,nop,timestamp 1048187 10595352>
> 11:24:47.227810 IP 192.168.1.99.39028 > 192.168.1.1.3306: . ack 2 win 92
> <nop,nop,timestamp 10595353 1048187>
>
> tcpdump on the client:
>
> 11:24:44.045735 IP 192.168.1.99.39028 > 192.168.1.1.3306: S
> 3068401272:3068401272(0) win 5840 <mss 1460,sackOK,timestamp 10592352
> 0,nop,wscale 6>
> 11:24:47.045710 IP 192.168.1.99.39028 > 192.168.1.1.3306: S
> 3068401272:3068401272(0) win 5840 <mss 1460,sackOK,timestamp 10595352
> 0,nop,wscale 6>
> 11:24:47.045911 IP 192.168.1.1.3306 > 192.168.1.99.39028: S
> 3052139931:3052139931(0) ack 3068401273 win 5792 <mss
> 1460,sackOK,timestamp 1048187 10592352,nop,wscale 7>
> 11:24:47.046015 IP 192.168.1.99.39028 > 192.168.1.1.3306: . ack 1 win 92
> <nop,nop,timestamp 10595352 1048187>
> 11:24:47.046541 IP 192.168.1.99.39028 > 192.168.1.1.3306: F 1:1(0) ack 1
> win 92 <nop,nop,timestamp 10595352 1048187>
> 11:24:47.046845 IP 192.168.1.1.3306 > 192.168.1.99.39028: F 1:1(0) ack 2
> win 46 <nop,nop,timestamp 1048187 10595352>
> 11:24:47.046880 IP 192.168.1.99.39028 > 192.168.1.1.3306: . ack 2 win 92
> <nop,nop,timestamp 10595353 1048187>
>
> I can also provide a strace if required (alternatively you can find the
> tcpdump and the strace in my original posting, e.g.
> http://marc.info/?l=linux-net&m=120214156112338&w=2).
>
> Kind regards,
> Leo
>

Well, the server sends back the SYN/ACK. Your client is not receiving it...
Normally I will say a network problem (switch / router), please check
that once more.

(There is / was a bug in PCAP before which lost packages in tcpdump,
but this seams not be the case here)

Br H.Willstrand


>
>
>
> H. Willstrand wrote:
> > On Mon, Mar 31, 2008 at 11:33 AM, Leo <neleo@xxxxxxx> wrote:
> >
> >> Yes, we have changed the back_log settings in my.cnf. We also tried
> >> various combinations of the kernel parameters you mentioned in your
> >> email - without success.
> >>
> >> Have you made any changes on the client (=webserver)? Perhaps it's not a
> >> problem of the server (=mysql). As I figured out in my original posting
> >> it looks as if the client didn't receive the SYN/ACK package from the
> >> server (keep in mind that it is reproducable with lo interface,
> >> therefore it's not a hardware problem with cables, switches, ...):
> >>
> >> "When you look at the TCP pakets transmitted between server and client
> >> (tcpdump) you will see that the client sends a SYN and the server
> >> answers with a SYN/ACK. However the client will never receive the
> >> SYN/ACK. Therefore the client resends a SYN after exactly 3 seconds. The
> >> server answers with a SYN/ACK again and this time the client receives
> >> the paket. Now the connection is working."
> >>
> >>
> >
> > Hi!
> >
> > Can you please post that tcpdump? (Just to check the first ACK).
> >
> > Br H.Willstrand
> >
> >
> >> On friday I received an email from another user having the same problem.
> >> As I mentioned in my last email: we are not alone ;-)
> >>
> >> Many thanks for your help.
> >>
> >> Kind regards,
> >> Leo
> >>
> >>
> >>
> >> Marlon de Boer wrote:
> >> > Leo wrote:
> >> >
> >> >> We discovered the same problem two month ago (even 3 sec connect
> >> >> problems between webserver and mysql backend). I have posted it to
> >> >> the list (subject "TCP connect hangs for 3 seconds", 2008-02-04)
> >> >> but didn't get any answer yet. When I read your mail it was a
> >> >> flicker of hope.
> >> >>
> >> >> This is not a mysql issue! You can easily reproduce it with a few
> >> >> lines of C code (see below, it's similar to your program) running
> >> >> against any open port on the server (e.g. netcat: nc -k -l
> >> >> <PORTNUMBER>). We have also tested it on different hardware and
> >> >> kernel versions (up to 2.6.24.4) and we can deliver further
> >> >> information if requested ...
> >> >>
> >> >> Unfortunately the kernel parameter you mentioned in your second
> >> >> mail (net.core.somaxconn) didn't solve the problem.
> >> >>
> >> >> I think there are many other people out there having this problem
> >> >> unconsciously. Can anybody help?
> >> >>
> >> >>
> >> > I used the c code below to test the server side. After playing with
> >> > the kernel settings, some code,
> >> > setsockopt(sd, SOL_SOCKET, SO_REUSEADDR, &opt, sizeof(opt)) and
> >> > listen(sd, 8192) the timeouts disappeared.
> >> >
> >> > Did you changed the settings in mysql.cnf? back_log = 8192, and
> >> > restarted the mysql daemon? Else try to raise it to 16384 or even higher.
> >> >
> >> > Regards
> >> > Marlon
> >> >
> >> >
> >>
> >>
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe linux-net" in
> >> the body of a message to majordomo@xxxxxxxxxxxxxxx
> >> More majordomo info at http://vger.kernel.org/majordomo-info.html
> >>
> >>
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-net" in
> > the body of a message to majordomo@xxxxxxxxxxxxxxx
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
> >
> >
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-net" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html