Re: tcp_fastretrans_alert since 2.6.32 ?

[Eric Dumazet]

CC netdev and Ilpo

Le mardi 21 septembre 2010 Ã 23:37 -0700, Sunny a Ãcrit :
> Hi,
> 
> After upgrading to the latest ubuntu kernel, I started to have this
> alert showing up, this is a vm running on ESXi host with vmxnet3
> adapter, mainly serving large traffic ~70Mbps towards overboard
> customers, so I expect packet loss and retrans is pretty high. So I'm
> posting to ask whether others been seeing the same issue? When this
> happens, machine freeze several seconds for about three times in a row
> then seems to start working again.
> 
> Please cc me on the thread, thanks in advance.
> 
> 
> [212670.327617] ------------[ cut here ]------------
> [212670.327689] WARNING: at
> /build/buildd/linux-2.6.32/net/ipv4/tcp_input.c:2919 tcp_fastret
> rans_alert+0x3f9/0xd90()
> [212670.327692] Hardware name: VMware Virtual Platform
> [212670.327693] Modules linked in: btrfs zlib_deflate crc32c libcrc32c
> ufs qnx4 hfsplus hfs  minix ntfs vfat msdos fat jfs xfs exportfs
> reiserfs sch_sfq cls_u32 sch_htb nf_conntrack_ipv 6 ip6table_filter
> ip6_tables xt_tcpudp xt_state xt_multiport iptable_filter iptable_nat
> nf_n at nf_conntrack_ipv4 sit nf_conntrack nf_defrag_ipv4 tunnel4
> ip_tables x_tables acpiphp ppde v psmouse parport_pc serio_raw fbcon
> tileblit font bitblit softcursor vmxnet3 vga16fb vgasta te lp parport
> floppy vmw_pvscsi
> [212670.327741] Pid: 0, comm: swapper Not tainted 2.6.32-24-server #42-Ubuntu
> [212670.327742] Call Trace:
> [212670.327744]  <IRQ>  [<ffffffff81066dab>] warn_slowpath_common+0x7b/0xc0
> [212670.327754]  [<ffffffff81066e04>] warn_slowpath_null+0x14/0x20
> [212670.327757]  [<ffffffff814b98f9>] tcp_fastretrans_alert+0x3f9/0xd90
> [212670.327759]  [<ffffffff814bad24>] tcp_ack+0x284/0x6b0
> [212670.327762]  [<ffffffff814bbab0>] tcp_rcv_established+0x400/0x730
> [212670.327766]  [<ffffffff814c3553>] tcp_v4_do_rcv+0xf3/0x160
> [212670.327769]  [<ffffffff814c4ca5>] tcp_v4_rcv+0x5b5/0x7e0
> [212670.327772]  [<ffffffff814a2c00>] ? ip_local_deliver_finish+0x0/0x2d0
> [212670.327776]  [<ffffffff8149a6b4>] ? nf_hook_slow+0x74/0x100
> [212670.327779]  [<ffffffff814a2c00>] ? ip_local_deliver_finish+0x0/0x2d0
> [212670.327781]  [<ffffffff814a2cdd>] ip_local_deliver_finish+0xdd/0x2d0
> [212670.327783]  [<ffffffff814a2f60>] ip_local_deliver+0x90/0xa0
> [212670.327785]  [<ffffffff814a241d>] ip_rcv_finish+0x12d/0x440
> [212670.327787]  [<ffffffff814a29a5>] ip_rcv+0x275/0x360
> [212670.327794]  [<ffffffff8147319a>] netif_receive_skb+0x38a/0x5d0
> [212670.327801]  [<ffffffffa005203c>]
> vmxnet3_rq_rx_complete+0x23c/0x610 [vmxnet3]
> [212670.327806]  [<ffffffffa005245f>] vmxnet3_do_poll+0x4f/0xf0 [vmxnet3]
> [212670.327809]  [<ffffffffa005252d>] vmxnet3_poll+0x2d/0x60 [vmxnet3]
> [212670.327816]  [<ffffffff810397a9>] ? default_spin_lock_flags+0x9/0x10
> [212670.327820]  [<ffffffff81473c8f>] net_rx_action+0x10f/0x250
> [212670.327824]  [<ffffffff8106e2f7>] __do_softirq+0xb7/0x1e0
> [212670.327828]  [<ffffffff810c4d40>] ? handle_IRQ_event+0x60/0x170
> [212670.327833]  [<ffffffff810142ec>] call_softirq+0x1c/0x30
> [212670.327836]  [<ffffffff81015cb5>] do_softirq+0x65/0xa0
> [212670.327837]  [<ffffffff8106e195>] irq_exit+0x85/0x90
> [212670.327845]  [<ffffffff8155fd45>] do_IRQ+0x75/0xf0
> [212670.327848]  [<ffffffff81013b13>] ret_from_intr+0x0/0x11
> [212670.327849]  <EOI>  [<ffffffff81038acb>] ? native_safe_halt+0xb/0x10
> [212670.327855]  [<ffffffff8101b69d>] ? default_idle+0x3d/0x90
> [212670.327860]  [<ffffffff81011e63>] ? cpu_idle+0xb3/0x110
> [212670.327867]  [<ffffffff81552b53>] ? start_secondary+0xa8/0xaa
> [212670.327869] ---[ end trace e26694ca94def524 ]---
> 
> 
> Here's netstat -s output
> 
> Ip:
>     362267935 total packets received
>     0 forwarded
>     0 incoming packets discarded
>     362257901 incoming packets delivered
>     326868094 requests sent out
>     9 fragments dropped after timeout
>     2777 reassemblies required
>     1357 packets reassembled ok
>     9 packet reassembles failed
>     490 fragments received ok
>     1034 fragments created
> Icmp:
>     22430 ICMP messages received
>     283 input ICMP message failed.
>     ICMP input histogram:
>         destination unreachable: 17113
>         timeout in transit: 613
>         redirects: 9
>         echo requests: 4685
>         echo replies: 10
>     7383 ICMP messages sent
>     0 ICMP messages failed
>     ICMP output histogram:
>         destination unreachable: 2695
>         echo request: 11
>         echo replies: 4677
> IcmpMsg:
>         InType0: 10
>         InType3: 17113
>         InType5: 9
>         InType8: 4685
>         InType11: 613
>         OutType0: 4677
>         OutType3: 2695
>         OutType8: 11
> Tcp:
>     6563076 active connections openings
>     1610183 passive connection openings
>     1646225 failed connection attempts
>     75246 connection resets received
>     1141 connections established
>     361192315 segments received
>     323797760 segments send out
>     1997456 segments retransmited
>     12990 bad segments received.
>     1851272 resets sent
> Udp:
>     4138313 packets received
>     186 packets to unknown port received.
>     0 packet receive errors
>     4161268 packets sent
> UdpLite:
> TcpExt:
>     5096 invalid SYN cookies received
>     167 resets received for embryonic SYN_RECV sockets
>     58826 packets pruned from receive queue because of socket buffer overrun
>     312 packets pruned from receive queue
>     19 packets dropped from out-of-order queue because of socket buffer overrun
>     365 ICMP packets dropped because they were out-of-window
>     2712111 TCP sockets finished time wait in fast timer
>     82068 time wait sockets recycled by time stamp
>     311 packets rejects in established connections because of timestamp
>     11516250 delayed acks sent
>     6415 delayed acks further delayed because of locked socket
>     Quick ack mode was activated 208750 times
>     41789 packets directly queued to recvmsg prequeue.
>     1 bytes directly in process context from backlog
>     2969 bytes directly received in process context from prequeue
>     155660989 packet headers predicted
>     55448128 acknowledgments not containing data payload received
>     89389461 predicted acknowledgments
>     9544 times recovered from packet loss due to fast retransmit
>     585991 times recovered from packet loss by selective acknowledgements
>     710 bad SACK blocks received
>     Detected reordering 1944 times using FACK
>     Detected reordering 26599 times using SACK
>     Detected reordering 267 times using reno fast retransmit
>     Detected reordering 1819 times using time stamp
>     3169 congestion windows fully recovered without slow start
>     26810 congestion windows partially recovered using Hoe heuristic
>     14402 congestion windows recovered without slow start by DSACK
>     40392 congestion windows recovered without slow start after partial ack
>     115772 TCP data loss events
>     TCPLostRetransmit: 5320
>     1922 timeouts after reno fast retransmit
>     69095 timeouts after SACK recovery
>     11101 timeouts in loss state
>     1151834 fast retransmits
>     74479 forward retransmits
>     225380 retransmits in slow start
>     275631 other TCP timeouts
>     2606 classic Reno fast retransmits failed
>     30507 SACK retransmits failed
>     4862645 packets collapsed in receive queue due to low socket buffer
>     260934 DSACKs sent for old packets
>     5109 DSACKs sent for out of order packets
>     95650 DSACKs received
>     819 DSACKs for out of order packets received
>     28142 connections reset due to unexpected data
>     22325 connections reset due to early user close
>     85423 connections aborted due to timeout
>     TCPSACKDiscard: 2319
>     TCPDSACKIgnoredOld: 52587
>     TCPDSACKIgnoredNoUndo: 29859
>     TCPSpuriousRTOs: 2605
>     TCPSackShifted: 3539688
>     TCPSackMerged: 3241543
>     TCPSackShiftFallback: 8938458
> IpExt:
>     InMcastPkts: 2105
>     InBcastPkts: 12
>     InOctets: 1670966745
>     OutOctets: -417434189
>     InMcastOctets: 58940
>     InBcastOctets: 1792


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/