Re: Problem on SCTP

From: Xin Long
Date: Wed Mar 01 2017 - 02:46:06 EST

Next message: Richard Guy Briggs: "[PATCH ALT4] audit: show fstype:pathname for entries with anonymous parents"
Previous message: Michael Hennerich: "Re: [PATCH 4/4] Input: ad7879 - do not manipulate capability bits directly"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

On Wed, Mar 1, 2017 at 12:15 PM, Sun Paul <paulrbk@xxxxxxxxx> wrote:
> Hi Xin
>
> I have used 3.10.0-514.6.2.el7.x86_64 on Centos7 and tessted. the
> same issue still occur.
>
> any idea?
>
what I can only see is :
03:32:37.928679 IP 10.165.250.22.3868 > 192.168.2.13.40001: sctp (1) [HB REQ]
03:32:37.928717 IP 192.168.2.14.3868 > 192.168.2.13.40001: sctp (1) [HB REQ]
03:32:37.929247 IP 192.168.2.13.40001 > 192.168.2.14.3868: sctp (1) [ABORT]

this ABORT might be caused by no assoc.

can you show more details about your env ?
1. the iptables rule for nat in router (# iptables -L -t nat)

2. the asocs info in client (# cat /proc/net/sctp/assocs)

3. more packet you captured, like 4-shake and some packets
right before these 3 packets.

4. conntrack info in router (# cat /proc/net/nf_conntrack)

> On Mon, Feb 27, 2017 at 11:06 PM, Xin Long <lucien.xin@xxxxxxxxx> wrote:
>> On Mon, Feb 27, 2017 at 5:01 PM, Sun Paul <paulrbk@xxxxxxxxx> wrote:
>>> Hi, can I confirm that the problem is on the Linux router itself or on
>>> both server and client?
>>>
>> try with a new kernel in router, like build and install a upstream
>> kernel on your route machine.
>>
>> git://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git
>>
>> https://kernelnewbies.org/KernelBuild :
>> part "Setting up your kernel configuration" and
>> part "Building the kernel"
>>
>>> On Thu, Feb 23, 2017 at 2:07 PM, Xin Long <lucien.xin@xxxxxxxxx> wrote:
>>>> On Thu, Feb 23, 2017 at 1:30 PM, Sun Paul <paulrbk@xxxxxxxxx> wrote:
>>>>> does this fixed in RHEL7?
>>>> yes, I think so.
>>>>
>>>>>
>>>>> On Wed, Feb 22, 2017 at 11:03 AM, Xin Long <lucien.xin@xxxxxxxxx> wrote:
>>>>>> On Wed, Feb 22, 2017 at 10:29 AM, Sun Paul <paulrbk@xxxxxxxxx> wrote:
>>>>>>> Hi Xin
>>>>>>>
>>>>>>> do you mean we need to patch the kernel?
>>>>>> Yups, pls comment on that bz if it's really needed for your env.
>>>>>> A z-stream kernel may be available for that issue soon.
>>>>>>
>>>>>> Thanks.
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Feb 22, 2017 at 10:00 AM, Xin Long <lucien.xin@xxxxxxxxx> wrote:
>>>>>>>> On Wed, Feb 22, 2017 at 9:12 AM, Sun Paul <paulrbk@xxxxxxxxx> wrote:
>>>>>>>>> Hi
>>>>>>>>>
>>>>>>>>> the router is actually is a linux running on RHEL6.8
>>>>>>>>> (2.6.32-642.4.2.el6.x86_64). it uses iptables to do SNAT aand DNAT
>>>>>>>>> forward.
>>>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1412038
>>>>>>>>
>>>>>>>> sctp_manip_pkt->sctp_compute_cksum:
>>>>>>>> struct sctphdr *sh = sctp_hdr(sub);
>>>>>>>>
>>>>>>>> But in rhel6, skb->transport_header is not yet set at that time.
>>>>>>>> This patch should be backported into rhel6.
>>>>>>>>
>>>>>>>> commit 21d1196a35f5686c4323e42a62fdb4b23b0ab4a3
>>>>>>>> Author: Eric Dumazet <edumazet@xxxxxxxxxx>
>>>>>>>> Date: Mon Jul 15 20:03:19 2013 -0700
>>>>>>>>
>>>>>>>> ipv4: set transport header earlier
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, Feb 21, 2017 at 11:53 PM, Xin Long <lucien.xin@xxxxxxxxx> wrote:
>>>>>>>>>> On Tue, Feb 21, 2017 at 12:26 PM, Sun Paul <paulrbk@xxxxxxxxx> wrote:
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> sorry to get back late, the platform is running on KVM. and this
>>>>>>>>>>> problem is resolved by moving to VMware environment, however, a new
>>>>>>>>>>> problem is coming out, we noticed that the HB REQ is being ABORT from
>>>>>>>>>>> client.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 03:32:35.233572 IP 10.165.250.22.3868 > 192.168.2.13.40001: sctp (1)
>>>>>>>>>>> [HB REQ] (from server to sctp router)
>>>>>>>>>>> 03:32:35.233603 IP 192.168.2.14.3868 > 192.168.2.13.40001: sctp (1)
>>>>>>>>>>> [HB REQ] (from sctp router to client)
>>>>>>>>>>> 03:32:35.233852 IP 192.168.2.13.40001 > 192.168.2.14.3868: sctp (1)
>>>>>>>>>>> [ABORT] (from client to sctp router)
>>>>>>>>>>>
>>>>>>>>>>> 03:32:37.928679 IP 10.165.250.22.3868 > 192.168.2.13.40001: sctp (1) [HB REQ]
>>>>>>>>>>> 03:32:37.928717 IP 192.168.2.14.3868 > 192.168.2.13.40001: sctp (1) [HB REQ]
>>>>>>>>>>> 03:32:37.929247 IP 192.168.2.13.40001 > 192.168.2.14.3868: sctp (1) [ABORT]
>>>>>>>>>>>
>>>>>>>>>>> For the above packet flow, 10.165.250.22 is the server and
>>>>>>>>>>> 192.168.2.13 is the client, the server 10.165.250.22 sends HB REQ to
>>>>>>>>>>> client 192.168.2.13 through 192.168.2.14 (the SCTP router), and the
>>>>>>>>>>> SCTP router change the src address before forward the HB REQ to the
>>>>>>>>>>> client.
>>>>>>>>>>>
>>>>>>>>>>> But somehow the client is ABORT the HB REQ, any idea? is it related to
>>>>>>>>>>> the HEARTBEAT information? or the checksum again>?
>>>>>>>>>> The incorrect checksum won't cause ABORT, but the abnormal HB REQ
>>>>>>>>>> could be, if HB information was modified when calculating the checksum
>>>>>>>>>> on router, the ABORT may be caused in client process.
>>>>>>>>>>
>>>>>>>>>> is your SCTP router linux ? if yes, what's the kernel version ?
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Jan 13, 2017 at 9:19 PM, Michael Tuexen
>>>>>>>>>>> <Michael.Tuexen@xxxxxxxxxxxxxxxxx> wrote:
>>>>>>>>>>>>> On 13 Jan 2017, at 10:43, Michael Tuexen <Michael.Tuexen@xxxxxxxxxxxxxxxxx> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Your router does NOT change any field in the SCTP packet, but the
>>>>>>>>>>>>> SCTP checksum was modified from
>>>>>>>>>>>>> Checksum: 0xbaea49e5 (not verified)
>>>>>>>>>>>>> to
>>>>>>>>>>>>> Checksum: 0xa9a86d3f (not verified)
>>>>>>>>>>>>> At least one of these is wrong. Read the tracefiles in wireshark and
>>>>>>>>>>>>> enable checksum validation and wireshark will tell you which one is
>>>>>>>>>>>>> correct. (That is why I asked for .pcap file instead of a .txt).
>>>>>>>>>>>>>
>>>>>>>>>>>>> My guess is that the initial checksum is correct and your box middlebox
>>>>>>>>>>>>> not only changes the destination address and ttl field and header
>>>>>>>>>>>>> checksum in the IP-header (which is expected) but also incorrectly the
>>>>>>>>>>>>> SCTP checksum. Since no field in the SCTP packet has changed, the checksum
>>>>>>>>>>>>> must be the same.
>>>>>>>>>>>> At the server have a look at the SNMP counters:
>>>>>>>>>>>> cat /proc/net/sctp/snmp
>>>>>>>>>>>> You should find a line staring with
>>>>>>>>>>>> SctpChecksumErrors
>>>>>>>>>>>> If the number reported there is positive, the node received packets
>>>>>>>>>>>> with checksum errors.
>>>>>>>>>>>>
>>>>>>>>>>>> Best regards
>>>>>>>>>>>> Michael
>>>>>>>>>>>>>
>>>>>>>>>>>>> Best regards
>>>>>>>>>>>>> Michael
>>>>>>>>>>>>>> On 13 Jan 2017, at 04:29, Sun Paul <paulrbk@xxxxxxxxx> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Frame 2: 98 bytes on wire (784 bits), 98 bytes captured (784 bits)
>>>>>>>>>>>>>> Encapsulation type: Ethernet (1)
>>>>>>>>>>>>>> Arrival Time: Jan 6, 2017 16:52:49.662321000 Malay Peninsula Standard Time
>>>>>>>>>>>>>> [Time shift for this packet: 0.000000000 seconds]
>>>>>>>>>>>>>> Epoch Time: 1483692769.662321000 seconds
>>>>>>>>>>>>>> [Time delta from previous captured frame: 0.000179000 seconds]
>>>>>>>>>>>>>> [Time delta from previous displayed frame: 0.000179000 seconds]
>>>>>>>>>>>>>> [Time since reference or first frame: 0.000179000 seconds]
>>>>>>>>>>>>>> Frame Number: 2
>>>>>>>>>>>>>> Frame Length: 98 bytes (784 bits)
>>>>>>>>>>>>>> Capture Length: 98 bytes (784 bits)
>>>>>>>>>>>>>> [Frame is marked: False]
>>>>>>>>>>>>>> [Frame is ignored: False]
>>>>>>>>>>>>>> [Protocols in frame: eth:ethertype:ip:sctp]
>>>>>>>>>>>>>> Ethernet II, Src: Vmware_81:41:6b (00:50:56:81:41:6b), Dst:
>>>>>>>>>>>>>> Vmware_81:a6:a3 (00:50:56:81:a6:a3)
>>>>>>>>>>>>>> Destination: Vmware_81:a6:a3 (00:50:56:81:a6:a3)
>>>>>>>>>>>>>> Address: Vmware_81:a6:a3 (00:50:56:81:a6:a3)
>>>>>>>>>>>>>> .... ..0. .... .... .... .... = LG bit: Globally unique
>>>>>>>>>>>>>> address (factory default)
>>>>>>>>>>>>>> .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
>>>>>>>>>>>>>> Source: Vmware_81:41:6b (00:50:56:81:41:6b)
>>>>>>>>>>>>>> Address: Vmware_81:41:6b (00:50:56:81:41:6b)
>>>>>>>>>>>>>> .... ..0. .... .... .... .... = LG bit: Globally unique
>>>>>>>>>>>>>> address (factory default)
>>>>>>>>>>>>>> .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
>>>>>>>>>>>>>> Type: IPv4 (0x0800)
>>>>>>>>>>>>>> Internet Protocol Version 4, Src: 192.168.206.83, Dst: 192.168.206.66
>>>>>>>>>>>>>> 0100 .... = Version: 4
>>>>>>>>>>>>>> .... 0101 = Header Length: 20 bytes (5)
>>>>>>>>>>>>>> Differentiated Services Field: 0x02 (DSCP: CS0, ECN: ECT(0))
>>>>>>>>>>>>>> 0000 00.. = Differentiated Services Codepoint: Default (0)
>>>>>>>>>>>>>> .... ..10 = Explicit Congestion Notification: ECN-Capable
>>>>>>>>>>>>>> Transport codepoint '10' (2)
>>>>>>>>>>>>>> Total Length: 84
>>>>>>>>>>>>>> Identification: 0x0000 (0)
>>>>>>>>>>>>>> Flags: 0x02 (Don't Fragment)
>>>>>>>>>>>>>> 0... .... = Reserved bit: Not set
>>>>>>>>>>>>>> .1.. .... = Don't fragment: Set
>>>>>>>>>>>>>> ..0. .... = More fragments: Not set
>>>>>>>>>>>>>> Fragment offset: 0
>>>>>>>>>>>>>> Time to live: 63
>>>>>>>>>>>>>> Protocol: SCTP (132)
>>>>>>>>>>>>>> Header checksum: 0x1d3d [validation disabled]
>>>>>>>>>>>>>> [Good: False]
>>>>>>>>>>>>>> [Bad: False]
>>>>>>>>>>>>>> Source: 192.168.206.83
>>>>>>>>>>>>>> Destination: 192.168.206.66
>>>>>>>>>>>>>> [Source GeoIP: Unknown]
>>>>>>>>>>>>>> [Destination GeoIP: Unknown]
>>>>>>>>>>>>>> Stream Control Transmission Protocol, Src Port: 50001 (50001), Dst
>>>>>>>>>>>>>> Port: 3868 (3868)
>>>>>>>>>>>>>> Source port: 50001
>>>>>>>>>>>>>> Destination port: 3868
>>>>>>>>>>>>>> Verification tag: 0x00000000
>>>>>>>>>>>>>> [Assocation index: 0]
>>>>>>>>>>>>>> Checksum: 0xa9a86d3f (not verified)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> INIT chunk (Outbound streams: 3000, inbound streams: 3000)
>>>>>>>>>>>>>> Chunk type: INIT (1)
>>>>>>>>>>>>>> 0... .... = Bit: Stop processing of the packet
>>>>>>>>>>>>>> .0.. .... = Bit: Do not report
>>>>>>>>>>>>>> Chunk flags: 0x00
>>>>>>>>>>>>>> Chunk length: 52
>>>>>>>>>>>>>> Initiate tag: 0xe79f40cb
>>>>>>>>>>>>>> Advertised receiver window credit (a_rwnd): 62464
>>>>>>>>>>>>>> Number of outbound streams: 3000
>>>>>>>>>>>>>> Number of inbound streams: 3000
>>>>>>>>>>>>>> Initial TSN: 176990880
>>>>>>>>>>>>>> IPv4 address parameter (Address: 192.168.206.83)
>>>>>>>>>>>>>> Parameter type: IPv4 address (0x0005)
>>>>>>>>>>>>>> 0... .... .... .... = Bit: Stop processing of chunk
>>>>>>>>>>>>>> .0.. .... .... .... = Bit: Do not report
>>>>>>>>>>>>>> Parameter length: 8
>>>>>>>>>>>>>> IP Version 4 address: 192.168.206.83
>>>>>>>>>>>>>> IPv4 address parameter (Address: 192.168.1.83)
>>>>>>>>>>>>>> Parameter type: IPv4 address (0x0005)
>>>>>>>>>>>>>> 0... .... .... .... = Bit: Stop processing of chunk
>>>>>>>>>>>>>> .0.. .... .... .... = Bit: Do not report
>>>>>>>>>>>>>> Parameter length: 8
>>>>>>>>>>>>>> IP Version 4 address: 192.168.1.83
>>>>>>>>>>>>>> Supported address types parameter (Supported types: IPv6, IPv4)
>>>>>>>>>>>>>> Parameter type: Supported address types (0x000c)
>>>>>>>>>>>>>> 0... .... .... .... = Bit: Stop processing of chunk
>>>>>>>>>>>>>> .0.. .... .... .... = Bit: Do not report
>>>>>>>>>>>>>> Parameter length: 8
>>>>>>>>>>>>>> Supported address type: IPv6 address (6)
>>>>>>>>>>>>>> Supported address type: IPv4 address (5)
>>>>>>>>>>>>>> ECN parameter
>>>>>>>>>>>>>> Parameter type: ECN (0x8000)
>>>>>>>>>>>>>> 1... .... .... .... = Bit: Skip parameter and continue
>>>>>>>>>>>>>> processing of the chunk
>>>>>>>>>>>>>> .0.. .... .... .... = Bit: Do not report
>>>>>>>>>>>>>> Parameter length: 4
>>>>>>>>>>>>>> Forward TSN supported parameter
>>>>>>>>>>>>>> Parameter type: Forward TSN supported (0xc000)
>>>>>>>>>>>>>> 1... .... .... .... = Bit: Skip parameter and continue
>>>>>>>>>>>>>> processing of the chunk
>>>>>>>>>>>>>> .1.. .... .... .... = Bit: Do report
>>>>>>>>>>>>>> Parameter length: 4
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> To unsubscribe from this list: send the line "unsubscribe linux-sctp" in
>>>>>>>>>>>>> the body of a message to majordomo@xxxxxxxxxxxxxxx
>>>>>>>>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> To unsubscribe from this list: send the line "unsubscribe linux-sctp" in
>>>>>>>>>>> the body of a message to majordomo@xxxxxxxxxxxxxxx
>>>>>>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html

Next message: Richard Guy Briggs: "[PATCH ALT4] audit: show fstype:pathname for entries with anonymous parents"
Previous message: Michael Hennerich: "Re: [PATCH 4/4] Input: ad7879 - do not manipulate capability bits directly"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]