[syzbot] [sctp?] UBSAN: shift-out-of-bounds in sctp_transport_update_rto

[Moon Hee Lee]

#syz test git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git main
From 7c472b40b901a12cfc8dc00ca0a99ac2d992c338 Mon Sep 17 00:00:00 2001
From: Moon Hee Lee <moonhee.lee.ca@xxxxxxxxx>
Date: Tue, 22 Jul 2025 12:31:16 -0700
Subject: [PATCH net] sctp: guard rto_alpha and rto_beta against unsafe shift
 values

rto_alpha and rto_beta are used as shift amounts in the RTT smoothing
calculation, where they represent inverse powers of two (e.g. 3 means 1/8).

Currently, the code uses net->sctp.rto_alpha and rto_beta directly in shift
expressions without validating them. If user-controlled or corrupted values
exceed valid shift bounds for 32-bit integers (e.g. 237), this leads to
undefined behavior and runtime faults.

syzbot reported such a case via UBSAN:

  UBSAN: shift-out-of-bounds in net/sctp/transport.c:509:41
  shift exponent 237 is too large for 32-bit type 'unsigned int'

This patch ensures both values are within the safe shift range [0, 31].
If not, the code falls back to the default constants SCTP_RTO_ALPHA and
SCTP_RTO_BETA to ensure correctness and system stability.

This preserves SCTP tunability while preventing undefined behavior.

Signed-off-by: Moon Hee Lee <moonhee.lee.ca@xxxxxxxxx>
---
 net/sctp/transport.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/net/sctp/transport.c b/net/sctp/transport.c
index 6946c1462793..848311bb7a9f 100644
--- a/net/sctp/transport.c
+++ b/net/sctp/transport.c
@@ -495,6 +495,8 @@ void sctp_transport_update_rto(struct sctp_transport *tp, __u32 rtt)
 
 	if (tp->rttvar || tp->srtt) {
 		struct net *net = tp->asoc->base.net;
+		int rto_alpha = net->sctp.rto_alpha;
+		int rto_beta = net->sctp.rto_beta;
 		/* 6.3.1 C3) When a new RTT measurement R' is made, set
 		 * RTTVAR <- (1 - RTO.Beta) * RTTVAR + RTO.Beta * |SRTT - R'|
 		 * SRTT <- (1 - RTO.Alpha) * SRTT + RTO.Alpha * R'
@@ -505,7 +507,16 @@ void sctp_transport_update_rto(struct sctp_transport *tp, __u32 rtt)
 		 * of two.
 		 * For example, assuming the default value of RTO.Alpha of
 		 * 1/8, rto_alpha would be expressed as 3.
+		 *
+		 * Guard rto_alpha and rto_beta to ensure they are within
+		 * valid shift bounds [0, 31] to avoid undefined behavior.
 		 */
+		if (unlikely(rto_alpha < 0 || rto_alpha >= 32))
+			rto_alpha = SCTP_RTO_ALPHA;
+
+		if (unlikely(rto_beta < 0 || rto_beta >= 32))
+			rto_beta = SCTP_RTO_BETA;
+
 		tp->rttvar = tp->rttvar - (tp->rttvar >> net->sctp.rto_beta)
 			+ (((__u32)abs((__s64)tp->srtt - (__s64)rtt)) >> net->sctp.rto_beta);
 		tp->srtt = tp->srtt - (tp->srtt >> net->sctp.rto_alpha)
-- 
2.43.0