cannot fill 100Mbps pipe (over 20ms rtt via netem)?

[slashdev]

hi,

i am experimenting with LFN's using "netem".

without adding any delay (using netem), with default (LAN-ish) rtt,
i am able to almost fill a 100Mbps link between two servers. both
nuttcp and iperf report ~94Mbps. thats nice.

then i added 10ms delay using netem on both nodes. making the
rtt as 20ms.

$ tc -s qdisc show dev eth0
qdisc netem 8005: limit 1000 delay 10.0ms
 Sent 1199824317 bytes 148119 pkt (dropped 0, overlimits 0 requeues 7)
 rate 0bit 0pps backlog 0b 1p requeues 7

i also increased the socket buffers to large values on both nodes:

$ sysctl net | grep mem
net.ipv4.tcp_rmem = 400000      400000  37500000
net.ipv4.tcp_wmem = 400000      400000  37500000
net.ipv4.tcp_mem = 196608       262144  393216
..
net.core.rmem_default = 126976
net.core.wmem_default = 126976
net.core.rmem_max = 37500000
net.core.wmem_max = 37500000

(yes i know those values are really really large. but i finally want
to experiment with 300ms rtts so changed the values to large ones.
both servers have 16GB ram and are dual-core xeons).

and my sles10 kernel looks to supports receiver side autotuning too:

$ uname -a
Linux Node1 2.6.16.21-0.8-smp #1 SMP Mon Jul 3 18:25:39 UTC 2006
x86_64 x86_64 x86_64 GNU/Linux

$ syctl net | grep rcvbuf
net.ipv4.tcp_moderate_rcvbuf = 1

given all this is set (and looks sufficient to me), ttcp or iperf
gives me only 30Mbps.
i tried setting socket buf sizes explicitly too (and turn off
autotuning) via cmdline
options in ttcp/iperf -- but that made no difference.

i expected that i will be able to fill the whole pipe (given i've
tuned all buffers
sufficiently) and get around 90Mbps similar to earlier test (where rtt
was really
small).

am i missing something here? or mis-understanding something?

i can provide the tcpdump trace. but that too looks ok. i.e. window
scale is set to
healthy "10" and the receiver does endup opening large window. but i
may be missing
something.

if you need any other information please do ask.

i am bit puzzled at the moment :-)

thanks in advance,
s.

ps: i did subscribe to linux-net, but have not got any confirmation
yet (waited for 2hrs).
      please Cc: me any replies so i dont loose any valuable inputs.
--
To unsubscribe from this list: send the line "unsubscribe linux-net" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html