Re: Performance regression in IO scheduler still there

[Corrado Zoccolo]

Hi Jeff,
what hardware are you using for tests?
I see aggregated random read bandwidth is larger than sequential read
bandwidth, and write bandwidth greater than read.
Is this a SAN with multiple independent spindles?

On Thu, Nov 5, 2009 at 9:10 PM, Jeff Moyer <jmoyer@xxxxxxxxxx> wrote:
> Jan Kara <jack@xxxxxxx> writes:
>
>> Â Hi,
>>
>> Â I took time and remeasured tiobench results on recent kernel. A short
>> conclusion is that there is still a performance regression which I reported
>> few months ago. The machine is Intel 2 CPU with 2 GB RAM and plain SATA
>> drive. tiobench sequential write performance numbers with 16 threads:
>> 2.6.29: Â Â Â Â Â Â ÂAVG Â Â Â STDERR
>> 37.80 38.54 39.48 -> 38.606667 0.687475
>>
>> 2.6.32-rc5:
>> 37.36 36.41 36.61 -> 36.793333 0.408928
>>
>> So about 5% regression. The regression happened sometime between 2.6.29 and
>> 2.6.30 and stays the same since then... With deadline scheduler, there's
>> no regression. Shouldn't we do something about it?
>
> Sorry it took so long, but I've been flat out lately. ÂI ran some
> numbers against 2.6.29 and 2.6.32-rc5, both with low_latency set to 0
> and to 1. ÂHere are the results (average of two runs):
>
>                              Ârlat   Â|   rrlat    |   wlat    | Ârwlat
> kernel   | Thr | read Â| randr Â| write Â| randw Â|  Âavg, max   |  Âavg, max   |  avg, max   | avg,max
> ------------------------------------------------------------------------------------------------------------------------
> 2.6.29 Â Â | Â8 Â| 72.95 | Â20.06 | 269.66 | 231.59 | Â6.625, 1683.66 | 23.241, 1547.97 | 1.761, Â698.10 | 0.720, 443.64
> Â Â Â Â Â | 16 Â| 72.33 | Â20.03 | 278.85 | 228.81 | 13.643, 2499.77 | 46.575, 1717.10 | 3.304, 1149.29 | 1.011, 140.30
> ------------------------------------------------------------------------------------------------------------------------
> 2.6.32-rc5 | Â8 Â| 86.58 | Â19.80 | 198.82 | 205.06 | Â5.694, 977.26 Â| 22.559, Â870.16 | 2.359, Â693.88 | 0.530, Â24.32
> Â Â Â Â Â | 16 Â| 86.82 | Â21.10 | 199.00 | 212.02 | 11.010, 1958.78 | 40.195, 1662.35 | 4.679, 1351.27 | 1.007, Â25.36
> ------------------------------------------------------------------------------------------------------------------------
> 2.6.32-rc5 | Â8 Â| 87.65 | 117.65 | 298.27 | 212.35 | Â5.615, Â984.89 | Â4.060, Â 97.39 | 1.535, Â311.14 | 0.534, Â24.29
> low_lat=0 Â| 16 Â| 95.60 | 119.95*| 302.48 | 213.27 | 10.263, 1750.19 | 13.899, 1006.21 | 3.221, Â734.22 | 1.062, Â40.40
> ------------------------------------------------------------------------------------------------------------------------
>
> Legend:
> rlat - read latency
> rrlat - random read latency
> wlat - write lancy
> rwlat - random write latency
> * - the two runs reported vastly different numbers: 67.53 and 172.46
>
> So, as you can see, if we turn off the low_latency tunable, we get
> better numbers across the board with the exception of random writes.
> It's also interesting to note that the latencies reported by tiobench
> are more favorable with low_latency set to 0, which is
> counter-intuitive.
>
> So, now it seems we don't have a regression in sequential read
> bandwidth, but we do have a regression in random read bandwidth (though
> the random write latencies look better). ÂSo, I'll look into that, as it
> is almost 10%, which is significant.
>

Sorry, I don't see a 10% regression in random read from your numbers.
I see a larger one  in sequential write for low_latency=1 (this was
the regression Jan reported in the original message), but not for
low_latency=0. And a 10% regression in random writes, that is not
completely fixed even by disabling low_latency.

I guess your seemingly counter-intuitive results for low_latency are
due to the uncommon hardware (low_latency was intended mainly for
desktop-class disks). Luckily, the patches queued for 2.6.33 already
address this low_latency misbehaviour.

Thanks,
Corrado.

> Cheers,
> Jeff
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@xxxxxxxxxxxxxxx
> More majordomo info at Âhttp://vger.kernel.org/majordomo-info.html
> Please read the FAQ at Âhttp://www.tux.org/lkml/
>
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/