Re: unfair io behaviour for high load interactive use still present in 2.6.31

[Corrado Zoccolo]

On Tue, Sep 15, 2009 at 9:30 AM, Tobias Oetiker <tobi@xxxxxxxxxx> wrote:
> Experts,
>
> We run several busy NFS file servers with Areca HW Raid + LVM2 + ext3
>
> We find that the read bandwidth falls dramatically as well as the
> response times going up to several seconds as soon as the system
> comes under heavy write strain.
>
> With the release of 2.6.31 and all the io fixes that went in there,
> I was hoping for a solution and set out to do some tests ...
>
> I have seen the io problem posted on LKML a few times, and normally
> it is simulated by running several concurrent dd processes one
> reading and one writing. It seems that cfq with a low slice_async
> can deal pretty well with competing dds.
>
> Unfortunately our use case is not users running dd but rather a lot
> of processes accessing many small to medium sized files for reading
> and writing.
>
> I have written a test program that unpacks Linux 2.6.30.5 a few
> times into a file system, flushes the cache and then tars it up
> again while unpacking some more tars in parallel. While this is
> running I use iostat to watch the activity on the block devices. As
> I am interested in interactive performance of the system, the await
> row as well as the rMB/s row are of special interest to me
>
> Âiostat -m -x 5
>
> Even with a low 'resolution' of 5 seconds, the performance figures
> are jumping all over the place. This concurs with the user
> experience when working with the system interactively.
>
> I tried to optimize the configuration systematically turning all
> the knobs I know of (/proc/sys/vm, /sys/block/*/queue/scheduler,
> data=journal, data=ordered, external journal on a ssd device) one
> at a time. I found that cfq helps the read performance quite a lot
> as far as total run-time is concerned, but the jerky nature of the
> measurements does not change, and also the read performance keeps
> dropping dramatically as soon as it is in competition with writers.
>
> I would love to get some hints on how to make such a setup perform
> without these huge performance fluctuations.
>
> While testing, I saw that iostat reports huge wMB/s numbers and
> ridiculously low rMB/s numbers. Looking at the actual amount of
> data in the tar files as well as the run time of the tar processes
> the numbers MB/s numbers reported by iostat do seem strange. The
> read numbers are too low and the write numbers are too high.
>
> Â12 * 1.3 GB reading in 270s = 14 MB/s sustained
> Â12 * 0.3 GB writing in 180s = 1.8 MB/s sustained
>
> Since I am looking at relative performance figures this does not
> matter so much, but it is still a bit disconcerting.
>
> My tests script is available on http://tobi.oetiker.ch/fspunisher/
>
> Below is an excerpt from iostat while the test is in full swing:
>
> * 2.6.31 (8 cpu x86_64, 24 GB Ram)
> * scheduler = cfq
> * iostat -m -x dm-5 5
> * running in parallel on 3 lvm logical volumes
> Âon a single physical volume
>
> Device:     rrqm/s  wrqm/s   r/s   w/s  ÂrMB/s  ÂwMB/s avgrq-sz avgqu-sz  await Âsvctm Â%util
> ---------------------------------------------------------------------------------------------------------
> dm-5 Â Â Â Â Â Â 0.00 Â Â 0.00 Â 26.60 7036.60 Â Â 0.10 Â Â27.49 Â Â 8.00 Â 669.04 Â 91.58 Â 0.09 Â66.96
> dm-5 Â Â Â Â Â Â 0.00 Â Â 0.00 Â 63.80 2543.00 Â Â 0.25 Â Â 9.93 Â Â 8.00 Â1383.63 Â539.28 Â 0.36 Â94.64
> dm-5 Â Â Â Â Â Â 0.00 Â Â 0.00 Â 78.00 5084.60 Â Â 0.30 Â Â19.86 Â Â 8.00 Â1007.41 Â195.12 Â 0.15 Â77.36
> dm-5 Â Â Â Â Â Â 0.00 Â Â 0.00 Â 44.00 5588.00 Â Â 0.17 Â Â21.83 Â Â 8.00 Â 516.27 Â 91.69 Â 0.17 Â95.44
> dm-5 Â Â Â Â Â Â 0.00 Â Â 0.00 Â Â0.00 6014.20 Â Â 0.00 Â Â23.49 Â Â 8.00 Â1331.42 Â 66.25 Â 0.13 Â76.48
> dm-5 Â Â Â Â Â Â 0.00 Â Â 0.00 Â 28.80 4491.40 Â Â 0.11 Â Â17.54 Â Â 8.00 Â1000.37 Â412.09 Â 0.17 Â78.24
> dm-5 Â Â Â Â Â Â 0.00 Â Â 0.00 Â 36.60 6486.40 Â Â 0.14 Â Â25.34 Â Â 8.00 Â 765.12 Â128.07 Â 0.11 Â72.16
> dm-5 Â Â Â Â Â Â 0.00 Â Â 0.00 Â 33.40 5095.60 Â Â 0.13 Â Â19.90 Â Â 8.00 Â 431.38 Â 43.78 Â 0.17 Â85.20
>
> for comparison these are the numbers for seen when running the test
> with just the writing enabled
>
> Device:     rrqm/s  wrqm/s   r/s   w/s  ÂrMB/s  ÂwMB/s avgrq-sz avgqu-sz  await Âsvctm Â%util
> ---------------------------------------------------------------------------------------------------------
> dm-5 Â Â Â Â Â Â 0.00 Â Â 0.00 Â Â4.00 12047.40 Â Â0.02 Â Â47.06 Â Â 8.00 Â 989.81 Â 79.55 Â 0.07 Â79.20
> dm-5 Â Â Â Â Â Â 0.00 Â Â 0.00 Â Â3.40 12399.00 Â Â0.01 Â Â48.43 Â Â 8.00 Â 977.13 Â 72.15 Â 0.05 Â58.08
> dm-5 Â Â Â Â Â Â 0.00 Â Â 0.00 Â Â3.80 13130.00 Â Â0.01 Â Â51.29 Â Â 8.00 Â1130.48 Â 95.11 Â 0.04 Â58.48
> dm-5 Â Â Â Â Â Â 0.00 Â Â 0.00 Â Â2.40 5109.20 Â Â 0.01 Â Â19.96 Â Â 8.00 Â 427.75 Â 47.41 Â 0.16 Â79.92
> dm-5 Â Â Â Â Â Â 0.00 Â Â 0.00 Â Â3.20 Â Â0.00 Â Â 0.01 Â Â 0.00 Â Â 8.00 Â 290.33 148653.75 282.50 Â90.40
> dm-5 Â Â Â Â Â Â 0.00 Â Â 0.00 Â Â3.40 5103.00 Â Â 0.01 Â Â19.93 Â Â 8.00 Â 168.75 Â 33.06 Â 0.13 Â67.84
>
> And also with just the reading:
>
> Device:     rrqm/s  wrqm/s   r/s   w/s  ÂrMB/s  ÂwMB/s avgrq-sz avgqu-sz  await Âsvctm Â%util
> ---------------------------------------------------------------------------------------------------------
> dm-5 Â Â Â Â Â Â 0.00 Â Â 0.00 Â463.80 Â Â0.00 Â Â 1.81 Â Â 0.00 Â Â 8.00 Â Â 3.90 Â Â8.41 Â 2.16 100.00
> dm-5 Â Â Â Â Â Â 0.00 Â Â 0.00 Â434.20 Â Â0.00 Â Â 1.70 Â Â 0.00 Â Â 8.00 Â Â 3.89 Â Â8.95 Â 2.30 100.00
> dm-5 Â Â Â Â Â Â 0.00 Â Â 0.00 Â540.80 Â Â0.00 Â Â 2.11 Â Â 0.00 Â Â 8.00 Â Â 3.88 Â Â7.18 Â 1.85 100.00
> dm-5 Â Â Â Â Â Â 0.00 Â Â 0.00 Â591.60 Â Â0.00 Â Â 2.31 Â Â 0.00 Â Â 8.00 Â Â 3.84 Â Â6.50 Â 1.68 Â99.68
> dm-5 Â Â Â Â Â Â 0.00 Â Â 0.00 Â793.20 Â Â0.00 Â Â 3.10 Â Â 0.00 Â Â 8.00 Â Â 3.81 Â Â4.80 Â 1.26 100.00
> dm-5 Â Â Â Â Â Â 0.00 Â Â 0.00 Â592.80 Â Â0.00 Â Â 2.32 Â Â 0.00 Â Â 8.00 Â Â 3.84 Â Â6.47 Â 1.68 Â99.60
> dm-5 Â Â Â Â Â Â 0.00 Â Â 0.00 Â578.80 Â Â0.00 Â Â 2.26 Â Â 0.00 Â Â 8.00 Â Â 3.85 Â Â6.66 Â 1.73 100.00
> dm-5 Â Â Â Â Â Â 0.00 Â Â 0.00 Â771.00 Â Â0.00 Â Â 3.01 Â Â 0.00 Â Â 8.00 Â Â 3.81 Â Â4.93 Â 1.30 Â99.92
>

>From the numbers, it appears that your controller (or the disks) is
lying about writes. They get cached (service time around 0.1ms) and
reported as completed. When the cache is full, or a flush is issued,
you get >200ms delay.
Reads, instead, have a more reasonable service time of around 2ms.

Can you disable write cache, and see if the response times for writes
become reasonable?

> I also tested 2.6.31 for what happens when I run the same 'load' on a single lvm
> logical volume. Interestingly enough it is even worse than when doing it on three
> volumes:
>
> Device:     rrqm/s  wrqm/s   r/s   w/s  ÂrMB/s  ÂwMB/s avgrq-sz avgqu-sz  await Âsvctm Â%util
> ---------------------------------------------------------------------------------------------------------
> dm-5 Â Â Â Â Â Â 0.00 Â Â 0.00 Â 19.20 4566.80 Â Â 0.07 Â Â17.84 Â Â 8.00 Â2232.12 Â486.87 Â 0.22 Â99.92
> dm-5 Â Â Â Â Â Â 0.00 Â Â 0.00 Â Â6.80 9410.40 Â Â 0.03 Â Â36.76 Â Â 8.00 Â1827.47 Â187.99 Â 0.10 Â92.00
> dm-5 Â Â Â Â Â Â 0.00 Â Â 0.00 Â Â4.00 Â Â0.00 Â Â 0.02 Â Â 0.00 Â Â 8.00 Â 685.26 185618.40 249.60 Â99.84
> dm-5 Â Â Â Â Â Â 0.00 Â Â 0.00 Â Â4.20 4968.20 Â Â 0.02 Â Â19.41 Â Â 8.00 Â1426.45 Â286.86 Â 0.20 Â99.84
> dm-5 Â Â Â Â Â Â 0.00 Â Â 0.00 Â 10.60 9886.00 Â Â 0.04 Â Â38.62 Â Â 8.00 Â 167.57 Â Â5.74 Â 0.09 Â88.72
> dm-5 Â Â Â Â Â Â 0.00 Â Â 0.00 Â Â5.00 Â Â0.00 Â Â 0.02 Â Â 0.00 Â Â 8.00 Â1103.98 242774.88 199.68 Â99.84
> dm-5 Â Â Â Â Â Â 0.00 Â Â 0.00 Â 38.20 14794.60 Â Â0.15 Â Â57.79 Â Â 8.00 Â1171.75 Â 74.25 Â 0.06 Â87.20
>
> I also tested 2.6.31 with io-controller v9 patches. It seems to help a
> bit with the read rate, but the figures still jumps all over the place
>
> * 2.6.31 with io-controller patches v9 (8 cpu x86_64, 24 GB Ram)
> * fairness set to 1 on all block devices
> * scheduler = cfq
> * iostat -m -x dm-5 5
> * running in parallel on 3 lvm logical volumes
> Âon a single physical volume
>
> Device:     rrqm/s  wrqm/s   r/s   w/s  ÂrMB/s  ÂwMB/s avgrq-sz avgqu-sz  await Âsvctm Â%util
> ---------------------------------------------------------------------------------------------------------
> dm-5 Â Â Â Â Â Â Â0.00 Â Â 0.00 Â412.00 1640.60 Â Â 1.61 Â Â 6.41 Â Â 8.00 Â1992.54 1032.27 Â 0.49 Â99.84
> dm-5 Â Â Â Â Â Â Â0.00 Â Â 0.00 Â362.40 Â576.40 Â Â 1.42 Â Â 2.25 Â Â 8.00 Â 456.13 Â612.67 Â 1.07 100.00
> dm-5 Â Â Â Â Â Â Â0.00 Â Â 0.00 Â211.80 1004.40 Â Â 0.83 Â Â 3.92 Â Â 8.00 Â1186.20 Â995.00 Â 0.82 100.00
> dm-5 Â Â Â Â Â Â Â0.00 Â Â 0.00 Â 44.20 Â719.60 Â Â 0.17 Â Â 2.81 Â Â 8.00 Â 788.56 Â574.81 Â 1.31 Â99.76
> dm-5 Â Â Â Â Â Â Â0.00 Â Â 0.00 Â Â0.00 1274.80 Â Â 0.00 Â Â 4.98 Â Â 8.00 Â1584.07 1317.32 Â 0.78 100.00
> dm-5 Â Â Â Â Â Â Â0.00 Â Â 0.00 Â Â0.00 Â946.91 Â Â 0.00 Â Â 3.70 Â Â 8.00 Â 989.09 Â911.30 Â 1.05 Â99.72
> dm-5 Â Â Â Â Â Â Â0.00 Â Â 0.00 Â Â7.20 2526.00 Â Â 0.03 Â Â 9.87 Â Â 8.00 Â2085.57 Â201.72 Â 0.37 Â92.88
>
>
> For completeness sake I did the tests on 2.4.24 as well not much different.
>
> * 2.6.24 (8 cpu x86_64, 24 GB Ram)
> * scheduler = cfq
> * iostat -m -x dm-5 5
> * running in parallel on 3 lvm logical volumes
> Âsituated on a single physical volume
>
> Device:     rrqm/s  wrqm/s   r/s   w/s  ÂrMB/s  ÂwMB/s avgrq-sz avgqu-sz  await Âsvctm Â%util
> ---------------------------------------------------------------------------------------------------------
> dm-5 Â Â Â Â Â Â Â0.00 Â Â 0.00 Â144.60 5498.00 Â Â 0.56 Â Â21.48 Â Â 8.00 Â1215.82 Â215.48 Â 0.14 Â76.80
> dm-5 Â Â Â Â Â Â Â0.00 Â Â 0.00 Â 30.00 9831.40 Â Â 0.12 Â Â38.40 Â Â 8.00 Â1729.16 Â142.37 Â 0.09 Â88.60
> dm-5 Â Â Â Â Â Â Â0.00 Â Â 0.00 Â 27.60 4126.40 Â Â 0.11 Â Â16.12 Â Â 8.00 Â2245.24 Â618.77 Â 0.21 Â86.00
> dm-5 Â Â Â Â Â Â Â0.00 Â Â 0.00 Â Â2.00 3981.20 Â Â 0.01 Â Â15.55 Â Â 8.00 Â1069.07 Â268.40 Â 0.23 Â91.60
> dm-5 Â Â Â Â Â Â Â0.00 Â Â 0.00 Â 40.60 Â 13.20 Â Â 0.16 Â Â 0.05 Â Â 8.00 Â Â 3.98 Â 74.83 Â15.02 Â80.80
> dm-5 Â Â Â Â Â Â Â0.00 Â Â 0.00 Â Â5.60 5085.20 Â Â 0.02 Â Â19.86 Â Â 8.00 Â2586.65 Â508.10 Â 0.18 Â94.00
> dm-5 Â Â Â Â Â Â Â0.00 Â Â 0.00 Â 20.80 5344.60 Â Â 0.08 Â Â20.88 Â Â 8.00 Â 985.51 Â148.96 Â 0.17 Â92.60
>
>
> cheers
> tobi
>
> --
> Tobi Oetiker, OETIKER+PARTNER AG, Aarweg 15 CH-4600 Olten, Switzerland
> http://it.oetiker.ch tobi@xxxxxxxxxx ++41 62 775 9902 / sb: -9900
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@xxxxxxxxxxxxxxx
> More majordomo info at Âhttp://vger.kernel.org/majordomo-info.html
> Please read the FAQ at Âhttp://www.tux.org/lkml/
>



-- 
__________________________________________________________________________

dott. Corrado Zoccolo                          mailto:czoccolo@xxxxxxxxx
PhD - Department of Computer Science - University of Pisa, Italy
--------------------------------------------------------------------------
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/