Re: [PATCH] cfq-iosched: reduce write depth only if sync was delayed

From: Corrado Zoccolo
Date: Tue Dec 08 2009 - 15:43:25 EST

On Tue, Dec 8, 2009 at 1:07 AM, Jeff Moyer <jmoyer@xxxxxxxxxx> wrote:
> Corrado Zoccolo <czoccolo@xxxxxxxxx> writes:
>
>> Hi Jeff, I remember you saw large performance drop on your SAN for
>> sequential writes with low_latency=1. ÂCan you test if Shaohua's and
>> this patch fix allow to recover some bandwidth? ÂI think that enabling
>> the queue depth ramp up only if a sync request was delayed should
>> disable it for fast hardware like yours, so you should not be seeing
>> the slowdown any more.
>
> Average of 10 runs. ÂLow latency set to 0:
>
> Unit information
> ================
> File size = megabytes
> Blk Size Â= bytes
> Rate   Â= megabytes per second
> CPU% Â Â Â= percentage of CPU used during the test
> Latency  = milliseconds
> Lat% Â Â Â= percent of requests that took longer than X seconds
> CPU Eff  = Rate divided by CPU% - throughput per cpu load
>
> Sequential Reads
>               ÂFile ÂBlk  Num          Avg   ÂMaximum   ÂLat%   Lat%  ÂCPU
> Identifier          ÂSize ÂSize ÂThr  Rate Â(CPU%) ÂLatency  ÂLatency   Â>2s   Â>10s  ÂEff
> ---------------------------- ------ ----- --- Â------ ------ --------- ----------- Â-------- -------- -----
> 2.6.32 Â Â Â Â Â Â Â Â Â Â Â Â8192 Â65536 Â 8 Â 90.08 94.51% Â Â51.316 Â Â10268.25 Â 0.00000 Â0.00000 Â Â95
> 2.6.32 Â Â Â Â Â Â Â Â Â Â Â Â8192 Â65536 Â16 Â 99.36 199.8% Â Â89.248 Â Â13883.81 Â 0.00000 Â0.00000 Â Â50
>
> Random Reads
>               ÂFile ÂBlk  Num          Avg   ÂMaximum   ÂLat%   Lat%  ÂCPU
> Identifier          ÂSize ÂSize ÂThr  Rate Â(CPU%) ÂLatency  ÂLatency   Â>2s   Â>10s  ÂEff
> ---------------------------- ------ ----- --- Â------ ------ --------- ----------- Â-------- -------- -----
> 2.6.32 Â Â Â Â Â Â Â Â Â Â Â Â8192 Â65536 Â 8 Â 81.16 85.90% Â Â45.672 Â Â 5963.19 Â 0.00000 Â0.00000 Â Â94
> 2.6.32 Â Â Â Â Â Â Â Â Â Â Â Â8192 Â65536 Â16 Â116.32 230.3% Â Â58.371 Â Â 6098.36 Â 0.00000 Â0.00000 Â Â51
>
> Sequential Writes
>               ÂFile ÂBlk  Num          Avg   ÂMaximum   ÂLat%   Lat%  ÂCPU
> Identifier          ÂSize ÂSize ÂThr  Rate Â(CPU%) ÂLatency  ÂLatency   Â>2s   Â>10s  ÂEff
> ---------------------------- ------ ----- --- Â------ ------ --------- ----------- Â-------- -------- -----
> 2.6.32 Â Â Â Â Â Â Â Â Â Â Â Â8192 Â65536 Â 8 Â112.17 1085.% Â Â42.623 Â Â17114.54 Â 0.00152 Â0.00000 Â Â10
> 2.6.32 Â Â Â Â Â Â Â Â Â Â Â Â8192 Â65536 Â16 Â111.26 2117.% Â Â84.964 Â Â26480.60 Â 0.03202 Â0.00000 Â Â 5
>
> Random Writes
>               ÂFile ÂBlk  Num          Avg   ÂMaximum   ÂLat%   Lat%  ÂCPU
> Identifier          ÂSize ÂSize ÂThr  Rate Â(CPU%) ÂLatency  ÂLatency   Â>2s   Â>10s  ÂEff
> ---------------------------- ------ ----- --- Â------ ------ --------- ----------- Â-------- -------- -----
> 2.6.32 Â Â Â Â Â Â Â Â Â Â Â Â8192 Â65536 Â 8 Â162.48 887.1% Â Â 6.106 Â Â Â313.54 Â 0.00000 Â0.00000 Â Â18
> 2.6.32 Â Â Â Â Â Â Â Â Â Â Â Â8192 Â65536 Â16 Â156.38 1767.% Â Â14.077 Â Â 1254.62 Â 0.00000 Â0.00000 Â Â 9
>
>
> Average of 10 runs. ÂLow latency set to 1:
>
> Unit information
> ================
> File size = megabytes
> Blk Size Â= bytes
> Rate   Â= megabytes per second
> CPU% Â Â Â= percentage of CPU used during the test
> Latency  = milliseconds
> Lat% Â Â Â= percent of requests that took longer than X seconds
> CPU Eff  = Rate divided by CPU% - throughput per cpu load
>
> Sequential Reads
>               ÂFile ÂBlk  Num          Avg   ÂMaximum   ÂLat%   Lat%  ÂCPU
> Identifier          ÂSize ÂSize ÂThr  Rate Â(CPU%) ÂLatency  ÂLatency   Â>2s   Â>10s  ÂEff
> ---------------------------- ------ ----- --- Â------ ------ --------- ----------- Â-------- -------- -----
> 2.6.32 Â Â Â Â Â Â Â Â Â Â Â Â8192 Â65536 Â 8 Â 92.23 100.1% Â Â52.119 Â Â 6920.18 Â 0.00000 Â0.00000 Â Â92
> 2.6.32 Â Â Â Â Â Â Â Â Â Â Â Â8192 Â65536 Â16 Â 97.88 217.0% Â Â99.691 Â Â 7453.18 Â 0.00000 Â0.00000 Â Â45
>
> Random Reads
>               ÂFile ÂBlk  Num          Avg   ÂMaximum   ÂLat%   Lat%  ÂCPU
> Identifier          ÂSize ÂSize ÂThr  Rate Â(CPU%) ÂLatency  ÂLatency   Â>2s   Â>10s  ÂEff
> ---------------------------- ------ ----- --- Â------ ------ --------- ----------- Â-------- -------- -----
> 2.6.32 Â Â Â Â Â Â Â Â Â Â Â Â8192 Â65536 Â 8 Â 98.70 107.5% Â Â42.994 Â Â 3409.08 Â 0.00000 Â0.00000 Â Â92
> 2.6.32 Â Â Â Â Â Â Â Â Â Â Â Â8192 Â65536 Â16 Â140.41 323.9% Â Â59.616 Â Â 4525.46 Â 0.00000 Â0.00000 Â Â43
>
> Sequential Writes
>               ÂFile ÂBlk  Num          Avg   ÂMaximum   ÂLat%   Lat%  ÂCPU
> Identifier          ÂSize ÂSize ÂThr  Rate Â(CPU%) ÂLatency  ÂLatency   Â>2s   Â>10s  ÂEff
> ---------------------------- ------ ----- --- Â------ ------ --------- ----------- Â-------- -------- -----
> 2.6.32 Â Â Â Â Â Â Â Â Â Â Â Â8192 Â65536 Â 8 Â112.33 1076.% Â Â42.617 Â Â17072.28 Â 0.00076 Â0.00000 Â Â10
> 2.6.32 Â Â Â Â Â Â Â Â Â Â Â Â8192 Â65536 Â16 Â111.84 2097.% Â Â85.156 Â Â28221.77 Â 0.02976 Â0.00000 Â Â 5
>
> Random Writes
>               ÂFile ÂBlk  Num          Avg   ÂMaximum   ÂLat%   Lat%  ÂCPU
> Identifier          ÂSize ÂSize ÂThr  Rate Â(CPU%) ÂLatency  ÂLatency   Â>2s   Â>10s  ÂEff
> ---------------------------- ------ ----- --- Â------ ------ --------- ----------- Â-------- -------- -----
> 2.6.32 Â Â Â Â Â Â Â Â Â Â Â Â8192 Â65536 Â 8 Â159.33 870.5% Â Â 6.469 Â Â Â765.50 Â 0.00000 Â0.00000 Â Â18
> 2.6.32 Â Â Â Â Â Â Â Â Â Â Â Â8192 Â65536 Â16 Â141.60 1632.% Â Â15.364 Â Â 2337.57 Â 0.00000 Â0.00000 Â Â 9
>
>
> Cheers,
> Jeff
>

The numbers look good. Now, there is no penalty in having low_latency
set for sequential writes, and just a small penalty for random ones.
The fact that random reads are faster with low_latency set is interesting.
Is the test is running with your patched tiobench (so that the number
of random operations is comparable with sequential ones)?

Thanks,
Corrado

--
__________________________________________________________________________

dott. Corrado Zoccolo mailto:czoccolo@xxxxxxxxx
PhD - Department of Computer Science - University of Pisa, Italy
--------------------------------------------------------------------------
The self-confidence of a warrior is not the self-confidence of the average
man. The average man seeks certainty in the eyes of the onlooker and calls
that self-confidence. The warrior seeks impeccability in his own eyes and
calls that humbleness.
Tales of Power - C. Castaneda
èº{.nÇ+‰·Ÿ®‰­†+%ŠËlzwm…ébëæìr¸›zX§»®w¥Š{ayºÊÚë,j­¢f£¢·hš‹àz¹®w¥¢¸ ¢·¦j:+v‰¨ŠwèjØm¶Ÿÿ¾«‘êçzZ+ƒùšŽŠÝj"ú!¶iO•æ¬z·švØ^¶m§ÿðà nÆàþY&—