Re: Linux Software RAID 5 Performance Optimizations: 2.6.19.1:(211MB/s read & 195MB/s write)

From: Justin Piszcz
Date: Fri Jan 12 2007 - 15:17:10 EST

Btw, max sectors did improve my performance a little bit but
stripe_cache+read_ahead were the main optimizations that made everything
go faster by about ~1.5x. I have individual bonnie++ benchmarks of
[only] the max_sector_kb tests as well, it improved the times from 8min/bonnie
run -> 7min 11 seconds or so, see below and then after that is what you
requested.

# Options used:
# blockdev --setra 1536 /dev/md3 (back to default)
# cat /sys/block/sd{e,g,i,k}/queue/max_sectors_kb
# value: 512
# value: 512
# value: 512
# value: 512
# Test with, chunksize of raid array (128)
# echo 128 > /sys/block/sde/queue/max_sectors_kb
# echo 128 > /sys/block/sdg/queue/max_sectors_kb
# echo 128 > /sys/block/sdi/queue/max_sectors_kb
# echo 128 > /sys/block/sdk/queue/max_sectors_kb

max_sectors_kb128_run1:max_sectors_kb128_run1,4000M,46522,98,109829,19,42776,12,46527,97,86206,14,647.7,1,16:100000:16/64,874,9,29123,97,2778,16,852,9,25399,86,1396,10
max_sectors_kb128_run2:max_sectors_kb128_run2,4000M,44037,99,107971,19,42420,12,46385,97,85773,14,628.8,1,16:100000:16/64,981,10,23006,77,3185,19,848,9,27891,94,1737,13
max_sectors_kb128_run3:max_sectors_kb128_run3,4000M,46501,98,108313,19,42558,12,46314,97,87697,15,617.0,1,16:100000:16/64,864,9,29795,99,2744,16,897,9,29021,98,1439,10
max_sectors_kb128_run4:max_sectors_kb128_run4,4000M,40750,98,108959,19,42519,12,45027,97,86484,14,637.0,1,16:100000:16/64,929,10,29641,98,2476,14,883,9,29529,99,1867,13
max_sectors_kb128_run5:max_sectors_kb128_run5,4000M,46664,98,108387,19,42801,12,46423,97,87379,14,642.5,0,16:100000:16/64,925,10,29756,99,2759,16,915,10,28694,97,1215,8

162.54user 43.96system 7:12.02elapsed 47%CPU (0avgtext+0avgdata
0maxresident)k
0inputs+0outputs (5major+1104minor)pagefaults 0swaps
168.75user 43.51system 7:14.49elapsed 48%CPU (0avgtext+0avgdata
0maxresident)k
0inputs+0outputs (13major+1092minor)pagefaults 0swaps
162.76user 44.18system 7:12.26elapsed 47%CPU (0avgtext+0avgdata
0maxresident)k
0inputs+0outputs (13major+1096minor)pagefaults 0swaps
178.91user 43.39system 7:24.39elapsed 50%CPU (0avgtext+0avgdata
0maxresident)k
0inputs+0outputs (13major+1094minor)pagefaults 0swaps
162.45user 43.86system 7:11.26elapsed 47%CPU (0avgtext+0avgdata
0maxresident)k
0inputs+0outputs (13major+1092minor)pagefaults 0swaps

---------------

# cat /sys/block/sd[abcdefghijk]/queue/*
cat: /sys/block/sda/queue/iosched: Is a directory
32767
512
128
128
noop [anticipatory]
cat: /sys/block/sdb/queue/iosched: Is a directory
32767
512
128
128
noop [anticipatory]
cat: /sys/block/sdc/queue/iosched: Is a directory
32767
128
128
128
noop [anticipatory]
cat: /sys/block/sdd/queue/iosched: Is a directory
32767
128
128
128
noop [anticipatory]
cat: /sys/block/sde/queue/iosched: Is a directory
32767
128
128
128
noop [anticipatory]
cat: /sys/block/sdf/queue/iosched: Is a directory
32767
128
128
128
noop [anticipatory]
cat: /sys/block/sdg/queue/iosched: Is a directory
32767
128
128
128
noop [anticipatory]
cat: /sys/block/sdh/queue/iosched: Is a directory
32767
128
128
128
noop [anticipatory]
cat: /sys/block/sdi/queue/iosched: Is a directory
32767
128
128
128
noop [anticipatory]
cat: /sys/block/sdj/queue/iosched: Is a directory
32767
128
128
128
noop [anticipatory]
cat: /sys/block/sdk/queue/iosched: Is a directory
32767
128
128
128
noop [anticipatory]
#

(note I am only using four of these (which are raptors, in raid5 for md3))

# cat /proc/meminfo
MemTotal: 2048904 kB
MemFree: 1299980 kB
Buffers: 1408 kB
Cached: 58032 kB
SwapCached: 0 kB
Active: 65012 kB
Inactive: 33796 kB
HighTotal: 1153312 kB
HighFree: 1061792 kB
LowTotal: 895592 kB
LowFree: 238188 kB
SwapTotal: 2200760 kB
SwapFree: 2200760 kB
Dirty: 8 kB
Writeback: 0 kB
AnonPages: 39332 kB
Mapped: 20248 kB
Slab: 37116 kB
SReclaimable: 10580 kB
SUnreclaim: 26536 kB
PageTables: 1284 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
CommitLimit: 3225212 kB
Committed_AS: 111056 kB
VmallocTotal: 114680 kB
VmallocUsed: 3828 kB
VmallocChunk: 110644 kB
#

# echo 3 > /proc/sys/vm/drop_caches
# dd if=/dev/md3 of=/dev/null bs=1M count=10240
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB) copied, 399.352 seconds, 26.9 MB/s
# for i in sde sdg sdi sdk; do echo 192 >
/sys/block/"$i"/queue/max_sectors_kb; echo "Set
/sys/block/"$i"/queue/max_sectors_kb to 192kb"; done
Set /sys/block/sde/queue/max_sectors_kb to 192kb
Set /sys/block/sdg/queue/max_sectors_kb to 192kb
Set /sys/block/sdi/queue/max_sectors_kb to 192kb
Set /sys/block/sdk/queue/max_sectors_kb to 192kb
# echo 3 > /proc/sys/vm/drop_caches
# dd if=/dev/md3 of=/dev/null bs=1M count=10240
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB) copied, 398.069 seconds, 27.0 MB/s

Awful performance with your numbers/drop_caches settings.. !

What were your tests designed to show?


Justin.

On Fri, 12 Jan 2007, Justin Piszcz wrote:

>
>
> On Fri, 12 Jan 2007, Al Boldi wrote:
>
> > Justin Piszcz wrote:
> > > RAID 5 TWEAKED: 1:06.41 elapsed @ 60% CPU
> > >
> > > This should be 1:14 not 1:06(was with a similarly sized file but not the
> > > same) the 1:14 is the same file as used with the other benchmarks. and to
> > > get that I used 256mb read-ahead and 16384 stripe size ++ 128
> > > max_sectors_kb (same size as my sw raid5 chunk size)
> >
> > max_sectors_kb is probably your key. On my system I get twice the read
> > performance by just reducing max_sectors_kb from default 512 to 192.
> >
> > Can you do a fresh reboot to shell and then:
> > $ cat /sys/block/hda/queue/*
> > $ cat /proc/meminfo
> > $ echo 3 > /proc/sys/vm/drop_caches
> > $ dd if=/dev/hda of=/dev/null bs=1M count=10240
> > $ echo 192 > /sys/block/hda/queue/max_sectors_kb
> > $ echo 3 > /proc/sys/vm/drop_caches
> > $ dd if=/dev/hda of=/dev/null bs=1M count=10240
> >
> >
> > Thanks!
> >
> > --
> > Al
> >
> > -
> > To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> > the body of a message to majordomo@xxxxxxxxxxxxxxx
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
> >
>
> Ok. sec
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@xxxxxxxxxxxxxxx
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/