Re: AIO, FIO and Threads ...

[Davide Libenzi]

On Wed, 21 Mar 2007, Jens Axboe wrote:

> On Tue, Mar 20 2007, Davide Libenzi wrote:
> > 
> > I was looking at Jens FIO stuff, and I decided to cook a quick patch for 
> > FIO to support GUASI (Generic Userspace Asyncronous Syscall Interface):
> > 
> > http://www.xmailserver.org/guasi-lib.html
> > 
> > I then ran a few tests on my Dual Opteron 252 with SATA drives (sata_nv) 
> > and 8GB of RAM.
> > Mind that I'm not FIO expert, like at all, but I got some interesting 
> > results when comparing GUASI with libaio at 8/1000/10000 depths.
> > If I read those result correctly (Jens may help), GUASI output is more 
> > then double the libaio one.
> > Lots of context switches, yes. But the throughput looks like 2+ times.
> > Can someone try to repeat the measures and/or spot the error?
> > Or tell me which other tests to run?
> > This is kinda a suprise for me ...
> 
> I don't know guasi at all, but libaio requires O_DIRECT to be async. I'm
> sure you know this, but you may not know that fio default to buffered IO
> so you have to tell it to use O_DIRECT :-)
> 
> So try adding a --direct=1 (or --buffered=0, same thing) as an extra
> option when comparing depths > 1.

This is a much smaller box. P4 with 1GB of RAM, and on a 1GB RW test. 
Now libaio performs well, even though GUASI can keep the pace. Quite a 
surprise, nonetheless ...




- Davide



*** fio --name=global --rw=randrw --size=1g --bs=4k --direct=1 
        --ioengine=guasi --name=job1 --iodepth=100 --thread --runtime=20
job1: (g=0): rw=randrw, bs=4K-4K/4K-4K, ioengine=guasi, iodepth=100
Starting 1 thread
Jobs: 1: [m] [100.0% done] [   609/     0 kb/s] [eta 00m:00s]
job1: (groupid=0, jobs=1): err= 0: pid=21862
  read : io=8,300KiB, bw=412KiB/s, iops=100, runt= 20599msec
    slat (msec): min=    0, max=    2, avg= 0.04, stdev= 0.29
    clat (msec): min=   11, max= 1790, avg=564.76, stdev=350.58
    bw (KiB/s) : min=   47, max=  692, per=121.80%, avg=501.83, stdev=196.86
  write: io=11,344KiB, bw=563KiB/s, iops=137, runt= 20599msec
    slat (msec): min=    0, max=    2, avg= 0.04, stdev= 0.28
    clat (msec): min=    2, max=  643, avg=311.86, stdev=108.85
    bw (KiB/s) : min=    0, max= 1695, per=143.52%, avg=808.00, stdev=632.11
  cpu          : usr=0.19%, sys=1.94%, ctx=28036
  IO depths    : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.2%, 16=0.3%, 32=0.7%, >=64=98.7%
     lat (msec): 2=0.0%, 4=0.0%, 10=0.0%, 20=0.3%, 50=1.2%, 100=2.2%
     lat (msec): 250=16.4%, 500=51.4%, 750=16.8%, 1000=6.5%, >=2000=5.1%

Run status group 0 (all jobs):
   READ: io=8,300KiB, aggrb=412KiB/s, minb=412KiB/s, maxb=412KiB/s, mint=20599msec, maxt=20599msec
  WRITE: io=11,344KiB, aggrb=563KiB/s, minb=563KiB/s, maxb=563KiB/s, mint=20599msec, maxt=20599msec

Disk stats (read/write):
  sda: ios=2074/2846, merge=1/26, ticks=945564/14060, in_queue=959624, util=97.65%



*** fio --name=global --rw=randrw --size=1g --bs=4k --direct=1 
        --ioengine=libaio --name=job1 --iodepth=100 --thread --runtime=20
job1: (g=0): rw=randrw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=100
Starting 1 thread
Jobs: 1: [m] [100.0% done] [   406/   438 kb/s] [eta 00m:00s]
job1: (groupid=0, jobs=1): err= 0: pid=21860
  read : io=8,076KiB, bw=403KiB/s, iops=98, runt= 20495msec
    slat (msec): min=    0, max=  494, avg= 0.55, stdev=14.75
    clat (msec): min=    0, max= 1788, avg=509.38, stdev=391.43
    bw (KiB/s) : min=   20, max=  682, per=104.55%, avg=421.32, stdev=153.91
  write: io=11,024KiB, bw=550KiB/s, iops=134, runt= 20495msec
    slat (msec): min=    0, max=  441, avg= 0.23, stdev= 8.40
    clat (msec): min=    0, max= 1695, avg=368.51, stdev=308.11
    bw (KiB/s) : min=    0, max= 1787, per=105.78%, avg=581.78, stdev=438.43
  cpu          : usr=0.06%, sys=0.76%, ctx=6185
  IO depths    : 1=0.1%, 2=0.2%, 4=0.3%, 8=0.7%, 16=1.3%, 32=2.7%, >=64=94.7%
     lat (msec): 2=0.4%, 4=0.1%, 10=0.7%, 20=2.5%, 50=7.0%, 100=9.4%
     lat (msec): 250=20.2%, 500=23.2%, 750=17.8%, 1000=10.9%, >=2000=7.9%

Run status group 0 (all jobs):
   READ: io=8,076KiB, aggrb=403KiB/s, minb=403KiB/s, maxb=403KiB/s, mint=20495msec, maxt=20495msec
  WRITE: io=11,024KiB, aggrb=550KiB/s, minb=550KiB/s, maxb=550KiB/s, mint=20495msec, maxt=20495msec

Disk stats (read/write):
  sda: ios=2019/2788, merge=0/38, ticks=988100/1048096, in_queue=2036196, util=99.38%



*** fio --name=global --rw=randrw --size=1g --bs=4k --direct=1 
        --ioengine=guasi --name=job1 --iodepth=1000 --thread --runtime=20
job1: (g=0): rw=randrw, bs=4K-4K/4K-4K, ioengine=guasi, iodepth=1000
Starting 1 thread
Jobs: 1: [m] [1.9% done] [  2348/  1710 kb/s] [eta 21m:00s]s]
job1: (groupid=0, jobs=1): err= 0: pid=19471
  read : io=10,640KiB, bw=436KiB/s, iops=106, runt= 24972msec
    slat (msec): min=    0, max=   26, avg= 3.46, stdev= 7.68
    clat (msec): min=   27, max= 9800, avg=5048.47, stdev=1728.19
    bw (KiB/s) : min=   44, max=  689, per=98.52%, avg=429.54, stdev=190.74
  write: io=9,748KiB, bw=399KiB/s, iops=97, runt= 24972msec
    slat (msec): min=    0, max=   26, avg= 4.04, stdev= 8.21
    clat (msec): min=    9, max= 9153, avg=4718.51, stdev=1692.57
    bw (KiB/s) : min=    0, max= 1586, per=109.27%, avg=436.00, stdev=395.51
  cpu          : usr=0.24%, sys=1.57%, ctx=20661
  IO depths    : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.2%, 16=0.3%, 32=0.6%, >=64=98.8%
     lat (msec): 2=0.0%, 4=0.0%, 10=0.0%, 20=0.0%, 50=0.2%, 100=0.1%
     lat (msec): 250=0.5%, 500=0.4%, 750=1.1%, 1000=1.0%, >=2000=3.8%

Run status group 0 (all jobs):
   READ: io=10,640KiB, aggrb=436KiB/s, minb=436KiB/s, maxb=436KiB/s, mint=24972msec, maxt=24972msec
  WRITE: io=9,748KiB, aggrb=399KiB/s, minb=399KiB/s, maxb=399KiB/s, mint=24972msec, maxt=24972msec

Disk stats (read/write):
  sda: ios=2660/2449, merge=0/27, ticks=1016188/22112, in_queue=1038296, util=97.60%


*** fio --name=global --rw=randrw --size=1g --bs=4k --direct=1 
        --ioengine=libaio --name=job1 --iodepth=1000 --thread --runtime=20
job1: (g=0): rw=randrw, bs=4K-4K/4K-4K, ioengine=libaio, iodepth=1000
Starting 1 thread
Jobs: 1: [m] [1.8% done] [  3502/     0 kb/s] [eta 20m:01s]s]
job1: (groupid=0, jobs=1): err= 0: pid=19723
  read : io=10,944KiB, bw=493KiB/s, iops=120, runt= 22697msec
    slat (msec): min=  363, max= 5490, avg=3917.46, stdev=1842.45
    clat (msec): min=    0, max= 5491, avg=980.56, stdev=1764.36
    bw (KiB/s) : min=    1, max= 4091, per=211.44%, avg=1042.38, stdev=1517.27
  write: io=8,468KiB, bw=382KiB/s, iops=93, runt= 22697msec
    slat (msec): min=    0, max= 3025, avg=2864.85, stdev=115.63
    clat (msec): min=    0, max= 6104, avg=381.86, stdev=1124.52
    bw (KiB/s) : min=    0, max= 3046, per=196.70%, avg=751.40, stdev=1305.17
  cpu          : usr=0.04%, sys=0.62%, ctx=2148
  IO depths    : 1=0.0%, 2=0.0%, 4=0.1%, 8=0.2%, 16=0.3%, 32=0.7%, >=64=98.7%
     lat (msec): 2=74.0%, 4=0.1%, 10=0.1%, 20=0.0%, 50=0.0%, 100=0.0%
     lat (msec): 250=0.0%, 500=2.8%, 750=0.0%, 1000=4.6%, >=2000=2.6%

Run status group 0 (all jobs):
   READ: io=10,944KiB, aggrb=493KiB/s, minb=493KiB/s, maxb=493KiB/s, mint=22697msec, maxt=22697msec
  WRITE: io=8,468KiB, aggrb=382KiB/s, minb=382KiB/s, maxb=382KiB/s, mint=22697msec, maxt=22697msec

Disk stats (read/write):
  sda: ios=2734/2135, merge=2/36, ticks=2345836/1359304, in_queue=3722228, util=99.23%



-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/