Re: performance changes on 4400755e: 200.0% fsmark.files_per_sec, -18.1% fsmark.files_per_sec, and few more

[NeilBrown]

On Wed, 18 Mar 2015 13:00:30 +0800 Yuanahn Liu <yuanhan.liu@xxxxxxxxxxxxxxx>
wrote:

> Hi,
> 
> FYI, we noticed performance changes on `fsmark.files_per_sec' by 4400755e356f9a2b0b7ceaa02f57b1c7546c3765:
> 
>     > commit 4400755e356f9a2b0b7ceaa02f57b1c7546c3765
>     > Author:     NeilBrown <neilb@xxxxxxx>
>     > AuthorDate: Thu Feb 26 12:47:56 2015 +1100
>     > Commit:     NeilBrown <neilb@xxxxxxx>
>     > CommitDate: Wed Mar 4 13:40:19 2015 +1100
>     > 
>     >     md/raid5: allow the stripe_cache to grow and shrink.

Thanks a lot for this testing!!! I was wondering how I could do some proper
testing of this patch, and you've done it for me :-)

The large number of improvements is very encouraging - that is what I was
hoping for of course.

The few regressions could be a concern.  I note that are all NoSync.
That seems to suggest that they could just be writing more data.
i.e. the data is written a bit earlier (certainly possible) so it happen to
introduce more delay ....

I guess I'm not really sure how to interpret NoSync results, and suspect that
poor NoSync result don't really reflect much on the underlying block device.
Could that be right?

Also, I'm a little confused by the
   fsmark.time.involuntary_context_switches
statistic:

>       1235 Â  2%     -47.5%        649 Â  3%  fsmark.time.percent_of_cpu_this_job_got

>        399 Â  4%     -20.0%        319 Â  3%  fsmark.time.percent_of_cpu_this_job_got

Does that means that the ext4 test changed from 12.4 cpus to 6.4, and that
the btrfs test chnages from 4 cpus to 3.2 ???

Or does it just not mean anything?

Thanks,
NeilBrown




> 
> 26089f4902595a2f64c512066af07af6e82eb096     4400755e356f9a2b0b7ceaa02f57b1c7546c3765
> ----------------------------------------     ----------------------------------------
> run time(m)     metric_value     Âstddev     run time(m)     metric_value     Âstddev     change   testbox/benchmark/sub-testcase
> --- ------  ----------------------------     --- ------  ----------------------------     -------- ------------------------------
> 3   18.6               6.400     Â0.0%       5   9.2               19.200     Â0.0%         200.0% ivb44/fsmark/1x-64t-3HDD-RAID5-ext4-4M-40G-fsyncBeforeClose
> 3   24.7               6.400     Â0.0%       3   13.7              12.800     Â0.0%         100.0% ivb44/fsmark/1x-64t-3HDD-RAID5-f2fs-4M-40G-fsyncBeforeClose
> 3   17.5              28.267     Â9.6%       3   12.3              42.833     Â6.5%          51.5% ivb44/fsmark/1x-64t-3HDD-RAID5-f2fs-4M-120G-NoSync
> 3   16.7              30.700     Â1.5%       3   12.6              40.733     Â2.4%          32.7% ivb44/fsmark/1x-1t-3HDD-RAID5-f2fs-4M-120G-NoSync
> 3   29.0               5.867     Â0.8%       5   23.6               7.240     Â0.7%          23.4% ivb44/fsmark/1x-1t-3HDD-RAID5-btrfs-4M-40G-fsyncBeforeClose
> 3   28.5               6.000     Â0.0%       3   23.2               7.367     Â0.6%          22.8% ivb44/fsmark/1x-1t-3HDD-RAID5-f2fs-4M-40G-fsyncBeforeClose
> 5   11.7              14.600     Â0.0%       5   9.7               17.500     Â0.4%          19.9% ivb44/fsmark/1x-1t-3HDD-RAID5-xfs-4M-40G-fsyncBeforeClose
> 3   22.4              25.600     Â0.0%       5   17.9              30.120     Â4.1%          17.7% ivb44/fsmark/1x-64t-3HDD-RAID5-xfs-4M-120G-NoSync
> 5   10.8              47.320     Â0.6%       5   9.3               54.820     Â0.2%          15.8% ivb44/fsmark/1x-1t-3HDD-RAID5-xfs-4M-120G-NoSync
> 1   0.5              252.400     Â0.0%       1   0.5              263.300     Â0.0%           4.3% ivb44/fsmark/1x-1t-4BRD_12G-RAID5-ext4-4M-30G-NoSync
> 
> 3   0.5              273.100     Â4.3%       3   0.6              223.567     Â6.5%         -18.1% ivb44/fsmark/1x-64t-4BRD_12G-RAID5-btrfs-4M-30G-NoSync
> 3   8.1               63.133     Â0.5%       3   9.2               55.633     Â0.2%         -11.9% ivb44/fsmark/1x-1t-3HDD-RAID5-btrfs-4M-120G-NoSync
> 3   8.2               64.000     Â0.0%       3   9.2               57.600     Â0.0%         -10.0% ivb44/fsmark/1x-64t-3HDD-RAID5-btrfs-4M-120G-NoSync
> 
> 
> NOTE: here are some more info about those test parameters for you to
>       understand the testcase better:
> 
>       1x: where 'x' means iterations or loop, corresponding to the 'L' option of fsmark
>       1t, 64t: where 't' means thread
>       4M: means the single file size, corresponding to the '-s' option of fsmark
>       40G, 30G, 120G: means the total test size
> 
>       4BRD_12G: BRD is the ramdisk, where '4' means 4 ramdisk, and where '12G' means
>                 the size of one ramdisk. So, it would be 48G in total. And we made a
> 		raid on those ramdisk.
> 
> 
> As you can see from above data, interestingly, all performance
> regressions come from btrfs testing. That's why Chris is also
> in the cc list, with which just FYI.
> 
> 
> FYI, here I listed more detailed changes for the maximal postive and negtive changes.
> 
> more detailed changes about ivb44/fsmark/1x-64t-3HDD-RAID5-ext4-4M-40G-fsyncBeforeClose
> ---------
> 
> 26089f4902595a2f  4400755e356f9a2b0b7ceaa02f  
> ----------------  --------------------------  
>          %stddev     %change         %stddev
>              \          |                \  
>       6.40 Â  0%    +200.0%      19.20 Â  0%  fsmark.files_per_sec
>  1.015e+08 Â  1%     -73.6%   26767355 Â  3%  fsmark.time.voluntary_context_switches
>      13793 Â  1%     -73.9%       3603 Â  5%  fsmark.time.system_time
>      78473 Â  6%     -64.3%      28016 Â  7%  fsmark.time.involuntary_context_switches
>   15789555 Â  9%     -54.7%    7159485 Â 13%  fsmark.app_overhead
>       1115 Â  0%     -50.3%        554 Â  1%  fsmark.time.elapsed_time.max
>       1115 Â  0%     -50.3%        554 Â  1%  fsmark.time.elapsed_time
>       1235 Â  2%     -47.5%        649 Â  3%  fsmark.time.percent_of_cpu_this_job_got
>     456465 Â  1%     -26.7%     334594 Â  4%  fsmark.time.minor_page_faults
>        275 Â  0%   +1257.7%       3733 Â  2%  slabinfo.raid5-md0.num_objs
>        275 Â  0%   +1257.7%       3733 Â  2%  slabinfo.raid5-md0.active_objs
>         11 Â  0%   +1250.9%        148 Â  2%  slabinfo.raid5-md0.active_slabs
>         11 Â  0%   +1250.9%        148 Â  2%  slabinfo.raid5-md0.num_slabs
>       2407 Â  4%    +293.4%       9471 Â 26%  numa-meminfo.node0.Writeback
>        600 Â  4%    +294.9%       2372 Â 26%  numa-vmstat.node0.nr_writeback
>    1114505 Â  0%     -77.4%     251696 Â  2%  softirqs.TASKLET
>    1808027 Â  1%     -77.7%     402378 Â  4%  softirqs.RCU
>   12158665 Â  1%     -77.1%    2786069 Â  4%  cpuidle.C3-IVT.usage
>    1119433 Â  0%     -77.3%     254192 Â  2%  softirqs.BLOCK
>   37824202 Â  1%     -75.1%    9405078 Â  4%  cpuidle.C6-IVT.usage
>  1.015e+08 Â  1%     -73.6%   26767355 Â  3%  time.voluntary_context_switches
>      13793 Â  1%     -73.9%       3603 Â  5%  time.system_time
>    5971084 Â  1%     -73.6%    1574912 Â  5%  softirqs.SCHED
>   10539492 Â  3%     -72.0%    2956258 Â  6%  cpuidle.C1E-IVT.usage
>          2 Â  0%    +230.0%          6 Â 12%  vmstat.procs.b
>      14064 Â  1%     -71.2%       4049 Â  6%  softirqs.HRTIMER
>    7388306 Â  1%     -71.2%    2129929 Â  4%  softirqs.TIMER
>  3.496e+09 Â  1%     -70.3%   1.04e+09 Â  1%  cpuidle.C3-IVT.time
>       0.88 Â  6%    +224.9%       2.87 Â 11%  turbostat.Pkg%pc6
>   19969464 Â  2%     -66.2%    6750675 Â  5%  cpuidle.C1-IVT.usage
>      78473 Â  6%     -64.3%      28016 Â  7%  time.involuntary_context_switches
>       4.23 Â  5%    +181.4%      11.90 Â  3%  turbostat.Pkg%pc2
>  2.551e+09 Â  1%     -61.4%  9.837e+08 Â  3%  cpuidle.C1E-IVT.time
>       8084 Â  3%    +142.6%      19608 Â  3%  meminfo.Writeback
>       2026 Â  4%    +141.6%       4895 Â  4%  proc-vmstat.nr_writeback
>        165 Â  4%     -56.9%         71 Â 14%  numa-vmstat.node1.nr_inactive_anon
>  7.748e+09 Â  3%     -50.3%  3.852e+09 Â  3%  cpuidle.C1-IVT.time
>        175 Â  5%     -53.2%         82 Â 13%  numa-vmstat.node1.nr_shmem
>       1115 Â  0%     -50.3%        554 Â  1%  time.elapsed_time.max
>       1115 Â  0%     -50.3%        554 Â  1%  time.elapsed_time
>       1147 Â  0%     -49.0%        585 Â  1%  uptime.boot
>    2260889 Â  0%     -48.8%    1157272 Â  1%  proc-vmstat.pgfree
>      16805 Â  2%     -35.9%      10776 Â 23%  numa-vmstat.node1.nr_dirty
>       1235 Â  2%     -47.5%        649 Â  3%  time.percent_of_cpu_this_job_got
>      67245 Â  2%     -35.9%      43122 Â 23%  numa-meminfo.node1.Dirty
>      39041 Â  0%     -45.7%      21212 Â  2%  uptime.idle
>         13 Â  9%     -49.0%          6 Â 11%  vmstat.procs.r
>       3072 Â 10%     -40.3%       1833 Â  9%  cpuidle.POLL.usage
>    3045115 Â  0%     -46.1%    1642053 Â  1%  proc-vmstat.pgfault
>        202 Â  1%     -45.2%        110 Â  0%  proc-vmstat.nr_inactive_anon
>    4583079 Â  2%     -31.4%    3143602 Â 16%  numa-vmstat.node1.numa_hit
>      28.03 Â  0%     +69.1%      47.39 Â  1%  turbostat.CPU%c6
>        223 Â  1%     -41.1%        131 Â  1%  proc-vmstat.nr_shmem
>    4518820 Â  3%     -30.8%    3128304 Â 16%  numa-vmstat.node1.numa_local
>    3363496 Â  3%     -27.4%    2441619 Â 20%  numa-vmstat.node1.nr_dirtied
>    3345346 Â  3%     -27.4%    2428396 Â 20%  numa-vmstat.node1.nr_written
>       0.18 Â 18%    +105.6%       0.37 Â 36%  turbostat.Pkg%pc3
>    3427913 Â  3%     -27.3%    2492563 Â 20%  numa-vmstat.node1.nr_inactive_file
>   13712431 Â  3%     -27.3%    9971152 Â 20%  numa-meminfo.node1.Inactive
>   13711768 Â  3%     -27.3%    9970866 Â 20%  numa-meminfo.node1.Inactive(file)
>    3444598 Â  3%     -27.2%    2508920 Â 20%  numa-vmstat.node1.nr_file_pages
>   13778510 Â  3%     -27.2%   10036287 Â 20%  numa-meminfo.node1.FilePages
>    8819175 Â  1%     -28.3%    6320188 Â 19%  numa-numastat.node1.numa_hit
>    8819051 Â  1%     -28.3%    6320152 Â 19%  numa-numastat.node1.local_node
>   14350918 Â  3%     -26.8%   10504070 Â 19%  numa-meminfo.node1.MemUsed
>     100892 Â  3%     -26.0%      74623 Â 19%  numa-vmstat.node1.nr_slab_reclaimable
>     403571 Â  3%     -26.0%     298513 Â 19%  numa-meminfo.node1.SReclaimable
>       3525 Â 13%     +36.6%       4817 Â 14%  slabinfo.blkdev_requests.active_objs
>       3552 Â 13%     +36.3%       4841 Â 14%  slabinfo.blkdev_requests.num_objs
>      30779 Â  4%     -34.7%      20084 Â 12%  proc-vmstat.pgmigrate_success
>      30779 Â  4%     -34.7%      20084 Â 12%  proc-vmstat.numa_pages_migrated
>     447400 Â  2%     -23.2%     343701 Â 16%  numa-meminfo.node1.Slab
>  2.532e+10 Â  0%     -33.1%  1.694e+10 Â  1%  cpuidle.C6-IVT.time
>       3081 Â  9%     +28.0%       3945 Â 12%  slabinfo.mnt_cache.num_objs
>       3026 Â  9%     +28.8%       3898 Â 12%  slabinfo.mnt_cache.active_objs
>       5822 Â  4%     +77.8%      10350 Â 25%  numa-meminfo.node1.Writeback
>       1454 Â  4%     +77.3%       2579 Â 25%  numa-vmstat.node1.nr_writeback
>     424984 Â  1%     -26.5%     312255 Â  3%  proc-vmstat.numa_pte_updates
>     368001 Â  1%     -26.8%     269440 Â  3%  proc-vmstat.numa_hint_faults
>     456465 Â  1%     -26.7%     334594 Â  4%  time.minor_page_faults
>       3.86 Â  3%     -24.4%       2.92 Â  2%  turbostat.CPU%c3
>    4661151 Â  2%     +20.6%    5622999 Â  9%  numa-vmstat.node1.nr_free_pages
>   18644452 Â  2%     +20.6%   22491300 Â  9%  numa-meminfo.node1.MemFree
>        876 Â  2%     +28.2%       1124 Â  5%  slabinfo.kmalloc-4096.num_objs
>        858 Â  3%     +24.0%       1064 Â  5%  slabinfo.kmalloc-4096.active_objs
>   17767832 Â  8%     -25.4%   13249545 Â 17%  cpuidle.POLL.time
>     285093 Â  1%     -23.1%     219372 Â  5%  proc-vmstat.numa_hint_faults_local
>     105423 Â  2%     -16.1%      88498 Â  0%  meminfo.Dirty
>      26365 Â  1%     -16.0%      22152 Â  1%  proc-vmstat.nr_dirty
>      41.04 Â  1%     -14.1%      35.26 Â  1%  turbostat.CPU%c1
>       9385 Â  4%     -14.3%       8043 Â  6%  slabinfo.kmalloc-192.active_objs
>       9574 Â  3%     -13.9%       8241 Â  6%  slabinfo.kmalloc-192.num_objs
>       2411 Â  3%     +17.0%       2820 Â  4%  slabinfo.kmalloc-2048.active_objs
>   12595574 Â  0%     -10.0%   11338368 Â  1%  proc-vmstat.pgalloc_normal
>       5262 Â  1%     +13.3%       5962 Â  1%  slabinfo.kmalloc-1024.num_objs
>       5262 Â  1%     +12.7%       5932 Â  1%  slabinfo.kmalloc-1024.active_objs
>       2538 Â  3%     +13.7%       2885 Â  4%  slabinfo.kmalloc-2048.num_objs
>    5299546 Â  0%      -9.9%    4776351 Â  0%  slabinfo.buffer_head.active_objs
>    5299546 Â  0%      -9.9%    4776351 Â  0%  slabinfo.buffer_head.num_objs
>     135885 Â  0%      -9.9%     122470 Â  0%  slabinfo.buffer_head.num_slabs
>     135885 Â  0%      -9.9%     122470 Â  0%  slabinfo.buffer_head.active_slabs
>      28.04 Â  2%    +715.6%     228.69 Â  3%  iostat.sdb.avgrq-sz
>      28.05 Â  2%    +708.1%     226.72 Â  2%  iostat.sdc.avgrq-sz
>       2245 Â  3%     -81.6%        413 Â  1%  iostat.sda.w/s
>       5.33 Â  1%   +1008.2%      59.07 Â  1%  iostat.sda.w_await
>       5.85 Â  1%   +1126.4%      71.69 Â  4%  iostat.sda.r_await
>       5.36 Â  1%    +978.6%      57.79 Â  3%  iostat.sdc.w_await
>       1263 Â  4%     -85.8%        179 Â  6%  iostat.sdc.r/s
>       2257 Â  3%     -81.6%        414 Â  2%  iostat.sdb.w/s
>       1264 Â  4%     -85.8%        179 Â  6%  iostat.sdb.r/s
>       5.55 Â  0%   +1024.2%      62.37 Â  4%  iostat.sdb.await
>       5.89 Â  1%   +1125.9%      72.16 Â  6%  iostat.sdb.r_await
>       5.36 Â  0%   +1014.3%      59.75 Â  3%  iostat.sdb.w_await
>       5.57 Â  1%    +987.9%      60.55 Â  3%  iostat.sdc.await
>       5.51 Â  0%   +1017.3%      61.58 Â  1%  iostat.sda.await
>       1264 Â  4%     -85.8%        179 Â  6%  iostat.sda.r/s
>      28.09 Â  2%    +714.2%     228.73 Â  2%  iostat.sda.avgrq-sz
>       5.95 Â  2%   +1091.0%      70.82 Â  6%  iostat.sdc.r_await
>       2252 Â  3%     -81.5%        417 Â  2%  iostat.sdc.w/s
>       4032 Â  2%    +151.6%      10143 Â  1%  iostat.sdb.wrqm/s
>       4043 Â  2%    +151.0%      10150 Â  1%  iostat.sda.wrqm/s
>       4035 Â  2%    +151.2%      10138 Â  1%  iostat.sdc.wrqm/s
>      26252 Â  1%     -54.0%      12077 Â  4%  vmstat.system.in
>      37813 Â  0%    +101.0%      75998 Â  1%  vmstat.io.bo
>      37789 Â  0%    +101.0%      75945 Â  1%  iostat.md0.wkB/s
>        205 Â  0%     +96.1%        402 Â  1%  iostat.md0.w/s
>     164286 Â  1%     -46.2%      88345 Â  2%  vmstat.system.cs
>      27.07 Â  2%     -46.7%      14.42 Â  3%  turbostat.%Busy
>        810 Â  2%     -46.7%        431 Â  3%  turbostat.Avg_MHz
>      15.56 Â  2%     +71.7%      26.71 Â  1%  iostat.sda.avgqu-sz
>      15.65 Â  2%     +69.1%      26.46 Â  2%  iostat.sdc.avgqu-sz
>      15.67 Â  2%     +72.7%      27.06 Â  2%  iostat.sdb.avgqu-sz
>      25151 Â  0%     +68.3%      42328 Â  1%  iostat.sda.wkB/s
>      25153 Â  0%     +68.2%      42305 Â  1%  iostat.sdb.wkB/s
>      25149 Â  0%     +68.2%      42292 Â  1%  iostat.sdc.wkB/s
>      97.45 Â  0%     -21.1%      76.90 Â  0%  turbostat.CorWatt
>      12517 Â  0%     -20.2%       9994 Â  1%  iostat.sdc.rkB/s
>      12517 Â  0%     -20.0%      10007 Â  1%  iostat.sda.rkB/s
>      12512 Â  0%     -19.9%      10018 Â  1%  iostat.sdb.rkB/s
>       1863 Â  3%     +24.7%       2325 Â  1%  iostat.sdb.rrqm/s
>       1865 Â  3%     +24.3%       2319 Â  1%  iostat.sdc.rrqm/s
>       1864 Â  3%     +24.6%       2322 Â  1%  iostat.sda.rrqm/s
>        128 Â  0%     -16.4%        107 Â  0%  turbostat.PkgWatt
>     150569 Â  0%      -8.7%     137525 Â  0%  iostat.md0.avgqu-sz
>       4.29 Â  0%      -5.1%       4.07 Â  0%  turbostat.RAMWatt
> 
> 
> more detailed changes about ivb44/fsmark/1x-64t-4BRD_12G-RAID5-btrfs-4M-30G-NoSync
> ---------
> 
> 26089f4902595a2f  4400755e356f9a2b0b7ceaa02f  
> ----------------  --------------------------  
>          %stddev     %change         %stddev
>              \          |                \  
>        273 Â  4%     -18.1%        223 Â  6%  fsmark.files_per_sec
>      29.24 Â  1%     +27.2%      37.20 Â  8%  fsmark.time.elapsed_time.max
>      29.24 Â  1%     +27.2%      37.20 Â  8%  fsmark.time.elapsed_time
>        399 Â  4%     -20.0%        319 Â  3%  fsmark.time.percent_of_cpu_this_job_got
>     129891 Â 20%     -28.9%      92334 Â 15%  fsmark.time.voluntary_context_switches
>        266 Â  0%    +413.4%       1365 Â  5%  slabinfo.raid5-md0.num_objs
>        266 Â  0%    +413.4%       1365 Â  5%  slabinfo.raid5-md0.active_objs
>       0.23 Â 27%     +98.6%       0.46 Â 35%  turbostat.CPU%c3
>   56612063 Â  9%     +36.7%   77369763 Â 20%  cpuidle.C1-IVT.time
>    5579498 Â 14%     -36.0%    3571516 Â  6%  cpuidle.C1E-IVT.time
>       4668 Â 38%     +64.7%       7690 Â 19%  numa-vmstat.node0.nr_unevictable
>      18674 Â 38%     +64.7%      30762 Â 19%  numa-meminfo.node0.Unevictable
>       9298 Â 37%     +64.4%      15286 Â 19%  proc-vmstat.nr_unevictable
>       4629 Â 37%     +64.1%       7596 Â 19%  numa-vmstat.node1.nr_unevictable
>      18535 Â 37%     +63.9%      30385 Â 19%  numa-meminfo.node1.Unevictable
>    4270894 Â 19%     +65.6%    7070923 Â 21%  cpuidle.C3-IVT.time
>      38457 Â 37%     +59.0%      61148 Â 19%  meminfo.Unevictable
>    3748226 Â 17%     +26.6%    4743674 Â 16%  numa-vmstat.node0.numa_local
>    4495283 Â 13%     -24.8%    3382315 Â 17%  numa-vmstat.node0.nr_free_pages
>    3818432 Â 16%     +26.5%    4830938 Â 16%  numa-vmstat.node0.numa_hit
>   17966826 Â 13%     -24.7%   13537228 Â 17%  numa-meminfo.node0.MemFree
>   14901309 Â 15%     +29.7%   19330906 Â 12%  numa-meminfo.node0.MemUsed
>         26 Â 21%     -32.9%         17 Â 14%  cpuidle.POLL.usage
>  1.183e+09 Â  1%     +29.6%  1.533e+09 Â  8%  cpuidle.C6-IVT.time
>      29.24 Â  1%     +27.2%      37.20 Â  8%  time.elapsed_time
>      29.24 Â  1%     +27.2%      37.20 Â  8%  time.elapsed_time.max
>        399 Â  4%     -20.0%        319 Â  3%  time.percent_of_cpu_this_job_got
>        850 Â  4%      -8.6%        777 Â  5%  slabinfo.blkdev_requests.num_objs
>        850 Â  4%      -8.6%        777 Â  5%  slabinfo.blkdev_requests.active_objs
>      14986 Â  9%     +17.1%      17548 Â  8%  numa-vmstat.node0.nr_slab_reclaimable
>      11943 Â  5%     -12.6%      10441 Â  2%  slabinfo.kmalloc-192.num_objs
>      59986 Â  9%     +17.0%      70186 Â  8%  numa-meminfo.node0.SReclaimable
>       3703 Â  6%     +10.2%       4082 Â  7%  slabinfo.btrfs_delayed_data_ref.num_objs
>     133551 Â  6%     +16.1%     154995 Â  1%  proc-vmstat.pgfault
>     129891 Â 20%     -28.9%      92334 Â 15%  time.voluntary_context_switches
>      11823 Â  4%     -12.0%      10409 Â  3%  slabinfo.kmalloc-192.active_objs
>       3703 Â  6%      +9.7%       4061 Â  7%  slabinfo.btrfs_delayed_data_ref.active_objs
>      19761 Â  2%     -11.2%      17542 Â  6%  slabinfo.anon_vma.active_objs
>      19761 Â  2%     -11.2%      17544 Â  6%  slabinfo.anon_vma.num_objs
>      13002 Â  3%     +14.9%      14944 Â  5%  slabinfo.kmalloc-256.num_objs
>      12695 Â  3%     +13.8%      14446 Â  7%  slabinfo.kmalloc-256.active_objs
>       1190 Â  1%     -11.8%       1050 Â  3%  slabinfo.mnt_cache.num_objs
>       1190 Â  1%     -11.8%       1050 Â  3%  slabinfo.mnt_cache.active_objs
>     136862 Â  1%     -13.8%     117938 Â  7%  cpuidle.C6-IVT.usage
>    1692630 Â  3%     +12.3%    1900854 Â  0%  numa-vmstat.node0.nr_written
>       1056 Â  2%      +8.8%       1149 Â  3%  slabinfo.mm_struct.active_objs
>       1056 Â  2%      +8.8%       1149 Â  3%  slabinfo.mm_struct.num_objs
>      24029 Â 11%     -30.6%      16673 Â  8%  vmstat.system.cs
>       8859 Â  2%     -15.0%       7530 Â  8%  vmstat.system.in
>     905630 Â  2%     -16.8%     753097 Â  4%  iostat.md0.wkB/s
>     906433 Â  2%     -16.9%     753482 Â  4%  vmstat.io.bo
>       3591 Â  2%     -16.9%       2982 Â  4%  iostat.md0.w/s
>      13.22 Â  5%     -16.3%      11.07 Â  1%  turbostat.%Busy
>        402 Â  4%     -15.9%        338 Â  1%  turbostat.Avg_MHz
>      54236 Â  3%     +10.4%      59889 Â  4%  iostat.md0.avgqu-sz
>       7.67 Â  1%      +4.5%       8.01 Â  1%  turbostat.RAMWatt
> 
> 
> 
> 	--yliu
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@xxxxxxxxxxxxxxx
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Attachment: pgpeLU7LEcZqV.pgp
Description: OpenPGP digital signature