Re: [performance bug] kernel building regression on 64 LCPUs machine

[Corrado Zoccolo]

On Wed, Jan 19, 2011 at 2:55 AM, Alex,Shi <alex.shi@xxxxxxxxx> wrote:
> Shaohua and I tested kernel building performance on latest kernel. and
> found it is drop about 15% on our 64 LCPUs NHM-EX machine on ext4 file
> system. We find this performance dropping is due to commit
> 749ef9f8423054e326f. If we revert this patch or just change the
> WRITE_SYNC back to WRITE in jbd2/commit.c file. the performance can be
> recovered.
>
> iostat report show with the commit, read request merge number increased
> and write request merge dropped. The total request size increased and
> queue length dropped. So we tested another patch: only change WRITE_SYNC
> to WRITE_SYNC_PLUG in jbd2/commit.c, but nothing effected.
>
> we didn't test deadline IO mode, just test cfq. seems insert write
> request into sync queue effect much read performance, but we don't know
> details. What's your comments of this?
>
> iostat of .37 kernel:
> rrqm/s  wrqm/s r/s   w/s   rMB/s ÂwMB/s avgrq-sz avgqu-sz  await Âsvctm Â%util
> 22.5772 96.46 92.3742 14.747 Â1.0048 Â0.439474 34.8557 0.18078 3.8076 0.30447 2.94302
> iostat of commit reverted .37:
> 26.6223 80.21 107.875 6.03538 1.51415 0 41.3275 0.153385 3.80569 0.377231 3.22323

>From these numbers, it seems to me that with the patch reverted, the
write bandwidth is really low, and probably you are keeping most
written files in the buffer cache during the whole compile, while the
non-reverted kernel is making progress in writing out the files. So
the 'improved' read bandwidth is due to unfairness w.r.t. writes.
Does the result change if you add a final sync and time that as well,
in order to see the full time to make it on disk?

I think that in a more extreme test where you end up filling all the
buffer cache with written data, you will see much longer stalls with
the revert than without.

Thanks,
Corrado

>
> vmstat report show, read bandwidth dropping:
> vmstat of .37:
> Âr Âb  swpd  free  buff Âcache  si  so  Âbi  Âbo  in  cs us sy id wa st
> 3.4 52.6 0.0 64303937.0 16466.7 121544.5 0.0 0.0 2102.7 1914.6 7414.1 3185.7 2.0 1.0 80.3 16.7 0.0
> vmstat of revert all from .37
> 2.2 35.8 0.0 64306767.4 17265.6 126101.2 0.0 0.0 2415.8 1619.1 8532.2 3556.2 2.5 1.1 83.0 13.3 0.0
>
> Regards
> Alex
>
> ===
> diff --git a/fs/jbd/commit.c b/fs/jbd/commit.c
> index 34a4861..27ac2f3 100644
> --- a/fs/jbd/commit.c
> +++ b/fs/jbd/commit.c
> @@ -294,7 +294,7 @@ void journal_commit_transaction(journal_t *journal)
> Â Â Â Âint first_tag = 0;
> Â Â Â Âint tag_flag;
> Â Â Â Âint i;
> - Â Â Â int write_op = WRITE_SYNC;
> + Â Â Â int write_op = WRITE;
>
> Â Â Â Â/*
> Â Â Â Â * First job: lock down the current transaction and wait for
> diff --git a/fs/jbd2/commit.c b/fs/jbd2/commit.c
> index f3ad159..69ff08e 100644
> --- a/fs/jbd2/commit.c
> +++ b/fs/jbd2/commit.c
> @@ -329,7 +329,7 @@ void jbd2_journal_commit_transaction(journal_t *journal)
> Â Â Â Âint tag_bytes = journal_tag_bytes(journal);
> Â Â Â Âstruct buffer_head *cbh = NULL; /* For transactional checksums */
> Â Â Â Â__u32 crc32_sum = ~0;
> - Â Â Â int write_op = WRITE_SYNC;
> + Â Â Â int write_op = WRITE;
>
> Â Â Â Â/*
> Â Â Â Â * First job: lock down the current transaction and wait for
>
>
>



-- 
__________________________________________________________________________

dott. Corrado Zoccolo             mailto:czoccolo@xxxxxxxxx
PhD - Department of Computer Science - University of Pisa, Italy
--------------------------------------------------------------------------
The self-confidence of a warrior is not the self-confidence of the average
man. The average man seeks certainty in the eyes of the onlooker and calls
that self-confidence. The warrior seeks impeccability in his own eyes and
calls that humbleness.
              Â Tales of Power - C. Castaneda
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/