Re: [Bugme-new] [Bug 12564] New: poor performance whilepreprocessing source code

[Andrew Morton]

On Wed, 28 Jan 2009 14:43:37 -0800
Steven Patrick <steven@xxxxxxxx> wrote:

> > > Problem Description:
> > >   Here's my situation.  I've recently upgraded the kernels
> > > on ~30 computers at work (from 2.6.21 to 2.6.27).  These 
> > > computers are used to build and test software we develop.  
> > > We speed up the building process using distcc.  However,
> > > after the kernel upgrade, the builds are much much slower.
> > > The preprocessing stage seems to be at least 10 times
> > > slower.
> > >   As evidence of this slowdown I am attaching two images created
> > > using distccmon-gnome.  Both snapshots were taken shortly 
> > > after starting builds in a clean sandbox.  The only difference
> > > is the kernel.  "fast.png" was generated while running
> > > kernel 2.6.25.20.  "slow.png" was generated with 2.6.26.
> > > The light purple sections indicate the preprocessing times
> > > for each file.  
> > >   This slowdown is observed on both 32 and 64 bit computers
> > > and using either gcc or the intel compiler. (The intel compiler
> > > builds do not use distcc, but that are also slower.)  Strangely 
> > > enough, it's still faster to use an NFS mounted sandbox on a
> > > machine with an older kernel than the same sandbox on the local
> > > machine with a newer kernel.  (This suggests to me that it
> > > is neither a disk or network IO problem.)
> > >   I've used git bisect to narrow it down to a single commit:
> > > # bad: [b0b539739fe9b7d75002412a787cfdf4efddbc33] NFS: Ensure that 'noac'
> > > and/or 'actimeo=0' turn off attribute caching
> > 
> > And thanks for bisecting it.
> 
>   I think it might be possible to bisect further, but I
> don't know how get git to do it.  

OK..

> > > This is the first commit after v2.6.26-rc3.  I'm not an
> > > experienced git user, so I don't know how to narrow it down
> > > further.  Distccmon-gnome snapshots from the bisection
> > > process are similar to the ones noted above.
> > >   Naturally, I would like the newer kernels to have similar
> > > performance to the older kernels.
> > >   I will be attaching various items.  Let me know what other 
> > > information might be helpful.
> > > 
> > > Steps to reproduce:
> > > distccmon-gnome &
> > > # using a makefile setup to use distcc:
> > > make -j 5 all
> > > # note preprocessing times in distccmon
> > > 
> > 
> > Something I don't understand from this:
> > 
> > If your normal setup is using distcc then what role does NFS have to
> > play?  I mean, distcc kind-of replaces NFS in this workload.
> 
>   I don't believe that this actually has anything to do
> with NFS.  The only way I could see NFS coming into this is the
> fact that I use amd to automount home directories.
>   The commit mentioned above modifies just over 100 files, very
> few of which appear related to NFS.  I've attached the diifs.
> It appears to be a number of fixes rolled up into one patch.
> This is why I believe it might be possible to continue bisecting
> with git.  But, to do that I suspect I would need to use a
> different repository and/or tag or version names.
>   BTW, the diffs were generated with "git diff v2.6.26-rc3"
> after having done the bisection.  This is why it looks to me
> like this one commit includes multiple updates.  Since all
> bisection iterations were bad, the diffs against the good
> version must be the diffs of the one commit.  Of course, I 
> reserve the right to have made a mistake or not understand 
> things completely.
>   There was one other detail that seemed strange.  In the
> last four bisection iterations, the kernel identified
> itself as -rc2 instead of -rc3.  I assumed an update
> accidentally changed it back.

Heaven knows.

OK, "This is the first commit after v2.6.26-rc3", so NFS is innocent. 
Presumably your git bisect landed on a big merge commit.

I'm afaid I'm not the person who can tell you how to fix this - it
hasn't happened to me in a while, but I don't know what I did to
prevent it happening :(

Here's the diffstat from your diffs.txt.gz:

 arch/m68k/configs/multi_defconfig              | 1269 -------------------------
 b/MAINTAINERS                                  |    4 
 b/Makefile                                     |    2 
 b/arch/arm/common/locomo.c                     |   66 -
 b/arch/arm/kernel/armksyms.c                   |    2 
 b/arch/arm/kernel/arthur.c                     |    2 
 b/arch/arm/mach-omap1/board-palmte.c           |    2 
 b/arch/arm/mach-omap1/board-palmz71.c          |    2 
 b/arch/arm/mach-omap2/board-2430sdp.c          |    1 
 b/arch/arm/mach-omap2/board-apollon.c          |    1 
 b/arch/arm/mach-omap2/board-generic.c          |    1 
 b/arch/arm/mach-omap2/board-h4.c               |    1 
 b/arch/arm/mach-omap2/clock.c                  |    4 
 b/arch/arm/mach-omap2/clock34xx.h              |   21 
 b/arch/arm/mach-omap2/cm-regbits-34xx.h        |    1 
 b/arch/arm/mach-omap2/mailbox.c                |   25 
 b/arch/arm/mach-omap2/prm.h                    |    2 
 b/arch/arm/mach-orion5x/dns323-setup.c         |    2 
 b/arch/arm/mach-orion5x/kurobox_pro-setup.c    |    2 
 b/arch/arm/mach-pxa/colibri.c                  |    3 
 b/arch/arm/mach-pxa/spitz.c                    |    1 
 b/arch/arm/mm/proc-arm925.S                    |    2 
 b/arch/arm/mm/proc-arm926.S                    |    2 
 b/arch/arm/mm/proc-arm940.S                    |    2 
 b/arch/arm/mm/proc-arm946.S                    |    2 
 b/arch/arm/plat-omap/clock.c                   |   10 
 b/arch/arm/plat-omap/dma.c                     |    2 
 b/arch/arm/plat-omap/mailbox.c                 |    1 
 b/arch/blackfin/mach-bf527/boards/ezkit.c      |    2 
 b/arch/m68k/Kconfig                            |    9 
 b/arch/m68k/configs/amiga_defconfig            |  159 +--
 b/arch/m68k/configs/apollo_defconfig           |  140 --
 b/arch/m68k/configs/atari_defconfig            |  143 +-
 b/arch/m68k/configs/bvme6000_defconfig         |  138 --
 b/arch/m68k/configs/hp300_defconfig            |  142 +-
 b/arch/m68k/configs/mac_defconfig              |  144 +-
 b/arch/m68k/configs/mvme147_defconfig          |  138 --
 b/arch/m68k/configs/mvme16x_defconfig          |  138 --
 b/arch/m68k/configs/q40_defconfig              |  159 +--
 b/arch/m68k/configs/sun3_defconfig             |  140 --
 b/arch/m68k/configs/sun3x_defconfig            |  140 --
 b/arch/m68k/kernel/head.S                      |    2 
 b/arch/m68k/kernel/setup.c                     |   15 
 b/arch/powerpc/platforms/pasemi/misc.c         |    7 
 b/arch/sparc64/defconfig                       |   40 
 b/arch/sparc64/mm/init.c                       |    2 
 b/arch/x86/kernel/process.c                    |   36 
 b/arch/x86/mm/pat.c                            |    2 
 b/drivers/block/amiflop.c                      |    6 
 b/drivers/block/z2ram.c                        |    2 
 b/drivers/char/snsc_event.c                    |    2 
 b/drivers/char/vme_scc.c                       |    4 
 b/drivers/i2c/busses/i2c-amd756.c              |    2 
 b/drivers/i2c/busses/i2c-nforce2.c             |   28 
 b/drivers/i2c/chips/max6875.c                  |    3 
 b/drivers/i2c/i2c-core.c                       |   22 
 b/drivers/ide/legacy/macide.c                  |    3 
 b/drivers/input/keyboard/hilkbd.c              |    4 
 b/drivers/input/misc/hp_sdc_rtc.c              |    5 
 b/drivers/input/serio/hp_sdc_mlc.c             |    5 
 b/drivers/input/serio/q40kbd.c                 |    2 
 b/drivers/media/video/cs5345.c                 |    7 
 b/drivers/media/video/cs53l32a.c               |   10 
 b/drivers/media/video/cx18/cx18-i2c.c          |    9 
 b/drivers/media/video/cx25840/cx25840-core.c   |    7 
 b/drivers/media/video/et61x251/et61x251_core.c |    2 
 b/drivers/media/video/ivtv/ivtv-i2c.c          |   13 
 b/drivers/media/video/m52790.c                 |    9 
 b/drivers/media/video/msp3400-driver.c         |   17 
 b/drivers/media/video/saa7115.c                |   40 
 b/drivers/media/video/saa7127.c                |    9 
 b/drivers/media/video/saa717x.c                |    9 
 b/drivers/media/video/sn9c102/sn9c102_core.c   |    2 
 b/drivers/media/video/tuner-core.c             |   17 
 b/drivers/media/video/upd64031a.c              |    6 
 b/drivers/media/video/upd64083.c               |    6 
 b/drivers/media/video/vp27smpx.c               |    9 
 b/drivers/media/video/wm8739.c                 |    7 
 b/drivers/media/video/wm8775.c                 |    7 
 b/drivers/media/video/zc0301/zc0301_core.c     |    2 
 b/drivers/media/video/zoran_device.c           |    2 
 b/drivers/media/video/zoran_driver.c           |    2 
 b/drivers/net/82596.c                          |    7 
 b/drivers/net/apne.c                           |    3 
 b/drivers/net/mac89x0.c                        |    3 
 b/drivers/net/macmace.c                        |    3 
 b/drivers/net/sun3lance.c                      |    3 
 b/drivers/net/wireless/atmel.c                 |    2 
 b/drivers/video/Kconfig                        |    4 
 b/drivers/video/amifb.c                        |    4 
 b/drivers/video/dnfb.c                         |    3 
 b/drivers/video/hpfb.c                         |    2 
 b/drivers/video/pxafb.c                        |    5 
 b/fs/befs/endian.h                             |    2 
 b/fs/nfs/inode.c                               |    7 
 b/include/asm-arm/arch-omap/common.h           |    4 
 b/include/asm-arm/arch-omap/control.h          |    2 
 b/include/asm-arm/arch-omap/mmc.h              |   24 
 b/include/asm-arm/arch-sa1100/irqs.h           |    2 
 b/include/asm-arm/hardware/locomo.h            |   19 
 b/include/asm-m68k/bug.h                       |    4 
 b/include/asm-m68k/io.h                        |   44 
 b/include/asm-m68k/setup.h                     |    2 
 b/include/asm-m68k/uaccess.h                   |    6 
 b/include/linux/i2c.h                          |    7 
 b/include/linux/i2c/pcf857x.h                  |    3 
 106 files changed, 833 insertions(+), 2743 deletions(-)

Now, distcc has a long track record of tripping up bugs and regressions
in the networking code - it's very clever that way.  But it doesn't
look like there were any relevant networking changes in that window.

So I think the next step is to ask you to continue with (or redo) that
bisection search.

Is there someone who is more git-competent than I who could help with
this please?


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/