Re: Tricks to speed up kernel builds

[david]

On Tue, 15 Sep 2009, Thomas Fjellstrom wrote:

> On Tue September 15 2009, Ozan ?a?layan wrote:
> > Hi,
> >
> > I'd googled a lot about the $subject but couldn't find any detailed
> > guide about it. So, the followings are what I understand from what I've
> > found:
> >
> > 1. Build inside a tmpfs to avoid I/O bottlenecks,

if you have the ram the system will use it to cache the data, avoiding the 
I/O bottlenecks.

the first time you compile you have to wait for the source to be read into 
ram, but you would have to copy the source to the tmpfs anyway

> > 2. Use ccache,
> > 3. Switch governor to performance if supported,
> > 4. Pass -j to gnu make to parallelize the build process,
> > 5. Don't build debug symbols a.k.a set CONFIG_DEBUG_INFO=n
> > 6. There are a bunch of sched patches in tip which apparently improves
> > things during kbuild
> > 7. Distribute the work into the local network using icecream, distcc.
> >
> > For the -j part, how many jobs should we pass to it? I've seen people
> > passing
> >
> > - NR_of_cores+1
> > - NR_of_cores x 2 (I think that this is useful when there's 2 threads of
> > execution per core like HyperThreading?)

this is not due to hyperthreading.

if you do not have the data cached in ram from a prior build, individual 
compile threads will stall waiting for the source to be read from disk. by 
having more threads than you have cores you can have some stalled waiting 
for data while others run and use the cpu cycles.

exactly how much benfit you get, and where things taper off is very 
dependant on your system (how fast is your I/O, how effectivly does it 
handle parallel requests)

it's probably higher than NR_of_cores +1, but if you have a 8-core or 16 
core box it's probably less than NR_of_cores x 2 (unless you have a 
storage subsystem that works well in parallel, say something like a 8 or 
16 disk raid array)

> > - 64, 128, etc. regardless of the number of cores (Is it really useful
> > regardless of the CPU)

this will overload your system with lots of processes competing for the 
CPU. it's a good stress test of the system, but inefficiant in processor 
utilization due to the fact that each time a new thread takes the CPU the 
cache has been wiped out by other threads.

> > The curious question is *what build times can you get* with your typical
> > systems using your speedup tricks (quad core, dual core, etc.) ? What
> > are other tricks that I can use to even more speed up the build?
> >
> > Do you really use icecream/distcc on your daily test builds?

most people don't, some people do. some people have systems that can build 
the kernel in a handful of seconds (I think the record was 7.5 seconds 
several years ago) if it is in the memory disk cache and don't need to. 
others build on architecures that don't have systems that fast and use 
distcc all the time to get acceptable speed

since many virtualization options will not let a guest OS use more than 
one real CPU at a time, people who build in virtual systems commonly use 
this sort of thing to use all the real cores on their system (same as the 
-j option above)

Linus has been heard to complain that it takes longer to boot a system 
with a new kernel than to compile it (this was on an enterprise-class 
system that took a long time for the bios/raid/etc initialization)


> > How about nicing/ionicing make on an idle system? Will it make any
> > difference?

if there is nothing to compete with the kernel build, these don't help.

David Lang

> I've only done a few compiles recently, and I only really use icecream and
> make -jX to speed up compile times.
>
> I have a 4 node icecc cluster, three quad cores (one limited to 2 cores), and
> one dual core (limited to a single core), which gives 11 total cores I can
> distribute to, and I usually pass: make -j12
>
> Any higher and I don't really see any speedups, any lower, same difference.
>
> My last kernel compile (2.6.31-git4) took about 9 minutes.
>
> > Thanks a lot
> > Ozan Caglayan
> >
> >
> >
> >
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> > the body of a message to majordomo@xxxxxxxxxxxxxxx
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > Please read the FAQ at  http://www.tux.org/lkml/
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/