In the FWIW department, you can get horribly misleading profile data
when using "slow" sampling rates (ie 100Hz) that are triggered by the
timer interrupt. No surprise when you think about it, as lots of things
are somehow implicitly related to the timer interrupt, and so you may
end up always missing the IP of some routine that runs shortly after
every n'th timer interrupt (where n is an integer).
There are two ways around this sort of problem:
(1) Use a high(er) sampling rate as per David above, so that the
period between samples is much less than the typical execution
time of the functions of interest. This stops functions that
appear as "harmonics" of the timer interrupt from escaping being
sampled.
(2) Use a sampling trigger that is comparable to 100Hz, but much
more independent of general kernel activity than the timer
interrupt, so that a functions execution time in relation
to the timer interrupt will not effect the profiling data.
In addition, you don't incurr the higher overhead of having
an excessive timer interrupt rate as in (1).
I spent some time implementing and testing a variation of (2) a few
months ago and and intended to write it up a bit (LJ article?) but
haven't had the free time as of yet. In a nutshell, I grabbed the
profiling data from the RTC interrupt handler instead of the timer
interrupt. This has the advantage of letting you change the sampling
rate on the fly without having to recompile everything that has HZ
stuck in the source.
I found that the data collected at low rates (e.g. RTC at 128Hz) was
in agreement with that collected with the RTC at higher rates
(e.g 2048Hz and above) and hence there was no need for a high rate
so long as the sampling trigger was (reasonably) independent of
general kernel activity. Furthermore, the data appeared reasonable,
whereas the data obtained from using the timer IRQ @ 100Hz was
clearly in error, and also inconsistent from one run to the next.
(FWIW, I was profiling ethercard drivers at the time.)
Paul.