Re: [Bug #11308] tbench regression on each kernel release from 2.6.22-> 2.6.28

From: Eric Dumazet
Date: Wed Sep 17 2008 - 10:47:42 EST


Mike Galbraith a écrit :
On Wed, 2008-09-17 at 14:49 +0200, Ingo Molnar wrote:
* Mike Galbraith <efault@xxxxxx> wrote:

It looks like a potentially bogus bisection result, but _maybe_ it has relevance: changes the size of "struct security_operations", which could have alignment and layout effects on all sorts of kernel variables, kmalloc sizes, etc.
This may well be a mythical creature infestation for all I know ;-), but it's address is somewhere in the 2069f45..847106f block, 316 commits, none of which look like they should be the least bit interesting to netperf. I reverted this particular commit in 27.git, got the expected result. Looks like I'll keep poking at it, can't seem to resist. Grr.
are you sure it's 2069f45..847106f? Filtering out the likely-uninteresting commits:

Yeah, as sure as I can be. I've built both (et al) kernels several
times, and it has repeated every time. Would be nice if someone would
try to confirm/deny though. For my little quad, I do..

#!/bin/sh

echo 0 > /proc/sys/kernel/sched_wakeup_granularity_ns

netserver -p 12865
netserver -p 12866
netserver -p 12867
netserver -p 12868

netperf -p 12865 -t TCP_RR -l 60 -H 127.0.0.1 -T 0,0 -- -r 1,1&
netperf -p 12866 -t TCP_RR -l 60 -H 127.0.0.1 -T 1,1 -- -r 1,1&
netperf -p 12867 -t TCP_RR -l 60 -H 127.0.0.1 -T 2,2 -- -r 1,1&
netperf -p 12868 -t TCP_RR -l 60 -H 127.0.0.1 -T 3,3 -- -r 1,1&

wait
killall netserver


git log --pretty=format:"%h: %s" 2069f45..847106f | grep -viE \
'block|alsa|pcmcia|sound|Merge|iosched|blk|DAC960|scsi|s390|paride|pktcdvd|filter|cdrom|drm'

gives us:

7daf705: Start using the new '%pS' infrastructure to print symbols
6f0f0fd: security: remove register_security hook
93cbace: security: remove dummy module fix
5915eb5: security: remove dummy module
b478a9f: security: remove unused sb_get_mnt_opts hook
32502b8: splice: fix generic_file_splice_read() race with page invalidation
8b3d356: ramfs: enable splice write
a144ff0: xen: Avoid allocations causing swap activity on the resume path

which really only leaves that security commit your bisection fingered. Which _slightly_ raises its likelyhood of being implicated. Structure size changes can move two formerly far-apart netperf-relevant symbols on the same cacheline, which can start cache ping-pong-ing badly.

I sure hope it's something like ping-pong, it's driving me NUTS.

Could you please try following patch ?

[PATCH] security_ops moved to read_mostly section

"struct security_operations *security_ops" should be moved to read_mostly section in order to NOT let it share a cache line with higly modified variables.

Signed-off-by: Eric Dumazet <dada1@xxxxxxxxxxxxx>

diff --git a/security/security.c b/security/security.c
index 3a4b4f5..0b13d65 100644
--- a/security/security.c
+++ b/security/security.c
@@ -24,7 +24,7 @@ static __initdata char chosen_lsm[SECURITY_NAME_MAX + 1];
extern struct security_operations default_security_ops;
extern void security_fixup_ops(struct security_operations *ops);

-struct security_operations *security_ops; /* Initialized to NULL */
+struct security_operations *security_ops __read_mostly;e

/* amount of vm to protect from userspace access */
unsigned long mmap_min_addr = CONFIG_SECURITY_DEFAULT_MMAP_MIN_ADDR;



--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/