Re: swapper: page allocation failure. order:3, mode:0x20

[Anton Blanchard]

Hi,

> After running 2.6.0 on a server for a few days, I met an interesting and
> annoying problem. I was playing with NFS over gigabit ethernet (e1000) and
> it was a bit slow. I tried to find out why by running top and I saw syslog-ng
> eating almost 10% of the 1,3GHz Duron. Looked at the log and there was a huge
> flood of these messages:
> 
> swapper: page allocation failure. order:3, mode:0x20
> irssi: page allocation failure. order:3, mode:0x20
> swapper: page allocation failure. order:3, mode:0x20
> vim: page allocation failure. order:3, mode:0x20
> swapper: page allocation failure. order:3, mode:0x20

Its sounds like you are using either a large MTU (9k?) or TSO. TSO
causes the networking stack to think it has a massive MTU and the e1000
card busts it up into proper MTU sized packets. The problem is that
it places much more stress on the allocator by asking for these large
chunks of memory in interrupt context.

Now e1000 uses TSO (and can regularly ask for 32kB+ kmallocs in
interrupt context) perhaps we should look moving the rx buffer refill code
into a context that can sleep. Then again its not like we can tolerate
much latency in this code path, your rx ring will run out quite quickly :)

BTW We have found increasing /proc/sys/vm/min_free_kbytes can help the
situation a bit. Bumping the slab limits for the larger kmallocs (via
echo X Y Z > /proc/slab) might be useful too.

We should probably rate limit that printk. Andrew: I was thinking of
stealing net_ratelimit and calling it core_ratelimit or whatever. Then
wrap these non critical things with it. Overkill?

Anton
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/