Re: Bug: Apparent memory exhaustion WTF?

[Phil Turmel]

On 04/01/2011 04:17 PM, haael wrote:
> 
> Hello, guys. My server keeps hanging up with the error output as seen on the bottom of this message. I run non-patched vanilla kernel. Full system upgrade didn't help. Kernel upgrade from 2.6.28.7 to 2.6.35.10 didn't help. I tried also turning swap on/off, upgrading memory and adding new CPU. After the CPU upgrade things actually went worse, now my system blows up every few hours.
> 
> Here you see the squid process running out of memory, but nothing changed when I killed squid; some other process would always cause this error. Just before a hangup, all processes are being killed, as if OOM killer went wild. But the OOM killer shouldn't disable the network stack, right?
> 
> After a hangup, the system becomes completely unresponsive, doesn't answer on ping and even on ARP requests. The only thing that still works is the serial console from which I get the following error. This message is being printed continuously, one per second, for infinity. The only thing I can do with my server is to turn off the power.
> 
> Tell me, guys, what to do? The server is 2x Intel dual core, 16GB RAM, HP Proliant. It has RAID-1 disk and a Broadcom network adapter, which is the most suspicious for me. Attached: lspci, /proc/meminfo, /proc/cpuinfo, kernel config and the actual error message from the serial console.

I think this is part of your problem:

> # CONFIG_64BIT is not set
> CONFIG_X86_32=y
> # CONFIG_X86_64 is not set
> CONFIG_X86=y

Everything I've seen on these lists says 32bit kernels and large amounts of RAM are a bad combination.  As I understand it, the information the kernel needs to track all of the memory above 4G must fit in kernel memory below 4G, which is usually just the 1G under the kernel/user split.

> [83155.718040] Free swap  = 0kB
> [83155.718040] Total swap = 0kB

And no swap to fall back on.  Which might be OK on a 64bit kernel.

Just my $0.02.

Phil
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/