Re: Limit hash table size
From: Steve Lord
Date: Thu Feb 05 2004 - 23:08:57 EST
Andrew Morton wrote:
Steve Lord <lord@xxxxxxx> wrote:
I have seen some dire cases with the dcache, SGI had some boxes with
millions of files out there, and every night a cron job would come
along and suck them all into memory. Resources got tight at some point,
and as more inodes and dentries were being read in, the try to free
pages path was continually getting called. There was always something
in filesystem cache which could get freed, and the inodes and dentries
kept getting more and more of the memory.
There are a number of variables here. Certainly, the old
inodes-pinned-by-highmem pagecache will cause this to happen - badly. 2.6
is pretty aggressive at killing off those inodes.
What kernel was it?
Was it a highmem box? If so, was the filesystem in question placing
directory pagecache in highmem? If so, that was really bad on older 2.4:
the directory pagecache in highmem pins down all directory inodes.
This is where my memory gets a little hazy, its been a few months.
This would have been a 2.4 kernel (probably around 2.4.21)
on an Altix, the filesystem was XFS. So no highmem, but definitely
not your standard kernel.
I never had time to dig into it too much, but I always thought that
on machines with large amounts of memory it was too easy for the
inode and dcache pools to get very large at the expense of the
regular memory zones. While there were pages available, the dcache
and inode zones could just keep on growing. If you run a big
enough find you get lots of memory into the dcache zone and
have a hard time getting it out again.
And it was try_to_free_pages I was referring to.
It does look like 2.6 does better, but I don't have quite the
amount of memory on my laptop....
Steve
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/