Re: Alpha 2.2.9: Too many open files problem

Matti Aarnio (matti.aarnio@sonera.fi)
Wed, 30 Jun 1999 20:54:56 +0300

On Wed, Jun 30, 1999 at 11:16:55AM +0100, John Connett wrote:
> Can anyone suggest how to diagnose a "Too many open files in system"
> problem?
...
> Symptoms are as follows:
>
> Running the same workload repeatedly and monitoring /proc/sys/fs/file-nr
> the file-nr count gradually increases until it reaches (4096 10 4096).
> If file-max is then raised by "echo 5000 > /proc/sys/fs/file-max" and
> lsof(8) is run then ~500 files are reported as open. What can be
> happening with the other ~3500 files that the kernel thinks are open?
>
Look for /proc/nnn/maps listed inodes.
They don't show up in open files, because they aren't "open"
in usual filedescriptor sense... (open file, mmap it, close the
fd --> mmapped region without open file, but the inode (and file)
is in use.)

The symptoms you have are very familiar to me as glibc 2.1.1
contained BSD DB 2.x BTREE seems to leak mmap()ed file regions.
You may have the same, or some other leakage.

Look for things like these in the mmap lists:

0000020000058000-0000020000072000 rw-s 0000000000000000 08:03 167939
/var/tmp/027015 (deleted)
0000020000072000-000002000008c000 rw-s 0000000000000000 08:03 167940
/var/tmp/027015 (deleted)
0000020000090000-00000200000aa000 rw-s 0000000000000000 08:03 167941
/var/tmp/027015 (deleted)
00000200000ae000-00000200000c8000 rw-s 0000000000000000 08:03 167942
/var/tmp/027015 (deleted)

> Are any other utilities more suitable than lsof(8) for investigating
> this?

I would love to see a tool capable to do this thing smartly - lsof
tries to be too low-level tool in most cases, and refuses to work
if kernel symbol tables don't match the compiled in data...

However, "cat" is quite potent tool in itself..

> The system is a Alpha 21164, UX2/Ruffian motherboard running Red Hat 6.0
> for Alpha. The kernel is 2.2.9 with the Linux Virtual Server
> (ipvs-0.4-2.2.9.tar.gz) patch applied.
>
> Thanks in anticipation
> John Connett (jrc@art-render.com)

/Matti Aarnio <matti.aarnio@sonera.fi>

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.rutgers.edu
Please read the FAQ at http://www.tux.org/lkml/