Re: multiply files in one (was GNU/Linux stance by Richard Stallman)

Richard Gooch (rgooch@atnf.csiro.au)
Tue, 30 Mar 1999 10:38:25 +1000


Larry McVoy writes:
> : > If you have 200 small files and you have to go get 200 inodes and
> : > 200 blocks to read them in, the inodes and the blocks aren't next to
> : > each other, that's 400 disk transactions instead of 1.
> :
> : OK, so please explain the scale of what you have in mind. How big is
> : the compressed tarball, and how big is it uncompressed?
> :
> : Do you plan on having a large FS (say /usr) stored in this way, or are
> : you thinking of a selected subset?
>
> Do it on a per directory basis, only for files where links == 1.
>
> I wouldn't bother to compress - just putting all the files into a
> ``tarball'' will compress them quite nicely because of the lack of
> fragmentation. Compression isn't the point, disk access / file read
> is the point. For a large group of small files I want 2 disk
> accesses (inode + tarball) for all files, instead of 2/file.

OK, fine. I'm agreeing with that principle: two seeks per file is
expensive. Better two seeks for a whole bunch of files.

> : Note that I'm thinking about reading-ahead the inodes *and* the
> : blocks. If you have RAM to burn, then this looks like it should
> : work. I'm not thinking in terms of reading ahead a few piddling
> : blocks. I'm talking megabytes or dozens of megabytes.
> :
> : So with two seek operations, you can read-ahead lots of files and lots
> : of data.
>
> More brain cells, more, more, more, I want more. :-)
>
> It's a transaction cost, not a platter speed cost. Read ahead does
> not help you on small files and inodes.

I'm still missing your point, then. The way I see it, if you read in
dozens of megabytes of disc blocks into your buffer cache (and say a
megabyte of inode blocks), then things should speed up a lot.

On the first file read, you go to the inode area and read in a
megabyte. So now a truckload of inodes blocks are sitting in RAM.
Then you go to the data block area, and read in dozens of
megabytes. Now truckloads of data blocks are sitting in RAM.

On a subsequent read of another file, there is a high chance that the
inode and data blocks are sitting in the buffer cache, so there is no
transaction with the device.

Combine this with reiserfs and it may perform even better.

> Look, if things worked the way I'm describing then you should see a
> quantum leap in small file performance. Think 10x faster and you'll
> see that the number of I/O's have to go down by at least a factor of
> 10.

I don't see why my suggestion won't yield that performance
increase. Tell me why not.

Regards,

Richard....

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.rutgers.edu
Please read the FAQ at http://www.tux.org/lkml/