Re: [RFC][PATCH 0/9] File descriptor hot-unplug support

[Eric W. Biederman]

Jamie Lokier <jamie@xxxxxxxxxxxxx> writes:

> Eric W. Biederman wrote:
>> > I don't have anything at hand but multithread/process server accepting
>> > on the same socket comes to mind.  I don't think it would be a very
>> > rare thing.  If you confine the scope to character devices or sysfs,
>> > it could be quite rare tho.
>> 
>> Yes.  I think I can safely exclude sockets, and not bother with
>> reference counting them.
>
> Good idea.  As well as many processes calling accept(), it's not
> unusual to have two threads or processes for reading and writing
> concurrently to TCP sockets, and to have a single UDP socket shared
> among threads/processes for sendto.

I have been playing with what I can see when I instrument up my code.

The first thing that popped up was that we have a lots of reads/writes
to files with f_count > 1.  Which defeats my micro optimization in
fops_read_lock.  So in those cases I still have to pay the full cost
of an atomic even if I have an exclusive cache line.

I have found that for make -j N I tend to get N processes all
reading from the same pipe at the same time.  Not a smoking
gun that my assumption that only one process will be using
a file descriptor at a time in performance paths but it certainly
shows that things are nowhere near as rare as I thought.

The good news is that I have found a much better/cheaper optimization.
Instead of per cpu or per file memory, use per task memory.  It is
always uncontended, and a task appears to never use more than two files
simultaneously (stacking?).

I have just prototyped that and things are looking very promising.
Now I just need to clean everything up and resend my patches.

>> The only strong evidence I have that multi-threading on a single file
>> descriptor is likely to be common is that we have pread and pwrite
>> syscalls.  At the same time the number of races we have in struct file
>> if it is accessed by multiple threads at the same time, suggests
>> that at least for cases where you have an offset it doesn't happen often.
>
> Notice the preadv and pwritev syscalls added recently?  They were
> added because QEMU and KVM need them for performance.  Those programs
> have multiple threads doing I/O to the same file concurrently.  It's
> like a poor man's AIO, except it's more reliable than real Linux AIO :-)
>
> Databases probably should use concurrent p{read,write}{,v} if they're
> not using direct I/O and AIO.  I'm not sure if the well-known
> databases do.  In the past there have been some poor quality
> "emulations" of those syscalls prone to races, on Linux and BSD I believe.
>
> What are the races you've noticed?

Besides the f_pos (which pread variants handle) there is no locking on
the file read ahead state, and f_flags only got locking a month or two
ago.


Eric
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/