> > One advantage (?) that NT has is the fact that it can throw files out to a
> > socket without any user-mode/kernel-mode transitions (and buffer copying)
> > using the TransmitFile APIs. IIS running in a single process probably
> > doesn't hurt either.
>
> First of all, threading. Naively threading apache is not going to be a
> win on linux. Tell me if I'm wrong: thread creation and process creation
> speeds are about the same. Additionally, there's no huge memory savings
> to be had -- I see about 80k to 100k unshared data per httpd-child on
> a server with hundreds of children serving millions of hits per day.
> This is essentially only the request-specific data, plus a few other
> unshared pages.
Apache originally was not threaded, and I don't know, which versions are
threaded -- they definitely are not suitable for any platform but NT or
Solaris, where native threads are more efficient compared to processes.
Multithreading is not a good thing for heavy-loaded servers unless the
platform has extremely poor context switching, so multiplexed/multitasked
servers perform poorly.
> Another fun NT optimization is the whole WaitForCompletion() thing,
> which is like select() on steroids. This is where the big threading
> wins come in. WaitForCompletion has the semantic that when an i/o
> event happens it wakes up exactly one thread and hands it the event.
> Whereas select() has the semantic that it wakes up all threads and they
> fight to see who gets to deal with the event. Not only that, NT will
> wake up the most-recently awake thread which increases cache-coolness.
> To utilize this in the Apache programming model would require a user
> level threads package layered on top of kernel threads.
select() (BSD) and poll() (SysV) are designed for multiplexed i/o and
are supposed to be the only operations performed on selected file
descriptors at the time. With threads they will get the worst of both
worlds.
> Unix can do almost the equivalent of TransmitFile using mmap(). You have
> open/mmap/write instead of open/TransmitFile. So to see winnings of
> TransmitFile, open/mmap once and cache the sucker.
Right now I'm rewriting my ftp/http server's file i/o to avoid
unnecessary process creation (it already uses mmap()/send()), and the
whole design that I use, is completely "anti-threads" -- normal
multiplexed i/o, pipes, etc. My "video streaming" webcam server is
implemented using its API, and the performance is good enough, but since
file I/O still used "fork-process-for every file" model, performance on
sending plain non-cached files (it also has asyncronously processed cached
files) was lower than Apache, and that kept me from announcing it widely.
Now I've modified API to allow moving file i/o into the module from the
main server without introducing additional buffering that was good for
processing modules, but will be bad for files, so now module that is
running as a separate process can choose to send data to the client using
passed file descriptor and bypass buffering that otherwise will be
provided by the main server if the same data was passed through the pipe
to it. So far the approach looks promising, "heavy-processing" modules can
use pipes and leave buffering to the main server, and "light-processing"
file i/o can be sone directly by sending file into socket to the client.
> Threaded apache is some unknown time away, so I've been trying to
> figure out how to dynamically build (and adapt) an open/mmapped file
> cache in a multiprocess server. Brilliant ideas welcome. Oh yeah,
> you can assume that it's relatively cheap, and definately safe for the
> parent to blow away hundreds of children and respawn them occasionally.
> It can even do that without interrupting a hit in progress.
A lot of servers used "fork-process-for-every-file" approach, and
changed it to either "pre-fork-then-pass-descriptors" (ncsa) or
"fork-then-every-process-accept()s" (apache). Which version of apache uses
threads, I have no idea, but it definitely should be inferior to
non-threaded one on every platform that does not have poorly-designed
process context switching, that should be compensated by use of threads. I
know two platforms of that kind, and they are NT and Solaris.
However, simultaneous accept() in multiple processes can cause problem
that was called "thundering herd" on freebsd-hackers ML -- all processes
awakened, but only one succeeds in accept() when connection arrives.
FreeBSD fixed that in 2.2.?, although I don't know if such problem exists
in Linux.
> The NT port in 1.3-dev is multithreaded, and went in relatively easily.
> It hasn't been stressed at all though, and hasn't been optimized.
> Look for an alpha/beta in a month or so. Threaded apache on unix
> won't happen for a while, we're hoping to learn from the NT experience.
> There are other arguments for delaying threaded unix code... in particular
> the severe lack of third party libraries that support threading (such
> as dns, various database libraries, perl).
I hope, the idea to make everything threaded will be abandoned. There
are already enough multithreaded servers (ex: Netscape) that perform
"well" on NT and Solaris, and something should exists for systems with
better designed kernels, too.
-- Alex