I'll provide more details in this note and answer some of
the questions asked. I'll try to combine replies in this
note.
>From: "David Schwartz" <davids@webmaster.com>
>Subject: RE: web server (and TCP) performance tuning?
>Date: Mon, 12 Jun 2000 22:32:30 -0700
>
> > I spent the day implementing a poll()-based version of
> > the server and making sure it was correct. To be efficient,
> > only the file descriptors that are in use are in the
> > pollfd array. However, this isn't a great savings for
> > me, since about 80% of the descriptors turn out to be
> > "active" in this server. I tested it using 4000 descriptors.
>
> Then you're probably wasting more time putting things into and out of the
>array than the cost to just leave the array as it is.
Most of the pollfd array stays the same between calls - only those
entries that have changed are added to or removed from the list.
Alan - I was comparing this version using poll() to my
earlier kernel build where I changed FD_SETSIZE for select().
That was how I was able to run 4000 descriptors in the select()
version.
> > This new implementation using poll() seemed to actually
> > have higher overhead than the select()-based version. On
> > some level, this makes sense - select uses 2 bits per
> > descriptor for the read/write sets, and since they're
> > in/out parameters, that's 4 bits each. Poll uses one int
> > plus two shorts, for a total of 64 bits per descriptor.
>
> You're probably doing something wrong. For example, you should be building
>the 'poll' array just once. You don't need to rebuild the entire array each
>time you call 'poll'.
I should have specified that I meant read/write parameters
across the system call. I was trying to figure out how much
data the kernel has to copy in and out for each invocation
of poll and select. I'm making the assumption that the kernel
has to copy the fdsets and the pollfd in parameters into its
own buffers rather than use them directly in user space.
> > Even so, that wouldn't explain the performance I'm getting
> > with either version. Are there any tunable parameters
> > relating to the networking code? Hash table sizes,
> > enabling/disabling delayed acks, etc? The same program on
> > a different operating system gets about 50% higher performance.
> > I'm sure I'm overlooking something small. Any suggestions
> > would be appreciated.
>
> Odds are there's something bad about your program that hurts it more on
>Linux than other platforms.
System details - This is a uniprocessor 500 MHz Celeron with a
3Com 3C905C 10/100 card and 512MB of memory. I've verified that
memory consumption is low and there's no paging activity. I'm
running 2.2.14-5.0 from a RedHat 6.2 install.
The same code/hardware has been testing using FreeBSD 4.x,
which got the higher performance result I cited earlier.
I'd like to see if I can bring up the Linux numbers.
> It may help to use multiple threads and have one thread that polls on the
>busiest file descriptors and one that polls on the less busy ones. This
>way,
>your most frequent calls to poll will be for a smaller number of
>descriptors, minimizing the number of wait queues the kernel has to put you
>on and pop you off of.
>
> It may also help to use one thread per 1,000 file descriptors. This way,
>each call to poll requires the kernel to process at most 1,000 file
>descriptors.
Thanks for the suggestions. Both require more changes to this code
than I'd like, and since it's supposed to be portable, this may be
a problem. I think I'll first try the /dev/poll and RT-signal
approaches mentioned on Dan Kegel's pages (thanks Dan). It looks
like they require fewer invasive changes to the code. Unfortunately,
neither seems to be available on a 2.2 kernel.
Thanks again,
Lucius Chang
________________________________________________________________________
Get Your Private, Free E-mail from MSN Hotmail at http://www.hotmail.com
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.rutgers.edu
Please read the FAQ at http://www.tux.org/lkml/
This archive was generated by hypermail 2b29 : Thu Jun 15 2000 - 21:00:28 EST