Re: Data corruption issue with splice() on

From: Willy Tarreau
Date: Mon Jan 19 2009 - 04:53:47 EST

On Mon, Jan 19, 2009 at 09:39:21AM +0100, Lennert Buytenhek wrote:
> On Tue, Jan 06, 2009 at 07:50:12PM +0100, Willy Tarreau wrote:
> > > Thanks a lot for the test application, it will greatly help to resolve
> > > this issue.
> >
> > I figured it was an absolute necessity. The original code in my proxy is in
> > an experimental state and far too hard to debug for these purposes. It was
> > enough to detect the problem, but I could run a lot more tests with this
> > small test app ! An who know, maybe it will serve as an example for
> > non-blocking splice ;-)
> :-)
> Just to throw some more (hacky) example code into the pool, below is
> an echo server that I hacked up to test nonblocking splice(). (You'll
> need to use it.) I also have a splice()
> discard server and a patch to my intercept-connection-via-iptables-and-
> forward-it-to-a-remote-SOCKS5-server-to-deal-with-crappy-VPNs app to
> use splice() somewhere.
> My main annoyances with splice(2) are/were:
> 1. -EAGAIN return on splice from socket/pipe to socket/pipe doesn't
> directly tell you whether the source ran out of data or the
> destination can't accept more data, which means you can't e.g. use
> epoll in edge triggered mode without jumping through some minor
> number of extra hoops. (For a pipe you can keep track of how many
> bytes are in it by hand, but for a socket->pipe splice -EAGAIN return
> you'll have to assume that the pipe is full if there are >0 bytes in
> it.)

I proceeded the same way : if EAGAIN and data still in the pipe, then
stop polling.

> 2. Because of (1), and because when splicing from a socket to a pipe
> it returns after the first bit of data (you mentioned this as well),
> you don't know at that point whether your pipe is full or not.

In fact this is fixed now. tcp_splice_read() returns all available data,
which somewhat hides problem #1. I'm running with 23 kB in a push/pull
method all the time, so it remains optimal.

> 3. Always returns -EAGAIN even if there was a FIN or error on the
> source socket. (Now fixed.)

Yes I saw your fix, it was indeed very annoying because the only workaround
I found was to perform an recv(MSG_PEEK) on the socket after each EAGAIN
to check whether the connection was closed or not.

For these reasons, I'd really love to see the few recent fixes backported
to -stable ASAP. It will boost splice() adoption among products.


To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at
Please read the FAQ at