Re: Pipe buffers' limit of 16 * 4K

[Fausto Richetti Blanco]

I think you did not receive it so I'm sending again....

On Mon, May 26, 2008 at 2:43 PM, Fausto Richetti Blanco
<fausto.blanco@xxxxxxxxx> wrote:
> Hi Rik,
>
>    I'll try to explain my problem...
>
>    Well, we have a cgi which uses a lib. This lib is used by a lot of
> other systems and we don't want to change it nor create a specific
> version of the lib to solve our problem. The webserver passes the post
> content to the cgi via stdin which, in this case, is a pipe.
>
>    Our problem is that our application must look at the post content
> (ie.: by reading stdin and, consequently, removing its content from
> the pipe buffer), take some decisions and then call the lib. The lib
> itself will also look at the stdin and here is the problem: since our
> application had already consumed the input content, we must restore it
> in order to the lib to be able to read it again.
>
>    We can't write to stdin because we only have access to the 'read
> side' of the pipe. The way we find to circumvent this is by creating
> another pipe. So, our application reads the stdin and saves its
> content in a buffer. To restore the stdin, we are doing this:
>
>        void restore_input(void){
>                int filedes[2];
>                pipe(filedes);
>                write(filedes[1], newstdin, stdbuffsz); // *1
>                close(filedes[1]);
>                close(STDIN_FILENO);
>                dup2(filedes[0], STDIN_FILENO);
>                close(filedes[0]);
>        }
>        //  *1 newstdin is the buffer we have saved the input
>
>    So, we can:
>        read_input_and_save_it()
>        take_our_decisions()
>        restore_input()
>        call_the_lib()
>
>    It works very well, except when the input has more than 4K (or 16
> * 4K in more recent kernels) because the restore_input() blocks at
> this limit.
>
>    I kwow there are other solutions to my problem (e.g: using a
> thread, moving our decisions to the lib, etc...) by I'm wondering if
> making the pipe buffers' limit adjustable is not a good idea. Maybe it
> should be helpful for another things too (like Jan Engelhardt said in
> his email).
>
>    In fact, I didn't find any way of restoring the input (with the
> input being the 'read side' of a pipe) other than using pipes. That's
> because I've decided to ask this in the linux-kernel list. Is there a
> reason for this limit not to be an adjustable parameter ?
>
>    Thanks in advance,
>
>          Fausto Richetti Blanco
>
> On Fri, May 23, 2008 at 9:56 PM, Rik van Riel <riel@xxxxxxxxxx> wrote:
>> On Fri, 23 May 2008 21:19:13 -0300
>> "Fausto Richetti Blanco" <fausto.blanco@xxxxxxxxx> wrote:
>>
>>> Hello guys,
>>>
>>>        I'm working with a 2.6.9 kernel (ok, I know it's quite old) and
>>> faced a problem with the 4K (one page) buffer limit for the pipes.
>>> I've found that in the 2.6.11 the pipes' buffers were changed to a
>>> circular list of pages which increased this limit to 16 * 4K. This
>>> limit is hardcoded here /usr/src/linux/include/linux/pipe_fs_i.h:6
>>> #define PIPE_BUFFERS (16)
>>>
>>>        Is there a reason for this not to be an adjustable parameter
>>> (eg.: by an ulimit in the userspace) ?
>>
>> What is the problem you found?
>>
>> Why do you need to change the limit from 16?
>>
>> Did it bring you any performance enhancements?
>>
>> If so, how much?
>>
>> --
>> All rights reversed.
>>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/