Yep, it's so much fun having to modify the code to increase the arena
size when you 1) get more CPUs, 2) launch more threads.
> My hope for the development of clone() was originally:
> - I laid the basic framework (done)
> - people started using it, primarily for
> - pthreads (pretty much done)
> - asynchronous IO (nope)
In fact, the application that really started me thinking in earnest
about clone() problems is sort-of asynchronous I/O: deep in my support
library I have "Channel" object, a bit like C library "FILE *"
streams, but properly full duplex (i.e. a separate read and write
buffer) plus a lot more smarts. Set an attribute and suddenly any
blocking writes as well as read-ahead can be done by a child thread,
leaving the main thread to write without blocking but also without
having to worry about rescheduling writes if you get EAGAIN.
> - interesting new uses that nobody thought about before (nope)
> - people started looking at what was missing and add kernel support
>
> The basic problem I had was that I had lots of ideas on what I wanted to
> be possible, but I was also aware that while I wanted a very specific
> basic design, I was by no means sure exactly which details need to be
> handled where.
>
> For example, the exact bits in the clone() flags field were made up by me
> not because I wanted those exact bits, but because I imagined that those
> bits might possibly make sense. For example, I am certain that the
> CLONE_VM bit makes sense, but does CLONE_FS make sense? I don't know.
> Maybe it would be better to include the current CLONE_FS information into
> the CLONE_FILES stuff?
Does it matter? It's there now, you don't have to use it, and it only
costs you one bit. Who knows: someone might figure out a reason to use
one and not the other. Actually, now that I think about it, I realise
that I'm now doing just that: I use CLONE_FS but not CLONE_FILES
because of that trick I mentioned earlier with pipes so that threads
can know when the parent dies. I still want to CLONE_FS, since
changing directory should still be global. There you go.
> Or take a look at the low 8 bits. I decided that to implement aio_xxx() on
> top of clone() we might be better off not using SIGCHLD, but instead have
> the death of the clone'd child send SIGIO directly. So now the low eight
> bits of clone_flags is the signal to be sent at exit time.
I like that feature: I use SIGKILL so that if a child dies, it takes
down the parent (there are reasons I don't want to deal with
SIGCHLD). With my "death pipe" scheme, when the parent is killed the
remaining child threads kill themselves off.
> I still don't know if people actually use this feature - or the feature
> that when the signal is something else than SIGCHLD, you have to use a
> special flag to "waitpid()" to get it to recognize the cloned children.
> Again, this was so that "waitpid()" wouldn't end up waiting for
> asynchronous IO.
>
> In short, I tried to make a basic framework that would fit my ideas of
> what a light-weight clone() would be good for, but at the same time I
> didn't want to set the design in stone - the intention was for it to
> develop as people found new ways of (mis)using the new cool thread
> feature.
I don't think you've made the interface over-flexible. I think all
those features are being used somehow.
> > > * modifications to the linuxthreads package for testing all this out
> > > * other things that may be needed -- linuxthreads needs a little bit of
> > > help in the kernel in the few places and presumably other more general
> > > clone based threads need the same sort of stuff. For example,
> > > linuxthreads uses SIGUSR1, SIGUSR2, and it also needs a "manager thread"
> > > to handle things like stack cleanup at termination.
> >
> > None of these address the problem of killing child threads. Although
> > it's all good stuff you mention: nice to see someone thinking about
> > all this.
> > What was Linus' view on encoding IDs in the upper pid bits? It's
> > certainly good for grouping processes together, though it may be more
> > prone to introducing bugs than a simple CLONE_NO_PROC_ENTRY like I
> > suggested, by the simple rule that anything more complicated is likely
> > to have more bugs :-)
>
> Encoding the thread ID in the high bits was one of the ideas from the very
> beginning. That's what CLONE_PID is there for: the _intent_ was that
> CLONE_PID would change only the high bits, and then you could do a global
> kill (anything with the high bits zero would send a signal to _all_
> threads that shared the same low bits).
>
> Again, this is still an interesting approach. I'd like to see it done some
> day. The fact that the /proc fs makes it a bit harder is a misfeature, but
> that's actually due to bad /proc design (which used to make a lot of sense
> back when inode numbers was all we had, but we could do better these
> days).
>
> My personal favourite for /proc would be that any CLONE_PID threads would
> show up _inside_ the original parent (that's kind of the basic idea with
> CLONE_PID). So you'd have
>
> /proc/155/ "original" process (ie something that was
> created without the CLONE_PID bit)
> /proc/155/1 "1st CLONE_PID child"
> /proc/155/2 "2nd CLONE_PID child"
>
> or something like that.
YES, YES, YES. The way it should be. Better than the Solaris scheme,
where you need a special syscall to dig into the process and get the
LWPs, but without cluttering ps output.
> > Can I suggest that, where possible, improvements to thread support in
> > Linux is made as a set of separate patches? We've seen before that
> > Linus rejects omnibus patches if he doesn't like some bits, even if
> > other bits are OK. Giving them as separate (independent) patches makes
> > life easier for him (and hence more likely that "good" patches are
> > applied quickly).
>
> Indeed.
Thought that approach might help :-)
> > * new flag to clone() to allow pids to be "grouped" so that part of
> > the bitrange for pids are shared within a group. Also needs pid
> > allocation algorithm to change
>
> CLONE_PID is that. It currently has a very limited use: the kernel uses it
> to allocate the SMP idle processes for each task, and those all have to
> have pid 0 (also high bits). But my real intent was to have something like
> this:
>
> if (flags & CLONE_PID) {
> newpid = current->pid;
> /* zero is special - the idle process */
> if (newpid) {
> create linked list of processes
> sharing the same low 16 bits,
> make "newpid" be the largest to
> date plus 0x10000 (ie "increment" the
> high 16 bit counter on a per-PID basis)
> }
> } else
> newpid = traditional_newpid();
>
> > * new flag to clone() to either hide a process in /proc or reflect
> > that the flag was set (thinking about this more makes me think that
> > this scheme can give much the same flexibility as the above with less
> > work in the kernel and possibly a little more work in userspace procps
> > tools)
>
> See above about how I'd like this to work. With the /proc/155/1 setup, the
> old tools would only ever see the original parent, so to "ps" the threaded
> application would look like just one process.
>
> Additionally note that "kill -1 155" would send SIGHUP to _all_ the
> threads, and if you wanted to kill just one subthread you'd have to name
> it completely in 32 bits (ie "kill -1 $((0x10000+155))" would kill 155/1,
> and we'd probably make an extension to bash so that you can say just that:
> "kill -1 155/1" would do the math for you).
If we can get such an extension, that would do, IMHO. Might be messy
otherwise.
> Yes, this does imply that the first thread is special, but I don't see
> anything really wrong with that. If you don't want the first thread to be
> special, just don't use CLONE_PID - then all threads will have a full life
> of their own.
I think the first thread being special is fine. Same as in Solaris.
> > * new signals for LinuxThreads support (no more stealing of SIGUSR1
> > and SIGUSR2)
>
> Yes. This is separate from threads, though. We need this for RT signals
> anyway.
>
> > * new flag to clone() or new syscall prctl() so that when a processes'
> > parent dies, it is sent a signal
>
> Agreed. It's actually technically very easy to send a SIGPARENT (just do
> it in "forget_original_parent()", I think), and it needs another bit in
> the clone flags.
But then you need a new signal? Won't that take some revamping of the
signal handling and libc interface? Not that I have a problem with the
concept of SIGPARENT. I think adding a flag to clone() or a new
syscall to specify a signal to be sent when the parent dies has more
flexibility: there may be times when you want a different signal than
SIGPARENT. I imagine that the default would be that you get SIGPARENT
(which is ignored by default), but you can use clone() or prctl() to
change the signal. In my apps, I would use SIGKILL.
Doing it the way I suggest (i.e do both) has the advantage of getting
a quick patch for delivering a signal to a child. Later the more
difficult (?) job of extending the signal list can be done.
Is that reasonable?
Regards,
Richard....