Re: [PATCH v2] Add /proc/pid_gen

[Andrew Morton]

On Wed, 21 Nov 2018 15:21:40 -0800 Daniel Colascione <dancol@xxxxxxxxxx> wrote:

> On Wed, Nov 21, 2018 at 2:50 PM Andrew Morton <akpm@xxxxxxxxxxxxxxxxxxxx> wrote:
> >
> > On Wed, 21 Nov 2018 14:40:28 -0800 Daniel Colascione <dancol@xxxxxxxxxx> wrote:
> >
> > > On Wed, Nov 21, 2018 at 2:12 PM Andrew Morton <akpm@xxxxxxxxxxxxxxxxxxxx> wrote:
>
> ...
>
> > > I wouldn't call tracing a specialized thing: it's important enough to
> > > justify its own summit and a whole ecosystem of trace collection and
> > > analysis tools. We use it in every day in Android. It's tremendously
> > > helpful for understanding system behavior, especially in cases where
> > > multiple components interact in ways that we can't readily predict or
> > > replicate. Reliability and precision in this area are essential:
> > > retrospective analysis of difficult-to-reproduce problems involves
> > > puzzling over trace files and testing hypothesis, and when the trace
> > > system itself is occasionally unreliable, the set of hypothesis to
> > > consider grows. I've tried to keep the amount of kernel infrastructure
> > > needed to support this precision and reliability to a minimum, pushing
> > > most of the complexity to userspace. But we do need, from the kernel,
> > > reliable process disambiguation.
> > >
> > > Besides: things like checkpoint and restart are also non-core
> > > features, but the kernel has plenty of infrastructure to support them.
> > > We're talking about a very lightweight feature in this thread.
> >
> > I'm still not understanding the seriousness of the problem.  Presumably
> > you've hit problems in real-life which were serious and frequent enough
> > to justify getting down and writing the code.  Please share some sob stories
> > with us!
> 
> The problem here is the possibility of confusion, even if it's rare.
> Does the naive approach of just walking /proc and ignoring the
> possibility of PID reuse races work most of the time? Sure. But "most
> of the time" isn't good enough. It's not that there are tons of sob
> stories: it's that without completely robust reporting, we can't rule
> out of the possibility that weirdness we observe in a given trace is
> actually just an artifact from a kinda-sort-working best-effort trace
> collection system instead of a real anomaly in behavior. Tracing,
> essentially, gives us deltas for system state, and without an accurate
> baseline, collected via some kind of scan on trace startup, it's
> impossible to use these deltas to robustly reconstruct total system
> state at a given time. And this matters, because errors in
> reconstruction (e.g., assigning a thread to the wrong process because
> the IDs happen to be reused) can affect processing of the whole trace.
> If it's 3am and I'm analyzing the lone trace from a dogfooder
> demonstrating a particularly nasty problem, I don't want to find out
> that the trace I'm analyzing ended up being useless because the
> kernel's trace system is merely best effort. It's very cheap to be
> 100% reliable here, so let's be reliable and rule out sources of
> error.

So we're solving a problem which isn't known to occur, but solving it
provides some peace-of-mind?  Sounds thin!

btw, how should tool developers test their pid_gen-based disambiguation
code?

> > > > Which userspace tools will be using pid_gen?  Are the developers of
> > > > those tools signed up to use pid_gen?
> > >
> > > I'll be changing Android tracing tools to capture process snapshots
> > > using pid_gen, using the algorithm in the commit message.
> >
> > Which other tools could use this and what was the feedback from their
> > developers?
> 
> I'm going to have Android's systrace and Perfetto use this approach.
> Exactly how many tools signed up to use this feature do you need?

What other ones are there?

> > Those people are the intended audience and the
> > best-positioned reviewers so let's hear from them?
> 
> I'm writing plenty of trace analysis tools myself, so I'm part of this
> intended audience. Other tracing tool authors have told me about
> out-of-tree hacks for process atomic snapshots via ftrace events. This
> approach avoids the necessity of these more-invasive hacks.

Those authors would make great reviewers!  Adding a cc is cheap.