Re: AUTOSEL process

[Sasha Levin]

On Mon, Feb 27, 2023 at 07:41:31PM -0800, Eric Biggers wrote:
> On Mon, Feb 27, 2023 at 08:53:31PM -0500, Sasha Levin wrote:
> > >
> > > I'm shocked that these are the statistics you use to claim the current AUTOSEL
> > > process is working.  I think they actually show quite the opposite!
> > >
> > > First, since many AUTOSEL commits aren't actually fixes but nearly all
> > > stable-tagged commits *are* fixes, the rate of regressions per commit would need
> > > to be lower for AUTOSEL commits than for stable-tagged commits in order for
> > > AUTOSEL commits to have the same rate of regressions *per fix*.  Your numbers
> > > suggest a similar regression rate *per commit*.  Thus, AUTOSEL probably
> > > introduces more regressions *per fix* than stable-tagged commits.
> >
> > Interesting claim. How many of the AUTOSEL commits are "actual" fixes?
> > How do you know if a commit is a fix for anything or not?
> >
> > Could you try and back claims with some evidence?
> >
> > Yes, in a perfect world where we know if a commit is a fix we could
> > avoid introducing regressions into the stable trees. Heck, maybe we could
> > even stop writing buggy code to begin with?
>
> Are you seriously trying to claim that a random commit your neural network
> picked up is just as likely to be a fix as a commit that the author explicitly
> tagged as a fix and/or for stable?

I'd like to think that this is the case after the initial selection and
multiple rounds of reviews, yes.

> That's quite an extraordinary claim, and it's not true from my experience.  Lots
> of AUTOSEL patches that get Cc'ed to me, if I'm familiar enough with the area to
> understand fairly well whether the patch is a "fix", are not actually fixes.  Or
> are very borderline "fixes" that don't meet stable criteria.  (Note, I generally
> only bother responding to AUTOSEL if I think a patch is actually going to cause
> a problem.  So a lack of response isn't necessarily agreement that a patch is
> really suitable for stable...)
>
> Oh sorry, personal experience is not "evidence".  Please disregard my invalid
> non-evidence-based opinion.
>
> > > (Of course, stable-tagged commits sometimes have missing prerequisite bugs too.
> > > But it's expected to be at a lower rate, since the original developers and
> > > maintainers are directly involved in adding the stable tags.  These are the
> > > people who are more familiar than anyone else with prerequisites.)
> >
> > You'd be surprised. There is documentation around how one would annotate
> > dependencies for stable tagged commits, something along the lines of:
> >
> > 	cc: stable@xxxxxxxxxx # dep1 dep2
> >
> > Grep through the git log and see how often this is actually used.
>
> Well, probably more common is that prerequisites are in the same patchset, and
> the prerequisites are tagged for stable too.  Whereas AUTOSEL often just picks
> patch X of N.  Also, developers and maintainers who tag patches for stable are
> probably more likely to help with the stable process in general and make sure
> patches are backported correctly...
>
> Anyway, the point is, AUTOSEL needs to be fixed to stop inappropriately
> cherry-picking patch X of N so often.

That's a fair point.

> > > a multi-patch series, and if so are earlier patches needed as prerequisites".
> > > There also needs to be more soak time in mainline, and more review time.
> >
> > Tricky bit with mainline/review time is that very few of our users
> > actually run -rc trees.
> >
> > We end up hitting many of the regressions because the commits actually
> > end up in stable trees. Should it work that way? No, but our testing
> > story around -rc releases is quite lacking.
>
> Well, in the bug that affected me, it *was* found on mainline almost
> immediately.  It just took a bit longer than the extremely aggressive 7-day
> AUTOSEL period to be fixed.
>
> Oh sorry again, one example is not "evidence".  Please disregard my invalid
> non-evidence-based opinion.

I'm happy that we're in agreement that significant process changes can't
happen because of opinions or anecdotal examples.

In all seriousness, I will work on addressing the issues that happened
around the commit(s) you've pointed out and improve our existing
process.

--
Thanks,
Sasha