Re: AUTOSEL process

[Eric Biggers]

On Mon, Feb 27, 2023 at 08:53:31PM -0500, Sasha Levin wrote:
> > 
> > I'm shocked that these are the statistics you use to claim the current AUTOSEL
> > process is working.  I think they actually show quite the opposite!
> > 
> > First, since many AUTOSEL commits aren't actually fixes but nearly all
> > stable-tagged commits *are* fixes, the rate of regressions per commit would need
> > to be lower for AUTOSEL commits than for stable-tagged commits in order for
> > AUTOSEL commits to have the same rate of regressions *per fix*.  Your numbers
> > suggest a similar regression rate *per commit*.  Thus, AUTOSEL probably
> > introduces more regressions *per fix* than stable-tagged commits.
> 
> Interesting claim. How many of the AUTOSEL commits are "actual" fixes?
> How do you know if a commit is a fix for anything or not?
> 
> Could you try and back claims with some evidence?
> 
> Yes, in a perfect world where we know if a commit is a fix we could
> avoid introducing regressions into the stable trees. Heck, maybe we could
> even stop writing buggy code to begin with?

Are you seriously trying to claim that a random commit your neural network
picked up is just as likely to be a fix as a commit that the author explicitly
tagged as a fix and/or for stable?

That's quite an extraordinary claim, and it's not true from my experience.  Lots
of AUTOSEL patches that get Cc'ed to me, if I'm familiar enough with the area to
understand fairly well whether the patch is a "fix", are not actually fixes.  Or
are very borderline "fixes" that don't meet stable criteria.  (Note, I generally
only bother responding to AUTOSEL if I think a patch is actually going to cause
a problem.  So a lack of response isn't necessarily agreement that a patch is
really suitable for stable...)

Oh sorry, personal experience is not "evidence".  Please disregard my invalid
non-evidence-based opinion.

> > (Of course, stable-tagged commits sometimes have missing prerequisite bugs too.
> > But it's expected to be at a lower rate, since the original developers and
> > maintainers are directly involved in adding the stable tags.  These are the
> > people who are more familiar than anyone else with prerequisites.)
> 
> You'd be surprised. There is documentation around how one would annotate
> dependencies for stable tagged commits, something along the lines of:
> 
> 	cc: stable@xxxxxxxxxx # dep1 dep2
> 
> Grep through the git log and see how often this is actually used.

Well, probably more common is that prerequisites are in the same patchset, and
the prerequisites are tagged for stable too.  Whereas AUTOSEL often just picks
patch X of N.  Also, developers and maintainers who tag patches for stable are
probably more likely to help with the stable process in general and make sure
patches are backported correctly...

Anyway, the point is, AUTOSEL needs to be fixed to stop inappropriately
cherry-picking patch X of N so often.

> > a multi-patch series, and if so are earlier patches needed as prerequisites".
> > There also needs to be more soak time in mainline, and more review time.
> 
> Tricky bit with mainline/review time is that very few of our users
> actually run -rc trees.
> 
> We end up hitting many of the regressions because the commits actually
> end up in stable trees. Should it work that way? No, but our testing
> story around -rc releases is quite lacking.

Well, in the bug that affected me, it *was* found on mainline almost
immediately.  It just took a bit longer than the extremely aggressive 7-day
AUTOSEL period to be fixed.

Oh sorry again, one example is not "evidence".  Please disregard my invalid
non-evidence-based opinion.

> I'm not sure how feedback in the form of "this sucks but I'm sure it
> could be much better" is useful.

I've already given you some specific suggestions.

I can't force you to listen to them, of course.

- Eric