Re: Long standing kernel warning: perfevents: irq loop stuck!

[Josh Hunt]

On Mon, Aug 12, 2019 at 12:42 PM Josh Hunt <joshhunt00@xxxxxxxxx> wrote:
>
> On Mon, Aug 12, 2019 at 12:34 PM Thomas Gleixner <tglx@xxxxxxxxxxxxx> wrote:
> >
> > On Mon, 12 Aug 2019, Josh Hunt wrote:
> > > On Mon, Aug 12, 2019 at 10:55 AM Thomas Gleixner <tglx@xxxxxxxxxxxxx> wrote:
> > > >
> > > > On Mon, 12 Aug 2019, Josh Hunt wrote:
> > > > > Was there any progress made on debugging this issue? We are still
> > > > > seeing it on 4.19.44:
> > > >
> > > > I haven't seen anyone looking at this.
> > > >
> > > > Can you please try the patch Ingo posted:
> > > >
> > > >   https://lore.kernel.org/lkml/20150501070226.GB18957@xxxxxxxxx/
> > > >
> > > > and if it fixes the issue decrease the value from 128 to the point where it
> > > > comes back, i.e. 128 -> 64 -> 32 ...
> > > >
> > > > Thanks,
> > > >
> > > >         tglx
> > >
> > > I just checked the machines where this problem occurs and they're both
> > > Nehalem boxes. I think Ingo's patch would only help Haswell machines.
> > > Please let me know if I misread the patch or if what I'm seeing is a
> > > different issue than the one Cong originally reported.
> >
> > Find the NHM hack below.
> >
> > Thanks,
> >
> >         tglx
> >
> > 8<----------------
> >
> > diff --git a/arch/x86/events/intel/core.c b/arch/x86/events/intel/core.c
> > index 648260b5f367..93c1a4f0e73e 100644
> > --- a/arch/x86/events/intel/core.c
> > +++ b/arch/x86/events/intel/core.c
> > @@ -3572,6 +3572,11 @@ static u64 bdw_limit_period(struct perf_event *event, u64 left)
> >         return left;
> >  }
> >
> > +static u64 nhm_limit_period(struct perf_event *event, u64 left)
> > +{
> > +       return max(left, 128ULL);
> > +}
> > +
> >  PMU_FORMAT_ATTR(event, "config:0-7"    );
> >  PMU_FORMAT_ATTR(umask, "config:8-15"   );
> >  PMU_FORMAT_ATTR(edge,  "config:18"     );
> > @@ -4606,6 +4611,7 @@ __init int intel_pmu_init(void)
> >                 x86_pmu.pebs_constraints = intel_nehalem_pebs_event_constraints;
> >                 x86_pmu.enable_all = intel_pmu_nhm_enable_all;
> >                 x86_pmu.extra_regs = intel_nehalem_extra_regs;
> > +               x86_pmu.limit_period = nhm_limit_period;
> >
> >                 mem_attr = nhm_mem_events_attrs;
> >
> Thanks Thomas. Will try this and let you know.
>
> --
> Josh

Thomas

I found on my setup that setting the value to 32 was the lowest value
I could use to keep the problem from happening. Let me know if you
want me to send a patch with the updated value, etc.

I saw in the original thread from Ingo and Vince that this was seen on
Haswell, but I checked our Haswell boxes and so far we have not
reproduced the problem there.

-- 
Josh