Re: Yet more softlockups.

From: Dave Jones
Date: Fri Jul 12 2013 - 13:19:16 EST


On Fri, Jul 12, 2013 at 11:12:13AM -0600, David Ahern wrote:
> On 7/12/13 9:45 AM, Dave Jones wrote:
> > Here's a fun trick:
> >
> > trinity -c perf_event_open -C4 -q -l off
> >
> > Within about a minute, that brings any of my boxes to its knees.
> > The softlockup detector starts going nuts, and then the box wedges solid.
>
> I tried that in a VM running latest Linus tree. I see trinity children
> getting nuked regularly from oom.

Weird. I'm curious what the backtrace looks like in those cases.
Where is it trying to allocate memory ?
(Though that isn't usually too helpful in most cases, but in absense of
anything else..)

> I was dumping Vm elements using:
>
> while [ 1 ]; do echo $(date) $(egrep Vm /proc/$pid/status); sleep 1; done
>
> And right before the process is killed was the line:
>
> Fri Jul 12 11:00:19 MDT 2013 VmPeak: 2867472 kB VmSize: 2867472 kB
> VmLck: 0 kB VmPin: 0 kB VmHWM: 1493092 kB VmRSS: 1493092 kB VmData:
> 2857944 kB VmStk: 136 kB VmExe: 100 kB VmLib: 1844 kB VmPTE: 5628 kB
> VmSwap: 0 kB
>
> The VmData is growing fairly steadily and strace shows a lot of brk
> calls. Is that normal for trinity - or this command line?

Hmm, there are a few known leaks in trinity that I still haven't got
around to fixing, but none that should be triggered from this.

I wonder if something about being virtualised is perturbing its behaviour
somehow. Though I can't think what would matter.

> Looking at the perf_event_open calls I see a lot of E2BIG errors in
> addition to EINVAL. e.g,
>
> ...
> perf_event_open(0xba9000, 0, 0x4c, 0xcc, 0) = -1 EINVAL (Invalid argument)
> alarm(0) = 1
> getppid() = 9031
> alarm(1) = 0
> perf_event_open(0xba9000, 0x2a6e, 0xe, 0xfd, 0) = -1 E2BIG (Argument
> list too long)
> alarm(0) = 1
> getppid() = 9031
> alarm(1) = 0

Yep, that'll happen.

Perhaps try CONFIG_MEMLEAK ?

Dave

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/