Re: [PATCH][GIT PULL] tracing: Fix compile issue fortrace_sched_wakeup.c

From: Jason Baron
Date: Thu Oct 21 2010 - 21:45:27 EST


On Thu, Oct 21, 2010 at 01:26:14PM +0200, Ingo Molnar wrote:
> * Steven Rostedt <rostedt@xxxxxxxxxxx> wrote:
>
> > On Wed, 2010-10-20 at 17:40 +0200, Ingo Molnar wrote:
> > > FYI, there's a new mystery hang (sometimes crash) that triggers in -tip - and which
> > > seems to be tracing related. See the crashlog below - config attached.
> > >
> > > It's not bisectable - small changes in the kernel make the bug come/go. (might be a
> > > race of some sorts)
> > >
> >
> >
> > > [ 42.324027] Testing all events:
> > > [ 245.668090] INFO: task swapper:1 blocked for more than 120 seconds.
> > > [ 245.672051] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > > [ 245.676026] swapper D f6420b40 6544 1 0 0x00000000
> > > [ 245.684051] f6437dac 00000046 f694aac0 f6420b40 f6438000 f6437d74 f6438294 f6438290
> > > [ 245.692237] c2192ac0 c204e6c0 c2192ac0 c2192ac0 f6438290 00000000 f6438000 ff2ffa7d
> > > [ 245.701068] 00000009 f6420b40 f6437e5c 7fffffff f6438000 f6437dfc f6437e5c 7fffffff
> > > [ 245.709071] Call Trace:
> > > [ 245.711551] [<c1a7f561>] schedule_timeout+0x1c/0x1e7
> > > [ 245.712036] [<c1a818b6>] ? _raw_spin_unlock_irq+0x2d/0x43
> > > [ 245.716037] [<c1027f2d>] ? sub_preempt_count+0x4/0x98
> > > [ 245.720061] [<c1a818b6>] ? _raw_spin_unlock_irq+0x2d/0x43
> > > [ 245.724036] [<c1027fb4>] ? sub_preempt_count+0x8b/0x98
> > > [ 245.728036] [<c1a7e76b>] wait_for_common+0xc1/0x11a
> > > [ 245.732062] [<c102de32>] ? default_wake_function+0x0/0x12
> > > [ 245.736041] [<c1a7e863>] wait_for_completion+0x17/0x19
> > > [ 245.740069] [<c10667a2>] __stop_cpus+0xdd/0x103
> > > [ 245.744072] [<c1a7e6db>] ? wait_for_common+0x31/0x11a
> > > [ 245.748040] [<c10665a4>] ? stop_machine_cpu_stop+0x0/0x9a
> > > [ 245.752040] [<c106683d>] stop_cpus+0x2c/0x3f
> > > [ 245.756069] [<c10668af>] __stop_machine+0x5f/0x67
> > > [ 245.760186] [<c1006240>] ? stop_machine_text_poke+0x0/0x43
> > > [ 245.764040] [<c1006240>] ? stop_machine_text_poke+0x0/0x43
> > > [ 245.768071] [<c19f0a73>] ? cfdgml_create+0x2b/0xde
> > > [ 245.772040] [<c10060fd>] text_poke_smp+0x3a/0x42
> > > [ 245.776039] [<c19f0a73>] ? cfdgml_create+0x2b/0xde
> >
> >
> > > [ 245.780098] [<c1005b9c>] arch_jump_label_transform+0x53/0x67
> > > [ 245.784042] [<c104ef0d>] jump_label_update+0x49/0x98
> >
> > Looks like this code had jump labels enabled. Do you have a dump where
> > they are not enabled?
>
> No. Good find - and the timeline agrees too, these crashes started triggering when i
> pulled jump labels from you.
>
> Thanks,
>
> Ingo

Hi,

(adding Tejun to the 'cc list)

I finally found that we actually continue to run after the above
apparent 'hang'. That is, we continue to make progress updating the jump
labels. And doing a dump of all the system tasks at the time of the hang
showed the processes in various places besides the stop machine threads.
Thus, I thought that perhaps, for some reason the stop machine threads
weren't being scheduled.

Thus, I tried commenting out the special scheduling that is set up for
stop machine threads, and that fixed the hang. I haven't yet looked into
what might be going wrong with that scheduling...but maybe somebody else
knows...

thanks,

-Jason


diff --git a/kernel/stop_machine.c b/kernel/stop_machine.c
index 090c288..3013b85 100644
--- a/kernel/stop_machine.c
+++ b/kernel/stop_machine.c
@@ -307,7 +307,7 @@ static int __cpuinit cpu_stop_cpu_callback(struct notifier_block *nfb,
return NOTIFY_BAD;
get_task_struct(p);
kthread_bind(p, cpu);
- sched_set_stop_task(cpu, p);
+ //sched_set_stop_task(cpu, p);
stopper->thread = p;
break;

@@ -326,7 +326,7 @@ static int __cpuinit cpu_stop_cpu_callback(struct notifier_block *nfb,
{
struct cpu_stop_work *work;

- sched_set_stop_task(cpu, NULL);
+ //sched_set_stop_task(cpu, NULL);
/* kill the stopper */
kthread_stop(stopper->thread);
/* drain remaining works */
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/