[regression] cpuidle_get_cpu_driver livelocks idle system

From: Russ Anderson
Date: Mon Dec 17 2012 - 14:36:02 EST

The 3.7 kernel grinds to a halt on boot of a system with
2048 cpus. NMI showed most of the cpus in
_raw_spin_lock in cpuidle_get_cpu_driver(). (backtrace below)

A quick look at cpuidle_get_cpu_driver() shows the hot lock.

In drivers/cpuidle/driver.c:
* cpuidle_get_cpu_driver - return the driver tied with a cpu
struct cpuidle_driver *cpuidle_get_cpu_driver(struct cpuidle_device *dev)
struct cpuidle_driver *drv;

if (!dev)
return NULL;

drv = __cpuidle_get_cpu_driver(dev->cpu);

return drv;

This change was added in on Nov 14th, 2012.

The patch says it adds support for cpus with different characteristics,
but adds a big global lock. The comment claims "no impact for the other
platforms if the option is disabled", which leads me to believe the
spin_lock was added inadvertently. CPU_IDLE_MULTIPLE_DRIVERS is off
in my config file.

linux$ grep CPU_IDLE_MULTIPLE_DRIVERS .config

As more cpus become idle, more cpus fight over the lock until
the system livelocks on the crushing weight of idle.

The fix may be to move the spin_lock into __cpuidle_get_cpu_driver,
which has different versions for CONFIG_CPU_IDLE_MULTIPLE_DRIVERS,
to avoid impacting the disabled case, or get rid of the spin_lock
all together.

== UV NMI process trace cpu 12: ==
CPU 12
Pid: 0, comm: swapper/12 Tainted: G O 3.7.0.rja-sgi+ #38
RIP: 0010:[<ffffffff81614e45>] [<ffffffff81614e45>] _raw_spin_lock+0x25/0x30
Call Trace:
[<ffffffff814c891c>] cpuidle_get_cpu_driver+0x1c/0x30
[<ffffffff814c871d>] cpuidle_idle_call+0x7d/0x1b0
[<ffffffff8101d08d>] cpu_idle+0xdd/0x130
[<ffffffff8160a3ea>] start_secondary+0xc6/0xcc

Russ Anderson, OS RAS/Partitioning Project Lead
SGI - Silicon Graphics Inc rja@xxxxxxx
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/