Re: soft lockup - CPU#1 stuck for 15s! [swapper:0]

From: Parag Warudkar
Date: Mon Dec 10 2007 - 23:49:58 EST

Next message: Eric Dumazet: "[PATCH] RCU : move three variables to __read_mostly to save space"
Previous message: Richard Knutsson: "[PATCH 6/6] pcmcia/pcnet_cs: Fix 'shadow variable' warning"
In reply to: Thomas Gleixner: "Re: soft lockup - CPU#1 stuck for 15s! [swapper:0]"
Next in thread: Len Brown: "Re: soft lockup - CPU#1 stuck for 15s! [swapper:0]"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

On Dec 10, 2007 12:49 AM, Thomas Gleixner <tglx@xxxxxxxxxxxxx> wrote:
> Can you please apply the patch below ? It prints out the internal
> state of the clockevents/timer system when the softlockup is detected.
>

Below is the relevant output after multiple soft lockups.

Thanks
Parag

BUG: soft lockup - CPU#1 stuck for 14s! [squid_redirect:3109]

Pid: 3109, comm: squid_redirect Not tainted (2.6.24-rc4 #15)
EIP: 0073:[<0077a362>] EFLAGS: 00000202 CPU: 1
EIP is at 0x77a362
EAX: 0804c008 EBX: 007ca5cc ECX: 08199947 EDX: 0846da74
ESI: 0846da74 EDI: 08199968 EBP: bfcc2d58 ESP: bfcc2d20
DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 007b
CR0: 80050033 CR2: b7ea2c84 CR3: 3721b000 CR4: 000006d0
DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
DR6: ffff0ff0 DR7: 00000400
=======================
Timer List Version: v0.3
HRTIMER_MAX_CLOCK_BASES: 2
now at 2899263521779 nsecs

cpu: 0
clock 0:
.index: 0
.resolution: 1 nsecs
.get_time: ktime_get_real
.offset: 1197343983635176004 nsecs
active timers:
clock 1:
.index: 1
.resolution: 1 nsecs
.get_time: ktime_get
.offset: 0 nsecs
active timers:
#0: <f6eb1e9c>, tick_sched_timer, S:01, tick_nohz_restart_sched_tick, swapper/0
# expires at 2899384000000 nsecs [in 120478221 nsecs]
#1: <f6eb1e9c>, hrtimer_wakeup, S:01, do_nanosleep, irqbalance/2663
# expires at 2902364827067 nsecs [in 3101305288 nsecs]
#2: <f6eb1e9c>, it_real_fn, S:01, do_setitimer, syslogd/2643
# expires at 2918995045613 nsecs [in 19731523834 nsecs]
#3: <f6eb1e9c>, hrtimer_wakeup, S:01, do_nanosleep, crond/3083
# expires at 2938118668932 nsecs [in 38855147153 nsecs]
#4: <f6eb1e9c>, hrtimer_wakeup, S:01, do_nanosleep, atd/3184
# expires at 3037524193866 nsecs [in 138260672087 nsecs]
#5: <f6eb1e9c>, hrtimer_wakeup, S:01, do_nanosleep, smartd/3313
# expires at 3645576708660 nsecs [in 746313186881 nsecs]
.expires_next : 2899500000000 nsecs
.hres_active : 1
.nr_events : 113175
.nohz_mode : 2
.idle_tick : 2899000000000 nsecs
.tick_stopped : 0
.idle_jiffies : 649750
.idle_calls : 848768
.idle_sleeps : 642128
.idle_entrytime : 2899580013124 nsecs
.idle_sleeptime : 2675153362437 nsecs
.last_jiffies : 649900
.next_jiffies : 649918
.idle_expires : 2899496000000 nsecs
jiffies: 649906

cpu: 1
clock 0:
.index: 0
.resolution: 1 nsecs
.get_time: ktime_get_real
.offset: 1197343983635176004 nsecs
active timers:
clock 1:
.index: 1
.resolution: 1 nsecs
.get_time: ktime_get
.offset: 0 nsecs
active timers:
#0: <f6eb1e9c>, it_real_fn, S:01, do_setitimer, sendmail/3036
# expires at 3634545118202 nsecs [in 735281596423 nsecs]
#1: <f6eb1e9c>, it_real_fn, S:01, do_setitimer, sendmail/3047
# expires at 3634633106605 nsecs [in 735369584826 nsecs]
.expires_next : 2899261000000 nsecs
.hres_active : 1
.nr_events : 51469
.nohz_mode : 2
.idle_tick : 2883861000000 nsecs
.tick_stopped : 0
.idle_jiffies : 645965
.idle_calls : 59946
.idle_sleeps : 21647
.idle_entrytime : 2883857038167 nsecs
.idle_sleeptime : 2760707498759 nsecs
.last_jiffies : 645965
.next_jiffies : 652750
.idle_expires : 2910996000000 nsecs
jiffies: 649967

Tick Device: mode: 1
Clock Event Device: hpet
max_delta_ns: 2147483647
min_delta_ns: 3352
mult: 61496110
shift: 32
mode: 3
next_event: 2899924000000 nsecs
set_next_event: hpet_legacy_next_event
set_mode: hpet_legacy_set_mode
event_handler: tick_handle_oneshot_broadcast
tick_broadcast_mask: 00000003
tick_broadcast_oneshot_mask: 00000001

Tick Device: mode: 1
Clock Event Device: lapic
max_delta_ns: 805384969
min_delta_ns: 1440
mult: 44734871
shift: 32
mode: 1
next_event: 2900012000000 nsecs
set_next_event: lapic_next_event
set_mode: lapic_timer_setup
event_handler: hrtimer_interrupt

Tick Device: mode: 1
Clock Event Device: lapic
max_delta_ns: 805384969
min_delta_ns: 1440
mult: 44734871
shift: 32
mode: 3
next_event: 9223372036854775807 nsecs
set_next_event: lapic_next_event
set_mode: lapic_timer_setup
event_handler: hrtimer_interrupt

BUG: soft lockup - CPU#1 stuck for 13s! [squid_redirect:3110]

Pid: 3110, comm: squid_redirect Not tainted (2.6.24-rc4 #15)
EIP: 0073:[<00725c40>] EFLAGS: 00000246 CPU: 1
EIP is at 0x725c40
EAX: 0829da00 EBX: 007ca5cc ECX: 080bb420 EDX: 00000000
ESI: 0804c008 EDI: 00000003 EBP: bfd26158 ESP: bfd2613c
DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 007b
CR0: 80050033 CR2: b7ea2c84 CR3: 37c06000 CR4: 000006d0
DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
DR6: ffff0ff0 DR7: 00000400
=======================
Timer List Version: v0.3
HRTIMER_MAX_CLOCK_BASES: 2
now at 2925319278498 nsecs

cpu: 0
clock 0:
.index: 0
.resolution: 1 nsecs
.get_time: ktime_get_real
.offset: 1197343983635176004 nsecs
active timers:
clock 1:
.index: 1
.resolution: 1 nsecs
.get_time: ktime_get
.offset: 0 nsecs
active timers:
#0: <f6eb5e9c>, tick_sched_timer, S:01, tick_nohz_restart_sched_tick, swapper/0
# expires at 2925436000000 nsecs [in 116721502 nsecs]
#1: <f6eb5e9c>, hrtimer_wakeup, S:01, do_nanosleep, irqbalance/2663
# expires at 2932364827220 nsecs [in 7045548722 nsecs]
#2: <f6eb5e9c>, hrtimer_wakeup, S:01, do_nanosleep, crond/3083
# expires at 2938118668932 nsecs [in 12799390434 nsecs]
#3: <f6eb5e9c>, it_real_fn, S:01, do_setitimer, syslogd/2643
# expires at 2948995090537 nsecs [in 23675812039 nsecs]
#4: <f6eb5e9c>, hrtimer_wakeup, S:01, do_nanosleep, atd/3184
# expires at 3037524193866 nsecs [in 112204915368 nsecs]
#5: <f6eb5e9c>, hrtimer_wakeup, S:01, do_nanosleep, smartd/3313
# expires at 3645576708660 nsecs [in 720257430162 nsecs]
.expires_next : 2925532000000 nsecs
.hres_active : 1
.nr_events : 113548
.nohz_mode : 2
.idle_tick : 2925000000000 nsecs
.tick_stopped : 0
.idle_jiffies : 656250
.idle_calls : 849830
.idle_sleeps : 642265
.idle_entrytime : 2925608012256 nsecs
.idle_sleeptime : 2700129054859 nsecs
.last_jiffies : 656311
.next_jiffies : 656321
.idle_expires : 2925496000000 nsecs
jiffies: 656311

cpu: 1
clock 0:
.index: 0
.resolution: 1 nsecs
.get_time: ktime_get_real
.offset: 1197343983635176004 nsecs
active timers:
clock 1:
.index: 1
.resolution: 1 nsecs
.get_time: ktime_get
.offset: 0 nsecs
active timers:
#0: <f6eb5e9c>, it_real_fn, S:01, do_setitimer, sendmail/3036
# expires at 3634545118202 nsecs [in 709225839704 nsecs]
#1: <f6eb5e9c>, it_real_fn, S:01, do_setitimer, sendmail/3047
# expires at 3634633106605 nsecs [in 709313828107 nsecs]
.expires_next : 2925241000000 nsecs
.hres_active : 1
.nr_events : 51474
.nohz_mode : 2
.idle_tick : 2911001000000 nsecs
.tick_stopped : 0
.idle_jiffies : 652750
.idle_calls : 59951
.idle_sleeps : 21649
.idle_entrytime : 2910997024596 nsecs
.idle_sleeptime : 2785834078493 nsecs
.last_jiffies : 652750
.next_jiffies : 660250
.idle_expires : 2940996000000 nsecs
jiffies: 656311

Tick Device: mode: 1
Clock Event Device: hpet
max_delta_ns: 2147483647
min_delta_ns: 3352
mult: 61496110
shift: 32
mode: 3
next_event: 2925952000000 nsecs
set_next_event: hpet_legacy_next_event
set_mode: hpet_legacy_set_mode
event_handler: tick_handle_oneshot_broadcast
tick_broadcast_mask: 00000003
tick_broadcast_oneshot_mask: 00000001

Tick Device: mode: 1
Clock Event Device: lapic
max_delta_ns: 805384969
min_delta_ns: 1440
mult: 44734871
shift: 32
mode: 1
next_event: 2926044000000 nsecs
set_next_event: lapic_next_event
set_mode: lapic_timer_setup
event_handler: hrtimer_interrupt

Tick Device: mode: 1
Clock Event Device: lapic
max_delta_ns: 805384969
min_delta_ns: 1440
mult: 44734871
shift: 32
mode: 3
next_event: 9223372036854775807 nsecs
set_next_event: lapic_next_event
set_mode: lapic_timer_setup
event_handler: hrtimer_interrupt

BUG: soft lockup - CPU#1 stuck for 107s! [swapper:0]

Pid: 0, comm: swapper Not tainted (2.6.24-rc4 #15)
EIP: 0060:[<c0533d06>] EFLAGS: 00000202 CPU: 1
EIP is at acpi_idle_enter_simple+0x166/0x1d0
EAX: f7829f88 EBX: 00000dab ECX: 00000266 EDX: 00000000
ESI: 00000000 EDI: 00bcbed8 EBP: 00bcb12d ESP: f7829f88
DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
CR0: 8005003b CR2: 086e902c CR3: 3720a000 CR4: 000006d0
DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
DR6: ffff0ff0 DR7: 00000400
[<c058cf03>] cpuidle_idle_call+0x53/0x78
[<c058ceb0>] cpuidle_idle_call+0x0/0x78
[<c0402575>] cpu_idle+0x97/0xb8
=======================
Timer List Version: v0.3
HRTIMER_MAX_CLOCK_BASES: 2
now at 3175981092730 nsecs

cpu: 0
clock 0:
.index: 0
.resolution: 1 nsecs
.get_time: ktime_get_real
.offset: 1197343983635176004 nsecs
active timers:
clock 1:
.index: 1
.resolution: 1 nsecs
.get_time: ktime_get
.offset: 0 nsecs
active timers:
#0: <f7829e34>, tick_sched_timer, S:01, tick_nohz_stop_sched_tick, swapper/0
# expires at 3175984000000 nsecs [in 2907270 nsecs]
#1: <f7829e34>, hrtimer_wakeup, S:01, do_nanosleep, crond/3083
# expires at 3178119140931 nsecs [in 2138048201 nsecs]
#2: <f7829e34>, hrtimer_wakeup, S:01, do_nanosleep, irqbalance/2663
# expires at 3182364827430 nsecs [in 6383734700 nsecs]
#3: <f7829e34>, it_real_fn, S:01, do_setitimer, syslogd/2643
# expires at 3188995455070 nsecs [in 13014362340 nsecs]
#4: <f7829e34>, hrtimer_wakeup, S:01, do_nanosleep, atd/3184
# expires at 3337524287270 nsecs [in 161543194540 nsecs]
#5: <f7829e34>, hrtimer_wakeup, S:01, do_nanosleep, smartd/3313
# expires at 3645576708660 nsecs [in 469595615930 nsecs]
.expires_next : 3175984000000 nsecs
.hres_active : 1
.nr_events : 116977
.nohz_mode : 2
.idle_tick : 3175872000000 nsecs
.tick_stopped : 0
.idle_jiffies : 718969
.idle_calls : 982424
.idle_sleeps : 768351
.idle_entrytime : 3175868036812 nsecs
.idle_sleeptime : 2942681121008 nsecs
.last_jiffies : 718968
.next_jiffies : 718995
.idle_expires : 3175976000000 nsecs
jiffies: 718996

cpu: 1
clock 0:
.index: 0
.resolution: 1 nsecs
.get_time: ktime_get_real
.offset: 1197343983635176004 nsecs
active timers:
clock 1:
.index: 1
.resolution: 1 nsecs
.get_time: ktime_get
.offset: 0 nsecs
active timers:
#0: <f7829e34>, it_real_fn, S:01, do_setitimer, sendmail/3036
# expires at 3634545118202 nsecs [in 458564025472 nsecs]
#1: <f7829e34>, it_real_fn, S:01, do_setitimer, sendmail/3047
# expires at 3634633106605 nsecs [in 458652013875 nsecs]
.expires_next : 3175981000000 nsecs
.hres_active : 1
.nr_events : 52243
.nohz_mode : 2
.idle_tick : 3061841000000 nsecs
.tick_stopped : 0
.idle_jiffies : 690460
.idle_calls : 61325
.idle_sleeps : 22249
.idle_entrytime : 3175980020722 nsecs
.idle_sleeptime : 3034270703089 nsecs
.last_jiffies : 718996
.next_jiffies : 732728
.idle_expires : 3230908000000 nsecs
jiffies: 718996

Tick Device: mode: 1
Clock Event Device: hpet
max_delta_ns: 2147483647
min_delta_ns: 3352
mult: 61496110
shift: 32
mode: 3
next_event: 9223372036854775807 nsecs
set_next_event: hpet_legacy_next_event
set_mode: hpet_legacy_set_mode
event_handler: tick_handle_oneshot_broadcast
tick_broadcast_mask: 00000003
tick_broadcast_oneshot_mask: 00000002

Tick Device: mode: 1
Clock Event Device: lapic
max_delta_ns: 805384969
min_delta_ns: 1440
mult: 44734871
shift: 32
mode: 3
next_event: 3175984000000 nsecs
set_next_event: lapic_next_event
set_mode: lapic_timer_setup
event_handler: hrtimer_interrupt

Tick Device: mode: 1
Clock Event Device: lapic
max_delta_ns: 805384969
min_delta_ns: 1440
mult: 44734871
shift: 32
mode: 1
next_event: 9223372036854775807 nsecs
set_next_event: lapic_next_event
set_mode: lapic_timer_setup
event_handler: hrtimer_interrupt

BUG: soft lockup - CPU#1 stuck for 13s! [swapper:0]

Pid: 0, comm: swapper Not tainted (2.6.24-rc4 #15)
EIP: 0060:[<c0533d06>] EFLAGS: 00000206 CPU: 1
EIP is at acpi_idle_enter_simple+0x166/0x1d0
EAX: f7829f88 EBX: 00000da2 ECX: 00000266 EDX: 00000000
ESI: 00000000 EDI: 009232c2 EBP: 00922520 ESP: f7829f88
DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
CR0: 8005003b CR2: 086e902c CR3: 37890000 CR4: 000006d0
DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
DR6: ffff0ff0 DR7: 00000400
[<c058cf03>] cpuidle_idle_call+0x53/0x78
[<c058ceb0>] cpuidle_idle_call+0x0/0x78
[<c0402575>] cpu_idle+0x97/0xb8
=======================
Timer List Version: v0.3
HRTIMER_MAX_CLOCK_BASES: 2
now at 3315910720388 nsecs

cpu: 0
clock 0:
.index: 0
.resolution: 1 nsecs
.get_time: ktime_get_real
.offset: 1197343983635176004 nsecs
active timers:
clock 1:
.index: 1
.resolution: 1 nsecs
.get_time: ktime_get
.offset: 0 nsecs
active timers:
#0: <f7829e34>, tick_sched_timer, S:01, tick_nohz_restart_sched_tick, swapper/0
# expires at 3316024000000 nsecs [in 113279612 nsecs]
#1: <f7829e34>, hrtimer_wakeup, S:01, do_nanosleep, irqbalance/2663
# expires at 3322364826957 nsecs [in 6454106569 nsecs]
#2: <f7829e34>, hrtimer_wakeup, S:01, do_nanosleep, atd/3184
# expires at 3337524287270 nsecs [in 21613566882 nsecs]
#3: <f7829e34>, it_real_fn, S:01, do_setitimer, syslogd/2643
# expires at 3338995684982 nsecs [in 23084964594 nsecs]
#4: <f7829e34>, hrtimer_wakeup, S:01, do_nanosleep, crond/3083
# expires at 3358119467677 nsecs [in 42208747289 nsecs]
#5: <f7829e34>, hrtimer_wakeup, S:01, do_nanosleep, smartd/3313
# expires at 3645576708660 nsecs [in 329665988272 nsecs]
.expires_next : 3316124000000 nsecs
.hres_active : 1
.nr_events : 118223
.nohz_mode : 2
.idle_tick : 3315500000000 nsecs
.tick_stopped : 0
.idle_jiffies : 753875
.idle_calls : 985645
.idle_sleeps : 770122
.idle_entrytime : 3316196013563 nsecs
.idle_sleeptime : 3079535676376 nsecs
.last_jiffies : 754054
.next_jiffies : 754085
.idle_expires : 3315996000000 nsecs
jiffies: 754060

cpu: 1
clock 0:
.index: 0
.resolution: 1 nsecs
.get_time: ktime_get_real
.offset: 1197343983635176004 nsecs
active timers:
clock 1:
.index: 1
.resolution: 1 nsecs
.get_time: ktime_get
.offset: 0 nsecs
active timers:
#0: <f7829e34>, it_real_fn, S:01, do_setitimer, sendmail/3036
# expires at 3634545118202 nsecs [in 318634397814 nsecs]
#1: <f7829e34>, it_real_fn, S:01, do_setitimer, sendmail/3047
# expires at 3634633106605 nsecs [in 318722386217 nsecs]
.expires_next : 3315813000000 nsecs
.hres_active : 1
.nr_events : 52555
.nohz_mode : 2
.idle_tick : 3302001000000 nsecs
.tick_stopped : 0
.idle_jiffies : 750500
.idle_calls : 61676
.idle_sleeps : 22325
.idle_entrytime : 3315812024367 nsecs
.idle_sleeptime : 3172329298971 nsecs
.last_jiffies : 753954
.next_jiffies : 825443
.idle_expires : 3601768000000 nsecs
jiffies: 754122

Tick Device: mode: 1
Clock Event Device: hpet
max_delta_ns: 2147483647
min_delta_ns: 3352
mult: 61496110
shift: 32
mode: 3
next_event: 3316544000000 nsecs
set_next_event: hpet_legacy_next_event
set_mode: hpet_legacy_set_mode
event_handler: tick_handle_oneshot_broadcast
tick_broadcast_mask: 00000003
tick_broadcast_oneshot_mask: 00000003

Tick Device: mode: 1
Clock Event Device: lapic
max_delta_ns: 805384969
min_delta_ns: 1440
mult: 44734871
shift: 32
mode: 1
next_event: 3316636000000 nsecs
set_next_event: lapic_next_event
set_mode: lapic_timer_setup
event_handler: hrtimer_interrupt

Tick Device: mode: 1
Clock Event Device: lapic
max_delta_ns: 805384969
min_delta_ns: 1440
mult: 44734871
shift: 32
mode: 1
next_event: 9223372036854775807 nsecs
set_next_event: lapic_next_event
set_mode: lapic_timer_setup
event_handler: hrtimer_interrupt

BUG: soft lockup - CPU#1 stuck for 17s! [swapper:0]

Pid: 0, comm: swapper Not tainted (2.6.24-rc4 #15)
EIP: 0060:[<c0533d06>] EFLAGS: 00000206 CPU: 1
EIP is at acpi_idle_enter_simple+0x166/0x1d0
EAX: f7829f88 EBX: 0000000a ECX: 00000266 EDX: 00000000
ESI: 00000000 EDI: 00855ac8 EBP: 00855abe ESP: f7829f88
DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
CR0: 8005003b CR2: 086e902c CR3: 3721b000 CR4: 000006d0
DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
DR6: ffff0ff0 DR7: 00000400
[<c058cf03>] cpuidle_idle_call+0x53/0x78
[<c058ceb0>] cpuidle_idle_call+0x0/0x78
[<c0402575>] cpu_idle+0x97/0xb8
=======================
Timer List Version: v0.3
HRTIMER_MAX_CLOCK_BASES: 2
now at 3339110116699 nsecs

cpu: 0
clock 0:
.index: 0
.resolution: 1 nsecs
.get_time: ktime_get_real
.offset: 1197343983635176004 nsecs
active timers:
clock 1:
.index: 1
.resolution: 1 nsecs
.get_time: ktime_get
.offset: 0 nsecs
active timers:
#0: <f7829e34>, tick_sched_timer, S:01, tick_nohz_restart_sched_tick, swapper/0
# expires at 3339224000000 nsecs [in 113883301 nsecs]
#1: <f7829e34>, hrtimer_wakeup, S:01, do_nanosleep, irqbalance/2663
# expires at 3342364827041 nsecs [in 3254710342 nsecs]
#2: <f7829e34>, hrtimer_wakeup, S:01, do_nanosleep, crond/3083
# expires at 3358119467677 nsecs [in 19009350978 nsecs]
#3: <f7829e34>, it_real_fn, S:01, do_setitimer, syslogd/2643
# expires at 3368995727615 nsecs [in 29885610916 nsecs]
#4: <f7829e34>, hrtimer_wakeup, S:01, do_nanosleep, atd/3184
# expires at 3637524360558 nsecs [in 298414243859 nsecs]
#5: <f7829e34>, hrtimer_wakeup, S:01, do_nanosleep, smartd/3313
# expires at 3645576708660 nsecs [in 306466591961 nsecs]
.expires_next : 3339320000000 nsecs
.hres_active : 1
.nr_events : 118638
.nohz_mode : 2
.idle_tick : 3339000000000 nsecs
.tick_stopped : 0
.idle_jiffies : 759750
.idle_calls : 986993
.idle_sleeps : 770348
.idle_entrytime : 3339396057527 nsecs
.idle_sleeptime : 3101499603762 nsecs
.last_jiffies : 759854
.next_jiffies : 759875
.idle_expires : 3339396000000 nsecs
jiffies: 759860

cpu: 1
clock 0:
.index: 0
.resolution: 1 nsecs
.get_time: ktime_get_real
.offset: 1197343983635176004 nsecs
active timers:
clock 1:
.index: 1
.resolution: 1 nsecs
.get_time: ktime_get
.offset: 0 nsecs
active timers:
#0: <f7829e34>, it_real_fn, S:01, do_setitimer, sendmail/3036
# expires at 3634545118202 nsecs [in 295435001503 nsecs]
#1: <f7829e34>, it_real_fn, S:01, do_setitimer, sendmail/3047
# expires at 3634633106605 nsecs [in 295522989906 nsecs]
.expires_next : 3339013000000 nsecs
.hres_active : 1
.nr_events : 52575
.nohz_mode : 2
.idle_tick : 3320845000000 nsecs
.tick_stopped : 0
.idle_jiffies : 755211
.idle_calls : 61702
.idle_sleeps : 22333
.idle_entrytime : 3339010504626 nsecs
.idle_sleeptime : 3194546573260 nsecs
.last_jiffies : 759753
.next_jiffies : 825443
.idle_expires : 3601768000000 nsecs
jiffies: 759921

Tick Device: mode: 1
Clock Event Device: hpet
max_delta_ns: 2147483647
min_delta_ns: 3352
mult: 61496110
shift: 32
mode: 3
next_event: 3339744000000 nsecs
set_next_event: hpet_legacy_next_event
set_mode: hpet_legacy_set_mode
event_handler: tick_handle_oneshot_broadcast
tick_broadcast_mask: 00000003
tick_broadcast_oneshot_mask: 00000003

Tick Device: mode: 1
Clock Event Device: lapic
max_delta_ns: 805384969
min_delta_ns: 1440
mult: 44734871
shift: 32
mode: 1
next_event: 3339832000000 nsecs
set_next_event: lapic_next_event
set_mode: lapic_timer_setup
event_handler: hrtimer_interrupt

Tick Device: mode: 1
Clock Event Device: lapic
max_delta_ns: 805384969
min_delta_ns: 1440
mult: 44734871
shift: 32
mode: 1
next_event: 9223372036854775807 nsecs
set_next_event: lapic_next_event
set_mode: lapic_timer_setup
event_handler: hrtimer_interrupt

BUG: soft lockup - CPU#1 stuck for 15s! [swapper:0]

Pid: 0, comm: swapper Not tainted (2.6.24-rc4 #15)
EIP: 0060:[<c0533d06>] EFLAGS: 00000206 CPU: 1
EIP is at acpi_idle_enter_simple+0x166/0x1d0
EAX: f7829f88 EBX: 00000da7 ECX: 00000266 EDX: 00000000
ESI: 00000000 EDI: 009dd0d4 EBP: 009dc32d ESP: f7829f88
DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
CR0: 8005003b CR2: 086e902c CR3: 3721b000 CR4: 000006d0
DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
DR6: ffff0ff0 DR7: 00000400
[<c058cf03>] cpuidle_idle_call+0x53/0x78
[<c058ceb0>] cpuidle_idle_call+0x0/0x78
[<c0402575>] cpu_idle+0x97/0xb8
=======================
Timer List Version: v0.3
HRTIMER_MAX_CLOCK_BASES: 2
now at 3358306003372 nsecs

cpu: 0
clock 0:
.index: 0
.resolution: 1 nsecs
.get_time: ktime_get_real
.offset: 1197343983635176004 nsecs
active timers:
clock 1:
.index: 1
.resolution: 1 nsecs
.get_time: ktime_get
.offset: 0 nsecs
active timers:
#0: <f7829e34>, tick_sched_timer, S:01, tick_nohz_restart_sched_tick, swapper/0
# expires at 3358420000000 nsecs [in 113996628 nsecs]
#1: <f7829e34>, hrtimer_wakeup, S:01, do_nanosleep, irqbalance/2663
# expires at 3362364827247 nsecs [in 4058823875 nsecs]
#2: <f7829e34>, it_real_fn, S:01, do_setitimer, syslogd/2643
# expires at 3368995727615 nsecs [in 10689724243 nsecs]
#3: <f7829e34>, hrtimer_wakeup, S:01, do_nanosleep, crond/3083
# expires at 3418119574321 nsecs [in 59813570949 nsecs]
#4: <f7829e34>, hrtimer_wakeup, S:01, do_nanosleep, atd/3184
# expires at 3637524360558 nsecs [in 279218357186 nsecs]
#5: <f7829e34>, hrtimer_wakeup, S:01, do_nanosleep, smartd/3313
# expires at 3645576708660 nsecs [in 287270705288 nsecs]
.expires_next : 3358516000000 nsecs
.hres_active : 1
.nr_events : 119037
.nohz_mode : 2
.idle_tick : 3358132000000 nsecs
.tick_stopped : 0
.idle_jiffies : 764533
.idle_calls : 988268
.idle_sleeps : 770593
.idle_entrytime : 3358592013279 nsecs
.idle_sleeptime : 3119453550713 nsecs
.last_jiffies : 764653
.next_jiffies : 764735
.idle_expires : 3358288000000 nsecs
jiffies: 764659

cpu: 1
clock 0:
.index: 0
.resolution: 1 nsecs
.get_time: ktime_get_real
.offset: 1197343983635176004 nsecs
active timers:
clock 1:
.index: 1
.resolution: 1 nsecs
.get_time: ktime_get
.offset: 0 nsecs
active timers:
#0: <f7829e34>, it_real_fn, S:01, do_setitimer, sendmail/3036
# expires at 3634545118202 nsecs [in 276239114830 nsecs]
#1: <f7829e34>, it_real_fn, S:01, do_setitimer, sendmail/3047
# expires at 3634633106605 nsecs [in 276327103233 nsecs]
.expires_next : 3358209000000 nsecs
.hres_active : 1
.nr_events : 52601
.nohz_mode : 2
.idle_tick : 3343153000000 nsecs
.tick_stopped : 0
.idle_jiffies : 760788
.idle_calls : 61735
.idle_sleeps : 22340
.idle_entrytime : 3358208022533 nsecs
.idle_sleeptime : 3212747374331 nsecs
.last_jiffies : 764553
.next_jiffies : 825443
.idle_expires : 3601768000000 nsecs
jiffies: 764720

Tick Device: mode: 1
Clock Event Device: hpet
max_delta_ns: 2147483647
min_delta_ns: 3352
mult: 61496110
shift: 32
mode: 3
next_event: 3358940000000 nsecs
set_next_event: hpet_legacy_next_event
set_mode: hpet_legacy_set_mode
event_handler: tick_handle_oneshot_broadcast
tick_broadcast_mask: 00000003
tick_broadcast_oneshot_mask: 00000003

Tick Device: mode: 1
Clock Event Device: lapic
max_delta_ns: 805384969
min_delta_ns: 1440
mult: 44734871
shift: 32
mode: 1
next_event: 3359028000000 nsecs
set_next_event: lapic_next_event
set_mode: lapic_timer_setup
event_handler: hrtimer_interrupt

Tick Device: mode: 1
Clock Event Device: lapic
max_delta_ns: 805384969
min_delta_ns: 1440
mult: 44734871
shift: 32
mode: 1
next_event: 9223372036854775807 nsecs
set_next_event: lapic_next_event
set_mode: lapic_timer_setup
event_handler: hrtimer_interrupt

BUG: soft lockup - CPU#1 stuck for 13s! [squid_redirect:3109]

Pid: 3109, comm: squid_redirect Not tainted (2.6.24-rc4 #15)
EIP: 0060:[<c05a0ca2>] EFLAGS: 00010246 CPU: 1
EIP is at __alloc_skb+0x5e/0xfb
EAX: 00000000 EBX: f6ebb300 ECX: 00000022 EDX: f7006800
ESI: 00000080 EDI: f6ebb304 EBP: 000004d0 ESP: f6eb1dd8
DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
CR0: 80050033 CR2: 086e902c CR3: 3721b000 CR4: 000006d0
DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
DR6: ffff0ff0 DR7: 00000400
[<c059d70b>] sock_alloc_send_skb+0x6e/0x196
[<c05fae2d>] unix_stream_sendmsg+0x15c/0x317
[<c059b0d5>] sock_aio_write+0xe3/0xef
[<c046ced1>] do_sync_write+0xc7/0x10a
[<c0430e29>] autoremove_wake_function+0x0/0x35
[<c0447e34>] audit_syscall_exit+0x2c7/0x2e3
[<c046d6aa>] vfs_write+0xb5/0x117
[<c046dc05>] sys_write+0x41/0x67
[<c0403e26>] syscall_call+0x7/0xb
=======================
Timer List Version: v0.3
HRTIMER_MAX_CLOCK_BASES: 2
now at 3378758554114 nsecs

cpu: 0
clock 0:
.index: 0
.resolution: 1 nsecs
.get_time: ktime_get_real
.offset: 1197343983635176004 nsecs
active timers:
clock 1:
.index: 1
.resolution: 1 nsecs
.get_time: ktime_get
.offset: 0 nsecs
active timers:
#0: <f6eb1c84>, tick_sched_timer, S:01, tick_nohz_restart_sched_tick, swapper/0
# expires at 3378880000000 nsecs [in 121445886 nsecs]
#1: <f6eb1c84>, hrtimer_wakeup, S:01, do_nanosleep, irqbalance/2663
# expires at 3382364827171 nsecs [in 3606273057 nsecs]
#2: <f6eb1c84>, it_real_fn, S:01, do_setitimer, syslogd/2643
# expires at 3398995770136 nsecs [in 20237216022 nsecs]
#3: <f6eb1c84>, hrtimer_wakeup, S:01, do_nanosleep, crond/3083
# expires at 3418119574321 nsecs [in 39361020207 nsecs]
#4: <f6eb1c84>, hrtimer_wakeup, S:01, do_nanosleep, atd/3184
# expires at 3637524360558 nsecs [in 258765806444 nsecs]
#5: <f6eb1c84>, hrtimer_wakeup, S:01, do_nanosleep, smartd/3313
# expires at 3645576708660 nsecs [in 266818154546 nsecs]
.expires_next : 3378980000000 nsecs
.hres_active : 1
.nr_events : 119459
.nohz_mode : 2
.idle_tick : 3378480000000 nsecs
.tick_stopped : 0
.idle_jiffies : 769621
.idle_calls : 989598
.idle_sleeps : 770848
.idle_entrytime : 3379048011858 nsecs
.idle_sleeptime : 3138622137644 nsecs
.last_jiffies : 769655
.next_jiffies : 769735
.idle_expires : 3378936000000 nsecs
jiffies: 769655

cpu: 1
clock 0:
.index: 0
.resolution: 1 nsecs
.get_time: ktime_get_real
.offset: 1197343983635176004 nsecs
active timers:
clock 1:
.index: 1
.resolution: 1 nsecs
.get_time: ktime_get
.offset: 0 nsecs
active timers:
#0: <f6eb1c84>, it_real_fn, S:01, do_setitimer, sendmail/3036
# expires at 3634545118202 nsecs [in 255786564088 nsecs]
#1: <f6eb1c84>, it_real_fn, S:01, do_setitimer, sendmail/3047
# expires at 3634633106605 nsecs [in 255874552491 nsecs]
.expires_next : 3378617000000 nsecs
.hres_active : 1
.nr_events : 52630
.nohz_mode : 2
.idle_tick : 3365009000000 nsecs
.tick_stopped : 0
.idle_jiffies : 766253
.idle_calls : 61772
.idle_sleeps : 22349
.idle_entrytime : 3365008034288 nsecs
.idle_sleeptime : 3232136157785 nsecs
.last_jiffies : 766253
.next_jiffies : 825443
.idle_expires : 3601768000000 nsecs
jiffies: 769655

Tick Device: mode: 1
Clock Event Device: hpet
max_delta_ns: 2147483647
min_delta_ns: 3352
mult: 61496110
shift: 32
mode: 3
next_event: 3379392000000 nsecs
set_next_event: hpet_legacy_next_event
set_mode: hpet_legacy_set_mode
event_handler: tick_handle_oneshot_broadcast
tick_broadcast_mask: 00000003
tick_broadcast_oneshot_mask: 00000001

Tick Device: mode: 1
Clock Event Device: lapic
max_delta_ns: 805384969
min_delta_ns: 1440
mult: 44734871
shift: 32
mode: 1
next_event: 3379480000000 nsecs
set_next_event: lapic_next_event
set_mode: lapic_timer_setup
event_handler: hrtimer_interrupt

Tick Device: mode: 1
Clock Event Device: lapic
max_delta_ns: 805384969
min_delta_ns: 1440
mult: 44734871
shift: 32
mode: 3
next_event: 9223372036854775807 nsecs
set_next_event: lapic_next_event
set_mode: lapic_timer_setup
event_handler: hrtimer_interrupt

BUG: soft lockup - CPU#1 stuck for 13s! [swapper:0]

Pid: 0, comm: swapper Not tainted (2.6.24-rc4 #15)
EIP: 0060:[<c0533d06>] EFLAGS: 00000206 CPU: 1
EIP is at acpi_idle_enter_simple+0x166/0x1d0
EAX: f7829f88 EBX: 000003d2 ECX: 00000266 EDX: 00000000
ESI: 00000000 EDI: 004d7480 EBP: 004d70ae ESP: f7829f88
DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
CR0: 8005003b CR2: b7cb8eb4 CR3: 3721b000 CR4: 000006d0
DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
DR6: ffff0ff0 DR7: 00000400
[<c058cf03>] cpuidle_idle_call+0x53/0x78
[<c058ceb0>] cpuidle_idle_call+0x0/0x78
[<c0402575>] cpu_idle+0x97/0xb8
=======================
Timer List Version: v0.3
HRTIMER_MAX_CLOCK_BASES: 2
now at 3399017378968 nsecs

cpu: 0
clock 0:
.index: 0
.resolution: 1 nsecs
.get_time: ktime_get_real
.offset: 1197343983635176004 nsecs
active timers:
clock 1:
.index: 1
.resolution: 1 nsecs
.get_time: ktime_get
.offset: 0 nsecs
active timers:
#0: <f7829e34>, tick_sched_timer, S:01, tick_nohz_restart_sched_tick, swapper/0
# expires at 3399132000000 nsecs [in 114621032 nsecs]
#1: <f7829e34>, hrtimer_wakeup, S:01, do_nanosleep, irqbalance/2663
# expires at 3402364827246 nsecs [in 3347448278 nsecs]
#2: <f7829e34>, hrtimer_wakeup, S:01, do_nanosleep, crond/3083
# expires at 3418119574321 nsecs [in 19102195353 nsecs]
#3: <f7829e34>, it_real_fn, S:01, do_setitimer, syslogd/2643
# expires at 3428995802046 nsecs [in 29978423078 nsecs]
#4: <f7829e34>, hrtimer_wakeup, S:01, do_nanosleep, atd/3184
# expires at 3637524360558 nsecs [in 238506981590 nsecs]
#5: <f7829e34>, hrtimer_wakeup, S:01, do_nanosleep, smartd/3313
# expires at 3645576708660 nsecs [in 246559329692 nsecs]
.expires_next : 3399228000000 nsecs
.hres_active : 1
.nr_events : 119857
.nohz_mode : 2
.idle_tick : 3398596000000 nsecs
.tick_stopped : 0
.idle_jiffies : 774649
.idle_calls : 990978
.idle_sleeps : 771180
.idle_entrytime : 3399300011523 nsecs
.idle_sleeptime : 3157671876745 nsecs
.last_jiffies : 774731
.next_jiffies : 774741
.idle_expires : 3398996000000 nsecs
jiffies: 774731

cpu: 1
clock 0:
.index: 0
.resolution: 1 nsecs
.get_time: ktime_get_real
.offset: 1197343983635176004 nsecs
active timers:
clock 1:
.index: 1
.resolution: 1 nsecs
.get_time: ktime_get
.offset: 0 nsecs
active timers:
#0: <f7829e34>, it_real_fn, S:01, do_setitimer, sendmail/3036
# expires at 3634545118202 nsecs [in 235527739234 nsecs]
#1: <f7829e34>, it_real_fn, S:01, do_setitimer, sendmail/3047
# expires at 3634633106605 nsecs [in 235615727637 nsecs]
.expires_next : 3398921000000 nsecs
.hres_active : 1
.nr_events : 52664
.nohz_mode : 2
.idle_tick : 3384993000000 nsecs
.tick_stopped : 0
.idle_jiffies : 771248
.idle_calls : 61816
.idle_sleeps : 22362
.idle_entrytime : 3398920723452 nsecs
.idle_sleeptime : 3251373016329 nsecs
.last_jiffies : 774731
.next_jiffies : 825443
.idle_expires : 3601768000000 nsecs
jiffies: 774731

Tick Device: mode: 1
Clock Event Device: hpet
max_delta_ns: 2147483647
min_delta_ns: 3352
mult: 61496110
shift: 32
mode: 3
next_event: 3399644000000 nsecs
set_next_event: hpet_legacy_next_event
set_mode: hpet_legacy_set_mode
event_handler: tick_handle_oneshot_broadcast
tick_broadcast_mask: 00000003
tick_broadcast_oneshot_mask: 00000003

Tick Device: mode: 1
Clock Event Device: lapic
max_delta_ns: 805384969
min_delta_ns: 1440
mult: 44734871
shift: 32
mode: 1
next_event: 3399736000000 nsecs
set_next_event: lapic_next_event
set_mode: lapic_timer_setup
event_handler: hrtimer_interrupt

Tick Device: mode: 1
Clock Event Device: lapic
max_delta_ns: 805384969
min_delta_ns: 1440
mult: 44734871
shift: 32
mode: 1
next_event: 9223372036854775807 nsecs
set_next_event: lapic_next_event
set_mode: lapic_timer_setup
event_handler: hrtimer_interrupt

BUG: soft lockup - CPU#1 stuck for 21s! [swapper:0]

Pid: 0, comm: swapper Not tainted (2.6.24-rc4 #15)
EIP: 0060:[<c0533d06>] EFLAGS: 00000206 CPU: 1
EIP is at acpi_idle_enter_simple+0x166/0x1d0
EAX: f7829f88 EBX: 0000026a ECX: 00000266 EDX: 00000000
ESI: 00000000 EDI: 000dc235 EBP: 000dbfcb ESP: f7829f88
DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
CR0: 8005003b CR2: b7cb8eb4 CR3: 3721b000 CR4: 000006d0
DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
DR6: ffff0ff0 DR7: 00000400
[<c058cf03>] cpuidle_idle_call+0x53/0x78
[<c058ceb0>] cpuidle_idle_call+0x0/0x78
[<c0402575>] cpu_idle+0x97/0xb8
=======================
Timer List Version: v0.3
HRTIMER_MAX_CLOCK_BASES: 2
now at 3425973316433 nsecs

cpu: 0
clock 0:
.index: 0
.resolution: 1 nsecs
.get_time: ktime_get_real
.offset: 1197343983635176004 nsecs
active timers:
clock 1:
.index: 1
.resolution: 1 nsecs
.get_time: ktime_get
.offset: 0 nsecs
active timers:
#0: <f7829e34>, tick_sched_timer, S:01, tick_nohz_restart_sched_tick, swapper/0
# expires at 3426088000000 nsecs [in 114683567 nsecs]
#1: <f7829e34>, it_real_fn, S:01, do_setitimer, syslogd/2643
# expires at 3428995802046 nsecs [in 3022485613 nsecs]
#2: <f7829e34>, hrtimer_wakeup, S:01, do_nanosleep, irqbalance/2663
# expires at 3432364826941 nsecs [in 6391510508 nsecs]
#3: <f7829e34>, hrtimer_wakeup, S:01, do_nanosleep, crond/3083
# expires at 3478119686256 nsecs [in 52146369823 nsecs]
#4: <f7829e34>, hrtimer_wakeup, S:01, do_nanosleep, atd/3184
# expires at 3637524360558 nsecs [in 211551044125 nsecs]
#5: <f7829e34>, hrtimer_wakeup, S:01, do_nanosleep, smartd/3313
# expires at 3645576708660 nsecs [in 219603392227 nsecs]
.expires_next : 3426184000000 nsecs
.hres_active : 1
.nr_events : 120297
.nohz_mode : 2
.idle_tick : 3425876000000 nsecs
.tick_stopped : 0
.idle_jiffies : 781469
.idle_calls : 992318
.idle_sleeps : 771457
.idle_entrytime : 3426256011783 nsecs
.idle_sleeptime : 3183364803024 nsecs
.last_jiffies : 781470
.next_jiffies : 781480
.idle_expires : 3425996000000 nsecs
jiffies: 781470

cpu: 1
clock 0:
.index: 0
.resolution: 1 nsecs
.get_time: ktime_get_real
.offset: 1197343983635176004 nsecs
active timers:
clock 1:
.index: 1
.resolution: 1 nsecs
.get_time: ktime_get
.offset: 0 nsecs
active timers:
#0: <f7829e34>, it_real_fn, S:01, do_setitimer, sendmail/3036
# expires at 3634545118202 nsecs [in 208571801769 nsecs]
#1: <f7829e34>, it_real_fn, S:01, do_setitimer, sendmail/3047
# expires at 3634633106605 nsecs [in 208659790172 nsecs]
.expires_next : 3425877000000 nsecs
.hres_active : 1
.nr_events : 52690
.nohz_mode : 2
.idle_tick : 3404129000000 nsecs
.tick_stopped : 0
.idle_jiffies : 776032
.idle_calls : 61850
.idle_sleeps : 22372
.idle_entrytime : 3425876824416 nsecs
.idle_sleeptime : 3277331921223 nsecs
.last_jiffies : 781470
.next_jiffies : 825443
.idle_expires : 3601768000000 nsecs
jiffies: 781470

Tick Device: mode: 1
Clock Event Device: hpet
max_delta_ns: 2147483647
min_delta_ns: 3352
mult: 61496110
shift: 32
mode: 3
next_event: 3426600000000 nsecs
set_next_event: hpet_legacy_next_event
set_mode: hpet_legacy_set_mode
event_handler: tick_handle_oneshot_broadcast
tick_broadcast_mask: 00000003
tick_broadcast_oneshot_mask: 00000003

Tick Device: mode: 1
Clock Event Device: lapic
max_delta_ns: 805384969
min_delta_ns: 1440
mult: 44734871
shift: 32
mode: 1
next_event: 3426692000000 nsecs
set_next_event: lapic_next_event
set_mode: lapic_timer_setup
event_handler: hrtimer_interrupt

Tick Device: mode: 1
Clock Event Device: lapic
max_delta_ns: 805384969
min_delta_ns: 1440
mult: 44734871
shift: 32
mode: 1
next_event: 9223372036854775807 nsecs
set_next_event: lapic_next_event
set_mode: lapic_timer_setup
event_handler: hrtimer_interrupt

BUG: soft lockup - CPU#1 stuck for 13s! [squid_redirect:3109]

Pid: 3109, comm: squid_redirect Not tainted (2.6.24-rc4 #15)
EIP: 0060:[<c0406798>] EFLAGS: 00000202 CPU: 1
EIP is at do_syscall_trace+0xfe/0x16a
EAX: f6de15c0 EBX: 00000000 ECX: f6de15c0 EDX: 00000000
ESI: 00000000 EDI: 00000000 EBP: f6eb1fb8 ESP: f6eb1f94
DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068
CR0: 80050033 CR2: b7cb8eb4 CR3: 3721b000 CR4: 000006d0
DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
DR6: ffff0ff0 DR7: 00000400
[<c046db9e>] sys_read+0x41/0x67
[<c0403f26>] syscall_exit_work+0x16/0x1b
=======================
Timer List Version: v0.3
HRTIMER_MAX_CLOCK_BASES: 2
now at 3445397166939 nsecs

cpu: 0
clock 0:
.index: 0
.resolution: 1 nsecs
.get_time: ktime_get_real
.offset: 1197343983635176004 nsecs
active timers:
clock 1:
.index: 1
.resolution: 1 nsecs
.get_time: ktime_get
.offset: 0 nsecs
active timers:
#0: <f6eb1e40>, tick_sched_timer, S:01, tick_nohz_restart_sched_tick, swapper/0
# expires at 3445516000000 nsecs [in 118833061 nsecs]
#1: <f6eb1e40>, hrtimer_wakeup, S:01, do_nanosleep, irqbalance/2663
# expires at 3452364827341 nsecs [in 6967660402 nsecs]
#2: <f6eb1e40>, it_real_fn, S:01, do_setitimer, syslogd/2643
# expires at 3458995868510 nsecs [in 13598701571 nsecs]
#3: <f6eb1e40>, hrtimer_wakeup, S:01, do_nanosleep, crond/3083
# expires at 3478119686256 nsecs [in 32722519317 nsecs]
#4: <f6eb1e40>, hrtimer_wakeup, S:01, do_nanosleep, atd/3184
# expires at 3637524360558 nsecs [in 192127193619 nsecs]
#5: <f6eb1e40>, hrtimer_wakeup, S:01, do_nanosleep, smartd/3313
# expires at 3645576708660 nsecs [in 200179541721 nsecs]
.expires_next : 3445616000000 nsecs
.hres_active : 1
.nr_events : 120695
.nohz_mode : 2
.idle_tick : 3445000000000 nsecs
.tick_stopped : 0
.idle_jiffies : 786250
.idle_calls : 993728
.idle_sleeps : 771801
.idle_entrytime : 3445688011834 nsecs
.idle_sleeptime : 3201567185186 nsecs
.last_jiffies : 786327
.next_jiffies : 786337
.idle_expires : 3445496000000 nsecs
jiffies: 786327

cpu: 1
clock 0:
.index: 0
.resolution: 1 nsecs
.get_time: ktime_get_real
.offset: 1197343983635176004 nsecs
active timers:
clock 1:
.index: 1
.resolution: 1 nsecs
.get_time: ktime_get
.offset: 0 nsecs
active timers:
#0: <f6eb1e40>, it_real_fn, S:01, do_setitimer, sendmail/3036
# expires at 3634545118202 nsecs [in 189147951263 nsecs]
#1: <f6eb1e40>, it_real_fn, S:01, do_setitimer, sendmail/3047
# expires at 3634633106605 nsecs [in 189235939666 nsecs]
.expires_next : 3445305000000 nsecs
.hres_active : 1
.nr_events : 52725
.nohz_mode : 2
.idle_tick : 3432009000000 nsecs
.tick_stopped : 0
.idle_jiffies : 783003
.idle_calls : 61890
.idle_sleeps : 22382
.idle_entrytime : 3432008033864 nsecs
.idle_sleeptime : 3295603622377 nsecs
.last_jiffies : 783003
.next_jiffies : 825443
.idle_expires : 3601768000000 nsecs
jiffies: 786327

Tick Device: mode: 1
Clock Event Device: hpet
max_delta_ns: 2147483647
min_delta_ns: 3352
mult: 61496110
shift: 32
mode: 3
next_event: 3446036000000 nsecs
set_next_event: hpet_legacy_next_event
set_mode: hpet_legacy_set_mode
event_handler: tick_handle_oneshot_broadcast
tick_broadcast_mask: 00000003
tick_broadcast_oneshot_mask: 00000001

Tick Device: mode: 1
Clock Event Device: lapic
max_delta_ns: 805384969
min_delta_ns: 1440
mult: 44734871
shift: 32
mode: 1
next_event: 3446124000000 nsecs
set_next_event: lapic_next_event
set_mode: lapic_timer_setup
event_handler: hrtimer_interrupt

Tick Device: mode: 1
Clock Event Device: lapic
max_delta_ns: 805384969
min_delta_ns: 1440
mult: 44734871
shift: 32
mode: 3
next_event: 9223372036854775807 nsecs
set_next_event: lapic_next_event
set_mode: lapic_timer_setup
event_handler: hrtimer_interrupt

BUG: soft lockup - CPU#1 stuck for 11s! [swapper:0]

Pid: 0, comm: swapper Not tainted (2.6.24-rc4 #15)
EIP: 0060:[<c0533d06>] EFLAGS: 00000206 CPU: 1
EIP is at acpi_idle_enter_simple+0x166/0x1d0
EAX: f7829f88 EBX: 00000da6 ECX: 00000266 EDX: 00000000
ESI: 00000000 EDI: 00804711 EBP: 0080396b ESP: f7829f88
DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
CR0: 8005003b CR2: b7cb8eb4 CR3: 3721b000 CR4: 000006d0
DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
DR6: ffff0ff0 DR7: 00000400
[<c058cf03>] cpuidle_idle_call+0x53/0x78
[<c058ceb0>] cpuidle_idle_call+0x0/0x78
[<c0402575>] cpu_idle+0x97/0xb8
=======================
Timer List Version: v0.3
HRTIMER_MAX_CLOCK_BASES: 2
now at 3470253369529 nsecs

cpu: 0
clock 0:
.index: 0
.resolution: 1 nsecs
.get_time: ktime_get_real
.offset: 1197343983635176004 nsecs
active timers:
clock 1:
.index: 1
.resolution: 1 nsecs
.get_time: ktime_get
.offset: 0 nsecs
active timers:
#0: <f7829e34>, tick_sched_timer, S:01, tick_nohz_restart_sched_tick, swapper/0
# expires at 3470368000000 nsecs [in 114630471 nsecs]
#1: <f7829e34>, hrtimer_wakeup, S:01, do_nanosleep, irqbalance/2663
# expires at 3472364827179 nsecs [in 2111457650 nsecs]
#2: <f7829e34>, hrtimer_wakeup, S:01, do_nanosleep, crond/3083
# expires at 3478119686256 nsecs [in 7866316727 nsecs]
#3: <f7829e34>, it_real_fn, S:01, do_setitimer, syslogd/2643
# expires at 3488995915727 nsecs [in 18742546198 nsecs]
#4: <f7829e34>, hrtimer_wakeup, S:01, do_nanosleep, atd/3184
# expires at 3637524360558 nsecs [in 167270991029 nsecs]
#5: <f7829e34>, hrtimer_wakeup, S:01, do_nanosleep, smartd/3313
# expires at 3645576708660 nsecs [in 175323339131 nsecs]
.expires_next : 3470464000000 nsecs
.hres_active : 1
.nr_events : 121145
.nohz_mode : 2
.idle_tick : 3470000000000 nsecs
.tick_stopped : 0
.idle_jiffies : 792500
.idle_calls : 995079
.idle_sleeps : 772053
.idle_entrytime : 3470532015086 nsecs
.idle_sleeptime : 3225097811941 nsecs
.last_jiffies : 792639
.next_jiffies : 792643
.idle_expires : 3470496000000 nsecs
jiffies: 792645

cpu: 1
clock 0:
.index: 0
.resolution: 1 nsecs
.get_time: ktime_get_real
.offset: 1197343983635176004 nsecs
active timers:
clock 1:
.index: 1
.resolution: 1 nsecs
.get_time: ktime_get
.offset: 0 nsecs
active timers:
#0: <f7829e34>, it_real_fn, S:01, do_setitimer, sendmail/3036
# expires at 3634545118202 nsecs [in 164291748673 nsecs]
#1: <f7829e34>, it_real_fn, S:01, do_setitimer, sendmail/3047
# expires at 3634633106605 nsecs [in 164379737076 nsecs]
.expires_next : 3470157000000 nsecs
.hres_active : 1
.nr_events : 52762
.nohz_mode : 2
.idle_tick : 3458613000000 nsecs
.tick_stopped : 0
.idle_jiffies : 789653
.idle_calls : 61939
.idle_sleeps : 22394
.idle_entrytime : 3470156022499 nsecs
.idle_sleeptime : 3319306712683 nsecs
.last_jiffies : 792540
.next_jiffies : 825443
.idle_expires : 3601768000000 nsecs
jiffies: 792706

Tick Device: mode: 1
Clock Event Device: hpet
max_delta_ns: 2147483647
min_delta_ns: 3352
mult: 61496110
shift: 32
mode: 3
next_event: 3470880000000 nsecs
set_next_event: hpet_legacy_next_event
set_mode: hpet_legacy_set_mode
event_handler: tick_handle_oneshot_broadcast
tick_broadcast_mask: 00000003
tick_broadcast_oneshot_mask: 00000003

Tick Device: mode: 1
Clock Event Device: lapic
max_delta_ns: 805384969
min_delta_ns: 1440
mult: 44734871
shift: 32
mode: 1
next_event: 3470968000000 nsecs
set_next_event: lapic_next_event
set_mode: lapic_timer_setup
event_handler: hrtimer_interrupt

Tick Device: mode: 1
Clock Event Device: lapic
max_delta_ns: 805384969
min_delta_ns: 1440
mult: 44734871
shift: 32
mode: 1
next_event: 9223372036854775807 nsecs
set_next_event: lapic_next_event
set_mode: lapic_timer_setup
event_handler: hrtimer_interrupt

BUG: soft lockup - CPU#1 stuck for 13s! [swapper:0]

Pid: 0, comm: swapper Not tainted (2.6.24-rc4 #15)
EIP: 0060:[<c0603eaa>] EFLAGS: 00000202 CPU: 1
EIP is at _spin_lock_irqsave+0x16/0x27
EAX: c06b4110 EBX: 00000001 ECX: f7873808 EDX: 00000293
ESI: 00000005 EDI: f7873808 EBP: 00000000 ESP: f7829f10
DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
CR0: 8005003b CR2: b7cb8eb4 CR3: 37dcc000 CR4: 000006d0
DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
DR6: ffff0ff0 DR7: 00000400
[<c0438293>] tick_broadcast_oneshot_control+0x10/0xda
[<c0437ce2>] tick_notify+0x1d4/0x2eb
[<c04281bc>] get_next_timer_interrupt+0x143/0x1b4
[<c06058a1>] notifier_call_chain+0x2a/0x47
[<c04345c0>] raw_notifier_call_chain+0x17/0x1a
[<c043781e>] clockevents_notify+0x19/0x4f
[<c0533d23>] acpi_idle_enter_simple+0x183/0x1d0
[<c058cf03>] cpuidle_idle_call+0x53/0x78
[<c058ceb0>] cpuidle_idle_call+0x0/0x78
[<c0402575>] cpu_idle+0x97/0xb8
=======================
Timer List Version: v0.3
HRTIMER_MAX_CLOCK_BASES: 2
now at 3489148567147 nsecs

cpu: 0
clock 0:
.index: 0
.resolution: 1 nsecs
.get_time: ktime_get_real
.offset: 1197343983635176004 nsecs
active timers:
clock 1:
.index: 1
.resolution: 1 nsecs
.get_time: ktime_get
.offset: 0 nsecs
active timers:
#0: <f7829dbc>, tick_sched_timer, S:01, tick_nohz_stop_sched_tick, swapper/0
# expires at 3489004000000 nsecs [in 18446744073564984469 nsecs]
#1: <f7829dbc>, hrtimer_wakeup, S:01, do_nanosleep, irqbalance/2663
# expires at 3492364827517 nsecs [in 3216260370 nsecs]
#2: <f7829dbc>, hrtimer_wakeup, S:01, do_nanosleep, crond/3083
# expires at 3538119802465 nsecs [in 48971235318 nsecs]
#3: <f7829dbc>, hrtimer_wakeup, S:01, do_nanosleep, atd/3184
# expires at 3637524360558 nsecs [in 148375793411 nsecs]
#4: <f7829dbc>, hrtimer_wakeup, S:01, do_nanosleep, smartd/3313
# expires at 3645576708660 nsecs [in 156428141513 nsecs]
.expires_next : 3489004000000 nsecs
.hres_active : 1
.nr_events : 121552
.nohz_mode : 2
.idle_tick : 3488500000000 nsecs
.tick_stopped : 0
.idle_jiffies : 797126
.idle_calls : 996547
.idle_sleeps : 772606
.idle_entrytime : 3489000016669 nsecs
.idle_sleeptime : 3242430478734 nsecs
.last_jiffies : 797251
.next_jiffies : 797375
.idle_expires : 3488996000000 nsecs
jiffies: 797251

cpu: 1
clock 0:
.index: 0
.resolution: 1 nsecs
.get_time: ktime_get_real
.offset: 1197343983635176004 nsecs
active timers:
clock 1:
.index: 1
.resolution: 1 nsecs
.get_time: ktime_get
.offset: 0 nsecs
active timers:
#0: <f7829dbc>, it_real_fn, S:01, do_setitimer, syslogd/2643
# expires at 3518995976435 nsecs [in 29847409288 nsecs]
#1: <f7829dbc>, it_real_fn, S:01, do_setitimer, sendmail/3036
# expires at 3634545118202 nsecs [in 145396551055 nsecs]
#2: <f7829dbc>, it_real_fn, S:01, do_setitimer, sendmail/3047
# expires at 3634633106605 nsecs [in 145484539458 nsecs]
.expires_next : 3488997000000 nsecs
.hres_active : 1
.nr_events : 52842
.nohz_mode : 2
.idle_tick : 3474001000000 nsecs
.tick_stopped : 0
.idle_jiffies : 793501
.idle_calls : 62051
.idle_sleeps : 22430
.idle_entrytime : 3488995988657 nsecs
.idle_sleeptime : 3336928804329 nsecs
.last_jiffies : 797249
.next_jiffies : 825443
.idle_expires : 3601768000000 nsecs
jiffies: 797251

Tick Device: mode: 1
Clock Event Device: hpet
max_delta_ns: 2147483647
min_delta_ns: 3352
mult: 61496110
shift: 32
mode: 3
next_event: 9223372036854775807 nsecs
set_next_event: hpet_legacy_next_event
set_mode: hpet_legacy_set_mode
event_handler: tick_handle_oneshot_broadcast
tick_broadcast_mask: 00000003
tick_broadcast_oneshot_mask: 00000003

Tick Device: mode: 1
Clock Event Device: lapic
max_delta_ns: 805384969
min_delta_ns: 1440
mult: 44734871
shift: 32
mode: 1
next_event: 3489004000000 nsecs
set_next_event: lapic_next_event
set_mode: lapic_timer_setup
event_handler: hrtimer_interrupt

Tick Device: mode: 1
Clock Event Device: lapic
max_delta_ns: 805384969
min_delta_ns: 1440
mult: 44734871
shift: 32
mode: 1
next_event: 9223372036854775807 nsecs
set_next_event: lapic_next_event
set_mode: lapic_timer_setup
event_handler: hrtimer_interrupt

BUG: soft lockup - CPU#1 stuck for 18s! [squid_redirect:3109]

Pid: 3109, comm: squid_redirect Not tainted (2.6.24-rc4 #15)
EIP: 0073:[<00778e26>] EFLAGS: 00000286 CPU: 1
EIP is at 0x778e26
EAX: 00000000 EBX: 007ca5cc ECX: 0819993b EDX: 00000000
ESI: 082eafb0 EDI: 08199963 EBP: bfcc2d18 ESP: bfcc2ca0
DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 007b
CR0: 80050033 CR2: b7cb8eb4 CR3: 3721b000 CR4: 000006d0
DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
DR6: ffff0ff0 DR7: 00000400
=======================
Timer List Version: v0.3
HRTIMER_MAX_CLOCK_BASES: 2
now at 3509959588108 nsecs

cpu: 0
clock 0:
.index: 0
.resolution: 1 nsecs
.get_time: ktime_get_real
.offset: 1197343983635176004 nsecs
active timers:
clock 1:
.index: 1
.resolution: 1 nsecs
.get_time: ktime_get
.offset: 0 nsecs
active timers:
#0: <f6eb1e9c>, tick_sched_timer, S:01, tick_nohz_restart_sched_tick, swapper/0
# expires at 3510076000000 nsecs [in 116411892 nsecs]
#1: <f6eb1e9c>, hrtimer_wakeup, S:01, do_nanosleep, irqbalance/2663
# expires at 3512364827593 nsecs [in 2405239485 nsecs]
#2: <f6eb1e9c>, hrtimer_wakeup, S:01, do_nanosleep, crond/3083
# expires at 3538119802465 nsecs [in 28160214357 nsecs]
#3: <f6eb1e9c>, hrtimer_wakeup, S:01, do_nanosleep, atd/3184
# expires at 3637524360558 nsecs [in 127564772450 nsecs]
#4: <f6eb1e9c>, hrtimer_wakeup, S:01, do_nanosleep, smartd/3313
# expires at 3645576708660 nsecs [in 135617120552 nsecs]
.expires_next : 3510156000000 nsecs
.hres_active : 1
.nr_events : 121736
.nohz_mode : 2
.idle_tick : 3509500000000 nsecs
.tick_stopped : 0
.idle_jiffies : 802375
.idle_calls : 997613
.idle_sleeps : 773233
.idle_entrytime : 3510228013028 nsecs
.idle_sleeptime : 3262173789757 nsecs
.last_jiffies : 802562
.next_jiffies : 802590
.idle_expires : 3509976000000 nsecs
jiffies: 802568

cpu: 1
clock 0:
.index: 0
.resolution: 1 nsecs
.get_time: ktime_get_real
.offset: 1197343983635176004 nsecs
active timers:
clock 1:
.index: 1
.resolution: 1 nsecs
.get_time: ktime_get
.offset: 0 nsecs
active timers:
#0: <f6eb1e9c>, it_real_fn, S:01, do_setitimer, syslogd/2643
# expires at 3518995976435 nsecs [in 9036388327 nsecs]
#1: <f6eb1e9c>, it_real_fn, S:01, do_setitimer, sendmail/3036
# expires at 3634545118202 nsecs [in 124585530094 nsecs]
#2: <f6eb1e9c>, it_real_fn, S:01, do_setitimer, sendmail/3047
# expires at 3634633106605 nsecs [in 124673518497 nsecs]
.expires_next : 3509881000000 nsecs
.hres_active : 1
.nr_events : 52891
.nohz_mode : 2
.idle_tick : 3490217000000 nsecs
.tick_stopped : 0
.idle_jiffies : 797554
.idle_calls : 62239
.idle_sleeps : 22472
.idle_entrytime : 3509880022423 nsecs
.idle_sleeptime : 3356666831101 nsecs
.last_jiffies : 802471
.next_jiffies : 825443
.idle_expires : 3601768000000 nsecs
jiffies: 802632

Tick Device: mode: 1
Clock Event Device: hpet
max_delta_ns: 2147483647
min_delta_ns: 3352
mult: 61496110
shift: 32
mode: 3
next_event: 3510588000000 nsecs
set_next_event: hpet_legacy_next_event
set_mode: hpet_legacy_set_mode
event_handler: tick_handle_oneshot_broadcast
tick_broadcast_mask: 00000003
tick_broadcast_oneshot_mask: 00000001

Tick Device: mode: 1
Clock Event Device: lapic
max_delta_ns: 805384969
min_delta_ns: 1440
mult: 44734871
shift: 32
mode: 1
next_event: 3510680000000 nsecs
set_next_event: lapic_next_event
set_mode: lapic_timer_setup
event_handler: hrtimer_interrupt

Tick Device: mode: 1
Clock Event Device: lapic
max_delta_ns: 805384969
min_delta_ns: 1440
mult: 44734871
shift: 32
mode: 3
next_event: 9223372036854775807 nsecs
set_next_event: lapic_next_event
set_mode: lapic_timer_setup
event_handler: hrtimer_interrupt

BUG: soft lockup - CPU#1 stuck for 73s! [swapper:0]

Pid: 0, comm: swapper Not tainted (2.6.24-rc4 #15)
EIP: 0060:[<c0533d06>] EFLAGS: 00000206 CPU: 1
EIP is at acpi_idle_enter_simple+0x166/0x1d0
EAX: f7829f88 EBX: 00000009 ECX: 00000266 EDX: 00000000
ESI: 00000000 EDI: 00cd838d EBP: 00cd8384 ESP: f7829f88
DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
CR0: 8005003b CR2: b7cb8eb4 CR3: 3721b000 CR4: 000006d0
DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
DR6: ffff0ff0 DR7: 00000400
[<c058cf03>] cpuidle_idle_call+0x53/0x78
[<c058ceb0>] cpuidle_idle_call+0x0/0x78
[<c0402575>] cpu_idle+0x97/0xb8
=======================
Timer List Version: v0.3
HRTIMER_MAX_CLOCK_BASES: 2
now at 3598217317189 nsecs

cpu: 0
clock 0:
.index: 0
.resolution: 1 nsecs
.get_time: ktime_get_real
.offset: 1197343983635176004 nsecs
active timers:
clock 1:
.index: 1
.resolution: 1 nsecs
.get_time: ktime_get
.offset: 0 nsecs
active timers:
#0: <f7829e34>, tick_sched_timer, S:01, tick_nohz_restart_sched_tick, swapper/0
# expires at 3598332000000 nsecs [in 114682811 nsecs]
#1: <f7829e34>, hrtimer_wakeup, S:01, do_nanosleep, irqbalance/2663
# expires at 3602364827289 nsecs [in 4147510100 nsecs]
#2: <f7829e34>, it_real_fn, S:01, do_setitimer, syslogd/2643
# expires at 3608996157924 nsecs [in 10778840735 nsecs]
#3: <f7829e34>, hrtimer_wakeup, S:01, do_nanosleep, atd/3184
# expires at 3637524360558 nsecs [in 39307043369 nsecs]
#4: <f7829e34>, hrtimer_wakeup, S:01, do_nanosleep, smartd/3313
# expires at 3645576708660 nsecs [in 47359391471 nsecs]
#5: <f7829e34>, hrtimer_wakeup, S:01, do_nanosleep, crond/3083
# expires at 3658120033497 nsecs [in 59902716308 nsecs]
.expires_next : 3598428000000 nsecs
.hres_active : 1
.nr_events : 122506
.nohz_mode : 2
.idle_tick : 3598000000000 nsecs
.tick_stopped : 0
.idle_jiffies : 824500
.idle_calls : 999222
.idle_sleeps : 773694
.idle_entrytime : 3598504013419 nsecs
.idle_sleeptime : 3348824164217 nsecs
.last_jiffies : 824631
.next_jiffies : 824750
.idle_expires : 3598320000000 nsecs
jiffies: 824636

cpu: 1
clock 0:
.index: 0
.resolution: 1 nsecs
.get_time: ktime_get_real
.offset: 1197343983635176004 nsecs
active timers:
clock 1:
.index: 1
.resolution: 1 nsecs
.get_time: ktime_get
.offset: 0 nsecs
active timers:
#0: <f7829e34>, it_real_fn, S:01, do_setitimer, sendmail/3036
# expires at 3634545118202 nsecs [in 36327801013 nsecs]
#1: <f7829e34>, it_real_fn, S:01, do_setitimer, sendmail/3047
# expires at 3634633106605 nsecs [in 36415789416 nsecs]
.expires_next : 3598121000000 nsecs
.hres_active : 1
.nr_events : 52922
.nohz_mode : 2
.idle_tick : 3519005000000 nsecs
.tick_stopped : 0
.idle_jiffies : 804751
.idle_calls : 62274
.idle_sleeps : 22480
.idle_entrytime : 3598120023719 nsecs
.idle_sleeptime : 3443870779570 nsecs
.last_jiffies : 824531
.next_jiffies : 825443
.idle_expires : 3601768000000 nsecs
jiffies: 824698

Tick Device: mode: 1
Clock Event Device: hpet
max_delta_ns: 2147483647
min_delta_ns: 3352
mult: 61496110
shift: 32
mode: 3
next_event: 3598848000000 nsecs
set_next_event: hpet_legacy_next_event
set_mode: hpet_legacy_set_mode
event_handler: tick_handle_oneshot_broadcast
tick_broadcast_mask: 00000003
tick_broadcast_oneshot_mask: 00000003

Tick Device: mode: 1
Clock Event Device: lapic
max_delta_ns: 805384969
min_delta_ns: 1440
mult: 44734871
shift: 32
mode: 1
next_event: 3598940000000 nsecs
set_next_event: lapic_next_event
set_mode: lapic_timer_setup
event_handler: hrtimer_interrupt

Tick Device: mode: 1
Clock Event Device: lapic
max_delta_ns: 805384969
min_delta_ns: 1440
mult: 44734871
shift: 32
mode: 1
next_event: 9223372036854775807 nsecs
set_next_event: lapic_next_event
set_mode: lapic_timer_setup
event_handler: hrtimer_interrupt

BUG: soft lockup - CPU#1 stuck for 16s! [swapper:0]

Pid: 0, comm: swapper Not tainted (2.6.24-rc4 #15)
EIP: 0060:[<c0603eaa>] EFLAGS: 00000202 CPU: 1
EIP is at _spin_lock_irqsave+0x16/0x27
EAX: c06b4110 EBX: 00000001 ECX: f7873808 EDX: 00000293
ESI: 00000005 EDI: f7873808 EBP: 00000000 ESP: f7829f10
DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
CR0: 8005003b CR2: bfef28a8 CR3: 37370000 CR4: 000006d0
DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
DR6: ffff0ff0 DR7: 00000400
[<c0438293>] tick_broadcast_oneshot_control+0x10/0xda
[<c0437ce2>] tick_notify+0x1d4/0x2eb
[<c04281bc>] get_next_timer_interrupt+0x143/0x1b4
[<c06058a1>] notifier_call_chain+0x2a/0x47
[<c04345c0>] raw_notifier_call_chain+0x17/0x1a
[<c043781e>] clockevents_notify+0x19/0x4f
[<c0533d23>] acpi_idle_enter_simple+0x183/0x1d0
[<c058cf03>] cpuidle_idle_call+0x53/0x78
[<c058ceb0>] cpuidle_idle_call+0x0/0x78
[<c0402575>] cpu_idle+0x97/0xb8
=======================
Timer List Version: v0.3
HRTIMER_MAX_CLOCK_BASES: 2
now at 4167069767337 nsecs

cpu: 0
clock 0:
.index: 0
.resolution: 1 nsecs
.get_time: ktime_get_real
.offset: 1197343983635176004 nsecs
active timers:
clock 1:
.index: 1
.resolution: 1 nsecs
.get_time: ktime_get
.offset: 0 nsecs
active timers:
#0: <f7829dbc>, tick_sched_timer, S:01, tick_nohz_restart_sched_tick, swapper/0
# expires at 4166924000000 nsecs [in 18446744073563784279 nsecs]
#1: <f7829dbc>, hrtimer_wakeup, S:01, do_nanosleep, irqbalance/2663
# expires at 4172364827191 nsecs [in 5295059854 nsecs]
#2: <f7829dbc>, it_real_fn, S:01, do_setitimer, syslogd/2643
# expires at 4178996866281 nsecs [in 11927098944 nsecs]
#3: <f7829dbc>, hrtimer_wakeup, S:01, do_nanosleep, crond/3083
# expires at 4198120966920 nsecs [in 31051199583 nsecs]
#4: <f7829dbc>, hrtimer_wakeup, S:01, do_nanosleep, atd/3184
# expires at 4237524532033 nsecs [in 70454764696 nsecs]
#5: <f7829dbc>, hrtimer_wakeup, S:01, do_nanosleep, smartd/3313
# expires at 5445738202858 nsecs [in 1278668435521 nsecs]
.expires_next : 4166924000000 nsecs
.hres_active : 1
.nr_events : 125855
.nohz_mode : 2
.idle_tick : 4166872000000 nsecs
.tick_stopped : 0
.idle_jiffies : 966718
.idle_calls : 1003972
.idle_sleeps : 777081
.idle_entrytime : 4166921107361 nsecs
.idle_sleeptime : 3914047603252 nsecs
.last_jiffies : 966731
.next_jiffies : 966740
.idle_expires : 4166996000000 nsecs
jiffies: 966731

cpu: 1
clock 0:
.index: 0
.resolution: 1 nsecs
.get_time: ktime_get_real
.offset: 1197343983635176004 nsecs
active timers:
clock 1:
.index: 1
.resolution: 1 nsecs
.get_time: ktime_get
.offset: 0 nsecs
active timers:
#0: <f7829dbc>, it_real_fn, S:01, do_setitimer, sendmail/3036
# expires at 7234546066766 nsecs [in 3067476299429 nsecs]
#1: <f7829dbc>, it_real_fn, S:01, do_setitimer, sendmail/3047
# expires at 7234633977840 nsecs [in 3067564210503 nsecs]
.expires_next : 4166921000000 nsecs
.hres_active : 1
.nr_events : 53499
.nohz_mode : 2
.idle_tick : 4150253000000 nsecs
.tick_stopped : 0
.idle_jiffies : 962563
.idle_calls : 62865
.idle_sleeps : 22602
.idle_entrytime : 4166918657364 nsecs
.idle_sleeptime : 4009849868806 nsecs
.last_jiffies : 966730
.next_jiffies : 970000
.idle_expires : 4179996000000 nsecs
jiffies: 966731

Tick Device: mode: 1
Clock Event Device: hpet
max_delta_ns: 2147483647
min_delta_ns: 3352
mult: 61496110
shift: 32
mode: 3
next_event: 9223372036854775807 nsecs
set_next_event: hpet_legacy_next_event
set_mode: hpet_legacy_set_mode
event_handler: tick_handle_oneshot_broadcast
tick_broadcast_mask: 00000003
tick_broadcast_oneshot_mask: 00000003

Tick Device: mode: 1
Clock Event Device: lapic
max_delta_ns: 805384969
min_delta_ns: 1440
mult: 44734871
shift: 32
mode: 1
next_event: 4166924000000 nsecs
set_next_event: lapic_next_event
set_mode: lapic_timer_setup
event_handler: hrtimer_interrupt

Tick Device: mode: 1
Clock Event Device: lapic
max_delta_ns: 805384969
min_delta_ns: 1440
mult: 44734871
shift: 32
mode: 1
next_event: 9223372036854775807 nsecs
set_next_event: lapic_next_event
set_mode: lapic_timer_setup
event_handler: hrtimer_interrupt

BUG: soft lockup - CPU#1 stuck for 20s! [swapper:0]

Pid: 0, comm: swapper Not tainted (2.6.24-rc4 #15)
EIP: 0060:[<c0533d06>] EFLAGS: 00000202 CPU: 1
EIP is at acpi_idle_enter_simple+0x166/0x1d0
EAX: f7829f88 EBX: 0000000a ECX: 00000266 EDX: 00000000
ESI: 00000000 EDI: 0053a918 EBP: 0053a90e ESP: f7829f88
DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
CR0: 8005003b CR2: 080ff4a8 CR3: 36f0e000 CR4: 000006d0
DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
DR6: ffff0ff0 DR7: 00000400
[<c058cf03>] cpuidle_idle_call+0x53/0x78
[<c058ceb0>] cpuidle_idle_call+0x0/0x78
[<c0402575>] cpu_idle+0x97/0xb8
=======================
Timer List Version: v0.3
HRTIMER_MAX_CLOCK_BASES: 2
now at 4261546668428 nsecs

cpu: 0
clock 0:
.index: 0
.resolution: 1 nsecs
.get_time: ktime_get_real
.offset: 1197343983635176004 nsecs
active timers:
clock 1:
.index: 1
.resolution: 1 nsecs
.get_time: ktime_get
.offset: 0 nsecs
active timers:
#0: <f7829e34>, tick_sched_timer, S:01, tick_nohz_restart_sched_tick, swapper/0
# expires at 4261660000000 nsecs [in 113331572 nsecs]
#1: <f7829e34>, hrtimer_wakeup, S:01, do_nanosleep, irqbalance/2663
# expires at 4262364827121 nsecs [in 818158693 nsecs]
#2: <f7829e34>, it_real_fn, S:01, do_setitimer, syslogd/2643
# expires at 4268996983467 nsecs [in 7450315039 nsecs]
#3: <f7829e34>, hrtimer_wakeup, S:01, do_nanosleep, crond/3083
# expires at 4318121181735 nsecs [in 56574513307 nsecs]
#4: <f7829e34>, hrtimer_wakeup, S:01, do_nanosleep, atd/3184
# expires at 4537524611720 nsecs [in 275977943292 nsecs]
#5: <f7829e34>, hrtimer_wakeup, S:01, do_nanosleep, smartd/3313
# expires at 5445738202858 nsecs [in 1184191534430 nsecs]
.expires_next : 4261756000000 nsecs
.hres_active : 1
.nr_events : 126606
.nohz_mode : 2
.idle_tick : 4261356000000 nsecs
.tick_stopped : 0
.idle_jiffies : 990339
.idle_calls : 1007304
.idle_sleeps : 779634
.idle_entrytime : 4261828014556 nsecs
.idle_sleeptime : 4006637937696 nsecs
.last_jiffies : 990462
.next_jiffies : 990500
.idle_expires : 4261496000000 nsecs
jiffies: 990468

cpu: 1
clock 0:
.index: 0
.resolution: 1 nsecs
.get_time: ktime_get_real
.offset: 1197343983635176004 nsecs
active timers:
clock 1:
.index: 1
.resolution: 1 nsecs
.get_time: ktime_get
.offset: 0 nsecs
active timers:
#0: <f7829e34>, it_real_fn, S:01, do_setitimer, sendmail/3036
# expires at 7234546066766 nsecs [in 2972999398338 nsecs]
#1: <f7829e34>, it_real_fn, S:01, do_setitimer, sendmail/3047
# expires at 7234633977840 nsecs [in 2973087309412 nsecs]
.expires_next : 4261449000000 nsecs
.hres_active : 1
.nr_events : 53573
.nohz_mode : 2
.idle_tick : 4240017000000 nsecs
.tick_stopped : 0
.idle_jiffies : 985004
.idle_calls : 62997
.idle_sleeps : 22668
.idle_entrytime : 4261446283566 nsecs
.idle_sleeptime : 4103195796783 nsecs
.last_jiffies : 990362
.next_jiffies : 992500
.idle_expires : 4269996000000 nsecs
jiffies: 990529

Tick Device: mode: 1
Clock Event Device: hpet
max_delta_ns: 2147483647
min_delta_ns: 3352
mult: 61496110
shift: 32
mode: 3
next_event: 4262172000000 nsecs
set_next_event: hpet_legacy_next_event
set_mode: hpet_legacy_set_mode
event_handler: tick_handle_oneshot_broadcast
tick_broadcast_mask: 00000003
tick_broadcast_oneshot_mask: 00000003

Tick Device: mode: 1
Clock Event Device: lapic
max_delta_ns: 805384969
min_delta_ns: 1440
mult: 44734871
shift: 32
mode: 1
next_event: 4262264000000 nsecs
set_next_event: lapic_next_event
set_mode: lapic_timer_setup
event_handler: hrtimer_interrupt

Tick Device: mode: 1
Clock Event Device: lapic
max_delta_ns: 805384969
min_delta_ns: 1440
mult: 44734871
shift: 32
mode: 1
next_event: 9223372036854775807 nsecs
set_next_event: lapic_next_event
set_mode: lapic_timer_setup
event_handler: hrtimer_interrupt

BUG: soft lockup - CPU#1 stuck for 11s! [swapper:0]

Pid: 0, comm: swapper Not tainted (2.6.24-rc4 #15)
EIP: 0060:[<c0603eaa>] EFLAGS: 00000202 CPU: 1
EIP is at _spin_lock_irqsave+0x16/0x27
EAX: c06b4110 EBX: 00000001 ECX: f7873808 EDX: 00000293
ESI: 00000005 EDI: f7873808 EBP: 00000000 ESP: f7829f10
DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
CR0: 8005003b CR2: 080ff4a8 CR3: 37344000 CR4: 000006d0
DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
DR6: ffff0ff0 DR7: 00000400
[<c0438293>] tick_broadcast_oneshot_control+0x10/0xda
[<c0437ce2>] tick_notify+0x1d4/0x2eb
[<c04281bc>] get_next_timer_interrupt+0x143/0x1b4
[<c06058a1>] notifier_call_chain+0x2a/0x47
[<c04345c0>] raw_notifier_call_chain+0x17/0x1a
[<c043781e>] clockevents_notify+0x19/0x4f
[<c0533d23>] acpi_idle_enter_simple+0x183/0x1d0
[<c058cf03>] cpuidle_idle_call+0x53/0x78
[<c058ceb0>] cpuidle_idle_call+0x0/0x78
[<c0402575>] cpu_idle+0x97/0xb8
=======================
Timer List Version: v0.3
HRTIMER_MAX_CLOCK_BASES: 2
now at 4281836818905 nsecs

cpu: 0
clock 0:
.index: 0
.resolution: 1 nsecs
.get_time: ktime_get_real
.offset: 1197343983635176004 nsecs
active timers:
clock 1:
.index: 1
.resolution: 1 nsecs
.get_time: ktime_get
.offset: 0 nsecs
active timers:
#0: <f7829dbc>, tick_sched_timer, S:01, tick_nohz_restart_sched_tick, swapper/0
# expires at 4281692000000 nsecs [in 18446744073564732711 nsecs]
#1: <f7829dbc>, hrtimer_wakeup, S:01, do_nanosleep, irqbalance/2663
# expires at 4282364827815 nsecs [in 528008910 nsecs]
#2: <f7829dbc>, it_real_fn, S:01, do_setitimer, syslogd/2643
# expires at 4298997022577 nsecs [in 17160203672 nsecs]
#3: <f7829dbc>, hrtimer_wakeup, S:01, do_nanosleep, crond/3083
# expires at 4318121181735 nsecs [in 36284362830 nsecs]
#4: <f7829dbc>, hrtimer_wakeup, S:01, do_nanosleep, atd/3184
# expires at 4537524611720 nsecs [in 255687792815 nsecs]
#5: <f7829dbc>, hrtimer_wakeup, S:01, do_nanosleep, smartd/3313
# expires at 5445738202858 nsecs [in 1163901383953 nsecs]
.expires_next : 4281692000000 nsecs
.hres_active : 1
.nr_events : 126862
.nohz_mode : 2
.idle_tick : 4281500000000 nsecs
.tick_stopped : 0
.idle_jiffies : 995375
.idle_calls : 1007974
.idle_sleeps : 779749
.idle_entrytime : 4281689136364 nsecs
.idle_sleeptime : 4025848101273 nsecs
.last_jiffies : 995423
.next_jiffies : 995438
.idle_expires : 4281996000000 nsecs
jiffies: 995423

cpu: 1
clock 0:
.index: 0
.resolution: 1 nsecs
.get_time: ktime_get_real
.offset: 1197343983635176004 nsecs
active timers:
clock 1:
.index: 1
.resolution: 1 nsecs
.get_time: ktime_get
.offset: 0 nsecs
active timers:
#0: <f7829dbc>, it_real_fn, S:01, do_setitimer, sendmail/3036
# expires at 7234546066766 nsecs [in 2952709247861 nsecs]
#1: <f7829dbc>, it_real_fn, S:01, do_setitimer, sendmail/3047
# expires at 7234633977840 nsecs [in 2952797158935 nsecs]
.expires_next : 4281689000000 nsecs
.hres_active : 1
.nr_events : 53582
.nohz_mode : 2
.idle_tick : 4270017000000 nsecs
.tick_stopped : 0
.idle_jiffies : 992504
.idle_calls : 63006
.idle_sleeps : 22670
.idle_entrytime : 4281688022031 nsecs
.idle_sleeptime : 4122505836637 nsecs
.last_jiffies : 995423
.next_jiffies : 1000000
.idle_expires : 4299996000000 nsecs
jiffies: 995423

Tick Device: mode: 1
Clock Event Device: hpet
max_delta_ns: 2147483647
min_delta_ns: 3352
mult: 61496110
shift: 32
mode: 3
next_event: 9223372036854775807 nsecs
set_next_event: hpet_legacy_next_event
set_mode: hpet_legacy_set_mode
event_handler: tick_handle_oneshot_broadcast
tick_broadcast_mask: 00000003
tick_broadcast_oneshot_mask: 00000003

Tick Device: mode: 1
Clock Event Device: lapic
max_delta_ns: 805384969
min_delta_ns: 1440
mult: 44734871
shift: 32
mode: 1
next_event: 4281692000000 nsecs
set_next_event: lapic_next_event
set_mode: lapic_timer_setup
event_handler: hrtimer_interrupt

Tick Device: mode: 1
Clock Event Device: lapic
max_delta_ns: 805384969
min_delta_ns: 1440
mult: 44734871
shift: 32
mode: 1
next_event: 9223372036854775807 nsecs
set_next_event: lapic_next_event
set_mode: lapic_timer_setup
event_handler: hrtimer_interrupt
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Next message: Eric Dumazet: "[PATCH] RCU : move three variables to __read_mostly to save space"
Previous message: Richard Knutsson: "[PATCH 6/6] pcmcia/pcnet_cs: Fix 'shadow variable' warning"
In reply to: Thomas Gleixner: "Re: soft lockup - CPU#1 stuck for 15s! [swapper:0]"
Next in thread: Len Brown: "Re: soft lockup - CPU#1 stuck for 15s! [swapper:0]"
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]