Re: [PATCHv3 09/10] clocksource: time-armada-370-xp: Divorce fromlocal timer API

From: Gregory CLEMENT
Date: Wed Mar 20 2013 - 13:21:36 EST

On 03/20/2013 06:09 PM, Gregory CLEMENT wrote:
> On 03/13/2013 07:17 PM, Stephen Boyd wrote:
>> Separate the armada 370xp local timers from the local timer API.
>> This will allow us to remove ARM local timer support in the near
>> future and makes this driver multi-architecture friendly.
>
> At first view the code looks good, but when I applied your patch set on
> linux-next, build it and run it on a Armada XP based board (AX3 with 2 cores),
> it crashed:
>
> Booting Linux on physical CPU 0x0
> Linux version 3.9.0-rc3-next-20130319-00010-g728b448 (gclement@FE-greg-laptop) (gcc version 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5) ) #153 SMP Wed Mar 20 17:58:57 CET 2013
> CPU: ARMv7 Processor [562f5842] revision 2 (ARMv7), cr=10c53c7d
> CPU: PIPT / VIPT nonaliasing data cache, PIPT instruction cache
> Machine: Marvell Armada 370/XP (Device Tree), model: PlatHome OpenBlocks AX3-4 board
> bootconsole [earlycon0] enabled
> Memory policy: ECC disabled, Data cache writealloc
> PERCPU: Embedded 7 pages/cpu @c22b0000 s7104 r8192 d13376 u32768
> Built 1 zonelists in Zone order, mobility grouping on. Total pages: 784912
> Kernel command line: console=ttyS0,115200 earlyprintk
> PID hash table entries: 4096 (order: 2, 16384 bytes)
> Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
> Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
> __ex_table already sorted, skipping sort
> Memory: 3072MB = 3072MB total
> Memory: 3113520k/3113520k available, 32208k reserved, 2367488K highmem
> Virtual kernel memory layout:
> vector : 0xffff0000 - 0xffff1000 ( 4 kB)
> fixmap : 0xfff00000 - 0xfffe0000 ( 896 kB)
> vmalloc : 0xf0000000 - 0xff000000 ( 240 MB)
> lowmem : 0xc0000000 - 0xef800000 ( 760 MB)
> pkmap : 0xbfe00000 - 0xc0000000 ( 2 MB)
> modules : 0xbf000000 - 0xbfe00000 ( 14 MB)
> .text : 0xc0008000 - 0xc041647c (4154 kB)
> .init : 0xc0417000 - 0xc0633bc0 (2163 kB)
> .data : 0xc0634000 - 0xc06605c0 ( 178 kB)
> .bss : 0xc06605c0 - 0xc0686a2c ( 154 kB)
> Hierarchical RCU implementation.
> RCU restricting CPUs from NR_CPUS=4 to nr_cpu_ids=2.
> NR_IRQS:16 nr_irqs:16 16
> Aurora cache controller enabled
> l2x0: 16 ways, CACHE_ID 0x00000100, AUX_CTRL 0x1a696b12, Cache size: 1048576 B
> sched_clock: 32 bits at 25MHz, resolution 40ns, wraps every 171798ms
> Console: colour dummy device 80x30
> Calibrating delay loop...
> Internal error: Oops - undefined instruction: 0 [#1] SMP ARM
> Modules linked in:
> CPU: 0 Not tainted (3.9.0-rc3-next-20130319-00010-g728b448 #153)
> PC is at 0xe92d45f0
> LR is at armada_370_xp_timer_interrupt+0x3c/0x4c
> pc : [<e92d45f0>] lr : [<c023c2bc>] psr: 600001d3
> sp : c0635eb8 ip : 00000000 fp : c063c3f0
> r10: 000003ff r9 : 00000000 r8 : 00000010
> r7 : c22b3c40 r6 : ef007c00 r5 : c0640fcc r4 : c0053e30
> r3 : e92d45f0 r2 : fffffffe r1 : c22b3c40 r0 : c0053e30
> Flags: nZCv IRQs off FIQs off Mode SVC_32 ISA ARM Segment kernel
> Control: 10c53c7d Table: 0000406a DAC: 00000015
> Process swapper/0 (pid: 0, stack limit = 0xc0634238)
> Stack: (0xc0635eb8 to 0xc0636000)
> 5ea0: ef004c80 c0063224
> 5ec0: 00000010 00000010 00000000 c0660ac0 c0635f18 c005fcb8 c0632b90 c000ed94
> 5ee0: c0313c60 60000153 00000001 c00085a8 c0313c54 c0313c60 60000153 ffffffff
> 5f00: c0635f4c 00000000 562f5842 c06360c0 00000000 c000db60 0000001a ffff8ad0
> 5f20: ffff8ad0 c06360c0 00000000 00000000 c04379c0 c22ad780 00000000 562f5842
> 5f40: c06360c0 00000000 60000153 c0635f60 c0313c54 c0313c60 60000153 ffffffff
> 5f60: 00000021 00000000 00000003 00000004 0000006e c065fcc0 c067924c c063ceb8
> 5f80: c063cc84 c006d8c0 00000005 c065fcc0 c067924c c0421764 c22ad780 c063c42c
> 5fa0: 562f5842 c063cca8 c06605c0 c04379c0 c22ad780 00000000 562f5842 00000000
> 5fc0: 00000000 c0417754 ffffffff ffffffff c04172dc 00000000 00000000 c04379c0
> 5fe0: 10c53c7d c063c414 c04379bc c063febc 0000406a 00008074 00000000 00000000
> [<c023c2bc>] (armada_370_xp_timer_interrupt+0x3c/0x4c) from [<c0063224>] (handle_percpu_devid_irq+0x64/0x80)
> [<c0063224>] (handle_percpu_devid_irq+0x64/0x80) from [<c005fcb8>] (generic_handle_irq+0x20/0x30)
> [<c005fcb8>] (generic_handle_irq+0x20/0x30) from [<c000ed94>] (handle_IRQ+0x38/0x90)
> [<c000ed94>] (handle_IRQ+0x38/0x90) from [<c00085a8>] (armada_370_xp_handle_irq+0xa4/0xb0)
> [<c00085a8>] (armada_370_xp_handle_irq+0xa4/0xb0) from [<c000db60>] (__irq_svc+0x40/0x50)
> Exception stack(0xc0635f18 to 0xc0635f60)
> 5f00: 0000001a ffff8ad0
> 5f20: ffff8ad0 c06360c0 00000000 00000000 c04379c0 c22ad780 00000000 562f5842
> 5f40: c06360c0 00000000 60000153 c0635f60 c0313c54 c0313c60 60000153 ffffffff
> [<c000db60>] (__irq_svc+0x40/0x50) from [<c0313c60>] (calibrate_delay+0x378/0x528)
> [<c0313c60>] (calibrate_delay+0x378/0x528) from [<c0417754>] (start_kernel+0x250/0x2dc)
> [<c0417754>] (start_kernel+0x250/0x2dc) from [<00008074>] (0x8074)
> Code: 1fe7deb7 cd5772dd fff5692e ed55f79e (7ed5a5f7)
> ---[ end trace 1b75b31a2719ed1c ]---
> Kernel panic - not syncing: Fatal exception in interrupt
> Internal error: Oops - undefined instruction: 0 [#2] SMP ARM
> Modules linked in:
> CPU: 0 Tainted: G D (3.9.0-rc3-next-20130319-00010-g728b448 #153)
> PC is at 0xe92d45f0
> LR is at armada_370_xp_timer_interrupt+0x3c/0x4c
> pc : [<e92d45f0>] lr : [<c023c2bc>] psr: 600001d3
> sp : c0635cc0 ip : 00000000 fp : c063c3f0
> r10: 000003ff r9 : 00000000 r8 : 00000010
> r7 : c22b3c40 r6 : ef007c00 r5 : c0640fcc r4 : c0053e30
> r3 : e92d45f0 r2 : fffffffe r1 : c22b3c40 r0 : c0053e30
> Flags: nZCv IRQs off FIQs off Mode SVC_32 ISA ARM Segment kernel
> Control: 10c53c7d Table: 0000406a DAC: 00000015
> Process swapper/0 (pid: 0, stack limit = 0xc0634238)
> Stack: (0xc0635cc0 to 0xc0636000)
> 5cc0: ef004c80 c0063224 00000010 00000010 c0635f18 c0660ac0 c0635d20 c005fcb8
> 5ce0: c0632b90 c000ed94 c0316d54 60000153 00000001 c00085a8 c0316ca4 c0316d54
> 5d00: 60000153 ffffffff c0635d54 c03a53b0 00000000 c0660b70 600001d3 c000db60
> 5d20: c0660fc0 00000000 c0642f80 00000000 00000000 00000000 0000000b c06400e0
> 5d40: c03a53b0 00000000 c0660b70 600001d3 00000000 c0635d68 c0316ca4 c0316d54
> 5d60: 60000153 ffffffff 600001d3 c0635d94 c06608c0 c0634000 0000000b c06400e0
> 5d80: c03a53b0 00000000 c063f180 c0011624 c03a5310 200001d3 c0642f80 00010000
> 5da0: c0634238 0000000b 00100100 c0635e70 e92d45f0 7ed5a5f7 00000001 00000500
> 5dc0: c000dbcc c0634000 c063c3f0 c00082a0 00000006 c22b0768 00000004 00000000
> 5de0: 00030001 e92d45f0 ffffffff 00000001 c22b076c 00000000 c065fa1c c065f9c0
> 5e00: 00100100 00000002 c065f9c0 00000000 c063cca0 00000000 00000002 c065f9c0
> 5e20: c0661ca4 c06618e6 00000041 00000002 0000000a 00000002 ffffffff 00000000
> 5e40: 00000000 00000000 00000000 00000000 c06602cc 00000000 c00146d4 e92d45f4
> 5e60: 600001d3 c0634050 00000001 c000dbcc c0053e30 c22b3c40 fffffffe e92d45f0
> 5e80: c0053e30 c0640fcc ef007c00 c22b3c40 00000010 00000000 000003ff c063c3f0
> 5ea0: 00000000 c0635eb8 c023c2bc e92d45f0 600001d3 ffffffff ef004c80 c0063224
> 5ec0: 00000010 00000010 00000000 c0660ac0 c0635f18 c005fcb8 c0632b90 c000ed94
> 5ee0: c0313c60 60000153 00000001 c00085a8 c0313c54 c0313c60 60000153 ffffffff
> 5f00: c0635f4c 00000000 562f5842 c06360c0 00000000 c000db60 0000001a ffff8ad0
> 5f20: ffff8ad0 c06360c0 00000000 00000000 c04379c0 c22ad780 00000000 562f5842
> 5f40: c06360c0 00000000 60000153 c0635f60 c0313c54 c0313c60 60000153 ffffffff
> 5f60: 00000021 00000000 00000003 00000004 0000006e c065fcc0 c067924c c063ceb8
> 5f80: c063cc84 c006d8c0 00000005 c065fcc0 c067924c c0421764 c22ad780 c063c42c
> 5fa0: 562f5842 c063cca8 c06605c0 c04379c0 c22ad780 00000000 562f5842 00000000
> 5fc0: 00000000 c0417754 ffffffff ffffffff c04172dc 00000000 00000000 c04379c0
> 5fe0: 10c53c7d c063c414 c04379bc c063febc 0000406a 00008074 00000000 00000000
> [<c023c2bc>] (armada_370_xp_timer_interrupt+0x3c/0x4c) from [<c0063224>] (handle_percpu_devid_irq+0x64/0x80)
> [<c0063224>] (handle_percpu_devid_irq+0x64/0x80) from [<c005fcb8>] (generic_handle_irq+0x20/0x30)
> [<c005fcb8>] (generic_handle_irq+0x20/0x30) from [<c000ed94>] (handle_IRQ+0x38/0x90)
> [<c000ed94>] (handle_IRQ+0x38/0x90) from [<c00085a8>] (armada_370_xp_handle_irq+0xa4/0xb0)
> [<c00085a8>] (armada_370_xp_handle_irq+0xa4/0xb0) from [<c000db60>] (__irq_svc+0x40/0x50)
> Exception stack(0xc0635d20 to 0xc0635d68)
> 5d20: c0660fc0 00000000 c0642f80 00000000 00000000 00000000 0000000b c06400e0
> 5d40: c03a53b0 00000000 c0660b70 600001d3 00000000 c0635d68 c0316ca4 c0316d54
> 5d60: 60000153 ffffffff
> [<c000db60>] (__irq_svc+0x40/0x50) from [<c0316d54>] (panic+0x144/0x1b0)
> [<c0316d54>] (panic+0x144/0x1b0) from [<c0011624>] (die+0x278/0x2a8)
> [<c0011624>] (die+0x278/0x2a8) from [<c00082a0>] (do_undefinstr+0x9c/0x1c4)
> [<c00082a0>] (do_undefinstr+0x9c/0x1c4) from [<c000dbcc>] (__und_svc_finish+0x0/0x14)
> Exception stack(0xc0635e70 to 0xc0635eb8)
> 5e60: c0053e30 c22b3c40 fffffffe e92d45f0
> 5e80: c0053e30 c0640fcc ef007c00 c22b3c40 00000010 00000000 000003ff c063c3f0
> 5ea0: 00000000 c0635eb8 c023c2bc e92d45f0 600001d3 ffffffff
> [<c000dbcc>] (__und_svc_finish+0x0/0x14) from [<e92d45f0>] (0xe92d45f0)
> Code: 1fe7deb7 cd5772dd fff5692e ed55f79e (7ed5a5f7)
> ---[ end trace 1b75b31a2719ed1d ]---
>
>
> I am trying to figure out what happened.

Ok I found it, you forgot to also change the indirection in interrupt handler,
if you add the following chunk to your patch, it fixes the issue:


--- a/drivers/clocksource/time-armada-370-xp.c
+++ b/drivers/clocksource/time-armada-370-xp.c
@@ -150,7 +150,7 @@ static irqreturn_t armada_370_xp_timer_interrupt(int irq, void *dev_id)
/*
* ACK timer interrupt and call event handler.
*/
- struct clock_event_device *evt = *(struct clock_event_device **)dev_id;
+ struct clock_event_device *evt = (struct clock_event_device *)dev_id;

writel(TIMER0_CLR_MASK, local_base + LCL_TIMER_EVENTS_STATUS);
evt->event_handler(evt);


>
>
>>
>> Cc: Gregory CLEMENT <gregory.clement@xxxxxxxxxxxxxxxxxx>
>> Signed-off-by: Stephen Boyd <sboyd@xxxxxxxxxxxxxx>
>> ---
>> drivers/clocksource/time-armada-370-xp.c | 85 ++++++++++++++------------------
>> 1 file changed, 38 insertions(+), 47 deletions(-)
>>
>> diff --git a/drivers/clocksource/time-armada-370-xp.c b/drivers/clocksource/time-armada-370-xp.c
>> index efe4aef..ee2e50c5 100644
>> --- a/drivers/clocksource/time-armada-370-xp.c
>> +++ b/drivers/clocksource/time-armada-370-xp.c
>> @@ -19,6 +19,7 @@
>> #include <linux/platform_device.h>
>> #include <linux/kernel.h>
>> #include <linux/clk.h>
>> +#include <linux/cpu.h>
>> #include <linux/timer.h>
>> #include <linux/clockchips.h>
>> #include <linux/interrupt.h>
>> @@ -31,7 +32,6 @@
>> #include <linux/time-armada-370-xp.h>
>>
>> #include <asm/sched_clock.h>
>> -#include <asm/localtimer.h>
>> /*
>> * Timer block registers.
>> */
>> @@ -70,7 +70,7 @@ static bool timer25Mhz = true;
>> */
>> static u32 ticks_per_jiffy;
>>
>> -static struct clock_event_device __percpu **percpu_armada_370_xp_evt;
>> +static struct clock_event_device __percpu *armada_370_xp_evt;
>>
>> static u32 notrace armada_370_xp_read_sched_clock(void)
>> {
>> @@ -143,14 +143,7 @@ armada_370_xp_clkevt_mode(enum clock_event_mode mode,
>> }
>> }
>>
>> -static struct clock_event_device armada_370_xp_clkevt = {
>> - .name = "armada_370_xp_per_cpu_tick",
>> - .features = CLOCK_EVT_FEAT_ONESHOT | CLOCK_EVT_FEAT_PERIODIC,
>> - .shift = 32,
>> - .rating = 300,
>> - .set_next_event = armada_370_xp_clkevt_next_event,
>> - .set_mode = armada_370_xp_clkevt_mode,
>> -};
>> +static int armada_370_xp_clkevt_irq;
>>
>> static irqreturn_t armada_370_xp_timer_interrupt(int irq, void *dev_id)
>> {
>> @@ -173,42 +166,53 @@ static int __cpuinit armada_370_xp_timer_setup(struct clock_event_device *evt)
>> u32 u;
>> int cpu = smp_processor_id();
>>
>> - /* Use existing clock_event for cpu 0 */
>> - if (!smp_processor_id())
>> - return 0;
>> -
>> u = readl(local_base + TIMER_CTRL_OFF);
>> if (timer25Mhz)
>> writel(u | TIMER0_25MHZ, local_base + TIMER_CTRL_OFF);
>> else
>> writel(u & ~TIMER0_25MHZ, local_base + TIMER_CTRL_OFF);
>>
>> - evt->name = armada_370_xp_clkevt.name;
>> - evt->irq = armada_370_xp_clkevt.irq;
>> - evt->features = armada_370_xp_clkevt.features;
>> - evt->shift = armada_370_xp_clkevt.shift;
>> - evt->rating = armada_370_xp_clkevt.rating,
>> + evt->name = "armada_370_xp_per_cpu_tick",
>> + evt->features = CLOCK_EVT_FEAT_ONESHOT |
>> + CLOCK_EVT_FEAT_PERIODIC;
>> + evt->shift = 32,
>> + evt->rating = 300,
>> evt->set_next_event = armada_370_xp_clkevt_next_event,
>> evt->set_mode = armada_370_xp_clkevt_mode,
>> + evt->irq = armada_370_xp_clkevt_irq;
>> evt->cpumask = cpumask_of(cpu);
>>
>> - *__this_cpu_ptr(percpu_armada_370_xp_evt) = evt;
>> -
>> clockevents_config_and_register(evt, timer_clk, 1, 0xfffffffe);
>> enable_percpu_irq(evt->irq, 0);
>>
>> return 0;
>> }
>>
>> -static void armada_370_xp_timer_stop(struct clock_event_device *evt)
>> +static void __cpuinit armada_370_xp_timer_stop(struct clock_event_device *evt)
>> {
>> evt->set_mode(CLOCK_EVT_MODE_UNUSED, evt);
>> disable_percpu_irq(evt->irq);
>> }
>>
>> -static struct local_timer_ops armada_370_xp_local_timer_ops __cpuinitdata = {
>> - .setup = armada_370_xp_timer_setup,
>> - .stop = armada_370_xp_timer_stop,
>> +static int __cpuinit armada_370_xp_timer_cpu_notify(struct notifier_block *self,
>> + unsigned long action, void *hcpu)
>> +{
>> + struct clock_event_device *evt = this_cpu_ptr(armada_370_xp_evt);
>> +
>> + switch (action & ~CPU_TASKS_FROZEN) {
>> + case CPU_STARTING:
>> + armada_370_xp_timer_setup(evt);
>> + break;
>> + case CPU_DYING:
>> + armada_370_xp_timer_stop(evt);
>> + break;
>> + }
>> +
>> + return NOTIFY_OK;
>> +}
>> +
>> +static struct notifier_block armada_370_xp_timer_cpu_nb __cpuinitdata = {
>> + .notifier_call = armada_370_xp_timer_cpu_notify,
>> };
>>
>> void __init armada_370_xp_timer_init(void)
>> @@ -224,9 +228,6 @@ void __init armada_370_xp_timer_init(void)
>>
>> if (of_find_property(np, "marvell,timer-25Mhz", NULL)) {
>> /* The fixed 25MHz timer is available so let's use it */
>> - u = readl(local_base + TIMER_CTRL_OFF);
>> - writel(u | TIMER0_25MHZ,
>> - local_base + TIMER_CTRL_OFF);
>> u = readl(timer_base + TIMER_CTRL_OFF);
>> writel(u | TIMER0_25MHZ,
>> timer_base + TIMER_CTRL_OFF);
>> @@ -236,9 +237,6 @@ void __init armada_370_xp_timer_init(void)
>> struct clk *clk = of_clk_get(np, 0);
>> WARN_ON(IS_ERR(clk));
>> rate = clk_get_rate(clk);
>> - u = readl(local_base + TIMER_CTRL_OFF);
>> - writel(u & ~(TIMER0_25MHZ),
>> - local_base + TIMER_CTRL_OFF);
>>
>> u = readl(timer_base + TIMER_CTRL_OFF);
>> writel(u & ~(TIMER0_25MHZ),
>> @@ -252,7 +250,7 @@ void __init armada_370_xp_timer_init(void)
>> * We use timer 0 as clocksource, and private(local) timer 0
>> * for clockevents
>> */
>> - armada_370_xp_clkevt.irq = irq_of_parse_and_map(np, 4);
>> + armada_370_xp_clkevt_irq = irq_of_parse_and_map(np, 4);
>>
>> ticks_per_jiffy = (timer_clk + HZ / 2) / HZ;
>>
>> @@ -277,26 +275,19 @@ void __init armada_370_xp_timer_init(void)
>> "armada_370_xp_clocksource",
>> timer_clk, 300, 32, clocksource_mmio_readl_down);
>>
>> - /* Register the clockevent on the private timer of CPU 0 */
>> - armada_370_xp_clkevt.cpumask = cpumask_of(0);
>> - clockevents_config_and_register(&armada_370_xp_clkevt,
>> - timer_clk, 1, 0xfffffffe);
>> + register_cpu_notifier(&armada_370_xp_timer_cpu_nb);
>>
>> - percpu_armada_370_xp_evt = alloc_percpu(struct clock_event_device *);
>> + armada_370_xp_evt = alloc_percpu(struct clock_event_device);
>>
>>
>> /*
>> * Setup clockevent timer (interrupt-driven).
>> */
>> - *__this_cpu_ptr(percpu_armada_370_xp_evt) = &armada_370_xp_clkevt;
>> - res = request_percpu_irq(armada_370_xp_clkevt.irq,
>> + res = request_percpu_irq(armada_370_xp_clkevt_irq,
>> armada_370_xp_timer_interrupt,
>> - armada_370_xp_clkevt.name,
>> - percpu_armada_370_xp_evt);
>> - if (!res) {
>> - enable_percpu_irq(armada_370_xp_clkevt.irq, 0);
>> -#ifdef CONFIG_LOCAL_TIMERS
>> - local_timer_register(&armada_370_xp_local_timer_ops);
>> -#endif
>> - }
>> + "armada_370_xp_per_cpu_tick",
>> + armada_370_xp_evt);
>> + /* Immediately configure the timer on the boot CPU */
>> + if (!res)
>> + armada_370_xp_timer_setup(this_cpu_ptr(armada_370_xp_evt));
>> }
>>
>
>


--
Gregory Clement, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/