Re: [PATCH 3/4] kgdb,clocksource: Prevent kernel hang in kernel debugger

[Xiaotian Feng]

On Tue, Jan 26, 2010 at 6:01 PM, Dongdong Deng <libfetion@xxxxxxxxx> wrote:
> On Tue, Jan 26, 2010 at 4:50 PM, Thomas Gleixner <tglx@xxxxxxxxxxxxx> wrote:
>> On Tue, 26 Jan 2010, Martin Schwidefsky wrote:
>>> On Mon, 25 Jan 2010 22:26:39 -0600
>>> Jason Wessel <jason.wessel@xxxxxxxxxxxxx> wrote:
>>>
>>> > This is a regression fix against: 0f8e8ef7c204988246da5a42d576b7fa5277a8e4
>>> >
>>> > Spin locks were added to the clocksource_resume_watchdog() which cause
>>> > the kernel debugger to deadlock on an SMP system frequently.
>>> >
>>> > The kernel debugger can try for the lock, but if it fails it should
>>> > continue to touch the clocksource watchdog anyway, else it will trip
>>> > if the general kernel execution has been paused for too long.
>>> >
>>> > This introduces an possible race condition where the kernel debugger
>>> > might not process the list correctly if a clocksource is being added
>>> > or removed at the time of this call. ÂThis race is sufficiently rare vs
>>> > having the kernel debugger hang the kernel
>>> >
>>> > CC: Thomas Gleixner <tglx@xxxxxxxxxxxxx>
>>> > CC: Martin Schwidefsky <schwidefsky@xxxxxxxxxx>
>>> > CC: John Stultz <johnstul@xxxxxxxxxx>
>>> > CC: Andrew Morton <akpm@xxxxxxxxxxxxxxxxxxxx>
>>> > CC: Magnus Damm <damm@xxxxxxxxxx>
>>> > Signed-off-by: Jason Wessel <jason.wessel@xxxxxxxxxxxxx>
>>>
>>> The first question I would ask is why does the kernel deadlock? Can we
>>> have a backchain of a deadlock please?
>>
>> The problem arises when the kernel is stopped inside the watchdog code
>> with watchdog_lock held. When kgdb restarts execution then it touches
>> the watchdog to avoid that TSC gets marked unstable.
>>
>>> Hmm, there are all kinds of races if the watchdog code gets interrupted
>>> by the kernel debugger. Wouldn't it be better to just disable the
>>> watchdog while the kernel debugger is active?
>>
>> No, we can keep it and in most cases it clocksource_touch_watchdog()
>> helps to keep TSC alive. A simple "if (!trylock) return;" should solve
>> the deadlock problem for kgdb without opening a can of worms.
>
> Is it possible that we reset the clocksource watchdog during in
> clocksource_watchdog() ?
>
> From the code view, The action of reset clocksource watchdog is just
> set the CLOCK_SOURCE_WATCHDOG flag.
> thus if we reset it before using, I think the logic will be right.

Is this logic right for resume case?

>
>
> ---
>
> diff --git a/kernel/time/clocksource.c b/kernel/time/clocksource.c
> index e85c234..0590983 100644
> --- a/kernel/time/clocksource.c
> +++ b/kernel/time/clocksource.c
> @@ -187,6 +187,8 @@ static DEFINE_SPINLOCK(watchdog_lock);
> Âstatic cycle_t watchdog_last;
> Âstatic int watchdog_running;
>
> +static int volatile clocksource_resume_watchdog;
> +
> Âstatic int clocksource_watchdog_kthread(void *data);
> Âstatic void __clocksource_change_rating(struct clocksource *cs, int rating);
>
> @@ -253,6 +255,11 @@ static void clocksource_watchdog(unsigned long data)
> Â Â Â Âif (!watchdog_running)
> Â Â Â Â Â Â Â Âgoto out;
>
> + Â Â Â if (unlikely(clocksource_resume_watchdog)) {
> + Â Â Â Â Â Â Â clocksource_reset_watchdog();
> + Â Â Â Â Â Â Â clocksource_resume_watchdog = 0;
> + Â Â Â }
> +
> Â Â Â Âwdnow = watchdog->read(watchdog);
> Â Â Â Âwd_nsec = clocksource_cyc2ns((wdnow - watchdog_last) & watchdog->mask,
> Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â watchdog->mult, watchdog->shift);
> @@ -341,11 +348,7 @@ static inline void clocksource_reset_watchdog(void)
>
> Âstatic void clocksource_resume_watchdog(void)
> Â{
> - Â Â Â unsigned long flags;
> -
> - Â Â Â spin_lock_irqsave(&watchdog_lock, flags);
> - Â Â Â clocksource_reset_watchdog();
> - Â Â Â spin_unlock_irqrestore(&watchdog_lock, flags);
> + Â Â Â clocksource_resume_watchdog = 1;
> Â}
>
> Âstatic void clocksource_enqueue_watchdog(struct clocksource *cs)
>
>
>
>>
>> Thanks,
>>
>> Â Â Â Âtglx
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>> the body of a message to majordomo@xxxxxxxxxxxxxxx
>> More majordomo info at Âhttp://vger.kernel.org/majordomo-info.html
>> Please read the FAQ at Âhttp://www.tux.org/lkml/
>>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/