Re: [tip:core/rcu] Revert "rcu: Decrease memory-barrier usage basedon semi-formal proof"

From: Yinghai Lu
Date: Wed May 25 2011 - 18:16:20 EST


On 05/24/2011 09:52 PM, Paul E. McKenney wrote:
> On Tue, May 24, 2011 at 05:10:11PM -0700, Yinghai Lu wrote:
>> On 05/24/2011 02:23 PM, Yinghai Lu wrote:
>>> On 05/23/2011 06:35 PM, Paul E. McKenney wrote:
>>>> On Mon, May 23, 2011 at 06:26:23PM -0700, Yinghai Lu wrote:
>>>>> On 05/23/2011 06:18 PM, Paul E. McKenney wrote:
>>>>>
>>>>>> OK, so it looks like I need to get this out of the way in order to track
>>>>>> down the delays. Or does reverting PeterZ's patch get you a stable
>>>>>> system, but with the longish delays in memory_dev_init()? If the latter,
>>>>>> it might be more productive to handle the two problems separately.
>>>>>>
>>>>>> For whatever it is worth, I do see about 5% increase in grace-period
>>>>>> duration when switching to kthreads. This is acceptable -- your
>>>>>> 30x increase clearly is completely unacceptable and must be fixed.
>>>>>> Other than that, the main thing that affects grace period duration is
>>>>>> the setting of CONFIG_HZ -- the smaller the HZ value, the longer the
>>>>>> grace-period duration.
>>>>>
>>>>> for my 1024g system when memory hotadd is enabled in kernel config:
>>>>> 1. current linus tree + tip tree: memory_dev_init will take about 100s.
>>>>> 2. current linus tree + tip tree + your tree - Peterz patch:
>>>>> a. on fedora 14 gcc: will cost about 4s: like old times
>>>>> b. on opensuse 11.3 gcc: will cost about 10s.
>>>>
>>>> So some patch in my tree that is not yet in tip makes things better?
>>>>
>>>> If so, could you please see which one? Maybe that would give me a hint
>>>> that could make things better on opensuse 11.3 as well.
>>>
>>> today's tip:
>>>
>>> [ 31.795597] cpu_dev_init done
>>> [ 40.930202] memory_dev_init done
>>>
>>
>> another boot from tip got:
>>
>> [ 35.211927] cpu_dev_init done
>> [ 136.053698] memory_dev_init done
>>
>> wonder if you can have clean revert for
>>
>> commit a26ac2455ffcf3be5c6ef92bc6df7182700f2114
>>> Author: Paul E. McKenney <paul.mckenney@xxxxxxxxxx>
>>> Date: Wed Jan 12 14:10:23 2011 -0800
>>>
>>> rcu: move TREE_RCU from softirq to kthread
>>>
>>> If RCU priority boosting is to be meaningful, callback invocation must
>>> be boosted in addition to preempted RCU readers. Otherwise, in presence
>>> of CPU real-time threads, the grace period ends, but the callbacks don't
>>> get invoked. If the callbacks don't get invoked, the associated memory
>>> doesn't get freed, so the system is still subject to OOM.
>>>
>>> But it is not reasonable to priority-boost RCU_SOFTIRQ, so this commit
>>> moves the callback invocations to a kthread, which can be boosted easily.
>>>
>>> Also add comments and properly synchronized all accesses to
>>> rcu_cpu_kthread_task, as suggested by Lai Jiangshan.
>>>
>>> Signed-off-by: Paul E. McKenney <paul.mckenney@xxxxxxxxxx>
>>> Signed-off-by: Paul E. McKenney <paulmck@xxxxxxxxxxxxxxxxxx>
>>> Reviewed-by: Josh Triplett <josh@xxxxxxxxxxxxxxxx>
>
> There is a new branch yinghai.2011.05.24a on:
>
> git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-2.6-rcu.git
>
> Or will be as soon as kernel.org updates its mirrors.
>
> I am not sure I could call this "clean", but it does revert that commit
> and 11 of the subsequent commits that depend on it. It does build,
> and I will test it once my currently running tests complete.

yes, with those revert, there is no delay in 10 times booting.

Thanks

Yinghai
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/