Re: sched: CPU #1's llc-sibling CPU #0 is not on the same node!

From: Tang Chen
Date: Tue Feb 26 2013 - 23:33:27 EST

On 02/27/2013 10:24 AM, Yinghai Lu wrote:
After looked at the code more, thought that theory that does not let
kernel use ram
on hotplug area is not right.

after that commit, following range can not use movable ram:
1. real_mode code.... well..funny, legacy cpu0 [0,1M) could be
2. dma_continguous ?
3. log buff ring.
4. initrd... why it will be freed after booting, so it could be on
5. crashkernel for kdump...: : looks like we can not put kdump kernel
above 4G anymore
6. initmem_init: it will allocate page table to setup kernel mapping
for memory..., it should
be with BRK and near end of max_pfn....

AFAIK, Linux kernel now cannot migrate memory used by the kernel because. So
any memory
used by the kernel should not be on movable area.

that depends.

initrd will be freed later, so it should be put anywhere that is under
max_pfn during boot.

OKïbut initrd is not that big. Actually, before my code start to work, memblock
has reserved some memory. But it is not that big. On the other hand, it is not that
easy to find out which memory should be kept in unmovable area, and which should not.

If node is hotplugable, the mem related stuff like page table and
vmemmap could be
on the that node without problem and should be on that node.

page tables and vmemmap are kernel memory. They should not be movable, I

why do you need to migrate pagetable and vmemmap for the memory range
that will be
offline ?

Hum, you are right. :)

True, we can store pagetable and vmemmap on the node that is hot-pluggable.
But just like the page_cgroup structs, we need additional work to handle it.

But based on the existing code, we didn't do any special handling. I think
we can improve it if needed. ïï

assume first cpu only have 1G ram, and other 31 socket will have bunch of
and those cpu with ram could be hotadd and hotremoved.
Now you want to put page table and vmemmap on first node.
The system would not boot as not enough memory for cover whole system RAM.

Yes, you are right. And a more extreme situation has been talked about by

"If all the memory is hot-pluggable, then the kernel won't be able to boot."

So, please refer to commit 01a178a94e8eaec351b29ee49fbb3d1c124cb7fb:
acpi, memory-hotplug: support getting hotplug info from SRAT

I have excluded all the memory reserved by memblock, and any node that has
reserved by memblock will be set to un-hot-pluggable, which means we will
enough memory (all the memory on the node) to boot the kernel. So I think
the problem
you are talking about has been solved.

I don't think that you understand the problem.

for the system that will put all pagetable and vmemmap on the 1G ram
of first cpu.
as all other ram are MOVABLE, so memblock_find_in_range will not use any local
ram on those nodes.

Yes, I konw that. :)

In this case, the kernel will not able to use local ram on those nodes. It will
cause some performance down.

I mean if the 1G ram is not enough for the kernel to boot, the current code will
set all the ram on the same node as un-hot-pluggable.

If all the ram on the node is not enough for kernel to boot, it is a really extreme
situation, IIUC.

I think users can solve this problem in two ways:
1) add more ram to the node.
2) use movablemem_map=nn[KMG]@ss[KMG] to configure more ram as unmovable.

Thanks. :)

To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at
Please read the FAQ at