Re: [Part3 PATCH v2 0/4] Support hot-remove local pagetable pages.

From: Vasilis Liaskovitis
Date: Tue Jun 18 2013 - 13:05:31 EST


Hi,

On Thu, Jun 13, 2013 at 09:03:52PM +0800, Tang Chen wrote:
> The following patch-set from Yinghai allocates pagetables to local nodes.
> v1: https://lkml.org/lkml/2013/3/7/642
> v2: https://lkml.org/lkml/2013/3/10/47
> v3: https://lkml.org/lkml/2013/4/4/639
> v4: https://lkml.org/lkml/2013/4/11/829
>
> Since pagetable pages are used by the kernel, they cannot be offlined.
> As a result, they cannot be hot-remove.
>
> This patch fix this problem with the following solution:
>
> 1. Introduce a new bootmem type LOCAL_NODE_DATAL, and register local
> pagetable pages as LOCAL_NODE_DATAL by setting page->lru.next to
> LOCAL_NODE_DATAL, just like we register SECTION_INFO pages.
>
> 2. Skip LOCAL_NODE_DATAL pages in offline/online procedures. When the
> whole memory block they reside in is offlined, the kernel can
> still access the pagetables.
> (This changes the semantics of offline/online a little bit.)

This could be a design problem of part3: if we allow local pagetable memory
to not be offlined but allow the offlining to return successfully, then
hot-remove is going to succeed. But the direct mapped pagetable pages are still
mapped in the kernel. The hot-removed memblocks will suddenly disappear (think
physical DIMMs getting disabled in real hardware, or in a VM case the
corresponding guest memory getting freed from the emulator e.g. qemu/kvm). The
system can crash as a result.

I think these local pagetables do need to be unmapped from kernel, offlined and
removed somehow - otherwise hot-remove should fail. Could they be migrated
alternatively e.g. to node 0 memory? But Iiuc direct mapped pages cannot be
migrated, correct?

What is the original reason for local node pagetable allocation with regards
to memory hotplug? I assume we want to have hotplugged nodes use only their local
memory, so that there are no inter-node memory dependencies for hot-add/remove.
Are there other reasons that I am missing?


I get a crash with the following:
- boot 2-node VM with 4G on node0 and 0G on node1
- guest kernel linux_next20130607 + part1 + part2 + part3 patchsets
- hotplugged 2 dimms on node1 (previosuly node 1 had no memory)
- reboot with movablecore=acpi, ZONE_MOVABLE is created for node1
- I verified that in this case LOCAL_NODE_DATA pages are allocated on node1
- remove a dimm on node1

[ 260.615677] Offlined Pages 32768
[ 260.619650] Offlined Pages 32768
[ 260.630514] Offlined Pages 32768
[ 260.634342] Offlined Pages 32768
[ 260.637989] Offlined Pages 32768
[ 260.642021] Offlined Pages 32768
[ 260.643852] Offlined Pages 32768
[ 260.645555] Offlined Pages 32768
[ 260.817960] general protection fault: 0000 [#1] SMP
[ 260.818384] Modules linked in: netconsole pci_hotplug lp loop kvm_amd kvm ppdev psmouse parport_pc i2c_piix4 parport i2c_core serio_raw evdev processor microcode button thermal_sys ext3 jbd mbcache sr_mod cdrom ata_generic virtio_blk virtio_net ata_piix libata scsi_mod virtio_pci virtio_ring virtio
[ 260.820449] CPU: 0 PID: 177 Comm: kworker/0:2 Not tainted 3.10.0-rc4-next-20130607-guest #1
[ 260.820449] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[ 260.820449] Workqueue: kacpi_hotplug acpi_os_execute_deferred
[ 260.820449] task: ffff880115c80080 ti: ffff880115564000 task.ti: ffff880115564000
[ 260.820449] RIP: 0010:[<ffffffff812b92b9>] [<ffffffff812b92b9>] memchr_inv+0x89/0x110
[ 260.820449] RSP: 0018:ffff880115565b90 EFLAGS: 00010206
[ 260.820449] RAX: 0000000008000000 RBX: ffff8801e0000000 RCX: 0000000000000000
[ 260.820449] RDX: 0000000040000000 RSI: 00000000000000fd RDI: 7fff8801c0000000
[ 260.820449] RBP: ffff8801e0000000 R08: 00000000000000fd R09: 0101010101010101
[ 260.820449] R10: 0000000000000000 R11: ffff880116e38e50 R12: ffff880220000000
[ 260.820449] R13: ffff8801c0000000 R14: 00003ffffffff000 R15: ffff880116e38e01
[ 260.820449] FS: 00007fa6d8de37c0(0000) GS:ffff88011f000000(0000) knlGS:0000000000000000
[ 260.820449] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 260.820449] CR2: 00007fa6d8dea000 CR3: 000000000180b000 CR4: 00000000000006f0
[ 260.820449] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 260.820449] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 260.820449] Stack:
[ 260.820449] ffffffff8151e3bc 0000160000000000 ffffc00000000fff ffff880001a5d038
[ 260.820449] ffffffff8150e39e ffff880116e38e50 0000000000000100 0000000000000000
[ 260.820449] ffff8801e0000000 ffff8801181fb060 ffffffff8150e424 ffff88021ffff000
[ 260.820449] Call Trace:
[ 260.820449] [<ffffffff8151e3bc>] ? remove_pagetable+0x20c/0x7f7
[ 260.820449] [<ffffffff8150e39e>] ? klist_put+0x4e/0xc0
[ 260.820449] [<ffffffff8150e424>] ? klist_iter_exit+0x14/0x20
[ 260.820449] [<ffffffff8150f385>] ? arch_remove_memory+0x75/0xd0
[ 260.820449] [<ffffffff81510e18>] ? remove_memory+0x68/0x80
[ 260.820449] [<ffffffff8132baa5>] ? acpi_memory_device_remove+0x90/0xea
[ 260.820449] [<ffffffff81308157>] ? acpi_bus_device_detach+0x37/0x5c
[ 260.820449] [<ffffffff813081b5>] ? acpi_bus_trim+0x39/0x6e
[ 260.820449] [<ffffffff81308c74>] ? acpi_scan_hot_remove+0x136/0x264
[ 260.820449] [<ffffffff81304b15>] ? acpi_evaluate_hotplug_ost+0x86/0x8b
[ 260.820449] [<ffffffff81308e3a>] ? acpi_bus_device_eject+0x98/0xcc
[ 260.820449] [<ffffffff813042ea>] ? acpi_os_execute_deferred+0x1f/0x2b
[ 260.820449] [<ffffffff8105a183>] ? process_one_work+0x153/0x480
[ 260.820449] [<ffffffff8105b2b6>] ? worker_thread+0x116/0x3d0
[ 260.820449] [<ffffffff8105b1a0>] ? manage_workers+0x2b0/0x2b0
[ 260.820449] [<ffffffff8105b1a0>] ? manage_workers+0x2b0/0x2b0
[ 260.820449] [<ffffffff81060b56>] ? kthread+0xc6/0xd0
[ 260.820449] [<ffffffff81060a90>] ? kthread_freezable_should_stop+0x60/0x60
[ 260.820449] [<ffffffff8152c6ec>] ? ret_from_fork+0x7c/0xb0
[ 260.820449] [<ffffffff81060a90>] ? kthread_freezable_should_stop+0x60/0x60
[ 260.820449] Code: 75 f0 45 89 c0 4c 01 c7 4c 29 c2 0f 1f 80 00 00 00 00 48 89 d0 48 c1 e8 03 85 c0 74 2a 44 0f b6 c6 49 b9 01 01 01 01 01 01 01 01 <48> 8b 0f 4d 0f af c1 4c 39 c1 74 08 eb 41 90 48 3b 0f 75 3b 48
[ 260.820449] RIP [<ffffffff812b92b9>] memchr_inv+0x89/0x110
[ 260.820449] RSP <ffff880115565b90>
[ 260.852908] ---[ end trace 5e9b131e0294cfb6 ]---

I haven't analyzed more yet.

Other times instead of a crash on hot-remove I get a "fork: cannot allocate memory"
always after hot-remove.

thanks,

- Vasilis
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/