Re: [RFC PATCH 0/5] crash dump bitmap: scan memory pages in kernelto speedup kernel dump process

[Vivek Goyal]

On Thu, Mar 07, 2013 at 10:58:18PM +0800, Jingbai Ma wrote:
> This patch intend to speedup the memory pages scanning process in
> selective dump mode.
> 
> Test result (On HP ProLiant DL980 G7 with 1TB RAM, makedumpfile
> v1.5.3):
> 
> 						Total scan Time
> Original kernel
> 	+ makedumpfile v1.5.3 cyclic mode	1958.05 seconds
> Original kernel
> 	+ makedumpfile v1.5.3 non-cyclic mode	1151.50 seconds
> Patched kernel
> 	+ patched makedumpfile v1.5.3		17.50 seconds
> 
> Traditionally, to reduce the size of dump file, dumper scans all memory
> pages to exclude the unnecessary memory pages after capture kernel
> booted, and scan it in userspace code (makedumpfile).

I think this is not a good idea. It has several issues.

- First of all it is doing more stuff in first kernel. And that runs
  contrary to kdump design where we want to do stuff in second kernel.
  After a kernel crash, you can't trust running kernel's data structures.
  So to improve reliability just do minial stuff in crashed kernel and
  get out quickly.

- Secondly, it moves filetering policy in kernel. I think keeping it
  in user space gives us the extra flexibility.
> 
> It introduces several problems:
> 
> 1. Requires more memory to store memory bitmap on systems with large
> amount of memory installed. And in capture kernel there is only a few
> free memory available, it will cause an out of memory error and fail.
> (Non-cyclic mode)

makedumpfile requires 2bits per 4K page. That is 64MB per TB. In your
patches also you are reserving 1bit per page and that is 32MB per TB
in first kernel.

So memory is anyway being reserved, just that makedumpfile seems to be
needing this extra bit. Not sure if that can be optimized or not.

First of all 64MB per TB should not be a huge deal. And makedumpfile
also has this cyclic mode where you process a map, discard it and then
move on to next section. So memory usage remains constant at the expense
of processing time.

Looks like now hpa and yinghai have done the work to be able to load
kdump kernel above 4GB. I am assuming this also removes the restriction
that we can only reserve 512MB or 896MB in second kernel. If that's
the case, then I don't see why people can't get away with reserving
64MB per TB.

> 
> 2. Scans all memory pages in makedumpfile is a very slow process. On
> system with 1TB or more memory installed, the scanning process is very
> long. Typically on 1TB idle system, it takes about 19 minutes. On system
> with 4TB or more memory installed, it even doesn't work. To address the
> out of memory issue on system with big memory (4TB or more memory
> installed),  makedumpfile v1.5.1 introduces a new cyclic mode. It only
> scans a piece of memory pages each time, and do it cyclically to scan
> all memory pages. But it runs more slowly, on 1TB system, takes about 33
> minutes.

One of the reasons it is slow because we don't support mmpa() interface.
That means for every read, we map 4K page, flush TLB, read it, unmap it
and flush TLB again. This is lot of processing overhead per 4K page.

Hatayama is now working on making mmap() interface and allow user space
to bigger chunks of memory in one so. So that in one mmap() call we can
map a bigger range instead of just 4K. And his numbers show that it
has helped a lot.

So instead of trying to move filtering logic in kernel, I think it
might be better if we try to optimize things in makedumpfile or second
kernel.

> 
> 3. Scans memory pages code in makedumpfile is very complicated, without
> kernel memory management related data structure, makedumpfile has to
> build up its own data structure, and will not able to use some macros
> that only be available in kernel (e.g. page_to_pfn), and has to use some
> slow lookup algorithm instead.
> 
> This patch introduces a new way to scan memory pages. It reserves a
> piece of memory (1 bit for each page, 32MB per TB memory on x86 systems)
> in the first kernel. During the kernel crash process, it scans all
> memory pages, clear the bit for all excluded memory pages in the
> reserved memory.

I think this is not a good idea. It has several issues.

- First of all it is doing more stuff in first kernel. And that runs
  contrary to kdump design where we want to do stuff in second kernel.
  After a kernel crash, you can't trust running kernel's data structures.
  So to improve reliability just do minial stuff in crashed kernel and
  get out quickly.

- Secondly, it moves filetering policy in kernel. I think keeping it
  in user space gives us the extra flexibility.

> 

> We have several benefits by this new approach:
> 
> 1. It's extremely fast, on 1TB system only takes about 17.5 seconds to
> scan all memory pages!
> 
> 2. Reduces the memory requirement of makedumpfile by putting the
> reserved memory in the first kernel memory space.

Even the second kernel's memory comes from first kernel. So that really
does not help.

Thanks
Vivek
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/