[PATCH] x86/mm: fix cpu stuck issue in __change_page_attr_set_clr

From: Bin Yang
Date: Thu Jun 28 2018 - 06:05:46 EST


This issue can be easily triggered by free_initmem() functuion on
x86_64 cpu.

When changing page attr, __change_page_attr_set_clr will call
__change_page_attr for every 4K page. And try_preserve_large_page
will be called to check whether it needs to split the large page.

If cpu supports "pdpe1gb", kernel will try to use 1G large page.
In worst case, it needs to check every 4K page in 1G range in
try_preserve_large_page function. If __change_page_attr_set_clr
needs to change lots of 4K pages, cpu will be stuck for long time.

This patch try to cache the last address which had been checked just
now. If the next address is in same big page, the cache will be used
without full range check.

Signed-off-by: Bin Yang <bin.yang@xxxxxxxxx>
---
arch/x86/mm/pageattr.c | 26 ++++++++++++++++++++++----
1 file changed, 22 insertions(+), 4 deletions(-)

diff --git a/arch/x86/mm/pageattr.c b/arch/x86/mm/pageattr.c
index 3bded76e..b9241ac 100644
--- a/arch/x86/mm/pageattr.c
+++ b/arch/x86/mm/pageattr.c
@@ -552,16 +552,20 @@ static int
try_preserve_large_page(pte_t *kpte, unsigned long address,
struct cpa_data *cpa)
{
+ static unsigned long address_cache;
+ static unsigned long do_split_cache = 1;
unsigned long nextpage_addr, numpages, pmask, psize, addr, pfn, old_pfn;
pte_t new_pte, old_pte, *tmp;
pgprot_t old_prot, new_prot, req_prot;
int i, do_split = 1;
enum pg_level level;

- if (cpa->force_split)
+ spin_lock(&pgd_lock);
+ if (cpa->force_split) {
+ do_split_cache = 1;
return 1;
+ }

- spin_lock(&pgd_lock);
/*
* Check for races, another CPU might have split this page
* up already:
@@ -627,13 +631,25 @@ try_preserve_large_page(pte_t *kpte, unsigned long address,

new_prot = static_protections(req_prot, address, pfn);

+ addr = address & pmask;
+ pfn = old_pfn;
+ /*
+ * If an address in same range had been checked just now, re-use the
+ * cache value without full range check. In the worst case, it needs to
+ * check every 4K page in 1G range, which causes cpu stuck for long
+ * time.
+ */
+ if (!do_split_cache &&
+ address_cache >= addr && address_cache < nextpage_addr &&
+ pgprot_val(new_prot) == pgprot_val(old_prot)) {
+ do_split = do_split_cache;
+ goto out_unlock;
+ }
/*
* We need to check the full range, whether
* static_protection() requires a different pgprot for one of
* the pages in the range we try to preserve:
*/
- addr = address & pmask;
- pfn = old_pfn;
for (i = 0; i < (psize >> PAGE_SHIFT); i++, addr += PAGE_SIZE, pfn++) {
pgprot_t chk_prot = static_protections(req_prot, addr, pfn);

@@ -670,6 +686,8 @@ try_preserve_large_page(pte_t *kpte, unsigned long address,
}

out_unlock:
+ address_cache = address;
+ do_split_cache = do_split;
spin_unlock(&pgd_lock);

return do_split;
--
2.7.4