Re: Panic caused by PTE corruption - 2.6.32 distro kernel

[Pasi Kärkkäinen]

On Sat, Jan 08, 2011 at 10:48:48PM -0600, Jack Steiner wrote:
> 
> A long shot but I'm hoping someone has seen corruption like this
> before.
> 
> System is running a 2.6.32 distro kernel on a large x86_64 Nehalem system.
>

Which exact kernel version/distro is this? 

-- Pasi

> During the last 4+ months we have seen 3 instances of panic's caused by a
> spurious bit 48 being set in a page table. So far, we are unable to
> reliably duplicate the problem but it has occured on multiple systems. We
> have not seen any other strange failures caused by memory corruption.  The
> bug only hits bit 48 in PTEs.
> 
> Has anyone seen anything like this before or have any ideas? Note
> that the system has several non-distro drivers. We have no
> reason to believe these drivers are related but can't rule it out.
> 
> Here is a detailed analysis of the first failure. The other failures are
> similar.
> 
> --------------------------------
> 
> Failure occurred because a user-mode page-table-walk found a page table
> entry with a reserved bit set.
> 
>         <1> engine_par: Corrupted page table at address 20705808
>         <4> PGD 7cf83c8b067 PUD 7ccd3170067 PMD 7e7f8c46067 PTE 800107e1d43f5067
>         <0> Bad pagetable: 000f [#1] SMP
>                               ^--- bit 3 ==> reserved bit is set in a PT entry
> 
>         PGD      7cf83c8b067
>         PUD      7ccd3170067
>         PMD      7e7f8c46067
>         PTE 800107e1d43f5067
>                ^------- ???
> 
> 
> Note bit 48 in the PTE. This bit should not be set. Failure left no trace
> that I could find.  Other observations:
> 
> Failing process is:
>         0xffff8fcff6942140   237757   237716  1 1010   R  0xffff8fcff69427d0 *engine_par
> 
> Appears to be a big MPI job (hybrid???)
> 
> 
> Other entries in the PT close to the corrupted entry look reasonable:
>         0xffff8fe7f8c46800 800007e1d43f0067 800007e1d43f1067
>         0xffff8fe7f8c46810 800007e1d43f2067 800007e1d43f3067
>         0xffff8fe7f8c46820 800007e1d43f4067 800107e1d43f5067 << has bad entry
>         0xffff8fe7f8c46830 800007e1d43f6067 800007e1d43f7067
>         0xffff8fe7f8c46840 800007e1ab590067 800007e1ab591067
>         0xffff8fe7f8c46850 800007e1ab592067 800007e1ab593067
>         0xffff8fe7f8c46860 800007e1ab594067 800007e1ab595067
>         0xffff8fe7f8c46870 800007e1ab596067 800007e1ab597067
> 
> 
> A second failure:
> 	0xffff8dbff5a34a70 800005bcdd382067 800005bcdd383067
> 	0xffff8dbff5a34a80 800005bcdd384067 800005bcdd385067
> 	0xffff8dbff5a34a90 800005bcdd386067 800005bcdd387067
> 	0xffff8dbff5a34aa0 800005bcdd388067 800105bcdd389067  << has bad entry
> 	0xffff8dbff5a34ab0 800005bcdd38a067 800005bcdd38b067
> 	0xffff8dbff5a34ac0 800005bcdd38c067 800005bcdd38d067
> 	0xffff8dbff5a34ad0 800005bcdd38e067 800005bcdd38f067
> 	0xffff8dbff5a34ae0 800005bcdd390067 800005bcdd391067
> 
> ---
> Jack Steiner (steiner@xxxxxxx)
> SGI - Silicon Graphics, Inc.
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@xxxxxxxxxxxxxxx
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/