Re: 2.6.38.8 kernel bug in XFS or megaraid driver with heavy I/O load

[Anders Ossowicki]

On Wed, Oct 12, 2011 at 02:35:26AM +0200, Dave Chinner wrote:
> Memory corruption can be caused by more than just a bad memory
> stick. You've got a brand new driver running your brand new
> controller and it may still have bugs - it might be scribbling over
> memory it doesn't own because of off-by-one index errors, etc. It's
> much more likely that that new hardware or driver code is the cause
> of your problem than an undetected ECC memory error or core VM
> problem.
Ah, now that I agree on. A few more observations from today's experiments:

First of all, there are two MegaRAID controllers in the machine. The 
old'n'reliable 8888ELP and the new'n'wonky 9285-8e. Both are using the megaraid
driver and the 8888ELP card ran with the megaraid driver prior to the 
refactoring that introduced support for 9285-8e without a hitch for about a
year.

We've gotten to a point where we can reliably reproduce this by running certain
queries in postgresql when data from the disk is cached. E.g 
 foo=# select count(*) from foo.sequence;
 ERROR:  invalid page header in block 529134 of relation base/16385/58318945

If we echo 3 >/proc/sys/vm/drop_caches and reload postgres, the same queries
work. This does indeed smell like memory corruption.

The 9285-8e controller has FastPath enabled.

> FWIW, if it's a repeatable problem, you might want to update the
> driver and controller firmware to something more recent and see if
> that solves the problem....
I upgraded the firmware (post-accident) but we're still seeing the corruption.
-- 
Anders

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/