Re: [fuse-devel] Horrible mmap write performance (kernel writeback issue?)

From: Miklos Szeredi
Date: Wed Mar 16 2016 - 05:44:17 EST


On Tue, Mar 15, 2016 at 8:55 AM, Jakob Unterwurzacher
<jakobunt@xxxxxxxxx> wrote:
> Just for anybody finding this thread: This still happens in v4.4, it
> just took longer to trigger.
>
> I have posted more details to linux-kernel (copy-pasted below),
> http://thread.gmane.org/gmane.linux.kernel/2132944

Okay, so you can reproduce this relatively quickly. Can you try "git
bisect" to find exactly which commit is responsible?

Thanks,
Miklos

>
> -------- copy of the email to linux-kernel -------------------
>
> 2016-01-22 21:10:59
>
> I have noticed an annoying regression that was introduced in 4.2 and is
> still there in 4.4. mmap writes to FUSE filesystems are throttled down
> to basically zero.
>
> Reproducer: https://github.com/rfjakob/mmapwrite , testing against encfs:
>
> $ mmapwrite /tmp/encfs-mnt/foo
> 1 .................................................. 107.01 MB/s
> 2 .................................................. 101.98 MB/s
> [...]
> 68 .................................................. 106.79 MB/s
> 69 .................................................. 105.09 MB/s
> 70 .................................................. 2.02 MB/s
> 71 .................................................. 1.77 MB/s
> 72 .................................................. 0.42 MB/s
> 73 .................................... (hangs)
>
> I have tested kernels from 4.0 and this seems to have been introduced in
> 4.2:
>
> 4.0 ....... 140MB/s permanent
> 4.1 ....... 140MB/s permanent
> 4.2 ....... 100MB/s at the start, sudden slowdown to 1MB/s after ~5GB
> 4.3 ....... 100MB/s at the start, sudden slowdown to 1MB/s after ~1.5GB
> 4.4-rc4 ... 100MB/s at the start, slowly ramps down, 0.3MB/s after ~2GB
> 4.4 ....... 100MB/s at the start, sudden slowdown after ~3GB
>
> Is there a way to disable the throttling? Or at least exempt FUSE until
> there is a proper fix?
>
> Thanks,
> Jakob
>
>
>
>
> On 17.12.2015 00:26, Jakob Unterwurzacher wrote:
>> This seems to be fixed in v4.4-rc5-18-gedb42dc. mmap writes now proceed
>> at solid 100MB/s with full CPU saturation.
>>
>> Thanks,
>> Jakob
>>
>> On Mon, Dec 14, 2015 at 9:06 AM, Jakob Unterwurzacher
>> <jakobunt@xxxxxxxxx <mailto:jakobunt@xxxxxxxxx>> wrote:
>>
>> I am the developer of https://github.com/rfjakob/gocryptfs (an
>> encrypted overlay filesystem like EncFS)
>> and have ported xfstests over for regression testing
>> ( https://github.com/rfjakob/fuse-xfstests ).
>>
>> xfstests generic/074 is how I noticed that mmap write performance
>> plummeted when Fedora upgraded my kernel to 4.2.5.
>> It used to complete in 10 minutes and now it will probably take days.
>> I am on kernel 4.4-rc1 now and still seeing the same issue.
>>
>> It looks like the kernel at some point the kernel writeback
>> mechanism gets stuck (or throttles?)
>>
>> Testing against encfs:
>>
>> ./mmapwrite /tmp/e2/foo #
>> https://github.com/rfjakob/mmapwrite .
>> ....................................................................................................
>> 98.91 MB/s
>> ....................................................................................................
>> 93.74 MB/s
>> ....................................................................................................
>> 103.89 MB/s
>> ....................................................................................................
>> 100.20 MB/s
>> ....................................................................................................
>> 104.03 MB/s
>> ....................................................................................................
>> 98.06 MB/s
>> ....................................................................................................
>> 10.17 MB/s
>> ....................................................................................................
>> 9.50 MB/s
>> .................................. (hangs)
>>
>> At this point no write requests are submitted to encfs and the
>> mmapwrite process is
>> stuck in the kernel in balance_dirty_pages.isra.22.
>>
>> Bisecting this will be a pain, I would appreciate any suggestions.
>>
>> Note that this seems to affect every FUSE filesystem, also ntfs-3g.
>>
>> Best regards,
>> Jakob
>>
>>
>
>
> ------------------------------------------------------------------------------
> Transform Data into Opportunity.
> Accelerate data analysis in your applications with
> Intel Data Analytics Acceleration Library.
> Click to learn more.
> http://pubads.g.doubleclick.net/gampad/clk?id=278785231&iu=/4140
> --
> fuse-devel mailing list
> To unsubscribe or subscribe, visit https://lists.sourceforge.net/lists/listinfo/fuse-devel