Re: [PATCH] erofs: support adjust lz4 history window size

[Huang Jianan]

Hi Xiang,

On 2021/2/22 12:44, Gao Xiang wrote:
> Hi Jianan,
>
> On Thu, Feb 18, 2021 at 08:00:49PM +0800, Huang Jianan via Linux-erofs wrote:
> > From: huangjianan <huangjianan@xxxxxxxx>
> >
> > lz4 uses LZ4_DISTANCE_MAX to record history preservation. When
> > using rolling decompression, a block with a higher compression
> > ratio will cause a larger memory allocation (up to 64k). It may
> > cause a large resource burden in extreme cases on devices with
> > small memory and a large number of concurrent IOs. So appropriately
> > reducing this value can improve performance.
> >
> > Decreasing this value will reduce the compression ratio (except
> > when input_size <LZ4_DISTANCE_MAX). But considering that erofs
> > currently only supports 4k output, reducing this value will not
> > significantly reduce the compression benefits.
> >
> > Signed-off-by: Huang Jianan <huangjianan@xxxxxxxx>
> > Signed-off-by: Guo Weichao <guoweichao@xxxxxxxx>
> > ---
> >   fs/erofs/decompressor.c | 13 +++++++++----
> >   fs/erofs/erofs_fs.h     |  3 ++-
> >   fs/erofs/internal.h     |  3 +++
> >   fs/erofs/super.c        |  3 +++
> >   4 files changed, 17 insertions(+), 5 deletions(-)
> >
> > diff --git a/fs/erofs/decompressor.c b/fs/erofs/decompressor.c
> > index 1cb1ffd10569..94ae56b3ff71 100644
> > --- a/fs/erofs/decompressor.c
> > +++ b/fs/erofs/decompressor.c
> > @@ -36,22 +36,27 @@ static int z_erofs_lz4_prepare_destpages(struct z_erofs_decompress_req *rq,
> >   	struct page *availables[LZ4_MAX_DISTANCE_PAGES] = { NULL };
> >   	unsigned long bounced[DIV_ROUND_UP(LZ4_MAX_DISTANCE_PAGES,
> >   					   BITS_PER_LONG)] = { 0 };
> > +	unsigned int lz4_distance_pages = LZ4_MAX_DISTANCE_PAGES;
> >   	void *kaddr = NULL;
> >   	unsigned int i, j, top;
> >   
> > +	if (EROFS_SB(rq->sb)->compr_alg)
> > +		lz4_distance_pages = DIV_ROUND_UP(EROFS_SB(rq->sb)->compr_alg,
> > +						  PAGE_SIZE) + 1;
> > +
> Thanks for your patch, I agree that will reduce runtime memory
> footpoint. and keep max sliding window ondisk in bytes (rather
> than in blocks) is better., but could we calculate lz4_distance_pages
> ahead when reading super_block?
Thanks for suggestion, i will update it soon.
> Also, in the next cycle, I'd like to introduce a bitmap for available
> algorithms (maximum 16-bit) for the next LZMA algorithm, and for each
> available algorithm introduces an on-disk variable-array like below:
> bitmap(16-bit)    2       1       0
>                  ...     LZMA    LZ4
> __le16		compr_opt_off;      /* get the opt array start offset
>                                         (I think also in 4-byte) */
>
> compr alg 0 (lz4)	__le16	alg_opt_size;
> 	/* next opt off = roundup(off + alg_opt_size, 4); */
> 			__le16	lz4_max_distance;
>
> /* 4-byte aligned */
> compr alg x (if available)	u8	alg_opt_size;
> 				...
>
> ...
>
> When reading sb, first, it scans the whole bitmap, and get all the
> available algorithms in the image at once. And then read such compr
> opts one-by-one.
>
> Do you have some interest and extra time to implement it? :) That
> makes me work less since I'm debugging mbpcluster compression now...

Sounds good, I will try to do this part of the work.

Thanks,

Jianan

> Thanks,
> Gao Xiang