Re: [PATCH 20/20] Get rid of the concept of hot/cold page freeing

[Mel Gorman]

On Tue, Feb 24, 2009 at 04:01:03PM -0800, Andrew Morton wrote:
> On Tue, 24 Feb 2009 11:51:26 +0000
> Mel Gorman <mel@xxxxxxxxx> wrote:
> 
> > > > Almost the opposite with steady improvements almost all the way through.
> > > > 
> > > > With the patch applied, we are still using hot/cold information on the
> > > > allocation side so I'm somewhat surprised the patch even makes much of a
> > > > difference. I'd have expected the pages being freed to be mostly hot.
> > > 
> > > Oh yeah.  Back in the ancient days, hot-cold-pages was using separate
> > > magazines for hot and cold pages.  Then Christoph went and mucked with
> > > it, using a single queue.  That might have affected things.
> > > 
> > 
> > It might have. The impact is that requests for cold pages can get hot pages
> > if there are not enough cold pages in the queue so readahead could prevent
> > an active process getting cache hot pages. I don't think that would have
> > showed up in the microbenchmark though.
> 
> We switched to doing non-temporal stores in copy_from_user(), didn't
> we? 

We do? I would have missed something like that but luckily I took a profile
of the microbenchmark and what do you know, we spent 17053 profiles samples in
__copy_user_nocache(). It's not quite copy_from_user() but it's close. Thanks
for pointing that out!

For anyone watching, copy_from_user() itself and the functions it calls do
not use non-temporal stores. At least, I am not seeing the nt variants of
mov in the assembly I looked at. __copy_user_nocache() on the other hand
uses movnt and it's called in the generic_file_buffered_write() path which
this micro-benchmark is optimising.

> That would rub out the benefit which that microbenchmark
> demonstrated?
> 

It'd impact it for sure. Due to the non-temporal stores, I'm surprised
there is any measurable impact from the patch.  This has likely been the
case since commit 0812a579c92fefa57506821fa08e90f47cb6dbdd. My reading of
this (someone correct/enlighten) is that even if the data was cache hot,
it is pushed out as a result of the non-temporal access.

The changelog doesn't give the reasoning for using uncached accesses but maybe
it's because for filesystem writes, it is not expected that the data will be
accessed by the CPU any more and the storage device driver has less work to
do to ensure the data in memory is not dirty in cache (this is speculation,
I don't know for sure what the expected benefit is meant to be but it might
be in the manual, I'll check later).

Thinking of alternative microbenchmarks that might show this up....

Repeated setup, populate and teardown of pagetables might show up something
as it should benefit if the pages were cache hot but the cost of faulting
and zeroing might hide it.

Maybe a microbenchmark that creates/deletes many small (or empty) files
or reads large directories might benefit from cache hotness as the slab
pages would have to be allocated, populated and then freed back to the
allocator. I'll give it a shot but alternative suggestions are welcome.

-- 
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/