Re: [RFC] SLAB : NUMA cache_free_alien() very expensive because ofvirt_to_slab(objp); nodeid = slabp->nodeid;

From: Christoph Lameter
Date: Tue Mar 20 2007 - 16:20:41 EST


On Tue, 20 Mar 2007, Eric Dumazet wrote:

>
> I noticed on a small x86_64 NUMA setup (2 nodes) that cache_free_alien() is very expensive.
> This is because of a cache miss on struct slab.
> At the time an object is freed (call to kmem_cache_free() for example), the underlying 'struct slab' is not anymore cache-hot.
>
> struct slab *slabp = virt_to_slab(objp);
> nodeid = slabp->nodeid; // cache miss
>
> So we currently need slab only to lookup nodeid, to be able to use the cachep cpu cache, or not.
>
> Couldn't we use something less expensive, like pfn_to_nid() ?

Nodeid describes the node that the slab was allocated for which may
not be equal to the node that the page came from. But if GFP_THISNODE is
used then this should always be the same. That those two nodeid are
different was certainly frequent before the GFP_THISNODE work went in. Now
it may just occur in corner cases. Perhaps during bootup on
some machines that boot with empty nodes. I vaguely recall a powerpc
issue.

> Is it possible virt_to_slab(objp)->nodeid being different from pfn_to_nid(objp) ?

It is possible the page allocator falls back to another node than
requested. We would need to check that this never occurs.

If we are sure then we could drop the nodeid field completely.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/