[PATCH] mm: Fix int overflow in callers of do_shrink_slab()

From: Kirill Tkhai
Date: Fri Sep 28 2018 - 07:29:54 EST


do_shrink_slab() returns unsigned long value, and
the placing into int variable cuts high bytes off.
Then we compare ret and 0xfffffffe (since SHRINK_EMPTY
is converted to ret type).

Thus, big number of objects returned by do_shrink_slab()
may be interpreted as SHRINK_EMPTY, if low bytes of
their value are equal to 0xfffffffe. Fix that
by declaration ret as unsigned long in these functions.

Reported-by: Cyrill Gorcunov <gorcunov@xxxxxxxxxx>
Signed-off-by: Kirill Tkhai <ktkhai@xxxxxxxxxxxxx>
---
mm/vmscan.c | 7 +++----
1 file changed, 3 insertions(+), 4 deletions(-)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index 0b63d9a2dc17..8ea87586925e 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -581,8 +581,8 @@ static unsigned long shrink_slab_memcg(gfp_t gfp_mask, int nid,
struct mem_cgroup *memcg, int priority)
{
struct memcg_shrinker_map *map;
- unsigned long freed = 0;
- int ret, i;
+ unsigned long ret, freed = 0;
+ int i;

if (!memcg_kmem_enabled() || !mem_cgroup_online(memcg))
return 0;
@@ -678,9 +678,8 @@ static unsigned long shrink_slab(gfp_t gfp_mask, int nid,
struct mem_cgroup *memcg,
int priority)
{
+ unsigned long ret, freed = 0;
struct shrinker *shrinker;
- unsigned long freed = 0;
- int ret;

if (!mem_cgroup_is_root(memcg))
return shrink_slab_memcg(gfp_mask, nid, memcg, priority);