Re: [PATCH 0/4] Fix ebizzy performance regression due to X86 TLBrange flush v2

[Mel Gorman]

On Mon, Dec 16, 2013 at 09:17:35AM -0800, Linus Torvalds wrote:
> On Mon, Dec 16, 2013 at 2:39 AM, Mel Gorman <mgorman@xxxxxxx> wrote:
> >
> > First was Alex's microbenchmark from https://lkml.org/lkml/2012/5/17/59
> > and ran it for a range of thread numbers, 320 iterations per thread with
> > random number of entires to flush. Results are from two machines
> 
> There's something wrong with that benchmark, it sometimes gets stuck,

It's not a thread-safe benchmark. The parent unmapping thread can finish
before the children start and it infinite loops.

> and the profile numbers are just random (and mostly in user space).
> 

Yep, it's why when I used it I ran a large number of iterations with
semi-randomised number of entries trying to knock some sense out of it.
I was hoping that the Intel folk might come back with more details on
what their testing methodology was.

> I think you mentioned fixing a bug in it, mind pointing at the fixed benchmark?
> 

Ugh, I'm embarassed by this. I did not properly fix the benchmark, just
bodged around the part that can lockup. Patch is below. Actual testing was
run using mmtests with the configs/config-global-dhp__tlbflush-performance
configuration file using something like this

# build boot kernel 1
./run-mmtests.sh --run-monitor --config configs/config-global-dhp__tlbflush-performance test-kernel-1
# build boot kernel 2
./run-mmtests.sh --run-monitor --config configs/config-global-dhp__tlbflush-performance test-kernel-2
cd work/log
../../compare-kernels.sh

> Looking at the kernel footprint, it seems to depend on what parameters
> you ran that benchmark with. Under certain loads, it seems to spend
> most of the time in clearing pages and in the page allocation ("-t 8
> -n 320"). And in other loads, it hits smp_call_function_many() and the
> TLB flushers ("-t 8 -n 8"). So exactly what parameters did you use?
> 

A range of parameters. The test effectively does this

TLBFLUSH_MAX_ENTRIES=256
for_each_thread_count
	for iteration in `seq 1 320`
        	# Select a range of entries to randomly select from. This is to ensure
        	# an evenish spread of entries to be tested
        	NR_SECTION=$((ITERATION%8))
        	RANGE=$((TLBFLUSH_MAX_ENTRIES/8))
        	THIS_MIN_ENTRIES=$((RANGE*NR_SECTION+1))
        	THIS_MAX_ENTRIES=$((THIS_MIN_ENTRIES+RANGE))

        	NR_ENTRIES=$((THIS_MIN_ENTRIES+(RANDOM%RANGE)))
        	if [ $NR_ENTRIES -gt $THIS_MAX_ENTRIES ]; then
                	NR_ENTRIES=$THIS_MAX_ENTRIES
        	fi

		RESULT=`tlbflush -n $NR_ENTRIES -t $NR_THREADS 2>&1`
	done
done

It splits the values for nr_entries (-n switch) into 8 segments and randomly
selects values within them. This results in noise but ensures the test hits
the best, average and worst cases for TLB range flushing. Writing this,
I realise I should have made MAX_ENTRIES 512 to hit the original shift
values. The original mail indicated that this test was run once for a very
limited number of threads and entries and I really hope this is not what
actually happened to tune that shift value.

> Because we've had things that change those two things (and they are
> totally independent).
> 

Indeed and tuning on specifics would be a bad idea -- hence why my
testing took a randomised selection of ranges to test with and a large
number of iterations.

> And does anything stand out in the profiles of ebizzy? For example, in
> between 3.4.x and 3.11, we've converted the anon_vma locking from a
> mutex to a rwsem, and we know that caused several issues, possibly
> causing unfairness. There are other potential sources of unfairness.
> It would be good to perhaps bisect things at least *somewhat*, because
> *so* much has changed in 3.4 to 3.11 that it's impossible to guess.
> 

I'll check. Right now, the machines are still occupied running bisections
which is still finding bugs. When that has found the obvious stuff, I'll use
profiles to identify what's left. FWIW, I would be surprised if ebizzy was
affected by the anon_vma locking. I do not think the threads are operating
within the same VMAs in a manner that would contend on those locks. If there
is a lock being contended, it's going to be on mmap_sem for creating mappings
just slightly larger than MMAP_THRESHOLD.  Guessing though, not proven.

This is bodge that stops Alex's benchmark locking up. It's the wrong way to
fix a problem like this. I was not even convinced this benchmark was useful
to begin with and was unmotivated to spending time on fixing it up properly.

--- tlbflush.c.orig	2013-12-15 11:05:08.813821030 +0000
+++ tlbflush.c	2013-12-15 11:04:46.504926426 +0000
@@ -67,13 +67,17 @@
 	char x;
 	int i, k;
 	int randn[PAGE_SIZE];
+	int count = 0;
 	
 	for (i=0;i<PAGE_SIZE; i++)
 		randn[i] = rand();
 
 	actimes = malloc(sizeof(long));
 
-	while (*threadstart == 0 )
+	while (*threadstart == 0) {
+		if (++count > 1000000)
+			break;
 		usleep(1);
+	}
 
 	if (d->rw == 0)
@@ -180,6 +181,7 @@
 	threadstart = malloc(sizeof(int));
 	*threadstart = 0;
 	data.readp = &p; data.startaddr = startaddr; data.rw = rw; data.loop = l;
+	sleep(1);
 	for (i=0; i< t; i++)
 		if(pthread_create(&pid[i], NULL, accessmm, &data))
 			perror("pthread create");


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/