cross-compiling vm, confusion over output directory

[Bird, Tim]

I was working on fixing the cross-compilation for the selftests/vm tests.
Currently, there are two issues in my testing:

1) problem: required library missing from some cross-compile environments:
tools/testing/selftests/vm/mlock-random-test.c requires libcap
be installed.  The target line for mlock-random-test in
tools/testing/selftests/vm/Makefile looks like this:

$(OUTPUT)/mlock-random-test: LDLIBS += -lcap

and mlock-random-test.c has this include line:
#include <sys/capability.h>

this is confusing, since this is different from the header file
linux/capability.h.  It is associated with the capability library (libcap)
and not the kernel.  In any event, on some distros and in some
cross-compile SDKs the package containing these files is not installed
by default.

Once this library is installed, things progress farther.  Using an Ubuntu
system, you can install the cross version of this library (for arm64) by doing:
$ sudo apt install libcap-dev:arm64

1) solution:
I would like to add some meta-data about this build dependency, by putting
something in the settings file as a hint to CI build systems.  Specifically, I'd like to
create the file 'tools/testing/selftests/vm/settings', with the content:
NEED_LIB=cap

We already use settings for other meta-data about a test (right now, just a
non-default timeout value), but I don't want to create a new file or syntax
for this build dependency data.

Let me know what you think.

I may follow up with some script in the kernel source tree to check these
dependencies, independent of any CI system.  I have such a script in Fuego
that I could submit, but it would need some work to fit into the kernel build
flow for kselftest.  The goal would be to provide a nicely formatted warning,
with a recommendation for a package install.  But that's more work than
I think is needed right now just to let developers know there's a build dependency
here.

2) problem: reference to source-relative header file
the Makefile for vm uses a relative path for include directories.
Specifically, it has the line:
CFLAGS = -Wall -I ../../../../../usr/include $(EXTRA_CFLAGS)

I believe this needs to reference kernel include files from the
output directory, not the source directory.

With the relative include directory path, the program userfaultfd.c
gets compilation error like this:

userfaultfd.c:267:21: error: 'UFFD_API_RANGE_IOCTLS_BASIC' undeclared here (not in a function)
  .expected_ioctls = UFFD_API_RANGE_IOCTLS_BASIC,
                     ^
userfaultfd.c: In function 'uffd_poll_thread':
userfaultfd.c:529:8: error: 'UFFD_EVENT_FORK' undeclared (first use in this function)
   case UFFD_EVENT_FORK:
        ^
userfaultfd.c:529:8: note: each undeclared identifier is reported only once for each function it appears in
userfaultfd.c:531:18: error: 'union <anonymous>' has no member named 'fork'
    uffd = msg.arg.fork.ufd;
                  ^

2) incomplete solution:
I originally changed this line to read:
CFLAGS = -Wall -I $(KBUILD_OUTPUT)/usr/include $(EXTRA_CFLAGS)

This works when the output directory is specified using KBUILD_OUTPUT,
but not when the output directory is specified using O=
I'm not sure what happens when the output directory is specified
with a non-source-tree current working directory.

In any event, while researching a proper solution to this, I found
the following in tools/testing/selftests/Makefile:

If compiling with ifneq ($(O),)
    BUILD := $(O)
else
    ifneq ($(KBUILD_OUTPUT),)
        BUILD := $(KBUILD_OUTPUT)/kselftest
    else
        BUILD := $(shell pwd)
        DEFAULT_INSTALL_HDR_PATH := 1
    endif
endif

This doesn't seem right.  It looks like the selftests Makefile treats a directory
passed in using O= different from one specified using KBUILD_OUTPUT
or the current working directory.
In the KBUILD_OUTPUT case, you get an extra 'kselftest' directory layer
that you don't get for the other two.

In contrast, the kernel top-level Makefile has this:
ifeq ("$(origin O)", "command line")
  KBUILD_OUTPUT := $(O)
endif
(and from then on, the top-level Makefile appears to only use KBUILD_OUTPUT)

This makes it look like the rest of the kernel build system treats O= and KBUILD_OUTPUT
identically.

Am I missing something, or is there a flaw in the O=/KBUILD_OUTPUT handling in
kselftest?  Please let me know and I'll try to work out an appropriate fix for
cross-compiling the vm tests.
 -- Tim