Re: [RFC 00/19] Integration of Kernel Test Framework (KTF) into the kernel tree
From: Brendan Higgins
Date: Tue Aug 13 2019 - 04:10:49 EST
On Mon, Aug 12, 2019 at 11:11 PM Knut Omang <knut.omang@xxxxxxxxxx> wrote:
>
> KTF has already been available for a while as a separate git repository with
> means to facilitate use with any kernel version.
>
> KTF can be used both for "pure" unit testing and for more pragmatic
> approaches to component testing. Apart from some useful features that
> KTF uses from the kernel toolbox (such as kallsyms, kprobes),
> KTF does not depend on any special environment such as UML or on a
> large set of mocked up APIs to be useful. KTF basically allows test
> code to be inserted and managed as separate kernel modules, while
> providing the tests convenient access to almost the full range of kernel
> APIs, both exposed and private. And once a KTF test set exists, it
> should be fairly easy to compile and run it against older versions of
> the kernel as well.
>
> This series proposes a non-intrusive integration of KTF into the kernel
> hopefully presented in digestable pieces.
> For convenience, the patch set is also available on top of v5.2
> at https://github.com/knuto/linux/pull/new/ktf_v1 .
>
> The high level structure of the KTF code is as follows:
>
> External dependencies for the user land side:
> * libnl3 for netlink communication
> * googletest for test runner, reporting and test selection support.
So I take it that KTF depends on a fully booted kernel with an up and
running userspace?
> Kernel components:
> * Simple core test and test suite related abstractions:
> core data structures ktf_case, ktf_test, an assertion macro "infrastructure"
> with ASSERT_* and EXPECT_* macros and helper functions
> * Bookkeeping data structures:
> - ktf_map - a (key, value) mapping container used to implement management
> of instances of the higher level abstraction needs, such as ktf_handle and ktf_context.
> - ktf_handle: A global environment that a test runs within.
> - ktf_context: a test suite specific abstraction to allow a test to execute within
> one or more contexts. An example use of context can be a device
> instance: A test can be instantiated to run on all available such
> devices, according to test suite defined criteria.
> * A generic netlink protocol (ktf_nl, ktf_unlproto) which defines operations to
> allow a user space part of a test to query the kernel for test information,
> invoke tests and get feedback reports about results.
> * An alternative debugfs interface to allow examining and executing kernel-only tests
> without a user level program.
> * Support for overriding and modifying behaviour of kernel calls.
>
> User mode components:
> * A test executor based on and integrated with Googletest.
> Googletest is one of several mature user land unit test suites for
> C/C++. The choice allowed us to focus on kernel specific
> functionality rather than having to reinvent too many wheels.
> * Tools to aid in creating new test modules (suites):
> To facilitate a developer friendly way of testing internals of a module or the
> kernel itself, one of the important features of KTF, we often need to access
> symbols deliberately not exposed from a module.
> KTF contains a script used to create definitions based on kallsyms
> lookup for easy access to symbols not exposed by a module or the kernel.
> The user just provides a simple text file with a list of the symbols by
> module.
>
> This series is an attempt to address feedback from several people
> that having the functionality readily available within the kernel repository
> is desired.
>
> An in-tree KTF allows test suites to be provided with the kernel, which makes
> it easy to use KTF based suites as part of the kernel selftests and driver
> test suites. Having the ability to still build and run the latest versions of
> test suites against older kernels should be of great value to stable maintainers,
> distributions and driver maintainers, who would want to have an easy path,
> with minimal backporting efforts to make sure that criterias implemented by
> new test logic is also valid for these kernels.
>
> Our definite goal moving forward is to try to satisfy both needs in a
> transparent way. The plan is to let the standalone KTF repository follow the
> in-kernel one, and to allow test suites to be maintained similarly,
> and to support maintenance by proper tooling.
>
> Mode of integration into the kernel
> ===================================
>
> One feature of KTF is that it allows tests to work, look and feel similar
> whether they execute entirely in user mode, entirely in kernel mode,
> or half and half (hybrid tests). KTF consist of both user space
> and kernel code. Unlike e.g. kselftest, KTF in the Github version
> does not attempt to address the test runner aspects of testing.
>
> Due to the need for building modules, KTF requires access to kernel module
> build facilities (obj-m). But KTF also has nontrivial needs for user
> land building, and we think it is good to keep the build structure in a way that
> allows KTF to be built both in-tree and out-of-tree without
> necessarily having to reconfigure the kernel.
>
> This first version of kernel integration of KTF solves this challenge
> by co-locating everything associated with KTF under ktf/ as in the
> github version, but use the little used hostprogs-y and hostlibs-y
> features to build the user space side. The first patch in the series is
> fixes to make it work in a natural way to suit our needs.
>
> Positioning for natural building within the kernel tree
> =======================================================
>
> Currently we find significant amount of C level tests within the following paths:
>
> tools/testing/selftests/ (kselftests, almost entirely user space so far)
> lib/ (various kernel level mostly unit tests)
>
> and in the making::
>
> kunit/ (kernel only (UML))
>
> So all kernel code are currently located directly within the kernel
> build paths, accessed from the top level Makefile, to allow everything
> to be controlled by config and from the main build targets for the
> kernel. But this also poses challenges, in that .config has to be
> modified to build tests. And once a .config is changed, we no longer
> in principle logically operate on the same kernel.
>
> A better approach seems to be to follow the principle
> taken by kselftest: To have all the logic associated with the test
> inside the test tree, and make it available for building separately
> from the kernel itself. This require us to have a means to build
> kernel modules from within the test tree, separately from the main
> kernel paths. Currently this seems to only by supported via the M=
> option used to build out-of-tree modules. This was also easy to get to work
> for the kernel parts, based on the Github version of KTF, where we
> already do this. With the additional need to compile user land code,
> using the corresponding hostprogs-y and hostNNlibs-y seemed natural,
> but this has been challenging: The build macros does not really
> support hostprogs-y etc as "first class citizens" so some amount of
> hacking is in there in this first draft version.
> Using hostprogs-y etc is also a different approach that what is used
> for C code in kselftest today, but we imagine that there's room for
> unification here to get the best of both worlds, with the help of
> the wider Kbuild community.
>
> As an initial proposal, we have positioned ktf as an additional
> kselftest target, under tools/testing/selftests/ktf, and the recommended:
>
> make TARGETS="ktf" kselftest
>
> way of building and running should work, even from normal user accounts,
> if the user running it has sudo rights (for the kernel module insertion
> and removal). This will run the selftests for KTF itself, and should
> be a good starting point for adding more test cases. We also have had
> activities going to take some of the existing test suites under lib/
> and convert them into KTF based test suites, and will get back to this
> later.
>
> A trimmed down output from the above make target would look like this:
>
> ...
> CC [M] tools/testing/selftests/ktf/kernel/ktf_override.o
> LD [M] tools/testing/selftests/ktf/kernel/ktf.o
> HOSTCC -fPIC tools/testing/selftests/ktf/lib/ktf_unlproto.o
> HOSTCXX -fPIC tools/testing/selftests/ktf/lib/ktf_int.o
> KTFSYMS tools/testing/selftests/ktf/selftest/ktf_syms.h
> CC [M] tools/testing/selftests/ktf/selftest/self.o
> LD [M] tools/testing/selftests/ktf/selftest/selftest.o
> HOSTCXX tools/testing/selftests/ktf/user/ktftest.o
> HOSTCXX tools/testing/selftests/ktf/user/hybrid.o
> HOSTLD tools/testing/selftests/ktf/user/ktftest
> Building modules, stage 2.
> MODPOST 7 modules
> LD [M] tools/testing/selftests/ktf/kernel/ktf.ko
> LD [M] tools/testing/selftests/ktf/selftest/selftest.ko
> running tests
> make BUILD=/net/abi/local/abi/build/kernel/ktf/tools/testing/selftests -f scripts/runtests.mk run_tests
> TAP version 13
> 1..1
> ...
> ok 1 selftests: ktf: runtests.sh
>
> We're looking forward to feedback on this, and also to more discussion
> around unit testing at the testing & fuzzing workshop at LPC!
Sounds good! Glad to have this on the lists!
> Alan Maguire (3):
> ktf: Implementation of ktf support for overriding function entry and return.
> ktf: A simple debugfs interface to test results
> ktf: Simple coverage support
>
> Knut Omang (16):
> kbuild: Fixes to rules for host-cshlib and host-cxxshlib
> ktf: Introduce the main part of the kernel side of ktf
> ktf: Introduce a generic netlink protocol for test result communication
> ktf: An implementation of a generic associative array container
> ktf: Configurable context support for network info setup
> ktf: resolve: A helper utility to aid in exposing private kernel symbols to KTF tests.
> ktf: Add documentation for Kernel Test Framework (KTF)
> ktf: Add a small test suite with a few tests to test KTF itself
> ktf: Main part of user land library for executing tests
> ktf: Integration logic for running ktf tests from googletest
> ktf: Internal debugging facilities
> ktf: Some simple examples
> ktf: Some user applications to run tests
> ktf: Toplevel ktf Makefile/makefile includes and scripts to run from kselftest
> kselftests: Enable building ktf
> Documentation/dev-tools: Add index entry for KTF documentation
>
> Documentation/dev-tools/index.rst | 1 +-
> Documentation/dev-tools/ktf/concepts.rst | 242 +++-
> Documentation/dev-tools/ktf/debugging.rst | 248 +++-
> Documentation/dev-tools/ktf/examples.rst | 26 +-
> Documentation/dev-tools/ktf/features.rst | 307 ++++-
> Documentation/dev-tools/ktf/implementation.rst | 70 +-
> Documentation/dev-tools/ktf/index.rst | 14 +-
> Documentation/dev-tools/ktf/installation.rst | 73 +-
> Documentation/dev-tools/ktf/introduction.rst | 134 ++-
> Documentation/dev-tools/ktf/progref.rst | 144 ++-
> scripts/Makefile.host | 17 +-
> tools/testing/selftests/Makefile | 1 +-
> tools/testing/selftests/ktf/Makefile | 21 +-
> tools/testing/selftests/ktf/examples/Makefile | 17 +-
> tools/testing/selftests/ktf/examples/h2.c | 45 +-
> tools/testing/selftests/ktf/examples/h3.c | 84 +-
> tools/testing/selftests/ktf/examples/h4.c | 62 +-
> tools/testing/selftests/ktf/examples/hello.c | 38 +-
> tools/testing/selftests/ktf/examples/kgdemo.c | 61 +-
> tools/testing/selftests/ktf/kernel/Makefile | 15 +-
> tools/testing/selftests/ktf/kernel/ktf.h | 604 +++++++-
> tools/testing/selftests/ktf/kernel/ktf_context.c | 409 +++++-
> tools/testing/selftests/ktf/kernel/ktf_cov.c | 690 ++++++++-
> tools/testing/selftests/ktf/kernel/ktf_cov.h | 94 +-
> tools/testing/selftests/ktf/kernel/ktf_debugfs.c | 356 ++++-
> tools/testing/selftests/ktf/kernel/ktf_debugfs.h | 34 +-
> tools/testing/selftests/ktf/kernel/ktf_map.c | 261 +++-
> tools/testing/selftests/ktf/kernel/ktf_map.h | 154 ++-
> tools/testing/selftests/ktf/kernel/ktf_netctx.c | 132 ++-
> tools/testing/selftests/ktf/kernel/ktf_netctx.h | 64 +-
> tools/testing/selftests/ktf/kernel/ktf_nl.c | 516 ++++++-
> tools/testing/selftests/ktf/kernel/ktf_nl.h | 15 +-
> tools/testing/selftests/ktf/kernel/ktf_override.c | 45 +-
> tools/testing/selftests/ktf/kernel/ktf_override.h | 15 +-
> tools/testing/selftests/ktf/kernel/ktf_test.c | 397 +++++-
> tools/testing/selftests/ktf/kernel/ktf_test.h | 381 ++++-
> tools/testing/selftests/ktf/kernel/ktf_unlproto.h | 105 +-
> tools/testing/selftests/ktf/lib/Makefile | 21 +-
> tools/testing/selftests/ktf/lib/ktf.h | 114 +-
> tools/testing/selftests/ktf/lib/ktf_debug.cc | 20 +-
> tools/testing/selftests/ktf/lib/ktf_debug.h | 59 +-
> tools/testing/selftests/ktf/lib/ktf_int.cc | 1031 ++++++++++++-
> tools/testing/selftests/ktf/lib/ktf_int.h | 84 +-
> tools/testing/selftests/ktf/lib/ktf_run.cc | 177 ++-
> tools/testing/selftests/ktf/lib/ktf_unlproto.c | 21 +-
> tools/testing/selftests/ktf/scripts/ktf_syms.mk | 16 +-
> tools/testing/selftests/ktf/scripts/resolve | 188 ++-
> tools/testing/selftests/ktf/scripts/runtests.mk | 3 +-
> tools/testing/selftests/ktf/scripts/runtests.sh | 100 +-
> tools/testing/selftests/ktf/scripts/top_make.mk | 14 +-
> tools/testing/selftests/ktf/selftest/Makefile | 17 +-
> tools/testing/selftests/ktf/selftest/context.c | 149 ++-
> tools/testing/selftests/ktf/selftest/context.h | 15 +-
> tools/testing/selftests/ktf/selftest/context_self.h | 34 +-
> tools/testing/selftests/ktf/selftest/hybrid.c | 35 +-
> tools/testing/selftests/ktf/selftest/hybrid.h | 24 +-
> tools/testing/selftests/ktf/selftest/hybrid_self.h | 27 +-
> tools/testing/selftests/ktf/selftest/ktf_syms.txt | 17 +-
> tools/testing/selftests/ktf/selftest/self.c | 661 ++++++++-
> tools/testing/selftests/ktf/user/Makefile | 26 +-
> tools/testing/selftests/ktf/user/hybrid.cc | 39 +-
> tools/testing/selftests/ktf/user/ktfcov.cc | 68 +-
> tools/testing/selftests/ktf/user/ktfrun.cc | 20 +-
> tools/testing/selftests/ktf/user/ktftest.cc | 46 +-
> 64 files changed, 8909 insertions(+), 9 deletions(-)
> create mode 100644 Documentation/dev-tools/ktf/concepts.rst
> create mode 100644 Documentation/dev-tools/ktf/debugging.rst
> create mode 100644 Documentation/dev-tools/ktf/examples.rst
> create mode 100644 Documentation/dev-tools/ktf/features.rst
> create mode 100644 Documentation/dev-tools/ktf/implementation.rst
> create mode 100644 Documentation/dev-tools/ktf/index.rst
> create mode 100644 Documentation/dev-tools/ktf/installation.rst
> create mode 100644 Documentation/dev-tools/ktf/introduction.rst
> create mode 100644 Documentation/dev-tools/ktf/progref.rst
> create mode 100644 tools/testing/selftests/ktf/Makefile
> create mode 100644 tools/testing/selftests/ktf/examples/Makefile
> create mode 100644 tools/testing/selftests/ktf/examples/h2.c
> create mode 100644 tools/testing/selftests/ktf/examples/h3.c
> create mode 100644 tools/testing/selftests/ktf/examples/h4.c
> create mode 100644 tools/testing/selftests/ktf/examples/hello.c
> create mode 100644 tools/testing/selftests/ktf/examples/kgdemo.c
> create mode 100644 tools/testing/selftests/ktf/kernel/Makefile
> create mode 100644 tools/testing/selftests/ktf/kernel/ktf.h
> create mode 100644 tools/testing/selftests/ktf/kernel/ktf_context.c
> create mode 100644 tools/testing/selftests/ktf/kernel/ktf_cov.c
> create mode 100644 tools/testing/selftests/ktf/kernel/ktf_cov.h
> create mode 100644 tools/testing/selftests/ktf/kernel/ktf_debugfs.c
> create mode 100644 tools/testing/selftests/ktf/kernel/ktf_debugfs.h
> create mode 100644 tools/testing/selftests/ktf/kernel/ktf_map.c
> create mode 100644 tools/testing/selftests/ktf/kernel/ktf_map.h
> create mode 100644 tools/testing/selftests/ktf/kernel/ktf_netctx.c
> create mode 100644 tools/testing/selftests/ktf/kernel/ktf_netctx.h
> create mode 100644 tools/testing/selftests/ktf/kernel/ktf_nl.c
> create mode 100644 tools/testing/selftests/ktf/kernel/ktf_nl.h
> create mode 100644 tools/testing/selftests/ktf/kernel/ktf_override.c
> create mode 100644 tools/testing/selftests/ktf/kernel/ktf_override.h
> create mode 100644 tools/testing/selftests/ktf/kernel/ktf_test.c
> create mode 100644 tools/testing/selftests/ktf/kernel/ktf_test.h
> create mode 100644 tools/testing/selftests/ktf/kernel/ktf_unlproto.h
> create mode 100644 tools/testing/selftests/ktf/lib/Makefile
> create mode 100644 tools/testing/selftests/ktf/lib/ktf.h
> create mode 100644 tools/testing/selftests/ktf/lib/ktf_debug.cc
> create mode 100644 tools/testing/selftests/ktf/lib/ktf_debug.h
> create mode 100644 tools/testing/selftests/ktf/lib/ktf_int.cc
> create mode 100644 tools/testing/selftests/ktf/lib/ktf_int.h
> create mode 100644 tools/testing/selftests/ktf/lib/ktf_run.cc
> create mode 100644 tools/testing/selftests/ktf/lib/ktf_unlproto.c
> create mode 100644 tools/testing/selftests/ktf/scripts/ktf_syms.mk
> create mode 100755 tools/testing/selftests/ktf/scripts/resolve
> create mode 100644 tools/testing/selftests/ktf/scripts/runtests.mk
> create mode 100755 tools/testing/selftests/ktf/scripts/runtests.sh
> create mode 100644 tools/testing/selftests/ktf/scripts/top_make.mk
> create mode 100644 tools/testing/selftests/ktf/selftest/Makefile
> create mode 100644 tools/testing/selftests/ktf/selftest/context.c
> create mode 100644 tools/testing/selftests/ktf/selftest/context.h
> create mode 100644 tools/testing/selftests/ktf/selftest/context_self.h
> create mode 100644 tools/testing/selftests/ktf/selftest/hybrid.c
> create mode 100644 tools/testing/selftests/ktf/selftest/hybrid.h
> create mode 100644 tools/testing/selftests/ktf/selftest/hybrid_self.h
> create mode 100644 tools/testing/selftests/ktf/selftest/ktf_syms.txt
> create mode 100644 tools/testing/selftests/ktf/selftest/self.c
> create mode 100644 tools/testing/selftests/ktf/user/Makefile
> create mode 100644 tools/testing/selftests/ktf/user/hybrid.cc
> create mode 100644 tools/testing/selftests/ktf/user/ktfcov.cc
> create mode 100644 tools/testing/selftests/ktf/user/ktfrun.cc
> create mode 100644 tools/testing/selftests/ktf/user/ktftest.cc
>
> base-commit: 0ecfebd2b52404ae0c54a878c872bb93363ada36
> --
> git-series 0.9.1