Re: [PATCH v2 1/2] kbuild: make single targets work more correctly

[Masahiro Yamada]

On Thu, Aug 15, 2019 at 12:19 AM Masahiro Yamada
<yamada.masahiro@xxxxxxxxxxxxx> wrote:
>
> Currently, the single target build directly descends into the directory
> of the target. For example,
>
>   $ make foo/bar/baz.o
>
> ... directly descends into foo/bar/.
>
> On the other hand, the normal build usually descends one directory at
> a time, i.e. descends into foo/, and then foo/bar/.
>
> This difference causes some problems.
>
> [1] miss subdir-asflags-y, subdir-ccflags-y in upper Makefiles
>
>     The options in subdir-{as,cc}flags-y take effect in the current
>     and its sub-directories. In other words, they are inherited
>     downward. In the example above, the single target will miss
>     subdir-{as,cc}flags-y if they are defined in foo/Makefile.
>
> [2] could be built in a different directory
>
>     As Documentation/kbuild/modules.rst section 4.3 says, Kbuild can
>     handle files that are spread over several sub-directories.
>
>     The build rule of foo/bar/baz.o may not necessarily be specified in
>     foo/bar/Makefile. It might be specifies in foo/Makefile as follows:
>
>     [foo/Makefile]
>     obj-y := bar/baz.o
>
>     This often happens when a module is so big that its source files
>     are divided into sub-directories.
>
>     In this case, there is no Makefile in the foo/bar/ directory, yet
>     the single target descends into foo/bar/, then fails due to the
>     missing Makefile. You can still do 'make foo/bar/' for partial
>     building, but cannot do 'make foo/bar/baz.s'. I believe the single
>     target '%.s' is a useful feature for inspecting the compiler output.
>
>     Some modules work around this issue by putting an empty Makefile
>     in every sub-directory.
>
> This commit fixes those problems by making the single target build
> descend in the same way as the normal build does.
>
> Another change is the single target build will observe the CONFIG
> options. Previously, it allowed users to build the foo.o even when
> the corresponding CONFIG_FOO is disabled:
>
>    obj-$(CONFIG_FOO) += foo.o
>
> In the new behavior, the single target build will just fail and show
> "No rule to make target ..." (or "Nothing to be done for ..." if the
> stale object already exists, but cannot be updated).
>
> The disadvantage of this commit is the build speed. Now that the
> single target build visits every directory and parses lots of
> Makefiles, it is slower than before. (But, I hope it will not be
> too slow.)
>
> Signed-off-by: Masahiro Yamada <yamada.masahiro@xxxxxxxxxxxxx>
> ---

Both applied.


> Changes in v2:
>  - Fix single targets for external module build
>
>  Makefile               | 84 ++++++++++++++++++++++++++----------------
>  scripts/Makefile.build | 45 +++++++++++++++++++---
>  2 files changed, 92 insertions(+), 37 deletions(-)
>
> diff --git a/Makefile b/Makefile
> index 9661fa37158f..164ca615e2f6 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -230,6 +230,8 @@ endif
>
>  export KBUILD_CHECKSRC KBUILD_EXTMOD
>
> +extmod-prefix = $(if $(KBUILD_EXTMOD),$(KBUILD_EXTMOD)/)
> +
>  ifeq ($(abs_srctree),$(abs_objtree))
>          # building in the source tree
>          srctree := .
> @@ -271,11 +273,16 @@ no-dot-config-targets := $(clean-targets) \
>                          %asm-generic kernelversion %src-pkg
>  no-sync-config-targets := $(no-dot-config-targets) install %install \
>                            kernelrelease
> +single-targets := %.a %.i %.ko %.lds %.lst %.mod %.o %.s %.symtypes %/
> +ifdef CONFIG_CC_IS_CLANG
> +single-targets += %.ll
> +endif
>
>  config-build   :=
>  mixed-build    :=
>  need-config    := 1
>  may-sync-config        := 1
> +single-build   :=
>
>  ifneq ($(filter $(no-dot-config-targets), $(MAKECMDGOALS)),)
>         ifeq ($(filter-out $(no-dot-config-targets), $(MAKECMDGOALS)),)
> @@ -302,6 +309,14 @@ ifeq ($(KBUILD_EXTMOD),)
>          endif
>  endif
>
> +# We cannot build single targets and the others at the same time
> +ifneq ($(filter $(single-targets), $(MAKECMDGOALS)),)
> +       single-build := 1
> +       ifneq ($(filter-out $(single-targets), $(MAKECMDGOALS)),)
> +               mixed-build := 1
> +       endif
> +endif
> +
>  # For "make -j clean all", "make -j mrproper defconfig all", etc.
>  ifneq ($(filter $(clean-targets),$(MAKECMDGOALS)),)
>          ifneq ($(filter-out $(clean-targets),$(MAKECMDGOALS)),)
> @@ -1003,7 +1018,7 @@ endif
>
>  PHONY += prepare0
>
> -export MODORDER := $(if $(KBUILD_EXTMOD),$(KBUILD_EXTMOD)/)modules.order
> +export MODORDER := $(extmod-prefix)modules.order
>
>  ifeq ($(KBUILD_EXTMOD),)
>  core-y         += kernel/ certs/ mm/ fs/ ipc/ security/ crypto/ block/
> @@ -1655,7 +1670,7 @@ endif # KBUILD_EXTMOD
>  PHONY += descend $(build-dirs)
>  descend: $(build-dirs)
>  $(build-dirs): prepare
> -       $(Q)$(MAKE) $(build)=$@ need-builtin=1 need-modorder=1
> +       $(Q)$(MAKE) $(build)=$@ single-build=$(single-build) need-builtin=1 need-modorder=1
>
>  clean-dirs := $(addprefix _clean_, $(clean-dirs))
>  PHONY += $(clean-dirs) clean
> @@ -1752,40 +1767,47 @@ tools/%: FORCE
>
>  # Single targets
>  # ---------------------------------------------------------------------------
> -# Single targets are compatible with:
> -# - build with mixed source and output
> -# - build with separate output dir 'make O=...'
> -# - external modules
> +# To build individual files in subdirectories, you can do like this:
> +#
> +#   make foo/bar/baz.s
> +#
> +# The supported suffixes for single-target are listed in 'single-targets'
>  #
> -#  target-dir => where to store outputfile
> -#  build-dir  => directory in kernel source tree to use
> -
> -build-target = $(if $(KBUILD_EXTMOD), $(KBUILD_EXTMOD)/)$@
> -build-dir = $(patsubst %/,%,$(dir $(build-target)))
> -
> -%.i: prepare FORCE
> -       $(Q)$(MAKE) $(build)=$(build-dir) $(build-target)
> -%.ll: prepare FORCE
> -       $(Q)$(MAKE) $(build)=$(build-dir) $(build-target)
> -%.lst: prepare FORCE
> -       $(Q)$(MAKE) $(build)=$(build-dir) $(build-target)
> -%.o: prepare FORCE
> -       $(Q)$(MAKE) $(build)=$(build-dir) $(build-target)
> -%.s: prepare FORCE
> -       $(Q)$(MAKE) $(build)=$(build-dir) $(build-target)
> -%.symtypes: prepare FORCE
> -       $(Q)$(MAKE) $(build)=$(build-dir) $(build-target)
> +# To build only under specific subdirectories, you can do like this:
> +#
> +#   make foo/bar/baz/
> +
> +ifdef single-build
> +
> +single-all := $(filter $(single-targets), $(MAKECMDGOALS))
> +
> +# .ko is special because modpost is needed
> +single-ko := $(sort $(filter %.ko, $(single-all)))
> +single-no-ko := $(sort $(patsubst %.ko,%.mod, $(single-all)))
> +
> +$(single-ko): single_modpost
> +       @:
> +$(single-no-ko): descend
> +       @:
> +
>  ifeq ($(KBUILD_EXTMOD),)
> -# For the single build of an in-tree module, use a temporary file to avoid
> +# For the single build of in-tree modules, use a temporary file to avoid
>  # the situation of modules_install installing an invalid modules.order.
> -%.ko: MODORDER := .modules.tmp
> +MODORDER := .modules.tmp
>  endif
> -%.ko: prepare FORCE
> -       $(Q)$(MAKE) $(build)=$(build-dir) $(build-target:.ko=.mod)
> -       $(Q)echo $(build-target) > $(MODORDER)
> +
> +PHONY += single_modpost
> +single_modpost: $(single-no-ko)
> +       $(Q){ $(foreach m, $(single-ko), echo $(extmod-prefix)$m;) } > $(MODORDER)
>         $(Q)$(MAKE) -f $(srctree)/scripts/Makefile.modpost
> -%/: prepare FORCE
> -       $(Q)$(MAKE) KBUILD_MODULES=1 $(build)=$(build-dir) need-modorder=1
> +
> +KBUILD_MODULES := 1
> +
> +export KBUILD_SINGLE_TARGETS := $(addprefix $(extmod-prefix), $(single-no-ko))
> +
> +single-build = $(if $(filter-out $@/, $(single-no-ko)),1)
> +
> +endif
>
>  # FIXME Should go into a make.lib or something
>  # ===========================================================================
> diff --git a/scripts/Makefile.build b/scripts/Makefile.build
> index f84ccca8d74f..10adf3b558de 100644
> --- a/scripts/Makefile.build
> +++ b/scripts/Makefile.build
> @@ -52,7 +52,7 @@ ifndef obj
>  $(warning kbuild: Makefile.build is included improperly)
>  endif
>
> -ifeq ($(MAKECMDGOALS)$(need-modorder),)
> +ifeq ($(need-modorder),)
>  ifneq ($(obj-m),)
>  $(warning $(patsubst %.o,'%.ko',$(obj-m)) will not be built even though obj-m is specified.)
>  $(warning You cannot use subdir-y/m to visit a module Makefile. Use obj-y/m instead.)
> @@ -76,11 +76,6 @@ endif
>
>  mod-targets := $(patsubst %.o, %.mod, $(obj-m))
>
> -__build: $(if $(KBUILD_BUILTIN),$(builtin-target) $(lib-target) $(extra-y)) \
> -        $(if $(KBUILD_MODULES),$(obj-m) $(mod-targets) $(modorder-target)) \
> -        $(subdir-ym) $(always)
> -       @:
> -
>  # Linus' kernel sanity checking tool
>  ifeq ($(KBUILD_CHECKSRC),1)
>    quiet_cmd_checksrc       = CHECK   $<
> @@ -487,12 +482,50 @@ targets += $(call intermediate_targets, .asn1.o, .asn1.c .asn1.h) \
>            $(call intermediate_targets, .lex.o, .lex.c) \
>            $(call intermediate_targets, .tab.o, .tab.c .tab.h)
>
> +# Build
> +# ---------------------------------------------------------------------------
> +
> +ifdef single-build
> +
> +curdir-single := $(sort $(foreach x, $(KBUILD_SINGLE_TARGETS), \
> +                       $(if $(filter $(x) $(basename $(x)).o, $(targets)), $(x))))
> +
> +# Handle single targets without any rule: show "Nothing to be done for ..." or
> +# "No rule to make target ..." depending on whether the target exists.
> +unknown-single := $(filter-out $(addsuffix /%, $(subdir-ym)), \
> +                       $(filter $(obj)/%, \
> +                               $(filter-out $(curdir-single), \
> +                                       $(KBUILD_SINGLE_TARGETS))))
> +
> +__build: $(curdir-single) $(subdir-ym)
> +ifneq ($(unknown-single),)
> +       $(Q)$(MAKE) -f /dev/null $(unknown-single)
> +endif
> +       @:
> +
> +ifeq ($(curdir-single),)
> +# Nothing to do in this directory. Do not include any .*.cmd file for speed-up
> +targets :=
> +else
> +targets += $(curdir-single)
> +endif
> +
> +else
> +
> +__build: $(if $(KBUILD_BUILTIN),$(builtin-target) $(lib-target) $(extra-y)) \
> +        $(if $(KBUILD_MODULES),$(obj-m) $(mod-targets) $(modorder-target)) \
> +        $(subdir-ym) $(always)
> +       @:
> +
> +endif
> +
>  # Descending
>  # ---------------------------------------------------------------------------
>
>  PHONY += $(subdir-ym)
>  $(subdir-ym):
>         $(Q)$(MAKE) $(build)=$@ \
> +       $(if $(filter $@/, $(KBUILD_SINGLE_TARGETS)),single-build=) \
>         need-builtin=$(if $(filter $@/built-in.a, $(subdir-obj-y)),1) \
>         need-modorder=$(if $(need-modorder),$(if $(filter $@/modules.order, $(modorder)),1))
>
> --
> 2.17.1
>


-- 
Best Regards
Masahiro Yamada