[REGRESSION] 4.11-rc: systemd doesn't see most devices

[Theodore Ts'o]

There is a frustrating regression in 4.11 that I've been trying to
track down.  The symptoms are that a large number of systemd devices
don't show up.  So instead of "systemctl | grep .device | wc -l"
listing some 50+ lines which look like this:

sys-devices-pci0000:00-0000:00:14.0-usb1-1\x2d7-1\x2d7:1.0-bluetooth-hci0.device         loaded active plugged   /sys/devices/pci0000:00/0000:00:14.0/usb1/1-7/1-7:1.0/bluetooth/hci0

I only get 5-10 lines of devices.  This is problematic because it
means that the wifi firmware is not automatically loaded.  More
annoyingly, because the device mapper systemd devices are missing:

sys-devices-virtual-block-dm\x2d0.device                                                 loaded active plugged   /sys/devices/virtual/block/dm-0

... the boot hangs for 90 seconds because it can't fsck devices that
systemd doesn't think exists yet.  (I'm using LVM on top of an
encrypted block device, and it doesn't think the dm-crypt device is
created, although given that the root file system is an LVM volume,
obviously LVM and the LUKS setup had worked just fine --- and people
wonder why some folks hate systemd.  :-)

The failure past v4.10-5879-gcaa59428971d starts starts becoming
flaky, so sometimes I have to reboot three times or more before the
failure shows up.  This is why the bisect has been taking so long, and
so while I'm *faily* certain that the failure is somewhere in the
staging branch merge, it's possible that one of the earlier "git
bisect good"'s are in error.  I have been trying multiple reboots
before concluding that a bisection point is "good" but this takes a
huge amount of time, since having GRUB unlock an encrypted LVM volume
takes a long time, and I have to type the decryption password twice at
each boot.

The end of the bisection doesn't make any sense, and so at this point
I've given up, and am posting this to LKML with Linus and Greg cc'ed,
in the hopes that someone else has seen this, or understands what sort
of failure would cause systemd to not think various devices are
present and/or finished initializing.  I'm using a Debian testing
distribution, and it would be really good to figure out what the ?!@#
is going on, since if 4.11 releases with this, I suspect a lot of
people will be affected.  Unfortunately, while it's not particularly
reliable deep into the bisection, at -rc3 or -rc5 it's **damned**
reproducible.  I know how to work around the systemd brain damage for
now by using rc.local, futzing with the dependencies, manually loading
the Wifi module by hand, and (sometimes) living without Audio, but
this is requires a lot of hacking, and it's not, shall we say, a
particularly nice user experience.  :-(

      	      		     	       		    - Ted

P.S.  I've also attached the output of "systemd | grep devices" so you
can see what happens in a good and bad case, in case that helps.

git bisect start
# good: [v4.10] Linux 4.10
git bisect good c470abd4fde40ea6a0846a2beab642a578c0b8cd
# bad: [v4.10-5879-gcaa59428971d] Merge tag 'staging-4.11-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging
git bisect bad caa59428971d5ad81d19512365c9ba580d83268c
# good: [v4.10-2518-g1e74a2eb1f5c] Merge tag 'gcc-plugins-v4.11-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/kees/linux
git bisect good 1e74a2eb1f5cc7f2f2b5aa9c9eeecbcf352220a3
# good: [v4.10-4456-g3051bf36c25d] Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next
git bisect good 3051bf36c25d5153051704291782f8d44e744d36
# good: [v4.10-5190-ge30aee9e10bb] Merge tag 'char-misc-4.11-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/char-misc
git bisect good e30aee9e10bb5168579e047f05c3d13d09e23356
# good: [v4.10-5202-gb2064617c74f] Merge tag 'driver-core-4.11-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/driver-core
git bisect good b2064617c74f301dab1448f1f9c8dbb3c8021058
# good: [v4.10-rc3-325-g994261dc8f3d] devicetree: sort the Garmin vendor prefix properly.
git bisect good 994261dc8f3dfcd19feecafae3040e932c8f90cf
# good: [v4.10-rc3-501-gc2351249f140] staging: lustre: libcfs: avoid stomping on module param cpu_pattern
git bisect good c2351249f140aebdf911d86d7a1542c40b20fca3
# good: [v4.10-rc7-589-g1b2d7f198140] staging: comedi: dt2815: usleep_range is preferred over udelay
git bisect good 1b2d7f198140aae48dd93c41abde37283312a98c
# bad: [v4.10-rc7-633-g4d0bdcb10c43] staging: rtl8192e: Aligning the * on each line in block comments
git bisect bad 4d0bdcb10c43056489b69186ee43669f2a73b8f9
# bad: [v4.10-rc7-611-g30d69ada0771] Staging: rtl8192u: r819xU_firmware.c - style fix
git bisect bad 30d69ada0771ed25b63ec56887faea31d8f551bd
# good: [v4.10-rc7-600-g8a0e4b9e469c] staging: android: ion: fix coding style issue
git bisect good 8a0e4b9e469c7bbaa206d81e2e7515e1abb1aa00
# bad: [v4.10-rc7-605-gdc223652c6de] staging: r8712u: Fix macros used to read/write the TX/RX descriptors
git bisect bad dc223652c6de409fe5e073b0f631f0413a90e69f
# good: [v4.10-rc7-602-g76b94eb1eb88] staging: set msi_domain_ops as __ro_after_init
git bisect good 76b94eb1eb882e3a8beb51b7d287a21a38a56938
# bad: [v4.10-rc7-604-g221c46d28957] staging: rtl8712u: Fix endian settings for structs describing network packets
git bisect bad 221c46d28957bd6e2158abc2179ce4a8c9ce07d3
# bad: [v4.10-rc7-603-ge2288bce8ec8] staging: rtl8712: Fix some Sparse endian messages
git bisect bad e2288bce8ec828ce36f6443e298e04d41fddff7e
# first bad commit: [v4.10-rc7-603-ge2288bce8ec8] staging: rtl8712: Fix some Sparse endian messages
sys-devices-pci0000:00-0000:00:02.0-drm-card0-card0\x2deDP\x2d1-intel_backlight.device   loaded active plugged   /sys/devices/pci0000:00/0000:00:02.0/drm/card0/card0-eDP-1/intel_backlight     
sys-devices-pci0000:00-0000:00:14.0-usb1-1\x2d3-1\x2d3:1.0-net-wwp0s20f0u3.device        loaded active plugged   Sierra_Wireless_EM7345_4G_LTE                                                  
sys-devices-pci0000:00-0000:00:14.0-usb1-1\x2d3-1\x2d3:1.2-tty-ttyACM0.device            loaded active plugged   Sunrise Point-LP USB 3.0 xHCI Controller                                       
sys-devices-pci0000:00-0000:00:14.0-usb1-1\x2d5.device                                   loaded active plugged   AU9540 Smartcard Reader                                                        
sys-devices-pci0000:00-0000:00:14.0-usb1-1\x2d6-1\x2d6:1.0.device                        loaded active plugged   Yubikey NEO(-N) U2F+CCID                                                       
sys-devices-pci0000:00-0000:00:14.0-usb1-1\x2d6-1\x2d6:1.1.device                        loaded active plugged   Yubikey NEO(-N) U2F+CCID                                                       
sys-devices-pci0000:00-0000:00:14.0-usb1-1\x2d6.device                                   loaded active plugged   Yubikey NEO(-N) U2F+CCID                                                       
sys-devices-pci0000:00-0000:00:14.0-usb1-1\x2d7-1\x2d7:1.0-bluetooth-hci0.device         loaded active plugged   /sys/devices/pci0000:00/0000:00:14.0/usb1/1-7/1-7:1.0/bluetooth/hci0           
sys-devices-pci0000:00-0000:00:17.0-ata2-host1-target1:0:0-1:0:0:0-block-sda-sda1.device loaded active plugged   Samsung_SSD_850_PRO_2TB 1                                                      
sys-devices-pci0000:00-0000:00:17.0-ata2-host1-target1:0:0-1:0:0:0-block-sda-sda2.device loaded active plugged   Samsung_SSD_850_PRO_2TB 2                                                      
sys-devices-pci0000:00-0000:00:17.0-ata2-host1-target1:0:0-1:0:0:0-block-sda.device      loaded active plugged   Samsung_SSD_850_PRO_2TB                                                        
sys-devices-pci0000:00-0000:00:1c.2-0000:04:00.0-net-wlp4s0.device                       loaded active plugged   Wireless 8260                                                                  
sys-devices-pci0000:00-0000:00:1f.3-sound-card0.device                                   loaded active plugged   Sunrise Point-LP HD Audio                                                      
sys-devices-pci0000:00-0000:00:1f.6-net-enp0s31f6.device                                 loaded active plugged   Ethernet Connection I219-V                                                     
sys-devices-platform-serial8250-tty-ttyS0.device                                         loaded active plugged   /sys/devices/platform/serial8250/tty/ttyS0                                     
sys-devices-platform-serial8250-tty-ttyS1.device                                         loaded active plugged   /sys/devices/platform/serial8250/tty/ttyS1                                     
sys-devices-platform-serial8250-tty-ttyS2.device                                         loaded active plugged   /sys/devices/platform/serial8250/tty/ttyS2                                     
sys-devices-platform-serial8250-tty-ttyS3.device                                         loaded active plugged   /sys/devices/platform/serial8250/tty/ttyS3                                     
sys-devices-platform-thinkpad_acpi-leds-tpacpi::kbd_backlight.device                     loaded active plugged   /sys/devices/platform/thinkpad_acpi/leds/tpacpi::kbd_backlight                 
sys-devices-virtual-block-dm\x2d0.device                                                 loaded active plugged   /sys/devices/virtual/block/dm-0                                                
sys-devices-virtual-block-dm\x2d1.device                                                 loaded active plugged   /sys/devices/virtual/block/dm-1                                                
sys-devices-virtual-block-dm\x2d10.device                                                loaded active plugged   /sys/devices/virtual/block/dm-10                                               
sys-devices-virtual-block-dm\x2d11.device                                                loaded active plugged   /sys/devices/virtual/block/dm-11                                               
sys-devices-virtual-block-dm\x2d2.device                                                 loaded active plugged   /sys/devices/virtual/block/dm-2                                                
sys-devices-virtual-block-dm\x2d3.device                                                 loaded active plugged   /sys/devices/virtual/block/dm-3                                                
sys-devices-virtual-block-dm\x2d4.device                                                 loaded active plugged   /sys/devices/virtual/block/dm-4                                                
sys-devices-virtual-block-dm\x2d5.device                                                 loaded active plugged   /sys/devices/virtual/block/dm-5                                                
sys-devices-virtual-block-dm\x2d6.device                                                 loaded active plugged   /sys/devices/virtual/block/dm-6                                                
sys-devices-virtual-block-dm\x2d7.device                                                 loaded active plugged   /sys/devices/virtual/block/dm-7                                                
sys-devices-virtual-block-dm\x2d8.device                                                 loaded active plugged   /sys/devices/virtual/block/dm-8                                                
sys-devices-virtual-block-dm\x2d9.device                                                 loaded active plugged   /sys/devices/virtual/block/dm-9                                                
sys-devices-virtual-block-ram0.device                                                    loaded active plugged   /sys/devices/virtual/block/ram0                                                
sys-devices-virtual-block-ram1.device                                                    loaded active plugged   /sys/devices/virtual/block/ram1                                                
sys-devices-virtual-block-ram10.device                                                   loaded active plugged   /sys/devices/virtual/block/ram10                                               
sys-devices-virtual-block-ram11.device                                                   loaded active plugged   /sys/devices/virtual/block/ram11                                               
sys-devices-virtual-block-ram12.device                                                   loaded active plugged   /sys/devices/virtual/block/ram12                                               
sys-devices-virtual-block-ram13.device                                                   loaded active plugged   /sys/devices/virtual/block/ram13                                               
sys-devices-virtual-block-ram14.device                                                   loaded active plugged   /sys/devices/virtual/block/ram14                                               
sys-devices-virtual-block-ram15.device                                                   loaded active plugged   /sys/devices/virtual/block/ram15                                               
sys-devices-virtual-block-ram2.device                                                    loaded active plugged   /sys/devices/virtual/block/ram2                                                
sys-devices-virtual-block-ram3.device                                                    loaded active plugged   /sys/devices/virtual/block/ram3                                                
sys-devices-virtual-block-ram4.device                                                    loaded active plugged   /sys/devices/virtual/block/ram4                                                
sys-devices-virtual-block-ram5.device                                                    loaded active plugged   /sys/devices/virtual/block/ram5                                                
sys-devices-virtual-block-ram6.device                                                    loaded active plugged   /sys/devices/virtual/block/ram6                                                
sys-devices-virtual-block-ram7.device                                                    loaded active plugged   /sys/devices/virtual/block/ram7                                                
sys-devices-virtual-block-ram8.device                                                    loaded active plugged   /sys/devices/virtual/block/ram8                                                
sys-devices-virtual-block-ram9.device                                                    loaded active plugged   /sys/devices/virtual/block/ram9                                                
sys-devices-virtual-misc-rfkill.device                                                   loaded active plugged   /sys/devices/virtual/misc/rfkill                                               
sys-devices-virtual-net-docker0.device                                                   loaded active plugged   /sys/devices/virtual/net/docker0                                               
sys-module-fuse.device                                                                   loaded active plugged   /sys/module/fuse                                                               
sys-subsystem-bluetooth-devices-hci0.device                                              loaded active plugged   /sys/subsystem/bluetooth/devices/hci0                                          
sys-subsystem-net-devices-docker0.device                                                 loaded active plugged   /sys/subsystem/net/devices/docker0                                             
sys-subsystem-net-devices-enp0s31f6.device                                               loaded active plugged   Ethernet Connection I219-V                                                     
sys-subsystem-net-devices-wlp4s0.device                                                  loaded active plugged   Wireless 8260                                                                  
sys-subsystem-net-devices-wwp0s20f0u3.device                                             loaded active plugged   Sierra_Wireless_EM7345_4G_LTE                                                  
kmod-static-nodes.service                                                                loaded active exited    Create list of required static device nodes for the current kernel             
sys-devices-virtual-block-dm\x2d3.device loaded active     plugged   /sys/devices/virtual/block/dm-3                                                
sys-devices-virtual-block-dm\x2d4.device loaded active     plugged   /sys/devices/virtual/block/dm-4                                                
sys-devices-virtual-block-dm\x2d5.device loaded active     plugged   /sys/devices/virtual/block/dm-5                                                
sys-devices-virtual-net-docker0.device   loaded active     plugged   /sys/devices/virtual/net/docker0                                               
sys-subsystem-net-devices-docker0.device loaded active     plugged   /sys/subsystem/net/devices/docker0