Re: SCHED_DEADLINE tasks missing their deadline with SCHED_FLAG_RECLAIM jobs in the mix (using GRUB)

[Marcel Ziswiler]

Hi Juri

On Thu, 2025-05-29 at 11:39 +0200, Juri Lelli wrote:
> Hi Marcel,
> 
> On 25/05/25 21:29, Marcel Ziswiler wrote:
> > Hi Luca
> > 
> > On Fri, 2025-05-23 at 21:46 +0200, luca abeni wrote:
> > > Hi Marcel,
> > > 
> > > sorry, but I have some additional questions to fully understand your
> > > setup...
> > 
> > No Problem, I am happy to answer any questions :)
> > 
> > > On Mon, 19 May 2025 15:32:27 +0200
> > > Marcel Ziswiler <marcel.ziswiler@xxxxxxxxxxxxxxx> wrote:
> > > [...]
> > > > > just a quick question to better understand your setup (and check
> > > > > where the issue comes from):
> > > > > in the email below, you say that tasks are statically assigned to
> > > > > cores; how did you do this? Did you use isolated cpusets,  
> > > > 
> > > > Yes, we use the cpuset controller from the cgroup-v2 APIs in the
> > > > linux kernel in order to partition CPUs and memory nodes. In detail,
> > > > we use the AllowedCPUs and AllowedMemoryNodes in systemd's slice
> > > > configurations.
> > > 
> > > How do you configure systemd? I am having troubles in reproducing your
> > > AllowedCPUs configuration... This is an example of what I am trying:
> > > 	sudo systemctl set-property --runtime custom-workload.slice AllowedCPUs=1
> > > 	sudo systemctl set-property --runtime init.scope AllowedCPUs=0,2,3
> > > 	sudo systemctl set-property --runtime system.slice AllowedCPUs=0,2,3
> > > 	sudo systemctl set-property --runtime user.slice AllowedCPUs=0,2,3
> > > and then I try to run a SCHED_DEADLINE application with
> > > 	sudo systemd-run --scope -p Slice=custom-workload.slice <application>
> > 
> > We just use a bunch of systemd configuration files as follows:
> > 
> 
> ...
> 
> > > How are you configuring the cpusets?
> > 
> > See above.
> > 
> 
> Could you please add 'debug sched_debug sched_verbose' to your kernel
> cmdline and share the complete dmesg before starting your tests?

Sure, here you go [1].

> Also, I am attaching a script that should be able to retrieve cpuset
> information if you run it with
> 
> # python3 get_cpuset_info.py > cpuset.out
> 
> Could you please also do that and share the collected information?

[root@localhost ~]# python3 get_cpuset_info.py > cpuset.out
[root@localhost ~]# cat cpuset.out 
Recursively retrieving cpuset information from /sys/fs/cgroup (cgroup v2):

Cgroup: /
  cpuset.cpus.effective: 0
  cpuset.mems.effective: 0
------------------------------
Cgroup: /safety3.slice
  cpuset.cpus: 3
  cpuset.mems: 
  cpuset.cpus.effective: 3
  cpuset.mems.effective: 0
  cpuset.cpus.exclusive: 
  cpuset.cpus.exclusive.effective: 3
  cpuset.cpus.partition: root
------------------------------
Cgroup: /sys-fs-fuse-connections.mount
  cpuset.cpus: 
  cpuset.mems: 
  cpuset.cpus.effective: 0
  cpuset.mems.effective: 0
  cpuset.cpus.exclusive: 
  cpuset.cpus.exclusive.effective: 
  cpuset.cpus.partition: member
------------------------------
Cgroup: /sys-kernel-debug.mount
  cpuset.cpus: 
  cpuset.mems: 
  cpuset.cpus.effective: 0
  cpuset.mems.effective: 0
  cpuset.cpus.exclusive: 
  cpuset.cpus.exclusive.effective: 
  cpuset.cpus.partition: member
------------------------------
Cgroup: /dev-mqueue.mount
  cpuset.cpus: 
  cpuset.mems: 
  cpuset.cpus.effective: 0
  cpuset.mems.effective: 0
  cpuset.cpus.exclusive: 
  cpuset.cpus.exclusive.effective: 
  cpuset.cpus.partition: member
------------------------------
Cgroup: /user.slice
  cpuset.cpus: 0
  cpuset.mems: 
  cpuset.cpus.effective: 0
  cpuset.mems.effective: 0
  cpuset.cpus.exclusive: 
  cpuset.cpus.exclusive.effective: 
  cpuset.cpus.partition: member
------------------------------
Cgroup: /monitor.slice
  cpuset.cpus: 0
  cpuset.mems: 
  cpuset.cpus.effective: 0
  cpuset.mems.effective: 0
  cpuset.cpus.exclusive: 
  cpuset.cpus.exclusive.effective: 
  cpuset.cpus.partition: member
------------------------------
Cgroup: /safety1.slice
  cpuset.cpus: 1
  cpuset.mems: 
  cpuset.cpus.effective: 1
  cpuset.mems.effective: 0
  cpuset.cpus.exclusive: 
  cpuset.cpus.exclusive.effective: 1
  cpuset.cpus.partition: root
------------------------------
Cgroup: /sys-kernel-tracing.mount
  cpuset.cpus: 
  cpuset.mems: 
  cpuset.cpus.effective: 0
  cpuset.mems.effective: 0
  cpuset.cpus.exclusive: 
  cpuset.cpus.exclusive.effective: 
  cpuset.cpus.partition: member
------------------------------
Cgroup: /init.scope
  cpuset.cpus: 0
  cpuset.mems: 
  cpuset.cpus.effective: 0
  cpuset.mems.effective: 0
  cpuset.cpus.exclusive: 
  cpuset.cpus.exclusive.effective: 
  cpuset.cpus.partition: member
------------------------------
Cgroup: /system.slice
  cpuset.cpus: 0
  cpuset.mems: 
  cpuset.cpus.effective: 0
  cpuset.mems.effective: 0
  cpuset.cpus.exclusive: 
  cpuset.cpus.exclusive.effective: 
  cpuset.cpus.partition: member
------------------------------
Cgroup: /system.slice/systemd-networkd.service
  cpuset.cpus: 
  cpuset.mems: 
  cpuset.cpus.effective: 0
  cpuset.mems.effective: 0
  cpuset.cpus.exclusive: 
  cpuset.cpus.exclusive.effective: 
  cpuset.cpus.partition: member
------------------------------
Cgroup: /system.slice/systemd-udevd.service
  cpuset.cpus: 
  cpuset.mems: 
  cpuset.cpus.effective: 0
  cpuset.mems.effective: 0
  cpuset.cpus.exclusive: 
  cpuset.cpus.exclusive.effective: 
  cpuset.cpus.partition: member
------------------------------
Cgroup: /system.slice/system-serial\x2dgetty.slice
  cpuset.cpus: 
  cpuset.mems: 
  cpuset.cpus.effective: 0
  cpuset.mems.effective: 0
  cpuset.cpus.exclusive: 
  cpuset.cpus.exclusive.effective: 
  cpuset.cpus.partition: member
------------------------------
Cgroup: /system.slice/boot.mount
  cpuset.cpus: 
  cpuset.mems: 
  cpuset.cpus.effective: 0
  cpuset.mems.effective: 0
  cpuset.cpus.exclusive: 
  cpuset.cpus.exclusive.effective: 
  cpuset.cpus.partition: member
------------------------------
Cgroup: /system.slice/var-lib-containers.mount
  cpuset.cpus: 
  cpuset.mems: 
  cpuset.cpus.effective: 0
  cpuset.mems.effective: 0
  cpuset.cpus.exclusive: 
  cpuset.cpus.exclusive.effective: 
  cpuset.cpus.partition: member
------------------------------
Cgroup: /system.slice/auditd.service
  cpuset.cpus: 
  cpuset.mems: 
  cpuset.cpus.effective: 0
  cpuset.mems.effective: 0
  cpuset.cpus.exclusive: 
  cpuset.cpus.exclusive.effective: 
  cpuset.cpus.partition: member
------------------------------
Cgroup: /system.slice/system-modprobe.slice
  cpuset.cpus: 
  cpuset.mems: 
  cpuset.cpus.effective: 0
  cpuset.mems.effective: 0
  cpuset.cpus.exclusive: 
  cpuset.cpus.exclusive.effective: 
  cpuset.cpus.partition: member
------------------------------
Cgroup: /system.slice/systemd-journald.service
  cpuset.cpus: 
  cpuset.mems: 
  cpuset.cpus.effective: 0
  cpuset.mems.effective: 0
  cpuset.cpus.exclusive: 
  cpuset.cpus.exclusive.effective: 
  cpuset.cpus.partition: member
------------------------------
Cgroup: /system.slice/systemd-nsresourced.service
  cpuset.cpus: 
  cpuset.mems: 
  cpuset.cpus.effective: 0
  cpuset.mems.effective: 0
  cpuset.cpus.exclusive: 
  cpuset.cpus.exclusive.effective: 
  cpuset.cpus.partition: member
------------------------------
Cgroup: /system.slice/sshd.service
  cpuset.cpus: 
  cpuset.mems: 
  cpuset.cpus.effective: 0
  cpuset.mems.effective: 0
  cpuset.cpus.exclusive: 
  cpuset.cpus.exclusive.effective: 
  cpuset.cpus.partition: member
------------------------------
Cgroup: /system.slice/var-tmp.mount
  cpuset.cpus: 
  cpuset.mems: 
  cpuset.cpus.effective: 0
  cpuset.mems.effective: 0
  cpuset.cpus.exclusive: 
  cpuset.cpus.exclusive.effective: 
  cpuset.cpus.partition: member
------------------------------
Cgroup: /system.slice/test-audio.service
  cpuset.cpus: 
  cpuset.mems: 
  cpuset.cpus.effective: 0
  cpuset.mems.effective: 0
  cpuset.cpus.exclusive: 
  cpuset.cpus.exclusive.effective: 
  cpuset.cpus.partition: member
------------------------------
Cgroup: /system.slice/tmp.mount
  cpuset.cpus: 
  cpuset.mems: 
  cpuset.cpus.effective: 0
  cpuset.mems.effective: 0
  cpuset.cpus.exclusive: 
  cpuset.cpus.exclusive.effective: 
  cpuset.cpus.partition: member
------------------------------
Cgroup: /system.slice/systemd-userdbd.service
  cpuset.cpus: 
  cpuset.mems: 
  cpuset.cpus.effective: 0
  cpuset.mems.effective: 0
  cpuset.cpus.exclusive: 
  cpuset.cpus.exclusive.effective: 
  cpuset.cpus.partition: member
------------------------------
Cgroup: /system.slice/test-speaker.service
  cpuset.cpus: 
  cpuset.mems: 
  cpuset.cpus.effective: 0
  cpuset.mems.effective: 0
  cpuset.cpus.exclusive: 
  cpuset.cpus.exclusive.effective: 
  cpuset.cpus.partition: member
------------------------------
Cgroup: /system.slice/systemd-oomd.service
  cpuset.cpus: 
  cpuset.mems: 
  cpuset.cpus.effective: 0
  cpuset.mems.effective: 0
  cpuset.cpus.exclusive: 
  cpuset.cpus.exclusive.effective: 
  cpuset.cpus.partition: member
------------------------------
Cgroup: /system.slice/systemd-resolved.service
  cpuset.cpus: 
  cpuset.mems: 
  cpuset.cpus.effective: 0
  cpuset.mems.effective: 0
  cpuset.cpus.exclusive: 
  cpuset.cpus.exclusive.effective: 
  cpuset.cpus.partition: member
------------------------------
Cgroup: /system.slice/dbus.service
  cpuset.cpus: 
  cpuset.mems: 
  cpuset.cpus.effective: 0
  cpuset.mems.effective: 0
  cpuset.cpus.exclusive: 
  cpuset.cpus.exclusive.effective: 
  cpuset.cpus.partition: member
------------------------------
Cgroup: /system.slice/systemd-timesyncd.service
  cpuset.cpus: 
  cpuset.mems: 
  cpuset.cpus.effective: 0
  cpuset.mems.effective: 0
  cpuset.cpus.exclusive: 
  cpuset.cpus.exclusive.effective: 
  cpuset.cpus.partition: member
------------------------------
Cgroup: /system.slice/system-getty.slice
  cpuset.cpus: 
  cpuset.mems: 
  cpuset.cpus.effective: 0
  cpuset.mems.effective: 0
  cpuset.cpus.exclusive: 
  cpuset.cpus.exclusive.effective: 
  cpuset.cpus.partition: member
------------------------------
Cgroup: /system.slice/systemd-logind.service
  cpuset.cpus: 
  cpuset.mems: 
  cpuset.cpus.effective: 0
  cpuset.mems.effective: 0
  cpuset.cpus.exclusive: 
  cpuset.cpus.exclusive.effective: 
  cpuset.cpus.partition: member
------------------------------
Cgroup: /system.slice/system-disk\x2dstat\x2dmonitoring.slice
  cpuset.cpus: 
  cpuset.mems: 
  cpuset.cpus.effective: 0
  cpuset.mems.effective: 0
  cpuset.cpus.exclusive: 
  cpuset.cpus.exclusive.effective: 
  cpuset.cpus.partition: member
------------------------------
Cgroup: /safety2.slice
  cpuset.cpus: 2
  cpuset.mems: 
  cpuset.cpus.effective: 2
  cpuset.mems.effective: 0
  cpuset.cpus.exclusive: 
  cpuset.cpus.exclusive.effective: 2
  cpuset.cpus.partition: root
------------------------------
Cgroup: /dev-hugepages.mount
  cpuset.cpus: 
  cpuset.mems: 
  cpuset.cpus.effective: 0
  cpuset.mems.effective: 0
  cpuset.cpus.exclusive: 
  cpuset.cpus.exclusive.effective: 
  cpuset.cpus.partition: member
------------------------------

> It should help us to better understand your setup and possibly reproduce
> the problem you are seeing.

Sure, I am happy to help.

> Thanks!

Thanks you!

> Juri

[1] https://pastebin.com/khFApYgf

Cheers

Marcel