Re: Query regarding work scheduling

[Steven Rostedt]

On Thu, 14 Aug 2025 03:54:58 +0000
Subbaraya Sundeep <sbhatta@xxxxxxxxxxx> wrote:

> > Difficult to tell where the latencies are coming from. Maybe you can use
> > something like https://github.com/josefbacik/systing to look further into
> > it? All the scheduling events are tracked by default and you should be able
> > to add tracepoints and other events relatively easily. You can also set  

> Thanks for the reply. I am using simple busybox to avoid overhead of any other apps
> or deamons running in background and taking CPU time in between.
> I will try building systing and running it. 6.16 histogram shows that it
> is not one high latency event causing overall latency but bunch of small
> latencies are adding up and causing big latency.
> I suspect this has something to do with EEVDF scheduling since this behavior is
> seen from 6.6 (please note I may be wrong completly).
> Are there any methods or options with which I can bring back CFS scheduling behavior
> maybe with the knobs in /sys/kernel/debug/sched/features as a quick check? 

You could also use tracefs as that works on busybox:

 # echo 0 > /sys/kernel/tracing/tracing_on
 # echo 1 > /sys/kernel/tracing/events/sched/sched_switch/enable
[ and perhaps even more events ]
 # echo 1 > /sys/kernel/tracing/tracing_on
 # <run test>; echo 0 > /sys/kernel/tracing/tracing_on
 # cat /sys/kernel/tracing/trace

You could even make it a trace.dat file:

 # mkdir /tmp/tracing
 # cp -r /sys/kernel/tracing/events /tmp/tracing/
 # cp -r /proc/kallsyms /tmp/tracing/
[ have bs be PAGE_SIZE for your architecture ]
 # dd bs=4096 if=/sys/kernel/tracing/per_cpu/cpu0/trace_pipe_raw of=/tmp/tracing/trace0.raw
 # cd /tmp
 # tar cvf trace.tar tracing

Copy trace.tar to a desktop and extract it.

 $ cd /tmp
 $ tar xvf trace.tar
[ Make sure you have the latest trace-cmd installed ]
 $ trace-cmd restore -t /tmp/tracing/ -k /tmp/tracing/kallsyms -o /tmp/trace.dat /tmp/tracing/trace0.raw 
 $ trace-cmp report /tmp/trace.dat

Now you can send us the trace.dat file and we could analyze it more.

You could also enable more events than just sched_switch, like sched_waking
and such.

-- Steve