Re: [RFC] tracing: Adding cgroup aware tracing functionality

[David Sharp]

On Thu, Apr 7, 2011 at 5:06 AM, Frederic Weisbecker <fweisbec@xxxxxxxxx> wrote:
> On Wed, Apr 06, 2011 at 08:17:33PM -0700, Vaibhav Nagarnaik wrote:
>> On Wed, Apr 6, 2011 at 6:33 PM, Frederic Weisbecker <fweisbec@xxxxxxxxx> wrote:
>> > On Wed, Apr 06, 2011 at 11:50:21AM -0700, Vaibhav Nagarnaik wrote:
>> >> All
>> >> The cgroup functionality is being used widely in different scenarios. It also
>> >> is being integrated with other parts of kernel to take advantage of its
>> >> features. One of the areas that is not yet aware of cgroup functionality is
>> >> the ftrace framework.
>> >>
>> >> Although ftrace provides a way to filter based on PIDs of tasks to be traced,
>> >> it is restricted to specific tracers, like function tracer. Also it becomes
>> >> difficult to keep track of all PIDs in a dynamic environment with processes
>> >> being created and destroyed in a short amount of time.
>> >>
>> >> An application that creates many processes/tasks is convenient to track and
>> >> control with cgroups, but it is difficult to track these processes for the
>> >> purposes of tracing. And if child processes are moved to another cgroup, it
>> >> makes sense to trace only the original cgroup.
>> >>
>> >> This proposal is to create a file in the tracing directory called
>> >> set_trace_cgroup to which a user can write the path of an active cgroup, one
>> >> at a time. If no cgroups are specified, no filtering is done and all tasks are
>> >> traced. When a cgroup path is added in, it sets a boolean tracing_enabled for
>> >> the enabled cgroup in all the hierarchies, which enables tracing for all the
>> >> assigned tasks under the specified cgroup.
>> >>
>> >> Though creating a new file in the directory is not desirable, but this
>> >> interface seems the most appropriate change required to implement the new
>> >> feature.
>> >>
>> >> This tracing_enabled flag is also exported in the cgroupfs directory structure
>> >> which can be turned on/off for a specific hierarchy/cgroup combination. This
>> >> gives control to enable/disable tracing over a cgroup in a specific hierarchy
>> >> only.
>> >>
>> >> This gives more fine-grained control over the tasks being traced. I would like
>> >> to know your thoughts on this interface and the approach to make tracing
>> >> cgroup aware.
>> >
>> > So I have to ask, why can't you use perf events to do tracing limited on cgroups?
>> > It has this cgroup context awareness.

Perf doesn't have the same latency characteristics as ftrace. It costs
a full microsecond for every trace event.

https://lkml.org/lkml/2010/10/28/261

It's possible these results need to be updated. Has any effort been
made to improve the tracing latency of perf?

>> The perf event cgroup awareness comes from creating a different hierarchy for
>> perf events. When the events and the current task's cgroup match, the events
>> are logged. So the changes are pretty specific to the perf events.
>>
>> Even in the case where changes are made to handle trace events, the interface
>> files are still needed. The interface used to specify perf events uses the
>> perf_event syscall which isn't available to specify trace events.
>>
>> This is based on my limited understanding of the perf_events cgroup awareness
>> patch. Please correct me if I am missing anything.
>
>
> Ah but perf events can do much more than counting and sampling
> hardware events. Trace events can be used as perf events too.
>
> List the events:
>
> Â Â Â Âperf list -e tracepoints
>
> List of pre-defined events (to be used in -e):
>
> Âskb:kfree_skb               Â[Tracepoint event]
> Âskb:consume_skb              Â[Tracepoint event]
> Âskb:skb_copy_datagram_iovec        Â[Tracepoint event]
> Ânet:net_dev_xmit              [Tracepoint event]
> Ânet:net_dev_queue             Â[Tracepoint event]
> Ânet:netif_receive_skb           Â[Tracepoint event]
> Ânet:netif_rx                [Tracepoint event]
> Ânapi:napi_poll               [Tracepoint event]
> Âscsi:scsi_dispatch_cmd_start        [Tracepoint event]
> Âscsi:scsi_dispatch_cmd_error        [Tracepoint event]
> Âscsi:scsi_dispatch_cmd_done        Â[Tracepoint event]
> Âscsi:scsi_dispatch_cmd_timeout       [Tracepoint event]
> Âscsi:scsi_eh_wakeup            Â[Tracepoint event]
> Âdrm:drm_vblank_event            [Tracepoint event]
> Âdrm:drm_vblank_event_queued        Â[Tracepoint event]
> Âdrm:drm_vblank_event_delivered       [Tracepoint event]
> Âblock:block_rq_abort            [Tracepoint event]
> Âblock:block_rq_requeue           [Tracepoint event]
> Âblock:block_rq_complete          Â[Tracepoint event]
> Âblock:block_rq_insert           Â[Tracepoint event]
> Âetc...
>
>
> Trace sched switch events:
>
> Â Â Â Âperf record -e sched:sched_switch -a
> Â Â Â Â^C
>
>
> Print them:
>
> Â Â Â Âperf script
>
>     swapper   0 [000] Â1132.964598: sched_switch: prev_comm=swapper prev_pid=0 prev_prio=120 prev_state=R ==> next_comm
> Â Â kworker/0:1 Â4358 [000] Â1132.964641: sched_switch: prev_comm=kworker/0:1 prev_pid=4358 prev_prio=120 prev_state=S ==> ne
> Â Â Â Â syslogd Â2703 [000] Â1132.964720: sched_switch: prev_comm=syslogd prev_pid=2703 prev_prio=120 prev_state=D ==> next_c
>     swapper   0 [000] Â1132.965100: sched_switch: prev_comm=swapper prev_pid=0 prev_prio=120 prev_state=R ==> next_comm
> Â Â Â Â Â Âperf Â4725 [001] Â1132.965178: sched_switch: prev_comm=perf prev_pid=4725 prev_prio=120 prev_state=D ==> next_comm
>     swapper   0 [001] Â1132.965227: sched_switch: prev_comm=kworker/0:0 prev_pid=0 prev_prio=120 prev_state=R ==> next_
> Â Â Â Â Â Âperf Â4725 [001] Â1132.965246: sched_switch: prev_comm=perf prev_pid=4725 prev_prio=120 prev_state=D ==> next_comm
> Â Â Â Âetc...
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/