Re: [PATCH v7 0/4] perf: add support for profiling jitted code

[Brendan Gregg]

G'Day,

On Wed, Sep 30, 2015 at 11:45 PM, Stephane Eranian <eranian@xxxxxxxxxx> wrote:
>
> This patch series extends perf record/report/annotate to enable
> profiling of jitted (just-in-time compiled) code. The current
> perf tool provides very limited support for profiling jitted
> code for some runtime environments. But the support is experimental
> and cannot be used in complex environments. It relies on files
> in /tmp, for instance. It does not support annotate mode or
> rejitted code.
>
> This patch series adds a better way of profiling jitted code
> with the following advantages:
>    - support any jitted code environment (some with modifications)
>    - support Java runtime with JVMTI interface with no modifications
>    - provides a portable JVMTI agent library
>    - known to support V8 runtime
>    - known to support DART runtime
>    - supports code rejitting and code movements
>    - no files in /tmp
>    - meta-data file is unique to each run
>    - no changes to perf report/annotate
>    - support per-thread and system-wide profiling
>    - support monitoring of multiple simultaneous Jit runtimes
>    - source level view in perf annotate
>
> The support is based on cooperation with the runtime. For Java runtimes,
> supporting the JVMTI interface, there is no change necessary. For other
> runtimes, modifications are necessary to emit the meta-data to support
> symbolization, annotation, source lines correlation of the samples.
> Those modifications are relatively straighforward, some have been
> implemented in V8 and DART.
>
> The jit environment emits a binary dump file which contains the jitted
> code (in raw format) and meta-data describing the mapping of functions.
> The binary format is documented in the jitdump.h header file. It is
> adapted from the OProfile jitdump format.

While this is impressive work, I don't think I'd use it very much, and
I wouldn't encourage others too either. Is it right that this approach
needs to be turned on from runtime start, and will constantly emit
timestamped JIT records? I'd use that as a backup for existing
techniques for perf_events and jitted runtimes.

Right now we (Netflix) can profile Java in production using
perf_events, using an on-demand symbol dump: the perf-map-agent
JVMTI[1]. This agent used to be loaded on startup, like this patchset,
which cost overhead: CPU, filesystem I/O, storage. The problem was
that we didn't know which of the tens of thousands of instances we'd
want to profile, so loading an always-on JIT symbol dumper on all
instances would cost resources. With on-demand, we only dump symbols
on the few instances needed. On-demand has caveats, of course, and
symbols can change during the profile. But so far that's not been a
problem.

So this seems like a lot of changes for what won't be our primary way
of using perf_events on Java or Node.js, which we currently can
profile with perf_events effectively. So long as we're aware of that.

Brendan

[1] http://techblog.netflix.com/2015/07/java-in-flames.html
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/