systemtap 1.7 release

From: Frank Ch. Eigler
Date: Wed Feb 01 2012 - 21:42:07 EST

The systemtap team announces release 1.7

The new group and privilege level "stapsys" has been added,
debuginfo can now be automatically downloaded, quiet runtime error
handling was added, parser error handling has been improved, device
driver initialization functions can now be probed, user and kernel
dwarf backtraces are now enabled on ppc, s390x and arm, new
stap-server customizing options have been added. Associative arrays
are much faster. Many diagnostic messages have been improved.
Preliminary user-space probing with inode-uprobes, without utrace.
Faster (compiled) *printf formatting.

= Where to get it - our project page
git tag release-1.7 (commit 83bd2699)

There have been over 700 commits since the last release.
There have been over 90 bugs/features fixed since the last release.

= How to build it

See the README and NEWS files at;a=tree
Further information at

= Systemtap frontend (stap) changes

- New Options

+ Systemtap loads extra command line options from $SYSTEMTAP_DIR/rc
($HOME/.systemtap/rc by default) before the normal argc/argv. This
may be useful to activate site options such as --use-server or
--download-debuginfo or --modinfo.
+ The translator supports a new --suppress-handler-errors option, which
causes most runtime errors to be turned into quiet skipped probes. This
also disables the MAXERRORS and MAXSKIPPED limits.

+ Translator warnings have been standardized and controlled by the -w / -W

+ The translator supports a new --modinfo NAME=VALUE option, which emits
additional MODULE_INFO(n,v) macros into the generated code.

+ The stap-server service (initscript) now supports three new options:
--log LOGFILE allow specifying a log file
--port PORT-NUMBER allow specifying a network port number
--SSL CERT-DATABASE allow specifying an NSS certificate database location
These options are also supported within individual server configuration
files. See stap-server and initscript/README.stap-server for details.
The stap-server is no longer activated by default.

+ The translator supports a new --dump-probe-types option, which dumps a list
of supported probe types. If --unprivileged is also specified, the list
will be limited to probe types which are available to unprivileged users.

+ Systemtap supports a new --download-debuginfo[=OPTION] option, which can be
used to automatically download required debuginfo. Possible values for
OPTION are: 'yes', 'no', 'ask', and a positive number representing the
timeout desired. The default behavior is to not automatically download the

+ Systemtap supports a new option --privilege=[stapusr|stapsys|stapdev].
Systemtap now supports a new group and privilege level "stapsys", which is
equivalent to the privilege afforded by membership in the group "stapdev",
except that guru mode (-g) functionality may not be used.
--privilege=stapusr is equivalent to specifying the existing --unprivileged
option. --privilege=stapdev is the default.

+ The systemtap module cache is cleaned less frequently, governed by the
number of seconds in the $SYSTEMTAP_DIR/cache/cache_clean_interval_s file.

- The stap-server has seen many improvements, and is no longer considered

- The fixed maximum number of VMA pages that will be tracked at runtime has
been removed. This reduces memory use for those scripts that either do not
need any or need only limited target process VMA tracking, and also allows
easier system wide probes that inspect shared library variables and/or user
backtraces. stap will now silently ignore -DTASK_FINDER_VMA_ENTRY_ITEMS.

- Scripts that use kernel.trace("...") probes compile much faster.

- Parse errors no longer generate a cascade of false errors. Instead, a
parse error skips the rest of the current probe or function, and resumes
at the next one. This should generate fewer and better messages.

- On 2.6.29+ kernels, systemtap can now probe kernel modules that
arrive and/or depart during session run-time. This allows
probing of device driver initialization functions, which had formerly been

- The environment variable MAXBACKTRACE, which did work in earlier releases,
has now been documented in the stap 1 manual page.

- Kernel backtraces can now be generated for non-pt_regs probe context
if the kernel support dump_trace(). This enables backtraces from
certain timer probes and tracepoints.

- There is some limited preliminary support for user-space probing
against kernels such as linux-next, which have no utrace but do have
the newer inode-uprobes work by Srikar Dronamraju and colleagues.

- Architecture Specific Changes

+ ubacktrace() should now also work for some x86 kernel probes which can
use the dwarf unwinder to recover the user registers to provide
more accurate user backtraces.

+ The s390x systemtap runtime now properly splits kernel and user
addresses, which are in separate address spaces on that architecture,
which enables user space introspection.

+ ppc, s390x, and arm now support user backtraces through the DWARF unwinder.

+ ppc now handles function descriptors as symbol names correctly.

+ arm now has a uprobes port which enables user probes. This still
requires some kernel patches (user_regsets and tracehook support for

= Systemtap script language changes

- process("PATH").[library("PATH")].function("NAME").exported probes are now
supported to filter function() to only exported instances.

- process("PATH").library("PATH").plt("NAME") probes are now supported.
Wildcards are supported in the plt-name part, to refer to any function in the
program linkage table which matches the glob pattern and the rest of the
probe point.

- Global array wrapping is now supported for both associative and
statistics typed arrays using the '%' character to signify a wrapped
array. For example, 'global foo%[100]' would allow the array 'foo'
to be wrapped if more than 100 elements are inserted.

- The following probe types are deprecated in release 1.7 and will be removed in release 1.8:

- SDT can now define up to 12 arguments in a probe point.

- Formatted printing is faster by compiling the formatting directives
to C code rather than interpreting at run time.

- The translator has better support for probing C++ applications by
better undertanding of compilation units, nested types, templates,
as used in probe point and @cast constructs.

= Systemtap tapset changes

- Many prior tapset functions have had documentation added for the generated
man pages and tapset reference guide.

- The tapset functions remote_id() and remote_uri() identify the member of a
swarm of "stap --remote FOO --remote BAR baz.stp" concurrent executions.

- The task_backtrace() tapset function was deprecated in 1.7 and has been
removed in 1.7.

- The tapset function probe_type() has been added, to return a short string
describing the low level probe handler type for the current probe point.

- Both unwind and symbol data is now only collected and emitted for
scripts actually using backtracing or function/data symbols.
Tapset functions are marked with /* pragma:symbols */ or
/* pragma:unwind */ to indicate they need the specific data.

- The functions cpu_clock_ns(cpu), local_clock_ns(), and their coarser variants
have been added. These are monotonic wallclock time sources, rather than the
absolute time-since-epoch sort. On kernels too old for these functions, it
falls back to stap's gettimeofday.

- The tapset speculative.stp has been added. Strings can be speculatively
written to an output buffer and then later committed or discarded.

- The tapset stopwatch.stp has been added. It allows named virtual stopwatches
to be stopped/started/read, as a substitute for manual timestamp subtraction.

- The tapset function pstrace() has been added. It allows printing of
a process ancestry list.

- The tapset functions cpu_clock_ns(cpu), local_clock_ns(), and coarser
variants were added, as monotonic wallclock time sources.

= Systemtap sample scripts

+ eventcount.stp uses wrapped maps to avoid runtime errors due to too many
processes being encountered.

+ New sample watchdog.stp: implement a resettable system watchdog timer
that prints a backtrace if it expires.

+ New sample func_time_stats.stp: print histogram of execution duration of
some named function.

+ New sample stopwatches.stp: demonstrates timekeeping using distinct

+ New sample nfsd_unlink.stp: lists which clients are removing files across

+ New sample pstrace_exec.stp: print process ancestry for selected execs.

+ New sample syscalltimes: a shell+systemtap script to summarize/filter
system call times.

+ New sample periodic.stp: use the timer_expire_entry tracepoint to summarize
software timer activity in the kernel.

+ New sample kvm_service_time.stp: report on vm-exit operations.

+ New sample qemu_count.stp: report on userspace qemu sys/sdt.h activity.

+ New sample qemu_io.stp: report on i/o port access by virtual machine.

= Contributors for this release

Chris Meek, Dave Brolley, David Smith, *Diego Elio Pettenò, Frank
Ch. Eigler, *Frederic Turgis, *Fredrik Östman, *Han Pingtian,
Jim Keniston, Josh Stone, Lukas Berk, Mark Wielaard, *Otavio Pontes,
*Peter Robinson, Srikar Dronamraju, Stan Cox, Timo Juhani Lindfors,
*Wade Farnsworth, William Cohen

Special thanks to Stan Cox for drafting these notes.
Special thanks to new contributors, marked with '*' above.

= Examples of tested kernel versions

3.3.0-rc1 (x86_64)
3.2.0-0 (i686,x86_64)
3.1.4-1 (i686,x86_64)
3.0.0-1 (x86_64) (i686) (armv7l) (i686) (x86_64)
2.6.32-220 (x86_64,i686,ppc64,s390x)
2.6.18-274 (x86_64)
2.6.9-101 (i686)

= Known issues with this release

- Some kernel crashes continue to be reported when a script probes
broad kernel function wildcards. (PR2725)

- Some kernel crashes have been reported when intensely probing many
short-lived user-space programs with uprobes. (PR13634, in progress.)

= Bugs fixed for this release <>

3217 compiled binary tracing
4794 backtrace of yyy_func3 and print_stack of yyy_func3 fails on ppc64.
5450 backtrace does not work for systemtap on s390x
5908 conversion test fails on arm
6503 permit probes on module .init and __exit functions
6961 backtrace from non-pt_regs probe context
10787 Probe exported functions only
10994 utrace-free task_finder
11209 Provide an alternative time tapset
11441 stap-server Roadmap to Legitimacy
11671 Allocate __stp_tf_vma_entry dynamically
11769 process.mark probes losing arguments
11775 ensure all tapsets are referenced in tapsets.tmpl
11859 print_ubacktrace() from kernel return probe prints kretprobe addresses
12135 field search is wrong when inheritance masks names
12136 @cast can't find namespaced types
12215 probe process functions defined in shared libraries
12221 Include build-ids in the script hash
12413 kprocess.* support PIDs but not TIDs
12741 rip-relative instruction is not detected as such
12773 invoke abrt-action-debuginfo-install --id=XXXXX if missing debuginfo
12888 stap-serverd should be weaned from -k
12895 stapconf should be using -nostdinc
12916 Aliased process probes aren't allowed for unprivileged
12947 Use of hardware watchpoint causing oops
13006 numa_faults.stp crashes 2.6.9 kernels
13026 staprun security bugs: CVE-2011-2502 CVE-2011-2503
13032 ring_buffer.c broken because ring_buffer_peek changed signature
13055 Replace kern_path_parent in runtime/procfs.c
13057 Provide a Command Line Option to Dump Supported Probe Points
13072 sdt.h STAP_PROBE doesn't build against gcc 3.4.6 on i386
13076 tweak module cache aging defaults & logic
13083 debug 3.1 kernel doesn't like mmput() in stap_utrace_attach
13100 Building tracepoint cache all at once fails when kernel debuginfo installed
13101 Networking tapsets should have structured comments
13102 nfs tapsets should have structured comments
13103 substr man page flubs 3rd arg
13107 simplify compile-server testsuite
13108 kprobing some paravirt stuff seems unsafe
13112 kprobing some port i/o stuff seems unsafe
13128 intermediate privilege between stapdev and stapusr
13132 buildok/pr10678.stp fails if system doesn't have ne2k-pci module
13133 sdt_misc.exp's wildcard criterion should be changed
13136 a guru-mode printk function
13142 Stopping script execution on Android (ARM) does not unload kernel module
13146 memory allocation can sleep
13155 conflicting tracepoint headers block stap -l / stap -p4
13156 improve hash_XXX.log
13158 rename_module.exp leaving module behind
13164 Deprecate process.function(number).inline
13200 apparent user-space memory corruption, unprivileged_myproc.exp
13210 vdso runtime tracking is broken
13216 AM_GNU_GETTEXT_VERSION needs a version
13241 too much dump-unwindsyms processing for modules even if !need_symbols
13261 syntax for global arrays activating map->wrap mode
13266 Implement DW_CFA_restore in the dwarf unwinder
13272 printf "quoting" broken
13274 standardize warning processing
13283 script -> C translator incorrect comparison (using ==) of kernel_string() operands
13284 failure to resolve kernel global e.g. jiffies
13289 %m/%M printf formatting operators access memory incorrectly
13306 option to skip probes with alias prologue problems
13312 on 2.6.18 s390x kernels, PTRACE_GET_LAST_BREAK isn't defined
13313 Can't parse SDT_V3 operand '(%rcx,%rax,8)'
13317 improve parse error recovery heuristics
13318 make it possible to set a default -DMAXMEMORY for unprivileged builds
13319 startup printk message omits stap -sNNN buffer sizes
13323 staprun should check return code from send_request when sending messages via the transport layer
13338 possible infinite recursion for process wildcard
13353 --remote arch/release maybe not respected for build-id calculations
13354 remote metadata / management
13373 FAILs for function.label(string) and new plt probe constructs in unprivileged myproc/probes tests
13386 BUG: using smp_processor_id() in preemptible code
13394 gcc 4.6 doesn't inline ibar for unprivileged testcase
13398 Allow Compile-Server Specs to Resolve to localhost and localhost.localdomain
13404 DTRACE_PROBE for more that 10 arguments needed
13428 nfsd.stp failure on f16 because of renamed kernel function
13440 cast.exp fails on s390x umod cast in (kernel) begin probe
13445 control.c:137: error: the frame size of 264 bytes is larger than 256 bytes
13458 sort generated tapset documentation
13460 expose all stap-serverd options to stap-server
13461 bring back systemtap-client subpackage
13462 symdata() returns ARM mapping symbols
13479 ARM tapset/scheduler.stp refers to kernel.function("__switch_to")
13496 stap could show some progress of the communication with the server
13499 prepare to deprecate grapher
13513 undo PR11759 for newer PREEMPT_RT kernels
13520 .systemtap/rc file
13536 staprun -u race condition
13539 occasional oops, kernel SEGV, RHEL5, :uprobes:uprobe_free_process+0xba/0x131
13546 improve map hashing
13626 gcc 4.7 doesn't compile sdt.h with -std=c++0x
13628 stap-prep script doesn't work in pt_BR environments
13631 Detection of Reqested Privilege Level in stap-serverd
13640 stap-server: don't start by default

To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at
Please read the FAQ at