Traces events with high-resolution timestamp, a user-provided value,
and the current I-pipe domain states
Supported events are
kernel function entries
entry and exit of a critical section (with separate longest-path
PID + current process priority
user-provided event IDs
Comes with instrumentation for hard IRQs-off tracing
Kernel and user-space API (the latter via Xenomai)
Strictly deterministic, causes no undefined latency peaks during
The tracer is a built-in feature of the
I-pipe. To enable tracing support, go the the
"Kernel hacking" section of the kernel configuration and switch on
"I-pipe debugging" and then "Latency tracing".
The following options are available at kernel configuration time:
"Enable tracing on boot" CONFIG_IPIPE_TRACE_ENABLE
Defines if the tracer is active by default when booting the system or
shall be later enabled via /proc/ipipe/trace/enable. Specifically if
function tracing is enabled, deferring to switch on the tracer reduces
the boot time on low-end systems.
"Instrument function entries" CONFIG_IPIPE_TRACE_MCOUNT
Trace each entry of a kernel function. Note that this instrumentation,
though it is the most valuable one, has a significant performance impact
on low-end systems (~50% larger worst-case latencies on a Pentium-I 133
"Trace IRQs-off times" CONFIG_IPIPE_TRACE_IRQSOFF
Instrument each disable and re-enable of hardware IRQs. This allows to
identify the longest path in a system with IRQs disabled.
"Depth of trace log" CONFIG_IPIPE_TRACE_SHIFT
Controls the number of trace points. The I-pipe tracer maintains four
ring buffers per CPU of the given capacity in order to switch traces in
a lock-less fashion with respect to potentially pending output requests
on those buffers. If you run short on memory, try reducing the trace log
depth which is set to 16000 trace points by default.
"Use vmalloc’ed trace buffer" CONFIG_IPIPE_TRACE_VMALLOC
Instead of reserving static kernel data, the required buffer is
allocated via vmalloc during boot-up when this option is enabled. This
can help to start systems that are low on memory, but it slightly
degrades overall performance. Try this option when a traced kernel hangs
unexpectedly at boot time.
Note: Since 2.6.29, the I-pipe tracer makes use of ftrace lower layers
for function tracing. CONFIG_FUNCTION_TRACE is enabled for that
reason, and /proc/sys/kernel/ftrace_enabled also influences the I-pipe
function tracer. This switch can be used to dynamically enable or
disable the tracer, reducing the overhead of CONFIG_IPIPE_TRACE_MCOUNT
to an absolute minimum when disabled. Ftrace’s function and graph
tracers themselves are still not usable over I-pipe kernels, latest
I-pipe will prevent conflicting use.
Traces can be generated in many ways. One happens automatically when
CONFIG_IPIPE_TRACE_IRQSOFF is enabled. Another way is to invoke the
testsuite, namely latency or irqbench with the option -f to
trigger a freeze on each new maximum latency. And finally one can
instrument kernel or user space code with
Xenomai:I-pipe Tracer#API[explicite tracer invocations].
All tracing related controls and information can be found under
/proc/ipipe/trace/. Here is an overview:
The longest trace path between all begin and end marks. Read from it for
dumping the content, write any data to it for resetting.
The first frozen back-trace triggered by an instrumented application (or
via echo 1 > frozen). Read from it for dumping the content, write 0
to it for resetting, write an integer value > 0 to additionally trigger
Global switch to turn all tracing on or off. Default depends on
CONFIG_IPIPE_TRACE_ENABLE. Read from it for dumping the state, write 0
to disable and > 0 to enable.
The number of additional trace points to be reported before the longest
trace path. Default: 10. Read from it for dumping the state, write an
integer >= 0 to set a new value.
The number of additional trace points to be captured and reported both
for the longest trace path and the frozen path. Default: 10. Read from
it for dumping the state, write an integer >= 0 to set a new value. Note: This value affects the latency of the tracer when it freezes a
new "max" or "frozen" path, because post_trace_points will be copied
from the frozen trace buffer to the new working buffer. Keep the number
small to avoid noticeable impact on the system latency.
The number of trace points backwards to be reported for the frozen trace
path (including the freeze event itself). Default: 30. Read from it for
dumping the state, write an integer >= 0 to set a new value.
Report more details in "max" or "frozen" listings, e.g. the I-pipe
domain states, user-provided values, or the related name and priority of
traced PIDs. Just give it a try. Default: off. Read from it for dumping
the state, write 0 to disable and > 0 to enable.
Freeze the trace when a specific function entry is recorded. On writing
a function’s symbolic name to this entry, the kernel will try to resolve
it and arm the trigger on success. Write an empty string to disable the
trigger, read the value to obtain its current state.
Write 0 to it for disabling the function tracer’s hooks in all kernel
functions, 1 to enable it. Disabling widely eliminates the overhead of
CONFIG_IPIPE_TRACE_MCOUNT so that instrumented kernels can be
considered for production systems.
The following traces were taken on Pentium M 1.3 GHz system by running
latency -f. The first one represents the longest path with hard IRQs
I-pipe worst-case tracing service on 18.104.22.168-adeos/ipipe-1.5-00 ------------------------------------------------------------ Begin: ` cycles, Trace Points: `` (``/``), Length: `
Back to Linux (prio -1). [... snipped by author ...]
Xenomai Tracer API
The following services are available both in kernel and user space by
including nucleus/trace.h. If the tracer is not enabled, all service
calls return -ENOSYS. This abstraction of the I-pipe tracer is also
intended to map on Ingo Molnar’s latency tracer for -rt one day.
int xntrace_max_begin(unsigned long v)
int xntrace_max_end(unsigned long v)
int xntrace_user_stop(unsigned long v)
int xntrace_user_freeze(unsigned long v, int once)
int xntrace_special(unsigned char id, unsigned long v)
int xntrace_special_u64(unsigned char id, unsigned long long v)
The next services are only available to kernel space users.
int xntrace_pid(pid_t pid, short prio)
I-pipe Tracer API
This is the low-level I-pipe tracer API. It is only available if
CONFIG_IPIPE_TRACE is enabled.
void ipipe_trace_begin(unsigned long v)
void ipipe_trace_end(unsigned long v)
void ipipe_trace_freeze(unsigned long v)
void ipipe_trace_special(unsigned char special_id, unsigned long v)
void ipipe_trace_pid(pid_t pid, short prio)
void ipipe_trace_event(unsigned char id, unsigned long delay_tsc)