1. 15 Jun, 2011 1 commit
    • Jiri Olsa's avatar
      tracing, function_graph: Remove dependency of abstime and duration fields on latency · 321e68b0
      Jiri Olsa authored
      The display of absolute time and duration fields is based on the
      latency field. This was added during the irqsoff/wakeup tracers
      graph support changes.
      It's causing confusion in what fields will be displayed for the
      function_graph tracer itself. So I'm removing this depency, and
      adding absolute time and duration fields to the preemptirqsoff
      preemptoff irqsoff wakeup tracers.
      With following commands:
      	# echo function_graph > ./current_tracer
      	# cat trace
      This is what it looked like before:
      # tracer: function_graph
      #     TIME        CPU  DURATION                  FUNCTION CALLS
      #      |          |     |   |                     |   |   |   |
       0)   0.068 us    |          } /* page_add_file_rmap */
       0)               |          _raw_spin_unlock() {
      This is what it looks like now:
      # tracer: function_graph
      # CPU  DURATION                  FUNCTION CALLS
      # |     |   |                     |   |   |   |
       0)   0.068 us    |                } /* add_preempt_count */
       0)   0.993 us    |              } /* vfsmount_lock_local_lock */
      For preemptirqsoff preemptoff irqsoff wakeup tracers,
      this is what it looked like before:
      #                       _-----=> irqs-off
      #                      / _----=> need-resched
      #                     | / _---=> hardirq/softirq
      #                     || / _--=> preempt-depth
      #                     ||| / _-=> lock-depth
      #                     |||| /
      # CPU  TASK/PID       |||||  DURATION                  FUNCTION CALLS
      # |     |    |        |||||   |   |                     |   |   |   |
       1)    <idle>-0    |  d..1  0.000 us    |  acpi_idle_enter_simple();
      This is what it looks like now:
      #                                       _-----=> irqs-off
      #                                      / _----=> need-resched
      #                                     | / _---=> hardirq/softirq
      #                                     || / _--=> preempt-depth
      #                                     ||| /
      #     TIME        CPU  TASK/PID       ||||  DURATION                  FUNCTION CALLS
      #      |          |     |    |        ||||   |   |                     |   |   |   |
         19.847735 |   1)    <idle>-0    |  d..1  0.000 us    |  acpi_idle_enter_simple();
      Signed-off-by: default avatarJiri Olsa <jolsa@redhat.com>
      Link: http://lkml.kernel.org/r/1307113131-10045-2-git-send-email-jolsa@redhat.comSigned-off-by: default avatarSteven Rostedt <rostedt@goodmis.org>
  2. 31 Mar, 2011 1 commit
  3. 18 Oct, 2010 3 commits
    • Steven Rostedt's avatar
      tracing: Remove parent recording in latency tracer graph options · 78c89ba1
      Steven Rostedt authored
      Even though the parent is recorded with the normal function tracing
      of the latency tracers (irqsoff and wakeup), the function graph
      recording is bogus.
      This is due to the function graph messing with the return stack.
      The latency tracers pass in as the parent CALLER_ADDR0, which
      works fine for plain function tracing. But this causes bogus output
      with the graph tracer:
       3)    <idle>-0    |  d.s3.  0.000 us    |  return_to_handler();
       3)    <idle>-0    |  d.s3.  0.000 us    |  _raw_spin_unlock_irqrestore();
       3)    <idle>-0    |  d.s3.  0.000 us    |  return_to_handler();
       3)    <idle>-0    |  d.s3.  0.000 us    |  trace_hardirqs_on();
      The "return_to_handle()" call is the trampoline of the
      function graph tracer, and is meaningless in this context.
      Cc: Jiri Olsa <jolsa@redhat.com>
      Signed-off-by: default avatarSteven Rostedt <rostedt@goodmis.org>
    • Jiri Olsa's avatar
      tracing: Make graph related irqs/preemptsoff functions global · 0a772620
      Jiri Olsa authored
      Move trace_graph_function() and print_graph_headers_flags() functions
      to the trace_function_graph.c to be globaly available.
      Signed-off-by: default avatarJiri Olsa <jolsa@redhat.com>
      LKML-Reference: <1285243253-7372-3-git-send-email-jolsa@redhat.com>
      Signed-off-by: default avatarSteven Rostedt <rostedt@goodmis.org>
    • Jiri Olsa's avatar
      tracing: Add proper check for irq_depth routines · a9d61173
      Jiri Olsa authored
      The check_irq_entry and check_irq_return could be called
      from graph event context. In such case there's no graph
      private data allocated. Adding checks to handle this case.
      Signed-off-by: default avatarJiri Olsa <jolsa@redhat.com>
      LKML-Reference: <20100924154102.GB1818@jolsa.brq.redhat.com>
      [ Fixed some grammar in the comments ]
      Signed-off-by: default avatarSteven Rostedt <rostedt@goodmis.org>
  4. 13 Oct, 2010 1 commit
    • Borislav Petkov's avatar
      tracing: Fix function-graph build warning on 32-bit · 14cae9bd
      Borislav Petkov authored
      kernel/trace/trace_functions_graph.c: In function ‘trace_print_graph_duration’:
      kernel/trace/trace_functions_graph.c:652: warning: comparison of distinct pointer types lacks a cast
      when building 36-rc6 on a 32-bit due to the strict type check failing
      in the min() macro.
      Signed-off-by: default avatarBorislav Petkov <bp@alien8.de>
      Cc: Chase Douglas <chase.douglas@canonical.com>
      Cc: Steven Rostedt <rostedt@goodmis.org>
      Cc: Ingo Molnar <mingo@elte.hu>
      LKML-Reference: <20100929080823.GA13595@liondog.tnic>
      Signed-off-by: default avatarFrederic Weisbecker <fweisbec@gmail.com>
  5. 15 Sep, 2010 2 commits
    • Steven Rostedt's avatar
      tracing: Do not trace in irq when funcgraph-irq option is zero · b304d044
      Steven Rostedt authored
      When the function graph tracer funcgraph-irq option is zero, disable
      tracing in IRQs. This makes the option have two effects.
      1) When reading the trace file, do not display the functions that
         happen in interrupt context (when detected)
      2) [*new*] When recording a trace, skip those that are detected
         to be in interrupt by the 'in_irq()' function
      Note, in_irq() is updated at irq_enter() and irq_exit(). There are
      still functions that are recorded by the function graph tracer that
      is in interrupt context but outside the irq_enter/exit() routines.
      Signed-off-by: default avatarSteven Rostedt <rostedt@goodmis.org>
    • Jiri Olsa's avatar
      tracing: Add funcgraph-irq option for function graph tracer. · 2bd16212
      Jiri Olsa authored
      It's handy to be able to disable the irq related output
      and not to have to jump over each irq related code, when
      you have no interrest in it.
      The option is by default enabled, so there's no change to
      current behaviour. It affects only the final output, so all
      the irq related data stay in the ring buffer.
      Signed-off-by: default avatarJiri Olsa <jolsa@redhat.com>
      LKML-Reference: <20100907145344.GC1912@jolsa.brq.redhat.com>
      Signed-off-by: default avatarSteven Rostedt <rostedt@goodmis.org>
  6. 18 Aug, 2010 1 commit
  7. 06 Aug, 2010 1 commit
  8. 29 Jun, 2010 1 commit
    • Chase Douglas's avatar
      tracing/function-graph: Use correct string size for snprintf · d62f85d1
      Chase Douglas authored
      The nsecs_str string is a local variable defined as:
      char nsecs_str[5];
      It is possible for the snprintf call to use a size value larger than the
      size of the string. This should not cause a buffer overrun as it is
      written now due to the value for the string format "%03lu" can not be
      larger than 1000. However, this change makes it correct. By making the
      size correct we guard against potential future changes that could actually
      cause a buffer overrun.
      Signed-off-by: default avatarChase Douglas <chase.douglas@canonical.com>
      LKML-Reference: <1276619355-18116-1-git-send-email-chase.douglas@canonical.com>
      [ added 'UL' to number 8 to fix gcc warning comparing it to sizeof() ]
      Signed-off-by: default avatarSteven Rostedt <rostedt@goodmis.org>
  9. 14 May, 2010 1 commit
    • Steven Rostedt's avatar
      tracing: Allow events to share their print functions · a9a57763
      Steven Rostedt authored
      Multiple events may use the same method to print their data.
      Instead of having all events have a pointer to their print funtions,
      the trace_event structure now points to a trace_event_functions structure
      that will hold the way to print ouf the event.
      The event itself is now passed to the print function to let the print
      function know what kind of event it should print.
      This opens the door to consolidating the way several events print
      their output.
         text	   data	    bss	    dec	    hex	filename
      4913961	1088356	 861512	6863829	 68bbd5	vmlinux.orig
      4900382	1048964	 861512	6810858	 67ecea	vmlinux.init
      4900446	1049028	 861512	6810986	 67ed6a	vmlinux.preprint
      This change slightly increases the size but is needed for the next change.
      v3: Fix the branch tracer events to handle this change.
      v2: Fix the new function graph tracer event calls to handle this change.
      Acked-by: default avatarMathieu Desnoyers <mathieu.desnoyers@efficios.com>
      Acked-by: default avatarMasami Hiramatsu <mhiramat@redhat.com>
      Acked-by: default avatarFrederic Weisbecker <fweisbec@gmail.com>
      Signed-off-by: default avatarSteven Rostedt <rostedt@goodmis.org>
  10. 27 Apr, 2010 1 commit
  11. 26 Apr, 2010 2 commits
  12. 01 Apr, 2010 1 commit
    • Steven Rostedt's avatar
      ring-buffer: Add place holder recording of dropped events · 66a8cb95
      Steven Rostedt authored
      Currently, when the ring buffer drops events, it does not record
      the fact that it did so. It does inform the writer that the event
      was dropped by returning a NULL event, but it does not put in any
      place holder where the event was dropped.
      This is not a trivial thing to add because the ring buffer mostly
      runs in overwrite (flight recorder) mode. That is, when the ring
      buffer is full, new data will overwrite old data.
      In a produce/consumer mode, where new data is simply dropped when
      the ring buffer is full, it is trivial to add the placeholder
      for dropped events. When there's more room to write new data, then
      a special event can be added to notify the reader about the dropped
      But in overwrite mode, any new write can overwrite events. A place
      holder can not be inserted into the ring buffer since there never
      may be room. A reader could also come in at anytime and miss the
      Luckily, the way the ring buffer works, the read side can find out
      if events were lost or not, and how many events. Everytime a write
      takes place, if it overwrites the header page (the next read) it
      updates a "overrun" variable that keeps track of the number of
      lost events. When a reader swaps out a page from the ring buffer,
      it can record this number, perfom the swap, and then check to
      see if the number changed, and take the diff if it has, which would be
      the number of events dropped. This can be stored by the reader
      and returned to callers of the reader.
      Since the reader page swap will fail if the writer moved the head
      page since the time the reader page set up the swap, this gives room
      to record the overruns without worrying about races. If the reader
      sets up the pages, records the overrun, than performs the swap,
      if the swap succeeds, then the overrun variable has not been
      updated since the setup before the swap.
      For binary readers of the ring buffer, a flag is set in the header
      of each sub page (sub buffer) of the ring buffer. This flag is embedded
      in the size field of the data on the sub buffer, in the 31st bit (the size
      can be 32 or 64 bits depending on the architecture), but only 27
      bits needs to be used for the actual size (less actually).
      We could add a new field in the sub buffer header to also record the
      number of events dropped since the last read, but this will change the
      format of the binary ring buffer a bit too much. Perhaps this change can
      be made if the information on the number of events dropped is considered
      important enough.
      Note, the notification of dropped events is only used by consuming reads
      or peeking at the ring buffer. Iterating over the ring buffer does not
      keep this information because the necessary data is only available when
      a page swap is made, and the iterator does not swap out pages.
      Cc: Robert Richter <robert.richter@amd.com>
      Cc: Andi Kleen <andi@firstfloor.org>
      Cc: Li Zefan <lizf@cn.fujitsu.com>
      Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
      Cc: "Luis Claudio R. Goncalves" <lclaudio@uudg.org>
      Cc: Frederic Weisbecker <fweisbec@gmail.com>
      Signed-off-by: default avatarSteven Rostedt <rostedt@goodmis.org>
  13. 30 Mar, 2010 1 commit
    • Tejun Heo's avatar
      include cleanup: Update gfp.h and slab.h includes to prepare for breaking... · 5a0e3ad6
      Tejun Heo authored
      include cleanup: Update gfp.h and slab.h includes to prepare for breaking implicit slab.h inclusion from percpu.h
      percpu.h is included by sched.h and module.h and thus ends up being
      included when building most .c files.  percpu.h includes slab.h which
      in turn includes gfp.h making everything defined by the two files
      universally available and complicating inclusion dependencies.
      percpu.h -> slab.h dependency is about to be removed.  Prepare for
      this change by updating users of gfp and slab facilities include those
      headers directly instead of assuming availability.  As this conversion
      needs to touch large number of source files, the following script is
      used as the basis of conversion.
      The script does the followings.
      * Scan files for gfp and slab usages and update includes such that
        only the necessary includes are there.  ie. if only gfp is used,
        gfp.h, if slab is used, slab.h.
      * When the script inserts a new include, it looks at the include
        blocks and try to put the new include such that its order conforms
        to its surrounding.  It's put in the include block which contains
        core kernel includes, in the same order that the rest are ordered -
        alphabetical, Christmas tree, rev-Xmas-tree or at the end if there
        doesn't seem to be any matching order.
      * If the script can't find a place to put a new include (mostly
        because the file doesn't have fitting include block), it prints out
        an error message indicating which .h file needs to be added to the
      The conversion was done in the following steps.
      1. The initial automatic conversion of all .c files updated slightly
         over 4000 files, deleting around 700 includes and adding ~480 gfp.h
         and ~3000 slab.h inclusions.  The script emitted errors for ~400
      2. Each error was manually checked.  Some didn't need the inclusion,
         some needed manual addition while adding it to implementation .h or
         embedding .c file was more appropriate for others.  This step added
         inclusions to around 150 files.
      3. The script was run again and the output was compared to the edits
         from #2 to make sure no file was left behind.
      4. Several build tests were done and a couple of problems were fixed.
         e.g. lib/decompress_*.c used malloc/free() wrappers around slab
         APIs requiring slab.h to be added manually.
      5. The script was run on all .h files but without automatically
         editing them as sprinkling gfp.h and slab.h inclusions around .h
         files could easily lead to inclusion dependency hell.  Most gfp.h
         inclusion directives were ignored as stuff from gfp.h was usually
         wildly available and often used in preprocessor macros.  Each
         slab.h inclusion directive was examined and added manually as
      6. percpu.h was updated not to include slab.h.
      7. Build test were done on the following configurations and failures
         were fixed.  CONFIG_GCOV_KERNEL was turned off for all tests (as my
         distributed build env didn't work with gcov compiles) and a few
         more options had to be turned off depending on archs to make things
         build (like ipr on powerpc/64 which failed due to missing writeq).
         * x86 and x86_64 UP and SMP allmodconfig and a custom test config.
         * powerpc and powerpc64 SMP allmodconfig
         * sparc and sparc64 SMP allmodconfig
         * ia64 SMP allmodconfig
         * s390 SMP allmodconfig
         * alpha SMP allmodconfig
         * um on x86_64 SMP allmodconfig
      8. percpu.h modifications were reverted so that it could be applied as
         a separate patch and serve as bisection point.
      Given the fact that I had only a couple of failures from tests on step
      6, I'm fairly confident about the coverage of this conversion patch.
      If there is a breakage, it's likely to be something in one of the arch
      headers which should be easily discoverable easily on most builds of
      the specific arch.
      Signed-off-by: default avatarTejun Heo <tj@kernel.org>
      Guess-its-ok-by: default avatarChristoph Lameter <cl@linux-foundation.org>
      Cc: Ingo Molnar <mingo@redhat.com>
      Cc: Lee Schermerhorn <Lee.Schermerhorn@hp.com>
  14. 06 Mar, 2010 2 commits
    • Tim Bird's avatar
      function-graph: Add tracing_thresh support to function_graph tracer · 0e950173
      Tim Bird authored
      Add support for tracing_thresh to the function_graph tracer.  This
      version of this feature isolates the checks into new entry and
      return functions, to avoid adding more conditional code into the
      main function_graph paths.
      When the tracing_thresh is set and the function graph tracer is
      enabled, only the functions that took longer than the time in
      microseconds that was set in tracing_thresh are recorded. To do this
      efficiently, only the function exits are recorded:
       [tracing]# echo 100 > tracing_thresh
       [tracing]# echo function_graph > current_tracer
       [tracing]# cat trace
       # tracer: function_graph
       # CPU  DURATION                  FUNCTION CALLS
       # |     |   |                     |   |   |   |
        1) ! 119.214 us  |  } /* smp_apic_timer_interrupt */
        1)   <========== |
        0) ! 101.527 us  |              } /* __rcu_process_callbacks */
        0) ! 126.461 us  |            } /* rcu_process_callbacks */
        0) ! 145.111 us  |          } /* __do_softirq */
        0) ! 149.667 us  |        } /* do_softirq */
        0) ! 168.817 us  |      } /* irq_exit */
        0) ! 248.254 us  |    } /* smp_apic_timer_interrupt */
      Also, add support for specifying tracing_thresh on the kernel
      command line.  When used like so: "tracing_thresh=200 ftrace=function_graph"
      this can be used to analyse system startup.  It is important to disable
      tracing soon after boot, in order to avoid losing the trace data.
      Acked-by: default avatarFrederic Weisbecker <fweisbec@gmail.com>
      Signed-off-by: default avatarTim Bird <tim.bird@am.sony.com>
      LKML-Reference: <4B87098B.4040308@am.sony.com>
      Signed-off-by: default avatarSteven Rostedt <rostedt@goodmis.org>
    • Steven Rostedt's avatar
      function-graph: Use comment notation for func names of dangling '}' · a094fe04
      Steven Rostedt authored
      When a '}' does not have a matching function start, the name is printed
      within parenthesis. But this makes it confusing between ending '}'
      and function starts. This patch makes the function name appear in C comment
      Old view:
       3)   1.281 us    |            } (might_fault)
       3)   3.620 us    |          } (filldir)
       3)   5.251 us    |        } (call_filldir)
       3)               |        call_filldir() {
       3)               |          filldir() {
      New view:
       3)   1.281 us    |            } /* might_fault */
       3)   3.620 us    |          } /* filldir */
       3)   5.251 us    |        } /* call_filldir */
       3)               |        call_filldir() {
       3)               |          filldir() {
      Requested-by: default avatarIngo Molnar <mingo@elte.hu>
      Signed-off-by: default avatarSteven Rostedt <rostedt@goodmis.org>
  15. 27 Feb, 2010 1 commit
    • Steven Rostedt's avatar
      ftrace: Add function names to dangling } in function graph tracer · f1c7f517
      Steven Rostedt authored
      The function graph tracer is currently the most invasive tracer
      in the ftrace family. It can easily overflow the buffer even with
      10megs per CPU. This means that events can often be lost.
      On start up, or after events are lost, if the function return is
      recorded but the function enter was lost, all we get to see is the
      exiting '}'.
      Here is how a typical trace output starts:
       [tracing] cat trace
       # tracer: function_graph
       # CPU  DURATION                  FUNCTION CALLS
       # |     |   |                     |   |   |   |
        0) + 91.897 us   |                  }
        0) ! 567.961 us  |                }
        0)   <========== |
        0) ! 579.083 us  |                _raw_spin_lock_irqsave();
        0)   4.694 us    |                _raw_spin_unlock_irqrestore();
        0) ! 594.862 us  |              }
        0) ! 603.361 us  |            }
        0) ! 613.574 us  |          }
        0) ! 623.554 us  |        }
        0)   3.653 us    |        fget_light();
        0)               |        sock_poll() {
      There are a series of '}' with no matching "func() {". There's no information
      to what functions these ending brackets belong to.
      This patch adds a stack on the per cpu structure used in outputting
      the function graph tracer to keep track of what function was outputted.
      Then on a function exit event, it checks the depth to see if the
      function exit has a matching entry event. If it does, then it only
      prints the '}', otherwise it adds the function name after the '}'.
      This allows function exit events to show what function they belong to
      at trace output startup, when the entry was lost due to ring buffer
      overflow, or even after a new task is scheduled in.
      Here is what the above trace will look like after this patch:
       [tracing] cat trace
       # tracer: function_graph
       # CPU  DURATION                  FUNCTION CALLS
       # |     |   |                     |   |   |   |
        0) + 91.897 us   |                  } (irq_exit)
        0) ! 567.961 us  |                } (smp_apic_timer_interrupt)
        0)   <========== |
        0) ! 579.083 us  |                _raw_spin_lock_irqsave();
        0)   4.694 us    |                _raw_spin_unlock_irqrestore();
        0) ! 594.862 us  |              } (add_wait_queue)
        0) ! 603.361 us  |            } (__pollwait)
        0) ! 613.574 us  |          } (tcp_poll)
        0) ! 623.554 us  |        } (sock_poll)
        0)   3.653 us    |        fget_light();
        0)               |        sock_poll() {
      Signed-off-by: default avatarSteven Rostedt <rostedt@goodmis.org>
  16. 25 Feb, 2010 1 commit
  17. 29 Jan, 2010 1 commit
    • Lai Jiangshan's avatar
      tracing: Simplify test for function_graph tracing start point · ea2c68a0
      Lai Jiangshan authored
      In the function graph tracer, a calling function is to be traced
      only when it is enabled through the set_graph_function file,
      or when it is nested in an enabled function.
      Current code uses TSK_TRACE_FL_GRAPH to test whether it is nested
      or not. Looking at the code, we can get this:
      (trace->depth > 0) <==> (TSK_TRACE_FL_GRAPH is set)
      trace->depth is more explicit to tell that it is nested.
      So we use trace->depth directly and simplify the code.
      No functionality is changed.
      TSK_TRACE_FL_GRAPH is not removed yet, it is left for future usage.
      Signed-off-by: default avatarLai Jiangshan <laijs@cn.fujitsu.com>
      Cc: Ingo Molnar <mingo@elte.hu>
      Cc: Steven Rostedt <rostedt@goodmis.org>
      LKML-Reference: <4B4DB0B6.7040607@cn.fujitsu.com>
      Signed-off-by: default avatarFrederic Weisbecker <fweisbec@gmail.com>
  18. 17 Jan, 2010 1 commit
  19. 09 Dec, 2009 1 commit
    • Jiri Olsa's avatar
      tracing: Fix function graph trace_pipe to properly display failed entries · be1eca39
      Jiri Olsa authored
      There is a case where the graph tracer might get confused and omits
      displaying of a single record.  This applies mostly with the trace_pipe
      since it is unlikely that the trace_seq buffer will overflow with the
      trace file.
      As the function_graph tracer goes through the trace entries keeping a
      pointer to the current record:
      current ->  func1 ENTRY
                  func2 ENTRY
                  func2 RETURN
                  func1 RETURN
      When an function ENTRY is encountered, it moves the pointer to the
      next entry to check if the function is a nested or leaf function.
                  func1 ENTRY
      current ->  func2 ENTRY
                  func2 RETURN
                  func1 RETURN
      If the rest of the writing of the function fills the trace_seq buffer,
      then the trace_pipe read will ignore this entry. The next read will
      Now start at the current location, but the first entry (func1) will
      be discarded.
      This patch keeps a copy of the current entry in the iterator private
      storage and will keep track of when the trace_seq buffer fills. When
      the trace_seq buffer fills, it will reuse the copy of the entry in the
      next iteration.
        This patch has been largely modified by Steven Rostedt in order to
        clean it up and simplify it. The original idea and concept was from
        Jirka and for that, this patch will go under his name to give him
        the credit he deserves. But because this was modify by Steven Rostedt
        anything wrong with the patch should be blamed on Steven.
      Signed-off-by: default avatarJiri Olsa <jolsa@redhat.com>
      Cc: Frederic Weisbecker <fweisbec@gmail.com>
      LKML-Reference: <1259067458-27143-1-git-send-email-jolsa@redhat.com>
      Signed-off-by: default avatarSteven Rostedt <rostedt@goodmis.org>
  20. 29 Oct, 2009 1 commit
  21. 13 Oct, 2009 1 commit
  22. 17 Sep, 2009 1 commit
  23. 11 Sep, 2009 3 commits
    • Steven Rostedt's avatar
      tracing: consolidate code between trace_output.c and trace_function_graph.c · f81c972d
      Steven Rostedt authored
      Both trace_output.c and trace_function_graph.c do basically the same
      thing to handle the printing of the latency-format. This patch moves
      the code into one function that both can use.
      Signed-off-by: default avatarSteven Rostedt <rostedt@goodmis.org>
    • Steven Rostedt's avatar
      tracing: add lock depth to entries · 637e7e86
      Steven Rostedt authored
      This patch adds the lock depth of the big kernel lock to the generic
      entry header. This way we can see the depth of the lock and help
      in removing the BKL.
       #                  _------=> CPU#
       #                 / _-----=> irqs-off
       #                | / _----=> need-resched
       #                || / _---=> hardirq/softirq
       #                ||| / _--=> preempt-depth
       #                |||| /_--=> lock-depth
       #                |||||/     delay
       #  cmd     pid   |||||| time  |   caller
       #     \   /      ||||||   \   |   /
         <idle>-0       2.N..3 5902255250us+: lock_acquire: read rcu_read_lock
         <idle>-0       2.N..3 5902255253us+: lock_release: rcu_read_lock
         <idle>-0       2dN..3 5902255257us+: lock_acquire: xtime_lock
         <idle>-0       2dN..4 5902255259us : lock_acquire: clocksource_lock
         <idle>-0       2dN..4 5902255261us+: lock_release: clocksource_lock
      Signed-off-by: default avatarSteven Rostedt <rostedt@goodmis.org>
    • Steven Rostedt's avatar
      tracing: add latency format to function_graph tracer · 49ff5903
      Steven Rostedt authored
      While debugging something with the function_graph tracer, I found the
      need to see the preempt count of the traces. Unfortunately, since
      the function graph tracer has its own output formatting, it does not
      honor the latency-format option.
      This patch makes the function_graph tracer honor the latency-format
      option, but still keeps control of the output. But now we have the
      same details that the latency-format supplies.
       # tracer: function_graph
       #      _-----=> irqs-off
       #     / _----=> need-resched
       #    | / _---=> hardirq/softirq
       #    || / _--=> preempt-depth
       #    ||| /
       #    ||||
       # CPU||||  DURATION                  FUNCTION CALLS
       # |  ||||   |   |                     |   |   |   |
        3)  d..1  1.333 us    |        idle_cpu();
        3)  d.h1              |        tick_check_idle() {
        3)  d.h1  0.550 us    |          tick_check_oneshot_broadcast();
        3)  d.h1              |          tick_nohz_stop_idle() {
        3)  d.h1              |            ktime_get() {
        3)  d.h1              |              ktime_get_ts() {
      Signed-off-by: default avatarSteven Rostedt <rostedt@goodmis.org>
  24. 04 Sep, 2009 1 commit
    • Steven Rostedt's avatar
      tracing: pass around ring buffer instead of tracer · e77405ad
      Steven Rostedt authored
      The latency tracers (irqsoff and wakeup) can swap trace buffers
      on the fly. If an event is happening and has reserved data on one of
      the buffers, and the latency tracer swaps the global buffer with the
      max buffer, the result is that the event may commit the data to the
      wrong buffer.
      This patch changes the API to the trace recording to be recieve the
      buffer that was used to reserve a commit. Then this buffer can be passed
      in to the commit.
      Signed-off-by: default avatarSteven Rostedt <rostedt@goodmis.org>
  25. 06 Aug, 2009 2 commits
  26. 28 Jul, 2009 1 commit
    • Lai Jiangshan's avatar
      tracing: Fix invalid function_graph entry · 38ceb592
      Lai Jiangshan authored
      When print_graph_entry() computes a function call entry event, it needs
      to also check the next entry to guess if it matches the return event of
      the current function entry.
      In order to look at this next event, it needs to consume the current
      entry before going ahead in the ring buffer.
      However, if the current event that gets consumed is the last one in the
      ring buffer head page, the ring_buffer may reuse the page for writers.
      The consumed entry will then become invalid because of possible
      racy overwriting.
      Me must then handle this entry by making a copy of it.
      The fix also applies on 2.6.30
      Signed-off-by: default avatarLai Jiangshan <laijs@cn.fujitsu.com>
      Cc: Steven Rostedt <rostedt@goodmis.org>
      Cc: stable@kernel.org
      LKML-Reference: <4A6EEAEC.3050508@cn.fujitsu.com>
      Signed-off-by: default avatarFrederic Weisbecker <fweisbec@gmail.com>
  27. 10 Jul, 2009 1 commit
  28. 18 Jun, 2009 1 commit
    • Steven Rostedt's avatar
      function-graph: add stack frame test · 71e308a2
      Steven Rostedt authored
      In case gcc does something funny with the stack frames, or the return
      from function code, we would like to detect that.
      An arch may implement passing of a variable that is unique to the
      function and can be saved on entering a function and can be tested
      when exiting the function. Usually the frame pointer can be used for
      this purpose.
      This patch also implements this for x86. Where it passes in the stack
      frame of the parent function, and will test that frame on exit.
      There was a case in x86_32 with optimize for size (-Os) where, for a
      few functions, gcc would align the stack frame and place a copy of the
      return address into it. The function graph tracer modified the copy and
      not the actual return address. On return from the funtion, it did not go
      to the tracer hook, but returned to the parent. This broke the function
      graph tracer, because the return of the parent (where gcc did not do
      this funky manipulation) returned to the location that the child function
      was suppose to. This caused strange kernel crashes.
      This test detected the problem and pointed out where the issue was.
      This modifies the parameters of one of the functions that the arch
      specific code calls, so it includes changes to arch code to accommodate
      the new prototype.
      Note, I notice that the parsic arch implements its own push_return_trace.
      This is now a generic function and the ftrace_push_return_trace should be
      used instead. This patch does not touch that code.
      Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
      Cc: Paul Mackerras <paulus@samba.org>
      Cc: Heiko Carstens <heiko.carstens@de.ibm.com>
      Cc: Martin Schwidefsky <schwidefsky@de.ibm.com>
      Cc: Frederic Weisbecker <fweisbec@gmail.com>
      Cc: Helge Deller <deller@gmx.de>
      Cc: Kyle McMartin <kyle@mcmartin.ca>
      Signed-off-by: default avatarSteven Rostedt <rostedt@goodmis.org>
  29. 02 Jun, 2009 1 commit
    • Steven Rostedt's avatar
      function-graph: enable the stack after initialization of other variables · 82310a32
      Steven Rostedt authored
      The function graph tracer checks if the task_struct has ret_stack defined
      to know if it is OK or not to use it. The initialization is done for
      all tasks by one process, but the idle tasks use the same initialization
      used by new tasks.
      If an interrupt happens on an idle task that just had the ret_stack
      created, but before the rest of the initialization took place, then
      we can corrupt the return address of the functions.
      This patch moves the setting of the task_struct's ret_stack to after
      the other variables have been initialized.
      [ Impact: prevent kernel panic on idle task when starting function graph ]
      Signed-off-by: default avatarSteven Rostedt <rostedt@goodmis.org>
  30. 25 Mar, 2009 2 commits
    • Steven Rostedt's avatar
      function-graph: add option to calculate graph time or not · a2a16d6a
      Steven Rostedt authored
      graph time is the time that a function is executing another function.
      Thus if function A calls B, if graph-time is set, then the time for
      A includes B. This is the default behavior. But if graph-time is off,
      then the time spent executing B is subtracted from A.
      Signed-off-by: default avatarSteven Rostedt <srostedt@redhat.com>
    • Steven Rostedt's avatar
      tracing: adding function timings to function profiler · 0706f1c4
      Steven Rostedt authored
      If the function graph trace is enabled, the function profiler will
      use it to take the timing of the functions.
       cat /debug/tracing/trace_stat/functions
        Function                               Hit    Time
        --------                               ---    ----
        mwait_idle                             127    183028.4 us
        schedule                                26    151997.7 us
        __schedule                              31    151975.1 us
        sys_wait4                                2    74080.53 us
        do_wait                                  2    74077.80 us
        sys_newlstat                           138    39929.16 us
        do_path_lookup                         179    39845.79 us
        vfs_lstat_fd                           138    39761.97 us
        user_path_at                           153    39469.58 us
        path_walk                              179    39435.76 us
        __link_path_walk                       189    39143.73 us
      Note the times are skewed due to the function graph tracer not taking
      into account schedules.
      Signed-off-by: default avatarSteven Rostedt <srostedt@redhat.com>
  31. 24 Mar, 2009 1 commit