Skip to content
  • Jiri Olsa's avatar
    sched/core: Move sched_entity::avg into separate cache line · 5a107804
    Jiri Olsa authored
    
    
    The sched_entity::avg collides with read-mostly sched_entity data.
    
    The perf c2c tool showed many read HITM accesses across
    many CPUs for sched_entity's cfs_rq and my_q, while having
    at the same time tons of stores for avg.
    
    After placing sched_entity::avg into separate cache line,
    the perf bench sched pipe showed around 20 seconds speedup.
    
    NOTE I cut out all perf events except for cycles and
    instructions from following output.
    
    Before:
      $ perf stat -r 5 perf bench sched pipe -l 10000000
      # Running 'sched/pipe' benchmark:
      # Executed 10000000 pipe operations between two processes
    
           Total time: 270.348 [sec]
    
            27.034805 usecs/op
                36989 ops/sec
       ...
    
         245,537,074,035      cycles                    #    1.433 GHz
         187,264,548,519      instructions              #    0.77  insns per cycle
    
           272.653840535 seconds time elapsed           ( +-  1.31% )
    
    After:
      $ perf stat -r 5 perf bench sched pipe -l 10000000
      # Running 'sched/pipe' benchmark:
      # Executed 10000000 pipe operations between two processes
    
           Total time: 251.076 [sec]
    
            25.107678 usecs/op
                39828 ops/sec
      ...
    
         244,573,513,928      cycles                    #    1.572 GHz
         187,409,641,157      instructions              #    0.76  insns per cycle
    
           251.679315188 seconds time elapsed           ( +-  0.31% )
    
    Signed-off-by: default avatarJiri Olsa <jolsa@kernel.org>
    Signed-off-by: default avatarPeter Zijlstra (Intel) <peterz@infradead.org>
    Cc: Arnaldo Carvalho de Melo <acme@kernel.org>
    Cc: Don Zickus <dzickus@redhat.com>
    Cc: Joe Mario <jmario@redhat.com>
    Cc: Linus Torvalds <torvalds@linux-foundation.org>
    Cc: Mike Galbraith <efault@gmx.de>
    Cc: Peter Zijlstra <peterz@infradead.org>
    Cc: Thomas Gleixner <tglx@linutronix.de>
    Link: http://lkml.kernel.org/r/1449606239-28602-1-git-send-email-jolsa@kernel.org
    
    
    Signed-off-by: default avatarIngo Molnar <mingo@kernel.org>
    5a107804