• Joonsoo Kim's avatar
    mm/compaction: speed up pageblock_pfn_to_page() when zone is contiguous · 7cf91a98
    Joonsoo Kim authored
    There is a performance drop report due to hugepage allocation and in
    there half of cpu time are spent on pageblock_pfn_to_page() in
    compaction [1].
    
    In that workload, compaction is triggered to make hugepage but most of
    pageblocks are un-available for compaction due to pageblock type and
    skip bit so compaction usually fails.  Most costly operations in this
    case is to find valid pageblock while scanning whole zone range.  To
    check if pageblock is valid to compact, valid pfn within pageblock is
    required and we can obtain it by calling pageblock_pfn_to_page().  This
    function checks whether pageblock is in a single zone and return valid
    pfn if possible.  Problem is that we need to check it every time before
    scanning pageblock even if we re-visit it and this turns out to be very
    expensive in this workload.
    
    Although we have no way to skip this pageblock check in the system where
    hole exists at arbitrary position, we can use cached value for zone
    continuity and just do pfn_to_page() in the system where hole doesn't
    exist.  This optimization considerably speeds up in above workload.
    
    Before vs After
      Max: 1096 MB/s vs 1325 MB/s
      Min: 635 MB/s 1015 MB/s
      Avg: 899 MB/s 1194 MB/s
    
    Avg is improved by roughly 30% [2].
    
    [1]: http://www.spinics.net/lists/linux-mm/msg97378.html
    [2]: https://lkml.org/lkml/2015/12/9/23
    
    [akpm@linux-foundation.org: don't forget to restore zone->contiguous on error path, per Vlastimil]
    Signed-off-by: 's avatarJoonsoo Kim <iamjoonsoo.kim@lge.com>
    Reported-by: 's avatarAaron Lu <aaron.lu@intel.com>
    Acked-by: 's avatarVlastimil Babka <vbabka@suse.cz>
    Tested-by: 's avatarAaron Lu <aaron.lu@intel.com>
    Cc: Mel Gorman <mgorman@suse.de>
    Cc: Rik van Riel <riel@redhat.com>
    Cc: David Rientjes <rientjes@google.com>
    Signed-off-by: 's avatarAndrew Morton <akpm@linux-foundation.org>
    Signed-off-by: 's avatarLinus Torvalds <torvalds@linux-foundation.org>
    7cf91a98
Name
Last commit
Last update
Documentation Loading commit data...
arch Loading commit data...
block Loading commit data...
certs Loading commit data...
crypto Loading commit data...
drivers Loading commit data...
firmware Loading commit data...
fs Loading commit data...
include Loading commit data...
init Loading commit data...
ipc Loading commit data...
kernel Loading commit data...
lib Loading commit data...
mm Loading commit data...
net Loading commit data...
samples Loading commit data...
scripts Loading commit data...
security Loading commit data...
sound Loading commit data...
tools Loading commit data...
usr Loading commit data...
virt Loading commit data...
.get_maintainer.ignore Loading commit data...
.gitignore Loading commit data...
.mailmap Loading commit data...
COPYING Loading commit data...
CREDITS Loading commit data...
Kbuild Loading commit data...
Kconfig Loading commit data...
MAINTAINERS Loading commit data...
Makefile Loading commit data...
README Loading commit data...
REPORTING-BUGS Loading commit data...