• Pavel Tatashin's avatar
    mm: stop zeroing memory during allocation in vmemmap · f7f99100
    Pavel Tatashin authored
    vmemmap_alloc_block() will no longer zero the block, so zero memory at
    its call sites for everything except struct pages.  Struct page memory
    is zero'd by struct page initialization.
    
    Replace allocators in sparse-vmemmap to use the non-zeroing version.
    So, we will get the performance improvement by zeroing the memory in
    parallel when struct pages are zeroed.
    
    Add struct page zeroing as a part of initialization of other fields in
    __init_single_page().
    
    This single thread performance collected on: Intel(R) Xeon(R) CPU E7-8895
    v3 @ 2.60GHz with 1T of memory (268400646 pages in 8 nodes):
    
                             BASE            FIX
    sparse_init     11.244671836s   0.007199623s
    zone_sizes_init  4.879775891s   8.355182299s
                      --------------------------
    Total           16.124447727s   8.362381922s
    
    sparse_init is where memory for struct pages is zeroed, and the zeroing
    part is moved later in this patch into __init_single_page(), which is
    called from zone_sizes_init().
    
    [akpm@linux-foundation.org: make vmemmap_alloc_block_zero() private to sparse-vmemmap.c]
    Link: http://lkml.kernel.org/r/20171013173214.27300-10-pasha.tatashin@oracle.comSigned-off-by: 's avatarPavel Tatashin <pasha.tatashin@oracle.com>
    Reviewed-by: 's avatarSteven Sistare <steven.sistare@oracle.com>
    Reviewed-by: 's avatarDaniel Jordan <daniel.m.jordan@oracle.com>
    Reviewed-by: 's avatarBob Picco <bob.picco@oracle.com>
    Tested-by: 's avatarBob Picco <bob.picco@oracle.com>
    Acked-by: 's avatarMichal Hocko <mhocko@suse.com>
    Cc: Alexander Potapenko <glider@google.com>
    Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
    Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org>
    Cc: Catalin Marinas <catalin.marinas@arm.com>
    Cc: Christian Borntraeger <borntraeger@de.ibm.com>
    Cc: David S. Miller <davem@davemloft.net>
    Cc: Dmitry Vyukov <dvyukov@google.com>
    Cc: Heiko Carstens <heiko.carstens@de.ibm.com>
    Cc: "H. Peter Anvin" <hpa@zytor.com>
    Cc: Ingo Molnar <mingo@redhat.com>
    Cc: Mark Rutland <mark.rutland@arm.com>
    Cc: Matthew Wilcox <willy@infradead.org>
    Cc: Mel Gorman <mgorman@techsingularity.net>
    Cc: Michal Hocko <mhocko@kernel.org>
    Cc: Sam Ravnborg <sam@ravnborg.org>
    Cc: Thomas Gleixner <tglx@linutronix.de>
    Cc: Will Deacon <will.deacon@arm.com>
    Signed-off-by: 's avatarAndrew Morton <akpm@linux-foundation.org>
    Signed-off-by: 's avatarLinus Torvalds <torvalds@linux-foundation.org>
    f7f99100