Skip to content
  • Nick Terrell's avatar
    btrfs: Add zstd support · 5c1aab1d
    Nick Terrell authored
    Add zstd compression and decompression support to BtrFS. zstd at its
    fastest level compresses almost as well as zlib, while offering much
    faster compression and decompression, approaching lzo speeds.
    
    I benchmarked btrfs with zstd compression against no compression, lzo
    compression, and zlib compression. I benchmarked two scenarios. Copying
    a set of files to btrfs, and then reading the files. Copying a tarball
    to btrfs, extracting it to btrfs, and then reading the extracted files.
    After every operation, I call `sync` and include the sync time.
    Between every pair of operations I unmount and remount the filesystem
    to avoid caching. The benchmark files can be found in the upstream
    zstd source repository under
    `contrib/linux-kernel/{btrfs-benchmark.sh,btrfs-extract-benchmark.sh}`
    [1] [2].
    
    I ran the benchmarks on a Ubuntu 14.04 VM with 2 cores and 4 GiB of RAM.
    The VM is running on a MacBook Pro with a 3.1 GHz Intel Core i7 processor,
    16 GB of RAM, and a SSD.
    
    The first compression benchmark is copying 10 copies of the unzipped
    Silesia corpus [3] into a BtrFS filesystem mounted with
    `-o compress-force=Method`. The decompression benchmark times how long
    it takes to `tar` all 10 copies into `/dev/null`. The compression ratio is
    measured by comparing the output of `df` and `du`. See the benchmark file
    [1] for details. I benchmarked multiple zstd compression levels, although
    the patch uses zstd level 1.
    
    | Method  | Ratio | Compression MB/s | Decompression speed |
    |---------|-------|------------------|---------------------|
    | None    |  0.99 |              504 |                 686 |
    | lzo     |  1.66 |              398 |                 442 |
    | zlib    |  2.58 |               65 |                 241 |
    | zstd 1  |  2.57 |              260 |                 383 |
    | zstd 3  |  2.71 |              174 |                 408 |
    | zstd 6  |  2.87 |               70 |                 398 |
    | zstd 9  |  2.92 |               43 |                 406 |
    | zstd 12 |  2.93 |               21 |                 408 |
    | zstd 15 |  3.01 |               11 |                 354 |
    
    The next benchmark first copies `linux-4.11.6.tar` [4] to btrfs. Then it
    measures the compression ratio, extracts the tar, and deletes the tar.
    Then it measures the compression ratio again, and `tar`s the extracted
    files into `/dev/null`. See the benchmark file [2] for details.
    
    | Method | Tar Ratio | Extract Ratio | Copy (s) | Extract (s)| Read (s) |
    |--------|-----------|---------------|----------|------------|----------|
    | None   |      0.97 |          0.78 |    0.981 |      5.501 |    8.807 |
    | lzo    |      2.06 |          1.38 |    1.631 |      8.458 |    8.585 |
    | zlib   |      3.40 |          1.86 |    7.750 |     21.544 |   11.744 |
    | zstd 1 |      3.57 |          1.85 |    2.579 |     11.479 |    9.389 |
    
    [1] https://github.com/facebook/zstd/blob/dev/contrib/linux-kernel/btrfs-benchmark.sh
    [2] https://github.com/facebook/zstd/blob/dev/contrib/linux-kernel/btrfs-extract-benchmark.sh
    [3] http://sun.aei.polsl.pl/~sdeor/index.php?page=silesia
    [4] https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.11.6.tar.xz
    
    zstd source repository: https://github.com/facebook/zstd
    
    
    
    Signed-off-by: default avatarNick Terrell <terrelln@fb.com>
    Signed-off-by: default avatarChris Mason <clm@fb.com>
    5c1aab1d