Skip to content
  • Linus Torvalds's avatar
    Merge branch 'zstd-minimal' of git://git.kernel.org/pub/scm/linux/kernel/git/mason/linux-btrfs · e7cdb60f
    Linus Torvalds authored
    Pull zstd support from Chris Mason:
     "Nick Terrell's patch series to add zstd support to the kernel has been
      floating around for a while. After talking with Dave Sterba, Herbert
      and Phillip, we decided to send the whole thing in as one pull
      request.
    
      zstd is a big win in speed over zlib and in compression ratio over
      lzo, and the compression team here at FB has gotten great results
      using it in production. Nick will continue to update the kernel side
      with new improvements from the open source zstd userland code.
    
      Nick has a number of benchmarks for the main zstd code in his lib/zstd
      commit:
    
          I ran the benchmarks on a Ubuntu 14.04 VM with 2 cores and 4 GiB
          of RAM. The VM is running on a MacBook Pro with a 3.1 GHz Intel
          Core i7 processor, 16 GB of RAM, and a SSD. I benchmarked using
          `silesia.tar` [3], which is 211,988,480 B large. Run the following
          commands for the benchmark:
    
            sudo modprobe zstd_compress_test
            sudo mknod zstd_compress_test c 245 0
            sudo cp silesia.tar zstd_compress_test
    
          The time is reported by the time of the userland `cp`.
          The MB/s is computed with
    
            1,536,217,008 B / time(buffer size, hash)
    
          which includes the time to copy from userland.
          The Adjusted MB/s is computed with
    
            1,536,217,088 B / (time(buffer size, hash) - time(buffer size, none)).
    
          The memory reported is the amount of memory the compressor
          requests.
    
            | Method   | Size (B) | Time (s) | Ratio | MB/s    | Adj MB/s | Mem (MB) |
            |----------|----------|----------|-------|---------|----------|----------|
            | none     | 11988480 |    0.100 |     1 | 2119.88 |        - |        - |
            | zstd -1  | 73645762 |    1.044 | 2.878 |  203.05 |   224.56 |     1.23 |
            | zstd -3  | 66988878 |    1.761 | 3.165 |  120.38 |   127.63 |     2.47 |
            | zstd -5  | 65001259 |    2.563 | 3.261 |   82.71 |    86.07 |     2.86 |
            | zstd -10 | 60165346 |   13.242 | 3.523 |   16.01 |    16.13 |    13.22 |
            | zstd -15 | 58009756 |   47.601 | 3.654 |    4.45 |     4.46 |    21.61 |
            | zstd -19 | 54014593 |  102.835 | 3.925 |    2.06 |     2.06 |    60.15 |
            | zlib -1  | 77260026 |    2.895 | 2.744 |   73.23 |    75.85 |     0.27 |
            | zlib -3  | 72972206 |    4.116 | 2.905 |   51.50 |    52.79 |     0.27 |
            | zlib -6  | 68190360 |    9.633 | 3.109 |   22.01 |    22.24 |     0.27 |
            | zlib -9  | 67613382 |   22.554 | 3.135 |    9.40 |     9.44 |     0.27 |
    
          I benchmarked zstd decompression using the same method on the same
          machine. The benchmark file is located in the upstream zstd repo
          under `contrib/linux-kernel/zstd_decompress_test.c` [4]. The
          memory reported is the amount of memory required to decompress
          data compressed with the given compression level. If you know the
          maximum size of your input, you can reduce the memory usage of
          decompression irrespective of the compression level.
    
            | Method   | Time (s) | MB/s    | Adjusted MB/s | Memory (MB) |
            |----------|----------|---------|---------------|-------------|
            | none     |    0.025 | 8479.54 |             - |           - |
            | zstd -1  |    0.358 |  592.15 |        636.60 |        0.84 |
            | zstd -3  |    0.396 |  535.32 |        571.40 |        1.46 |
            | zstd -5  |    0.396 |  535.32 |        571.40 |        1.46 |
            | zstd -10 |    0.374 |  566.81 |        607.42 |        2.51 |
            | zstd -15 |    0.379 |  559.34 |        598.84 |        4.61 |
            | zstd -19 |    0.412 |  514.54 |        547.77 |        8.80 |
            | zlib -1  |    0.940 |  225.52 |        231.68 |        0.04 |
            | zlib -3  |    0.883 |  240.08 |        247.07 |        0.04 |
            | zlib -6  |    0.844 |  251.17 |        258.84 |        0.04 |
            | zlib -9  |    0.837 |  253.27 |        287.64 |        0.04 |
    
      I ran a long series of tests and benchmarks on the btrfs side and the
      gains are very similar to the core benchmarks Nick ran"
    
    * 'zstd-minimal' of git://git.kernel.org/pub/scm/linux/kernel/git/mason/linux-btrfs:
      squashfs: Add zstd support
      btrfs: Add zstd support
      lib: Add zstd modules
      lib: Add xxhash module
    e7cdb60f