Commit 24fa0402 authored by Lasse Collin's avatar Lasse Collin Committed by Linus Torvalds

decompressors: add XZ decompressor module

In userspace, the .lzma format has become mostly a legacy file format that
got superseded by the .xz format.  Similarly, LZMA Utils was superseded by
XZ Utils.

These patches add support for XZ decompression into the kernel.  Most of
the code is as is from XZ Embedded <>.
It was written for the Linux kernel but is usable in other projects too.

Advantages of XZ over the current LZMA code in the kernel:
  - Nice API that can be used by other kernel modules; it's
    not limited to kernel, initramfs, and initrd decompression.
  - Integrity check support (CRC32)
  - BCJ filters improve compression of executable code on
    certain architectures. These together with LZMA2 can
    produce a few percent smaller kernel or Squashfs images
    than plain LZMA without making the decompression slower.

This patch: Add the main decompression code (xz_dec), testing module
(xz_dec_test), wrapper script ( for the xz command line tool,
and documentation.  The xz_dec module is enough to have a usable XZ
decompressor e.g.  for Squashfs.
Signed-off-by: default avatarLasse Collin <>
Cc: "H. Peter Anvin" <>
Cc: Alain Knaff <>
Cc: Albin Tonnerre <>
Cc: Phillip Lougher <>
Signed-off-by: default avatarAndrew Morton <>
Signed-off-by: default avatarLinus Torvalds <>
parent fb7fa589
XZ data compression in Linux
XZ is a general purpose data compression format with high compression
ratio and relatively fast decompression. The primary compression
algorithm (filter) is LZMA2. Additional filters can be used to improve
compression ratio even further. E.g. Branch/Call/Jump (BCJ) filters
improve compression ratio of executable data.
The XZ decompressor in Linux is called XZ Embedded. It supports
the LZMA2 filter and optionally also BCJ filters. CRC32 is supported
for integrity checking. The home page of XZ Embedded is at
<>, where you can find the
latest version and also information about using the code outside
the Linux kernel.
For userspace, XZ Utils provide a zlib-like compression library
and a gzip-like command line tool. XZ Utils can be downloaded from
XZ related components in the kernel
The xz_dec module provides XZ decompressor with single-call (buffer
to buffer) and multi-call (stateful) APIs. The usage of the xz_dec
module is documented in include/linux/xz.h.
The xz_dec_test module is for testing xz_dec. xz_dec_test is not
useful unless you are hacking the XZ decompressor. xz_dec_test
allocates a char device major dynamically to which one can write
.xz files from userspace. The decompressed output is thrown away.
Keep an eye on dmesg to see diagnostics printed by xz_dec_test.
See the xz_dec_test source code for the details.
For decompressing the kernel image, initramfs, and initrd, there
is a wrapper function in lib/decompress_unxz.c. Its API is the
same as in other decompress_*.c files, which is defined in
scripts/ is a wrapper for the xz command line tool found
from XZ Utils. The wrapper sets compression options to values suitable
for compressing the kernel image.
For kernel makefiles, two commands are provided for use with
$(call if_needed). The kernel image should be compressed with
$(call if_needed,xzkern) which will use a BCJ filter and a big LZMA2
dictionary. It will also append a four-byte trailer containing the
uncompressed size of the file, which is needed by the boot code.
Other things should be compressed with $(call if_needed,xzmisc)
which will use no BCJ filter and 1 MiB LZMA2 dictionary.
Notes on compression options
Since the XZ Embedded supports only streams with no integrity check or
CRC32, make sure that you don't use some other integrity check type
when encoding files that are supposed to be decoded by the kernel. With
liblzma, you need to use either LZMA_CHECK_NONE or LZMA_CHECK_CRC32
when encoding. With the xz command line tool, use --check=none or
Using CRC32 is strongly recommended unless there is some other layer
which will verify the integrity of the uncompressed data anyway.
Double checking the integrity would probably be waste of CPU cycles.
Note that the headers will always have a CRC32 which will be validated
by the decoder; you can only change the integrity check type (or
disable it) for the actual uncompressed data.
In userspace, LZMA2 is typically used with dictionary sizes of several
megabytes. The decoder needs to have the dictionary in RAM, thus big
dictionaries cannot be used for files that are intended to be decoded
by the kernel. 1 MiB is probably the maximum reasonable dictionary
size for in-kernel use (maybe more is OK for initramfs). The presets
in XZ Utils may not be optimal when creating files for the kernel,
so don't hesitate to use custom settings. Example:
xz --check=crc32 --lzma2=dict=512KiB inputfile
An exception to above dictionary size limitation is when the decoder
is used in single-call mode. Decompressing the kernel itself is an
example of this situation. In single-call mode, the memory usage
doesn't depend on the dictionary size, and it is perfectly fine to
use a big dictionary: for maximum compression, the dictionary should
be at least as big as the uncompressed data itself.
Future plans
Creating a limited XZ encoder may be considered if people think it is
useful. LZMA2 is slower to compress than e.g. Deflate or LZO even at
the fastest settings, so it isn't clear if LZMA2 encoder is wanted
into the kernel.
Support for limited random-access reading is planned for the
decompression code. I don't know if it could have any use in the
kernel, but I know that it would be useful in some embedded projects
outside the Linux kernel.
Conformance to the .xz file format specification
There are a couple of corner cases where things have been simplified
at expense of detecting errors as early as possible. These should not
matter in practice all, since they don't cause security issues. But
it is good to know this if testing the code e.g. with the test files
from XZ Utils.
Reporting bugs
Before reporting a bug, please check that it's not fixed already
at upstream. See <> to get the
latest code.
Report bugs to <> or visit #tukaani on
Freenode and talk to Larhzu. I don't actively read LKML or other
kernel-related mailing lists, so if there's something I should know,
you should email to me personally or use IRC.
Don't bother Igor Pavlov with questions about the XZ implementation
in the kernel or about XZ Utils. While these two implementations
include essential code that is directly based on Igor Pavlov's code,
these implementations aren't maintained nor supported by him.
This diff is collapsed.
......@@ -106,6 +106,8 @@ config LZO_COMPRESS
source "lib/xz/Kconfig"
# These all provide a common interface (hence the apparent duplication with
# ZLIB_INFLATE; DECOMPRESS_GZIP is just a wrapper.)
......@@ -69,6 +69,7 @@ obj-$(CONFIG_ZLIB_DEFLATE) += zlib_deflate/
obj-$(CONFIG_REED_SOLOMON) += reed_solomon/
obj-$(CONFIG_LZO_COMPRESS) += lzo/
obj-$(CONFIG_XZ_DEC) += xz/
obj-$(CONFIG_RAID6_PQ) += raid6/
lib-$(CONFIG_DECOMPRESS_GZIP) += decompress_inflate.o
config XZ_DEC
tristate "XZ decompression support"
select CRC32
LZMA2 compression algorithm and BCJ filters are supported using
the .xz file format as the container. For integrity checking,
CRC32 is supported. See Documentation/xz.txt for more information.
config XZ_DEC_X86
bool "x86 BCJ filter decoder" if EMBEDDED
default y
depends on XZ_DEC
select XZ_DEC_BCJ
bool "PowerPC BCJ filter decoder" if EMBEDDED
default y
depends on XZ_DEC
select XZ_DEC_BCJ
config XZ_DEC_IA64
bool "IA-64 BCJ filter decoder" if EMBEDDED
default y
depends on XZ_DEC
select XZ_DEC_BCJ
config XZ_DEC_ARM
bool "ARM BCJ filter decoder" if EMBEDDED
default y
depends on XZ_DEC
select XZ_DEC_BCJ
bool "ARM-Thumb BCJ filter decoder" if EMBEDDED
default y
depends on XZ_DEC
select XZ_DEC_BCJ
bool "SPARC BCJ filter decoder" if EMBEDDED
default y
depends on XZ_DEC
select XZ_DEC_BCJ
config XZ_DEC_BCJ
default n
config XZ_DEC_TEST
tristate "XZ decompressor tester"
default n
depends on XZ_DEC
This allows passing .xz files to the in-kernel XZ decoder via
a character special file. It calculates CRC32 of the decompressed
data and writes diagnostics to the system log.
Unless you are developing the XZ decoder, you don't need this
and should say N.
obj-$(CONFIG_XZ_DEC) += xz_dec.o
xz_dec-y := xz_dec_syms.o xz_dec_stream.o xz_dec_lzma2.o
xz_dec-$(CONFIG_XZ_DEC_BCJ) += xz_dec_bcj.o
obj-$(CONFIG_XZ_DEC_TEST) += xz_dec_test.o
* CRC32 using the polynomial from IEEE-802.3
* Authors: Lasse Collin <>
* Igor Pavlov <>
* This file has been put into the public domain.
* You can do whatever you want with this file.
* This is not the fastest implementation, but it is pretty compact.
* The fastest versions of xz_crc32() on modern CPUs without hardware
* accelerated CRC instruction are 3-5 times as fast as this version,
* but they are bigger and use more memory for the lookup table.
#include "xz_private.h"
* STATIC_RW_DATA is used in the pre-boot environment on some architectures.
* See <linux/decompress/mm.h> for details.
# define STATIC_RW_DATA static
STATIC_RW_DATA uint32_t xz_crc32_table[256];
XZ_EXTERN void xz_crc32_init(void)
const uint32_t poly = 0xEDB88320;
uint32_t i;
uint32_t j;
uint32_t r;
for (i = 0; i < 256; ++i) {
r = i;
for (j = 0; j < 8; ++j)
r = (r >> 1) ^ (poly & ~((r & 1) - 1));
xz_crc32_table[i] = r;
XZ_EXTERN uint32_t xz_crc32(const uint8_t *buf, size_t size, uint32_t crc)
crc = ~crc;
while (size != 0) {
crc = xz_crc32_table[*buf++ ^ (crc & 0xFF)] ^ (crc >> 8);
return ~crc;
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
* XZ decoder module information
* Author: Lasse Collin <>
* This file has been put into the public domain.
* You can do whatever you want with this file.
#include <linux/module.h>
#include <linux/xz.h>
MODULE_DESCRIPTION("XZ decompressor");
MODULE_AUTHOR("Lasse Collin <> and Igor Pavlov");
* This code is in the public domain, but in Linux it's simplest to just
* say it's GPL and consider the authors as the copyright holders.
* XZ decoder tester
* Author: Lasse Collin <>
* This file has been put into the public domain.
* You can do whatever you want with this file.
#include <linux/kernel.h>
#include <linux/module.h>
#include <linux/fs.h>
#include <linux/uaccess.h>
#include <linux/crc32.h>
#include <linux/xz.h>
/* Maximum supported dictionary size */
#define DICT_MAX (1 << 20)
/* Device name to pass to register_chrdev(). */
#define DEVICE_NAME "xz_dec_test"
/* Dynamically allocated device major number */
static int device_major;
* We reuse the same decoder state, and thus can decode only one
* file at a time.
static bool device_is_open;
/* XZ decoder state */
static struct xz_dec *state;
* Return value of xz_dec_run(). We need to avoid calling xz_dec_run() after
* it has returned XZ_STREAM_END, so we make this static.
static enum xz_ret ret;
* Input and output buffers. The input buffer is used as a temporary safe
* place for the data coming from the userspace.
static uint8_t buffer_in[1024];
static uint8_t buffer_out[1024];
* Structure to pass the input and output buffers to the XZ decoder.
* A few of the fields are never modified so we initialize them here.
static struct xz_buf buffers = {
.in = buffer_in,
.out = buffer_out,
.out_size = sizeof(buffer_out)
* CRC32 of uncompressed data. This is used to give the user a simple way
* to check that the decoder produces correct output.
static uint32_t crc;
static int xz_dec_test_open(struct inode *i, struct file *f)
if (device_is_open)
return -EBUSY;
device_is_open = true;
ret = XZ_OK;
crc = 0xFFFFFFFF;
buffers.in_pos = 0;
buffers.in_size = 0;
buffers.out_pos = 0;
printk(KERN_INFO DEVICE_NAME ": opened\n");
return 0;
static int xz_dec_test_release(struct inode *i, struct file *f)
device_is_open = false;
if (ret == XZ_OK)
printk(KERN_INFO DEVICE_NAME ": input was truncated\n");
printk(KERN_INFO DEVICE_NAME ": closed\n");
return 0;
* Decode the data given to us from the userspace. CRC32 of the uncompressed
* data is calculated and is printed at the end of successful decoding. The
* uncompressed data isn't stored anywhere for further use.
* The .xz file must have exactly one Stream and no Stream Padding. The data
* after the first Stream is considered to be garbage.
static ssize_t xz_dec_test_write(struct file *file, const char __user *buf,
size_t size, loff_t *pos)
size_t remaining;
if (ret != XZ_OK) {
if (size > 0)
printk(KERN_INFO DEVICE_NAME ": %zu bytes of "
"garbage at the end of the file\n",
return -ENOSPC;
printk(KERN_INFO DEVICE_NAME ": decoding %zu bytes of input\n",
remaining = size;
while ((remaining > 0 || buffers.out_pos == buffers.out_size)
&& ret == XZ_OK) {
if (buffers.in_pos == buffers.in_size) {
buffers.in_pos = 0;
buffers.in_size = min(remaining, sizeof(buffer_in));
if (copy_from_user(buffer_in, buf, buffers.in_size))
return -EFAULT;
buf += buffers.in_size;
remaining -= buffers.in_size;
buffers.out_pos = 0;
ret = xz_dec_run(state, &buffers);
crc = crc32(crc, buffer_out, buffers.out_pos);
switch (ret) {
case XZ_OK:
return size;
"CRC32 = 0x%08X\n", ~crc);
return size - remaining - (buffers.in_size - buffers.in_pos);
printk(KERN_INFO DEVICE_NAME ": Bug detected!\n");
return -EIO;
/* Allocate the XZ decoder state and register the character device. */
static int __init xz_dec_test_init(void)
static const struct file_operations fileops = {
.owner = THIS_MODULE,
.open = &xz_dec_test_open,
.release = &xz_dec_test_release,
.write = &xz_dec_test_write
state = xz_dec_init(XZ_PREALLOC, DICT_MAX);
if (state == NULL)
return -ENOMEM;
device_major = register_chrdev(0, DEVICE_NAME, &fileops);
if (device_major < 0) {
return device_major;
printk(KERN_INFO DEVICE_NAME ": module loaded\n");
printk(KERN_INFO DEVICE_NAME ": Create a device node with "
"'mknod " DEVICE_NAME " c %d 0' and write .xz files "
"to it.\n", device_major);
return 0;
static void __exit xz_dec_test_exit(void)
unregister_chrdev(device_major, DEVICE_NAME);
printk(KERN_INFO DEVICE_NAME ": module unloaded\n");
MODULE_DESCRIPTION("XZ decompressor tester");
MODULE_AUTHOR("Lasse Collin <>");
* This code is in the public domain, but in Linux it's simplest to just
* say it's GPL and consider the authors as the copyright holders.
* LZMA2 definitions
* Authors: Lasse Collin <>
* Igor Pavlov <>
* This file has been put into the public domain.
* You can do whatever you want with this file.
#ifndef XZ_LZMA2_H
#define XZ_LZMA2_H
/* Range coder constants */
#define RC_SHIFT_BITS 8
#define RC_TOP_BITS 24
#define RC_TOP_VALUE (1 << RC_TOP_BITS)
#define RC_MOVE_BITS 5
* Maximum number of position states. A position state is the lowest pb
* number of bits of the current uncompressed offset. In some places there
* are different sets of probabilities for different position states.
#define POS_STATES_MAX (1 << 4)
* This enum is used to track which LZMA symbols have occurred most recently
* and in which order. This information is used to predict the next symbol.
* Symbols:
* - Literal: One 8-bit byte
* - Match: Repeat a chunk of data at some distance
* - Long repeat: Multi-byte match at a recently seen distance
* - Short repeat: One-byte repeat at a recently seen distance
* The symbol names are in from STATE_oldest_older_previous. REP means
* either short or long repeated match, and NONLIT means any non-literal.
enum lzma_state {
/* Total number of states */
#define STATES 12
/* The lowest 7 states indicate that the previous state was a literal. */
#define LIT_STATES 7
/* Indicate that the latest symbol was a literal. */
static inline void lzma_state_literal(enum lzma_state *state)
*state = STATE_LIT_LIT;
else if (*state <= STATE_LIT_SHORTREP)
*state -= 3;
*state -= 6;
/* Indicate that the latest symbol was a match. */
static inline void lzma_state_match(enum lzma_state *state)
/* Indicate that the latest state was a long repeated match. */
static inline void lzma_state_long_rep(enum lzma_state *state)
/* Indicate that the latest symbol was a short match. */
static inline void lzma_state_short_rep(enum lzma_state *state)
/* Test if the previous symbol was a literal. */
static inline bool lzma_state_is_literal(enum lzma_state state)
return state < LIT_STATES;
/* Each literal coder is divided in three sections:
* - 0x001-0x0FF: Without match byte
* - 0x101-0x1FF: With match byte; match bit is 0
* - 0x201-0x2FF: With match byte; match bit is 1
* Match byte is used when the previous LZMA symbol was something else than
* a literal (that is, it was some kind of match).
#define LITERAL_CODER_SIZE 0x300
/* Maximum number of literal coders */
#define LITERAL_CODERS_MAX (1 << 4)
/* Minimum length of a match is two bytes. */
#define MATCH_LEN_MIN 2
/* Match length is encoded with 4, 5, or 10 bits.
* Length Bits
* 2-9 4 = Choice=0 + 3 bits
* 10-17 5 = Choice=1 + Choice2=0 + 3 bits
* 18-273 10 = Choice=1 + Choice2=1 + 8 bits
#define LEN_LOW_BITS 3
#define LEN_MID_BITS 3
#define LEN_HIGH_BITS 8
* Maximum length of a match is 273 which is a result of the encoding
* described above.
* Different sets of probabilities are used for match distances that have
* very short match length: Lengths of 2, 3, and 4 bytes have a separate
* set of probabilities for each length. The matches with longer length
* use a shared set of probabilities.
#define DIST_STATES 4
* Get the index of the appropriate probability array for decoding
* the distance slot.
static inline uint32_t lzma_get_dist_state(uint32_t len)
* The highest two bits of a 32-bit match distance are encoded using six bits.
* This six-bit value is called a distance slot. This way encoding a 32-bit
* value takes 6-36 bits, larger values taking more bits.
#define DIST_SLOT_BITS 6
/* Match distances up to 127 are fully encoded using probabilities. Since
* the highest two bits (distance slot) are always encoded using six bits,
* the distances 0-3 don't need any additional bits to encode, since the
* distance slot itself is the same as the actual distance. DIST_MODEL_START
* indicates the first distance slot where at least one additional bit is
* needed.
* Match distances greater than 127 are encoded in three pieces:
* - distance slot: the highest two bits
* - direct bits: 2-26 bits below the highest two bits
* - alignment bits: four lowest bits
* Direct bits don't use any probabilities.
* The distance slot value of 14 is for distances 128-191.
#define DIST_MODEL_END 14
/* Distance slots that indicate a distance <= 127. */
* For match distances greater than 127, only the highest two bits and the
* lowest four bits (alignment) is encoded using probabilities.
#define ALIGN_BITS 4
#define ALIGN_SIZE (1 << ALIGN_BITS)
/* Total number of all probability variables */
* LZMA remembers the four most recent match distances. Reusing these
* distances tends to take less space than re-encoding the actual
* distance value.
#define REPS 4
* Private includes and definitions
* Author: Lasse Collin <>
* This file has been put into the public domain.
* You can do whatever you want with this file.
#ifndef XZ_PRIVATE_H
#define XZ_PRIVATE_H
#ifdef __KERNEL__
# include <linux/xz.h>
# include <asm/byteorder.h>
# include <asm/unaligned.h>
/* XZ_PREBOOT may be defined only via decompress_unxz.c. */
# ifndef XZ_PREBOOT
# include <linux/slab.h>
# include <linux/vmalloc.h>
# include <linux/string.h>
# ifdef CONFIG_XZ_DEC_X86
# define XZ_DEC_X86
# endif