aboutsummaryrefslogtreecommitdiff
path: root/winsup/bz2lib/bzip2.txt
diff options
context:
space:
mode:
authorChristopher Faylor <me@cgf.cx>2001-04-18 20:00:34 +0000
committerChristopher Faylor <me@cgf.cx>2001-04-18 20:00:34 +0000
commita2d47fe9d983ec85075172211172af9e5ab4054f (patch)
treea7938fb0b0afe632f3ec65163742b7377cdfcd06 /winsup/bz2lib/bzip2.txt
parent77c45b121e815ab2a107f1e3e6f6b737da770d2a (diff)
downloadnewlib-a2d47fe9d983ec85075172211172af9e5ab4054f.zip
newlib-a2d47fe9d983ec85075172211172af9e5ab4054f.tar.gz
newlib-a2d47fe9d983ec85075172211172af9e5ab4054f.tar.bz2
*** empty log message ***
Diffstat (limited to 'winsup/bz2lib/bzip2.txt')
-rw-r--r--winsup/bz2lib/bzip2.txt376
1 files changed, 376 insertions, 0 deletions
diff --git a/winsup/bz2lib/bzip2.txt b/winsup/bz2lib/bzip2.txt
new file mode 100644
index 0000000..4f1ae86
--- /dev/null
+++ b/winsup/bz2lib/bzip2.txt
@@ -0,0 +1,376 @@
+
+
+NAME
+ bzip2, bunzip2 - a block-sorting file compressor, v1.0
+ bzcat - decompresses files to stdout
+ bzip2recover - recovers data from damaged bzip2 files
+
+
+SYNOPSIS
+ bzip2 [ -cdfkqstvzVL123456789 ] [ filenames ... ]
+ bunzip2 [ -fkvsVL ] [ filenames ... ]
+ bzcat [ -s ] [ filenames ... ]
+ bzip2recover filename
+
+
+DESCRIPTION
+ bzip2 compresses files using the Burrows-Wheeler block
+ sorting text compression algorithm, and Huffman coding.
+ Compression is generally considerably better than that
+ achieved by more conventional LZ77/LZ78-based compressors,
+ and approaches the performance of the PPM family of sta-
+ tistical compressors.
+
+ The command-line options are deliberately very similar to
+ those of GNU gzip, but they are not identical.
+
+ bzip2 expects a list of file names to accompany the com-
+ mand-line flags. Each file is replaced by a compressed
+ version of itself, with the name "original_name.bz2".
+ Each compressed file has the same modification date, per-
+ missions, and, when possible, ownership as the correspond-
+ ing original, so that these properties can be correctly
+ restored at decompression time. File name handling is
+ naive in the sense that there is no mechanism for preserv-
+ ing original file names, permissions, ownerships or dates
+ in filesystems which lack these concepts, or have serious
+ file name length restrictions, such as MS-DOS.
+
+ bzip2 and bunzip2 will by default not overwrite existing
+ files. If you want this to happen, specify the -f flag.
+
+ If no file names are specified, bzip2 compresses from
+ standard input to standard output. In this case, bzip2
+ will decline to write compressed output to a terminal, as
+ this would be entirely incomprehensible and therefore
+ pointless.
+
+ bunzip2 (or bzip2 -d) decompresses all specified files.
+ Files which were not created by bzip2 will be detected and
+ ignored, and a warning issued. bzip2 attempts to guess
+ the filename for the decompressed file from that of the
+ compressed file as follows:
+
+ filename.bz2 becomes filename
+ filename.bz becomes filename
+ filename.tbz2 becomes filename.tar
+ filename.tbz becomes filename.tar
+ anyothername becomes anyothername.out
+
+ If the file does not end in one of the recognised endings,
+ .bz2, .bz, .tbz2 or .tbz, bzip2 complains that it cannot
+ guess the name of the original file, and uses the original
+ name with .out appended.
+
+ As with compression, supplying no filenames causes decom-
+ pression from standard input to standard output.
+
+ bunzip2 will correctly decompress a file which is the con-
+ catenation of two or more compressed files. The result is
+ the concatenation of the corresponding uncompressed files.
+ Integrity testing (-t) of concatenated compressed files is
+ also supported.
+
+ You can also compress or decompress files to the standard
+ output by giving the -c flag. Multiple files may be com-
+ pressed and decompressed like this. The resulting outputs
+ are fed sequentially to stdout. Compression of multiple
+ files in this manner generates a stream containing multi-
+ ple compressed file representations. Such a stream can be
+ decompressed correctly only by bzip2 version 0.9.0 or
+ later. Earlier versions of bzip2 will stop after decom-
+ pressing the first file in the stream.
+
+ bzcat (or bzip2 -dc) decompresses all specified files to
+ the standard output.
+
+ bzip2 will read arguments from the environment variables
+ BZIP2 and BZIP, in that order, and will process them
+ before any arguments read from the command line. This
+ gives a convenient way to supply default arguments.
+
+ Compression is always performed, even if the compressed
+ file is slightly larger than the original. Files of less
+ than about one hundred bytes tend to get larger, since the
+ compression mechanism has a constant overhead in the
+ region of 50 bytes. Random data (including the output of
+ most file compressors) is coded at about 8.05 bits per
+ byte, giving an expansion of around 0.5%.
+
+ As a self-check for your protection, bzip2 uses 32-bit
+ CRCs to make sure that the decompressed version of a file
+ is identical to the original. This guards against corrup-
+ tion of the compressed data, and against undetected bugs
+ in bzip2 (hopefully very unlikely). The chances of data
+ corruption going undetected is microscopic, about one
+ chance in four billion for each file processed. Be aware,
+ though, that the check occurs upon decompression, so it
+ can only tell you that something is wrong. It can't help
+ you recover the original uncompressed data. You can use
+ bzip2recover to try to recover data from damaged files.
+
+ Return values: 0 for a normal exit, 1 for environmental
+ problems (file not found, invalid flags, I/O errors, &c),
+ 2 to indicate a corrupt compressed file, 3 for an internal
+ consistency error (eg, bug) which caused bzip2 to panic.
+
+
+OPTIONS
+ -c --stdout
+ Compress or decompress to standard output.
+
+ -d --decompress
+ Force decompression. bzip2, bunzip2 and bzcat are
+ really the same program, and the decision about
+ what actions to take is done on the basis of which
+ name is used. This flag overrides that mechanism,
+ and forces bzip2 to decompress.
+
+ -z --compress
+ The complement to -d: forces compression, regard-
+ less of the invokation name.
+
+ -t --test
+ Check integrity of the specified file(s), but don't
+ decompress them. This really performs a trial
+ decompression and throws away the result.
+
+ -f --force
+ Force overwrite of output files. Normally, bzip2
+ will not overwrite existing output files. Also
+ forces bzip2 to break hard links to files, which it
+ otherwise wouldn't do.
+
+ -k --keep
+ Keep (don't delete) input files during compression
+ or decompression.
+
+ -s --small
+ Reduce memory usage, for compression, decompression
+ and testing. Files are decompressed and tested
+ using a modified algorithm which only requires 2.5
+ bytes per block byte. This means any file can be
+ decompressed in 2300k of memory, albeit at about
+ half the normal speed.
+
+ During compression, -s selects a block size of
+ 200k, which limits memory use to around the same
+ figure, at the expense of your compression ratio.
+ In short, if your machine is low on memory (8
+ megabytes or less), use -s for everything. See
+ MEMORY MANAGEMENT below.
+
+ -q --quiet
+ Suppress non-essential warning messages. Messages
+ pertaining to I/O errors and other critical events
+ will not be suppressed.
+
+ -v --verbose
+ Verbose mode -- show the compression ratio for each
+ file processed. Further -v's increase the ver-
+ bosity level, spewing out lots of information which
+ is primarily of interest for diagnostic purposes.
+
+ -L --license -V --version
+ Display the software version, license terms and
+ conditions.
+
+ -1 to -9
+ Set the block size to 100 k, 200 k .. 900 k when
+ compressing. Has no effect when decompressing.
+ See MEMORY MANAGEMENT below.
+
+ -- Treats all subsequent arguments as file names, even
+ if they start with a dash. This is so you can han-
+ dle files with names beginning with a dash, for
+ example: bzip2 -- -myfilename.
+
+ --repetitive-fast --repetitive-best
+ These flags are redundant in versions 0.9.5 and
+ above. They provided some coarse control over the
+ behaviour of the sorting algorithm in earlier ver-
+ sions, which was sometimes useful. 0.9.5 and above
+ have an improved algorithm which renders these
+ flags irrelevant.
+
+
+MEMORY MANAGEMENT
+ bzip2 compresses large files in blocks. The block size
+ affects both the compression ratio achieved, and the
+ amount of memory needed for compression and decompression.
+ The flags -1 through -9 specify the block size to be
+ 100,000 bytes through 900,000 bytes (the default) respec-
+ tively. At decompression time, the block size used for
+ compression is read from the header of the compressed
+ file, and bunzip2 then allocates itself just enough memory
+ to decompress the file. Since block sizes are stored in
+ compressed files, it follows that the flags -1 to -9 are
+ irrelevant to and so ignored during decompression.
+
+ Compression and decompression requirements, in bytes, can
+ be estimated as:
+
+ Compression: 400k + ( 8 x block size )
+
+ Decompression: 100k + ( 4 x block size ), or
+ 100k + ( 2.5 x block size )
+
+ Larger block sizes give rapidly diminishing marginal
+ returns. Most of the compression comes from the first two
+ or three hundred k of block size, a fact worth bearing in
+ mind when using bzip2 on small machines. It is also
+ important to appreciate that the decompression memory
+ requirement is set at compression time by the choice of
+ block size.
+
+ For files compressed with the default 900k block size,
+ bunzip2 will require about 3700 kbytes to decompress. To
+ support decompression of any file on a 4 megabyte machine,
+ bunzip2 has an option to decompress using approximately
+ half this amount of memory, about 2300 kbytes. Decompres-
+ sion speed is also halved, so you should use this option
+ only where necessary. The relevant flag is -s.
+
+ In general, try and use the largest block size memory con-
+ straints allow, since that maximises the compression
+ achieved. Compression and decompression speed are virtu-
+ ally unaffected by block size.
+
+ Another significant point applies to files which fit in a
+ single block -- that means most files you'd encounter
+ using a large block size. The amount of real memory
+ touched is proportional to the size of the file, since the
+ file is smaller than a block. For example, compressing a
+ file 20,000 bytes long with the flag -9 will cause the
+ compressor to allocate around 7600k of memory, but only
+ touch 400k + 20000 * 8 = 560 kbytes of it. Similarly, the
+ decompressor will allocate 3700k but only touch 100k +
+ 20000 * 4 = 180 kbytes.
+
+ Here is a table which summarises the maximum memory usage
+ for different block sizes. Also recorded is the total
+ compressed size for 14 files of the Calgary Text Compres-
+ sion Corpus totalling 3,141,622 bytes. This column gives
+ some feel for how compression varies with block size.
+ These figures tend to understate the advantage of larger
+ block sizes for larger files, since the Corpus is domi-
+ nated by smaller files.
+
+ Compress Decompress Decompress Corpus
+ Flag usage usage -s usage Size
+
+ -1 1200k 500k 350k 914704
+ -2 2000k 900k 600k 877703
+ -3 2800k 1300k 850k 860338
+ -4 3600k 1700k 1100k 846899
+ -5 4400k 2100k 1350k 845160
+ -6 5200k 2500k 1600k 838626
+ -7 6100k 2900k 1850k 834096
+ -8 6800k 3300k 2100k 828642
+ -9 7600k 3700k 2350k 828642
+
+
+RECOVERING DATA FROM DAMAGED FILES
+ bzip2 compresses files in blocks, usually 900kbytes long.
+ Each block is handled independently. If a media or trans-
+ mission error causes a multi-block .bz2 file to become
+ damaged, it may be possible to recover data from the
+ undamaged blocks in the file.
+
+ The compressed representation of each block is delimited
+ by a 48-bit pattern, which makes it possible to find the
+ block boundaries with reasonable certainty. Each block
+ also carries its own 32-bit CRC, so damaged blocks can be
+ distinguished from undamaged ones.
+
+ bzip2recover is a simple program whose purpose is to
+ search for blocks in .bz2 files, and write each block out
+ into its own .bz2 file. You can then use bzip2 -t to test
+ the integrity of the resulting files, and decompress those
+ which are undamaged.
+
+ bzip2recover takes a single argument, the name of the dam-
+ aged file, and writes a number of files "rec0001file.bz2",
+ "rec0002file.bz2", etc, containing the extracted blocks.
+ The output filenames are designed so that the use of
+ wildcards in subsequent processing -- for example, "bzip2
+ -dc rec*file.bz2 > recovered_data" -- lists the files in
+ the correct order.
+
+ bzip2recover should be of most use dealing with large .bz2
+ files, as these will contain many blocks. It is clearly
+ futile to use it on damaged single-block files, since a
+ damaged block cannot be recovered. If you wish to min-
+ imise any potential data loss through media or transmis-
+ sion errors, you might consider compressing with a smaller
+ block size.
+
+
+PERFORMANCE NOTES
+ The sorting phase of compression gathers together similar
+ strings in the file. Because of this, files containing
+ very long runs of repeated symbols, like "aabaabaabaab
+ ..." (repeated several hundred times) may compress more
+ slowly than normal. Versions 0.9.5 and above fare much
+ better than previous versions in this respect. The ratio
+ between worst-case and average-case compression time is in
+ the region of 10:1. For previous versions, this figure
+ was more like 100:1. You can use the -vvvv option to mon-
+ itor progress in great detail, if you want.
+
+ Decompression speed is unaffected by these phenomena.
+
+ bzip2 usually allocates several megabytes of memory to
+ operate in, and then charges all over it in a fairly ran-
+ dom fashion. This means that performance, both for com-
+ pressing and decompressing, is largely determined by the
+ speed at which your machine can service cache misses.
+ Because of this, small changes to the code to reduce the
+ miss rate have been observed to give disproportionately
+ large performance improvements. I imagine bzip2 will per-
+ form best on machines with very large caches.
+
+
+CAVEATS
+ I/O error messages are not as helpful as they could be.
+ bzip2 tries hard to detect I/O errors and exit cleanly,
+ but the details of what the problem is sometimes seem
+ rather misleading.
+
+ This manual page pertains to version 1.0 of bzip2. Com-
+ pressed data created by this version is entirely forwards
+ and backwards compatible with the previous public
+ releases, versions 0.1pl2, 0.9.0 and 0.9.5, but with the
+ following exception: 0.9.0 and above can correctly decom-
+ press multiple concatenated compressed files. 0.1pl2 can-
+ not do this; it will stop after decompressing just the
+ first file in the stream.
+
+ bzip2recover uses 32-bit integers to represent bit posi-
+ tions in compressed files, so it cannot handle compressed
+ files more than 512 megabytes long. This could easily be
+ fixed.
+
+
+AUTHOR
+ Julian Seward, jseward@acm.org.
+
+ http://sourceware.cygnus.com/bzip2
+ http://www.muraroa.demon.co.uk
+
+ The ideas embodied in bzip2 are due to (at least) the fol-
+ lowing people: Michael Burrows and David Wheeler (for the
+ block sorting transformation), David Wheeler (again, for
+ the Huffman coder), Peter Fenwick (for the structured cod-
+ ing model in the original bzip, and many refinements), and
+ Alistair Moffat, Radford Neal and Ian Witten (for the
+ arithmetic coder in the original bzip). I am much
+ indebted for their help, support and advice. See the man-
+ ual in the source distribution for pointers to sources of
+ documentation. Christian von Roques encouraged me to look
+ for faster sorting algorithms, so as to speed up compres-
+ sion. Bela Lubkin encouraged me to improve the worst-case
+ compression performance. Many people sent patches, helped
+ with portability problems, lent machines, gave advice and
+ were generally helpful.
+