New feature: "Large Window Brotli" (#640)

* New feature: "Large Window Brotli"

By setting special encoder/decoder flag it is now possible to extend
LZ-window up to 30 bits; though produced stream will not be RFC7932
compliant.

Added new dictionary generator - "DSH". It combines speed of "Sieve"
and quality of "DM". Plus utilities to prepare train corpora
(remove unique strings).

Improved compression ratio: now two sub-blocks could be stitched:
the last copy command could be extended to span the next sub-block.

Fixed compression ineffectiveness caused by floating numbers rounding and
wrong cost heuristic.

Other C changes:
 - combined / moved `context.h` to `common`
 - moved transforms to `common`
 - unified some aspects of code formatting
 - added an abstraction for encoder (static) dictionary
 - moved default allocator/deallocator functions to `common`

brotli CLI:
 - window size is auto-adjusted if not specified explicitly

Java:
 - added "eager" decoding both to JNI wrapper and pure decoder
 - huge speed-up of `DictionaryData` initialization

* Add dictionaryless compressed dictionary

* Fix `sources.lst`

* Fix `sources.lst` and add a note that `libtool` is also required.

* Update setup.py

* Fix `EagerStreamTest`

* Fix BUILD file

* Add missing `libdivsufsort` dependency

* Fix "unused parameter" warning.

4 files changed, 14 insertions, 4 deletions

diff --git a/docs/brotli.1 b/docs/brotli.1
index c55b906..7242a32 100644
--- a/docs/brotli.1
+++ b/docs/brotli.1
@@ -1,4 +1,4 @@
-.TH "BROTLI" "1" "August 2017" "brotli 1.0.0" "User commands"
+.TH "BROTLI" "1" "February 2018" "brotli 1.0.0" "User commands"
 .SH "NAME"
 \fBbrotli\fR \- brotli, unbrotli \- compress or decompress files
 .SH SYNOPSIS
diff --git a/docs/decode.h.3 b/docs/decode.h.3
index 0948bf7..7b8581c 100644
--- a/docs/decode.h.3
+++ b/docs/decode.h.3
@@ -1,4 +1,4 @@
-.TH "decode.h" 3 "Fri Dec 8 2017" "Brotli" \" -*- nroff -*-
+.TH "decode.h" 3 "Thu Feb 22 2018" "Brotli" \" -*- nroff -*-
 .ad l
 .nh
 .SH NAME
@@ -143,6 +143,9 @@ Options to be used with \fBBrotliDecoderSetParameter\fP\&.
 .TP
 \fB\fIBROTLI_DECODER_PARAM_DISABLE_RING_BUFFER_REALLOCATION \fP\fP
 Disable 'canny' ring buffer allocation strategy\&. Ring buffer is allocated according to window size, despite the real size of the content\&. 
+.TP
+\fB\fIBROTLI_DECODER_PARAM_LARGE_WINDOW \fP\fP
+Flag that determines if 'Large Window Brotli' is used\&. 
 .SS "enum \fBBrotliDecoderResult\fP"
 
 .PP
diff --git a/docs/encode.h.3 b/docs/encode.h.3
index 1e1193e..906ce07 100644
--- a/docs/encode.h.3
+++ b/docs/encode.h.3
@@ -1,4 +1,4 @@
-.TH "encode.h" 3 "Fri Dec 8 2017" "Brotli" \" -*- nroff -*-
+.TH "encode.h" 3 "Thu Feb 22 2018" "Brotli" \" -*- nroff -*-
 .ad l
 .nh
 .SH NAME
@@ -23,6 +23,10 @@ encode.h \- API for Brotli compression\&.
 .br
 .RI "\fIDefault value for \fBBROTLI_PARAM_LGWIN\fP parameter\&. \fP"
 .ti -1c
+.RI "#define \fBBROTLI_LARGE_MAX_WINDOW_BITS\fP   30"
+.br
+.RI "\fIMaximal value for \fBBROTLI_PARAM_LGWIN\fP parameter in 'Large Window Brotli' (32-bit)\&. \fP"
+.ti -1c
 .RI "#define \fBBROTLI_MAX_INPUT_BLOCK_BITS\fP   24"
 .br
 .RI "\fIMaximal value for \fBBROTLI_PARAM_LGBLOCK\fP parameter\&. \fP"
@@ -287,6 +291,9 @@ Flag that affects usage of 'literal context modeling' format feature\&. This fla
 .TP
 \fB\fIBROTLI_PARAM_SIZE_HINT \fP\fP
 Estimated total input size for all \fBBrotliEncoderCompressStream\fP calls\&. The default value is 0, which means that the total input size is unknown\&. 
+.TP
+\fB\fIBROTLI_PARAM_LARGE_WINDOW \fP\fP
+Flag that determines if 'Large Window Brotli' is used\&. 
 .SH "Function Documentation"
 .PP 
 .SS "\fBBROTLI_BOOL\fP BrotliEncoderCompress (int quality, int lgwin, \fBBrotliEncoderMode\fP mode, size_t input_size, const uint8_t input_buffer[input_size], size_t * encoded_size, uint8_t encoded_buffer[*encoded_size])"
diff --git a/docs/types.h.3 b/docs/types.h.3
index e72ae6e..bef9313 100644
--- a/docs/types.h.3
+++ b/docs/types.h.3
@@ -1,4 +1,4 @@
-.TH "types.h" 3 "Wed Aug 2 2017" "Brotli" \" -*- nroff -*-
+.TH "types.h" 3 "Thu Feb 22 2018" "Brotli" \" -*- nroff -*-
 .ad l
 .nh
 .SH NAME