New feature: "Large Window Brotli" (#640)

* New feature: "Large Window Brotli"

By setting special encoder/decoder flag it is now possible to extend
LZ-window up to 30 bits; though produced stream will not be RFC7932
compliant.

Added new dictionary generator - "DSH". It combines speed of "Sieve"
and quality of "DM". Plus utilities to prepare train corpora
(remove unique strings).

Improved compression ratio: now two sub-blocks could be stitched:
the last copy command could be extended to span the next sub-block.

Fixed compression ineffectiveness caused by floating numbers rounding and
wrong cost heuristic.

Other C changes:
 - combined / moved `context.h` to `common`
 - moved transforms to `common`
 - unified some aspects of code formatting
 - added an abstraction for encoder (static) dictionary
 - moved default allocator/deallocator functions to `common`

brotli CLI:
 - window size is auto-adjusted if not specified explicitly

Java:
 - added "eager" decoding both to JNI wrapper and pure decoder
 - huge speed-up of `DictionaryData` initialization

* Add dictionaryless compressed dictionary

* Fix `sources.lst`

* Fix `sources.lst` and add a note that `libtool` is also required.

* Update setup.py

* Fix `EagerStreamTest`

* Fix BUILD file

* Add missing `libdivsufsort` dependency

* Fix "unused parameter" warning.

1 files changed, 9 insertions, 8 deletions

diff --git a/scripts/sources.lst b/scripts/sources.lst
index cd61a7f..cdddb37 100644
--- a/scripts/sources.lst
+++ b/scripts/sources.lst
@@ -5,11 +5,15 @@ BROTLI_CLI_C = \
   c/tools/brotli.c
 
 BROTLI_COMMON_C = \
-  c/common/dictionary.c
+  c/common/dictionary.c \
+  c/common/transform.c
 
 BROTLI_COMMON_H = \
   c/common/constants.h \
+  c/common/context.h \
   c/common/dictionary.h \
+  c/common/platform.h \
+  c/common/transform.h \
   c/common/version.h
 
 BROTLI_DEC_C = \
@@ -20,12 +24,9 @@ BROTLI_DEC_C = \
 
 BROTLI_DEC_H = \
   c/dec/bit_reader.h \
-  c/dec/context.h \
   c/dec/huffman.h \
-  c/dec/port.h \
   c/dec/prefix.h \
-  c/dec/state.h \
-  c/dec/transform.h
+  c/dec/state.h
 
 BROTLI_ENC_C = \
   c/enc/backward_references.c \
@@ -38,6 +39,7 @@ BROTLI_ENC_C = \
   c/enc/compress_fragment_two_pass.c \
   c/enc/dictionary_hash.c \
   c/enc/encode.c \
+  c/enc/encoder_dict.c \
   c/enc/entropy_encode.c \
   c/enc/histogram.c \
   c/enc/literal_cost.c \
@@ -61,14 +63,14 @@ BROTLI_ENC_H = \
   c/enc/command.h \
   c/enc/compress_fragment.h \
   c/enc/compress_fragment_two_pass.h \
-  c/enc/context.h \
   c/enc/dictionary_hash.h \
+  c/enc/encoder_dict.h \
   c/enc/entropy_encode.h \
   c/enc/entropy_encode_static.h \
   c/enc/fast_log.h \
   c/enc/find_match_length.h \
-  c/enc/hash_forgetful_chain_inc.h \
   c/enc/hash.h \
+  c/enc/hash_forgetful_chain_inc.h \
   c/enc/hash_longest_match64_inc.h \
   c/enc/hash_longest_match_inc.h \
   c/enc/hash_longest_match_quickly_inc.h \
@@ -79,7 +81,6 @@ BROTLI_ENC_H = \
   c/enc/memory.h \
   c/enc/metablock.h \
   c/enc/metablock_inc.h \
-  c/enc/port.h \
   c/enc/prefix.h \
   c/enc/quality.h \
   c/enc/ringbuffer.h \