New feature: "Large Window Brotli" (#640)

* New feature: "Large Window Brotli"

By setting special encoder/decoder flag it is now possible to extend
LZ-window up to 30 bits; though produced stream will not be RFC7932
compliant.

Added new dictionary generator - "DSH". It combines speed of "Sieve"
and quality of "DM". Plus utilities to prepare train corpora
(remove unique strings).

Improved compression ratio: now two sub-blocks could be stitched:
the last copy command could be extended to span the next sub-block.

Fixed compression ineffectiveness caused by floating numbers rounding and
wrong cost heuristic.

Other C changes:
 - combined / moved `context.h` to `common`
 - moved transforms to `common`
 - unified some aspects of code formatting
 - added an abstraction for encoder (static) dictionary
 - moved default allocator/deallocator functions to `common`

brotli CLI:
 - window size is auto-adjusted if not specified explicitly

Java:
 - added "eager" decoding both to JNI wrapper and pure decoder
 - huge speed-up of `DictionaryData` initialization

* Add dictionaryless compressed dictionary

* Fix `sources.lst`

* Fix `sources.lst` and add a note that `libtool` is also required.

* Update setup.py

* Fix `EagerStreamTest`

* Fix BUILD file

* Add missing `libdivsufsort` dependency

* Fix "unused parameter" warning.

1 files changed, 12 insertions, 8 deletions

diff --git a/c/dec/huffman.h b/c/dec/huffman.h
index 730af88..521ec6e 100644
--- a/c/dec/huffman.h
+++ b/c/dec/huffman.h
@@ -19,10 +19,11 @@ extern "C" {
 #define BROTLI_HUFFMAN_MAX_CODE_LENGTH 15
 
 /* Maximum possible Huffman table size for an alphabet size of (index * 32),
- * max code length 15 and root table bits 8. */
+   max code length 15 and root table bits 8. */
 static const uint16_t kMaxHuffmanTableSize[] = {
   256, 402, 436, 468, 500, 534, 566, 598, 630, 662, 694, 726, 758, 790, 822,
-  854, 886, 920, 952, 984, 1016, 1048, 1080};
+  854, 886, 920, 952, 984, 1016, 1048, 1080, 1112, 1144, 1176, 1208, 1240, 1272,
+  1304, 1336, 1368, 1400, 1432, 1464, 1496, 1528};
 /* BROTLI_NUM_BLOCK_LEN_SYMBOLS == 26 */
 #define BROTLI_HUFFMAN_MAX_SIZE_26 396
 /* BROTLI_MAX_BLOCK_TYPE_SYMBOLS == 258 */
@@ -41,23 +42,26 @@ typedef struct {
 BROTLI_INTERNAL void BrotliBuildCodeLengthsHuffmanTable(HuffmanCode* root_table,
     const uint8_t* const code_lengths, uint16_t* count);
 
-/* Builds Huffman lookup table assuming code lengths are in symbol order. */
-/* Returns size of resulting table. */
+/* Builds Huffman lookup table assuming code lengths are in symbol order.
+   Returns size of resulting table. */
 BROTLI_INTERNAL uint32_t BrotliBuildHuffmanTable(HuffmanCode* root_table,
     int root_bits, const uint16_t* const symbol_lists, uint16_t* count_arg);
 
-/* Builds a simple Huffman table. The num_symbols parameter is to be */
-/* interpreted as follows: 0 means 1 symbol, 1 means 2 symbols, 2 means 3 */
-/* symbols, 3 means 4 symbols with lengths 2,2,2,2, 4 means 4 symbols with */
-/* lengths 1,2,3,3. */
+/* Builds a simple Huffman table. The |num_symbols| parameter is to be
+   interpreted as follows: 0 means 1 symbol, 1 means 2 symbols,
+   2 means 3 symbols, 3 means 4 symbols with lengths [2, 2, 2, 2],
+   4 means 4 symbols with lengths [1, 2, 3, 3]. */
 BROTLI_INTERNAL uint32_t BrotliBuildSimpleHuffmanTable(HuffmanCode* table,
     int root_bits, uint16_t* symbols, uint32_t num_symbols);
 
 /* Contains a collection of Huffman trees with the same alphabet size. */
+/* max_symbol is needed due to simple codes since log2(alphabet_size) could be
+   greater than log2(max_symbol). */
 typedef struct {
   HuffmanCode** htrees;
   HuffmanCode* codes;
   uint16_t alphabet_size;
+  uint16_t max_symbol;
   uint16_t num_htrees;
 } HuffmanTreeGroup;