New feature: "Large Window Brotli" (#640)

* New feature: "Large Window Brotli"

By setting special encoder/decoder flag it is now possible to extend
LZ-window up to 30 bits; though produced stream will not be RFC7932
compliant.

Added new dictionary generator - "DSH". It combines speed of "Sieve"
and quality of "DM". Plus utilities to prepare train corpora
(remove unique strings).

Improved compression ratio: now two sub-blocks could be stitched:
the last copy command could be extended to span the next sub-block.

Fixed compression ineffectiveness caused by floating numbers rounding and
wrong cost heuristic.

Other C changes:
 - combined / moved `context.h` to `common`
 - moved transforms to `common`
 - unified some aspects of code formatting
 - added an abstraction for encoder (static) dictionary
 - moved default allocator/deallocator functions to `common`

brotli CLI:
 - window size is auto-adjusted if not specified explicitly

Java:
 - added "eager" decoding both to JNI wrapper and pure decoder
 - huge speed-up of `DictionaryData` initialization

* Add dictionaryless compressed dictionary

* Fix `sources.lst`

* Fix `sources.lst` and add a note that `libtool` is also required.

* Update setup.py

* Fix `EagerStreamTest`

* Fix BUILD file

* Add missing `libdivsufsort` dependency

* Fix "unused parameter" warning.

1 files changed, 7 insertions, 6 deletions

diff --git a/c/enc/block_encoder_inc.h b/c/enc/block_encoder_inc.h
index 2a08f90..8cbd5ea 100644
--- a/c/enc/block_encoder_inc.h
+++ b/c/enc/block_encoder_inc.h
@@ -13,9 +13,9 @@
    stream. */
 static void FN(BuildAndStoreEntropyCodes)(MemoryManager* m, BlockEncoder* self,
     const HistogramType* histograms, const size_t histograms_size,
-    HuffmanTree* tree, size_t* storage_ix, uint8_t* storage) {
-  const size_t alphabet_size = self->alphabet_size_;
-  const size_t table_size = histograms_size * alphabet_size;
+    const size_t alphabet_size, HuffmanTree* tree,
+    size_t* storage_ix, uint8_t* storage) {
+  const size_t table_size = histograms_size * self->histogram_length_;
   self->depths_ = BROTLI_ALLOC(m, uint8_t, table_size);
   self->bits_ = BROTLI_ALLOC(m, uint16_t, table_size);
   if (BROTLI_IS_OOM(m)) return;
@@ -23,9 +23,10 @@ static void FN(BuildAndStoreEntropyCodes)(MemoryManager* m, BlockEncoder* self,
   {
     size_t i;
     for (i = 0; i < histograms_size; ++i) {
-      size_t ix = i * alphabet_size;
-      BuildAndStoreHuffmanTree(&histograms[i].data_[0], alphabet_size, tree,
-          &self->depths_[ix], &self->bits_[ix], storage_ix, storage);
+      size_t ix = i * self->histogram_length_;
+      BuildAndStoreHuffmanTree(&histograms[i].data_[0], self->histogram_length_,
+          alphabet_size, tree, &self->depths_[ix], &self->bits_[ix],
+          storage_ix, storage);
     }
   }
 }