org.eclipse.jgit.storage.dht.spi
Interface ChunkTable

All Known Implementing Classes:
CacheChunkTable

public interface ChunkTable

Stores object data in compressed pack format.

Each chunk stores multiple objects, using the highly compressed and Git native pack file format. Chunks are sized during insertion, but average around 1 MB for historical chunks, and may be as small as a few KB for very recent chunks that were written in small bursts.

Objects whose compressed form is too large to fit into a single chunk are fragmented across multiple chunks, and the fragment information is used to put them back together in the correct order. Since the fragmenting occurs after data compression, random access to bytes of the large object is not currently possible.

Chunk keys are very well distributed, by embedding a uniformly random number at the start of the key, and also including a small time component. This layout permits chunks to be evenly spread across a cluster of disks or servers in a round-robin fashion (based on a hash of the leading bytes), but also offers some chance for older chunks to be located near each other and have that part of the storage system see less activity over time.


Method Summary
 void get(Context options, java.util.Set<ChunkKey> keys, AsyncCallback<java.util.Collection<PackChunk.Members>> callback)
          Asynchronously load one or more chunks
 void getMeta(Context options, java.util.Set<ChunkKey> keys, AsyncCallback<java.util.Map<ChunkKey,GitStore.ChunkMeta>> callback)
          Asynchronously load one or more chunk meta fields.
 void put(PackChunk.Members chunk, WriteBuffer buffer)
          Put some (or all) of a single chunk.
 void remove(ChunkKey key, WriteBuffer buffer)
          Completely remove a chunk and all of its data elements.
 

Method Detail

get

void get(Context options,
         java.util.Set<ChunkKey> keys,
         AsyncCallback<java.util.Collection<PackChunk.Members>> callback)
Asynchronously load one or more chunks

Callers are responsible for breaking up very large collections of chunk keys into smaller units, based on the reader's batch size option. Since chunks typically 1 MB each, 10-20 keys is a reasonable batch size, but depends on available JVM memory and performance of this method obtaining chunks from the database.

Parameters:
options - options to control reading.
keys - the chunk keys to obtain.
callback - receives the results when ready. If this is an instance of StreamingCallback, implementors should try to deliver results early.

getMeta

void getMeta(Context options,
             java.util.Set<ChunkKey> keys,
             AsyncCallback<java.util.Map<ChunkKey,GitStore.ChunkMeta>> callback)
Asynchronously load one or more chunk meta fields.

Usually meta is loaded by get(Context, Set, AsyncCallback), but some uses may require looking up the fragment data without having the entire chunk.

Parameters:
options - options to control reading.
keys - the chunk keys to obtain.
callback - receives the results when ready. If this is an instance of StreamingCallback, implementors should try to deliver results early.

put

void put(PackChunk.Members chunk,
         WriteBuffer buffer)
         throws DhtException
Put some (or all) of a single chunk.

The higher level storage layer typically stores chunks in pieces. Its common to first store the data, then much later store the fragments and index. Sometimes all of the members are ready at once, and can be put together as a single unit. This method handles both approaches to storing a chunk.

Implementors must use a partial writing approach, for example:

   ColumnUpdateList list = ...;
   if (chunk.getChunkData() != null)
     list.addColumn("chunk_data", chunk.getChunkData());
   if (chunk.getChunkIndex() != null)
     list.addColumn("chunk_index", chunk.getChunkIndex());
   if (chunk.getFragments() != null)
     list.addColumn("fragments", chunk.getFragments());
   createOrUpdateRow(chunk.getChunkKey(), list);
 

Parameters:
chunk - description of the chunk to be stored.
buffer - buffer to enqueue the put onto.
Throws:
DhtException - if the buffer flushed and an enqueued operation failed.

remove

void remove(ChunkKey key,
            WriteBuffer buffer)
            throws DhtException
Completely remove a chunk and all of its data elements.

Chunk removal should occur as quickly as possible after the flush has completed, as the caller has already ensured the chunk is not in use.

Parameters:
key - key of the chunk to remove.
buffer - buffer to enqueue the remove onto.
Throws:
DhtException - if the buffer flushed and an enqueued operation failed.


Copyright © 2011. All Rights Reserved.