Package org.elasticsearch.indices.recovery

Class MultiChunkTransfer<Source,​Request extends MultiChunkTransfer.ChunkRequest>

java.lang.Object
org.elasticsearch.indices.recovery.MultiChunkTransfer<Source,​Request>
All Implemented Interfaces:
Closeable, AutoCloseable
public abstract class MultiChunkTransfer<Source,​Request extends MultiChunkTransfer.ChunkRequest> extends Object implements Closeable
File chunks are sent/requested sequentially by at most one thread at any time. However, the sender/requestor won't wait for the response before processing the next file chunk request to reduce the recovery time especially on secure/compressed or high latency communication.

The sender/requestor can send up to maxConcurrentFileChunks file chunk requests without waiting for responses. Since the recovery target can receive file chunks out of order, it has to buffer those file chunks in memory and only flush to disk when there's no gap. To ensure the recover target never buffers more than maxConcurrentFileChunks file chunks, we allow the sender/requestor to send only up to maxConcurrentFileChunks file chunk requests from the last flushed (and acknowledged) file chunk. We leverage the local checkpoint tracker for this purpose. We generate a new sequence number and assign it to each file chunk request before sending; then mark that sequence number as processed when we receive a response for the corresponding file chunk request. With the local checkpoint tracker, we know the last acknowledged-flushed file-chunk is a file chunk whose requestSeqId equals to the local checkpoint because the recover target can flush all file chunks up to the local checkpoint.

When the number of un-replied file chunk requests reaches the limit (i.e. the gap between the max_seq_no and the local checkpoint is greater than maxConcurrentFileChunks), the sending/requesting thread will abort its execution. That process will be resumed by one of the networking threads which receive/handle the responses of the current pending file chunk requests. This process will continue until all chunk requests are sent/responded.

  • Constructor Details

    • MultiChunkTransfer

      protected MultiChunkTransfer​(org.apache.logging.log4j.Logger logger, ThreadContext threadContext, ActionListener<Void> listener, int maxConcurrentChunks, List<Source> sources)

  • Method Details

    • start

      public final void start()

    • assertOnSuccess

      protected boolean assertOnSuccess()

    • onNewResource

      protected void onNewResource​(Source resource) throws IOException
      This method is called when starting sending/requesting a new source. Subclasses should override this method to reset the file offset or close the previous file and open a new file if needed.
      Throws:
      IOException

    • nextChunkRequest

      protected abstract Request nextChunkRequest​(Source resource) throws IOException
      Throws:
      IOException

    • executeChunkRequest

      protected abstract void executeChunkRequest​(Request request, ActionListener<Void> listener)

    • handleError

      protected abstract void handleError​(Source resource, Exception e) throws Exception
      Throws:
      Exception