Module org.apache.lucene.core
Package org.apache.lucene.index

Class DocumentsWriterFlushControl

java.lang.Object
org.apache.lucene.index.DocumentsWriterFlushControl
All Implemented Interfaces:
Closeable, AutoCloseable, Accountable
final class DocumentsWriterFlushControl extends Object implements Accountable, Closeable
This class controls DocumentsWriterPerThread flushing during indexing. It tracks the memory consumption per DocumentsWriterPerThread and uses a configured FlushPolicy to decide if a DocumentsWriterPerThread must flush.

In addition to the FlushPolicy the flush control might set certain DocumentsWriterPerThread as flush pending iff a DocumentsWriterPerThread exceeds the IndexWriterConfig.getRAMPerThreadHardLimitMB() to prevent address space exhaustion.

  • Field Details

    • hardMaxBytesPerDWPT

      private final long hardMaxBytesPerDWPT

    • activeBytes

      private long activeBytes

    • flushBytes

      private volatile long flushBytes

    • numPending

      private volatile int numPending

    • numDocsSinceStalled

      private int numDocsSinceStalled

    • flushDeletes

      private final AtomicBoolean flushDeletes

    • fullFlush

      private boolean fullFlush

    • fullFlushMarkDone

      private boolean fullFlushMarkDone

    • flushQueue

      private final Queue<DocumentsWriterPerThread> flushQueue

    • blockedFlushes

      private final Queue<DocumentsWriterPerThread> blockedFlushes

    • flushingWriters

      private final List<DocumentsWriterPerThread> flushingWriters

    • maxConfiguredRamBuffer

      private double maxConfiguredRamBuffer

    • peakActiveBytes

      private long peakActiveBytes

    • peakFlushBytes

      private long peakFlushBytes

    • peakNetBytes

      private long peakNetBytes

    • peakDelta

      private long peakDelta

    • flushByRAMWasDisabled

      private boolean flushByRAMWasDisabled

    • stallControl

      final DocumentsWriterStallControl stallControl

    • perThreadPool

      private final DocumentsWriterPerThreadPool perThreadPool

    • flushPolicy

      private final FlushPolicy flushPolicy

    • closed

      private boolean closed

    • documentsWriter

      private final DocumentsWriter documentsWriter

    • config

      private final LiveIndexWriterConfig config

    • infoStream

      private final InfoStream infoStream

    • stallStartNS

      private long stallStartNS

  • Constructor Details

  • Method Details

    • activeBytes

      public long activeBytes()

    • getFlushingBytes

      long getFlushingBytes()

    • netBytes

      long netBytes()

    • stallLimitBytes

      private long stallLimitBytes()

    • assertMemory

      private boolean assertMemory()

    • updatePeaks

      private boolean updatePeaks(long delta)

    • ramBufferGranularity

      private long ramBufferGranularity()
      Return the smallest number of bytes that we would like to make sure to not miss from the global RAM accounting.

    • doAfterDocument

      DocumentsWriterPerThread doAfterDocument(DocumentsWriterPerThread perThread)

    • checkout

      private DocumentsWriterPerThread checkout(DocumentsWriterPerThread perThread, boolean markPending)

    • assertNumDocsSinceStalled

      private boolean assertNumDocsSinceStalled(boolean stalled)

    • doAfterFlush

      void doAfterFlush(DocumentsWriterPerThread dwpt)

    • updateStallState

      private boolean updateStallState()

    • waitForFlush

      public void waitForFlush()

    • setFlushPending

      public void setFlushPending(DocumentsWriterPerThread perThread)
      Sets flush pending state on the given DocumentsWriterPerThread. The DocumentsWriterPerThread must have indexed at least on Document and must not be already pending.

    • doOnAbort

      void doOnAbort(DocumentsWriterPerThread perThread)

    • checkoutAndBlock

      private void checkoutAndBlock(DocumentsWriterPerThread perThread)
      To be called only by the owner of this object's monitor lock

    • checkOutForFlush

      private DocumentsWriterPerThread checkOutForFlush(DocumentsWriterPerThread perThread)

    • addFlushingDWPT

      private void addFlushingDWPT(DocumentsWriterPerThread perThread)

    • toString

      public String toString()
      Overrides:
      toString in class Object

    • nextPendingFlush

      DocumentsWriterPerThread nextPendingFlush()

    • close

      public void close()
      Specified by:
      close in interface AutoCloseable
      Specified by:
      close in interface Closeable

    • allActiveWriters

      public Iterator<DocumentsWriterPerThread> allActiveWriters()
      Returns an iterator that provides access to all currently active DocumentsWriterPerThreads

    • doOnDelete

      void doOnDelete()

    • getDeleteBytesUsed

      public long getDeleteBytesUsed()
      Returns heap bytes currently consumed by buffered deletes/updates that would be freed if we pushed all deletes. This does not include bytes consumed by already pushed delete/update packets.

    • ramBytesUsed

      public long ramBytesUsed()
      Description copied from interface: Accountable
      Return the memory usage of this object in bytes. Negative values are illegal.
      Specified by:
      ramBytesUsed in interface Accountable

    • numFlushingDWPT

      int numFlushingDWPT()

    • getAndResetApplyAllDeletes

      public boolean getAndResetApplyAllDeletes()

    • getApplyAllDeletes

      public boolean getApplyAllDeletes()
      Check whether deletes need to be applied. This can be used as a pre-flight check before calling getAndResetApplyAllDeletes() to make sure that a single thread applies deletes.

    • setApplyAllDeletes

      public void setApplyAllDeletes()

    • obtainAndLock

      DocumentsWriterPerThread obtainAndLock()

    • markForFullFlush

      long markForFullFlush()

    • assertActiveDeleteQueue

      private boolean assertActiveDeleteQueue(DocumentsWriterDeleteQueue queue)

    • pruneBlockedQueue

      private void pruneBlockedQueue(DocumentsWriterDeleteQueue flushingQueue)
      Prunes the blockedQueue by removing all DWPTs that are associated with the given flush queue.

    • finishFullFlush

      void finishFullFlush()

    • assertBlockedFlushes

      boolean assertBlockedFlushes(DocumentsWriterDeleteQueue flushingQueue)

    • abortFullFlushes

      void abortFullFlushes()

    • abortPendingFlushes

      void abortPendingFlushes()

    • isFullFlush

      boolean isFullFlush()
      Returns true if a full flush is currently running

    • numQueuedFlushes

      int numQueuedFlushes()
      Returns the number of flushes that are already checked out but not yet actively flushing

    • numBlockedFlushes

      int numBlockedFlushes()
      Returns the number of flushes that are checked out but not yet available for flushing. This only applies during a full flush if a DWPT needs flushing but must not be flushed until the full flush has finished.

    • waitIfStalled

      void waitIfStalled()
      This method will block if too many DWPT are currently flushing and no checked out DWPT are available

    • anyStalledThreads

      boolean anyStalledThreads()
      Returns true iff stalled

    • getInfoStream

      public InfoStream getInfoStream()
      Returns the IndexWriter InfoStream

    • findLargestNonPendingWriter

      DocumentsWriterPerThread findLargestNonPendingWriter()

    • checkoutLargestNonPendingWriter

      final DocumentsWriterPerThread checkoutLargestNonPendingWriter()
      Returns the largest non-pending flushable DWPT or null if there is none.

    • getPeakActiveBytes

      long getPeakActiveBytes()

    • getPeakNetBytes

      long getPeakNetBytes()