Module org.apache.lucene.core
Package org.apache.lucene.index

Class DocumentsWriterDeleteQueue

java.lang.Object
org.apache.lucene.index.DocumentsWriterDeleteQueue
All Implemented Interfaces:
Closeable, AutoCloseable, Accountable
final class DocumentsWriterDeleteQueue extends Object implements Accountable, Closeable
DocumentsWriterDeleteQueue is a non-blocking linked pending deletes queue. In contrast to other queue implementation we only maintain the tail of the queue. A delete queue is always used in a context of a set of DWPTs and a global delete pool. Each of the DWPT and the global pool need to maintain their 'own' head of the queue (as a DeleteSlice instance per DocumentsWriterPerThread). The difference between the DWPT and the global pool is that the DWPT starts maintaining a head once it has added its first document since for its segments private deletes only the deletes after that document are relevant. The global pool instead starts maintaining the head once this instance is created by taking the sentinel instance as its initial head.

Since each DocumentsWriterDeleteQueue.DeleteSlice maintains its own head and the list is only single linked the garbage collector takes care of pruning the list for us. All nodes in the list that are still relevant should be either directly or indirectly referenced by one of the DWPT's private DocumentsWriterDeleteQueue.DeleteSlice or by the global BufferedUpdates slice.

Each DWPT as well as the global delete pool maintain their private DeleteSlice instance. In the DWPT case updating a slice is equivalent to atomically finishing the document. The slice update guarantees a "happens before" relationship to all other updates in the same indexing session. When a DWPT updates a document it:

  1. consumes a document and finishes its processing
  2. updates its private DocumentsWriterDeleteQueue.DeleteSlice either by calling updateSlice(DeleteSlice) or add(Node, DeleteSlice) (if the document has a delTerm)
  3. applies all deletes in the slice to its private BufferedUpdates and resets it
  4. increments its internal document id
The DWPT also doesn't apply its current documents delete term until it has updated its delete slice which ensures the consistency of the update. If the update fails before the DeleteSlice could have been updated the deleteTerm will also not be added to its private deletes neither to the global deletes.

  • Field Details

    • tail

      private volatile DocumentsWriterDeleteQueue.Node<?> tail

    • closed

      private volatile boolean closed

    • globalSlice

      private final DocumentsWriterDeleteQueue.DeleteSlice globalSlice
      Used to record deletes against all prior (already written to disk) segments. Whenever any segment flushes, we bundle up this set of deletes and insert into the buffered updates stream before the newly flushed segment(s).

    • globalBufferedUpdates

      private final BufferedUpdates globalBufferedUpdates

    • globalBufferLock

      final ReentrantLock globalBufferLock

    • generation

      final long generation

    • nextSeqNo

      private final AtomicLong nextSeqNo
      Generates the sequence number that IW returns to callers changing the index, showing the effective serialization of all operations.

    • infoStream

      private final InfoStream infoStream

    • maxSeqNo

      private volatile long maxSeqNo

    • startSeqNo

      private final long startSeqNo

    • previousMaxSeqId

      private final LongSupplier previousMaxSeqId

    • advanced

      private boolean advanced

  • Constructor Details

    • DocumentsWriterDeleteQueue

      DocumentsWriterDeleteQueue(InfoStream infoStream)

    • DocumentsWriterDeleteQueue

      private DocumentsWriterDeleteQueue(InfoStream infoStream, long generation, long startSeqNo, LongSupplier previousMaxSeqId)

  • Method Details

    • addDelete

      long addDelete(Query... queries)

    • addDelete

      long addDelete(Term... terms)

    • addDocValuesUpdates

      long addDocValuesUpdates(DocValuesUpdate... updates)

    • newNode

      static DocumentsWriterDeleteQueue.Node<Term> newNode(Term term)

    • newNode

      static DocumentsWriterDeleteQueue.Node<Query> newNode(Query query)

    • newNode

      static DocumentsWriterDeleteQueue.Node<DocValuesUpdate[]> newNode(DocValuesUpdate... updates)

    • add

      long add(DocumentsWriterDeleteQueue.Node<?> deleteNode, DocumentsWriterDeleteQueue.DeleteSlice slice)
      invariant for document update

    • add

      long add(DocumentsWriterDeleteQueue.Node<?> newNode)

    • anyChanges

      boolean anyChanges()

    • tryApplyGlobalSlice

      void tryApplyGlobalSlice()

    • freezeGlobalBuffer

      FrozenBufferedUpdates freezeGlobalBuffer(DocumentsWriterDeleteQueue.DeleteSlice callerSlice)

    • maybeFreezeGlobalBuffer

      FrozenBufferedUpdates maybeFreezeGlobalBuffer()
      This may freeze the global buffer unless the delete queue has already been closed. If the queue has been closed this method will return null

    • freezeGlobalBufferInternal

      private FrozenBufferedUpdates freezeGlobalBufferInternal(DocumentsWriterDeleteQueue.Node<?> currentTail)

    • newSlice

      DocumentsWriterDeleteQueue.DeleteSlice newSlice()

    • updateSlice

      long updateSlice(DocumentsWriterDeleteQueue.DeleteSlice slice)
      Negative result means there were new deletes since we last applied

    • updateSliceNoSeqNo

      boolean updateSliceNoSeqNo(DocumentsWriterDeleteQueue.DeleteSlice slice)
      Just like updateSlice, but does not assign a sequence number

    • ensureOpen

      private void ensureOpen()

    • isOpen

      public boolean isOpen()

    • close

      public void close()
      Specified by:
      close in interface AutoCloseable
      Specified by:
      close in interface Closeable

    • numGlobalTermDeletes

      int numGlobalTermDeletes()
      For test purposes.

    • clear

      void clear()

    • getBufferedUpdatesTermsSize

      public int getBufferedUpdatesTermsSize()

    • ramBytesUsed

      public long ramBytesUsed()
      Description copied from interface: Accountable
      Return the memory usage of this object in bytes. Negative values are illegal.
      Specified by:
      ramBytesUsed in interface Accountable

    • toString

      public String toString()
      Overrides:
      toString in class Object

    • getNextSequenceNumber

      public long getNextSequenceNumber()

    • getLastSequenceNumber

      long getLastSequenceNumber()

    • skipSequenceNumbers

      void skipSequenceNumbers(long jump)
      Inserts a gap in the sequence numbers. This is used by IW during flush or commit to ensure any in-flight threads get sequence numbers inside the gap

    • getMaxCompletedSeqNo

      long getMaxCompletedSeqNo()
      Returns the maximum completed seq no for this queue.

    • getPrevMaxSeqIdSupplier

      private static LongSupplier getPrevMaxSeqIdSupplier(AtomicLong nextSeqNo)

    • advanceQueue

      DocumentsWriterDeleteQueue advanceQueue(int maxNumPendingOps)
      Advances the queue to the next queue on flush. This carries over the the generation to the next queue and set the getMaxSeqNo() based on the given maxNumPendingOps. This method can only be called once, subsequently the returned queue should be used.
      Parameters:
      maxNumPendingOps - the max number of possible concurrent operations that will execute on this queue after it was advanced. This corresponds the the number of DWPTs that own the current queue at the moment when this queue is advanced since each these DWPTs can increment the seqId after we advanced it.
      Returns:
      a new queue as a successor of this queue.

    • getMaxSeqNo

      long getMaxSeqNo()
      Returns the maximum sequence number for this queue. This value will change once this queue is advanced.

    • isAdvanced

      boolean isAdvanced()
      Returns true if it was advanced.