Module org.apache.lucene.misc

Package org.apache.lucene.misc.index

Class BPReorderingMergePolicy

java.lang.Object

org.apache.lucene.index.MergePolicy

org.apache.lucene.index.FilterMergePolicy

org.apache.lucene.misc.index.BPReorderingMergePolicy

All Implemented Interfaces:

Unwrappable<MergePolicy>

public final class BPReorderingMergePolicy
extends FilterMergePolicy

A merge policy that reorders merged segments according to a BPIndexReorderer. When reordering doesn't have enough RAM, it simply skips reordering in order not to fail the merge. So make sure to give enough RAM to your BPIndexReorderer via BPIndexReorderer.setRAMBudgetMB(double).

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.lucene.index.MergePolicy

MergePolicy.MergeAbortedException, MergePolicy.MergeContext, MergePolicy.MergeException, MergePolicy.MergeSpecification, MergePolicy.OneMerge, MergePolicy.OneMergeProgress

Field Summary

Fields

Modifier and Type	Field	Description
private int	minNaturalMergeNumDocs
private float	minNaturalMergeRatioFromBiggestSegment
(package private) static final String	REORDERED	Whether a segment has been reordered.
private final BPIndexReorderer	reorderer

Fields inherited from class org.apache.lucene.index.FilterMergePolicy

in

Fields inherited from class org.apache.lucene.index.MergePolicy

DEFAULT_MAX_CFS_SEGMENT_SIZE, DEFAULT_NO_CFS_RATIO, maxCFSSegmentSize, noCFSRatio

Constructor Summary

Constructors

Constructor	Description
BPReorderingMergePolicy(MergePolicy in, BPIndexReorderer reorderer)	Sole constructor.

Method Summary

Modifier and Type	Method	Description
MergePolicy.MergeSpecification	findForcedDeletesMerges(SegmentInfos segmentInfos, MergePolicy.MergeContext mergeContext)	Determine what set of merge operations is necessary in order to expunge all deletes from the index.
MergePolicy.MergeSpecification	findForcedMerges(SegmentInfos segmentInfos, int maxSegmentCount, Map<SegmentCommitInfo,Boolean> segmentsToMerge, MergePolicy.MergeContext mergeContext)	Determine what set of merge operations is necessary in order to merge to <= the specified segment count.
MergePolicy.MergeSpecification	findFullFlushMerges(MergeTrigger mergeTrigger, SegmentInfos segmentInfos, MergePolicy.MergeContext mergeContext)	Identifies merges that we want to execute (synchronously) on commit.
MergePolicy.MergeSpecification	findMerges(CodecReader... readers)	Define the set of merge operations to perform on provided codec readers in IndexWriter.addIndexes(CodecReader...).
MergePolicy.MergeSpecification	findMerges(MergeTrigger mergeTrigger, SegmentInfos segmentInfos, MergePolicy.MergeContext mergeContext)	Determine what set of merge operations are now necessary on the index.
private MergePolicy.MergeSpecification	maybeReorder(MergePolicy.MergeSpecification spec, boolean forced, SegmentInfos infos)
void	setMinNaturalMergeNumDocs(int minNaturalMergeNumDocs)	Set the minimum number of docs that a merge must have for the resulting segment to be reordered.
void	setMinNaturalMergeRatioFromBiggestSegment(float minNaturalMergeRatioFromBiggestSegment)	Set the minimum number of docs that a merge must have for the resulting segment to be reordered, as a ratio of the total number of documents of the current biggest segment in the index.

Methods inherited from class org.apache.lucene.index.FilterMergePolicy

getMaxCFSSegmentSizeMB, getNoCFSRatio, keepFullyDeletedSegment, maxFullFlushMergeSize, numDeletesToMerge, setMaxCFSSegmentSizeMB, setNoCFSRatio, size, toString, unwrap, useCompoundFile

Methods inherited from class org.apache.lucene.index.MergePolicy

assertDelCount, isMerged, message, segString, verbose

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Details

REORDERED

static final String REORDERED

Whether a segment has been reordered.

See Also:

• Constant Field Values

reorderer

private final BPIndexReorderer reorderer

minNaturalMergeNumDocs

private int minNaturalMergeNumDocs

minNaturalMergeRatioFromBiggestSegment

private float minNaturalMergeRatioFromBiggestSegment

Constructor Details

BPReorderingMergePolicy

public BPReorderingMergePolicy(MergePolicy in,
 BPIndexReorderer reorderer)

Sole constructor. It takes the merge policy that should be used to compute merges, and will then reorder doc IDs from all merges above the configured minimum doc count, as well as all forced merges.

If you wish to only run reordering upon forced merges, pass Integer.MAX_VALUE as a minNaturalMergeNumDocs. Otherwise a default value of 2^18 = 262,144 is suggested. This should help retain merging optimizations on small merges while reordering the larger segments that are important for good search performance.

Parameters:

in - the merge policy to use to compute merges

reorderer - the BPIndexReorderer to use to renumber doc IDs

Method Details

setMinNaturalMergeNumDocs

public void setMinNaturalMergeNumDocs(int minNaturalMergeNumDocs)

Set the minimum number of docs that a merge must have for the resulting segment to be reordered.

setMinNaturalMergeRatioFromBiggestSegment

public void setMinNaturalMergeRatioFromBiggestSegment(float minNaturalMergeRatioFromBiggestSegment)

Set the minimum number of docs that a merge must have for the resulting segment to be reordered, as a ratio of the total number of documents of the current biggest segment in the index. This parameter helps only enable reordering on segments that are large enough that they will significantly contribute to overall search performance.

maybeReorder

private MergePolicy.MergeSpecification maybeReorder(MergePolicy.MergeSpecification spec,
 boolean forced,
 SegmentInfos infos)

findMerges

public MergePolicy.MergeSpecification findMerges(MergeTrigger mergeTrigger,
 SegmentInfos segmentInfos,
 MergePolicy.MergeContext mergeContext)
                                          throws IOException

Description copied from class: MergePolicy

Determine what set of merge operations are now necessary on the index. IndexWriter calls this whenever there is a change to the segments. This call is always synchronized on the IndexWriter instance so only one thread at a time will call this method.

Overrides:

findMerges in class FilterMergePolicy

Parameters:

mergeTrigger - the event that triggered the merge

segmentInfos - the total set of segments in the index

mergeContext - the IndexWriter to find the merges on

Throws:

IOException

findForcedMerges

public MergePolicy.MergeSpecification findForcedMerges(SegmentInfos segmentInfos,
 int maxSegmentCount,
 Map<SegmentCommitInfo,Boolean> segmentsToMerge,
 MergePolicy.MergeContext mergeContext)
                                                throws IOException

Description copied from class: MergePolicy

Determine what set of merge operations is necessary in order to merge to <= the specified segment count. IndexWriter calls this when its IndexWriter.forceMerge(int) method is called. This call is always synchronized on the IndexWriter instance so only one thread at a time will call this method.

Overrides:

findForcedMerges in class FilterMergePolicy

Parameters:

segmentInfos - the total set of segments in the index

maxSegmentCount - requested maximum number of segments in the index

segmentsToMerge - contains the specific SegmentInfo instances that must be merged away. This may be a subset of all SegmentInfos. If the value is True for a given SegmentInfo, that means this segment was an original segment present in the to-be-merged index; else, it was a segment produced by a cascaded merge.

mergeContext - the MergeContext to find the merges on

Throws:

IOException

findForcedDeletesMerges

public MergePolicy.MergeSpecification findForcedDeletesMerges(SegmentInfos segmentInfos,
 MergePolicy.MergeContext mergeContext)
                                                       throws IOException

Description copied from class: MergePolicy

Determine what set of merge operations is necessary in order to expunge all deletes from the index.

Overrides:

findForcedDeletesMerges in class FilterMergePolicy

Parameters:

segmentInfos - the total set of segments in the index

mergeContext - the MergeContext to find the merges on

Throws:

IOException

findFullFlushMerges

public MergePolicy.MergeSpecification findFullFlushMerges(MergeTrigger mergeTrigger,
 SegmentInfos segmentInfos,
 MergePolicy.MergeContext mergeContext)
                                                   throws IOException

Description copied from class: MergePolicy

Identifies merges that we want to execute (synchronously) on commit. By default, this will return natural merges whose segments are all less than the max segment size for full flushes.

Any merges returned here will make IndexWriter.commit(), IndexWriter.prepareCommit() or IndexWriter.getReader(boolean, boolean) block until the merges complete or until LiveIndexWriterConfig.getMaxFullFlushMergeWaitMillis() has elapsed. This may be used to merge small segments that have just been flushed, reducing the number of segments in the point in time snapshot. If a merge does not complete in the allotted time, it will continue to execute, and eventually finish and apply to future point in time snapshot, but will not be reflected in the current one.

If a MergePolicy.OneMerge in the returned MergePolicy.MergeSpecification includes a segment already included in a registered merge, then IndexWriter.commit() or IndexWriter.prepareCommit() will throw a IllegalStateException. Use MergePolicy.MergeContext.getMergingSegments() to determine which segments are currently registered to merge.

Overrides:

findFullFlushMerges in class FilterMergePolicy

Parameters:

mergeTrigger - the event that triggered the merge (COMMIT or GET_READER).

segmentInfos - the total set of segments in the index (while preparing the commit)

mergeContext - the MergeContext to find the merges on, which should be used to determine which segments are already in a registered merge (see MergePolicy.MergeContext.getMergingSegments()).

Throws:

IOException

findMerges

public MergePolicy.MergeSpecification findMerges(CodecReader... readers)
                                          throws IOException

Description copied from class: MergePolicy

Define the set of merge operations to perform on provided codec readers in IndexWriter.addIndexes(CodecReader...).

The merge operation is required to convert provided readers into segments that can be added to the writer. This API can be overridden in custom merge policies to control the concurrency for addIndexes. Default implementation creates a single merge operation for all provided readers (lowest concurrency). Creating a merge for each reader, would provide the highest level of concurrency possible with the configured merge scheduler.

Overrides:

findMerges in class FilterMergePolicy

Parameters:

readers - CodecReader(s) to merge into the main index

Throws:

IOException