Module org.apache.lucene.core

Package org.apache.lucene.util

Class OfflineSorter

java.lang.Object

org.apache.lucene.util.OfflineSorter

public class OfflineSorter
extends Object

On-disk sorting of byte arrays. Each byte array (entry) is composed of the following fields:

• (two bytes) length of the following byte array,
• exactly the above count of bytes for the sequence to be sorted.

See Also:

• sort(String)

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static final class	OfflineSorter.BufferSize	A bit more descriptive unit for constructors.
static class	OfflineSorter.ByteSequencesReader	Utility class to read length-prefixed byte[] entries from an input.
static class	OfflineSorter.ByteSequencesWriter	Utility class to emit length-prefixed byte[] entries to an output stream for sorting.
(package private) static class	OfflineSorter.FileAndTop
private class	OfflineSorter.MergePartitionsTask	Merges multiple file-based partitions to a single on-disk partition.
private static class	OfflineSorter.Partition	Holds one partition of items, either loaded into memory or based on a file.
class	OfflineSorter.SortInfo	Sort info (debugging mostly).
private class	OfflineSorter.SortPartitionTask	Sorts one in-memory partition, writes it to disk, and returns the resulting file-based partition.

Field Summary

Fields

Modifier and Type	Field	Description
static final long	ABSOLUTE_MIN_SORT_BUFFER_SIZE	Absolute minimum required buffer size for sorting.
private final Comparator<BytesRef>	comparator
static final Comparator<BytesRef>	DEFAULT_COMPARATOR	Default comparator: sorts in binary (codepoint) order
private final Directory	dir
private final ExecutorService	exec
static final long	GB	Convenience constant for gigabytes
static final int	MAX_TEMPFILES	Maximum number of temporary files before doing an intermediate merge.
private final int	maxTempFiles
static final long	MB	Convenience constant for megabytes
static final long	MIN_BUFFER_SIZE_MB	Minimum recommended buffer size for sorting.
private static final String	MIN_BUFFER_SIZE_MSG
private final Semaphore	partitionsInRAM
private final OfflineSorter.BufferSize	ramBufferSize
(package private) OfflineSorter.SortInfo	sortInfo
private final String	tempFileNamePrefix
private final int	valueLength

Constructor Summary

Constructors

Constructor	Description
OfflineSorter(Directory dir, String tempFileNamePrefix)	Defaults constructor.
OfflineSorter(Directory dir, String tempFileNamePrefix, Comparator<BytesRef> comparator)	Defaults constructor with a custom comparator.
OfflineSorter(Directory dir, String tempFileNamePrefix, Comparator<BytesRef> comparator, OfflineSorter.BufferSize ramBufferSize, int maxTempfiles, int valueLength, ExecutorService exec, int maxPartitionsInRAM)	All-details constructor.

Method Summary

Modifier and Type	Method	Description
Comparator<BytesRef>	getComparator()	Returns the comparator in use to sort entries
Directory	getDirectory()	Returns the Directory we use to create temp files.
private OfflineSorter.Partition	getPartition(Future<OfflineSorter.Partition> future)
protected OfflineSorter.ByteSequencesReader	getReader(ChecksumIndexInput in, String name)	Subclasses can override to change how byte sequences are read from disk.
String	getTempFileNamePrefix()	Returns the temp file name prefix passed to Directory.createTempOutput(java.lang.String, java.lang.String, org.apache.lucene.store.IOContext) to generate temporary files.
protected OfflineSorter.ByteSequencesWriter	getWriter(IndexOutput out, long itemCount)	Subclasses can override to change how byte sequences are written to disk.
(package private) void	mergePartitions(Directory trackingDir, List<Future<OfflineSorter.Partition>> segments)	Merge the most recent maxTempFile partitions into a new partition.
(package private) OfflineSorter.Partition	readPartition(OfflineSorter.ByteSequencesReader reader)	Read in a single partition of data, setting isExhausted[0] to true if there are no more items.
String	sort(String inputFileName)	Sort input to a new temp file, returning its name.
private void	verifyChecksum(Throwable priorException, OfflineSorter.ByteSequencesReader reader)	Called on exception, to check whether the checksum is also corrupt in this source, and add that information (checksum matched or didn't) as a suppressed exception.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

MB

public static final long MB

Convenience constant for megabytes

See Also:

• Constant Field Values

GB

public static final long GB

Convenience constant for gigabytes

See Also:

• Constant Field Values

MIN_BUFFER_SIZE_MB

public static final long MIN_BUFFER_SIZE_MB

Minimum recommended buffer size for sorting.

See Also:

• Constant Field Values

ABSOLUTE_MIN_SORT_BUFFER_SIZE

public static final long ABSOLUTE_MIN_SORT_BUFFER_SIZE

Absolute minimum required buffer size for sorting.

See Also:

• Constant Field Values

MIN_BUFFER_SIZE_MSG

private static final String MIN_BUFFER_SIZE_MSG

See Also:

• Constant Field Values

MAX_TEMPFILES

public static final int MAX_TEMPFILES

Maximum number of temporary files before doing an intermediate merge.

See Also:

• Constant Field Values

dir

private final Directory dir

valueLength

private final int valueLength

tempFileNamePrefix

private final String tempFileNamePrefix

exec

private final ExecutorService exec

partitionsInRAM

private final Semaphore partitionsInRAM

ramBufferSize

private final OfflineSorter.BufferSize ramBufferSize

sortInfo

OfflineSorter.SortInfo sortInfo

maxTempFiles

private final int maxTempFiles

comparator

private final Comparator<BytesRef> comparator

DEFAULT_COMPARATOR

public static final Comparator<BytesRef> DEFAULT_COMPARATOR

Default comparator: sorts in binary (codepoint) order

Constructor Details

OfflineSorter

public OfflineSorter(Directory dir,
 String tempFileNamePrefix)
              throws IOException

Defaults constructor.

Throws:

IOException

See Also:

• OfflineSorter.BufferSize.automatic()

OfflineSorter

public OfflineSorter(Directory dir,
 String tempFileNamePrefix,
 Comparator<BytesRef> comparator)
              throws IOException

Defaults constructor with a custom comparator.

Throws:

IOException

See Also:

• OfflineSorter.BufferSize.automatic()

OfflineSorter

public OfflineSorter(Directory dir,
 String tempFileNamePrefix,
 Comparator<BytesRef> comparator,
 OfflineSorter.BufferSize ramBufferSize,
 int maxTempfiles,
 int valueLength,
 ExecutorService exec,
 int maxPartitionsInRAM)

All-details constructor. If valueLength is -1 (the default), the length of each value differs; otherwise, all values have the specified length. If you pass a non-null ExecutorService then it will be used to run sorting operations that can be run concurrently, and maxPartitionsInRAM is the maximum concurrent in-memory partitions. Thus the maximum possible RAM used by this class while sorting is maxPartitionsInRAM * ramBufferSize.

Method Details

getDirectory

public Directory getDirectory()

Returns the Directory we use to create temp files.

getTempFileNamePrefix

public String getTempFileNamePrefix()

Returns the temp file name prefix passed to Directory.createTempOutput(java.lang.String, java.lang.String, org.apache.lucene.store.IOContext) to generate temporary files.

sort

public String sort(String inputFileName)
            throws IOException

Sort input to a new temp file, returning its name.

Throws:

IOException

verifyChecksum

private void verifyChecksum(Throwable priorException,
 OfflineSorter.ByteSequencesReader reader)
                     throws IOException

Called on exception, to check whether the checksum is also corrupt in this source, and add that information (checksum matched or didn't) as a suppressed exception.

Throws:

IOException

mergePartitions

void mergePartitions(Directory trackingDir,
 List<Future<OfflineSorter.Partition>> segments)
              throws IOException

Merge the most recent maxTempFile partitions into a new partition.

Throws:

IOException

readPartition

OfflineSorter.Partition readPartition(OfflineSorter.ByteSequencesReader reader)
                               throws IOException,
InterruptedException

Read in a single partition of data, setting isExhausted[0] to true if there are no more items.

Throws:

IOException

InterruptedException

getWriter

protected OfflineSorter.ByteSequencesWriter getWriter(IndexOutput out,
 long itemCount)
                                               throws IOException

Subclasses can override to change how byte sequences are written to disk.

Throws:

IOException

getReader

protected OfflineSorter.ByteSequencesReader getReader(ChecksumIndexInput in,
 String name)
                                               throws IOException

Subclasses can override to change how byte sequences are read from disk.

Throws:

IOException

getComparator

public Comparator<BytesRef> getComparator()

Returns the comparator in use to sort entries

getPartition

private OfflineSorter.Partition getPartition(Future<OfflineSorter.Partition> future)
                                      throws IOException

Throws:

IOException