Module org.apache.lucene.misc

Package org.apache.lucene.misc.store

Class DirectIODirectory

java.lang.Object

org.apache.lucene.store.Directory

org.apache.lucene.store.FilterDirectory

org.apache.lucene.misc.store.DirectIODirectory

All Implemented Interfaces:

Closeable, AutoCloseable

public class DirectIODirectory
extends FilterDirectory

A Directory implementation for all Unixes and Windows that uses DIRECT I/O to bypass OS level IO caching during merging. For all other cases (searching, writing) we delegate to the provided Directory instance.

See Overview for more details.

WARNING: this code is very new and quite easily could contain horrible bugs.

This directory passes Solr and Lucene tests on Linux, OS X, and Windows; other systems should work but have not been tested! Use at your own risk.

@throws UnsupportedOperationException if the operating system, file system or JDK does not support Direct I/O or a sufficient equivalent.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
private static final class	DirectIODirectory.DirectIOIndexInput
private static final class	DirectIODirectory.DirectIOIndexOutput

Field Summary

Fields

Modifier and Type	Field	Description
private final int	blockSize
static final int	DEFAULT_MERGE_BUFFER_SIZE	Default buffer size before writing to disk (256 KB); larger means less IO load but more RAM and direct buffer storage space consumed during merging.
static final long	DEFAULT_MIN_BYTES_DIRECT	Default min expected merge size before direct IO is used (10 MB):
(package private) static final OpenOption	ExtendedOpenOption_DIRECT	Reference to com.sun.nio.file.ExtendedOpenOption.DIRECT by reflective class and enum lookup.
(package private) boolean	isOpen
private final int	mergeBufferSize
private final long	minBytesDirect

Fields inherited from class org.apache.lucene.store.FilterDirectory

in

Constructor Summary

Constructors

Constructor	Description
DirectIODirectory(FSDirectory delegate)	Create a new DirectIODirectory for the named location.
DirectIODirectory(FSDirectory delegate, int mergeBufferSize, long minBytesDirect)	Create a new DirectIODirectory for the named location.

Method Summary

Modifier and Type	Method	Description
void	close()	Closes the directory.
IndexOutput	createOutput(String name, IOContext context)	Creates a new, empty file in the directory and returns an IndexOutput instance for appending data to this file.
protected void	ensureOpen()	Ensures this directory is still open.
private static OpenOption	getDirectOpenOption()
Path	getDirectory()
IndexInput	openInput(String name, IOContext context)	Opens a stream for reading an existing file.
protected boolean	useDirectIO(String name, IOContext context, OptionalLong fileLength)	Determines if direct IO should be used for a file.

Methods inherited from class org.apache.lucene.store.FilterDirectory

createTempOutput, deleteFile, fileLength, getDelegate, getPendingDeletions, listAll, obtainLock, rename, sync, syncMetaData, toString, unwrap

Methods inherited from class org.apache.lucene.store.Directory

copyFrom, getTempFileName, openChecksumInput

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Details

DEFAULT_MERGE_BUFFER_SIZE

public static final int DEFAULT_MERGE_BUFFER_SIZE

Default buffer size before writing to disk (256 KB); larger means less IO load but more RAM and direct buffer storage space consumed during merging.

See Also:

• Constant Field Values

DEFAULT_MIN_BYTES_DIRECT

public static final long DEFAULT_MIN_BYTES_DIRECT

Default min expected merge size before direct IO is used (10 MB):

See Also:

• Constant Field Values

blockSize

private final int blockSize

mergeBufferSize

private final int mergeBufferSize

minBytesDirect

private final long minBytesDirect

isOpen

volatile boolean isOpen

ExtendedOpenOption_DIRECT

static final OpenOption ExtendedOpenOption_DIRECT

Reference to com.sun.nio.file.ExtendedOpenOption.DIRECT by reflective class and enum lookup. There are two reasons for using this instead of directly referencing ExtendedOpenOption.DIRECT:

1. ExtendedOpenOption.DIRECT is OpenJDK's internal proprietary API. This API causes un-suppressible(?) warning to be emitted when compiling with --release flag and value N, where N is smaller than the the version of javac used for compilation. For details, please refer to https://bugs.java.com/bugdatabase/view_bug.do?bug_id=JDK-8259039.
2. It is possible that Lucene is run using JDK that does not support ExtendedOpenOption.DIRECT. In such a case, dynamic lookup allows us to bail out with UnsupportedOperationException with meaningful error message.

This reference is null, if the JDK does not support direct I/O.

Constructor Details

DirectIODirectory

public DirectIODirectory(FSDirectory delegate,
 int mergeBufferSize,
 long minBytesDirect)
                  throws IOException

Create a new DirectIODirectory for the named location.

Parameters:

delegate - Directory for non-merges, also used as reference to file system path.

mergeBufferSize - Size of buffer to use for merging.

minBytesDirect - Merges, or files to be opened for reading, smaller than this will not use direct IO. See DEFAULT_MIN_BYTES_DIRECT and useDirectIO(java.lang.String, org.apache.lucene.store.IOContext, java.util.OptionalLong).

Throws:

IOException - If there is a low-level I/O error

DirectIODirectory

public DirectIODirectory(FSDirectory delegate)
                  throws IOException

Create a new DirectIODirectory for the named location.

Parameters:

delegate - Directory for non-merges, also used as reference to file system path.

Throws:

IOException - If there is a low-level I/O error

Method Details

getDirectory

public Path getDirectory()

Returns:

the underlying file system directory

ensureOpen

protected void ensureOpen()
                   throws AlreadyClosedException

Description copied from class: Directory

Ensures this directory is still open.

Overrides:

ensureOpen in class FilterDirectory

Throws:

AlreadyClosedException - if this directory is closed.

useDirectIO

protected boolean useDirectIO(String name,
 IOContext context,
 OptionalLong fileLength)

Determines if direct IO should be used for a file. By default this tests if it is a merge context and if the merge or file length extends the minimum size (see DEFAULT_MIN_BYTES_DIRECT). Subclasses may override method to enforce direct IO for specific file types.

Parameters:

name - file name (unused by default implementation)

context - information about merge size

fileLength - if available, gives the file length. Will be empty when requesting an IndexOutput.

Returns:

true if direct IO should be used; false if input/output should be requested from delegate directory.

openInput

public IndexInput openInput(String name,
 IOContext context)
                     throws IOException

Description copied from class: Directory

Opens a stream for reading an existing file.

This method must throw either NoSuchFileException or FileNotFoundException if name points to a non-existing file.

Overrides:

openInput in class FilterDirectory

Parameters:

name - the name of an existing file.

Throws:

IOException - in case of I/O error

createOutput

public IndexOutput createOutput(String name,
 IOContext context)
                         throws IOException

Description copied from class: Directory

Creates a new, empty file in the directory and returns an IndexOutput instance for appending data to this file.

This method must throw FileAlreadyExistsException if the file already exists.

Overrides:

createOutput in class FilterDirectory

Parameters:

name - the name of the file to create.

Throws:

IOException - in case of I/O error

close

public void close()
           throws IOException

Description copied from class: Directory

Closes the directory.

Specified by:

close in interface AutoCloseable

Specified by:

close in interface Closeable

Overrides:

close in class FilterDirectory

Throws:

IOException

getDirectOpenOption

private static OpenOption getDirectOpenOption()