org.mapdb

Class AsyncWriteEngine

java.lang.Object

org.mapdb.EngineWrapper

org.mapdb.AsyncWriteEngine

All Implemented Interfaces:

Engine

public class AsyncWriteEngine
extends EngineWrapper
implements Engine

Engine wrapper which provides asynchronous serialization and asynchronous write. This class takes an object instance, passes it to background writer thread (using Write Queue) where it is serialized and written to disk. Async write does not affect commit durability, Write Queue is flushed into disk on each commit. Modified records are held in small instance cache, until they are written into disk. This feature is disabled by default and can be enabled by calling DBMaker.asyncWriteEnable(). Write Cache is flushed in regular intervals or when it becomes full. Flush interval is 100 ms by default and can be controlled by DBMaker.asyncWriteFlushDelay(int). Increasing this interval may improve performance in scenarios where frequently modified items should be cached, typically BTreeMap import where keys are presorted. Asynchronous write does not affect commit durability. Write Queue is flushed during each commit, rollback and close call. Those method also block until all records are written. You may flush Write Queue manually by using clearCache() method. There is global lock which prevents record being updated while commit is in progress. This wrapper starts one threads named `MapDB writer #N` (where N is static counter). Async Writer takes modified records from Write Queue and writes them into store. It also preallocates new recids, as finding empty `recids` takes time so small stash is pre-allocated. It runs as `daemon`, so it does not prevent JVM to exit. Asynchronous Writes have several advantages (especially for single threaded user). But there are two things user should be aware of: * Because data are serialized on back-ground thread, they need to be thread safe or better immutable. When you insert record into MapDB and modify it latter, this modification may happen before item was serialized and you may not be sure what version was persisted * Inter-thread communication has some overhead. There is also only single Writer Thread, which may create single bottle-neck. This usually not issue for single or two threads, but in multi-threaded environment it may decrease performance. So in truly concurrent environments with many updates (network servers, parallel computing ) you should keep Asynchronous Writes disabled.

Author:

Jan Kotek

See Also:

Engine, EngineWrapper

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected static class	AsyncWriteEngine.WriterRunnable

Nested classes/interfaces inherited from class org.mapdb.EngineWrapper

EngineWrapper.CloseOnJVMShutdown, EngineWrapper.ImmutabilityCheckEngine, EngineWrapper.ReadOnlyEngine, EngineWrapper.SerializerCheckEngineWrapper, EngineWrapper.SynchronizedEngineWrapper

Field Summary

Fields

Modifier and Type	Field and Description
protected AtomicReference<CountDownLatch>	action
protected CountDownLatch	activeThreadsCount number of active threads running, used to await thread termination on close
protected int	asyncFlushDelay flush Write Queue every N milliseconds
protected boolean	closeInProgress indicates that `close()` was called and background threads are being terminated
protected ReentrantReadWriteLock	commitLock Each insert to Write Queue must hold read lock.
protected int	maxSize
protected AtomicInteger	size
protected static AtomicLong	threadCounter ensures thread name is followed by number
protected Throwable	threadFailedException If background thread fails with exception, it is stored here, and rethrown to all callers.
protected static Object	TOMBSTONE used to signal that object was deleted
protected LongConcurrentHashMap<Fun.Tuple2<Object,Serializer>>	writeCache Associates `recid` from Write Queue with record data and serializer.

Fields inherited from class org.mapdb.EngineWrapper

CLOSED

Fields inherited from interface org.mapdb.Engine

CATALOG_RECID, CHECK_RECORD, CLASS_INFO_RECID, LAST_RESERVED_RECID

Constructor Summary

Constructors

Constructor and Description
AsyncWriteEngine(Engine engine)
AsyncWriteEngine(Engine engine, int _asyncFlushDelay, int queueSize, Executor executor) Construct new class and starts background threads.

Method Summary

Modifier and Type	Method and Description
protected void	checkState() checks that background threads are ready and throws exception if not
void	clearCache() clears any underlying cache
void	close() Close store/cache.
void	commit() Makes all changes made since the previous commit/rollback permanent.
void	compact() This method blocks all put/update/delete operations until it finishes (via global ReadWrite Commit Lock).
<A> boolean	compareAndSwap(long recid, A expectedOldValue, A newValue, Serializer<A> serializer) Updates existing record in atomic (Compare And Swap) manner.
<A> void	delete(long recid, Serializer<A> serializer) Remove existing record from store/cache Recid must be a number returned by 'put' method.
<A> A	get(long recid, Serializer<A> serializer) Get existing record.
<A> long	put(A value, Serializer<A> serializer) Insert new record.
void	rollback() Undoes all changes made in the current transaction.
protected boolean	runWriter() runs on background thread.
protected void	startThreads(Executor executor) Starts background threads.
<A> void	update(long recid, A value, Serializer<A> serializer) Update existing record with new value.
protected void	waitForAction(int actionNumber)

Methods inherited from class org.mapdb.EngineWrapper

canRollback, canSnapshot, checkClosed, closeListenerRegister, closeListenerUnregister, getSerializerPojo, getWrappedEngine, isClosed, isReadOnly, preallocate, preallocate, snapshot

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.mapdb.Engine

canRollback, canSnapshot, closeListenerRegister, closeListenerUnregister, getSerializerPojo, isClosed, isReadOnly, preallocate, preallocate, snapshot

Field Detail

threadCounter

protected static final AtomicLong threadCounter

ensures thread name is followed by number

TOMBSTONE

protected static final Object TOMBSTONE

used to signal that object was deleted

maxSize

protected final int maxSize

size

protected final AtomicInteger size

writeCache

protected final LongConcurrentHashMap<Fun.Tuple2<Object,Serializer>> writeCache

Associates `recid` from Write Queue with record data and serializer.

commitLock

protected final ReentrantReadWriteLock commitLock

Each insert to Write Queue must hold read lock. Commit, rollback and close operations must hold write lock

activeThreadsCount

protected final CountDownLatch activeThreadsCount

number of active threads running, used to await thread termination on close

threadFailedException

protected volatile Throwable threadFailedException

If background thread fails with exception, it is stored here, and rethrown to all callers.

closeInProgress

protected volatile boolean closeInProgress

indicates that `close()` was called and background threads are being terminated

asyncFlushDelay

protected final int asyncFlushDelay

flush Write Queue every N milliseconds

action

protected final AtomicReference<CountDownLatch> action

Constructor Detail

AsyncWriteEngine

public AsyncWriteEngine(Engine engine,
                        int _asyncFlushDelay,
                        int queueSize,
                        Executor executor)

Construct new class and starts background threads. User may provide executor in which background tasks will be executed, otherwise MapDB starts two daemon threads.

Parameters:

engine - which stores data.

_asyncFlushDelay - flush Write Queue every N milliseconds

executor - optional executor to run tasks. If null daemon threads will be created

AsyncWriteEngine

public AsyncWriteEngine(Engine engine)

Method Detail

startThreads

protected void startThreads(Executor executor)

Starts background threads. You may override this if you wish to start thread different way

Parameters:

executor - optional executor to run tasks, if null deamon threads will be created

runWriter

protected boolean runWriter()
                     throws InterruptedException

runs on background thread. Takes records from Write Queue, serializes and writes them.

Throws:

InterruptedException

checkState

protected void checkState()

checks that background threads are ready and throws exception if not

put

public <A> long put(A value,
                    Serializer<A> serializer)

Insert new record. Recids are managed by underlying Engine. Finding free or allocating new recids may take some time, so for this reason recids are preallocated by Writer Thread and stored in queue. This method just takes preallocated recid from queue with minimal delay. Newly inserted records are not written synchronously, but forwarded to background Writer Thread via queue. 

Specified by:

put in interface Engine

Overrides:

put in class EngineWrapper

Parameters:

value - records to be added

serializer - used to convert record into/from binary form

Returns:

recid (record identifier) under which record is stored.

get

public <A> A get(long recid,
                 Serializer<A> serializer)

Get existing record.

Recid must be a number returned by 'put' method. Behaviour for invalid recid (random number or already deleted record) is not defined, typically it returns null or throws 'EndOfFileException' This method first looks up into Write Cache if record is not currently being written. If not it continues as usually

Specified by:

get in interface Engine

Overrides:

get in class EngineWrapper

Parameters:

recid - (record identifier) under which record was persisted

serializer - used to deserialize record from binary form

Returns:

record matching given recid, or null if record is not found under given recid.

update

public <A> void update(long recid,
                       A value,
                       Serializer<A> serializer)

Update existing record with new value.

Recid must be a number returned by 'put' method. Behaviour for invalid recid (random number or already deleted record) is not defined, typically it throws 'EndOfFileException', but it may also corrupt store. This methods forwards record into Writer Thread and returns asynchronously. 

Specified by:

update in interface Engine

Overrides:

update in class EngineWrapper

Parameters:

recid - (record identifier) under which record was persisted.

value - new record value to be stored

serializer - used to serialize record into binary form

compareAndSwap

public <A> boolean compareAndSwap(long recid,
                                  A expectedOldValue,
                                  A newValue,
                                  Serializer<A> serializer)

Updates existing record in atomic (Compare And Swap) manner. Value is modified only if old value matches expected value. There are three ways to match values, MapDB may use any of them:

1. Equality check oldValue==expectedOldValue when old value is found in instance cache
2. Deserializing oldValue using serializer and checking oldValue.equals(expectedOldValue)
3. Serializing expectedOldValue using serializer and comparing binary array with already serialized oldValue

Recid must be a number returned by 'put' method. Behaviour for invalid recid (random number or already deleted record) is not defined, typically it throws 'EndOfFileException', but it may also corrupt store. This method first looks up Write Cache if record is not currently being written. Successful modifications are forwarded to Write Thread and method returns asynchronously. Asynchronicity does not affect atomicity.

Specified by:

compareAndSwap in interface Engine

Overrides:

compareAndSwap in class EngineWrapper

Parameters:

recid - (record identifier) under which record was persisted.

expectedOldValue - old value to be compared with existing record

newValue - to be written if values are matching

serializer - used to serialize record into binary form

Returns:

true if values matched and newValue was written

delete

public <A> void delete(long recid,
                       Serializer<A> serializer)

Remove existing record from store/cache

Recid must be a number returned by 'put' method. Behaviour for invalid recid (random number or already deleted record) is not defined, typically it throws 'EndOfFileException', but it may also corrupt store. This method places 'tombstone' into Write Queue so record is eventually deleted asynchronously. However record is visible as deleted immediately.

Specified by:

delete in interface Engine

Overrides:

delete in class EngineWrapper

Parameters:

recid - (record identifier) under which was record persisted

serializer - which may be used in some circumstances to deserialize and store old object

close

public void close()

Close store/cache. This method must be called before JVM exits to flush all caches and prevent store corruption. Also it releases resources used by MapDB (disk, memory..).

Engine can no longer be used after this method was called. If Engine is used after closing, it may throw any exception including NullPointerException

There is an configuration option DBMaker.closeOnJvmShutdown() which uses shutdown hook to automatically close Engine when JVM shutdowns. This method blocks until Write Queue is flushed and Writer Thread writes all records and finishes. When this method was called `closeInProgress` is set and no record can be modified.

Specified by:

close in interface Engine

Overrides:

close in class EngineWrapper

waitForAction

protected void waitForAction(int actionNumber)

commit

public void commit()

Makes all changes made since the previous commit/rollback permanent. In transactional mode (on by default) it means creating journal file and replaying it to storage. In other modes it may flush disk caches or do nothing at all (check your config options) This method blocks until Write Queue is flushed. All put/update/delete methods are blocked while commit is in progress (via global ReadWrite Commit Lock). After this method returns, commit lock is released and other operations may continue

Specified by:

commit in interface Engine

Overrides:

commit in class EngineWrapper

rollback

public void rollback()

Undoes all changes made in the current transaction. If transactions are disabled it throws UnsupportedOperationException. This method blocks until Write Queue is cleared. All put/update/delete methods are blocked while rollback is in progress (via global ReadWrite Commit Lock). After this method returns, commit lock is released and other operations may continue

Specified by:

rollback in interface Engine

Overrides:

rollback in class EngineWrapper

compact

public void compact()

This method blocks all put/update/delete operations until it finishes (via global ReadWrite Commit Lock).

Specified by:

compact in interface Engine

Overrides:

compact in class EngineWrapper

clearCache

public void clearCache()

clears any underlying cache This method blocks until Write Queue is empty (written into disk). It also blocks any put/update/delete operations until it finishes (via global ReadWrite Commit Lock).

Specified by:

clearCache in interface Engine

Overrides:

clearCache in class EngineWrapper