StoreDirect (mapdb 1.0.7 API)

java.lang.Object
- org.mapdb.Store
- - org.mapdb.StoreDirect

All Implemented Interfaces:

Engine

Direct Known Subclasses:

StoreWAL
```
public class StoreDirect
extends Store
```
Storage Engine which saves record directly into file. It has zero protection from data corruption and must be closed properly after modifications. It is used when Write-Ahead-Log transactions are disabled. Storage format ---------------- `StoreDirect` is composed of two files: Index file is sequence of 8-byte longs, it translates `recid` (offset in index file) to record size and offset in physical file. Records position may change, but it requires stable ID, so the index file is used for translation. This store uses data structure called `Long Stack` to manage (and reuse) free space, it is is linked LIFO queue of 8-byte longs. Index file -------------- Index file is translation table between permanent record ID (recid) and mutable location in physical file. Index file is sequence of 8-byte longs, one for each record. It also has some extra longs to manage free space and other metainfo. Index table and physical data could be stored in single file, but keeping index table separate simplifies compaction. Basic **structure of index file** is bellow. Each slot is 8-bytes long so `offset=slot*8` slot | in code | description --- | --- | --- 0 | HEADER | File header, format version and flags 1 | IO_INDEX_SIZE | Allocated file size of index file in bytes. 2 | IO_PHYS_SIZE | Allocated file size of physical file in bytes. 3 | IO_FREE_SIZE | Space occupied by free records in physical file in bytes. 4 | IO_INDEX_SUM | Checksum of all Index file headers. Checks if store was closed correctly 5..9 | | Reserved for future use 10..14 | | For usage by user 15 | IO_FREE_RECID |Long Stack of deleted recids, those will be reused and returned by Engine.put(Object, Serializer) 16..4111 | |Long Stack of free physical records. This contains free space released by record update or delete. Each slots corresponds to free record size. TODO check 4111 is right 4112 | IO_USER_START |Record size and offset in physical file for recid=1 4113 | |Record size and offset in physical file for recid=2 ... | ... |... snip ... N+4111 | |Record size and offset in physical file for recid=N Long Stack ------------ Long Stack is data structure used to store free records. It is LIFO queue which uses linked records to store 8-byte longs. Long Stack is identified by slot in Index File, which stores pointer to Long Stack head. The structure of of index pointer is following: byte | description --- |--- 0..1 | relative offset in head Long Stack Record to take value from. This value decreases by 8 each take 2..7 | physical file offset of head Long Stack Record, zero if Long Stack is empty Each Long Stack Record is sequence of 8-byte longs, first slot is header. Long Stack Record structure is following: byte | description --- |--- 0..1 | length of current Long Stack Record in bytes 2..7 | physical file offset of next Long Stack Record, zero of this record is last 8-15 | Long Stack value 16-23 | Long Stack value ... | and so on until end of Long Stack Record Physical pointer ---------------- Index slot value typically contains physical pointer (information about record location and size in physical file). First 2 bytes are record size (max 65536). Then there is 6 byte offset in physical file (max store size is 281 TB). Physical file offset must always be multiple of 16, so last 4 bites are used to flag extra record information. Structure of **physical pointer**: bite | in code | description --- | --- | --- 0-15 |`val>>>48` | record size 16-59 |`val&MASK_OFFSET` | physical offset 60 |`val&MASK_LINKED!=0` | linked record flag 61 |`val&MASK_DISCARD!=0` | to be discarded while storage is offline flag 62 |`val&MASK_ARCHIVE!=0` | record modified since last backup flag 63 | | not used yet Records in Physical File --------------------------- Records are stored in physical file. Maximal record size size is 64KB, so larger records must be stored in form of the linked list. Each record starts by Physical Pointer from Index File. There is flag in Physical Pointer indicating if record is linked. If record is not linked you may just read ByteBuffer from given size and offset. If record is linked, each record starts with Physical Pointer to next record. So actual data payload is record size-8. The last linked record does not have the Physical Pointer header to next record, there is MASK_LINKED flag which indicates if next record is the last one.

Author:

Jan Kotek

Field Summary

Fields
Modifier and Type	Field and Description
`static String`	`DATA_FILE_EXT`
`protected boolean`	`deleteFilesAfterClose`
`protected long`	`freeSize`
`protected static int`	`HEADER` 4 byte file header
`protected Volume`	`index`
`protected long`	`indexSize`
`protected static int`	`IO_FREE_RECID` index file offset where reference to longstack of free recid is stored
`protected static int`	`IO_FREE_SIZE` index file offset where space occupied by free phys records is stored
`protected static int`	`IO_INDEX_SIZE` index file offset where current size of index file is stored
`protected static int`	`IO_INDEX_SUM` checksum of all index file headers.
`protected static int`	`IO_PHYS_SIZE` index file offset where current size of phys file is stored
`protected static int`	`IO_USER_START` index file offset where first recid available to user is stored
`protected static int`	`LONG_STACK_PREF_COUNT`
`protected static int`	`LONG_STACK_PREF_COUNT_ALTER`
`protected static long`	`LONG_STACK_PREF_SIZE`
`protected static long`	`LONG_STACK_PREF_SIZE_ALTER`
`protected static long`	`MASK_ARCHIVE`
`protected static long`	`MASK_DISCARD`
`protected static long`	`MASK_LINKED`
`protected static long`	`MASK_OFFSET`
`protected static int`	`MAX_REC_SIZE` maximal non linked record size
`protected long`	`maxUsedIoList` maximal non zero slot in free phys record, access requires `structuralLock`
`protected Volume`	`phys`
`protected static int`	`PHYS_FREE_SLOTS_COUNT` number of free physical slots
`protected long`	`physSize`
`protected boolean`	`readOnly`
`protected long`	`sizeLimit`
`protected boolean`	`spaceReclaimReuse`
`protected boolean`	`spaceReclaimSplit`
`protected boolean`	`spaceReclaimTrack`
`protected static short`	`STORE_VERSION` 2 byte store version
`protected boolean`	`syncOnCommitDisabled`

Fields inherited from class org.mapdb.Store
checksum, CHECKSUM_FLAG_MASK, CHUNK_SIZE, CHUNK_SIZE_MOD_MASK, compress, COMPRESS_FLAG_MASK, encrypt, ENCRYPT_FLAG_MASK, encryptionXTEA, locks, LOG, LZF, newRecidLock, password, recycledDataOuts, serializerPojo, serializerPojoInitLock, structuralLock

Fields inherited from interface org.mapdb.Engine
CATALOG_RECID, CHECK_RECORD, CLASS_INFO_RECID, LAST_RESERVED_RECID

Constructor Summary

Constructors
Constructor and Description
`StoreDirect(Volume.Factory volFac)`
`StoreDirect(Volume.Factory volFac, boolean readOnly, boolean deleteFilesAfterClose, int spaceReclaimMode, boolean syncOnCommitDisabled, long sizeLimit, boolean checksum, boolean compress, byte[] password, boolean disableLocks, int sizeIncrement)`

Method Summary

All Methods Static Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`String`	`calculateStatistics()` get some statistics about store.
`boolean`	`canRollback()`
`protected void`	`checkHeaders()`
`void`	`clearCache()` clears any underlying cache
`void`	`close()` Close store/cache.
`void`	`commit()` Makes all changes made since the previous commit/rollback permanent.
`void`	`compact()`
`protected void`	`compactPostUnderLock()` subclasses put additional cleanup after compaction finishes here
`protected void`	`compactPreUnderLock()` subclasses put additional checks before compaction starts here
`<A> boolean`	`compareAndSwap(long recid, A expectedOldValue, A newValue, Serializer<A> serializer)` Updates existing record in atomic (Compare And Swap) manner.
`protected long`	`countLongStackItems(long ioList)`
`protected void`	`createStructure()`
`<A> void`	`delete(long recid, Serializer<A> serializer)` Remove existing record from store/cache Recid must be a number returned by 'put' method.
`protected void`	`freeIoRecidPut(long ioRecid)`
`protected long`	`freeIoRecidTake(boolean ensureAvail)`
`protected void`	`freePhysPut(long indexVal, boolean recursive)`
`protected long`	`freePhysTake(int size, boolean ensureAvail, boolean recursive)`
`<A> A`	`get(long recid, Serializer<A> serializer)` Get existing record.
`protected <A> A`	`get2(long ioRecid, Serializer<A> serializer)`
`long`	`getCurrSize()` returns current size occupied by physical store (does not include index).
`Iterator<Long>`	`getFreeRecids()`
`long`	`getFreeSize()` returns free size in physical store (does not include index).
`protected long[]`	`getLinkedRecordsIndexVals(long indexVal)`
`long`	`getMaxRecid()`
`ByteBuffer`	`getRaw(long recid)`
`long`	`getSizeLimit()` returns maximal store size or `0` if there is no limit
`protected long`	`indexHeaderChecksum()`
`boolean`	`isClosed()` Checks whether Engine was closed.
`boolean`	`isReadOnly()` Check if you can write into this Engine.
`protected void`	`longStackPut(long ioList, long offset, boolean recursive)`
`protected long`	`longStackTake(long ioList, boolean recursive)`
`protected long[]`	`physAllocate(int size, boolean ensureAvail, boolean recursive)`
`long`	`preallocate()` Preallocates recid for not yet created record.
`void`	`preallocate(long[] recids)` Preallocates recids for not yet created record.
`<A> long`	`put(A value, Serializer<A> serializer)` Insert new record.
`protected void`	`put2(DataOutput2 out, long ioRecid, long[] indexVals)`
`void`	`rollback()` Undoes all changes made in the current transaction.
`protected static long`	`roundTo16(long offset)`
`protected static long`	`size2ListIoRecid(long size)`
`<A> void`	`update(long recid, A value, Serializer<A> serializer)` Update existing record with new value.
`protected void`	`update2(DataOutput2 out, long ioRecid)`
`void`	`updateRaw(long recid, ByteBuffer data)`

Methods inherited from class org.mapdb.Store
canSnapshot, closeListenerRegister, closeListenerUnregister, deserialize, expectedMasks, forDB, forEngine, getSerializerPojo, lockAllWrite, lockPos, newDataOut2, printStatistics, serialize, snapshot, unlockAllWrite

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Field Detail
  - MASK_OFFSET
```
protected static final long MASK_OFFSET
```
    See Also:
    
    Constant Field Values
  - MASK_LINKED
```
protected static final long MASK_LINKED
```
    See Also:
    
    Constant Field Values
  - MASK_DISCARD
```
protected static final long MASK_DISCARD
```
    See Also:
    
    Constant Field Values
  - MASK_ARCHIVE
```
protected static final long MASK_ARCHIVE
```
    See Also:
    
    Constant Field Values
  - HEADER
```
protected static final int HEADER
```
    4 byte file header
    
    See Also:
    
    Constant Field Values
  - STORE_VERSION
```
protected static final short STORE_VERSION
```
    2 byte store version
    
    See Also:
    
    Constant Field Values
  - MAX_REC_SIZE
```
protected static final int MAX_REC_SIZE
```
    maximal non linked record size
    
    See Also:
    
    Constant Field Values
  - PHYS_FREE_SLOTS_COUNT
```
protected static final int PHYS_FREE_SLOTS_COUNT
```
    number of free physical slots
    
    See Also:
    
    Constant Field Values
  - IO_INDEX_SIZE
```
protected static final int IO_INDEX_SIZE
```
    index file offset where current size of index file is stored
    
    See Also:
    
    Constant Field Values
  - IO_PHYS_SIZE
```
protected static final int IO_PHYS_SIZE
```
    index file offset where current size of phys file is stored
    
    See Also:
    
    Constant Field Values
  - IO_FREE_SIZE
```
protected static final int IO_FREE_SIZE
```
    index file offset where space occupied by free phys records is stored
    
    See Also:
    
    Constant Field Values
  - IO_INDEX_SUM
```
protected static final int IO_INDEX_SUM
```
    checksum of all index file headers. Used to verify store was closed correctly
    
    See Also:
    
    Constant Field Values
  - IO_FREE_RECID
```
protected static final int IO_FREE_RECID
```
    index file offset where reference to longstack of free recid is stored
    
    See Also:
    
    Constant Field Values
  - IO_USER_START
```
protected static final int IO_USER_START
```
    index file offset where first recid available to user is stored
    
    See Also:
    
    Constant Field Values
  - DATA_FILE_EXT
```
public static final String DATA_FILE_EXT
```
    See Also:
    
    Constant Field Values
  - LONG_STACK_PREF_COUNT
```
protected static final int LONG_STACK_PREF_COUNT
```
    See Also:
    
    Constant Field Values
  - LONG_STACK_PREF_SIZE
```
protected static final long LONG_STACK_PREF_SIZE
```
    See Also:
    
    Constant Field Values
  - LONG_STACK_PREF_COUNT_ALTER
```
protected static final int LONG_STACK_PREF_COUNT_ALTER
```
    See Also:
    
    Constant Field Values
  - LONG_STACK_PREF_SIZE_ALTER
```
protected static final long LONG_STACK_PREF_SIZE_ALTER
```
    See Also:
    
    Constant Field Values
  - index
```
protected Volume index
```
  - phys
```
protected Volume phys
```
  - physSize
```
protected long physSize
```
  - indexSize
```
protected long indexSize
```
  - freeSize
```
protected long freeSize
```
  - deleteFilesAfterClose
```
protected final boolean deleteFilesAfterClose
```
  - readOnly
```
protected final boolean readOnly
```
  - syncOnCommitDisabled
```
protected final boolean syncOnCommitDisabled
```
  - spaceReclaimReuse
```
protected final boolean spaceReclaimReuse
```
  - spaceReclaimSplit
```
protected final boolean spaceReclaimSplit
```
  - spaceReclaimTrack
```
protected final boolean spaceReclaimTrack
```
  - sizeLimit
```
protected final long sizeLimit
```
  - maxUsedIoList
```
protected long maxUsedIoList
```
    maximal non zero slot in free phys record, access requires `structuralLock`
- Constructor Detail
  - StoreDirect
```
public StoreDirect(Volume.Factory volFac,
                   boolean readOnly,
                   boolean deleteFilesAfterClose,
                   int spaceReclaimMode,
                   boolean syncOnCommitDisabled,
                   long sizeLimit,
                   boolean checksum,
                   boolean compress,
                   byte[] password,
                   boolean disableLocks,
                   int sizeIncrement)
```
  - StoreDirect
```
public StoreDirect(Volume.Factory volFac)
```
- Method Detail
  - checkHeaders
```
protected void checkHeaders()
```
  - createStructure
```
protected void createStructure()
```
  - indexHeaderChecksum
```
protected long indexHeaderChecksum()
```
  - preallocate
```
public long preallocate()
```
    Description copied from interface: Engine
    
    Preallocates recid for not yet created record. It does not insert any data into it.
    
    Returns:
    
    new recid
  - preallocate
```
public void preallocate(long[] recids)
```
    Description copied from interface: Engine
    
    Preallocates recids for not yet created record. It does not insert any data into it. This is done in batch of given size (determied by size of array in argument)
    
    Parameters:
    
    recids - array to put result into
  - put
```
public <A> long put(A value,
                    Serializer<A> serializer)
```
    Description copied from interface: Engine
    
    Insert new record.
    
    Parameters:
    
    value - records to be added
    
    serializer - used to convert record into/from binary form
    
    Returns:
    
    recid (record identifier) under which record is stored.
  - put2
```
protected void put2(DataOutput2 out,
                    long ioRecid,
                    long[] indexVals)
```
  - get
```
public <A> A get(long recid,
                 Serializer<A> serializer)
```
    Description copied from interface: Engine
    
    Get existing record.
    Recid must be a number returned by 'put' method. Behaviour for invalid recid (random number or already deleted record) is not defined, typically it returns null or throws 'EndOfFileException'
    
    Parameters:
    
    recid - (record identifier) under which record was persisted
    
    serializer - used to deserialize record from binary form
    
    Returns:
    
    record matching given recid, or null if record is not found under given recid.
  - get2
```
protected <A> A get2(long ioRecid,
                     Serializer<A> serializer)
              throws IOException
```
    Throws:
    
    IOException
  - update
```
public <A> void update(long recid,
                       A value,
                       Serializer<A> serializer)
```
    Description copied from interface: Engine
    
    Update existing record with new value.
    Recid must be a number returned by 'put' method. Behaviour for invalid recid (random number or already deleted record) is not defined, typically it throws 'EndOfFileException', but it may also corrupt store.
    
    Parameters:
    
    recid - (record identifier) under which record was persisted.
    
    value - new record value to be stored
    
    serializer - used to serialize record into binary form
  - update2
```
protected void update2(DataOutput2 out,
                       long ioRecid)
```
  - compareAndSwap
```
public <A> boolean compareAndSwap(long recid,
                                  A expectedOldValue,
                                  A newValue,
                                  Serializer<A> serializer)
```
    Description copied from interface: Engine
    Updates existing record in atomic (Compare And Swap) manner. Value is modified only if old value matches expected value. There are three ways to match values, MapDB may use any of them:
    1. Equality check oldValue==expectedOldValue when old value is found in instance cache
    2. Deserializing oldValue using serializer and checking oldValue.equals(expectedOldValue)
    3. Serializing expectedOldValue using serializer and comparing binary array with already serialized oldValue
    Recid must be a number returned by 'put' method. Behaviour for invalid recid (random number or already deleted record) is not defined, typically it throws 'EndOfFileException', but it may also corrupt store.
    Parameters:
    
    recid - (record identifier) under which record was persisted.
    
    expectedOldValue - old value to be compared with existing record
    
    newValue - to be written if values are matching
    
    serializer - used to serialize record into binary form
    
    Returns:
    
    true if values matched and newValue was written
  - delete
```
public <A> void delete(long recid,
                       Serializer<A> serializer)
```
    Description copied from interface: Engine
    
    Remove existing record from store/cache
    Recid must be a number returned by 'put' method. Behaviour for invalid recid (random number or already deleted record) is not defined, typically it throws 'EndOfFileException', but it may also corrupt store.
    
    Parameters:
    
    recid - (record identifier) under which was record persisted
    
    serializer - which may be used in some circumstances to deserialize and store old object
  - getLinkedRecordsIndexVals
```
protected long[] getLinkedRecordsIndexVals(long indexVal)
```
  - physAllocate
```
protected long[] physAllocate(int size,
                              boolean ensureAvail,
                              boolean recursive)
```
  - roundTo16
```
protected static long roundTo16(long offset)
```
  - close
```
public void close()
```
    Description copied from interface: Engine
    
    Close store/cache. This method must be called before JVM exits to flush all caches and prevent store corruption. Also it releases resources used by MapDB (disk, memory..).
    Engine can no longer be used after this method was called. If Engine is used after closing, it may throw any exception including NullPointerException
    There is an configuration option DBMaker.closeOnJvmShutdown() which uses shutdown hook to automatically close Engine when JVM shutdowns.
  - isClosed
```
public boolean isClosed()
```
    Description copied from interface: Engine
    
    Checks whether Engine was closed.
    
    Returns:
    
    true if engine was closed
  - commit
```
public void commit()
```
    Description copied from interface: Engine
    
    Makes all changes made since the previous commit/rollback permanent. In transactional mode (on by default) it means creating journal file and replaying it to storage. In other modes it may flush disk caches or do nothing at all (check your config options)
  - rollback
```
public void rollback()
              throws UnsupportedOperationException
```
    Description copied from interface: Engine
    
    Undoes all changes made in the current transaction. If transactions are disabled it throws UnsupportedOperationException.
    
    Throws:
    
    UnsupportedOperationException - if transactions are disabled
  - isReadOnly
```
public boolean isReadOnly()
```
    Description copied from interface: Engine
    
    Check if you can write into this Engine. It may be readonly in some cases (snapshot, read-only files).
    
    Returns:
    
    true if engine is read-only
  - canRollback
```
public boolean canRollback()
```
    Returns:
    
    true if engine supports rollback
  - clearCache
```
public void clearCache()
```
    Description copied from interface: Engine
    
    clears any underlying cache
  - compact
```
public void compact()
```
  - compactPreUnderLock
```
protected void compactPreUnderLock()
```
    subclasses put additional checks before compaction starts here
  - compactPostUnderLock
```
protected void compactPostUnderLock()
```
    subclasses put additional cleanup after compaction finishes here
  - longStackTake
```
protected long longStackTake(long ioList,
                             boolean recursive)
```
  - longStackPut
```
protected void longStackPut(long ioList,
                            long offset,
                            boolean recursive)
```
  - freeIoRecidPut
```
protected void freeIoRecidPut(long ioRecid)
```
  - freeIoRecidTake
```
protected long freeIoRecidTake(boolean ensureAvail)
```
  - size2ListIoRecid
```
protected static long size2ListIoRecid(long size)
```
  - freePhysPut
```
protected void freePhysPut(long indexVal,
                           boolean recursive)
```
  - freePhysTake
```
protected long freePhysTake(int size,
                            boolean ensureAvail,
                            boolean recursive)
```
  - getMaxRecid
```
public long getMaxRecid()
```
    Specified by:
    
    getMaxRecid in class Store
  - getRaw
```
public ByteBuffer getRaw(long recid)
```
    Specified by:
    
    getRaw in class Store
  - getFreeRecids
```
public Iterator<Long> getFreeRecids()
```
    Specified by:
    
    getFreeRecids in class Store
  - updateRaw
```
public void updateRaw(long recid,
                      ByteBuffer data)
```
    Specified by:
    
    updateRaw in class Store
  - getSizeLimit
```
public long getSizeLimit()
```
    Description copied from class: Store
    
    returns maximal store size or `0` if there is no limit
    
    Specified by:
    
    getSizeLimit in class Store
  - getCurrSize
```
public long getCurrSize()
```
    Description copied from class: Store
    
    returns current size occupied by physical store (does not include index). It means file allocated by physical file
    
    Specified by:
    
    getCurrSize in class Store
  - getFreeSize
```
public long getFreeSize()
```
    Description copied from class: Store
    
    returns free size in physical store (does not include index).
    
    Specified by:
    
    getFreeSize in class Store
  - calculateStatistics
```
public String calculateStatistics()
```
    Description copied from class: Store
    
    get some statistics about store. This may require traversing entire store, so it can take some time.
    
    Specified by:
    
    calculateStatistics in class Store
  - countLongStackItems
```
protected long countLongStackItems(long ioList)
```

Class StoreDirect

Field Summary

Fields inherited from class org.mapdb.Store

Fields inherited from interface org.mapdb.Engine

Constructor Summary

Method Summary

Methods inherited from class org.mapdb.Store

Methods inherited from class java.lang.Object

Field Detail

MASK_OFFSET

MASK_LINKED

MASK_DISCARD

MASK_ARCHIVE

HEADER

STORE_VERSION

MAX_REC_SIZE

PHYS_FREE_SLOTS_COUNT

IO_INDEX_SIZE

IO_PHYS_SIZE

IO_FREE_SIZE

IO_INDEX_SUM

IO_FREE_RECID

IO_USER_START

DATA_FILE_EXT

LONG_STACK_PREF_COUNT

LONG_STACK_PREF_SIZE

LONG_STACK_PREF_COUNT_ALTER

LONG_STACK_PREF_SIZE_ALTER

index

phys

physSize

indexSize

freeSize

deleteFilesAfterClose

readOnly

syncOnCommitDisabled

spaceReclaimReuse

spaceReclaimSplit

spaceReclaimTrack

sizeLimit

maxUsedIoList

Constructor Detail

StoreDirect

StoreDirect

Method Detail

checkHeaders

createStructure

indexHeaderChecksum

preallocate

preallocate

put

put2

get

get2

update

update2

compareAndSwap

delete

getLinkedRecordsIndexVals

physAllocate

roundTo16

close

isClosed

commit

rollback

isReadOnly

canRollback

clearCache

compact

compactPreUnderLock

compactPostUnderLock

longStackTake

longStackPut

freeIoRecidPut

freeIoRecidTake

size2ListIoRecid

freePhysPut

freePhysTake

getMaxRecid

getRaw