Tkrzw
|
File database manager implementation based on skip list. More...
#include <tkrzw_dbm_skip.h>
Classes | |
class | Iterator |
Iterator for each record. More... | |
struct | TuningParameters |
Tuning parameters for the database. More... | |
Public Types | |
typedef std::vector< std::string >(* | ReducerType) (const std::string &, const std::vector< std::string > &) |
Interface of a function to reduce records of the same key in the at-random build mode. More... | |
Public Member Functions | |
SkipDBM () | |
Default constructor. More... | |
SkipDBM (std::unique_ptr< File > file) | |
Constructor with a file object. More... | |
virtual | ~SkipDBM () |
Destructor. More... | |
SkipDBM (const SkipDBM &rhs)=delete | |
Copy and assignment are disabled. More... | |
SkipDBM & | operator= (const SkipDBM &rhs)=delete |
Status | Open (const std::string &path, bool writable, int32_t options=File::OPEN_DEFAULT) override |
Opens a database file. More... | |
Status | OpenAdvanced (const std::string &path, bool writable, int32_t options=File::OPEN_DEFAULT, const TuningParameters &tuning_params=TuningParameters()) |
Opens a database file, in an advanced way. More... | |
Status | Close () override |
Closes the database file. More... | |
Status | Process (std::string_view key, RecordProcessor *proc, bool writable) override |
Processes a record with a processor. More... | |
Status | Set (std::string_view key, std::string_view value, bool overwrite=true) override |
Sets a record of a key and a value. More... | |
Status | Remove (std::string_view key) override |
Removes a record of a key. More... | |
Status | GetByIndex (int64_t index, std::string *key=nullptr, std::string *value=nullptr) |
Gets the key and the value of the record of an index. More... | |
Status | ProcessEach (RecordProcessor *proc, bool writable) override |
Processes each and every record in the database with a processor. More... | |
Status | Count (int64_t *count) override |
Gets the number of records. More... | |
Status | GetFileSize (int64_t *size) override |
Gets the current file size of the database. More... | |
Status | GetFilePath (std::string *path) override |
Gets the path of the database file. More... | |
Status | Clear () override |
Removes all records. More... | |
Status | Rebuild () override |
Rebuilds the entire database. More... | |
Status | RebuildAdvanced (const TuningParameters &tuning_params=TuningParameters()) |
Rebuilds the entire database, in an advanced way. More... | |
Status | ShouldBeRebuilt (bool *tobe) override |
Checks whether the database should be rebuilt. More... | |
Status | Synchronize (bool hard, FileProcessor *proc=nullptr) override |
Synchronizes the content of the database to the file system. More... | |
Status | SynchronizeAdvanced (bool hard, FileProcessor *proc=nullptr, ReducerType reducer=nullptr) |
Synchronizes the content of the database to the file system. More... | |
std::vector< std::pair< std::string, std::string > > | Inspect () override |
Inspects the database. More... | |
bool | IsOpen () const override |
Checks whether the database is open. More... | |
bool | IsWritable () const override |
Checks whether the database is writable. More... | |
bool | IsHealthy () const override |
Checks whether the database condition is healthy. More... | |
bool | IsOrdered () const override |
Checks whether ordered operations are supported. More... | |
std::unique_ptr< DBM::Iterator > | MakeIterator () override |
Makes an iterator for each record. More... | |
std::unique_ptr< DBM > | MakeDBM () const override |
Makes a new DBM object of the same concrete class. More... | |
const File * | GetInternalFile () const |
Gets the pointer to the internal file object. More... | |
int64_t | GetEffectiveDataSize () |
Gets the effective data size. More... | |
double | GetModificationTime () |
Gets the last modification time of the database. More... | |
int32_t | GetDatabaseType () |
Gets the database type. More... | |
Status | SetDatabaseType (uint32_t db_type) |
Sets the database type. More... | |
std::string | GetOpaqueMetadata () |
Gets the opaque metadata. More... | |
Status | SetOpaqueMetadata (const std::string &opaque) |
Sets the opaque metadata. More... | |
Status | Revert () |
Reverts all updates. More... | |
bool | IsUpdated () |
Checks whether the database has been updated after being opened. More... | |
Status | MergeSkipDatabase (const std::string &src_path) |
Merges the contents of another skip database file. More... | |
![]() | |
virtual | ~DBM ()=default |
Destructor. More... | |
virtual Status | Get (std::string_view key, std::string *value=nullptr) |
Gets the value of a record of a key. More... | |
virtual std::string | GetSimple (std::string_view key, std::string_view default_value="") |
Gets the value of a record of a key, in a simple way. More... | |
virtual std::map< std::string, std::string > | GetMulti (const std::initializer_list< std::string > &keys) |
Gets the values of multiple records of keys. More... | |
virtual std::map< std::string, std::string > | GetMulti (const std::vector< std::string > &keys) |
Gets the values of multiple records of keys, with a vector. More... | |
virtual Status | SetMulti (const std::initializer_list< std::pair< std::string, std::string >> &records, bool overwrite=true) |
Sets multiple records. More... | |
virtual Status | SetMulti (const std::map< std::string, std::string > &records, bool overwrite=true) |
Sets multiple records, with a map of strings. More... | |
virtual Status | Append (std::string_view key, std::string_view value, std::string_view delim="") |
Appends data at the end of a record of a key. More... | |
virtual Status | CompareExchange (std::string_view key, std::string_view expected, std::string_view desired, std::string *actual=nullptr) |
Compares the value of a record and exchanges if the condition meets. More... | |
virtual Status | Increment (std::string_view key, int64_t increment=1, int64_t *current=nullptr, int64_t initial=0) |
Increments the numeric value of a record. More... | |
int64_t | IncrementSimple (std::string_view key, int64_t increment=1, int64_t initial=0) |
Increments the numeric value of a record, in a simple way. More... | |
virtual int64_t | CountSimple () |
Gets the number of records, in a simple way. More... | |
virtual int64_t | GetFileSizeSimple () |
Gets the current file size of the database, in a simple way. More... | |
virtual std::string | GetFilePathSimple () |
Gets the path of the database file, in a simple way. More... | |
virtual bool | ShouldBeRebuiltSimple () |
Checks whether the database should be rebuilt, in a simple way. More... | |
virtual Status | CopyFile (const std::string &dest_path) |
Copies the content of the database file to another file. More... | |
virtual Status | Export (DBM *dbm) |
Exports all records to another database. More... | |
Static Public Member Functions | |
static std::vector< std::string > | ReduceRemove (const std::string &key, const std::vector< std::string > &values) |
Reduces the values of records of the same key by removing REMOVING_VALUE and past values. More... | |
static std::vector< std::string > | ReduceToFirst (const std::string &key, const std::vector< std::string > &values) |
Reduces the values of records of the same key by keeping the first value only. More... | |
static std::vector< std::string > | ReduceToSecond (const std::string &key, const std::vector< std::string > &values) |
Reduces the values of records of the same key by keeping the second or first value only. More... | |
static std::vector< std::string > | ReduceToLast (const std::string &key, const std::vector< std::string > &values) |
Reduces the values of records of the same key by keeping the last value only. More... | |
static std::vector< std::string > | ReduceConcat (const std::string &key, const std::vector< std::string > &values) |
Reduces the values of records of the same key by concatenating all values. More... | |
static std::vector< std::string > | ReduceConcatWithNull (const std::string &key, const std::vector< std::string > &values) |
Reduces the values of records of the same key by concatenating all values with null code. More... | |
static std::vector< std::string > | ReduceConcatWithTab (const std::string &key, const std::vector< std::string > &values) |
Reduces the values of records of the same key by concatenating all values with tab. More... | |
static std::vector< std::string > | ReduceConcatWithLine (const std::string &key, const std::vector< std::string > &values) |
Reduces the values of records of the same key by concatenating all values with linefeed. More... | |
static std::vector< std::string > | ReduceToTotal (const std::string &key, const std::vector< std::string > &values) |
Reduces the values of records of the same key by totaling numeric expressions. More... | |
static Status | RestoreDatabase (const std::string &old_file_path, const std::string &new_file_path) |
Restores a broken database as a new healthy database. More... | |
Static Public Attributes | |
static constexpr int32_t | DEFAULT_OFFSET_WIDTH = 4 |
The default value of the offset width. More... | |
static constexpr int32_t | DEFAULT_STEP_UNIT = 4 |
The default value of the step unit. More... | |
static constexpr int32_t | DEFAULT_MAX_LEVEL = 14 |
The default value of the maximum level. More... | |
static constexpr int64_t | DEFAULT_SORT_MEM_SIZE = 256LL << 20 |
The default value of the memory size used for sorting. More... | |
static constexpr int32_t | DEFAULT_MAX_CACHED_RECORDS = 65536 |
The default value of the maximum cached records. More... | |
static constexpr int32_t | OPAQUE_METADATA_SIZE = 64 |
The size of the opaque metadata. More... | |
static const std::string | REMOVING_VALUE |
The special removing value. More... | |
File database manager implementation based on skip list.
All operations are thread-safe; Multiple threads can access the same database concurrently. Every opened database must be closed explicitly to avoid data corruption.
typedef std::vector<std::string>(* tkrzw::SkipDBM::ReducerType) (const std::string &, const std::vector< std::string > &) |
Interface of a function to reduce records of the same key in the at-random build mode.
The first parameter is the key. The second parameter is a vecgtor of record values. The return value is a vector of new values which are stored in the database.
tkrzw::SkipDBM::SkipDBM | ( | ) |
Default constructor.
MemoryMapParallelFile is used to handle the data.
tkrzw::SkipDBM::SkipDBM | ( | std::unique_ptr< File > | file | ) |
Constructor with a file object.
file | The file object to handle the data. The ownership is taken. |
|
virtual |
Destructor.
|
explicitdelete |
Copy and assignment are disabled.
|
overridevirtual |
Opens a database file.
path | A path of the file. |
writable | If true, the file is writable. If false, it is read-only. |
options | Bit-sum options for opening the file. |
Precondition: The database is not opened.
Implements tkrzw::DBM.
Status tkrzw::SkipDBM::OpenAdvanced | ( | const std::string & | path, |
bool | writable, | ||
int32_t | options = File::OPEN_DEFAULT , |
||
const TuningParameters & | tuning_params = TuningParameters() |
||
) |
Opens a database file, in an advanced way.
path | A path of the file. |
writable | If true, the file is writable. If false, it is read-only. |
options | Bit-sum options for opening the file. |
tuning_params | A structure for tuning parameters. |
Precondition: The database is not opened.
|
overridevirtual |
Closes the database file.
Precondition: The database is opened.
Implements tkrzw::DBM.
|
overridevirtual |
Processes a record with a processor.
key | The key of the record. |
proc | The pointer to the processor object. |
writable | True if the processor can edit the record. |
Precondition: The database is opened. The writable parameter should be consistent to the open mode.
If the specified record exists, the ProcessFull of the processor is called. Otherwise, the ProcessEmpty of the processor is called. Inserted records are not visible until the database is synchronized. The database allows duplication of keys. Duplication can be solved by the reducer applied when the database is synchronized. Removing records is implemented as inserting a special value REMOVING_VALUE, which is handled when the database is synchronized.
Implements tkrzw::DBM.
|
overridevirtual |
Sets a record of a key and a value.
key | The key of the record. |
value | The value of the record. |
overwrite | Whether to overwrite the existing value if there's a record with the same key. If true, the existing value is ovewritten by the new value. If false, the operation is given up and an error status is returned. |
Precondition: The database is opened as writable.
Reimplemented from tkrzw::DBM.
|
overridevirtual |
Removes a record of a key.
key | The key of the record. |
Even if there's no matching record, this doesn't report failure.
Precondition: The database is opened as writable.
Reimplemented from tkrzw::DBM.
Status tkrzw::SkipDBM::GetByIndex | ( | int64_t | index, |
std::string * | key = nullptr , |
||
std::string * | value = nullptr |
||
) |
Gets the key and the value of the record of an index.
index | The index of the record to retrieve. |
key | The pointer to a string object to contain the record key. If it is nullptr, the key data is ignored. |
value | The pointer to a string object to contain the record value. If it is nullptr, the value data is ignored. |
|
overridevirtual |
Processes each and every record in the database with a processor.
proc | The pointer to the processor object. |
writable | True if the processor can edit the record. |
Precondition: The database is opened. The writable parameter should be consistent to the open mode.
The ProcessFull of the processor is called repeatedly for each record. The ProcessEmpty of the processor is called once before the iteration and once after the iteration.
Implements tkrzw::DBM.
|
overridevirtual |
Gets the number of records.
count | The pointer to an integer object to contain the result count. |
Precondition: The database is opened.
Implements tkrzw::DBM.
|
overridevirtual |
Gets the current file size of the database.
size | The pointer to an integer object to contain the result size. |
Precondition: The database is opened.
Implements tkrzw::DBM.
|
overridevirtual |
Gets the path of the database file.
path | The pointer to a string object to contain the result path. |
Precondition: The database is opened.
Implements tkrzw::DBM.
|
overridevirtual |
Removes all records.
Precondition: The database is opened as writable.
Implements tkrzw::DBM.
|
overridevirtual |
Rebuilds the entire database.
Precondition: The database is opened as writable.
Rebuilding a database is useful to optimize the size of the file. All tuning parameters are succeeded or calculated implicitly.
Implements tkrzw::DBM.
Status tkrzw::SkipDBM::RebuildAdvanced | ( | const TuningParameters & | tuning_params = TuningParameters() | ) |
Rebuilds the entire database, in an advanced way.
tuning_params | A structure for tuning parameters. The default value of each parameter means that the current setting is succeeded or calculated implicitly. |
Precondition: The database is opened as writable.
Rebuilding a database is useful to optimize the size of the file.
|
overridevirtual |
Checks whether the database should be rebuilt.
tobe | The pointer to a boolean object to contain the result decision. |
Precondition: The database is opened.
Implements tkrzw::DBM.
|
overridevirtual |
Synchronizes the content of the database to the file system.
hard | True to do physical synchronization with the hardware or false to do only logical synchronization with the file system. |
proc | The pointer to the file processor object, whose Process method is called while the content of the file is synchronized. If it is nullptr, it is ignored. |
Precondition: The database is opened as writable.
All inserted records are written in the database file and they become visible. This operation rebuilds the entire database by merging the existing records and inserted records. No reducer is applied to records so that all records with duplicated keys are kept intact.
Implements tkrzw::DBM.
Status tkrzw::SkipDBM::SynchronizeAdvanced | ( | bool | hard, |
FileProcessor * | proc = nullptr , |
||
ReducerType | reducer = nullptr |
||
) |
Synchronizes the content of the database to the file system.
hard | True to do physical synchronization with the hardware or false to do only logical synchronization with the file system. |
proc | The pointer to the file processor object, whose Process method is called while the content of the file is synchronized. If it is nullptr, it is ignored. |
reducer | A reducer applied before stroing records. If it is nullptr, no reducer is applied and all records are kept intact. |
Precondition: The database is opened as writable.
All inserted records are written in the database file and they become visible. This operation rebuilds the entire database by merging the existing records and inserted records. Records of the same key given to the reducer are ordered in a way that the old value comes earlier than the new value. Thus, if you want to keep the old value, ReduceToFirst should be set. If you want to replace the old value with the new value, ReduceToLast should be set.
|
overridevirtual |
Inspects the database.
Implements tkrzw::DBM.
|
overridevirtual |
Checks whether the database is open.
Implements tkrzw::DBM.
|
overridevirtual |
Checks whether the database is writable.
Implements tkrzw::DBM.
|
overridevirtual |
Checks whether the database condition is healthy.
Precondition: The database is opened.
Implements tkrzw::DBM.
|
overridevirtual |
Checks whether ordered operations are supported.
Implements tkrzw::DBM.
|
overridevirtual |
Makes an iterator for each record.
Precondition: The database is opened.
Implements tkrzw::DBM.
|
overridevirtual |
Makes a new DBM object of the same concrete class.
Implements tkrzw::DBM.
const File* tkrzw::SkipDBM::GetInternalFile | ( | ) | const |
Gets the pointer to the internal file object.
Accessing the internal file viorates encapsulation policy. This should be used only for testing and debugging.
int64_t tkrzw::SkipDBM::GetEffectiveDataSize | ( | ) |
Gets the effective data size.
Precondition: The database is opened.
The effective data size means the total size of the keys and the values.
double tkrzw::SkipDBM::GetModificationTime | ( | ) |
Gets the last modification time of the database.
Precondition: The database is opened.
int32_t tkrzw::SkipDBM::GetDatabaseType | ( | ) |
Gets the database type.
Precondition: The database is opened.
Status tkrzw::SkipDBM::SetDatabaseType | ( | uint32_t | db_type | ) |
Sets the database type.
db_type | The database type. |
Precondition: The database is opened as writable.
This data is just for applications and not used by the database implementation.
std::string tkrzw::SkipDBM::GetOpaqueMetadata | ( | ) |
Gets the opaque metadata.
Precondition: The database is opened.
Status tkrzw::SkipDBM::SetOpaqueMetadata | ( | const std::string & | opaque | ) |
Sets the opaque metadata.
opaque | The opaque metadata, of which leading 64 bytes are stored in the file. |
Precondition: The database is opened as writable.
This data is just for applications and not used by the database implementation.
Status tkrzw::SkipDBM::Revert | ( | ) |
Reverts all updates.
Precondition: The database is opened as writable and not finished.
All inserted records are discarded and the contents are returned to the state when the database was synchronized at the last time.
bool tkrzw::SkipDBM::IsUpdated | ( | ) |
Checks whether the database has been updated after being opened.
Precondition: The database is opened.
Status tkrzw::SkipDBM::MergeSkipDatabase | ( | const std::string & | src_path | ) |
Merges the contents of another skip database file.
src_path | A path to the source database file. |
Precondition: The database is opened as writable.
Merged records are shown after the database is synchronized. Among records of the same key, synchronized existing records come first, added records come second, and unsynchronized existing records come last.
|
static |
Reduces the values of records of the same key by removing REMOVING_VALUE and past values.
key | The common key of the records. |
values | The values of the records of the same key. |
|
static |
Reduces the values of records of the same key by keeping the first value only.
key | The common key of the records. |
values | The values of the records of the same key. |
|
static |
Reduces the values of records of the same key by keeping the second or first value only.
key | The common key of the records. |
values | The values of the records of the same key. |
|
static |
Reduces the values of records of the same key by keeping the last value only.
key | The common key of the records. |
values | The values of the records of the same key. |
|
static |
Reduces the values of records of the same key by concatenating all values.
key | The common key of the records. |
values | The values of the records of the same key. |
|
static |
Reduces the values of records of the same key by concatenating all values with null code.
key | The common key of the records. |
values | The values of the records of the same key. |
|
static |
Reduces the values of records of the same key by concatenating all values with tab.
key | The common key of the records. |
values | The values of the records of the same key. |
|
static |
Reduces the values of records of the same key by concatenating all values with linefeed.
key | The common key of the records. |
values | The values of the records of the same key. |
|
static |
Reduces the values of records of the same key by totaling numeric expressions.
key | The common key of the records. |
values | The values of the records of the same key. |
|
static |
Restores a broken database as a new healthy database.
old_file_path | The path of the broken database. |
new_file_path | The path of the new database to be created. |
|
staticconstexpr |
The default value of the offset width.
|
staticconstexpr |
The default value of the step unit.
|
staticconstexpr |
The default value of the maximum level.
|
staticconstexpr |
The default value of the memory size used for sorting.
|
staticconstexpr |
The default value of the maximum cached records.
|
staticconstexpr |
The size of the opaque metadata.
|
static |
The special removing value.