Buffer objects that need to be written out to disk so as to optimize writing operations. More...

#include <DiskBuffer.h>

Inheritance diagram for Mantid::Kernel::DiskBuffer:

Public Types
using	freeSpace_bySize_t = freeSpace_t::nth_index< 1 >::type
	A way to index the free space by their size. More...

using	freeSpace_t = boost::multi_index::multi_index_container< FreeBlock, boost::multi_index::indexed_by< boost::multi_index::ordered_non_unique< ::boost::multi_index::const_mem_fun< FreeBlock, uint64_t, &FreeBlock::getFilePosition > >, boost::multi_index::ordered_non_unique< ::boost::multi_index::const_mem_fun< FreeBlock, uint64_t, &FreeBlock::getSize > > > >
	A map for the list of free space blocks in the file. More...

Public Member Functions
uint64_t	allocate (uint64_t const newSize)
	Allocate a block of the given size in a free spot in the file, or at the end of the file if there is no space. More...

void	defragFreeBlocks ()
	Method that defrags free blocks by combining adjacent ones together NOTE: This is not necessary to run since the freeBlock() methods automatically defrags neighboring blocks. More...

	DiskBuffer ()
	Constructor. More...

	DiskBuffer (const DiskBuffer &)=delete

	DiskBuffer (uint64_t m_writeBufferSize)
	Constructor. More...

void	flushCache ()
	Flush out all the data in the memory; and writes out everything in the to-write cache. More...

void	freeBlock (uint64_t const pos, uint64_t const size)
	This method is called by this->relocate when object that has shrunk and so has left a bit of free space after itself on the file; or when an object gets moved to a new spot. More...

uint64_t	getFileLength () const

freeSpace_t &	getFreeSpaceMap ()

void	getFreeSpaceVector (std::vector< uint64_t > &free) const
	Returns a vector with two entries per free block: position and size. More...

std::string	getMemoryStr () const

uint64_t	getWriteBufferSize () const

uint64_t	getWriteBufferUsed () const

void	objectDeleted (ISaveable *item)
	Call this method when an object that might be in the cache is getting deleted. More...

DiskBuffer &	operator= (const DiskBuffer &)=delete

uint64_t	relocate (uint64_t const oldPos, uint64_t const oldSize, const uint64_t newSize)
	This method is called by an ISaveable object that has outgrown its space allocated on file and needs to relocate. More...

void	setFileLength (const uint64_t length) const
	Set the length of the file that this MRU writes to. More...

void	setFreeSpaceVector (std::vector< uint64_t > &free)
	Sets the free space map. More...

void	setWriteBufferSize (uint64_t buffer)
	Set the size of the to-write buffer, in number of events. More...

void	toWrite (ISaveable *item)
	Call this method when an object is ready to be written out to disk. More...

virtual	~DiskBuffer ()=default

Protected Member Functions
void	writeOldObjects ()
	Method to write out the old objects that have been stored in the "toWrite" buffer. More...

Protected Attributes
uint64_t	m_fileLength
	Length of the file. This is where new blocks that don't fit get placed. More...

freeSpace_t	m_free
	Map of the free blocks in the file. More...

freeSpace_bySize_t &	m_free_bySize
	Index into m_free, but indexed by block size. More...

std::mutex	m_freeMutex
	Mutex for modifying the free space list. More...

std::mutex	m_mutex
	Mutex for modifying the the toWrite buffer. More...

size_t	m_nObjectsToWrite
	number of objects stored in to write buffer list More...

std::list< ISaveable * >	m_toWriteBuffer
	A forward list for the buffer of "toWrite" objects. More...

size_t	m_writeBufferSize
	Do we use the write buffer? Always now. More...

size_t	m_writeBufferUsed
	Total amount of memory in the "toWrite" buffer. More...

Detailed Description

Buffer objects that need to be written out to disk so as to optimize writing operations.

This will be used by file-backed MDEventWorkspaces to store boxes (lists of events) before writing them out.

It also stores a list of "free" blocks in the output file, to allow new blocks to fill them later.

Date: 2011-12-30

Definition at line 42 of file DiskBuffer.h.

Member Typedef Documentation

◆ freeSpace_bySize_t

using Mantid::Kernel::DiskBuffer::freeSpace_bySize_t = freeSpace_t::nth_index<1>::type

A way to index the free space by their size.

Definition at line 56 of file DiskBuffer.h.

◆ freeSpace_t

using Mantid::Kernel::DiskBuffer::freeSpace_t = boost::multi_index::multi_index_container< FreeBlock, boost::multi_index::indexed_by< boost::multi_index::ordered_non_unique< ::boost::multi_index::const_mem_fun<FreeBlock, uint64_t, &FreeBlock::getFilePosition> >, boost::multi_index::ordered_non_unique< ::boost::multi_index::const_mem_fun<FreeBlock, uint64_t, &FreeBlock::getSize> >> >

A map for the list of free space blocks in the file.

Index 1: Position in the file. Index 2: Size of the free block

Definition at line 48 of file DiskBuffer.h.

Constructor & Destructor Documentation

◆ DiskBuffer() [1/3]

Mantid::Kernel::DiskBuffer::DiskBuffer ( )

Constructor.

Definition at line 19 of file DiskBuffer.cpp.

References m_free.

◆ DiskBuffer() [2/3]

Mantid::Kernel::DiskBuffer::DiskBuffer ( uint64_t m_writeBufferSize )

Constructor.

Parameters

m_writeBufferSize :: Amount of memory to accumulate in the write buffer before writing.

Returns

Definition at line 32 of file DiskBuffer.cpp.

References m_free.

◆ DiskBuffer() [3/3]

Mantid::Kernel::DiskBuffer::DiskBuffer ( const DiskBuffer & )

delete

◆ ~DiskBuffer()

virtual Mantid::Kernel::DiskBuffer::~DiskBuffer ( )

virtualdefault

Member Function Documentation

◆ allocate()

uint64_t Mantid::Kernel::DiskBuffer::allocate ( uint64_t const newSize )

Allocate a block of the given size in a free spot in the file, or at the end of the file if there is no space.

Parameters

newSize :: new size of the data

Returns: a new position at which the data can be saved.

Definition at line 294 of file DiskBuffer.cpp.

References freeBlock(), m_fileLength, m_free, m_free_bySize, and m_freeMutex.

Referenced by relocate(), and writeOldObjects().

◆ defragFreeBlocks()

void Mantid::Kernel::DiskBuffer::defragFreeBlocks ( )

Method that defrags free blocks by combining adjacent ones together NOTE: This is not necessary to run since the freeBlock() methods automatically defrags neighboring blocks.

Definition at line 260 of file DiskBuffer.cpp.

References m_free, m_freeMutex, and Mantid::Kernel::FreeBlock::merge().

◆ flushCache()

void Mantid::Kernel::DiskBuffer::flushCache ( )

Flush out all the data in the memory; and writes out everything in the to-write cache.

Definition at line 192 of file DiskBuffer.cpp.

References writeOldObjects().

Referenced by Mantid::DataObjects::BoxControllerNeXusIO::closeFile(), Mantid::MDAlgorithms::MergeMDFiles::doExecByCloning(), and Mantid::MDAlgorithms::LoadSQW::readEvents().

◆ freeBlock()

void Mantid::Kernel::DiskBuffer::freeBlock	(	uint64_t const	pos,
		uint64_t const	size
	)

This method is called by this->relocate when object that has shrunk and so has left a bit of free space after itself on the file; or when an object gets moved to a new spot.

Parameters

pos	:: position in the file of the START of the new free block
size	:: size of the free block

Definition at line 205 of file DiskBuffer.cpp.

References m_free, m_freeMutex, and Mantid::Kernel::FreeBlock::merge().

Referenced by allocate(), objectDeleted(), and relocate().

◆ getFileLength()

uint64_t Mantid::Kernel::DiskBuffer::getFileLength ( ) const

inline

Returns: the position of the last allocated point in the file (for testing only!)

Definition at line 106 of file DiskBuffer.h.

Referenced by MantidTestHelpers::BoxControllerDummyIO::loadBlock(), Mantid::DataObjects::BoxControllerNeXusIO::loadGenericBlock(), and Mantid::DataObjects::BoxControllerNeXusIO::saveGenericBlock().

◆ getFreeSpaceMap()

freeSpace_t & Mantid::Kernel::DiskBuffer::getFreeSpaceMap ( )

inline

Returns: reference to the free space map (for testing only!)

Definition at line 101 of file DiskBuffer.h.

◆ getFreeSpaceVector()

void Mantid::Kernel::DiskBuffer::getFreeSpaceVector ( std::vector< uint64_t > & free ) const

Returns a vector with two entries per free block: position and size.

Parameters

[out] free :: vector to fill

Definition at line 352 of file DiskBuffer.cpp.

References m_free, and m_free_bySize.

Referenced by Mantid::DataObjects::BoxControllerNeXusIO::closeFile(), and Mantid::DataObjects::BoxControllerNeXusIO::getDiskBufferFileData().

◆ getMemoryStr()

std::string Mantid::Kernel::DiskBuffer::getMemoryStr ( ) const

Returns: a string describing the memory buffers, for debugging.

Definition at line 383 of file DiskBuffer.cpp.

References m_nObjectsToWrite, and m_writeBufferUsed.

◆ getWriteBufferSize()

uint64_t Mantid::Kernel::DiskBuffer::getWriteBufferSize ( ) const

inline

Returns: the size of the to-write buffer, in number of events

Definition at line 94 of file DiskBuffer.h.

◆ getWriteBufferUsed()

uint64_t Mantid::Kernel::DiskBuffer::getWriteBufferUsed ( ) const

inline

Returns: the memory used in the "toWrite" buffer, in number of events

Definition at line 97 of file DiskBuffer.h.

◆ objectDeleted()

void Mantid::Kernel::DiskBuffer::objectDeleted ( ISaveable * item )

Call this method when an object that might be in the cache is getting deleted.

The object is removed from the to-write buffer (if present). The space it uses on disk is marked as free.

Parameters

item	:: ISaveable object that is getting deleted.

Definition at line 83 of file DiskBuffer.cpp.

References Mantid::Kernel::ISaveable::clearBufferState(), freeBlock(), Mantid::Kernel::ISaveable::getBufferSize(), Mantid::Kernel::ISaveable::getBufPostion(), Mantid::Kernel::ISaveable::getFilePosition(), Mantid::Kernel::ISaveable::getFileSize(), m_mutex, m_toWriteBuffer, m_writeBufferUsed, and Mantid::Kernel::ISaveable::wasSaved().

◆ operator=()

DiskBuffer & Mantid::Kernel::DiskBuffer::operator= ( const DiskBuffer & )

delete

◆ relocate()

uint64_t Mantid::Kernel::DiskBuffer::relocate	(	uint64_t const	oldPos,
		uint64_t const	oldSize,
		const uint64_t	newSize
	)

This method is called by an ISaveable object that has outgrown its space allocated on file and needs to relocate.

This should only be called when the MRU is saving a block to disk, i.e. the ISaveable cannot be in either the MRU buffer or the toWrite buffer.

Parameters

oldPos	:: original position in the file
oldSize	:: original size in the file. This will be marked as "free"
newSize	:: new size of the data

Returns: a new position at which the data can be saved.

Definition at line 341 of file DiskBuffer.cpp.

References allocate(), and freeBlock().

Referenced by writeOldObjects().

◆ setFileLength()

void Mantid::Kernel::DiskBuffer::setFileLength ( const uint64_t length ) const

inline

Set the length of the file that this MRU writes to.

Parameters

length :: length in the same units as the cache, etc. (not necessarily bytes)

Definition at line 111 of file DiskBuffer.h.

Referenced by MantidTestHelpers::BoxControllerDummyIO::openFile(), Mantid::DataObjects::BoxControllerNeXusIO::prepareNxSdata_CurVersion(), Mantid::DataObjects::BoxControllerNeXusIO::prepareNxSToWrite_CurVersion(), MantidTestHelpers::BoxControllerDummyIO::saveBlock(), and Mantid::DataObjects::BoxControllerNeXusIO::saveGenericBlock().

◆ setFreeSpaceVector()

void Mantid::Kernel::DiskBuffer::setFreeSpaceVector ( std::vector< uint64_t > & free )

Sets the free space map.

Should only be used when loading a file.

Parameters

[in] free :: vector containing free space index to set

Definition at line 364 of file DiskBuffer.cpp.

References m_free.

Referenced by Mantid::DataObjects::BoxControllerNeXusIO::getDiskBufferFileData().

◆ setWriteBufferSize()

void Mantid::Kernel::DiskBuffer::setWriteBufferSize ( uint64_t buffer )

inline

Set the size of the to-write buffer, in number of events.

Parameters

buffer :: number of events to accumulate before writing. 0 to NOT use the write buffer

Definition at line 85 of file DiskBuffer.h.

◆ toWrite()

void Mantid::Kernel::DiskBuffer::toWrite ( ISaveable * item )

Call this method when an object is ready to be written out to disk.

When the to-write buffer is full, all of it gets written out to disk using writeOldObjects()

Parameters

item	:: item that can be written to disk.

Definition at line 47 of file DiskBuffer.cpp.

References Mantid::Kernel::ISaveable::getBufferSize(), Mantid::Kernel::ISaveable::getBufPostion(), Mantid::Kernel::ISaveable::getDataMemorySize(), m_mutex, m_nObjectsToWrite, m_toWriteBuffer, m_writeBufferSize, m_writeBufferUsed, Mantid::Kernel::ISaveable::setBufferPosition(), Mantid::Kernel::ISaveable::setBufferSize(), and writeOldObjects().

Referenced by Mantid::MDAlgorithms::DivideMD::execEventScalar(), Mantid::MDAlgorithms::MultiplyMD::execEventScalar(), and Mantid::MDAlgorithms::FlippingRatioCorrectionMD::executeTemplatedMDE().

◆ writeOldObjects()

void Mantid::Kernel::DiskBuffer::writeOldObjects ( )

inlineprotected

Method to write out the old objects that have been stored in the "toWrite" buffer.

Definition at line 109 of file DiskBuffer.cpp.

References allocate(), Mantid::Kernel::ISaveable::getFilePosition(), m_fileLength, m_mutex, m_nObjectsToWrite, m_toWriteBuffer, m_writeBufferUsed, obj, and relocate().

Referenced by flushCache(), and toWrite().

Member Data Documentation

◆ m_fileLength

uint64_t Mantid::Kernel::DiskBuffer::m_fileLength

mutableprotected

Length of the file. This is where new blocks that don't fit get placed.

Definition at line 149 of file DiskBuffer.h.

Referenced by allocate(), and writeOldObjects().

◆ m_free

freeSpace_t Mantid::Kernel::DiskBuffer::m_free

protected

Map of the free blocks in the file.

Definition at line 139 of file DiskBuffer.h.

Referenced by allocate(), defragFreeBlocks(), DiskBuffer(), freeBlock(), getFreeSpaceVector(), and setFreeSpaceVector().

◆ m_free_bySize

freeSpace_bySize_t& Mantid::Kernel::DiskBuffer::m_free_bySize

protected

Index into m_free, but indexed by block size.

Definition at line 142 of file DiskBuffer.h.

Referenced by allocate(), and getFreeSpaceVector().

◆ m_freeMutex

std::mutex Mantid::Kernel::DiskBuffer::m_freeMutex

protected

Mutex for modifying the free space list.

Definition at line 145 of file DiskBuffer.h.

Referenced by allocate(), defragFreeBlocks(), and freeBlock().

◆ m_mutex

std::mutex Mantid::Kernel::DiskBuffer::m_mutex

protected

Mutex for modifying the the toWrite buffer.

Definition at line 134 of file DiskBuffer.h.

Referenced by objectDeleted(), toWrite(), and writeOldObjects().

◆ m_nObjectsToWrite

size_t Mantid::Kernel::DiskBuffer::m_nObjectsToWrite

protected

number of objects stored in to write buffer list

Definition at line 129 of file DiskBuffer.h.

Referenced by getMemoryStr(), toWrite(), and writeOldObjects().

◆ m_toWriteBuffer

std::list<ISaveable *> Mantid::Kernel::DiskBuffer::m_toWriteBuffer

protected

A forward list for the buffer of "toWrite" objects.

Definition at line 131 of file DiskBuffer.h.

Referenced by objectDeleted(), toWrite(), and writeOldObjects().

◆ m_writeBufferSize

size_t Mantid::Kernel::DiskBuffer::m_writeBufferSize

protected

Do we use the write buffer? Always now.

Amount of memory to accumulate in the write buffer before writing.

Definition at line 124 of file DiskBuffer.h.

Referenced by toWrite().

◆ m_writeBufferUsed

size_t Mantid::Kernel::DiskBuffer::m_writeBufferUsed

protected

Total amount of memory in the "toWrite" buffer.

Definition at line 127 of file DiskBuffer.h.

Referenced by getMemoryStr(), objectDeleted(), toWrite(), and writeOldObjects().

The documentation for this class was generated from the following files:

Framework/Kernel/inc/MantidKernel/DiskBuffer.h
Framework/Kernel/src/DiskBuffer.cpp

Public Types

Public Member Functions

Protected Member Functions

Protected Attributes

Detailed Description

Member Typedef Documentation

◆ freeSpace_bySize_t

◆ freeSpace_t

Constructor & Destructor Documentation

◆ DiskBuffer() [1/3]

◆ DiskBuffer() [2/3]

◆ DiskBuffer() [3/3]

◆ ~DiskBuffer()

Member Function Documentation

◆ allocate()

◆ defragFreeBlocks()

◆ flushCache()

◆ freeBlock()

◆ getFileLength()

◆ getFreeSpaceMap()

◆ getFreeSpaceVector()

◆ getMemoryStr()

◆ getWriteBufferSize()

◆ getWriteBufferUsed()

◆ objectDeleted()

◆ operator=()

◆ relocate()

◆ setFileLength()

◆ setFreeSpaceVector()

◆ setWriteBufferSize()

◆ toWrite()

◆ writeOldObjects()

Member Data Documentation

◆ m_fileLength

◆ m_free

◆ m_free_bySize

◆ m_freeMutex

◆ m_mutex

◆ m_nObjectsToWrite

◆ m_toWriteBuffer

◆ m_writeBufferSize

◆ m_writeBufferUsed