Distributed array located primarily on disk. More...

#include <DistrArrayDisk.h>

Inheritance diagram for molpro::linalg::array::DistrArrayDisk:

[legend]

Collaboration diagram for molpro::linalg::array::DistrArrayDisk:

[legend]

Detailed Description

Distributed array located primarily on disk.

This class stores the full array on disk and implements RMA and more efficient linear algebra operations.

RMA operations read/write directly to disk. Care must be taken that they do not overlap with local buffer modifications.

LocalBuffer reads the whole local section into memory. This might be prohivitively expensive for large arrays, so care must be taken.

BufferManager reads the local section in chunks using a separate thread for I/O. This is more memory efficient and allows overlap of communication and computation. The walk through the local section is via iterators.

IO can be done in a separate thread using util::Task.

#include <molpro/linalg/array/util.h>
auto da = DistrArrayDisk{...};
da.put(lo,hi, data); // Normal put same as in the base class is guaranteed to finish
auto t = util::Task::create([&](){da.put(lo, hi, data)}); // does I/O in a new thread
// do something time consuming
t.wait(); // wait for the thread to finish the I/O

Classes
class	LocalBufferDisk

Public Types
using	disk_array = void
	a compile time tag that this is a distributed disk array More...

Public Types inherited from molpro::linalg::array::DistrArray
using	distributed_array = void
	a compile time tag that this is a distributed array More...

using	value_type = double

using	index_type = size_t

using	SparseArray = std::map< index_type, double >

using	Distribution = util::Distribution< index_type >

Public Member Functions
virtual void	erase ()=0
	Erase the array from disk. More...

const Distribution &	distribution () const override
	Access distribution of the array among processes. More...

value_type	dot (const DistrArrayDisk &y) const

value_type	dot (const DistrArray &y) const override
	Scalar product of two arrays. Collective. Throws error if any array is empty. Both arrays should be part of the same processor group (same communicator). The result is broadcast to each process. More...

value_type	dot (const SparseArray &y) const override

void	set_buffer_size (size_t buffer_size)

void	copy (const DistrArray &y) override

std::unique_ptr< LocalBuffer >	local_buffer () override
	Access the buffer local to this process. More...

std::unique_ptr< const LocalBuffer >	local_buffer () const override

std::unique_ptr< LocalBuffer >	local_buffer (const span::Span< value_type > &buffer)
	Access local section, reading it into the provided buffer. More...

std::unique_ptr< const LocalBuffer >	local_buffer (const span::Span< value_type > &buffer) const
	Read-only access to the local section, reading it into the provided buffer. More...

Public Member Functions inherited from molpro::linalg::array::DistrArray
virtual	~DistrArray ()=default

MPI_Comm	communicator () const
	return a copy of the communicator More...

virtual void	sync () const
	Synchronizes all process in this group and ensures any outstanding operations on the array have completed. More...

size_t	size () const
	total number of elements, same as overall dimension of array More...

bool	compatible (const DistrArray &other) const
	Checks that arrays are of the same dimensionality. More...

virtual void	zero ()
	Set all local elements to zero. More...

virtual void	error (const std::string &message) const
	stops application with an error More...

value_type	operator[] (size_t index)

virtual value_type	at (index_type ind) const =0

virtual void	set (index_type ind, value_type val)=0
	Set one element to a scalar. Global operation. More...

virtual void	get (index_type lo, index_type hi, value_type *buf) const =0
	Gets buffer[lo:hi) from global array (hi is past-the-end). Blocking. More...

virtual std::vector< value_type >	get (index_type lo, index_type hi) const =0

virtual void	put (index_type lo, index_type hi, const value_type *data)=0
	array[lo:hi) = data[:] (hi is past-the-end). Blocking More...

virtual void	acc (index_type lo, index_type hi, const value_type *data)=0
	array[lo:hi) += scaling_constant * data[:] (hi is past-the-end). Blocking More...

virtual std::vector< value_type >	gather (const std::vector< index_type > &indices) const =0
	gets elements with discontinuous indices from array. Blocking More...

virtual void	scatter (const std::vector< index_type > &indices, const std::vector< value_type > &data)=0
	array[indices[i]] = data[i] Puts vals of elements with discontinuous indices of array. Blocking. More...

virtual void	scatter_acc (std::vector< index_type > &indices, const std::vector< value_type > &data)=0
	array[indices[i]] += vals[i] Accumulates vals of elements into discontinuous indices of array. Atomic, blocking, with on-sided communication More...

virtual std::vector< value_type >	vec () const =0
	Copies the whole buffer into a vector. Blocking. More...

virtual void	fill (value_type val)

virtual void	copy_patch (const DistrArray &y, index_type start, index_type end)
	Copies elements in a patch of y. If both arrays are empty than does nothing. If only one is empty, throws an error. More...

virtual void	axpy (value_type a, const DistrArray &y)
	this[:] += a * y[:]. Throws an error if any array is empty. Add a multiple of another array to this one. Blocking, collective. More...

virtual void	axpy (value_type a, const SparseArray &y)

virtual void	scal (value_type a)
	Scale by a constant. Local. More...

virtual void	add (const DistrArray &y)
	Add another array to this. Local. Throws error if any array is empty. More...

virtual void	add (value_type a)
	Add a constant. Local. More...

virtual void	sub (const DistrArray &y)
	Subtract another array from this. Local. Throws error if any array is empty. More...

virtual void	sub (value_type a)
	Subtract a constant. Local. More...

virtual void	recip ()
	Take element-wise reciprocal of this. Local. No checks are made for zero values. More...

virtual void	times (const DistrArray &y)
	this[i] *= y[i]. Throws error if any array is empty. More...

virtual void	times (const DistrArray &y, const DistrArray &z)
	this[i] = y[i]*z[i]. Throws error if any array is empty. More...

void	divide (const DistrArray &y, const DistrArray &z, value_type shift=0, bool append=false, bool negative=false)
	this[i] = y[i]/(z[i]+shift). Collective. Throws error if any array is empty. More...

std::list< std::pair< index_type, value_type > >	min_n (int n) const
	returns n smallest elements in array x Collective operation, must be called by all processes in the group. More...

std::list< std::pair< index_type, value_type > >	max_n (int n) const
	returns n largest elements in array x Collective operation, must be called by all processes in the group. More...

std::list< std::pair< index_type, value_type > >	min_abs_n (int n) const
	returns n elements that are largest by absolute value in array x Collective operation, must be called by all processes in the group. More...

std::list< std::pair< index_type, value_type > >	max_abs_n (int n) const
	returns n elements that are largest by absolute value in array x Collective operation, must be called by all processes in the group. More...

std::vector< index_type >	min_loc_n (int n) const
	find the index of n smallest components in array x Collective operation, must be called by all processes in the group. More...

std::map< size_t, value_type >	select_max_dot (size_t n, const DistrArray &y) const

std::map< size_t, value_type >	select_max_dot (size_t n, const SparseArray &y) const

std::map< size_t, value_type >	select (size_t n, bool max=false, bool ignore_sign=false) const

Protected Member Functions
	DistrArrayDisk (std::unique_ptr< Distribution > distr, MPI_Comm commun)

	DistrArrayDisk ()

	DistrArrayDisk (const DistrArray &source)

	DistrArrayDisk (DistrArrayDisk &&source) noexcept

	~DistrArrayDisk () override

	DistrArray (size_t dimension, MPI_Comm commun)
	Initializes array without allocating any memory. More...

	DistrArray ()=default

Protected Member Functions inherited from molpro::linalg::array::DistrArray
	DistrArray (size_t dimension, MPI_Comm commun)
	Initializes array without allocating any memory. More...

	DistrArray ()=default

virtual void	_divide (const DistrArray &y, const DistrArray &z, value_type shift, bool append, bool negative)

Protected Attributes
bool	m_allocated = false
	Flags that the memory view buffer has been allocated. More...

std::unique_ptr< Distribution >	m_distribution
	describes distribution of array among processes More...

size_t	m_buffer_size = 8192
	buffer size for paged access via BufferManager More...

Protected Attributes inherited from molpro::linalg::array::DistrArray
index_type	m_dimension = 0
	number of elements in the array More...

MPI_Comm	m_communicator

Member Typedef Documentation

◆ disk_array

using molpro::linalg::array::DistrArrayDisk::disk_array = void

a compile time tag that this is a distributed disk array

Constructor & Destructor Documentation

◆ DistrArrayDisk() [1/4]

molpro::linalg::array::DistrArrayDisk::DistrArrayDisk	(	std::unique_ptr< Distribution >	distr,
		MPI_Comm	commun
	)

protected

◆ DistrArrayDisk() [2/4]

molpro::linalg::array::DistrArrayDisk::DistrArrayDisk ( )

protecteddefault

◆ DistrArrayDisk() [3/4]

molpro::linalg::array::DistrArrayDisk::DistrArrayDisk ( const DistrArray & source )

protected

◆ DistrArrayDisk() [4/4]

molpro::linalg::array::DistrArrayDisk::DistrArrayDisk ( DistrArrayDisk && source )

protectednoexcept

◆ ~DistrArrayDisk()

molpro::linalg::array::DistrArrayDisk::~DistrArrayDisk ( )

overrideprotecteddefault

Member Function Documentation

◆ copy()

void molpro::linalg::array::DistrArrayDisk::copy ( const DistrArray & y )

overridevirtual

Copies all elements of y. If both arrays are empty than does nothing. If only one is empty, throws an error.

Reimplemented from molpro::linalg::array::DistrArray.

◆ DistrArray() [1/2]

molpro::linalg::array::DistrArray::DistrArray ( )

protecteddefault

◆ DistrArray() [2/2]

molpro::linalg::array::DistrArray::DistrArray	(	size_t	dimension,
		MPI_Comm	commun
	)

protected

Initializes array without allocating any memory.

◆ distribution()

const DistrArray::Distribution & molpro::linalg::array::DistrArrayDisk::distribution ( ) const

overridevirtual

Access distribution of the array among processes.

Implements molpro::linalg::array::DistrArray.

◆ dot() [1/3]

DistrArray::value_type molpro::linalg::array::DistrArrayDisk::dot ( const DistrArray & y ) const

overridevirtual

Scalar product of two arrays. Collective. Throws error if any array is empty. Both arrays should be part of the same processor group (same communicator). The result is broadcast to each process.

Reimplemented from molpro::linalg::array::DistrArray.

◆ dot() [2/3]

DistrArray::value_type molpro::linalg::array::DistrArrayDisk::dot ( const DistrArrayDisk & y ) const

◆ dot() [3/3]

DistrArray::value_type molpro::linalg::array::DistrArrayDisk::dot ( const SparseArray & y ) const

overridevirtual

Reimplemented from molpro::linalg::array::DistrArray.

◆ erase()

virtual void molpro::linalg::array::DistrArrayDisk::erase ( )

pure virtual

Erase the array from disk.

Implemented in molpro::linalg::array::DistrArrayFile.

◆ local_buffer() [1/4]

std::unique_ptr< const DistrArray::LocalBuffer > molpro::linalg::array::DistrArrayDisk::local_buffer ( ) const

overridevirtual

Implements molpro::linalg::array::DistrArray.

◆ local_buffer() [2/4]

std::unique_ptr< DistrArray::LocalBuffer > molpro::linalg::array::DistrArrayDisk::local_buffer ( )

overridevirtual

Access the buffer local to this process.

Implements molpro::linalg::array::DistrArray.

◆ local_buffer() [3/4]

std::unique_ptr< LocalBuffer > molpro::linalg::array::DistrArrayDisk::local_buffer ( const span::Span< value_type > & buffer )

Access local section, reading it into the provided buffer.

◆ local_buffer() [4/4]

std::unique_ptr< const LocalBuffer > molpro::linalg::array::DistrArrayDisk::local_buffer ( const span::Span< value_type > & buffer ) const

Read-only access to the local section, reading it into the provided buffer.

◆ set_buffer_size()

void molpro::linalg::array::DistrArrayDisk::set_buffer_size ( size_t buffer_size )

inline

Member Data Documentation

◆ m_allocated

bool molpro::linalg::array::DistrArrayDisk::m_allocated = false

protected

Flags that the memory view buffer has been allocated.

◆ m_buffer_size

size_t molpro::linalg::array::DistrArrayDisk::m_buffer_size = 8192

protected

buffer size for paged access via BufferManager

◆ m_distribution

std::unique_ptr<Distribution> molpro::linalg::array::DistrArrayDisk::m_distribution

protected

describes distribution of array among processes

Detailed Description

Classes

Public Types

Public Member Functions

Protected Member Functions

Protected Attributes

Member Typedef Documentation

◆ disk_array

Constructor & Destructor Documentation

◆ DistrArrayDisk() [1/4]

◆ DistrArrayDisk() [2/4]

◆ DistrArrayDisk() [3/4]

◆ DistrArrayDisk() [4/4]

◆ ~DistrArrayDisk()

Member Function Documentation

◆ copy()

◆ DistrArray() [1/2]

◆ DistrArray() [2/2]

◆ distribution()

◆ dot() [1/3]

◆ dot() [2/3]

◆ dot() [3/3]

◆ erase()

◆ local_buffer() [1/4]

◆ local_buffer() [2/4]

◆ local_buffer() [3/4]

◆ local_buffer() [4/4]

◆ set_buffer_size()

Member Data Documentation

◆ m_allocated

◆ m_buffer_size

◆ m_distribution