Distributed array located primarily on disk. More...

#include <DistrArrayDisk.h>

Inheritance diagram for molpro::linalg::array::DistrArrayDisk:

[legend]

Collaboration diagram for molpro::linalg::array::DistrArrayDisk:

[legend]

Detailed Description

Distributed array located primarily on disk.

This class stores the full array on disk and implements RMA and more efficient linear algebra operations.

RMA operations read/write directly to disk.

There is no local buffer: the memory used must not scale with the size of the array, so local_buffer() throws. Elementwise operations, including those with arrays held in memory, page through the local section disk_page_size() elements at a time.

BufferManager reads the local section in chunks using a separate thread for I/O. This is more memory efficient and allows overlap of communication and computation. The walk through the local section is via iterators.

IO can be done in a separate thread using util::Task.

#include <molpro/linalg/array/util.h>
auto da = DistrArrayDisk{...};
da.put(lo,hi, data); // Normal put same as in the base class is guaranteed to finish
auto t = util::Task::create([&](){da.put(lo, hi, data)}); // does I/O in a new thread
// do something time consuming
t.wait(); // wait for the thread to finish the I/O

Public Types
using	disk_array = void
	a compile time tag that this is a distributed disk array

Public Types inherited from molpro::linalg::array::DistrArray
using	distributed_array = void
	a compile time tag that this is a distributed array

using	value_type = double

using	index_type = size_t

using	SparseArray = std::map< index_type, double >

using	Distribution = util::Distribution< index_type >

Public Member Functions
virtual void	erase ()=0
	Erase the array from disk.

const Distribution &	distribution () const override
	Access distribution of the array among processes.

value_type	dot (const DistrArrayDisk &y) const

void	set_buffer_size (size_t buffer_size)

size_t	disk_page_size () const override

std::unique_ptr< LocalBuffer >	local_buffer () override
	Not available: an array on disk is not held in memory.

std::unique_ptr< const LocalBuffer >	local_buffer () const override
	Not available: an array on disk is not held in memory.

virtual value_type	dot (const DistrArray &y) const
	Scalar product of two arrays. Collective. Throws error if any array is empty. Both arrays should be part of the same processor group (same communicator). The result is broadcast to each process.

virtual value_type	dot (const SparseArray &y) const

Public Member Functions inherited from molpro::linalg::array::DistrArray
virtual	~DistrArray ()=default

	DistrArray (const DistrArray &)=default

	DistrArray (DistrArray &&) noexcept=default

DistrArray &	operator= (const DistrArray &)=default

DistrArray &	operator= (DistrArray &&) noexcept=default

MPI_Comm	communicator () const
	return a copy of the communicator

virtual void	sync () const
	Synchronizes all process in this group and ensures any outstanding operations on the array have completed.

size_t	size () const
	total number of elements, same as overall dimension of array

bool	compatible (const DistrArray &other) const
	Checks that arrays are of the same dimensionality.

virtual void	zero ()
	Set all local elements to zero.

virtual void	error (const std::string &message) const
	stops application with an error

virtual value_type	at (index_type ind) const =0

value_type	operator[] (index_type ind) const

util::ValueProxy< DistrArray >	operator[] (index_type ind)

virtual void	set (index_type ind, value_type val)=0
	Set one element to a scalar. Global operation.

virtual void	get (index_type lo, index_type hi, value_type *buf) const =0
	Gets buffer[lo:hi) from global array (hi is past-the-end). Blocking.

virtual std::vector< value_type >	get (index_type lo, index_type hi) const =0

virtual void	put (index_type lo, index_type hi, const value_type *data)=0
	array[lo:hi) = data[:] (hi is past-the-end). Blocking

virtual void	acc (index_type lo, index_type hi, const value_type *data)=0
	array[lo:hi) += scaling_constant * data[:] (hi is past-the-end). Blocking

virtual std::vector< value_type >	gather (const std::vector< index_type > &indices) const =0
	gets elements with discontinuous indices from array. Blocking

virtual void	scatter (const std::vector< index_type > &indices, const std::vector< value_type > &data)=0
	array[indices[i]] = data[i] Puts vals of elements with discontinuous indices of array. Blocking.

virtual void	scatter_acc (std::vector< index_type > &indices, const std::vector< value_type > &data)=0
	array[indices[i]] += vals[i] Accumulates vals of elements into discontinuous indices of array. Atomic, blocking, with on-sided communication

virtual std::vector< value_type >	vec () const =0
	Copies the whole buffer into a vector. Blocking.

virtual void	fill (value_type val)

virtual void	copy (const DistrArray &y)

virtual void	copy_patch (const DistrArray &y, index_type start, index_type end)
	Copies elements in a patch of y. If both arrays are empty than does nothing. If only one is empty, throws an error.

virtual void	axpy (value_type a, const DistrArray &y)
	this[:] += a * y[:]. Throws an error if any array is empty. Add a multiple of another array to this one. Blocking, collective.

virtual void	axpy (value_type a, const SparseArray &y)

virtual void	scal (value_type a)
	Scale by a constant. Local.

virtual void	add (const DistrArray &y)
	Add another array to this. Local. Throws error if any array is empty.

virtual void	add (value_type a)
	Add a constant. Local.

virtual void	sub (const DistrArray &y)
	Subtract another array from this. Local. Throws error if any array is empty.

virtual void	sub (value_type a)
	Subtract a constant. Local.

virtual void	recip ()
	Take element-wise reciprocal of this. Local. No checks are made for zero values.

virtual void	times (const DistrArray &y)
	this[i] *= y[i]. Throws error if any array is empty.

virtual void	times (const DistrArray &y, const DistrArray &z)
	this[i] = y[i]*z[i]. Throws error if any array is empty.

void	divide (const DistrArray &y, const DistrArray &z, value_type shift=0, bool append=false, bool negative=false)
	this[i] = y[i]/(z[i]+shift). Collective. Throws error if any array is empty.

std::list< std::pair< index_type, value_type > >	min_n (int n) const
	returns n smallest elements in array x Collective operation, must be called by all processes in the group.

std::list< std::pair< index_type, value_type > >	max_n (int n) const
	returns n largest elements in array x Collective operation, must be called by all processes in the group.

std::list< std::pair< index_type, value_type > >	min_abs_n (int n) const
	returns n elements that are largest by absolute value in array x Collective operation, must be called by all processes in the group.

std::list< std::pair< index_type, value_type > >	max_abs_n (int n) const
	returns n elements that are largest by absolute value in array x Collective operation, must be called by all processes in the group.

std::vector< index_type >	min_loc_n (int n) const
	find the index of n smallest components in array x Collective operation, must be called by all processes in the group.

std::map< size_t, value_type >	select_max_dot (size_t n, const DistrArray &y) const

std::map< size_t, value_type >	select_max_dot (size_t n, const SparseArray &y) const

std::map< size_t, value_type >	select (size_t n, bool max=false, bool ignore_sign=false) const

Protected Member Functions
	DistrArrayDisk (std::unique_ptr< Distribution > distr, MPI_Comm commun)

	DistrArrayDisk ()

	DistrArrayDisk (const DistrArray &source)

	DistrArrayDisk (DistrArrayDisk &&source) noexcept

	~DistrArrayDisk () override

	DistrArray (size_t dimension, MPI_Comm commun)
	Initializes array without allocating any memory.

	DistrArray ()=default

	DistrArray (const DistrArray &)=default

	DistrArray (DistrArray &&) noexcept=default

Protected Member Functions inherited from molpro::linalg::array::DistrArray
	DistrArray (size_t dimension, MPI_Comm commun)
	Initializes array without allocating any memory.

	DistrArray ()=default

virtual void	_divide (const DistrArray &y, const DistrArray &z, value_type shift, bool append, bool negative)

Protected Attributes
bool	m_allocated = false
	Flags that the memory view buffer has been allocated.

std::unique_ptr< Distribution >	m_distribution
	describes distribution of array among processes

size_t	m_buffer_size = 8192
	buffer size for paged access via BufferManager

Protected Attributes inherited from molpro::linalg::array::DistrArray
index_type	m_dimension = 0
	number of elements in the array

MPI_Comm	m_communicator

Member Typedef Documentation

◆ disk_array

using molpro::linalg::array::DistrArrayDisk::disk_array = void

a compile time tag that this is a distributed disk array

Constructor & Destructor Documentation

◆ DistrArrayDisk() [1/4]

molpro::linalg::array::DistrArrayDisk::DistrArrayDisk	(	std::unique_ptr< Distribution >	distr,
		MPI_Comm	commun
	)

protected

◆ DistrArrayDisk() [2/4]

molpro::linalg::array::DistrArrayDisk::DistrArrayDisk ( )

protecteddefault

◆ DistrArrayDisk() [3/4]

molpro::linalg::array::DistrArrayDisk::DistrArrayDisk ( const DistrArray & source )

protected

◆ DistrArrayDisk() [4/4]

molpro::linalg::array::DistrArrayDisk::DistrArrayDisk ( DistrArrayDisk && source )

protectednoexcept

◆ ~DistrArrayDisk()

molpro::linalg::array::DistrArrayDisk::~DistrArrayDisk ( )

overrideprotecteddefault

Member Function Documentation

◆ disk_page_size()

size_t molpro::linalg::array::DistrArrayDisk::disk_page_size ( ) const

inlineoverridevirtual

For an array held on disk, which has no local buffer, the number of elements that elementwise operations read and write at a time; 0 for an array held in memory

Reimplemented from molpro::linalg::array::DistrArray.

◆ DistrArray() [1/4]

molpro::linalg::array::DistrArray::DistrArray ( )

protecteddefault

◆ DistrArray() [2/4]

molpro::linalg::array::DistrArray::DistrArray ( const DistrArray & )

protecteddefault

◆ DistrArray() [3/4]

molpro::linalg::array::DistrArray::DistrArray ( DistrArray && )

protecteddefaultnoexcept

◆ DistrArray() [4/4]

molpro::linalg::array::DistrArray::DistrArray	(	size_t	dimension,
		MPI_Comm	commun
	)

protected

Initializes array without allocating any memory.

◆ distribution()

const DistrArray::Distribution & molpro::linalg::array::DistrArrayDisk::distribution ( ) const

overridevirtual

Access distribution of the array among processes.

Implements molpro::linalg::array::DistrArray.

◆ dot() [1/3]

DistrArray::value_type molpro::linalg::array::DistrArray::dot ( const DistrArray & y ) const

virtual

Scalar product of two arrays. Collective. Throws error if any array is empty. Both arrays should be part of the same processor group (same communicator). The result is broadcast to each process.

Reimplemented from molpro::linalg::array::DistrArray.

◆ dot() [2/3]

value_type molpro::linalg::array::DistrArrayDisk::dot ( const DistrArrayDisk & y ) const

inline

◆ dot() [3/3]

DistrArray::value_type molpro::linalg::array::DistrArray::dot ( const SparseArray & y ) const

virtual

Reimplemented from molpro::linalg::array::DistrArray.

◆ erase()

virtual void molpro::linalg::array::DistrArrayDisk::erase ( )

pure virtual

Erase the array from disk.

Implemented in molpro::linalg::array::DistrArrayFile.

◆ local_buffer() [1/2]

std::unique_ptr< const DistrArray::LocalBuffer > molpro::linalg::array::DistrArrayDisk::local_buffer ( ) const

overridevirtual

Not available: an array on disk is not held in memory.

Exceptions

std::logic_error

Implements molpro::linalg::array::DistrArray.

◆ local_buffer() [2/2]

std::unique_ptr< DistrArray::LocalBuffer > molpro::linalg::array::DistrArrayDisk::local_buffer ( )

overridevirtual

Not available: an array on disk is not held in memory.

Exceptions

std::logic_error

Implements molpro::linalg::array::DistrArray.

◆ set_buffer_size()

void molpro::linalg::array::DistrArrayDisk::set_buffer_size ( size_t buffer_size )

inline

Member Data Documentation

◆ m_allocated

bool molpro::linalg::array::DistrArrayDisk::m_allocated = false

protected

Flags that the memory view buffer has been allocated.

◆ m_buffer_size

size_t molpro::linalg::array::DistrArrayDisk::m_buffer_size = 8192

protected

buffer size for paged access via BufferManager

◆ m_distribution

std::unique_ptr<Distribution> molpro::linalg::array::DistrArrayDisk::m_distribution

protected

describes distribution of array among processes

Detailed Description

Public Types

Public Member Functions

Protected Member Functions

Protected Attributes

Member Typedef Documentation

◆ disk_array

Constructor & Destructor Documentation

◆ DistrArrayDisk() [1/4]

◆ DistrArrayDisk() [2/4]

◆ DistrArrayDisk() [3/4]

◆ DistrArrayDisk() [4/4]

◆ ~DistrArrayDisk()

Member Function Documentation

◆ disk_page_size()

◆ DistrArray() [1/4]

◆ DistrArray() [2/4]

◆ DistrArray() [3/4]

◆ DistrArray() [4/4]

◆ distribution()

◆ dot() [1/3]

◆ dot() [2/3]

◆ dot() [3/3]

◆ erase()

◆ local_buffer() [1/2]

◆ local_buffer() [2/2]

◆ set_buffer_size()

Member Data Documentation

◆ m_allocated

◆ m_buffer_size

◆ m_distribution