Distributed array which uses Global Arrays for managing the array buffer and RMA calls. More...

#include <DistrArrayGA.h>

Inheritance diagram for molpro::linalg::array::DistrArrayGA:

[legend]

Collaboration diagram for molpro::linalg::array::DistrArrayGA:

[legend]

Detailed Description

Distributed array which uses Global Arrays for managing the array buffer and RMA calls.

Note: Thread safety: construction, copying and allocation are collective over the communicator (they call MPI and GA collectives) and are not thread safe. Within each process they must be called from one thread at a time, and in the same order on every process, as for any other GA collective operation.

Classes
struct	LocalBufferGA

Public Member Functions
	DistrArrayGA ()=delete

	DistrArrayGA (size_t dimension, MPI_Comm commun)

	DistrArrayGA (const DistrArrayGA &source)
	Make a copy of source, allocating buffer if necessary.

DistrArrayGA &	operator= (const DistrArrayGA &source)
	Copy source, allocating buffer if necessary.

	DistrArrayGA (DistrArrayGA &&source) noexcept
	Take ownership of source leaving it in undefined state. Not collective, no allocation/deallocation.

DistrArrayGA &	operator= (DistrArrayGA &&source) noexcept
	Take ownership of source leaving it in undefined state. Not collective, no allocation/deallocation.

	~DistrArrayGA () override

void	sync () const override
	Synchronizes all process in this group and ensures any outstanding operations on the array have completed.

const Distribution &	distribution () const override
	Access distribution of the array among processes.

std::unique_ptr< LocalBuffer >	local_buffer () override
	Access the buffer local to this process.

std::unique_ptr< const LocalBuffer >	local_buffer () const override

value_type	at (index_type ind) const override

void	set (index_type ind, value_type val) override
	Set one element to a scalar. Global operation.

void	get (index_type lo, index_type hi, value_type *buf) const override
	Gets buffer[lo:hi) from global array (hi is past-the-end). Blocking.

std::vector< value_type >	get (index_type lo, index_type hi) const override

void	put (index_type lo, index_type hi, const value_type *data) override
	array[lo:hi) = data[:] (hi is past-the-end). Blocking

void	acc (index_type lo, index_type hi, const value_type *data) override
	array[lo:hi) += scaling_constant * data[:] (hi is past-the-end). Blocking

std::vector< value_type >	gather (const std::vector< index_type > &indices) const override
	gets elements with discontinuous indices from array. Blocking

void	scatter (const std::vector< index_type > &indices, const std::vector< value_type > &data) override
	array[indices[i]] = data[i] Puts vals of elements with discontinuous indices of array. Blocking.

void	scatter_acc (std::vector< index_type > &indices, const std::vector< value_type > &data) override
	array[indices[i]] += vals[i] Accumulates vals of elements into discontinuous indices of array. Atomic, blocking, with on-sided communication

std::vector< value_type >	vec () const override
	Copies the whole buffer into a vector. Blocking.

void	error (const std::string &message) const override
	stops application with an error

Public Member Functions inherited from molpro::linalg::array::DistrArray
virtual	~DistrArray ()=default

	DistrArray (const DistrArray &)=default

	DistrArray (DistrArray &&) noexcept=default

DistrArray &	operator= (const DistrArray &)=default

DistrArray &	operator= (DistrArray &&) noexcept=default

MPI_Comm	communicator () const
	return a copy of the communicator

size_t	size () const
	total number of elements, same as overall dimension of array

bool	compatible (const DistrArray &other) const
	Checks that arrays are of the same dimensionality.

virtual void	zero ()
	Set all local elements to zero.

virtual size_t	disk_page_size () const

value_type	operator[] (index_type ind) const

util::ValueProxy< DistrArray >	operator[] (index_type ind)

virtual void	fill (value_type val)

virtual void	copy (const DistrArray &y)

virtual void	copy_patch (const DistrArray &y, index_type start, index_type end)
	Copies elements in a patch of y. If both arrays are empty than does nothing. If only one is empty, throws an error.

virtual void	axpy (value_type a, const DistrArray &y)
	this[:] += a * y[:]. Throws an error if any array is empty. Add a multiple of another array to this one. Blocking, collective.

virtual void	axpy (value_type a, const SparseArray &y)

void	copy (const SparseArray &y)
	this[:] = y[:], i.e. the elements of y, and zero elsewhere. Local, each process sets its local section.

virtual void	scal (value_type a)

virtual void	add (const DistrArray &y)
	Add another array to this. Local. Throws error if any array is empty.

virtual void	add (value_type a)
	Add a constant. Local.

virtual void	sub (const DistrArray &y)
	Subtract another array from this. Local. Throws error if any array is empty.

virtual void	sub (value_type a)
	Subtract a constant. Local.

virtual void	recip ()
	Take element-wise reciprocal of this. Local. No checks are made for zero values.

virtual void	times (const DistrArray &y)
	this[i] *= y[i]. Throws error if any array is empty.

virtual void	times (const DistrArray &y, const DistrArray &z)
	this[i] = y[i]*z[i]. Throws error if any array is empty.

void	copy (Span< const value_type > y)

void	axpy (value_type a, Span< const value_type > y)
	this[:] += a * y[:] over the local section. Local.

void	copy_to (Span< value_type > y) const
	y[:] = this[:]. Collective: each process copies its local section, and then y is replicated over all of them.

void	axpy_to (value_type a, Span< value_type > y) const

value_type	dot (Span< const value_type > y) const
	Scalar product <this\|y>. Collective, the result is available to each process.

std::map< size_t, value_type >	select_max_dot (size_t n, Span< const value_type > y) const
	The n indices with the largest \|this[i]*y[i]\|. Collective.

virtual value_type	dot (const DistrArray &y) const
	Scalar product of two arrays. Collective. Throws error if any array is empty. Both arrays should be part of the same processor group (same communicator). The result is broadcast to each process.

virtual value_type	dot (const SparseArray &y) const

void	divide (const DistrArray &y, const DistrArray &z, value_type shift=0, bool append=false, bool negative=false)
	this[i] = y[i]/(z[i]+shift). Collective. Throws error if any array is empty.

std::list< std::pair< index_type, value_type > >	min_n (int n) const
	returns n smallest elements in array x Collective operation, must be called by all processes in the group.

std::list< std::pair< index_type, value_type > >	max_n (int n) const
	returns n largest elements in array x Collective operation, must be called by all processes in the group.

std::list< std::pair< index_type, value_type > >	min_abs_n (int n) const
	returns n elements that are largest by absolute value in array x Collective operation, must be called by all processes in the group.

std::list< std::pair< index_type, value_type > >	max_abs_n (int n) const
	returns n elements that are largest by absolute value in array x Collective operation, must be called by all processes in the group.

std::vector< index_type >	min_loc_n (int n) const
	find the index of n smallest components in array x Collective operation, must be called by all processes in the group.

std::map< size_t, value_type >	select_max_dot (size_t n, const DistrArray &y) const

std::map< size_t, value_type >	select_max_dot (size_t n, const SparseArray &y) const

std::map< size_t, value_type >	select (size_t n, bool max=false, bool ignore_sign=false) const

Protected Member Functions
void	check_ga_ind_overlow (index_type ind) const
	Checks if index will overflow when converted to int. Calls error() if it does.

Distribution	make_distribution () const

void	allocate_buffer ()

Protected Member Functions inherited from molpro::linalg::array::DistrArray
	DistrArray (size_t dimension, MPI_Comm commun)
	Initializes array without allocating any memory.

	DistrArray ()=default

void	check_compatible (Span< const value_type > y, const std::string &name) const
	Stops with an error unless the contiguous array y has as many elements as this.

virtual void	_divide (const DistrArray &y, const DistrArray &z, value_type shift, bool append, bool negative)

Protected Attributes
int	m_comm_rank = 0
	rank in process group

int	m_comm_size = 0
	size of process group

int	m_ga_handle = 0
	Global Array handle, needed by GA libary.

int	m_ga_pgroup = 0
	Global Array processor group handle.

int	m_ga_chunk = 1
	GA chunck size.

bool	m_ga_allocated = false

std::unique_ptr< Distribution >	m_distribution

Protected Attributes inherited from molpro::linalg::array::DistrArray
index_type	m_dimension = 0
	number of elements in the array

MPI_Comm	m_communicator

Static Protected Attributes
static std::map< MPI_Comm, int >	_ga_pgroups

Friends
void	swap (DistrArrayGA &a1, DistrArrayGA &a2) noexcept
	swap content of two arrays. Not collective.

Additional Inherited Members
Public Types inherited from molpro::linalg::array::DistrArray
using	distributed_array = void
	a compile time tag that this is a distributed array

using	value_type = double

using	index_type = size_t

using	SparseArray = std::map< index_type, double >

using	Distribution = util::Distribution< index_type >

Constructor & Destructor Documentation

◆ DistrArrayGA() [1/4]

molpro::linalg::array::DistrArrayGA::DistrArrayGA ( )

delete

◆ DistrArrayGA() [2/4]

molpro::linalg::array::DistrArrayGA::DistrArrayGA	(	size_t	dimension,
		MPI_Comm	commun
	)

◆ DistrArrayGA() [3/4]

molpro::linalg::array::DistrArrayGA::DistrArrayGA ( const DistrArrayGA & source )

Make a copy of source, allocating buffer if necessary.

◆ DistrArrayGA() [4/4]

molpro::linalg::array::DistrArrayGA::DistrArrayGA ( DistrArrayGA && source )

noexcept

Take ownership of source leaving it in undefined state. Not collective, no allocation/deallocation.

◆ ~DistrArrayGA()

molpro::linalg::array::DistrArrayGA::~DistrArrayGA ( )

override

Member Function Documentation

◆ acc()

void molpro::linalg::array::DistrArrayGA::acc	(	index_type	lo,
		index_type	hi,
		const value_type *	data
	)

overridevirtual

array[lo:hi) += scaling_constant * data[:] (hi is past-the-end). Blocking

Implements molpro::linalg::array::DistrArray.

◆ allocate_buffer()

void molpro::linalg::array::DistrArrayGA::allocate_buffer ( )

protected

◆ at()

value_type molpro::linalg::array::DistrArrayGA::at ( index_type ind ) const

overridevirtual

get element at the offset. Blocking.

Implements molpro::linalg::array::DistrArray.

◆ check_ga_ind_overlow()

void molpro::linalg::array::DistrArrayGA::check_ga_ind_overlow ( index_type ind ) const

protected

Checks if index will overflow when converted to int. Calls error() if it does.

◆ distribution()

const Distribution & molpro::linalg::array::DistrArrayGA::distribution ( ) const

overridevirtual

Access distribution of the array among processes.

Implements molpro::linalg::array::DistrArray.

◆ error()

void molpro::linalg::array::DistrArrayGA::error ( const std::string & message ) const

overridevirtual

stops application with an error

Reimplemented from molpro::linalg::array::DistrArray.

◆ gather()

std::vector< value_type > molpro::linalg::array::DistrArrayGA::gather ( const std::vector< index_type > & indices ) const

overridevirtual

gets elements with discontinuous indices from array. Blocking

Returns: res[i] = array[indices[i]]

Implements molpro::linalg::array::DistrArray.

◆ get() [1/2]

std::vector< value_type > molpro::linalg::array::DistrArrayGA::get	(	index_type	lo,
		index_type	hi
	)		const

overridevirtual

Implements molpro::linalg::array::DistrArray.

◆ get() [2/2]

void molpro::linalg::array::DistrArrayGA::get	(	index_type	lo,
		index_type	hi,
		value_type *	buf
	)		const

overridevirtual

Gets buffer[lo:hi) from global array (hi is past-the-end). Blocking.

Implements molpro::linalg::array::DistrArray.

◆ local_buffer() [1/2]

std::unique_ptr< const LocalBuffer > molpro::linalg::array::DistrArrayGA::local_buffer ( ) const

overridevirtual

Implements molpro::linalg::array::DistrArray.

◆ local_buffer() [2/2]

std::unique_ptr< LocalBuffer > molpro::linalg::array::DistrArrayGA::local_buffer ( )

overridevirtual

Access the buffer local to this process.

Implements molpro::linalg::array::DistrArray.

◆ make_distribution()

Distribution molpro::linalg::array::DistrArrayGA::make_distribution ( ) const

protected

◆ operator=() [1/2]

DistrArrayGA & molpro::linalg::array::DistrArrayGA::operator= ( const DistrArrayGA & source )

Copy source, allocating buffer if necessary.

◆ operator=() [2/2]

DistrArrayGA & molpro::linalg::array::DistrArrayGA::operator= ( DistrArrayGA && source )

noexcept

Take ownership of source leaving it in undefined state. Not collective, no allocation/deallocation.

◆ put()

void molpro::linalg::array::DistrArrayGA::put	(	index_type	lo,
		index_type	hi,
		const value_type *	data
	)

overridevirtual

array[lo:hi) = data[:] (hi is past-the-end). Blocking

Implements molpro::linalg::array::DistrArray.

◆ scatter()

void molpro::linalg::array::DistrArrayGA::scatter	(	const std::vector< index_type > &	indices,
		const std::vector< value_type > &	data
	)

overridevirtual

array[indices[i]] = data[i] Puts vals of elements with discontinuous indices of array. Blocking.

Implements molpro::linalg::array::DistrArray.

◆ scatter_acc()

void molpro::linalg::array::DistrArrayGA::scatter_acc	(	std::vector< index_type > &	indices,
		const std::vector< value_type > &	data
	)

overridevirtual

array[indices[i]] += vals[i] Accumulates vals of elements into discontinuous indices of array. Atomic, blocking, with on-sided communication

Implements molpro::linalg::array::DistrArray.

◆ set()

void molpro::linalg::array::DistrArrayGA::set	(	index_type	ind,
		value_type	val
	)

overridevirtual

Set one element to a scalar. Global operation.

Implements molpro::linalg::array::DistrArray.

◆ sync()

void molpro::linalg::array::DistrArrayGA::sync ( ) const

overridevirtual

Synchronizes all process in this group and ensures any outstanding operations on the array have completed.

Reimplemented from molpro::linalg::array::DistrArray.

◆ vec()

std::vector< value_type > molpro::linalg::array::DistrArrayGA::vec ( ) const

overridevirtual

Copies the whole buffer into a vector. Blocking.

Note: This is only meant for debugging small arrays!

Implements molpro::linalg::array::DistrArray.

Friends And Related Symbol Documentation

◆ swap

void swap	(	DistrArrayGA &	a1,
		DistrArrayGA &	a2
	)

friend

swap content of two arrays. Not collective.

Member Data Documentation

◆ _ga_pgroups

std::map<MPI_Comm, int> molpro::linalg::array::DistrArrayGA::_ga_pgroups

staticprotected

Record every process group created, because GA can only allocate a fixed number of them. Not synchronised: a lock here would not make allocation thread safe, because it is surrounded by MPI and GA collectives, and holding a lock across collectives could deadlock. See the thread-safety note on the class.

◆ m_comm_rank

int molpro::linalg::array::DistrArrayGA::m_comm_rank = 0

protected

rank in process group

◆ m_comm_size

int molpro::linalg::array::DistrArrayGA::m_comm_size = 0

protected

size of process group

◆ m_distribution

std::unique_ptr<Distribution> molpro::linalg::array::DistrArrayGA::m_distribution

protected

◆ m_ga_allocated

bool molpro::linalg::array::DistrArrayGA::m_ga_allocated = false

protected

Flags that GA has been allocated

◆ m_ga_chunk

int molpro::linalg::array::DistrArrayGA::m_ga_chunk = 1

protected

GA chunck size.

◆ m_ga_handle

int molpro::linalg::array::DistrArrayGA::m_ga_handle = 0

protected

Global Array handle, needed by GA libary.

◆ m_ga_pgroup

int molpro::linalg::array::DistrArrayGA::m_ga_pgroup = 0

protected

Global Array processor group handle.

Detailed Description

Classes

Public Member Functions

Protected Member Functions

Protected Attributes

Static Protected Attributes

Friends

Additional Inherited Members

Constructor & Destructor Documentation

◆ DistrArrayGA() [1/4]

◆ DistrArrayGA() [2/4]

◆ DistrArrayGA() [3/4]

◆ DistrArrayGA() [4/4]

◆ ~DistrArrayGA()

Member Function Documentation

◆ acc()

◆ allocate_buffer()

◆ at()

◆ check_ga_ind_overlow()

◆ distribution()

◆ error()

◆ gather()

◆ get() [1/2]

◆ get() [2/2]

◆ local_buffer() [1/2]

◆ local_buffer() [2/2]

◆ make_distribution()

◆ operator=() [1/2]

◆ operator=() [2/2]

◆ put()

◆ scatter()

◆ scatter_acc()

◆ set()

◆ sync()

◆ vec()

Friends And Related Symbol Documentation

◆ swap

Member Data Documentation

◆ _ga_pgroups

◆ m_comm_rank

◆ m_comm_size

◆ m_distribution

◆ m_ga_allocated

◆ m_ga_chunk

◆ m_ga_handle

◆ m_ga_pgroup