Implementation of distributed array using MPI3 RMA operations. More...

#include <DistrArrayMPI3.h>

Inheritance diagram for molpro::linalg::array::DistrArrayMPI3:

[legend]

Collaboration diagram for molpro::linalg::array::DistrArrayMPI3:

[legend]

Detailed Description

Implementation of distributed array using MPI3 RMA operations.

The array buffer is a window which is open for all processes on creation and remains open until the destruction of the array.

Warning: Care must be taken that overlapping put and get operations or local buffer modifications do not cause undefined behaviour.

Classes
struct	LocalBufferMPI3

Public Member Functions
	DistrArrayMPI3 ()=delete

	DistrArrayMPI3 (size_t dimension, MPI_Comm commun)
	Creates distributed array using MPI3 RMA routines and default distribution. More...

	DistrArrayMPI3 (std::unique_ptr< Distribution > distribution, MPI_Comm commun)
	Creates distributed array using MPI3 RMA routines and specified distribution. More...

	DistrArrayMPI3 (std::unique_ptr< Distribution > distribution, MPI_Comm commun, Span< value_type > buffer)
	Create distributed array using external buffer for the local section. More...

	DistrArrayMPI3 (size_t dimension, MPI_Comm commun, Span< value_type > buffer)
	Create distributed array using external buffer for the local section. More...

	DistrArrayMPI3 (const DistrArrayMPI3 &source)
	Copy constructor allocates the buffer if source is not empty. More...

	DistrArrayMPI3 (const DistrArray &source)

	DistrArrayMPI3 (DistrArrayMPI3 &&source) noexcept

DistrArrayMPI3 &	operator= (const DistrArrayMPI3 &source)

DistrArrayMPI3 &	operator= (DistrArrayMPI3 &&source) noexcept

	~DistrArrayMPI3 () override

void	sync () const override
	Synchronizes all process in this group and ensures any outstanding operations on the array have completed. More...

const Distribution &	distribution () const override
	Access distribution of the array among processes. More...

std::unique_ptr< LocalBuffer >	local_buffer () override
	Access the buffer local to this process. More...

std::unique_ptr< const LocalBuffer >	local_buffer () const override

value_type	at (index_type ind) const override

void	set (index_type ind, value_type val) override
	Set one element to a scalar. Global operation. More...

void	get (index_type lo, index_type hi, value_type *buf) const override
	Gets buffer[lo:hi) from global array (hi is past-the-end). Blocking. More...

std::vector< value_type >	get (index_type lo, index_type hi) const override

void	put (index_type lo, index_type hi, const value_type *data) override
	array[lo:hi) = data[:] (hi is past-the-end). Blocking More...

void	acc (index_type lo, index_type hi, const value_type *data) override
	array[lo:hi) += scaling_constant * data[:] (hi is past-the-end). Blocking More...

std::vector< value_type >	gather (const std::vector< index_type > &indices) const override
	gets elements with discontinuous indices from array. Blocking More...

void	scatter (const std::vector< index_type > &indices, const std::vector< value_type > &data) override
	array[indices[i]] = data[i] Puts vals of elements with discontinuous indices of array. Blocking. More...

void	scatter_acc (std::vector< index_type > &indices, const std::vector< value_type > &data) override
	array[indices[i]] += vals[i] Accumulates vals of elements into discontinuous indices of array. Atomic, blocking, with on-sided communication More...

std::vector< value_type >	vec () const override
	Copies the whole buffer into a vector. Blocking. More...

void	error (const std::string &message) const override
	stops application with an error More...

Public Member Functions inherited from molpro::linalg::array::DistrArray
virtual	~DistrArray ()=default

MPI_Comm	communicator () const
	return a copy of the communicator More...

virtual void	sync () const
	Synchronizes all process in this group and ensures any outstanding operations on the array have completed. More...

size_t	size () const
	total number of elements, same as overall dimension of array More...

bool	compatible (const DistrArray &other) const
	Checks that arrays are of the same dimensionality. More...

virtual void	zero ()
	Set all local elements to zero. More...

virtual void	error (const std::string &message) const
	stops application with an error More...

value_type	operator[] (size_t index)

virtual void	fill (value_type val)

virtual void	copy (const DistrArray &y)

virtual void	copy_patch (const DistrArray &y, index_type start, index_type end)
	Copies elements in a patch of y. If both arrays are empty than does nothing. If only one is empty, throws an error. More...

virtual void	axpy (value_type a, const DistrArray &y)
	this[:] += a * y[:]. Throws an error if any array is empty. Add a multiple of another array to this one. Blocking, collective. More...

virtual void	axpy (value_type a, const SparseArray &y)

virtual void	scal (value_type a)
	Scale by a constant. Local. More...

virtual void	add (const DistrArray &y)
	Add another array to this. Local. Throws error if any array is empty. More...

virtual void	add (value_type a)
	Add a constant. Local. More...

virtual void	sub (const DistrArray &y)
	Subtract another array from this. Local. Throws error if any array is empty. More...

virtual void	sub (value_type a)
	Subtract a constant. Local. More...

virtual void	recip ()
	Take element-wise reciprocal of this. Local. No checks are made for zero values. More...

virtual void	times (const DistrArray &y)
	this[i] *= y[i]. Throws error if any array is empty. More...

virtual void	times (const DistrArray &y, const DistrArray &z)
	this[i] = y[i]*z[i]. Throws error if any array is empty. More...

virtual value_type	dot (const DistrArray &y) const
	Scalar product of two arrays. Collective. Throws error if any array is empty. Both arrays should be part of the same processor group (same communicator). The result is broadcast to each process. More...

virtual value_type	dot (const SparseArray &y) const

void	divide (const DistrArray &y, const DistrArray &z, value_type shift=0, bool append=false, bool negative=false)
	this[i] = y[i]/(z[i]+shift). Collective. Throws error if any array is empty. More...

std::list< std::pair< index_type, value_type > >	min_n (int n) const
	returns n smallest elements in array x Collective operation, must be called by all processes in the group. More...

std::list< std::pair< index_type, value_type > >	max_n (int n) const
	returns n largest elements in array x Collective operation, must be called by all processes in the group. More...

std::list< std::pair< index_type, value_type > >	min_abs_n (int n) const
	returns n elements that are largest by absolute value in array x Collective operation, must be called by all processes in the group. More...

std::list< std::pair< index_type, value_type > >	max_abs_n (int n) const
	returns n elements that are largest by absolute value in array x Collective operation, must be called by all processes in the group. More...

std::vector< index_type >	min_loc_n (int n) const
	find the index of n smallest components in array x Collective operation, must be called by all processes in the group. More...

std::map< size_t, value_type >	select_max_dot (size_t n, const DistrArray &y) const

std::map< size_t, value_type >	select_max_dot (size_t n, const SparseArray &y) const

std::map< size_t, value_type >	select (size_t n, bool max=false, bool ignore_sign=false) const

Protected Types
enum class	RMAType { get , put , acc , gather , scatter , scatter_acc }

Protected Member Functions
void	_get_put (index_type lo, index_type hi, const value_type *buf, RMAType option)
	does get or put or accumulate More...

void	_gather_scatter (const std::vector< index_type > &indices, std::vector< value_type > &data, RMAType option)
	does gather or scatter or scatter_acc More...

void	allocate_buffer ()

Protected Member Functions inherited from molpro::linalg::array::DistrArray
	DistrArray (size_t dimension, MPI_Comm commun)
	Initializes array without allocating any memory. More...

	DistrArray ()=default

virtual void	_divide (const DistrArray &y, const DistrArray &z, value_type shift, bool append, bool negative)

Protected Attributes
MPI_Win	m_win = MPI_WIN_NULL
	window object More...

std::unique_ptr< Distribution >	m_distribution
	describes distribution of array among processes More...

bool	m_allocated = false
	whether the window has been created More...

Protected Attributes inherited from molpro::linalg::array::DistrArray
index_type	m_dimension = 0
	number of elements in the array More...

MPI_Comm	m_communicator

Friends
void	swap (DistrArrayMPI3 &a1, DistrArrayMPI3 &a2) noexcept

Additional Inherited Members
Public Types inherited from molpro::linalg::array::DistrArray
using	distributed_array = void
	a compile time tag that this is a distributed array More...

using	value_type = double

using	index_type = size_t

using	SparseArray = std::map< index_type, double >

using	Distribution = util::Distribution< index_type >

Member Enumeration Documentation

◆ RMAType

enum class molpro::linalg::array::DistrArrayMPI3::RMAType

strongprotected

Enumerator
get
put
acc
gather
scatter
scatter_acc

Constructor & Destructor Documentation

◆ DistrArrayMPI3() [1/8]

molpro::linalg::array::DistrArrayMPI3::DistrArrayMPI3 ( )

delete

◆ DistrArrayMPI3() [2/8]

molpro::linalg::array::DistrArrayMPI3::DistrArrayMPI3	(	size_t	dimension,
		MPI_Comm	commun
	)

Creates distributed array using MPI3 RMA routines and default distribution.

By default array is distributed uniformly with any remainder spread over the first processes so their size is larger by 1.

Parameters

dimension	overall size of array
commun	MPI communicator

◆ DistrArrayMPI3() [3/8]

molpro::linalg::array::DistrArrayMPI3::DistrArrayMPI3	(	std::unique_ptr< Distribution >	distribution,
		MPI_Comm	commun
	)

Creates distributed array using MPI3 RMA routines and specified distribution.

Parameters

distribution	specification of how array is distributed among processes
commun	MPI communicator

◆ DistrArrayMPI3() [4/8]

molpro::linalg::array::DistrArrayMPI3::DistrArrayMPI3	(	std::unique_ptr< Distribution >	distribution,
		MPI_Comm	commun,
		Span< value_type >	buffer
	)

Create distributed array using external buffer for the local section.

◆ DistrArrayMPI3() [5/8]

molpro::linalg::array::DistrArrayMPI3::DistrArrayMPI3	(	size_t	dimension,
		MPI_Comm	commun,
		Span< value_type >	buffer
	)

Create distributed array using external buffer for the local section.

◆ DistrArrayMPI3() [6/8]

molpro::linalg::array::DistrArrayMPI3::DistrArrayMPI3 ( const DistrArrayMPI3 & source )

Copy constructor allocates the buffer if source is not empty.

◆ DistrArrayMPI3() [7/8]

molpro::linalg::array::DistrArrayMPI3::DistrArrayMPI3 ( const DistrArray & source )

◆ DistrArrayMPI3() [8/8]

molpro::linalg::array::DistrArrayMPI3::DistrArrayMPI3 ( DistrArrayMPI3 && source )

noexcept

◆ ~DistrArrayMPI3()

molpro::linalg::array::DistrArrayMPI3::~DistrArrayMPI3 ( )

override

Member Function Documentation

◆ _gather_scatter()

void molpro::linalg::array::DistrArrayMPI3::_gather_scatter	(	const std::vector< index_type > &	indices,
		std::vector< value_type > &	data,
		RMAType	option
	)

protected

does gather or scatter or scatter_acc

◆ _get_put()

void molpro::linalg::array::DistrArrayMPI3::_get_put	(	index_type	lo,
		index_type	hi,
		const value_type *	buf,
		RMAType	option
	)

protected

does get or put or accumulate

◆ acc()

void molpro::linalg::array::DistrArrayMPI3::acc	(	index_type	lo,
		index_type	hi,
		const value_type *	data
	)

overridevirtual

array[lo:hi) += scaling_constant * data[:] (hi is past-the-end). Blocking

Implements molpro::linalg::array::DistrArray.

◆ allocate_buffer()

void molpro::linalg::array::DistrArrayMPI3::allocate_buffer ( )

protected

◆ at()

DistrArray::value_type molpro::linalg::array::DistrArrayMPI3::at ( index_type ind ) const

overridevirtual

get element at the offset. Blocking.

Implements molpro::linalg::array::DistrArray.

◆ distribution()

const DistrArray::Distribution & molpro::linalg::array::DistrArrayMPI3::distribution ( ) const

overridevirtual

Access distribution of the array among processes.

Implements molpro::linalg::array::DistrArray.

◆ error()

void molpro::linalg::array::DistrArrayMPI3::error ( const std::string & message ) const

overridevirtual

stops application with an error

Reimplemented from molpro::linalg::array::DistrArray.

◆ gather()

std::vector< DistrArrayMPI3::value_type > molpro::linalg::array::DistrArrayMPI3::gather ( const std::vector< index_type > & indices ) const

overridevirtual

gets elements with discontinuous indices from array. Blocking

Returns: res[i] = array[indices[i]]

Implements molpro::linalg::array::DistrArray.

◆ get() [1/2]

std::vector< DistrArrayMPI3::value_type > molpro::linalg::array::DistrArrayMPI3::get	(	index_type	lo,
		index_type	hi
	)		const

overridevirtual

Implements molpro::linalg::array::DistrArray.

◆ get() [2/2]

void molpro::linalg::array::DistrArrayMPI3::get	(	index_type	lo,
		index_type	hi,
		value_type *	buf
	)		const

overridevirtual

Gets buffer[lo:hi) from global array (hi is past-the-end). Blocking.

Implements molpro::linalg::array::DistrArray.

◆ local_buffer() [1/2]

std::unique_ptr< const DistrArray::LocalBuffer > molpro::linalg::array::DistrArrayMPI3::local_buffer ( ) const

overridevirtual

Implements molpro::linalg::array::DistrArray.

◆ local_buffer() [2/2]

std::unique_ptr< DistrArrayMPI3::LocalBuffer > molpro::linalg::array::DistrArrayMPI3::local_buffer ( )

overridevirtual

Access the buffer local to this process.

Implements molpro::linalg::array::DistrArray.

◆ operator=() [1/2]

DistrArrayMPI3 & molpro::linalg::array::DistrArrayMPI3::operator= ( const DistrArrayMPI3 & source )

◆ operator=() [2/2]

DistrArrayMPI3 & molpro::linalg::array::DistrArrayMPI3::operator= ( DistrArrayMPI3 && source )

noexcept

◆ put()

void molpro::linalg::array::DistrArrayMPI3::put	(	index_type	lo,
		index_type	hi,
		const value_type *	data
	)

overridevirtual

array[lo:hi) = data[:] (hi is past-the-end). Blocking

Implements molpro::linalg::array::DistrArray.

◆ scatter()

void molpro::linalg::array::DistrArrayMPI3::scatter	(	const std::vector< index_type > &	indices,
		const std::vector< value_type > &	data
	)

overridevirtual

array[indices[i]] = data[i] Puts vals of elements with discontinuous indices of array. Blocking.

Implements molpro::linalg::array::DistrArray.

◆ scatter_acc()

void molpro::linalg::array::DistrArrayMPI3::scatter_acc	(	std::vector< index_type > &	indices,
		const std::vector< value_type > &	data
	)

overridevirtual

array[indices[i]] += vals[i] Accumulates vals of elements into discontinuous indices of array. Atomic, blocking, with on-sided communication

Implements molpro::linalg::array::DistrArray.

◆ set()

void molpro::linalg::array::DistrArrayMPI3::set	(	index_type	ind,
		value_type	val
	)

overridevirtual

Set one element to a scalar. Global operation.

Implements molpro::linalg::array::DistrArray.

◆ sync()

void molpro::linalg::array::DistrArrayMPI3::sync ( ) const

overridevirtual

Synchronizes all process in this group and ensures any outstanding operations on the array have completed.

Reimplemented from molpro::linalg::array::DistrArray.

◆ vec()

std::vector< DistrArrayMPI3::value_type > molpro::linalg::array::DistrArrayMPI3::vec ( ) const

overridevirtual

Copies the whole buffer into a vector. Blocking.

Note: This is only meant for debugging small arrays!

Implements molpro::linalg::array::DistrArray.

Friends And Related Function Documentation

◆ swap

void swap	(	DistrArrayMPI3 &	a1,
		DistrArrayMPI3 &	a2
	)

friend

Member Data Documentation

◆ m_allocated

bool molpro::linalg::array::DistrArrayMPI3::m_allocated = false

protected

whether the window has been created

◆ m_distribution

std::unique_ptr<Distribution> molpro::linalg::array::DistrArrayMPI3::m_distribution

protected

describes distribution of array among processes

◆ m_win

MPI_Win molpro::linalg::array::DistrArrayMPI3::m_win = MPI_WIN_NULL

protected

window object

Detailed Description

Classes

Public Member Functions

Protected Types

Protected Member Functions

Protected Attributes

Friends

Additional Inherited Members

Member Enumeration Documentation

◆ RMAType

Constructor & Destructor Documentation

◆ DistrArrayMPI3() [1/8]

◆ DistrArrayMPI3() [2/8]

◆ DistrArrayMPI3() [3/8]

◆ DistrArrayMPI3() [4/8]

◆ DistrArrayMPI3() [5/8]

◆ DistrArrayMPI3() [6/8]

◆ DistrArrayMPI3() [7/8]

◆ DistrArrayMPI3() [8/8]

◆ ~DistrArrayMPI3()

Member Function Documentation

◆ _gather_scatter()

◆ _get_put()

◆ acc()

◆ allocate_buffer()

◆ at()

◆ distribution()

◆ error()

◆ gather()

◆ get() [1/2]

◆ get() [2/2]

◆ local_buffer() [1/2]

◆ local_buffer() [2/2]

◆ operator=() [1/2]

◆ operator=() [2/2]

◆ put()

◆ scatter()

◆ scatter_acc()

◆ set()

◆ sync()

◆ vec()

Friends And Related Function Documentation

◆ swap

Member Data Documentation

◆ m_allocated

◆ m_distribution

◆ m_win