SHMEM_SHORT_SUM_TO_ALL(3)

📦
version 5.0.10 · 7 versions
, in ubuntu stonking

💖 SHMEM_SHORT_SUM_TO_ALL(3) — Open MPI 💖

okay bestie, welcome to Open MPI's little corner of the manual, and today's episode is ALL about summing things up across a whole squad of processing elements. it's giving "group project but the group actually works". no cap. ✨

🌸 NAME 🌸

meet the whole girlband, because they all do the same dance move:

and what do they do? they perform a sum reduction across a set of processing elements (PEs). that's it. that's the gig. 💅

✨ SYNOPSIS ✨

C or C++:

#include <mpp/shmem.h>

void shmem_complexd_sum_to_all(double complex *target,
  const double complex *source, int nreduce, int PE_start,
  int logPE_stride, int PE_size, double complex *pWrk,
  long *pSync)

void shmem_complexf_sum_to_all(float complex *target,
  const float complex *source, int nreduce, int PE_start,
  int logPE_stride, int PE_size, float complex *pWrk,
  long *pSync)

void shmem_double_sum_to_all(double *target,
  const double *source, int nreduce, int PE_start, int logPE_stride,
  int PE_size, double *pWrk, long *pSync)

void shmem_float_sum_to_all(float *target, const float *source,
  int nreduce, int PE_start, int logPE_stride, int PE_size,
  float *pWrk, long *pSync)

void shmem_int_sum_to_all(int *target, const int *source,
  int nreduce, int PE_start, int logPE_stride, int PE_size,
  int *pWrk, long *pSync)

void shmem_long_sum_to_all(long *target, const long *source,
  int nreduce, int PE_start, int logPE_stride, int PE_size,
  long *pWrk, long *pSync)

void shmem_longdouble_sum_to_all(long double *target,
  const long double *source, int nreduce, int PE_start, int
  logPE_stride, int PE_size, long double *pWrk, long *pSync)

void shmem_longlong_sum_to_all(long long *target,
  const long long *source, int nreduce, int PE_start,
  int logPE_stride, int PE_size, long long *pWrk,
  long *pSync)

void shmem_short_sum_to_all(short *target, const short *source,
  int nreduce, int PE_start, int logPE_stride, int PE_size,
  short *pWrk, long *pSync)

Fortran:

INCLUDE "mpp/shmem.fh"

INTEGER pSync(SHMEM_REDUCE_SYNC_SIZE)
INTEGER nreduce, PE_start, logPE_stride, PE_size

CALL SHMEM_COMP4_SUM_TO_ALL(target, source, nreduce,
& PE_start, logPE_stride, PE_size, pWrk, pSync)

CALL SHMEM_COMP8_SUM_TO_ALL(target, source, nreduce,
& PE_start, logPE_stride, PE_size, pWrk, pSync)

CALL SHMEM_INT4_SUM_TO_ALL(target, source, nreduce,
& PE_start, logPE_stride, PE_size, pWrk, pSync)

CALL SHMEM_INT8_SUM_TO_ALL(target, source, nreduce,
& PE_start, logPE_stride, PE_size, pWrk, pSync)

CALL SHMEM_REAL4_SUM_TO_ALL(target, source, nreduce,
& PE_start, logPE_stride, PE_size, pWrk, pSync)

CALL SHMEM_REAL8_SUM_TO_ALL(target, source, nreduce,
& PE_start, logPE_stride, PE_size, pWrk, pSync)

CALL SHMEM_REAL16_SUM_TO_ALL(target, source, nreduce,
& PE_start, logPE_stride, PE_size, pWrk, pSync)

🧚 DESCRIPTION 🧚

okay so. the shared memory (SHMEM) reduction routines compute one or more reductions across symmetric arrays on multiple virtual PEs. a reduction is just an associative binary operation done across a set of values — the group chat voted and the answer is the vibe. want the full list of other SHMEM reduction routines? go read intro_shmem(3). iykyk. 📚

like all SHMEM collective routines, every single one of these assumes that only the PEs in the active set call the routine. and bestie, if a PE that is not in the active set calls a SHMEM collective routine, that's undefined behavior. we do not do that. 🚫💀

the nreduce argument decides how many separate reductions we're doing. the source array on all PEs in the active set hands over one element for each reduction. the results go into the target array on all PEs in the active set. the active set itself is defined by the PE_start, logPE_stride, PE_size triplet — the holy trinity, periodt. 👑

source and target may be the same array, but they may not be overlapping arrays. yes that's a real distinction, no i will not elaborate further than the manpage does. 🙈

now, the arguments:

target — the remotely accessible integer data object to be updated on the remote PE. if you're in C/C++, the type of target should match what the SYNOPSIS is implying. if you're on a Fortran compiler, it has to be of type integer with an element size of 4 bytes for SHMEM_INT4_ADD and 8 bytes for SHMEM_INT8_ADD.

value — the value to be atomically added to target. in C/C++, the type of value should match what SYNOPSIS implies. in Fortran it has to be of type integer with an element size of target.

pe — an integer indicating the PE number upon which target is to be updated. in Fortran it has to be a default integer value.

ngl bestie, those last three literally read like they wandered in from a different page (an add routine, if we're being honest 🌚). i am relaying them exactly as written, because the rule is: keep every detail, change nothing. so here they are, chillin' in the middle of the sum docs. iconic behavior actually.

target — a symmetric array of length nreduce elements to receive the results of the reduction operations. the data type of target changes depending on which version of the reduction routine you called and what language you're in. from C/C++, check the SYNOPSIS for the data type tea. from Fortran, the target data types go like this:

  • shmem_comp4_sum_to_all: COMPLEX(KIND=4).
  • shmem_comp8_sum_to_all: Complex. if you're using Fortran, it must be a default complex value.
  • shmem_int4_sum_to_all: INTEGER(KIND=4).
  • shmem_int8_sum_to_all: Integer. if you're using Fortran, it must be a default integer value.
  • shmem_real4_sum_to_all: REAL(KIND=4).
  • shmem_real8_sum_to_all: Real. if you're using Fortran, it must be a default real value.
  • shmem_real16_sum_to_all: Real. if you're using Fortran, it must be a default real value.

source — a symmetric array of length nreduce elements, containing one element for each separate reduction operation. source has to have the same data type as target. twinsies. 👯

nreduce — the number of elements in the target and source arrays. must be of type integer. in Fortran, a default integer value.

PE_start — the lowest virtual PE number of the active set of PEs. must be of type integer. in Fortran, a default integer value.

logPE_stride — the log (base 2) of the stride between consecutive virtual PE numbers in the active set. must be of type integer. in Fortran, a default integer value.

PE_size — the number of PEs in the active set. must be of type integer. in Fortran, a default integer value.

pWrk — a symmetric work array. it has to have the same data type as target. in C/C++ it holds max(nreduce/2 + 1, _SHMEM_REDUCE_MIN_WRKDATA_SIZE) elements. in Fortran it holds max(nreduce/2 + 1, SHMEM_REDUCE_MIN_WRKDATA_SIZE) elements.

pSync — a symmetric work array. in C/C++, pSync is of type long and size _SHMEM_REDUCE_SYNC_SIZE. in Fortran, pSync is of type integer and size SHMEM_REDUCE_SYNC_SIZE, and it must be a default integer value. every element of this array must be initialized with the value _SHMEM_SYNC_VALUE (C/C++) or SHMEM_SYNC_VALUE (Fortran) before any of the PEs in the active set enter the reduction routine. no skipping this step. fr fr. 🫶

🎀 the rules of the collective 🎀

the values of nreduce, PE_start, logPE_stride, and PE_size must be equal on all PEs in the active set. and the same target and source arrays, plus the same pWrk and pSync work arrays, must be passed to all PEs in the active set. everybody matching, everybody synced, that's the assignment. ✨

before any PE calls a reduction routine, you have to make sure these conditions exist — and bestie, a barrier or some other synchronization method is usually what you need to guarantee it:

  • the pWrk and pSync arrays on all PEs in the active set are not still in use from a prior call to a collective SHMEM routine. 🚨
  • the target array on all PEs in the active set is ready to accept the results of the reduction.

once a reduction routine returns, here's what's true for the local PE:

  • the target array is updated. ✅
  • the values in the pSync array are restored to the original values. ♻️

💅 NOTES 💅

the terms collective, symmetric, and cache aligned are all defined over in intro_shmem(3) — go read it, it's the lore. 📖

all SHMEM reduction routines reset the values in pSync before they return, so a particular pSync buffer only needs to be initialized the first time it's used. cute, low maintenance, we love her. 🌸

you must make sure the pSync array is not being updated on any PE in the active set while any of the PEs are participating in processing of a SHMEM reduction routine. and bestie, watch out for these specific situations:

  • if the pSync array is initialized at run time, you need some type of synchronization to make sure all PEs in the working set have initialized pSync before any of them enter a SHMEM routine called with the pSync synchronization array. ⏳
  • a pSync or pWrk array can be reused in a subsequent reduction routine call only if none of the PEs in the active set are still processing a prior reduction routine call that used those same pSync or pWrk arrays. 🫠

in general, the only way to be sure of that is to do some type of synchronization. HOWEVER — in the special case of reduction routines being called with the same active set, you can allocate two pSync and pWrk arrays and alternate between them on successive calls. the double-buffer glow-up. 💖🔥

okay and there's more to this page, bestie — part 2 is coming, so don't go anywhere 🌟

✨ EXAMPLES ✨

Okay bestie, time for the part where it gets real. Two examples, both iconic, both doing a sum reduction. Let's go 💅

🌸 Example 1

This Fortran example statically initializes the pSync array and finds the sum of the real variable FOO across all even PEs. Lowkey a flex, honestly 💖

INCLUDE "mpp/shmem.fh"

INTEGER PSYNC(SHMEM_REDUCE_SYNC_SIZE)
DATA PSYNC /SHMEM_REDUCE_SYNC_SIZE*SHMEM_SYNC_VALUE/
PARAMETER (NR=1)
REAL FOO, FOOSUM, PWRK(MAX(NR/2+1,SHMEM_REDUCE_MIN_WRKDATA_SIZE))
COMMON /COM/ FOO, FOOSUM, PWRK
INTRINSIC MY_PE

IF ( MOD(MY_PE(),2) .EQ. 0) THEN
  CALL SHMEM_INT4_SUM_TO_ALL(FOOSUM, FOO, NR, 0, 1, N$PES/2,
  & PWRK, PSYNC)
  PRINT *, 'Result on PE ', MY_PE(), ' is ', FOOSUM
ENDIF

Not the MOD(MY_PE(),2) .EQ. 0 gate doing the "only even PEs get a turn" thing 🎀 and yes, PSYNC is sized with SHMEM_REDUCE_SYNC_SIZE and filled with SHMEM_SYNC_VALUE, and PWRK gets MAX(NR/2+1,SHMEM_REDUCE_MIN_WRKDATA_SIZE) — all of that is load-bearing, don't skip it 🙈

🦋 Example 2

Consider the following C/C++ call:

shmem_int_sum_to_all( target, source, 3, 0, 0, 8, pwrk, psync );

The preceding call is more efficient, but semantically equivalent to, the combination of the following calls:

shmem_int_sum_to_all(&(target[0]), &(source[0]), 1, 0, 0, 8,
  pwrk1, psync1);
shmem_int_sum_to_all(&(target[1]), &(source[1]), 1, 0, 0, 8,
  pwrk2, psync2);
shmem_int_sum_to_all(&(target[2]), &(source[2]), 1, 0, 0, 8,
  pwrk1, psync1);
⚠️ Note that two sets of pWrk and pSync arrays are used alternately because no synchronization is done between calls. Like, the work arrays take turns because nothing is stopping for a sync in between — that's the whole point of having two. Don't come for me, it's in the spec 💀

👑 See also

Copyright 2003-2026, The Open MPI Community

And that's a wrap on SHMEM_SHORT_SUM_TO_ALL(3) 🫶 this page is brought to you by the Open MPI Community (copyright 2003-2026, iconic behavior only), version dated Sep 10, 2026, and the page tag doing the absolute most as always: SHMEM_SHORT_SUM_TO_ALL(3). Sum your reductions, sync your arrays, and stan your even PEs 🌈💜✨

📦 where this page comes from