SHMEM_FCOLLECT64(3)

📦
version 5.0.10 · 4 versions
, in ubuntu stonking

💖 SHMEM_FCOLLECT64(3) — Open MPI, but make it make sense 💖

okay bestie, gather round, because this is the glow-up of the Open MPI manual page for shmem_fcollect64 (and its whole little squad of friends). no cap, this one is a collective, and collective means everybody's in on it. let's go 🫶

✨ NAME ✨

so like, this one page covers a whole friend group 💅

and what do they all do? they concatenate blocks of data from multiple processing elements (PEs) to an array in every PE. yes, every PE. it's giving "we all get a copy, nobody's left out" energy. 👑

✨ SYNOPSIS ✨

C or C++ besties 🎀

#include <mpp/shmem.h>

void shmem_collect32(void *target, const void *source,
  size_t nelems, int PE_start, int logPE_stride, int PE_size,
  long *pSync)

void shmem_collect64(void *target, const void *source,
  size_t nelems, int PE_start, int logPE_stride, int PE_size,
  long *pSync)

void shmem_fcollect32(void *target, const void *source,
  size_t nelems, int PE_start, int logPE_stride, int PE_size,
  long *pSync)

void shmem_fcollect64(void *target, const void *source,
  size_t nelems, int PE_start, int logPE_stride, int PE_size,
  long *pSync)

Fortran besties 🌸

INCLUDE "mpp/shmem.fh"

INTEGER nelems
INTEGER PE_start, logPE_stride, PE_size
INTEGER pSync(SHMEM_COLLECT_SYNC_SIZE)

CALL SHMEM_COLLECT4(target, source, nelems, PE_start,
& logPE_stride, PE_size, pSync)

CALL SHMEM_COLLECT8(target, source, nelems, PE_start,
& logPE_stride, PE_size, pSync)

CALL SHMEM_FCOLLECT4(target, source, nelems, PE_start,
& logPE_stride, PE_size, pSync)

CALL SHMEM_FCOLLECT8(target, source, nelems, PE_start,
& logPE_stride, PE_size, pSync)

like, pick your fighter. there's a 32-bit one and a 64-bit one and they're all right there. 🌈

✨ DESCRIPTION ✨

okay so here's the tea ☕

the shared memory (SHMEM) collect and fcollect routines concatenate nelems 64-bit or 32-bit data items from the source array into the target array, over the set of PEs defined by PE_start, log2PE_stride, and PE_size, in processor number order. and the order is very much intentional: the resultant target array contains the contribution from PE PE_start first, then the contribution from PE PE_start + PE_stride second, and so on. the collected result is written to the target array for all PEs in the active set. periodt. 💅

the fcollect routines require that nelems be the same value in all participating PEs, while the collect routines allow nelems to vary from PE to PE. (not me being flexible vs. not me being flexible, iykyk.)

the resulting target array is as follows:

----------------------------------------------------------
   source(1..nelems)
       from PE (PE_start + 0 * (2**logPE_stride))
----------------------------------------------------------
   source(1..nelems)
       from PE (PE_start + 1 * (2**logPE_stride))
----------------------------------------------------------
   ...
----------------------------------------------------------
   source(1..nelems) from
       PE (PE_start + (PE_size - 1) * (2**logPE_stride))
----------------------------------------------------------

look at that. stacked up all neat like a little friendship bracelet. 🧚

as with all SHMEM collective routines, each of these routines assumes that only PEs in the active set call the routine. if a PE not in the active set calls a SHMEM collective routine, undefined behavior results. 🚨 not the vibe. do not do that.

the arguments, one by one 🍓

  • target — A symmetric array. the target argument must be large enough to accept the concatenation of the source arrays on all PEs. the data types are as follows:
    • [shmem_collect8, shmem_collect64, shmem_fcollect8, and shmem_fcollect64] — any data type with an element size of 64 bits. Fortran derived types, Fortran character type, and C/C++ structures are not permitted. 🙅
    • [shmem_collect4, shmem_collect32, shmem_fcollect4, and shmem_fcollect32] — any data type with an element size of 32 bits. Fortran derived types, Fortran character type, and C/C++ structures are not permitted. 🙅
  • source — A symmetric data object that can be of any type permissible for the target argument. so basically whatever target is allowed to be, source can be too. matching outfits. 💞
  • nelems — The number of elements in the source array. nelems must be of type integer. if you're using Fortran, it must be a default integer value. (the collect girlies may vary this per PE; the fcollect girlies may not.)
  • PE_start — The lowest virtual PE number of the active set of PEs. PE_start must be of type integer. if you're using Fortran, it must be a default integer value.
  • logPE_stride — The log (base 2) of the stride between consecutive virtual PE numbers in the active set. logPE_stride must be of type integer. if you're using Fortran, it must be a default integer value. yes bestie it's a log, base 2, that's just how they do it. 📐
  • PE_size — The number of PEs in the active set. PE_size must be of type integer. if you're using Fortran, it must be a default integer value.
  • pSync — A symmetric work array. in C/C++, pSync must be of type int and size _SHMEM_COLLECT_SYNC_SIZE. in Fortran, pSync must be of type integer and size SHMEM_COLLECT_SYNC_SIZE. if you're using Fortran, it must be a default integer value. every element of this array must be initialized with the value _SHMEM_SYNC_VALUE in C/C++ or SHMEM_SYNC_VALUE in Fortran before any of the PEs in the active set enter shmem_barrier(). do not skip this step i am begging you 😭

the rules of the group chat 📋

  • The values of arguments PE_start, logPE_stride, and PE_size must be equal on all PEs in the active set.
  • The same target and source arrays and the same pSync work array must be passed to all PEs in the active set.

like, everyone has to agree. it's a group project and everyone is doing their part. shocking, i know.

upon return from a collective routine, the following are true for the local PE:

  • The target array is updated. ✨
  • The values in the pSync array are restored to the original values.

so it cleans up after itself. main character with manners. 💅

✨ NOTES ✨

the terms collective and symmetric are defined in intro_shmem(3). go read it, it's giving required reading. 📚

all SHMEM collective routines reset the values in pSync before they return, so a particular pSync buffer need only be initialized the first time it is used. cute, right? one-time setup. 🎀

WARNING bestie, this is the important part: you must ensure that the pSync array is not being updated on any PE in the active set while any of the PEs participate in processing of a SHMEM collective routine.

be careful to avoid these situations:

  • If the pSync array is initialized at run time, some type of synchronization is needed to ensure that all PEs in the working set have initialized pSync before any of them enter a SHMEM routine called with the pSync synchronization array.
  • A pSync array can be reused on a subsequent SHMEM collective routine only if none of the PEs in the active set are still processing a prior SHMEM collective routine call that used the same pSync array.

in general, this may be ensured only by doing some type of synchronization. however, in the special case of SHMEM routines being called with the same active set, you can allocate two pSync arrays and alternate between them on successive calls. see? two buffers, taking turns, no drama. 🦋

also, performance tea: the collective routines operate on active PE sets that have a non-power-of-two PE_size with some performance degradation. they operate with no performance degradation when nelems is a non-power-of-two value. so like, one of these is fine and one of these costs you. the manpage said what it said. 😌

✨ EXAMPLES ✨

C/C++ 💜

for (i=0; i < _SHMEM_COLLECT_SYNC_SIZE; i++) {
  pSync[i] = _SHMEM_SYNC_VALUE;
}
shmem_barrier_all(); /* Wait for all PEs to initialize pSync */
shmem_collect32(target, source, 64, pe_start, logPE_stride,
   pe_size, pSync);

see how they init pSync and then shmem_barrier_all() to wait for everybody before anyone moves? that's the whole "no PE is updating pSync while others are working" thing from the notes. iconic behavior tbh. 👑

Fortran 🌷

INTEGER PSYNC(SHMEM_COLLECT_SYNC_SIZE)
DATA PSYNC /SHMEM_COLLECT_SYNC_SIZE*SHMEM_SYNC_VALUE/

CALL SHMEM_COLLECT4(TARGET, SOURCE, 64, PE_START,
& LOGPE_STRIDE, PE_SIZE, PSYNC)

same energy, different syntax. we love range. 🌈

✨ SEE ALSO ✨

  • intro_shmem(3) — go read it bestie, it explains collective and symmetric and you're gonna want that context.

💖 the cute sign-off 💖

this page is copyright 2003-2026, The Open MPI Community, and it was last touched on Sep 10, 2026, and honestly? that's the whole thing. thanks for coming to my ted talk, now go initialize your pSync array. 🫶✨💅

📦 where this page comes from