SHMEM_ALLTOALL64(3)

🦋 SHMEM_ALLTOALL64(3) — the Open MPI glow-up ✨

okay bestie, sit down, hydrate, because this is the collective routine where every single PE exchanges a fixed amount of data with literally all the other PEs in the Active set. no cap. the whole girl group is trading fits at once 💅

it's shmem_alltoall32(3), shmem_alltoall64(3), shmem_alltoalls32(3), and shmem_alltoalls64(3), all living under the Open MPI roof 🌈

---

✨ SYNOPSIS ✨

C or C++:

#include <mpp/shmem.h>

void shmem_alltoall32(void *target, const void *source,
  size_t nelems, int PE_start, int logPE_stride,
  int PE_size, long *pSync)

void shmem_alltoall64(void *target, const void *source,
  size_t nelems, int PE_start, int logPE_stride,
  int PE_size, long *pSync)

void shmem_alltoalls32(void *target, const void *source,
  ptrdiff_t dst, ptrdiff_t sst,
  size_t nelems, int PE_start, int logPE_stride,
  int PE_size, long *pSync)

void shmem_alltoalls64(void *target, const void *source,
  ptrdiff_t dst, ptrdiff_t sst,
  size_t nelems, int PE_start, int logPE_stride,
  int PE_size, long *pSync)

Fortran:

INCLUDE "mpp/shmem.fh"

INTEGER dst, sst, nelems, PE_root, PE_start, logPE_stride, PE_size
INTEGER pSync(SHMEM_ALLTOALL_SYNC_SIZE)

CALL SHMEM_ALLTOALL32(target, source, nelems,
& PE_start, logPE_stride, PE_size, fIpSync)

CALL SHMEM_ALLTOALL64(target, source, nelems,
& PE_start, logPE_stride, PE_size, pSync)

CALL SHMEM_ALLTOALLS32(target, source, dst, sst, nelems,
& PE_start, logPE_stride, PE_size, pSync)

CALL SHMEM_ALLTOALLS64(target, source, dst, sst, nelems,
& PE_start, logPE_stride, PE_size, pSync)

---

💖 DESCRIPTION 💖

so. the shmem_alltoalls routines are collective routines. every PE in the Active set swaps nelems strided data elements — each one sized 32 bits (that's shmem_alltoalls32) or 64 bits (that's shmem_alltoalls64) — with all the other PEs in the set. everyone eats, nobody is left on read 🍓

the tea on the strides: both strides, dst and sst, must be greater than or equal to 1. and here's the placement rule, write it on your mirror: the sst*jth block sent from PE i to PE j gets placed in the dst*ith block of the dest data object on PE j. iconic choreography, no notes 👑

and like every OpenSHMEM collective routine ever, these assume that only PEs in the Active set call the routine. if a PE that is not in the Active set calls an OpenSHMEM collective routine, you get undefined behavior. periodt, don't test it 💀

the values of the arguments dst, sst, nelems, PE_start, logPE_stride, and PE_size must be equal on all PEs in the Active set. the same dest and source data objects, and the same pSync work array, must be passed to all PEs in the Active set. matching outfits only. 🙈

before any PE calls a shmem_alltoalls routine, these conditions must already exist (and yes, synchronization via a barrier or some other method is often needed to make that true):

• the pSync array on all PEs in the Active set is not still in use from a prior call to a shmem_alltoalls routine
• the dest data object on all PEs in the Active set is ready to accept the shmem_alltoalls data

upon return from a shmem_alltoalls routine, here's what's true for the local PE:

• its dest symmetric data object is completely updated and the data has been copied out of the source data object
• the values in the pSync array are restored to the original values

the arguments are as follows:

• a symmetric data object with one of the following data types:
  • shmem_alltoall32: any noncharacter type that has an element size of 32 bits. no Fortran derived types or C/C++ structures are allowed.
  • shmem_alltoall64: any noncharacter type that has an element size of 64 bits. no Fortran derived types or C/C++ structures are allowed.
• target — a symmetric data object big enough to receive the combined total of nelems elements from each PE in the Active set. it's giving spacious 💅
• source — a symmetric data object holding nelems elements of data for each PE in the Active set, ordered according to destination PE.
• dst — the stride between consecutive elements of the dest data object. the stride is scaled by the element size. a value of 1 means contiguous data. dst must be of type ptrdiff_t. if you're in Fortran, it must be a default integer value.
• sst — the stride between consecutive elements of the source data object. the stride is scaled by the element size. a value of 1 means contiguous data. sst must be of type ptrdiff_t. if you're in Fortran, it must be a default integer value.
• nelems — the number of elements to exchange for each PE. nelems must be of type size_t for C/C++. if you're in Fortran, it must be a default integer value.
• PE_start — the lowest virtual PE number of the active set of PEs. PE_start must be of type integer. if you're in Fortran, it must be a default integer value.
• logPE_stride — the log (base 2) of the stride between consecutive virtual PE numbers in the active set. log_PE_stride must be of type integer. if you're in Fortran, it must be a default integer value.
• PE_size — the number of PEs in the active set. PE_size must be of type integer. if you're in Fortran, it must be a default integer value.
• pSync — a symmetric work array. in C/C++, pSync must be of type long and size _SHMEM_ALLTOALL_SYNC_SIZE. in Fortran, pSync must be of type integer and size SHMEM_ALLTOALL_SYNC_SIZE. every element of this array must be initialized with the value _SHMEM_SYNC_VALUE (in C/C++) or SHMEM_SYNC_VALUE (in Fortran) before any of the PEs in the active set enter shmem_barrier().

upon return from a shmem_alltoalls routine, the following is true for the local PE: its dest symmetric data object is completely updated and the data has been copied out of the source data object. the values in the pSync array are restored to the original values. (yes bestie, this is the vibe again, it's that important 💜)

the values of arguments PE_root, PE_start, logPE_stride, and PE_size must be equal on all PEs in the active set. the same target and source data objects and the same pSync work array must be passed to all PEs in the active set.

before any PE calls a alltoall routine, you must make sure the following conditions exist (synchronization via a barrier or some other method is often needed to ensure this):

• the pSync array on all PEs in the active set is not still in use from a prior call to a alltoall routine
• the target array on all PEs in the active set is ready to accept the alltoall data

---

🌸 NOTES 🌸

the terms collective and symmetric are defined in intro_shmem(3). go read her, she's the blueprint 📖

all SHMEM alltoall routines restore pSync to its original contents. multiple calls to SHMEM routines that use the same pSync array do not require that pSync be reinitialized after the first call. so efficient, so slay ✨

⚠️ you must ensure the that the pSync array is not being updated by any PE in the active set while any of the PEs participates in processing of a SHMEM broadcast routine. be careful to avoid these situations, bestie, this is the part where people get got 😭

• if the pSync array is initialized at run time, some type of synchronization is needed to ensure that all PEs in the working set have initialized pSync before any of them enter a SHMEM routine called with the pSync synchronization array.
• a pSync array may be reused on a subsequent SHMEM broadcast routine only if none of the PEs in the active set are still processing a prior SHMEM alltoall routine call that used the same pSync array.
• in general, this can be ensured only by doing some type of synchronization. however, in the special case of SHMEM routines being called with the same active set, you can allocate two pSync arrays and alternate between them on successive calls. two arrays, twin energy, we love a rotation 🎀

---

🧚 EXAMPLES 🧚

C/C++ example:

#include <shmem.h>
#include <stdio.h>

long pSync[SHMEM_ALLTOALL_SYNC_SIZE];
int main(void)
{
int64_t *source, *dest;
int i, count, pe;
shmem_init();
count = 2;
dest = (int64_t*) shmem_malloc(count * shmem_n_pes() * sizeof(int64_t));
source = (int64_t*) shmem_malloc(count * shmem_n_pes() * sizeof(int64_t));
/* assign source values */
for (pe=0; pe <shmem_n_pes(); pe++){
for (i=0; i<count; i++){
source[(pe*count)+i] = shmem_my_pe() + pe;
dest[(pe*count)+i] = 9999;
}
}
for (i=0; i< SHMEM_ALLTOALLS_SYNC_SIZE; i++) {
pSync[i] = SHMEM_SYNC_VALUE;
}
/* wait for all PEs to initialize pSync */
shmem_barrier_all();
/* alltoalls on all PES */
shmem_alltoalls64(dest, source, 1, 1, count, 0, 0, shmem_n_pes(), pSync);
/* verify results */
for (pe=0; pe<shmem_n_pes(); pe++) {
for (i=0; i<count; i++){
if (dest[(pe*count)+i] != shmem_my_pe() + pe) {
printf("[%d] ERROR: dest[%d]=%ld, should be %d,
shmem_my_pe(),(pe*count)+i,dest[(pe*count)+i],
shmem_n_pes() + pe);
}
}
}
shmem_barrier_all();
shmem_free(dest);
shmem_free(source);
shmem_finalize();
return 0;
}

See also:

• intro_shmem(3)

---

👑 Copyright 👑

Copyright 2003-2026, The Open MPI Community

---

and that's the whole page, bestie 🫶 this was SHMEM_ALLTOALL64(3) from Open MPI, dated Sep 10, 2026 — copyright 2003-2026, The Open MPI Community. go forth and alltoall responsibly, ily, mwah 💖✨🎀