MPI_ALLTOALLW_INIT(3)

📦
version 5.0.10 · 4 versions
, in ubuntu stonking

🌸 MPI_ALLTOALLW_INIT(3) — Open MPI 🌸

okay bestie, gather round, because this one is the glow-up of all-to-all and i need you to be seated for it 💅

MPI_Alltoallw, MPI_Ialltoallw, MPI_Alltoallw_init — literally every process sends data of different types to, and receives data of different types from, all processes. yes, the "w" stands for "we're not doing the simple version", and honestly? iconic. 👑

✨ SYNTAX ✨

💖 C Syntax

#include <mpi.h>

int MPI_Alltoallw(const void *sendbuf, const int sendcounts[],
     const int sdispls[], const MPI_Datatype sendtypes[],
     void *recvbuf, const int recvcounts[], const int rdispls[],
     const MPI_Datatype recvtypes[], MPI_Comm comm)

int MPI_Ialltoallw(const void *sendbuf, const int sendcounts[],
     const int sdispls[], const MPI_Datatype sendtypes[],
     void *recvbuf, const int recvcounts[], const int rdispls[],
     const MPI_Datatype recvtypes[], MPI_Comm comm,
     MPI_Request *request)

int MPI_Alltoallw_init(const void *sendbuf, const int sendcounts[],
     const int sdispls[], const MPI_Datatype sendtypes[],
     void *recvbuf, const int recvcounts[], const int rdispls[],
     const MPI_Datatype recvtypes[], MPI_Comm comm, MPI_Info info,
     MPI_Request *request)

💖 Fortran Syntax

USE MPI
! or the older form: INCLUDE 'mpif.h'
MPI_ALLTOALLW(SENDBUF, SENDCOUNTS, SDISPLS, SENDTYPES,
     RECVBUF, RECVCOUNTS, RDISPLS, RECVTYPES, COMM, IERROR)

     <type>  SENDBUF(*), RECVBUF(*)
     INTEGER SENDCOUNTS(*), SDISPLS(*), SENDTYPES(*)
     INTEGER RECVCOUNTS(*), RDISPLS(*), RECVTYPES(*)
     INTEGER COMM, IERROR

MPI_IALLTOALLW(SENDBUF, SENDCOUNTS, SDISPLS, SENDTYPES,
     RECVBUF, RECVCOUNTS, RDISPLS, RECVTYPES, COMM, REQUEST, IERROR)

     <type>  SENDBUF(*), RECVBUF(*)
     INTEGER SENDCOUNTS(*), SDISPLS(*), SENDTYPES(*)
     INTEGER RECVCOUNTS(*), RDISPLS(*), RECVTYPES(*)
     INTEGER COMM, REQUEST, IERROR

MPI_ALLTOALLW_INIT(SENDBUF, SENDCOUNTS, SDISPLS, SENDTYPES,
     RECVBUF, RECVCOUNTS, RDISPLS, RECVTYPES, COMM, INFO, REQUEST, IERROR)

     <type>  SENDBUF(*), RECVBUF(*)
     INTEGER SENDCOUNTS(*), SDISPLS(*), SENDTYPES(*)
     INTEGER RECVCOUNTS(*), RDISPLS(*), RECVTYPES(*)
     INTEGER COMM, INFO, REQUEST, IERROR

💖 Fortran 2008 Syntax

USE mpi_f08
MPI_Alltoallw(sendbuf, sendcounts, sdispls, sendtypes, recvbuf, recvcounts,
             rdispls, recvtypes, comm, ierror)

     TYPE(*), DIMENSION(..), INTENT(IN) :: sendbuf
     TYPE(*), DIMENSION(..) :: recvbuf
     INTEGER, INTENT(IN) :: sendcounts(*), sdispls(*), recvcounts(*),
     rdispls(*)
     TYPE(MPI_Datatype), INTENT(IN) :: sendtypes(*)
     TYPE(MPI_Datatype), INTENT(IN) :: recvtypes(*)
     TYPE(MPI_Comm), INTENT(IN) :: comm
     INTEGER, OPTIONAL, INTENT(OUT) :: ierror

MPI_Ialltoallw(sendbuf, sendcounts, sdispls, sendtypes, recvbuf,
             recvcounts, rdispls, recvtypes, comm, request, ierror)

     TYPE(*), DIMENSION(..), INTENT(IN), ASYNCHRONOUS :: sendbuf
     TYPE(*), DIMENSION(..), ASYNCHRONOUS :: recvbuf
     INTEGER, INTENT(IN), ASYNCHRONOUS :: sendcounts(*), sdispls(*),
     recvcounts(*), rdispls(*)
     TYPE(MPI_Datatype), INTENT(IN), ASYNCHRONOUS :: sendtypes(*),
     recvtypes(*)
     TYPE(MPI_Comm), INTENT(IN) :: comm
     TYPE(MPI_Request), INTENT(OUT) :: request
     INTEGER, OPTIONAL, INTENT(OUT) :: ierror

MPI_Alltoallw_init(sendbuf, sendcounts, sdispls, sendtypes, recvbuf,
             recvcounts, rdispls, recvtypes, comm, fIinfo, request, ierror)

     TYPE(*), DIMENSION(..), INTENT(IN), ASYNCHRONOUS :: sendbuf
     TYPE(*), DIMENSION(..), ASYNCHRONOUS :: recvbuf
     INTEGER, INTENT(IN), ASYNCHRONOUS :: sendcounts(*), sdispls(*),
     recvcounts(*), rdispls(*)
     TYPE(MPI_Datatype), INTENT(IN), ASYNCHRONOUS :: sendtypes(*),
     recvtypes(*)
     TYPE(MPI_Comm), INTENT(IN) :: comm
     TYPE(MPI_Info), INTENT(IN) :: info
     TYPE(MPI_Request), INTENT(OUT) :: request
     INTEGER, OPTIONAL, INTENT(OUT) :: ierror

🦋 INPUT PARAMETERS 🦋

  • sendbuf — starting address of send buffer. the origin point, the birthplace, the beginning 🫶
  • sendcounts — integer array, where entry i specifies the number of elements to send to rank i. one number per bestie, no skipping.
  • sdispls — integer array, where entry i specifies the displacement (in bytes, offset from sendbuf) from which to send data to rank i. yes, bytes. we'll get to that, don't worry.
  • sendtypes — datatype array, where entry i specifies the datatype to use when sending data to rank i. this is the ✨special✨ part, you get to customize per rank.
  • recvcounts — integer array, where entry j specifies the number of elements to receive from rank j.
  • rdispls — integer array, where entry j specifies the displacement (in bytes, offset from recvbuf) to which data from rank j should be written.
  • recvtypes — datatype array, where entry j specifies the datatype to use when receiving data from rank j.
  • comm — communicator over which data is to be exchanged. the group chat, essentially 💬
  • info — Info (handle, persistent only). she's lowkey just vibing there unless you're doing the persistent thing.

🎀 OUTPUT PARAMETERS 🎀

  • recvbuf — address of receive buffer. where the goodies land 💖
  • request — Request (handle, non-blocking only). your little receipt, your claim ticket, your "i'ma come back for this".
  • ierror — Fortran only: error status. classic.

💅 DESCRIPTION 💅

MPI_Alltoallw is a generalized collective operation in which all processes send data to and receive data from all other processes. she's the upgraded version, no cap.

Like, it adds flexibility to MPI_Alltoallv by allowing the user to specify the datatype of individual data blocks (on top of displacement and element count). so it's not just "how much" and "where", it's also "what kind" — three layers of customization, she's doing the most and we love her for it. 🔥

Her operation can be thought of in the following way, where each process performs 2n (n being the number of processes in communicator comm) independent point-to-point communications (including communication with itself, yes, she talks to herself, relatable 🙈):

MPI_Comm_size(comm, &n);
for (i = 0, i < n; i++)
    MPI_Send(sendbuf + sdispls[i], sendcounts[i],
        sendtypes[i], i, ..., comm);
for (i = 0, i < n; i++)
    MPI_Recv(recvbuf + rdispls[i], recvcounts[i],
        recvtypes[i], i, ..., comm);

so process j sends the k-th block of its local sendbuf to process k, which places the data in the j-th block of its local recvbuf. cute little swap situation. 🔄

When a pair of processes exchanges data, each may pass different element count and datatype arguments so long as the sender specifies the same amount of data to send (in bytes) as the receiver expects to receive. so like — different shapes are fine, but the byte total has to match. it's giving ✨math✨.

Note that process i may send a different amount of data to process j than it receives from process j. ALSO, a process may send entirely different amounts and types of data to different processes in the communicator. she's not doing fairness, she's doing whatever she wants. slay. 💅

🌈 WHEN COMMUNICATOR IS AN INTER-COMMUNICATOR 🌈

When the communicator is an inter-communicator, the gather operation occurs in two phases. The data is gathered from all the members of the first group and received by all the members of the second group. Then the data is gathered from all the members of the second group and received by all the members of the first. The operation exhibits a symmetric, full-duplex behavior. she's doing both directions at once, main character energy fr fr.

The first group defines the root process. The root process uses MPI_ROOT as the value of root. All other processes in the first group use MPI_PROC_NULL as the value of root. All processes in the second group use the rank of the root process in the first group as the value of root.

When the communicator is an intra-communicator, these groups are the same, and the operation occurs in a single phase. simpler era.

🧚 USE OF IN-PLACE OPTION 🧚

When the communicator is an intracommunicator, you can perform an all-to-all operation in-place (the output buffer is used as the input buffer). Use the variable MPI_IN_PLACE as the value of sendbuf.

In this case, sendcounts, sdispls, and sendtypes are ignored. The input data of each process is assumed to be in the area where that process would receive its own contribution to the receive buffer. so your own stuff is just... already there. living in the recv area. cozy. 🛏️

⭐ NOTES ⭐

The specification of counts, types, and displacements should not cause any location to be written more than once. seriously bestie, do not overlap your writes, that's chaos and chaos is not the vibe here.

All arguments on all processes are significant. The comm argument, in particular, must describe the same communicator on all processes. everybody has to be on the same page — no rogue communicators, no cap.

The offsets of sdispls and rdispls are measured in bytes. Compare this to MPI_Alltoallv, where these offsets are measured in units of sendtype and recvtype, respectively. ⚠️ this is the trap, the little gotcha, the "why is my data in the wrong place" moment. bytes, bestie. bytes. 🫠

💖 okay that's part 1, and we are not done yet — more of this absolute queen comes next, so don't go anywhere 💖

😭 ERRORS 😭

okay bestie, real talk time 💅 — basically almost every MPI routine gives you back an error value. How? For C routines, it's literally the return value of the function. For Fortran routines? It's chillin' in the last argument. No cap.

Before that error value even comes back to you, the current MPI error handler that's attached to the communication object (think: communicator, window, file — you know the vibe) gets called. And if there's no communication object associated with the MPI call at all? Then the call is considered attached to MPI_COMM_SELF, and it calls the associated MPI error handler. Periodt.

Now here's the tea 🍵 — when MPI_COMM_SELF isn't initialized (like before MPI_Init / MPI_Init_thread, after MPI_Finalize, or when you're using the Sessions Model exclusively), the error raises the initial error handler. You can change that initial error handler by:

  • calling MPI_Comm_set_errhandler on MPI_COMM_SELF when using the World model, or
  • using the mpi_initial_errhandler CLI argument to mpiexec, or
  • passing an info key to MPI_Comm_spawn / MPI_Comm_spawn_multiple ✨

And if no other appropriate error handler has been set? Then MPI_ERRORS_RETURN gets called for MPI I/O functions, and MPI_ERRORS_ABORT gets called for all other MPI functions. wild, but that's the rule 🤷‍♀️

Open MPI ships with three predefined error handlers you can use, and they're kinda iconic:

  • MPI_ERRORS_ARE_FATAL — causes the program to abort all connected MPI processes. it's giving nuclear option 💀
  • MPI_ERRORS_ABORT — an error handler that can be invoked on a communicator, window, file, or session. When it's called on a communicator, it acts as if MPI_Abort was called on that communicator. If it's called on a window or file, it acts as if MPI_Abort was called on a communicator containing the group of processes in the corresponding window or file. If it's called on a session, it aborts only the local process. okayyy selective chaos 🫶
  • MPI_ERRORS_RETURN — returns an error code to the application. simple, polite, respectful 💖

MPI applications can also implement their own error handlers by calling these lil duos:

  • MPI_Comm_create_errhandler then MPI_Comm_set_errhandler
  • MPI_File_create_errhandler then MPI_File_set_errhandler
  • MPI_Session_create_errhandler then MPI_Session_set_errhandler or at MPI_Session_init
  • MPI_Win_create_errhandler then MPI_Win_set_errhandler
⚠️ note that MPI does not guarantee that an MPI program can continue past an error. so like… don't just assume everything's fine bestie 😭

See the MPI man page for a full list of MPI error codes 👑

See the Error Handling section of the MPI-3.1 standard for more information 📚

See also:

  • MPI_Alltoall
  • MPI_Alltoallv

2003-2026, The Open MPI Community

and that's the whole page, bestie 🫶 this was MPI_ALLTOALLW_INIT(3), dated Sep 10, 2026 — go forth and alltoallw-init responsibly, mwah 💋✨

📦 where this page comes from