OPENSHMEM(3)

📦
version 5.0.10 · 4 versions
, in ubuntu stonking

🌈 OPENSHMEM(3) 🌈

Open MPI presents… the OpenSHMEM programming model, but explained by someone who actually loves you 💖 This is part 1 of 2, so bestie do not close the tab, there's more coming.

🎀 NAME 🎀

intro_shmem — Introduction to the OpenSHMEM programming model

💖 DESCRIPTION 💖

okay so. The SHMEM programming model is literally a bundle of library routines that hand you low-latency, high-bandwidth communication for highly parallelized, scalable programs. That's the whole point. 🌸

The routines in the OpenSHMEM application programming interface (API) give you a whole programming model for exchanging data between cooperating parallel processes. And the programs you end up writing? Style-wise they're giving very much Message Passing Interface (MPI) energy. But here's the twist: you can use the SHMEM API either all on its own, or mix it with MPI routines in the same parallel program. Pick your fighter. 💅

An OpenSHMEM program is SPMD (single program, multiple data) in style — that's the vibe. The SHMEM processes, which we call processing elements, or PEs if you're in a hurry, all start at the same time and they all run the same program. Usually the PEs do their computation on their own subdomains of the larger problem, and then every so often they talk to the other PEs to exchange information that the next computation phase depends on. 🫶

The OpenSHMEM routines exist to minimize the overhead associated with data transfer requests, maximize bandwidth and minimize data latency. And what even is data latency? It's the period of time that starts when a PE initiates a transfer of data and ends when a PE can use the data. Iconic definition, no notes.

OpenSHMEM routines support remote data transfer through:

  • put operations — which transfer data to a different PE
  • get operations — which transfer data from a different PE
  • remote pointers — which allow direct references to data objects owned by another PE

Other operations supported are collective broadcast and reduction, barrier synchronization, and atomic memory operations. An atomic memory operation is an atomic read-and-update operation, such as a fetch-and-increment, on a remote or local data object. periodt. 💅

🔥 OPENSHMEM ROUTINES 🔥

This section lists the significant OpenSHMEM message-passing routines. It's a long one. We're listing them all, we're not skipping, we're not summarising, we're committed. 💪

👯 PE queries

🍓 Elemental data put routines

  • C/C++ only:
    • shmem_double_p <#shmem-double-p>(3)
    • shmem_float_p <#shmem-float-p>(3)
    • shmem_int_p <#shmem-int-p>(3)
    • shmem_long_p <#shmem-long-p>(3)
    • shmem_short_p <#shmem-short-p>.*(3)

🌸 Block data put routines

  • C/C++ and Fortran:
    • shmem_put32 <#shmem-put32>(3)
    • shmem_put64 <#shmem-put64>(3)
    • shmem_put128 <#shmem-put128>(3)
  • C/C++ only:
    • shmem_double_put <#shmem-double-put>(3)
    • shmem_float_put <#shmem-float-put>(3)
    • shmem_int_put <#shmem-int-put>(3)
    • shmem_long_put <#shmem-long-put>(3)
    • shmem_short_put <#shmem-short-put>.*(3)
  • Fortran only:
    • shmem_complex_put(3)
    • shmem_integer_put(3)
    • shmem_logical_put(3)
    • shmem_real_put(3)

🦋 Elemental data get routines

  • C/C++ only:
    • shmem_double_g <#shmem-double-g>(3)
    • shmem_float_g <#shmem-float-g>(3)
    • shmem_int_g <#shmem-int-g>(3)
    • shmem_long_g <#shmem-long-g>(3)
    • shmem_short_g <#shmem-short-g>(3)

🎀 Block data get routines

  • C/C++ and Fortran:
    • shmem_get32 <#shmem-get32>(3)
    • shmem_get64 <#shmem-get64>(3)
    • shmem_get128 <#shmem-get128>(3)
  • C/C++ only:
    • shmem_double_get <#shmem-double-get>(3)
    • shmem_float_get <#shmem-float-get>(3)
    • shmem_int_get <#shmem-int-get>(3)
    • shmem_long_get <#shmem-long-get>(3)
    • shmem_short_get <#shmem-short-get>(3)
  • Fortran only:
    • shmem_complex_get(3)
    • shmem_integer_get(3)
    • shmem_logical_get(3)
    • shmem_real_get(3)

✨ Strided put routines

  • C/C++ and Fortran:
    • shmem_iput32 <#shmem-iput32>(3)
    • shmem_iput64 <#shmem-iput64>(3)
    • shmem_iput128 <#shmem-iput128>(3)
  • C/C++ only:
    • shmem_double_iput <#shmem-double-iput>(3)
    • shmem_float_iput <#shmem-float-iput>(3)
    • shmem_int_iput <#shmem-int-iput>(3)
    • shmem_long_iput <#shmem-long-iput>(3)
    • shmem_short_iput <#shmem-short-iput>(3)
  • Fortran only:
    • shmem_complex_iput(3)
    • shmem_integer_iput(3)
    • shmem_logical_iput(3)
    • shmem_real_iput(3)

💜 Strided get routines

  • C/C++ and Fortran:
    • shmem_iget32 <#shmem-iget32>(3)
    • shmem_iget64 <#shmem-iget64>(3)
    • shmem_iget128 <#shmem-iget128>(3)
  • C/C++ only:
    • shmem_double_iget <#shmem-double-iget>(3)
    • shmem_float_iget <#shmem-float-iget>(3)
    • shmem_int_iget <#shmem-int-iget>(3)
    • shmem_long_iget <#shmem-long-iget>(3)
    • shmem_short_iget <#shmem-short-iget>(3)
  • Fortran only:
    • shmem_complex_iget(3)
    • shmem_integer_iget(3)
    • shmem_logical_iget(3)
    • shmem_real_iget(3)

⏳ Point-to-point synchronization routines

  • C/C++ only:
    • shmem_int_wait <#shmem-int-wait>(3)
    • shmem_int_wait_until <#shmem-int-wait-until>(3)
    • shmem_long_wait <#shmem-long-wait>(3)
    • shmem_long_wait_until <#shmem-long-wait-until>(3)
    • shmem_longlong_wait <#shmem-longlong-wait>(3)
    • shmem_longlong_wait_until <#shmem-longlong-wait-until>(3)
    • shmem_short_wait <#shmem-short-wait>(3)
    • shmem_short_wait_until <#shmem-short-wait-until>(3)
  • Fortran:
    • shmem_int4_wait(3)
    • shmem_int4_wait_until(3)
    • shmem_int8_wait(3)
    • shmem_int8_wait_until(3)

🚧 Barrier synchronization routines

  • C/C++ and Fortran:
    • shmem_barrier_all <#shmem-barrier-all>(3)
    • shmem_barrier <#shmem-barrier>(3)

⚛️ Atomic memory fetch-and-operate (fetch-op) routines

  • C/C++ and Fortran:
    • shmem_swap <#shmem-swap>

🧮 Reduction routines

  • C/C++ only:
    • shmem_int_and_to_all <#shmem-int-and-to-all>(3)
    • shmem_long_and_to_all <#shmem-long-and-to-all>(3)
    • shmem_longlong_and_to_all <#shmem-longlong-and-to-all>(3)
    • shmem_short_and_to_all <#shmem-short-and-to-all>(3)
    • shmem_double_max_to_all <#shmem-double-max-to-all>(3)
    • shmem_float_max_to_all <#shmem-float-max-to-all>(3)
    • shmem_int_max_to_all <#shmem-int-max-to-all>(3)
    • shmem_long_max_to_all <#shmem-long-max-to-all>(3)
    • shmem_longlong_max_to_all <#shmem-longlong-max-to-all>(3)
    • shmem_short_max_to_all <#shmem-short-max-to-all>(3)
    • shmem_double_min_to_all <#shmem-double-min-to-all>(3)
    • shmem_float_min_to_all <#shmem-float-min-to-all>(3)
    • shmem_int_min_to_all <#shmem-int-min-to-all>(3)
    • shmem_long_min_to_all <#shmem-long-min-to-all>(3)
    • shmem_longlong_min_to_all <#shmem-longlong-min-to-all>(3)
    • shmem_short_min_to_all <#shmem-short-min-to-all>(3)
    • shmem_double_sum_to_all <#shmem-double-sum-to-all>(3)
    • shmem_float_sum_to_all <#shmem-float-sum-to-all>(3)
    • shmem_int_sum_to_all <#shmem-int-sum-to-all>(3)
    • shmem_long_sum_to_all <#shmem-long-sum-to-all>(3)
    • shmem_longlong_sum_to_all <#shmem-longlong-sum-to-all>(3)
    • shmem_short_sum_to_all <#shmem-short-sum-to-all>(3)
    • shmem_double_prod_to_all <#shmem-double-prod-to-all>(3)
    • shmem_float_prod_to_all <#shmem-float-prod-to-all>(3)
    • shmem_int_prod_to_all <#shmem-int-prod-to-all>(3)
    • shmem_long_prod_to_all <#shmem-long-prod-to-all>(3)
    • shmem_longlong_prod_to_all <#shmem-longlong-prod-to-all>(3)
    • shmem_short_prod_to_all <#shmem-short-prod-to-all>(3)
    • shmem_int_or_to_all <#shmem-int-or-to-all>(3)
    • shmem_long_or_to_all <#shmem-long-or-to-all>(3)
    • shmem_longlong_or_to_all <#shmem-longlong-or-to-all>(3)
    • shmem_short_or_to_all <#shmem-short-or-to-all>(3)
    • shmem_int_xor_to_all <#shmem-int-xor-to-all>(3)
    • shmem_long_xor_to_all <#shmem-long-xor-to-all>(3)
    • shmem_longlong_xor_to_all <#shmem-longlong-xor-to-all>(3)
    • shmem_short_xor_to_all <#shmem-short-xor-to-all>(3)
  • Fortran only:
    • shmem_int4_and_to_all(3)
    • shmem_int8_and_to_all(3)
    • shmem_real4_max_to_all(3)
    • shmem_real8_max_to_all(3)
    • shmem_int4_max_to_all(3)
    • shmem_int8_max_to_all(3)
    • shmem_real4_min_to_all(3)
    • shmem_real8_min_to_all(3)
    • shmem_int4_min_to_all(3)
    • shmem_int8_min_to_all(3)
    • shmem_real4_sum_to_all(3)
    • shmem_real8_sum_to_all(3)
    • shmem_int4_sum_to_all(3)
    • shmem_int8_sum_to_all(3)
    • shmem_real4_prod_to_all(3)
    • shmem_real8_prod_to_all(3)
    • shmem_int4_prod_to_all(3)
    • shmem_int8_prod_to_all(3)
    • shmem_int4_or_to_all(3)
    • shmem_int8_or_to_all(3)
    • shmem_int4_xor_to_all(3)
    • shmem_int8_xor_to_all(3)

📢 Broadcast routines

  • C/C++ and Fortran:
    • shmem_broadcast32 <#shmem-broadcast32>(3)
    • shmem_broadcast64 <#shmem-broadcast64>(3)

🧹 Cache management routines

  • C/C++ and Fortran:
    • shmem_udcflush <#shmem-udcflush>(3)
    • shmem_udcflush_line <#shmem-udcflush-line>(3)

🪡 Byte-granularity block put routines

  • C/C++ and Fortran
    • shmem_putmem <#shmem-putmem>(3)
    • shmem_getmem <#shmem-getmem>(3)
  • Fortran only:
    • shmem_character_put(3)
    • shmem_character_get(3)

🧺 Collect routines

  • C/C++ and Fortran:
    • shmem_collect32 <#shmem-collect32>(3)
    • shmem_collect64 <#shmem-collect64>(3)
    • shmem_fcollect32 <#shmem-fcollect32>(3)
    • shmem_fcollect64 <#shmem-fcollect64>(3)

⚛️ Atomic memory fetch-and-operate (fetch-op) routines

  • C/C++ only:
    • shmem_double_swap <#shmem-double-swap>(3)

...and the routine list keeps going, bestie 💖 here's the rest of them, same energy:

⚛️ Atomic memory operation routines ⚛️

  • Fortran only:
    • shmem_int4_add(3)
    • shmem_int4_inc(3)

🧭 Remote memory pointer function 🧭

  • C/C++ and Fortran (everyone's invited, periodt):

🎀 Reduction routines 🎀

🔍 Accessibility query routines 🔍

🌸 SYMMETRIC DATA OBJECTS 🌸

okay so this is lowkey the whole personality of OpenSHMEM. Because OpenSHMEM is SPMD at its core, it's got this whole concept of symmetric data objects — these are arrays or variables that exist with the same size, same type, and same relative address on all PEs. Another name for them is "remotely accessible data objects", and honestly that name eats. 🫶

In the interface definitions for OpenSHMEM data transfer routines, one or more of the parameters are typically required to be symmetric or remotely accessible. That's just how she is.

So which data objects are symmetric? These ones, bestie:

  • Fortran data objects in common blocks or with the SAVE attribute. These data objects must not be defined in a dynamic shared object (DSO).
  • Non-stack C and C++ variables. These data objects must not be defined in a DSO.
  • Fortran arrays allocated with shpalloc(3)
  • C and C++ data allocated by shmalloc(3)

🎉 COLLECTIVE ROUTINES 🎉

Some SHMEM routines, for example shmem_broadcast(3) and shmem_float_sum_to_all(3), are classified as collective routines because they distribute work across a set of PEs. The catch? They must be called concurrently by all PEs in the active set defined by the PE_start, logPE_stride, PE_size argument triplet. No stragglers, no solo missions, it's giving group project energy 💅

The following man pages describe the OpenSHMEM collective routines:

  • shmem_and(3)
  • shmem_barrier(3)
  • shmem_broadcast(3)
  • shmem_collect(3)
  • shmem_max(3)
  • shmem_min(3)
  • shmem_or(3)
  • shmem_prod(3)
  • shmem_sum(3)
  • shmem_xor(3)

🧚 USING THE SYMMETRIC WORK ARRAY, PSYNC 🧚

Multiple pSync arrays are often needed if a particular PE calls an OpenSHMEM collective routine twice without intervening barrier synchronization. Like, chaos happens if some PEs in the active set for call 2 rock up to call 2 before processing of call 1 is complete by all PEs in the call 1 active set. 😭 You can use shmem_barrier(3) or shmem_barrier_all(3) to perform a barrier synchronization between consecutive calls to OpenSHMEM collective routines — that's your little reset button.

There are two special cases, and they're iconic:

  • The shmem_barrier(3) routine allows the same pSync array to be used on consecutive calls as long as the active PE set does not change.
  • If the same collective routine is called multiple times with the same active set, the calls may alternate between two pSync arrays. The SHMEM routines guarantee that a first call is completely finished by all PEs by the time processing of a third call begins on any PE.

And because the SHMEM routines restore pSync to its original contents, multiple calls that use the same pSync array do not require that pSync be reinitialized after the first call. We love a self-cleaning queen. ✨

💻 SHMEM ENVIRONMENT VARIABLES 💻

This section lists the significant SHMEM environment variables. Screenshot these, bestie:

  • SMA_VERSION — print the library version at start-up.
  • SMA_INFO — print helpful text about all these environment variables.
  • SMA_SYMMETRIC_SIZE — number of bytes to allocate for the symmetric heap.
  • SMA_DEBUG — enable debugging messages.
listen up 🚨 The first call to SHMEM must be start_pes(3). That routine initializes the SHMEM runtime. Calling any other SHMEM routines beforehand has undefined behavior. And multiple calls to this routine are not allowed. One shot only, no cap.

🌈 COMPILING AND RUNNING OPENSHMEM PROGRAMS 🌈

ngl, the OpenSHMEM specification is silent about how OpenSHMEM programs are compiled, linked and run. So this section just shows some examples of how wrapper programs could be utilized to compile and launch applications. The commands are styled after wrapper programs found in many MPI implementations. Cute, right?

The following sample command line demonstrates running an OpenSHMEM Program using a wrapper script (oshrun in this case):

  • C/C++:
oshcc c_program.c
  • FORTRAN:
oshfort fortran_program.f

The following sample command line demonstrates running an OpenSHMEM Program assuming that the library provides a wrapper script for such purpose (named oshrun for this example):

oshrun -n 32 ./a.out

💜 EXAMPLES 💜

Example 1: The following Fortran OpenSHMEM program directs all PEs to sum simultaneously the numbers in the VALUES variable across all PEs:

PROGRAM REDUCTION
  REAL VALUES, SUM
  COMMON /C/ VALUES
  REAL WORK

  CALL START_PES(0)
  VALUES = MY_PE()
  CALL SHMEM_BARRIER_ALL ! Synchronize all PEs
  SUM = 0.0
  DO I = 0, NUM_PES()-1
    CALL SHMEM_REAL_GET(WORK, VALUES, 1, I) ! Get next value
    SUM = SUM + WORK                ! Sum it
  ENDDO
  PRINT *, 'PE ', MY_PE(), ' COMPUTED SUM=', SUM
  CALL SHMEM_BARRIER_ALL
END

Example 2: The following C OpenSHMEM program transfers an array of 10 longs from PE 0 to PE 1:

#include <mpp/shmem.h>

main() {
  long source[10] = { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 };
  static long target[10];

  shmem_init();
  if (shmem_my_pe() == 0) {
    /* put 10 elements into target on PE 1 */
    shmem_long_put(target, source, 10, 1);
  }
  shmem_barrier_all(); /* sync sender and receiver */
  if (shmem_my_pe() == 1)
    printf("target[0] on PE %d is %d\n", shmem_my_pe(), target[0]);
}

👀 See also 👀

The following man pages also contain information on OpenSHMEM routines. See the specific man pages for implementation information.

2003-2026, The Open MPI Community

okay that's the whole page, bestie, we ate and left no crumbs 🍓 This icon is brought to you by The Open MPI Community, copyright 2003-2026, and the page itself is dated Sep 10, 2026 — go forth and shmem_barrier_all(), mwah 💖🎀🦋

📦 where this page comes from