cufile.h(3)

📦
version 1.9.1.3~12.4.1 · 7 versions
, in debian trixie, ubuntu questing, ubuntu resolute, ubuntu stonking

okay bestie, welcome to cufile.h(3) from the Library Functions Manual — the cuFile C APIs page, and it is a lot in the best way. 💅

✨ NAME ✨

cufile.h — cuFile C APIs. That's the whole vibe. 🫶

✨ SYNOPSIS ✨

Data Structures 🎀

  • struct CUfileDescr_t
  • struct CUfileDrvProps
  • struct CUfileError — note: cufileop status string
  • struct CUfileFSOps
  • struct CUfileIOEvents
  • struct CUfileIOParams
  • struct cufileRDMAInfo

Macros 🧚

  • CU_FILE_CUDA_ERR(status) — pulls out the CUDA error:
#define CU_FILE_CUDA_ERR(status)   ((status).cu_err)
  • CU_FILE_RDMA_REGISTER — the value is 1:
#define CU_FILE_RDMA_REGISTER   1
  • CU_FILE_RDMA_RELAXED_ORDERING — the value is (1<<1):
#define CU_FILE_RDMA_RELAXED_ORDERING   (1<<1)
  • CU_FILE_STREAM_FIXED_BUF_OFFSET — the value is 1:
#define CU_FILE_STREAM_FIXED_BUF_OFFSET   1
  • CU_FILE_STREAM_FIXED_FILE_OFFSET — the value is 2:
#define CU_FILE_STREAM_FIXED_FILE_OFFSET   2
  • CU_FILE_STREAM_FIXED_FILE_SIZE — the value is 4:
#define CU_FILE_STREAM_FIXED_FILE_SIZE   4
  • CU_FILE_STREAM_PAGE_ALIGNED_INPUTS — the value is 8:
#define CU_FILE_STREAM_PAGE_ALIGNED_INPUTS   8
  • CUFILE_ERRSTR(err) — this one is giving error-string realness:
#define CUFILE_ERRSTR(err)
    cufileop_status_error((CUfileOpError)abs((err)))
  • cuFileDriverClose — aliases straight to cuFileDriverClose_v2:
#define cuFileDriverClose   cuFileDriverClose_v2
  • IS_CUDA_ERR(status) — checks for CU_FILE_CUDA_DRIVER_ERROR:
#define IS_CUDA_ERR(status)   ((status).err ==
    CU_FILE_CUDA_DRIVER_ERROR)
  • IS_CUFILE_ERR(err) — checks (abs((err)) > CUFILEOP_BASE_ERR):
#define IS_CUFILE_ERR(err)   (abs((err)) > CUFILEOP_BASE_ERR)
  • and these are the error macros to inspect error status of type CUfileOpError — yes, that whole little note is part of the package, bestie.

Typedefs 🦋

  • typedef void * CUfileBatchHandle_t
  • typedef enum cufileBatchMode CUfileBatchMode_t
  • typedef struct CUfileDescr_t CUfileDescr_t
  • typedef enum CUfileDriverControlFlags CUfileDriverControlFlags_t
  • typedef enum CUfileDriverStatusFlags CUfileDriverStatusFlags_t
  • typedef struct CUfileDrvProps CUfileDrvProps_t
  • typedef struct CUfileError CUfileError_t — note: cufileop status string
  • typedef enum CUfileFeatureFlags CUfileFeatureFlags_t
  • typedef struct CUfileFSOps CUfileFSOps_t
  • typedef void * CUfileHandle_t — note: File handle type.
  • typedef struct CUfileIOEvents CUfileIOEvents_t
  • typedef struct CUfileIOParams CUfileIOParams_t
  • typedef enum CUfileOpcode CUfileOpcode_t
  • typedef struct cufileRDMAInfo cufileRDMAInfo_t
  • typedef enum CUFILEStatus_enum CUfileStatus_t
  • typedef struct sockaddr sockaddr_t

Enumerations 🔥

  • enum cufileBatchMode:
enum cufileBatchMode { CUFILE_BATCH = 1 }
  • enum CUfileDriverControlFlags:
enum CUfileDriverControlFlags { CU_FILE_USE_POLL_MODE = 0,
    CU_FILE_ALLOW_COMPAT_MODE = 1 }
  • enum CUfileDriverStatusFlags:
enum CUfileDriverStatusFlags { CU_FILE_LUSTRE_SUPPORTED = 0,
    CU_FILE_WEKAFS_SUPPORTED = 1, CU_FILE_NFS_SUPPORTED = 2,
    CU_FILE_GPFS_SUPPORTED = 3, CU_FILE_NVME_SUPPORTED = 4,
    CU_FILE_NVMEOF_SUPPORTED = 5, CU_FILE_SCSI_SUPPORTED = 6,
    CU_FILE_SCALEFLUX_CSD_SUPPORTED = 7, CU_FILE_NVMESH_SUPPORTED = 8,
    CU_FILE_BEEGFS_SUPPORTED = 9 }
  • enum CUfileFeatureFlags:
enum CUfileFeatureFlags { CU_FILE_DYN_ROUTING_SUPPORTED = 0,
    CU_FILE_BATCH_IO_SUPPORTED = 1, CU_FILE_STREAMS_SUPPORTED = 2,
    CU_FILE_PARALLEL_IO_SUPPORTED = 3 }
  • enum CUfileFileHandleType:
enum CUfileFileHandleType { CU_FILE_HANDLE_TYPE_OPAQUE_FD = 1,
    CU_FILE_HANDLE_TYPE_OPAQUE_WIN32 = 2,
    CU_FILE_HANDLE_TYPE_USERSPACE_FS = 3 }
  • enum CUfileOpcode:
enum CUfileOpcode { CUFILE_READ = 0, CUFILE_WRITE }
  • enum CUFILEStatus_enum:
enum CUFILEStatus_enum { CUFILE_WAITING = 0x000001, CUFILE_PENDING =
    0x000002, CUFILE_INVALID = 0x000004, CUFILE_CANCELED = 0x000008,
    CUFILE_COMPLETE = 0x0000010, CUFILE_TIMEOUT = 0x0000020,
    CUFILE_FAILED = 0x0000040 }

Functions 💖

  • CUfileError_t cuFileBatchIOCancel (CUfileBatchHandle_t batch_idp)
  • void cuFileBatchIODestroy (CUfileBatchHandle_t batch_idp)
  • CUfileError_t cuFileBatchIOGetStatus (CUfileBatchHandle_t batch_idp, unsigned min_nr, unsigned *nr, CUfileIOEvents_t *iocbp, struct timespec *timeout)
  • CUfileError_t cuFileBatchIOSetUp (CUfileBatchHandle_t *batch_idp, unsigned nr)
  • CUfileError_t cuFileBatchIOSubmit (CUfileBatchHandle_t batch_idp, unsigned nr, CUfileIOParams_t *iocbp, unsigned int flags)
  • CUfileError_t cuFileBufDeregister (const void *bufPtr_base) — deregister an already registered device or host memory from cuFile
  • CUfileError_t cuFileBufRegister (const void *bufPtr_base, size_t length, int flags) — register an existing cudaMalloced memory with cuFile to pin for GPUDirect Storage access, or register host allocated memory with cuFile.
  • CUfileError_t cuFileDriverClose (void)
  • CUfileError_t cuFileDriverClose_v2 (void) — reset the cuFile library and release the nvidia-fs driver
  • CUfileError_t cuFileDriverGetProperties (CUfileDrvProps_t *props) — gets the Driver session properties.
  • CUfileError_t cuFileDriverOpen (void) — initialize the cuFile library and open the nvidia-fs driver.
  • CUfileError_t cuFileDriverSetMaxCacheSize (size_t max_cache_size) — control parameter to set maximum GPU memory reserved per device by the library for internal buffering.
  • CUfileError_t cuFileDriverSetMaxDirectIOSize (size_t max_direct_io_size) — control parameter to set max IO size(KB) used by the library to talk to nvidia-fs driver.
  • CUfileError_t cuFileDriverSetMaxPinnedMemSize (size_t max_pinned_size) — sets maximum buffer space that is pinned in KB for use by cuFileBufRegister.
  • CUfileError_t cuFileDriverSetPollMode (bool poll, size_t poll_threshold_size) — sets whether the Read/Write APIs use polling to do IO operations.
  • CUfileError_t cuFileGetVersion (int *version)
  • void cuFileHandleDeregister (CUfileHandle_t fh) — releases a registered filehandle from cuFile
  • CUfileError_t cuFileHandleRegister (CUfileHandle_t *fh, CUfileDescr_t *descr) — cuFileHandleRegister is required, and performs extra checking that is memoized to provide increased performance on later cuFile operations.
  • ssize_t cuFileRead (CUfileHandle_t fh, void *bufPtr_base, size_t size, off_t file_offset, off_t bufPtr_offset) — read data from a registered file handle to a specified device or host memory
  • CUfileError_t cuFileReadAsync (CUfileHandle_t fh, void *bufPtr_base, size_t *size_p, off_t *file_offset_p, off_t *bufPtr_offset_p, ssize_t *bytes_read_p, CUstream stream)
  • CUfileError_t cuFileStreamDeregister (CUstream stream)
  • CUfileError_t cuFileStreamRegister (CUstream stream, unsigned flags)
  • long cuFileUseCount (void) — returns use count of cufile drivers at that moment by the process.
  • ssize_t cuFileWrite (CUfileHandle_t fh, const void *bufPtr_base, size_t size, off_t file_offset, off_t bufPtr_offset) — write data from a specified device or host memory to a registered file handle
  • CUfileError_t cuFileWriteAsync (CUfileHandle_t fh, void *bufPtr_base, size_t *size_p, off_t *file_offset_p, off_t *bufPtr_offset_p, ssize_t *bytes_written_p, CUstream stream)

Variables 🍓

  • CUfileOpError

🌸 DETAILED DESCRIPTION 🌸

cuFile C APIs.

This file contains all the C APIs to perform GPUDirect Storage supported IO operations. Like, all of them. 💅

🧚 MACRO DEFINITION DOCUMENTATION 🧚

Yep, same macro squad, now in documentation form:

  • CU_FILE_CUDA_ERR(status):
#define CU_FILE_CUDA_ERR(status)   ((status).cu_err)
  • CU_FILE_RDMA_REGISTER:
#define CU_FILE_RDMA_REGISTER   1
  • CU_FILE_RDMA_RELAXED_ORDERING:
#define CU_FILE_RDMA_RELAXED_ORDERING   (1<<1)
  • CU_FILE_STREAM_FIXED_BUF_OFFSET:
#define CU_FILE_STREAM_FIXED_BUF_OFFSET   1
  • CU_FILE_STREAM_FIXED_FILE_OFFSET:
#define CU_FILE_STREAM_FIXED_FILE_OFFSET   2
  • CU_FILE_STREAM_FIXED_FILE_SIZE:
#define CU_FILE_STREAM_FIXED_FILE_SIZE   4
  • CU_FILE_STREAM_PAGE_ALIGNED_INPUTS:
#define CU_FILE_STREAM_PAGE_ALIGNED_INPUTS   8
  • CUFILE_ERRSTR(err):
#define CUFILE_ERRSTR(err)
    cufileop_status_error((CUfileOpError)abs((err)))
  • cuFileDriverClose:
#define cuFileDriverClose   cuFileDriverClose_v2
  • IS_CUDA_ERR(status):
#define IS_CUDA_ERR(status)   ((status).err == CU_FILE_CUDA_DRIVER_ERROR)
  • IS_CUFILE_ERR(err):
#define IS_CUFILE_ERR(err)   (abs((err)) > CUFILEOP_BASE_ERR)
  • and again, these are the error macros to inspect error status of type CUfileOpError. Iconic. ✨

🎀 TYPEDEF DOCUMENTATION 🎀

  • typedef void* CUfileBatchHandle_t
  • typedef enum cufileBatchMode CUfileBatchMode_t
  • typedef struct CUfileDescr_t CUfileDescr_t
  • typedef enum CUfileDriverControlFlags CUfileDriverControlFlags_t
  • typedef enum CUfileDriverStatusFlags CUfileDriverStatusFlags_t
  • typedef struct CUfileDrvProps CUfileDrvProps_t
  • typedef struct CUfileError CUfileError_t — note: cufileop status string
  • typedef enum CUfileFeatureFlags CUfileFeatureFlags_t
  • typedef struct CUfileFSOps CUfileFSOps_t
  • typedef void* CUfileHandle_t — note: File handle type.
  • typedef struct CUfileIOEvents CUfileIOEvents_t
  • typedef struct CUfileIOParams CUfileIOParams_t
  • typedef enum CUfileOpcode CUfileOpcode_t
  • typedef struct cufileRDMAInfo cufileRDMAInfo_t
  • typedef enum CUFILEStatus_enum CUfileStatus_t
  • typedef struct sockaddr sockaddr_t

🔥 ENUMERATION TYPE DOCUMENTATION 🔥

enum cufileBatchMode

Enumerator ✨

  • CUFILE_BATCH — no extra description on the page, just the name.

enum CUfileDriverControlFlags

Enumerator ✨

  • CU_FILE_USE_POLL_MODE — use POLL mode. properties.use_poll_mode
  • CU_FILE_ALLOW_COMPAT_MODE — allow COMPATIBILITY mode. properties.allow_compat_mode

enum CUfileDriverStatusFlags

Enumerator ✨

  • CU_FILE_LUSTRE_SUPPORTED — Support for DDN LUSTRE
  • CU_FILE_WEKAFS_SUPPORTED — Support for WEKAFS
  • CU_FILE_NFS_SUPPORTED — Support for NFS
  • CU_FILE_GPFS_SUPPORTED — no description on the page, bestie
  • CU_FILE_NVME_SUPPORTED — < Support for GPFS Support for NVMe
  • CU_FILE_NVMEOF_SUPPORTED — Support for NVMeOF
  • CU_FILE_SCSI_SUPPORTED — Support for SCSI
  • CU_FILE_SCALEFLUX_CSD_SUPPORTED — Support for Scaleflux CSD
  • CU_FILE_NVMESH_SUPPORTED — Support for NVMesh Block Dev
  • CU_FILE_BEEGFS_SUPPORTED — Support for BeeGFS

enum CUfileFeatureFlags

Enumerator ✨

  • CU_FILE_DYN_ROUTING_SUPPORTED — Support for Dynamic routing to handle devices across the PCIe bridges
  • CU_FILE_BATCH_IO_SUPPORTED — Unsupported
  • CU_FILE_STREAMS_SUPPORTED — Unsupported
  • CU_FILE_PARALLEL_IO_SUPPORTED — Unsupported

enum CUfileFileHandleType

Enumerator ✨

  • CU_FILE_HANDLE_TYPE_OPAQUE_FD — Linux based fd
  • CU_FILE_HANDLE_TYPE_OPAQUE_WIN32 — Windows based handle (unsupported)
  • CU_FILE_HANDLE_TYPE_USERSPACE_FS — no description on the page

enum CUfileOpcode

Enumerator ✨

  • CUFILE_READ — no extra description on the page
  • CUFILE_WRITE — no extra description on the page

enum CUFILEStatus_enum

Enumerator ✨

  • CUFILE_WAITING
  • CUFILE_PENDING
  • CUFILE_INVALID
  • CUFILE_CANCELED
  • CUFILE_COMPLETE
  • CUFILE_TIMEOUT
  • CUFILE_FAILED

💖 FUNCTION DOCUMENTATION 💖

CUfileError_t cuFileBatchIOCancel (CUfileBatchHandle_t batch_idp)

No extra description on the page for this one — it's just the signature and main-character energy.

void cuFileBatchIODestroy (CUfileBatchHandle_t batch_idp)

Also just the signature here.

CUfileError_t cuFileBatchIOGetStatus (CUfileBatchHandle_t batch_idp, unsigned min_nr, unsigned * nr, CUfileIOEvents_t * iocbp, struct timespec * timeout)

Signature only, bestie.

CUfileError_t cuFileBatchIOSetUp (CUfileBatchHandle_t * batch_idp, unsigned nr)

Signature only. Slay.

CUfileError_t cuFileBatchIOSubmit (CUfileBatchHandle_t batch_idp, unsigned nr, CUfileIOParams_t * iocbp, unsigned int flags)

Signature only.

CUfileError_t cuFileBufDeregister (const void * bufPtr_base)

deregister an already registered device or host memory from cuFile

Parameters

  • bufPtr_base — buffer pointer to deregister

Returns

  • CU_FILE_SUCCESS on success
  • CU_FILE_INVALID_VALUE on invalid memory pointer or unregistered memory pointer

See also

  • cuFileBufRegister
  • cuFileRead
  • cuFileWrite

CUfileError_t cuFileBufRegister (const void * bufPtr_base, size_t length, int flags)

register an existing cudaMalloced memory with cuFile to pin for GPUDirect Storage access or register host allocated memory with cuFile.

Parameters:

  • bufPtr_base — the buffer pointer you allocated 🫶
  • length — the size of the memory region, counting from that bufPtr_base we just mentioned
  • flags — CU_FILE_RDMA_REGISTER

Returns:

  • CU_FILE_SUCCESS when it goes through ✨
  • CU_FILE_NVFS_DRIVER_ERROR
  • CU_FILE_INVALID_VALUE
  • CU_FILE_CUDA_ERROR — for unsupported memory type
  • CU_FILE_MEMORY_ALREADY_REGISTERED — on error
  • CU_FILE_GPU_MEMORY_PINNING_FAILED — if there's not enough pinned memory available

Note: this memory is gonna be used to perform GPU direct DMA from the supported storage.

Warning: this API is intended for usecases where the memory is used as a streaming buffer that gets reused across multiple cuFile IO operations before you call cuFileBufDeregister. iykyk 🔥

See also: cuFileBufDeregister, cuFileRead, cuFileWrite

cuFileDriverClose / cuFileDriverClose_v2

CUfileError_t cuFileDriverClose (void)
CUfileError_t cuFileDriverClose_v2 (void)

okay so these two literally reset the cuFile library and release the nvidia-fs driver 💅

Returns:

  • CU_FILE_SUCCESS on success
  • CU_FILE_DRIVER_CLOSING — if there are any active IO operations using cuFileRead or cuFileWrite

See also: cuFileDriverOpen

cuFileDriverGetProperties

CUfileError_t cuFileDriverGetProperties (CUfileDrvProps_t * props)

Gets the Driver session properties 👑

Returns:

  • CU_FILE_SUCCESS on success

See also: cuFileDriverSetPollMode, cuFileDriverSetMaxDirectIOSize, cuFileDriverSetMaxCacheSize, cuFileDriverSetMaxPinnedMemSize

cuFileDriverOpen

CUfileError_t cuFileDriverOpen (void)

Initialize the cuFile library and open the nvidia-fs driver 🚀 this is like the main character entrance of the whole API ngl

Returns:

  • CU_FILE_SUCCESS on success
  • CU_FILE_DRIVER_NOT_INITIALIZED
  • CU_FILE_DRIVER_VERSION_MISMATCH — on driver version mismatch error

See also: cuFileDriverClose

cuFileDriverSetMaxCacheSize

CUfileError_t cuFileDriverSetMaxCacheSize (size_t max_cache_size)

Control parameter to set the maximum GPU memory reserved per device by the library for its internal buffering 🎀

Parameters:

  • max_cache_size — the maximum GPU buffer space per device used for internal use, in KB

Returns:

  • CU_FILE_SUCCESS on success
  • CU_FILE_DRIVER_NOT_INITIALIZED — if the driver is not initialized
  • CU_FILE_DRIVER_VERSION_MISMATCH, CU_FILE_DRIVER_UNSUPPORTED_LIMIT — on error
Warning: this is an advanced command and should be tuned based on supported GPU memory. no cap, don't just yeet a number in there 💀

See also: cuFileDriverGetProperties

cuFileDriverSetMaxDirectIOSize

CUfileError_t cuFileDriverSetMaxDirectIOSize (size_t max_direct_io_size)

Control parameter to set the max IO size (KB) that the library uses to talk to the nvidia-fs driver 🌈

Parameters:

  • max_direct_io_size — maximum allowed direct io size in KB

Returns:

  • CU_FILE_SUCCESS on success
  • CU_FILE_DRIVER_NOT_INITIALIZED — if the driver is not initialized
  • CU_FILE_DRIVER_VERSION_MISMATCH, CU_FILE_DRIVER_UNSUPPORTED_LIMIT — on error
Warning: this is an advanced command and should be tuned based on available system memory 🫶

See also: cuFileDriverGetProperties

cuFileDriverSetMaxPinnedMemSize

CUfileError_t cuFileDriverSetMaxPinnedMemSize (size_t max_pinned_size)

Sets the maximum buffer space that is pinned, in KB, for use by cuFileBufRegister 🧚

Parameters:

  • max_pinned_size — maximum buffer space that is pinned, in KB

Returns:

  • CU_FILE_SUCCESS on success
  • CU_FILE_DRIVER_NOT_INITIALIZED — if the driver is not initialized
  • CU_FILE_DRIVER_VERSION_MISMATCH, CU_FILE_DRIVER_UNSUPPORTED_LIMIT — on error
Warning: this is an advanced command and should be tuned based on supported GPU memory 💜

See also: cuFileDriverGetProperties

cuFileDriverSetPollMode

CUfileError_t cuFileDriverSetPollMode (bool poll, size_t poll_threshold_size)

Sets whether the Read/Write APIs use polling to do IO operations ⭐ like, do we spin or do we not spin, that is the question

Parameters:

  • poll — boolean to indicate whether to use poll mode or not
  • poll_threshold_size — max IO size to use for POLLING mode, in KB

Returns:

  • CU_FILE_SUCCESS on success
  • CU_FILE_DRIVER_NOT_INITIALIZED — if the driver is not initialized
  • CU_FILE_DRIVER_VERSION_MISMATCH, CU_FILE_DRIVER_UNSUPPORTED_LIMIT — on error
Warning: this is an advanced command and should be tuned based on available system memory 🌸

See also: cuFileDriverGetProperties

cuFileGetVersion

CUfileError_t cuFileGetVersion (int * version)

Returns: the cufile library version 💖

version comes back as (1000 major + 10 minor). example: CUFILE 1.7.0 would be represented by 1070. math ate, honestly 🧮✨

Note: this is useful for applications that need to inquire the library.

Returns:

  • CU_FILE_SUCCESS on success
  • CU_FILE_INVALID_VALUE — if the input parameter is null
  • CU_FILE_DRIVER_VERSION_READ_ERROR — if the version is not available

cuFileHandleDeregister

void cuFileHandleDeregister (CUfileHandle_t fh)

releases a registered filehandle from cuFile 🦋 bye bestie

Parameters:

  • fh — CUfileHandle_t file handle

Returns: void

See also: cuFileHandleRegister

cuFileHandleRegister

CUfileError_t cuFileHandleRegister (CUfileHandle_t * fh, CUfileDescr_t * descr)

cuFileHandleRegister is required, and it performs extra checking that's memoized so that later cuFile operations go faster 🔥 (memoized!! we love a caching king)

Parameters:

  • fh — CUfileHandle_t opaque file handle for IO operations
  • descr — CUfileDescr_t file descriptor (OS agnostic)

Returns:

  • CU_FILE_SUCCESS on successful completion. fh will be updated for use in cuFileRead, cuFileWrite, cuFileHandleDeregister
  • CU_FILE_DRIVER_NOT_INITIALIZED — on failure to load driver
  • CU_FILE_IO_NOT_SUPPORTED — if filesystem is not supported
  • CU_FILE_INVALID_VALUE — if null or bad api arguments
  • CU_FILE_INVALID_FILE_OPEN_FLAG — if file is opened with unsupported modes like no O_DIRECT
  • CU_FILE_INVALID_FILE_TYPE — if filepath is not valid or is not a regular file
  • CU_FILE_HANDLE_ALREADY_REGISTERED — if file handle/descriptor is already registered

Description: cuFileHandleRegister registers the open file descriptor for use with cuFile IO operations 💅

This API will ensure that the file's descriptor is checked for GPUDirect Storage support and returns a valid file handle on CU_FILE_SUCCESS.

Note: the file needs to be opened in O_DIRECT mode to support GPUDirect Storage. do not skip this one, fr fr 😭

See also: cuFileRead, cuFileWrite, cuFileHandleDeregister

cuFileRead

ssize_t cuFileRead (CUfileHandle_t fh, void * bufPtr_base, size_t size, off_t file_offset, off_t bufPtr_offset)

read data from a registered file handle into a specified device or host memory 📖✨

Parameters:

  • fh — CUfileHandle_t opaque file handle
  • bufPtr_base — base address of buffer in device or host memory
  • size — size bytes to read
  • file_offset — file-offset from beginning of the file
  • bufPtr_offset — offset relative to the bufPtr_base pointer to read into.

Returns:

  • size of bytes successfully read
  • -1 on error, in which case errno is set to indicate filesystem errors.
  • all other errors will return a negative integer value of CUfileOpError enum value.

Note: if the bufPtr is not registered with cuFileBufRegister, the data will be buffered through preallocated pinned buffers if needed 🫶

This is useful for applications that need to perform IO to unaligned file offsets and/or size. This is also recommended for cases where the BAR1 memory size is smaller than the size of the allocated memory 👑

See also: cuFileBufRegister, cuFileHandleRegister, cuFileWrite

cuFileReadAsync

CUfileError_t cuFileReadAsync (CUfileHandle_t fh, void * bufPtr_base, size_t * size_p, off_t * file_offset_p, off_t * bufPtr_offset_p, ssize_t * bytes_read_p, CUstream stream)

Parameters:

  • fh — The cuFile handle for the file.
  • bufPtr_base — base address of buffer in device or host memory
  • size_p — pointer to size bytes to read
Note: *size_p — if the size is not known at the time of submission, then you must provide the max possible size for the I/O request. plan ahead bestie 📋

Parameters:

  • file_offset_p — pointer to file-offset from beginning of the file
  • bufPtr_offset_p — pointer to offset relative to the bufPtr_base pointer to read into.
  • bytes_read_p — pointer to the number of bytes that were successfully read.
  • CUstream stream — cuda stream for the operation.

Returns:

  • size of bytes successfully read in *bytes_read_p
  • -1 on error, in which case errno is set to indicate filesystem errors.
  • all other errors will return a negative integer value of CUfileOpError enum value.

Note: if the bufPtr_base is not registered with cuFileBufRegister, the data will be buffered through preallocated pinned buffers.

This is useful for applications that need to perform IO to unaligned file offsets and/or size. This is also recommended for cases where the BAR1 memory size is smaller than the size of the allocated memory 🌈

If the stream is registered with cuFileStreamRegister, the IO setup and teardown overhead will be reduced ✨ (less overhead = more slay).

on cuda stream errors, the user must call cuFileStreamDeregister to release any outstanding cuFile resources for the stream ⚠️ like, actually must.

See also: cuFileBufRegister, cuFileHandleRegister, cuFileRead, cuFileStreamRegister, cuFileStreamDeregister

cuFileStreamDeregister

CUfileError_t cuFileStreamDeregister (CUstream stream)

Parameters:

  • CUstream stream — cuda stream for the operation.

Note:

  • deallocates resources used by previous cuFile asynchronous operations for the cuda stream 🧹
  • highly recommend to call after cuda stream errors to release any outstanding cuFile resources for this stream
  • must be called before cuStreamDestroy call for the specified stream. 👈 this one's non-negotiable
  • This is useful for applications that need to perform IO to unaligned file offsets and/or size. This is also recommended for cases where the BAR1 memory size is smaller than the size of the allocated memory.

Returns:

  • CU_FILE_SUCCESS on success
  • CU_FILE_DRIVER_NOT_INITIALIZED — if the driver is not initialized
  • CU_FILE_INVALID_VALUE — if the stream is invalid

See also: cuFileReadAsync, cuFileWriteAsync, cuFileStreamRegister

cuFileStreamRegister

CUfileError_t cuFileStreamRegister (CUstream stream, unsigned flags)

Parameters:

  • CUstream stream — cuda stream for the operation.
  • flags — for the stream to improve the stream execution of IO based on input parameters.

Note: supported FLAGS are 🚩

  • CU_FILE_STREAM_FIXED_BUF_OFFSET — buffer pointer offset is set at submission time
  • CU_FILE_STREAM_FIXED_FILE_OFFSET — file offset is set at submission time
  • CU_FILE_STREAM_FIXED_FILE_SIZE — the file size is locked in at submission time, no take-backs 💅
  • CU_FILE_STREAM_PAGE_ALIGNED_INPUTS — size, offset and buffer offset are all 4k aligned, iconic behavior tbh ✨

...and bestie, that call is what allocates resources needed to support cuFile operations asynchronously for the cuda stream 🦋

This is genuinely useful for applications that need to perform IO to unaligned file offsets and/or size. It's also the recommended move for cases where the BAR1 memory size is smaller than the size of the allocated memory — yeah, that scenario, we love a practical queen 👑

Returns:

  • CU_FILE_SUCCESS on success 🎀
  • CU_FILE_DRIVER_NOT_INITIALIZED if the driver is not initialized
  • CU_FILE_INVALID_VALUE if the stream is invalid

See also:

  • cuFileReadAsync
  • cuFileWriteAsync
  • cuFileStreamDeregister

🌸 long cuFileUseCount (void)

Okay so this one is literally just: returns use count of cufile drivers at that moment by the process. That's it. Snapshot of the vibe check on how many cufile drivers are in use by your process right this second 🔥

🦋 ssize_t cuFileWrite (CUfileHandle_t fh, const void * bufPtr_base, size_t size, off_t file_offset, off_t bufPtr_offset)

Write data from a specified device or host memory to a registered file handle. Get it, girl, write it out 💜

Parameters:

  • fh — CUfileHandle_t opaque file handle
  • bufPtr_base — base address of buffer in device or host memory
  • size — size bytes to write
  • file_offset — file-offset from beginning of the file
  • bufPtr_offset — offset relative to the bufPtr_base pointer to write from.

Returns:

  • size of bytes successfully written ✅
  • -1 on error, in which case errno is set to indicate filesystem errors. 😭
  • all other errors will return a negative integer value of CUfileOpError enum value.
Note: If the bufPtr is not registered with cuFileBufRegister, the data will be buffered through preallocated pinned buffers if needed.

This is useful for applications that need to perform IO to unaligned file offsets and/or size. It's also recommended for cases where the BAR1 memory size is smaller than the size of the allocated memory — same energy as before, no cap 🌈

See also:

  • cuFileBufRegister
  • cuFileHandleRegister
  • cuFileRead

⭐ CUfileError_t cuFileWriteAsync (CUfileHandle_t fh, void * bufPtr_base, size_t * size_p, off_t * file_offset_p, off_t * bufPtr_offset_p, ssize_t * bytes_written_p, CUstream stream)

Parameters:

  • fh — The cuFile handle for the file.
  • bufPtr_base — base address of buffer in device or host memory
  • size_p — pointer to size bytes to write.
Note: *size_p — if the size is not known at the time of submission, then must provide the max possible size for I/O request. So like, guess big, bestie 🙈

Parameters: (yes we're back to parameters, the doc did the split thing, we respect it)

  • file_offset_p — pointer to file-offset from beginning of the file
  • bufPtr_offset_p — pointer to offset relative to the bufPtr_base pointer to write from.
  • bytes_written_p — pointer to the number of bytes that were successfully written.
  • CUstream — cuda stream for the operation.

Returns:

  • size of bytes successfully written in *bytes_written_p
  • -1 on error, in which case errno is set to indicate filesystem errors. 💀
  • all other errors will return a negative integer value of CUfileOpError enum value.
Note: If the bufPtr_base is not registered with cuFileBufRegister, the data will be buffered through preallocated pinned buffers.

This is useful for applications that need to perform IO to unaligned file offsets and/or size. This is also recommended for cases where the BAR1 memory size is smaller than the size of the allocated memory 🫶

If the stream is registered with cuFileStreamRegister prior to this call, the IO setup and teardown overhead will be reduced — free performance, we ate 🍓

on cuda stream errors, the user must call cuFileStreamDeregister to release any outstanding cuFile resources for the stream. Don't skip this one, fr fr, it's giving leak if you ignore it 😭

See also:

  • cuFileBufRegister
  • cuFileHandleRegister
  • cuFileWrite
  • cuFileStreamRegister
  • cuFileStreamDeregister

🎀 Variable Documentation 🎀

CUfileOpError

That's the one variable doc, and honestly? She speaks for herself 👑

👑 Author 👑

Generated automatically by Doxygen for GPUDirect Storage from the source code. A whole bot wrote this and we're out here annotating it like it's a group chat, love that for us 💖

And that's a wrap on cufile.h(3)! 🌟 This lil beauty came to you from GPUDirect Storage, dated Wed Mar 13 2024, page cufile.h(3) — go forth and async your IO, bestie, you're literally the main character now 🧚💅

📦 where this page comes from