AI/rocm-systems
2
0
Forc 0
Cód Saincheisteanna Iarratais Tarraingthe Gníomhartha Pacáistí Tionscadail Scaoileann Vicí Gníomhaíocht

2.22.3-1

Brabhsáil Foinse 
Rework core for NVIDIA Trusted Computing
 * Compress work structs so that they are shared between channels
 * Utilize the full amount of kernel argument space permitted (4k)
   before resorting to work fifo.
 * Rework the task preprocessing phase.
 * Use a separate abortDevFlag which is kept in sync with abortFlag
   using cudaMemcpy operations.
 * Rename src/include/align.h to src/include/bitops.h

Add lazy connection establishment for collective operations
 * Move buffer allocation and connection establishment to the first
   collective operation using that algorithm.
 * Accelerate init time and reduce memory usage.
 * Avoid allocating NVLS buffers if all calls are registered.
 * Compute algo/proto in ncclLaunchCollTasksInfo early on.
 * Connect peers in ncclCollPreconnectFunc if not connected already.
 * Also move shared buffer creation to the first send/recv call.

Accelerate intra-node NVLink detection
 * Make each rank only detect NVLinks attached to its GPU.
 * Fuse XMLs to reconstruct the full NVLink topology

Add init profiling to report time spend in different init phases.
 * Report timings of bootstrap, allgather, search, connect, etc.
 * Add new "PROFILE" category for NCCL_DEBUG_SUBSYS.

Add support for PCI p2p on split PCI switches
 * Detect split PCI switches through a kernel module exposing
   switch information.
 * Update the topology XML and graph to add those inter-switch
   connections.

Add cost estimation API
 * Add a new ncclGroupEndSimulate primitive to return the estimated
   time a group would take.

Net/IB: Add separate traffic class for fifo messages
 * Add NCCL_IB_FIFO_TC to control the traffic class of fifo messages
   independently from NCCL_IB_TC.
   Merges PR #1194

Net/IB: Add support for IB router
 * Use flid instead of lid if subnets do not match
 * Warn if flid is 0

Optimizations and fixes for device network offload (unpack)
 * Double the default number of channels
 * Cache netDeviceType
 * Fix save/increment head logic to enable Tree support.

Support ncclGroupStart/End for ncclCommAbort/Destroy
 * Allow Abort/Destroy to be called within a group when managing
   multiple GPUs with a single process.

Improve Tuner API
 * Provide to the plugin the original cost table so that the plugin
   can leave unknown or disabled algo/proto combinations untouched.
 * Remove nvlsSupport and collnetSupport.

Do not print version to stdout when using a debug file
 * Also print version from all processes with INFO debug level.
   Fixes issue #1271

Fix clang warnings in NVTX headers
 * Update NVTX headers to the latest version
   Fixes issue #1270

Disable port fusion in heterogeneous systems
 * Do not fuse ports if a mix of multi-port and single port are detected.

Fix NVLS graphs search for dual NICs.
 * Fix NVLS graph search when we have more than one NIC per GPU.

Fix crash with collnetDirect
 * Add separate graph search for collnetDirect, testing alltoall paths
   and working similarly to the NVLS search.

Fix hang when nodes have different CPU types
 * Add the CPU type to the rank peer info.
 * Align all ranks on the CPU type after the first allgather.
 * Only use the aligned CPU type for all tuning operations.
   Fixes issue #1136
   Fixes issue #1184

Fix performance of registered send/recv operations
 * Allow for single full size operations
 * Add INFO to confirm the registration of send/recv buffers.

Move all sync ops to finalize stage
 * Ensure ncclCommDestroy is non-blocking if ncclCommFinalize has
   been called.

Improve error reporting during SHM segment creation

Improve support of various compilers
   Merges PR #1177
   Merges PR #1228

Allow net and tuner plugins to be statically linked
 * Search for ncclNet or ncclTuner symbols in the main binary.
   Merges PR #979

Plugin examples includes cleanup
 * Harmonize err.h and common.h usage.
 * Add mixed plugin with both net and tuner.
Tá an tiomantas seo le fáil i:
Sylvain Jeaugey
2024-06-11 01:28:01 -07:00
tuismitheoir 529ee691c3
tiomantas 178b6b7590
D'athraigh 115 comhad le 8672 breiseanna agus 4403 scriosta
Íoslódáil an comhad paiste Íoslódáil Comhad Diff Leathnaigh gach comhaid Laghdaigh gach comhaid
src/include/align.h
-47
Féach ar an gComhad
@@ -1,47 +0,0 @@
/*************************************************************************
* Copyright (c) 2015-2019, NVIDIA CORPORATION. All rights reserved.
*
* See LICENSE.txt for license information
************************************************************************/
#ifndef NCCL_ALIGN_H_
#define NCCL_ALIGN_H_
#define DIVUP(x, y) \
(((x)+(y)-1)/(y))
#define ROUNDUP(x, y) \
(DIVUP((x), (y))*(y))
#define ALIGN_POWER(x, y) \
((x) > (y) ? ROUNDUP(x, y) : ((y)/((y)/(x))))
#define ALIGN_SIZE(size, align) \
size = ((size + (align) - 1) / (align)) * (align);
#if !__CUDA_ARCH__
#ifndef __host__
#define __host__
#endif
#ifndef __device__
#define __device__
#endif
#endif
template<typename X, typename Y, typename Z = decltype(X()+Y())>
__host__ __device__ constexpr Z divUp(X x, Y y) {
return (x+y-1)/y;
}
template<typename X, typename Y, typename Z = decltype(X()+Y())>
__host__ __device__ constexpr Z roundUp(X x, Y y) {
return (x+y-1) - (x+y-1)%y;
}
// assumes second argument is a power of 2
template<typename X, typename Z = decltype(X()+int())>
__host__ __device__ constexpr Z alignUp(X x, int a) {
return (x+a-1) & Z(-a);
}
#endif
src/include/alloc.h
+73 -52
Féach ar an gComhad
@@ -9,7 +9,7 @@
#include "nccl.h"
#include "checks.h"
#include "align.h"
#include "bitops.h"
#include "utils.h"
#include "p2p.h"
#include <sys/mman.h>
@@ -19,18 +19,25 @@
uint64_t clockNano(); // from utils.h with which we have a circular dependency
template<typename T>
constexpr size_t ncclSizeOfT() { return sizeof(T); }
template<>
constexpr size_t ncclSizeOfT<void>() { return 1; }
template <typename T>
ncclResult_t ncclCudaHostCallocDebug(T** ptr, size_t nelem, const char *filefunc, int line) {
ncclResult_t result = ncclSuccess;
cudaStreamCaptureMode mode = cudaStreamCaptureModeRelaxed;
*ptr = nullptr;
CUDACHECK(cudaThreadExchangeStreamCaptureMode(&mode));
CUDACHECKGOTO(cudaHostAlloc(ptr, nelem*sizeof(T), cudaHostAllocMapped), result, finish);
memset(*ptr, 0, nelem*sizeof(T));
if (nelem > 0) {
CUDACHECKGOTO(cudaHostAlloc(ptr, nelem*ncclSizeOfT<T>(), cudaHostAllocMapped), result, finish);
memset(*ptr, 0, nelem*ncclSizeOfT<T>());
}
finish:
CUDACHECK(cudaThreadExchangeStreamCaptureMode(&mode));
if (*ptr == nullptr) WARN("Failed to CUDA host alloc %ld bytes", nelem*sizeof(T));
INFO(NCCL_ALLOC, "%s:%d Cuda Host Alloc Size %ld pointer %p", filefunc, line, nelem*sizeof(T), *ptr);
if (*ptr == nullptr && nelem > 0) WARN("Failed to CUDA host alloc %ld bytes", nelem*ncclSizeOfT<T>());
INFO(NCCL_ALLOC, "%s:%d Cuda Host Alloc Size %ld pointer %p", filefunc, line, nelem*ncclSizeOfT<T>(), *ptr);
return result;
}
#define ncclCudaHostCalloc(...) ncclCudaHostCallocDebug(__VA_ARGS__, __FILE__, __LINE__)
@@ -42,14 +49,18 @@ inline ncclResult_t ncclCudaHostFree(void* ptr) {
template <typename T>
ncclResult_t ncclCallocDebug(T** ptr, size_t nelem, const char *filefunc, int line) {
void* p = malloc(nelem*sizeof(T));
if (p == NULL) {
WARN("Failed to malloc %ld bytes", nelem*sizeof(T));
return ncclSystemError;
if (nelem > 0) {
void* p = malloc(nelem*ncclSizeOfT<T>());
if (p == NULL) {
WARN("Failed to malloc %ld bytes", nelem*ncclSizeOfT<T>());
return ncclSystemError;
}
//INFO(NCCL_ALLOC, "%s:%d malloc Size %ld pointer %p", filefunc, line, nelem*ncclSizeOfT<T>(), p);
memset(p, 0, nelem*ncclSizeOfT<T>());
*ptr = (T*)p;
} else {
*ptr = NULL;
}
//INFO(NCCL_ALLOC, "%s:%d malloc Size %ld pointer %p", filefunc, line, nelem*sizeof(T), p);
memset(p, 0, nelem*sizeof(T));
*ptr = (T*)p;
return ncclSuccess;
}
#define ncclCalloc(...) ncclCallocDebug(__VA_ARGS__, __FILE__, __LINE__)
@@ -60,16 +71,16 @@ ncclResult_t ncclRealloc(T** ptr, size_t oldNelem, size_t nelem) {
if (nelem == oldNelem) return ncclSuccess;
T* oldp = *ptr;
T* p = (T*)malloc(nelem*sizeof(T));
T* p = (T*)malloc(nelem*ncclSizeOfT<T>());
if (p == NULL) {
WARN("Failed to malloc %ld bytes", nelem*sizeof(T));
WARN("Failed to malloc %ld bytes", nelem*ncclSizeOfT<T>());
return ncclSystemError;
}
memcpy(p, oldp, oldNelem*sizeof(T));
memcpy(p, oldp, oldNelem*ncclSizeOfT<T>());
free(oldp);
memset(p+oldNelem, 0, (nelem-oldNelem)*sizeof(T));
memset(p+oldNelem, 0, (nelem-oldNelem)*ncclSizeOfT<T>());
*ptr = (T*)p;
INFO(NCCL_ALLOC, "Mem Realloc old size %ld, new size %ld pointer %p", oldNelem*sizeof(T), nelem*sizeof(T), *ptr);
INFO(NCCL_ALLOC, "Mem Realloc old size %ld, new size %ld pointer %p", oldNelem*ncclSizeOfT<T>(), nelem*ncclSizeOfT<T>(), *ptr);
return ncclSuccess;
}
@@ -111,7 +122,7 @@ static inline ncclResult_t ncclCuMemAlloc(void **ptr, CUmemGenericAllocationHand
accessDesc.flags = CU_MEM_ACCESS_FLAGS_PROT_READWRITE;
CUCHECK(cuMemSetAccess((CUdeviceptr)*ptr, size, &accessDesc, 1));
if (handlep) *handlep = handle;
TRACE(NCCL_ALLOC, "CuMem Alloc Size %zi pointer %p handle %llx", size, *ptr, handle);
TRACE(NCCL_ALLOC, "CuMem Alloc Size %zu pointer %p handle %llx", size, *ptr, handle);
return result;
}
@@ -123,7 +134,7 @@ static inline ncclResult_t ncclCuMemFree(void *ptr) {
CUCHECK(cuMemRetainAllocationHandle(&handle, ptr));
CUCHECK(cuMemRelease(handle));
CUCHECK(cuMemGetAddressRange(NULL, &size, (CUdeviceptr)ptr));
TRACE(NCCL_ALLOC, "CuMem Free Size %zi pointer %p handle 0x%llx", size, ptr, handle);
TRACE(NCCL_ALLOC, "CuMem Free Size %zu pointer %p handle 0x%llx", size, ptr, handle);
CUCHECK(cuMemUnmap((CUdeviceptr)ptr, size));
CUCHECK(cuMemRelease(handle));
CUCHECK(cuMemAddressFree((CUdeviceptr)ptr, size));
@@ -151,15 +162,17 @@ ncclResult_t ncclCudaMallocDebug(T** ptr, size_t nelem, const char *filefunc, in
cudaStreamCaptureMode mode = cudaStreamCaptureModeRelaxed;
*ptr = nullptr;
CUDACHECK(cudaThreadExchangeStreamCaptureMode(&mode));
if (ncclCuMemEnable()) {
NCCLCHECKGOTO(ncclCuMemAlloc((void **)ptr, NULL, nelem*sizeof(T)), result, finish);
} else {
CUDACHECKGOTO(cudaMalloc(ptr, nelem*sizeof(T)), result, finish);
if (nelem > 0) {
if (ncclCuMemEnable()) {
NCCLCHECKGOTO(ncclCuMemAlloc((void **)ptr, NULL, nelem*ncclSizeOfT<T>()), result, finish);
} else {
CUDACHECKGOTO(cudaMalloc(ptr, nelem*ncclSizeOfT<T>()), result, finish);
}
}
finish:
CUDACHECK(cudaThreadExchangeStreamCaptureMode(&mode));
if (*ptr == nullptr) WARN("Failed to CUDA malloc %ld bytes", nelem*sizeof(T));
INFO(NCCL_ALLOC, "%s:%d Cuda Alloc Size %ld pointer %p", filefunc, line, nelem*sizeof(T), *ptr);
if (*ptr == nullptr && nelem > 0) WARN("Failed to CUDA malloc %ld bytes", nelem*ncclSizeOfT<T>());
INFO(NCCL_ALLOC, "%s:%d Cuda Alloc Size %ld pointer %p", filefunc, line, nelem*ncclSizeOfT<T>(), *ptr);
return result;
}
#define ncclCudaMalloc(...) ncclCudaMallocDebug(__VA_ARGS__, __FILE__, __LINE__)
@@ -170,21 +183,23 @@ ncclResult_t ncclCudaCallocDebug(T** ptr, size_t nelem, const char *filefunc, in
cudaStreamCaptureMode mode = cudaStreamCaptureModeRelaxed;
*ptr = nullptr;
CUDACHECK(cudaThreadExchangeStreamCaptureMode(&mode));
// Need a side stream so as not to interfere with graph capture.
cudaStream_t stream;
CUDACHECK(cudaStreamCreateWithFlags(&stream, cudaStreamNonBlocking));
if (ncclCuMemEnable()) {
NCCLCHECKGOTO(ncclCuMemAlloc((void **)ptr, NULL, nelem*sizeof(T)), result, finish);
} else {
CUDACHECKGOTO(cudaMalloc(ptr, nelem*sizeof(T)), result, finish);
if (nelem > 0) {
// Need a side stream so as not to interfere with graph capture.
cudaStream_t stream;
CUDACHECK(cudaStreamCreateWithFlags(&stream, cudaStreamNonBlocking));
if (ncclCuMemEnable()) {
NCCLCHECKGOTO(ncclCuMemAlloc((void **)ptr, NULL, nelem*ncclSizeOfT<T>()), result, finish);
} else {
CUDACHECKGOTO(cudaMalloc(ptr, nelem*ncclSizeOfT<T>()), result, finish);
}
CUDACHECKGOTO(cudaMemsetAsync(*ptr, 0, nelem*ncclSizeOfT<T>(), stream), result, finish);
CUDACHECKGOTO(cudaStreamSynchronize(stream), result, finish);
CUDACHECKGOTO(cudaStreamDestroy(stream), result, finish);
}
CUDACHECKGOTO(cudaMemsetAsync(*ptr, 0, nelem*sizeof(T), stream), result, finish);
CUDACHECKGOTO(cudaStreamSynchronize(stream), result, finish);
CUDACHECKGOTO(cudaStreamDestroy(stream), result, finish);
finish:
CUDACHECK(cudaThreadExchangeStreamCaptureMode(&mode));
if (*ptr == nullptr) WARN("Failed to CUDA calloc %ld bytes", nelem*sizeof(T));
INFO(NCCL_ALLOC, "%s:%d Cuda Alloc Size %ld pointer %p", filefunc, line, nelem*sizeof(T), *ptr);
if (*ptr == nullptr && nelem > 0) WARN("Failed to CUDA calloc %ld bytes", nelem*ncclSizeOfT<T>());
INFO(NCCL_ALLOC, "%s:%d Cuda Alloc Size %ld pointer %p", filefunc, line, nelem*ncclSizeOfT<T>(), *ptr);
return result;
}
#define ncclCudaCalloc(...) ncclCudaCallocDebug(__VA_ARGS__, __FILE__, __LINE__)
@@ -195,16 +210,18 @@ ncclResult_t ncclCudaCallocAsyncDebug(T** ptr, size_t nelem, cudaStream_t stream
cudaStreamCaptureMode mode = cudaStreamCaptureModeRelaxed;
*ptr = nullptr;
CUDACHECK(cudaThreadExchangeStreamCaptureMode(&mode));
if (ncclCuMemEnable()) {
NCCLCHECKGOTO(ncclCuMemAlloc((void **)ptr, NULL, nelem*sizeof(T)), result, finish);
} else {
CUDACHECKGOTO(cudaMalloc(ptr, nelem*sizeof(T)), result, finish);
if (nelem > 0) {
if (ncclCuMemEnable()) {
NCCLCHECKGOTO(ncclCuMemAlloc((void **)ptr, NULL, nelem*ncclSizeOfT<T>()), result, finish);
} else {
CUDACHECKGOTO(cudaMalloc(ptr, nelem*ncclSizeOfT<T>()), result, finish);
}
CUDACHECKGOTO(cudaMemsetAsync(*ptr, 0, nelem*ncclSizeOfT<T>(), stream), result, finish);
}
CUDACHECKGOTO(cudaMemsetAsync(*ptr, 0, nelem*sizeof(T), stream), result, finish);
finish:
CUDACHECK(cudaThreadExchangeStreamCaptureMode(&mode));
if (*ptr == nullptr) WARN("Failed to CUDA calloc async %ld bytes", nelem*sizeof(T));
INFO(NCCL_ALLOC, "%s:%d Cuda Alloc Size %ld pointer %p", filefunc, line, nelem*sizeof(T), *ptr);
if (*ptr == nullptr && nelem > 0) WARN("Failed to CUDA calloc async %ld bytes", nelem*ncclSizeOfT<T>());
INFO(NCCL_ALLOC, "%s:%d Cuda Alloc Size %ld pointer %p", filefunc, line, nelem*ncclSizeOfT<T>(), *ptr);
return result;
}
#define ncclCudaCallocAsync(...) ncclCudaCallocAsyncDebug(__VA_ARGS__, __FILE__, __LINE__)
@@ -230,7 +247,7 @@ ncclResult_t ncclCudaMemcpyAsync(T* dst, T* src, size_t nelem, cudaStream_t stre
ncclResult_t result = ncclSuccess;
cudaStreamCaptureMode mode = cudaStreamCaptureModeRelaxed;
CUDACHECK(cudaThreadExchangeStreamCaptureMode(&mode));
CUDACHECKGOTO(cudaMemcpyAsync(dst, src, nelem*sizeof(T), cudaMemcpyDefault, stream), result, finish);
CUDACHECKGOTO(cudaMemcpyAsync(dst, src, nelem*ncclSizeOfT<T>(), cudaMemcpyDefault, stream), result, finish);
finish:
CUDACHECK(cudaThreadExchangeStreamCaptureMode(&mode));
return result;
@@ -256,13 +273,17 @@ finish:
// allocated on separate pages as those pages will be marked DONTFORK
// and if they are shared, that could cause a crash in a child process
inline ncclResult_t ncclIbMallocDebug(void** ptr, size_t size, const char *filefunc, int line) {
size_t page_size = sysconf(_SC_PAGESIZE);
void* p;
int size_aligned = ROUNDUP(size, page_size);
int ret = posix_memalign(&p, page_size, size_aligned);
if (ret != 0) return ncclSystemError;
memset(p, 0, size);
*ptr = p;
if (size > 0) {
size_t page_size = sysconf(_SC_PAGESIZE);
void* p;
int size_aligned = ROUNDUP(size, page_size);
int ret = posix_memalign(&p, page_size, size_aligned);
if (ret != 0) return ncclSystemError;
memset(p, 0, size);
*ptr = p;
} else {
*ptr = NULL;
}
INFO(NCCL_ALLOC, "%s:%d Ib Alloc Size %ld pointer %p", filefunc, line, size, *ptr);
return ncclSuccess;
}
src/include/bitops.h
+277
Féach ar an gComhad
@@ -0,0 +1,277 @@
/*************************************************************************
* Copyright (c) 2019-2022, NVIDIA CORPORATION. All rights reserved.
*
* See LICENSE.txt for license information
************************************************************************/
#ifndef NCCL_BITOPS_H_
#define NCCL_BITOPS_H_
#include <stdint.h>
#if !__NVCC__
#ifndef __host__
#define __host__
#endif
#ifndef __device__
#define __device__
#endif
#endif
#define DIVUP(x, y) \
(((x)+(y)-1)/(y))
#define ROUNDUP(x, y) \
(DIVUP((x), (y))*(y))
#define ALIGN_POWER(x, y) \
((x) > (y) ? ROUNDUP(x, y) : ((y)/((y)/(x))))
#define ALIGN_SIZE(size, align) \
size = ((size + (align) - 1) / (align)) * (align);
template<typename X, typename Y, typename Z = decltype(X()+Y())>
__host__ __device__ constexpr Z divUp(X x, Y y) {
return (x+y-1)/y;
}
template<typename X, typename Y, typename Z = decltype(X()+Y())>
__host__ __device__ constexpr Z roundUp(X x, Y y) {
return (x+y-1) - (x+y-1)%y;
}
template<typename X, typename Y, typename Z = decltype(X()+Y())>
__host__ __device__ constexpr Z roundDown(X x, Y y) {
return x - x%y;
}
// assumes second argument is a power of 2
template<typename X, typename Z = decltype(X()+int())>
__host__ __device__ constexpr Z alignUp(X x, int a) {
return (x + a-1) & Z(-a);
}
// assumes second argument is a power of 2
template<typename X, typename Z = decltype(X()+int())>
__host__ __device__ constexpr Z alignDown(X x, int a) {
return x & Z(-a);
}
template<typename Int>
inline __host__ __device__ int countOneBits(Int x) {
#if __CUDA_ARCH__
if (sizeof(Int) <= sizeof(unsigned int)) {
return __popc((unsigned int)x);
} else if (sizeof(Int) <= sizeof(unsigned long long)) {
return __popcll((unsigned long long)x);
} else {
static_assert(sizeof(Int) <= sizeof(unsigned long long), "Unsupported integer size.");
return -1;
}
#else
if (sizeof(Int) <= sizeof(unsigned int)) {
return __builtin_popcount((unsigned int)x);
} else if (sizeof(Int) <= sizeof(unsigned long)) {
return __builtin_popcountl((unsigned long)x);
} else if (sizeof(Int) <= sizeof(unsigned long long)) {
return __builtin_popcountll((unsigned long long)x);
} else {
static_assert(sizeof(Int) <= sizeof(unsigned long long), "Unsupported integer size.");
return -1;
}
#endif
}
// Returns index of first one bit or returns -1 if mask is zero.
template<typename Int>
inline __host__ __device__ int firstOneBit(Int mask) {
int i;
#if __CUDA_ARCH__
if (sizeof(Int) <= sizeof(int)) {
i = __ffs((int)mask);
} else if (sizeof(Int) <= sizeof(long long)) {
i = __ffsll((long long)mask);
} else {
static_assert(sizeof(Int) <= sizeof(long long), "Unsupported integer size.");
}
#else
if (sizeof(Int) <= sizeof(int)) {
i = __builtin_ffs((int)mask);
} else if (sizeof(Int) <= sizeof(long)) {
i = __builtin_ffsl((long)mask);
} else if (sizeof(Int) <= sizeof(long long)) {
i = __builtin_ffsll((long long)mask);
} else {
static_assert(sizeof(Int) <= sizeof(long long), "Unsupported integer size.");
}
#endif
return i-1;
}
template<typename Int>
inline __host__ __device__ int popFirstOneBit(Int* mask) {
Int tmp = *mask;
*mask &= *mask-1;
return firstOneBit(tmp);
}
template<typename Int>
inline __host__ __device__ int log2Down(Int x) {
int w, n;
#if __CUDA_ARCH__
if (sizeof(Int) <= sizeof(int)) {
w = 8*sizeof(int);
n = __clz((int)x);
} else if (sizeof(Int) <= sizeof(long long)) {
w = 8*sizeof(long long);
n = __clzll((long long)x);
} else {
static_assert(sizeof(Int) <= sizeof(long long), "Unsupported integer size.");
}
#else
if (x == 0) {
return -1;
} else if (sizeof(Int) <= sizeof(unsigned int)) {
w = 8*sizeof(unsigned int);
n = __builtin_clz((unsigned int)x);
} else if (sizeof(Int) <= sizeof(unsigned long)) {
w = 8*sizeof(unsigned long);
n = __builtin_clzl((unsigned long)x);
} else if (sizeof(Int) <= sizeof(unsigned long long)) {
w = 8*sizeof(unsigned long long);
n = __builtin_clzll((unsigned long long)x);
} else {
static_assert(sizeof(Int) <= sizeof(unsigned long long), "Unsupported integer size.");
}
#endif
return (w-1)-n;
}
template<typename Int>
inline __host__ __device__ int log2Up(Int x) {
int w, n;
if (x != 0) x -= 1;
#if __CUDA_ARCH__
if (sizeof(Int) <= sizeof(int)) {
w = 8*sizeof(int);
n = __clz((int)x);
} else if (sizeof(Int) <= sizeof(long long)) {
w = 8*sizeof(long long);
n = __clzll((long long)x);
} else {
static_assert(sizeof(Int) <= sizeof(long long), "Unsupported integer size.");
}
#else
if (x == 0) {
return 0;
} else if (sizeof(Int) <= sizeof(unsigned int)) {
w = 8*sizeof(unsigned int);
n = __builtin_clz((unsigned int)x);
} else if (sizeof(Int) <= sizeof(unsigned long)) {
w = 8*sizeof(unsigned long);
n = __builtin_clzl((unsigned long)x);
} else if (sizeof(Int) <= sizeof(unsigned long long)) {
w = 8*sizeof(unsigned long long);
n = __builtin_clzll((unsigned long long)x);
} else {
static_assert(sizeof(Int) <= sizeof(unsigned long long), "Unsupported integer size.");
}
#endif
return w-n;
}
template<typename Int>
inline __host__ __device__ Int pow2Up(Int x) {
return Int(1)<<log2Up(x);
}
template<typename Int>
inline __host__ __device__ Int pow2Down(Int x) {
return Int(1)<<log2Down(x);
}
template<typename UInt, int nSubBits>
inline __host__ UInt reverseSubBits(UInt x) {
if (nSubBits >= 16 && 8*sizeof(UInt) == nSubBits) {
switch (8*sizeof(UInt)) {
case 16: x = __builtin_bswap16(x); break;
case 32: x = __builtin_bswap32(x); break;
case 64: x = __builtin_bswap64(x); break;
default: static_assert(8*sizeof(UInt) <= 64, "Unsupported integer type.");
}
return reverseSubBits<UInt, 8>(x);
} else if (nSubBits == 1) {
return x;
} else {
UInt m = UInt(-1)/((UInt(1)<<(nSubBits/2))+1);
x = (x & m)<<(nSubBits/2) | (x & ~m)>>(nSubBits/2);
return reverseSubBits<UInt, nSubBits/2>(x);
}
}
template<typename T> struct ncclToUnsigned;
template<> struct ncclToUnsigned<char> { using type = unsigned char; };
template<> struct ncclToUnsigned<signed char> { using type = unsigned char; };
template<> struct ncclToUnsigned<unsigned char> { using type = unsigned char; };
template<> struct ncclToUnsigned<signed short> { using type = unsigned short; };
template<> struct ncclToUnsigned<unsigned short> { using type = unsigned short; };
template<> struct ncclToUnsigned<signed int> { using type = unsigned int; };
template<> struct ncclToUnsigned<unsigned int> { using type = unsigned int; };
template<> struct ncclToUnsigned<signed long> { using type = unsigned long; };
template<> struct ncclToUnsigned<unsigned long> { using type = unsigned long; };
template<> struct ncclToUnsigned<signed long long> { using type = unsigned long long; };
template<> struct ncclToUnsigned<unsigned long long> { using type = unsigned long long; };
// Reverse the bottom nBits bits of x. The top bits will be overwritten with 0's.
template<typename Int>
inline __host__ __device__ Int reverseBits(Int x, int nBits) {
using UInt = typename ncclToUnsigned<Int>::type;
union { UInt ux; Int sx; };
sx = x;
#if __CUDA_ARCH__
if (sizeof(Int) <= sizeof(unsigned int)) {
ux = __brev(ux);
} else if (sizeof(Int) <= sizeof(unsigned long long)) {
ux = __brevll(ux);
} else {
static_assert(sizeof(Int) <= sizeof(unsigned long long), "Unsupported integer type.");
}
#else
ux = reverseSubBits<UInt, 8*sizeof(UInt)>(ux);
#endif
ux = nBits==0 ? 0 : ux>>(8*sizeof(UInt)-nBits);
return sx;
}
////////////////////////////////////////////////////////////////////////////////
// Custom 8 bit floating point format for approximating 32 bit uints. This format
// has nearly the full range of uint32_t except it only keeps the top 3 bits
// beneath the leading 1 bit and thus has a max value of 0xf0000000.
inline __host__ __device__ uint32_t u32fpEncode(uint32_t x, int bitsPerPow2) {
int log2x;
#if __CUDA_ARCH__
log2x = 31-__clz(x|1);
#else
log2x = 31-__builtin_clz(x|1);
#endif
uint32_t mantissa = x>>(log2x >= bitsPerPow2 ? log2x-bitsPerPow2 : 0) & ((1u<<bitsPerPow2)-1);
uint32_t exponent = log2x >= bitsPerPow2 ? log2x-(bitsPerPow2-1) : 0;
return exponent<<bitsPerPow2 | mantissa;
}
inline __host__ __device__ uint32_t u32fpDecode(uint32_t x, int bitsPerPow2) {
uint32_t exponent = x>>bitsPerPow2;
uint32_t mantissa = (x & ((1u<<bitsPerPow2)-1)) | (exponent!=0 ? 0x8 : 0);
if (exponent != 0) exponent -= 1;
return mantissa<<exponent;
}
constexpr uint32_t u32fp8MaxValue() { return 0xf0000000; }
inline __host__ __device__ uint8_t u32fp8Encode(uint32_t x) {
return u32fpEncode(x, 3);
}
inline __host__ __device__ uint32_t u32fp8Decode(uint8_t x) {
return u32fpDecode(x, 3);
}
#endif
src/include/channel.h
+12 -29
Féach ar an gComhad
@@ -7,42 +7,25 @@
#ifndef NCCL_CHANNEL_H_
#define NCCL_CHANNEL_H_
#include "comm.h"
#include "utils.h"
#include <algorithm>
ncclResult_t initChannel(struct ncclComm* comm, int channelid);
ncclResult_t initNvlsChannel(struct ncclComm* comm, int channelId, struct ncclComm* parent, bool share);
ncclResult_t initCollnetChannel(struct ncclComm* comm, int channelId, struct ncclComm* parent, bool share);
ncclResult_t freeChannel(struct ncclChannel* channel, int nRanks, int collnetNRanks, int nvlsNRanks);
static ncclResult_t ncclChannelComputeBase(struct ncclComm* comm, int peer, int coll, int*channelBase) {
int p2pGroupSize = NCCL_MAX_WORK_ELEMENTS_P2P/2;
int peerNode = comm->rankToNode[peer];
int peerIndex = comm->rankToLocalRank[peer];
int nsteps = comm->maxLocalRanks;
int rankIndex = comm->rankToLocalRank[comm->rank];
int step, delta;
if (coll == ncclFuncSend) {
step = (nsteps + peerIndex - rankIndex)%nsteps;
delta = (comm->nNodes + peerNode - comm->node) % comm->nNodes;
} else if (coll == ncclFuncRecv) {
step = (nsteps + rankIndex - peerIndex)%nsteps;
delta = (comm->nNodes + comm->node - peerNode) % comm->nNodes;
inline uint8_t ncclP2pChannelBaseForRound(struct ncclComm* comm, int p2pRound) {
if (comm->nNodes > 1) {
int nodeDelta = p2pRound/comm->maxLocalRanks;
int localDelta = p2pRound%comm->maxLocalRanks;
int base = nodeDelta*divUp(comm->maxLocalRanks, NCCL_MAX_DEV_WORK_P2P_PER_BATCH);
base += localDelta/NCCL_MAX_DEV_WORK_P2P_PER_BATCH;
return base & 0xff;
} else {
return ncclInternalError;
return p2pRound & 0xff;
}
*channelBase = comm->nNodes > 1 ? delta+(step/p2pGroupSize) : step;
return ncclSuccess;
}
static ncclResult_t ncclChannelComputeFromBase(struct ncclComm* comm, int base, int channelInc, int*channelId) {
//*channelId = (base+comm->p2pChannels[channelInc]) % comm->p2pnChannels;
*channelId = (comm->p2pChannels[base%comm->p2pnChannels]+channelInc) % comm->p2pnChannels;
return ncclSuccess;
}
static ncclResult_t ncclChannelCompute(struct ncclComm* comm, int peer, int channelInc, int coll, int*channelId) {
int base;
NCCLCHECK(ncclChannelComputeBase(comm, peer, coll, &base));
NCCLCHECK(ncclChannelComputeFromBase(comm, base, channelInc, channelId));
return ncclSuccess;
}
#endif
src/include/checks.h
+4 -4
Féach ar an gComhad
@@ -123,23 +123,23 @@
} while (0);
#define NCCLWAIT(call, cond, abortFlagPtr) do { \
volatile uint32_t* tmpAbortFlag = (abortFlagPtr); \
uint32_t* tmpAbortFlag = (abortFlagPtr); \
ncclResult_t RES = call; \
if (RES != ncclSuccess && RES != ncclInProgress) { \
if (ncclDebugNoWarn == 0) INFO(NCCL_ALL,"%s:%d -> %d", __FILE__, __LINE__, RES); \
return ncclInternalError; \
} \
if (tmpAbortFlag) NEQCHECK(*tmpAbortFlag, 0); \
if (__atomic_load(tmpAbortFlag, __ATOMIC_ACQUIRE)) NEQCHECK(*tmpAbortFlag, 0); \
} while (!(cond));
#define NCCLWAITGOTO(call, cond, abortFlagPtr, RES, label) do { \
volatile uint32_t* tmpAbortFlag = (abortFlagPtr); \
uint32_t* tmpAbortFlag = (abortFlagPtr); \
RES = call; \
if (RES != ncclSuccess && RES != ncclInProgress) { \
if (ncclDebugNoWarn == 0) INFO(NCCL_ALL,"%s:%d -> %d", __FILE__, __LINE__, RES); \
goto label; \
} \
if (tmpAbortFlag) NEQCHECKGOTO(*tmpAbortFlag, 0, RES, label); \
if (__atomic_load(tmpAbortFlag, __ATOMIC_ACQUIRE)) NEQCHECKGOTO(*tmpAbortFlag, 0, RES, label); \
} while (!(cond));
#define NCCLCHECKTHREAD(a, args) do { \
src/include/collectives.h
+8
Féach ar an gComhad
@@ -8,6 +8,8 @@
#define NCCL_COLLECTIVES_H_
#include "nccl.h"
#include "nccl_common.h"
#include "device.h"
// CHUNKSIZE must be a multiple of SLICESIZE
#define ALLREDUCE_SLICESTEPS (NCCL_STEPS/4)
@@ -22,6 +24,12 @@
#define REDUCE_CHUNKSTEPS 1
#define NCCL_MAX_SLICE_PER_CHUNK 2 // max value for CHUNKSTEPS/SLICESTEPS, must accord with above
const char* ncclFuncToString(ncclFunc_t op);
const char* ncclDevRedOpToString(ncclDevRedOp_t op);
const char* ncclDatatypeToString(ncclDataType_t type);
const char* ncclAlgoToString(int algo);
const char* ncclProtoToString(int proto);
inline int ncclTypeSize(ncclDataType_t type) {
switch (type) {
case ncclInt8:
src/include/comm.h
+222 -60
Féach ar an gComhad
@@ -7,7 +7,7 @@
#ifndef NCCL_COMM_H_
#define NCCL_COMM_H_
#include "transport.h"
//#include "transport.h"
#include "p2p.h"
#include "collectives.h"
#include "nccl_tuner.h"
@@ -15,6 +15,7 @@
#include "strongstream.h"
#include "nccl_net.h"
#include "register.h"
#include "graph.h"
#if CUDART_VERSION < 9000
struct cudaLaunchParams {
@@ -144,7 +145,7 @@ struct ncclChannel {
struct ncclNvls nvls;
int id; // index of this channel
uint32_t workFifoSent; // last used work index+1
uint32_t workFifoProduced; // +1 successor of last used work fifo byte
/* comm split sharable resources */
struct ncclChannelPeer* collnetPeers;
@@ -153,22 +154,15 @@ struct ncclChannel {
struct ncclDevChannelPeer* nvlsDevPeers;
};
struct ncclWorkList {
struct ncclWorkBatchList {
struct ncclWorkBatchList* next;
struct ncclDevWorkBatch batch;
};
struct alignas(16) ncclWorkList {
struct ncclWorkList* next;
struct ncclWork work;
};
struct ncclPointerList {
struct ncclPointerList* next;
void *ptr;
};
struct ncclNvlsMcHandleList {
struct ncclNvlsMcHandleList *next;
CUmemGenericAllocationHandle mcHandle;
CUdeviceptr ptr;
int dev;
size_t size;
enum ncclDevWorkType workType;
int size; // Size of struct following this node
// ncclDevWorkColl, ncclDevWorkColLReg, ncclDevWorkP2p[]...
};
struct ncclCollnetHandleList {
@@ -188,33 +182,190 @@ struct ncclKernelPlan {
struct ncclKernelPlan* next;
bool persistent; // aka captured in a graph
enum ncclDevWorkStorageType workStorageType;
bool kernelSpecialized;
void *kernelFn;
int channelUbound; // only channels c < channelUbound are present
int channelCount; // number of channels present
uint64_t channelMask; // which channels are present, channelCount == popcount(channelMask)
struct ncclDevKernelArgs* kernelArgs;
size_t kernelArgsSize;
uint64_t channelMask; // bitset of which channels are present
bool hasProxyOps; // does any channel have a non-empty proxyOpQueue
int threadPerBlock;
// workHeap fields are null until uploadWorkFifo() or preparePersistentKernel()
struct ncclWork* workHead;
int collOpCount; // zero based for this plan
int collOpCount; // Number of collectives in this plan.
int nWorkBatches; // Number of work batches.
size_t workBytes; // Sum size of all work (in the fifo) in bytes.
struct ncclIntruQueue<struct ncclWorkList, &ncclWorkList::next> workQueue;
struct ncclIntruQueue<struct ncclCommCallback, &ncclCommCallback::next> cleanupQueue;
void* workBufPersistent;
struct ncclIntruQueue<struct ncclPointerList, &ncclPointerList::next> ipcMemQueue;
struct ncclIntruQueue<struct ncclNvlsMcHandleList, &ncclNvlsMcHandleList::next> nvlsMcHandleQueue;
struct ncclIntruQueue<struct ncclCollnetHandleList, &ncclCollnetHandleList::next> collnetHandleQueue;
struct ncclIntruQueue<struct ncclProxyOp, &ncclProxyOp::enqNext> proxyOpQueue;
};
struct Channel {
int nWork;
union {
int nWorkElem; // used for coll and reg coll
int p2pTailElem[2]; // used for p2p, indexed by ncclWorkElemP2pType-1
};
size_t collBytes;
struct ncclIntruQueue<struct ncclWorkList, &ncclWorkList::next> workQueue;
struct ncclIntruQueue<struct ncclProxyOp, &ncclProxyOp::enqNext> proxyOpQueue;
} channels[MAXCHANNELS];
size_t maxBytesPerChannel;
////////////////////////////////////////////////////////////////////////////////
struct ncclTaskColl {
struct ncclTaskColl* next;
ncclFunc_t func;
void const* sendbuff;
void* recvbuff;
size_t count;
int root;
ncclDataType_t datatype;
ncclRedOp_t opHost;
struct ncclDevRedOpFull opDev;
int chunkSteps, sliceSteps;
// Computed later:
size_t trafficBytes;
int32_t nMaxChannels:8;
int32_t nWarps:8;
int32_t algorithm:8, protocol:8;
uint32_t isCollnet:1, isNvls:1;
uint32_t devFuncId:30;
enum ncclRegBufferType regBufType;
// number of elements in planner->ipcMemQueue associated with this collective
int nCleanupQueueElts;
void* sendMhandle;
void* recvMhandle;
};
struct ncclTaskP2p {
struct ncclTaskP2p* next;
void* buff;
size_t bytes;
};
////////////////////////////////////////////////////////////////////////////////
// Roughly sorts ncclTaskColl's by their size descending. This structure is
// self-referential, meaning that pointers it contains internally may point
// into the structure itself. This means that it is NOT memcpy-moveable:
struct ncclTaskCollSorter {
static constexpr int UnitLog2 = 10; // 1K
static constexpr size_t UnitSize = 1<<UnitLog2;
static constexpr int MaxLog2 = 30; // 1GB
static constexpr size_t MaxSize = 1ull<<MaxLog2;
// Number of bins between powers of 2. For 4 bins, the worst case out-of-order
// relative magnitude is (5/4)-1 = 25%
static constexpr int BitsPerPow2 = 2;
static constexpr int BinsPerPow2 = 1<<BitsPerPow2;
static constexpr int BinCount = 1 + (MaxLog2-UnitLog2)*BinsPerPow2;
struct ncclTaskColl* head;
struct ncclTaskColl* tail;
// Least bin such that it and all above are empty.
int binEdge;
// Pointer to the pointer to this bin's head node which is either the
// previous node's `next` field or `head`.
struct ncclTaskColl** bins[BinCount];
};
inline void ncclTaskCollSorterInsert(
struct ncclTaskCollSorter* me, struct ncclTaskColl* x, size_t size
) {
constexpr int UnitLog2 = ncclTaskCollSorter::UnitLog2;
constexpr size_t MaxSize = ncclTaskCollSorter::MaxSize;
constexpr int BitsPerPow2 = ncclTaskCollSorter::BitsPerPow2;
constexpr int BinCount = ncclTaskCollSorter::BinCount;
int bin = u32fpEncode(std::min(MaxSize, size)>>UnitLog2, BitsPerPow2);
bin = BinCount-1 - bin; // descending bin
if (me->bins[bin] == nullptr) {
if (me->binEdge <= bin) {
me->binEdge = bin+1;
me->bins[bin] = me->tail ? &me->tail->next : &me->head;
me->tail = x;
} else {
// Find successor non-empty bin after this one.
int succ = bin+1;
while (me->bins[succ] == nullptr) succ++;
// What was our successor's head's previous is now our head's previous.
me->bins[bin] = me->bins[succ];
// The first node we insert is our tail, so that becomes our successor's
// head's new previous.
me->bins[succ] = &x->next;
}
}
// Push a new head for this bin.
x->next = *me->bins[bin];
*me->bins[bin] = x;
}
inline bool ncclTaskCollSorterEmpty(struct ncclTaskCollSorter* me) {
return me->head == nullptr;
}
// Reset sorter and return sorted linked list of its coll tasks.
inline struct ncclTaskColl* ncclTaskCollSorterDequeueAll(struct ncclTaskCollSorter* me) {
struct ncclTaskColl* head = me->head;
if (head != nullptr) memset(me, 0, sizeof(*me));
return head;
}
////////////////////////////////////////////////////////////////////////////////
struct ncclCudaStreamList {
struct ncclCudaStreamList *next;
cudaStream_t stream;
};
struct ncclKernelPlanner {
//////////////////////////////////////////////////////////////////////////////
// State for accumulating tasks between ncclGroupStart/End()
//////////////////////////////////////////////////////////////////////////////
struct Peer {
bool sendSeen, recvSeen;
struct ncclIntruQueue<struct ncclTaskP2p, &ncclTaskP2p::next> sendQueue;
struct ncclIntruQueue<struct ncclTaskP2p, &ncclTaskP2p::next> recvQueue;
};
struct ncclTaskCollSorter collSorter;
struct Peer* peers/*[nRanks]*/;
int nTasksColl, nTasksP2p;
bool persistent;
// The list of user streams aggregated over all tasks present.
struct ncclCudaStreamList* streams;
// The most recent user stream. Ignored if streams==nullptr
cudaStream_t streamRecent;
// The graph capturing all user streams or invalid if none. Thus we restrict the
// user that all streams must be captured in the same graph or not captured
// at all. Technically we could probably relax this, but that would mean
// collecting a different `ncclTasks` per graph and one for non-graph.
struct ncclCudaGraph capturingGraph;
//////////////////////////////////////////////////////////////////////////////
// Lists of tasks to be assembled into plans.
//////////////////////////////////////////////////////////////////////////////
struct ncclIntruQueue<struct ncclTaskColl, &ncclTaskColl::next> collTaskQueue;
struct ncclIntruQueue<struct ncclWorkList, &ncclWorkList::next> collWorkQueue;
struct ncclIntruQueue<struct ncclCommCallback, &ncclCommCallback::next> collCleanupQueue;
//////////////////////////////////////////////////////////////////////////////
// State for building current (Work-In-Progress) plan:
//////////////////////////////////////////////////////////////////////////////
struct WipPlan {
struct Channel {
struct {
int workBytes; // Sum size of work metadata referenced by this batch.
int nP2ps; // Number of p2p works in this batch
int p2pRounds[NCCL_MAX_DEV_WORK_P2P_PER_BATCH]; // which rounds are present in this batch.
} wipBatch; // work-in-progress batch which will be next tail of workBatchQueue
int nWorkBatchesP2p; // number of p2p batches for this channel.
struct ncclIntruQueue<struct ncclWorkBatchList, &ncclWorkBatchList::next> workBatchQueue;
struct ncclIntruQueue<struct ncclProxyOp, &ncclProxyOp::enqNext> proxyOpQueue;
} channels[MAXCHANNELS];
} wipPlan;
//////////////////////////////////////////////////////////////////////////////
// State for launching built plans:
//////////////////////////////////////////////////////////////////////////////
// List of kernel plans built form tasks.
struct ncclIntruQueue<struct ncclKernelPlan, &ncclKernelPlan::next> planQueue;
// First of the unlaunched kernels in `planQueue`
struct ncclKernelPlan* unlaunchedPlansHead;
};
#define NCCL_MAGIC 0x0280028002800280 // Nickel atomic number is 28.
@@ -233,12 +384,18 @@ struct ncclComm {
struct ncclPeerInfo* peerInfo;
struct ncclTopoSystem* topo;
int netPluginLoaded;
ncclNet_t* ncclNet;
ncclNetDeviceType netDeviceType;
ncclCollNet_t* ncclCollNet;
void* bootstrap;
// Bitmasks for ncclTransportP2pSetup
uint64_t* connectSend;
uint64_t* connectRecv;
struct ncclTopoGraph graphs[NCCL_NUM_ALGORITHMS];
bool initAlgoChannels[NCCL_NUM_ALGORITHMS];
bool runtimeConn; // if dynamic connection is supported
int cuMemSupport;
uint64_t magic; // Magic number for all network communication. Not a security key -- only goal is to detect mismatches.
@@ -253,6 +410,9 @@ struct ncclComm {
cpu_set_t cpuAffinity; // CPU affinity of the GPU
int cudaArch; // matches __CUDA_ARCH__ of device
int cpuArch; // architecture - As defined in src/include/graph.h, e.g. x86/arm/ppc/mixed
int cpuVendor; // vendor - As defined in src/include/graph.h
int node;
int nNodes;
int localRank;
@@ -278,10 +438,11 @@ struct ncclComm {
int nChannels; // connection nChannels
int collChannels; // enqueue nChannels
int nvlsChannels; // enqueue nChannels
// all nvls heads stored to check if we can splitShare
int nvlsHeads[MAXCHANNELS];
// Channels (per peer) for p2p
int p2pnChannels;
int p2pnChannelsPerPeer;
int p2pChannels[MAXCHANNELS];
// Should this comm allocate LL buffers for network P2P connections?
bool allocP2pNetLLBuffers;
@@ -303,23 +464,28 @@ struct ncclComm {
ncclResult_t asyncResult;
// Flag to ask NCCL kernels to abort
volatile uint32_t *abortFlag;
volatile uint32_t *childAbortFlag;
uint32_t *abortFlagRefCount;
uint32_t* abortFlag;
uint32_t* abortFlagDev;
int* abortFlagRefCount;
uint32_t* childAbortFlag;
uint32_t* childAbortFlagDev;
uint32_t destroyFlag;
// Device side of the communicator (for cudaFree's)
struct ncclDevComm* devComm; // actually = &ncclDevCommAndChannels::comm
// Operation pool.
int workFifoDepth; // size of workFifoHeap[], power of 2
struct ncclWork* workFifoHeap;
struct ncclWork* devWorkFifoHeap;
void* workFifoHeapGdrHandle;
uint32_t workArgsBytes; // max size of kernel args
uint32_t workFifoBytes; // size of workFifoBuf, power of 2
void* workFifoBuf;
void* workFifoBufDev;
void* workFifoBufGdrHandle;
// Work completion notificaion
uint32_t* workFifoDone/*[MAXCHANNELS]*/; // in cudaHost memory
uint32_t workFifoSent; // Monotonic (mod 1<<32) index of next unused fifo slot.
uint32_t workFifoAckdMin; // Monotonic index of least unprocessed fifo slot over all channels.
// Monotonic number of bytes (mod 1<<32) consumed per channel. In cudaHost memory.
uint32_t* workFifoConsumed/*[MAXCHANNELS]*/;
// Last observed value of: min(workFifoConsumed[c] for c < MAXCHANNELS)
uint32_t workFifoConsumedLeast;
// Monotonic number of bytes (mod 1<<32) sent to fifo.
uint32_t workFifoProduced;
// Intra-process sync
struct ncclComm* intraComm0; // leader of intra-process comms (self possible)
@@ -337,7 +503,7 @@ struct ncclComm {
// Whether this communicator uses collNet
int collNetSupport;
bool collNetRegSupport;
uint8_t collNetSupportMatrix[4/*sum,prod,min,max*/][ncclNumTypes];
uint8_t collNetSupportMatrix[4/*sum,prod,max,min*/][ncclNumTypes];
int intraHighestTransportType;
int* collNetHeads;
int collNetHeadsNum;
@@ -355,16 +521,16 @@ struct ncclComm {
// pools backed by comm->memPermanent
struct ncclMemoryPool memPool_ncclProxyOp;
struct ncclMemoryPool memPool_ncclKernelPlan;
struct ncclMemoryPool memPool_ncclPointerList;
struct ncclMemoryPool memPool_ncclNvlsHandleList;
struct ncclMemoryPool memPool_ncclCollnetHandleList;
// Next comm in this thread's active ncclGroup[Start|End](). Holds "0x1" when
// this comm is not yet in a group.
struct ncclComm* groupNext;
// Subset of those in groupNext list. Holds 0x1 if not needing preconnect.
struct ncclComm* preconnectNext;
int persistentRefs; // number of persistent plan-lists capturing this comm
struct ncclTasks tasks;
struct P2pSchedulePair { int sendRank; int recvRank; } *p2pSchedule;
struct ncclKernelPlanner planner;
// user-created reduction ops
int userRedOpCapacity, userRedOpFreeHead;
@@ -373,11 +539,6 @@ struct ncclComm {
// Queue of things for the main thread to do
struct ncclIntruQueueMpsc<struct ncclCommCallback, &ncclCommCallback::next> callbackQueue;
// List of kernel plans built form tasks.
struct ncclIntruQueue<struct ncclKernelPlan, &ncclKernelPlan::next> planQueue;
// First of the unlaunched kernels in `planQueue`
struct ncclKernelPlan* unlaunchedPlansHead;
ncclConfig_t config;
// initState is to more conveniently reclaim resources when errors happen.
ncclResult_t initState;
@@ -389,6 +550,7 @@ struct ncclComm {
struct ncclGroupJob *groupJob;
// Tuning plugin
int tunerPluginLoaded;
ncclTuner_t* tuner;
void *tunerContext;
// buffer registration cache
src/include/cudawrap.h
+4
Féach ar an gComhad
@@ -80,6 +80,10 @@ DECLARE_CUDA_PFN_EXTERN(cuCtxGetCurrent);
DECLARE_CUDA_PFN_EXTERN(cuCtxSetCurrent);
DECLARE_CUDA_PFN_EXTERN(cuCtxGetDevice);
DECLARE_CUDA_PFN_EXTERN(cuPointerGetAttribute);
DECLARE_CUDA_PFN_EXTERN(cuLaunchKernel);
#if CUDART_VERSION >= 11080
DECLARE_CUDA_PFN_EXTERN(cuLaunchKernelEx);
#endif
// cuMem API support
DECLARE_CUDA_PFN_EXTERN(cuMemAddressReserve);
DECLARE_CUDA_PFN_EXTERN(cuMemAddressFree);
src/include/debug.h
+1 -8
Féach ar an gComhad
@@ -10,21 +10,14 @@
#include "nccl.h"
#include "nccl_common.h"
#include <stdio.h>
#include <chrono>
#include <type_traits>
#include <limits.h>
#include <string.h>
#include <pthread.h>
// Conform to pthread and NVTX standard
#define NCCL_THREAD_NAMELEN 16
extern int ncclDebugLevel;
extern uint64_t ncclDebugMask;
extern pthread_mutex_t ncclDebugLock;
extern FILE *ncclDebugFile;
extern ncclResult_t getHostName(char* hostname, int maxlen, const char delim);
void ncclDebugLog(ncclDebugLogLevel level, unsigned long flags, const char *filefunc, int line, const char *fmt, ...) __attribute__ ((format (printf, 5, 6)));
@@ -32,13 +25,13 @@ void ncclDebugLog(ncclDebugLogLevel level, unsigned long flags, const char *file
extern thread_local int ncclDebugNoWarn;
extern char ncclLastError[];
#define VERSION(...) ncclDebugLog(NCCL_LOG_VERSION, NCCL_ALL, __FILE__, __LINE__, __VA_ARGS__)
#define WARN(...) ncclDebugLog(NCCL_LOG_WARN, NCCL_ALL, __FILE__, __LINE__, __VA_ARGS__)
#define INFO(FLAGS, ...) ncclDebugLog(NCCL_LOG_INFO, (FLAGS), __func__, __LINE__, __VA_ARGS__)
#define TRACE_CALL(...) ncclDebugLog(NCCL_LOG_TRACE, NCCL_CALL, __func__, __LINE__, __VA_ARGS__)
#ifdef ENABLE_TRACE
#define TRACE(FLAGS, ...) ncclDebugLog(NCCL_LOG_TRACE, (FLAGS), __func__, __LINE__, __VA_ARGS__)
extern std::chrono::steady_clock::time_point ncclEpoch;
#else
#define TRACE(...)
#endif
src/include/device.h
+182 -95
Féach ar an gComhad
@@ -9,8 +9,10 @@
#include "nccl.h"
#include "nccl_common.h"
#include "align.h"
#include "bitops.h"
#include <algorithm>
#include <stdint.h>
#include <sys/types.h>
extern const char* ncclFuncStr[NCCL_NUM_FUNCTIONS];
@@ -21,6 +23,12 @@ extern const char* ncclProtoStr[NCCL_NUM_PROTOCOLS];
#define NCCL_MAX_OPS 2048
#define NCCL_STEPS 8
#ifdef __CUDA_ARCH__
#define NCCL_CUDA_ARCH __CUDA_ARCH__
#else
#define NCCL_CUDA_ARCH 0
#endif
#include "net_device.h"
enum ncclDevRedOp_t {
@@ -52,8 +60,11 @@ union ncclLLFifoLine {
#define WARP_SIZE 32
#define MAXCHANNELS 32
#define NCCL_MAX_LOCAL_RANKS 64
#define NCCL_MAX_NTHREADS 640
#define NCCL_MIN_NTHREADS (4*WARP_SIZE)
#define NCCL_SIMPLE_MAX_NTHREADS 512
#define NCCL_SIMPLE_EXTRA_GROUP_IF_NTHREADS_GE (3*WARP_SIZE)
#define NCCL_LL_MAX_NTHREADS 512
#define NCCL_LL_LINES_PER_THREAD 8
#ifdef TEST_LL_CLEANUP
@@ -84,6 +95,9 @@ static_assert(NCCL_LL_CLEAN_MASK % NCCL_STEPS == 0, "Invalid NCCL_LL_CLEAN_MASK
#define NCCL_IPC_READ 0x10
#define NCCL_NVLS_MIN_POLL 0x20
// Number of named barriers supported by CUDA
#define NCCL_MAX_GROUPS 16
#define NCCL_MAX_COLLNET_SIZE (1L << 29)
enum ncclRegBufferType {
@@ -196,112 +210,155 @@ struct ncclChannelPeer {
struct ncclDevComm;
/* ncclWork is to be a power of two, currently 8x64 bytes, */
/* to make sure reads to host from the CUDA kernel are aligned. */
/* Make sure to adjust padding at the end of ncclWorkElem. */
#define NCCL_WORK_SIZE 512
struct alignas(16) ncclDevWorkP2p {
void *sendAddr, *recvAddr;
size_t sendBytes, recvBytes;
int sendRank, recvRank;
// From the part index, nP2pChannels, and channelBase the device code can
// calculate which part of the transfer a channel is responsible for.
uint8_t nP2pChannels; // Always equal to comm->p2pnChannels
uint8_t channelBase; // Channel owning first part.
// Zero channels indicates no work in that direction.
uint8_t nSendChannels, nRecvChannels;
// Chunk size stored in 8 bits via u32fp8Encode/Decode.
uint8_t sendChunkSize_u32fp8, recvChunkSize_u32fp8;
enum ncclWorkType : uint8_t {
ncclWorkTypeUnused=0,
ncclWorkTypeColl=1,
ncclWorkTypeP2p=2,
ncclWorkTypeRegColl=3
};
enum ncclWorkP2PType : uint8_t {
ncclWorkP2pTypeUnused=0,
ncclWorkP2pTypeSend,
ncclWorkP2pTypeRecv
uint8_t sendProtoLL:1, recvProtoLL:1;
uint8_t sendRegistered:1, recvRegistered:1;
};
struct ncclWorkHeader {
union {
int32_t workNext; // when isLast=0: Offset from kernel argument workHead
uint32_t doneAcks; // when isLast=1: Monotonic (mod 1<<32) ack value to send back.
};
uint16_t funcIndex;
uint8_t isLast:1; // last work for this kernel
uint8_t inFifo:1; // is this work in the fifo
enum ncclWorkType type;
};
// Compute the subset of the data transfer corresponding to the given part index.
inline __host__ __device__ void ncclP2pPartBounds(int nParts, int part, size_t bytes, size_t* partBeg, size_t* partEnd) {
size_t partBytes = alignUp(divUp(bytes, nParts), 4<<10);
#if __CUDA_ARCH__
*partBeg = min((part+0)*partBytes, bytes);
*partEnd = min((part+1)*partBytes, bytes);
#else
*partBeg = std::min<size_t>((part+0)*partBytes, bytes);
*partEnd = std::min<size_t>((part+1)*partBytes, bytes);
#endif
}
struct ncclWorkElem {
union {
uint8_t flagBits;
struct {
uint8_t isUsed:1, redOpArgIsPtr:1, oneNode:1;
};
};
uint8_t regUsed;
uint8_t nWarps;
uint8_t direct;
// implemented in channel.h
inline __host__ uint8_t ncclP2pChannelBaseForRound(struct ncclComm* comm, int p2pRound);
// ncclP2pChannelToPart and ncclP2pChannelForPart are inverses. The device code
// uses ncclP2pChannelToPart to determine which part "this" channel is responsible for.
inline __host__ int ncclP2pChannelForPart(int nP2pChannels, int base, int part) {
// Only works because nP2pChannels is pow2
int nChannelsLog2 = countOneBits(nP2pChannels-1);
int delta = reverseBits(part, nChannelsLog2);
return (base + delta) & (nP2pChannels-1);
}
inline __device__ int ncclP2pChannelToPart(int nP2pChannels, int base, int channel) {
// Only works because nP2pChannels is pow2
int nChannelsLog2 = countOneBits(nP2pChannels-1);
int delta = (channel-base) & (nP2pChannels-1);
return reverseBits(delta, nChannelsLog2);
}
struct alignas(16) ncclDevWorkColl {
// Running on channels [channelLo..channelHi], hi is inclusive.
// nChannels == (channelHi - channelLo) + 1
uint32_t channelLo:8, channelHi:8;
uint32_t nWarps:8;
uint32_t redOpArgIsPtr:1, regUsed:2, oneNode:1, direct:4;
uint32_t root;
const void *sendbuff;
void *recvbuff;
size_t count;
uint64_t redOpArg;
uint64_t chunkCount:25, workCount:39;
void* recvbuff;
void* sendbuff;
union {
// Continuous-byte-distribution scheduling. The lo and hi channels are of
// different size than the channels in the middle.
struct {
uint64_t lastChunkCount:25;
uint64_t workOffset:39;
};
size_t countLo, countMid, countHi;
// Chunk counts where units are ncclProtoGrainSize(protocol) bytes
uint64_t chunkGrainsLo:21, chunkGrainsMid:21, chunkGrainsHi:21;
} cbd;
// Collnet scheduling. All channels divide work evenly.
struct {
uint64_t bid:32;
uint64_t nChannels:32;
};
size_t count; // Total size, not divided per channel.
uint32_t chunkCount;
} collnet;
};
uint64_t redOpArg;
};
#define NCCL_MAX_WORK_ELEMENTS ((NCCL_WORK_SIZE - alignUp(sizeof(ncclWorkHeader), alignof(ncclWorkElem)))/sizeof(ncclWorkElem))
static_assert(NCCL_MAX_WORK_ELEMENTS == 9, "Sanity check: NCCL_MAX_WORK_ELEMENTS == 9");
struct ncclWorkElemP2p {
int peer : 30;
int proto : 2;
__host__ __device__ constexpr int ncclProtoGrainSize(int proto) {
return proto == NCCL_PROTO_LL ? 16 :
proto == NCCL_PROTO_LL128 ? WARP_SIZE*NCCL_LL128_SHMEM_ELEMS_PER_THREAD/NCCL_LL128_LINEELEMS*NCCL_LL128_DATAELEMS*sizeof(uint64_t) :
proto == NCCL_PROTO_SIMPLE ? 512 :
-1;
}
enum ncclWorkP2PType p2pType;
uint8_t reg:1;
uint8_t nWarps:5;
uint8_t warpStart;
uint8_t ngroups;
// Important not to use any fields with greater than 4-byte alignment since
// we need sizeof(ncclWorkElemP2p)==28, but that would be padded up to 32 if
// there were 8-byte fields.
//void* buff;
uint32_t buffHi32, buffLo32; // buff = buffHi32<<32 | buffLo32;
//size_t count;
uint32_t countHi32, countLo32; // count = countHi32<<32 | countLo32;
int chunkSize;
};
template<typename Int>
__host__ __device__ inline void ncclCollCbdPart(
struct ncclDevWorkColl* work, uint32_t channelId, int proto, int eltSize,
Int* count, Int* partOffset, Int* partCount, Int* chunkCount
) {
int eltPerGrain = ncclProtoGrainSize(proto)/eltSize;
int nMidChannels = work->channelHi - work->channelLo - 1;
// We can assum that nMidChannels<0 implies countMid==0, which let's us assume
// that countMid*nMidChannels == 0.
if (count != nullptr) {
*count = work->cbd.countLo + work->cbd.countMid*nMidChannels + work->cbd.countHi;
}
if (channelId == work->channelLo) {
*partOffset = 0;
*partCount = work->cbd.countLo;
*chunkCount = work->cbd.chunkGrainsLo*eltPerGrain;
} else if (channelId == work->channelHi) {
*partOffset = work->cbd.countLo + nMidChannels*work->cbd.countMid;
*partCount = work->cbd.countHi;
*chunkCount = work->cbd.chunkGrainsHi*eltPerGrain;
} else {
int mid = channelId - work->channelLo - 1;
*partOffset = work->cbd.countLo + mid*work->cbd.countMid;
*partCount = work->cbd.countMid;
*chunkCount = work->cbd.chunkGrainsMid*eltPerGrain;
}
}
static_assert(((NCCL_WORK_SIZE - alignUp(sizeof(ncclWorkHeader), alignof(ncclWorkElemP2p)))/sizeof(ncclWorkElemP2p)) >= 16, "Sanity check: NCCL_MAX_WORK_ELEMENTS_P2P == 16");
#define NCCL_MAX_WORK_ELEMENTS_P2P 16
struct ncclWorkElemReg {
struct ncclWorkElem elem;
struct alignas(16) ncclDevWorkCollReg {
struct ncclDevWorkColl coll;
void* dnInputs[NCCL_MAX_DIRECT_ARITY+1];
void* dnOutputs[NCCL_MAX_DIRECT_ARITY+1];
void* upOutputs[NCCL_MAX_DIRECT_ARITY+1];
};
#define NCCL_MAX_WORK_ELEMENTS_REG ((NCCL_WORK_SIZE - alignUp(sizeof(ncclWorkHeader), alignof(ncclWorkElemReg)))/sizeof(ncclWorkElemReg))
static_assert(NCCL_MAX_WORK_ELEMENTS_REG == 2, "Sanity check: NCCL_MAX_WORK_ELEMENTS_REG == 2");
// Number of named barriers supported by CUDA
#define NCCL_MAX_GROUPS 16
struct ncclWork {
struct ncclWorkHeader header;
union {
char pad[NCCL_WORK_SIZE - sizeof(struct ncclWorkHeader)];
struct ncclWorkElem elems[NCCL_MAX_WORK_ELEMENTS];
struct ncclWorkElemP2p p2pElems[NCCL_MAX_WORK_ELEMENTS_P2P];
struct ncclWorkElemReg regElems[NCCL_MAX_WORK_ELEMENTS_REG];
};
enum ncclDevWorkType: uint8_t {
ncclDevWorkTypeP2p,
ncclDevWorkTypeColl,
ncclDevWorkTypeCollReg
};
constexpr size_t ncclDevWorkSize(enum ncclDevWorkType type) {
return type == ncclDevWorkTypeP2p ? sizeof(ncclDevWorkP2p) :
type == ncclDevWorkTypeColl ? sizeof(ncclDevWorkColl) : sizeof(ncclDevWorkCollReg);
}
#define NCCL_MAX_DEV_WORK_BATCH_BYTES 1024
#define NCCL_MAX_DEV_WORK_BATCH_COLLS (NCCL_MAX_DEV_WORK_BATCH_BYTES/sizeof(ncclDevWorkColl))
#define NCCL_MAX_DEV_WORK_P2P_PER_BATCH 8
struct alignas(16) ncclDevWorkBatch {
union {
struct {
// nextExtends: should next one be merged into this one.
// nextJump=0: end of this channel's batch list
// nextJump>0: batches[thisIndex+nextJump] is next batch in this list
uint32_t nextJump:14, nextExtends:1;
uint32_t workType:2, funcId:15;
};
// Unioning bitfields with underlying type hints compiler to emit the best
// SASS LD/ST accesses.
uint32_t flags;
};
// Rolling offset in fifo where this batch's work structs begin
uint32_t offsetBase;
// Set of relative offsets from offsetBase for this channel's subset of the batch:
// For each bit index i in offsetMask, find work at fifo offset: offsetBase + i*sizeof(WorkStructType)
uint64_t offsetBitset;
};
static_assert(sizeof(struct ncclWork) == NCCL_WORK_SIZE, "Sanity check: sizeof(struct ncclWork) == NCCL_WORK_SIZE");
static_assert(sizeof(struct ncclWork)%16 == 0, "Sanity check: sizeof(struct ncclWork)%16 == 0");
struct ncclDevChannelPeer {
// Stripped version of ncclChannelPeer where we only keep the ncclConnInfo
@@ -328,9 +385,8 @@ struct ncclDevComm {
int buffSizes[NCCL_NUM_PROTOCOLS];
int p2pChunkSize;
// Operation list for aggregation
int workFifoDepth;
struct ncclWork* workFifoHeap; // may be cudaHost or GDR memory
// Work fifo return credits
uint32_t* workConsumed/*[MAXCHANNELS]*/;
int* collNetDenseToUserRank;
@@ -346,11 +402,37 @@ struct alignas(16) ncclDevCommAndChannels {
struct ncclDevChannel channels[MAXCHANNELS];
};
#ifdef __CUDA_ARCH__
#define NCCL_CUDA_ARCH __CUDA_ARCH__
#else
#define NCCL_CUDA_ARCH 0
#endif
enum ncclDevWorkStorageType: uint8_t {
ncclDevWorkStorageTypeArgs=0,
ncclDevWorkStorageTypeFifo=1,
ncclDevWorkStorageTypePersistent=2
};
struct alignas(16) ncclDevKernelArgs {
struct ncclDevComm* comm;
uint64_t channelMask;
enum ncclDevWorkStorageType workStorageType;
uint32_t workMask;
void* workBuf;
// A channel's first batch is at `blockIdx.x`. Use `nextJump` to follow rest of list.
// struct ncclDevWorkBatch batches[];
};
__host__ __device__ constexpr int ncclMaxKernelArgsSize(/*int cudaDriver, */int cudaArch=NCCL_CUDA_ARCH) {
//return (cudaArch < 700 || cudaDriver < 12010) ? 4<<10 : (32<<10)-4;
return 4<<10;
}
template<size_t capacity>
struct alignas(16) ncclDevKernelArgsStorage {
union {
struct ncclDevKernelArgs args;
ulong2 storage[capacity/sizeof(ulong2)];
};
};
typedef ncclDevKernelArgsStorage<(4<<10)> ncclDevKernelArgs4K;
//typedef ncclDevKernelArgsStorage<(32<<10)-4> ncclDevKernelArgs31K;
template<typename T>
__host__ __device__ constexpr T min_constexpr(T a) { return a; }
@@ -366,6 +448,10 @@ __host__ __device__ constexpr T max_constexpr(T a, T b, Ts ...c) {
return max_constexpr<T>((a > b ? a : b), c...);
}
constexpr int ncclDevMaxChannelsForArgsBytes(size_t argsBytes) {
return min_constexpr<size_t>(MAXCHANNELS, (argsBytes - sizeof(struct ncclDevKernelArgs))/sizeof(struct ncclDevWorkBatch));
}
// Calculate the unroll factor given:
// * bytePerPack: number of bytes accessed per instruction
// * insns: max permissible unroll value
@@ -412,6 +498,7 @@ extern int const ncclDevKernelCount;
extern void* const ncclDevKernelList[/*ncclDevKernelCount*/];
// Table of most specialized kernel function to run given func index.
extern int const ncclDevFuncIdCount;
extern int const ncclDevFuncRowToId[];
extern void* const ncclDevKernelForFunc[/*funcIndex*/];
extern bool const ncclDevKernelForFuncIsSpecialized[/*funcIndex*/];
src/include/enqueue.h
+1
Féach ar an gComhad
@@ -24,5 +24,6 @@ ncclResult_t ncclLaunchKernelBefore_NoUncapturedCuda(struct ncclComm* comm, stru
ncclResult_t ncclLaunchKernel(struct ncclComm* comm, struct ncclKernelPlan* plan);
ncclResult_t ncclLaunchKernelAfter_NoCuda(struct ncclComm* comm, struct ncclKernelPlan* plan);
ncclResult_t ncclLaunchFinish(struct ncclComm* comm);
ncclResult_t ncclPrepareTasks(struct ncclComm* comm, bool* algoNeedConnect, bool* needConnect, ncclSimInfo_t* simInfo);
#endif // End include guard
src/include/gdrwrap.h
+5 -4
Féach ar an gComhad
@@ -8,6 +8,7 @@
#define NCCL_GDRWRAP_H_
#include "nccl.h"
#include "alloc.h"
#include <stdint.h> // for standard [u]intX_t types
#include <stdio.h>
#include <stdlib.h>
@@ -194,7 +195,7 @@ static ncclResult_t ncclGdrCudaCalloc(T** ptr, T** devPtr, size_t nelem, void**
char *devMem;
void *gdrMap;
mapSize = sizeof(T)*nelem;
mapSize = ncclSizeOfT<T>()*nelem;
// GDRCOPY Pinned buffer has to be a minimum of a GPU_PAGE_SIZE
ALIGN_SIZE(mapSize, GPU_PAGE_SIZE);
@@ -203,7 +204,7 @@ static ncclResult_t ncclGdrCudaCalloc(T** ptr, T** devPtr, size_t nelem, void**
uint64_t alignedAddr = (((uint64_t) devMem) + GPU_PAGE_OFFSET) & GPU_PAGE_MASK;
size_t align = alignedAddr - (uint64_t)devMem;
//TRACE(NCCL_INIT, "GDRCOPY: Pin buffer 0x%lx (%p) align %zi size %zi", alignedAddr, devMem, align, mapSize);
//TRACE(NCCL_INIT, "GDRCOPY: Pin buffer 0x%lx (%p) align %zu size %zu", alignedAddr, devMem, align, mapSize);
NCCLCHECK(wrap_gdr_pin_buffer(ncclGdrCopy, alignedAddr, mapSize, 0, 0, &mh));
NCCLCHECK(wrap_gdr_map(ncclGdrCopy, mh, &gdrMap, mapSize));
@@ -226,7 +227,7 @@ static ncclResult_t ncclGdrCudaCalloc(T** ptr, T** devPtr, size_t nelem, void**
*ptr = (T *)((char *)gdrMap+off);
if (devPtr) *devPtr = (T *)(devMem+off+align);
TRACE(NCCL_INIT, "GDRCOPY : allocated devMem %p gdrMap %p offset %lx mh %lx mapSize %zi at %p",
TRACE(NCCL_INIT, "GDRCOPY : allocated devMem %p gdrMap %p offset %lx mh %lx mapSize %zu at %p",
md->gdrDevMem, md->gdrMap, md->gdrOffset, md->gdrMh.h, md->gdrMapSize, *ptr);
return ncclSuccess;
@@ -235,7 +236,7 @@ static ncclResult_t ncclGdrCudaCalloc(T** ptr, T** devPtr, size_t nelem, void**
template <typename T>
static ncclResult_t ncclGdrCudaCopy(void *gdrHandle, T* dst, T* src, size_t nelem) {
gdr_mem_desc_t *md = (gdr_mem_desc_t*)gdrHandle;
NCCLCHECK(wrap_gdr_copy_to_mapping(md->gdrMh, dst, src, nelem*sizeof(T)));
NCCLCHECK(wrap_gdr_copy_to_mapping(md->gdrMh, dst, src, nelem*ncclSizeOfT<T>()));
return ncclSuccess;
}
src/include/graph.h
+5 -2
Féach ar an gComhad
@@ -29,6 +29,7 @@ ncclResult_t ncclTopoTrimSystem(struct ncclTopoSystem* system, struct ncclComm*
ncclResult_t ncclTopoComputeP2pChannels(struct ncclComm* comm);
ncclResult_t ncclTopoGetNvbGpus(struct ncclTopoSystem* system, int rank, int* nranks, int** ranks);
int ncclTopoPathAllNVLink(struct ncclTopoSystem* system);
ncclResult_t ncclTopoComputeCommCPU(struct ncclComm* comm);
// Query topology
ncclResult_t ncclTopoGetNetDev(struct ncclComm* comm, int rank, struct ncclTopoGraph* graph, int channelId, int peerRank, int64_t* id, int* dev, int* proxyRank);
@@ -46,9 +47,11 @@ ncclResult_t ncclTopoGetCpuAffinity(struct ncclTopoSystem* system, int rank, cpu
#define NCCL_TOPO_CPU_ARCH_X86 1
#define NCCL_TOPO_CPU_ARCH_POWER 2
#define NCCL_TOPO_CPU_ARCH_ARM 3
#define NCCL_TOPO_CPU_ARCH_MIXED 4
#define NCCL_TOPO_CPU_VENDOR_INTEL 1
#define NCCL_TOPO_CPU_VENDOR_AMD 2
#define NCCL_TOPO_CPU_VENDOR_ZHAOXIN 3
#define NCCL_TOPO_CPU_VENDOR_MIXED 4
#define NCCL_TOPO_CPU_TYPE_BDW 1
#define NCCL_TOPO_CPU_TYPE_SKL 2
#define NCCL_TOPO_CPU_TYPE_YONGFENG 1
@@ -70,6 +73,7 @@ ncclResult_t ncclTopoSearchInit(struct ncclTopoSystem* system);
#define NCCL_TOPO_PATTERN_TREE 3 // All NIC traffic going to/from the same GPU
#define NCCL_TOPO_PATTERN_RING 4 // Ring
#define NCCL_TOPO_PATTERN_NVLS 5 // NVLS+SHARP and NVLS+Tree
#define NCCL_TOPO_PATTERN_COLLNET_DIRECT 6 // Collnet Direct
struct ncclTopoGraph {
// Input / output
int id; // ring : 0, tree : 1, collnet : 2
@@ -113,7 +117,6 @@ ncclResult_t ncclTopoPostset(struct ncclComm* comm, int* firstRanks, int* treePa
struct ncclTopoRanks** allTopoRanks, int* rings, struct ncclTopoGraph** graphs, struct ncclComm* parent);
ncclResult_t ncclTopoTuneModel(struct ncclComm* comm, int minCompCap, int maxCompCap, struct ncclTopoGraph** graphs);
#include "info.h"
ncclResult_t ncclTopoGetAlgoTime(struct ncclInfo* info, int algorithm, int protocol, int numPipeOps, float* time, bool* backup = NULL);
ncclResult_t ncclTopoGetAlgoTime(struct ncclComm* comm, int coll, int algorithm, int protocol, size_t nBytes, int numPipeOps, float* time, bool* backup=nullptr);
#endif
src/include/group.h
+11 -4
Féach ar an gComhad
@@ -35,9 +35,12 @@ struct ncclAsyncJob {
void(*undo)(struct ncclAsyncJob*);
void(*destructor)(void*);
ncclGroupJobState_t state;
volatile uint32_t *abortFlag; /* point to comm abortFlag */
volatile uint32_t *childAbortFlag; /* point to child abortFlag */
uint32_t* abortFlag; /* point to comm abortFlag */
uint32_t* abortFlagDev; /* point to comm abortFlagDev */
uint32_t* childAbortFlag; /* point to child abortFlag */
uint32_t* childAbortFlagDev; /* point to child abortFlagDev */
ncclComm_t comm;
int destroyFlag;
};
ncclResult_t ncclAsyncLaunch(
@@ -52,14 +55,14 @@ struct ncclGroupJob {
struct ncclComm **groupCommHeadPtr;
struct ncclComm **groupCommPreconnectHeadPtr;
ncclResult_t *groupErrorPtr;
volatile bool *abortFlagPtr;
bool *abortFlagPtr;
int *groupBlockingPtr;
struct ncclIntruQueue<struct ncclAsyncJob, &ncclAsyncJob::next> *asyncJobsPtr;
bool initialized;
};
ncclResult_t ncclGroupStartInternal();
ncclResult_t ncclGroupEndInternal();
ncclResult_t ncclGroupEndInternal(ncclSimInfo_t* simInfo = NULL);
ncclResult_t ncclAsyncJobComplete(struct ncclAsyncJob* job);
////////////////////////////////////////////////////////////////////////////////
@@ -114,6 +117,10 @@ inline void ncclGroupCommJoin(struct ncclComm* comm) {
// Comms gets a new memory stack scope upon joining. Each task batched for
// this comm is allocated there.
ncclMemoryStackPush(&comm->memScoped);
// Initialize planner
ncclKernelPlanner::Peer* tmp = comm->planner.peers;
memset(&comm->planner, 0, sizeof(comm->planner));
comm->planner.peers = tmp;
}
ncclGroupBlocking = comm->config.blocking;
src/include/info.h
-123
Féach ar an gComhad
@@ -8,28 +8,9 @@
#define NCCL_INFO_H_
#include "nccl.h"
#include "device.h"
#include "collectives.h"
#include "core.h"
#include "utils.h"
#include "strongstream.h"
#define NCCL_MAX_LOCAL_RANKS 64
typedef enum : uint8_t {
ncclPatternRing,
ncclPatternRingTwice,
ncclPatternPipelineFrom,
ncclPatternPipelineTo,
ncclPatternTreeUp,
ncclPatternTreeDown,
ncclPatternTreeUpDown,
ncclPatternCollnetChain,
ncclPatternCollnetDirect,
ncclPatternNvls,
ncclPatternNvlsTree,
ncclPatternSend,
ncclPatternRecv
} ncclPattern_t;
// Used to pass NCCL call information between functions
struct ncclInfo {
@@ -47,110 +28,6 @@ struct ncclInfo {
// Algorithm details
int chunkSteps;
int sliceSteps;
// Computed later
ncclDevRedOpFull opFull;
ncclPattern_t pattern;
size_t nBytes;
size_t aggnBytes;
size_t workBytes;
size_t sendbuffSize;
size_t recvbuffSize;
int stepSize;
int chunkCount;
int chunkSize;
int channelId;
int workFuncIndex;
ncclRegBufferType regBufType;
void* regBufSend[NCCL_MAX_LOCAL_RANKS];
void* regBufRecv[NCCL_MAX_LOCAL_RANKS];
// collnet buffer reg handles
void* sendMhandle;
void* recvMhandle;
// Need to initialize
int nThreads;
int nChannels;
int algorithm;
int protocol;
bool userTuned;
struct ncclInfo *next;
};
inline ncclResult_t ncclInfoSetDerived(struct ncclInfo* info, int nRanks) {
info->nBytes = info->workBytes = info->count * ncclTypeSize(info->datatype);
if (info->coll == ncclFuncAllGather || info->coll == ncclFuncBroadcast) {
info->count = info->workBytes;
info->datatype = ncclInt8;
}
if (info->coll == ncclFuncAllGather || info->coll == ncclFuncReduceScatter) info->nBytes *= nRanks; // count is per rank
/* compute buffer size for NVLS buffer registration */
if (info->coll == ncclFuncAllGather) {
info->sendbuffSize = info->workBytes;
info->recvbuffSize = info->sendbuffSize * nRanks;
} else if (info->coll == ncclFuncReduceScatter) {
info->recvbuffSize = info->workBytes;
info->sendbuffSize = info->recvbuffSize * nRanks;
} else {
info->sendbuffSize = info->recvbuffSize = info->workBytes;
}
return ncclSuccess;
}
struct ncclTaskColl {
struct ncclTaskColl* next;
ncclFunc_t func;
void const* sendbuff;
void* recvbuff;
size_t count;
int root;
ncclDataType_t datatype;
ncclDevRedOpFull op;
int chunkSteps, sliceSteps;
struct ncclInfo info;
};
struct ncclTaskP2p {
ncclTaskP2p *next;
void *buff;
size_t bytes;
// Stateful chunk index. If a p2p gets "cut" over two plans this keeps track
// of where it left off.
int chunk;
};
struct ncclCudaStreamList {
struct ncclCudaStreamList *next;
cudaStream_t stream;
};
struct ncclTasks {
struct Peer {
bool sendSeen, recvSeen;
struct ncclIntruQueue<struct ncclTaskP2p, &ncclTaskP2p::next> sendQueue;
struct ncclIntruQueue<struct ncclTaskP2p, &ncclTaskP2p::next> recvQueue;
};
struct ncclIntruQueue<struct ncclInfo, &ncclInfo::next> collQueue;
// Queue for user-tuned executed collectives
struct ncclIntruQueue<struct ncclInfo, &ncclInfo::next> collTunedQueue;
// Queue for continuous bytes distribution (CBD) collectives
struct ncclIntruQueue<struct ncclInfo, &ncclInfo::next> collCBDQueue;
// Queue for collnet
struct ncclIntruQueue<struct ncclInfo, &ncclInfo::next> collnetQueue;
size_t workBytesTotal;
int usableChannels;
bool sorted;
struct Peer* peers/*[nRanks]*/;
int *p2pSendOrder, *p2pRecvOrder;
int p2pOrderSteps;
int nTasksColl, nTasksP2p;
// The list of user streams aggregated over all tasks present.
struct ncclCudaStreamList* streams;
// The most recent user stream. Ignored if streams==nullptr
cudaStream_t streamRecent;
// The graph capturing all user streams or invalid if none. Thus we restrict the
// user that all streams must be captured in the same graph or not captured
// at all. Technically we could probably relax this, but that would mean
// collecting a different `ncclTasks` per graph and one for non-graph.
struct ncclCudaGraph capturingGraph;
};
#endif
src/include/nccl_common.h
+28 -2
Féach ar an gComhad
@@ -7,8 +7,33 @@
#ifndef NCCL_DEBUG_H_
#define NCCL_DEBUG_H_
typedef enum {NCCL_LOG_NONE=0, NCCL_LOG_VERSION=1, NCCL_LOG_WARN=2, NCCL_LOG_INFO=3, NCCL_LOG_ABORT=4, NCCL_LOG_TRACE=5} ncclDebugLogLevel;
typedef enum {NCCL_INIT=1, NCCL_COLL=2, NCCL_P2P=4, NCCL_SHM=8, NCCL_NET=16, NCCL_GRAPH=32, NCCL_TUNING=64, NCCL_ENV=128, NCCL_ALLOC=256, NCCL_CALL=512, NCCL_PROXY=1024, NCCL_NVLS=2048, NCCL_BOOTSTRAP=4096, NCCL_REG=8192, NCCL_ALL=~0} ncclDebugLogSubSys;
typedef enum {
NCCL_LOG_NONE = 0,
NCCL_LOG_VERSION = 1,
NCCL_LOG_WARN = 2,
NCCL_LOG_INFO = 3,
NCCL_LOG_ABORT = 4,
NCCL_LOG_TRACE = 5
} ncclDebugLogLevel;
typedef enum {
NCCL_INIT = 0x1,
NCCL_COLL = 0x2,
NCCL_P2P = 0x4,
NCCL_SHM = 0x8,
NCCL_NET = 0x10,
NCCL_GRAPH = 0x20,
NCCL_TUNING = 0x40,
NCCL_ENV = 0x80,
NCCL_ALLOC = 0x100,
NCCL_CALL = 0x200,
NCCL_PROXY = 0x400,
NCCL_NVLS = 0x800,
NCCL_BOOTSTRAP = 0x1000,
NCCL_REG = 0x2000,
NCCL_PROFILE = 0x4000,
NCCL_ALL = ~0
} ncclDebugLogSubSys;
typedef void (*ncclDebugLogger_t)(ncclDebugLogLevel level, unsigned long flags, const char *file, int line, const char *fmt, ...);
@@ -40,4 +65,5 @@ typedef enum {
#define NCCL_PROTO_LL128 1
#define NCCL_PROTO_SIMPLE 2
#define NCCL_ALGO_PROTO_IGNORE -1.0
#endif
src/include/nccl_tuner.h
+50 -6
Féach ar an gComhad
@@ -11,6 +11,54 @@
#include "nccl.h"
#include "nccl_common.h"
// API to be implemented by external tuner
typedef struct {
// Name of the tuner
const char* name;
// Initializes tuner states.
// Inputs:
// - nRanks: number of ranks in current communicator. Each communicator initialize its own tuner.
// - nNodes: number of nodes in current communicator.
// - logFunction: a logFunction can be useful to integrate logging together with NCCL core.
// Outputs:
// - context: tuner context object
ncclResult_t (*init)(size_t nRanks, size_t nNodes, ncclDebugLogger_t logFunction, void **context);
// Gets info (algo, protocol, number of ctas and threads) for a given collective.
// Inputs:
// - context: tuner context object
// - collType: collective type , e.g., allreduce, allgather…
// - nBytes: collective size in bytes
// - numPipeOps: number of operations in the group
// - numAlgo: number of algorithms in collCostTable
// - numProto: number of protocols in collCostTable
//
// Outputs:
// - nChannels: number of channels (hence SMs) to be used.
//
// InOut:
// - collCostTable: collective cost table, generated by NCCL core, containing algo|proto|time entries for collType.
// NCCL core sets ignored algo/proto cost table entries to -1.0 (NCCL_ALGO_PROTO_IGNORE).
//
// If getCollInfo() does not return ncclSuccess, NCCL will fall back to the
// default tuning for the given collective.
// Also, the plugin is allowed to not set any output, or set only the
// algorithm and protocol, but not only the algorithm or only the protocol.
// Unset fields will be set automatically by NCCL.
ncclResult_t (*getCollInfo)(void* context, ncclFunc_t collType, size_t nBytes,
int numPipeOps, float** collCostTable, int numAlgo, int numProto,
int* nChannels);
// Terminates the plugin and cleans up any resources that the plugin allocated.
// context: tuner context object
ncclResult_t (*destroy)(void* context);
} ncclTuner_v3_t;
typedef ncclTuner_v3_t ncclTuner_t;
#define NCCL_TUNER_PLUGIN_SYMBOL "ncclTunerPlugin_v3"
// API to be implemented by external tuner
typedef struct {
// Name of the tuner
@@ -36,7 +84,7 @@ typedef struct {
//
// Outputs:
// - algorithm: selected algorithm to be used for the given collective
// - protocol: selected protocol to be used for the given collective
// - protocol: selected protocol to be used for the give collective
// - nChannels: number of channels (hence SMs) to be used.
//
// If getCollInfo() does not return ncclSuccess, NCCL will fall back to the
@@ -46,15 +94,11 @@ typedef struct {
// Unset fields will be set automatically by NCCL.
ncclResult_t (*getCollInfo)(void* context, ncclFunc_t collType, size_t nBytes,
int collNetSupport, int nvlsSupport, int numPipeOps,
int *algorithm, int *protocol, int* nChannels);
int* algorithm, int* protocol, int* nChannels);
// Terminates the plugin and cleans up any resources that the plugin allocated.
// context: tuner context object
ncclResult_t (*destroy)(void* context);
} ncclTuner_v2_t;
typedef ncclTuner_v2_t ncclTuner_t;
#define NCCL_TUNER_PLUGIN_SYMBOL "ncclTunerPlugin_v2"
#endif
src/include/net.h
+3 -1
Féach ar an gComhad
@@ -14,8 +14,10 @@
typedef char ncclNetHandle_t[NCCL_NET_HANDLE_MAXSIZE];
ncclResult_t ncclNetPluginInit();
ncclResult_t ncclNetPluginLoad(struct ncclComm* comm);
ncclResult_t ncclNetPluginUnload(struct ncclComm* comm);
ncclResult_t ncclNetInit(struct ncclComm* comm);
ncclResult_t ncclNetFinalize(struct ncclComm* comm);
int ncclNetVersion(struct ncclComm* comm);
// Test whether the current GPU support GPU Direct RDMA.
src/include/nvmlwrap.h
+38
Féach ar an gComhad
@@ -253,6 +253,38 @@ typedef nvmlGpuFabricInfo_v2_t nvmlGpuFabricInfoV_t;
*/
#define nvmlGpuFabricInfo_v2 NVML_STRUCT_VERSION(GpuFabricInfo, 2)
/**
* Confidential Compute Feature Status values
*/
#define NVML_CC_SYSTEM_FEATURE_DISABLED 0
#define NVML_CC_SYSTEM_FEATURE_ENABLED 1
typedef struct nvmlConfComputeSystemState_st {
unsigned int environment;
unsigned int ccFeature;
unsigned int devToolsMode;
} nvmlConfComputeSystemState_t;
/**
* Confidential Compute Multigpu mode values
*/
#define NVML_CC_SYSTEM_MULTIGPU_NONE 0
#define NVML_CC_SYSTEM_MULTIGPU_PROTECTED_PCIE 1
/**
* Confidential Compute System settings
*/
typedef struct {
unsigned int version;
unsigned int environment;
unsigned int ccFeature;
unsigned int devToolsMode;
unsigned int multiGpuMode;
} nvmlSystemConfComputeSettings_v1_t;
typedef nvmlSystemConfComputeSettings_v1_t nvmlSystemConfComputeSettings_t;
#define nvmlSystemConfComputeSettings_v1 NVML_STRUCT_VERSION(SystemConfComputeSettings, 1)
/* End of nvml.h */
#endif // NCCL_NVML_DIRECT
@@ -268,6 +300,11 @@ extern int ncclNvmlDeviceCount;
extern ncclNvmlDeviceInfo ncclNvmlDevices[ncclNvmlMaxDevices];
extern ncclNvmlDevicePairInfo ncclNvmlDevicePairs[ncclNvmlMaxDevices][ncclNvmlMaxDevices];
struct ncclNvmlCCStatus {
bool CCEnabled;
bool multiGpuCCEnabled;
};
// All ncclNvmlFoo() functions call ncclNvmlEnsureInitialized() implicitly.
// Outsiders need only call it if they want to inspect the ncclNvml global
// tables above.
@@ -283,5 +320,6 @@ ncclResult_t ncclNvmlDeviceGetCudaComputeCapability(nvmlDevice_t device, int* ma
ncclResult_t ncclNvmlDeviceGetP2PStatus(nvmlDevice_t device1, nvmlDevice_t device2, nvmlGpuP2PCapsIndex_t p2pIndex, nvmlGpuP2PStatus_t* p2pStatus);
ncclResult_t ncclNvmlDeviceGetFieldValues(nvmlDevice_t device, int valuesCount, nvmlFieldValue_t *values);
ncclResult_t ncclNvmlDeviceGetGpuFabricInfoV(nvmlDevice_t device, nvmlGpuFabricInfoV_t *gpuFabricInfo);
ncclResult_t ncclNvmlGetCCStatus(struct ncclNvmlCCStatus *status);
#endif // End include guard
src/include/nvtx.h
+1 -1
Féach ar an gComhad
@@ -63,7 +63,7 @@ class payload_schema {
nullptr,
NVTX_PAYLOAD_SCHEMA_TYPE_STATIC,
NVTX_PAYLOAD_SCHEMA_FLAG_NONE,
nullptr, 0, 0, 0};
nullptr, 0, 0, 0, 0, nullptr};
};
// Create NVTX push/pop range with parameters
src/include/nvtx3/nvToolsExt.h
+50 -46
Féach ar an gComhad
@@ -25,9 +25,9 @@
*
* \section INITIALIZATION_SECTION Initialization
*
* Typically the tool's library that plugs into NVTX is indirectly
* loaded via enviromental properties that are platform specific.
* For some platform or special cases, the user may be required
* Typically the tool's library that plugs into NVTX is indirectly
* loaded via enviromental properties that are platform specific.
* For some platform or special cases, the user may be required
* to instead explicity initialize instead though. This can also
* be helpful to control when the API loads a tool's library instead
* of what would typically be the first function call to emit info.
@@ -37,16 +37,16 @@
*
* Markers and ranges are used to describe events at a specific time (markers)
* or over a time span (ranges) during the execution of the application
* respectively.
* respectively.
*
* \subsection MARKERS Markers
*
*
* Markers denote specific moments in time.
*
*
*
*
* See \ref DOMAINS and \ref EVENT_ATTRIBUTES for additional information on
* how to specify the domain.
*
*
* \subsection THREAD_RANGES Thread Ranges
*
* Thread ranges denote nested time ranges. Nesting is maintained per thread
@@ -59,9 +59,9 @@
*
* \subsection PROCESS_RANGES Process Ranges
*
* Process ranges denote a time span that can expose arbitrary concurrency, as
* Process ranges denote a time span that can expose arbitrary concurrency, as
* opposed to thread ranges that only support nesting. In addition the range
* start event can happen on a different thread than the end marker. For the
* start event can happen on a different thread than the end marker. For the
* correlation of a start/end pair an unique correlation ID is used that is
* returned from the start API call and needs to be passed into the end API
* call.
@@ -87,15 +87,15 @@
*
* The function ::nvtxDomainCreateA or ::nvtxDomainCreateW is used to create
* a named domain.
*
*
* Each domain maintains its own
* - categories
* - thread range stacks
* - registered strings
*
* The function ::nvtxDomainDestroy marks the end of the domain. Destroying
* a domain unregisters and destroys all objects associated with it such as
* registered strings, resource objects, named categories, and started ranges.
* The function ::nvtxDomainDestroy marks the end of the domain. Destroying
* a domain unregisters and destroys all objects associated with it such as
* registered strings, resource objects, named categories, and started ranges.
*
* \section RESOURCE_NAMING Resource Naming
*
@@ -105,41 +105,41 @@
* The functions can be called multiple times during the execution of an
* application, however, in that case it is implementation dependent which
* name will be reported by the tool.
*
*
* \subsection CATEGORY_NAMING Category Naming
*
* Some function in this library support associating an integer category
* to enable filtering and sorting. The category naming functions allow
* the application to associate a user friendly name with the integer
* category. Support for domains have been added in NVTX_VERSION_2 to
* avoid collisions when domains are developed independantly.
* Some function in this library support associating an integer category
* to enable filtering and sorting. The category naming functions allow
* the application to associate a user friendly name with the integer
* category. Support for domains have been added in NVTX_VERSION_2 to
* avoid collisions when domains are developed independantly.
*
* \subsection RESOURCE_OBJECTS Resource Objects
*
* Resource objects are a generic mechanism for attaching data to an application
* resource. The identifier field makes the association to a pointer or handle,
* while the type field helps provide deeper understanding of the identifier as
* Resource objects are a generic mechanism for attaching data to an application
* resource. The identifier field makes the association to a pointer or handle,
* while the type field helps provide deeper understanding of the identifier as
* well as enabling differentiation in cases where handles generated by different
* APIs may collide. The resource object may also have an associated message to
* associate with the application resource, enabling further annotation of this
* associate with the application resource, enabling further annotation of this
* object and how it is used.
*
*
* The resource object was introduced in NVTX_VERSION_2 to supersede existing naming
* functions and allow the application resource identified by those functions to be
* associated to a domain. The other naming functions are still supported for backward
* compatibility but will be associated only to the default domain.
*
* \subsection RESOURCE_NAMING_OS Resource Naming
*
* Some operating system resources creation APIs do not support providing a user friendly
* name, such as some OS thread creation APIs. This API support resource naming though
* both through resource objects and functions following the pattern
* nvtxName[RESOURCE_TYPE][A|W](identifier, name). Resource objects introduced in NVTX_VERSION 2
*
* Some operating system resources creation APIs do not support providing a user friendly
* name, such as some OS thread creation APIs. This API support resource naming though
* both through resource objects and functions following the pattern
* nvtxName[RESOURCE_TYPE][A|W](identifier, name). Resource objects introduced in NVTX_VERSION 2
* supersede the other functions with a a more general method of assigning names to OS resources,
* along with associating them to domains too. The older nvtxName* functions are only associated
* along with associating them to domains too. The older nvtxName* functions are only associated
* with the default domain.
* \section EXTENSIONS Optional Extensions
* Optional extensions will either appear within the existing sections the extend or appear
* Optional extensions will either appear within the existing sections the extend or appear
* in the "Related Pages" when they introduce new concepts.
*/
@@ -159,7 +159,11 @@
#define NVTX_INLINE_STATIC __inline static
#else /*defined(__GNUC__)*/
#define NVTX_API
#if defined(__cplusplus) || (defined(__STDC_VERSION__) && __STDC_VERSION__ >= 199901L)
#define NVTX_INLINE_STATIC inline static
#else
#define NVTX_INLINE_STATIC __inline__ static
#endif
#endif /* Platform */
#if defined(NVTX_NO_IMPL)
@@ -212,7 +216,7 @@
extern "C" {
#endif /* __cplusplus */
/**
/**
* Result Codes
*/
@@ -281,12 +285,12 @@ typedef enum nvtxColorType_t
* ------------------------------------------------------------------------- */
typedef enum nvtxMessageType_t
{
NVTX_MESSAGE_UNKNOWN = 0, /**< Message payload is unused. */
NVTX_MESSAGE_UNKNOWN = 0, /**< Message attribute is unused. */
NVTX_MESSAGE_TYPE_ASCII = 1, /**< A character sequence is used as payload. */
NVTX_MESSAGE_TYPE_UNICODE = 2, /**< A wide character sequence is used as payload. */
/* NVTX_VERSION_2 */
NVTX_MESSAGE_TYPE_REGISTERED = 3, /**< A unique string handle that was registered
with \ref nvtxDomainRegisterStringA() or
with \ref nvtxDomainRegisterStringA() or
\ref nvtxDomainRegisterStringW(). */
} nvtxMessageType_t;
@@ -338,7 +342,7 @@ NVTX_DECLSPEC void NVTX_API nvtxInitialize(const void* reserved);
* ------------------------------------------------------------------------- */
typedef enum nvtxPayloadType_t
{
NVTX_PAYLOAD_UNKNOWN = 0, /**< Color payload is unused. */
NVTX_PAYLOAD_UNKNOWN = 0, /**< Payload attribute is unused. */
NVTX_PAYLOAD_TYPE_UNSIGNED_INT64 = 1, /**< A 64 bit unsigned integer value is used as payload. */
NVTX_PAYLOAD_TYPE_INT64 = 2, /**< A 64 bit signed integer value is used as payload. */
NVTX_PAYLOAD_TYPE_DOUBLE = 3, /**< A 64 bit floating point value is used as payload. */
@@ -714,10 +718,10 @@ NVTX_DECLSPEC nvtxRangeId_t NVTX_API nvtxRangeStartW(const wchar_t* message);
/* ------------------------------------------------------------------------- */
/** \brief Ends a process range.
*
* \param domain - The domain
* \param domain - The domain
* \param id - The correlation ID returned from a nvtxRangeStart call.
*
* \remarks This function is offered completeness but is an alias for ::nvtxRangeEnd.
* \remarks This function is offered completeness but is an alias for ::nvtxRangeEnd.
* It does not need a domain param since that is associated iwth the range ID at ::nvtxDomainRangeStartEx
*
* \par Example:
@@ -929,10 +933,10 @@ NVTX_DECLSPEC int NVTX_API nvtxRangePop(void);
/* ------------------------------------------------------------------------- */
/** \cond SHOW_HIDDEN
* \brief Resource typing helpers.
* \brief Resource typing helpers.
*
* Classes are used to make it easy to create a series of resource types
* per API without collisions
* Classes are used to make it easy to create a series of resource types
* per API without collisions
*/
#define NVTX_RESOURCE_MAKE_TYPE(CLASS, INDEX) ((((uint32_t)(NVTX_RESOURCE_CLASS_ ## CLASS))<<16)|((uint32_t)(INDEX)))
#define NVTX_RESOURCE_CLASS_GENERIC 1
@@ -1062,7 +1066,7 @@ typedef struct nvtxResourceAttributes_v0
int32_t identifierType; /* values from enums following the pattern nvtxResource[name]Type_t */
/**
* \brief Identifier for the resource.
* \brief Identifier for the resource.
* \anchor RESOURCE_IDENTIFIER_FIELD
*
* An identifier may be a pointer or a handle to an OS or middleware API object.
@@ -1093,7 +1097,7 @@ typedef struct nvtxResourceAttributes_v0
typedef struct nvtxResourceAttributes_v0 nvtxResourceAttributes_t;
/* \cond SHOW_HIDDEN
/* \cond SHOW_HIDDEN
* \version \NVTX_VERSION_2
*/
#define NVTX_RESOURCE_ATTRIB_STRUCT_SIZE ( (uint16_t)( sizeof(nvtxResourceAttributes_v0) ) )
@@ -1106,7 +1110,7 @@ typedef struct nvtxResourceHandle* nvtxResourceHandle_t;
/** \brief Create a resource object to track and associate data with OS and middleware objects
*
* Allows users to associate an API handle or pointer with a user-provided name.
*
*
*
* \param domain - Domain to own the resource object
* \param attribs - Attributes to be associated with the resource
@@ -1240,7 +1244,7 @@ NVTX_DECLSPEC void NVTX_API nvtxNameCategoryW(uint32_t category, const wchar_t*
* POSIX pthread_t type returned by pthread_self() may not comply with these
* expectations. Please use OS-specific thread ID instead of pthread_t.
*
* The thread name is associated to the default domain. To support domains
* The thread name is associated to the default domain. To support domains
* use resource objects via ::nvtxDomainResourceCreate.
*
* \param threadId - The ID of the thread to name.
@@ -1457,7 +1461,7 @@ NVTX_DECLSPEC void NVTX_API nvtxDomainDestroy(nvtxDomainHandle_t domain);
} /* extern "C" */
#endif /* __cplusplus */
#define NVTX_IMPL_GUARD /* Ensure other headers cannot included directly */
#define NVTX_IMPL_GUARD /* Ensure other headers cannot be included directly */
#include "nvtxDetail/nvtxTypes.h"
src/include/nvtx3/nvToolsExtCounters.h
+335
Féach ar an gComhad
@@ -0,0 +1,335 @@
/**
* The NVTX counters extension is intended to collect counter values of various
* sources. It uses the NVTX payload extension to specify the data layout a
* counter group.
*
* A counter group is a set of counters that are collected together (at the same
* time). Counters are always registered as a group. Hence, a single counter is
* represented by a group with one counter.
*
* A sample refers to all values for a given timestamp. These values must
* include counter values and may include multiple instances of a counter group.
*
* The NVTX domain handle is the first argument to all counter collect
* functions. 0/NULL/nullptr represents the default domain (no domain).
*/
#include "nvToolsExtPayload.h"
#ifndef NVTOOLSEXT_COUNTERS_H
#define NVTOOLSEXT_COUNTERS_H
/**
* \brief The compatibility ID is used for versioning of this extension.
*/
#ifndef NVTX_EXT_COUNTERS_COMPATID
#define NVTX_EXT_COUNTERS_COMPATID 0x0101
#endif
/**
* \brief The module ID identifies the payload extension. It has to be unique
* among the extension modules.
*/
#ifndef NVTX_EXT_COUNTERS_MODULEID
#define NVTX_EXT_COUNTERS_MODULEID 4
#endif
/** Identifies an invalid scope and indicates an error if returned by `nvtxScopeRegister`. */
#define NVTX_SCOPE_NONE 0 /* no scope */
#define NVTX_SCOPE_ROOT 1
#define NVTX_SCOPE_CURRENT_HW_MACHINE 2 /* Node/machine name, Device? */
#define NVTX_SCOPE_CURRENT_HW_SOCKET 3
#define NVTX_SCOPE_CURRENT_HW_CPU 4
#define NVTX_SCOPE_CURRENT_HW_CPU_LOGICAL 5
/* Innermost HW execution context at registration time */
#define NVTX_SCOPE_CURRENT_HW_INNERMOST 6
/* Virtualized hardware, virtual machines, OS (if you don't know any better) */
#define NVTX_SCOPE_CURRENT_HYPERVISOR 7
#define NVTX_SCOPE_CURRENT_VM 8
#define NVTX_SCOPE_CURRENT_KERNEL 9
#define NVTX_SCOPE_CURRENT_CONTAINER 10
#define NVTX_SCOPE_CURRENT_OS 11
/* Software scopes */
#define NVTX_SCOPE_CURRENT_SW_PROCESS 12 /* Process scope */
#define NVTX_SCOPE_CURRENT_SW_THREAD 13 /* Thread scope */
#define NVTX_SCOPE_CURRENT_SW_FIBER 14
/* Innermost SW execution context at registration time */
#define NVTX_SCOPE_CURRENT_SW_INNERMOST 15
/** Static (user-provided) scope IDs (feed forward) */
#define NVTX_SCOPE_ID_STATIC_START (1 << 24)
/** Dynamically (tool) generated scope IDs */
#define NVTX_SCOPE_ID_DYNAMIC_START 4294967296 /* 1 << 32 */
/** Identifier of the semantic extension for counters. */
#define NVTX_SEMANTIC_ID_COUNTERS_V1 5
/*** Flags to augment the counter value. ***/
#define NVTX_COUNTERS_FLAG_NONE 0
/**
* Convert the fixed point value to a normalized floating point.
* Use the sign/unsign from the underlying type this flag is applied to.
* Unsigned [0f : 1f] or signed [-1f : 1f]
*/
#define NVTX_COUNTERS_FLAG_NORM (1 << 1)
/**
* Tools should apply scale and limits when graphing, ideally in a "soft" way to
* to see when limits are exceeded.
*/
#define NVTX_COUNTERS_FLAG_LIMIT_MIN (1 << 2)
#define NVTX_COUNTERS_FLAG_LIMIT_MAX (1 << 3)
#define NVTX_COUNTERS_FLAG_LIMITS \
(NVTX_COUNTERS_FLAG_LIMIT_MIN | NVTX_COUNTERS_FLAG_LIMIT_MAX)
/** Counter time scope **/
#define NVTX_COUNTERS_FLAG_TIME_POINT (1 << 5)
#define NVTX_COUNTERS_FLAG_TIME_SINCE_LAST (2 << 5)
#define NVTX_COUNTERS_FLAG_TIME_UNTIL_NEXT (3 << 5)
#define NVTX_COUNTERS_FLAG_TIME_SINCE_START (4 << 5)
/** Counter value type **/
#define NVTX_COUNTERS_FLAG_VALUE_ABSOLUTE (1 << 10)
#define NVTX_COUNTERS_FLAG_VALUE_DELTA (2 << 10) // delta to previous counter sample
/** Counter visualization hints **/
#define NVTX_COUNTERS_FLAG_INTERPOLATE (1 << 14)
/** Datatypes for limits union (value of `limitType`). */
#define NVTX_COUNTERS_LIMIT_I64 0
#define NVTX_COUNTERS_LIMIT_U64 1
#define NVTX_COUNTERS_LIMIT_F64 2
/** Reasons for the missing sample value. */
#define NVTX_COUNTERS_SAMPLE_ZERO 0
#define NVTX_COUNTERS_SAMPLE_UNCHANGED 1
#define NVTX_COUNTERS_SAMPLE_UNAVAILABLE 2
#ifdef __cplusplus
extern "C" {
#endif /* __cplusplus */
/**
* \brief Specify additional properties of a counter or counter group.
*/
typedef struct nvtxSemanticsCounter_v1
{
/** Header of the semantic extension (with identifier, version, etc.). */
struct nvtxSemanticsHeader_v1 header;
/**
* Flag if normalization, scale limits, etc. should be applied to counter
* values.
*/
uint64_t flags;
/** Unit of the counter value (case insensitive) */
const char* unit;
/** Should be 1 if not used. */
uint64_t unitScaleNumerator;
/** Should be 1 if not used. */
uint64_t unitScaleDenominator;
/** Determines the used union member. Use defines `NVTX_COUNTERS_LIMIT_*`. */
int64_t limitType;
/** Soft graph limit. */
union limits_t {
int64_t i64[2];
uint64_t u64[2];
double d[2];
} limits;
} nvtxSemanticsCounter_t;
typedef struct nvtxCountersAttr_v1
{
size_t structSize;
/**
* A schema ID referring to the data layout of the counter group or a
* predefined NVTX payloads number type.
*/
uint64_t schemaId;
/** Name of the counter group. */
const char* name;
/** Identifier of the scope of the counters. */
uint64_t scopeId;
/**
* (Optional) Specify additional semantics for a counter (group). The
* semantics provided are applied to the all counters in a group. If the
* semantics should only refer to a single counter in a group, the semantics
* field of the payload entry has to be used. Accepted semantics are
* `nvtxSemanticsCounter_t` and `nvtxSemanticsTime_t`.
*/
const nvtxSemanticsHeader_t* semantics;
} nvtxCountersAttr_t;
/* Forward declaration of opaque counter group registration structure */
struct nvtxCountersRegistration_st;
typedef struct nvtxCountersRegistration_st nvtxCountersRegistration;
/* \brief Counters Handle Structure.
* \anchor COUNTERS_HANDLE_STRUCTURE
*
* This structure is opaque to the user and is used as a handle to reference a counter group.
* This type is returned from tools when using the NVTX API to create a counters group.
*/
typedef nvtxCountersRegistration* nvtxCountersHandle_t;
typedef struct nvtxCountersBatch_v1
{
/** Handle to attributes (data layout, scope, etc.) of a counter (group). */
nvtxCountersHandle_t hCounter;
/** Array of counter samples. */
const void* counters;
/** Size of the `counters` array (in bytes). */
size_t cntArrSize;
/** Array of timestamps or reference-time plus delta pair. `NULL` is used, if
timestamps are part of the counter (group) layout.) */
const void* timestamps;
/** Size of the `timestamps` array or definition (in bytes). */
size_t tsSize;
} nvtxCountersBatch_t;
/**
* \brief Register a counter group.
*
* @param hDomain NVTX domain handle.
* @param attr Pointer to the attributes of the counter (group).
*
* @return Counter handle identifying a counter or counter (group).
* The counter handle is unique within the NVTX domain.
*/
NVTX_DECLSPEC nvtxCountersHandle_t NVTX_API nvtxCountersRegister(
nvtxDomainHandle_t hDomain,
const nvtxCountersAttr_t* attr);
/**
* \brief Sample one integer counter by value immediately (the NVTX tool determines the timestamp).
*
* @param hDomain handle of the NVTX domain.
* @param hCounter handle of the NVTX counter (group).
* @param value 64-bit integer counter value.
*/
NVTX_DECLSPEC void NVTX_API nvtxCountersSampleInt64(
nvtxDomainHandle_t hDomain,
nvtxCountersHandle_t hCounter,
int64_t value);
/**
* \brief Sample one floating point counter by value immediately (the NVTX tool determines the timestamp).
*
* @param hDomain handle of the NVTX domain.
* @param hCounter handle of the NVTX counter (group).
* @param value 64-bit floating-point counter value.
*/
NVTX_DECLSPEC void NVTX_API nvtxCountersSampleFloat64(
nvtxDomainHandle_t hDomain,
nvtxCountersHandle_t hCounter,
double value);
/**
* \brief Sample a counter group by reference immediately (the NVTX tool determines the timestamp).
*
* @param hDomain handle of the NVTX domain.
* @param hCounter handle of the NVTX counter (group).
* @param counters pointer to one or more counter values.
* @param size size of the counter value(s) in bytes.
*/
NVTX_DECLSPEC void NVTX_API nvtxCountersSample(
nvtxDomainHandle_t hDomain,
nvtxCountersHandle_t hCounter,
void* values,
size_t size);
/**
* \brief Sample without value.
*
* @param hDomain handle of the NVTX domain.
* @param hCounter handle of the NVTX counter (group).
* @param reason reason for the missing sample value.
*/
NVTX_DECLSPEC void NVTX_API nvtxCountersSampleNoValue(
nvtxDomainHandle_t hDomain,
nvtxCountersHandle_t hCounter,
uint8_t reason);
/**
* \brief Submit a batch of counters in the given domain.
* Timestamps are part of the counter sample data.
*
* The size of a data sampling point is defined by the `staticSize` field of the
* payload schema. An NVTX tool can assume that the counter samples are stored
* as an array with each entry being `staticSize` bytes.
*
* @param hDomain handle of the NVTX domain
* @param hCounter handle of the counter group (includes counter data decoding schema)
* @param counters blob containing counter data and timestamps
* @param size size of the counter data blob in bytes
*/
NVTX_DECLSPEC void NVTX_API nvtxCountersSubmitBatch(
nvtxDomainHandle_t hDomain,
nvtxCountersHandle_t hCounter,
const void* counters,
size_t size);
/**
* \brief Submit a batch of counters in the given domain.
* Timestamps are separated from the counter data.
*
* @param hDomain handle of the NVTX domain
* @param counterBatch Pointer to the counter data to be submitted.
*/
NVTX_DECLSPEC void NVTX_API nvtxCountersSubmitBatchEx(
nvtxDomainHandle_t hDomain,
const nvtxCountersBatch_t* counterBatch);
#define NVTX3EXT_CBID_nvtxCountersRegister 0
#define NVTX3EXT_CBID_nvtxCountersSampleInt64 1
#define NVTX3EXT_CBID_nvtxCountersSampleFloat64 2
#define NVTX3EXT_CBID_nvtxCountersSample 3
#define NVTX3EXT_CBID_nvtxCountersSampleNoValue 4
#define NVTX3EXT_CBID_nvtxCountersSubmitBatch 5
#define NVTX3EXT_CBID_nvtxCountersSubmitBatchEx 6
#ifdef __GNUC__
#pragma GCC visibility push(internal)
#endif
#define NVTX_EXT_TYPES_GUARD /* Ensure other headers cannot be included directly */
#include "nvtxDetail/nvtxExtTypes.h"
#undef NVTX_EXT_TYPES_GUARD
#ifndef NVTX_NO_IMPL
#define NVTX_EXT_IMPL_COUNTERS_GUARD /* Ensure other headers cannot be included directly */
#include "nvtxDetail/nvtxExtImplCounters_v1.h"
#undef NVTX_EXT_IMPL_COUNTERS_GUARD
#endif /*NVTX_NO_IMPL*/
#ifdef __GNUC__
#pragma GCC visibility pop
#endif
#ifdef __cplusplus
}
#endif /* __cplusplus */
#endif /* NVTOOLSEXT_COUNTERS_H */
src/include/nvtx3/nvToolsExtCuda.h
+2 -2
Féach ar an gComhad
@@ -30,7 +30,7 @@ extern "C" {
*/
/* ------------------------------------------------------------------------- */
/* \cond SHOW_HIDDEN
/* \cond SHOW_HIDDEN
* \brief Used to build a non-colliding value for resource types separated class
* \version \NVTX_VERSION_2
*/
@@ -133,7 +133,7 @@ NVTX_DECLSPEC void NVTX_API nvtxNameCuEventW(CUevent event, const wchar_t* name)
#endif /* __cplusplus */
#ifndef NVTX_NO_IMPL
#define NVTX_IMPL_GUARD_CUDA /* Ensure other headers cannot included directly */
#define NVTX_IMPL_GUARD_CUDA /* Ensure other headers cannot be included directly */
#include "nvtxDetail/nvtxImplCuda_v3.h"
#undef NVTX_IMPL_GUARD_CUDA
#endif /*NVTX_NO_IMPL*/
src/include/nvtx3/nvToolsExtCudaRt.h
+2 -2
Féach ar an gComhad
@@ -31,7 +31,7 @@ extern "C" {
*/
/* ------------------------------------------------------------------------- */
/* \cond SHOW_HIDDEN
/* \cond SHOW_HIDDEN
* \brief Used to build a non-colliding value for resource types separated class
* \version \NVTX_VERSION_2
*/
@@ -109,7 +109,7 @@ NVTX_DECLSPEC void NVTX_API nvtxNameCudaEventW(cudaEvent_t event, const wchar_t*
#endif /* __cplusplus */
#ifndef NVTX_NO_IMPL
#define NVTX_IMPL_GUARD_CUDART /* Ensure other headers cannot included directly */
#define NVTX_IMPL_GUARD_CUDART /* Ensure other headers cannot be included directly */
#include "nvtxDetail/nvtxImplCudaRt_v3.h"
#undef NVTX_IMPL_GUARD_CUDART
#endif /*NVTX_NO_IMPL*/
src/include/nvtx3/nvToolsExtMem.h
+694
Féach ar an gComhad
@@ -0,0 +1,694 @@
/*
* Copyright 2009-2020 NVIDIA Corporation. All rights reserved.
*
* Licensed under the Apache License v2.0 with LLVM Exceptions.
* See https://llvm.org/LICENSE.txt for license information.
* SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
*/
#include "nvToolsExt.h"
#ifndef NVTOOLSEXTV3_MEM_V1
#define NVTOOLSEXTV3_MEM_V1
#ifdef __cplusplus
extern "C" {
#endif /* __cplusplus */
#define NVTX_EXT_MODULEID_MEM 1
/* \cond SHOW_HIDDEN
* \brief A compatibility ID value used in structures and initialization to
* identify version differences.
*/
#define NVTX_EXT_COMPATID_MEM 0x0102
/* \cond SHOW_HIDDEN
* \brief This value is returned by functions that return `nvtxMemHeapHandle_t`,
* if a tool is not attached.
*/
#define NVTX_MEM_HEAP_HANDLE_NO_TOOL ((nvtxMemHeapHandle_t)(intptr_t)-1)
/* \cond SHOW_HIDDEN
* \brief This value is returned by functions that return `nvtxMemRegionHandle_t`
* if a tool is not attached.
*/
#define NVTX_MEM_REGION_HANDLE_NO_TOOL ((nvtxMemRegionHandle_t)(intptr_t)-1)
/* \cond SHOW_HIDDEN
* \brief This value is returned by functions that return `nvtxMemPermissionsHandle_t`
* if a tool is not attached.
*/
#define NVTX_MEM_PERMISSIONS_HANDLE_NO_TOOL ((nvtxMemPermissionsHandle_t)-1)
/* \cond SHOW_HIDDEN
* \brief This should not be used and is considered an error but defined to
* detect an accidental use of zero or NULL.
*/
#define NVTX_MEM_HEAP_USAGE_UNKNOWN 0x0
/* \cond SHOW_HIDDEN
* \brief This should not be used and is considered an error but defined to
* detect an accidental use of zero or NULL.
*/
#define NVTX_MEM_TYPE_UNKNOWN 0x0
/* ------------------------------------------------------------------------- */
/** \defgroup MEMORY Memory
* See page \ref PAGE_MEMORY.
* @{
*/
/**
* \brief To indicate the full process virtual address space as a heap for
* functions where a nvtxMemHeapHandle_t is accepted.
*
* The heap by default is always read-write-execute permissions without creating regions.
* Regions created in this heap have read-write access by default but not execute.
*/
#define NVTX_MEM_HEAP_HANDLE_PROCESS_WIDE ((nvtxMemHeapHandle_t)0)
/** \brief This heap is a sub-allocator.
*
* Heap created with this usage should not be accessed by the user until regions are registered.
* Regions from a heap with this usage have read-write access by default but not execute.
*/
#define NVTX_MEM_HEAP_USAGE_TYPE_SUB_ALLOCATOR 0x1
/**
* \brief This is a heap of memory that has an explicit layout.
*
* The layout could be static or dynamic (calculated). This often represents an algorithm's
* structures that are packed together. By default this heap is assumed to be accessible for
* scopes where the memory is naturally accessible by hardware. Regions may be use to further
* annotate or restrict access. A tool may have an option to be more strict, but special
* consideration must be made for `NVTX_MEM_HEAP_HANDLE_PROCESS_WIDE`.
*
* The behavior of this usage is similar to NVTX_MEM_HEAP_HANDLE_PROCESS_WIDE but
* a tool can use it to track special behaviors and reservation.
*
* Memory in a heap with this usage has read-write permissions by default but not execute without
* creating regions. Regions created in this heap have the same default permission access.
*/
#define NVTX_MEM_HEAP_USAGE_TYPE_LAYOUT 0x2
/**
* \brief Standard process userspace virtual addresses for linear allocations.
*
* APIs that map into this space, such as CUDA UVA should use this type.
*
* Relevant functions: cudaMalloc, cudaMallocManaged, cudaHostAlloc, cudaMallocHost
* NVTX_MEM_HEAP_HANDLE_PROCESS_WIDE is supported
*
* nvtxMemHeapRegister receives a heapDesc of type nvtxMemVirtualRangeDesc_t
*/
#define NVTX_MEM_TYPE_VIRTUAL_ADDRESS 0x1
/**
* \brief To indicate you are modifying permissions to the process-wide
* full virtual address space.
*
* This is a companion object to `NVTX_MEM_HEAP_HANDLE_PROCESS_WIDE`.
*/
#define NVTX_MEM_PERMISSIONS_HANDLE_PROCESS_WIDE ((nvtxMemPermissionsHandle_t)0)
#define NVTX_MEM_PERMISSIONS_CREATE_FLAGS_NONE 0x0
#define NVTX_MEM_PERMISSIONS_CREATE_FLAGS_EXCLUDE_GLOBAL_READ 0x1
#define NVTX_MEM_PERMISSIONS_CREATE_FLAGS_EXCLUDE_GLOBAL_WRITE 0x2
#define NVTX_MEM_PERMISSIONS_CREATE_FLAGS_EXCLUDE_GLOBAL_ATOMIC 0x4
/* \cond SHOW_HIDDEN
* \brief Forward declaration of opaque memory heap structure.
*/
struct nvtxMemHeap_v1;
typedef struct nvtxMemHeap_v1 nvtxMemHeap_t;
/** \brief A handle returned by a tool to represent a memory heap. */
typedef nvtxMemHeap_t* nvtxMemHeapHandle_t;
/* \cond SHOW_HIDDEN
* \brief Forward declaration of opaque memory heap structure.
*/
struct nvtxMemRegion_v1;
typedef struct nvtxMemRegion_v1 nvtxMemRegion_t;
/** \brief A handle returned by a tool to represent a memory region. */
typedef nvtxMemRegion_t* nvtxMemRegionHandle_t;
/** \brief A reference to a memory region (by pointer or handle).
* Which member of the union will be determined by a type or flag field outside.
*/
typedef union nvtxMemRegionRef_t
{
void const* pointer;
nvtxMemRegionHandle_t handle;
} nvtxMemRegionRef_t;
/* \cond SHOW_HIDDEN
* \brief Forward declaration of opaque memory permissions structure
*/
struct nvtxMemPermissions_v1;
typedef struct nvtxMemPermissions_v1 nvtxMemPermissions_t;
/** \brief A handle returned by a tool to represent a memory permissions mask. */
typedef nvtxMemPermissions_t* nvtxMemPermissionsHandle_t;
typedef struct nvtxMemVirtualRangeDesc_v1
{
size_t size;
void const* ptr;
} nvtxMemVirtualRangeDesc_v1 ;
typedef nvtxMemVirtualRangeDesc_v1 nvtxMemVirtualRangeDesc_t;
/** \brief structure to describe a heap in process virtual memory. */
typedef struct nvtxMemHeapDesc_v1
{
uint16_t extCompatID; /* Set to NVTX_EXT_COMPATID_MEM */
uint16_t structSize; /* Size of the structure. */
uint32_t reserved0;
/** \brief Usage characteristics of the heap
*
* Usage characteristics help tools like memcheckers, santiizer,
* as well as other debugging and profiling tools to determine some
* special behaviors they should apply to the heap and it's regions.
* The value follows the convention NVTX_MEM_HEAP_USAGE_*
*
* Default Value is 0, which is invalid.
*/
uint32_t usage;
/** \brief Memory type characteristics of the heap
*
* The 'type' indicates how to interpret the ptr field of the heapDesc.
* This is intended to support many additional types of memory, beyond
* standard process virtual memory, such as API specific memory only
* addressed by handles or multi-dimensional memory requiring more complex
* descriptions to handle features like strides, tiling, or interlace.
*
* The values conforms to NVTX_MEM_TYPE_*
*
* The value in the field 'type' identifies the descriptor type that will
* be in the field 'typeSpecificDesc'. 'typeSpecificDesc' is void* because
* it is extensible. Example usage is if type is NVTX_MEM_TYPE_VIRTUAL_ADDRESS,
* then typeSpecificDesc points to a nvtxMemVirtualRangeDesc_t.
*
* Default Value is 0, which is invalid.
*/
uint32_t type;
/** \brief size of the heap memory descriptor pointed to by typeSpecificDesc
*
* Default Value is 0 which is invalid.
*/
size_t typeSpecificDescSize;
/** \brief Pointer to the heap memory descriptor
*
* The value in the field 'type' identifies the descriptor type that will
* be in the field 'typeSpecificDesc'. 'typeSpecificDesc' is void* because
* it is extensible. Example usage is if type is NVTX_MEM_TYPE_VIRTUAL_ADDRESS,
* then typeSpecificDesc points to a nvtxMemVirtualRangeDesc_t.
*
* Default Value is 0, which is invalid.
*/
void const* typeSpecificDesc;
/** \brief ID of the category the event is assigned to.
*
* A category is a user-controlled ID that can be used to group
* events. The tool may use category IDs to improve filtering or
* enable grouping of events in the same category. The functions
* \ref ::nvtxNameCategoryA or \ref ::nvtxNameCategoryW can be used
* to name a category.
*
* Default Value is 0.
*/
uint32_t category;
/** \brief Message type specified in this attribute structure.
*
* Defines the message format of the attribute structure's \ref MESSAGE_FIELD
* "message" field.
*
* Default Value is `NVTX_MESSAGE_UNKNOWN`.
*/
uint32_t messageType; /* nvtxMessageType_t */
/** \brief Message assigned to this attribute structure. \anchor MESSAGE_FIELD
*
* The text message that is attached to an event.
*/
nvtxMessageValue_t message;
} nvtxMemHeapDesc_v1 ;
typedef nvtxMemHeapDesc_v1 nvtxMemHeapDesc_t;
/**
* \brief Create a memory heap to represent a object or range of memory that will be further
* sub-divided into regions.
*
* The handle used to addrss the heap will depend on the heap's type. Where the heap is virtual
* memory accessible, the addrss of the heap's memory itself is it's handle. This will likewise
* be returned from the function.
*
* For more advanced types, where the heap is not virtual memory accessible the tools may be
* responsible for returning a void const * that that uniquely identifies the object. Please see
* the description of each heap type for more details on whether this is expected to be a uniquely
* generated by the tool or otherwise.
*/
NVTX_DECLSPEC nvtxMemHeapHandle_t NVTX_API nvtxMemHeapRegister(
nvtxDomainHandle_t domain,
nvtxMemHeapDesc_t const* desc);
/** \brief Destroy a memory heap. */
NVTX_DECLSPEC void NVTX_API nvtxMemHeapUnregister(
nvtxDomainHandle_t domain,
nvtxMemHeapHandle_t heap);/* NVTX_MEM_HEAP_HANDLE_PROCESS_WIDE is not supported */
/**
* \brief Reset the memory heap wipes out any changes, as if it were a fresh heap.
*
* This includes invalidating all regions and their handles.
*/
NVTX_DECLSPEC void NVTX_API nvtxMemHeapReset(
nvtxDomainHandle_t domain,
nvtxMemHeapHandle_t heap); /* NVTX_MEM_HEAP_HANDLE_PROCESS_WIDE is supported */
/**
* \brief Register a region of memory inside of a heap.
*
* The heap refers the the heap within which the region resides. This can be from
* `nvtxMemHeapRegister`, `NVTX_MEM_HEAP_HANDLE_PROCESS_WIDE`, or one provided
* from other extension API.
*
* The regionType arg will define which type is used in regionDescArray.
* The most commonly used type is `NVTX_MEM_TYPE_VIRTUAL_ADDRESS`.
* In this case regionDescElements is an array of `nvtxMemVirtualRangeDesc_t`.
*
* The regionCount arg is how many element are in regionDescArray and regionHandleArrayOut.
*
* The regionHandleArrayOut arg points to an array where the tool will provide region handles. If
* a pointer is provided, it is expected to have regionCount elements. This pointer can be NULL if
* regionType is NVTX_MEM_TYPE_VIRTUAL_ADDRESS. In this case, the user can use the pointer to the
* virtual memory to reference the region in other related functions which accept nvtMemRegionRef_t.
*/
typedef struct nvtxMemRegionsRegisterBatch_v1
{
uint16_t extCompatID; /* Set to NVTX_EXT_COMPATID_MEM */
uint16_t structSize; /* Size of the structure. */
uint32_t regionType; /* NVTX_MEM_TYPE_* */
nvtxMemHeapHandle_t heap;
size_t regionCount;
size_t regionDescElementSize;
void const* regionDescElements; /* This will also become the handle for this region. */
nvtxMemRegionHandle_t* regionHandleElementsOut; /* This will also become the handle for this region. */
} nvtxMemRegionsRegisterBatch_v1;
typedef nvtxMemRegionsRegisterBatch_v1 nvtxMemRegionsRegisterBatch_t;
/** \brief Register a region of memory inside of a heap of linear process virtual memory
*/
NVTX_DECLSPEC void NVTX_API nvtxMemRegionsRegister(
nvtxDomainHandle_t domain,
nvtxMemRegionsRegisterBatch_t const* desc);
/**
* \brief Register a region of memory inside of a heap.
*
* The heap refers the the heap within which the region resides.
* This can be from nvtxMemHeapRegister, NVTX_MEM_HEAP_HANDLE_PROCESS_WIDE, or
* one provided from other extension API.
*
* The regionType arg will define which type is used in regionDescArray.
* The most commonly used type is NVTX_MEM_TYPE_VIRTUAL_ADDRESS.
*
* The regionCount arg is how many element are in regionDescArray and regionHandleArrayOut.
*
* The regionHandleArrayOut arg points to an array where the tool will provide region handles. If
* a pointer if provided, it is expected to have regionCount elements. This pointer can be NULL if
* regionType is NVTX_MEM_TYPE_VIRTUAL_ADDRESS. In this case, the user can use the pointer to the
* virtual memory to reference the region in other related functions which accept nvtMemRegionRef_t.
*/
typedef struct nvtxMemRegionsResizeBatch_v1
{
uint16_t extCompatID; /* Set to NVTX_EXT_COMPATID_MEM */
uint16_t structSize; /* Size of the structure. */
uint32_t regionType; /* NVTX_MEM_TYPE_* */
size_t regionDescCount;
size_t regionDescElementSize;
void const* regionDescElements; /* This will also become the handle for this region. */
} nvtxMemRegionsResizeBatch_v1;
typedef nvtxMemRegionsResizeBatch_v1 nvtxMemRegionsResizeBatch_t;
/** \brief Register a region of memory inside of a heap of linear process virtual memory
*/
NVTX_DECLSPEC void NVTX_API nvtxMemRegionsResize(
nvtxDomainHandle_t domain,
nvtxMemRegionsResizeBatch_t const* desc);
#define NVTX_MEM_REGION_REF_TYPE_UNKNOWN 0x0
#define NVTX_MEM_REGION_REF_TYPE_POINTER 0x1
#define NVTX_MEM_REGION_REF_TYPE_HANDLE 0x2
/**
* \brief Register a region of memory inside of a heap.
*
* The heap refers the the heap within which the region resides.
* This can be from nvtxMemHeapRegister, `NVTX_MEM_HEAP_HANDLE_PROCESS_WIDE`, or
* one provided from other extension API.
*
* The regionType arg will define which type is used in `regionDescArray`.
* The most commonly used type is NVTX_MEM_TYPE_VIRTUAL_ADDRESS.
*
* The regionCount arg is how many element are in regionDescArray and regionHandleArrayOut.
*
* The regionHandleArrayOut arg points to an array where the tool will provide region handles.
* If a pointer if provided, it is expected to have regionCount elements.
* This pointer can be NULL if regionType is NVTX_MEM_TYPE_VIRTUAL_ADDRESS. In this case,
* the user can use the pointer to the virtual memory to reference the region in other
* related functions which accept a nvtMemRegionRef_t.
*/
typedef struct nvtxMemRegionsUnregisterBatch_v1
{
uint16_t extCompatID; /* Set to NVTX_EXT_COMPATID_MEM */
uint16_t structSize; /* Size of the structure. */
uint32_t refType; /* NVTX_MEM_REGION_REF_TYPE_* */
size_t refCount; /* count of elements in refArray */
size_t refElementSize;
nvtxMemRegionRef_t const* refElements; /* This will also become the handle for this region. */
} nvtxMemRegionsUnregisterBatch_v1;
typedef nvtxMemRegionsUnregisterBatch_v1 nvtxMemRegionsUnregisterBatch_t;
/**
* \brief Unregistration for regions of process virtual memory
*
* This is not necessary if the nvtx heap destroy function has been called that
* contains this object.
*/
NVTX_DECLSPEC void NVTX_API nvtxMemRegionsUnregister(
nvtxDomainHandle_t domain,
nvtxMemRegionsUnregisterBatch_t const* desc);
typedef struct nvtxMemRegionNameDesc_v1
{
uint32_t regionRefType; /* NVTX_MEM_REGION_REF_TYPE_* */
uint32_t nameType; /* nvtxMessageType_t */
nvtxMemRegionRef_t region;
nvtxMessageValue_t name;
uint32_t category;
uint32_t reserved0;
} nvtxMemRegionNameDesc_v1;
typedef nvtxMemRegionNameDesc_v1 nvtxMemRegionNameDesc_t;
typedef struct nvtxMemRegionsNameBatch_v1
{
uint16_t extCompatID; /* Set to NVTX_EXT_COMPATID_MEM */
uint16_t structSize; /* Size of the structure. */
uint32_t reserved0;
size_t regionCount;
size_t regionElementSize;
nvtxMemRegionNameDesc_t const* regionElements;
size_t reserved1;
} nvtxMemRegionsNameBatch_v1 ;
typedef nvtxMemRegionsNameBatch_v1 nvtxMemRegionsNameBatch_t;
/** \brief Name or rename a region. */
NVTX_DECLSPEC void NVTX_API nvtxMemRegionsName(
nvtxDomainHandle_t domain,
nvtxMemRegionsNameBatch_t const* desc);
/** \brief There are no permissions for this memory. */
#define NVTX_MEM_PERMISSIONS_REGION_FLAGS_NONE 0x0
/** \brief The memory is readable. */
#define NVTX_MEM_PERMISSIONS_REGION_FLAGS_READ 0x1
/** \brief The memory is writable. */
#define NVTX_MEM_PERMISSIONS_REGION_FLAGS_WRITE 0x2
/** \brief The memory is for atomic RW. */
#define NVTX_MEM_PERMISSIONS_REGION_FLAGS_ATOMIC 0x4
/**
* \brief The memory access permissions are reset for a region.
*
* This is as if never set, rather than documented defaults. As as result any flags
* indicating how unspecified regions are handle will affect this area.
*
* This should not be used with READ, WRITE, nor ATOMIC, as those flags would have no effect.
*/
#define NVTX_MEM_PERMISSIONS_REGION_FLAGS_RESET 0x8
typedef struct nvtxMemPermissionsAssignRegionDesc_v1
{
uint32_t flags; /* NVTX_MEM_PERMISSIONS_REGION_FLAGS_* */
uint32_t regionRefType; /* NVTX_MEM_REGION_REF_TYPE_* */
nvtxMemRegionRef_t region;
} nvtxMemPermissionsAssignRegionDesc_v1 ;
typedef nvtxMemPermissionsAssignRegionDesc_v1 nvtxMemPermissionsAssignRegionDesc_t;
typedef struct nvtxMemPermissionsAssignBatch_v1
{
uint16_t extCompatID; /* Set to NVTX_EXT_COMPATID_MEM */
uint16_t structSize; /* Size of the structure. */
uint32_t reserved0;
nvtxMemPermissionsHandle_t permissions;
size_t regionCount;
size_t regionElementSize;
nvtxMemPermissionsAssignRegionDesc_t const* regionElements;
size_t reserved1;
} nvtxMemPermissionsAssignBatch_v1 ;
typedef nvtxMemPermissionsAssignBatch_v1 nvtxMemPermissionsAssignBatch_t;
/** \brief Change the permissions of a region of process virtual memory. */
NVTX_DECLSPEC void NVTX_API nvtxMemPermissionsAssign(
nvtxDomainHandle_t domain,
nvtxMemPermissionsAssignBatch_t const* desc);
/**
* \brief Create a permissions object for fine grain thread-local control in
* multi-threading scenarios
*
* Unlike the global permissions object (NVTX_MEM_PERMISSIONS_HANDLE_PROCESS_WIDE), a new
* permissions object is empty. There are no regions registered to it, so more memory is accessible
* if bound(bind) without calls to nvtxMemPermissionsSetAccess* first. The permissions are not
* active until nvtxMemPermissionsBind. See `nvtxMemPermissionsBind` for more details.
*
* Use the flags NVTX_MEM_PERMISSIONS_CREATE_FLAGS_EXCLUDE_GLOBAL_* to control how the regions in
* this permission object will interact with global permissions when bound. You may choose to
* either replace global memory regions setting or overlay on top of them. The most common uses are
* as follows:
* * To limit tools to validate writing exclusively specified in this object but inherit all
* global read access regions use `NVTX_MEM_PERMISSIONS_CREATE_FLAGS_EXCLUDE_GLOBAL_WRITE`
* * To limit tools to validate both read & write permissions exclusively specified in this
* object use NVTX_MEM_PERMISSIONS_CREATE_FLAGS_EXCLUDE_GLOBAL_READ
* & NVTX_MEM_PERMISSIONS_CREATE_FLAGS_EXCLUDE_GLOBAL_WRITE
*
* Also see `nvtxMemPermissionsBind` & `nvtxMemPermissionsSetAccess*`.
*/
NVTX_DECLSPEC nvtxMemPermissionsHandle_t NVTX_API nvtxMemPermissionsCreate(
nvtxDomainHandle_t domain,
int32_t creationflags); /* NVTX_MEM_PERMISSIONS_CREATE_FLAGS_* */
/**
* \brief Destroy the permissions object.
*
* If bound(bind), destroy will also unbind it.
*/
NVTX_DECLSPEC void NVTX_API nvtxMemPermissionsDestroy(
nvtxDomainHandle_t domain,
nvtxMemPermissionsHandle_t permissionsHandle); /* only supported on objects from nvtxMemPermissionsCreate */
/** \brief Reset the permissions object back to its created state. */
NVTX_DECLSPEC void NVTX_API nvtxMemPermissionsReset(
nvtxDomainHandle_t domain,
nvtxMemPermissionsHandle_t permissionsHandle);
/* NVTX_MEM_PERMISSIONS_HANDLE_PROCESS_WIDE and other special handles are supported */
#define NVTX_MEM_PERMISSIONS_BIND_FLAGS_NONE 0x0
/** \brief Upon binding, with the thread, exclude parent scope write regions instead of overlaying on top of them.
*
* EX A developer may chose to first prevent all writes except the ones specified to avoid
* OOB writes, since there are typically less regions written to than read from.
**/
#define NVTX_MEM_PERMISSIONS_BIND_FLAGS_STRICT_WRITE 0x2
/** \brief Upon binding, with the thread, exclude parent scope read regions instead of overlaying on top of them.
*
* EX After eliminating any errors when applying strict writes, a developer may then choose to
* annotate and enforce strict reads behaviors in segments of code.
**/
#define NVTX_MEM_PERMISSIONS_BIND_FLAGS_STRICT_READ 0x1
/** \brief Upon binding, with the thread, exclude parent scope atomic RW regions instead of overlaying on top of them.
*
* EX After eliminating any errors from read and write, a developer may chose to ensure
* that atomics are in their own region, removing standard read/write, and replacing with
* this strict atomic only access. This way they know that conventional reads or writes
* will not cause unepected issues.
**/
#define NVTX_MEM_PERMISSIONS_BIND_FLAGS_STRICT_ATOMIC 0x4
#define NVTX_MEM_PERMISSIONS_BIND_SCOPE_UNKNOWN 0x0
/** \brief Bind to thread scope. In this case, tools should validate that local thread's
* execution is honoring the permissions as well as the state of NVTX_MEM_PERMISSIONS_HANDLE_PROCESS_WIDE
* at the time of binding. If this is not bound then NVTX_MEM_PERMISSIONS_HANDLE_PROCESS_WIDE should be
* used to validate the memory.
*
* Not all tools will support every scope, such a GPU sanitizer.
**/
#define NVTX_MEM_PERMISSIONS_BIND_SCOPE_CPU_THREAD 0x1
/**
* \brief Bind to CUDA stream scope.
*
* In this case, work enqueued to a CUDA stream should be validated by the tool,
* when it executes, that it respect the permission of the permission at the point
* of binding, as well as the appropriate nvtxMemCudaGetDevicePermissions at the
* time of binding. If this is not bound then nvtxMemCudaGetDevicePermissions at
* the time of stream enqueue should be used to validate the memory.
*
* This could apply to work done either on the GPU like a kernel launch or to
* CPU based callbacks like cudaStreamAddCallback if the tools supports it.
*
* Binding is applies locally to a CPU thread so that if N CPU threads are enqueing
* work to the same stream (like the default stream) that there cannot be a race
* condition between thread binding vs launching their work. IE users should
* expect the permissions bound in the thread to be honored by the proceeding
* work (launches, copies, etc) invoked from in the CPU thread until unbound.
*/
#define NVTX_MEM_PERMISSIONS_BIND_SCOPE_CUDA_STREAM 0x2
/**
* \brief Bind the permissions object into a particular scope on the caller thread
*
* Permissions do not take affect until binding. Binding permissions is a thread local
* activity that overrides global behaviors. This is to avoid multi-threaded race conditions,
*
* The scope dictates what type of processing it applies to, and when in some cases.
* EX1: NVTX_MEM_PERMISSIONS_BIND_SCOPE_CPU_THREAD applies to CPU code accessing memory while bound.
* EX2: NVTX_MEM_PERMISSIONS_BIND_SCOPE_CUDA_STREAM applies to CUDA streams, and the permissions
* must be recorded and applied when the work in the stream dequeues to executes. In this case
* it could be GPU or CPU, if the tool support both.
*
* Bind can be called again on the same object and thread to take any updates to the
* specified permission object or the inherited properties.
*
* Bind flags support changing how the binding process inherits region access control.
* In the case of thread scope this is NVTX_MEM_PERMISSIONS_HANDLE_PROCESS_WIDE and from CUDA_STREAM
* this is nvtxMemCudaGetDevicePermissions. Choosing stricter modes allows the user to
* further reduce the access with less work, since memory by default, behaves as natural
* until the NVTX annotations instructs a tool to treat it anther way. See strict flags
* for more details.
*
* Also see nvtxMemPermissionsUnbind
*/
NVTX_DECLSPEC void NVTX_API nvtxMemPermissionsBind(
nvtxDomainHandle_t domain,
nvtxMemPermissionsHandle_t permissions, /* special object like NVTX_MEM_PERMISSIONS_HANDLE_PROCESS_WIDE are not supported */
uint32_t bindScope, /* NVTX_MEM_PERMISSIONS_BIND_SCOPE_* */
uint32_t bindFlags); /* NVTX_MEM_PERMISSIONS_BIND_FLAGS_* */
/**
* \brief Unbind the permissions object bound to the caller thread.
*
* Upon unbind, the thread local permissions for a scope are restored to the default
* behavior defined by the scope.
*/
NVTX_DECLSPEC void NVTX_API nvtxMemPermissionsUnbind(
nvtxDomainHandle_t domain,
uint32_t bindScope);
/** @} */ /*END defgroup*/
typedef enum NvtxExtMemCallbackId
{
/* CBID 0 is invalid */
NVTX3EXT_CBID_nvtxMemHeapRegister = 1,
NVTX3EXT_CBID_nvtxMemHeapUnregister = 2,
NVTX3EXT_CBID_nvtxMemHeapReset = 3,
NVTX3EXT_CBID_nvtxMemRegionsRegister = 4,
NVTX3EXT_CBID_nvtxMemRegionsResize = 5,
NVTX3EXT_CBID_nvtxMemRegionsUnregister = 6,
NVTX3EXT_CBID_nvtxMemRegionsName = 7,
NVTX3EXT_CBID_nvtxMemPermissionsAssign = 8,
NVTX3EXT_CBID_nvtxMemPermissionsCreate = 9,
NVTX3EXT_CBID_nvtxMemPermissionsDestroy = 10,
NVTX3EXT_CBID_nvtxMemPermissionsReset = 11,
NVTX3EXT_CBID_nvtxMemPermissionsBind = 12,
NVTX3EXT_CBID_nvtxMemPermissionsUnbind = 13,
/* 14-16 in nvtExtImplMemCudaRt1.h */
NVTX3EXT_CBID_nvtxMemCudaGetProcessWidePermissions = 14,
NVTX3EXT_CBID_nvtxMemCudaGetDeviceWidePermissions = 15,
NVTX3EXT_CBID_nvtxMemCudaSetPeerAccess = 16,
NVTX3EXT_CBID_MEM_FN_NUM = 17
} NvtxExtMemCallbackId;
#ifdef __GNUC__
#pragma GCC visibility push(internal)
#endif
/* Extension types are required for the implementation and the NVTX handler. */
#define NVTX_EXT_TYPES_GUARD /* Ensure other headers cannot be included directly */
#include "nvtxDetail/nvtxExtTypes.h"
#undef NVTX_EXT_TYPES_GUARD
#ifndef NVTX_NO_IMPL
/* Ensure other headers cannot be included directly */
#define NVTX_EXT_IMPL_MEM_GUARD
#include "nvtxDetail/nvtxExtImplMem_v1.h"
#undef NVTX_EXT_IMPL_MEM_GUARD
#endif /*NVTX_NO_IMPL*/
#ifdef __GNUC__
#pragma GCC visibility pop
#endif
#ifdef __cplusplus
}
#endif /* __cplusplus */
#endif /* NVTOOLSEXTV3_MEM_V1 */
src/include/nvtx3/nvToolsExtMemCudaRt.h
+150
Féach ar an gComhad
@@ -0,0 +1,150 @@
/*
* Copyright 2009-2020 NVIDIA Corporation. All rights reserved.
*
* Licensed under the Apache License v2.0 with LLVM Exceptions.
* See https://llvm.org/LICENSE.txt for license information.
* SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
*/
#ifndef NVTOOLSEXTV3_MEM_CUDART_V1
#define NVTOOLSEXTV3_MEM_CUDART_V1
#include "nvToolsExtMem.h"
#include "cuda.h"
#include "cuda_runtime.h"
#ifdef __cplusplus
extern "C" {
#endif /* __cplusplus */
/** \brief The memory is from a CUDA runtime array.
*
* Relevant functions: cudaMallocArray, cudaMalloc3DArray
* Also cudaArray_t from other types such as cudaMipmappedArray_t
*
* NVTX_MEM_HEAP_HANDLE_PROCESS_WIDE is not supported
*
* nvtxMemHeapRegister receives a heapDesc of type cudaArray_t because the description can be retrieved by tools through cudaArrayGetInfo()
* nvtxMemRegionRegisterEx receives a regionDesc of type nvtxMemCudaArrayRangeDesc_t
*/
#define NVTX_MEM_TYPE_CUDA_ARRAY 0x11
/** \brief structure to describe memory in a CUDA array object
*/
typedef struct nvtxMemCudaArrayRangeDesc_v1
{
uint16_t extCompatID; /* Set to NVTX_EXT_COMPATID_MEM */
uint16_t structSize; /* Size of the structure. */
uint32_t reserved0;
cudaArray_t src;
size_t offset[3];
size_t extent[3];
} nvtxMemCudaArrayRangeDesc_v1;
typedef nvtxMemCudaArrayRangeDesc_v1 nvtxMemCudaArrayRangeDesc_t;
/** \brief The memory is from a CUDA device array.
*
* Relevant functions: cuArrayCreate, cuArray3DCreate
* Also CUarray from other types such as CUmipmappedArray
*
* NVTX_MEM_HEAP_HANDLE_PROCESS_WIDE is not supported
*
* nvtxMemHeapRegister receives a heapDesc of type cudaArray_t because the description can be retrieved by tools through cudaArrayGetInfo()
* nvtxMemRegionRegisterEx receives a regionDesc of type nvtxMemCuArrayRangeDesc_t
*/
#define NVTX_MEM_TYPE_CU_ARRAY 0x12
/** \brief structure to describe memory in a CUDA array object
*/
typedef struct nvtxMemCuArrayRangeDesc_v1
{
uint16_t extCompatID; /* Set to NVTX_EXT_COMPATID_MEM */
uint16_t structSize; /* Size of the structure. */
uint32_t reserved0;
CUarray src;
size_t offset[3];
size_t extent[3];
} nvtxMemCuArrayRangeDesc_v1;
typedef nvtxMemCuArrayRangeDesc_v1 nvtxMemCuArrayRangeDesc_t;
/* Reserving 0x2-0xF for more common types */
#define NVTX_MEM_CUDA_PEER_ALL_DEVICES -1
/** \brief Get the permission object that represent the CUDA runtime device
* or cuda driver context
*
* This object will allow developers to adjust permissions applied to work executed
* on the GPU. It may be inherited or overridden by permissions object bound
* with NVTX_MEM_PERMISSIONS_BIND_SCOPE_CUDA_STREAM, depending on the binding flags.
*
* Ex. change the peer to peer access permissions between devices in entirety
* or punch through special holes
*
* By default, all memory is accessible that naturally would be to a CUDA kernel until
* modified otherwise by nvtxMemCudaSetPeerAccess or changing regions.
*
* This object should also represent the CUDA driver API level context.
*/
NVTX_DECLSPEC nvtxMemPermissionsHandle_t NVTX_API nvtxMemCudaGetProcessWidePermissions(
nvtxDomainHandle_t domain);
/** \brief Get the permission object that represent the CUDA runtime device
* or cuda driver context
*
* This object will allow developers to adjust permissions applied to work executed
* on the GPU. It may be inherited or overridden by permissions object bound
* with NVTX_MEM_PERMISSIONS_BIND_SCOPE_CUDA_STREAM, depending on the binding flags.
*
* Ex. change the peer to peer access permissions between devices in entirety
* or punch through special holes
*
* By default, all memory is accessible that naturally would be to a CUDA kernel until
* modified otherwise by nvtxMemCudaSetPeerAccess or changing regions.
*
* This object should also represent the CUDA driver API level context.
*/
NVTX_DECLSPEC nvtxMemPermissionsHandle_t NVTX_API nvtxMemCudaGetDeviceWidePermissions(
nvtxDomainHandle_t domain,
int device);
/** \brief Change the default behavior for all memory mapped in from a particular device.
*
* While typically all memory defaults to readable and writable, users may desire to limit
* access to reduced default permissions such as read-only and a per-device basis.
*
* Regions can used to further override smaller windows of memory.
*
* devicePeer can be NVTX_MEM_CUDA_PEER_ALL_DEVICES
*
*/
NVTX_DECLSPEC void NVTX_API nvtxMemCudaSetPeerAccess(
nvtxDomainHandle_t domain,
nvtxMemPermissionsHandle_t permissions,
int devicePeer, /* device number such as from cudaGetDevice() or NVTX_MEM_CUDA_PEER_ALL_DEVICES */
uint32_t flags); /* NVTX_MEM_PERMISSIONS_REGION_FLAGS_* */
/** @} */ /*END defgroup*/
#ifdef __GNUC__
#pragma GCC visibility push(internal)
#endif
#ifndef NVTX_NO_IMPL
#define NVTX_EXT_IMPL_MEM_CUDART_GUARD /* Ensure other headers cannot be included directly */
#include "nvtxDetail/nvtxExtImplMemCudaRt_v1.h"
#undef NVTX_EXT_IMPL_MEM_CUDART_GUARD
#endif /*NVTX_NO_IMPL*/
#ifdef __GNUC__
#pragma GCC visibility pop
#endif
#ifdef __cplusplus
}
#endif /* __cplusplus */
#endif /* NVTOOLSEXTV3_MEM_CUDART_V1 */
src/include/nvtx3/nvToolsExtOpenCL.h
+3 -3
Féach ar an gComhad
@@ -30,11 +30,11 @@ extern "C" {
*/
/* ------------------------------------------------------------------------- */
/* \cond SHOW_HIDDEN
/* \cond SHOW_HIDDEN
* \brief Used to build a non-colliding value for resource types separated class
* \version \NVTX_VERSION_2
*/
#define NVTX_RESOURCE_CLASS_OPENCL 6
#define NVTX_RESOURCE_CLASS_OPENCL 6
/** \endcond */
/* ------------------------------------------------------------------------- */
@@ -183,7 +183,7 @@ NVTX_DECLSPEC void NVTX_API nvtxNameClEventW(cl_event evnt, const wchar_t* name)
#endif /* __cplusplus */
#ifndef NVTX_NO_IMPL
#define NVTX_IMPL_GUARD_OPENCL /* Ensure other headers cannot included directly */
#define NVTX_IMPL_GUARD_OPENCL /* Ensure other headers cannot be included directly */
#include "nvtxDetail/nvtxImplOpenCL_v3.h"
#undef NVTX_IMPL_GUARD_OPENCL
#endif /*NVTX_NO_IMPL*/
src/include/nvtx3/nvToolsExtPayload.h
+713 -388
Féach ar an gComhad
Tá difríocht comhad cosc orthu toisc go bhfuil sé ró-mhór Difríocht Luchtaigh
src/include/nvtx3/nvToolsExtPayloadHelper.h
+170
Féach ar an gComhad
@@ -0,0 +1,170 @@
/*
* Copyright 2023 NVIDIA Corporation. All rights reserved.
*
* Licensed under the Apache License v2.0 with LLVM Exceptions.
* See https://llvm.org/LICENSE.txt for license information.
* SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
*/
#include "nvtxDetail/nvtxExtPayloadHelperInternal.h"
/* This is just an empty marker (for readability), which can be omitted. */
/* TODO: Fix issue with trailing comma at end of entry list. */
#define NVTX_PAYLOAD_ENTRIES
/**
* Use this macro for payload entries that are defined by a schema (nested
* payload schema).
*/
#define NVTX_PAYLOAD_NESTED(schemaId) _NVTX_PAYLOAD_NESTED(schemaId)
/**
* \brief Define a payload schema for an existing C `struct` definition.
*
* This macro does
* 1) create schema description (array of schema entries).
* 2) set the schema attributes for a static data layout.
*
* It can be used in static code or within a function context.
*
* Example:
* NVTX_DEFINE_SCHEMA_FOR_STRUCT(your_struct, "SchemaName",
* NVTX_PAYLOAD_ENTRIES(
* (index, TYPE_INT, "integer value"),
* (dpfloat, TYPE_DOUBLE, "fp64 value"),
* (text, TYPE_CSTRING, "text", NULL, 24)
* )
* )
*
* It is required to at least provide the struct name and the payload entries.
* The first two fields (member name and NVTX entry type) of each payload entry
* are required.
*
* The optional parameters are only allowed to be passed in the predefined order.
* Hence, `payload_flags` requires `payload_schema` to be given and
* `prefix` requires `payload_flags` and `payload_schema` to be given.
* The payload entries are always the last parameter. A maximum of 16 schema
* entries is supported.
*
* It is recommended to use `NVTX_PAYLOAD_SCHEMA_REGISTER` to register the schema.
*
* @param struct_id The name of the struct.
* @param schema_name (Optional 1) name of the payload schema. Default is `NULL`.
* @param prefix (Optional 2) prefix before the schema and attributes variables,
* e.g. `static const`. Leave this empty, if no prefix is desired.
* @param schema_flags (Optional 2) flags to augment the payload schema.
* Default is `NVTX_PAYLOAD_SCHEMA_FLAG_NONE`.
* @param schema_id (Optional 4) User-defined payload schema ID.
* @param entries (Mandatory) Payload schema entries. This is always the last
* parameter to the macro.
*/
#define NVTX_DEFINE_SCHEMA_FOR_STRUCT(struct_id, ...) \
_NVTX_DEFINE_SCHEMA_FOR_STRUCT(struct_id, __VA_ARGS__)
/**
* \brief Define a C struct together with a matching schema.
*
* This macro does
* 1) define the payload type (typedef struct).
* 2) create schema description (array of schema entries).
* 3) set the schema attributes for a static data layout.
*
* The macro can be used in static code or within a function context.
*
* It defines the schema attributes in `struct_id##Attr`. Thus, it is recommended
* to use `NVTX_PAYLOAD_SCHEMA_REGISTER(domain, struct_id)` to register the schema.
*
* Example:
* NVTX_DEFINE_STRUCT_WITH_SCHEMA(your_struct_name, "Your schema name",
* NVTX_PAYLOAD_ENTRIES(
* (int, index, TYPE_INT, "integer value"),
* (double, dpfloat, TYPE_DOUBLE, "fp64 value"),
* (const char, (text, 24), TYPE_CSTRING, "text", NULL, 24)
* )
* )
*
* The first three fields (C type, member, entry type) of each entry are required.
* A fixed-size array or string requires a special notation with the member
* name and the size separated by comma and put into brackets (see last entry
* in the example).
*
* The optional parameters are positional (only allowed to be passed in the
* predefined order). A maximum of 16 schema entries is supported.
*
* @param struct_id The name of the struct.
* @param schema_name (Optional 1) name of the payload schema. Default is `NULL`.
* @param prefix (Optional 2) prefix before the schema and attributes variables,
* e.g. `static const`. Leave this empty, if no prefix is desired.
* @param schema_flags (Optional 3) flags to augment the payload schema.
* Default is `NVTX_PAYLOAD_SCHEMA_FLAG_NONE`.
* @param schema_id (Optional 4) User-defined payload schema ID.
* @param entries (Mandatory) The schema entries. This is always the last
* parameter to the macro.
*/
#define NVTX_DEFINE_STRUCT_WITH_SCHEMA(struct_id, ...) \
_NVTX_DEFINE_STRUCT_WITH_SCHEMA(struct_id, __VA_ARGS__)
/**
* \brief Initialize and register the NVTX binary payload schema.
*
* This does essentially the same as `NVTX_DEFINE_STRUCT_WITH_SCHEMA`, but in
* addition the schema is registered. The schema ID will be defined as follows:
* `const uint64_t struct_id##_schemaId`.
*
* @param domain The NVTX domain handle (0 for default domain).
* All other parameters are similar to `NVTX_DEFINE_STRUCT_WITH_SCHEMA`.
*/
#define NVTX_DEFINE_STRUCT_WITH_SCHEMA_AND_REGISTER(domain, struct_id, ...) \
_NVTX_DEFINE_STRUCT_WITH_SCHEMA(struct_id, __VA_ARGS__) \
const uint64_t struct_id##_schemaId = nvtxPayloadSchemaRegister(domain, &struct_id##Attr);
/**
* \brief Define payload schema for an existing `struct` and register the schema.
*
* This does essentially the same as `NVTX_PAYLOAD_STATIC_SCHEMA_DEFINE`, but in
* addition, the schema is registered and `uint64_t struct_id##_schemaId` set.
*
* @param domain The NVTX domain handle (0 for default domain).
* All other parameters are similar to `NVTX_PAYLOAD_STATIC_SCHEMA_DEFINE`.
*/
#define NVTX_DEFINE_SCHEMA_FOR_STRUCT_AND_REGISTER(domain, struct_id, ...) \
_NVTX_DEFINE_SCHEMA_FOR_STRUCT(struct_id, __VA_ARGS__) \
const uint64_t struct_id##_schemaId = nvtxPayloadSchemaRegister(domain, &struct_id##Attr);
/**
* \brief Create a type definition for the given struct ID and members.
*
* This is a convenience macro. A normal `typedef` can be used instead.
*
* Example usage:
* NVTX_DEFINE_STRUCT(your_struct,
* (double, fp64),
* (uint8_t, u8),
* (float, fp32[3])
* )
*
* @param struct_id The name of the struct.
* @param members The members of the struct.
*/
#define NVTX_DEFINE_STRUCT(struct_id, ...) \
_NVTX_PAYLOAD_TYPEDEF_STRUCT(struct_id, __VA_ARGS__)
/**
* \brief Register an NVTX binary payload schema.
*
* This is a convenience macro, which takes the same `struct_id` that has been
* used in other helper macros. Instead, `nvtxPayloadSchemaRegister` can also be
* used, but `&struct_id##Attr` has to be passed.
*
* @param domain The NVTX domain handle (0 for default domain).
* @param struct_id The name of the struct.
*
* @return NVTX schema ID
*/
#define NVTX_PAYLOAD_SCHEMA_REGISTER(domain, struct_id) \
nvtxPayloadSchemaRegister(domain, &struct_id##Attr);
src/include/nvtx3/nvToolsExtSemanticsCounters.h
+88
Féach ar an gComhad
@@ -0,0 +1,88 @@
/*
* Copyright 2024 NVIDIA Corporation. All rights reserved.
*
* Licensed under the Apache License v2.0 with LLVM Exceptions.
* See https://llvm.org/LICENSE.txt for license information.
* SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
*/
/**
* NVTX semantic headers require nvToolsExtPayload.h to be included beforehand.
*/
#ifndef NVTX_SEMANTIC_ID_COUNTERS_V1
#define NVTX_SEMANTIC_ID_COUNTERS_V1 2
/**
* Flags to extend the semantics of counters.
*/
#define NVTX_COUNTERS_FLAGS_NONE 0
/**
* Convert the fixed point value to a normalized floating point value.
* Unsigned [0f : 1f] or signed [-1f : 1f] is determined by the underlying type
* this flag is applied to.
*/
#define NVTX_COUNTERS_FLAG_NORMALIZE (1 << 1)
/**
* Visual tools should apply scale and limits when graphing.
*/
#define NVTX_COUNTERS_FLAG_LIMIT_MIN (1 << 2)
#define NVTX_COUNTERS_FLAG_LIMIT_MAX (1 << 3)
#define NVTX_COUNTERS_FLAG_LIMITS \
(NVTX_COUNTERS_FLAG_LIMIT_MIN | NVTX_COUNTERS_FLAG_LIMIT_MAX)
/**
* Counter time scopes.
*/
#define NVTX_COUNTERS_FLAG_TIMESCOPE_POINT (1 << 5)
#define NVTX_COUNTERS_FLAG_TIMESCOPE_SINCE_LAST (2 << 5)
#define NVTX_COUNTERS_FLAG_TIMESCOPE_UNTIL_NEXT (3 << 5)
#define NVTX_COUNTERS_FLAG_TIMESCOPE_SINCE_START (4 << 5)
/**
* Counter value types.
*/
#define NVTX_COUNTERS_FLAG_VALUETYPE_ABSOLUTE (1 << 10)
/** Delta to previous value of same counter type. */
#define NVTX_COUNTERS_FLAG_VALUETYPE_DELTA (2 << 10)
/**
* Datatypes for the `limits` union.
*/
#define NVTX_COUNTERS_LIMIT_I64 0
#define NVTX_COUNTERS_LIMIT_U64 1
#define NVTX_COUNTERS_LIMIT_F64 2
/**
*\brief Specify counter semantics.
*/
typedef struct nvtxSemanticsCounter_v1 {
/** Header of the semantic extensions (with identifier, version, etc.). */
struct nvtxSemanticsHeader_v1 header;
/** Flags to provide more context about the counter value. */
uint64_t flags;
/** Unit of the counter value (case-insensitive). */
const char* unit;
/** Should be 1 if not used. */
uint64_t unitScaleNumerator;
/** Should be 1 if not used. */
uint64_t unitScaleDenominator;
/** Determines the used union member. Use defines `NVTX_COUNTER_LIMIT_*`. */
int64_t limitType;
/** Graph limits {minimum, maximum}. */
union limits_t {
int64_t i64[2];
uint64_t u64[2];
double d[2];
} limits;
} nvtxSemanticsCounter_t;
#endif /* NVTX_SEMANTIC_ID_COUNTERS_V1 */
src/include/nvtx3/nvToolsExtSemanticsScope.h
+30
Féach ar an gComhad
@@ -0,0 +1,30 @@
/*
* Copyright 2024 NVIDIA Corporation. All rights reserved.
*
* Licensed under the Apache License v2.0 with LLVM Exceptions.
* See https://llvm.org/LICENSE.txt for license information.
* SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
*/
/**
* NVTX semantic headers require nvToolsExtPayload.h to be included beforehand.
*/
#ifndef NVTX_SEMANTIC_ID_SCOPE_V1
#define NVTX_SEMANTIC_ID_SCOPE_V1 1
/**
* \brief Specify the NVTX scope for a payload entry.
*
* This allows the scope to be set for a specific value or counter in a payload.
* The scope must be known at schema registration time.
*/
typedef struct nvtxSemanticsScope_v1
{
struct nvtxSemanticsHeader_v1 header;
/** Specifies the scope of a payload entry, e.g. a counter or timestamp. */
uint64_t scopeId;
} nvtxSemanticsScope_t;
#endif /* NVTX_SEMANTIC_ID_SCOPE_V1 */
src/include/nvtx3/nvToolsExtSync.h
+13 -13
Féach ar an gComhad
@@ -15,23 +15,23 @@
extern "C" {
#endif /* __cplusplus */
/* \cond SHOW_HIDDEN
/* \cond SHOW_HIDDEN
* \version \NVTX_VERSION_2
*/
#define NVTX_SYNCUSER_ATTRIB_STRUCT_SIZE ( (uint16_t)( sizeof(nvtxSyncUserAttributes_v0) ) )
/** \endcond */
/**
/**
* \page PAGE_SYNCHRONIZATION Synchronization
*
* This section covers a subset of the API that allow users to track additional
* synchronization details of their application. Naming OS synchronization primitives
* may allow users to better understand the data collected by traced synchronization
* synchronization details of their application. Naming OS synchronization primitives
* may allow users to better understand the data collected by traced synchronization
* APIs. Additionally, a user defined synchronization object can allow the users to
* to tell the tools when the user is building their own synchronization system
* that do not rely on the OS to provide behaviors and instead use techniques like
* atomic operations and spinlocks.
* atomic operations and spinlocks.
*
* See module \ref SYNCHRONIZATION for details.
*
@@ -59,7 +59,7 @@ extern "C" {
*
* bool Lock() {
* nvtxDomainSyncUserAcquireStart(hSync);
* bool acquired = __sync_bool_compare_and_swap(&bLocked, 0, 1);//atomic compiler intrinsic
* bool acquired = __sync_bool_compare_and_swap(&bLocked, 0, 1);//atomic compiler intrinsic
* if (acquired) {
* nvtxDomainSyncUserAcquireSuccess(hSync);
@@ -76,12 +76,12 @@ extern "C" {
* }
* };
* \endcode
*
*
* \version \NVTX_VERSION_2
*/
/* ------------------------------------------------------------------------- */
/* \cond SHOW_HIDDEN
/* \cond SHOW_HIDDEN
* \brief Used to build a non-colliding value for resource types separated class
* \version \NVTX_VERSION_2
*/
@@ -154,8 +154,8 @@ typedef struct nvtxSyncUser* nvtxSyncUser_t;
/** \brief User Defined Synchronization Object Attributes Structure.
* \anchor USERDEF_SYNC_ATTRIBUTES_STRUCTURE
*
* This structure is used to describe the attributes of a user defined synchronization
* object. The layout of the structure is defined by a specific version of the tools
* This structure is used to describe the attributes of a user defined synchronization
* object. The layout of the structure is defined by a specific version of the tools
* extension library and can change between different versions of the Tools Extension
* library.
*
@@ -259,7 +259,7 @@ typedef struct nvtxSyncUserAttributes_v0
typedef struct nvtxSyncUserAttributes_v0 nvtxSyncUserAttributes_t;
/* ------------------------------------------------------------------------- */
/** \brief Create a user defined synchronization object
/** \brief Create a user defined synchronization object
* This is used to track non-OS synchronization working with spinlocks and atomics
*
* \param domain - Domain to own the resource
@@ -317,7 +317,7 @@ NVTX_DECLSPEC void NVTX_API nvtxDomainSyncUserAcquireStart(nvtxSyncUser_t handle
/* ------------------------------------------------------------------------- */
/** \brief Signal to tools of failure in acquiring a user defined synchronization object
* This should be called after \ref nvtxDomainSyncUserAcquireStart
*
*
* \param handle - A handle to the object to operate on.
*
* \sa
@@ -374,7 +374,7 @@ NVTX_DECLSPEC void NVTX_API nvtxDomainSyncUserReleasing(nvtxSyncUser_t handle);
#endif /* __cplusplus */
#ifndef NVTX_NO_IMPL
#define NVTX_IMPL_GUARD_SYNC /* Ensure other headers cannot included directly */
#define NVTX_IMPL_GUARD_SYNC /* Ensure other headers cannot be included directly */
#include "nvtxDetail/nvtxImplSync_v3.h"
#undef NVTX_IMPL_GUARD_SYNC
#endif /*NVTX_NO_IMPL*/
src/include/nvtx3/nvtx3.hpp
+13 -8
Féach ar an gComhad
@@ -12,6 +12,11 @@
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*
* Licensed under the Apache License v2.0 with LLVM Exceptions.
* See https://llvm.org/LICENSE.txt for license information.
*
* SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
*/
/* Temporary helper #defines, #undef'ed at end of header */
@@ -1937,9 +1942,9 @@ class event_attributes {
0, // color value
NVTX_PAYLOAD_UNKNOWN, // payload type
0, // reserved 4B
0, // payload value (union)
{0}, // payload value (union)
NVTX_MESSAGE_UNKNOWN, // message type
0 // message value (union)
{0} // message value (union)
}
{
}
@@ -2003,20 +2008,20 @@ class event_attributes {
attributes_.messageType = m.get_type();
}
/**
* @brief Variadic constructor where the first argument is a binary payload.
/**
* @brief Variadic constructor where the first argument is an extended payload.
*
* Sets the value of the `EventAttribute`s message based on `m` and forwards
* Sets the `ullValue` of the `EventAttribute`s payload and forwards
* the remaining variadic parameter pack to the next constructor.
*
*/
template <typename... Args>
NVTX3_CONSTEXPR_IF_CPP14 explicit event_attributes(nvtxPayloadData_t const* bpl, Args const&... args) noexcept
NVTX3_CONSTEXPR_IF_CPP14 explicit event_attributes(nvtxPayloadData_t const* p, Args const&... args) noexcept
: event_attributes(args...)
{
attributes_.payloadType = NVTX_PAYLOAD_TYPE_BINARY;
attributes_.payloadType = NVTX_PAYLOAD_TYPE_EXT;
attributes_.reserved0 = 1; // NCCL uses only a single binary payload per event.
attributes_.payload.ullValue = NVTX_POINTER_AS_PAYLOAD_ULLVALUE(bpl);
attributes_.payload.ullValue = NVTX_POINTER_AS_PAYLOAD_ULLVALUE(p);
}
~event_attributes() = default;
src/include/nvtx3/nvtxDetail/nvtxExtHelperMacros.h
+31
Féach ar an gComhad
@@ -0,0 +1,31 @@
/*
* Copyright 2023 NVIDIA Corporation. All rights reserved.
*
* Licensed under the Apache License v2.0 with LLVM Exceptions.
* See https://llvm.org/LICENSE.txt for license information.
* SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
*/
#ifndef NVTX_EXT_HELPER_MACROS_H
#define NVTX_EXT_HELPER_MACROS_H
/* Combine tokens */
#define _NVTX_EXT_CONCAT(a, b) a##b
#define NVTX_EXT_CONCAT(a, b) _NVTX_EXT_CONCAT(a, b)
/* Resolves to the number of arguments passed. */
#define NVTX_EXT_NUM_ARGS(...) \
NVTX_EXT_SELECTA16(__VA_ARGS__, 15, 14, 13, 12, 11, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1, throwaway)
#define NVTX_EXT_SELECTA16(a1, a2, a3, a4, a5, a6, a7, a8, a9, a10, a11, a12, a13, a14, a15, a16, ...) a16
/* Cast argument(s) to void to prevent unused variable warnings. */
#define _NVTX_EXT_VOIDIFY1(a1) (void)a1;
#define _NVTX_EXT_VOIDIFY2(a1, a2) (void)a1; (void)a2;
#define _NVTX_EXT_VOIDIFY3(a1, a2, a3) (void)a1; (void)a2; (void)a3;
#define _NVTX_EXT_VOIDIFY4(a1, a2, a3, a4) (void)a1; (void)a2; (void)a3; (void)a4;
/* Mark function arguments as unused. */
#define NVTX_EXT_HELPER_UNUSED_ARGS(...) \
NVTX_EXT_CONCAT(_NVTX_EXT_VOIDIFY, NVTX_EXT_NUM_ARGS(__VA_ARGS__))(__VA_ARGS__)
#endif /* NVTX_EXT_HELPER_MACROS_H */
src/include/nvtx3/nvtxExtDetail/nvtxExtImpl.h → src/include/nvtx3/nvtxDetail/nvtxExtImpl.h
+28 -22
Féach ar an gComhad
@@ -14,7 +14,12 @@
#define NVTX_EXT_IMPL_H
/* ---- Include required platform headers ---- */
#if defined(_WIN32)
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <wchar.h>
#if defined(_WIN32)
#include <Windows.h>
@@ -22,27 +27,19 @@
#include <unistd.h>
#if defined(__ANDROID__)
#include <android/api-level.h>
#include <android/api-level.h>
#endif
#if defined(__linux__) || defined(__CYGWIN__)
#include <sched.h>
#endif
#include <sys/types.h>
#include <limits.h>
#include <dlfcn.h>
#include <fcntl.h>
#include <stdlib.h>
#include <stdio.h>
#include <sys/types.h>
#include <unistd.h>
#include <errno.h>
#include <string.h>
#include <sys/types.h>
#include <pthread.h>
#include <stdlib.h>
#include <wchar.h>
#endif
@@ -66,26 +63,35 @@
#ifdef __cplusplus
extern "C" {
#endif /* __cplusplus */
// #ifdef __GNUC__
// #pragma GCC visibility push(hidden)
// #endif
/*
#ifdef __GNUC__
#pragma GCC visibility push(hidden)
#endif
*/
#define NVTX_EXTENSION_FRESH 0
#define NVTX_EXTENSION_DISABLED 1
#define NVTX_EXTENSION_STARTING 2
#define NVTX_EXTENSION_LOADED 3
NVTX_LINKONCE_DEFINE_GLOBAL NvtxExtInitializeInjectionFunc_t NVTX_VERSIONED_IDENTIFIER(injectionFnPtr) = (NvtxExtInitializeInjectionFunc_t)0;
/* Function slots are local to each extension */
typedef struct nvtxExtGlobals1_t
{
NvtxExtInitializeInjectionFunc_t injectionFnPtr;
} nvtxExtGlobals1_t;
NVTX_LINKONCE_DEFINE_GLOBAL nvtxExtGlobals1_t NVTX_VERSIONED_IDENTIFIER(nvtxExtGlobals1) =
{
(NvtxExtInitializeInjectionFunc_t)0
};
#define NVTX_EXT_INIT_GUARD
#include "nvtxExtInit.h"
#undef NVTX_EXT_INIT_GUARD
// #ifdef __GNUC__
// #pragma GCC visibility pop
// #endif
/*
#ifdef __GNUC__
#pragma GCC visibility pop
#endif
*/
#ifdef __cplusplus
} /* extern "C" */
#endif /* __cplusplus */
src/include/nvtx3/nvtxDetail/nvtxExtImplCounters_v1.h
+148
Féach ar an gComhad
@@ -0,0 +1,148 @@
/*
* Copyright 2023-2024 NVIDIA Corporation. All rights reserved.
*
* Licensed under the Apache License v2.0 with LLVM Exceptions.
* See https://llvm.org/LICENSE.txt for license information.
* SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
*/
#ifndef NVTX_EXT_IMPL_COUNTERS_GUARD
#error Never include this file directly -- it is automatically included by nvToolsExtCounters.h (except when NVTX_NO_IMPL is defined).
#endif
#define NVTX_EXT_IMPL_GUARD
#include "nvtxExtImpl.h"
#undef NVTX_EXT_IMPL_GUARD
#ifndef NVTX_EXT_IMPL_COUNTERS_V1
#define NVTX_EXT_IMPL_COUNTERS_V1
#ifdef __cplusplus
extern "C" {
#endif /* __cplusplus */
/* Macros to create versioned symbols. */
#define NVTX_EXT_COUNTERS_VERSIONED_IDENTIFIER_L3(NAME, VERSION, COMPATID) \
NAME##_v##VERSION##_bpl##COMPATID
#define NVTX_EXT_COUNTERS_VERSIONED_IDENTIFIER_L2(NAME, VERSION, COMPATID) \
NVTX_EXT_COUNTERS_VERSIONED_IDENTIFIER_L3(NAME, VERSION, COMPATID)
#define NVTX_EXT_COUNTERS_VERSIONED_ID(NAME) \
NVTX_EXT_COUNTERS_VERSIONED_IDENTIFIER_L2(NAME, NVTX_VERSION, NVTX_EXT_COUNTERS_COMPATID)
#ifdef NVTX_DISABLE
#include "nvtxExtHelperMacros.h"
#define NVTX_EXT_COUNTERS_IMPL_FN_V1(ret_val, fn_name, signature, arg_names) \
ret_val fn_name signature { \
NVTX_EXT_HELPER_UNUSED_ARGS arg_names \
return ((ret_val)(intptr_t)-1); \
}
#else /* NVTX_DISABLE */
/*
* Function slots for the counters extension. First entry is the module state,
* initialized to `0` (`NVTX_EXTENSION_FRESH`).
*/
#define NVTX_EXT_COUNTERS_SLOT_COUNT 63
NVTX_LINKONCE_DEFINE_GLOBAL intptr_t
NVTX_EXT_COUNTERS_VERSIONED_ID(nvtxExtCountersSlots)[NVTX_EXT_COUNTERS_SLOT_COUNT + 1]
= {0};
/* Avoid warnings about missing prototype. */
NVTX_LINKONCE_FWDDECL_FUNCTION void NVTX_EXT_COUNTERS_VERSIONED_ID(nvtxExtCountersInitOnce)(void);
NVTX_LINKONCE_DEFINE_FUNCTION void NVTX_EXT_COUNTERS_VERSIONED_ID(nvtxExtCountersInitOnce)()
{
intptr_t* fnSlots = NVTX_EXT_COUNTERS_VERSIONED_ID(nvtxExtCountersSlots) + 1;
nvtxExtModuleSegment_t segment = {
0, /* unused (only one segment) */
NVTX_EXT_COUNTERS_SLOT_COUNT,
fnSlots
};
nvtxExtModuleInfo_t module = {
NVTX_VERSION, sizeof(nvtxExtModuleInfo_t),
NVTX_EXT_COUNTERS_MODULEID, NVTX_EXT_COUNTERS_COMPATID,
1, &segment, /* number of segments, segments */
NULL, /* no export function needed */
/* bake type sizes and alignment information into program binary */
NULL
};
NVTX_INFO( "%s\n", __FUNCTION__ );
NVTX_VERSIONED_IDENTIFIER(nvtxExtInitOnce)(&module,
NVTX_EXT_COUNTERS_VERSIONED_ID(nvtxExtCountersSlots));
}
#define NVTX_EXT_COUNTERS_IMPL_FN_V1(ret_type, fn_name, signature, arg_names) \
typedef ret_type (*fn_name##_impl_fntype)signature; \
NVTX_DECLSPEC ret_type NVTX_API fn_name signature { \
intptr_t slot = NVTX_EXT_COUNTERS_VERSIONED_ID(nvtxExtCountersSlots)[NVTX3EXT_CBID_##fn_name + 1]; \
if (slot != NVTX_EXTENSION_DISABLED) { \
if (slot != NVTX_EXTENSION_FRESH) { \
return (*(fn_name##_impl_fntype)slot) arg_names; \
} else { \
NVTX_EXT_COUNTERS_VERSIONED_ID(nvtxExtCountersInitOnce)(); \
/* Re-read function slot after extension initialization. */ \
slot = NVTX_EXT_COUNTERS_VERSIONED_ID(nvtxExtCountersSlots)[NVTX3EXT_CBID_##fn_name + 1]; \
if (slot != NVTX_EXTENSION_DISABLED && slot != NVTX_EXTENSION_FRESH) { \
return (*(fn_name##_impl_fntype)slot) arg_names; \
} \
} \
} \
NVTX_EXT_FN_RETURN_INVALID(ret_type) \
}
#endif /*NVTX_DISABLE*/
/* Non-void functions. */
#define NVTX_EXT_FN_RETURN_INVALID(rtype) return ((rtype)(intptr_t)-1);
NVTX_EXT_COUNTERS_IMPL_FN_V1(nvtxCountersHandle_t, nvtxCountersRegister,
(nvtxDomainHandle_t domain, const nvtxCountersAttr_t* attr),
(domain, attr))
#undef NVTX_EXT_FN_RETURN_INVALID
/* END: Non-void functions. */
/* void functions. */
#define NVTX_EXT_FN_RETURN_INVALID(rtype)
#define return
NVTX_EXT_COUNTERS_IMPL_FN_V1(void, nvtxCountersSampleInt64,
(nvtxDomainHandle_t domain, nvtxCountersHandle_t hCounter, int64_t value),
(domain, hCounter, value))
NVTX_EXT_COUNTERS_IMPL_FN_V1(void, nvtxCountersSampleFloat64,
(nvtxDomainHandle_t domain, nvtxCountersHandle_t hCounter, double value),
(domain, hCounter, value))
NVTX_EXT_COUNTERS_IMPL_FN_V1(void, nvtxCountersSample,
(nvtxDomainHandle_t domain, nvtxCountersHandle_t hCounter, void* values, size_t size),
(domain, hCounter, values, size))
NVTX_EXT_COUNTERS_IMPL_FN_V1(void, nvtxCountersSampleNoValue,
(nvtxDomainHandle_t domain, nvtxCountersHandle_t hCounter, uint8_t reason),
(domain, hCounter, reason))
NVTX_EXT_COUNTERS_IMPL_FN_V1(void, nvtxCountersSubmitBatch,
(nvtxDomainHandle_t domain, nvtxCountersHandle_t hCounters,
const void* counters, size_t size), (domain, hCounters, counters, size))
NVTX_EXT_COUNTERS_IMPL_FN_V1(void, nvtxCountersSubmitBatchEx,
(nvtxDomainHandle_t domain, const nvtxCountersBatch_t* countersBatch),
(domain, countersBatch))
#undef return
#undef NVTX_EXT_FN_RETURN_INVALID
/* END: void functions. */
/* Keep NVTX_EXT_COUNTERS_IMPL_FN_V1 defined for a future version of this extension. */
#ifdef __cplusplus
} /* extern "C" */
#endif /* __cplusplus */
#endif /* NVTX_EXT_IMPL_COUNTERS_V1 */
src/include/nvtx3/nvtxDetail/nvtxExtImplMemCudaRt_v1.h
+74
Féach ar an gComhad
@@ -0,0 +1,74 @@
/*
* Copyright 2009-2020 NVIDIA Corporation. All rights reserved.
*
* Licensed under the Apache License v2.0 with LLVM Exceptions.
* See https://llvm.org/LICENSE.txt for license information.
* SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
*/
#ifndef NVTX_EXT_IMPL_MEM_CUDART_GUARD
#error Never include this file directly -- it is automatically included by nvToolsExtMemCudaRt.h (except when NVTX_NO_IMPL is defined).
#endif
#ifdef __cplusplus
extern "C" {
#endif /* __cplusplus */
#ifdef NVTX_DISABLE
#include "nvtxExtHelperMacros.h"
#define NVTX_EXT_FN_IMPL(ret_val, fn_name, signature, arg_names) \
ret_val fn_name signature { \
NVTX_EXT_HELPER_UNUSED_ARGS arg_names \
return ((ret_val)(intptr_t)-1); \
}
#else /* NVTX_DISABLE */
#define NVTX_EXT_FN_IMPL(ret_type, fn_name, signature, arg_names) \
typedef ret_type ( * fn_name##_impl_fntype )signature; \
NVTX_DECLSPEC ret_type NVTX_API fn_name signature { \
intptr_t slot = NVTX_EXT_MEM_VERSIONED_ID(nvtxExtMemSlots)[NVTX3EXT_CBID_##fn_name + 1]; \
if (slot != NVTX_EXTENSION_DISABLED) { \
if (slot != NVTX_EXTENSION_FRESH) { \
return (*(fn_name##_impl_fntype)slot) arg_names; \
} else { \
NVTX_EXT_MEM_VERSIONED_ID(nvtxExtMemInitOnce)(); \
/* Re-read function slot after extension initialization. */ \
slot = NVTX_EXT_MEM_VERSIONED_ID(nvtxExtMemSlots)[NVTX3EXT_CBID_##fn_name + 1]; \
if (slot != NVTX_EXTENSION_DISABLED && slot != NVTX_EXTENSION_FRESH) { \
return (*(fn_name##_impl_fntype)slot) arg_names; \
} \
} \
} \
NVTX_EXT_FN_RETURN_INVALID(ret_type) \
}
#endif /*NVTX_DISABLE*/
/* Non-void functions. */
#define NVTX_EXT_FN_RETURN_INVALID(rtype) return ((rtype)(intptr_t)-1);
NVTX_EXT_FN_IMPL(nvtxMemPermissionsHandle_t, nvtxMemCudaGetProcessWidePermissions, (nvtxDomainHandle_t domain), (domain))
NVTX_EXT_FN_IMPL(nvtxMemPermissionsHandle_t, nvtxMemCudaGetDeviceWidePermissions, (nvtxDomainHandle_t domain, int device), (domain, device))
#undef NVTX_EXT_FN_RETURN_INVALID
/* END: Non-void functions. */
/* void functions. */
#define NVTX_EXT_FN_RETURN_INVALID(rtype)
#define return
NVTX_EXT_FN_IMPL(void, nvtxMemCudaSetPeerAccess, (nvtxDomainHandle_t domain, nvtxMemPermissionsHandle_t permissions, int devicePeer, uint32_t flags), (domain, permissions, devicePeer, flags))
#undef return
#undef NVTX_EXT_FN_RETURN_INVALID
/* END: void functions. */
#undef NVTX_EXT_FN_IMPL
#ifdef __cplusplus
} /* extern "C" */
#endif /* __cplusplus */
src/include/nvtx3/nvtxDetail/nvtxExtImplMem_v1.h
+133
Féach ar an gComhad
@@ -0,0 +1,133 @@
/*
* Copyright 2009-2020,2023 NVIDIA Corporation. All rights reserved.
*
* Licensed under the Apache License v2.0 with LLVM Exceptions.
* See https://llvm.org/LICENSE.txt for license information.
* SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
*/
#ifndef NVTX_EXT_IMPL_MEM_GUARD
#error Never include this file directly -- it is automatically included by nvToolsExtMem.h (except when NVTX_NO_IMPL is defined).
#endif
#define NVTX_EXT_IMPL_GUARD
#include "nvtxExtImpl.h"
#undef NVTX_EXT_IMPL_GUARD
#ifdef __cplusplus
extern "C" {
#endif /* __cplusplus */
#define NVTXMEM_VERSIONED_IDENTIFIER_L3(NAME, VERSION, COMPATID) NAME##_v##VERSION##_mem##COMPATID
#define NVTXMEM_VERSIONED_IDENTIFIER_L2(NAME, VERSION, COMPATID) NVTXMEM_VERSIONED_IDENTIFIER_L3(NAME, VERSION, COMPATID)
#define NVTX_EXT_MEM_VERSIONED_ID(NAME) NVTXMEM_VERSIONED_IDENTIFIER_L2(NAME, NVTX_VERSION, NVTX_EXT_COMPATID_MEM)
#ifdef NVTX_DISABLE
#include "nvtxExtHelperMacros.h"
#define NVTX_EXT_FN_IMPL(ret_val, fn_name, signature, arg_names) \
ret_val fn_name signature { \
NVTX_EXT_HELPER_UNUSED_ARGS arg_names \
return ((ret_val)(intptr_t)-1); \
}
#else /* NVTX_DISABLE */
/*
* Function slots for the memory extension. First entry is the module
* state, initialized to `0` (`NVTX_EXTENSION_FRESH`).
*/
NVTX_LINKONCE_DEFINE_GLOBAL intptr_t
NVTX_EXT_MEM_VERSIONED_ID(nvtxExtMemSlots)[NVTX3EXT_CBID_MEM_FN_NUM + 2]
= {0};
NVTX_LINKONCE_DEFINE_FUNCTION void NVTX_EXT_MEM_VERSIONED_ID(nvtxExtMemInitOnce)()
{
intptr_t* fnSlots = NVTX_EXT_MEM_VERSIONED_ID(nvtxExtMemSlots) + 1;
nvtxExtModuleSegment_t segment = {
0, /* unused (only one segment) */
NVTX3EXT_CBID_MEM_FN_NUM,
fnSlots
};
nvtxExtModuleInfo_t module = {
NVTX_VERSION, sizeof(nvtxExtModuleInfo_t),
NVTX_EXT_MODULEID_MEM, NVTX_EXT_COMPATID_MEM,
1, &segment,
NULL, /* no export function needed */
NULL
};
NVTX_INFO( "%s\n", __FUNCTION__ );
NVTX_VERSIONED_IDENTIFIER(nvtxExtInitOnce)(&module,
NVTX_EXT_MEM_VERSIONED_ID(nvtxExtMemSlots));
}
#define NVTX_EXT_FN_IMPL(ret_type, fn_name, signature, arg_names) \
typedef ret_type ( * fn_name##_impl_fntype )signature; \
NVTX_DECLSPEC ret_type NVTX_API fn_name signature { \
intptr_t slot = NVTX_EXT_MEM_VERSIONED_ID(nvtxExtMemSlots)[NVTX3EXT_CBID_##fn_name + 1]; \
if (slot != NVTX_EXTENSION_DISABLED) { \
if (slot != NVTX_EXTENSION_FRESH) { \
return (*(fn_name##_impl_fntype)slot) arg_names; \
} else { \
NVTX_EXT_MEM_VERSIONED_ID(nvtxExtMemInitOnce)(); \
/* Re-read function slot after extension initialization. */ \
slot = NVTX_EXT_MEM_VERSIONED_ID(nvtxExtMemSlots)[NVTX3EXT_CBID_##fn_name + 1]; \
if (slot != NVTX_EXTENSION_DISABLED && slot != NVTX_EXTENSION_FRESH) { \
return (*(fn_name##_impl_fntype)slot) arg_names; \
} \
} \
} \
NVTX_EXT_FN_RETURN_INVALID(ret_type) \
}
#endif /*NVTX_DISABLE*/
/* Non-void functions. */
#define NVTX_EXT_FN_RETURN_INVALID(rtype) return ((rtype)(intptr_t)-1);
NVTX_EXT_FN_IMPL(nvtxMemHeapHandle_t, nvtxMemHeapRegister, (nvtxDomainHandle_t domain, nvtxMemHeapDesc_t const* desc), (domain, desc))
NVTX_EXT_FN_IMPL(nvtxMemPermissionsHandle_t, nvtxMemPermissionsCreate, (nvtxDomainHandle_t domain, int32_t creationflags), (domain, creationflags))
#undef NVTX_EXT_FN_RETURN_INVALID
/* END: Non-void functions. */
/* void functions. */
#define NVTX_EXT_FN_RETURN_INVALID(rtype)
#define return
NVTX_EXT_FN_IMPL(void, nvtxMemHeapUnregister, (nvtxDomainHandle_t domain, nvtxMemHeapHandle_t heap), (domain, heap))
NVTX_EXT_FN_IMPL(void, nvtxMemHeapReset, (nvtxDomainHandle_t domain, nvtxMemHeapHandle_t heap), (domain, heap))
NVTX_EXT_FN_IMPL(void, nvtxMemRegionsRegister, (nvtxDomainHandle_t domain, nvtxMemRegionsRegisterBatch_t const* desc), (domain, desc))
NVTX_EXT_FN_IMPL(void, nvtxMemRegionsResize, (nvtxDomainHandle_t domain,nvtxMemRegionsResizeBatch_t const* desc), (domain, desc))
NVTX_EXT_FN_IMPL(void, nvtxMemRegionsUnregister, (nvtxDomainHandle_t domain,nvtxMemRegionsUnregisterBatch_t const* desc), (domain, desc))
NVTX_EXT_FN_IMPL(void, nvtxMemRegionsName, (nvtxDomainHandle_t domain,nvtxMemRegionsNameBatch_t const* desc), (domain, desc))
NVTX_EXT_FN_IMPL(void, nvtxMemPermissionsAssign, (nvtxDomainHandle_t domain,nvtxMemPermissionsAssignBatch_t const* desc), (domain, desc))
NVTX_EXT_FN_IMPL(void, nvtxMemPermissionsDestroy, (nvtxDomainHandle_t domain, nvtxMemPermissionsHandle_t permissions), (domain, permissions))
NVTX_EXT_FN_IMPL(void, nvtxMemPermissionsReset, (nvtxDomainHandle_t domain, nvtxMemPermissionsHandle_t permissions), (domain, permissions))
NVTX_EXT_FN_IMPL(void, nvtxMemPermissionsBind, (nvtxDomainHandle_t domain, nvtxMemPermissionsHandle_t permissions, uint32_t bindScope, uint32_t bindFlags), (domain, permissions, bindScope, bindFlags))
NVTX_EXT_FN_IMPL(void, nvtxMemPermissionsUnbind, (nvtxDomainHandle_t domain, uint32_t bindScope), (domain, bindScope))
#undef return
#undef NVTX_EXT_FN_RETURN_INVALID
/* END: void functions. */
#undef NVTX_EXT_FN_IMPL
#ifdef __cplusplus
} /* extern "C" */
#endif /* __cplusplus */
src/include/nvtx3/nvtxDetail/nvtxExtImplPayload_v1.h
+155
Féach ar an gComhad
@@ -0,0 +1,155 @@
/*
* Copyright 2021-2023 NVIDIA Corporation. All rights reserved.
*
* Licensed under the Apache License v2.0 with LLVM Exceptions.
* See https://llvm.org/LICENSE.txt for license information.
* SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
*/
#ifndef NVTX_EXT_IMPL_PAYLOAD_GUARD
#error Never include this file directly -- it is automatically included by nvToolsExtPayload.h (except when NVTX_NO_IMPL is defined).
#endif
#define NVTX_EXT_IMPL_GUARD
#include "nvtxExtImpl.h"
#undef NVTX_EXT_IMPL_GUARD
#ifndef NVTX_EXT_IMPL_PAYLOAD_V1
#define NVTX_EXT_IMPL_PAYLOAD_V1
#ifdef __cplusplus
extern "C" {
#endif /* __cplusplus */
/* Macros to create versioned symbols. */
#define NVTX_EXT_PAYLOAD_VERSIONED_IDENTIFIER_L3(NAME, VERSION, COMPATID) \
NAME##_v##VERSION##_bpl##COMPATID
#define NVTX_EXT_PAYLOAD_VERSIONED_IDENTIFIER_L2(NAME, VERSION, COMPATID) \
NVTX_EXT_PAYLOAD_VERSIONED_IDENTIFIER_L3(NAME, VERSION, COMPATID)
#define NVTX_EXT_PAYLOAD_VERSIONED_ID(NAME) \
NVTX_EXT_PAYLOAD_VERSIONED_IDENTIFIER_L2(NAME, NVTX_VERSION, NVTX_EXT_PAYLOAD_COMPATID)
#ifdef NVTX_DISABLE
#include "nvtxExtHelperMacros.h"
#define NVTX_EXT_PAYLOAD_IMPL_FN_V1(ret_val, fn_name, signature, arg_names) \
ret_val fn_name signature { \
NVTX_EXT_HELPER_UNUSED_ARGS arg_names \
return ((ret_val)(intptr_t)-1); \
}
#else /* NVTX_DISABLE */
#include "nvtxExtPayloadTypeInfo.h"
/*
* Function slots for the payload extension. First entry is the module state,
* initialized to `0` (`NVTX_EXTENSION_FRESH`).
*/
#define NVTX_EXT_PAYLOAD_SLOT_COUNT 63
NVTX_LINKONCE_DEFINE_GLOBAL intptr_t
NVTX_EXT_PAYLOAD_VERSIONED_ID(nvtxExtPayloadSlots)[NVTX_EXT_PAYLOAD_SLOT_COUNT + 1]
= {0};
/* Avoid warnings about missing prototype. */
NVTX_LINKONCE_FWDDECL_FUNCTION void NVTX_EXT_PAYLOAD_VERSIONED_ID(nvtxExtPayloadInitOnce)(void);
NVTX_LINKONCE_DEFINE_FUNCTION void NVTX_EXT_PAYLOAD_VERSIONED_ID(nvtxExtPayloadInitOnce)()
{
intptr_t* fnSlots = NVTX_EXT_PAYLOAD_VERSIONED_ID(nvtxExtPayloadSlots) + 1;
nvtxExtModuleSegment_t segment = {
0, /* unused (only one segment) */
NVTX_EXT_PAYLOAD_SLOT_COUNT,
fnSlots
};
nvtxExtModuleInfo_t module = {
NVTX_VERSION, sizeof(nvtxExtModuleInfo_t),
NVTX_EXT_PAYLOAD_MODULEID, NVTX_EXT_PAYLOAD_COMPATID,
1, &segment, /* number of segments, segments */
NULL, /* no export function needed */
/* bake type sizes and alignment information into program binary */
&(NVTX_EXT_PAYLOAD_VERSIONED_ID(nvtxExtPayloadTypeInfo))
};
NVTX_INFO( "%s\n", __FUNCTION__ );
NVTX_VERSIONED_IDENTIFIER(nvtxExtInitOnce)(&module,
NVTX_EXT_PAYLOAD_VERSIONED_ID(nvtxExtPayloadSlots));
}
#define NVTX_EXT_PAYLOAD_IMPL_FN_V1(ret_type, fn_name, signature, arg_names) \
typedef ret_type (*fn_name##_impl_fntype)signature; \
NVTX_DECLSPEC ret_type NVTX_API fn_name signature { \
intptr_t slot = NVTX_EXT_PAYLOAD_VERSIONED_ID(nvtxExtPayloadSlots)[NVTX3EXT_CBID_##fn_name + 1]; \
if (slot != NVTX_EXTENSION_DISABLED) { \
if (slot != NVTX_EXTENSION_FRESH) { \
return (*(fn_name##_impl_fntype)slot) arg_names; \
} else { \
NVTX_EXT_PAYLOAD_VERSIONED_ID(nvtxExtPayloadInitOnce)(); \
/* Re-read function slot after extension initialization. */ \
slot = NVTX_EXT_PAYLOAD_VERSIONED_ID(nvtxExtPayloadSlots)[NVTX3EXT_CBID_##fn_name + 1]; \
if (slot != NVTX_EXTENSION_DISABLED && slot != NVTX_EXTENSION_FRESH) { \
return (*(fn_name##_impl_fntype)slot) arg_names; \
} \
} \
} \
NVTX_EXT_FN_RETURN_INVALID(ret_type) \
}
#endif /*NVTX_DISABLE*/
/* Non-void functions. */
#define NVTX_EXT_FN_RETURN_INVALID(rtype) return ((rtype)(intptr_t)-1);
NVTX_EXT_PAYLOAD_IMPL_FN_V1(uint64_t, nvtxPayloadSchemaRegister,
(nvtxDomainHandle_t domain, const nvtxPayloadSchemaAttr_t* attr),
(domain, attr))
NVTX_EXT_PAYLOAD_IMPL_FN_V1(uint64_t, nvtxPayloadEnumRegister,
(nvtxDomainHandle_t domain, const nvtxPayloadEnumAttr_t* attr),
(domain, attr))
NVTX_EXT_PAYLOAD_IMPL_FN_V1(int, nvtxRangePushPayload,
(nvtxDomainHandle_t domain, const nvtxPayloadData_t* payloadData, size_t count),
(domain, payloadData, count))
NVTX_EXT_PAYLOAD_IMPL_FN_V1(int, nvtxRangePopPayload,
(nvtxDomainHandle_t domain, const nvtxPayloadData_t* payloadData, size_t count),
(domain, payloadData, count))
NVTX_EXT_PAYLOAD_IMPL_FN_V1(nvtxRangeId_t, nvtxRangeStartPayload,
(nvtxDomainHandle_t domain, const nvtxPayloadData_t* payloadData, size_t count),
(domain, payloadData, count))
NVTX_EXT_PAYLOAD_IMPL_FN_V1(uint8_t, nvtxDomainIsEnabled, (nvtxDomainHandle_t domain), (domain))
NVTX_EXT_PAYLOAD_IMPL_FN_V1(uint64_t, nvtxScopeRegister, (nvtxDomainHandle_t domain,
const nvtxScopeAttr_t* attr), (domain, attr))
#undef NVTX_EXT_FN_RETURN_INVALID
/* END: Non-void functions. */
/* void functions. */
#define NVTX_EXT_FN_RETURN_INVALID(rtype)
#define return
NVTX_EXT_PAYLOAD_IMPL_FN_V1(void, nvtxMarkPayload, (nvtxDomainHandle_t domain,
const nvtxPayloadData_t* payloadData, size_t count), (domain, payloadData, count))
NVTX_EXT_PAYLOAD_IMPL_FN_V1(void, nvtxRangeEndPayload, (nvtxDomainHandle_t domain,
nvtxRangeId_t id, const nvtxPayloadData_t* payloadData, size_t count),
(domain, id, payloadData, count))
#undef return
#undef NVTX_EXT_FN_RETURN_INVALID
/* END: void functions. */
/* Keep NVTX_EXT_PAYLOAD_IMPL_FN_V1 defined for a future version of this extension. */
#ifdef __cplusplus
} /* extern "C" */
#endif /* __cplusplus */
#endif /* NVTX_EXT_IMPL_PAYLOAD_V1 */
src/include/nvtx3/nvtxExtDetail/nvtxExtInit.h → src/include/nvtx3/nvtxDetail/nvtxExtInit.h
+88 -73
Féach ar an gComhad
@@ -1,5 +1,5 @@
/*
* Copyright 2009-2020 NVIDIA Corporation. All rights reserved.
* Copyright 2009-2023 NVIDIA Corporation. All rights reserved.
*
* Licensed under the Apache License v2.0 with LLVM Exceptions.
* See https://llvm.org/LICENSE.txt for license information.
@@ -22,7 +22,7 @@ extern "C" {
#define NVTX_PATHCHAR wchar_t
#define NVTX_STR(x) L##x
#define NVTX_GETENV _wgetenv
#define NVTX_BUFSIZE MAX_PATH
#define NVTX_BUFSIZE 16384
#define NVTX_DLLHANDLE HMODULE
#define NVTX_DLLOPEN(x) LoadLibraryW(x)
#define NVTX_DLLFUNC GetProcAddress
@@ -39,14 +39,14 @@ extern "C" {
#define NVTX_PATHCHAR char
#define NVTX_STR(x) x
#define NVTX_GETENV getenv
#define NVTX_BUFSIZE PATH_MAX
#define NVTX_BUFSIZE 16384
#define NVTX_DLLHANDLE void*
#define NVTX_DLLOPEN(x) dlopen(x, RTLD_LAZY)
#define NVTX_DLLFUNC dlsym
#define NVTX_DLLCLOSE dlclose
#define NVTX_YIELD() sched_yield()
#define NVTX_MEMBAR() __sync_synchronize()
/* Ensure full memory barrier for atomics, to match Windows functions */
/* Ensure full memory barrier for atomics, to match Windows functions. */
#define NVTX_ATOMIC_WRITE_32(address, value) __sync_synchronize(); __sync_lock_test_and_set(address, value)
#define NVTX_ATOMIC_CAS_32(old, address, exchange, comparand) __sync_synchronize(); old = __sync_val_compare_and_swap(address, exchange, comparand)
#define NVTX_ATOMIC_WRITE_PTR(address, value) __sync_synchronize(); __sync_lock_test_and_set(address, value)
@@ -63,7 +63,7 @@ extern "C" {
#define NVTX_SUPPORT_ALREADY_INJECTED_LIBRARY 0
#endif
/* Define this to 1 for platforms that support environment variables */
/* Define this to 1 for platforms that support environment variables. */
/* TODO: Detect UWP, a.k.a. Windows Store app, and set this to 0. */
/* Try: #if defined(WINAPI_FAMILY_PARTITION) && WINAPI_FAMILY_PARTITION(WINAPI_PARTITION_APP) */
#define NVTX_SUPPORT_ENV_VARS 1
@@ -72,16 +72,16 @@ extern "C" {
#define NVTX_SUPPORT_DYNAMIC_INJECTION_LIBRARY 1
/* Injection libraries implementing InitializeInjectionNvtxExtension may be statically linked,
* and this will override any dynamic injection. Useful for platforms where dynamic
* injection is not available. Since weak symbols not explicitly marked extern are
* guaranteed to be initialized to zero if no definitions are found by the linker, the
* dynamic injection process proceeds normally if pfnInitializeInjectionNvtx2 is 0. */
* which will override any dynamic injection. This is useful for platforms, where dynamic
* injection is not available. Since weak symbols, not explicitly marked extern, are
* guaranteed to be initialized to zero, if no definitions are found by the linker, the
* dynamic injection process proceeds normally, if pfnInitializeInjectionNvtx2 is 0. */
#if defined(__GNUC__) && !defined(_WIN32) && !defined(__CYGWIN__)
#define NVTX_SUPPORT_STATIC_INJECTION_LIBRARY 1
/* To statically inject an NVTX library, define InitializeInjectionNvtxExtension_fnptr as a normal
* symbol (not weak) pointing to the implementation of InitializeInjectionNvtxExtension (which
* does not need to be named "InitializeInjectionNvtxExtension" as is necessary in a dynamic
* injection library. */
* symbol (not weak) pointing to the implementation of InitializeInjectionNvtxExtension, which
* does not need to be named "InitializeInjectionNvtxExtension" as it is necessary in a dynamic
* injection library. */
__attribute__((weak)) NvtxExtInitializeInjectionFunc_t InitializeInjectionNvtxExtension_fnptr;
#else
#define NVTX_SUPPORT_STATIC_INJECTION_LIBRARY 0
@@ -89,35 +89,37 @@ __attribute__((weak)) NvtxExtInitializeInjectionFunc_t InitializeInjectionNvtxEx
/* This function tries to find or load an NVTX injection library and get the
* address of its InitializeInjectionExtension function. If such a function pointer
* is found, it is called, and passed the address of this NVTX instance's
* nvtxGetExportTable function, so the injection can attach to this instance.
* If the initialization fails for any reason, any dynamic library loaded will
* be freed, and all NVTX implementation functions will be set to no-ops. If
* initialization succeeds, NVTX functions not attached to the tool will be set
* to no-ops. This is implemented as one function instead of several small
* functions to minimize the number of weak symbols the linker must resolve.
* Order of search is:
* - Pre-injected library exporting InitializeInjectionNvtxExtension
* - Loadable library exporting InitializeInjectionNvtxExtension
* - Path specified by env var NVTX_INJECTION??_PATH (?? is 32 or 64)
* - On Android, libNvtxInjection??.so within the package (?? is 32 or 64)
* - Statically-linked injection library defining InitializeInjectionNvtx2_fnptr
*/
NVTX_LINKONCE_FWDDECL_FUNCTION int NVTX_VERSIONED_IDENTIFIER(nvtxExtLoadInjectionLibrary)(NvtxExtInitializeInjectionFunc_t* out_init_fnptr);
NVTX_LINKONCE_DEFINE_FUNCTION int NVTX_VERSIONED_IDENTIFIER(nvtxExtLoadInjectionLibrary)(NvtxExtInitializeInjectionFunc_t* out_init_fnptr)
/* This function tries to find or load an NVTX injection library and get the address of its
* `InitializeInjectionExtension` function. If such a function pointer is found, it is called and
* passed the address of this NVTX instance's `nvtxGetExportTable` function, so that the injection
* can attach to this instance.
* If the initialization fails for any reason, any dynamic library loaded will be freed, and all
* NVTX implementation functions will be set to no-ops. If the initialization succeeds, NVTX
* functions that are not attached to the tool will be set to no-ops. This is implemented as one
* function instead of several small functions to minimize the number of weak symbols the linker
* must resolve. The order of search is:
* 1) Pre-injected library exporting InitializeInjectionNvtxExtension
* 2) Loadable library exporting InitializeInjectionNvtxExtension
* - Path specified by env var NVTX_INJECTION??_PATH (?? is 32 or 64)
* - On Android, libNvtxInjection??.so within the package (?? is 32 or 64)
* 3) Statically-linked injection library defining InitializeInjectionNvtx2_fnptr
*/
NVTX_LINKONCE_FWDDECL_FUNCTION int NVTX_VERSIONED_IDENTIFIER(nvtxExtLoadInjectionLibrary)(
NvtxExtInitializeInjectionFunc_t* out_init_fnptr);
NVTX_LINKONCE_DEFINE_FUNCTION int NVTX_VERSIONED_IDENTIFIER(nvtxExtLoadInjectionLibrary)(
NvtxExtInitializeInjectionFunc_t* out_init_fnptr)
{
const char* const initFuncName = "InitializeInjectionNvtxExtension";
NvtxExtInitializeInjectionFunc_t init_fnptr = (NvtxExtInitializeInjectionFunc_t)0;
NVTX_DLLHANDLE injectionLibraryHandle = (NVTX_DLLHANDLE)0;
if(out_init_fnptr){
if (out_init_fnptr)
{
*out_init_fnptr = (NvtxExtInitializeInjectionFunc_t)0;
}
#if NVTX_SUPPORT_ALREADY_INJECTED_LIBRARY
/* Use POSIX global symbol chain to query for init function from any module */
/* Use POSIX global symbol chain to query for init function from any module. */
init_fnptr = (NvtxExtInitializeInjectionFunc_t)NVTX_DLLFUNC(0, initFuncName);
#endif
@@ -127,7 +129,7 @@ NVTX_LINKONCE_DEFINE_FUNCTION int NVTX_VERSIONED_IDENTIFIER(nvtxExtLoadInjection
{
#if NVTX_SUPPORT_ENV_VARS
/* If env var NVTX_INJECTION64_PATH is set, it should contain the path
* to a 64-bit dynamic NVTX injection library (and similar for 32-bit). */
to a 64-bit dynamic NVTX injection library (and similar for 32-bit). */
const NVTX_PATHCHAR* const nvtxEnvVarName = (sizeof(void*) == 4)
? NVTX_STR("NVTX_INJECTION32_PATH")
: NVTX_STR("NVTX_INJECTION64_PATH");
@@ -135,12 +137,12 @@ NVTX_LINKONCE_DEFINE_FUNCTION int NVTX_VERSIONED_IDENTIFIER(nvtxExtLoadInjection
NVTX_PATHCHAR injectionLibraryPathBuf[NVTX_BUFSIZE];
const NVTX_PATHCHAR* injectionLibraryPath = (const NVTX_PATHCHAR*)0;
/* Refer to this variable explicitly in case all references to it are #if'ed out */
/* Refer to this variable explicitly in case all references to it are #if'ed out. */
(void)injectionLibraryPathBuf;
#if NVTX_SUPPORT_ENV_VARS
/* Disable the warning for getenv & _wgetenv -- this usage is safe because
* these functions are not called again before using the returned value. */
these functions are not called again before using the returned value. */
#if defined(_MSC_VER)
#pragma warning( push )
#pragma warning( disable : 4996 )
@@ -188,7 +190,8 @@ NVTX_LINKONCE_DEFINE_FUNCTION int NVTX_VERSIONED_IDENTIFIER(nvtxExtLoadInjection
pkgName[bytesRead] = 0;
/* String can contain colon as a process separator. In this case the package name is before the colon. */
/* String can contain colon as a process separator. In this case the
package name is before the colon. */
pos = 0;
while (pos < bytesRead && pkgName[pos] != ':' && pkgName[pos] != '\0')
{
@@ -223,8 +226,8 @@ NVTX_LINKONCE_DEFINE_FUNCTION int NVTX_VERSIONED_IDENTIFIER(nvtxExtLoadInjection
}
#endif
/* At this point, injectionLibraryPath is specified if a dynamic
* injection library was specified by a tool. */
/* At this point, `injectionLibraryPath` is specified if a dynamic
injection library was specified by a tool. */
if (injectionLibraryPath)
{
/* Load the injection library */
@@ -236,7 +239,7 @@ NVTX_LINKONCE_DEFINE_FUNCTION int NVTX_VERSIONED_IDENTIFIER(nvtxExtLoadInjection
}
else
{
/* Attempt to get the injection library's entry-point */
/* Attempt to get the injection library's entry-point. */
init_fnptr = (NvtxExtInitializeInjectionFunc_t)NVTX_DLLFUNC(injectionLibraryHandle, initFuncName);
if (!init_fnptr)
{
@@ -252,8 +255,8 @@ NVTX_LINKONCE_DEFINE_FUNCTION int NVTX_VERSIONED_IDENTIFIER(nvtxExtLoadInjection
#if NVTX_SUPPORT_STATIC_INJECTION_LIBRARY
if (!init_fnptr)
{
/* Check weakly-defined function pointer. A statically-linked injection can define this as
* a normal symbol and it will take precedence over a dynamic injection. */
/* Check weakly-defined function pointer. A statically-linked injection can define
this as a normal symbol and it will take precedence over a dynamic injection. */
if (InitializeInjectionNvtxExtension_fnptr)
{
init_fnptr = InitializeInjectionNvtxExtension_fnptr;
@@ -261,13 +264,13 @@ NVTX_LINKONCE_DEFINE_FUNCTION int NVTX_VERSIONED_IDENTIFIER(nvtxExtLoadInjection
}
#endif
if(out_init_fnptr){
if (out_init_fnptr)
{
*out_init_fnptr = init_fnptr;
}
/* At this point, if init_fnptr is not set, then no tool has specified
* an NVTX injection library -- return non-success result so all NVTX
* API functions will be set to no-ops. */
/* At this point, if `init_fnptr` is not set, no tool has specified an NVTX injection library.
Non-success result is returned, so that all NVTX API functions will be set to no-ops. */
if (!init_fnptr)
{
return NVTX_ERR_NO_INJECTION_LIBRARY_AVAILABLE;
@@ -276,16 +279,19 @@ NVTX_LINKONCE_DEFINE_FUNCTION int NVTX_VERSIONED_IDENTIFIER(nvtxExtLoadInjection
return NVTX_SUCCESS;
}
/* Avoid warnings about missing prototypes. */
NVTX_LINKONCE_FWDDECL_FUNCTION void NVTX_VERSIONED_IDENTIFIER(nvtxExtInitOnce) (
nvtxExtModuleInfo_t* moduleInfo, intptr_t* moduleState);
NVTX_LINKONCE_DEFINE_FUNCTION void NVTX_VERSIONED_IDENTIFIER(nvtxExtInitOnce) (
nvtxExtModuleInfo_t* moduleInfo,
intptr_t* moduleState
)
nvtxExtModuleInfo_t* moduleInfo, intptr_t* moduleState)
{
intptr_t old;
NVTX_INFO( "%s\n", __FUNCTION__ );
if( *moduleState == NVTX_EXTENSION_LOADED) {
if (*moduleState == NVTX_EXTENSION_LOADED)
{
NVTX_INFO("Module loaded\n");
return;
}
@@ -296,45 +302,55 @@ NVTX_LINKONCE_DEFINE_FUNCTION void NVTX_VERSIONED_IDENTIFIER(nvtxExtInitOnce) (
NVTX_EXTENSION_FRESH);
if (old == NVTX_EXTENSION_FRESH)
{
NvtxExtInitializeInjectionFunc_t init_fnptr = NVTX_VERSIONED_IDENTIFIER(injectionFnPtr);
NvtxExtInitializeInjectionFunc_t init_fnptr =
NVTX_VERSIONED_IDENTIFIER(nvtxExtGlobals1).injectionFnPtr;
int entryPointStatus = 0;
int forceAllToNoops = 0;
size_t s;
/* Load & initialize injection library -- it will assign the function pointers */
if(init_fnptr == 0){
/* Load and initialize injection library, which will assign the function pointers. */
if (init_fnptr == 0)
{
int result = 0;
/* try to load vanilla NVTX first*/
/* Try to load vanilla NVTX first. */
nvtxInitialize(0);
result = NVTX_VERSIONED_IDENTIFIER(nvtxExtLoadInjectionLibrary)(&init_fnptr);
/*at this point init_fnptr will be either 0 or a real function*/
/* At this point `init_fnptr` will be either 0 or a real function. */
if(result == NVTX_SUCCESS) {
NVTX_VERSIONED_IDENTIFIER(injectionFnPtr) = init_fnptr;
if (result == NVTX_SUCCESS)
{
NVTX_VERSIONED_IDENTIFIER(nvtxExtGlobals1).injectionFnPtr = init_fnptr;
}
else {
else
{
NVTX_ERR("Failed to load injection library\n");
}
}
if(init_fnptr != 0) {
/* Invoke injection library's initialization function. If it returns
* 0 (failure) and a dynamic injection was loaded, unload it. */
if (init_fnptr != 0)
{
/* Invoke injection library's initialization function. If it returns
0 (failure) and a dynamic injection was loaded, unload it. */
entryPointStatus = init_fnptr(moduleInfo);
if (entryPointStatus == 0) {
if (entryPointStatus == 0)
{
NVTX_ERR("Failed to initialize injection library -- initialization function returned 0\n");
}
}
/* Clean up any functions that are still uninitialized so that they are skipped.
* Set all to null if injection init function failed as well.
*/
/* Clean up any functions that are still uninitialized so that they are
skipped. Set all to null if injection init function failed as well. */
forceAllToNoops = (init_fnptr == 0) || (entryPointStatus == 0);
for(size_t s = 0; s < moduleInfo->segmentsCount; ++s){
nvtxExtModuleSegment_t* segment = moduleInfo->segments+s;
for(size_t i = 0; i < segment->slotCount; ++i){
if(forceAllToNoops || (segment->functionSlots[i] == NVTX_EXTENSION_FRESH)){
for (s = 0; s < moduleInfo->segmentsCount; ++s)
{
nvtxExtModuleSegment_t* segment = moduleInfo->segments + s;
size_t i;
for (i = 0; i < segment->slotCount; ++i)
{
if (forceAllToNoops || (segment->functionSlots[i] == NVTX_EXTENSION_FRESH))
{
segment->functionSlots[i] = NVTX_EXTENSION_DISABLED;
}
}
@@ -342,12 +358,11 @@ NVTX_LINKONCE_DEFINE_FUNCTION void NVTX_VERSIONED_IDENTIFIER(nvtxExtInitOnce) (
NVTX_MEMBAR();
/* Signal that initialization has finished, so now the assigned function pointers will be used */
NVTX_ATOMIC_WRITE_PTR(
moduleState,
NVTX_EXTENSION_LOADED);
/* Signal that initialization has finished and the assigned function
pointers will be used. */
NVTX_ATOMIC_WRITE_PTR(moduleState, NVTX_EXTENSION_LOADED);
}
else /* Spin-wait until initialization has finished */
else /* Spin-wait until initialization has finished. */
{
NVTX_MEMBAR();
while (*moduleState != NVTX_EXTENSION_LOADED)
src/include/nvtx3/nvtxDetail/nvtxExtPayloadHelperInternal.h
+272
Féach ar an gComhad
@@ -0,0 +1,272 @@
/*
* Copyright 2023 NVIDIA Corporation. All rights reserved.
*
* Licensed under the Apache License v2.0 with LLVM Exceptions.
* See https://llvm.org/LICENSE.txt for license information.
* SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
*/
#ifndef NVTX_EXT_PAYLOAD_HELPER_INTERNAL_H
#define NVTX_EXT_PAYLOAD_HELPER_INTERNAL_H
/* General helper macros */
#include "nvtxExtHelperMacros.h"
/* Get variable name with line number (almost unique per file). */
#define _NVTX_PAYLOAD_DATA_VAR NVTX_EXT_CONCAT(nvtxDFDB,__LINE__)
/* Create real arguments from just pasting tokens next to each other. */
#define _NVTX_PAYLOAD_PASS_THROUGH(...) __VA_ARGS__
/* Avoid prefixing `NVTX_PAYLOAD_ENTRY_` for nested payloads. */
#define NVTX_PAYLOAD_ENTRY_THROWAWAY
#define _NVTX_PAYLOAD_NESTED(id) THROWAWAY id
/*
* Create the NVTX binary payloads schema attributes.
*
* @param struct_id The name of the struct.
* @param schema_name The name of the schema.
* @param schema_flags Additional schema flags
* @param mask_add Fields to be added to the mask.
* @param num_entries The number schema entries.
*/
#define NVTX_PAYLOAD_SCHEMA_ATTR(struct_id, schema_name, schema_flags, schema_id, mask_add, num_entries) \
nvtxPayloadSchemaAttr_t struct_id##Attr = { \
/*.fieldMask = */NVTX_PAYLOAD_SCHEMA_ATTR_TYPE | mask_add \
NVTX_PAYLOAD_SCHEMA_ATTR_ENTRIES | \
NVTX_PAYLOAD_SCHEMA_ATTR_NUM_ENTRIES | \
NVTX_PAYLOAD_SCHEMA_ATTR_STATIC_SIZE, \
/*.name = */schema_name, \
/*.type = */NVTX_PAYLOAD_SCHEMA_TYPE_STATIC, \
/*.flags = */schema_flags, \
/*.entries = */struct_id##Schema, /*.numEntries = */num_entries, \
/*.payloadStaticSize = */sizeof(struct_id), \
/*.packAlign = */0, /*.schemaId = */schema_id};
/*****************************************************************/
/*** Helper for `NVTX_DEFINE_SCHEMA_FOR_STRUCT[_AND_REGISTER]` ***/
/* First part of schema entry for different number of arguments. */
#define _NVTX_PAYLOAD_SCHEMA_EF2(member, etype) \
0, NVTX_PAYLOAD_ENTRY_##etype, NULL, NULL, 0,
#define _NVTX_PAYLOAD_SCHEMA_EF3(member, etype, name) \
0, NVTX_PAYLOAD_ENTRY_##etype, name, NULL, 0,
#define _NVTX_PAYLOAD_SCHEMA_EF4(member, etype, name, desc) \
0, NVTX_PAYLOAD_ENTRY_##etype, name, desc, 0,
#define _NVTX_PAYLOAD_SCHEMA_EF5(member, etype, name, desc, arraylen) \
0, NVTX_PAYLOAD_ENTRY_##etype, name, desc, arraylen,
#define _NVTX_PAYLOAD_SCHEMA_EF6(member, etype, name, desc, arraylen, flags) \
NVTX_PAYLOAD_ENTRY_FLAG_##flags, NVTX_PAYLOAD_ENTRY_##etype, name, desc, arraylen,
#define _NVTX_PAYLOAD_SCHEMA_ENTRY_FRONT(...) \
NVTX_EXT_CONCAT(_NVTX_PAYLOAD_SCHEMA_EF, NVTX_EXT_NUM_ARGS(__VA_ARGS__))(__VA_ARGS__)
/* Second part of schema entry (append struct member).
(At least two arguments are passed (`member` and `etype`). */
#define _NVTX_PAYLOAD_SCHEMA_ENTRY_END(member, ...) member
/* Resolve to schema entry. `entry` is `(ctype, name, ...)`. */
#define _NVTX_PAYLOAD_SCHEMA_ENTRY(struct_id, entry) \
{_NVTX_PAYLOAD_SCHEMA_ENTRY_FRONT entry \
offsetof(struct_id, _NVTX_PAYLOAD_SCHEMA_ENTRY_END entry)},
/* Handle up to 16 schema entries. */
#define _NVTX_PAYLOAD_SME1(s,e1,...) _NVTX_PAYLOAD_SCHEMA_ENTRY(s,e1)
#define _NVTX_PAYLOAD_SME2(s,e1,...) _NVTX_PAYLOAD_SCHEMA_ENTRY(s,e1) _NVTX_PAYLOAD_SME1(s,__VA_ARGS__)
#define _NVTX_PAYLOAD_SME3(s,e1,...) _NVTX_PAYLOAD_SCHEMA_ENTRY(s,e1) _NVTX_PAYLOAD_SME2(s,__VA_ARGS__)
#define _NVTX_PAYLOAD_SME4(s,e1,...) _NVTX_PAYLOAD_SCHEMA_ENTRY(s,e1) _NVTX_PAYLOAD_SME3(s,__VA_ARGS__)
#define _NVTX_PAYLOAD_SME5(s,e1,...) _NVTX_PAYLOAD_SCHEMA_ENTRY(s,e1) _NVTX_PAYLOAD_SME4(s,__VA_ARGS__)
#define _NVTX_PAYLOAD_SME6(s,e1,...) _NVTX_PAYLOAD_SCHEMA_ENTRY(s,e1) _NVTX_PAYLOAD_SME5(s,__VA_ARGS__)
#define _NVTX_PAYLOAD_SME7(s,e1,...) _NVTX_PAYLOAD_SCHEMA_ENTRY(s,e1) _NVTX_PAYLOAD_SME6(s,__VA_ARGS__)
#define _NVTX_PAYLOAD_SME8(s,e1,...) _NVTX_PAYLOAD_SCHEMA_ENTRY(s,e1) _NVTX_PAYLOAD_SME7(s,__VA_ARGS__)
#define _NVTX_PAYLOAD_SME9(s,e1,...) _NVTX_PAYLOAD_SCHEMA_ENTRY(s,e1) _NVTX_PAYLOAD_SME8(s,__VA_ARGS__)
#define _NVTX_PAYLOAD_SME10(s,e1,...) _NVTX_PAYLOAD_SCHEMA_ENTRY(s,e1) _NVTX_PAYLOAD_SME9(s,__VA_ARGS__)
#define _NVTX_PAYLOAD_SME11(s,e1,...) _NVTX_PAYLOAD_SCHEMA_ENTRY(s,e1) _NVTX_PAYLOAD_SME10(s,__VA_ARGS__)
#define _NVTX_PAYLOAD_SME12(s,e1,...) _NVTX_PAYLOAD_SCHEMA_ENTRY(s,e1) _NVTX_PAYLOAD_SME11(s,__VA_ARGS__)
#define _NVTX_PAYLOAD_SME13(s,e1,...) _NVTX_PAYLOAD_SCHEMA_ENTRY(s,e1) _NVTX_PAYLOAD_SME12(s,__VA_ARGS__)
#define _NVTX_PAYLOAD_SME14(s,e1,...) _NVTX_PAYLOAD_SCHEMA_ENTRY(s,e1) _NVTX_PAYLOAD_SME13(s,__VA_ARGS__)
#define _NVTX_PAYLOAD_SME15(s,e1,...) _NVTX_PAYLOAD_SCHEMA_ENTRY(s,e1) _NVTX_PAYLOAD_SME14(s,__VA_ARGS__)
#define _NVTX_PAYLOAD_SME16(s,e1,...) _NVTX_PAYLOAD_SCHEMA_ENTRY(s,e1) _NVTX_PAYLOAD_SME15(s,__VA_ARGS__)
#define _NVTX_PAYLOAD_SCHEMA_ENTRIES(struct_id, ...) \
nvtxPayloadSchemaEntry_t struct_id##Schema[] = { \
NVTX_EXT_CONCAT(_NVTX_PAYLOAD_SME, NVTX_EXT_NUM_ARGS(__VA_ARGS__))(struct_id, __VA_ARGS__) \
{0, 0} \
};
/*
* Handle optional parameters for `NVTX_DEFINE_SCHEMA_FOR_STRUCT[_AND_REGISTER]`.
*/
#define _NVTX_DEFINE_S4S_6(struct_id, schema_name, prefix, schema_flags, schema_id, entries) \
prefix _NVTX_PAYLOAD_SCHEMA_ENTRIES(struct_id, _NVTX_PAYLOAD_PASS_THROUGH entries) \
prefix NVTX_PAYLOAD_SCHEMA_ATTR(struct_id, schema_name, schema_flags, schema_id, \
NVTX_PAYLOAD_SCHEMA_ATTR_NAME | NVTX_PAYLOAD_SCHEMA_ATTR_FLAGS | NVTX_PAYLOAD_SCHEMA_ATTR_SCHEMA_ID |,\
NVTX_EXT_NUM_ARGS(_NVTX_PAYLOAD_PASS_THROUGH entries))
#define _NVTX_DEFINE_S4S_5(struct_id, schema_name, prefix, schema_flags, entries) \
prefix _NVTX_PAYLOAD_SCHEMA_ENTRIES(struct_id, _NVTX_PAYLOAD_PASS_THROUGH entries) \
prefix NVTX_PAYLOAD_SCHEMA_ATTR(struct_id, schema_name, schema_flags, 0, \
NVTX_PAYLOAD_SCHEMA_ATTR_NAME | NVTX_PAYLOAD_SCHEMA_ATTR_FLAGS |, \
NVTX_EXT_NUM_ARGS(_NVTX_PAYLOAD_PASS_THROUGH entries))
#define _NVTX_DEFINE_S4S_4(struct_id, schema_name, prefix, entries) \
prefix _NVTX_PAYLOAD_SCHEMA_ENTRIES(struct_id, _NVTX_PAYLOAD_PASS_THROUGH entries) \
prefix NVTX_PAYLOAD_SCHEMA_ATTR(struct_id, schema_name, NVTX_PAYLOAD_SCHEMA_FLAG_NONE, 0, \
NVTX_PAYLOAD_SCHEMA_ATTR_NAME |, \
NVTX_EXT_NUM_ARGS(_NVTX_PAYLOAD_PASS_THROUGH entries))
#define _NVTX_DEFINE_S4S_3(struct_id, schema_name, entries) \
_NVTX_DEFINE_S4S_4(struct_id, schema_name, /*prefix*/, entries)
#define _NVTX_DEFINE_S4S_2(struct_id, entries) \
_NVTX_PAYLOAD_SCHEMA_ENTRIES(struct_id, _NVTX_PAYLOAD_PASS_THROUGH entries) \
NVTX_PAYLOAD_SCHEMA_ATTR(struct_id, NULL, NVTX_PAYLOAD_SCHEMA_FLAG_NONE, 0, ,\
NVTX_EXT_NUM_ARGS(_NVTX_PAYLOAD_PASS_THROUGH entries))
#define _NVTX_DEFINE_SCHEMA_FOR_STRUCT(struct_id, ...) \
NVTX_EXT_CONCAT(_NVTX_DEFINE_S4S_, \
NVTX_EXT_NUM_ARGS(struct_id, __VA_ARGS__))(struct_id, __VA_ARGS__)
/*** END: Helper for `NVTX_PAYLOAD_STATIC_SCHEMA_{DEFINE,SETUP}` ***/
/******************************************************************/
/*** Helper for `NVTX_DEFINE_STRUCT_WITH_SCHEMA[_AND_REGISTER]` ***/
/* Extract struct member for fixed-size arrays. */
#define _NVTX_PAYLOAD_STRUCT_ARR_MEM1(name) name
#define _NVTX_PAYLOAD_STRUCT_ARR_MEM2(name, count) name[count]
/* Extract type and member name and handle special case of fixed-size array. */
#define _NVTX_PAYLOAD_STRUCT_E2(type, member) type member;
#define _NVTX_PAYLOAD_STRUCT_E3(type, member, etype) type member;
#define _NVTX_PAYLOAD_STRUCT_E4(type, member, etype, name) type member;
#define _NVTX_PAYLOAD_STRUCT_E5(type, member, etype, name, desc) type member;
#define _NVTX_PAYLOAD_STRUCT_E6(type, member, etype, name, desc, arraylen) \
type NVTX_EXT_CONCAT(_NVTX_PAYLOAD_STRUCT_ARR_MEM, NVTX_EXT_NUM_ARGS member) member;
#define _NVTX_PAYLOAD_STRUCT_E7(type, member, etype, name, desc, arraylen, flags) \
_NVTX_PAYLOAD_STRUCT_E6(type, member, etype, name, desc, arraylen)
/* Handle different number of arguments per struct entry. */
#define _NVTX_PAYLOAD_STRUCT_ENTRY_(...) \
NVTX_EXT_CONCAT(_NVTX_PAYLOAD_STRUCT_E, NVTX_EXT_NUM_ARGS(__VA_ARGS__))(__VA_ARGS__)
/* Handle up to 16 struct members. */
#define _NVTX_PAYLOAD_STRUCT_ENTRY(entry) _NVTX_PAYLOAD_STRUCT_ENTRY_ entry
#define _NVTX_PAYLOAD_STRUCT1(e1, ...) _NVTX_PAYLOAD_STRUCT_ENTRY(e1)
#define _NVTX_PAYLOAD_STRUCT2(e1, ...) _NVTX_PAYLOAD_STRUCT_ENTRY(e1) _NVTX_PAYLOAD_STRUCT1(__VA_ARGS__)
#define _NVTX_PAYLOAD_STRUCT3(e1, ...) _NVTX_PAYLOAD_STRUCT_ENTRY(e1) _NVTX_PAYLOAD_STRUCT2(__VA_ARGS__)
#define _NVTX_PAYLOAD_STRUCT4(e1, ...) _NVTX_PAYLOAD_STRUCT_ENTRY(e1) _NVTX_PAYLOAD_STRUCT3(__VA_ARGS__)
#define _NVTX_PAYLOAD_STRUCT5(e1, ...) _NVTX_PAYLOAD_STRUCT_ENTRY(e1) _NVTX_PAYLOAD_STRUCT4(__VA_ARGS__)
#define _NVTX_PAYLOAD_STRUCT6(e1, ...) _NVTX_PAYLOAD_STRUCT_ENTRY(e1) _NVTX_PAYLOAD_STRUCT5(__VA_ARGS__)
#define _NVTX_PAYLOAD_STRUCT7(e1, ...) _NVTX_PAYLOAD_STRUCT_ENTRY(e1) _NVTX_PAYLOAD_STRUCT6(__VA_ARGS__)
#define _NVTX_PAYLOAD_STRUCT8(e1, ...) _NVTX_PAYLOAD_STRUCT_ENTRY(e1) _NVTX_PAYLOAD_STRUCT7(__VA_ARGS__)
#define _NVTX_PAYLOAD_STRUCT9(e1, ...) _NVTX_PAYLOAD_STRUCT_ENTRY(e1) _NVTX_PAYLOAD_STRUCT8(__VA_ARGS__)
#define _NVTX_PAYLOAD_STRUCT10(e1, ...) _NVTX_PAYLOAD_STRUCT_ENTRY(e1) _NVTX_PAYLOAD_STRUCT9(__VA_ARGS__)
#define _NVTX_PAYLOAD_STRUCT11(e1, ...) _NVTX_PAYLOAD_STRUCT_ENTRY(e1) _NVTX_PAYLOAD_STRUCT10(__VA_ARGS__)
#define _NVTX_PAYLOAD_STRUCT12(e1, ...) _NVTX_PAYLOAD_STRUCT_ENTRY(e1) _NVTX_PAYLOAD_STRUCT11(__VA_ARGS__)
#define _NVTX_PAYLOAD_STRUCT13(e1, ...) _NVTX_PAYLOAD_STRUCT_ENTRY(e1) _NVTX_PAYLOAD_STRUCT12(__VA_ARGS__)
#define _NVTX_PAYLOAD_STRUCT14(e1, ...) _NVTX_PAYLOAD_STRUCT_ENTRY(e1) _NVTX_PAYLOAD_STRUCT13(__VA_ARGS__)
#define _NVTX_PAYLOAD_STRUCT15(e1, ...) _NVTX_PAYLOAD_STRUCT_ENTRY(e1) _NVTX_PAYLOAD_STRUCT14(__VA_ARGS__)
#define _NVTX_PAYLOAD_STRUCT16(e1, ...) _NVTX_PAYLOAD_STRUCT_ENTRY(e1) _NVTX_PAYLOAD_STRUCT15(__VA_ARGS__)
/* Generate the typedef. */
#define _NVTX_PAYLOAD_TYPEDEF_STRUCT(struct_id, ...) \
typedef struct { \
NVTX_EXT_CONCAT(_NVTX_PAYLOAD_STRUCT, NVTX_EXT_NUM_ARGS(__VA_ARGS__))(__VA_ARGS__) \
} struct_id;
/* Generate first part of the schema entry. */
#define _NVTX_PAYLOAD_INIT_SCHEMA_N3(type, memberId, etype) \
0, NVTX_PAYLOAD_ENTRY_##etype, NULL, NULL, 0,
#define _NVTX_PAYLOAD_INIT_SCHEMA_N4(type, memberId, etype, name) \
0, NVTX_PAYLOAD_ENTRY_##etype, name, NULL, 0,
#define _NVTX_PAYLOAD_INIT_SCHEMA_N5(type, memberId, etype, name, desc) \
0, NVTX_PAYLOAD_ENTRY_##etype, name, desc, 0,
#define _NVTX_PAYLOAD_INIT_SCHEMA_N6(type, memberId, etype, name, desc, arraylen) \
0, NVTX_PAYLOAD_ENTRY_##etype, name, desc, arraylen,
#define _NVTX_PAYLOAD_INIT_SCHEMA_N7(type, memberId, etype, name, desc, arraylen, flags) \
NVTX_PAYLOAD_ENTRY_FLAG_##flags, NVTX_PAYLOAD_ENTRY_##etype, name, desc, arraylen,
#define _NVTX_PAYLOAD_SCHEMA_INIT_ENTRY_FRONT(...) \
NVTX_EXT_CONCAT(_NVTX_PAYLOAD_INIT_SCHEMA_N, NVTX_EXT_NUM_ARGS(__VA_ARGS__))(__VA_ARGS__)
#define _NVTX_PAYLOAD_ARRAY_MEMBER1(name) name
#define _NVTX_PAYLOAD_ARRAY_MEMBER2(name, count) name
/* Resolve to last part of schema entry (append struct member). */
#define _NVTX_PAYLOAD_INIT_SCHEMA_NX3(type, memberId, ...) memberId
#define _NVTX_PAYLOAD_INIT_SCHEMA_NX4(type, memberId, ...) memberId
#define _NVTX_PAYLOAD_INIT_SCHEMA_NX5(type, memberId, ...) memberId
#define _NVTX_PAYLOAD_INIT_SCHEMA_NX6(type, memberId, ...) \
NVTX_EXT_CONCAT(_NVTX_PAYLOAD_ARRAY_MEMBER, NVTX_EXT_NUM_ARGS memberId) memberId
#define _NVTX_PAYLOAD_INIT_SCHEMA_NX7(type, memberId, ...) \
_NVTX_PAYLOAD_INIT_SCHEMA_NX6(type, memberId, __VA_ARGS__)
#define _NVTX_PAYLOAD_SCHEMA_INIT_ENTRY_END(...) \
NVTX_EXT_CONCAT(_NVTX_PAYLOAD_INIT_SCHEMA_NX, NVTX_EXT_NUM_ARGS(__VA_ARGS__))(__VA_ARGS__)
/* Resolve to schema entry. `entry` is `(ctype, name, ...)`. */
#define _NVTX_PAYLOAD_SCHEMA_INIT_ENTRY(struct_id, entry) \
{_NVTX_PAYLOAD_SCHEMA_INIT_ENTRY_FRONT entry \
offsetof(struct_id, _NVTX_PAYLOAD_SCHEMA_INIT_ENTRY_END entry)},
/* Handle up to 16 schema entries. */
#define _NVTX_PAYLOAD_INIT_SME1(s, e1, ...) _NVTX_PAYLOAD_SCHEMA_INIT_ENTRY(s, e1)
#define _NVTX_PAYLOAD_INIT_SME2(s, e1, ...) _NVTX_PAYLOAD_SCHEMA_INIT_ENTRY(s, e1) _NVTX_PAYLOAD_INIT_SME1(s, __VA_ARGS__)
#define _NVTX_PAYLOAD_INIT_SME3(s, e1, ...) _NVTX_PAYLOAD_SCHEMA_INIT_ENTRY(s, e1) _NVTX_PAYLOAD_INIT_SME2(s, __VA_ARGS__)
#define _NVTX_PAYLOAD_INIT_SME4(s, e1, ...) _NVTX_PAYLOAD_SCHEMA_INIT_ENTRY(s, e1) _NVTX_PAYLOAD_INIT_SME3(s, __VA_ARGS__)
#define _NVTX_PAYLOAD_INIT_SME5(s, e1, ...) _NVTX_PAYLOAD_SCHEMA_INIT_ENTRY(s, e1) _NVTX_PAYLOAD_INIT_SME4(s, __VA_ARGS__)
#define _NVTX_PAYLOAD_INIT_SME6(s, e1, ...) _NVTX_PAYLOAD_SCHEMA_INIT_ENTRY(s, e1) _NVTX_PAYLOAD_INIT_SME5(s, __VA_ARGS__)
#define _NVTX_PAYLOAD_INIT_SME7(s, e1, ...) _NVTX_PAYLOAD_SCHEMA_INIT_ENTRY(s, e1) _NVTX_PAYLOAD_INIT_SME6(s, __VA_ARGS__)
#define _NVTX_PAYLOAD_INIT_SME8(s, e1, ...) _NVTX_PAYLOAD_SCHEMA_INIT_ENTRY(s, e1) _NVTX_PAYLOAD_INIT_SME7(s, __VA_ARGS__)
#define _NVTX_PAYLOAD_INIT_SME9(s, e1, ...) _NVTX_PAYLOAD_SCHEMA_INIT_ENTRY(s, e1) _NVTX_PAYLOAD_INIT_SME8(s, __VA_ARGS__)
#define _NVTX_PAYLOAD_INIT_SME10(s, e1, ...) _NVTX_PAYLOAD_SCHEMA_INIT_ENTRY(s, e1) _NVTX_PAYLOAD_INIT_SME9(s, __VA_ARGS__)
#define _NVTX_PAYLOAD_INIT_SME11(s, e1, ...) _NVTX_PAYLOAD_SCHEMA_INIT_ENTRY(s, e1) _NVTX_PAYLOAD_INIT_SME10(s, __VA_ARGS__)
#define _NVTX_PAYLOAD_INIT_SME12(s, e1, ...) _NVTX_PAYLOAD_SCHEMA_INIT_ENTRY(s, e1) _NVTX_PAYLOAD_INIT_SME11(s, __VA_ARGS__)
#define _NVTX_PAYLOAD_INIT_SME13(s, e1, ...) _NVTX_PAYLOAD_SCHEMA_INIT_ENTRY(s, e1) _NVTX_PAYLOAD_INIT_SME12(s, __VA_ARGS__)
#define _NVTX_PAYLOAD_INIT_SME14(s, e1, ...) _NVTX_PAYLOAD_SCHEMA_INIT_ENTRY(s, e1) _NVTX_PAYLOAD_INIT_SME13(s, __VA_ARGS__)
#define _NVTX_PAYLOAD_INIT_SME15(s, e1, ...) _NVTX_PAYLOAD_SCHEMA_INIT_ENTRY(s, e1) _NVTX_PAYLOAD_INIT_SME14(s, __VA_ARGS__)
#define _NVTX_PAYLOAD_INIT_SME16(s, e1, ...) _NVTX_PAYLOAD_SCHEMA_INIT_ENTRY(s, e1) _NVTX_PAYLOAD_INIT_SME15(s, __VA_ARGS__)
#define _NVTX_PAYLOAD_SCHEMA_INIT_ENTRIES(struct_id, ...) \
nvtxPayloadSchemaEntry_t struct_id##Schema[] = { \
NVTX_EXT_CONCAT(_NVTX_PAYLOAD_INIT_SME, NVTX_EXT_NUM_ARGS(__VA_ARGS__))(struct_id, __VA_ARGS__) \
{0, 0} \
};
/*
* Handle optional parameters for `NVTX_DEFINE_STRUCT_WITH_SCHEMA[_AND_REGISTER]`.
*/
#define _NVTX_DEFINE_SWS_6(struct_id, schema_name, prefix, schema_flags, schema_id, entries) \
_NVTX_PAYLOAD_TYPEDEF_STRUCT(struct_id, _NVTX_PAYLOAD_PASS_THROUGH entries) \
prefix _NVTX_PAYLOAD_SCHEMA_INIT_ENTRIES(struct_id, _NVTX_PAYLOAD_PASS_THROUGH entries) \
prefix NVTX_PAYLOAD_SCHEMA_ATTR(struct_id, schema_name, schema_flags, schema_id, \
NVTX_PAYLOAD_SCHEMA_ATTR_NAME | NVTX_PAYLOAD_SCHEMA_ATTR_FLAGS | \
NVTX_PAYLOAD_SCHEMA_ATTR_SCHEMA_ID |, \
NVTX_EXT_NUM_ARGS(_NVTX_PAYLOAD_PASS_THROUGH entries))
#define _NVTX_DEFINE_SWS_5(struct_id, schema_name, prefix, schema_flags, entries) \
_NVTX_PAYLOAD_TYPEDEF_STRUCT(struct_id, _NVTX_PAYLOAD_PASS_THROUGH entries) \
prefix _NVTX_PAYLOAD_SCHEMA_INIT_ENTRIES(struct_id, _NVTX_PAYLOAD_PASS_THROUGH entries) \
prefix NVTX_PAYLOAD_SCHEMA_ATTR(struct_id, schema_name, schema_flags, 0, \
NVTX_PAYLOAD_SCHEMA_ATTR_NAME | NVTX_PAYLOAD_SCHEMA_ATTR_FLAGS |, \
NVTX_EXT_NUM_ARGS(_NVTX_PAYLOAD_PASS_THROUGH entries))
#define _NVTX_DEFINE_SWS_4(struct_id, schema_name, prefix, entries) \
_NVTX_PAYLOAD_TYPEDEF_STRUCT(struct_id, _NVTX_PAYLOAD_PASS_THROUGH entries) \
prefix _NVTX_PAYLOAD_SCHEMA_INIT_ENTRIES(struct_id, _NVTX_PAYLOAD_PASS_THROUGH entries) \
prefix NVTX_PAYLOAD_SCHEMA_ATTR(struct_id, schema_name, NVTX_PAYLOAD_SCHEMA_FLAG_NONE, 0, \
NVTX_PAYLOAD_SCHEMA_ATTR_NAME |, \
NVTX_EXT_NUM_ARGS(_NVTX_PAYLOAD_PASS_THROUGH entries))
#define _NVTX_DEFINE_SWS_3(struct_id, schema_name, entries) \
_NVTX_DEFINE_SWS_4(struct_id, schema_name, /* no prefix */, entries)
#define _NVTX_DEFINE_SWS_2(struct_id, entries) \
_NVTX_PAYLOAD_TYPEDEF_STRUCT(struct_id, _NVTX_PAYLOAD_PASS_THROUGH entries) \
_NVTX_PAYLOAD_SCHEMA_INIT_ENTRIES(struct_id, _NVTX_PAYLOAD_PASS_THROUGH entries) \
NVTX_PAYLOAD_SCHEMA_ATTR(struct_id, NULL, NVTX_PAYLOAD_SCHEMA_FLAG_NONE, 0, , \
NVTX_EXT_NUM_ARGS(_NVTX_PAYLOAD_PASS_THROUGH entries))
#define _NVTX_DEFINE_STRUCT_WITH_SCHEMA(struct_id, ...) \
NVTX_EXT_CONCAT(_NVTX_DEFINE_SWS_, \
NVTX_EXT_NUM_ARGS(struct_id, __VA_ARGS__))(struct_id, __VA_ARGS__)
/*** END: Helper for `NVTX_PAYLOAD_STATIC_SCHEMA_{INIT,CREATE}` */
#endif /* NVTX_EXT_PAYLOAD_HELPER_INTERNAL_H */
src/include/nvtx3/nvtxExtDetail/nvtxExtPayloadTypeInfo.h → src/include/nvtx3/nvtxDetail/nvtxExtPayloadTypeInfo.h
+14 -6
Féach ar an gComhad
@@ -10,14 +10,14 @@
#error Never include this file directly -- it is automatically included by nvToolsExtPayload.h (except when NVTX_NO_IMPL is defined).
#endif
typedef void* pointer_type;
typedef void* nvtx_payload_pointer_type;
#if (defined(__STDC_VERSION__) && __STDC_VERSION__ >= 201112L)
#include <uchar.h>
#include <stdalign.h>
#endif
/* `alignof` is available as of C11 or C++11 */
/* `alignof` is available as of C11 or C++11. */
#if (defined(__STDC_VERSION__) && (__STDC_VERSION__ >= 201112L)) || (defined(__cplusplus) && __cplusplus >= 201103L)
#define nvtx_alignof(type) alignof(type)
@@ -54,7 +54,7 @@ MKTYPEDEF(double);
MKTYPEDEF2(long double, longdouble);
MKTYPEDEF(size_t);
MKTYPEDEF(pointer_type);
MKTYPEDEF(nvtx_payload_pointer_type);
MKTYPEDEF(wchar_t);
@@ -85,8 +85,16 @@ MKTYPEDEF(wchar_t);
/*
* Helper array to get the alignment for each predefined C/C++ language type.
* The order of entries must match the values in`enum nvtxPayloadSchemaEntryType`.
*
* In C++, `const` variables use internal linkage by default, but we need it to
* be public (extern) since weak declarations must be public.
*/
const nvtxPayloadEntryTypeInfo_t nvtxExtPayloadTypeInfo[NVTX_PAYLOAD_ENTRY_TYPE_INFO_ARRAY_SIZE] =
NVTX_LINKONCE_DEFINE_GLOBAL
#ifdef __cplusplus
extern
#endif
const nvtxPayloadEntryTypeInfo_t
NVTX_EXT_PAYLOAD_VERSIONED_ID(nvtxExtPayloadTypeInfo)[NVTX_PAYLOAD_ENTRY_TYPE_INFO_ARRAY_SIZE] =
{
/* The first entry contains this array's length and the size of each entry in this array. */
{NVTX_PAYLOAD_ENTRY_TYPE_INFO_ARRAY_SIZE, sizeof(nvtxPayloadEntryTypeInfo_t)},
@@ -119,7 +127,7 @@ const nvtxPayloadEntryTypeInfo_t nvtxExtPayloadTypeInfo[NVTX_PAYLOAD_ENTRY_TYPE_
/* NVTX_PAYLOAD_ENTRY_TYPE_LONGDOUBLE */ {sizeof(long double), nvtx_alignof2(long double, longdouble)},
/* NVTX_PAYLOAD_ENTRY_TYPE_SIZE */ {sizeof(size_t), nvtx_alignof(size_t)},
/* NVTX_PAYLOAD_ENTRY_TYPE_ADDRESS */ {sizeof(pointer_type), nvtx_alignof(pointer_type)},
/* NVTX_PAYLOAD_ENTRY_TYPE_ADDRESS */ {sizeof(nvtx_payload_pointer_type), nvtx_alignof(nvtx_payload_pointer_type)},
/*** Special character types ***/
/* NVTX_PAYLOAD_ENTRY_TYPE_WCHAR */ {sizeof(wchar_t), nvtx_alignof(wchar_t)},
@@ -140,4 +148,4 @@ const nvtxPayloadEntryTypeInfo_t nvtxExtPayloadTypeInfo[NVTX_PAYLOAD_ENTRY_TYPE_
};
#undef nvtx_alignof
#undef nvtx_alignof2
#undef nvtx_alignof2
src/include/nvtx3/nvtxExtDetail/nvtxExtTypes.h → src/include/nvtx3/nvtxDetail/nvtxExtTypes.h
Féach ar an gComhad
src/include/nvtx3/nvtxDetail/nvtxImpl.h
+9 -12
Féach ar an gComhad
@@ -10,37 +10,34 @@
#error Never include this file directly -- it is automatically included by nvToolsExt.h (except when NVTX_NO_IMPL is defined).
#endif
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <wchar.h>
/* ---- Include required platform headers ---- */
#if defined(_WIN32)
#if defined(_WIN32)
#include <Windows.h>
#include <windows.h>
#else
#include <unistd.h>
#if defined(__ANDROID__)
#include <android/api-level.h>
#include <android/api-level.h>
#endif
#if defined(__linux__) || defined(__CYGWIN__)
#include <sched.h>
#endif
#include <sys/types.h>
#include <limits.h>
#include <dlfcn.h>
#include <fcntl.h>
#include <stdlib.h>
#include <stdio.h>
#include <sys/types.h>
#include <unistd.h>
#include <errno.h>
#include <string.h>
#include <sys/types.h>
#include <pthread.h>
#include <stdlib.h>
#include <wchar.h>
#endif
src/include/nvtx3/nvtxDetail/nvtxInit.h
+3 -3
Féach ar an gComhad
@@ -14,11 +14,11 @@
/* Prefer macros over inline functions to reduce symbol resolution at link time */
#if defined(_WIN32)
#if defined(_WIN32)
#define NVTX_PATHCHAR wchar_t
#define NVTX_STR(x) L##x
#define NVTX_GETENV _wgetenv
#define NVTX_BUFSIZE MAX_PATH
#define NVTX_BUFSIZE 16384
#define NVTX_DLLHANDLE HMODULE
#define NVTX_DLLOPEN(x) LoadLibraryW(x)
#define NVTX_DLLFUNC GetProcAddress
@@ -31,7 +31,7 @@
#define NVTX_PATHCHAR char
#define NVTX_STR(x) x
#define NVTX_GETENV getenv
#define NVTX_BUFSIZE PATH_MAX
#define NVTX_BUFSIZE 16384
#define NVTX_DLLHANDLE void*
#define NVTX_DLLOPEN(x) dlopen(x, RTLD_LAZY)
#define NVTX_DLLFUNC dlsym
src/include/nvtx3/nvtxDetail/nvtxLinkOnce.h
+1 -1
Féach ar an gComhad
@@ -23,7 +23,7 @@
* In some situations it is desirable to declare a variable without initializing
* it, refer to it in code or other variables' initializers, and then initialize
* it later. Similarly, functions can be prototyped, have their address taken,
* and then have their body defined later. In such cases, use the FWDDECL macros
* and then have their body defined later. In such cases, use the FWDDECL macros
* when forward-declaring LINKONCE global variables without initializers and
* function prototypes, and then use the DEFINE macros when later defining them.
* Although in many cases the FWDDECL macro is equivalent to the DEFINE macro,
src/include/nvtx3/nvtxExtDetail/nvtxExtImplPayload_v1.h
-86
Féach ar an gComhad
@@ -1,86 +0,0 @@
/*
* Copyright 2021 NVIDIA Corporation. All rights reserved.
*
* Licensed under the Apache License v2.0 with LLVM Exceptions.
* See https://llvm.org/LICENSE.txt for license information.
* SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
*/
#ifndef NVTX_EXT_IMPL_PAYLOAD_GUARD
#error Never include this file directly -- it is automatically included by nvToolsExtPayload.h (except when NVTX_NO_IMPL is defined).
#endif
#define NVTX_EXT_IMPL_GUARD
#include "nvtxExtImpl.h"
#undef NVTX_EXT_IMPL_GUARD
#ifdef __cplusplus
extern "C" {
#endif /* __cplusplus */
#define NVTX_EXT_PAYLOAD_VERSIONED_IDENTIFIER_L3(NAME, VERSION, COMPATID) \
NAME##_v##VERSION##_mem##COMPATID
#define NVTX_EXT_PAYLOAD_VERSIONED_IDENTIFIER_L2(NAME, VERSION, COMPATID) \
NVTX_EXT_PAYLOAD_VERSIONED_IDENTIFIER_L3(NAME, VERSION, COMPATID)
#define NVTX_EXT_PAYLOAD_VERSIONED_ID(NAME) \
NVTX_EXT_PAYLOAD_VERSIONED_IDENTIFIER_L2(NAME, NVTX_VERSION, NVTX_EXT_COMPATID_PAYLOAD)
/*
* Function slots for the binary payload extension. First entry is the module
* state, initialized to `0` (`NVTX_EXTENSION_FRESH`).
*/
NVTX_LINKONCE_DEFINE_GLOBAL intptr_t
NVTX_EXT_PAYLOAD_VERSIONED_ID(nvtxExtPayloadSlots)[NVTX3EXT_CBID_PAYLOAD_FN_NUM + 1]
= {0};
NVTX_LINKONCE_DEFINE_FUNCTION void NVTX_EXT_PAYLOAD_VERSIONED_ID(nvtxExtPayloadInitOnce)()
{
intptr_t* fnSlots = NVTX_EXT_PAYLOAD_VERSIONED_ID(nvtxExtPayloadSlots) + 1;
nvtxExtModuleSegment_t segment = {
0, // unused (only one segment)
NVTX3EXT_CBID_PAYLOAD_FN_NUM,
fnSlots
};
nvtxExtModuleInfo_t module = {
NVTX_VERSION, sizeof(nvtxExtModuleInfo_t),
NVTX_EXT_MODULEID_PAYLOAD, NVTX_EXT_COMPATID_PAYLOAD,
1, &segment, // number of segments, segments
NULL, // no export function needed
// bake type sizes and alignment information into program binary
&nvtxExtPayloadTypeInfo
};
NVTX_INFO( "%s\n", __FUNCTION__ );
NVTX_VERSIONED_IDENTIFIER(nvtxExtInitOnce)(&module,
NVTX_EXT_PAYLOAD_VERSIONED_ID(nvtxExtPayloadSlots));
}
#define NVTX_EXT_FN_IMPL(ret_val, fn_name, signature, arg_names) \
typedef ret_val ( * fn_name##_impl_fntype )signature; \
NVTX_LINKONCE_DEFINE_FUNCTION ret_val fn_name signature { \
intptr_t slot = NVTX_EXT_PAYLOAD_VERSIONED_ID(nvtxExtPayloadSlots)[NVTX3EXT_CBID_##fn_name + 1]; \
if (slot != NVTX_EXTENSION_DISABLED) { \
if (slot) { \
return (*(fn_name##_impl_fntype)slot) arg_names; \
} else { \
NVTX_EXT_PAYLOAD_VERSIONED_ID(nvtxExtPayloadInitOnce)(); \
slot = NVTX_EXT_PAYLOAD_VERSIONED_ID(nvtxExtPayloadSlots)[NVTX3EXT_CBID_##fn_name + 1]; \
if (slot != NVTX_EXTENSION_DISABLED && slot) { \
return (*(fn_name##_impl_fntype)slot) arg_names; \
} \
} \
} \
return ((ret_val)(intptr_t)-1); \
}
NVTX_EXT_FN_IMPL(uint64_t, nvtxPayloadSchemaRegister, (nvtxDomainHandle_t domain, const nvtxPayloadSchemaAttr_t* attr), (domain, attr))
NVTX_EXT_FN_IMPL(uint64_t, nvtxPayloadEnumRegister, (nvtxDomainHandle_t domain, const nvtxPayloadEnumAttr_t* attr), (domain, attr))
#undef NVTX_EXT_FN_IMPL
#ifdef __cplusplus
} /* extern "C" */
#endif /* __cplusplus */
src/include/p2p.h
+3
Féach ar an gComhad
@@ -10,6 +10,9 @@
#define NCCL_P2P_H_
#include <cuda.h>
#include <cuda_runtime.h>
#include "core.h"
#if CUDART_VERSION < 12030
// MNNVL: FABRIC handle support lifted from CUDA 12.3
src/include/proxy.h
+21 -5
Féach ar an gComhad
@@ -16,13 +16,29 @@
#include "shm.h"
#include "p2p.h"
typedef enum : uint8_t {
ncclPatternRing,
ncclPatternRingTwice,
ncclPatternPipelineFrom,
ncclPatternPipelineTo,
ncclPatternTreeUp,
ncclPatternTreeDown,
ncclPatternTreeUpDown,
ncclPatternCollnetChain,
ncclPatternCollnetDirect,
ncclPatternNvls,
ncclPatternNvlsTree,
ncclPatternSend,
ncclPatternRecv
} ncclPattern_t;
enum ncclProxyOpState { ncclProxyOpNone, ncclProxyOpReady, ncclProxyOpProgress };
struct ncclProxyArgs;
typedef ncclResult_t (*proxyProgressFunc_t)(struct ncclProxyState*, struct ncclProxyArgs*);
#define NCCL_PROXY_MAX_SUBS MAXCHANNELS
static_assert(NCCL_MAX_WORK_ELEMENTS <= MAXCHANNELS, "Not enough sub space for max work elements");
static_assert(2*NCCL_MAX_DEV_WORK_P2P_PER_BATCH <= MAXCHANNELS, "Not enough sub space for max work elements");
union ncclProxyOpSpecifics {
struct {
@@ -124,8 +140,9 @@ struct ncclProxyArgs {
// ProxyOps are used to communicate between main thread and service thread
// Make sure we have enough to store two full rounds of operations on all channels.
// Otherwise we'd be unable to post half of them to free new elements.
#define MAX_OPS_PER_PEER (2*MAXCHANNELS*NCCL_MAX_WORK_ELEMENTS_P2P)
// Otherwise we'd be unable to post half of them to free new elements. Each
// p2p work contains a send and recv proxy op hence the 2x before it.
#define MAX_OPS_PER_PEER (2*MAXCHANNELS*2*NCCL_MAX_DEV_WORK_P2P_PER_BATCH)
struct ncclProxyOpsPool {
struct ncclProxyOp ops[MAX_OPS_PER_PEER*NCCL_MAX_LOCAL_RANKS];
@@ -243,7 +260,7 @@ struct ncclProxyState {
bool dmaBufSupport;
ncclNet_t* ncclNet;
ncclCollNet_t* ncclCollNet;
volatile uint32_t* abortFlag;
uint32_t* abortFlag;
// Service threads
pthread_t thread;
pthread_t threadUDS;
@@ -301,7 +318,6 @@ enum proxyMode {
};
ncclResult_t ncclProxySaveOp(struct ncclComm* comm, struct ncclProxyOp* proxyOp, bool *justInquire);
ncclResult_t ncclProxyComputeP2p(struct ncclInfo* info, struct ncclProxyOp* proxyOp, int reg);
ncclResult_t ncclProxyStart(struct ncclComm* comm);
ncclResult_t ncclProxyInit(struct ncclComm* comm, struct ncclSocket* sock, union ncclSocketAddress* peerAddresses, uint64_t *peerAddressesUDS);
ncclResult_t ncclProxyCreate(struct ncclComm* comm);
src/include/register.h
+5
Féach ar an gComhad
@@ -1,6 +1,11 @@
#ifndef NCCL_REGISTER_H_
#define NCCL_REGISTER_H_
#include "device.h"
#include <cuda.h>
#include <stdint.h>
enum {
NET_REG_COMPLETE = 0x01,
NVLS_REG_COMPLETE = 0x02,
src/include/transport.h
+27 -9
Féach ar an gComhad
@@ -13,12 +13,14 @@
#include "core.h"
#define NTRANSPORTS 4
#define TRANSPORT_UNDEFINED -1
#define TRANSPORT_P2P 0
#define TRANSPORT_SHM 1
#define TRANSPORT_NET 2
#define TRANSPORT_COLLNET 3
#include "proxy.h"
#include "comm.h"
extern struct ncclTransport p2pTransport;
extern struct ncclTransport shmTransport;
@@ -45,6 +47,7 @@ struct ncclPeerInfo {
int cudaCompCap;
// MNNVL support
nvmlGpuFabricInfoV_t fabricInfo;
int cuMemSupport;
};
#define CONNECT_SIZE 128
@@ -57,17 +60,21 @@ struct ncclConnect {
#define NVLS_HANDLE_SIZE 64
struct ncclNvlsSharedRes {
int refCount;
CUmulticastObjectProp properties;
bool inited;
CUmulticastObjectProp bufProp;
CUmulticastObjectProp signalProp;
CUmemAccessDesc accessDesc;
int dev;
size_t size;
size_t granularity;
CUmemGenericAllocationHandle mcHandle; // Multicast handle for NVLS buffer
size_t buffSize;
size_t creditSize;
CUmemGenericAllocationHandle mcBuffHandle; // Multicast handle for NVLS buffer
CUmemGenericAllocationHandle mcCreditHandle; // Multicast handle for NVLS credit buffer
char* mcBuff; // Multicast NVLS buffer address
CUmemGenericAllocationHandle ucHandle; // Unicast Handle for NVLS buffer
char* mcCredit; // Multicast NVLS credit address
CUmemGenericAllocationHandle ucBuffHandle; // Unicast Handle for NVLS buffer
CUmemGenericAllocationHandle ucCreditHandle; // Unicast Handle for NVLS credit buffer
char* ucBuff; // Unicast NVLS buffer address
char shareableHandle[NVLS_HANDLE_SIZE];
size_t ucGran;
char* ucCredit; // Unicast NVLS credit address
int nChannels;
struct ncclShmemCollBuff nvlsShmem;
void *nvlsShmemHandle;
@@ -84,6 +91,7 @@ struct ncclCollNetSharedRes {
void* resources;
int nChannels;
size_t buffSize;
int intraHighestTransportType;
};
struct ncclTransportComm {
@@ -111,7 +119,9 @@ ncclResult_t ncclTransportP2pSetup(struct ncclComm* comm, struct ncclTopoGraph*
ncclResult_t ncclNvlsInit(struct ncclComm* comm);
ncclResult_t ncclNvlsSetup(struct ncclComm* comm, struct ncclComm* parent);
ncclResult_t ncclNvlsGraphRegisterBuffer(struct ncclComm *comm, struct ncclKernelPlan *plan, const void *sendbuff, void *recvbuff, size_t sendbuffSize, size_t recvbuffSize, bool *outRegBufUsed, void **outRegBufSend, void **outRegBufRecv);
ncclResult_t ncclNvlsBufferSetup(struct ncclComm* comm);
ncclResult_t ncclNvlsTreeConnect(struct ncclComm* comm);
ncclResult_t ncclNvlsGraphRegisterBuffer(struct ncclComm *comm, const void *sendbuff, void *recvbuff, size_t sendbuffSize, size_t recvbuffSize, bool *outRegBufUsed, void **outRegBufSend, void **outRegBufRecv, struct ncclIntruQueue<struct ncclCommCallback, &ncclCommCallback::next>* cleanupQueue, int* nCleanupQueueElts);
ncclResult_t ncclNvlsLocalRegisterBuffer(struct ncclComm *comm, const void *sendbuff, void *recvbuff, size_t sendbuffSize, size_t recvbuffSize, bool *outRegBufUsed, void **outRegBufSend, void **outRegBufRecv);
ncclResult_t ncclNvlsDeregBuffer(CUmemGenericAllocationHandle *mcHandler, CUdeviceptr ptr, int dev, size_t size);
ncclResult_t ncclNvlsFree(struct ncclComm* comm);
@@ -121,6 +131,14 @@ int ncclTransportCollNetSetup(struct ncclComm* comm, struct ncclTopoGraph* collN
ncclResult_t ncclTransportCollNetCheck(struct ncclComm* comm, int collNetSetupFail);
ncclResult_t ncclTransportCollNetFree(struct ncclComm* comm);
ncclResult_t ncclCollnetLocalRegisterBuffer(struct ncclComm* comm, const void* userbuff, size_t buffSize, int type, int* outRegBufUsed, void** outHandle);
ncclResult_t ncclCollnetGraphRegisterBuffer(struct ncclComm* comm, struct ncclKernelPlan *plan, const void* userbuff, size_t buffSize, int type, int* outRegBufFlag, void** outHandle);
ncclResult_t ncclCollnetGraphRegisterBuffer(struct ncclComm* comm, const void* userbuff, size_t buffSize, int type, int* outRegBufFlag, void** outHandle, struct ncclIntruQueue<struct ncclCommCallback, &ncclCommCallback::next>* cleanupQueue, int* nCleanupQueueElts);
ncclResult_t ncclCollnetDeregBuffer(struct ncclComm* comm, struct ncclProxyConnector* proxyconn, void* handle);
ncclResult_t ncclTransportRingConnect(struct ncclComm* comm);
ncclResult_t ncclTransportTreeConnect(struct ncclComm* comm);
ncclResult_t ncclCollNetSetup(ncclComm_t comm, ncclComm_t parent, struct ncclTopoGraph* graphs[]);
ncclResult_t ncclCollNetChainBufferSetup(ncclComm_t comm);
ncclResult_t ncclCollNetDirectBufferSetup(ncclComm_t comm);
#endif
src/include/tuner.h
+3 -2
Féach ar an gComhad
@@ -9,14 +9,15 @@
#define NCCL_INT_TUNER_H_
#include "nccl_tuner.h"
#include "comm.h"
// Tuning plugin to override NCCL's default algorithm/protocol tuning.
// Attempts to load NCCL tuner from environmental variable.
// Returns ncclSuccess if the correct tuner symbol has been found and
// successully loaded. Otherwise returns an error and also logs the error.
ncclResult_t ncclTunerPluginLoad(ncclTuner_t** tuner);
ncclResult_t ncclTunerPluginLoad(struct ncclComm* comm);
// Cleans up NCCL tuner plugin.
ncclResult_t ncclTunerPluginUnload(ncclTuner_t** tuner);
ncclResult_t ncclTunerPluginUnload(struct ncclComm* comm);
#endif
src/include/utils.h
+39 -48
Féach ar an gComhad
@@ -9,12 +9,14 @@
#include "nccl.h"
#include "alloc.h"
#include "bitops.h"
#include "checks.h"
#include <stdint.h>
#include <time.h>
#include <sched.h>
#include <algorithm>
#include <new>
#include <type_traits>
int ncclCudaCompCap();
@@ -30,11 +32,6 @@ uint64_t getHostHash();
uint64_t getPidHash();
ncclResult_t getRandomData(void* buffer, size_t bytes);
const char* ncclOpToString(ncclRedOp_t op);
const char* ncclDatatypeToString(ncclDataType_t type);
const char* ncclAlgoToString(int algo);
const char* ncclProtoToString(int proto);
struct netIf {
char prefix[64];
int port;
@@ -44,9 +41,7 @@ int parseStringList(const char* string, struct netIf* ifList, int maxList);
bool matchIfList(const char* string, int port, struct netIf* ifList, int listSize, bool matchExact);
static long log2i(long n) {
long l = 0;
while (n>>=1) l++;
return l;
return log2Down(n);
}
inline uint64_t clockNano() {
@@ -96,8 +91,11 @@ void ncclMemoryStackConstruct(struct ncclMemoryStack* me);
void ncclMemoryStackDestruct(struct ncclMemoryStack* me);
void ncclMemoryStackPush(struct ncclMemoryStack* me);
void ncclMemoryStackPop(struct ncclMemoryStack* me);
void* ncclMemoryStackAlloc(struct ncclMemoryStack* me, size_t size, size_t align);
template<typename T>
T* ncclMemoryStackAlloc(struct ncclMemoryStack* me, size_t n=1);
template<typename Header, typename Element>
inline Header* ncclMemoryStackAllocInlineArray(struct ncclMemoryStack* me, size_t nElt);
////////////////////////////////////////////////////////////////////////////////
/* ncclMemoryPool: A free-list of same-sized allocations. It is an invalid for
@@ -140,11 +138,14 @@ T* ncclIntruQueueHead(ncclIntruQueue<T,next> *me);
template<typename T, T *T::*next>
void ncclIntruQueueEnqueue(ncclIntruQueue<T,next> *me, T *x);
template<typename T, T *T::*next>
void ncclIntruQueueEnqueueFront(ncclIntruQueue<T,next> *me, T *x);
template<typename T, T *T::*next>
T* ncclIntruQueueDequeue(ncclIntruQueue<T,next> *me);
template<typename T, T *T::*next>
T* ncclIntruQueueTryDequeue(ncclIntruQueue<T,next> *me);
template<typename T, T *T::*next>
void ncclIntruQueueFreeAll(ncclIntruQueue<T,next> *me, ncclMemoryPool *memPool);
void ncclIntruQueueTransfer(ncclIntruQueue<T,next> *dst, ncclIntruQueue<T,next> *src);
////////////////////////////////////////////////////////////////////////////////
/* ncclThreadSignal: Couples a pthread mutex and cond together. The "mutex"
@@ -233,6 +234,12 @@ inline void* ncclMemoryStack::allocate(struct ncclMemoryStack* me, size_t size,
return obj;
}
inline void* ncclMemoryStackAlloc(struct ncclMemoryStack* me, size_t size, size_t align) {
void *obj = ncclMemoryStack::allocate(me, size, align);
memset(obj, 0, size);
return obj;
}
template<typename T>
inline T* ncclMemoryStackAlloc(struct ncclMemoryStack* me, size_t n) {
void *obj = ncclMemoryStack::allocate(me, n*sizeof(T), alignof(T));
@@ -240,6 +247,17 @@ inline T* ncclMemoryStackAlloc(struct ncclMemoryStack* me, size_t n) {
return (T*)obj;
}
template<typename Header, typename Element>
inline Header* ncclMemoryStackAllocInlineArray(struct ncclMemoryStack* me, size_t nElt) {
size_t size = sizeof(Header);
size = (size + alignof(Element)-1) & -alignof(Element);
size += nElt*sizeof(Element);
size_t align = alignof(Header) < alignof(Element) ? alignof(Element) : alignof(Header);
void *obj = ncclMemoryStack::allocate(me, size, align);
memset(obj, 0, size);
return (Header*)obj;
}
inline void ncclMemoryStackPush(struct ncclMemoryStack* me) {
using Frame = ncclMemoryStack::Frame;
Frame tmp = me->topFrame;
@@ -343,6 +361,13 @@ inline void ncclIntruQueueEnqueue(ncclIntruQueue<T,next> *me, T *x) {
me->tail = x;
}
template<typename T, T *T::*next>
inline void ncclIntruQueueEnqueueFront(ncclIntruQueue<T,next> *me, T *x) {
if (me->head == nullptr) me->tail = x;
x->*next = me->head;
me->head = x;
}
template<typename T, T *T::*next>
inline T* ncclIntruQueueDequeue(ncclIntruQueue<T,next> *me) {
T *ans = me->head;
@@ -388,45 +413,11 @@ inline T* ncclIntruQueueTryDequeue(ncclIntruQueue<T,next> *me) {
}
template<typename T, T *T::*next>
void ncclIntruQueueFreeAll(ncclIntruQueue<T,next> *me, ncclMemoryPool *pool) {
T *head = me->head;
me->head = nullptr;
me->tail = nullptr;
while (head != nullptr) {
T *tmp = head->*next;
ncclMemoryPoolFree(pool, tmp);
head = tmp;
}
}
/* cmp function determines the sequence of objects in the queue. If cmp returns value >= 0, it means a > b,
* and we should put a before b; otherwise, b should be put ahead of a. */
template<typename T, T *T::*next>
inline void ncclIntruQueueSortEnqueue(ncclIntruQueue<T,next> *me, T *x, int (*cmp)(T *a, T *b)) {
T *cur = me->head;
T *prev = NULL;
if (cur == NULL) {
x->*next = nullptr;
me->tail = me->head = x;
} else {
while (cur) {
if (cmp(cur, x) > 0) {
prev = cur;
cur = cur->next;
} else {
break;
}
}
x->*next = cur;
if (prev) {
prev->*next = x;
if (cur == NULL) me->tail = x;
} else {
me->head = x;
}
}
void ncclIntruQueueTransfer(ncclIntruQueue<T,next> *dst, ncclIntruQueue<T,next> *src) {
(dst->tail ? dst->tail->next : dst->head) = src->head;
if (src->tail) dst->tail = src->tail;
src->head = nullptr;
src->tail = nullptr;
}
////////////////////////////////////////////////////////////////////////////////