rocm-systems/source/docs/api-reference/callback_services.md

# Callback tracing services

## Overview

Callback tracing services provide immediate callbacks to a tool on the current CPU thread when a given event occurs.
For example, when tracing an API function, e.g. `hipSetDevice`, callback tracing invokes a user-specified callback
before and after the traced function executes on the thread which is invoking the API function.

## Subscribing to Callback Tracing Services

During tool initialization, tools configure callback tracing via the `rocprofiler_configure_callback_tracing_service`
function:

```cpp
rocprofiler_status_t
rocprofiler_configure_callback_tracing_service(rocprofiler_context_id_t            context_id,
                                               rocprofiler_callback_tracing_kind_t kind,
                                               rocprofiler_tracing_operation_t*    operations,
                                               size_t                              operations_count,
                                               rocprofiler_callback_tracing_cb_t   callback,
                                               void*                               callback_args);
```

The `kind` parameter is a high-level specifier of which service to trace (also known as a "domain").
Domain examples include, but are not limited to, the HIP API, the HSA API, and kernel dispatches.
For each domain, there are (often) various "operations", which can be used to restrict the callbacks
to a subset within the domain. For domains which correspond to APIs, the "operations" are the functions
which compose the API. If all operations in a domain should be traced, the `operations` and `operations_count`
parameters can be set to `nullptr` and `0`, respectively. If the tracing domain should be restricted to a subset
of operations, the tool library should specify a C-array of type `rocprofiler_tracing_operation_t` and the
size of the array for the `operations` and `operations_count` parameter.

`rocprofiler_configure_callback_tracing_service` will return an error if a callback service for given context
and given domain is configured more than once. For example, if one only wanted to trace two functions within
the HIP runtime API, `hipGetDevice` and `hipSetDevice`, the following code would accomplish this objective:

```cpp
{
    auto ctx = rocprofiler_context_id_t{};
    // ... creation of context, etc. ...

    // array of operations (i.e. API functions)
    auto operations = std::array<rocprofiler_tracing_operation_t, 2>{
        ROCPROFILER_HIP_RUNTIME_API_ID_hipSetDevice,
        ROCPROFILER_HIP_RUNTIME_API_ID_hipGetDevice
    };

    rocprofiler_configure_callback_tracing_service(ctx,
                                                   ROCPROFILER_CALLBACK_TRACING_HIP_RUNTIME_API,
                                                   operations.data(),
                                                   operations.size(),
                                                   callback_func,
                                                   nullptr);
    // ... etc. ...
}
```

But the following code would be invalid:

```cpp
{
    auto ctx = rocprofiler_context_id_t{};
    // ... creation of context, etc. ...

    // array of operations (i.e. API functions)
    auto operations = std::array<rocprofiler_tracing_operation_t, 2>{
        ROCPROFILER_HIP_RUNTIME_API_ID_hipSetDevice,
        ROCPROFILER_HIP_RUNTIME_API_ID_hipGetDevice
    };

    for(auto op : operations)
    {
        // after the first iteration, will return ROCPROFILER_STATUS_ERROR_SERVICE_ALREADY_CONFIGURED
        rocprofiler_configure_callback_tracing_service(ctx,
                                                       ROCPROFILER_CALLBACK_TRACING_HIP_RUNTIME_API,
                                                       &op,
                                                       1,
                                                       callback_func,
                                                       nullptr);
    }

    // ... etc. ...
}
```

## Callback Tracing Callback Function

Rocprofiler-sdk callback tracing callback functions have the signature:

```cpp
typedef void (*rocprofiler_callback_tracing_cb_t)(rocprofiler_callback_tracing_record_t record,
                                                  rocprofiler_user_data_t*              user_data,
                                                  void* callback_data)
```

The `record` parameter contains the information to uniquely identify a tracing record type and has the
following definition:

```cpp
typedef struct rocprofiler_callback_tracing_record_t
{
    rocprofiler_context_id_t            context_id;
    rocprofiler_thread_id_t             thread_id;
    rocprofiler_correlation_id_t        correlation_id;
    rocprofiler_callback_tracing_kind_t kind;
    uint32_t                            operation;
    rocprofiler_callback_phase_t        phase;
    void*                               payload;
} rocprofiler_callback_tracing_record_t;
```

The underlying type of `payload` field above is typically unique to a domain and, less frequently, an operation.
For example, for the `ROCPROFILER_CALLBACK_TRACING_HIP_RUNTIME_API` and `ROCPROFILER_CALLBACK_TRACING_HIP_COMPILER_API`,
the payload should be casted to `rocprofiler_callback_tracing_hip_api_data_t*` -- which will contain the arguments
to the function and (in the exit phase) the return value of the function. The payload field will only be a valid
pointer during the invocation of the callback function(s).

The `user_data` parameter can be used to store data in between callback phases. It is a unique for every
instance of an operation. For example, if the tool library wishes to store the timestamp of the
`ROCPROFILER_CALLBACK_PHASE_ENTER` phase for the ensuing `ROCPROFILER_CALLBACK_PHASE_EXIT` callback,
this data can be stored in a method similar to below:

```cpp
void
callback_func(rocprofiler_callback_tracing_record_t record,
              rocprofiler_user_data_t*              user_data,
              void*                                 cb_data)
{
    auto ts = rocprofiler_timestamp_t{};
    rocprofiler_get_timestamp(&ts);

    if(record.phase == ROCPROFILER_CALLBACK_PHASE_ENTER)
    {
        user_data->value = ts;
    }
    else if(record.phase == ROCPROFILER_CALLBACK_PHASE_EXIT)
    {
        auto delta_ts = (ts - user_data->value);
        // ... etc. ...
    }
    else
    {
        // ... etc. ...
    }
}
```

The `callback_data` argument will be the value of `callback_args` passed to `rocprofiler_configure_callback_tracing_service`
in [the previous section](#subscribing-to-callback-tracing-services).

## Callback Tracing Record

The name of a tracing kind can be obtained via the `rocprofiler_query_callback_tracing_kind_name` function.
The name of an operation specific to a tracing kind can be obtained via the `rocprofiler_query_callback_tracing_kind_operation_name`
function. One can also iterate over all the callback tracing kinds and operations for each tracing kind via the
`rocprofiler_iterate_callback_tracing_kinds` and `rocprofiler_iterate_callback_tracing_kind_operations` functions.
Lastly, for a given `rocprofiler_callback_tracing_record_t` object, rocprofiler-sdk supports generically iterating over
the arguments of the payload field for many domains.

As mentioned above, within the `rocprofiler_callback_tracing_record_t` object,
an opaque `void* payload` is provided for accessing domain specific information.
The data types generally follow the naming convention of `rocprofiler_callback_tracing_<DOMAIN>_data_t`,
e.g., for the tracing kinds `ROCPROFILER_BUFFER_TRACING_HSA_{CORE,AMD_EXT,IMAGE_EXT,FINALIZE_EXT}_API`,
the payload should be casted to `rocprofiler_callback_tracing_hsa_api_data_t*`:

```cpp
void
callback_func(rocprofiler_callback_tracing_record_t record,
              rocprofiler_user_data_t*              user_data,
              void*                                 cb_data)
{
    static auto hsa_domains = std::unordered_set<rocprofiler_buffer_tracing_kind_t>{
        ROCPROFILER_BUFFER_TRACING_HSA_CORE_API,
        ROCPROFILER_BUFFER_TRACING_HSA_AMD_EXT_API,
        ROCPROFILER_BUFFER_TRACING_HSA_IMAGE_EXT_API,
        ROCPROFILER_BUFFER_TRACING_HSA_FINALIZER_API};

    if(hsa_domains.count(record.kind) > 0)
    {
        auto* payload = static_cast<rocprofiler_callback_tracing_hsa_api_data_t*>(record.payload);

        hsa_status_t status = payload->retval.hsa_status_t_retval;
        if(record.phase == ROCPROFILER_CALLBACK_PHASE_EXIT && status != HSA_STATUS_SUCCESS)
        {
            const char* _kind = nullptr;
            const char* _operation = nullptr;

            rocprofiler_query_callback_tracing_kind_name(record.kind, &_kind, nullptr);
            rocprofiler_query_callback_tracing_kind_operation_name(
                record.kind, record.operation, &_operation, nullptr);

            // message that
            fprintf(stderr, "[domain=%s] %s returned a non-zero exit code: %i\n", _kind, _operation, status);
        }
    }
    else if(record.phase == ROCPROFILER_CALLBACK_PHASE_EXIT)
    {
        auto delta_ts = (ts - user_data->value);
        // ... etc. ...
    }
    else
    {
        // ... etc. ...
    }
}
```

### Sample `rocprofiler_iterate_callback_tracing_kind_operation_args`

```cpp
int
print_args(rocprofiler_callback_tracing_kind_t domain_idx,
           uint32_t                            op_idx,
           uint32_t                            arg_num,
           const void* const                   arg_value_addr,
           int32_t                             arg_indirection_count,
           const char*                         arg_type,
           const char*                         arg_name,
           const char*                         arg_value_str,
           int32_t                             arg_dereference_count,
           void*                               data)
{
    if(arg_num == 0)
    {
        const char* _kind      = nullptr;
        const char* _operation = nullptr;

        rocprofiler_query_callback_tracing_kind_name(domain_idx, &_kind, nullptr);
        rocprofiler_query_callback_tracing_kind_operation_name(
            domain_idx, op_idx, &_operation, nullptr);

        fprintf(stderr, "\n[%s] %s\n", _kind, _operation);
    }

    char* _arg_type = abi::__cxa_demangle(arg_type, nullptr, nullptr, nullptr);

    fprintf(stderr, "    %u: %-18s %-16s = %s\n", arg_num, _arg_type, arg_name, arg_value_str);

    free(_arg_type);

    // unused in example
    (void) arg_value_addr;
    (void) arg_indirection_count;
    (void) arg_dereference_count;
    (void) data;

    return 0;
}

void
callback_func(rocprofiler_callback_tracing_record_t record,
              rocprofiler_user_data_t*              user_data,
              void*                                 cb_data)
{
    if(record.phase == ROCPROFILER_CALLBACK_PHASE_EXIT &&
       record.kind == ROCPROFILER_CALLBACK_TRACING_HIP_RUNTIME_API &&
       (record.operation == ROCPROFILER_HIP_RUNTIME_API_ID_hipLaunchKernel ||
        record.operation == ROCPROFILER_HIP_RUNTIME_API_ID_hipMemcpyAsync))
    {
        rocprofiler_iterate_callback_tracing_kind_operation_args(
                             record, print_args, record.phase, nullptr));
    }
}
```

Sample Output:

```console

[HIP_RUNTIME_API] hipLaunchKernel
    0: void const*        function_address = 0x219308
    1: rocprofiler_dim3_t numBlocks        = {z=1, y=310, x=310}
    2: rocprofiler_dim3_t dimBlocks        = {z=1, y=32, x=32}
    3: void**             args             = 0x7ffe6d8dd3c0
    4: unsigned long      sharedMemBytes   = 0
    5: hipStream_t*      stream           = 0x17b40c0

[HIP_RUNTIME_API] hipMemcpyAsync
    0: void*              dst              = 0x7f06c7bbb010
    1: void const*        src              = 0x7f0698800000
    2: unsigned long      sizeBytes        = 393625600
    3: hipMemcpyKind      kind             = DeviceToHost
    4: hipStream_t*      stream           = 0x25dfcf0
```

## Code Object Tracing

The code object tracing service is a critical component for obtaining information regarding
asynchronous activity on the GPU. The `rocprofiler_callback_tracing_code_object_load_data_t`
payload (kind=`ROCPROFILER_CALLBACK_TRACING_CODE_OBJECT`, operation=`ROCPROFILER_CODE_OBJECT_LOAD`)
provides a unique identifier for a bundle of one or more GPU kernel symbols which have been loaded
for a specific GPU agent. For example, if your application is leveraging a multi-GPU system system
containing 4 Vega20 GPUs and 4 MI100 GPUs, there will at least 8 code objects loaded: one code
object for each GPU. Each code object will be associated with a set of kernel symbols:
the `rocprofiler_callback_tracing_code_object_kernel_symbol_register_data_t` payload
(kind=`ROCPROFILER_CALLBACK_TRACING_CODE_OBJECT`, operation=`ROCPROFILER_CODE_OBJECT_DEVICE_KERNEL_SYMBOL_REGISTER`)
provides a globally unique identifier for the specific kernel symbol along with the kernel name and
several other static properties of the kernel (e.g. scratch size, scalar general purpose register count, etc.).
Note: two otherwise identical kernel symbols (same kernel name, scratch size, etc.) which are part of
otherwise identical code objects but the code objects are loaded for different GPU agents ***will*** have unique
kernel identifiers. Furthermore, if the same code object (and it's kernel symbols) are unloaded and then
re-loaded, that code object and all of it's kernel symbols ***will*** be given new unique identifiers.

In general, when a code object is loaded and unloaded, here is the sequence of events:

1. Callback: code object load
    - kind=`ROCPROFILER_CALLBACK_TRACING_CODE_OBJECT`
    - operation=`ROCPROFILER_CODE_OBJECT_LOAD`
    - phase=`ROCPROFILER_CALLBACK_PHASE_LOAD`
2. Callback: kernel symbol load
    - kind=`ROCPROFILER_CALLBACK_TRACING_CODE_OBJECT`
    - operation=`ROCPROFILER_CODE_OBJECT_DEVICE_KERNEL_SYMBOL_REGISTER`
    - phase=`ROCPROFILER_CALLBACK_PHASE_LOAD`
    - Repeats for each kernel symbol in code object
3. Application Execution
4. Callback: kernel symbol unload
    - kind=`ROCPROFILER_CALLBACK_TRACING_CODE_OBJECT`
    - operation=`ROCPROFILER_CODE_OBJECT_DEVICE_KERNEL_SYMBOL_REGISTER`
    - phase=`ROCPROFILER_CALLBACK_PHASE_UNLOAD`
    - Repeats for each kernel symbol in code object
5. Callback: code object unload
    - kind=`ROCPROFILER_CALLBACK_TRACING_CODE_OBJECT`
    - operation=`ROCPROFILER_CODE_OBJECT_LOAD`
    - phase=`ROCPROFILER_CALLBACK_PHASE_UNLOAD`

Note: rocprofiler-sdk does not provide an interface to query this information outside of the
code object tracing service. If you wish to be able to associate kernel names with kernel tracing records,
a tool is personally responsible for making a copy of the relevant information when the code objects and
kernel symbol are loaded (however, any constant string fields like the (`const char* kernel_name` field)
need not to be copied, these are guaranteed to be valid pointers until after rocprofiler-sdk finalization).
If a tool decides to delete their copy of the data associated with a given code object or kernel symbol
identifier when the code object and kernel symbols are unloaded, it is highly recommended to flush
any/all buffers which might contain references to that code object or kernel symbol identifiers before
deleting the associated data.

For a sample of code object tracing, please see the `samples/code_object_tracing` example in the
[rocprofiler-sdk GitHub repository](https://github.com/ROCm/rocprofiler-sdk).