AI/rocm-systems
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

[rocprofiler-sdk][rocprofiler-register] Initial Attachment Support (#316)

Browse Source 
* attach: milestone: API tracing

- This pairs with another commit in rocprofiler-sdk to fully
  function
- Add ptrace entry points for tool attachment
- API tracing works at this commit
- Queue tracing not supported yet

* attach: cleanup

- Remove hardcode for loading of tool library
- Make invoke registration functions public again

* attach: proxy queue first draft

- Adds ability to trace with queues during attachment
- Must be paired with updated rocprofiler-sdk

* attach: prestore overhaul

- Must be paired with commit in rocprofiler-sdk

* attach: add dispatch table rework

- Register will load the prestore library and provide entrypoints to sdk

* attach: formatting and cleanup

* attach: revise dispatch table scheme

* attach: formatting

* attach: milestone: API tracing

- This change must be paired with a change in rocprofiler-register to
  fully function.
- API tracing works at this commit
- Queue tracing not supported yet

* attach: cleanup and comments

* attach: Formatting and crash fixes

* attach: add attach duration

- Add option attach-duration-msec for attachment

* Formatting + sglang hang fix via signal handling

* Changed FATAL_IF to DFATAL_IF for scratch_memory due to persistent crash when iterating queues

* attach: proxy queue first draft

- Adds ability to trace with queues during attachment
- Must be paired with updated rocprofiler-register

* Allow null agents for scratch output

* attach: improve queue library interface

- Significant changes to force exported interfaces back to C
- Fixes bug with unknown agents at attachment
- Code objects' names may still be incorrect

* attach: add code_object support

- Kernel traces will now have names and all other information for launches
- Add capture of hsa_executable to the queue library
- Various logging improvements

* attach: rename queue library to prestore

* attach: prestore overhaul

- Must be paired with commit from rocprofiler-register
- Massive overhaul of code organization in prestore library
  - Separates registrations for different object types
  - Sets up future changes for initialization

* attach: add prestore dispatch table

- Removes linkage to prestore library from sdk

* attach: cleanup

* attach: formatting

* attach: fix input prompt not appearing

* attach: fix component name in cmake

* attach: revert change to export level

* Make prestore API public

* attach: update sdk attachment library WIP

- This commit is NONFUNCTIONAL

- Changes around structure to remove classes
- Seperate C linkage where needed
- Still needs updates to register for correct usage

* attach: update register with dispatch table WIP
- This commit is NONFUNCTIONAL

- Changes rocprofiler_register to handle dispatch table from attach
  library.
- Still needs changes in SDK with dispatch table usage

* attach: dispatch table wip
- This commit is NONFUNCTIONAL

* attach: move attach component into core

* attach: rename to rocprofv3-attach

* attach: add callbacks for new queues and code objects

* attach: finish dispatch table implementation

- Fixes kernel tracing

* attach: add cmake variable for attachment support

* feat: Add --attach alias for rocprofv3 with comprehensive attachment tests

- Add `--attach` as an alias to existing `-p/--pid` functionality in rocprofv3.py
- Create comprehensive attachment test suite with CSV and JSON output validation:
- New attachment-test application for testing dynamic profiling scenarios
- Unified test script supporting both CSV and JSON output formats
- Pytest-based validation for kernel traces, memory copies, HSA API calls, and agent info
- Add CMake integration for automated attachment testing
- Support parameterized output directory and filename specification
- Implement proper environment setup for attachment queue registration

Tests verify successful attachment to running processes and capture of:
- Kernel dispatch traces with workgroup/grid dimensions
- Memory copy operations (H2D/D2H) with size validation
- HSA API call traces across multiple domains
- GPU/CPU agent information and capabilities

* Documentation Update

* attach: make attach script callable

* Added ROCPROFILER_REGISTER_ATTACHMENT_TOOL_LIB to remove hardcoded name

* attach: revert metrics library path changes

* Generic Attachment in Register (#942)

Remove tool references in register

* Add second param to attach call in rocprof register

* Add experimental reattachment support for ROCprofiler-SDK

This commit introduces experimental reattachment functionality allowing tools
to dynamically reattach to running processes with comprehensive design changes
to support multiple attach/detach cycles:

**Core Reattachment API:**
- Add rocprofiler_tool_configure_result_experimental_t with tool_reattach/tool_detach callbacks
- Add rocprofiler_call_client_reattach and rocprofiler_call_client_detach C exports
- Implement reattachment tracking in rocprofiler_register_attach to differentiate
initial attachment from reattachment cycles
- Add rocprofiler_register_invoke_reattach for handling reattachment requests

**Design Changes - Registration System Flow:**
The registration system now supports a dual-path initialization:

1. Initial Attachment Flow:
    - rocprofiler_register_attach() -> rocprofiler_register_invoke_all_registrations()
    - Full tool initialization with complete context setup
    - Sets prev_attached atomic flag to track state

2. Reattachment Flow:
    - rocprofiler_register_attach() detects prev_attached=true -> rocprofiler_register_invoke_reattach()
    - Bypasses full re-initialization, calls client reattach callbacks instead
    - Preserves existing contexts and buffers, only reactivates profiling services

**Design Changes - Tool Library Loading:**
Enhanced rocprofiler-register library loading with function pointer resolution:
- Extended rocp_set_api_table_data_t tuple to include reattach/detach function pointers
- Automatic symbol resolution for rocprofiler_call_client_reattach/detach functions
- Support for both LD_PRELOAD and dlopen scenarios with consistent callback availability

**Design Changes - Context Management:**
Introduced dual context systems for attachment scenarios:
- get_contexts() - Original contexts for standard tool initialization
- get_attach_contexts() - Separate context map for attachment-specific lifecycle
- attach_init() - Creates contexts for ALL buffer tracing services using existing buffers
- attach_start() - Selectively starts contexts based on configuration options
- attach_detach() - Cleanly stops and destroys attachment contexts

**Design Changes - Buffer Management:**
Added reset_tmp_file_buffer() template for clean reattachment state:
- Properly closes and removes old temporary files
- Deletes existing file_buffer instances to prevent stale file position tracking
- Creates fresh file_buffer instances for clean reattachment cycles
- Addresses core issue where file position metadata becomes stale between cycles

**Design Changes - Environment Variable Injection:**
Added ROCP_REGISTERED_TOOL_ATTACH environment variable:
- Distinguishes attachment-loaded tools from LD_PRELOAD scenarios
- Enables registration system to apply attachment-specific logic
- Helps tools adapt behavior for attachment vs standard initialization

**Attachment Context Management:**
- Add attach_init/attach_start/attach_detach functions for dynamic context lifecycle
- Add reset_tmp_file_buffer template for clean reattachment state management
- Implement get_attach_contexts() for tracking active attachment contexts

**Test Infrastructure:**
- Add projects/rocprofiler-sdk/tests/rocprofv3/reattach/ comprehensive test suite
- Include reattachment test scripts with unified attachment/detachment cycles
- Add validate.py with trace data validation for kernel, memory copy, HSA API, and agent info
- Add conftest.py for JSON and CSV data loading utilities

**Configuration Updates:**
- Update CMakeLists.txt to include reattachment tests in build system
- Add environment variable ROCP_REGISTERED_TOOL_ATTACH for attachment state tracking
- Enhance rocprofiler-register library loading with reattach/detach function resolution

**Flow Impact Analysis:**
This design enables robust multi-cycle attachment by:
1. Preventing duplicate initialization on reattachment
2. Maintaining separate context lifecycles for attachment vs standard operation
3. Ensuring clean temporary file state between attachment cycles
4. Providing tools with explicit reattach/detach callback hooks
5. Supporting both programmatic and environment-based tool configuration

The experimental nature allows for iteration on the API while establishing
the foundation for production-ready dynamic profiling capabilities.

* Fix misc clang-tidy warnings/errors

* CMake Option and Environment Variable Updates

- CMake: ROCPROFILER_REGISTER_ALWAYS_SUPPORT_ATTACH -> ROCPROFILER_REGISTER_BUILD_DEFAULT_ATTACHMENT
- Env: ROCPROFILER_REGISTER_ATTACHMENT_ENABLED ->

* Source reorganization

* Formatting + new lines at EOF

* Fix flake8 F841: local variable is assigned to but never used

* Update attachment test

- get rid of 5 second start delay
- add roctx

* Rework implementation

- Remove rocprofiler_tool_configure_result_experimental_t in lieu of rocprofiler_configure_attach
- Add <rocprofiler-sdk/experimental/registration.h>
- TODO: Update process_attachment.rst

* Handle re-attachment options

- inherit options from previous attachment
- check previous options do not modify data collection services

* Fix support for tools w/o rocprofiler_configure_attach

- fix segfault when rocprofiler_configure_attach does not exist
- fix naming convention for functions accepting attach dispatch table
- cleanup rocprofiler_configure_attach implementation in rocprofv3 tool

* attach: remove unknown agent handling

- Change was from earlier commit, no longer needed

* attach: add error for attaching without library loaded

* attach: revise version numbering

* attach: register header revisions

* attach: clang format register

* attach: formatting

* attach: fix build failure

- Remove cross dependency into rocprofiler-sdk, fixes build on some systems

* attach: revise register library detection

* Update rocprofiler-register and attach library

- formatting
- proper signature of register_functor for rocprofiler-sdk-attach library callback
- remove get_dispatch_registration_table()

* Bump rocprofiler-register version to 0.6.0 + AnyNewerVersion

* Fix output support for rocprofiler-sdk-tool

* Fix formatting

* Fix clang tidy errors

* Misc rocprofiler-sdk-attach fixes

* attach: add sigint handling to attach python

* tool README.md formatting

Co-authored-by: Jonathan R. Madsen <jrmadsen@users.noreply.github.com>

* Fix buffered output issue

* attach: add errors for tool attach

* CI Fixes

* Rework tests

* attach: improve library loading in rocprofv3 attach

* formatting

* Update tests to use pytest framework

* Fix test_attachment_hsa_api_trace

* attach: catch ctypes exceptions

* attach: fix leak in registration

* attach: fix sanitizer tests

* attach: fix sanitizer tests further

* attach: disable attach asan tests

* attach: disable ubsan test

* attach: fix permissions in installed test package

* attach: formatting

---------

Co-authored-by: Ian Trowbridge <Ian.Trowbridge@amd.com>
Co-authored-by: Tim Gu <Tim.Gu@amd.com>
Co-authored-by: Claude Code <claude@anthropic.com>
Co-authored-by: Benjamin Welton <bwelton@amd.com>
Co-authored-by: Jonathan R. Madsen <jonathanrmadsen@gmail.com>
Co-authored-by: Jonathan R. Madsen <jrmadsen@users.noreply.github.com>
Co-authored-by: Benjamin Welton <bewelton@amd.com>
This commit is contained in:
Mark Meserve
2025-09-18 18:10:45 -05:00
committed by GitHub
parent f3d672d507
commit bf49039005
72 changed files with 6456 additions and 140 deletions
Download Patch File Download Diff File Expand all files Collapse all files
projects/rocprofiler-sdk/source/lib/rocprofiler-sdk-tool/tool.cpp
+139 -22
View File
@@ -66,6 +66,7 @@
#include <rocprofiler-sdk/defines.h>
#include <rocprofiler-sdk/dispatch_counting_service.h>
#include <rocprofiler-sdk/experimental/counters.h>
#include <rocprofiler-sdk/experimental/registration.h>
#include <rocprofiler-sdk/experimental/thread_trace.h>
#include <rocprofiler-sdk/external_correlation.h>
#include <rocprofiler-sdk/fwd.h>
@@ -85,6 +86,7 @@
#include <cassert>
#include <chrono>
#include <csignal>
#include <cstdint>
#include <cstdlib>
#include <cstring>
#include <fstream>
@@ -114,6 +116,7 @@ __gcov_dump(void);
namespace common = ::rocprofiler::common;
namespace tool = ::rocprofiler::tool;
namespace fs = ::rocprofiler::common::filesystem;
extern "C" {
void
@@ -1767,6 +1770,47 @@ get_tracing_callbacks()
return tracing_callbacks_t{use_real_callbacks};
}
int
tool_attach(rocprofiler_client_detach_t /*detach_func*/,
rocprofiler_context_id_t* context_ids,
uint64_t context_ids_length,
void* /*tool_data*/)
{
// save the existing config for comparison
auto original_config = tool::get_config();
// reset config for attach (i.e. re-parse environment variables)
tool::get_config() = tool::config{};
// ensure the config has not changed which services were requested.
// NOTE: this is a temporary restriction
ROCP_FATAL_IF(!tool::is_attach_invariant(tool::get_config(), original_config))
<< "configuration mismatch between initial tool load and attach. rocprofv3 does not "
"support changing the set of enabled tracing services between initial load and attach. "
"After the initial attachment, it is recommended to just use `rocprofv3 --pid=<pid> [-o "
"<output_file> -d <output_directory> ...]` to attach to a new process.";
pid_t target_pid = getppid(); // The target process we're attaching to
pid_t tool_pid = getpid(); // The rocprofv3 tool process
ROCP_INFO << "Attach mode: Setting process_id to target PID " << target_pid
<< " (tool PID: " << tool_pid << ")";
tool_metadata->set_process_id(target_pid, 0); // Set target as main process
for(uint64_t i = 0; i < context_ids_length; ++i)
{
if(int status = 0;
rocprofiler_context_is_active(context_ids[i], &status) == ROCPROFILER_STATUS_SUCCESS &&
status == 0)
{
ROCP_INFO << "Attach mode: starting context ID " << context_ids[i].handle;
ROCPROFILER_CALL(rocprofiler_start_context(context_ids[i]),
"failed to start received context");
}
}
return 0;
}
int
tool_init(rocprofiler_client_finalize_t fini_func, void* tool_data)
{
@@ -2224,6 +2268,7 @@ tool_init(rocprofiler_client_finalize_t fini_func, void* tool_data)
}
tool_metadata->set_process_id(getpid(), getppid());
// set_process_id should set process_start_ns unless it cannot read from /proc/<pid>/stat
if(tool_metadata->process_start_ns == 0)
rocprofiler_get_timestamp(&(tool_metadata->process_start_ns));
@@ -2264,10 +2309,16 @@ api_timestamps_callback(rocprofiler_intercept_table_t table_id,
});
}
enum class cleanup_mode
{
destroy,
reset,
};
using stats_data_t = tool::stats_data_t;
using stats_entry_t = tool::stats_entry_t;
using domain_stats_vec_t = tool::domain_stats_vec_t;
using cleanup_vec_t = std::vector<std::function<void()>>;
using cleanup_vec_t = std::vector<std::function<void(cleanup_mode)>>;
struct output_data
{
@@ -2366,7 +2417,26 @@ generate_output(tool::buffered_output<Tp, DomainT>& output_v,
domain_stats_vec_t& contributions_v,
cleanup_vec_t& cleanups_v)
{
cleanups_v.emplace_back([&output_v]() { output_v.destroy(); });
cleanups_v.emplace_back([&output_v](cleanup_mode _mode) {
switch(_mode)
{
case cleanup_mode::destroy:
{
// ROCP_INFO << fmt::format("destroying buffer for {}",
// get_domain_column_name(DomainT));
output_v.destroy();
return;
}
case cleanup_mode::reset:
{
// ROCP_INFO << fmt::format("resetting buffer for {}",
// get_domain_column_name(DomainT));
output_v.reset();
return;
}
}
ROCP_CI_LOG(WARNING) << fmt::format("invalid cleanup mode {}", static_cast<int>(_mode));
});
if(!output_v) return;
@@ -2402,23 +2472,9 @@ generate_output(tool::buffered_output<Tp, DomainT>& output_v,
}
void
tool_fini(void* /*tool_data*/)
generate_output(cleanup_mode _cleanup_mode)
{
static bool _first = true;
if(!_first) return;
_first = false;
client_identifier = nullptr;
client_finalizer = nullptr;
auto _fini_timer = common::simple_timer{"[rocprofv3] tool finalization"};
if(tool_metadata->process_end_ns == 0)
rocprofiler_get_timestamp(&(tool_metadata->process_end_ns));
flush();
rocprofiler_stop_context(get_client_ctx());
flush();
auto _output_gen_timer = common::simple_timer{"[rocprofv3] output generation"};
auto kernel_dispatch_output =
rocprofiler::tool::kernel_dispatch_buffered_output_ext_t{tool::get_config().kernel_trace};
@@ -2457,10 +2513,10 @@ tool_fini(void* /*tool_data*/)
auto contributions = domain_stats_vec_t{};
auto cleanups = cleanup_vec_t{};
auto run_cleanup = [&cleanups]() {
auto run_cleanup = [&cleanups, _cleanup_mode]() {
for(const auto& itr : cleanups)
{
if(itr) itr();
if(itr) itr(_cleanup_mode);
}
cleanups.clear();
};
@@ -2645,6 +2701,43 @@ tool_fini(void* /*tool_data*/)
}
run_cleanup();
}
void
tool_detach(void* /*tool_data*/)
{
auto _detach_timer = common::simple_timer{"[rocprofv3] tool detachment"};
// Flush all buffers (same as tool_fini)
flush();
// Set process end timestamp for this detachment cycle
if(tool_metadata->process_end_ns == 0)
rocprofiler_get_timestamp(&(tool_metadata->process_end_ns));
generate_output(cleanup_mode::reset);
}
void
tool_fini(void* /*tool_data*/)
{
static bool _first = true;
if(!_first) return;
_first = false;
client_identifier = nullptr;
client_finalizer = nullptr;
auto _fini_timer = common::simple_timer{"[rocprofv3] tool finalization"};
if(tool_metadata->process_end_ns == 0)
rocprofiler_get_timestamp(&(tool_metadata->process_end_ns));
flush();
rocprofiler_stop_context(get_client_ctx());
flush();
generate_output(cleanup_mode::destroy);
if(destructors)
{
@@ -2654,6 +2747,14 @@ tool_fini(void* /*tool_data*/)
destructors = nullptr;
}
// remove the attach arguments file if it exists
if(auto attach_args_fname = fmt::format("/tmp/rocprofv3_attach_{}.pkl", getpid());
fs::exists(attach_args_fname))
{
ROCP_INFO << "removing attach arguments file: " << attach_args_fname;
fs::remove(attach_args_fname);
}
#if defined(CODECOV) && CODECOV > 0
__gcov_dump();
#endif
@@ -3072,13 +3173,29 @@ rocprofiler_configure(uint32_t version,
ROCP_INFO << id->name << " is using rocprofiler-sdk v" << major << "." << minor << "." << patch
<< " (" << runtime_version << ")";
// create configure data
// create configure data using experimental struct with attach/detach support
static auto cfg = rocprofiler_tool_configure_result_t{
sizeof(rocprofiler_tool_configure_result_t), &tool_init, &tool_fini, nullptr};
// return pointer to configure data
return &cfg;
// data passed around all the callbacks
}
rocprofiler_tool_configure_attach_result_t*
rocprofiler_configure_attach(uint32_t /*version*/,
const char* /*runtime_version*/,
uint32_t /*priority*/,
rocprofiler_client_id_t* /*id*/)
{
// This function is called right after rocprofiler_configure with the same parameters.
// The data returned is only used when attaching to a running process.
// create configure data using experimental struct with attach/detach support
static auto cfg = rocprofiler_tool_configure_attach_result_t{
sizeof(rocprofiler_tool_configure_attach_result_t), &tool_attach, &tool_detach, nullptr};
// return pointer to configure data
return &cfg;
}
void