Files
rocm-systems/projects/rocprofiler-sdk/source/lib/rocprofv3-attach/rocprofv3_attach.cpp
T
Mark Meserve bf49039005 [rocprofiler-sdk][rocprofiler-register] Initial Attachment Support (#316)
* attach: milestone: API tracing

- This pairs with another commit in rocprofiler-sdk to fully
  function
- Add ptrace entry points for tool attachment
- API tracing works at this commit
- Queue tracing not supported yet

* attach: cleanup

- Remove hardcode for loading of tool library
- Make invoke registration functions public again

* attach: proxy queue first draft

- Adds ability to trace with queues during attachment
- Must be paired with updated rocprofiler-sdk

* attach: prestore overhaul

- Must be paired with commit in rocprofiler-sdk

* attach: add dispatch table rework

- Register will load the prestore library and provide entrypoints to sdk

* attach: formatting and cleanup

* attach: revise dispatch table scheme

* attach: formatting

* attach: milestone: API tracing

- This change must be paired with a change in rocprofiler-register to
  fully function.
- API tracing works at this commit
- Queue tracing not supported yet

* attach: cleanup and comments

* attach: Formatting and crash fixes

* attach: add attach duration

- Add option attach-duration-msec for attachment

* Formatting + sglang hang fix via signal handling

* Changed FATAL_IF to DFATAL_IF for scratch_memory due to persistent crash when iterating queues

* attach: proxy queue first draft

- Adds ability to trace with queues during attachment
- Must be paired with updated rocprofiler-register

* Allow null agents for scratch output

* attach: improve queue library interface

- Significant changes to force exported interfaces back to C
- Fixes bug with unknown agents at attachment
- Code objects' names may still be incorrect

* attach: add code_object support

- Kernel traces will now have names and all other information for launches
- Add capture of hsa_executable to the queue library
- Various logging improvements

* attach: rename queue library to prestore

* attach: prestore overhaul

- Must be paired with commit from rocprofiler-register
- Massive overhaul of code organization in prestore library
  - Separates registrations for different object types
  - Sets up future changes for initialization

* attach: add prestore dispatch table

- Removes linkage to prestore library from sdk

* attach: cleanup

* attach: formatting

* attach: fix input prompt not appearing

* attach: fix component name in cmake

* attach: revert change to export level

* Make prestore API public

* attach: update sdk attachment library WIP

- This commit is NONFUNCTIONAL

- Changes around structure to remove classes
- Seperate C linkage where needed
- Still needs updates to register for correct usage

* attach: update register with dispatch table WIP
- This commit is NONFUNCTIONAL

- Changes rocprofiler_register to handle dispatch table from attach
  library.
- Still needs changes in SDK with dispatch table usage

* attach: dispatch table wip
- This commit is NONFUNCTIONAL

* attach: move attach component into core

* attach: rename to rocprofv3-attach

* attach: add callbacks for new queues and code objects

* attach: finish dispatch table implementation

- Fixes kernel tracing

* attach: add cmake variable for attachment support

* feat: Add --attach alias for rocprofv3 with comprehensive attachment tests

- Add `--attach` as an alias to existing `-p/--pid` functionality in rocprofv3.py
- Create comprehensive attachment test suite with CSV and JSON output validation:
- New attachment-test application for testing dynamic profiling scenarios
- Unified test script supporting both CSV and JSON output formats
- Pytest-based validation for kernel traces, memory copies, HSA API calls, and agent info
- Add CMake integration for automated attachment testing
- Support parameterized output directory and filename specification
- Implement proper environment setup for attachment queue registration

Tests verify successful attachment to running processes and capture of:
- Kernel dispatch traces with workgroup/grid dimensions
- Memory copy operations (H2D/D2H) with size validation
- HSA API call traces across multiple domains
- GPU/CPU agent information and capabilities

* Documentation Update

* attach: make attach script callable

* Added ROCPROFILER_REGISTER_ATTACHMENT_TOOL_LIB to remove hardcoded name

* attach: revert metrics library path changes

* Generic Attachment in Register (#942)

Remove tool references in register

* Add second param to attach call in rocprof register

* Add experimental reattachment support for ROCprofiler-SDK

This commit introduces experimental reattachment functionality allowing tools
to dynamically reattach to running processes with comprehensive design changes
to support multiple attach/detach cycles:

**Core Reattachment API:**
- Add rocprofiler_tool_configure_result_experimental_t with tool_reattach/tool_detach callbacks
- Add rocprofiler_call_client_reattach and rocprofiler_call_client_detach C exports
- Implement reattachment tracking in rocprofiler_register_attach to differentiate
initial attachment from reattachment cycles
- Add rocprofiler_register_invoke_reattach for handling reattachment requests

**Design Changes - Registration System Flow:**
The registration system now supports a dual-path initialization:

1. Initial Attachment Flow:
    - rocprofiler_register_attach() -> rocprofiler_register_invoke_all_registrations()
    - Full tool initialization with complete context setup
    - Sets prev_attached atomic flag to track state

2. Reattachment Flow:
    - rocprofiler_register_attach() detects prev_attached=true -> rocprofiler_register_invoke_reattach()
    - Bypasses full re-initialization, calls client reattach callbacks instead
    - Preserves existing contexts and buffers, only reactivates profiling services

**Design Changes - Tool Library Loading:**
Enhanced rocprofiler-register library loading with function pointer resolution:
- Extended rocp_set_api_table_data_t tuple to include reattach/detach function pointers
- Automatic symbol resolution for rocprofiler_call_client_reattach/detach functions
- Support for both LD_PRELOAD and dlopen scenarios with consistent callback availability

**Design Changes - Context Management:**
Introduced dual context systems for attachment scenarios:
- get_contexts() - Original contexts for standard tool initialization
- get_attach_contexts() - Separate context map for attachment-specific lifecycle
- attach_init() - Creates contexts for ALL buffer tracing services using existing buffers
- attach_start() - Selectively starts contexts based on configuration options
- attach_detach() - Cleanly stops and destroys attachment contexts

**Design Changes - Buffer Management:**
Added reset_tmp_file_buffer() template for clean reattachment state:
- Properly closes and removes old temporary files
- Deletes existing file_buffer instances to prevent stale file position tracking
- Creates fresh file_buffer instances for clean reattachment cycles
- Addresses core issue where file position metadata becomes stale between cycles

**Design Changes - Environment Variable Injection:**
Added ROCP_REGISTERED_TOOL_ATTACH environment variable:
- Distinguishes attachment-loaded tools from LD_PRELOAD scenarios
- Enables registration system to apply attachment-specific logic
- Helps tools adapt behavior for attachment vs standard initialization

**Attachment Context Management:**
- Add attach_init/attach_start/attach_detach functions for dynamic context lifecycle
- Add reset_tmp_file_buffer template for clean reattachment state management
- Implement get_attach_contexts() for tracking active attachment contexts

**Test Infrastructure:**
- Add projects/rocprofiler-sdk/tests/rocprofv3/reattach/ comprehensive test suite
- Include reattachment test scripts with unified attachment/detachment cycles
- Add validate.py with trace data validation for kernel, memory copy, HSA API, and agent info
- Add conftest.py for JSON and CSV data loading utilities

**Configuration Updates:**
- Update CMakeLists.txt to include reattachment tests in build system
- Add environment variable ROCP_REGISTERED_TOOL_ATTACH for attachment state tracking
- Enhance rocprofiler-register library loading with reattach/detach function resolution

**Flow Impact Analysis:**
This design enables robust multi-cycle attachment by:
1. Preventing duplicate initialization on reattachment
2. Maintaining separate context lifecycles for attachment vs standard operation
3. Ensuring clean temporary file state between attachment cycles
4. Providing tools with explicit reattach/detach callback hooks
5. Supporting both programmatic and environment-based tool configuration

The experimental nature allows for iteration on the API while establishing
the foundation for production-ready dynamic profiling capabilities.

* Fix misc clang-tidy warnings/errors

* CMake Option and Environment Variable Updates

- CMake: ROCPROFILER_REGISTER_ALWAYS_SUPPORT_ATTACH -> ROCPROFILER_REGISTER_BUILD_DEFAULT_ATTACHMENT
- Env: ROCPROFILER_REGISTER_ATTACHMENT_ENABLED ->

* Source reorganization

* Formatting + new lines at EOF

* Fix flake8 F841: local variable is assigned to but never used

* Update attachment test

- get rid of 5 second start delay
- add roctx

* Rework implementation

- Remove rocprofiler_tool_configure_result_experimental_t in lieu of rocprofiler_configure_attach
- Add <rocprofiler-sdk/experimental/registration.h>
- TODO: Update process_attachment.rst

* Handle re-attachment options

- inherit options from previous attachment
- check previous options do not modify data collection services

* Fix support for tools w/o rocprofiler_configure_attach

- fix segfault when rocprofiler_configure_attach does not exist
- fix naming convention for functions accepting attach dispatch table
- cleanup rocprofiler_configure_attach implementation in rocprofv3 tool

* attach: remove unknown agent handling

- Change was from earlier commit, no longer needed

* attach: add error for attaching without library loaded

* attach: revise version numbering

* attach: register header revisions

* attach: clang format register

* attach: formatting

* attach: fix build failure

- Remove cross dependency into rocprofiler-sdk, fixes build on some systems

* attach: revise register library detection

* Update rocprofiler-register and attach library

- formatting
- proper signature of register_functor for rocprofiler-sdk-attach library callback
- remove get_dispatch_registration_table()

* Bump rocprofiler-register version to 0.6.0 + AnyNewerVersion

* Fix output support for rocprofiler-sdk-tool

* Fix formatting

* Fix clang tidy errors

* Misc rocprofiler-sdk-attach fixes

* attach: add sigint handling to attach python

* tool README.md formatting

Co-authored-by: Jonathan R. Madsen <jrmadsen@users.noreply.github.com>

* Fix buffered output issue

* attach: add errors for tool attach

* CI Fixes

* Rework tests

* attach: improve library loading in rocprofv3 attach

* formatting

* Update tests to use pytest framework

* Fix test_attachment_hsa_api_trace

* attach: catch ctypes exceptions

* attach: fix leak in registration

* attach: fix sanitizer tests

* attach: fix sanitizer tests further

* attach: disable attach asan tests

* attach: disable ubsan test

* attach: fix permissions in installed test package

* attach: formatting

---------

Co-authored-by: Ian Trowbridge <Ian.Trowbridge@amd.com>
Co-authored-by: Tim Gu <Tim.Gu@amd.com>
Co-authored-by: Claude Code <claude@anthropic.com>
Co-authored-by: Benjamin Welton <bwelton@amd.com>
Co-authored-by: Jonathan R. Madsen <jonathanrmadsen@gmail.com>
Co-authored-by: Jonathan R. Madsen <jrmadsen@users.noreply.github.com>
Co-authored-by: Benjamin Welton <bewelton@amd.com>
2025-09-18 18:10:45 -05:00

259 sor
8.3 KiB
C++

// MIT License
//
// Copyright (c) 2025 Advanced Micro Devices, Inc. All rights reserved.
//
// Permission is hereby granted, free of charge, to any person obtaining a copy
// of this software and associated documentation files (the "Software"), to deal
// in the Software without restriction, including without limitation the rights
// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
// copies of the Software, and to permit persons to whom the Software is
// furnished to do so, subject to the following conditions:
//
// The above copyright notice and this permission notice shall be included in all
// copies or substantial portions of the Software.
//
// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
// SOFTWARE.
#include "ptrace_session.hpp"
#include "lib/common/environment.hpp"
#include "lib/common/logging.hpp"
#include "lib/common/static_object.hpp"
#include <rocprofiler-sdk/defines.h>
#include <atomic>
#include <thread>
extern char** environ;
namespace common = ::rocprofiler::common;
namespace
{
std::unique_ptr<rocprofiler::attach::PTraceSession> ptrace_session;
std::thread ptrace_thread;
std::atomic<bool> finished_setup(false);
} // namespace
ROCPROFILER_EXTERN_C_INIT
int
attach(uint32_t pid) ROCPROFILER_EXPORT;
int
detach() ROCPROFILER_EXPORT;
ROCPROFILER_EXTERN_C_FINI
void
initialize_logging()
{
auto logging_cfg = rocprofiler::common::logging_config{.install_failure_handler = true};
common::init_logging("ROCPROF", logging_cfg);
FLAGS_colorlogtostderr = true;
}
namespace
{
// Helper function to allocate memory in target process and write data
bool
write_data_to_target(const std::string& description,
const std::vector<uint8_t>& data,
void*& allocated_addr)
{
// Allocate memory in target process
if(!ptrace_session->simple_mmap(allocated_addr, data.size()))
{
ROCP_ERROR << "Failed to allocate memory for " << description << " in target process";
return false;
}
ROCP_TRACE << "Allocated memory for " << description << " at " << allocated_addr;
// Stop target process for writing
if(!ptrace_session->stop())
{
ROCP_ERROR << "Failed to stop target process for " << description << " writing";
return false;
}
// Write data to target process memory
if(!ptrace_session->write(reinterpret_cast<size_t>(allocated_addr), data, data.size()))
{
ROCP_ERROR << "Failed to write " << description << " to target process";
return false;
}
// Continue target process
if(!ptrace_session->cont())
{
ROCP_ERROR << "Failed to continue target process after " << description << " writing";
return false;
}
ROCP_TRACE << "Wrote " << description << " to target process";
return true;
}
// Helper function to build environment buffer
std::vector<uint8_t>
build_environment_buffer()
{
std::vector<uint8_t> environment_buffer(4);
uint32_t var_count = 0;
char** invars = environ;
for(; *invars; invars++)
{
const char* var = *invars;
if(strncmp("ROCP", var, 4) != 0)
{
continue;
}
var_count++;
ROCP_TRACE << "Adding to environment buffer: " << var;
// Add variable name
while(*var != '=')
{
environment_buffer.emplace_back(*var++);
}
environment_buffer.emplace_back(0);
// Add variable value
var++;
while(*var != 0)
{
environment_buffer.emplace_back(*var++);
}
environment_buffer.emplace_back(0);
}
// Store count in first 4 bytes
const uint8_t* var_count_bytes = reinterpret_cast<uint8_t*>(&var_count);
std::copy(var_count_bytes, var_count_bytes + 4, environment_buffer.data());
return environment_buffer;
}
} // anonymous namespace
ROCPROFILER_EXTERN_C_INIT
void
handle_ptrace_operations(uint32_t pid)
{
// Setup attachement for rocprofiler
ROCP_TRACE << "Attachment library called for pid " << pid;
ptrace_session = std::make_unique<rocprofiler::attach::PTraceSession>(pid);
ROCP_TRACE << "Attempting attachment to pid " << pid;
if(!ptrace_session->attach())
{
ROCP_ERROR << "Attachment failed to pid " << pid;
ptrace_session->m_setup_status.store(ROCPROFILER_STATUS_ERROR_INVALID_ARGUMENT);
finished_setup.store(true);
return;
}
ROCP_TRACE << "Attachment success to pid " << pid;
// Build and write environment buffer to target process
auto environment_buffer = build_environment_buffer();
void* environment_buffer_addr = nullptr;
if(!write_data_to_target("environment buffer", environment_buffer, environment_buffer_addr))
{
ptrace_session->m_setup_status.store(ROCPROFILER_STATUS_ERROR);
finished_setup.store(true);
return;
}
// Build and write tool library path to target process
auto tool_lib_path_env =
rocprofiler::common::get_env("ROCPROF_ATTACH_TOOL_LIBRARY", "librocprofiler-sdk-tool.so");
const char* tool_lib_path = tool_lib_path_env.c_str();
ROCP_TRACE << "Tool library path: " << tool_lib_path;
size_t tool_lib_path_len = strlen(tool_lib_path) + 1;
std::vector<uint8_t> tool_lib_buffer(tool_lib_path, tool_lib_path + tool_lib_path_len);
void* tool_lib_path_addr = nullptr;
if(!write_data_to_target("tool library path", tool_lib_buffer, tool_lib_path_addr))
{
ptrace_session->m_setup_status.store(ROCPROFILER_STATUS_ERROR);
finished_setup.store(true);
return;
}
// Execute the attach function with both parameters
if(!ptrace_session->call_function("librocprofiler-register.so",
"rocprofiler_register_attach",
environment_buffer_addr,
tool_lib_path_addr))
{
ROCP_ERROR << "Failed to call attach function in target process " << pid;
ptrace_session->m_setup_status.store(ROCPROFILER_STATUS_ERROR);
finished_setup.store(true);
return;
}
// Clean up - free the tool library path memory in target process
if(!ptrace_session->simple_munmap(tool_lib_path_addr, tool_lib_path_len))
{
ROCP_ERROR << "Failed to free tool library path memory in target process";
// Continue anyway since the main operation succeeded
}
ROCP_TRACE << "Cleaned up tool library path memory in target process";
// Allow main thread to continue
finished_setup.store(true);
if(!ptrace_session->handle_signals())
{
ROCP_ERROR << "Signal handling loop terminated unexepectedly for pid " << pid;
// don't return, try to detach anyways
}
// Detach rocprofiler
ROCP_TRACE << "Detaching rocprofiler from pid " << pid;
if(!ptrace_session->call_function("librocprofiler-register.so", "rocprofiler_register_detach"))
{
ROCP_ERROR << "Failed to call detach function in target process";
// don't return, try to detach anyways
}
ptrace_session->stop();
ptrace_session->detach();
ptrace_session.reset();
}
int
attach(uint32_t pid)
{
initialize_logging();
ptrace_thread = std::thread(handle_ptrace_operations, pid);
// Wait for ptrace thread to finish setting up
while(!finished_setup.load())
std::this_thread::yield();
auto status = ptrace_session->m_setup_status.load();
if(status != ROCPROFILER_STATUS_SUCCESS)
{
ROCP_ERROR << "ptrace session failed with error code " << ptrace_session->m_setup_status;
ptrace_thread.join();
finished_setup.store(false);
return status;
}
return ROCPROFILER_STATUS_SUCCESS;
}
int
detach()
{
ptrace_session->detach_ptrace_session();
ptrace_thread.join();
finished_setup.store(false);
return ROCPROFILER_STATUS_SUCCESS;
}
ROCPROFILER_EXTERN_C_FINI