[rocprofiler-sdk][rocprofiler-register] Initial Attachment Support (#316)

* attach: milestone: API tracing

- This pairs with another commit in rocprofiler-sdk to fully
  function
- Add ptrace entry points for tool attachment
- API tracing works at this commit
- Queue tracing not supported yet

* attach: cleanup

- Remove hardcode for loading of tool library
- Make invoke registration functions public again

* attach: proxy queue first draft

- Adds ability to trace with queues during attachment
- Must be paired with updated rocprofiler-sdk

* attach: prestore overhaul

- Must be paired with commit in rocprofiler-sdk

* attach: add dispatch table rework

- Register will load the prestore library and provide entrypoints to sdk

* attach: formatting and cleanup

* attach: revise dispatch table scheme

* attach: formatting

* attach: milestone: API tracing

- This change must be paired with a change in rocprofiler-register to
  fully function.
- API tracing works at this commit
- Queue tracing not supported yet

* attach: cleanup and comments

* attach: Formatting and crash fixes

* attach: add attach duration

- Add option attach-duration-msec for attachment

* Formatting + sglang hang fix via signal handling

* Changed FATAL_IF to DFATAL_IF for scratch_memory due to persistent crash when iterating queues

* attach: proxy queue first draft

- Adds ability to trace with queues during attachment
- Must be paired with updated rocprofiler-register

* Allow null agents for scratch output

* attach: improve queue library interface

- Significant changes to force exported interfaces back to C
- Fixes bug with unknown agents at attachment
- Code objects' names may still be incorrect

* attach: add code_object support

- Kernel traces will now have names and all other information for launches
- Add capture of hsa_executable to the queue library
- Various logging improvements

* attach: rename queue library to prestore

* attach: prestore overhaul

- Must be paired with commit from rocprofiler-register
- Massive overhaul of code organization in prestore library
  - Separates registrations for different object types
  - Sets up future changes for initialization

* attach: add prestore dispatch table

- Removes linkage to prestore library from sdk

* attach: cleanup

* attach: formatting

* attach: fix input prompt not appearing

* attach: fix component name in cmake

* attach: revert change to export level

* Make prestore API public

* attach: update sdk attachment library WIP

- This commit is NONFUNCTIONAL

- Changes around structure to remove classes
- Seperate C linkage where needed
- Still needs updates to register for correct usage

* attach: update register with dispatch table WIP
- This commit is NONFUNCTIONAL

- Changes rocprofiler_register to handle dispatch table from attach
  library.
- Still needs changes in SDK with dispatch table usage

* attach: dispatch table wip
- This commit is NONFUNCTIONAL

* attach: move attach component into core

* attach: rename to rocprofv3-attach

* attach: add callbacks for new queues and code objects

* attach: finish dispatch table implementation

- Fixes kernel tracing

* attach: add cmake variable for attachment support

* feat: Add --attach alias for rocprofv3 with comprehensive attachment tests

- Add `--attach` as an alias to existing `-p/--pid` functionality in rocprofv3.py
- Create comprehensive attachment test suite with CSV and JSON output validation:
- New attachment-test application for testing dynamic profiling scenarios
- Unified test script supporting both CSV and JSON output formats
- Pytest-based validation for kernel traces, memory copies, HSA API calls, and agent info
- Add CMake integration for automated attachment testing
- Support parameterized output directory and filename specification
- Implement proper environment setup for attachment queue registration

Tests verify successful attachment to running processes and capture of:
- Kernel dispatch traces with workgroup/grid dimensions
- Memory copy operations (H2D/D2H) with size validation
- HSA API call traces across multiple domains
- GPU/CPU agent information and capabilities

* Documentation Update

* attach: make attach script callable

* Added ROCPROFILER_REGISTER_ATTACHMENT_TOOL_LIB to remove hardcoded name

* attach: revert metrics library path changes

* Generic Attachment in Register (#942)

Remove tool references in register

* Add second param to attach call in rocprof register

* Add experimental reattachment support for ROCprofiler-SDK

This commit introduces experimental reattachment functionality allowing tools
to dynamically reattach to running processes with comprehensive design changes
to support multiple attach/detach cycles:

**Core Reattachment API:**
- Add rocprofiler_tool_configure_result_experimental_t with tool_reattach/tool_detach callbacks
- Add rocprofiler_call_client_reattach and rocprofiler_call_client_detach C exports
- Implement reattachment tracking in rocprofiler_register_attach to differentiate
initial attachment from reattachment cycles
- Add rocprofiler_register_invoke_reattach for handling reattachment requests

**Design Changes - Registration System Flow:**
The registration system now supports a dual-path initialization:

1. Initial Attachment Flow:
    - rocprofiler_register_attach() -> rocprofiler_register_invoke_all_registrations()
    - Full tool initialization with complete context setup
    - Sets prev_attached atomic flag to track state

2. Reattachment Flow:
    - rocprofiler_register_attach() detects prev_attached=true -> rocprofiler_register_invoke_reattach()
    - Bypasses full re-initialization, calls client reattach callbacks instead
    - Preserves existing contexts and buffers, only reactivates profiling services

**Design Changes - Tool Library Loading:**
Enhanced rocprofiler-register library loading with function pointer resolution:
- Extended rocp_set_api_table_data_t tuple to include reattach/detach function pointers
- Automatic symbol resolution for rocprofiler_call_client_reattach/detach functions
- Support for both LD_PRELOAD and dlopen scenarios with consistent callback availability

**Design Changes - Context Management:**
Introduced dual context systems for attachment scenarios:
- get_contexts() - Original contexts for standard tool initialization
- get_attach_contexts() - Separate context map for attachment-specific lifecycle
- attach_init() - Creates contexts for ALL buffer tracing services using existing buffers
- attach_start() - Selectively starts contexts based on configuration options
- attach_detach() - Cleanly stops and destroys attachment contexts

**Design Changes - Buffer Management:**
Added reset_tmp_file_buffer() template for clean reattachment state:
- Properly closes and removes old temporary files
- Deletes existing file_buffer instances to prevent stale file position tracking
- Creates fresh file_buffer instances for clean reattachment cycles
- Addresses core issue where file position metadata becomes stale between cycles

**Design Changes - Environment Variable Injection:**
Added ROCP_REGISTERED_TOOL_ATTACH environment variable:
- Distinguishes attachment-loaded tools from LD_PRELOAD scenarios
- Enables registration system to apply attachment-specific logic
- Helps tools adapt behavior for attachment vs standard initialization

**Attachment Context Management:**
- Add attach_init/attach_start/attach_detach functions for dynamic context lifecycle
- Add reset_tmp_file_buffer template for clean reattachment state management
- Implement get_attach_contexts() for tracking active attachment contexts

**Test Infrastructure:**
- Add projects/rocprofiler-sdk/tests/rocprofv3/reattach/ comprehensive test suite
- Include reattachment test scripts with unified attachment/detachment cycles
- Add validate.py with trace data validation for kernel, memory copy, HSA API, and agent info
- Add conftest.py for JSON and CSV data loading utilities

**Configuration Updates:**
- Update CMakeLists.txt to include reattachment tests in build system
- Add environment variable ROCP_REGISTERED_TOOL_ATTACH for attachment state tracking
- Enhance rocprofiler-register library loading with reattach/detach function resolution

**Flow Impact Analysis:**
This design enables robust multi-cycle attachment by:
1. Preventing duplicate initialization on reattachment
2. Maintaining separate context lifecycles for attachment vs standard operation
3. Ensuring clean temporary file state between attachment cycles
4. Providing tools with explicit reattach/detach callback hooks
5. Supporting both programmatic and environment-based tool configuration

The experimental nature allows for iteration on the API while establishing
the foundation for production-ready dynamic profiling capabilities.

* Fix misc clang-tidy warnings/errors

* CMake Option and Environment Variable Updates

- CMake: ROCPROFILER_REGISTER_ALWAYS_SUPPORT_ATTACH -> ROCPROFILER_REGISTER_BUILD_DEFAULT_ATTACHMENT
- Env: ROCPROFILER_REGISTER_ATTACHMENT_ENABLED ->

* Source reorganization

* Formatting + new lines at EOF

* Fix flake8 F841: local variable is assigned to but never used

* Update attachment test

- get rid of 5 second start delay
- add roctx

* Rework implementation

- Remove rocprofiler_tool_configure_result_experimental_t in lieu of rocprofiler_configure_attach
- Add <rocprofiler-sdk/experimental/registration.h>
- TODO: Update process_attachment.rst

* Handle re-attachment options

- inherit options from previous attachment
- check previous options do not modify data collection services

* Fix support for tools w/o rocprofiler_configure_attach

- fix segfault when rocprofiler_configure_attach does not exist
- fix naming convention for functions accepting attach dispatch table
- cleanup rocprofiler_configure_attach implementation in rocprofv3 tool

* attach: remove unknown agent handling

- Change was from earlier commit, no longer needed

* attach: add error for attaching without library loaded

* attach: revise version numbering

* attach: register header revisions

* attach: clang format register

* attach: formatting

* attach: fix build failure

- Remove cross dependency into rocprofiler-sdk, fixes build on some systems

* attach: revise register library detection

* Update rocprofiler-register and attach library

- formatting
- proper signature of register_functor for rocprofiler-sdk-attach library callback
- remove get_dispatch_registration_table()

* Bump rocprofiler-register version to 0.6.0 + AnyNewerVersion

* Fix output support for rocprofiler-sdk-tool

* Fix formatting

* Fix clang tidy errors

* Misc rocprofiler-sdk-attach fixes

* attach: add sigint handling to attach python

* tool README.md formatting

Co-authored-by: Jonathan R. Madsen <jrmadsen@users.noreply.github.com>

* Fix buffered output issue

* attach: add errors for tool attach

* CI Fixes

* Rework tests

* attach: improve library loading in rocprofv3 attach

* formatting

* Update tests to use pytest framework

* Fix test_attachment_hsa_api_trace

* attach: catch ctypes exceptions

* attach: fix leak in registration

* attach: fix sanitizer tests

* attach: fix sanitizer tests further

* attach: disable attach asan tests

* attach: disable ubsan test

* attach: fix permissions in installed test package

* attach: formatting

---------

Co-authored-by: Ian Trowbridge <Ian.Trowbridge@amd.com>
Co-authored-by: Tim Gu <Tim.Gu@amd.com>
Co-authored-by: Claude Code <claude@anthropic.com>
Co-authored-by: Benjamin Welton <bwelton@amd.com>
Co-authored-by: Jonathan R. Madsen <jonathanrmadsen@gmail.com>
Co-authored-by: Jonathan R. Madsen <jrmadsen@users.noreply.github.com>
Co-authored-by: Benjamin Welton <bewelton@amd.com>
このコミットが含まれているのは:
Mark Meserve
2025-09-18 18:10:45 -05:00
committed by GitHub
コミット bf49039005
72個のファイルの変更、6456行の追加、140行の削除
+10
ファイルの表示
@@ -26,6 +26,16 @@ install(
WORLD_EXECUTE
COMPONENT tools)
configure_file(rocprofv3-attach.py
${PROJECT_BINARY_DIR}/${CMAKE_INSTALL_BINDIR}/rocprofv3-attach COPYONLY)
install(
FILES ${PROJECT_BINARY_DIR}/${CMAKE_INSTALL_BINDIR}/rocprofv3-attach
DESTINATION ${CMAKE_INSTALL_BINDIR}
PERMISSIONS OWNER_READ OWNER_WRITE OWNER_EXECUTE GROUP_READ GROUP_EXECUTE WORLD_READ
WORLD_EXECUTE
COMPONENT tools)
# for each entry here there must be a ROCPD_BIN_<entry> list
set(ROCPD_EXECUTABLES "all" "csv" "otf2" "pftrace" "summary")
+88
ファイルの表示
@@ -0,0 +1,88 @@
#!/usr/bin/env python3
# MIT License
#
# Copyright (c) 2025 Advanced Micro Devices, Inc. All rights reserved.
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
# THE SOFTWARE.
import ctypes
import os
import signal
import sys
import time
ROCPROFV3_ATTACH_DIR = os.path.dirname(os.path.realpath(__file__))
ROCM_DIR = os.path.dirname(ROCPROFV3_ATTACH_DIR)
ROCPROF_ATTACH_TOOL_LIBRARY = f"{ROCM_DIR}/lib/rocprofiler-sdk/librocprofv3-attach.so"
def main(
pid=os.environ.get("ROCPROF_ATTACH_PID", None),
attach_library=os.environ.get(
"ROCPROF_ATTACH_TOOL_LIBRARY", ROCPROF_ATTACH_TOOL_LIBRARY
),
duration=os.environ.get("ROCPROF_ATTACH_DURATION", None),
):
if pid is None:
raise RuntimeError("rocprofv3_attach called with no PID specified")
print(f"Attaching to PID {pid} using library {attach_library}")
# Load the shared library into ctypes and attach
try:
c_lib = ctypes.CDLL(attach_library)
c_lib.attach.restype = ctypes.c_int
c_lib.attach.argtypes = [ctypes.c_uint]
attach_status = c_lib.attach(int(pid))
except Exception as e:
raise RuntimeError(f"Exception during library load and attachment: {e}")
if attach_status != 0:
raise RuntimeError(
f"Calling attach in {attach_library} returned non-zero status {attach_status}"
)
print(f"Attaching to PID {pid} using library {attach_library} :: success")
def detach():
try:
c_lib.detach()
except Exception as e:
print(f"Exception during detachment: {e}")
def signal_handler(sig, frame):
print("\nCaught signal SIGINT, detaching")
detach()
sys.exit(0)
signal.signal(signal.SIGINT, signal_handler)
if duration is None:
sys.stdout.write("Press Enter to detach...")
sys.stdout.flush() # Force the prompt to appear immediately
input() # Now wait for input
else:
time.sleep(int(duration) / 1000)
detach()
if __name__ == "__main__":
main()
+118 -25
ファイルの表示
@@ -60,6 +60,12 @@ class dotdict(dict):
[dotdict(i) if isinstance(i, (dict)) else i for i in v],
)
def __getstate__(self):
return self.__dict__
def __setstate__(self, d):
self.__dict__ = d
def patch_message(msg, *args):
msg = textwrap.dedent(msg)
@@ -72,14 +78,14 @@ def patch_message(msg, *args):
def fatal_error(msg, *args, exit_code=1):
msg = patch_message(msg, *args)
sys.stderr.write(f"Fatal error: {msg}\n")
sys.stderr.write(f"[rocprofv3] Fatal error: {msg}\n")
sys.stderr.flush()
sys.exit(exit_code)
def warning(msg, *args):
msg = patch_message(msg, *args)
sys.stderr.write(f"Warning: {msg}\n")
sys.stderr.write(f"[rocprofv3] Warning: {msg}\n")
sys.stderr.flush()
@@ -224,6 +230,11 @@ For MPI applications (or other job launchers such as SLURM), place rocprofv3 ins
$ mpirun -n 4 rocprofv3 --hip-trace -- ./mympiapp
For attachment profiling of running processes:
$ rocprofv3 --attach <PID> --hip-trace --kernel-trace
$ rocprofv3 --attach 1234 --attach-duration 10 --hsa-trace
"""
# Create the parser
@@ -725,13 +736,19 @@ For MPI applications (or other job launchers such as SLURM), place rocprofv3 ins
metavar="KB",
)
reserved_options = parser.add_argument_group("Reserved options")
reserved_options.add_argument(
advanced_options.add_argument(
"-p",
"--pid",
help=argparse.SUPPRESS,
type=str,
nargs="+",
"--attach",
help="""Attach to a target process by pid and execute as a tool from within said process.""",
type=int,
default=None,
)
advanced_options.add_argument(
"--attach-duration-msec",
help="""When --pid is used, sets the amount of time in milliseconds the profiler will be attached before detaching. When unset, the profiler will wait until Enter is pressed to detach.""",
type=int,
default=None,
)
@@ -940,18 +957,27 @@ def patch_args(data):
return data
def get_args(cmd_args, inp_args):
def get_args(cmd_args, inp_args, filter=[]):
def ensure_type(name, var, type_id):
if not isinstance(var, type_id):
raise TypeError(
f"{name} is of type {type(var).__name__}, expected {type(type_id).__name__}"
f"{name} is of type {type(var).__name__}, expected {type_id.__name__}"
)
ensure_type("cmd_args", cmd_args, argparse.Namespace)
ensure_type("inp_args", inp_args, dotdict)
if isinstance(cmd_args, argparse.Namespace):
ensure_type("cmd_args", cmd_args, argparse.Namespace)
ensure_type("inp_args", inp_args, dotdict)
cmd_keys = list(cmd_args.__dict__.keys())
inp_keys = list(inp_args.keys())
else:
ensure_type("cmd_args", cmd_args, dotdict)
ensure_type("inp_args", inp_args, dotdict)
cmd_keys = list(cmd_args.keys())
inp_keys = list(inp_args.keys())
cmd_keys = list(cmd_args.__dict__.keys())
inp_keys = list(inp_args.keys())
data = {}
def get_attr(key):
@@ -967,9 +993,30 @@ def get_args(cmd_args, inp_args):
and has_set_attr(inp_args, itr)
and getattr(cmd_args, itr) != getattr(inp_args, itr)
):
raise RuntimeError(
f"conflicting value for {itr} : {getattr(cmd_args, itr)} vs {getattr(inp_args, itr)}"
)
should_raise = True
if filter:
is_filtered = False
for fitr in filter:
import re
if re.match(fitr, itr):
is_filtered = True
break
if not is_filtered:
warning(
f"Option '{itr}' has been modified. {itr}={getattr(cmd_args, itr)} (previously {itr}={getattr(inp_args, itr)})"
)
should_raise = False
# should raise error if not in filter list
if should_raise:
raise RuntimeError(
f"conflicting value for {itr} : {getattr(cmd_args, itr)} vs {getattr(inp_args, itr)}"
)
else:
# has preference towards command line args
data[itr] = get_attr(itr)
else:
data[itr] = get_attr(itr)
@@ -982,13 +1029,6 @@ def run(app_args, args, **kwargs):
use_execv = kwargs.get("use_execv", True)
app_pass = kwargs.get("pass_id", None)
if args.pid is not None:
fatal_error(
"""The -p shorthand option for --collection-period is now an upper-case -P
In the future, rocprofv3 plans to support debugger-like process attachment and -p
is de-facto standard shorthand option for this feature"""
)
def setattrifnone(obj, attr, value):
if getattr(obj, f"{attr}") is None:
setattr(obj, f"{attr}", value)
@@ -1075,6 +1115,7 @@ def run(app_args, args, **kwargs):
ROCPROF_LIST_AVAIL_TOOL_LIBRARY = (
f"{ROCM_DIR}/lib/rocprofiler-sdk/librocprofv3-list-avail.so"
)
ROCPROF_ATTACH_TOOL_LIBRARY = f"{ROCM_DIR}/lib/rocprofiler-sdk/librocprofv3-attach.so"
ROCPROF_TOOL_LIBRARY = resolve_library_path(ROCPROF_TOOL_LIBRARY, args)
ROCPROF_SDK_LIBRARY = resolve_library_path(ROCPROF_SDK_LIBRARY, args)
@@ -1083,6 +1124,7 @@ def run(app_args, args, **kwargs):
ROCPROF_LIST_AVAIL_TOOL_LIBRARY = resolve_library_path(
ROCPROF_LIST_AVAIL_TOOL_LIBRARY, args
)
ROCPROF_ATTACH_TOOL_LIBRARY = resolve_library_path(ROCPROF_ATTACH_TOOL_LIBRARY, args)
prepend_preload = [itr for itr in args.preload if itr]
append_preload = [
@@ -1090,8 +1132,9 @@ def run(app_args, args, **kwargs):
ROCPROF_SDK_LIBRARY,
]
update_env("LD_PRELOAD", ":".join(prepend_preload), prepend=True)
update_env("LD_PRELOAD", ":".join(append_preload), append=True)
if not args.pid:
update_env("LD_PRELOAD", ":".join(prepend_preload), prepend=True)
update_env("LD_PRELOAD", ":".join(append_preload), append=True)
update_env(
"ROCP_TOOL_LIBRARIES",
@@ -1298,6 +1341,13 @@ def run(app_args, args, **kwargs):
overwrite_if_true=True,
)
if args.pid:
update_env(
"ROCPROF_ATTACH_TOOL_LIBRARY",
ROCPROF_ATTACH_TOOL_LIBRARY,
overwrite_if_true=True,
)
if args.collection_period:
factors = {
"hour": 60 * 60 * 1e9,
@@ -1430,6 +1480,16 @@ def run(app_args, args, **kwargs):
env=app_env,
)
elif args.pid:
update_env("ROCPROF_ATTACH_PID", args.pid)
if args.attach_duration_msec is not None:
update_env("ROCPROF_ATTACH_DURATION", f"{args.attach_duration_msec}")
path = os.path.join(f"{ROCM_DIR}", "bin/rocprofv3-attach")
if app_args:
exit_code = subprocess.check_call([sys.executable, path], env=app_env)
else:
app_args = [sys.executable, path]
elif not app_args and not args.echo:
log_config(app_env)
fatal_error("No application provided")
@@ -1673,6 +1733,39 @@ def main(argv=None):
if len(inp_args) == 1:
args = get_args(cmd_args, inp_args[0])
if args.pid:
import pickle
if args.collection_period:
fatal_error("--collection-period is not compatible with attach mode")
fname = f"/tmp/rocprofv3_attach_{args.pid}.pkl"
if os.path.exists(fname):
# load the configuration from the previous attachment
with open(fname, "rb") as ifs:
if args.log_level in ("config", "info", "trace"):
print(f"Loading attach configuration from {fname}...")
prev_args = pickle.load(ifs)
args = get_args(
args,
dotdict(prev_args),
filter=[
".*_trace",
"^pc_sampling_.*$",
"^att_.*$",
"^(pmc|pmc_groups|output_config|extra_counters)$",
"^kernel_(include_regex|exclude_regex|iteration_range)$",
],
)
# write the configuration for future attachments
with open(fname, "wb") as ofs:
if args.log_level in ("config", "info", "trace"):
print(f"Saving attach configuration to {fname}...")
pickle.dump(args, ofs)
pass_idx = None
if has_set_attr(args, "pmc") and len(args.pmc) > 0:
pass_idx = 1