* attach: rename librocprofv3-attach
- Renames library to librocprofiler-sdk-rocattach
- ROCAttach library will be formalized and documented in future commit
* Address review comments
- Rename rocprofv3-attach.py to rocprof-attach.py
- Use common filesystem.hpp in rocattach
* Fix component name typo
* Doc fixup
---------
Co-authored-by: JeniferC99 <150404595+JeniferC99@users.noreply.github.com>
* adding ROCpd database merge
* adding ROCpd database merge concatenating all tables
* update merge script
- copy all tables from files
* fix merge format
* Add package submodule, initial POC. Need to refine
* Minor fixes and clean up duplicated code in package.py
* Revamp metadata layout, add wildcard and .rpdb parsing
* Add auto merge & package when > 5 DBs, add examples, don't use auto_merge when using sub-commands merge & package
* - Extend package/yaml inputs to all rocpd modules
- Improve handling more corner cases for bad input files when parsing input parameters (bad yaml files, bad .rpdb folder, folders as input)
- Changed to use UUID in merged filename instead of the time, in auto-merge algorithm
* Minor text fixes for consistancy between modules
* Add more wildcard support and add package, merge tests
* Make changes based on review suggestions
* Move parsing packages into importer.py, simplified adding required params to a function
* fix package test by flattening input list before processing
* Integrate merge.py changes from Jonathan to add name-collision checks, recreating indexes, foreign key check (disabled for now, due to processing time)
* Rework rocpd.<submodule>.{add_args,process_args}
- add_args function returns a functor which accepts input and args
- time_window functor returned from add_args automatically applies time windowing of input
* change merge&package limit to 1, merge should create data views
* Move files by default instead of making copies
- copying can be enabled by passing "copy=True" or --copy cmdline argument
* refactor package to make the logic cleaner, set merge limit back to 5
* Allow automerge-limit param to override limit, change default back to 1. Tests updated to use query, much quicker
* Update --help instructions for package
---------
Co-authored-by: acanadas <acanadas@amd.com>
Co-authored-by: a-canadasruiz <Araceli.CanadasRuiz@amd.com>
Co-authored-by: Young Hui <young.hui@amd.com>
Co-authored-by: Jonathan R. Madsen <jonathanrmadsen@gmail.com>
* attach: rocprofv3-attach py improvements
- Handle error status during detachment
- Add detection and error for changing rocprofv3 configuration on reattachment
- Add and improve console messages during attachment and detachment
- Documentation update pass
* Fix for SWDEV-552584
Two calls to ompt_callback_task_scheduled were issued for the same
prior task. One of them was ompt_task_complete, which causes
internal storage to be release and a pointer zeroed. The other
was ompt_task_early_fulfill, which attempted to reference the
pointer. The callbacks could come in any order as they were
from different threads, thus causing a null pointer
dereference on occasion. The code was changed to do nothing
for the early_fulfill. Additional null pointer checks were
added.
* formatting
* Update ompt.cpp
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
---------
Co-authored-by: Jonathan R. Madsen <jrmadsen@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Fix dimension mismatch for multi-GPU systems with identical architectures
This change addresses an issue where counter dimensions were incorrectly
shared across all GPU agents with the same architecture name, even when
those agents had different hardware configurations (e.g., different CU counts).
Changes:
- Updated getBlockDimensions() to accept agent ID instead of architecture name
- Made dimension cache agent-specific instead of architecture-specific
- Updated set_dimensions() in AST evaluation to use specific agent ID
- Modified all API functions to handle agent-specific dimension lookups
- Updated tests to work with agent-specific dimensions
This fix ensures that dimensions accurately reflect the actual hardware
configuration of each individual GPU agent, preventing dimension mismatches
in multi-GPU systems where GPUs share the same architecture but have
different physical configurations.
Counter ID Representation Changes:
- Modified counter_id encoding to include agent information in bits 37-32
- Agent logical_node_id is encoded as (value + 1) to ensure agent 0 is detectable
- Counter records internally store only 16-bit base metric IDs (bits 15-0)
- Tool reconstructs agent-encoded counter IDs from base metric ID & agent info
- Instance record counter_id field uses bitwise AND mask to extract base metric ID
(counter_id.handle & 0xFFFF) to fit in 16-bit storage
- Output generators (CSV, JSON, Perfetto) use agent-encoded IDs for consistency
- Updated counter_config.cpp and metrics.cpp to extract base metric ID when needed
- All counter lookups now properly handle agent-encoded vs base metric IDs
This ensures counter IDs are consistent between metadata and output records while
maintaining compact storage in instance records.
- Remove unimplemented older API functions
- Remove mentions of reattach API
- Remove details on implementing a process attachment library
- This will return later as a theory of operation
* Changed stream error warning, remove regex search from attach execute test
* Formatting
* Revert accidental change
* Fix stream hang error due to grabbing same lock twice
* Updated add stream code, need to update tests
* Update attachment tests to use streams, threads, and multiple devices
* Update tests and fix stream issues
* Updated error messages to be more explicit, updated json to csv code in conftest to include streams and threads
* Formatting
* Add attachment label to attachment tests and update validation to fix errors
* Fix attach twice conftest
* Disabled thread san tests for attachment since they no longer work with bin file changes
* Updated for comment
* Added null check for getting attach status
* SWDEV-545950 - Add hipStreamCopyAttributes API Implementation
* Add unit test for hipStreamCopyAttributes API
* Add ChangeLog and nvidia mapping for the API
* Update rocprofiler-sdk with new HIP API details
* [rocprofiler-sdk] handle hipStreamCopyAttributes in stream tracing service
- this new HIP function has multiple stream arguments and needs to be skipped because it does not have an explicit create/destroy/set functionality
* Update HIP_RUNTIME_API_TABLE_STEP_VERSION in clr and rocprofiler-sdk
* Resolve merge conflicts
---------
Co-authored-by: Jonathan R. Madsen <jonathanrmadsen@gmail.com>
* Rollup of build changes needed for compat with TheRock.
* When built for a non-default ROCM location, the HIP headers can't be found by a few targets.
* Uses pkg_check for DRM libraries like ROCR-Runtime does (which avoids accidental fallback to system versions).
* Robust fix for nolink targets
* nolink targets essentially exist for include directories
* all nolink targets are automatically added to rocprofiler-sdk-headers with a $<BUILD_INTERFACE:...> generator expression
* Re-add previously used mechanism to find drm libs
---------
Co-authored-by: Marius Brehler <marius.brehler@amd.com>
Co-authored-by: Stella Laurenzo <stellaraccident@gmail.com>
Co-authored-by: Jonathan R. Madsen <jonathanrmadsen@gmail.com>
When users provide an incorrectly formatted YAML file to the -E/--extra-counters
option in rocprofv3, they now receive a clear error message showing:
- What went wrong (invalid YAML format)
- The correct rocprofiler-sdk YAML structure with example
- The actual content that failed to parse
This addresses confusion where users might use the legacy ROCProfiler YAML
format instead of the new rocprofiler-sdk schema format.
* Initial consecutive kernel WIP
* Updated logic after discussion, create context only when needed, change set of captured ids to dispatch_id_t type
* Updated to fix concurrency issues and revert kernel_iterations
* Add captured id in first lock capture
* Updated code to use wlock, added comments, removed some unecessary atomic
* Cleaned up, need to add test
* Add test to check that generated stats csv file is not empty
* Updated test to check if vector-ops kernels are being used
* Fix phase bug
* Updated for comments
* Flattened ATT logic a bit
* Fix incorrect if-statement
* Fix merge conflict
The GFX12 host-trap PC sampling support in SDK and V3.
Introducing parser tests specific to GFX12.
Co-authored-by: vlaindic_amdeng <vladimir.indic@amd.com>
* Write agent info to CSV
* Write kernel to CSV
* Write memory copy to CSV
* Write memory allocation to CSV
* Write hip api to CSV
* Write hsa api to CSV
* Write marker api to CSV
* Write counters to CSV
* Write scratch memory to CSV
* Write rccl api to CSV
* Write rocdecode api to CSV
* Write rocjpeg api to CSV
* Remove info_process joins
* Format agent id
* Compose full file name is sql writer function
* Add missing fields to kernel traces csv
* Rename vgpr_count to arch_vgpr_count
* Fix kernel name
* Skip empty query results
* Format csv.py
* Delete c++ CSV writer
* Add CSV header comparison test
* Fix comment spacing in csv.py
* Change ALLOC to ALLOCATE in memory allocation writer
* Do not append trace to agent info file name
* Revert changes for VGPR_Count
* Fix csv validation test
* Add sorting by guid
* Use EXISTS to check query results are not empty
* Merge API-specific queries
* Optimize regions query
* Column name mapping for agent info
* Pass config to sql writer
* Move agent id string building to a separate function
* add titled_headers argument
* Remove titled-columns argument
* Improvements for regions csv
* fix CSV validation test
* improve CSV validation test
* remove roctxMarkA from csv validation test
* fix capability field titles in agent info
* remove filter.py from query as that is still experimental
* Remove some aliases, now that query will auto-title the column headers
---------
Co-authored-by: Aleksei Tumakaev <atumakae@amd.com>
Co-authored-by: Young Hui <young.hui@amd.com>
Co-authored-by: Young Hui - AMD <145490163+yhuiYH@users.noreply.github.com>
* attach: milestone: API tracing
- This pairs with another commit in rocprofiler-sdk to fully
function
- Add ptrace entry points for tool attachment
- API tracing works at this commit
- Queue tracing not supported yet
* attach: cleanup
- Remove hardcode for loading of tool library
- Make invoke registration functions public again
* attach: proxy queue first draft
- Adds ability to trace with queues during attachment
- Must be paired with updated rocprofiler-sdk
* attach: prestore overhaul
- Must be paired with commit in rocprofiler-sdk
* attach: add dispatch table rework
- Register will load the prestore library and provide entrypoints to sdk
* attach: formatting and cleanup
* attach: revise dispatch table scheme
* attach: formatting
* attach: milestone: API tracing
- This change must be paired with a change in rocprofiler-register to
fully function.
- API tracing works at this commit
- Queue tracing not supported yet
* attach: cleanup and comments
* attach: Formatting and crash fixes
* attach: add attach duration
- Add option attach-duration-msec for attachment
* Formatting + sglang hang fix via signal handling
* Changed FATAL_IF to DFATAL_IF for scratch_memory due to persistent crash when iterating queues
* attach: proxy queue first draft
- Adds ability to trace with queues during attachment
- Must be paired with updated rocprofiler-register
* Allow null agents for scratch output
* attach: improve queue library interface
- Significant changes to force exported interfaces back to C
- Fixes bug with unknown agents at attachment
- Code objects' names may still be incorrect
* attach: add code_object support
- Kernel traces will now have names and all other information for launches
- Add capture of hsa_executable to the queue library
- Various logging improvements
* attach: rename queue library to prestore
* attach: prestore overhaul
- Must be paired with commit from rocprofiler-register
- Massive overhaul of code organization in prestore library
- Separates registrations for different object types
- Sets up future changes for initialization
* attach: add prestore dispatch table
- Removes linkage to prestore library from sdk
* attach: cleanup
* attach: formatting
* attach: fix input prompt not appearing
* attach: fix component name in cmake
* attach: revert change to export level
* Make prestore API public
* attach: update sdk attachment library WIP
- This commit is NONFUNCTIONAL
- Changes around structure to remove classes
- Seperate C linkage where needed
- Still needs updates to register for correct usage
* attach: update register with dispatch table WIP
- This commit is NONFUNCTIONAL
- Changes rocprofiler_register to handle dispatch table from attach
library.
- Still needs changes in SDK with dispatch table usage
* attach: dispatch table wip
- This commit is NONFUNCTIONAL
* attach: move attach component into core
* attach: rename to rocprofv3-attach
* attach: add callbacks for new queues and code objects
* attach: finish dispatch table implementation
- Fixes kernel tracing
* attach: add cmake variable for attachment support
* feat: Add --attach alias for rocprofv3 with comprehensive attachment tests
- Add `--attach` as an alias to existing `-p/--pid` functionality in rocprofv3.py
- Create comprehensive attachment test suite with CSV and JSON output validation:
- New attachment-test application for testing dynamic profiling scenarios
- Unified test script supporting both CSV and JSON output formats
- Pytest-based validation for kernel traces, memory copies, HSA API calls, and agent info
- Add CMake integration for automated attachment testing
- Support parameterized output directory and filename specification
- Implement proper environment setup for attachment queue registration
Tests verify successful attachment to running processes and capture of:
- Kernel dispatch traces with workgroup/grid dimensions
- Memory copy operations (H2D/D2H) with size validation
- HSA API call traces across multiple domains
- GPU/CPU agent information and capabilities
* Documentation Update
* attach: make attach script callable
* Added ROCPROFILER_REGISTER_ATTACHMENT_TOOL_LIB to remove hardcoded name
* attach: revert metrics library path changes
* Generic Attachment in Register (#942)
Remove tool references in register
* Add second param to attach call in rocprof register
* Add experimental reattachment support for ROCprofiler-SDK
This commit introduces experimental reattachment functionality allowing tools
to dynamically reattach to running processes with comprehensive design changes
to support multiple attach/detach cycles:
**Core Reattachment API:**
- Add rocprofiler_tool_configure_result_experimental_t with tool_reattach/tool_detach callbacks
- Add rocprofiler_call_client_reattach and rocprofiler_call_client_detach C exports
- Implement reattachment tracking in rocprofiler_register_attach to differentiate
initial attachment from reattachment cycles
- Add rocprofiler_register_invoke_reattach for handling reattachment requests
**Design Changes - Registration System Flow:**
The registration system now supports a dual-path initialization:
1. Initial Attachment Flow:
- rocprofiler_register_attach() -> rocprofiler_register_invoke_all_registrations()
- Full tool initialization with complete context setup
- Sets prev_attached atomic flag to track state
2. Reattachment Flow:
- rocprofiler_register_attach() detects prev_attached=true -> rocprofiler_register_invoke_reattach()
- Bypasses full re-initialization, calls client reattach callbacks instead
- Preserves existing contexts and buffers, only reactivates profiling services
**Design Changes - Tool Library Loading:**
Enhanced rocprofiler-register library loading with function pointer resolution:
- Extended rocp_set_api_table_data_t tuple to include reattach/detach function pointers
- Automatic symbol resolution for rocprofiler_call_client_reattach/detach functions
- Support for both LD_PRELOAD and dlopen scenarios with consistent callback availability
**Design Changes - Context Management:**
Introduced dual context systems for attachment scenarios:
- get_contexts() - Original contexts for standard tool initialization
- get_attach_contexts() - Separate context map for attachment-specific lifecycle
- attach_init() - Creates contexts for ALL buffer tracing services using existing buffers
- attach_start() - Selectively starts contexts based on configuration options
- attach_detach() - Cleanly stops and destroys attachment contexts
**Design Changes - Buffer Management:**
Added reset_tmp_file_buffer() template for clean reattachment state:
- Properly closes and removes old temporary files
- Deletes existing file_buffer instances to prevent stale file position tracking
- Creates fresh file_buffer instances for clean reattachment cycles
- Addresses core issue where file position metadata becomes stale between cycles
**Design Changes - Environment Variable Injection:**
Added ROCP_REGISTERED_TOOL_ATTACH environment variable:
- Distinguishes attachment-loaded tools from LD_PRELOAD scenarios
- Enables registration system to apply attachment-specific logic
- Helps tools adapt behavior for attachment vs standard initialization
**Attachment Context Management:**
- Add attach_init/attach_start/attach_detach functions for dynamic context lifecycle
- Add reset_tmp_file_buffer template for clean reattachment state management
- Implement get_attach_contexts() for tracking active attachment contexts
**Test Infrastructure:**
- Add projects/rocprofiler-sdk/tests/rocprofv3/reattach/ comprehensive test suite
- Include reattachment test scripts with unified attachment/detachment cycles
- Add validate.py with trace data validation for kernel, memory copy, HSA API, and agent info
- Add conftest.py for JSON and CSV data loading utilities
**Configuration Updates:**
- Update CMakeLists.txt to include reattachment tests in build system
- Add environment variable ROCP_REGISTERED_TOOL_ATTACH for attachment state tracking
- Enhance rocprofiler-register library loading with reattach/detach function resolution
**Flow Impact Analysis:**
This design enables robust multi-cycle attachment by:
1. Preventing duplicate initialization on reattachment
2. Maintaining separate context lifecycles for attachment vs standard operation
3. Ensuring clean temporary file state between attachment cycles
4. Providing tools with explicit reattach/detach callback hooks
5. Supporting both programmatic and environment-based tool configuration
The experimental nature allows for iteration on the API while establishing
the foundation for production-ready dynamic profiling capabilities.
* Fix misc clang-tidy warnings/errors
* CMake Option and Environment Variable Updates
- CMake: ROCPROFILER_REGISTER_ALWAYS_SUPPORT_ATTACH -> ROCPROFILER_REGISTER_BUILD_DEFAULT_ATTACHMENT
- Env: ROCPROFILER_REGISTER_ATTACHMENT_ENABLED ->
* Source reorganization
* Formatting + new lines at EOF
* Fix flake8 F841: local variable is assigned to but never used
* Update attachment test
- get rid of 5 second start delay
- add roctx
* Rework implementation
- Remove rocprofiler_tool_configure_result_experimental_t in lieu of rocprofiler_configure_attach
- Add <rocprofiler-sdk/experimental/registration.h>
- TODO: Update process_attachment.rst
* Handle re-attachment options
- inherit options from previous attachment
- check previous options do not modify data collection services
* Fix support for tools w/o rocprofiler_configure_attach
- fix segfault when rocprofiler_configure_attach does not exist
- fix naming convention for functions accepting attach dispatch table
- cleanup rocprofiler_configure_attach implementation in rocprofv3 tool
* attach: remove unknown agent handling
- Change was from earlier commit, no longer needed
* attach: add error for attaching without library loaded
* attach: revise version numbering
* attach: register header revisions
* attach: clang format register
* attach: formatting
* attach: fix build failure
- Remove cross dependency into rocprofiler-sdk, fixes build on some systems
* attach: revise register library detection
* Update rocprofiler-register and attach library
- formatting
- proper signature of register_functor for rocprofiler-sdk-attach library callback
- remove get_dispatch_registration_table()
* Bump rocprofiler-register version to 0.6.0 + AnyNewerVersion
* Fix output support for rocprofiler-sdk-tool
* Fix formatting
* Fix clang tidy errors
* Misc rocprofiler-sdk-attach fixes
* attach: add sigint handling to attach python
* tool README.md formatting
Co-authored-by: Jonathan R. Madsen <jrmadsen@users.noreply.github.com>
* Fix buffered output issue
* attach: add errors for tool attach
* CI Fixes
* Rework tests
* attach: improve library loading in rocprofv3 attach
* formatting
* Update tests to use pytest framework
* Fix test_attachment_hsa_api_trace
* attach: catch ctypes exceptions
* attach: fix leak in registration
* attach: fix sanitizer tests
* attach: fix sanitizer tests further
* attach: disable attach asan tests
* attach: disable ubsan test
* attach: fix permissions in installed test package
* attach: formatting
---------
Co-authored-by: Ian Trowbridge <Ian.Trowbridge@amd.com>
Co-authored-by: Tim Gu <Tim.Gu@amd.com>
Co-authored-by: Claude Code <claude@anthropic.com>
Co-authored-by: Benjamin Welton <bwelton@amd.com>
Co-authored-by: Jonathan R. Madsen <jonathanrmadsen@gmail.com>
Co-authored-by: Jonathan R. Madsen <jrmadsen@users.noreply.github.com>
Co-authored-by: Benjamin Welton <bewelton@amd.com>
* Added check in Queue::sync to verify that there is a callback for every dispatch
* Removed new atomic, using get_balanced_signal_slots() atomic with initial value of NUM_SIGNALS to verify dispatches complete
* Use queries instead of views in summary.py
* Export queries when created
* Remove HIP and HSA from output
* Fix domain query
* Export summary queries in the main function
* Fix comments and variable names
* Change syntax for old python versions
---------
Co-authored-by: Young Hui <young.hui@amd.com>
* add more derived metrics for navi4.
* addr comments
* addr comments, and add more derived counters.
* EOF.
* misc.
* remove duplicate counter.
* misc.
* Remove gfx12 architecture definition for ldslatency
* remove extra architectures for gfx12.
* use wgp for normalization
* move these changes to another PR.
---------
Co-authored-by: Venkateshwar Reddy Kandula <venkateshwar.kandula1306@gmail.com>
Swati Rawat
Mark Meserve
Jonathan R. Madsen
Gopesh Bhardwaj
Giovanni Lenzi Baraldi
systems-assistant[bot]
Aleksei Tumakaev
Satyanvesh Dittakavi
Todd tiantuo Li
usrihari123
Larry Meadows
sluzynsk-amd
Benjamin Welton
Vladimir Indic
Ammar ELWazir
itrowbri
Rahul Manocha
Venkateshwar Reddy Kandula
Marius Brehler
Jatin Chaudhary
Matt Williams
Julian Jose