AI/rocm-systems
2
0
派生 0
代码 工单 合并请求 工作流 软件包 项目 发布 百科 活动

Add 'projects/rocprofiler-sdk/' from commit 'bf0fad1d5406fbc51403ba1aa9621a9d4a9bce2b'

浏览代码 
git-subtree-dir: projects/rocprofiler-sdk
git-subtree-mainline: 50a90550e9
git-subtree-split: bf0fad1d54
这个提交包含在：
systems-assistant[bot]
2025-07-22 22:52:46 +00:00
父节点 50a90550e9 bf0fad1d54
当前提交 ad0fb25ed5
共修改 1272 个文件，包含 230117 行新增和 0 行删除
下载 Patch 文件 下载 Diff 文件 展开所有文件 折叠所有文件
projects/rocprofiler-sdk/source/docs/how-to/samples.rst
+49
查看文件
@@ -0,0 +1,49 @@
.. meta::
:description: "ROCprofiler-SDK is a tooling infrastructure for profiling general-purpose GPU compute applications running on the ROCm software."
:keywords: "ROCprofiler-SDK, ROCProfiler-SDK samples"
.. _rocprofiler-sdk-samples:
ROCprofiler-SDK samples
========================
The samples are provided to help you see the profiler in action.
Finding samples
---------------
The ROCm installation provides sample programs and ``rocprofv3`` tool.
- Sample programs are installed here:
.. code-block:: bash
/opt/rocm/share/rocprofiler-sdk/samples
- ``rocprofv3`` tool is installed here:
.. code-block:: bash
/opt/rocm/bin
Building Samples
----------------
To build samples from any directory, run:
.. code-block:: bash
cmake -B build-rocprofiler-sdk-samples /opt/rocm/share/rocprofiler-sdk/samples -DCMAKE_PREFIX_PATH=/opt/rocm
cmake --build build-rocprofiler-sdk-samples --target all --parallel 8
Running samples
---------------
To run the built samples, ``cd`` into the ``build-rocprofiler-sdk-samples`` directory and run:
.. code-block:: bash
ctest -V
The `-V` option enables verbose output, providing detailed information about the test execution.
projects/rocprofiler-sdk/source/docs/how-to/using-pc-sampling.rst
+357
查看文件
@@ -0,0 +1,357 @@
.. meta::
:description: Documentation of the usage of pc-sampling with rocprofv3 command-line tool
:keywords: Sampling PC, Sampling program counter, rocprofv3, rocprofv3 tool usage, Using rocprofv3, ROCprofiler-SDK command line tool, PC sampling
.. _using-pc-sampling:
==================
Using PC sampling
==================
PC (Program Counter) sampling service for GPU profiling is a profiling technique to periodically sample the program counter during GPU kernel execution. PC sampling helps in understanding code execution patterns and identifying hotspot(s).
Here are the benefits of using PC sampling:
- Identify performance bottlenecks
- Understand kernel execution behavior
- Analyze code coverage
- Find heavily executed code paths
To try out the PC sampling feature, you can use the command-line tool ``rocprofv3`` or the ROCprofiler-SDK library on `ROCm 6.4` or later.
.. note::
PC sampling is ONLY supported on AMD GPUs with architectures gfx90a and later.
PC sampling availability and configuration
===========================================
To check if the GPU supports PC sampling, use:
.. code-block:: bash
rocprofv3 -L
Or
.. code-block:: bash
rocprofv3 --list-avail
The output lists if ``rocprofv3`` supports PC sampling on the GPU and the supported configuration.
.. code-block:: bash
GPU:0
NAME:gfx90a
configs:
Method :host_trap
Unit :time
Min_Interval :1
Max_Interval :18446744073709551615
Flags :none
The preceding output shows that the GPU supports PC sampling with the ``ROCPROFILER_PC_SAMPLING_METHOD_HOST_TRAP`` method and the ``ROCPROFILER_PC_SAMPLING_UNIT_TIME`` unit. The minimum and maximum intervals are also displayed.
.. note::
Important firmware fixes to host-trap and stochastic PC-sampling for AMD Instinct MI300X have been made in ROCm 7.0.
To ensure that you have the latest fixes, check if you have the correct firmware versions installed:
For host-trap PC-sampling on MI300X: PSP TOS Firmware >= version 00.36.02.59 or 0x00360259
For stochastic PC-sampling on MI300X as described in the following section: MEC Firmware feature version: 50, firmware version >= 0x0000001a
To check the firmware versions, use:
.. code-block:: bash
# To check PSP TOS Firmware:
sudo cat /sys/kernel/debug/dri/0/amdgpu_firmware_info | grep SOS
# To check MEC Firmware:
sudo cat /sys/kernel/debug/dri/1/amdgpu_firmware_info | grep MEC
Based on the available PC-sampling configurations, use the following command to profile the application using PC-sampling:
.. code-block:: bash
rocprofv3 --pc-sampling-beta-enabled --pc-sampling-method host_trap --pc-sampling-unit time --pc-sampling-interval 1 --output-format csv -- <application_path>
The preceding command enables PC sampling with the ``host_trap`` method, ``time`` unit, and an interval of ``1`` μs (microsecond). Replace ``<application_path>`` with the path to the application you want to profile.
This generates two files, ``agent_info.csv`` and ``pc_sampling_host_trap.csv``. Both files are prefixed with the process ID.
Here are the contents of ``pc_sampling_host_trap.csv`` file generated for MatrixTranspose sample application:
.. csv-table:: PC sampling host trap
:file: /data/pc_sampling_host_trap.csv
:widths: 20,10,10,10,10,20
:header-rows: 1
For description of the fields in the output file, see :ref:`pc-sampling-fields`.
If you find the ``Instruction_Comment`` field in the output file to be empty, populate this field by compiling your application with debug symbols.
Enabling debug symbols while compiling the application maps back to the source line. This helps in understanding the code execution pattern and hotspots.
.. csv-table:: PC sampling host trap with debug symbols
:file: /data/pc_sampling_host_trap_debug.csv
:widths: 20,10,10,10,10,20
:header-rows: 1
The preceding output shows the ``Instruction_Comment`` field populated with the source-line information.
.. _pc-sampling-fields:
PC sampling fields
===================
Here are the fields in the output file generated by PC sampling:
- ``Sample_Timestamp``: Timestamp when sample is generated
- ``Exec_Mask``: Active SIMD lanes when sampled
- ``Dispatch_Id``: Originating kernel dispatch ID
- ``Instruction``: Assembly instruction such as ``s_load_dword s8, s[1:2], 0x10``
- ``Instruction_Comment``: Instruction comment that maps back to the source-line if debug symbols were enabled when application was compiled
- ``Correlation_Id``: API launch call ID that matches dispatch ID
To dump samples in a more comprehensive format, use JSON through ``--output-format json``:
.. code-block:: bash
rocprofv3 --pc-sampling-beta-enabled --pc-sampling-method host_trap --pc-sampling-unit time --pc-sampling-interval 1 --output-format json -- <application_path>
The preceding command generates a JSON file with the comprehensive output. Here is a trimmed down output with multiple records:
.. code-block:: text
{
"pc_sample_host_trap": [
{
"record": {
"hw_id": {
"chiplet": 0,
"wave_id": 0,
"simd_id": 2,
"pipe_id": 0,
"cu_or_wgp_id": 1,
"shader_array_id": 0,
"shader_engine_id": 2,
"workgroup_id": 0,
"vm_id": 3,
"queue_id": 2,
"microengine_id": 1
},
"pc": {
"code_object_id": 1,
"code_object_offset": 20228
},
"exec_mask": 18446744073709551615,
"timestamp": 51040126667689,
"dispatch_id": 1,
"corr_id": {
"internal": 1,
"external": 0
},
"wrkgrp_id": {
"x": 182,
"y": 0,
"z": 0
},
"wave_in_grp": 1
},
"inst_index": 0
},
{
"record": {
"hw_id": {
"chiplet": 0,
"wave_id": 0,
"simd_id": 2,
"pipe_id": 0,
"cu_or_wgp_id": 0,
"shader_array_id": 0,
"shader_engine_id": 2,
"workgroup_id": 0,
"vm_id": 3,
"queue_id": 2,
"microengine_id": 1
},
"pc": {
"code_object_id": 1,
"code_object_offset": 20236
},
"exec_mask": 18446744073709551615,
"timestamp": 51040126667689,
"dispatch_id": 1,
"corr_id": {
"internal": 1,
"external": 0
},
"wrkgrp_id": {
"x": 158,
"y": 0,
"z": 0
},
"wave_in_grp": 2
},
"inst_index": 1
}
]
}
For description of the fields in the JSON output, see :ref:`output-file-fields`.
An Arbitrary Host-Trap PC Sampling Skid
===============================================
Host-Trap PC sampling is a software-based technique that utilizes a background kernel thread
to periodically interrupt running waves in order to capture the program counter (PC).
This method is effective for gathering performance data without requiring specialized hardware
to snapshot the waves. However, it has limitations due to the potential delay between
when a wave receives an interrupt and when it processes the interrupt to capture the PC.
This delay can lead to a sampling skid, where the PC samples may be attributed to instructions
that are up to two instructions away from the actual source of latency.
This results in a non-precise intra-kernel sampling method.
When analyzing an application profile generated by host-trap PC sampling,
developers should consider not only the reported most costly instruction but
also the instructions immediately preceding or following it.
If the costly instruction is near a branch instruction, it is important
to also consider the instruction targeted by the branch and the one immediately following it.
To address the limitations of host-trap sampling, the hardware-based stochastic PC sampling method
has been developed. This method provides precise intra-kernel sampling with zero sampling skid,
offering more accurate performance insights.
It is important to note that the skid issue inherent in host-trap PC sampling will not be resolved
in its current form. Therefore, users are encouraged to adopt stochastic PC sampling,
starting with the GFX942 architecture, to achieve more precise performance profiling.
Hardware-Based (Stochastic) PC Sampling Method
===============================================
The new ``ROCPROFILER_PC_SAMPLING_METHOD_STOCHASTIC`` has been introduced for gfx942 architecture.
It employs a specific hardware for probing waves actively running on GPU.
Beside information already provided with ``ROCPROFILER_PC_SAMPLING_METHOD_HOST_TRAP`` useful for determining hot-spots within the kernel,
it delivers additional information that tells whether a sampled wave issued an instruction represented with particular PC.
If not, it provides the reason for not issuing the instruction (stall reason).
This type of information is particularly useful for understanding stalls during the kernel execution.
To use this method on gfx942, we recommend listing available PC sampling configurations to verify if the latest ROCm stack is installed
on the system by running:
.. code-block:: bash
rocprofv3 -L
Output similar to the following indicates that the ``ROCPROFILER_PC_SAMPLING_METHOD_STOCHASTIC`` method is available:
.. code-block:: bash
GPU:1
NAME:gfx942
configs:
Method :stochastic
Unit :cycle
Min_Interval :256
Max_Interval :2147483648
Flags :interval pow2
Please note that on gfx942, `ROCPROFILER_PC_SAMPLING_METHOD_STOCHASTIC` requires intervals to be specified in cycles, whose values are powers of 2
To profile a gfx942 accelerated application with ``ROCPROFILER_PC_SAMPLING_METHOD_STOCHASTIC`` PC sampling, one can use the following command:
.. code-block:: bash
rocprofv3 --pc-sampling-beta-enabled --pc-sampling-method stochastic --pc-sampling-unit cycles --pc-sampling-interval 1048576 --output-format csv, json -- <application_path>
The previous command serializes samples in both CSV and JSON output formats in the ``pc_sampling_stochastic.csv`` and ``out_results.json`` files, respectively.
Comparing the ``pc_sampling_stochastic.csv`` to ``pc_sampling_host_trap`` from previous section, one can notice that the ``ROCPROFILER_PC_SAMPLING_METHOD_STOCHASTIC`` method
generates additional fields:
- ``Wave_Issued_Instruction``: Indicates whether the wave issued an instruction (value 1) represented with particular PC or not (value 0)
- ``Instruction_Type``: If the value of ``Wave_Issued_Instruction`` is 1, this fields indicates the type of the issued instruction. Otherwise, this fields irrelevant.
- ``Stall_Reason``: If the value of ``Wave_Issued_Instruction`` is 0, this fields indicates the reason for not issuing the instruction (stall reason). Otherwise, this field is irrelevant.
- ``Wave_Count``: Total number of waves actively running on a compute unit when the sample was generated.
.. csv-table:: PC sampling stochastic with debug symbols
:file: /data/pc_sampling_stochastic_debug.csv
:widths: 20,10,10,10,10,20,10,20,20,10
:header-rows: 1
Similarly, ``ROCPROFILER_PC_SAMPLING_METHOD_STOCHASTIC`` method delivers additional information to every sample in the JSON output.
The following snippet shows one sample from ``out_results.json`` file.
.. code-block:: text
{
"record": {
"flags": {
"has_mem_cnt": 0
},
"hw_id": {
"chiplet": 4,
"wave_id": 0,
"simd_id": 2,
"pipe_id": 3,
"cu_or_wgp_id": 1,
"shader_array_id": 0,
"shader_engine_id": 3,
"workgroup_id": 0,
"vm_id": 3,
"queue_id": 2,
"microengine_id": 1
},
"pc": {
"code_object_id": 2,
"code_object_offset": 13880
},
"exec_mask": 18446744073709551615,
"timestamp": 390705261924637,
"dispatch_id": 29,
"corr_id": {
"internal": 29,
"external": 0
},
"wrkgrp_id": {
"x": 9,
"y": 489,
"z": 0
},
"wave_in_grp": 0,
"wave_issued": 1,
"inst_type": "ROCPROFILER_PC_SAMPLING_INSTRUCTION_TYPE_VALU",
"wave_cnt": 6,
"snapshot": {
"stall_reason": "ROCPROFILER_PC_SAMPLING_INSTRUCTION_NOT_ISSUED_REASON_OTHER_WAIT",
"dual_issue_valu": 0,
"arb_state_issue_valu": 1,
"arb_state_issue_matrix": 0,
"arb_state_issue_lds": 0,
"arb_state_issue_lds_direct": 0,
"arb_state_issue_scalar": 0,
"arb_state_issue_vmem_tex": 0,
"arb_state_issue_flat": 0,
"arb_state_issue_exp": 0,
"arb_state_issue_misc": 0,
"arb_state_issue_brmsg": 0,
"arb_state_stall_valu": 0,
"arb_state_stall_matrix": 0,
"arb_state_stall_lds": 0,
"arb_state_stall_lds_direct": 0,
"arb_state_stall_scalar": 0,
"arb_state_stall_vmem_tex": 0,
"arb_state_stall_flat": 0,
"arb_state_stall_exp": 0,
"arb_state_stall_misc": 0,
"arb_state_stall_brmsg": 0
}
},
"inst_index": 1
},
Fields starting with ``arb_state_`` are of particular interest as they indicate the state of the arbiter at the time of sampling.
Namely, ``arb_state_issue_`` fields indicate what type of instructions arbiter issued at the time of sampling.
On the other hand, ``arb_state_stall_`` fields indicate what type of instructions were stalled at the time of sampling.
This information is useful for understanding how many instructions per cycle (IPC) are issued.
projects/rocprofiler-sdk/source/docs/how-to/using-rocpd-output-format.rst
+193
查看文件
@@ -0,0 +1,193 @@
.. meta::
:description: "ROCprofiler-SDK is a tooling infrastructure for profiling general-purpose GPU compute applications running on the ROCm software."
:keywords: "ROCprofiler-SDK, ROCProfiler-SDK output formats, rocpd, SQLite3, CSV, JSON, PFTrace, OTF2"
.. _using-rocpd-output-format:
=========================
Using rocpd Output Format
=========================
``rocprofv3`` supports the following output formats:
- **rocpd** (SQLite3 Database, Default)
- **CSV**
- **JSON** (Custom format for programmatic analysis only)
- **PFTrace** (Perfetto trace for visualization with Perfetto)
- **OTF2** (Open Trace Format for visualization with compatible third-party tools)
The ``rocpd`` output format is the default for ``rocprofv3``. It stores profiling results in a SQLite3 database, providing a structured and efficient way to analyze and post-process profiling data. This format allows users to query and manipulate profiling data using SQL, making it easy to extract specific information or perform complex analyses.
Features
++++++++
- **Rich Data Model**: Stores all collected profiling data, including traces, counters, and metadata, in a single `.db` (SQLite3) file.
- **Programmatic Access**: Can be queried using standard SQL tools or libraries (e.g., `sqlite3` CLI, Python's `sqlite3` module).
- **Post-Processing**: Enables advanced analysis and visualization using custom scripts or third-party tools that support SQLite3.
Generating rocpd Output
+++++++++++++++++++++++
To generate output in rocpd format, simply use:
.. code-block:: bash
rocprofv3 --hip-trace -- <application>
Or use the ``--output-format`` option with ``rocpd``:
.. code-block:: bash
rocprofv3 --hip-trace --output-format rocpd -- <application>
The output will be saved as ``%hostname%/%pid%_results.db``, where ``%hostname%`` is the name of the host machine and ``%pid%`` is the process ID of the application being profiled.
Converting rocpd to Other Formats
+++++++++++++++++++++++++++++++++
The ``rocpd`` output format can be converted to other formats for further analysis or visualization.
First, ensure the ``rocpd`` Python module is available in your environment:
.. code-block:: bash
export PYTHONPATH=<install-path>/lib/pythonX.Y/site-packages:$PYTHONPATH
where ``<install-path>`` is the ROCm installation path (usually ``/opt/rocm-<major.minor.patch>``), and ``X.Y`` is your Python version.
Once the ``rocpd`` module is available, use the ``rocpd convert`` command to convert the output to other formats.
Convert to CSV format:
.. code-block:: bash
python3 -m rocpd convert -i <input-file>.db --output-format csv
The converted CSV will be saved as ``rocpd-output-data/out_hip_api_trace.csv`` in the current working directory.
Convert to OTF2 format:
.. code-block:: bash
python3 -m rocpd convert -i <input-file>.db --output-format otf2
Convert to PFTrace format:
.. code-block:: bash
python3 -m rocpd convert -i <input-file>.db --output-format pftrace
rocpd convert Command-Line Options
++++++++++++++++++++++++++++++++++
.. code-block:: none
usage: rocpd convert [-h] -i INPUT [INPUT ...] -f {csv,pftrace,otf2} [{csv,pftrace,otf2} ...]
[-o OUTPUT_FILE] [-d OUTPUT_PATH] [--kernel-rename]
[--agent-index-value {absolute,relative,type-relative}]
[--perfetto-backend {inprocess,system}]
[--perfetto-buffer-fill-policy {discard,ring_buffer}]
[--perfetto-buffer-size KB] [--perfetto-shmem-size-hint KB]
[--group-by-queue]
[--start START | --start-marker START_MARKER]
[--end END | --end-marker END_MARKER]
[--inclusive INCLUSIVE]
Options
-------
**Required Arguments:**
- ``-i INPUT [INPUT ...]``, ``--input INPUT [INPUT ...]``
Input path and filename to one or more database(s), separated by spaces.
- ``-f {csv,pftrace,otf2} [{csv,pftrace,otf2} ...]``, ``--output-format {csv,pftrace,otf2} [{csv,pftrace,otf2} ...]``
Specify one or more output formats. Supported: ``csv``, ``pftrace``, ``otf2``.
**I/O Options:**
- ``-o OUTPUT_FILE``, ``--output-file OUTPUT_FILE``
Sets the base output file name (default: ``out``).
- ``-d OUTPUT_PATH``, ``--output-path OUTPUT_PATH``
Sets the output directory (default: ``./rocpd-output-data``).
**Kernel Naming Options:**
- ``--kernel-rename``
Use ROCTx marker names instead of kernel names.
**Generic Options:**
- ``--agent-index-value {absolute,relative,type-relative}``
Device identification format in output:
- ``absolute``: Uses node_id (e.g., Agent-0, Agent-2, Agent-4), ignoring cgroups.
- ``relative``: Uses logical_node_id (e.g., Agent-0, Agent-1, Agent-2), considering cgroups. *(Default)*
- ``type-relative``: Uses logical_node_type_id (e.g., CPU-0, GPU-0, GPU-1), numbering resets for each device type.
**Perfetto Trace (pftrace) Options:**
- ``--perfetto-backend {inprocess,system}``
Perfetto data collection backend. ``system`` mode requires running ``traced`` and ``perfetto`` daemons (default: ``inprocess``).
- ``--perfetto-buffer-fill-policy {discard,ring_buffer}``
Policy for handling new records when buffer is full (default: ``discard``).
- ``--perfetto-buffer-size KB``
Buffer size for perfetto output in KB (default: 1 GB).
- ``--perfetto-shmem-size-hint KB``
Perfetto shared memory size hint in KB (default: 64 KB).
- ``--group-by-queue``
Display HIP streams that kernels and memory copy operations are submitted to, rather than HSA queues.
**Time Window Options:**
- ``--start START``
Start time as percentage or nanoseconds from trace file (e.g., ``50%`` or ``781470909013049``).
- ``--start-marker START_MARKER``
Named marker event to use as window start point.
- ``--end END``
End time as percentage or nanoseconds from trace file (e.g., ``75%`` or ``3543724246381057``).
- ``--end-marker END_MARKER``
Named marker event to use as window end point.
- ``--inclusive INCLUSIVE``
``True``: include events if START or END in window; ``False``: only if BOTH in window (default: ``True``).
**Help:**
- ``-h``, ``--help``
Show help message and exit.
Examples
++++++++
Convert one database to Perfetto trace:
.. code-block:: bash
python3 -m rocpd convert -i db1.db --output-format pftrace
Convert two databases to Perfetto trace, set output path and filename, and limit to last 70% of trace:
.. code-block:: bash
python3 -m rocpd convert -i db1.db db2.db --output-format pftrace -d "./output/" -o "twoFileTraces" --start 30% --end 100%
Convert six databases to CSV and Perfetto trace formats:
.. code-block:: bash
python3 -m rocpd convert -i db{0..5}.db --output-format csv pftrace -d "~/output_folder/" -o "sixFileTraces"
Convert two databases to CSV, OTF2, and Perfetto trace formats:
.. code-block:: bash
python3 -m rocpd convert -i db{3,4}.db --output-format csv otf2 pftrace
projects/rocprofiler-sdk/source/docs/how-to/using-rocprofiler-sdk-roctx.rst
+293
查看文件
@@ -0,0 +1,293 @@
.. meta::
:description: Documentation for the usage of rocprofiler-sdk-roctx library
:keywords: ROCprofiler-SDK tool, using-rocprofiler-sdk-roctx library, roctx, markers, ranges, rocprofv3, rocprofv3 tool usage, Using rocprofv3, ROCprofiler-SDK command line tool, marker-trace
.. _using-rocprofiler-sdk-roctx:
============
Using ROCTx
============
ROCTx is an AMD tools extension library, a cross platform API for annotating code with markers and ranges. The ROCTx API is written in C++.
In certain situations, such as debugging performance issues in large-scale GPU programs, API-level tracing might be too fine-grained to provide an overview of the program execution.
In such cases, it is helpful to define specific tasks to be traced. To specify the tasks for tracing, enclose the respective source code with the API calls provided by the ROCTx library.
This process is also known as instrumentation.
ROCTx annotations
++++++++++++++++++
ROCTx provides two types of annotations: markers and ranges.
Markers
========
Markers are used to insert a marker in the code with a message. Creating markers helps you see when a line of code is executed.
Ranges
=======
Ranges are used to define the scope of code for instrumentation using enclosing API calls.
A range is a programmer-defined task that has a well-defined start and end code scope.
You can further refine the scope specified within a range using nested ranges. ``rocprofv3`` also reports the timelines for these nested ranges.
These are the two types of ranges:
- **Push and Pop:** These can be nested to form a stack. The Pop call is automatically associated with a prior Push call on the same thread.
- **Start and End:** These may overlap with other ranges arbitrarily. The Start call returns a handle that must be passed to the End call. These ranges can start and end on different threads.
ROCTx APIs
===========
Here is the list of useful APIs for code instrumentation:
- ``roctxMark``: Inserts a marker in the code with a message. Creating marks help you see when a line of code is executed.
- ``roctxRangeStart``: Starts a range. Different threads can start ranges.
- ``roctxRangePush``: Starts a new nested range.
- ``roctxRangePop``: Stops the current nested range.
- ``roctxRangeStop``: Stops the given range.
- ``roctxProfilerPause``: Requests any currently running profiling tool to stop data collection.
- ``roctxProfilerResume``: Requests any currently running profiling tool to resume data collection.
- ``roctxGetThreadId``: Retrieves the ID for the current thread identical to the ID received using ``rocprofiler_get_thread_id(rocprofiler_thread_id_t*)``.
- ``roctxNameOsThread``: Labels the current CPU OS thread in the profiling tool output with the provided name.
- ``roctxNameHsaAgent``: Labels the given HSA agent in the profiling tool output with the provided name.
- ``roctxNameHipDevice``: Labels the HIP device ID in the profiling tool output with the provided name.
- ``roctxNameHipStream``: Labels the given HIP stream in the profiling tool output with the provided name.
Using ROCTx in the application
+++++++++++++++++++++++++++++++
The following sample code from the MatrixTranspose application shows the usage of ROCTx APIs:
.. code-block:: bash
#include <rocprofiler-sdk-roctx/roctx.h>
roctxMark("before hipLaunchKernel");
int rangeId = roctxRangeStart("hipLaunchKernel range");
roctxRangePush("hipLaunchKernel");
// Launching kernel from host
hipLaunchKernelGGL(matrixTranspose, dim3(WIDTH/THREADS_PER_BLOCK_X, WIDTH/THREADS_PER_BLOCK_Y), dim3(THREADS_PER_BLOCK_X, THREADS_PER_BLOCK_Y), 0,0,gpuTransposeMatrix,gpuMatrix, WIDTH);
roctxMark("after hipLaunchKernel");
// Memory transfer from device to host
roctxRangePush("hipMemcpy");
hipMemcpy(TransposeMatrix, gpuTransposeMatrix, NUM * sizeof(float), hipMemcpyDeviceToHost);
roctxRangePop(); // for "hipMemcpy"
roctxRangePop(); // for "hipLaunchKernel"
roctxRangeStop(rangeId);
To trace the API calls enclosed within the range, use:
.. code-block:: bash
rocprofv3 --marker-trace --output-format csv -- <application_path>
Running the preceding command generates a ``marker_api_trace.csv`` file prefixed with the process ID.
.. code-block:: shell
$ cat 210_marker_api_trace.csv
Here are the contents of ``marker_api_trace.csv`` file:
.. csv-table:: Marker api trace
:file: /data/marker_api_trace.csv
:widths: 10,10,10,10,10,20,20
:header-rows: 1
For the description of the fields in the output file, see :ref:`output-file-fields`.
``roctxProfilerPause`` and ``roctxProfilerResume`` can be used to hide the calls between them. This is useful when you want to hide the calls that are not relevant to your profiling session.
.. code-block:: bash
#include <rocprofiler-sdk-roctx/roctx.h>
// Memory transfer from host to device
HIP_API_CALL(hipMemcpy(gpuMatrix, Matrix, NUM * sizeof(float), hipMemcpyHostToDevice));
auto tid = roctx_thread_id_t{};
roctxGetThreadId(&tid);
roctxProfilerPause(tid);
// Memory transfer that should be hidden by profiling tool
HIP_API_CALL(
hipMemcpy(gpuTransposeMatrix, gpuMatrix, NUM * sizeof(float), hipMemcpyDeviceToDevice));
roctxProfilerResume(tid);
// Launching kernel from host
hipLaunchKernelGGL(matrixTranspose,
dim3(WIDTH / THREADS_PER_BLOCK_X, WIDTH / THREADS_PER_BLOCK_Y),
dim3(THREADS_PER_BLOCK_X, THREADS_PER_BLOCK_Y),
0,
0,
gpuTransposeMatrix,
gpuMatrix,
WIDTH);
// Memory transfer from device to host
HIP_API_CALL(
hipMemcpy(TransposeMatrix, gpuTransposeMatrix, NUM * sizeof(float), hipMemcpyDeviceToHost));
To trace the preceding code, use:
.. code-block:: shell
rocprofv3 --marker-trace --hip-trace --output-format csv -- <application_path>
The preceding command generates a ``hip_api_trace.csv`` file prefixed with the process ID. The file contains two ``hipMemcpy`` calls with the in-between ``hipMemcpyDeviceToHost`` call hidden .
.. code-block:: shell
"Domain","Function","Process_Id","Thread_Id","Correlation_Id","Start_Timestamp","End_Timestamp"
"HIP_COMPILER_API","__hipRegisterFatBinary",1643920,1643920,1,320301257609216,320301257636427
"HIP_COMPILER_API","__hipRegisterFunction",1643920,1643920,2,320301257650707,320301257678857
"HIP_RUNTIME_API","hipGetDevicePropertiesR0600",1643920,1643920,4,320301258114239,320301337764472
"HIP_RUNTIME_API","hipMalloc",1643920,1643920,5,320301338073823,320301338247374
"HIP_RUNTIME_API","hipMalloc",1643920,1643920,6,320301338248284,320301338399595
"HIP_RUNTIME_API","hipMemcpy",1643920,1643920,7,320301338410995,320301631549262
"HIP_COMPILER_API","__hipPushCallConfiguration",1643920,1643920,10,320301632131175,320301632134215
"HIP_COMPILER_API","__hipPopCallConfiguration",1643920,1643920,11,320301632137745,320301632139735
"HIP_RUNTIME_API","hipLaunchKernel",1643920,1643920,12,320301632142615,320301632898289
"HIP_RUNTIME_API","hipMemcpy",1643920,1643920,14,320301632901249,320301633934395
"HIP_RUNTIME_API","hipFree",1643920,1643920,15,320301643320908,320301643511479
"HIP_RUNTIME_API","hipFree",1643920,1643920,16,320301643512629,320301643585639
Resource naming
++++++++++++++++
``ROCTx`` provides APIs to rename certain resources in the output generated by the profiling tool. You can pass the desired label for a specific resource in the output as an argument to the API. Note that ROCprofiler-SDK doesn't provide any explicit support for how profiling tools handle this request. Support for this capability is tool-specific.
The following table lists the APIs available for labeling the given resources:
.. |br| raw:: html
<br />
.. list-table:: resource naming
:header-rows: 1
* - Resource
- API
- Description
* - OS thread
- ``roctxNameOsThread(const char* name)``
- Labels the current CPU OS thread with the given name in the output. Note that ROCTx does NOT rename the thread using ``pthread_setname_np``.
* - HIP runtime
- | ``roctxNameHipDevice(const char* name, int device_id)`` |br| |br|
| ``roctxNameHipStream(const char* name, const struct ihipStream_t* stream)``
- | Labels the given HIP device ID with the given name in the output. |br| |br|
| Labels the given HIP stream ID with the given name in the output.
* - HSA runtime
- ``roctxNameHsaAgent(const char* name, const struct hsa_agent_s*)``
- Labels the given HSA agent with the given name in the output.
Using ROCTx in the python application
++++++++++++++++++++++++++++++++++++++
ROCTx APIs can be used in a python application using the ``roctx`` module. The APIs are available as functions in the module. The API names are prefixed with ``roctx`` to avoid name conflicts with other libraries.
The following sample code from the MatrixTranspose application shows the usage of ROCTx APIs in a python application:
.. code-block:: python
import os
import roctx
import random
from roctx.context_decorators import RoctxRange
_prefix = os.path.basename(__file__)
@RoctxRange("matrix_transpose")
def matrix_transpose(matrix):
nrows = len(matrix)
ncols = len(matrix[0]) if nrows > 0 else 0
with RoctxRange(f"transpose(nrows={nrows}, ncols={ncols})"):
# Transpose the matrix
transposed = [[matrix[j][i] for j in range(nrows)] for i in range(ncols)]
return transposed
def generate_matrix(rows, cols):
with RoctxRange(f"generate_matrix(rows={rows}, cols={cols})"):
return [[random.randint(0, 100) for _ in range(cols)] for _ in range(rows)]
def run(rows, cols):
idx = roctx.rangeStart(f"run(rows={rows}, cols={cols})")
matrix = generate_matrix(rows, cols)
transposed = matrix_transpose(matrix)
roctx.rangeStop(idx)
return matrix, transposed
if __name__ == "__main__":
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("-r", "--rows", type=int, default=4, help="Number of rows")
parser.add_argument("-c", "--cols", type=int, default=5, help="Number of columns")
args = parser.parse_args()
roctx.mark(f"MatrixTranspose: rows={args.rows}, cols={args.cols}")
with RoctxRange("main"):
matrix, transposed = run(args.rows, args.cols)
print(f"[{_prefix}] Original matrix:")
for row in matrix:
print(row)
print(f"\n[{_prefix}] Transposed matrix:")
for row in transposed:
print(row)
Before using the ``roctx`` module for python application, ensure that the ``roctx`` module is built, installed and available in your python environment.
An example to build and install ``roctx`` module is as follows:
.. code-block:: shell
cmake -B build-sdk -DCMAKE_INSTALL_PREFIX=/opt/rocm -DROCPROFILER_PYTHON_VERSIONS="3.10" -DCMAKE_PREFIX_PATH=/opt/rocm
If you are using a different python version, replace ``3.10`` with the appropriate version in the above command.
Multiple python versions can be specified in the ``ROCPROFILER_PYTHON_VERSIONS`` variable. The roctx module will be built and installed for all the specified python versions.
.. code-block:: shell
``cmake -B build-sdk -DCMAKE_INSTALL_PREFIX=/opt/rocm -DROCPROFILER_PYTHON_VERSIONS="3.8;3.9;3.10;3.11;3.12" -DCMAKE_PREFIX_PATH=/opt/rocm``
Based on the python major.minor version and the roctx module install path ("/opt/rocm" in above example), set the ``PYTHONPATH`` environment variable to include the path to the ``roctx`` module.
.. code-block:: shell
export PYTHONPATH="<install-path>/lib/pythonX.Y/site-packages:$PYTHONPATH"
Above example will install the roctx module in ``/opt/rocm/lib/python3.10/site-packages``, set the ``PYTHONPATH`` as follows:
.. code-block:: shell
export PYTHONPATH=/opt/rocm/lib/python3.10/site-packages:$PYTHONPATH
Once the ``PYTHONPATH`` is set, user should be able to import the `roctx` package:
.. code-block:: shell
python3 -c "import roctx"
User can profile the python application which is annotated with ROCTx markers using ``rocprofv3`` as follows:
.. code-block:: shell
rocprofv3 --marker-trace --output-format csv -- $(which python) <python_application_path>
The preceding command generates a ``marker_api_trace.csv`` file prefixed with the process ID.
.. csv-table:: Marker api trace for python application
:file: /data/python_bindings.csv
:widths: 10,10,10,10,10,20,20
:header-rows: 1
projects/rocprofiler-sdk/source/docs/how-to/using-rocprofv3-avail.rst
+111
查看文件
@@ -0,0 +1,111 @@
.. meta::
:description: Documentation of the usage of rocprofv3-avail
:keywords: ROCprofiler-SDK tool usage, rocprofv3-avail usage, rocprofv3 user manual, rocprofv3 usage, rocprofv3 user guide, using rocprofv3, ROCprofiler-SDK tool user guide, ROCprofiler-SDK tool user manual, using ROCprofiler-SDK tool, ROCprofiler-SDK command-line tool, ROCprofiler-SDK CLI, ROCprofiler-SDK command line tool
.. _using-rocprofv3-avail:
======================
Using rocprofv3-avail
======================
``rocprofv3-avail`` is a CLI tool that helps you to query the features supported by the hardware and Rocprofiler SDK.
The following sections demonstrate the use of ``rocprofv3-avail`` for querying features using various command-line options.
``rocprofv3-avail`` is installed with ROCm under ``/opt/rocm/bin``. To use the tool from anywhere in the system, export ``PATH`` variable:
.. code-block:: bash
export PATH=$PATH:/opt/rocm/bin
.. _rocprofv3-avail_cli-options:
Command-line options
--------------------
The following table lists ``rocprofv3-avail`` command-line options categorized according to their purpose.
.. # COMMENT: The following lines define a line break for use in the table below.
.. |br| raw:: html
<br />
.. list-table:: rocprofv3-avail options
:header-rows: 1
* - Purpose
- Option
- Description
* - avail-aptions commands
- | ``info``
| ``list``
| ``pmc-check``
- | Info options for detailed information of counters, agents, and pc-sampling configurations.
| List options for hardware counters, agents and pc-sampling support.
| Checking if a set of counters can be collected together on agent.
.. code-block:: bash
rocprofv3-avail list
The preceding command generates an output listing agents and hardware counters.
.. code-block:: bash
rocprofv3-avail list --agent
The preceding command generates an output listing basic info for all agents, if used with ``-d``, only basic info for device ``-d`` is listed.
Following is the sample output
.. code-block:: bash
rocprofv3-avail list --pmc
The preceding command generates an output listing counters for all agents, if used with ``-d``, only counters on the ``-d`` device is listed.
Output contains following information: logical node id, name and list of PMC counters supported on the agent.
.. code-block:: bash
rocprofv3-avail list --pc-sampling
The preceding command generates an output listing agents that supports any kind of PC Sampling. ``-d`` option is not applicable here.
.. code-block:: bash
rocprofv3-avail info
The preceding command generates an output with agent information and listing all counters supported on each agent.
.. code-block:: bash
rocprofv3-avail info --pmc
The preceding command generates an output with the pmc info, if used with ``-d`` information of pmc for device ``-d`` is generated.
Output includes the following information: logical node id, name, counter_name, description of the counter, dimensions, block/expression for every counter.
.. code-block:: bash
rocprofv3-avail info --pc-sampling
The preceding command generates list of supported PC sampling configurations for each agent that supports PC sampling. ``-d`` option is not applicable here.
Output has following information: logical node id, method supported, unit, minimum sampling interval, maximum sampling interval
flags.
.. code-block:: bash
rocprofv3-avail pmc-check [pmc [pmc...]]
The preceding command checks if the pmc can be collected together
.. code-block:: bash
rocprofv3-avail pmc-check -d 0 <pmc1> <pmc2> <pmc3>:device=1
The preceding command checks if the pmc1 and pmc2 can be collected together on agent 0 and pmc3 on agent 1
.. note::
All commands writes to the standard output.
projects/rocprofiler-sdk/source/docs/how-to/using-rocprofv3-with-mpi.rst
+170
查看文件
@@ -0,0 +1,170 @@
.. meta::
:description: Documentation of the MPI usage for rocprofv3
:keywords: ROCprofiler-SDK tool, mpirun, rocprofv3, rocprofv3 tool usage, mpich, ROCprofiler-SDK command line tool, ROCprofiler-SDK CLI
.. _using-rocprofv3-with-mpi:
Using rocprofv3 with MPI
+++++++++++++++++++++++++++++
Message Passing Interface (MPI) is a standardized and portable message-passing system designed to function on a wide variety of parallel computing architectures. MPI is widely used for developing parallel applications and is considered the de facto standard for communication in high-performance computing (HPC) environments.
MPI applications are parallel programs that run across multiple processes, which can be distributed over one or more nodes.
For MPI applications or other job launchers such as `SLURM <https://slurm.schedmd.com/documentation.html>`_, place ``rocprofv3`` inside the job launcher. The following example demonstrates how to use ``rocprofv3`` with MPI:
.. code-block:: bash
mpirun -n 4 rocprofv3 --hip-trace --output-format csv -- <application_path>
The preceding command runs the application with ``rocprofv3`` and generates the trace file for each rank. The trace files are prefixed with the process ID.
.. code-block:: bash
2293213_agent_info.csv
2293213_hip_api_trace.csv
2293214_agent_info.csv
2293214_hip_api_trace.csv
2293212_agent_info.csv
2293212_hip_api_trace.csv
2293215_agent_info.csv
2293215_hip_api_trace.csv
Since the data collection is performed in-process, it's ideal to collect data from within the processes launched by MPI. When ``rocprofv3`` is run outside of ``mpirun``, the tool library is loaded into the `mpirun` executable..
Collecting data outside of ``mpirun`` works but fetches agent info for the ``mpirun`` process too. For example:
.. code-block:: bash
rocprofv3 --hip-trace -d %h.%p.%env{OMPI_COMM_WORLD_RANK}% --output-format csv -- mpirun -n 2 <application_path>
In the preceding example, an extra agent info file is generated for the ``mpirun`` process. The trace files are prefixed with the hostname, process ID, and the MPI rank.
.. code-block:: bash
ubuntu-latest.3000020.1/3000020_agent_info.csv
ubuntu-latest.3000020.0/3000019_agent_info.csv
ubuntu-latest.3000020.1/3000020_hip_api_trace.csv
ubuntu-latest.3000020.0/3000019_hip_api_trace.csv
ROCTx annotations
===================
For an MPI application, you can use ROCTx annotations to mark the start and end of the MPI code region. The following example demonstrates how to use ROCTx annotations with MPI:
.. code-block:: cpp
#include <roctx.h>
#include <mpi.h>
...
void run(int rank, int tid, int dev_id, int argc, char** argv)
{
auto roctx_run_id = roctxRangeStart("run");
const auto mark = [rank, tid, dev_id](std::string_view suffix) {
auto _ss = std::stringstream{};
_ss << "run/rank-" << rank << "/thread-" << tid << "/device-" << dev_id << "/" << suffix;
roctxMark(_ss.str().c_str());
};
mark("begin");
constexpr unsigned int M = 4960 * 2;
constexpr unsigned int N = 4960 * 2;
unsigned long long nitr = 0;
unsigned long long nsync = 0;
if(argc > 2) nitr = atoll(argv[2]);
if(argc > 3) nsync = atoll(argv[3]);
hipStream_t stream = {};
printf("[transpose] Rank %i, thread %i assigned to device %i\n", rank, tid, dev_id);
HIP_API_CALL(hipSetDevice(dev_id));
HIP_API_CALL(hipStreamCreate(&stream));
auto_lock_t _lk{print_lock};
std::cout << "[transpose][" << rank << "][" << tid << "] M: " << M << " N: " << N << std::endl;
_lk.unlock();
std::default_random_engine _engine{std::random_device{}() * (rank + 1) * (tid + 1)};
std::uniform_int_distribution<int> _dist{0, 1000};
...
auto t1 = std::chrono::high_resolution_clock::now();
for(size_t i = 0; i < nitr; ++i)
{
roctxRangePush("run/iteration");
transpose<<<grid, block, 0, stream>>>(in, out, M, N);
check_hip_error();
if(i % nsync == (nsync - 1))
{
roctxRangePush("run/iteration/sync");
HIP_API_CALL(hipStreamSynchronize(stream));
roctxRangePop();
}
roctxRangePop();
}
auto t2 = std::chrono::high_resolution_clock::now();
HIP_API_CALL(hipStreamSynchronize(stream));
HIP_API_CALL(hipMemcpyAsync(out_matrix, out, size, hipMemcpyDeviceToHost, stream));
double time = std::chrono::duration_cast<std::chrono::duration<double>>(t2 - t1).count();
float GB = (float) size * nitr * 2 / (1 << 30);
print_lock.lock();
std::cout << "[transpose][" << rank << "][" << tid << "] Runtime of transpose is " << time
<< " sec\n";
std::cout << "[transpose][" << rank << "][" << tid
<< "] The average performance of transpose is " << GB / time << " GBytes/sec"
<< std::endl;
print_lock.unlock();
...
mark("end");
roctxRangeStop(roctx_run_id);
}
This preceding sample generates output similar to the following:
.. code-block:: shell
"MARKER_CORE_API","run/rank-0/thread-0/device-0/begin",2936128,2936128,5,432927100747635,432927100747635
"MARKER_CORE_API","run/rank-0/thread-1/device-1/begin",2936128,2936397,7,432927100811475,432927100811475
"MARKER_CORE_API","run/iteration",2936128,2936397,22,432928615598809,432928648197081
"MARKER_CORE_API","run/iteration",2936128,2936397,61,432928648229081,432928648234041
"MARKER_CORE_API","run/iteration",2936128,2936397,67,432928648234701,432928648239621
"MARKER_CORE_API","run/iteration",2936128,2936397,73,432928648239971,432928648244141
"MARKER_CORE_API","run/iteration/sync",2936128,2936397,84,432928648249791,432928664871094
...
"MARKER_CORE_API","run/iteration",2936128,2936128,6313,432929397644269,432929397648369
"MARKER_CORE_API","run/iteration/sync",2936128,2936128,6324,432929397653119,432929401455250
"MARKER_CORE_API","run/iteration",2936128,2936128,6319,432929397648779,432929401455640
"MARKER_CORE_API","run/rank-0/thread-1/device-1/end",2936128,2936397,6339,432929527301990,432929527301990
"MARKER_CORE_API","run",2936128,2936397,6,432927100787035,432929527313480
"MARKER_CORE_API","run/rank-0/thread-0/device-0/end",2936128,2936128,6342,432929612438185,432929612438185
"MARKER_CORE_API","run",2936128,2936128,4,432927100729745,432929612448285
Output format features
=======================
To collect the profiles of the individual MPI processes, use ``rocprofv3`` with output directory option to send output to unique files.
.. code-block:: bash
mpirun -n 2 rocprofv3 --hip-trace -d %h.%p.%env{OMPI_COMM_WORLD_RANK}% --output-format csv -- <application_path>
To see the placeholders supported by the output directory option, see :ref:`output directory placeholders <output_field_format>`.
Assuming the hostname as `ubuntu-latest`, the process IDs as 3000020 and 3000019, the generated output file names are:
.. code-block:: bash
ubuntu-latest.3000020.1/ubuntu-latest/3000020_agent_info.csv
ubuntu-latest.3000019.0/ubuntu-latest/3000019_agent_info.csv
ubuntu-latest.3000020.1/ubuntu-latest/3000020_hip_api_trace.csv
ubuntu-latest.3000019.0/ubuntu-latest/3000019_hip_api_trace.csv
projects/rocprofiler-sdk/source/docs/how-to/using-rocprofv3-with-openmp.rst
+90
查看文件
@@ -0,0 +1,90 @@
.. meta::
:description: Documentation for using rocprofv3 with OpenMP applications
:keywords: ROCprofiler-SDK tool, OpenMP, rocprofv3, rocprofv3 tool usage, ROCprofiler-SDK command line tool, ROCprofiler-SDK CLI
.. _using-rocprofv3-with-openmp:
Using rocprofv3 with OpenMP
+++++++++++++++++++++++++++++
`rocprofv3` does not provide native support for profiling CPU-side OpenMP code. However, when OpenMP is used to offload computations to AMD GPUs (for example, via OpenMP target offload), `rocprofv3` can capture and profile GPU activities initiated by these offloaded regions. Note that profiling of CPU-side OpenMP parallel regions is not supported.
Example: Vector Addition Using OpenMP Offload on AMD GPUs
---------------------------------------------------------
The following example demonstrates how to perform vector addition using OpenMP target offload, enabling execution of the workload on AMD GPUs.
**Key Steps:**
- Initialize input arrays on the host.
- Offload the vector addition computation to the GPU using OpenMP directives.
- Retrieve and verify the results on the host.
.. code-block:: c
#include <stdio.h>
#include <omp.h>
#define N 1024
int main() {
float a[N], b[N], c[N];
// Initialize input arrays
for (int i = 0; i < N; ++i) {
a[i] = i * 1.0f;
b[i] = (N - i) * 1.0f;
}
// Offload vector addition to GPU
#pragma omp target teams distribute parallel for map(to: a[0:N], b[0:N]) map(from: c[0:N])
for (int i = 0; i < N; ++i) {
c[i] = a[i] + b[i];
}
// Verify results
int errors = 0;
for (int i = 0; i < N; ++i) {
if (c[i] != N * 1.0f) {
errors++;
}
}
if (errors == 0) {
printf("Vector addition successful!\\n");
} else {
printf("Vector addition failed with %d errors.\\n", errors);
}
return 0;
}
Building the OpenMP Offload Application
---------------------------------------
To compile the application for AMD GPU offload, use the following command:
.. code-block:: bash
amdclang++ -fopenmp -fopenmp-targets=amdgcn-amd-amdhsa -L/opt/rocm/lib --offload-arch=gfx9xx -o vector_add <application>
Profiling the Application with rocprofv3
----------------------------------------
To profile the GPU activity during execution, run the application with `rocprofv3`:
.. code-block:: bash
rocprofv3 -s --output-format csv -- ./vector_add
Upon execution, `rocprofv3` will generate several CSV trace files, such as:
- `<pid>_kernel_trace.csv`
- `<pid>_hsa_api_trace.csv`
- `<pid>_memory_copy_trace.csv`
- `<pid>_memory_allocation_trace.csv`
- `<pid>_scratch_memory_trace.csv`
These files contain detailed profiling information about GPU kernel execution, HSA API calls, memory operations, and more, enabling comprehensive analysis of the offloaded workload.
projects/rocprofiler-sdk/source/docs/how-to/using-rocprofv3.rst
+1921
查看文件
文件差异内容过多而无法显示 加载差异
projects/rocprofiler-sdk/source/docs/how-to/using-thread-trace.rst
+234
查看文件
@@ -0,0 +1,234 @@
.. meta::
:description: Documentation of the usage of thread trace with rocprofv3 command-line tool
:keywords: rocprofv3, rocprofv3 tool usage, Using rocprofv3, ROCprofiler-SDK command line tool, Thread Trace, SQTT, ATT, ROCprof Trace Decoder, ROCprof Compute Viewer
.. _using-thread-trace:
============================
Using thread trace
============================
Thread trace is a shader execution tracing technique capable of profiling wavefronts at the instruction timing level.
This is a low-level tracing and profiling feature that targets a single or a few kernel executions.
Thread trace features include:
* Near cycle-accurate instruction tracing
* Exact thread or wave execution path
* Wave scheduling and stall timing analysis
* Instruction and source level hotspots
* Extremely fast and granular counter collection (AMD Instinct)
Supported devices:
* AMD Instinct: MI200 and MI300 series
* AMD Radeon: gfx10, gfx11 and gfx12
Thread trace profiling is performed in the following steps:
1. Tracing (data collection) - Uses ROCprofiler-SDK thread trace service API
2. Decoding (analysis) - Uses ROCprof Trace Decoder API
3. Visualization - Requires ROCprof Compute Viewer
Tracing and decoding is handled by ``rocprofv3`` while visualization is handled by the ROCprof Compute Viewer.
Prerequisites
=========
- aqlprofile:
* ROCm 7.x build, or
* Early release can be `built from source <https://github.com/rocm/aqlprofile>`_
* Otherwise, ``rocprofv3`` throws error "INVALID_SHADER_DATA" or "Agent not supported".
- Installation of ROCprof Trace Decoder component:
* For binary files, see `ROCprof trace decoder release page <https://github.com/ROCm/rocprof-trace-decoder/releases>`_.
* Default install location is ``/opt/rocm/lib``
* For custom location, use:
* Parameter ``--att-library-path``, or
* Environment variable ``ROCPROF_ATT_LIBRARY_PATH``
.. _thread-trace-parameters:
rocprofv3 parameters for thread tracing
============================
To collect thread trace with default parameters, use:
.. code-block:: bash
rocprofv3 --att -d <output_dir> -- <application_path>
The following table lists the parameters relevant to thread tracing:
+--------------------------+---------+---------+-----------+--------------------------------------------------------------+
| Parameter | Type | Range | Typical | Description |
+==========================+=========+=========+===========+==============================================================+
| att-target-cu | Integer | 0 - 15 | 1 | Defines the CU used to gather detail tokens (WGP on Navi) |
+--------------------------+---------+---------+-----------+--------------------------------------------------------------+
| att-shader-engine-mask | Bitmask | 1 - ~0u | 0x1 | Defines the Shader Engines (SE) to be traced. Max 2^32 - 1 |
+--------------------------+---------+---------+-----------+--------------------------------------------------------------+
| att-simd-select | Integer | 0 - 0xF | gfx9: 0xF | Defines one or more SIMDs to be traced, out of four. |
| | | | Navi: 0x0 | Bitmask on GFX9 and SIMD_ID[0,3] on Navi. |
+--------------------------+---------+---------+-----------+--------------------------------------------------------------+
| kernel-iteration-range | List | | | Defines dispatch iteration of the kernel to be profiled |
+--------------------------+---------+---------+-----------+--------------------------------------------------------------+
| kernel-include-regex | String | Any | | Profiles kernel names matching the regex |
+--------------------------+---------+---------+-----------+--------------------------------------------------------------+
| kernel-exclude-regex | String | Any | | Doesn't profile kernel names matching the regex |
+--------------------------+---------+---------+-----------+--------------------------------------------------------------+
| att-buffer-size | Bytes | 1MB-2GB | 96MB | Specifies the trace buffer size. This is shared for all SEs. |
| | | | | Increase this value if the buffer tends to get full. |
+--------------------------+---------+---------+-----------+--------------------------------------------------------------+
| att-serialize-all | Bool | | False | If set to "True", turns on serialization for untraced kernels|
+--------------------------+---------+---------+-----------+--------------------------------------------------------------+
| att-perfcounter-ctrl | Integer | 1 - 32 | 2~8 | Available only in gfx9. Streams SQ performance counters to |
| | | | | the thread trace buffer in the given relative period. As |
| | | | | this uses high bandwidth, a value too low can cause or worsen|
| | | | | "Data Lost" events and warnings. |
+--------------------------+---------+---------+-----------+--------------------------------------------------------------+
| att-perfcounters | String | SQ-only | | Available only in gfx9. Specifies the list of SQ counters. |
| | | | | To list all counters, use "rocprofv3 --list-avail``. |
+--------------------------+---------+---------+-----------+--------------------------------------------------------------+
| att-activity | Integer | 1 - 16 | 5~10 | Available only in gfx9. |
| | | | | Shorthand for att-perfcounter-ctrl and the att-perfcounters |
| | | | | related to compute unit activity such as VALU, SALU, etc. |
+--------------------------+---------+---------+-----------+--------------------------------------------------------------+
For AMD Instinct accelerators, enable perfmon streaming using:
.. code-block:: bash
rocprofv3 --att --att-activity 8 -- <application_path>
For AMD Radeon, the ``simd-select`` parameter is a SIMD ID defaulting to 3. For some applications it's best to use:
.. code-block:: bash
rocprofv3 --att --att-simd-select 0x0 -- <application_path>
Using input file
===========
As explained in the preceding section, you can specify parameters on the command line or use a JSON input file:
.. code-block:: text
{
"jobs": [
{
"advanced_thread_trace": true,
"att_target_cu": 1,
"att_shader_engine_mask": "0x1",
"att_simd_select": "0xF",
"att_buffer_size": "0x6000000"
}
]
}
Thread tracing for multiple kernel instances
=============================
By default, ``rocprofv3`` enables thread trace only once per kernel instance. This implies that if an application launches the same kernel multiple times, only the first instance will be traced.
To enable thread trace for multiple kernel instances, use the ``kernel-iteration-range`` parameter.
It's recommended to use ``kernel-include-regex`` parameter to filter the desired kernel names instead of tracing everything.
.. _output-files:
rocprofv3 output files
===============
After the application finishes executing, ROCprof Trace Decoder runs automatically and the following output files are generated:
- stats_*.csv files:
* Contains a summary of instruction latency per kernel.
- ui_output_agent_{agent_id}_dispatch_{dispatch_id} directory:
* Contains detailed tracing information in the form of .json files.
* This directory can be opened using the `ROCprof Compute Viewer <https://rocm.docs.amd.com/projects/rocprof-compute-viewer/en/amd-mainline/>`_.
- Raw files:
* .att - Raw SQTT data. Can be used with the ROCprof Trace Decoder for further analysis.
* .out - Code object binaries (executable). Can be used with ISA analysis tools.
.. _csv-content:
Stats CSV
------------
Here is a sample stats_*.csv file that is generated by the rocprofv3 tool.
+---------+-------+---------------------------------------------+----------+---------+-------+------+-------------------+
| Codeobj | Vaddr | Instruction | Hitcount | Latency | Stall | Idle | Source |
+=========+=======+=============================================+==========+=========+=======+======+===================+
| 11 | 5888 | s_load_dwordx4 s[40:43], s[0:1], 0x18 | 48 | 276 | 96 | 48 | kernel.py:391 |
+---------+-------+---------------------------------------------+----------+---------+-------+------+-------------------+
| 11 | 5896 | s_load_dwordx2 s[38:39], s[0:1], 0x28 | 48 | 192 | 0 | 0 | kernel.py:391 |
+---------+-------+---------------------------------------------+----------+---------+-------+------+-------------------+
| 11 | 5904 | s_ashr_i32 s3, s2, 31 | 48 | 260 | 0 | 0 | kernel.py:395 |
+---------+-------+---------------------------------------------+----------+---------+-------+------+-------------------+
| 11 | 5908 | s_add_i32 s7, s2, s3 | 48 | 196 | 0 | 0 | kernel.py:395 |
+---------+-------+---------------------------------------------+----------+---------+-------+------+-------------------+
The columns of the stats_*.csv file are described here:
* **Codeobj:** The code object load ID assigned by ROCprofiler-SDK.
* **Vaddr:** ELF vaddr.
* **Hitcount:** The number of times a particular instruction is executed while adding all the traced waves.
* **Latency:** Total latency in cycles, defined as "Stall time + Issue time" for gfx9 or "Stall time + Execute time" for gfx10+.
* **Stall:** The total number of cycles the hardware pipe couldn't issue an instruction.
* Usually caused when the hardware unit is busy, such as TCP or LDS backpressure.
* **Idle:** The total time gap between the completion of previous instruction and the beginning of the current instruction. The idle time can be caused by:
* Arbiter loss
* Source or destination register dependency
* Instruction cache miss
* **Source:** The original source line of code assigned by the compiler.
* Requires compiling with debug symbols.
Troubleshooting
===============
For some applications, stats_*.csv file could be empty even for a valid kernel dispatch.
Thread trace is limited to a single CU per SE (``att-target-cu``). If a kernel dispatch doesn't launch enough waves to populate the whole GPU, there's a possibility of no wave getting assigned to the ``target_cu``. In such cases, there's nothing to be traced.
Here are some options to handle this:
* Launch more waves.
* Swap the ``target_cu``.
* Set the ``--att-shader-engine-mask`` to 0x11111111, or possibly to 0xFFFFFFFF
* A number too high can cause packet losses and/or lead to a full buffer.
* Set the ``HSA_CU_MASK`` to mask out all CUs but the target. For more details, see `setting CUs <https://rocm.docs.amd.com/en/latest/how-to/setting-cus.html>`_.
* If only the ``target_cu`` (or a few CUs) are not masked out, then all or most waves will be assigned to the ``target_cu``.
* This can potentially cause low performance in high-demanding kernels.