Add 'projects/rocprofiler-sdk/' from commit 'bf0fad1d5406fbc51403ba1aa9621a9d4a9bce2b'
git-subtree-dir: projects/rocprofiler-sdk git-subtree-mainline:50a90550e9git-subtree-split:bf0fad1d54
This commit is contained in:
systems-assistant[bot]
@@ -0,0 +1,357 @@
|
||||
.. meta::
|
||||
:description: Documentation of the usage of pc-sampling with rocprofv3 command-line tool
|
||||
:keywords: Sampling PC, Sampling program counter, rocprofv3, rocprofv3 tool usage, Using rocprofv3, ROCprofiler-SDK command line tool, PC sampling
|
||||
|
||||
.. _using-pc-sampling:
|
||||
|
||||
==================
|
||||
Using PC sampling
|
||||
==================
|
||||
|
||||
PC (Program Counter) sampling service for GPU profiling is a profiling technique to periodically sample the program counter during GPU kernel execution. PC sampling helps in understanding code execution patterns and identifying hotspot(s).
|
||||
|
||||
Here are the benefits of using PC sampling:
|
||||
|
||||
- Identify performance bottlenecks
|
||||
- Understand kernel execution behavior
|
||||
- Analyze code coverage
|
||||
- Find heavily executed code paths
|
||||
|
||||
To try out the PC sampling feature, you can use the command-line tool ``rocprofv3`` or the ROCprofiler-SDK library on `ROCm 6.4` or later.
|
||||
|
||||
.. note::
|
||||
PC sampling is ONLY supported on AMD GPUs with architectures gfx90a and later.
|
||||
|
||||
PC sampling availability and configuration
|
||||
===========================================
|
||||
|
||||
To check if the GPU supports PC sampling, use:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
rocprofv3 -L
|
||||
|
||||
Or
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
rocprofv3 --list-avail
|
||||
|
||||
The output lists if ``rocprofv3`` supports PC sampling on the GPU and the supported configuration.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
GPU:0
|
||||
NAME:gfx90a
|
||||
configs:
|
||||
Method :host_trap
|
||||
Unit :time
|
||||
Min_Interval :1
|
||||
Max_Interval :18446744073709551615
|
||||
Flags :none
|
||||
|
||||
The preceding output shows that the GPU supports PC sampling with the ``ROCPROFILER_PC_SAMPLING_METHOD_HOST_TRAP`` method and the ``ROCPROFILER_PC_SAMPLING_UNIT_TIME`` unit. The minimum and maximum intervals are also displayed.
|
||||
|
||||
.. note::
|
||||
Important firmware fixes to host-trap and stochastic PC-sampling for AMD Instinct MI300X have been made in ROCm 7.0.
|
||||
To ensure that you have the latest fixes, check if you have the correct firmware versions installed:
|
||||
|
||||
For host-trap PC-sampling on MI300X: PSP TOS Firmware >= version 00.36.02.59 or 0x00360259
|
||||
For stochastic PC-sampling on MI300X as described in the following section: MEC Firmware feature version: 50, firmware version >= 0x0000001a
|
||||
|
||||
To check the firmware versions, use:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
# To check PSP TOS Firmware:
|
||||
sudo cat /sys/kernel/debug/dri/0/amdgpu_firmware_info | grep SOS
|
||||
|
||||
# To check MEC Firmware:
|
||||
sudo cat /sys/kernel/debug/dri/1/amdgpu_firmware_info | grep MEC
|
||||
|
||||
Based on the available PC-sampling configurations, use the following command to profile the application using PC-sampling:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
rocprofv3 --pc-sampling-beta-enabled --pc-sampling-method host_trap --pc-sampling-unit time --pc-sampling-interval 1 --output-format csv -- <application_path>
|
||||
|
||||
The preceding command enables PC sampling with the ``host_trap`` method, ``time`` unit, and an interval of ``1`` μs (microsecond). Replace ``<application_path>`` with the path to the application you want to profile.
|
||||
|
||||
This generates two files, ``agent_info.csv`` and ``pc_sampling_host_trap.csv``. Both files are prefixed with the process ID.
|
||||
|
||||
Here are the contents of ``pc_sampling_host_trap.csv`` file generated for MatrixTranspose sample application:
|
||||
|
||||
.. csv-table:: PC sampling host trap
|
||||
:file: /data/pc_sampling_host_trap.csv
|
||||
:widths: 20,10,10,10,10,20
|
||||
:header-rows: 1
|
||||
|
||||
|
||||
For description of the fields in the output file, see :ref:`pc-sampling-fields`.
|
||||
|
||||
If you find the ``Instruction_Comment`` field in the output file to be empty, populate this field by compiling your application with debug symbols.
|
||||
Enabling debug symbols while compiling the application maps back to the source line. This helps in understanding the code execution pattern and hotspots.
|
||||
|
||||
.. csv-table:: PC sampling host trap with debug symbols
|
||||
:file: /data/pc_sampling_host_trap_debug.csv
|
||||
:widths: 20,10,10,10,10,20
|
||||
:header-rows: 1
|
||||
|
||||
|
||||
The preceding output shows the ``Instruction_Comment`` field populated with the source-line information.
|
||||
|
||||
.. _pc-sampling-fields:
|
||||
|
||||
PC sampling fields
|
||||
===================
|
||||
|
||||
Here are the fields in the output file generated by PC sampling:
|
||||
|
||||
- ``Sample_Timestamp``: Timestamp when sample is generated
|
||||
- ``Exec_Mask``: Active SIMD lanes when sampled
|
||||
- ``Dispatch_Id``: Originating kernel dispatch ID
|
||||
- ``Instruction``: Assembly instruction such as ``s_load_dword s8, s[1:2], 0x10``
|
||||
- ``Instruction_Comment``: Instruction comment that maps back to the source-line if debug symbols were enabled when application was compiled
|
||||
- ``Correlation_Id``: API launch call ID that matches dispatch ID
|
||||
|
||||
To dump samples in a more comprehensive format, use JSON through ``--output-format json``:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
rocprofv3 --pc-sampling-beta-enabled --pc-sampling-method host_trap --pc-sampling-unit time --pc-sampling-interval 1 --output-format json -- <application_path>
|
||||
|
||||
The preceding command generates a JSON file with the comprehensive output. Here is a trimmed down output with multiple records:
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
{
|
||||
"pc_sample_host_trap": [
|
||||
{
|
||||
"record": {
|
||||
"hw_id": {
|
||||
"chiplet": 0,
|
||||
"wave_id": 0,
|
||||
"simd_id": 2,
|
||||
"pipe_id": 0,
|
||||
"cu_or_wgp_id": 1,
|
||||
"shader_array_id": 0,
|
||||
"shader_engine_id": 2,
|
||||
"workgroup_id": 0,
|
||||
"vm_id": 3,
|
||||
"queue_id": 2,
|
||||
"microengine_id": 1
|
||||
},
|
||||
"pc": {
|
||||
"code_object_id": 1,
|
||||
"code_object_offset": 20228
|
||||
},
|
||||
"exec_mask": 18446744073709551615,
|
||||
"timestamp": 51040126667689,
|
||||
"dispatch_id": 1,
|
||||
"corr_id": {
|
||||
"internal": 1,
|
||||
"external": 0
|
||||
},
|
||||
"wrkgrp_id": {
|
||||
"x": 182,
|
||||
"y": 0,
|
||||
"z": 0
|
||||
},
|
||||
"wave_in_grp": 1
|
||||
},
|
||||
"inst_index": 0
|
||||
},
|
||||
{
|
||||
"record": {
|
||||
"hw_id": {
|
||||
"chiplet": 0,
|
||||
"wave_id": 0,
|
||||
"simd_id": 2,
|
||||
"pipe_id": 0,
|
||||
"cu_or_wgp_id": 0,
|
||||
"shader_array_id": 0,
|
||||
"shader_engine_id": 2,
|
||||
"workgroup_id": 0,
|
||||
"vm_id": 3,
|
||||
"queue_id": 2,
|
||||
"microengine_id": 1
|
||||
},
|
||||
"pc": {
|
||||
"code_object_id": 1,
|
||||
"code_object_offset": 20236
|
||||
},
|
||||
"exec_mask": 18446744073709551615,
|
||||
"timestamp": 51040126667689,
|
||||
"dispatch_id": 1,
|
||||
"corr_id": {
|
||||
"internal": 1,
|
||||
"external": 0
|
||||
},
|
||||
"wrkgrp_id": {
|
||||
"x": 158,
|
||||
"y": 0,
|
||||
"z": 0
|
||||
},
|
||||
"wave_in_grp": 2
|
||||
},
|
||||
"inst_index": 1
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
For description of the fields in the JSON output, see :ref:`output-file-fields`.
|
||||
|
||||
An Arbitrary Host-Trap PC Sampling Skid
|
||||
===============================================
|
||||
|
||||
Host-Trap PC sampling is a software-based technique that utilizes a background kernel thread
|
||||
to periodically interrupt running waves in order to capture the program counter (PC).
|
||||
This method is effective for gathering performance data without requiring specialized hardware
|
||||
to snapshot the waves. However, it has limitations due to the potential delay between
|
||||
when a wave receives an interrupt and when it processes the interrupt to capture the PC.
|
||||
This delay can lead to a sampling skid, where the PC samples may be attributed to instructions
|
||||
that are up to two instructions away from the actual source of latency.
|
||||
This results in a non-precise intra-kernel sampling method.
|
||||
|
||||
When analyzing an application profile generated by host-trap PC sampling,
|
||||
developers should consider not only the reported most costly instruction but
|
||||
also the instructions immediately preceding or following it.
|
||||
If the costly instruction is near a branch instruction, it is important
|
||||
to also consider the instruction targeted by the branch and the one immediately following it.
|
||||
|
||||
To address the limitations of host-trap sampling, the hardware-based stochastic PC sampling method
|
||||
has been developed. This method provides precise intra-kernel sampling with zero sampling skid,
|
||||
offering more accurate performance insights.
|
||||
|
||||
It is important to note that the skid issue inherent in host-trap PC sampling will not be resolved
|
||||
in its current form. Therefore, users are encouraged to adopt stochastic PC sampling,
|
||||
starting with the GFX942 architecture, to achieve more precise performance profiling.
|
||||
|
||||
Hardware-Based (Stochastic) PC Sampling Method
|
||||
===============================================
|
||||
|
||||
The new ``ROCPROFILER_PC_SAMPLING_METHOD_STOCHASTIC`` has been introduced for gfx942 architecture.
|
||||
It employs a specific hardware for probing waves actively running on GPU.
|
||||
Beside information already provided with ``ROCPROFILER_PC_SAMPLING_METHOD_HOST_TRAP`` useful for determining hot-spots within the kernel,
|
||||
it delivers additional information that tells whether a sampled wave issued an instruction represented with particular PC.
|
||||
If not, it provides the reason for not issuing the instruction (stall reason).
|
||||
This type of information is particularly useful for understanding stalls during the kernel execution.
|
||||
|
||||
To use this method on gfx942, we recommend listing available PC sampling configurations to verify if the latest ROCm stack is installed
|
||||
on the system by running:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
rocprofv3 -L
|
||||
|
||||
Output similar to the following indicates that the ``ROCPROFILER_PC_SAMPLING_METHOD_STOCHASTIC`` method is available:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
GPU:1
|
||||
NAME:gfx942
|
||||
configs:
|
||||
Method :stochastic
|
||||
Unit :cycle
|
||||
Min_Interval :256
|
||||
Max_Interval :2147483648
|
||||
Flags :interval pow2
|
||||
|
||||
Please note that on gfx942, `ROCPROFILER_PC_SAMPLING_METHOD_STOCHASTIC` requires intervals to be specified in cycles, whose values are powers of 2
|
||||
|
||||
To profile a gfx942 accelerated application with ``ROCPROFILER_PC_SAMPLING_METHOD_STOCHASTIC`` PC sampling, one can use the following command:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
rocprofv3 --pc-sampling-beta-enabled --pc-sampling-method stochastic --pc-sampling-unit cycles --pc-sampling-interval 1048576 --output-format csv, json -- <application_path>
|
||||
|
||||
The previous command serializes samples in both CSV and JSON output formats in the ``pc_sampling_stochastic.csv`` and ``out_results.json`` files, respectively.
|
||||
|
||||
Comparing the ``pc_sampling_stochastic.csv`` to ``pc_sampling_host_trap`` from previous section, one can notice that the ``ROCPROFILER_PC_SAMPLING_METHOD_STOCHASTIC`` method
|
||||
generates additional fields:
|
||||
- ``Wave_Issued_Instruction``: Indicates whether the wave issued an instruction (value 1) represented with particular PC or not (value 0)
|
||||
- ``Instruction_Type``: If the value of ``Wave_Issued_Instruction`` is 1, this fields indicates the type of the issued instruction. Otherwise, this fields irrelevant.
|
||||
- ``Stall_Reason``: If the value of ``Wave_Issued_Instruction`` is 0, this fields indicates the reason for not issuing the instruction (stall reason). Otherwise, this field is irrelevant.
|
||||
- ``Wave_Count``: Total number of waves actively running on a compute unit when the sample was generated.
|
||||
|
||||
.. csv-table:: PC sampling stochastic with debug symbols
|
||||
:file: /data/pc_sampling_stochastic_debug.csv
|
||||
:widths: 20,10,10,10,10,20,10,20,20,10
|
||||
:header-rows: 1
|
||||
|
||||
Similarly, ``ROCPROFILER_PC_SAMPLING_METHOD_STOCHASTIC`` method delivers additional information to every sample in the JSON output.
|
||||
The following snippet shows one sample from ``out_results.json`` file.
|
||||
|
||||
.. code-block:: text
|
||||
|
||||
{
|
||||
"record": {
|
||||
"flags": {
|
||||
"has_mem_cnt": 0
|
||||
},
|
||||
"hw_id": {
|
||||
"chiplet": 4,
|
||||
"wave_id": 0,
|
||||
"simd_id": 2,
|
||||
"pipe_id": 3,
|
||||
"cu_or_wgp_id": 1,
|
||||
"shader_array_id": 0,
|
||||
"shader_engine_id": 3,
|
||||
"workgroup_id": 0,
|
||||
"vm_id": 3,
|
||||
"queue_id": 2,
|
||||
"microengine_id": 1
|
||||
},
|
||||
"pc": {
|
||||
"code_object_id": 2,
|
||||
"code_object_offset": 13880
|
||||
},
|
||||
"exec_mask": 18446744073709551615,
|
||||
"timestamp": 390705261924637,
|
||||
"dispatch_id": 29,
|
||||
"corr_id": {
|
||||
"internal": 29,
|
||||
"external": 0
|
||||
},
|
||||
"wrkgrp_id": {
|
||||
"x": 9,
|
||||
"y": 489,
|
||||
"z": 0
|
||||
},
|
||||
"wave_in_grp": 0,
|
||||
"wave_issued": 1,
|
||||
"inst_type": "ROCPROFILER_PC_SAMPLING_INSTRUCTION_TYPE_VALU",
|
||||
"wave_cnt": 6,
|
||||
"snapshot": {
|
||||
"stall_reason": "ROCPROFILER_PC_SAMPLING_INSTRUCTION_NOT_ISSUED_REASON_OTHER_WAIT",
|
||||
"dual_issue_valu": 0,
|
||||
"arb_state_issue_valu": 1,
|
||||
"arb_state_issue_matrix": 0,
|
||||
"arb_state_issue_lds": 0,
|
||||
"arb_state_issue_lds_direct": 0,
|
||||
"arb_state_issue_scalar": 0,
|
||||
"arb_state_issue_vmem_tex": 0,
|
||||
"arb_state_issue_flat": 0,
|
||||
"arb_state_issue_exp": 0,
|
||||
"arb_state_issue_misc": 0,
|
||||
"arb_state_issue_brmsg": 0,
|
||||
"arb_state_stall_valu": 0,
|
||||
"arb_state_stall_matrix": 0,
|
||||
"arb_state_stall_lds": 0,
|
||||
"arb_state_stall_lds_direct": 0,
|
||||
"arb_state_stall_scalar": 0,
|
||||
"arb_state_stall_vmem_tex": 0,
|
||||
"arb_state_stall_flat": 0,
|
||||
"arb_state_stall_exp": 0,
|
||||
"arb_state_stall_misc": 0,
|
||||
"arb_state_stall_brmsg": 0
|
||||
}
|
||||
},
|
||||
"inst_index": 1
|
||||
},
|
||||
|
||||
Fields starting with ``arb_state_`` are of particular interest as they indicate the state of the arbiter at the time of sampling.
|
||||
Namely, ``arb_state_issue_`` fields indicate what type of instructions arbiter issued at the time of sampling.
|
||||
On the other hand, ``arb_state_stall_`` fields indicate what type of instructions were stalled at the time of sampling.
|
||||
This information is useful for understanding how many instructions per cycle (IPC) are issued.
|
||||
|
||||
Reference in New Issue
Block a user