``rocprofv3`` is a CLI tool that helps you quickly optimize applications and understand the low-level kernel details without requiring any modification in the source code.
It's backward compatible with its predecessor, ``rocprof``, and provides more features for application profiling with better accuracy.
cmake --build <build-directory> --target all --parallel <N>
Options
---------
Here is the list of ``rocprofv3`` command-line options. Some options are used for application tracing and some for kernel profiling while the output control options control the presentation and redirection of the generated output.
- pecifies the kernel names to target during counter collection.
- Kernel profiling
* - ``-L`` \| ``--list-metrics``
- List metrics for counter collection.
- Kernel profiling
* - ``-d`` \| ``--output-directory``
- Specifies the path for the output files.
- Output control
* - ``-o`` \| ``--output-file``
- Specifies the name of the output file. Note that this name is appended to the default names (_api_trace or counter_collection.csv) of the generated files'.
- Libraries to prepend to LD_PRELOAD (usually for sanitizers)
- Extension
You can also see all the ``rocprofv3`` options using:
..code-block::bash
rocprofv3 --help
Application tracing
---------------------
Application tracing provides the big picture of a program’s execution by collecting data on the execution times of API calls and GPU commands, such as kernel execution, async memory copy, and barrier packets. This information can be used as the first step in the profiling process to answer important questions, such as how much percentage of time was spent on memory copy and which kernel took the longest time to execute.
To use ``rocprofv3`` for application tracing, run:
..code-block::bash
rocprofv3 <tracing_option> -- <app_relative_path>
HIP trace
+++++++++++
HIP trace comprises execution traces for the entire application at the HIP level. This includes HIP API functions and their asynchronous activities at the runtime level. In general, HIP APIs directly interact with the user program. It is easier to analyze HIP traces as you can directly map them to the program.
To trace HIP runtime APIs, use:
..code-block::bash
rocprofv3 --hip-trace -- < app_relative_path >
The above command generates a `hip_api_trace.csv` file prefixed with the process ID.
..code-block::shell
$ cat 238_hip_api_trace.csv
Here are the contents of `hip_api_trace.csv` file:
For the description of the fields in the output file, see :ref:`output-file-fields`.
HSA trace
+++++++++++++
The HIP runtime library is implemented with the low-level HSA runtime. HSA API tracing is more suited for advanced users who want to understand the application behavior at the lower level. In general, tracing at the HIP level is recommended for most users. You should use HSA trace only if you are familiar with HSA runtime.
HSA trace contains the start and end time of HSA runtime API calls and their asynchronous activities.
The above command generates a `hsa_api_trace.csv` file prefixed with process ID. Note that the contents of this file have been truncated for demonstration purposes.
..code-block::shell
$ cat 197_hsa_api_trace.csv
Here are the contents of `hsa_api_trace.csv` file:
For the description of the fields in the output file, see :ref:`output-file-fields`.
Marker trace
++++++++++++++
In certain situations, such as debugging performance issues in large-scale GPU programs, API-level tracing might be too fine-grained to provide a big picture of the program execution. In such cases, it is helpful to define specific tasks to be traced.
To specify the tasks for tracing, enclose the respective source code with the API calls provided by the `ROCTX` library. This process is also known as instrumentation. As the scope of code for instrumentation is defined using the enclosing API calls, it is called a range. A range is a programmer-defined task that has a well-defined start and end code scope. You can also refine the scope specified within a range using further nested ranges. ``rocprofv3`` also reports the timelines for these nested ranges.
Here is a list of useful APIs for code instrumentation.
-``roctxMark``: Inserts a marker in the code with a message. Creating marks help you see when a line of code is executed.
-``roctxRangeStart``: Starts a range. Different threads can start ranges.
-``roctxRangePush``: Starts a new nested range.
-``roctxRangePop``: Stops the current nested range.
-``roctxRangeStop``: Stops the given range.
See how to use `rocTX` APIs in the MatrixTranspose application below:
..code-block::bash
roctxMark("before hipLaunchKernel");
int rangeId= roctxRangeStart("hipLaunchKernel range");
For the description of the fields in the output file, see :ref:`output-file-fields`.
Sys trace
+++++++++++
This is an all-inclusive option to collect all the above-mentioned traces.
..code-block::shell
rocprofv3 –-sys-trace -- < app_relative_path >
Running the above command generates `hip_api_trace.csv`, `hsa_api_trace.csv`, `kernel_trace.csv`, `memory_copy_trace.csv`, and `marker_api_trace.csv` (if `rocTX` APIs are specified in the application) files prefixed with the process ID.
Scratch memory trace
++++++++++++++++++++++
This option collects scratch memory operation's traces. Scratch is an address space on AMDGPUs, which is roughly equivalent to the `local memory` in NVIDIA CUDA. The `local memory` in CUDA is a thread-local global memory with interleaved addressing, which is used for register spills or stack space. With this option, you can trace when the ``rocr`` runtime allocates, frees, and tries to reclaim scratch memory.
The application tracing functionality allows you to evaluate the duration of kernel execution but is of little help in providing insight into kernel execution details. The kernel profiling functionality allows you to select kernels for profiling and choose the basic counters or derived metrics to be collected for each kernel execution, thus providing a greater insight into kernel execution.
For a comprehensive list of counters available on MI200, see `MI200 performance counters and metrics <https://rocm.docs.amd.com/en/latest/conceptual/gpu-arch/mi300-mi200-performance-counters.html>`_.
To collect the desired basic counters or derived metrics, mention them in an input file. In the input file, the line consisting of the counter or metric names must begin with ``pmc``. The input file could be in text (.txt), yaml (.yaml/.yml), or JSON (.json) format.
The number of basic counters or derived metrics that can be collected in one run of profiling are limited by the GPU hardware resources. If too many counters or metrics are selected, the kernels need to be executed multiple times to collect them. For multi-pass execution, include multiple ``pmc`` rows in the input file. Counters or metrics in each ``pmc`` row can be collected in each kernel run.
Kernel profiling output
+++++++++++++++++++++++++
To supply the input file for kernel profiling, use:
Running the above command generates a `./pmc_n/counter_collection.csv` file prefixed with the process ID. For each ``pmc`` row, a directory ``pmc_n`` containing a `counter_collection.csv` file is generated, where n = 1 for the first row and so on.
Running the above command generates a `./pmc_n/counter_collection.csv` file prefixed with the process ID. For each ``pmc`` row, a directory ``pmc_n`` containing a `counter_collection.csv` file is generated, where n = 1 for the first row and so on.
Kernel filtering allows you to filter the kernel profiling output based on the kernel name by specifying regex strings in the input file. To include kernel names matching the regex string in the kernel profiling output, use ``kernel_include_regex``. To exclude the kernel names matching the regex string from the kernel profiling output, use ``kernel_exclude_regex``.
You can also specify an iteration range for set of iterations of the included kernels. If the iteration range is not specified, then all iterations of the included kernels are profiled.
The following table lists the various fields or the columns in the output CSV files generated for application tracing and kernel profiling:
..list-table:: output file fields
:header-rows:1
* - Field
- Description
* - Agent_Id
- GPU identifier to which the kernel was submitted.
* - Correlation_Id
- Unique identifier for correlation between HIP and HSA async calls during activity tracing.
* - Start_Timestamp
- Begin time in nanoseconds (ns) when the kernel begins execution.
* - End_Timestamp
- End time in ns when the kernel finishes execution.
* - Queue_Id
- ROCm queue unique identifier to which the kernel was submitted.
* - Private_Segment_Size
- The amount of memory required in bytes for the combined private, spill, and arg segments for a work item.
* - Group_Segment_Size
- The group segment memory required by a workgroup in bytes. This does not include any dynamically allocated group segment memory that may be added when the kernel is dispatched.
* - Workgroup_Size
- Size of the workgroup as declared by the compute shader.
* - Workgroup_Size_n
- Size of the workgroup in the nth dimension as declared by the compute shader, where n = X, Y, or Z.
* - Grid_Size
- Number of thread blocks required to launch the kernel.
* - Grid_Size_n
- Number of thread blocks in the nth dimension required to launch the kernel, where n = X, Y, or Z.
* - LDS_Block_Size
- Thread block size for the kernel's Local Data Share (LDS) memory.
* - Scratch_Size
- Kernel’s scratch memory size.
* - SGPR_Count
- Kernel's Scalar General Purpose Register (SGPR) count.
* - VGPR_Count
- Kernel's Vector General Purpose Register (VGPR) count.
Output formats
----------------
``rocprofv3`` supports the following output formats:
- CSV (default)
- JSON
- PFTrace
You can specify the output format using the ``--output-format`` command-line option. Format selection is case-insensitive
and multiple output formats are supported. For example: ``--output-format json`` enables JSON output exclusively whereas
``--output-format csv json pftrace`` enables all three output formats for the run.
For trace visualization, use the PFTrace format and open the trace in `ui.perfetto.dev <https://ui.perfetto.dev/>`_.
JSON output schema
++++++++++++++++++++
``rocprofv3`` supports a custom JSON output format designed for programmatic analysis. The schema is optimized for size
while factoring in usability. You can generate the JSON output using ``--output-format json`` command-line option.
Properties
++++++++++++
-**`rocprofiler-sdk-tool`**`(array)`: rocprofv3 data per process (each element represents a process).
-**Items**`(object)`: Data for rocprofv3.
- **`metadata`**`(object, required)`: Metadata related to the profiler session.
- **`pid`**`(integer, required)`: Process ID.
- **`init_time`**`(integer, required)`: Initialization time in nanoseconds.
- **`fini_time`**`(integer, required)`: Finalization time in nanoseconds.
- **`agents`**`(array, required)`: List of agents.
- **Items**`(object)`: Data for an agent.
- **`size`**`(integer, required)`: Size of the agent data.
- **`id`**`(object, required)`: Identifier for the agent.
- **`handle`**`(integer, required)`: Handle for the agent.
- **`type`**`(integer, required)`: Type of the agent.
- **`cpu_cores_count`**`(integer)`: Number of CPU cores.
- **`simd_count`**`(integer)`: Number of SIMD units.
- **`mem_banks_count`**`(integer)`: Number of memory banks.
- **`caches_count`**`(integer)`: Number of caches.
- **`io_links_count`**`(integer)`: Number of I/O links.
- **`cpu_core_id_base`**`(integer)`: Base ID for CPU cores.
- **`simd_id_base`**`(integer)`: Base ID for SIMD units.
- **`max_waves_per_simd`**`(integer)`: Maximum waves per SIMD.
- **`lds_size_in_kb`**`(integer)`: Size of LDS in KB.
- **`gds_size_in_kb`**`(integer)`: Size of GDS in KB.
- **`num_gws`**`(integer)`: Number of GWS (global work size).
- **`wave_front_size`**`(integer)`: Size of the wave front.
- **`num_xcc`**`(integer)`: Number of XCC (execution compute units).
- **`cu_count`**`(integer)`: Number of compute units (CUs).
- **`array_count`**`(integer)`: Number of arrays.
- **`num_shader_banks`**`(integer)`: Number of shader banks.
- **`simd_arrays_per_engine`**`(integer)`: SIMD arrays per engine.
- **`cu_per_simd_array`**`(integer)`: CUs per SIMD array.
- **`simd_per_cu`**`(integer)`: SIMDs per CU.
- **`max_slots_scratch_cu`**`(integer)`: Maximum slots for scratch CU.
