As explained in the preceding section, you can specify parameters on the command line or use a JSON input file:
..code-block::text
{
"jobs": [
{
"advanced_thread_trace": true,
"att_target_cu": 1,
"att_shader_engine_mask": "0x1",
"att_simd_select": "0xF",
"att_buffer_size": "0x6000000"
}
]
}
Thread tracing for multiple kernel instances
=============================
By default, ``rocprofv3`` enables thread trace only once per kernel instance. This implies that if an application launches the same kernel multiple times, only the first instance will be traced.
To enable thread trace for multiple kernel instances, use the ``kernel-iteration-range`` parameter.
It's recommended to use ``kernel-include-regex`` parameter to filter the desired kernel names instead of tracing everything.
.._output-files:
rocprofv3 output files
===============
After the application finishes executing, ROCprof Trace Decoder runs automatically and the following output files are generated:
- stats_*.csv files:
* Contains a summary of instruction latency per kernel.
The columns of the stats_*.csv file are described here:
***Codeobj:** The code object load ID assigned by ROCprofiler-SDK.
***Vaddr:** ELF vaddr.
***Hitcount:** The number of times a particular instruction is executed while adding all the traced waves.
***Latency:** Total latency in cycles, defined as "Stall time + Issue time" for gfx9 or "Stall time + Execute time" for gfx10+.
***Stall:** The total number of cycles the hardware pipe couldn't issue an instruction.
* Usually caused when the hardware unit is busy, such as TCP or LDS backpressure.
***Idle:** The total time gap between the completion of previous instruction and the beginning of the current instruction. The idle time can be caused by:
* Arbiter loss
* Source or destination register dependency
* Instruction cache miss
***Source:** The original source line of code assigned by the compiler.
* Requires compiling with debug symbols.
Troubleshooting
===============
For some applications, stats_*.csv file could be empty even for a valid kernel dispatch.
Thread trace is limited to a single CU per SE (``att-target-cu``). If a kernel dispatch doesn't launch enough waves to populate the whole GPU, there's a possibility of no wave getting assigned to the ``target_cu``. In such cases, there's nothing to be traced.
Here are some options to handle this:
* Launch more waves.
* Swap the ``target_cu``.
* Set the ``--att-shader-engine-mask`` to 0x11111111, or possibly to 0xFFFFFFFF
* A number too high can cause packet losses and/or lead to a full buffer.
* Set the ``HSA_CU_MASK`` to mask out all CUs but the target. For more details, see `setting CUs <https://rocm.docs.amd.com/en/latest/how-to/setting-cus.html>`_.
* If only the ``target_cu`` (or a few CUs) are not masked out, then all or most waves will be assigned to the ``target_cu``.
* This can potentially cause low performance in high-demanding kernels.
