[Rocprof-Systems]: Documentation update for profiling modes and PAPI counter enablement (#1437)

* Documentation update for profiling modes and papi counter enablement Update the documentation to add more details regarding profiling modes. Update the Papi event and hardware counter collection documentation. * Change1 for review comments * Formatting changes for Examples * Apply suggestions from code review Co-authored-by: Pratik Basyal <pratik.basyal@amd.com> * Formatting and code block error fixed * Bold applied --------- Co-authored-by: Pratik Basyal <pratik.basyal@amd.com> Co-authored-by: prbasyal <prbasyal@amd.com>
2025-11-19 17:04:35 -05:00
@@ -145,3 +145,122 @@ Statistical sampling of the Fibonacci function

 .. image:: ../data/fibonacci-sampling.png
   :alt: Visualization of the output of a statistical sample of the Fibonacci function
+
+Overview of profiling modes
+----------------------------
+
+ROCm Systems Profiler provides several complementary profiling approaches that can be used independently or in combination. Profiling modes are controlled via the ``ROCPROFSYS_MODE`` environment variable, which determines the active backends and features.
+
+Available values for ``ROCPROFSYS_MODE``: `trace`, `sampling`, `causal`, and `coverage`.
+
+Primary collection modes
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. list-table::
+   :header-rows: 1
+   :widths: 25 75
+
+   * - **Mode**
+     - **Purpose**
+   * - **Trace mode**
+     - Event tracing
+   * - **Profile mode**
+     - High-level summary profiles
+   * - **Sampling mode**
+     - Statistical call-stack sampling
+   * - **Causal mode**
+     - Performance impact analysis
+   * - **Coverage mode**
+     - Code coverage analysis
+
+Trace mode (default)
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+Tracing mode generates comprehensive, deterministic traces of every event and measurement during application execution. This mode can be enabled using ``ROCPROFSYS_TRACE=true`` or ``ROCPROFSYS_MODE=trace`` setting.
+
+Additional configuration options to control the tracing behavior include:
+
+- ``ROCPROFSYS_TRACE_DELAY`` (``--trace-wait``): Delay before starting trace collection (in seconds).
+- ``ROCPROFSYS_TRACE_DURATION`` (``--trace-duration``): Duration of trace collection (in seconds).
+- ``ROCPROFSYS_TRACE_PERIODS`` (``--trace-periods``): Specifies multiple delay/duration periods in the format ``<DELAY>:<DURATION>``, ``<DELAY>:<DURATION>:<REPEAT>``, or ``<DELAY>:<DURATION>:<REPEAT>:<CLOCK_ID>``.
+- ``ROCPROFSYS_TRACE_PERIOD_CLOCK_ID`` (``--trace-clock-id``): Clock type for timing, such as ``realtime``, ``monotonic``, ``cputime``.
+
+Profile mode
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+Profile mode generates high-level summary profiles with statistical aggregations (mean, min, max, stddev). This mode can be enabled using ``ROCPROFSYS_PROFILE=true``. This mode uses the **Timemory** backend. When tracing is enabled, profiling is turned off by default, and vice versa. However, both modes can be turned on at the same time.
+
+By default, only wall-clock timing is collected. Additional metrics can be configured using ``ROCPROFSYS_TIMEMORY_COMPONENTS``, which enables hardware counters (via PAPI), CPU, memory, and system metrics. To view the available components, use: ``rocprof-sys-avail --components --description``.
+
+Profile types:
+
+- **Flat Profile** (``--flat-profile``): Aggregated metrics per function across all contexts.
+- **Hierarchical Profile** (``--profile``): Metrics organized by call-stack context.
+
+.. tip:: Start with a flat profile to identify high-impact functions, then use a hierarchical profile to analyze critical paths.
+
+Sampling mode
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+Sampling uses statistical call-stack sampling through periodic software interrupts per thread, as described in :doc:`Sampling the call stack <../how-to/sampling-call-stack>`.
+
+Sampling types:
+
+1. **CPU-Time sampling** (default)
+
+* Enabled using ``ROCPROFSYS_SAMPLING_CPUTIME=ON`` or ``--cputime`` (rocprof-sys-sample), ``--sample-cputime`` (rocprof-sys-run). The sampling can be controlled using:
+
+  * ``ROCPROFSYS_SAMPLING_CPUTIME_FREQ``
+  * ``ROCPROFSYS_SAMPLING_CPUTIME_DELAY``
+  * ``ROCPROFSYS_SAMPLING_CPUTIME_SIGNAL``
+
+2. **Real-Time sampling**
+
+* Enabled using ``ROCPROFSYS_SAMPLING_REALTIME=ON`` or ``--realtime`` (rocprof-sys-sample), ``--sample-realtime`` (rocprof-sys-run). The sampling can be controlled using:
+
+  * ``ROCPROFSYS_SAMPLING_REALTIME_FREQ``
+  * ``ROCPROFSYS_SAMPLING_REALTIME_DELAY``
+  * ``ROCPROFSYS_SAMPLING_REALTIME_SIGNAL``
+
+3. **Overflow sampling**
+
+* Enabled using ``ROCPROFSYS_SAMPLING_OVERFLOW=ON`` or ``--sample-overflow`` (rocprof-sys-run). It requires Linux ``perf`` support (``/proc/sys/kernel/perf_event_paranoid <= 2``) as described in :ref:`rocprof-sys_papi_events`. The sampling can be controlled using:
+
+  * ``ROCPROFSYS_SAMPLING_OVERFLOW_FREQ``
+  * ``ROCPROFSYS_SAMPLING_OVERFLOW_EVENT``
+  * ``ROCPROFSYS_SAMPLING_OVERFLOW_SIGNAL``
+  * ``ROCPROFSYS_SAMPLING_OVERFLOW_TIDS``
+
+4. **Process sampling**
+
+* Enabled using ``ROCPROFSYS_USE_PROCESS_SAMPLING=ON`` (default ON). The sampling can be controlled using:
+
+  * ``ROCPROFSYS_PROCESS_SAMPLING_FREQ``
+  * ``ROCPROFSYS_SAMPLING_CPUS``
+  * ``ROCPROFSYS_SAMPLING_GPUS``
+
+.. note:: If sampling is enabled but no specific type is selected, CPU-time sampling is used by default.
+
+To enable sampling:
+
+1. Use ``rocprof-sys-sample`` (auto-enables sampling).
+2. Set ``ROCPROFSYS_USE_SAMPLING=ON`` and ``ROCPROFSYS_MODE=sampling``. 
+3. Use ``-S`` or ``--sample`` with ``rocprof-sys-run``. 
+4. Use ``-M sampling`` or ``--mode sampling`` with ``rocprof-sys-instrument``. Use of ``rocprof-sys-sample`` is **recommended** over ``rocprof-sys-instrument -M sampling`` when binary instrumentation is not necessary. For more details, see :doc:`Sampling the call stack <../how-to/sampling-call-stack>`.
+
+Causal mode
+^^^^^^^^^^^^^^^^^^^^^^^^
+Causal profiling quantifies the potential impact of optimizations in parallel code and predicts where efforts should be focused as described in :doc:`Performing causal profiling <../how-to/performing-causal-profiling>`.
+This mode can be enabled using: ``ROCPROFSYS_USE_CAUSAL=true`` or ``ROCPROFSYS_MODE=causal``.
+
+Coverage mode
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+Coverage mode tracks which parts of your code are executed during a run. It uses binary instrumentation to record function and/or basic block execution. This mode can be enabled using: ``rocprof-sys-instrument -M coverage``.
+
+Granularity options:
+
+- Function-level: ``--coverage=function`` (``CODECOV_FUNCTION``)
+- Basic block-level: ``--coverage=basic_block`` (``CODECOV_BASIC_BLOCK``)
+
+.. note:: Coverage mode disables several other features and all other modes to reduce overhead.
@@ -67,7 +67,7 @@ GPU metrics
  
  .. note::
  
-  The availability of VCN, JPEG, XGMI, and PCIe metrics depends on device support and system topology. If unsupported, values will be reported as ``N/A`` in the output of ``amd-smi metric --usage``.
+    The availability of VCN, JPEG, XGMI, and PCIe metrics depends on device support and system topology. If unsupported, values will be reported as ``N/A`` in the output of ``amd-smi metric --usage``.

 CPU metrics
 ========================================
@@ -181,6 +181,51 @@ PAPI components from different namespaces:
   installed with the prefix ``rocprof-sys-`` with
   underscores replaced with hyphens, for example ``papi_avail`` becomes ``rocprof-sys-papi-avail``.

+There are two distinct approaches for collecting PAPI-based hardware counters, each with different characteristics and use cases:
+
+1. **Instrumentation-based collection:** Uses function instrumentation system to collect PAPI counters at function entry and exit points. This works with profiling mode via ``ROCPROFSYS_TIMEMORY_COMPONENTS``:
+
+**Example 1: Using ``papi_array`` for a fixed list of events**
+
+.. code-block:: shell   
+   
+   # Enable profiling mode (required)
+   export ROCPROFSYS_PROFILE=ON
+
+   # Specify papi_array in the timemory component list
+   export ROCPROFSYS_TIMEMORY_COMPONENTS="wall_clock,papi_array"
+
+   # Specify which PAPI events to collect
+   export ROCPROFSYS_PAPI_EVENTS="PAPI_TOT_CYC,PAPI_TOT_INS"
+   
+
+**Example 2: Using ``papi_vector`` for dynamically allocated array of events**
+
+.. code-block:: shell 
+   
+   # Include papi_vector for dynamic event lists
+   export ROCPROFSYS_TIMEMORY_COMPONENTS="wall_clock,papi_vector"
+
+   # Alternative: Use perf event names
+   export ROCPROFSYS_PAPI_EVENTS="perf::INSTRUCTIONS,perf::CACHE-REFERENCES,perf::CACHE-MISSES"
+   
+
+2. **Sampling-based collection:** Periodically interrupts program execution to capture hardware counters along with call stack information. This works with the sampling mode.
+
+.. code-block:: shell 
+   
+   # Enable sampling mode (required)
+   export ROCPROFSYS_USE_SAMPLING=ON
+
+   # Specify PAPI events for sampling
+   export ROCPROFSYS_PAPI_EVENTS="PAPI_TOT_CYC,PAPI_TOT_INS"
+
+You can also enable overflow sampling for PAPI events with ``ROCPROFSYS_SAMPLING_OVERFLOW_EVENT``: 
+
+.. code-block:: shell
+
+   export ROCPROFSYS_SAMPLING_OVERFLOW_EVENT="perf::PERF_COUNT_HW_CACHE_REFERENCES"
+
 ROCPROFSYS_ROCM_EVENTS
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^