Adding rocpd documenation (#449)

* Adding rocpd docuemenation

* rocpd format

* CHANGELOG update and indexing

* Fixing links

* format fixes

* fixing table

* major edits

* fixed logical error

* fixing rocprofv3 avail

CHANGELOG.md

@@ -189,7 +189,8 @@ Full documentation for ROCprofiler-SDK is available at [rocm.docs.amd.com/projec
   - Requires [ROCprof Trace Decoder plugin](https://github.com/rocm/rocprof-trace-decoder)
 - Added thread trace option to the rocprofv3 tool under the --att parameters
   - See [using thread trace with rocprofv3](https://rocm.docs.amd.com/projects/rocprofiler-sdk/en/amd-mainline/how-to/using-thread-trace.html)
-  - Requires the ROCprof Trace Decoder plugin installed (see above)
+  - Requires the ROCprof Trace Decoder plugin installed (see above).
+- Added `rocpd` output format documentation
 
 ### Changed
 

source/docs/_toc.yml.in

@@ -14,6 +14,7 @@ subtrees:
     - file: how-to/samples
       title: Samples
     - file: how-to/using-rocprofv3
+    - file: how-to/using-rocpd-output-format
     - file: how-to/using-rocprofv3-avail
     - file: how-to/using-rocprofiler-sdk-roctx
     - file: how-to/using-rocprofv3-with-mpi

source/docs/data/rocpd-to-csv.csv

@@ -0,0 +1,14 @@
+"Guid","Domain","Function","Process_Id","Thread_Id","Correlation_Id","Start_Timestamp","End_Timestamp"
+"0000ddb5-8903-7903-b269-cd8d3745bb8a","HIP_COMPILER_API_EXT","__hipRegisterFatBinary",137,137,1,3719662243949943,3719662243977763
+"0000ddb5-8903-7903-b269-cd8d3745bb8a","HIP_COMPILER_API_EXT","__hipRegisterFunction",137,137,2,3719662243992513,3719662244013373
+"0000ddb5-8903-7903-b269-cd8d3745bb8a","HIP_RUNTIME_API_EXT","hipGetDevicePropertiesR0600",137,137,3,3719662244489885,3719662300657772
+"0000ddb5-8903-7903-b269-cd8d3745bb8a","HIP_RUNTIME_API_EXT","hipMalloc",137,137,4,3719662301007843,3719662301139564
+"0000ddb5-8903-7903-b269-cd8d3745bb8a","HIP_RUNTIME_API_EXT","hipMalloc",137,137,5,3719662301140994,3719662301221024
+"0000ddb5-8903-7903-b269-cd8d3745bb8a","HIP_RUNTIME_API_EXT","hipMemcpy",137,137,6,3719662301228614,3719662416709451
+"0000ddb5-8903-7903-b269-cd8d3745bb8a","HIP_RUNTIME_API_EXT","hipMemcpy",137,137,7,3719662416720791,3719662416728291
+"0000ddb5-8903-7903-b269-cd8d3745bb8a","HIP_COMPILER_API_EXT","__hipPushCallConfiguration",137,137,8,3719662416731301,3719662416733001
+"0000ddb5-8903-7903-b269-cd8d3745bb8a","HIP_COMPILER_API_EXT","__hipPopCallConfiguration",137,137,9,3719662416734631,3719662416735421
+"0000ddb5-8903-7903-b269-cd8d3745bb8a","HIP_RUNTIME_API_EXT","hipLaunchKernel",137,137,10,3719662416752501,3719662417325443
+"0000ddb5-8903-7903-b269-cd8d3745bb8a","HIP_RUNTIME_API_EXT","hipMemcpy",137,137,11,3719662417327973,3719662418667169
+"0000ddb5-8903-7903-b269-cd8d3745bb8a","HIP_RUNTIME_API_EXT","hipFree",137,137,12,3719662428427823,3719662428488874
+"0000ddb5-8903-7903-b269-cd8d3745bb8a","HIP_RUNTIME_API_EXT","hipFree",137,137,13,3719662428491354,3719662428514084

source/docs/how-to/using-rocpd-output-format.rst

@@ -0,0 +1,193 @@
+.. meta::
+    :description: "ROCprofiler-SDK is a tooling infrastructure for profiling general-purpose GPU compute applications running on the ROCm software."
+    :keywords: "ROCprofiler-SDK, ROCProfiler-SDK output formats, rocpd, SQLite3, CSV, JSON, PFTrace, OTF2"
+
+.. _using-rocpd-output-format:
+
+=========================
+Using rocpd Output Format
+=========================
+
+``rocprofv3`` supports the following output formats:
+
+- **rocpd** (SQLite3 Database, Default)
+- **CSV**
+- **JSON** (Custom format for programmatic analysis only)
+- **PFTrace** (Perfetto trace for visualization with Perfetto)
+- **OTF2** (Open Trace Format for visualization with compatible third-party tools)
+
+The ``rocpd`` output format is the default for ``rocprofv3``. It stores profiling results in a SQLite3 database, providing a structured and efficient way to analyze and post-process profiling data. This format allows users to query and manipulate profiling data using SQL, making it easy to extract specific information or perform complex analyses.
+
+Features
+++++++++
+
+- **Rich Data Model**: Stores all collected profiling data, including traces, counters, and metadata, in a single `.db` (SQLite3) file.
+- **Programmatic Access**: Can be queried using standard SQL tools or libraries (e.g., `sqlite3` CLI, Python's `sqlite3` module).
+- **Post-Processing**: Enables advanced analysis and visualization using custom scripts or third-party tools that support SQLite3.
+
+Generating rocpd Output
++++++++++++++++++++++++
+
+To generate output in rocpd format, simply use:
+
+.. code-block:: bash
+
+   rocprofv3 --hip-trace -- <application>
+
+Or use the ``--output-format`` option with ``rocpd``:
+
+.. code-block:: bash
+
+   rocprofv3 --hip-trace --output-format rocpd -- <application>
+
+The output will be saved as ``%hostname%/%pid%_results.db``, where ``%hostname%`` is the name of the host machine and ``%pid%`` is the process ID of the application being profiled.
+
+Converting rocpd to Other Formats
++++++++++++++++++++++++++++++++++
+
+The ``rocpd`` output format can be converted to other formats for further analysis or visualization.  
+First, ensure the ``rocpd`` Python module is available in your environment:
+
+.. code-block:: bash
+
+   export PYTHONPATH=<install-path>/lib/pythonX.Y/site-packages:$PYTHONPATH
+
+where ``<install-path>`` is the ROCm installation path (usually ``/opt/rocm-<major.minor.patch>``), and ``X.Y`` is your Python version.
+
+Once the ``rocpd`` module is available, use the ``rocpd convert`` command to convert the output to other formats.
+
+Convert to CSV format:
+
+.. code-block:: bash
+
+   python3 -m rocpd convert -i <input-file>.db --output-format csv
+
+The converted CSV will be saved as ``rocpd-output-data/out_hip_api_trace.csv`` in the current working directory.
+
+Convert to OTF2 format:
+
+.. code-block:: bash
+
+   python3 -m rocpd convert -i <input-file>.db --output-format otf2
+
+Convert to PFTrace format:
+
+.. code-block:: bash
+
+   python3 -m rocpd convert -i <input-file>.db --output-format pftrace
+
+rocpd convert Command-Line Options
+++++++++++++++++++++++++++++++++++
+
+.. code-block:: none
+
+   usage: rocpd convert [-h] -i INPUT [INPUT ...] -f {csv,pftrace,otf2} [{csv,pftrace,otf2} ...]
+                        [-o OUTPUT_FILE] [-d OUTPUT_PATH] [--kernel-rename]
+                        [--agent-index-value {absolute,relative,type-relative}]
+                        [--perfetto-backend {inprocess,system}]
+                        [--perfetto-buffer-fill-policy {discard,ring_buffer}]
+                        [--perfetto-buffer-size KB] [--perfetto-shmem-size-hint KB]
+                        [--group-by-queue]
+                        [--start START | --start-marker START_MARKER]
+                        [--end END | --end-marker END_MARKER]
+                        [--inclusive INCLUSIVE]
+
+Options
+-------
+
+**Required Arguments:**
+
+- ``-i INPUT [INPUT ...]``, ``--input INPUT [INPUT ...]``  
+  Input path and filename to one or more database(s), separated by spaces.
+
+- ``-f {csv,pftrace,otf2} [{csv,pftrace,otf2} ...]``, ``--output-format {csv,pftrace,otf2} [{csv,pftrace,otf2} ...]``  
+  Specify one or more output formats. Supported: ``csv``, ``pftrace``, ``otf2``.
+
+**I/O Options:**
+
+- ``-o OUTPUT_FILE``, ``--output-file OUTPUT_FILE``  
+  Sets the base output file name (default: ``out``).
+
+- ``-d OUTPUT_PATH``, ``--output-path OUTPUT_PATH``  
+  Sets the output directory (default: ``./rocpd-output-data``).
+
+**Kernel Naming Options:**
+
+- ``--kernel-rename``  
+  Use ROCTx marker names instead of kernel names.
+
+**Generic Options:**
+
+- ``--agent-index-value {absolute,relative,type-relative}``  
+  Device identification format in output:
+  
+  - ``absolute``: Uses node_id (e.g., Agent-0, Agent-2, Agent-4), ignoring cgroups.
+  - ``relative``: Uses logical_node_id (e.g., Agent-0, Agent-1, Agent-2), considering cgroups. *(Default)*
+  - ``type-relative``: Uses logical_node_type_id (e.g., CPU-0, GPU-0, GPU-1), numbering resets for each device type.
+
+**Perfetto Trace (pftrace) Options:**
+
+- ``--perfetto-backend {inprocess,system}``  
+  Perfetto data collection backend. ``system`` mode requires running ``traced`` and ``perfetto`` daemons (default: ``inprocess``).
+
+- ``--perfetto-buffer-fill-policy {discard,ring_buffer}``  
+  Policy for handling new records when buffer is full (default: ``discard``).
+
+- ``--perfetto-buffer-size KB``  
+  Buffer size for perfetto output in KB (default: 1 GB).
+
+- ``--perfetto-shmem-size-hint KB``  
+  Perfetto shared memory size hint in KB (default: 64 KB).
+
+- ``--group-by-queue``  
+  Display HIP streams that kernels and memory copy operations are submitted to, rather than HSA queues.
+
+**Time Window Options:**
+
+- ``--start START``  
+  Start time as percentage or nanoseconds from trace file (e.g., ``50%`` or ``781470909013049``).
+
+- ``--start-marker START_MARKER``  
+  Named marker event to use as window start point.
+
+- ``--end END``  
+  End time as percentage or nanoseconds from trace file (e.g., ``75%`` or ``3543724246381057``).
+
+- ``--end-marker END_MARKER``  
+  Named marker event to use as window end point.
+
+- ``--inclusive INCLUSIVE``  
+  ``True``: include events if START or END in window; ``False``: only if BOTH in window (default: ``True``).
+
+**Help:**
+
+- ``-h``, ``--help``  
+  Show help message and exit.
+
+Examples
+++++++++
+
+Convert one database to Perfetto trace:
+
+.. code-block:: bash
+
+   python3 -m rocpd convert -i db1.db --output-format pftrace
+
+Convert two databases to Perfetto trace, set output path and filename, and limit to last 70% of trace:
+
+.. code-block:: bash
+
+   python3 -m rocpd convert -i db1.db db2.db --output-format pftrace -d "./output/" -o "twoFileTraces" --start 30% --end 100%
+
+Convert six databases to CSV and Perfetto trace formats:
+
+.. code-block:: bash
+
+   python3 -m rocpd convert -i db{0..5}.db --output-format csv pftrace -d "~/output_folder/" -o "sixFileTraces"
+
+Convert two databases to CSV, OTF2, and Perfetto trace formats:
+
+.. code-block:: bash
+
+   python3 -m rocpd convert -i db{3,4}.db --output-format csv otf2 pftrace
+

source/docs/how-to/using-rocprofv3.rst

@@ -49,13 +49,13 @@ The following table lists the commonly used ``rocprofv3`` command-line options c
      - | ``-i`` INPUT \| ``--input`` INPUT |br| |br| |br| |br| |br| |br|
        | ``-o`` OUTPUT_FILE \| ``--output-file`` OUTPUT_FILE |br| |br| |br|
        | ``-d`` OUTPUT_DIRECTORY \| ``--output-directory`` OUTPUT_DIRECTORY |br| |br|
-       | ``--output-format {csv,json,pftrace,otf2} [{csv,json,pftrace,otf2} ...]`` |br| |br|
+       | ``--output-format {csv,json,pftrace,otf2,rocpd} [{csv,json,pftrace,otf2,rocpd} ...]`` |br| |br|
        | ``--log-level {fatal,error,warning,info,trace,env}`` |br| |br|
        | ``-E`` EXTRA_COUNTERS \| ``--extra-counters`` EXTRA_COUNTERS
      - | Specifies the path to the input file. JSON and YAML formats support configuration of all command-line options for tracing and profiling whereas the text format supports only the specification of HW counters. |br| |br|
        | Specifies output file name. If nothing is specified, the default path is ``%hostname%/%pid%``. |br| |br|
        | Specifies the output path for saving the output files. If nothing is specified, the default path is ``%hostname%/%pid%``. |br| |br|
-       | Specifies output format. Supported formats: CSV, JSON, PFTrace, and OTF2. |br| |br| |br|
+       | Specifies output format. Supported formats: CSV, JSON, PFTrace, OTF2 and rocpd. |br| |br| |br|
        | Sets the desired log level. |br| |br| |br|
        | Specifies the path to a YAML file consisting of extra counter definitions.
 
@@ -192,6 +192,14 @@ To use ``rocprofv3`` for application tracing, run:
 
     rocprofv3 <tracing_option> -- <application_path>
 
+
+.. note::
+
+  All the tracing examples below use the ``--output-format csv`` option to generate output in CSV format.
+  However, the default output format is ``rocpd`` (SQLite3 database). You can simply omit the ``--output-format`` option to generate output in the default format.
+  ``rocpd`` format can be converted to other formats such as CSV, OTF2, and PFTrace using the ``rocpd`` module. 
+  To understand how to convert ``rocpd`` output to other formats, see :ref:`using-rocpd-output-format`.
+
 HIP trace
 +++++++++++
 
@@ -1513,21 +1521,22 @@ The following table lists the various fields or the columns in the output CSV fi
 Output formats
 ----------------
 
-``rocprofv3`` supports the following output formats:
 
-- SQLite3 Database (Default)
+- rocpd (SQLite3 Database (Default))
 - CSV
 - JSON (Custom format for programmatic analysis only)
 - PFTrace (Perfetto trace for visualization with Perfetto)
 - OTF2 (Open Trace Format for visualization with compatible third-party tools)
 
-To specify the output format, use:
+
+The default output format is ``rocpd``. To know more about the rocpd format, see :ref:`using-rocpd-output-format`. 
+To specify the particular output format, use the ``--output-format`` option followed by the desired format.
 
 .. code-block::
 
    rocprofv3 -i input.txt --output-format json -- <application_path>
 
-Format selection is case-insensitive and multiple output formats are supported. While ``--output-format json`` exclusively enables JSON output, ``--output-format csv json pftrace otf2`` enables all four output formats for the run.
+Format selection is case-insensitive and multiple output formats are supported. While ``--output-format json`` exclusively enables JSON output, ``--output-format csv json pftrace otf2, rocpd`` enables all four output formats for the run.
 
 For PFTrace trace visualization, use the PFTrace format and open the trace in `ui.perfetto.dev <https://ui.perfetto.dev/>`_.
 

source/docs/index.rst

@@ -31,6 +31,8 @@ The documentation is structured as follows:
 
     * :doc:`Samples <how-to/samples>`
     * :ref:`using-rocprofv3`
+    * :ref:`using-rocprofv3-avail`
+    * :ref:`using-rocpd-output-format`
     * :ref:`using-rocprofiler-sdk-roctx`
     * :ref:`using-rocprofv3-with-mpi`
     * :ref:`using-pc-sampling`