AI/rocm-systems
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update branding to ROCm Systems Profiler in documentation (#2)

Browse Source 
* Update branding in docs

* Rename image used in documentation

* Update names of code samples.

In the code snippets, the "-" is not valid. ex., rocprof-sys_ --> rocprofsys_

* Update ASCII art

* update Doxyfile strip_from_path

* Add a "Formerly known as" message.

* Fixed typo in product name

ROCm Systems Profiler, not ROCm Profiler System

* Add "Omnitrace" back to the metadata keywords

* Update "install via package manager" section

* Update paths to user API files

* Rename configuration and environment settings

* Update Doxyfiles

Update publisher name & ID to "AMD".
Update bundle ID to "rocprofiler-systems"

* Update docs/what-is-rocprof-sys.rst

Co-authored-by: Jeffrey Novotny <jnovotny@amd.com>

* Update docs/conceptual/data-collection-modes.rst

Co-authored-by: Jeffrey Novotny <jnovotny@amd.com>

* Update docs/tutorials/video-tutorials.rst

Co-authored-by: Jeffrey Novotny <jnovotny@amd.com>

* Update docs/conceptual/rocprof-sys-feature-set.rst

Co-authored-by: Jeffrey Novotny <jnovotny@amd.com>

* Update docs/how-to/configuring-runtime-options.rst

Co-authored-by: Jeffrey Novotny <jnovotny@amd.com>

* Update docs/how-to/configuring-validating-environment.rst

Co-authored-by: Jeffrey Novotny <jnovotny@amd.com>

* Update docs/how-to/general-tips-using-rocprof-sys.rst

Co-authored-by: Jeffrey Novotny <jnovotny@amd.com>

* Update docs/reference/rocprof-sys-glossary.rst

Co-authored-by: Jeffrey Novotny <jnovotny@amd.com>

* Update docs/reference/development-guide.rst

Co-authored-by: Jeffrey Novotny <jnovotny@amd.com>

* Update docs/how-to/instrumenting-rewriting-binary-application.rst

Co-authored-by: Jeffrey Novotny <jnovotny@amd.com>

* Update docs/install/quick-start.rst

Co-authored-by: Jeffrey Novotny <jnovotny@amd.com>

* Note that videos were recorded using the "Omnitrace" name.

* Rebase and update some file paths

* Update paths to doc images

* Update Omnitrace references in code snippets

* Rename examples still using the "omni" prefix.

* Update docs/how-to/performing-causal-profiling.rst

Co-authored-by: Jeffrey Novotny <jnovotny@amd.com>

* Update docs/how-to/profiling-python-scripts.rst

Co-authored-by: Jeffrey Novotny <jnovotny@amd.com>

* Update docs/how-to/sampling-call-stack.rst

Co-authored-by: Jeffrey Novotny <jnovotny@amd.com>

* Update docs/how-to/understanding-rocprof-sys-output.rst

Co-authored-by: Jeffrey Novotny <jnovotny@amd.com>

* Update docs/install/install.rst

Co-authored-by: Jeffrey Novotny <jnovotny@amd.com>

---------

Co-authored-by: Peter Park <peter.park@amd.com>
Co-authored-by: Jeffrey Novotny <jnovotny@amd.com>

[ROCm/rocprofiler-systems commit: 032d39f15c]
This commit is contained in:
David Galiffi
2024-10-17 15:19:19 -04:00
committed by GitHub
parent 181a782835
commit d13617cf91
59 changed files with 1340 additions and 10282 deletions
Download Patch File Download Diff File Expand all files Collapse all files
projects/rocprofiler-systems/docs/how-to/configuring-runtime-options.rst
+281 -279
View File
@@ -1,131 +1,133 @@
.. meta::
:description: Omnitrace documentation and reference
:keywords: Omnitrace, ROCm, profiler, tracking, visualization, tool, Instinct, accelerator, AMD
:description: ROCm Systems Profiler runtime options documentation and reference
:keywords: rocprof-sys, rocprofiler-systems, Omnitrace, ROCm, runtime options, profiler, tracking, visualization, tool, Instinct, accelerator, AMD
****************************************************
Configuring runtime options
****************************************************
The ``omnitrace.cfg`` file maintains a list of the `Omnitrace <https://github.com/ROCm/omnitrace>`_ runtime options. To create this configuration
file and view the current runtime options, use the ``omnitrace-avail`` executable.
The ``rocprof-sys.cfg`` file maintains a list of the
`ROCm Systems Profiler <https://github.com/ROCm/rocprofiler-systems>`_ runtime
options. To create this configuration
file and view the current runtime options, use the ``rocprof-sys-avail`` executable.
The omnitrace-avail executable
The rocprof-sys-avail executable
========================================
The ``omnitrace-avail`` executable provides information about the runtime settings,
The ``rocprof-sys-avail`` executable provides information about the runtime settings,
data collection capabilities, and, when built with PAPI support, the
available hardware counters. The executable is effectively
self-updating. As new capabilities and settings are added to the Omnitrace source code, they are
propagated to ``omnitrace-avail``. ``omnitrace-avail`` should be viewed as the ultimate authority
self-updating. As new capabilities and settings are added to the ROCm Systems Profiler source code, they are
propagated to ``rocprof-sys-avail``. ``rocprof-sys-avail`` should be viewed as the ultimate authority
in the event of any conflicts with this documentation.
It is recommended that you create a default configuration file in
``${HOME}/.omnitrace.cfg``. This can be done by
running the command ``omnitrace-avail -G ~/.omnitrace.cfg``. Alternatively,
use the ``omnitrace-avail -G ~/.omnitrace.cfg --all`` option
It is recommended that you create a default configuration file in
``${HOME}/.rocprof-sys.cfg``. This can be done by
running the command ``rocprof-sys-avail -G ~/.rocprof-sys.cfg``. Alternatively,
use the ``rocprof-sys-avail -G ~/.rocprof-sys.cfg --all`` option
for a verbose configuration file with descriptions, categories, and additional information.
Modify ``${HOME}/.omnitrace.cfg`` as required. For example, enable `Perfetto <https://perfetto.dev/>`_,
Modify ``${HOME}/.rocprof-sys.cfg`` as required. For example, enable `Perfetto <https://perfetto.dev/>`_,
`Timemory <https://github.com/NERSC/timemory>`_, sampling, and process-level sampling by default
and tweak the default sampling values.
.. code-block:: shell
# ...
OMNITRACE_TRACE = true
OMNITRACE_PROFILE = true
OMNITRACE_USE_SAMPLING = true
OMNITRACE_USE_PROCESS_SAMPLING = true
ROCPROFSYS_TRACE = true
ROCPROFSYS_PROFILE = true
ROCPROFSYS_USE_SAMPLING = true
ROCPROFSYS_USE_PROCESS_SAMPLING = true
# ...
OMNITRACE_SAMPLING_FREQ = 50
OMNITRACE_SAMPLING_CPUS = all
OMNITRACE_SAMPLING_GPUS = $env:HIP_VISIBLE_DEVICES
ROCPROFSYS_SAMPLING_FREQ = 50
ROCPROFSYS_SAMPLING_CPUS = all
ROCPROFSYS_SAMPLING_GPUS = $env:HIP_VISIBLE_DEVICES
Exploring runtime settings
-----------------------------------
Use the following command to view the list of the available runtime settings, their current values, and descriptions
Use the following command to view the list of the available runtime settings, their current values, and descriptions
for each setting:
.. code-block:: shell
omnitrace-avail --description
rocprof-sys-avail --description
.. note::
Use ``--brief`` to suppress printing the current value and/or ``-c 0`` to suppress truncation of the descriptions.
Any Boolean setting (``omnitrace-avail --settings --value --brief --filter bool``)
accepts a case insensitive match for nearly all common Boolean logic expressions:
Any Boolean setting (``rocprof-sys-avail --settings --value --brief --filter bool``)
accepts a case insensitive match for nearly all common Boolean logic expressions:
``ON``, ``OFF``, ``YES``, ``NO``, ``TRUE``, ``FALSE``, ``0``, ``1``, etc.
Exploring components
-----------------------------------
Omnitrace uses `Timemory <https://github.com/NERSC/timemory>`_ extensively to provide
ROCm Systems Profiler uses `Timemory <https://github.com/NERSC/timemory>`_ extensively to provide
various capabilities and manage
data and resources. By default, with ``OMNITRACE_PROFILE=ON``, Omnitrace only collects wall-clock
timing values. However, by modifying the ``OMNITRACE_TIMEMORY_COMPONENTS`` setting,
Omnitrace can be configured to
data and resources. By default, with ``ROCPROFSYS_PROFILE=ON``, ROCm Systems Profiler only collects wall-clock
timing values. However, by modifying the ``ROCPROFSYS_TIMEMORY_COMPONENTS`` setting,
ROCm Systems Profiler can be configured to
collect hardware counters, CPU-clock timers, memory usage, context switches, page faults, network statistics,
and much more. Omnitrace can even be used as a dynamic instrumentation vehicle
and much more. ROCm Systems Profiler can even be used as a dynamic instrumentation vehicle
for other third-party profiling
APIs such as `Caliper <https://github.com/LLNL/Caliper>`_ and `LIKWID <https://github.com/RRZE-HPC/likwid>`_.
To leverage this capability, build Omnitrace from source with the CMake
To leverage this capability, build ROCm Systems Profiler from source with the CMake
options ``TIMEMORY_USE_CALIPER=ON`` or ``TIMEMORY_USE_LIKWID=ON`` and then add
``caliper_marker``, ``likwid_marker``, or both to ``OMNITRACE_TIMEMORY_COMPONENTS``.
``caliper_marker``, ``likwid_marker``, or both to ``ROCPROFSYS_TIMEMORY_COMPONENTS``.
To view all possible components and their descriptions:
.. code-block:: shell
omnitrace-avail --components --description
rocprof-sys-avail --components --description
To restrict the output to available components and view the string identifiers for ``OMNITRACE_TIMEMORY_COMPONENTS``:
To restrict the output to available components and view the string identifiers for ``ROCPROFSYS_TIMEMORY_COMPONENTS``:
.. code-block:: shell
omnitrace-avail --components --available --string --brief
rocprof-sys-avail --components --available --string --brief
Exploring hardware counters
-----------------------------------
Omnitrace supports hardware counter collection via PAPI and ROCm.
ROCm Systems Profiler supports hardware counter collection via PAPI and ROCm.
Generally, PAPI is used to collect CPU-based hardware counters and ROCm is used to collect GPU-based hardware
counters. Although it is possible to install PAPI with ROCm support and use it to
collect GPU-based hardware counters, this is not recommended because PAPI
counters. Although it is possible to install PAPI with ROCm support and use it to
collect GPU-based hardware counters, this is not recommended because PAPI
cannot simultaneously collect CPU and GPU hardware counters.
To view all possible hardware counters and their descriptions, use the following command:
.. code-block:: shell
omnitrace-avail --hw-counters --description
rocprof-sys-avail --hw-counters --description
Appending the ``-c CPU`` option restricts the list of hardware counters to
Appending the ``-c CPU`` option restricts the list of hardware counters to
those available through PAPI, while ``-c GPU`` limits the list to those available from ROCm.
Enabling hardware counters
-----------------------------------
PAPI Hardware counters are configured with the ``OMNITRACE_PAPI_EVENTS`` configuration variable.
ROCm Hardware counters are configured with the ``OMNITRACE_ROCM_EVENTS`` configuration variable.
ROCm hardware counters also require the ``OMNITRACE_USE_ROCPROFILER`` configuration
variable to be enabled using ``OMNITRACE_USE_ROCPROFILER=ON``.
PAPI Hardware counters are configured with the ``ROCPROFSYS_PAPI_EVENTS`` configuration variable.
ROCm Hardware counters are configured with the ``ROCPROFSYS_ROCM_EVENTS`` configuration variable.
ROCm hardware counters also require the ``ROCPROFSYS_USE_ROCPROFILER`` configuration
variable to be enabled using ``ROCPROFSYS_USE_ROCPROFILER=ON``.
Here is a sample configuration for hardware counters:
.. code-block:: shell
# using papi identifiers
OMNITRACE_PAPI_EVENTS = PAPI_TOT_CYC PAPI_TOT_INS
ROCPROFSYS_PAPI_EVENTS = PAPI_TOT_CYC PAPI_TOT_INS
# using perf identifiers
OMNITRACE_PAPI_EVENTS = perf::INSTRUCTIONS perf::CACHE-REFERENCES perf::CACHE-MISSES
ROCPROFSYS_PAPI_EVENTS = perf::INSTRUCTIONS perf::CACHE-REFERENCES perf::CACHE-MISSES
.. _omnitrace_papi_events:
.. _rocprof-sys_papi_events:
OMNITRACE_PAPI_EVENTS
ROCPROFSYS_PAPI_EVENTS
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
In order to collect the majority of hardware counters via PAPI, ensure the ``/proc/sys/kernel/perf_event_paranoid``
@@ -135,18 +137,18 @@ has a value <= 2. If you have ``sudo`` access, use the following command to modi
echo 0 | sudo tee /proc/sys/kernel/perf_event_paranoid
However this value is not retained upon reboot.
However this value is not retained upon reboot.
Use the following command to preserve this setting after a reboot:
.. code-block:: shell
echo 'kernel.perf_event_paranoid=0' | sudo tee -a /etc/sysctl.conf
PAPI events use a concept similar to a namespace. All specified hardware
PAPI events use a concept similar to a namespace. All specified hardware
counters must be from the same namespace.
For hardware counters starting with the ``PAPI_`` prefix, these are high-level
For hardware counters starting with the ``PAPI_`` prefix, these are high-level
aggregates of multiple hardware counters.
Otherwise, most events use two or three colons (``::`` or ``:::``) between the
Otherwise, most events use two or three colons (``::`` or ``:::``) between the
component name and the counter name, for example,
``amd64_rapl::RAPL_ENERGY_PKG`` and ``perf::PERF_COUNT_HW_CPU_CYCLES``.
@@ -154,33 +156,33 @@ For example, the following is a valid configuration:
.. code-block:: shell
OMNITRACE_PAPI_EVENTS = perf::INSTRUCTIONS perf::CACHE-REFERENCES perf::CACHE-MISSES
ROCPROFSYS_PAPI_EVENTS = perf::INSTRUCTIONS perf::CACHE-REFERENCES perf::CACHE-MISSES
However, the following specification of a roughly equivalent set of hardware counters is an incorrect configuration because it mixes
PAPI components from different namespaces:
.. code-block:: shell
OMNITRACE_PAPI_EVENTS = PAPI_TOT_INS perf::CACHE-REFERENCES perf::CACHE-MISSES
ROCPROFSYS_PAPI_EVENTS = PAPI_TOT_INS perf::CACHE-REFERENCES perf::CACHE-MISSES
.. note::
If Omnitrace was configured with the default ``OMNITRACE_BUILD_PAPI=ON`` setting,
If ROCm Systems Profiler was configured with the default ``ROCPROFSYS_BUILD_PAPI=ON`` setting,
standard PAPI command-line tools such as
``papi_avail`` and ``papi_event_chooser`` are not able to provide information
about the PAPI library used by Omnitrace
(because Omnitrace statically links to ``libpapi``). However, all of these tools are
installed with the prefix ``omnitrace-`` with
underscores replaced with hypens, for example ``papi_avail`` becomes ``omnitrace-papi-avail``.
``papi_avail`` and ``papi_event_chooser`` are not able to provide information
about the PAPI library used by ROCm Systems Profiler
(because ROCm Systems Profiler statically links to ``libpapi``). However, all of these tools are
installed with the prefix ``rocprof-sys-`` with
underscores replaced with hypens, for example ``papi_avail`` becomes ``rocprof-sys-papi-avail``.
OMNITRACE_ROCM_EVENTS
ROCPROFSYS_ROCM_EVENTS
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Omnitrace reads the ROCm events from the ``${ROCM_PATH}/lib/rocprofiler/metrics.xml``
ROCm Systems Profiler reads the ROCm events from the ``${ROCM_PATH}/lib/rocprofiler/metrics.xml``
file. Use the ``ROCP_METRICS`` environment
variable to point Omnitrace to a different XML metrics file, for example,
variable to point ROCm Systems Profiler to a different XML metrics file, for example,
``export ROCP_METRICS=${PWD}/custom_metrics.xml``.
``omnitrace-avail -H -c GPU`` shows event names with a suffix of ``:device=N``
``rocprof-sys-avail -H -c GPU`` shows event names with a suffix of ``:device=N``
where ``N`` is the device number.
For example, if you have two devices, the output is:
@@ -190,7 +192,7 @@ For example, if you have two devices, the output is:
...
| Wavefronts:device=1 | Derived counter: SQ_WAVES |
To collect the event on all devices, specify the event,
To collect the event on all devices, specify the event,
such as ``Wavefronts``, without the ``:device=`` suffix.
To collect the event only on specific devices, use the ``:device=`` suffix.
@@ -202,12 +204,12 @@ The following example:
.. code-block:: shell
OMNITRACE_ROCM_EVENTS = GPUBusy SQ_WAVES:device=0 SQ_INSTS_VALU:device=1
ROCPROFSYS_ROCM_EVENTS = GPUBusy SQ_WAVES:device=0 SQ_INSTS_VALU:device=1
omnitrace-avail examples
rocprof-sys-avail examples
-----------------------------------
The following examples demonstrate how to use ``omnitrace-avail`` to perform several common
The following examples demonstrate how to use ``rocprof-sys-avail`` to perform several common
configuration tasks.
Generating a default configuration file
@@ -215,96 +217,96 @@ Generating a default configuration file
.. code-block:: shell
$ omnitrace-avail -G ~/.omnitrace.cfg
[omnitrace-avail] Outputting text configuration file '/home/user/.omnitrace.cfg'...
$ cat ~/.omnitrace.cfg
# auto-generated by omnitrace-avail (version 1.2.0) on 2022-06-27 @ 19:15
$ rocprof-sys-avail -G ~/.rocprof-sys.cfg
[rocprof-sys-avail] Outputting text configuration file '/home/user/.rocprof-sys.cfg'...
$ cat ~/.rocprof-sys.cfg
# auto-generated by rocprof-sys-avail (version 1.2.0) on 2022-06-27 @ 19:15
OMNITRACE_CONFIG_FILE =
OMNITRACE_MODE = trace
OMNITRACE_TRACE = true
OMNITRACE_PROFILE = false
OMNITRACE_USE_SAMPLING = false
OMNITRACE_USE_PROCESS_SAMPLING = true
OMNITRACE_USE_ROCTRACER = true
OMNITRACE_USE_ROCM_SMI = true
OMNITRACE_USE_KOKKOSP = false
OMNITRACE_USE_CODE_COVERAGE = false
OMNITRACE_USE_PID = true
OMNITRACE_OUTPUT_PATH = omnitrace-%tag%-output
OMNITRACE_OUTPUT_PREFIX =
OMNITRACE_CI = false
OMNITRACE_THREAD_POOL_SIZE = 8
OMNITRACE_DEBUG = false
OMNITRACE_DL_VERBOSE = 0
OMNITRACE_INSTRUMENTATION_INTERVAL = 1
OMNITRACE_KOKKOSP_KERNEL_LOGGER = false
OMNITRACE_PAPI_EVENTS = PAPI_TOT_CYC
OMNITRACE_PERFETTO_BACKEND = inprocess
OMNITRACE_PERFETTO_BUFFER_SIZE_KB = 1024000
OMNITRACE_PERFETTO_COMBINE_TRACES = false
OMNITRACE_PERFETTO_FILE = perfetto-trace.proto
OMNITRACE_PERFETTO_FILL_POLICY = discard
OMNITRACE_PERFETTO_SHMEM_SIZE_HINT_KB = 4096
OMNITRACE_ROCTRACER_HSA_ACTIVITY = false
OMNITRACE_ROCTRACER_HSA_API = false
OMNITRACE_ROCTRACER_HSA_API_TYPES =
OMNITRACE_SAMPLING_CPUS =
OMNITRACE_SAMPLING_DELAY = 0.5
OMNITRACE_SAMPLING_FREQ = 10
OMNITRACE_SAMPLING_GPUS = all
OMNITRACE_TIME_OUTPUT = true
OMNITRACE_TIMEMORY_COMPONENTS = wall_clock
OMNITRACE_TRACE_THREAD_LOCKS = false
OMNITRACE_VERBOSE = 0
OMNITRACE_COLLAPSE_PROCESSES = false
OMNITRACE_COLLAPSE_THREADS = false
OMNITRACE_COUT_OUTPUT = false
OMNITRACE_CPU_AFFINITY = false
OMNITRACE_DIFF_OUTPUT = false
OMNITRACE_ENABLE_SIGNAL_HANDLER = true
OMNITRACE_ENABLED = true
OMNITRACE_FILE_OUTPUT = true
OMNITRACE_FLAT_PROFILE = false
OMNITRACE_INPUT_EXTENSIONS = json,xml
OMNITRACE_INPUT_PATH =
OMNITRACE_INPUT_PREFIX =
OMNITRACE_JSON_OUTPUT = true
OMNITRACE_MAX_DEPTH = 65535
OMNITRACE_MAX_WIDTH = 120
OMNITRACE_MEMORY_PRECISION = -1
OMNITRACE_MEMORY_SCIENTIFIC = false
OMNITRACE_MEMORY_UNITS = MB
OMNITRACE_MEMORY_WIDTH = -1
OMNITRACE_NETWORK_INTERFACE =
OMNITRACE_NODE_COUNT = 0
OMNITRACE_PAPI_FAIL_ON_ERROR = false
OMNITRACE_PAPI_MULTIPLEXING = false
OMNITRACE_PAPI_OVERFLOW = 0
OMNITRACE_PAPI_QUIET = false
OMNITRACE_PAPI_THREADING = true
OMNITRACE_PRECISION = -1
OMNITRACE_SCIENTIFIC = false
OMNITRACE_STRICT_CONFIG = true
OMNITRACE_SUPPRESS_CONFIG = true
OMNITRACE_SUPPRESS_PARSING = true
OMNITRACE_TEXT_OUTPUT = true
OMNITRACE_TIME_FORMAT = %F_%H.%M
OMNITRACE_TIMELINE_PROFILE = false
OMNITRACE_TIMING_PRECISION = 6
OMNITRACE_TIMING_SCIENTIFIC = false
OMNITRACE_TIMING_UNITS = sec
OMNITRACE_TIMING_WIDTH = -1
OMNITRACE_TREE_OUTPUT = true
OMNITRACE_WIDTH = -1
ROCPROFSYS_CONFIG_FILE =
ROCPROFSYS_MODE = trace
ROCPROFSYS_TRACE = true
ROCPROFSYS_PROFILE = false
ROCPROFSYS_USE_SAMPLING = false
ROCPROFSYS_USE_PROCESS_SAMPLING = true
ROCPROFSYS_USE_ROCTRACER = true
ROCPROFSYS_USE_ROCM_SMI = true
ROCPROFSYS_USE_KOKKOSP = false
ROCPROFSYS_USE_CODE_COVERAGE = false
ROCPROFSYS_USE_PID = true
ROCPROFSYS_OUTPUT_PATH = rocprof-sys-%tag%-output
ROCPROFSYS_OUTPUT_PREFIX =
ROCPROFSYS_CI = false
ROCPROFSYS_THREAD_POOL_SIZE = 8
ROCPROFSYS_DEBUG = false
ROCPROFSYS_DL_VERBOSE = 0
ROCPROFSYS_INSTRUMENTATION_INTERVAL = 1
ROCPROFSYS_KOKKOSP_KERNEL_LOGGER = false
ROCPROFSYS_PAPI_EVENTS = PAPI_TOT_CYC
ROCPROFSYS_PERFETTO_BACKEND = inprocess
ROCPROFSYS_PERFETTO_BUFFER_SIZE_KB = 1024000
ROCPROFSYS_PERFETTO_COMBINE_TRACES = false
ROCPROFSYS_PERFETTO_FILE = perfetto-trace.proto
ROCPROFSYS_PERFETTO_FILL_POLICY = discard
ROCPROFSYS_PERFETTO_SHMEM_SIZE_HINT_KB = 4096
ROCPROFSYS_ROCTRACER_HSA_ACTIVITY = false
ROCPROFSYS_ROCTRACER_HSA_API = false
ROCPROFSYS_ROCTRACER_HSA_API_TYPES =
ROCPROFSYS_SAMPLING_CPUS =
ROCPROFSYS_SAMPLING_DELAY = 0.5
ROCPROFSYS_SAMPLING_FREQ = 10
ROCPROFSYS_SAMPLING_GPUS = all
ROCPROFSYS_TIME_OUTPUT = true
ROCPROFSYS_TIMEMORY_COMPONENTS = wall_clock
ROCPROFSYS_TRACE_THREAD_LOCKS = false
ROCPROFSYS_VERBOSE = 0
ROCPROFSYS_COLLAPSE_PROCESSES = false
ROCPROFSYS_COLLAPSE_THREADS = false
ROCPROFSYS_COUT_OUTPUT = false
ROCPROFSYS_CPU_AFFINITY = false
ROCPROFSYS_DIFF_OUTPUT = false
ROCPROFSYS_ENABLE_SIGNAL_HANDLER = true
ROCPROFSYS_ENABLED = true
ROCPROFSYS_FILE_OUTPUT = true
ROCPROFSYS_FLAT_PROFILE = false
ROCPROFSYS_INPUT_EXTENSIONS = json,xml
ROCPROFSYS_INPUT_PATH =
ROCPROFSYS_INPUT_PREFIX =
ROCPROFSYS_JSON_OUTPUT = true
ROCPROFSYS_MAX_DEPTH = 65535
ROCPROFSYS_MAX_WIDTH = 120
ROCPROFSYS_MEMORY_PRECISION = -1
ROCPROFSYS_MEMORY_SCIENTIFIC = false
ROCPROFSYS_MEMORY_UNITS = MB
ROCPROFSYS_MEMORY_WIDTH = -1
ROCPROFSYS_NETWORK_INTERFACE =
ROCPROFSYS_NODE_COUNT = 0
ROCPROFSYS_PAPI_FAIL_ON_ERROR = false
ROCPROFSYS_PAPI_MULTIPLEXING = false
ROCPROFSYS_PAPI_OVERFLOW = 0
ROCPROFSYS_PAPI_QUIET = false
ROCPROFSYS_PAPI_THREADING = true
ROCPROFSYS_PRECISION = -1
ROCPROFSYS_SCIENTIFIC = false
ROCPROFSYS_STRICT_CONFIG = true
ROCPROFSYS_SUPPRESS_CONFIG = true
ROCPROFSYS_SUPPRESS_PARSING = true
ROCPROFSYS_TEXT_OUTPUT = true
ROCPROFSYS_TIME_FORMAT = %F_%H.%M
ROCPROFSYS_TIMELINE_PROFILE = false
ROCPROFSYS_TIMING_PRECISION = 6
ROCPROFSYS_TIMING_SCIENTIFIC = false
ROCPROFSYS_TIMING_UNITS = sec
ROCPROFSYS_TIMING_WIDTH = -1
ROCPROFSYS_TREE_OUTPUT = true
ROCPROFSYS_WIDTH = -1
When creating a new configuration file, the following recommendations apply:
* Use the ``--all`` option to view all descriptions, choices, and other information in the configuration file.
* To create a new configuration without inheriting from an existing ``${HOME}/.omnitrace.cfg`` file,
set ``OMNITRACE_SUPPRESS_CONFIG=ON`` in the environment beforehand.
* To create a new configuration without inheriting from an existing ``${HOME}/.rocprof-sys.cfg`` file,
set ``ROCPROFSYS_SUPPRESS_CONFIG=ON`` in the environment beforehand.
* To create a new configuration that makes minor changes to an existing configuration,
set ``OMNITRACE_CONFIG_FILE=/path/to/existing/file`` and define the changes as environment
set ``ROCPROFSYS_CONFIG_FILE=/path/to/existing/file`` and define the changes as environment
variables before generating it.
Viewing the setting descriptions
@@ -312,89 +314,89 @@ Viewing the setting descriptions
.. code-block:: shell
$ omnitrace-avail -S -bd
$ rocprof-sys-avail -S -bd
|-----------------------------------------|-----------------------------------------|
| ENVIRONMENT VARIABLE | DESCRIPTION |
|-----------------------------------------|-----------------------------------------|
| OMNITRACE_CI | Enable some runtime validation check... |
| OMNITRACE_ADD_SECONDARY | Enable/disable components adding sec... |
| OMNITRACE_COLLAPSE_PROCESSES | Enable/disable combining process-spe... |
| OMNITRACE_COLLAPSE_THREADS | Enable/disable combining thread-spec... |
| OMNITRACE_CONFIG_FILE | Configuration file for omnitrace |
| OMNITRACE_COUT_OUTPUT | Write output to stdout |
| OMNITRACE_CPU_AFFINITY | Enable pinning threads to CPUs (Linu... |
| OMNITRACE_THREAD_POOL_SIZE | Number of threads to use when genera... |
| OMNITRACE_DEBUG | Enable debug output |
| OMNITRACE_DIFF_OUTPUT | Generate a difference output vs. a p... |
| OMNITRACE_DL_VERBOSE | Verbosity within the omnitrace-dl li... |
| OMNITRACE_ENABLED | Activation state of timemory |
| OMNITRACE_ENABLE_SIGNAL_HANDLER | Enable signals in timemory_init |
| OMNITRACE_FILE_OUTPUT | Write output to files |
| OMNITRACE_FLAT_PROFILE | Set the label hierarchy mode to defa... |
| OMNITRACE_INPUT_EXTENSIONS | File extensions used when searching ... |
| OMNITRACE_INPUT_PATH | Explicitly specify the input folder ... |
| OMNITRACE_INPUT_PREFIX | Explicitly specify the prefix for in... |
| OMNITRACE_INSTRUMENTATION_INTERVAL | Instrumentation only takes measureme... |
| OMNITRACE_JSON_OUTPUT | Write json output files |
| OMNITRACE_KOKKOSP_KERNEL_LOGGER | Enables kernel logging |
| OMNITRACE_MAX_DEPTH | Set the maximum depth of label hiera... |
| OMNITRACE_MAX_THREAD_BOOKMARKS | Maximum number of times a worker thr... |
| OMNITRACE_MAX_WIDTH | Set the maximum width for component ... |
| OMNITRACE_MEMORY_PRECISION | Set the precision for components wit... |
| OMNITRACE_MEMORY_SCIENTIFIC | Set the numerical reporting format f... |
| OMNITRACE_MEMORY_UNITS | Set the units for components with u... |
| OMNITRACE_MEMORY_WIDTH | Set the output width for components ... |
| OMNITRACE_NETWORK_INTERFACE | Default network interface |
| OMNITRACE_NODE_COUNT | Total number of nodes used in applic... |
| OMNITRACE_OUTPUT_FILE | Perfetto filename |
| OMNITRACE_OUTPUT_PATH | Explicitly specify the output folder... |
| OMNITRACE_OUTPUT_PREFIX | Explicitly specify a prefix for all ... |
| OMNITRACE_PAPI_EVENTS | PAPI presets and events to collect (... |
| OMNITRACE_PAPI_FAIL_ON_ERROR | Configure PAPI errors to trigger a r... |
| OMNITRACE_PAPI_MULTIPLEXING | Enable multiplexing when using PAPI |
| OMNITRACE_PAPI_OVERFLOW | Value at which PAPI hw counters trig... |
| OMNITRACE_PAPI_QUIET | Configure suppression of reporting P... |
| OMNITRACE_PAPI_THREADING | Enable multithreading support when u... |
| OMNITRACE_PERFETTO_BACKEND | Specify the perfetto backend to acti... |
| OMNITRACE_PERFETTO_BUFFER_SIZE_KB | Size of perfetto buffer (in KB) |
| OMNITRACE_PERFETTO_COMBINE_TRACES | Combine Perfetto traces. If not expl... |
| OMNITRACE_PERFETTO_FILL_POLICY | Behavior when perfetto buffer is ful... |
| OMNITRACE_PERFETTO_SHMEM_SIZE_HINT_KB | Hint for shared-memory buffer size i... |
| OMNITRACE_PRECISION | Set the global output precision for ... |
| OMNITRACE_ROCTRACER_HSA_ACTIVITY | Enable HSA activity tracing support |
| OMNITRACE_ROCTRACER_HSA_API | Enable HSA API tracing support |
| OMNITRACE_ROCTRACER_HSA_API_TYPES | HSA API type to collect |
| OMNITRACE_SAMPLING_CPUS | CPUs to collect frequency informatio... |
| OMNITRACE_SAMPLING_DELAY | Number of seconds to wait before the... |
| OMNITRACE_SAMPLING_FREQ | Number of software interrupts per se... |
| OMNITRACE_SAMPLING_GPUS | Devices to query when OMNITRACE_USE_... |
| OMNITRACE_SCIENTIFIC | Set the global numerical reporting t... |
| OMNITRACE_STRICT_CONFIG | Throw errors for unknown setting nam... |
| OMNITRACE_SUPPRESS_CONFIG | Disable processing of setting config... |
| OMNITRACE_SUPPRESS_PARSING | Disable parsing environment |
| OMNITRACE_TEXT_OUTPUT | Write text output files |
| OMNITRACE_TIMELINE_PROFILE | Set the label hierarchy mode to defa... |
| OMNITRACE_TIMEMORY_COMPONENTS | List of components to collect via ti... |
| OMNITRACE_TIME_FORMAT | Customize the folder generation when... |
| OMNITRACE_TIME_OUTPUT | Output data to subfolder w/ a timest... |
| OMNITRACE_TIMING_PRECISION | Set the precision for components wit... |
| OMNITRACE_TIMING_SCIENTIFIC | Set the numerical reporting format f... |
| OMNITRACE_TIMING_UNITS | Set the units for components with u... |
| OMNITRACE_TIMING_WIDTH | Set the output width for components ... |
| OMNITRACE_TRACE_THREAD_LOCKS | Enable tracking calls to pthread_mut... |
| OMNITRACE_TREE_OUTPUT | Write hierarchical json output files |
| OMNITRACE_USE_CODE_COVERAGE | Enable support for code coverage |
| OMNITRACE_USE_KOKKOSP | Enable support for Kokkos Tools |
| OMNITRACE_USE_OMPT | Enable support for OpenMP-Tools |
| OMNITRACE_TRACE | Enable perfetto backend |
| OMNITRACE_USE_PID | Enable tagging filenames with proces... |
| OMNITRACE_USE_ROCM_SMI | Enable sampling GPU power, temp, uti... |
| OMNITRACE_USE_ROCTRACER | Enable ROCM tracing |
| OMNITRACE_USE_SAMPLING | Enable statistical sampling of call-... |
| OMNITRACE_USE_PROCESS_SAMPLING | Enable a background thread which sam... |
| OMNITRACE_PROFILE | Enable timemory backend |
| OMNITRACE_VERBOSE | Verbosity level |
| OMNITRACE_WIDTH | Set the global output width for comp... |
| ROCPROFSYS_CI | Enable some runtime validation check... |
| ROCPROFSYS_ADD_SECONDARY | Enable/disable components adding sec... |
| ROCPROFSYS_COLLAPSE_PROCESSES | Enable/disable combining process-spe... |
| ROCPROFSYS_COLLAPSE_THREADS | Enable/disable combining thread-spec... |
| ROCPROFSYS_CONFIG_FILE | Configuration file for rocprof-sys |
| ROCPROFSYS_COUT_OUTPUT | Write output to stdout |
| ROCPROFSYS_CPU_AFFINITY | Enable pinning threads to CPUs (Linu... |
| ROCPROFSYS_THREAD_POOL_SIZE | Number of threads to use when genera... |
| ROCPROFSYS_DEBUG | Enable debug output |
| ROCPROFSYS_DIFF_OUTPUT | Generate a difference output vs. a p... |
| ROCPROFSYS_DL_VERBOSE | Verbosity within the rocprof-sys-dl ... |
| ROCPROFSYS_ENABLED | Activation state of timemory |
| ROCPROFSYS_ENABLE_SIGNAL_HANDLER | Enable signals in timemory_init |
| ROCPROFSYS_FILE_OUTPUT | Write output to files |
| ROCPROFSYS_FLAT_PROFILE | Set the label hierarchy mode to defa... |
| ROCPROFSYS_INPUT_EXTENSIONS | File extensions used when searching ... |
| ROCPROFSYS_INPUT_PATH | Explicitly specify the input folder ... |
| ROCPROFSYS_INPUT_PREFIX | Explicitly specify the prefix for in... |
| ROCPROFSYS_INSTRUMENTATION_INTERVAL | Instrumentation only takes measureme... |
| ROCPROFSYS_JSON_OUTPUT | Write json output files |
| ROCPROFSYS_KOKKOSP_KERNEL_LOGGER | Enables kernel logging |
| ROCPROFSYS_MAX_DEPTH | Set the maximum depth of label hiera... |
| ROCPROFSYS_MAX_THREAD_BOOKMARKS | Maximum number of times a worker thr... |
| ROCPROFSYS_MAX_WIDTH | Set the maximum width for component ... |
| ROCPROFSYS_MEMORY_PRECISION | Set the precision for components wit... |
| ROCPROFSYS_MEMORY_SCIENTIFIC | Set the numerical reporting format f... |
| ROCPROFSYS_MEMORY_UNITS | Set the units for components with u... |
| ROCPROFSYS_MEMORY_WIDTH | Set the output width for components ... |
| ROCPROFSYS_NETWORK_INTERFACE | Default network interface |
| ROCPROFSYS_NODE_COUNT | Total number of nodes used in applic... |
| ROCPROFSYS_OUTPUT_FILE | Perfetto filename |
| ROCPROFSYS_OUTPUT_PATH | Explicitly specify the output folder... |
| ROCPROFSYS_OUTPUT_PREFIX | Explicitly specify a prefix for all ... |
| ROCPROFSYS_PAPI_EVENTS | PAPI presets and events to collect (... |
| ROCPROFSYS_PAPI_FAIL_ON_ERROR | Configure PAPI errors to trigger a r... |
| ROCPROFSYS_PAPI_MULTIPLEXING | Enable multiplexing when using PAPI |
| ROCPROFSYS_PAPI_OVERFLOW | Value at which PAPI hw counters trig... |
| ROCPROFSYS_PAPI_QUIET | Configure suppression of reporting P... |
| ROCPROFSYS_PAPI_THREADING | Enable multithreading support when u... |
| ROCPROFSYS_PERFETTO_BACKEND | Specify the perfetto backend to acti... |
| ROCPROFSYS_PERFETTO_BUFFER_SIZE_KB | Size of perfetto buffer (in KB) |
| ROCPROFSYS_PERFETTO_COMBINE_TRACES | Combine Perfetto traces. If not expl... |
| ROCPROFSYS_PERFETTO_FILL_POLICY | Behavior when perfetto buffer is ful... |
| ROCPROFSYS_PERFETTO_SHMEM_SIZE_HINT_KB | Hint for shared-memory buffer size i... |
| ROCPROFSYS_PRECISION | Set the global output precision for ... |
| ROCPROFSYS_ROCTRACER_HSA_ACTIVITY | Enable HSA activity tracing support |
| ROCPROFSYS_ROCTRACER_HSA_API | Enable HSA API tracing support |
| ROCPROFSYS_ROCTRACER_HSA_API_TYPES | HSA API type to collect |
| ROCPROFSYS_SAMPLING_CPUS | CPUs to collect frequency informatio... |
| ROCPROFSYS_SAMPLING_DELAY | Number of seconds to wait before the... |
| ROCPROFSYS_SAMPLING_FREQ | Number of software interrupts per se... |
| ROCPROFSYS_SAMPLING_GPUS | Devices to query when ROCPROFSYS_USE_... |
| ROCPROFSYS_SCIENTIFIC | Set the global numerical reporting t... |
| ROCPROFSYS_STRICT_CONFIG | Throw errors for unknown setting nam... |
| ROCPROFSYS_SUPPRESS_CONFIG | Disable processing of setting config... |
| ROCPROFSYS_SUPPRESS_PARSING | Disable parsing environment |
| ROCPROFSYS_TEXT_OUTPUT | Write text output files |
| ROCPROFSYS_TIMELINE_PROFILE | Set the label hierarchy mode to defa... |
| ROCPROFSYS_TIMEMORY_COMPONENTS | List of components to collect via ti... |
| ROCPROFSYS_TIME_FORMAT | Customize the folder generation when... |
| ROCPROFSYS_TIME_OUTPUT | Output data to subfolder w/ a timest... |
| ROCPROFSYS_TIMING_PRECISION | Set the precision for components wit... |
| ROCPROFSYS_TIMING_SCIENTIFIC | Set the numerical reporting format f... |
| ROCPROFSYS_TIMING_UNITS | Set the units for components with u... |
| ROCPROFSYS_TIMING_WIDTH | Set the output width for components ... |
| ROCPROFSYS_TRACE_THREAD_LOCKS | Enable tracking calls to pthread_mut... |
| ROCPROFSYS_TREE_OUTPUT | Write hierarchical json output files |
| ROCPROFSYS_USE_CODE_COVERAGE | Enable support for code coverage |
| ROCPROFSYS_USE_KOKKOSP | Enable support for Kokkos Tools |
| ROCPROFSYS_USE_OMPT | Enable support for OpenMP-Tools |
| ROCPROFSYS_TRACE | Enable perfetto backend |
| ROCPROFSYS_USE_PID | Enable tagging filenames with proces... |
| ROCPROFSYS_USE_ROCM_SMI | Enable sampling GPU power, temp, uti... |
| ROCPROFSYS_USE_ROCTRACER | Enable ROCM tracing |
| ROCPROFSYS_USE_SAMPLING | Enable statistical sampling of call-... |
| ROCPROFSYS_USE_PROCESS_SAMPLING | Enable a background thread which sam... |
| ROCPROFSYS_PROFILE | Enable timemory backend |
| ROCPROFSYS_VERBOSE | Verbosity level |
| ROCPROFSYS_WIDTH | Set the global output width for comp... |
|-----------------------------------------|-----------------------------------------|
Viewing components
@@ -402,7 +404,7 @@ Viewing components
.. code-block:: shell
$ omnitrace-avail -C -bd
$ rocprof-sys-avail -C -bd
|-----------------------------------|----------------------------------------------|
| COMPONENT | DESCRIPTION |
|-----------------------------------|----------------------------------------------|
@@ -460,7 +462,7 @@ Viewing components
| wall_clock | Real-clock timer (i.e. wall-clock timer). |
| written_bytes | Number of bytes sent to the storage layer. |
| written_char | Number of bytes which this task has cause... |
| omnitrace | Invokes instrumentation functions omnitr... |
| rocprof-sys | Invokes instrumentation functions rocprof... |
| roctracer | High-precision ROCm API and kernel tracing. |
| sampling_wall_clock | Wall-clock timing. Derived from statistic... |
| sampling_cpu_clock | CPU-clock timing. Derived from statistica... |
@@ -476,7 +478,7 @@ Viewing hardware counters
.. code-block:: shell
$ omnitrace-avail -H -bd
$ rocprof-sys-avail -H -bd
|---------------------------------------|---------------------------------------|
| HARDWARE COUNTER | DESCRIPTION |
|---------------------------------------|---------------------------------------|
@@ -1197,17 +1199,17 @@ Viewing hardware counters
Creating a configuration file
========================================
Omnitrace supports three configuration file formats: JSON, XML, and plain text.
Use ``omnitrace-avail -G <filename> -F txt json xml`` to generate default
ROCm Systems Profiler supports three configuration file formats: JSON, XML, and plain text.
Use ``rocprof-sys-avail -G <filename> -F txt json xml`` to generate default
configuration files in each format. Optionally
include the ``--all`` flag to include full descriptions and other information.
Configuration files are specified by the ``OMNITRACE_CONFIG_FILE`` environment variable
which by default looks for ``${HOME}/.omnitrace.cfg`` and ``${HOME}/.omnitrace.json``.
Configuration files are specified by the ``ROCPROFSYS_CONFIG_FILE`` environment variable
which by default looks for ``${HOME}/.rocprof-sys.cfg`` and ``${HOME}/.rocprof-sys.json``.
Multiple configuration files can be concatenated using the ``:`` symbol, for example:
.. code-block:: shell
export OMNITRACE_CONFIG_FILE=~/.config/omnitrace.cfg:~/.config/omnitrace.json
export ROCPROFSYS_CONFIG_FILE=~/.config/rocprof-sys.cfg:~/.config/rocprof-sys.json
If a configuration variable is specified in both a configuration file and in the environment,
the environment variable takes precedence.
@@ -1220,7 +1222,7 @@ Variables are created when an lvalue starts with a ``$`` and are
de-referenced when they appear as rvalues.
Entries in the text configuration file which do not match a known setting
in ``omnitrace-avail`` but are prefixed with ``OMNITRACE_`` are interpreted as
in ``rocprof-sys-avail`` but are prefixed with ``ROCPROFSYS_`` are interpreted as
environment variables. They are exported via ``setenv``
but do not override an existing value for the environment variable.
@@ -1231,35 +1233,35 @@ but do not override an existing value for the environment variable.
$SAMPLE = OFF
# use fields
OMNITRACE_TRACE = $ENABLE
OMNITRACE_PROFILE = $ENABLE
OMNITRACE_USE_SAMPLING = $SAMPLE
OMNITRACE_USE_PROCESS_SAMPLING = $SAMPLE
ROCPROFSYS_TRACE = $ENABLE
ROCPROFSYS_PROFILE = $ENABLE
ROCPROFSYS_USE_SAMPLING = $SAMPLE
ROCPROFSYS_USE_PROCESS_SAMPLING = $SAMPLE
# debug
OMNITRACE_DEBUG = OFF
OMNITRACE_VERBOSE = 1
ROCPROFSYS_DEBUG = OFF
ROCPROFSYS_VERBOSE = 1
# output fields
OMNITRACE_OUTPUT_PATH = omnitrace-output
OMNITRACE_OUTPUT_PREFIX = %tag%/
OMNITRACE_TIME_OUTPUT = OFF
OMNITRACE_USE_PID = OFF
ROCPROFSYS_OUTPUT_PATH = rocprof-sys-output
ROCPROFSYS_OUTPUT_PREFIX = %tag%/
ROCPROFSYS_TIME_OUTPUT = OFF
ROCPROFSYS_USE_PID = OFF
# timemory fields
OMNITRACE_PAPI_EVENTS = PAPI_TOT_INS PAPI_FP_INS
OMNITRACE_TIMEMORY_COMPONENTS = wall_clock peak_rss trip_count
OMNITRACE_MEMORY_UNITS = MB
OMNITRACE_TIMING_UNITS = sec
ROCPROFSYS_PAPI_EVENTS = PAPI_TOT_INS PAPI_FP_INS
ROCPROFSYS_TIMEMORY_COMPONENTS = wall_clock peak_rss trip_count
ROCPROFSYS_MEMORY_UNITS = MB
ROCPROFSYS_TIMING_UNITS = sec
# sampling fields
OMNITRACE_SAMPLING_FREQ = 50
OMNITRACE_SAMPLING_DELAY = 0.1
OMNITRACE_SAMPLING_CPUS = 0-3
OMNITRACE_SAMPLING_GPUS = $env:HIP_VISIBLE_DEVICES
ROCPROFSYS_SAMPLING_FREQ = 50
ROCPROFSYS_SAMPLING_DELAY = 0.1
ROCPROFSYS_SAMPLING_CPUS = 0-3
ROCPROFSYS_SAMPLING_GPUS = $env:HIP_VISIBLE_DEVICES
# misc env variables (see metadata JSON file after run)
$env:OMNITRACE_SAMPLING_KEEP_DYNINST_SUFFIX = OFF
$env:ROCPROFSYS_SAMPLING_KEEP_DYNINST_SUFFIX = OFF
Sample JSON configuration file
-----------------------------------
@@ -1269,9 +1271,9 @@ The full JSON specification for a configuration value contains a lot of informat
.. code-block:: json
{
"omnitrace": {
"rocprof-sys": {
"settings": {
"OMNITRACE_ADD_SECONDARY": {
"ROCPROFSYS_ADD_SECONDARY": {
"count": -1,
"name": "add_secondary",
"data_type": "bool",
@@ -1279,9 +1281,9 @@ The full JSON specification for a configuration value contains a lot of informat
"value": true,
"max_count": 1,
"cmdline": [
"--omnitrace-add-secondary"
"--rocprof-sys-add-secondary"
],
"environ": "OMNITRACE_ADD_SECONDARY",
"environ": "ROCPROFSYS_ADD_SECONDARY",
"cereal_class_version": 1,
"categories": [
"component",
@@ -1294,15 +1296,15 @@ The full JSON specification for a configuration value contains a lot of informat
}
}
However when writing an JSON configuration file, the following example is minimally acceptable
for ``OMNITRACE_ADD_SECONDARY``:
However when writing an JSON configuration file, the following example is minimally acceptable
for ``ROCPROFSYS_ADD_SECONDARY``:
.. code-block:: json
{
"omnitrace": {
"rocprof-sys": {
"settings": {
"OMNITRACE_ADD_SECONDARY": {
"ROCPROFSYS_ADD_SECONDARY": {
"value": true
}
}
@@ -1318,19 +1320,19 @@ The full XML specification for a configuration value contains the same informati
<?xml version="1.0" encoding="utf-8"?>
<timemory_xml>
<omnitrace>
<rocprofiler-systems>
<settings>
<cereal_class_version>2</cereal_class_version>
<!-- Full setting specification -->
<OMNITRACE_ADD_SECONDARY>
<ROCPROFSYS_ADD_SECONDARY>
<cereal_class_version>1</cereal_class_version>
<name>add_secondary</name>
<environ>OMNITRACE_ADD_SECONDARY</environ>
<environ>ROCPROFSYS_ADD_SECONDARY</environ>
<description>...</description>
<count>-1</count>
<max_count>1</max_count>
<cmdline>
<value0>--omnitrace-add-secondary</value0>
<value0>--rocprof-sys-add-secondary</value0>
</cmdline>
<categories>
<value0>component</value0>
@@ -1340,24 +1342,24 @@ The full XML specification for a configuration value contains the same informati
<data_type>bool</data_type>
<initial>true</initial>
<value>true</value>
</OMNITRACE_ADD_SECONDARY>
</ROCPROFSYS_ADD_SECONDARY>
<!-- etc. -->
</settings>
</omnitrace>
</rocprofiler-systems>
</timemory_xml>
However, when writing an XML configuration file, it is minimally acceptable
to set ``OMNITRACE_ADD_SECONDARY=false``:
However, when writing an XML configuration file, it is minimally acceptable
to set ``ROCPROFSYS_ADD_SECONDARY=false``:
.. code-block:: xml
<?xml version="1.0" encoding="utf-8"?>
<timemory_xml>
<omnitrace>
<rocprofiler-systems>
<settings>
<OMNITRACE_ADD_SECONDARY>
<ROCPROFSYS_ADD_SECONDARY>
<value>false</value>
</OMNITRACE_ADD_SECONDARY>
</ROCPROFSYS_ADD_SECONDARY>
</settings>
</omnitrace>
</rocprofiler-systems>
</timemory_xml>
projects/rocprofiler-systems/docs/how-to/configuring-validating-environment.rst
+24 -24
View File
@@ -1,47 +1,47 @@
.. meta::
:description: Omnitrace documentation and reference
:keywords: Omnitrace, ROCm, profiler, tracking, visualization, tool, Instinct, accelerator, AMD
:description: ROCm Systems Profiler environment validation documentation and reference
:keywords: rocprof-sys, rocprofiler-systems, Omnitrace, ROCm, profiler, environment, tracking, visualization, tool, Instinct, accelerator, AMD
****************************************************
Configuring and validating the environment
****************************************************
After installing `Omnitrace <https://github.com/ROCm/omnitrace>`_, additional steps are required to set up
After installing `ROCm Systems Profiler <https://github.com/ROCm/rocprofiler-systems>`_, additional steps are required to set up
and validate the environment.
.. note::
The following instructions use the installation path ``/opt/omnitrace``. If
Omnitrace is installed elsewhere, substitute the actual installation path.
The following instructions use the installation path ``/opt/rocprofiler-systems``. If
ROCm Systems Profiler is installed elsewhere, substitute the actual installation path.
Configuring the environment
========================================
After Omnitrace is installed, source the ``setup-env.sh`` script to prefix the
After ROCm Systems Profiler is installed, source the ``setup-env.sh`` script to prefix the
``PATH``, ``LD_LIBRARY_PATH``, and other environment variables:
.. code-block:: shell
source /opt/omnitrace/share/omnitrace/setup-env.sh
source /opt/rocprofiler-systems/share/rocprofiler-systems/setup-env.sh
Alternatively, if environment modules are supported, add the ``<prefix>/share/modulefiles`` directory
to ``MODULEPATH``:
.. code-block:: shell
module use /opt/omnitrace/share/modulefiles
module use /opt/rocprofiler-systems/share/modulefiles
.. note::
As an alternative, the above line can be added to the ``${HOME}/.modulerc`` file.
After Omnitrace has been added to the ``MODULEPATH``, it can be loaded
using ``module load omnitrace/<VERSION>`` and unloaded using ``module unload omnitrace/<VERSION>``.
After ROCm Systems Profiler has been added to the ``MODULEPATH``, it can be loaded
using ``module load rocprofiler-systems/<VERSION>`` and unloaded using ``module unload rocprofiler-systems/<VERSION>``.
.. code-block:: shell
module load omnitrace/1.0.0
module unload omnitrace/1.0.0
module load rocprofiler-systems/1.0.0
module unload rocprofiler-systems/1.0.0
.. note::
@@ -51,21 +51,21 @@ using ``module load omnitrace/<VERSION>`` and unloaded using ``module unload omn
Validating the environment configuration
========================================
If the following commands all run successfully with the expected output,
then you are ready to use Omnitrace:
If the following commands all run successfully with the expected output,
then you are ready to use ROCm Systems Profiler:
.. code-block:: shell
which omnitrace
which omnitrace-avail
which omnitrace-sample
omnitrace-instrument --help
omnitrace-avail --all
omnitrace-sample --help
which rocprof-sys
which rocprof-sys-avail
which rocprof-sys-sample
rocprof-sys-instrument --help
rocprof-sys-avail --all
rocprof-sys-sample --help
If Omnitrace was built with Python support, validate these additional commands:
If ROCm Systems Profiler was built with Python support, validate these additional commands:
.. code-block:: shell
which omnitrace-python
omnitrace-python --help
which rocprof-sys-python
rocprof-sys-python --help
projects/rocprofiler-systems/docs/how-to/general-tips-using-omnitrace.rst → projects/rocprofiler-systems/docs/how-to/general-tips-using-rocprof-sys.rst
+17 -17
View File
@@ -1,19 +1,19 @@
.. meta::
:description: Omnitrace documentation and reference
:keywords: Omnitrace, ROCm, profiler, tracking, visualization, tool, Instinct, accelerator, AMD
:description: ROCm Systems Profiler general tips and usage documentation and reference
:keywords: rocprof-sys, rocprofiler-systems, Omnitrace, ROCm, tips, how to, profiler, tracking, visualization, tool, Instinct, accelerator, AMD
**********************************
General tips for using Omnitrace
General tips for using ROCm Systems Profiler
**********************************
Follow these general guidelines when using Omnitrace. For an explanation of the terms used in this topic, see
the :doc:`Omnitrace glossary <../reference/omnitrace-glossary>`.
Follow these general guidelines when using ROCm Systems Profiler. For an explanation of the terms used in this topic, see
the :doc:`ROCm Systems Profiler glossary <../reference/rocprof-sys-glossary>`.
* Use ``omnitrace-avail`` to look up configuration settings, hardware counters, and data collection components
* Use ``rocprof-sys-avail`` to look up configuration settings, hardware counters, and data collection components
* Use the ``-d`` flag for descriptions
* Generate a default configuration with ``omnitrace-avail -G ${HOME}/.omnitrace.cfg`` and adjust it
* Generate a default configuration with ``rocprof-sys-avail -G ${HOME}/.rocprof-sys.cfg`` and adjust it
to the desired default behavior
* **Decide whether binary instrumentation, statistical sampling, or both** provides the desired performance data (for non-Python applications)
* Compile code with optimization enabled (``-O2`` or higher), disable asserts (i.e. ``-DNDEBUG``), and include debug info (for instance, ``-g1`` at a minimum)
@@ -24,26 +24,26 @@ the :doc:`Omnitrace glossary <../reference/omnitrace-glossary>`.
* **Use binary instrumentation for characterizing the performance of every invocation of specific functions**
* **Use statistical sampling to characterize the performance of the entire application while minimizing overhead**
* Enable statistical sampling after binary instrumentation to help "fill in the gaps" between instrumented regions
* Use the user API to create custom regions and enable/disable Omnitrace for specific processes, threads, and regions
* Use the user API to create custom regions and enable/disable ROCm Systems Profiler for specific processes, threads, and regions
* Dynamic symbol interception, callback APIs, and the user API are always available with binary instrumentation and sampling
* Dynamic symbol interception and callback APIs are (generally) controlled through ``OMNITRACE_USE_<API>``
options, for example, ``OMNITRACE_USE_KOKKOSP`` and ``OMNITRACE_USE_OMPT`` enable Kokkos-Tools and OpenMP-Tools
* Dynamic symbol interception and callback APIs are (generally) controlled through ``ROCPROFSYS_USE_<API>``
options, for example, ``ROCPROFSYS_USE_KOKKOSP`` and ``ROCPROFSYS_USE_OMPT`` enable Kokkos-Tools and OpenMP-Tools
callbacks, respectively
* When generically seeking regions for performance improvement:
* **Start off by collecting a flat profile**
* Look for functions with high call counts, large cumulative runtimes/values, or large standard deviations
* When call counts are high, improving the performance of this function or "inlining" the function can result in quick and easy performance improvements
* When the standard deviation is high, collect a hierarchical profile and see if the high variation can be attributable to the calling context.
* When the standard deviation is high, collect a hierarchical profile and see if the high variation can be attributable to the calling context.
In this scenario, consider creating a specialized version of the function for the longer-running contexts
* **Collect a hierarchical profile** and verify the functions that are part of the "critical path" of your
* **Collect a hierarchical profile** and verify the functions that are part of the "critical path" of your
application, as indicated in the flat profile
* For example, functions with high call counts but which are part of a "setup" or "post-processing"
* For example, functions with high call counts but which are part of a "setup" or "post-processing"
phase that does not consume much time relative to the overall time are generally a lower priority for optimization
* **Use the information from the profiles when analyzing detailed traces**
@@ -54,7 +54,7 @@ the :doc:`Omnitrace glossary <../reference/omnitrace-glossary>`.
* When using binary instrumentation with MPI, avoid runtime instrumentation
* Runtime instrumentation requires a fork and a ``ptrace``, which is generally incompatible with how MPI applications spawn processes
* Perform a binary rewrite of the executable (and optionally, libraries used by the executable) using MPI and run
the generated instrumented executable using ``omnitrace-run`` instead of the original.
For example, instead of ``mpirun -n 2 ./myexe``, use ``mpirun -n 2 omnitrace-run -- ./myexe.inst``, where
* Perform a binary rewrite of the executable (and optionally, libraries used by the executable) using MPI and run
the generated instrumented executable using ``rocprof-sys-run`` instead of the original.
For example, instead of ``mpirun -n 2 ./myexe``, use ``mpirun -n 2 rocprof-sys-run -- ./myexe.inst``, where
``myexe.inst`` is the instrumented ``myexe`` executable that was generated.
projects/rocprofiler-systems/docs/how-to/instrumenting-rewriting-binary-application.rst
+94 -94
View File
@@ -1,12 +1,12 @@
.. meta::
:description: Omnitrace documentation and reference
:keywords: Omnitrace, ROCm, profiler, tracking, visualization, tool, Instinct, accelerator, AMD
:description: ROCm Systems Profiler binary instrumentation and rewrite documentation and reference
:keywords: rocprof-sys, rocprofiler-systems, Omnitrace, ROCm, binary instrumentation, binary rewrite, profiler, tracking, visualization, tool, Instinct, accelerator, AMD
****************************************************
Instrumenting and rewriting a binary application
****************************************************
There are three ways to perform instrumentation with the ``omnitrace-instrument`` executable:
There are three ways to perform instrumentation with the ``rocprof-sys-instrument`` executable:
* Runtime instrumentation
* Attaching to an already running process
@@ -14,11 +14,11 @@ There are three ways to perform instrumentation with the ``omnitrace-instrument`
Here is a comparison of the three modes:
* Runtime instrumentation of the application using the ``omnitrace-instrument`` executable
* Runtime instrumentation of the application using the ``rocprof-sys-instrument`` executable
(analogous to ``gdb --args <program> <args>``)
* This mode is the default if neither the ``-p`` nor ``-o`` command-line options are used
* Runtime instrumentation supports instrumenting not only the target executable but also
* Runtime instrumentation supports instrumenting not only the target executable but also
the shared libraries loaded by the target executable. Consequently, this mode consumes more memory,
takes longer to perform the instrumentation, and tends to add more significant overhead to the
runtime of the application.
@@ -26,7 +26,7 @@ Here is a comparison of the three modes:
libraries but also the performance of the library dependencies
* Attaching to a process that is currently running (analogous to ``gdb -p <PID>``)
* This mode is activated using ``-p <PID>``
* The same caveats from the first example apply with respect to memory and overhead
@@ -39,25 +39,25 @@ Here is a comparison of the three modes:
* This mode is activated through the ``-o <output-file>`` option
* Binary rewriting is limited to the text section of the target executable or library. It does not instrument
the dynamically-linked libraries. Consequently, this mode performs the
the dynamically-linked libraries. Consequently, this mode performs the
instrumentation significantly faster
and has a much lower overhead when running the instrumented executable and libraries.
* Binary rewriting is the recommended mode when the target executable uses
* Binary rewriting is the recommended mode when the target executable uses
process-level parallelism (for example, MPI)
* If the target executable has a minimal ``main`` routine and the bulk of your
* If the target executable has a minimal ``main`` routine and the bulk of your
application is in one specific dynamic library,
see :ref:`binary-rewriting-library-label` for help
The omnitrace-instrument executable
The rocprof-sys-instrument executable
========================================
Instrumentation is performed with the ``omnitrace-instrument`` executable. For more details, use the ``-h`` or ``--help`` option to
Instrumentation is performed with the ``rocprof-sys-instrument`` executable. For more details, use the ``-h`` or ``--help`` option to
view the help menu.
.. code-block:: shell
$ omnitrace-instrument --help
[omnitrace-instrument] Usage: omnitrace-instrument [ --help (count: 0, dtype: bool)
$ rocprof-sys-instrument --help
[rocprof-sys-instrument] Usage: rocprof-sys-instrument [ --help (count: 0, dtype: bool)
--version (count: 0, dtype: bool)
--verbose (max: 1, dtype: bool)
--error (max: 1, dtype: boolean)
@@ -161,8 +161,8 @@ view the help menu.
[MODE OPTIONS]
-o, --output Enable generation of a new executable (binary-rewrite). If a filename is not provided,
omnitrace will use the basename and output to the cwd, unless the target binary is in the
cwd. In the latter case, omnitrace will either use ${PWD}/<basename>.inst (non-libraries)
rocprof-sys will use the basename and output to the cwd, unless the target binary is in the
cwd. In the latter case, rocprof-sys will either use ${PWD}/<basename>.inst (non-libraries)
or ${PWD}/instrumented/<basename> (libraries)
-p, --pid Connect to running process
-M, --mode [ coverage | sampling | trace ]
@@ -177,7 +177,7 @@ view the help menu.
[LIBRARY OPTIONS]
--prefer [ shared | static ] Prefer this library types when available
-L, --library Libraries with instrumentation routines (default: "libomnitrace-dl")
-L, --library Libraries with instrumentation routines (default: "librocprof-sys-dl")
-m, --main-function The primary function to instrument around, e.g. \'main\'
--load Supplemental instrumentation library names w/o extension (e.g. \'libinstr\' for
\'libinstr.so\' or \'libinstr.a\')
@@ -200,17 +200,17 @@ view the help menu.
-ME, --module-exclude Regex(es) for excluding modules/files/libraries (always applied)
-MR, --module-restrict Regex(es) for restricting modules/files/libraries only to those that match the provided
regular-expressions
--internal-function-include Regex(es) for including functions which are (likely) utilized by omnitrace itself. Use
--internal-function-include Regex(es) for including functions which are (likely) utilized by rocprof-sys itself. Use
this option with care.
--internal-module-include Regex(es) for including modules/libraries which are (likely) utilized by omnitrace
--internal-module-include Regex(es) for including modules/libraries which are (likely) utilized by rocprof-sys
itself. Use this option with care.
--instruction-exclude Regex(es) for excluding functions containing certain instructions
--internal-library-deps Treat the libraries linked to the internal libraries as internal libraries. This increase
the internal library processing time and consume more memory (so use with care) but may
be useful when the application uses Boost libraries and Dyninst is dynamically linked
against the same boost libraries
--internal-library-append Append to the list of libraries which omnitrace treats as being used internally, e.g.
OmniTrace will find all the symbols in this library and prevent them from being
--internal-library-append Append to the list of libraries which rocprof-sys treats as being used internally, e.g.
ROCm Systems Profiler will find all the symbols in this library and prevent them from being
instrumented.
--internal-library-remove [ ld-linux-x86-64.so.2
libBrokenLocale.so.1
@@ -272,7 +272,7 @@ view the help menu.
libz.so
libzstd.so ]
Remove the specified libraries from being treated as being used internally, e.g.
OmniTrace will permit all the symbols in these libraries to be eligible for
ROCm System Profiler will permit all the symbols in these libraries to be eligible for
instrumentation.
--linkage [ global | local | unique | unknown | weak ]
Only instrument functions with specified linkage (default: global, local, unique)
@@ -287,11 +287,11 @@ view the help menu.
options to gain more information about the function signature or location of the
functions
-C, --config Read in a configuration file and encode these values as the defaults in the executable
-d, --default-components Default components to instrument (only useful when timemory is enabled in omnitrace
-d, --default-components Default components to instrument (only useful when timemory is enabled in rocprof-sys
library)
--env Environment variables to add to the runtime in form VARIABLE=VALUE. E.g. use \'--env
OMNITRACE_PROFILE=ON\' to default to using timemory instead of perfetto
--mpi Enable MPI support (requires omnitrace built w/ full or partial MPI support). NOTE: this
ROCPROFSYS_PROFILE=ON\' to default to using timemory instead of perfetto
--mpi Enable MPI support (requires rocprof-sys built w/ full or partial MPI support). NOTE: this
will automatically be activated if MPI_Init, MPI_Init_thread, MPI_Finalize,
MPI_Comm_rank, or MPI_Comm_size are found in the symbol table of target
@@ -322,8 +322,8 @@ view the help menu.
--allow-overlapping Allow dyninst to instrument either multiple functions which overlap (share part of same
function body) or single functions with multiple entry points. For more info, see Section
2 of the DyninstAPI documentation.
--parse-all-modules By default, omnitrace simply requests Dyninst to provide all the procedures in the
application image. If this option is enabled, omnitrace will iterate over all the modules
--parse-all-modules By default, rocprof-sys simply requests Dyninst to provide all the procedures in the
application image. If this option is enabled, rocprof-sys will iterate over all the modules
and extract the functions. Theoretically, it should be the same but the data is slightly
different, possibly due to weak binding scopes. In general, enabling option will probably
have no visible effect
@@ -344,17 +344,17 @@ view the help menu.
TypeChecking ]
Advanced dyninst options: BPatch::set<OPTION>(bool), e.g. bpatch->setTrampRecursive(true)
``omnitrace-instrument`` uses a similar syntax as LLVM to separate command-line arguments from the
application's arguments. It uses a standalone
double-hyphen (``--``) as a separator.
``rocprof-sys-instrument`` uses a similar syntax as LLVM to separate command-line arguments from the
application's arguments. It uses a standalone
double-hyphen (``--``) as a separator.
All arguments preceding the double-hyphen
are interpreted as belonging to Omnitrace and all arguments following the
are interpreted as belonging to ROCm Systems Profiler and all arguments following the
double-hyphen are interpreted as being part of the
application and its arguments. In binary rewrite mode, all application arguments after the first argument
are ignored. As an example, ``./omnitrace-instrument -o ls.inst -- ls -l`` interprets ``ls`` as
are ignored. As an example, ``./rocprof-sys-instrument -o ls.inst -- ls -l`` interprets ``ls`` as
the target to instrument, ignoring the ``-l`` argument,
and generates a ``ls.inst`` executable that you can subsequently run using the
``omnitrace-run -- ls.inst -l`` command.
and generates a ``ls.inst`` executable that you can subsequently run using the
``rocprof-sys-run -- ls.inst -l`` command.
Runtime instrumentation example
========================================
@@ -363,7 +363,7 @@ The following example shows how to enable runtime instrumentation.
.. code-block:: shell
omnitrace-instrument <omnitrace-options> -- <exe> [<exe-options>...]
rocprof-sys-instrument <rocprof-sys-options> -- <exe> [<exe-options>...]
Attaching to a running process
========================================
@@ -372,7 +372,7 @@ Use the following command to attach to an active process.
.. code-block:: shell
omnitrace-instrument <omnitrace-options> -p <PID> -- <exe-name>
rocprof-sys-instrument <rocprof-sys-options> -p <PID> -- <exe-name>
Binary rewrite
========================================
@@ -381,24 +381,24 @@ This example demonstrates how to rewrite a binary.
.. code-block:: shell
omnitrace-instrument <omnitrace-options> -o <name-of-new-exe-or-library> -- <exe-or-library>
rocprof-sys-instrument <rocprof-sys-options> -o <name-of-new-exe-or-library> -- <exe-or-library>
.. _binary-rewriting-library-label:
Binary rewrite of a library
-----------------------------------
Many applications bundle the bulk of their functionality into one or more
Many applications bundle the bulk of their functionality into one or more
dynamic libraries and have a relatively simple ``main``
which links to these libraries and serves as the "driver" for
which links to these libraries and serves as the "driver" for
setting up the workflow. If you perform a binary rewrite of an
executable like this and find there is insufficient information, you
executable like this and find there is insufficient information, you
can either switch to runtime instrumentation or perform a
binary rewrite on the relevant libraries.
Support for stand-alone binary rewriting of a dynamic library without a binary rewrite of
Support for stand-alone binary rewriting of a dynamic library without a binary rewrite of
the executable is a beta feature.
In general, it is supported as long as the library contains the ``_init`` and
In general, it is supported as long as the library contains the ``_init`` and
``_fini`` symbols but these symbols are not
standardized to the extent of ``main`` in an executable.
@@ -406,8 +406,8 @@ Here is the recommended workflow for the binary rewrite of a library:
#. Determine the names of the dynamically linked libraries of interest using ``ldd``
#. Generate a binary rewrite of the executable
#. Generate a binary rewrite of the desired libraries with the same base name as the
original library, for example, ``libfoo.so.2`` instead of ``libfoo.so``, and output the instrumented
#. Generate a binary rewrite of the desired libraries with the same base name as the
original library, for example, ``libfoo.so.2`` instead of ``libfoo.so``, and output the instrumented
library into a different folder than the original library.
#. Prefix the ``LD_LIBRARY_PATH`` executable with the output folder from the previous step
@@ -433,10 +433,10 @@ Generate binary rewrites of ``foo`` and ``libfoo.so.2``:
.. code-block:: shell
omnitrace-instrument -o ./foo.inst -- foo
omnitrace-instrument -o ./libfoo.so.2 -- /usr/local/lib/libfoo.so.2
rocprof-sys-instrument -o ./foo.inst -- foo
rocprof-sys-instrument -o ./libfoo.so.2 -- /usr/local/lib/libfoo.so.2
At this point, the instrumented ``foo.inst`` executable still dynamically loads the
At this point, the instrumented ``foo.inst`` executable still dynamically loads the
original ``libfoo.so.2`` in ``/usr/local/lib``:
.. code-block:: shell
@@ -446,7 +446,7 @@ original ``libfoo.so.2`` in ``/usr/local/lib``:
libfoo.so.2 => /usr/local/lib/libfoo.so.2 (...)
...
Prefix the ``LD_LIBRARY_PATH`` environment variable with the folder containing
Prefix the ``LD_LIBRARY_PATH`` environment variable with the folder containing
the instrumented ``libfoo.so.2``:
.. code-block:: shell
@@ -465,90 +465,90 @@ the instrumented ``libfoo.so.2``:
Selective instrumentation
========================================
The default behavior of ``omnitrace-instrument`` does not instrument every symbol in the binary.
The default behavior of ``rocprof-sys-instrument`` does not instrument every symbol in the binary.
The default rules are:
* Skip instrumenting dynamic call-sites (such as function pointers)
* The ``--dynamic-callsites`` option forces instrumentation for all dynamic call-sites
* The cost of a function can be loosely approximated by the number of
instructions. By default, ``omnitrace-instrument`` only instruments functions
* The cost of a function can be loosely approximated by the number of
instructions. By default, ``rocprof-sys-instrument`` only instruments functions
with at least 1024 instructions
* The ``--min-instructions`` option modifies this heuristic for all functions which do not contain loops
* The ``--min-instructions-loop`` option modifies this heuristic for functions which contain loops.
* The cost of a function can be also be loosely approximated by the size of the function
in the binary so this heuristic can be used in lieu of or in addition to the
* The cost of a function can be also be loosely approximated by the size of the function
in the binary so this heuristic can be used in lieu of or in addition to the
minimum number of instructions
* The ``--min-address-range`` option modifies this heuristic for all functions which do not contain loops
* The ``--min-address-range-loop`` option modifies this heuristic for functions which contain loops
* The ``--min-address-range-loop`` option modifies this heuristic for functions which contain loops
* Skip instrumentation points which require using a trap
* See the description for the ``--traps`` and ``--loop-traps`` options for more information
* Skip instrumenting loops within the body of a function
* The ``--instrument-loops`` option enables this behavior
* Skip instrumenting functions with overlapping function bodies and single
* Skip instrumenting functions with overlapping function bodies and single
functions with multiple entry point
* These behaviors arise from various optimizations. Enable instrumenting for these functions
* These behaviors arise from various optimizations. Enable instrumenting for these functions
by using the ``--allow-overlapping`` option
.. note::
The separate loop options ``--min-instructions-loop`` and ``--min-address-range-loop``
The separate loop options ``--min-instructions-loop`` and ``--min-address-range-loop``
are provided because functions with loops can be compact in the binary while also being costly
Viewing the available, instrumented, excluded, and overlapping functions
-------------------------------------------------------------------------
Whenever ``omnitrace-instrument`` runs with a verbosity of zero or higher,
it generates files that detail which functions
were available for instrumentation (along with the module they were defined in), actually instrumented,
Whenever ``rocprof-sys-instrument`` runs with a verbosity of zero or higher,
it generates files that detail which functions
were available for instrumentation (along with the module they were defined in), actually instrumented,
excluded, and which contained overlapping function bodies.
By default, these files are saved to the ``omnitrace-<NAME>-output`` folder
By default, these files are saved to the ``rocprof-sys-<NAME>-output`` folder
where ``<NAME>`` is the base name of the targeted binary (or
the base name of the resulting executable in the case of binary rewrite). For example,
``omnitrace-instrument -- ls`` outputs these files to ``omnitrace-ls-output``
whereas ``omnitrace-instrument -o ls.inst -- ls`` places them in ``omnitrace-ls.inst-output``.
``rocprof-sys-instrument -- ls`` outputs these files to ``rocprof-sys-ls-output``
whereas ``rocprof-sys-instrument -o ls.inst -- ls`` places them in ``rocprof-sys-ls.inst-output``.
To generate these files without running or generating an
To generate these files without running or generating an
executable, use the ``--simulate`` option:
.. code-block:: shell
omnitrace-instrument --simulate -- foo
omnitrace-instrument --simulate -o foo.inst -- foo
rocprof-sys-instrument --simulate -- foo
rocprof-sys-instrument --simulate -o foo.inst -- foo
Excluding and including modules and functions
----------------------------------------------
Omnitrace has a set of six command-line options which each accept one or more
ROCm Systems Profiler has a set of six command-line options which each accept one or more
regular expressions for customizing the scope of which module and/or functions are
instrumented. Multiple regex patterns per option are treated as an OR operation,
instrumented. Multiple regex patterns per option are treated as an OR operation,
for example, ``--module-include libfoo libbar`` is effectively the same as ``--module-include 'libfoo|libbar'``.
To force the inclusion of certain modules and/or function
To force the inclusion of certain modules and/or function
without changing any of the heuristics, use the ``--module-include`` and/or ``--function-include`` options.
These options do not exclude modules or functions which do
These options do not exclude modules or functions which do
not satisfy their regular expression.
To narrow the scope of the instrumentation to a specific set
To narrow the scope of the instrumentation to a specific set
of libraries and/or functions, use the ``--module-restrict`` and ``--function-restrict`` options.
These options let you exclusively select the union of one or more
These options let you exclusively select the union of one or more
regular expressions, regardless of whether or not the functions satisfy the
previously-mentioned default heuristics. Any function or module that is not within
previously-mentioned default heuristics. Any function or module that is not within
the union of these regular expressions is excluded from instrumentation.
To avoid instrumenting a set of modules and/or functions,
To avoid instrumenting a set of modules and/or functions,
use the ``--module-exclude`` and ``--function-exclude`` options.
These options are always applied, even if the module or function
These options are always applied, even if the module or function
satisfies the "restrict" or "include" regular expression.
.. _available-module-function-output:
@@ -558,7 +558,7 @@ An example of the available module and function info output
.. code-block:: shell
omnitrace-instrument -o lulesh.inst --label file line args --simulate -- lulesh
rocprof-sys-instrument -o lulesh.inst --label file line args --simulate -- lulesh
.. code-block:: shell
@@ -779,7 +779,7 @@ An example of instrumented module and function info output
.. code-block:: shell
omnitrace-instrument -o lulesh.inst --label file line args --simulate -- lulesh
rocprof-sys-instrument -o lulesh.inst --label file line args --simulate -- lulesh
After the heuristics are applied based on the pattern in :ref:`available-module-function-output`,
the selected module and functions are:
@@ -850,15 +850,15 @@ Sampling
This capability has been deprecated in favor of :doc:`Call stack sampling <./sampling-call-stack>`.
By default, ``omnitrace-instrument`` uses ``--mode trace`` for instrumentation. The ``--mode sampling`` option
By default, ``rocprof-sys-instrument`` uses ``--mode trace`` for instrumentation. The ``--mode sampling`` option
only instruments ``main`` in an executable. It activates both CPU call-stack sampling and
background system-level thread sampling by default.
Tracing capabilities which do not rely on instrumentation, such as the HIP API and kernel tracing
(which is collected by roctracer), are still available.
The Omnitrace sampling capabilities are always available, even in trace mode, but are deactivated by default.
To activate sampling in trace mode, set ``OMNITRACE_USE_SAMPLING=ON`` in the environment
or in an Omnitrace configuration file.
The ROCm Systems Profiler sampling capabilities are always available, even in trace mode, but are deactivated by default.
To activate sampling in trace mode, set ``ROCPROFSYS_USE_SAMPLING=ON`` in the environment
or in an ROCm Systems Profiler configuration file.
Embedding a default configuration
========================================
@@ -872,31 +872,31 @@ the configuration settings are not be preserved for subsequent sessions:
.. code-block:: shell
omnitrace-instrument -o ./foo.inst -- ./foo
export OMNITRACE_USE_SAMPLING=ON
export OMNITRACE_SAMPLING_FREQ=5
omnitrace-run -- ./foo.inst
rocprof-sys-instrument -o ./foo.inst -- ./foo
export ROCPROFSYS_USE_SAMPLING=ON
export ROCPROFSYS_SAMPLING_FREQ=5
rocprof-sys-run -- ./foo.inst
Whereas the following command preserves those environment variables:
.. code-block:: shell
omnitrace-instrument -o ./foo.samp --env OMNITRACE_USE_SAMPLING=ON OMNITRACE_SAMPLING_FREQ=5 -- ./foo
rocprof-sys-instrument -o ./foo.samp --env ROCPROFSYS_USE_SAMPLING=ON ROCPROFSYS_SAMPLING_FREQ=5 -- ./foo
They can now be used in future sessions.
.. code-block:: shell
# will sample 5x per second
omnitrace-run -- ./foo.samp
rocprof-sys-run -- ./foo.samp
Even though the environment variables are preserved, subsequent sessions can still override those defaults:
.. code-block:: shell
# will sample 100x per second
export OMNITRACE_SAMPLING_FREQ=100
omnitrace-run -- ./foo.samp
export ROCPROFSYS_SAMPLING_FREQ=100
rocprof-sys-run -- ./foo.samp
.. _rpath-troubleshooting:
@@ -906,10 +906,10 @@ Troubleshooting
Checking for RPATH
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
If ``ldd ./foo.inst`` from the :ref:`binary-rewriting-library-label`
section still returns ``/usr/local/lib/libfoo.so.2``, the executable could have
If ``ldd ./foo.inst`` from the :ref:`binary-rewriting-library-label`
section still returns ``/usr/local/lib/libfoo.so.2``, the executable could have
an rpath encoded in the binary.
This ELF entry results in the dynamic linker ignoring ``LD_LIBRARY_PATH`` if
This ELF entry results in the dynamic linker ignoring ``LD_LIBRARY_PATH`` if
it finds ``libfoo.so.2`` in the rpath.
Using the ``objdump`` tool, perform the following query:
@@ -923,13 +923,13 @@ If this produces output that appears similar to this output.:
RUNPATH $ORIGIN:$ORIGIN/../lib
Remove or modify the rpath to get ``foo.inst`` to resolve
Remove or modify the rpath to get ``foo.inst`` to resolve
to the instrumented ``libfoo.so.2`` as explained in the next section.
Modifying an RPATH
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
This code snippet uses the ``patchelf`` tool to modify the rpath of the given executable
This code snippet uses the ``patchelf`` tool to modify the rpath of the given executable
or library to ``/home/user``, which is where the instrumented libraries are located.
.. note::
projects/rocprofiler-systems/docs/how-to/performing-causal-profiling.rst
+110 -114
View File
@@ -1,6 +1,6 @@
.. meta::
:description: Omnitrace documentation and reference
:keywords: Omnitrace, ROCm, profiler, tracking, visualization, tool, Instinct, accelerator, AMD
:description: ROCm Systems Profiler causal profiling documentation and reference
:keywords: rocprof-sys, rocprofiler-systems, Omnitrace, ROCm, causal profiling, profiler, tracking, visualization, tool, Instinct, accelerator, AMD
****************************************************
Performing causal profiling
@@ -18,10 +18,6 @@ Thus, causal profiling works by performing experiments on blocks of code during
insert pauses to slow down all other concurrently running code. During post-processing, these experiments
are translated into calculations for the potential impact of speeding up this block of code.
.. note::
Causal profiling supersedes the original critical trace feature, which was removed in Omnitrace v1.11.0.
Consider the following C++ code executing ``foo`` and ``bar`` concurrently in two different threads
where ``foo`` is ideally 30% faster than ``bar``:
@@ -51,52 +47,52 @@ where ``foo`` is ideally 30% faster than ``bar``:
itr.join();
}
No matter how many optimizations are applied to ``foo``, the application will always
No matter how many optimizations are applied to ``foo``, the application will always
require the same amount of time
because the end-to-end performance is limited by ``bar``. However, a 5% speed-up
because the end-to-end performance is limited by ``bar``. However, a 5% speed-up
in ``bar`` results in the
end-to-end performance improving by 5%. This trend continues linearly, with a 10% speed-up
end-to-end performance improving by 5%. This trend continues linearly, with a 10% speed-up
in ``bar`` yielding a 10% speed-up in
end-to-end performance, and so on, up to a 30% speed-up, at which point ``bar`` runs as fast as ``foo``.
Any speed-up to ``bar`` beyond 30% still only yields an end-to-end performance
Any speed-up to ``bar`` beyond 30% still only yields an end-to-end performance
improvement of 30% because the application
is now limited by performance of ``foo``, as demonstrated below in the causal
is now limited by performance of ``foo``, as demonstrated below in the causal
profiling visualization:
.. image:: ../data/causal-foobar.png
:alt: Visualization of the performance improvements for two functions with causal profiling
The full details of the causal profiling methodology can be found in the paper
The full details of the causal profiling methodology can be found in the paper
`Coz: Finding Code that Counts with Causal Profiling <http://arxiv.org/pdf/1608.03676v1.pdf>`_.
The author's implementation is publicly available on `GitHub <https://github.com/plasma-umass/coz>`_.
Getting started
========================================
To effectively use causal profiling, it is important to understand a few key
To effectively use causal profiling, it is important to understand a few key
concepts, such as progress points.
Progress points
-----------------------------------
Causal profiling requires "progress points" to track progress through the code
Causal profiling requires "progress points" to track progress through the code
in between samples. Progress points must be triggered in a deterministic manner via instrumentation.
This can happen in three different ways:
* `Omnitrace <https://github.com/ROCm/omnitrace>`_ can leverage the callbacks from
Kokkos-Tools, OpenMP-Tools, roctracer, etc. and the wrappers around functions for
* `ROCm Systems Profiler <https://github.com/ROCm/rocprofiler-systems>`_ can leverage the callbacks from
Kokkos-Tools, OpenMP-Tools, roctracer, etc. and the wrappers around functions for
MPI, NUMA, RCCL, etc. to act as progress points
* Users can leverage the :doc:`runtime instrumentation capabilities <./instrumenting-rewriting-binary-application>`
* Users can leverage the :doc:`runtime instrumentation capabilities <./instrumenting-rewriting-binary-application>`
to insert progress points
* Users can leverage :doc:`User APIs <../how-to/using-omnitrace-api>`,
such as ``OMNITRACE_CAUSAL_PROGRESS``
* Users can leverage :doc:`User APIs <../how-to/using-rocprof-sys-api>`,
such as ``ROCPROFSYS_CAUSAL_PROGRESS``
.. note::
Binary rewrite to insert progress points is not supported. When a rewritten binary
runs, Dyninst translates the instruction pointer address in order to perform
the instrumentation. As a result, call stack samples never return instruction
pointer addresses within the valid Omnitrace range.
Binary rewrite to insert progress points is not supported. When a rewritten binary
runs, Dyninst translates the instruction pointer address in order to perform
the instrumentation. As a result, call stack samples never return instruction
pointer addresses within the valid ROCm Systems Profiler range.
Key concepts
-----------------------------------
@@ -104,26 +100,26 @@ Key concepts
+------------------+-------------------------------------+----------------------------------+--------------------------------------------+
| Concept | Setting | Options | Description |
+==================+=====================================+==================================+============================================+
| Backend | ``OMNITRACE_CAUSAL_BACKEND`` | ``perf``, ``timer`` | Backend for recording samples required |
| Backend | ``ROCPROFSYS_CAUSAL_BACKEND`` | ``perf``, ``timer`` | Backend for recording samples required |
| | | | to calculate the virtual speed-up |
+------------------+-------------------------------------+----------------------------------+--------------------------------------------+
| Mode | ``OMNITRACE_CAUSAL_MODE`` | ``function``, ``line`` | Select an entire function or individual |
| Mode | ``ROCPROFSYS_CAUSAL_MODE`` | ``function``, ``line`` | Select an entire function or individual |
| | | | line of code for causal experiments |
+------------------+-------------------------------------+----------------------------------+--------------------------------------------+
| End-to-end | ``OMNITRACE_CAUSAL_END_TO_END`` | Boolean | Perform a single experiment during the |
| End-to-end | ``ROCPROFSYS_CAUSAL_END_TO_END`` | Boolean | Perform a single experiment during the |
| | | | entire run (does not require |
| | | | progress points) |
+------------------+-------------------------------------+----------------------------------+--------------------------------------------+
| Fixed speed-up | ``OMNITRACE_CAUSAL_FIXED_SPEEDUP`` | one or more values from [0, 100] | Virtual speed-up or pool of virtual |
| Fixed speed-up | ``ROCPROFSYS_CAUSAL_FIXED_SPEEDUP`` | one or more values from [0, 100] | Virtual speed-up or pool of virtual |
| | | | speed-ups to randomly select |
+------------------+-------------------------------------+----------------------------------+--------------------------------------------+
| Binary scope | ``OMNITRACE_CAUSAL_BINARY_SCOPE`` | regular expression(s) | Dynamic binaries containing code for |
| Binary scope | ``ROCPROFSYS_CAUSAL_BINARY_SCOPE`` | regular expression(s) | Dynamic binaries containing code for |
| | | | experiments |
+------------------+-------------------------------------+----------------------------------+--------------------------------------------+
| Source scope | ``OMNITRACE_CAUSAL_SOURCE_SCOPE`` | regular expression(s) | ``<file>`` and/or ``<file>:<line>`` |
| Source scope | ``ROCPROFSYS_CAUSAL_SOURCE_SCOPE`` | regular expression(s) | ``<file>`` and/or ``<file>:<line>`` |
| | | | containing code to include in experiments |
+------------------+-------------------------------------+----------------------------------+--------------------------------------------+
| Function scope | ``OMNITRACE_CAUSAL_FUNCTION_SCOPE`` | regular expression(s) | Restricts experiments to matching |
| Function scope | ``ROCPROFSYS_CAUSAL_FUNCTION_SCOPE`` | regular expression(s) | Restricts experiments to matching |
| | | | functions (function mode) or lines of |
| | | | code within matching functions (line mode) |
+------------------+-------------------------------------+----------------------------------+--------------------------------------------+
@@ -137,30 +133,30 @@ Key concepts
Backends
-----------------------------------
There are two backends to choose from: ``perf`` and ``timer``.
They are used to record the samples required to calculate the virtual speedup.
There are two backends to choose from: ``perf`` and ``timer``.
They are used to record the samples required to calculate the virtual speedup.
Both backends interrupt each thread 1000 times per second (of CPU-time) to apply the virtual speed-ups.
The difference between each backend is how the samples are recorded.
There are three key differences between the two backends:
* the ``perf`` backend requires Linux Perf and elevated security priviledges
* the ``perf`` backend interrupts the application less frequently whereas the ``timer`` backend
* the ``perf`` backend interrupts the application less frequently whereas the ``timer`` backend
interrupts the application 1000 times per second of realtime
* the ``timer`` backend has less accurate call stacks due to instruction pointer skid
In general, the ``perf`` backend is preferred over the ``timer`` backend when sufficient
In general, the ``perf`` backend is preferred over the ``timer`` backend when sufficient
security priviledges permit its usage.
If ``OMNITRACE_CAUSAL_BACKEND`` is set to ``auto``, Omnitrace falls back
If ``ROCPROFSYS_CAUSAL_BACKEND`` is set to ``auto``, ROCm Systems Profiler falls back
to using the ``timer`` backend only if
the ``perf`` backend fails. If ``OMNITRACE_CAUSAL_BACKEND`` is
set to ``perf`` and using this backend fails, Omnitrace aborts.
the ``perf`` backend fails. If ``ROCPROFSYS_CAUSAL_BACKEND`` is
set to ``perf`` and using this backend fails, ROCm Systems Profiler aborts.
Instruction pointer skid
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Instruction pointer (IP) skid measures how many instructions run after the event of interest
before the program actually stops. The IP skid is calculated by subtracting
the location of the IP at the point of interest from the location of the IP
the location of the IP at the point of interest from the location of the IP
when the kernel finally stops the application.
For the ``timer`` backend, this translates to the
difference in the IP between when the timer generated a signal and when the
@@ -172,9 +168,9 @@ especially in ``line`` mode.
Installing Linux Perf
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Linux Perf is built into the kernel and may already be installed
Linux Perf is built into the kernel and may already be installed
(for instance, it is included in the default kernel for OpenSUSE).
The official method of checking whether Linux Perf is installed is
The official method of checking whether Linux Perf is installed is
checking for the existence of the file
``/proc/sys/kernel/perf_event_paranoid``. If the file exists, the kernel has Perf installed.
@@ -184,12 +180,12 @@ If this file does not exist, as with Debian-based systems like Ubuntu, run the f
apt-get install linux-tools-common linux-tools-generic linux-tools-$(uname -r)
and reboot your computer. In order to use the ``perf`` backend, the value
and reboot your computer. In order to use the ``perf`` backend, the value
of ``/proc/sys/kernel/perf_event_paranoid``
should be less than or equal to 2. If the value in this file is greater than 2, you can't
should be less than or equal to 2. If the value in this file is greater than 2, you can't
use the ``perf`` backend.
To update the paranoid level temporarily until the system is rebooted, run
To update the paranoid level temporarily until the system is rebooted, run
one of the following commands
as a superuser (where ``PARANOID_LEVEL=<N>`` has a value of ``<N>`` in the range ``[-1, 2]``):
@@ -206,18 +202,18 @@ or
To make the paranoid level persistent after a reboot, add ``kernel.perf_event_paranoid=<N>``
(where ``<N>`` is the desired paranoid level) to the ``/etc/sysctl.conf`` file.
Speed-up prediction variability and the omnitrace-causal executable
Speed-up prediction variability and the rocprof-sys-causal executable
-----------------------------------------------------------------------
Causal profiling typically requires running the application several times in
order to adequately sample all the code domains, experiment
Causal profiling typically requires running the application several times in
order to adequately sample all the code domains, experiment
with speed-ups and other techniques, and resolve statistical fluctuations.
The ``omnitrace-causal`` executable is designed to simplify this procedure:
The ``rocprof-sys-causal`` executable is designed to simplify this procedure:
.. code-block:: shell
$ omnitrace-causal --help
[omnitrace-causal] Usage: ./bin/omnitrace-causal [ --help (count: 0, dtype: bool)
$ rocprof-sys-causal --help
[rocprof-sys-causal] Usage: ./bin/rocprof-sys-causal [ --help (count: 0, dtype: bool)
--version (count: 0, dtype: bool)
--monochrome (max: 1, dtype: bool)
--debug (max: 1, dtype: bool)
@@ -246,21 +242,21 @@ The ``omnitrace-causal`` executable is designed to simplify this procedure:
This executable is designed to streamline that process.
For example (assume all commands end with \'-- <exe> <args>\'):
omnitrace-causal -n 5 -- <exe> # runs <exe> 5x with causal profiling enabled
rocprof-sys-causal -n 5 -- <exe> # runs <exe> 5x with causal profiling enabled
omnitrace-causal -s 0 5,10,15,20 # runs <exe> 2x with virtual speedups:
rocprof-sys-causal -s 0 5,10,15,20 # runs <exe> 2x with virtual speedups:
# - 0
# - randomly selected from 5, 10, 15, and 20
omnitrace-causal -F func_A func_B func_(A|B) # runs <exe> 3x with the function scope limited to:
rocprof-sys-causal -F func_A func_B func_(A|B) # runs <exe> 3x with the function scope limited to:
# 1. func_A
# 2. func_B
# 3. func_A or func_B
General tips:
- Insert progress points at hotspots in your code or use omnitrace\'s runtime instrumentation
- Insert progress points at hotspots in your code or use rocprof-sys\'s runtime instrumentation
- Note: binary rewrite will produce a incompatible new binary
- Run omnitrace-causal in "function" mode first (does not require debug info)
- Run omnitrace-causal in "line" mode when you are targeting one function (requires debug info)
- Run rocprof-sys-causal in "function" mode first (does not require debug info)
- Run rocprof-sys-causal in "line" mode when you are targeting one function (requires debug info)
- Preferably, use predictions from the "function" mode to determine which function to target
- Limit the virtual speedups to a smaller pool, e.g., 0,5,10,25,50, to get reliable predictions quicker
- Make use of the binary, source, and function scope to limit the functions/lines selected for experiments
@@ -280,15 +276,15 @@ The ``omnitrace-causal`` executable is designed to simplify this procedure:
[GENERAL OPTIONS]
-c, --config Base configuration file
-l, --launcher When running MPI jobs, omnitrace-causal needs to be *before* the executable which launches the MPI processes (i.e.
-l, --launcher When running MPI jobs, rocprof-sys-causal needs to be *before* the executable which launches the MPI processes (i.e.
before `mpirun`, `srun`, etc.). Pass the name of the target executable (or a regex for matching to the name of the
target) for causal profiling, e.g., `omnitrace-causal -l foo -- mpirun -n 4 foo`. This ensures that the omnitrace
target) for causal profiling, e.g., `rocprof-sys-causal -l foo -- mpirun -n 4 foo`. This ensures that the rocprof-sys
library is LD_PRELOADed on the proper target
-g, --generate-configs Generate config files instead of passing environment variables directly. If no arguments are provided, the config files
will be placed in ${PWD}/omnitrace-causal-config folder
will be placed in ${PWD}/rocprof-sys-causal-config folder
--no-defaults Do not activate default features which are recommended for causal profiling. For example: PID-tagging of output files
and timestamped subdirectories are disabled by default. Kokkos tools support is added by default
(OMNITRACE_USE_KOKKOSP=ON) because, for Kokkos applications, the Kokkos-Tools callbacks are used for progress points.
(ROCPROFSYS_USE_KOKKOSP=ON) because, for Kokkos applications, the Kokkos-Tools callbacks are used for progress points.
Activation of OpenMP tools support is similar
[CAUSAL PROFILING OPTIONS (General)]
@@ -335,20 +331,20 @@ Examples
#!/bin/bash -e
module load omnitrace
module load rocprofiler-systems
N=20
I=3
# when providing speedups to omnitrace-causal, speedup
# when providing speedups to rocprof-sys-causal, speedup
# groups are separated by a space so "0,10" results in
# one speedup group where omnitrace samples from
# one speedup group where rocprof-sys samples from
# the speedup set of {0, 10}. Passing "0 10" (without
# quotes to omnitrace-causal multiplies the
# quotes to rocprof-sys-causal multiplies the
# number of runs by 2, where the first half of the
# runs instruct omnitrace to only use 0 as the
# runs instruct rocprof-sys to only use 0 as the
# speedup and the second half of the runs instruct
# omnitrace to only use 10 as the speedup.
# rocprof-sys to only use 10 as the speedup.
SPEEDUPS="0,0,0,10,20,30,40,50,50,75,75,75,90,90,90"
# thus, -s ${SPEEDUPS} only multiplies the number
# of runs by 1 whereas -S ${SPEEDUPS_E2E} multiplies
@@ -370,14 +366,14 @@ Examples
#
# total executions: 20
#
omnitrace-causal \
rocprof-sys-causal \
-n ${N} \
-s ${SPEEDUPS} \
-m function \
-o experiments.func \
-S ".*\\.cpp" \
-- \
./causal-omni-cpu "${@}"
./causal-rocprofsys-cpu "${@}"
# 20 iterations in line mode with 1 speedup group
@@ -390,14 +386,14 @@ Examples
#
# total executions: 20
#
omnitrace-causal \
rocprof-sys-causal \
-n ${N} \
-s ${SPEEDUPS} \
-m line \
-o experiments.line \
-S "causal\\.cpp:(100|110)" \
-- \
./causal-omni-cpu "${@}"
./causal-rocprofsys-cpu "${@}"
# 3 iterations in function mode of 15 singular speedups
@@ -411,7 +407,7 @@ Examples
#
# total executions: 90
#
omnitrace-causal \
rocprof-sys-causal \
-n ${I} \
-s ${SPEEDUPS_E2E} \
-m func \
@@ -420,7 +416,7 @@ Examples
-F "cpu_slow_func" \
"cpu_fast_func" \
-- \
./causal-omni-cpu "${@}"
./causal-rocprofsys-cpu "${@}"
# 3 iterations in line mode of 15 singular speedups
# in end-to-end mode with 2 different source scopes
@@ -433,7 +429,7 @@ Examples
#
# total executions: 90
#
omnitrace-causal \
rocprof-sys-causal \
-n ${I} \
-s ${SPEEDUPS_E2E} \
-m line \
@@ -442,7 +438,7 @@ Examples
-S "causal\\.cpp:100" \
"causal\\.cpp:110" \
-- \
./causal-omni-cpu "${@}"
./causal-rocprofsys-cpu "${@}"
export OMP_NUM_THREADS=8
@@ -468,7 +464,7 @@ Examples
# existing causal/experiments.func.(coz|json)
# file due to "--reset" argument
#
omnitrace-causal \
rocprof-sys-causal \
--reset \
-n ${N} \
-s ${SPEEDUPS} \
@@ -477,7 +473,7 @@ Examples
-S "lulesh.*" \
-FE "^(Kokkos::|std::enable_if)" \
-- \
./lulesh-omni -i 50 -s 200 -r 20 -b 5 -c 5 -p
./lulesh-rocprofsys -i 50 -s 200 -r 20 -b 5 -c 5 -p
# 5 iterations in line mode of 1 speedup
@@ -498,7 +494,7 @@ Examples
# existing causal/experiments.line.(coz|json)
# file due to "--reset" argument
#
omnitrace-causal \
rocprof-sys-causal \
--reset \
-n ${N} \
-s ${SPEEDUPS} \
@@ -507,7 +503,7 @@ Examples
-S "lulesh.*" \
-FE "^(exec_range|execute);construct_shared_allocation;\\._omp_fn\\." \
-- \
./lulesh-omni -i 50 -s 200 -r 20 -b 5 -c 5 -p
./lulesh-rocprofsys -i 50 -s 200 -r 20 -b 5 -c 5 -p
# 5 iterations in line mode of 1 speedup
@@ -528,7 +524,7 @@ Examples
# existing causal/experiments.line.(coz|json)
# file due to "--reset" argument
#
omnitrace-causal \
rocprof-sys-causal \
--reset \
-n ${N} \
-s ${SPEEDUPS} \
@@ -539,30 +535,30 @@ Examples
"CalcVolumeForceForElems" \
-S "lulesh\\.cc" \
-- \
./lulesh-omni -i 50 -s 200 -r 20 -b 5 -c 5 -p
./lulesh-rocprofsys -i 50 -s 200 -r 20 -b 5 -c 5 -p
Using omnitrace-causal with other launchers like mpirun
Using rocprof-sys-causal with other launchers like mpirun
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The ``omnitrace-causal`` executable is intended to assist with application replay
The ``rocprof-sys-causal`` executable is intended to assist with application replay
and is designed to always be at the start of the command line as the primary process.
``omnitrace-causal`` typically adds a ``LD_PRELOAD`` of the Omnitrace libraries
``rocprof-sys-causal`` typically adds a ``LD_PRELOAD`` of the ROCm Systems Profiler libraries
into the environment before launching the command to inject the functionality
required to start the causal profiling tooling. However, this is problematic
when the target application for causal profiling uses a launcher, in which case
it is listed as an argument rather than as the main application. For example,
``foo`` is the target application for profiling, but the command to run it is
``mpirun -n 2 foo``. Running the command ``omnitrace-causal -- mpirun -n 2 foo``
applies the causal profiling to ``mpirun`` instead of ``foo``.
required to start the causal profiling tooling. However, this is problematic
when the target application for causal profiling uses a launcher, in which case
it is listed as an argument rather than as the main application. For example,
``foo`` is the target application for profiling, but the command to run it is
``mpirun -n 2 foo``. Running the command ``rocprof-sys-causal -- mpirun -n 2 foo``
applies the causal profiling to ``mpirun`` instead of ``foo``.
``omnitrace-causal`` remedies this by providing a command-line option ``-l` / `--launcher``
to indicate the target application is using a launcher script/executable. The
``rocprof-sys-causal`` remedies this by providing a command-line option ``-l` / `--launcher``
to indicate the target application is using a launcher script/executable. The
argument to the command-line option is the name of, or regular expression for, the target application
on the command line. When ``--launcher`` is used, ``omnitrace-causal`` generates
on the command line. When ``--launcher`` is used, ``rocprof-sys-causal`` generates
all the replay configurations and runs them but delays adding the ``LD_PRELOAD``. Instead it
inserts a call to itself into the command line right before the target
inserts a call to itself into the command line right before the target
application. This recursive call inherits the configuration from
the parent ``omnitrace-causal`` executable, inserts an ``LD_PRELOAD`` into the environment,
the parent ``rocprof-sys-causal`` executable, inserts an ``LD_PRELOAD`` into the environment,
and calls ``execv`` to replace itself with the new process launched by the target
application.
@@ -570,32 +566,32 @@ In other words, the following command:
.. code-block:: shell
omnitrace-causal -l foo -n 3 -- mpirun -n 2 foo`
rocprof-sys-causal -l foo -n 3 -- mpirun -n 2 foo`
Effectively results in:
.. code-block:: shell
mpirun -n 2 omnitrace-causal -- foo
mpirun -n 2 omnitrace-causal -- foo
mpirun -n 2 omnitrace-causal -- foo
mpirun -n 2 rocprof-sys-causal -- foo
mpirun -n 2 rocprof-sys-causal -- foo
mpirun -n 2 rocprof-sys-causal -- foo
Visualizing the causal output
-------------------------------------------------------------------------
Omnitrace generates ``causal/experiments.json`` and ``causal/experiments.coz`` in
``${OMNITRACE_OUTPUT_PATH}/${OMNITRACE_OUTPUT_PREFIX}``. Visit
ROCm Systems Profiler generates ``causal/experiments.json`` and ``causal/experiments.coz`` in
``${ROCPROFSYS_OUTPUT_PATH}/${ROCPROFSYS_OUTPUT_PREFIX}``. Visit
`plasma-umass.org/coz <https://plasma-umass.org/coz/>`_ to open the ``*.coz`` file.
Omnitrace versus Coz
ROCm Systems Profiler versus Coz
=======================================
This comparison is intended for readers who are familiar with the
This comparison is intended for readers who are familiar with the
`Coz profiler <https://github.com/plasma-umass/coz>`_.
Omnitrace provides several additional features and utilities for causal profiling:
ROCm Systems Profiler provides several additional features and utilities for causal profiling:
.. csv-table::
:header: "Feature", "Coz", "Omnitrace", "Notes"
.. csv-table::
:header: "Feature", "Coz", "ROCm Systems Profiler", "Notes"
:widths: 20, 60, 60, 30
"Debug info", "requires debug info in DWARF v3 format (``-gdwarf-3``)", "optional, supports any DWARF format version", "See Note #1 below"
@@ -608,23 +604,23 @@ Omnitrace provides several additional features and utilities for causal profilin
.. note::
#. Omnitrace supports a "function" mode which does not require debug info.
#. Omnitrace supports selecting an entire range of instruction pointers for a function instead
#. ROCm Systems Profiler supports a "function" mode which does not require debug info.
#. ROCm Systems Profiler supports selecting an entire range of instruction pointers for a function instead
of an instruction pointer for one line. In large code bases, "function" mode
can resolve in fewer iterations. After a target function is identified, you can
can resolve in fewer iterations. After a target function is identified, you can
switch to line mode and limit the function scope to the target function.
#. Omnitrace supports randomly sampling from subsets, e.g. { 0, 0, 5, 10 }
#. ROCm Systems Profiler supports randomly sampling from subsets, e.g. { 0, 0, 5, 10 }
where 0% is randomly selected 50% of time and 5% and 10% are randomly selected 25% of the time.
#. Omnitrace and COZ have the same definition for binary scope, which is the binaries
#. ROCm Systems Profiler and COZ have the same definition for binary scope, which is the binaries
loaded at runtime (the executable and linked libraries).
#. Omnitrace "source scope" supports both ``<file>`` and ``<file>:<line>`` formats
#. ROCm Systems Profiler "source scope" supports both ``<file>`` and ``<file>:<line>`` formats
in contrast to the COZ "source scope" which requires ``<file>:<line>`` format.
#. Omnitrace supports a "function" scope which narrows the function and lines
#. ROCm Systems Profiler supports a "function" scope which narrows the function and lines
which are eligible for causal experiments to those within the matching functions.
#. Omnitrace supports a second filter on scopes for removing binary/source/function
#. ROCm Systems Profiler supports a second filter on scopes for removing binary/source/function
caught by an inclusive match. For example ``BINARY_SCOPE=.*`` and ``BINARY_EXCLUDE=libmpi.*``
initially includes all binaries but exclude regex removes MPI libraries.
#. In Omnitrace, the Linux Perf backend is preferred over use libunwind. However,
#. In ROCm Systems Profiler, the Linux Perf backend is preferred over use libunwind. However,
Linux Perf usage can be restricted for security reasons.
Omnitrace falls back to using a second POSIX timer and libunwind if
ROCm Systems Profiler falls back to using a second POSIX timer and libunwind if
Linux Perf is not available.
projects/rocprofiler-systems/docs/how-to/profiling-python-scripts.rst
+53 -53
View File
@@ -1,80 +1,80 @@
.. meta::
:description: Omnitrace documentation and reference
:keywords: Omnitrace, ROCm, profiler, tracking, visualization, tool, Instinct, accelerator, AMD
:description: ROCm Systems Profiler Python profiling documentation and reference
:keywords: rocprof-sys, rocprofiler-systems, Omnitrace, ROCm, Python, profiling Python, profiler, tracking, visualization, tool, Instinct, accelerator, AMD
****************************************************
Profiling Python scripts
****************************************************
`Omnitrace <https://github.com/ROCm/omnitrace>`_ supports profiling Python code at the
`ROCm Systems Profiler <https://github.com/ROCm/rocprofiler-systems>`_ supports profiling Python code at the
source level and the script level.
Python support is enabled via the ``OMNITRACE_USE_PYTHON`` and the
``OMNITRACE_PYTHON_VERSIONS="<MAJOR>.<MINOR>`` CMake options.
Alternatively, to build multiple Python versions, use
``OMNITRACE_PYTHON_VERSIONS="<MAJOR>.<MINOR>;[<MAJOR>.<MINOR>]"``,
and ``OMNITRACE_PYTHON_ROOT_DIRS="/path/to/version;[/path/to/version]"`` instead of ``OMNITRACE_PYTHON_VERSION``.
When building multiple Python versions, the length of the ``OMNITRACE_PYTHON_VERSIONS``
and ``OMNITRACE_PYTHON_ROOT_DIRS`` lists must
Python support is enabled via the ``ROCPROFSYS_USE_PYTHON`` and the
``ROCPROFSYS_PYTHON_VERSIONS="<MAJOR>.<MINOR>`` CMake options.
Alternatively, to build multiple Python versions, use
``ROCPROFSYS_PYTHON_VERSIONS="<MAJOR>.<MINOR>;[<MAJOR>.<MINOR>]"``,
and ``ROCPROFSYS_PYTHON_ROOT_DIRS="/path/to/version;[/path/to/version]"`` instead of ``ROCPROFSYS_PYTHON_VERSION``.
When building multiple Python versions, the length of the ``ROCPROFSYS_PYTHON_VERSIONS``
and ``ROCPROFSYS_PYTHON_ROOT_DIRS`` lists must
be the same size.
.. note::
When using Omnitrace with Python programs, the Python interpreter major and minor version (e.g. 3.7)
When using ROCm Systems Profiler with Python programs, the Python interpreter major and minor version (e.g. 3.7)
must match the interpreter major and minor version
used when compiling the Python bindings. When building Omnitrace,
the shared object file ``libpyomnitrace.<IMPL>-<VERSION>-<ARCH>-<OS>-<ABI>.so`` is generated
where ``IMPL`` is the Python implementation, ``VERSION`` is the major and minor
used when compiling the Python bindings. When building ROCm Systems Profiler,
the shared object file ``libpyrocprofsys.<IMPL>-<VERSION>-<ARCH>-<OS>-<ABI>.so`` is generated
where ``IMPL`` is the Python implementation, ``VERSION`` is the major and minor
version, ``ARCH`` is the architecture,
``OS`` is the operating system, and ``ABI`` is the application binary interface,
for example, ``libpyomnitrace.cpython-38-x86_64-linux-gnu.so``.
``OS`` is the operating system, and ``ABI`` is the application binary interface,
for example, ``libpyrocprofsys.cpython-38-x86_64-linux-gnu.so``.
Getting Started
========================================
The Omnitrace Python package is installed in ``lib/pythonX.Y/site-packages/omnitrace``.
To ensure the Python interpreter can find the Omnitrace package,
The ROCm Systems Profiler Python package is installed in ``lib/pythonX.Y/site-packages/rocprofsys``.
To ensure the Python interpreter can find the ROCm Systems Profiler package,
add this path to the ``PYTHONPATH`` environment variable, as in the following example:
.. code-block:: shell
export PYTHONPATH=/opt/omnitrace/lib/python3.8/site-packages:${PYTHONPATH}
export PYTHONPATH=/opt/rocprofiler-systems/lib/python3.8/site-packages:${PYTHONPATH}
Both the ``share/omnitrace/setup-env.sh`` script and the module file in
``share/modulefiles/omnitrace`` automatically handle the prefixing of the ``PYTHONPATH``
Both the ``share/rocprofiler-systems/setup-env.sh`` script and the module file in
``share/modulefiles/rocprofiler-systems`` automatically handle the prefixing of the ``PYTHONPATH``
environment variable.
Running Omnitrace on a Python script
Running ROCm Systems Profiler on a Python script
========================================
Omnitrace provides an ``omnitrace-python`` helper bash script which
ROCm Systems Profiler provides an ``rocprof-sys-python`` helper bash script which
ensures ``PYTHONPATH`` is properly set and the correct Python interpreter is used.
This means the following commands are effectively equivalent:
.. code-block:: shell
omnitrace-python --help
rocprof-sys-python --help
and
.. code-block:: shell
export PYTHONPATH=/opt/omnitrace/lib/python3.8/site-packages:${PYTHONPATH}
python3.8 -m omnitrace --help
export PYTHONPATH=/opt/rocprofiler-systems/lib/python3.8/site-packages:${PYTHONPATH}
python3.8 -m rocprofsys --help
.. note::
``omnitrace-python`` and ``python -m omnitrace`` use the same command-line syntax
as the other ``omnitrace`` executables (``omnitrace-python <OMNITRACE_ARGS> -- <SCRIPT> <SCRIPT_ARGS>``)
``rocprof-sys-python`` and ``python -m rocprofsys`` use the same command-line syntax
as the other ``rocprof-sys`` executables (``rocprof-sys-python <ROCPROFSYS_ARGS> -- <SCRIPT> <SCRIPT_ARGS>``)
and has similar options.
Command line options
-----------------------------------
Use ``omnitrace-python --help`` to view the available options:
Use ``rocprof-sys-python --help`` to view the available options:
.. code-block:: shell
usage: omnitrace [-h] [-v VERBOSITY] [-b] [-c FILE] [-s FILE] [-F [BOOL]] [--label [{args,file,line} [{args,file,line} ...]]] [-I FUNC [FUNC ...]] [-E FUNC [FUNC ...]] [-R FUNC [FUNC ...]] [-MI FILE [FILE ...]] [-ME FILE [FILE ...]] [-MR FILE [FILE ...]] [--trace-c [BOOL]]
usage: rocprof-sys [-h] [-v VERBOSITY] [-b] [-c FILE] [-s FILE] [-F [BOOL]] [--label [{args,file,line} [{args,file,line} ...]]] [-I FUNC [FUNC ...]] [-E FUNC [FUNC ...]] [-R FUNC [FUNC ...]] [-MI FILE [FILE ...]] [-ME FILE [FILE ...]] [-MR FILE [FILE ...]] [--trace-c [BOOL]]
optional arguments:
-h, --help show this help message and exit
@@ -82,7 +82,7 @@ Use ``omnitrace-python --help`` to view the available options:
Logging verbosity
-b, --builtin Put 'profile' in the builtins. Use '@profile' to decorate a single function, or 'with profile:' to profile a single section of code.
-c FILE, --config FILE
OmniTrace configuration file
ROCm Systems Profiler configuration file
-s FILE, --setup FILE
Code to execute before the code to profile
-F [BOOL], --full-filepath [BOOL]
@@ -103,19 +103,19 @@ Use ``omnitrace-python --help`` to view the available options:
Select only entries from these files
--trace-c [BOOL] Enable profiling C functions
usage: python3 -m omnitrace <OMNITRACE_ARGS> -- <SCRIPT> <SCRIPT_ARGS>
usage: python3 -m rocprofsys <ROCPROFSYS_ARGS> -- <SCRIPT> <SCRIPT_ARGS>
.. note::
The ``--trace-c`` option does not incorporate Omnitrace's dynamic instrumentation support.
The ``--trace-c`` option does not incorporate ROCm Systems Profiler's dynamic instrumentation support.
It only enables profiling the underlying C function call within the Python interpreter.
Selective instrumentation
-----------------------------------
Similar to the ``omnitrace-instrument`` executable, command-line options exist for restricting,
Similar to the ``rocprof-sys-instrument`` executable, command-line options exist for restricting,
including, and excluding certain functions and modules, for example, ``--function-exclude "^__init__$"``.
Alternatively, add the ``@profile`` decorator to the primary function of interest
Alternatively, add the ``@profile`` decorator to the primary function of interest
in your program and use the ``-b`` / ``--builtin`` command-line option to narrow the scope of the
instrumentation to this function and its children.
@@ -145,8 +145,8 @@ Consider the following Python code (``example.py``):
if __name__ == "__main__":
run(20)
Running ``omnitrace-python ./example.py`` with ``OMNITRACE_PROFILE=ON`` and
``OMNITRACE_TIMEMORY_COMPONENTS=trip_count`` produces the following:
Running ``rocprof-sys-python ./example.py`` with ``ROCPROFSYS_PROFILE=ON`` and
``ROCPROFSYS_TIMEMORY_COMPONENTS=trip_count`` produces the following:
.. code-block:: shell
@@ -187,7 +187,7 @@ If the ``inefficient`` function is decorated with ``@profile`` as follows:
def inefficient(n):
# ...
And then run using the command ``omnitrace-python -b -- ./example.py``, Omnitrace produces this output:
And then run using the command ``rocprof-sys-python -b -- ./example.py``, ROCm Systems Profiler produces this output:
.. code-block:: shell
@@ -199,37 +199,37 @@ And then run using the command ``omnitrace-python -b -- ./example.py``, Omnitrac
| |0>>> inefficient | 1 | 0 | trip_count | 1 |
|-----------------------------------------------------------|
Omnitrace Python source instrumentation
ROCm Systems Profiler Python source instrumentation
========================================
Starting with the unmodified ``example.py`` script above, import the ``omnitrace`` module:
Starting with the unmodified ``example.py`` script above, import the ``rocprofsys`` module:
.. code-block:: python
import sys
import omnitrace # import omnitrace
import rocprofsys # import rocprofsys
def fib(n):
# ... etc. ...
Next, add ``@omnitrace.profile()`` to the ``run`` function:
Next, add ``@rocprofsys.profile()`` to the ``run`` function:
.. code-block:: python
@omnitrace.profile()
@rocprofsys.profile()
def run(n):
# ...
Alternatively, use ``omnitrace.profile()`` as a context-manager around ``run(20)``:
Alternatively, use ``rocprofsys.profile()`` as a context-manager around ``run(20)``:
.. code-block:: python
if __name__ == "__main__":
with omnitrace.profile():
with rocprofsys.profile():
run(20)
The results for both of the source-level instrumentation modes are identical to the
original ``omnitrace-python ./example.py`` results:
The results for both of the source-level instrumentation modes are identical to the
original ``rocprofsys-python ./example.py`` results:
.. code-block:: shell
@@ -264,14 +264,14 @@ original ``omnitrace-python ./example.py`` results:
.. note::
When ``omnitrace-python`` is used without built-ins, the profiling results can be cluttered by the
When ``rocprof-sys-python`` is used without built-ins, the profiling results can be cluttered by the
numerous functions called when more complex modules are imported, such as ``import numpy``.
Omnitrace Python source instrumentation configuration
ROCm Systems Profiler Python source instrumentation configuration
-------------------------------------------------------------
Within the Python source code, the profiler can be configured by directly
modifying the ``omnitrace.profiler.config`` data fields.
Within the Python source code, the profiler can be configured by directly
modifying the ``rocprof-sys.profiler.config`` data fields.
.. code-block:: python
@@ -295,8 +295,8 @@ modifying the ``omnitrace.profiler.config`` data fields.
if __name__ == "__main__":
from omnitrace.profiler import config
from omnitrace import profile
from rocprofsys.profiler import config
from rocprofsys import profile
config.include_args = True
config.include_filename = False
projects/rocprofiler-systems/docs/how-to/sampling-call-stack.rst
+229 -228
View File
@@ -1,77 +1,77 @@
.. meta::
:description: Omnitrace documentation and reference
:keywords: Omnitrace, ROCm, profiler, tracking, visualization, tool, Instinct, accelerator, AMD
:description: ROCm Systems Profiler call stack sampling documentation and reference
:keywords: rocprofiler-systems,rocprofsys, ROCm, profiler, sampling, call stack, tracking, visualization, tool, Instinct, accelerator, AMD
****************************************************
Sampling the call stack
****************************************************
`Omnitrace <https://github.com/ROCm/omnitrace>`_ can use call-stack sampling
on a binary instrumented with either the ``omnitrace`` executable
or the ``omnitrace-sample`` executable.
`ROCm Systems Profiler <https://github.com/ROCm/rocprofiler-systems>`_ can use call-stack sampling
on a binary instrumented with either the ``rocprof-sys`` executable
or the ``rocprof-sys-sample`` executable.
For example, all of the following commands are effectively equivalent:
* Binary rewrite with only the instrumentation necessary to start and stop sampling
.. code-block:: shell
omnitrace-instrument -M sampling -o foo.inst -- foo
omnitrace-run -- ./foo.inst
rocprof-sys-instrument -M sampling -o foo.inst -- foo
rocprof-sys-run -- ./foo.inst
* Runtime instrumentation with only the instrumentation necessary to start and stop sampling
.. code-block:: shell
omnitrace-instrument -M sampling -- foo
rocprof-sys-instrument -M sampling -- foo
* No instrumentation required
.. code-block:: shell
omnitrace-sample -- foo
rocprof-sys-sample -- foo
.. note::
Set ``OMNITRACE_USE_SAMPLING=ON`` to activate call-stack sampling when executing an instrumented binary.
Set ``ROCPROFSYS_USE_SAMPLING=ON`` to activate call-stack sampling when executing an instrumented binary.
All ``omnitrace-instrument -M sampling`` (subsequently referred to as "instrumented-sampling")
All ``rocprof-sys-instrument -M sampling`` (subsequently referred to as "instrumented-sampling")
does is wrap the ``main`` of the executable with initialization
before ``main`` starts and finalization after ``main`` ends.
This can be accomplished without instrumentation through a ``LD_PRELOAD``
This can be accomplished without instrumentation through a ``LD_PRELOAD``
of a library containing a dynamic symbol wrapper around ``__libc_start_main``.
The use of ``omnitrace-sample`` is **recommended** over
``omnitrace-instrument -M sampling`` when binary instrumentation
The use of ``rocprof-sys-sample`` is **recommended** over
``rocprof-sys-instrument -M sampling`` when binary instrumentation
is not necessary. This is for a number of reasons:
* ``omnitrace-sample`` provides command-line options for controlling the Omnitrace feature set instead of
* ``rocprof-sys-sample`` provides command-line options for controlling the ROCm Systems Profiler feature set instead of
requiring configuration files or environment variables
* Despite the fact that instrumented-sampling only requires inserting snippets
* Despite the fact that instrumented-sampling only requires inserting snippets
around one function (``main``), Dyninst
does not have a feature for specifying that parsing and processing all the
does not have a feature for specifying that parsing and processing all the
other symbols in the binary is unnecessary.
In the best-case scenario when the target binary is relatively small,
In the best-case scenario when the target binary is relatively small,
instrumented-sampling has a slightly slower launch time,
but in the worst case scenarios it requires a significant amount of time and memory to launch.
* ``omnitrace-sample`` is fully compatible with MPI. For example,
the command ``mpirun -n 2 omnitrace-sample -- foo`` is valid,
whereas ``mpirun -n 2 omnitrace-instrument -M sampling -- foo``
* ``rocprof-sys-sample`` is fully compatible with MPI. For example,
the command ``mpirun -n 2 rocprof-sys-sample -- foo`` is valid,
whereas ``mpirun -n 2 rocprof-sys-instrument -M sampling -- foo``
is incompatible with some MPI distributions (particularly OpenMPI). This is because
MPI prohibits forking within an MPI rank.
* When MPI and binary instrumentation are both involved, two steps are required:
performing a binary rewrite of the executable and then using the instrumented executable
in lieu of the original executable. ``omnitrace-sample`` is therefore much easier to use with MPI.
performing a binary rewrite of the executable and then using the instrumented executable
in lieu of the original executable. ``rocprof-sys-sample`` is therefore much easier to use with MPI.
The omnitrace-sample executable
The rocprof-sys-sample executable
========================================
View the help menu of ``omnitrace-sample`` with the ``-h`` / ``--help`` option:
View the help menu of ``rocprof-sys-sample`` with the ``-h`` / ``--help`` option:
.. code-block:: shell
$ omnitrace-sample --help
[omnitrace-sample] Usage: omnitrace-sample [ --help (count: 0, dtype: bool)
$ rocprof-sys-sample --help
[rocprof-sys-sample] Usage: rocprof-sys-sample [ --help (count: 0, dtype: bool)
--version (count: 0, dtype: bool)
--monochrome (max: 1, dtype: bool)
--debug (max: 1, dtype: bool)
@@ -111,47 +111,47 @@ View the help menu of ``omnitrace-sample`` with the ``-h`` / ``--help`` option:
--gpu-events (count: unlimited)
--inlines (max: 1, dtype: bool)
--hsa-interrupt (count: 1, dtype: int)
]
]
Options:
-h, -?, --help Shows this page (count: 0, dtype: bool)
--version Prints the version and exit (count: 0, dtype: bool)
[DEBUG OPTIONS]
--monochrome Disable colorized output (max: 1, dtype: bool)
--debug Debug output (max: 1, dtype: bool)
-v, --verbose Verbose output (count: 1)
[GENERAL OPTIONS] These are options which are ubiquitously applied
-c, --config Configuration file (min: 0, dtype: filepath)
-o, --output Output path. Accepts 1-2 parameters corresponding to the output path and the output prefix (min: 1)
-T, --trace Generate a detailed trace (perfetto output) (max: 1, dtype: bool)
-P, --profile Generate a call-stack-based profile (conflicts with --flat-profile) (max: 1, dtype: bool)
-F, --flat-profile Generate a flat profile (conflicts with --profile) (max: 1, dtype: bool)
-H, --host Enable sampling host-based metrics for the process. E.g. CPU frequency, memory usage, etc. (max: 1, dtype: bool)
-D, --device Enable sampling device-based metrics for the process. E.g. GPU temperature, memory usage, etc. (max: 1, dtype: bool)
-w, --wait This option is a combination of '--trace-wait' and '--sampling-wait'. See the descriptions for those two options.
(count: 1)
-d, --duration This option is a combination of '--trace-duration' and '--sampling-duration'. See the descriptions for those two
options. (count: 1)
[TRACING OPTIONS] Specific options controlling tracing (i.e. deterministic measurements of every event)
--trace-file Specify the trace output filename. Relative filepath will be with respect to output path and output prefix. (count: 1,
dtype: filepath)
--trace-buffer-size Size limit for the trace output (in KB) (count: 1, dtype: KB)
-h, -?, --help Shows this page (count: 0, dtype: bool)
--version Prints the version and exit (count: 0, dtype: bool)
[DEBUG OPTIONS]
--monochrome Disable colorized output (max: 1, dtype: bool)
--debug Debug output (max: 1, dtype: bool)
-v, --verbose Verbose output (count: 1)
[GENERAL OPTIONS] These are options which are ubiquitously applied
-c, --config Configuration file (min: 0, dtype: filepath)
-o, --output Output path. Accepts 1-2 parameters corresponding to the output path and the output prefix (min: 1)
-T, --trace Generate a detailed trace (perfetto output) (max: 1, dtype: bool)
-P, --profile Generate a call-stack-based profile (conflicts with --flat-profile) (max: 1, dtype: bool)
-F, --flat-profile Generate a flat profile (conflicts with --profile) (max: 1, dtype: bool)
-H, --host Enable sampling host-based metrics for the process. E.g. CPU frequency, memory usage, etc. (max: 1, dtype: bool)
-D, --device Enable sampling device-based metrics for the process. E.g. GPU temperature, memory usage, etc. (max: 1, dtype: bool)
-w, --wait This option is a combination of '--trace-wait' and '--sampling-wait'. See the descriptions for those two options.
(count: 1)
-d, --duration This option is a combination of '--trace-duration' and '--sampling-duration'. See the descriptions for those two
options. (count: 1)
[TRACING OPTIONS] Specific options controlling tracing (i.e. deterministic measurements of every event)
--trace-file Specify the trace output filename. Relative filepath will be with respect to output path and output prefix. (count: 1,
dtype: filepath)
--trace-buffer-size Size limit for the trace output (in KB) (count: 1, dtype: KB)
--trace-fill-policy [ discard | ring_buffer ]
Policy for new data when the buffer size limit is reached:
- discard : new data is ignored
- ring_buffer : new data overwrites oldest data (count: 1)
--trace-wait Set the wait time (in seconds) before collecting trace and/or profiling data(in seconds). By default, the duration is
in seconds of realtime but that can changed via --trace-clock-id. (count: 1)
--trace-duration Set the duration of the trace and/or profile data collection (in seconds). By default, the duration is in seconds of
realtime but that can changed via --trace-clock-id. (count: 1)
--trace-periods More powerful version of specifying trace delay and/or duration. Format is one or more groups of: <DELAY>:<DURATION>,
<DELAY>:<DURATION>:<REPEAT>, and/or <DELAY>:<DURATION>:<REPEAT>:<CLOCK_ID>. (min: 1)
--trace-wait Set the wait time (in seconds) before collecting trace and/or profiling data(in seconds). By default, the duration is
in seconds of realtime but that can changed via --trace-clock-id. (count: 1)
--trace-duration Set the duration of the trace and/or profile data collection (in seconds). By default, the duration is in seconds of
realtime but that can changed via --trace-clock-id. (count: 1)
--trace-periods More powerful version of specifying trace delay and/or duration. Format is one or more groups of: <DELAY>:<DURATION>,
<DELAY>:<DURATION>:<REPEAT>, and/or <DELAY>:<DURATION>:<REPEAT>:<CLOCK_ID>. (min: 1)
--trace-clock-id [ 0 (realtime|CLOCK_REALTIME)
1 (monotonic|CLOCK_MONOTONIC)
2 (cputime|CLOCK_PROCESS_CPUTIME_ID)
@@ -159,40 +159,40 @@ View the help menu of ``omnitrace-sample`` with the ``-h`` / ``--help`` option:
5 (realtime_coarse|CLOCK_REALTIME_COARSE)
6 (monotonic_coarse|CLOCK_MONOTONIC_COARSE)
7 (boottime|CLOCK_BOOTTIME) ]
Set the default clock ID for for trace delay/duration. Note: "cputime" is the *process* CPU time and might need to be
scaled based on the number of threads, i.e. 4 seconds of CPU-time for an application with 4 fully active threads would
equate to ~1 second of realtime. If this proves to be difficult to handle in practice, please file a feature request
for omnitrace to auto-scale based on the number of threads. (count: 1)
[PROFILE OPTIONS] Specific options controlling profiling (i.e. deterministic measurements which are aggregated into a summary)
Set the default clock ID for for trace delay/duration. Note: "cputime" is the *process* CPU time and might need to be
scaled based on the number of threads, i.e. 4 seconds of CPU-time for an application with 4 fully active threads would
equate to ~1 second of realtime. If this proves to be difficult to handle in practice, please file a feature request
for rocprof-sys to auto-scale based on the number of threads. (count: 1)
[PROFILE OPTIONS] Specific options controlling profiling (i.e. deterministic measurements which are aggregated into a summary)
--profile-format [ console | json | text ]
Data formats for profiling results (min: 1)
--profile-diff Generate a diff output b/t the profile collected and an existing profile from another run Accepts 1-2 parameters
corresponding to the input path and the input prefix (min: 1)
Data formats for profiling results (min: 1)
--profile-diff Generate a diff output b/t the profile collected and an existing profile from another run Accepts 1-2 parameters
corresponding to the input path and the input prefix (min: 1)
[HOST/DEVICE (PROCESS SAMPLING) OPTIONS]
Process sampling is background measurements for resources available to the entire process. These samples are not tied
to specific lines/regions of code
--process-freq Set the default host/device sampling frequency (number of interrupts per second) (count: 1)
--process-wait Set the default wait time (i.e. delay) before taking first host/device sample (in seconds of realtime) (count: 1)
--process-duration Set the duration of the host/device sampling (in seconds of realtime) (count: 1)
--cpus CPU IDs for frequency sampling. Supports integers and/or ranges (count: unlimited, dtype: int or range)
--gpus GPU IDs for SMI queries. Supports integers and/or ranges (count: unlimited, dtype: int or range)
[GENERAL SAMPLING OPTIONS] General options for timer-based sampling per-thread
-f, --freq Set the default sampling frequency (number of interrupts per second) (count: 1)
--sampling-wait Set the default wait time (i.e. delay) before taking first sample (in seconds). This delay time is based on the clock
of the sampler, i.e., a delay of 1 second for CPU-clock sampler may not equal 1 second of realtime (count: 1)
--sampling-duration Set the duration of the sampling (in seconds of realtime). I.e., it is possible (currently) to set a CPU-clock time
delay that exceeds the real-time duration... resulting in zero samples being taken (count: 1)
-t, --tids Specify the default thread IDs for sampling, where 0 (zero) is the main thread and each thread created by the target
application is assigned an atomically incrementing value. (min: 1)
[SAMPLING TIMER OPTIONS] These options determine the heuristic for deciding when to take a sample
Process sampling is background measurements for resources available to the entire process. These samples are not tied
to specific lines/regions of code
--process-freq Set the default host/device sampling frequency (number of interrupts per second) (count: 1)
--process-wait Set the default wait time (i.e. delay) before taking first host/device sample (in seconds of realtime) (count: 1)
--process-duration Set the duration of the host/device sampling (in seconds of realtime) (count: 1)
--cpus CPU IDs for frequency sampling. Supports integers and/or ranges (count: unlimited, dtype: int or range)
--gpus GPU IDs for SMI queries. Supports integers and/or ranges (count: unlimited, dtype: int or range)
[GENERAL SAMPLING OPTIONS] General options for timer-based sampling per-thread
-f, --freq Set the default sampling frequency (number of interrupts per second) (count: 1)
--sampling-wait Set the default wait time (i.e. delay) before taking first sample (in seconds). This delay time is based on the clock
of the sampler, i.e., a delay of 1 second for CPU-clock sampler may not equal 1 second of realtime (count: 1)
--sampling-duration Set the duration of the sampling (in seconds of realtime). I.e., it is possible (currently) to set a CPU-clock time
delay that exceeds the real-time duration... resulting in zero samples being taken (count: 1)
-t, --tids Specify the default thread IDs for sampling, where 0 (zero) is the main thread and each thread created by the target
application is assigned an atomically incrementing value. (min: 1)
[SAMPLING TIMER OPTIONS] These options determine the heuristic for deciding when to take a sample
--cputime Sample based on a CPU-clock timer (default). Accepts zero or more arguments:
0. Enables sampling based on CPU-clock timer.
1. Interrupts per second. E.g., 100 == sample every 10 milliseconds of CPU-time.
@@ -210,22 +210,22 @@ View the help menu of ``omnitrace-sample`` with the ``-h`` / ``--help`` option:
When sampling with a real-clock timer, please note that enabling this will cause threads which are typically "idle"
to consume more resources since, while idle, the real-clock time increases (and therefore triggers taking samples)
whereas the CPU-clock time does not. (min: 0)
[BACKEND OPTIONS] These options control region information captured w/o sampling or instrumentation
[BACKEND OPTIONS] These options control region information captured w/o sampling or instrumentation
-I, --include [ all | kokkosp | mpip | mutex-locks | ompt | rcclp | rocm-smi | rocprofiler | roctracer | roctx | rw-locks | spin-locks ]
Include data from these backends (count: unlimited)
Include data from these backends (count: unlimited)
-E, --exclude [ all | kokkosp | mpip | mutex-locks | ompt | rcclp | rocm-smi | rocprofiler | roctracer | roctx | rw-locks | spin-locks ]
Exclude data from these backends (count: unlimited)
[HARDWARE COUNTER OPTIONS] See also: omnitrace-avail -H
-C, --cpu-events Set the CPU hardware counter events to record (ref: `omnitrace-avail -H -c CPU`) (count: unlimited)
-G, --gpu-events Set the GPU hardware counter events to record (ref: `omnitrace-avail -H -c GPU`) (count: unlimited)
[MISCELLANEOUS OPTIONS]
-i, --inlines Include inline info in output when available (max: 1, dtype: bool)
Exclude data from these backends (count: unlimited)
[HARDWARE COUNTER OPTIONS] See also: rocprof-sys-avail -H
-C, --cpu-events Set the CPU hardware counter events to record (ref: `rocprof-sys-avail -H -c CPU`) (count: unlimited)
-G, --gpu-events Set the GPU hardware counter events to record (ref: `rocprof-sys-avail -H -c GPU`) (count: unlimited)
[MISCELLANEOUS OPTIONS]
-i, --inlines Include inline info in output when available (max: 1, dtype: bool)
--hsa-interrupt [ 0 | 1 ] Set the value of the HSA_ENABLE_INTERRUPT environment variable.
ROCm version 5.2 and older have a bug which will cause a deadlock if a sample is taken while waiting for the signal
that a kernel completed -- which happens when sampling with a real-clock timer. We require this option to be set to
@@ -235,147 +235,148 @@ View the help menu of ``omnitrace-sample`` with the ``-h`` / ``--help`` option:
0 avoid triggering the bug, potentially at the cost of reduced performance
1 do not modify how ROCm is notified about kernel completion (count: 1, dtype: int)
The general syntax for separating Omnitrace command-line arguments from the
following application arguments
is consistent with the LLVM style of using a stand-alone double hyphen (``--``).
The general syntax for separating ROCm Systems Profiler command-line arguments from the
following application arguments
is consistent with the LLVM style of using a stand-alone double hyphen (``--``).
All arguments preceding the double hyphen
are interpreted as belonging to Omnitrace and all arguments following it
are interpreted as belonging to ROCm Systems Profiler and all arguments following it
are interpreted as the
application and its arguments. The double hyphen is only necessary when passing
application and its arguments. The double hyphen is only necessary when passing
command-line arguments to a target
which also uses hyphens. For example, you can run ``omnitrace-sample ls``, but
to run ``ls -la``, use ``omnitrace-sample -- ls -la``.
which also uses hyphens. For example, you can run ``rocprof-sys-sample ls``, but
to run ``ls -la``, use ``rocprof-sys-sample -- ls -la``.
:doc:`Configuring the Omnitrace runtime options <./configuring-runtime-options>`
establishes the precedence of environment variable values over values specified
:doc:`Configuring the ROCm Systems Profiler runtime options <./configuring-runtime-options>`
establishes the precedence of environment variable values over values specified
in the configuration files. This enables
you to configure the Omnitrace runtime to your preferred default behavior
in a file such as ``~/.omnitrace.cfg`` and then easily override
those settings in the command line, for example, ``OMNITRACE_ENABLED=OFF omnitrace-sample -- foo``.
Similarly, the command-line arguments passed to ``omnitrace-sample`` take precedence
you to configure the ROCm Systems Profiler runtime to your preferred default behavior
in a file such as ``~/.rocprof-sys.cfg`` and then easily override
those settings in the command line, for example, ``ROCPROFSYS_ENABLED=OFF rocprof-sys-sample -- foo``.
Similarly, the command-line arguments passed to ``rocprof-sys-sample`` take precedence
over environment variables.
All of the command-line options above correlate to one or more configuration
settings, for example, ``--cpu-events`` correlates to the ``OMNITRACE_PAPI_EVENTS`` configuration variable.
``omnitrace-sample`` processes the arguments and outputs a summary of its configuration
before running the target application.
All of the command-line options above correlate to one or more configuration
settings, for example, ``--cpu-events`` correlates to the ``ROCPROFSYS_PAPI_EVENTS`` configuration variable.
``rocprof-sys-sample`` processes the arguments and outputs a summary of its configuration
before running the target application.
The following snippets show how ``omnitrace-sample`` runs with various environment updates.
The following snippets show how ``rocprof-sys-sample`` runs with various environment updates.
* This snippet shows the environment updates when ``omnitrace-sample`` is invoked with no arguments:
* This snippet shows the environment updates when ``rocprof-sys-sample`` is invoked with no arguments:
.. code-block:: shell
$ omnitrace-sample -- ./parallel-overhead-locks 30 4 100
$ rocprof-sys-sample -- ./parallel-overhead-locks 30 4 100
HSA_TOOLS_LIB=/opt/omnitrace/lib/libomnitrace-dl.so.1.7.1
HSA_TOOLS_LIB=/opt/rocprofiler-systems/lib/librocprof-sys-dl.so.1.7.1
HSA_TOOLS_REPORT_LOAD_FAILURE=1
LD_PRELOAD=/opt/omnitrace/lib/libomnitrace-dl.so.1.7.1
OMNITRACE_USE_PROCESS_SAMPLING=false
OMNITRACE_USE_SAMPLING=true
OMP_TOOL_LIBRARIES=/opt/omnitrace/lib/libomnitrace-dl.so.1.7.1
ROCP_TOOL_LIB=/opt/omnitrace/lib/libomnitrace.so.1.7.1
LD_PRELOAD=/opt/rocprofiler-systems/lib/librocprof-sys-dl.so.1.7.1
ROCPROFSYS_USE_PROCESS_SAMPLING=false
ROCPROFSYS_USE_SAMPLING=true
OMP_TOOL_LIBRARIES=/opt/rocprofiler-systems/lib/librocprof-sys-dl.so.1.7.1
ROCP_TOOL_LIB=/opt/rocprofiler-systems/lib/librocprof-sys.so.1.7.1
* The next snippet shows the environment updates when ``omnitrace-sample`` enables
* The next snippet shows the environment updates when ``rocprof-sys-sample`` enables
profiling, tracing, host process-sampling, device process-sampling, and all the available backends:
.. code-block:: shell
$ omnitrace-sample -PTDH -I all -- ./parallel-overhead-locks 30 4 100
$ rocprof-sys-sample -PTDH -I all -- ./parallel-overhead-locks 30 4 100
HSA_TOOLS_LIB=/opt/omnitrace/lib/libomnitrace-dl.so.1.7.1
HSA_TOOLS_LIB=/opt/rocprofiler-systems/lib/librocprof-sys-dl.so.1.7.1
HSA_TOOLS_REPORT_LOAD_FAILURE=1
KOKKOS_PROFILE_LIBRARY=/opt/omnitrace/lib/libomnitrace.so.1.7.1
LD_PRELOAD=/opt/omnitrace/lib/libomnitrace-dl.so.1.7.1
OMNITRACE_CPU_FREQ_ENABLED=true
OMNITRACE_TRACE_THREAD_LOCKS=true
OMNITRACE_TRACE_THREAD_RW_LOCKS=true
OMNITRACE_TRACE_THREAD_SPIN_LOCKS=true
OMNITRACE_USE_KOKKOSP=true
OMNITRACE_USE_MPIP=true
OMNITRACE_USE_OMPT=true
OMNITRACE_TRACE=true
OMNITRACE_USE_PROCESS_SAMPLING=true
OMNITRACE_USE_RCCLP=true
OMNITRACE_USE_ROCM_SMI=true
OMNITRACE_USE_ROCPROFILER=true
OMNITRACE_USE_ROCTRACER=true
OMNITRACE_USE_ROCTX=true
OMNITRACE_USE_SAMPLING=true
OMNITRACE_PROFILE=true
OMP_TOOL_LIBRARIES=/opt/omnitrace/lib/libomnitrace-dl.so.1.7.1
ROCP_TOOL_LIB=/opt/omnitrace/lib/libomnitrace.so.1.7.1
KOKKOS_PROFILE_LIBRARY=/opt/rocprofiler-systems/lib/librocprof-sys.so.1.7.1
LD_PRELOAD=/opt/rocprofiler-systems/lib/librocprof-sys-dl.so.1.7.1
ROCPROFSYS_CPU_FREQ_ENABLED=true
ROCPROFSYS_TRACE_THREAD_LOCKS=true
ROCPROFSYS_TRACE_THREAD_RW_LOCKS=true
ROCPROFSYS_TRACE_THREAD_SPIN_LOCKS=true
ROCPROFSYS_USE_KOKKOSP=true
ROCPROFSYS_USE_MPIP=true
ROCPROFSYS_USE_OMPT=true
ROCPROFSYS_TRACE=true
ROCPROFSYS_USE_PROCESS_SAMPLING=true
ROCPROFSYS_USE_RCCLP=true
ROCPROFSYS_USE_ROCM_SMI=true
ROCPROFSYS_USE_ROCPROFILER=true
ROCPROFSYS_USE_ROCTRACER=true
ROCPROFSYS_USE_ROCTX=true
ROCPROFSYS_USE_SAMPLING=true
ROCPROFSYS_PROFILE=true
OMP_TOOL_LIBRARIES=/opt/rocprofiler-systems/lib/librocprof-sys-dl.so.1.7.1
ROCP_TOOL_LIB=/opt/rocprofiler-systems/lib/librocprof-sys.so.1.7.1
...
* The final snippet shows the environment updates when ``omnitrace-sample`` enables
* The final snippet shows the environment updates when ``rocprof-sys-sample`` enables
profiling, tracing, host process-sampling, and device process-sampling,
sets the output path to ``omnitrace-output`` and the output prefix to ``%tag%``, and disables
sets the output path to ``rocprof-sys-output`` and the output prefix to ``%tag%``, and disables
all the available backends:
.. code-block:: shell
$ omnitrace-sample -PTDH -E all -o omnitrace-output %tag% -- ./parallel-overhead-locks 30 4 100
$ rocprof-sys-sample -PTDH -E all -o rocprof-sys-output %tag% -- ./parallel-overhead-locks 30 4 100
LD_PRELOAD=/opt/omnitrace/lib/libomnitrace-dl.so.1.7.1
OMNITRACE_CPU_FREQ_ENABLED=true
OMNITRACE_OUTPUT_PATH=omnitrace-output
OMNITRACE_OUTPUT_PREFIX=%tag%
OMNITRACE_TRACE_THREAD_LOCKS=false
OMNITRACE_TRACE_THREAD_RW_LOCKS=false
OMNITRACE_TRACE_THREAD_SPIN_LOCKS=false
OMNITRACE_USE_KOKKOSP=false
OMNITRACE_USE_MPIP=false
OMNITRACE_USE_OMPT=false
OMNITRACE_TRACE=true
OMNITRACE_USE_PROCESS_SAMPLING=true
OMNITRACE_USE_RCCLP=false
OMNITRACE_USE_ROCM_SMI=false
OMNITRACE_USE_ROCPROFILER=false
OMNITRACE_USE_ROCTRACER=false
OMNITRACE_USE_ROCTX=false
OMNITRACE_USE_SAMPLING=true
OMNITRACE_PROFILE=true
LD_PRELOAD=/opt/rocprofiler-systems/lib/librocprof-sys-dl.so.1.7.1
ROCPROFSYS_CPU_FREQ_ENABLED=true
ROCPROFSYS_OUTPUT_PATH=rocprof-sys-output
ROCPROFSYS_OUTPUT_PREFIX=%tag%
ROCPROFSYS_TRACE_THREAD_LOCKS=false
ROCPROFSYS_TRACE_THREAD_RW_LOCKS=false
ROCPROFSYS_TRACE_THREAD_SPIN_LOCKS=false
ROCPROFSYS_USE_KOKKOSP=false
ROCPROFSYS_USE_MPIP=false
ROCPROFSYS_USE_OMPT=false
ROCPROFSYS_TRACE=true
ROCPROFSYS_USE_PROCESS_SAMPLING=true
ROCPROFSYS_USE_RCCLP=false
ROCPROFSYS_USE_ROCM_SMI=false
ROCPROFSYS_USE_ROCPROFILER=false
ROCPROFSYS_USE_ROCTRACER=false
ROCPROFSYS_USE_ROCTX=false
ROCPROFSYS_USE_SAMPLING=true
ROCPROFSYS_PROFILE=true
...
An omnitrace-sample example
An rocprof-sys-sample example
========================================
Here is the full output from the previous
``omnitrace-sample -PTDH -E all -o omnitrace-output %tag% -- ./parallel-overhead-locks 30 4 100`` command:
Here is the full output from the previous
``rocprof-sys-sample -PTDH -E all -o rocprof-sys-output %tag% -- ./parallel-overhead-locks 30 4 100`` command:
.. code-block:: shell
$ omnitrace-sample -PTDH -E all -o omnitrace-output %tag% -c -- ./parallel-overhead-locks 30 4 100
$ rocprof-sys-sample -PTDH -E all -o rocprof-sys-output %tag% -c -- ./parallel-overhead-locks 30 4 100
LD_PRELOAD=/opt/omnitrace/lib/libomnitrace-dl.so.1.11.3
OMNITRACE_CONFIG_FILE=
OMNITRACE_CPU_FREQ_ENABLED=true
OMNITRACE_OUTPUT_PATH=omnitrace-output
OMNITRACE_OUTPUT_PREFIX=%tag%
OMNITRACE_PROFILE=true
OMNITRACE_TRACE=true
OMNITRACE_TRACE_THREAD_LOCKS=false
OMNITRACE_TRACE_THREAD_RW_LOCKS=false
OMNITRACE_TRACE_THREAD_SPIN_LOCKS=false
OMNITRACE_USE_KOKKOSP=false
OMNITRACE_USE_MPIP=false
OMNITRACE_USE_OMPT=false
OMNITRACE_USE_PROCESS_SAMPLING=true
OMNITRACE_USE_RCCLP=false
OMNITRACE_USE_ROCM_SMI=false
OMNITRACE_USE_ROCPROFILER=false
OMNITRACE_USE_ROCTRACER=false
OMNITRACE_USE_ROCTX=false
OMNITRACE_USE_SAMPLING=true
[omnitrace][dl][1785877] omnitrace_main
[omnitrace][1785877][omnitrace_init_tooling] Instrumentation mode: Sampling
______ .___ ___. .__ __. __ .___________..______ ___ ______ _______
/ __ \ | \/ | | \ | | | | | || _ \ / \ / || ____|
| | | | | \ / | | \| | | | `---| |----`| |_) | / ^ \ | ,----'| |__
| | | | | |\/| | | . ` | | | | | | / / /_\ \ | | | __|
| `--' | | | | | | |\ | | | | | | |\ \----./ _____ \ | `----.| |____
\______/ |__| |__| |__| \__| |__| |__| | _| `._____/__/ \__\ \______||_______|
omnitrace v1.11.2 (rev: 2586b74db8bf335742600010b8d9f1ce8da9cf89, compiler: GNU v11.4.1, rocm: v6.1.x)
LD_PRELOAD=/opt/rocprofiler-systems/lib/librocprof-sys-dl.so.1.11.3
ROCPROFSYS_CONFIG_FILE=
ROCPROFSYS_CPU_FREQ_ENABLED=true
ROCPROFSYS_OUTPUT_PATH=rocprof-sys-output
ROCPROFSYS_OUTPUT_PREFIX=%tag%
ROCPROFSYS_PROFILE=true
ROCPROFSYS_TRACE=true
ROCPROFSYS_TRACE_THREAD_LOCKS=false
ROCPROFSYS_TRACE_THREAD_RW_LOCKS=false
ROCPROFSYS_TRACE_THREAD_SPIN_LOCKS=false
ROCPROFSYS_USE_KOKKOSP=false
ROCPROFSYS_USE_MPIP=false
ROCPROFSYS_USE_OMPT=false
ROCPROFSYS_USE_PROCESS_SAMPLING=true
ROCPROFSYS_USE_RCCLP=false
ROCPROFSYS_USE_ROCM_SMI=false
ROCPROFSYS_USE_ROCPROFILER=false
ROCPROFSYS_USE_ROCTRACER=false
ROCPROFSYS_USE_ROCTX=false
ROCPROFSYS_USE_SAMPLING=true
[rocprof-sys][dl][1785877] rocprofsys_main
[rocprof-sys][1785877][rocprofsys_init_tooling] Instrumentation mode: Sampling
__
_ __ ___ ___ _ __ _ __ ___ / _| ___ _ _ ___
| '__| / _ \ / __| | '_ \ | '__| / _ \ | |_ _____ / __| | | | | / __|
| | | (_) | | (__ | |_) | | | | (_) | | _| |_____| \__ \ | |_| | \__ \
|_| \___/ \___| | .__/ |_| \___/ |_| |___/ \__, | |___/
|_| |___/
rocprof-sys v1.11.2 (rev: 2586b74db8bf335742600010b8d9f1ce8da9cf89, compiler: GNU v11.4.1, rocm: v6.1.x)
[988.958] perfetto.cc:58649 Configured tracing session 1, #sources:1, duration:0 ms, #buffers:1, total buffer size:1024000 KB, total sessions:1, uid:0 session name: ""
[parallel-overhead-locks] Threads: 4
[parallel-overhead-locks] Iterations: 100
@@ -386,19 +387,19 @@ Here is the full output from the previous
[4] number of iterations: 100
[parallel-overhead-locks] fibonacci(30) x 4 = 409221992
[parallel-overhead-locks] number of mutex locks = 400
[omnitrace][1785877][0][omnitrace_finalize] finalizing...
[omnitrace][1785877][0][omnitrace_finalize]
[omnitrace][1785877][0][omnitrace_finalize] omnitrace/process/1785877 : 0.294342 sec wall_clock, 4.776 MB peak_rss, 3.170 MB page_rss, 0.990000 sec cpu_clock, 336.3 % cpu_util [laps: 1]
[omnitrace][1785877][0][omnitrace_finalize] omnitrace/process/1785877/thread/0 : 0.291535 sec wall_clock, 0.002619 sec thread_cpu_clock, 0.9 % thread_cpu_util, 4.776 MB peak_rss [laps: 1]
[omnitrace][1785877][0][omnitrace_finalize] omnitrace/process/1785877/thread/1 : 0.271353 sec wall_clock, 0.222572 sec thread_cpu_clock, 82.0 % thread_cpu_util, 4.200 MB peak_rss [laps: 1]
[omnitrace][1785877][0][omnitrace_finalize] omnitrace/process/1785877/thread/2 : 0.238218 sec wall_clock, 0.206405 sec thread_cpu_clock, 86.6 % thread_cpu_util, 3.432 MB peak_rss [laps: 1]
[omnitrace][1785877][0][omnitrace_finalize] omnitrace/process/1785877/thread/3 : 0.209459 sec wall_clock, 0.193415 sec thread_cpu_clock, 92.3 % thread_cpu_util, 2.472 MB peak_rss [laps: 1]
[omnitrace][1785877][0][omnitrace_finalize] omnitrace/process/1785877/thread/4 : 0.212029 sec wall_clock, 0.211694 sec thread_cpu_clock, 99.8 % thread_cpu_util, 1.152 MB peak_rss [laps: 1]
[omnitrace][1785877][0][omnitrace_finalize]
[omnitrace][1785877][0][omnitrace_finalize] Finalizing perfetto...
[omnitrace][1785877][perfetto]> Outputting '/home/user/code/omnitrace/build-release/omnitrace-output/2024-07-15_16.21/parallel-overhead-locksperfetto-trace-1785877.proto' (39.12 KB / 0.04 MB / 0.00 GB)... Done
[omnitrace][1785877][wall_clock]> Outputting 'omnitrace-output/2024-07-15_16.21/parallel-overhead-lockswall_clock-1785877.json'
[omnitrace][1785877][wall_clock]> Outputting 'omnitrace-output/2024-07-15_16.21/parallel-overhead-lockswall_clock-1785877.txt'
[omnitrace][1785877][metadata]> Outputting 'omnitrace-output/2024-07-15_16.21/parallel-overhead-locksmetadata-1785877.json' and 'omnitrace-output/2024-07-15_16.21/parallel-overhead-locksfunctions-1785877.json'
[omnitrace][1785877][0][omnitrace_finalize] Finalized: 0.054582 sec wall_clock, 0.000 MB peak_rss, -1.798 MB page_rss, 0.040000 sec cpu_clock, 73.3 % cpu_util
[rocprof-sys][1785877][0][rocprofsys_finalize] finalizing...
[rocprof-sys][1785877][0][rocprofsys_finalize]
[rocprof-sys][1785877][0][rocprofsys_finalize] rocprof-sys/process/1785877 : 0.294342 sec wall_clock, 4.776 MB peak_rss, 3.170 MB page_rss, 0.990000 sec cpu_clock, 336.3 % cpu_util [laps: 1]
[rocprof-sys][1785877][0][rocprofsys_finalize] rocprof-sys/process/1785877/thread/0 : 0.291535 sec wall_clock, 0.002619 sec thread_cpu_clock, 0.9 % thread_cpu_util, 4.776 MB peak_rss [laps: 1]
[rocprof-sys][1785877][0][rocprofsys_finalize] rocprof-sys/process/1785877/thread/1 : 0.271353 sec wall_clock, 0.222572 sec thread_cpu_clock, 82.0 % thread_cpu_util, 4.200 MB peak_rss [laps: 1]
[rocprof-sys][1785877][0][rocprofsys_finalize] rocprof-sys/process/1785877/thread/2 : 0.238218 sec wall_clock, 0.206405 sec thread_cpu_clock, 86.6 % thread_cpu_util, 3.432 MB peak_rss [laps: 1]
[rocprof-sys][1785877][0][rocprofsys_finalize] rocprof-sys/process/1785877/thread/3 : 0.209459 sec wall_clock, 0.193415 sec thread_cpu_clock, 92.3 % thread_cpu_util, 2.472 MB peak_rss [laps: 1]
[rocprof-sys][1785877][0][rocprofsys_finalize] rocprof-sys/process/1785877/thread/4 : 0.212029 sec wall_clock, 0.211694 sec thread_cpu_clock, 99.8 % thread_cpu_util, 1.152 MB peak_rss [laps: 1]
[rocprof-sys][1785877][0][rocprofsys_finalize]
[rocprof-sys][1785877][0][rocprofsys_finalize] Finalizing perfetto...
[rocprof-sys][1785877][perfetto]> Outputting '/home/user/code/rocprofiler-systems/build-release/rocprofiler-systems-output/2024-07-15_16.21/parallel-overhead-locksperfetto-trace-1785877.proto' (39.12 KB / 0.04 MB / 0.00 GB)... Done
[rocprof-sys][1785877][wall_clock]> Outputting 'rocprof-sys-output/2024-07-15_16.21/parallel-overhead-lockswall_clock-1785877.json'
[rocprof-sys][1785877][wall_clock]> Outputting 'rocprof-sys-output/2024-07-15_16.21/parallel-overhead-lockswall_clock-1785877.txt'
[rocprof-sys][1785877][metadata]> Outputting 'rocprof-sys-output/2024-07-15_16.21/parallel-overhead-locksmetadata-1785877.json' and 'rocprof-sys-output/2024-07-15_16.21/parallel-overhead-locksfunctions-1785877.json'
[rocprof-sys][1785877][0][rocprofsys_finalize] Finalized: 0.054582 sec wall_clock, 0.000 MB peak_rss, -1.798 MB page_rss, 0.040000 sec cpu_clock, 73.3 % cpu_util
[989.312] perfetto.cc:60128 Tracing session 1 ended, total sessions:0
projects/rocprofiler-systems/docs/how-to/understanding-omnitrace-output.rst → projects/rocprofiler-systems/docs/how-to/understanding-rocprof-sys-output.rst
+68 -68
View File
@@ -1,63 +1,63 @@
.. meta::
:description: Omnitrace documentation and reference
:keywords: Omnitrace, ROCm, profiler, tracking, visualization, tool, Instinct, accelerator, AMD
:description: ROCm Systems Profiler system output documentation and reference
:keywords: rocprof-sys, rocprofiler-systems, Omnitrace, ROCm, profiler, system output, tracking, visualization, tool, Instinct, accelerator, AMD
****************************************************
Understanding the Omnitrace output
Understanding the Systems Profiler output
****************************************************
The general output form of `Omnitrace <https://github.com/ROCm/omnitrace>`_ is
The general output form of `ROCm Systems Profiler <https://github.com/ROCm/rocprofiler-systems>`_ is
``<OUTPUT_PATH>[/<TIMESTAMP>]/[<PREFIX>]<DATA_NAME>[-<OUTPUT_SUFFIX>].<EXT>``.
For example, starting with the following base configuration:
.. code-block:: shell
export OMNITRACE_OUTPUT_PATH=omnitrace-example-output
export OMNITRACE_TIME_OUTPUT=ON
export OMNITRACE_USE_PID=OFF
export OMNITRACE_PROFILE=ON
export OMNITRACE_TRACE=ON
export ROCPROFSYS_OUTPUT_PATH=rocprof-sys-example-output
export ROCPROFSYS_TIME_OUTPUT=ON
export ROCPROFSYS_USE_PID=OFF
export ROCPROFSYS_PROFILE=ON
export ROCPROFSYS_TRACE=ON
.. code-block:: shell
$ omnitrace-instrument -- ./foo
$ rocprof-sys-instrument -- ./foo
...
[omnitrace] Outputting 'omnitrace-example-output/perfetto-trace.proto'...
[rocprof-sys] Outputting 'rocprof-sys-example-output/perfetto-trace.proto'...
[omnitrace] Outputting 'omnitrace-example-output/wall-clock.txt'...
[omnitrace] Outputting 'omnitrace-example-output/wall-clock.json'...
[rocprof-sys] Outputting 'rocprof-sys-example-output/wall-clock.txt'...
[rocprof-sys] Outputting 'rocprof-sys-example-output/wall-clock.json'...
If the ``OMNITRACE_USE_PID`` option is enabled, then running a non-MPI executable
If the ``ROCPROFSYS_USE_PID`` option is enabled, then running a non-MPI executable
with a PID of ``63453`` results in the following output:
.. code-block:: shell
$ export OMNITRACE_USE_PID=ON
$ omnitrace-instrument -- ./foo
$ export ROCPROFSYS_USE_PID=ON
$ rocprof-sys-instrument -- ./foo
...
[omnitrace] Outputting 'omnitrace-example-output/perfetto-trace-63453.proto'...
[rocprof-sys] Outputting 'rocprof-sys-example-output/perfetto-trace-63453.proto'...
[omnitrace] Outputting 'omnitrace-example-output/wall-clock-63453.txt'...
[omnitrace] Outputting 'omnitrace-example-output/wall-clock-63453.json'...
[rocprof-sys] Outputting 'rocprof-sys-example-output/wall-clock-63453.txt'...
[rocprof-sys] Outputting 'rocprof-sys-example-output/wall-clock-63453.json'...
If ``OMNITRACE_TIME_OUTPUT`` is enabled, then a job that started on January 31, 2022 at 12:30 PM
If ``ROCPROFSYS_TIME_OUTPUT`` is enabled, then a job that started on January 31, 2022 at 12:30 PM
generates the following:
.. code-block:: shell
$ export OMNITRACE_TIME_OUTPUT=ON
$ omnitrace-instrument -- ./foo
$ export ROCPROFSYS_TIME_OUTPUT=ON
$ rocprof-sys-instrument -- ./foo
...
[omnitrace] Outputting 'omnitrace-example-output/2022-01-31_12.30_PM/perfetto-trace-63453.proto'...
[rocprof-sys] Outputting 'rocprof-sys-example-output/2022-01-31_12.30_PM/perfetto-trace-63453.proto'...
[omnitrace] Outputting 'omnitrace-example-output/2022-01-31_12.30_PM/wall-clock-63453.txt'...
[omnitrace] Outputting 'omnitrace-example-output/2022-01-31_12.30_PM/wall-clock-63453.json'...
[rocprof-sys] Outputting 'rocprof-sys-example-output/2022-01-31_12.30_PM/wall-clock-63453.txt'...
[rocprof-sys] Outputting 'rocprof-sys-example-output/2022-01-31_12.30_PM/wall-clock-63453.json'...
Metadata
========================================
Omnitrace outputs a ``metadata.json`` file. This metadata file contains
ROCm Systems Profiler outputs a ``metadata.json`` file. This metadata file contains
information about the settings, environment variables, output files, and info
about the system and the run, as follows:
@@ -77,7 +77,7 @@ Metadata JSON Sample
.. code-block:: json
{
"omnitrace": {
"rocprof-sys": {
"metadata": {
"info": {
"HW_L1_CACHE_SIZE": 32768,
@@ -161,13 +161,13 @@ Metadata JSON Sample
"text": [
{
"value": [
"omnitrace-tests-output/parallel-overhead-binary-rewrite/roctracer.txt"
"rocprof-sys-tests-output/parallel-overhead-binary-rewrite/roctracer.txt"
],
"key": "roctracer"
},
{
"value": [
"omnitrace-tests-output/parallel-overhead-binary-rewrite/wall_clock.txt"
"rocprof-sys-tests-output/parallel-overhead-binary-rewrite/wall_clock.txt"
],
"key": "wall_clock"
}
@@ -175,15 +175,15 @@ Metadata JSON Sample
"json": [
{
"value": [
"omnitrace-tests-output/parallel-overhead-binary-rewrite/roctracer.json",
"omnitrace-tests-output/parallel-overhead-binary-rewrite/roctracer.tree.json"
"rocprof-sys-tests-output/parallel-overhead-binary-rewrite/roctracer.json",
"rocprof-sys-tests-output/parallel-overhead-binary-rewrite/roctracer.tree.json"
],
"key": "roctracer"
},
{
"value": [
"omnitrace-tests-output/parallel-overhead-binary-rewrite/wall_clock.json",
"omnitrace-tests-output/parallel-overhead-binary-rewrite/wall_clock.tree.json"
"rocprof-sys-tests-output/parallel-overhead-binary-rewrite/wall_clock.json",
"rocprof-sys-tests-output/parallel-overhead-binary-rewrite/wall_clock.tree.json"
],
"key": "wall_clock"
}
@@ -208,7 +208,7 @@ Metadata JSON Sample
}
],
"settings": {
"OMNITRACE_JSON_OUTPUT": {
"ROCPROFSYS_JSON_OUTPUT": {
"count": -1,
"environ_updated": false,
"name": "json_output",
@@ -218,9 +218,9 @@ Metadata JSON Sample
"value": true,
"max_count": 1,
"cmdline": [
"--omnitrace-json-output"
"--rocprof-sys-json-output"
],
"environ": "OMNITRACE_JSON_OUTPUT",
"environ": "ROCPROFSYS_JSON_OUTPUT",
"config_updated": false,
"categories": [
"io",
@@ -237,10 +237,10 @@ Metadata JSON Sample
}
}
Configuring the Omnitrace output
Configuring the ROCm Systems Profiler output
========================================
Omnitrace includes a core set of options for controlling the format
ROCm Systems Profiler includes a core set of options for controlling the format
and contents of the output files. For additional information, see the guide on
:doc:`configuring runtime options <./configuring-runtime-options>`.
@@ -251,19 +251,19 @@ Core configuration settings
:header: "Setting", "Value", "Description"
:widths: 30, 30, 100
"``OMNITRACE_OUTPUT_PATH``", "Any valid path", "Path to folder where output files should be placed"
"``OMNITRACE_OUTPUT_PREFIX``", "String", "Useful for multiple runs with different arguments. See the next section on output prefix keys."
"``OMNITRACE_OUTPUT_FILE``", "Any valid filepath", "Specific location for the Perfetto output file"
"``OMNITRACE_TIME_OUTPUT``", "Boolean", "Place all output in a timestamped folder, timestamp format controlled via ``OMNITRACE_TIME_FORMAT``"
"``OMNITRACE_TIME_FORMAT``", "String", "See ``strftime`` man pages for valid identifiers"
"``OMNITRACE_USE_PID``", "Boolean", "Append either the PID or the MPI rank to all output files (before the extension)"
"``ROCPROFSYS_OUTPUT_PATH``", "Any valid path", "Path to folder where output files should be placed"
"``ROCPROFSYS_OUTPUT_PREFIX``", "String", "Useful for multiple runs with different arguments. See the next section on output prefix keys."
"``ROCPROFSYS_OUTPUT_FILE``", "Any valid filepath", "Specific location for the Perfetto output file"
"``ROCPROFSYS_TIME_OUTPUT``", "Boolean", "Place all output in a timestamped folder, timestamp format controlled via ``ROCPROFSYS_TIME_FORMAT``"
"``ROCPROFSYS_TIME_FORMAT``", "String", "See ``strftime`` man pages for valid identifiers"
"``ROCPROFSYS_USE_PID``", "Boolean", "Append either the PID or the MPI rank to all output files (before the extension)"
Output prefix keys
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Output prefix keys have many uses but are most helpful when dealing with multiple
profiling runs or large MPI jobs.
They are included in Omnitrace because they were introduced into Timemory
They are included in ROCm Systems Profiler because they were introduced into Timemory
for `compile-time-perf <https://github.com/jrmadsen/compile-time-perf>`_.
They are needed to create different output files for a generic wrapper around
compilation commands while still
@@ -271,8 +271,8 @@ overwriting the output from the last time a file was compiled.
When doing scaling studies and specifying options via the command line,
the recommended process is to
use a common ``OMNITRACE_OUTPUT_PATH``, disable ``OMNITRACE_TIME_OUTPUT``,
set ``OMNITRACE_OUTPUT_PREFIX="%argt%-"``, and let Omnitrace cleanly organize the output.
use a common ``ROCPROFSYS_OUTPUT_PATH``, disable ``ROCPROFSYS_TIME_OUTPUT``,
set ``ROCPROFSYS_OUTPUT_PREFIX="%argt%-"``, and let ROCm Systems Profiler cleanly organize the output.
.. csv-table::
:header: "String", "Encoding"
@@ -297,9 +297,9 @@ set ``OMNITRACE_OUTPUT_PREFIX="%argt%-"``, and let Omnitrace cleanly organize th
"``%rank%``", "Value of ``SLURM_PROCID`` environment variable if exists, else ``MPI_Comm_rank`` (or ``0`` non-mpi)"
"``%size%``", "``MPI_Comm_size`` or ``1`` if non-mpi"
"``%nid%``", "``%rank%`` if possible, otherwise ``%pid%``"
"``%launch_time%``", "Launch date and time (uses ``OMNITRACE_TIME_FORMAT``)"
"``%launch_time%``", "Launch date and time (uses ``ROCPROFSYS_TIME_FORMAT``)"
"``%env{NAME}%``", "Value of environment variable ``NAME`` (i.e. ``getenv(NAME)``)"
"``%cfg{NAME}%``", "Value of configuration variable ``NAME`` (e.g. ``%cfg{OMNITRACE_SAMPLING_FREQ}%`` would resolve to sampling frequency)"
"``%cfg{NAME}%``", "Value of configuration variable ``NAME`` (e.g. ``%cfg{ROCPROFSYS_SAMPLING_FREQ}%`` would resolve to sampling frequency)"
"``$env{NAME}``", "Alternative syntax to ``%env{NAME}%``"
"``$cfg{NAME}``", "Alternative syntax to ``%cfg{NAME}%``"
"``%m``", "Shorthand for ``%argt_hash%``"
@@ -318,8 +318,8 @@ set ``OMNITRACE_OUTPUT_PREFIX="%argt%-"``, and let Omnitrace cleanly organize th
Perfetto output
========================================
Use the ``OMNITRACE_OUTPUT_FILE`` to specify a specific location. If this is an
absolute path, then all ``OMNITRACE_OUTPUT_PATH`` and similar
Use the ``ROCPROFSYS_OUTPUT_FILE`` to specify a specific location. If this is an
absolute path, then all ``ROCPROFSYS_OUTPUT_PATH`` and similar
settings are ignored. Visit `ui.perfetto.dev <https://ui.perfetto.dev>`_ and open
this file.
@@ -328,26 +328,26 @@ this file.
If you are experiencing problems viewing your trace in the latest version of `Perfetto <http://ui.perfetto.dev>`_,
then try using `Perfetto UI v46.0 <https://ui.perfetto.dev/v46.0-35b3d9845/#!/>`_.
.. image:: ../data/omnitrace-perfetto.png
.. image:: ../data/rocprof-sys-perfetto.png
:alt: Visualization of a performance graph in Perfetto
.. image:: ../data/omnitrace-rocm.png
.. image:: ../data/rocprof-sys-rocm.png
:alt: Visualization of ROCm data in Perfetto
.. image:: ../data/omnitrace-rocm-flow.png
.. image:: ../data/rocprof-sys-rocm-flow.png
:alt: Visualization of ROCm flow data in Perfetto
.. image:: ../data/omnitrace-user-api.png
.. image:: ../data/rocprof-sys-user-api.png
:alt: Visualization of ROCm API calls in Perfetto
Timemory output
========================================
Use ``omnitrace-avail --components --filename`` to view the base filename for each component, as follows
Use ``rocprof-sys-avail --components --filename`` to view the base filename for each component, as follows
.. code-block:: shell
$ omnitrace-avail wall_clock -C -f
$ rocprof-sys-avail wall_clock -C -f
|---------------------------------|---------------|------------------------|
| COMPONENT | AVAILABLE | FILENAME |
|---------------------------------|---------------|------------------------|
@@ -355,16 +355,16 @@ Use ``omnitrace-avail --components --filename`` to view the base filename for ea
| sampling_wall_clock | true | sampling_wall_clock |
|---------------------------------|---------------|------------------------|
The ``OMNITRACE_COLLAPSE_THREADS`` and ``OMNITRACE_COLLAPSE_PROCESSES`` settings are
only valid when full `MPI support is enabled <../install/install.html#mpi-support-within-omnitrace>`_.
The ``ROCPROFSYS_COLLAPSE_THREADS`` and ``ROCPROFSYS_COLLAPSE_PROCESSES`` settings are
only valid when full `MPI support is enabled <../install/install.html#mpi-support-within-rocprof-sys>`_.
When they are set, Timemory combines the per-thread and per-rank data (respectively) of
identical call stacks.
The ``OMNITRACE_FLAT_PROFILE`` setting removes all call stack hierarchy.
Using ``OMNITRACE_FLAT_PROFILE=ON`` in combination
with ``OMNITRACE_COLLAPSE_THREADS=ON`` is a useful configuration for identifying
The ``ROCPROFSYS_FLAT_PROFILE`` setting removes all call stack hierarchy.
Using ``ROCPROFSYS_FLAT_PROFILE=ON`` in combination
with ``ROCPROFSYS_COLLAPSE_THREADS=ON`` is a useful configuration for identifying
min/max measurements regardless of the calling context.
The ``OMNITRACE_TIMELINE_PROFILE`` setting (with ``OMNITRACE_FLAT_PROFILE=OFF``) effectively
The ``ROCPROFSYS_TIMELINE_PROFILE`` setting (with ``ROCPROFSYS_FLAT_PROFILE=OFF``) effectively
generates similar data to that found
in Perfetto. Enabling timeline and flat profiling effectively generates
similar data to ``strace``. However, while Timemory generally
@@ -376,11 +376,11 @@ Timemory text output
Timemory text output files are meant for human consumption (while JSON formats are for analysis),
so some fields such as the ``LABEL`` might be truncated for readability.
The truncation settings be changed through the ``OMNITRACE_MAX_WIDTH`` setting.
The truncation settings be changed through the ``ROCPROFSYS_MAX_WIDTH`` setting.
.. note::
The generation of text output is configurable via ``OMNITRACE_TEXT_OUTPUT``.
The generation of text output is configurable via ``ROCPROFSYS_TEXT_OUTPUT``.
.. _text-output-example-label:
@@ -389,7 +389,7 @@ Timemory text output example
In the following example, the ``NN`` field in ``|NN>>>`` is the thread ID. If MPI support is enabled,
this becomes ``|MM|NN>>>`` where ``MM`` is the rank.
If ``OMNITRACE_COLLAPSE_THREADS=ON`` and ``OMNITRACE_COLLAPSE_PROCESSES=ON`` are configured,
If ``ROCPROFSYS_COLLAPSE_THREADS=ON`` and ``ROCPROFSYS_COLLAPSE_PROCESSES=ON`` are configured,
neither the ``MM`` nor the ``NN`` are present unless the
component explicitly sets type traits. Type traits specify that the data is only
relevant per-thread or per-process, such as the ``thread_cpu_clock`` clock component.
@@ -592,8 +592,8 @@ write a simple Python script for post-processing using this format than with the
.. note::
The generation of flat JSON output is configurable via ``OMNITRACE_JSON_OUTPUT``.
The generation of hierarchical JSON data is configurable via ``OMNITRACE_TREE_OUTPUT``
The generation of flat JSON output is configurable via ``ROCPROFSYS_JSON_OUTPUT``.
The generation of hierarchical JSON data is configurable via ``ROCPROFSYS_TREE_OUTPUT``
Timemory JSON output sample
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
projects/rocprofiler-systems/docs/how-to/using-omnitrace-api.rst → projects/rocprofiler-systems/docs/how-to/using-rocprof-sys-api.rst
+67 -66
View File
@@ -1,37 +1,38 @@
.. meta::
:description: Omnitrace documentation and reference
:keywords: Omnitrace, ROCm, profiler, tracking, visualization, tool, Instinct, accelerator, AMD
:description: ROCm Systems Profiler documentation and reference
:keywords: rocprof-sys, rocprofiler-systems, Omnitrace, ROCm, profiler, tracking, visualization, tool, Instinct, accelerator, AMD
****************************************************
Using the Omnitrace API
Using the ROCm Systems Profiler API
****************************************************
The following example shows how a program can use the Omnitrace API for run-time analysis.
The following example shows how a program can use the ROCm Systems Profiler API
for run-time analysis.
Omnitrace user API example program
ROCm Systems Profiler user API example program
========================================
You can use the Omnitrace API to define custom regions to profile and trace.
The following C++ program demonstrates this technique by calling several functions from the
Omnitrace API, such as ``omnitrace_user_push_region`` and
``omnitrace_user_stop_thread_trace``.
You can use the ROCm Systems Profiler API to define custom regions to profile and trace.
The following C++ program demonstrates this technique by calling several functions from the
ROCm Systems Profiler API, such as ``rocprofsys_user_push_region`` and
``rocprofsys_user_stop_thread_trace``.
.. note::
By default, when Omnitrace detects any ``omnitrace_user_start_*`` or
``omnitrace_user_stop_*`` function, instrumentation
is disabled at start up, which means ``omnitrace_user_stop_trace()`` is not
By default, when ROCm Systems Profiler detects any ``rocprofsys_user_start_*`` or
``rocprofsys_user_stop_*`` function, instrumentation
is disabled at start up, which means ``rocprofsys_user_stop_trace()`` is not
required at the beginning of ``main``. This behavior
can be manually controlled by using the ``OMNITRACE_INIT_ENABLED`` environment variable.
can be manually controlled by using the ``ROCPROFSYS_INIT_ENABLED`` environment variable.
User-defined regions are always
recorded, regardless of whether ``omnitrace_user_start_*`` or
``omnitrace_user_stop_*`` has been called.
recorded, regardless of whether ``rocprofsys_user_start_*`` or
``rocprofsys_user_stop_*`` has been called.
.. code-block:: shell
#include <omnitrace/categories.h>
#include <omnitrace/types.h>
#include <omnitrace/user.h>
#include <rocprofiler-systems/categories.h>
#include <rocprofiler-systems/types.h>
#include <rocprofiler-systems/user.h>
#include <atomic>
#include <cassert>
@@ -56,52 +57,52 @@ Omnitrace API, such as ``omnitrace_user_push_region`` and
namespace
{
omnitrace_user_callbacks_t custom_callbacks = OMNITRACE_USER_CALLBACKS_INIT;
omnitrace_user_callbacks_t original_callbacks = OMNITRACE_USER_CALLBACKS_INIT;
rocprofsys_user_callbacks_t custom_callbacks = ROCPROFSYS_USER_CALLBACKS_INIT;
rocprofsys_user_callbacks_t original_callbacks = ROCPROFSYS_USER_CALLBACKS_INIT;
} // namespace
int
main(int argc, char** argv)
{
custom_callbacks.push_region = &custom_push_region;
omnitrace_user_configure(OMNITRACE_USER_UNION_CONFIG, custom_callbacks,
rocprofsys_user_configure(ROCPROFSYS_USER_UNION_CONFIG, custom_callbacks,
&original_callbacks);
omnitrace_user_push_region(argv[0]);
omnitrace_user_push_region("initialization");
rocprofsys_user_push_region(argv[0]);
rocprofsys_user_push_region("initialization");
size_t nthread = std::min<size_t>(16, std::thread::hardware_concurrency());
size_t nitr = 50000;
long nfib = 10;
if(argc > 1) nfib = atol(argv[1]);
if(argc > 2) nthread = atol(argv[2]);
if(argc > 3) nitr = atol(argv[3]);
omnitrace_user_pop_region("initialization");
rocprofsys_user_pop_region("initialization");
printf("[%s] Threads: %zu\n[%s] Iterations: %zu\n[%s] fibonacci(%li)...\n", argv[0],
nthread, argv[0], nitr, argv[0], nfib);
omnitrace_user_push_region("thread_creation");
rocprofsys_user_push_region("thread_creation");
std::vector<std::thread> threads{};
threads.reserve(nthread);
// disable instrumentation for child threads
omnitrace_user_stop_thread_trace();
rocprofsys_user_stop_thread_trace();
for(size_t i = 0; i < nthread; ++i)
{
threads.emplace_back(&run, nitr, nfib);
}
// re-enable instrumentation
omnitrace_user_start_thread_trace();
omnitrace_user_pop_region("thread_creation");
rocprofsys_user_start_thread_trace();
rocprofsys_user_pop_region("thread_creation");
omnitrace_user_push_region("thread_wait");
rocprofsys_user_push_region("thread_wait");
for(auto& itr : threads)
itr.join();
omnitrace_user_pop_region("thread_wait");
rocprofsys_user_pop_region("thread_wait");
run(nitr, nfib);
printf("[%s] fibonacci(%li) x %lu = %li\n", argv[0], nfib, nthread, total.load());
omnitrace_user_pop_region(argv[0]);
rocprofsys_user_pop_region(argv[0]);
return 0;
}
@@ -120,19 +121,19 @@ Omnitrace API, such as ``omnitrace_user_push_region`` and
void
run(size_t nitr, long n)
{
omnitrace_user_push_region(RUN_LABEL);
rocprofsys_user_push_region(RUN_LABEL);
long local = 0;
for(size_t i = 0; i < nitr; ++i)
local += fib(n);
total += local;
omnitrace_user_pop_region(RUN_LABEL);
rocprofsys_user_pop_region(RUN_LABEL);
}
int
custom_push_region(const char* name)
{
if(!original_callbacks.push_region || !original_callbacks.push_annotated_region)
return OMNITRACE_USER_ERROR_NO_BINDING;
return ROCPROFSYS_USER_ERROR_NO_BINDING;
printf("Pushing custom region :: %s\n", name);
@@ -143,22 +144,22 @@ Omnitrace API, such as ``omnitrace_user_push_region`` and
char _buff[1024];
if(_err != 0) _msg = strerror_r(_err, _buff, sizeof(_buff));
omnitrace_annotation_t _annotations[] = {
{ "errno", OMNITRACE_INT32, &_err }, { "strerror", OMNITRACE_STRING, _msg }
rocprofsys_annotation_t _annotations[] = {
{ "errno", ROCPROFSYS_INT32, &_err }, { "strerror", ROCPROFSYS_STRING, _msg }
};
errno = 0; // reset errno
return (*original_callbacks.push_annotated_region)(
name, _annotations, sizeof(_annotations) / sizeof(omnitrace_annotation_t));
name, _annotations, sizeof(_annotations) / sizeof(rocprofsys_annotation_t));
}
return (*original_callbacks.push_region)(name);
}
Linking the Omnitrace libraries to another program
Linking the ROCm Systems Profiler libraries to another program
=======================================================
To link the ``omnitrace-user-library`` to another program,
To link the ``rocprofiler-systems-user-library`` to another program,
use the following CMake and ``g++`` directives.
CMake
@@ -166,19 +167,19 @@ CMake
.. code-block:: cmake
find_package(omnitrace REQUIRED COMPONENTS user)
find_package(rocprofiler-systems REQUIRED COMPONENTS user)
add_executable(foo foo.cpp)
target_link_libraries(foo PRIVATE omnitrace::omnitrace-user-library)
target_link_libraries(foo PRIVATE rocprofiler-systems::rocprofiler-systems-user-library)
g++ compilation
-------------------------------------------------------
Assuming Omnitrace is installed in ``/opt/omnitrace``, use the ``g++`` compiler
Assuming ROCm Systems Profiler is installed in ``/opt/rocprofiler-systems``, use the ``g++`` compiler
to build the application.
.. code-block:: shell
g++ -I/opt/omnitrace foo.cpp -o foo -lomnitrace-user
g++ -I/opt/rocprofiler-systems foo.cpp -o foo -lrocprofiler-systems-user
Output from the API example program
========================================
@@ -187,19 +188,19 @@ First, instrument and run the program.
.. code-block:: shell
$ omnitrace-instrument -l --min-instructions=8 -E custom_push_region -o -- ./user-api
$ rocprof-sys-instrument -l --min-instructions=8 -E custom_push_region -o -- ./user-api
...
$ omnitrace-run --profile --use-pid off --time-output off -- ./user-api.inst 20 4 100
$ rocprof-sys-run --profile --use-pid off --time-output off -- ./user-api.inst 20 4 100
Pushing custom region :: ./user-api.inst
[omnitrace][omnitrace_init_tooling] Instrumentation mode: Trace
[rocprof-sys][rocprofsys_init_tooling] Instrumentation mode: Trace
______ .___ ___. .__ __. __ .___________..______ ___ ______ _______
/ __ \ | \/ | | \ | | | | | || _ \ / \ / || ____|
| | | | | \ / | | \| | | | `---| |----`| |_) | / ^ \ | ,----'| |__
| | | | | |\/| | | . ` | | | | | | / / /_\ \ | | | __|
| `--' | | | | | | |\ | | | | | | |\ \----./ _____ \ | `----.| |____
\______/ |__| |__| |__| \__| |__| |__| | _| `._____/__/ \__\ \______||_______|
__
_ __ ___ ___ _ __ _ __ ___ / _| ___ _ _ ___
| '__| / _ \ / __| | '_ \ | '__| / _ \ | |_ _____ / __| | | | | / __|
| | | (_) | | (__ | |_) | | | | (_) | | _| |_____| \__ \ | |_| | \__ \
|_| \___/ \___| | .__/ |_| \___/ |_| |___/ \__, | |___/
|_| |___/
@@ -215,29 +216,29 @@ First, instrument and run the program.
Pushing custom region :: run(20) x 100
Pushing custom region :: run(20) x 100
[./user-api.inst] fibonacci(20) x 4 = 3382500
[omnitrace][86267][0][omnitrace_finalize] finalizing...
[rocprof-sys][86267][0][rocprofsys_finalize] finalizing...
[omnitrace][86267][0] omnitrace : 5.190895 sec wall_clock, 2.748 mb peak_rss, 6.330000 sec cpu_clock, 121.9 % cpu_util [laps: 1]
[omnitrace][86267][0] user-api.inst/thread-0 : 5.078713 sec wall_clock, 4.722415 sec thread_cpu_clock, 93.0 % thread_cpu_util, 1.276 mb peak_rss [laps: 1]
[omnitrace][86267][0] user-api.inst/thread-1 : 0.322248 sec wall_clock, 0.322191 sec thread_cpu_clock, 100.0 % thread_cpu_util, 1.000 mb peak_rss [laps: 1]
[omnitrace][86267][0] user-api.inst/thread-2 : 0.323255 sec wall_clock, 0.323194 sec thread_cpu_clock, 100.0 % thread_cpu_util, 0.000 mb peak_rss [laps: 1]
[omnitrace][86267][0] user-api.inst/thread-3 : 0.323569 sec wall_clock, 0.323484 sec thread_cpu_clock, 100.0 % thread_cpu_util, 1.092 mb peak_rss [laps: 1]
[omnitrace][86267][0] user-api.inst/thread-4 : 0.324178 sec wall_clock, 0.324057 sec thread_cpu_clock, 100.0 % thread_cpu_util, 1.184 mb peak_rss [laps: 1]
[omnitrace][86267][0] Post-processing 51 cpu frequency and memory usage entries...
[rocprof-sys][86267][0] rocprof-sys : 5.190895 sec wall_clock, 2.748 mb peak_rss, 6.330000 sec cpu_clock, 121.9 % cpu_util [laps: 1]
[rocprof-sys][86267][0] user-api.inst/thread-0 : 5.078713 sec wall_clock, 4.722415 sec thread_cpu_clock, 93.0 % thread_cpu_util, 1.276 mb peak_rss [laps: 1]
[rocprof-sys][86267][0] user-api.inst/thread-1 : 0.322248 sec wall_clock, 0.322191 sec thread_cpu_clock, 100.0 % thread_cpu_util, 1.000 mb peak_rss [laps: 1]
[rocprof-sys][86267][0] user-api.inst/thread-2 : 0.323255 sec wall_clock, 0.323194 sec thread_cpu_clock, 100.0 % thread_cpu_util, 0.000 mb peak_rss [laps: 1]
[rocprof-sys][86267][0] user-api.inst/thread-3 : 0.323569 sec wall_clock, 0.323484 sec thread_cpu_clock, 100.0 % thread_cpu_util, 1.092 mb peak_rss [laps: 1]
[rocprof-sys][86267][0] user-api.inst/thread-4 : 0.324178 sec wall_clock, 0.324057 sec thread_cpu_clock, 100.0 % thread_cpu_util, 1.184 mb peak_rss [laps: 1]
[rocprof-sys][86267][0] Post-processing 51 cpu frequency and memory usage entries...
[omnitrace][wall_clock]|0> Outputting 'omnitrace-user-api.inst-output/wall_clock.json'...
[omnitrace][wall_clock]|0> Outputting 'omnitrace-user-api.inst-output/wall_clock.tree.json'...
[omnitrace][wall_clock]|0> Outputting 'omnitrace-user-api.inst-output/wall_clock.txt'...
[rocprof-sys][wall_clock]|0> Outputting 'rocprof-sys-user-api.inst-output/wall_clock.json'...
[rocprof-sys][wall_clock]|0> Outputting 'rocprof-sys-user-api.inst-output/wall_clock.tree.json'...
[rocprof-sys][wall_clock]|0> Outputting 'rocprof-sys-user-api.inst-output/wall_clock.txt'...
[omnitrace][manager::finalize][metadata]> Outputting 'omnitrace-user-api.inst-output/metadata.json' and 'omnitrace-user-api.inst-output/functions.json'...
[omnitrace][86267][0][omnitrace_finalize] Finalized
[rocprof-sys][manager::finalize][metadata]> Outputting 'rocprof-sys-user-api.inst-output/metadata.json' and 'rocprof-sys-user-api.inst-output/functions.json'...
[rocprof-sys][86267][0][rocprofsys_finalize] Finalized
Then review the output.
.. code-block:: shell
$ cat omnitrace-example-output/wall_clock.txt
$ cat rocprof-sys-example-output/wall_clock.txt
|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| REAL-CLOCK TIMER (I.E. WALL-CLOCK TIMER) |
|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|