Update branding to ROCm Systems Profiler in documentation (#2)
* Update branding in docs
* Rename image used in documentation
* Update names of code samples.
In the code snippets, the "-" is not valid. ex., rocprof-sys_ --> rocprofsys_
* Update ASCII art
* update Doxyfile strip_from_path
* Add a "Formerly known as" message.
* Fixed typo in product name
ROCm Systems Profiler, not ROCm Profiler System
* Add "Omnitrace" back to the metadata keywords
* Update "install via package manager" section
* Update paths to user API files
* Rename configuration and environment settings
* Update Doxyfiles
Update publisher name & ID to "AMD".
Update bundle ID to "rocprofiler-systems"
* Update docs/what-is-rocprof-sys.rst
Co-authored-by: Jeffrey Novotny <jnovotny@amd.com>
* Update docs/conceptual/data-collection-modes.rst
Co-authored-by: Jeffrey Novotny <jnovotny@amd.com>
* Update docs/tutorials/video-tutorials.rst
Co-authored-by: Jeffrey Novotny <jnovotny@amd.com>
* Update docs/conceptual/rocprof-sys-feature-set.rst
Co-authored-by: Jeffrey Novotny <jnovotny@amd.com>
* Update docs/how-to/configuring-runtime-options.rst
Co-authored-by: Jeffrey Novotny <jnovotny@amd.com>
* Update docs/how-to/configuring-validating-environment.rst
Co-authored-by: Jeffrey Novotny <jnovotny@amd.com>
* Update docs/how-to/general-tips-using-rocprof-sys.rst
Co-authored-by: Jeffrey Novotny <jnovotny@amd.com>
* Update docs/reference/rocprof-sys-glossary.rst
Co-authored-by: Jeffrey Novotny <jnovotny@amd.com>
* Update docs/reference/development-guide.rst
Co-authored-by: Jeffrey Novotny <jnovotny@amd.com>
* Update docs/how-to/instrumenting-rewriting-binary-application.rst
Co-authored-by: Jeffrey Novotny <jnovotny@amd.com>
* Update docs/install/quick-start.rst
Co-authored-by: Jeffrey Novotny <jnovotny@amd.com>
* Note that videos were recorded using the "Omnitrace" name.
* Rebase and update some file paths
* Update paths to doc images
* Update Omnitrace references in code snippets
* Rename examples still using the "omni" prefix.
* Update docs/how-to/performing-causal-profiling.rst
Co-authored-by: Jeffrey Novotny <jnovotny@amd.com>
* Update docs/how-to/profiling-python-scripts.rst
Co-authored-by: Jeffrey Novotny <jnovotny@amd.com>
* Update docs/how-to/sampling-call-stack.rst
Co-authored-by: Jeffrey Novotny <jnovotny@amd.com>
* Update docs/how-to/understanding-rocprof-sys-output.rst
Co-authored-by: Jeffrey Novotny <jnovotny@amd.com>
* Update docs/install/install.rst
Co-authored-by: Jeffrey Novotny <jnovotny@amd.com>
---------
Co-authored-by: Peter Park <peter.park@amd.com>
Co-authored-by: Jeffrey Novotny <jnovotny@amd.com>
[ROCm/rocprofiler-systems commit: 032d39f15c]
@@ -1,17 +1,17 @@
|
||||
.. meta::
|
||||
:description: Omnitrace documentation and reference
|
||||
:keywords: Omnitrace, ROCm, profiler, tracking, visualization, tool, Instinct, accelerator, AMD
|
||||
:description: ROCm Systems Profiler data collection modes documentation
|
||||
:keywords: rocprof-sys, rocprofiler-systems, Omnitrace, ROCm, profiler, data collection, tracking, visualization, tool, Instinct, accelerator, AMD
|
||||
|
||||
**********************
|
||||
Data collection modes
|
||||
**********************
|
||||
|
||||
Omnitrace supports several modes of recording trace and profiling data for your application.
|
||||
ROCm Systems Profiler supports several modes of recording trace and profiling data for your application.
|
||||
|
||||
.. note::
|
||||
|
||||
For an explanation of the terms used in this topic, see
|
||||
the :doc:`Omnitrace glossary <../reference/omnitrace-glossary>`.
|
||||
|
||||
For an explanation of the terms used in this topic, see
|
||||
the :doc:`ROCm Systems Profiler glossary <../reference/rocprof-sys-glossary>`.
|
||||
|
||||
+-----------------------------+---------------------------------------------------------+
|
||||
| Mode | Description |
|
||||
@@ -23,61 +23,62 @@ Omnitrace supports several modes of recording trace and profiling data for your
|
||||
| | and records various metrics for the given call stack |
|
||||
+-----------------------------+---------------------------------------------------------+
|
||||
| Callback APIs | Parallelism frameworks such as ROCm, OpenMP, and Kokkos |
|
||||
| | make callbacks into Omnitrace to provide information |
|
||||
| | about the work the API is performing |
|
||||
| | make callbacks into ROCm Systems Profiler to provide |
|
||||
| | information about the work the API is performing |
|
||||
+-----------------------------+---------------------------------------------------------+
|
||||
| Dynamic Symbol Interception | Wrap function symbols defined in a position independent |
|
||||
| | dynamic library/executable, like ``pthread_mutex_lock`` |
|
||||
| | in ``libpthread.so`` or ``MPI_Init`` in the MPI library |
|
||||
+-----------------------------+---------------------------------------------------------+
|
||||
| User API | User-defined regions and controls for Omnitrace |
|
||||
| User API | User-defined regions and controls for ROCm Systems |
|
||||
| | Profiler |
|
||||
+-----------------------------+---------------------------------------------------------+
|
||||
|
||||
The two most generic and important modes are binary instrumentation and statistical sampling.
|
||||
The two most generic and important modes are binary instrumentation and statistical sampling.
|
||||
It is important to understand their advantages and disadvantages.
|
||||
Binary instrumentation and statistical sampling can be performed with the ``omnitrace-instrument``
|
||||
Binary instrumentation and statistical sampling can be performed with the ``rocprof-sys-instrument``
|
||||
executable. For statistical sampling, it's highly recommended to use the
|
||||
``omnitrace-sample`` executable instead if binary instrumentation isn't required or needed.
|
||||
``rocprof-sys-sample`` executable instead if binary instrumentation isn't required or needed.
|
||||
Callback APIs and dynamic symbol interception can be utilized with either tool.
|
||||
|
||||
Binary instrumentation
|
||||
-----------------------------------
|
||||
|
||||
Binary instrumentation lets you record deterministic measurements for
|
||||
Binary instrumentation lets you record deterministic measurements for
|
||||
every single invocation of a given function.
|
||||
Binary instrumentation effectively adds instructions to the target application to
|
||||
collect the required information. It therefore has the potential to cause performance
|
||||
changes which might, in some cases, lead to inaccurate results. The effect depends on
|
||||
the information being collected and which features are activated in Omnitrace.
|
||||
Binary instrumentation effectively adds instructions to the target application to
|
||||
collect the required information. It therefore has the potential to cause performance
|
||||
changes which might, in some cases, lead to inaccurate results. The effect depends on
|
||||
the information being collected and which features are activated in ROCm Systems Profiler.
|
||||
For example, collecting only the wall-clock timing data
|
||||
has less of an effect than collecting the wall-clock timing, CPU-clock timing,
|
||||
memory usage, cache-misses, and number of instructions that were run. Similarly,
|
||||
collecting a flat profile has less overhead than a hierarchical profile
|
||||
and collecting a trace OR a profile has less overhead than collecting a
|
||||
has less of an effect than collecting the wall-clock timing, CPU-clock timing,
|
||||
memory usage, cache-misses, and number of instructions that were run. Similarly,
|
||||
collecting a flat profile has less overhead than a hierarchical profile
|
||||
and collecting a trace OR a profile has less overhead than collecting a
|
||||
trace AND a profile.
|
||||
|
||||
In Omnitrace, the primary heuristic for controlling the overhead with binary
|
||||
instrumentation is the minimum number of instructions for selecting functions
|
||||
In ROCm Systems Profiler, the primary heuristic for controlling the overhead with binary
|
||||
instrumentation is the minimum number of instructions for selecting functions
|
||||
for instrumentation.
|
||||
|
||||
Statistical sampling
|
||||
-----------------------------------
|
||||
|
||||
Statistical call-stack sampling periodically interrupts the application at
|
||||
Statistical call-stack sampling periodically interrupts the application at
|
||||
regular intervals using operating system interrupts.
|
||||
Sampling is typically less numerically accurate and specific, but the
|
||||
Sampling is typically less numerically accurate and specific, but the
|
||||
target program runs at nearly full speed.
|
||||
In contrast to the data derived from binary instrumentation, the resulting
|
||||
In contrast to the data derived from binary instrumentation, the resulting
|
||||
data is not exact but is instead a statistical approximation.
|
||||
However, sampling often provides a more accurate picture of the application
|
||||
However, sampling often provides a more accurate picture of the application
|
||||
execution because it is less intrusive to the target application and has fewer
|
||||
side effects on memory caches or instruction decoding pipelines. Furthermore,
|
||||
side effects on memory caches or instruction decoding pipelines. Furthermore,
|
||||
because sampling does not affect the execution speed as much, is it
|
||||
relatively immune to over-evaluating the cost of small, frequently called
|
||||
relatively immune to over-evaluating the cost of small, frequently called
|
||||
functions or "tight" loops.
|
||||
|
||||
In Omnitrace, the overhead for statistical sampling depends on the
|
||||
sampling rate and whether the samples are taken with respect to the CPU time
|
||||
In ROCm Systems Profiler, the overhead for statistical sampling depends on the
|
||||
sampling rate and whether the samples are taken with respect to the CPU time
|
||||
and/or real time.
|
||||
|
||||
Binary instrumentation vs. statistical sampling example
|
||||
@@ -112,24 +113,24 @@ Consider the following code:
|
||||
return 0;
|
||||
}
|
||||
|
||||
Binary instrumentation of the ``fib`` function will record **every single invocation**
|
||||
Binary instrumentation of the ``fib`` function will record **every single invocation**
|
||||
of the function. For a very small function
|
||||
such as ``fib``, this results in **significant** overhead since this simple function
|
||||
such as ``fib``, this results in **significant** overhead since this simple function
|
||||
takes about 20 instructions, whereas the entry and
|
||||
exit snippets are ~1024 instructions. Therefore, you generally want to avoid
|
||||
exit snippets are ~1024 instructions. Therefore, you generally want to avoid
|
||||
instrumenting functions where the instrumented function has significantly fewer
|
||||
instructions than entry and exit instrumentation. (Note that many of the
|
||||
instructions than entry and exit instrumentation. (Note that many of the
|
||||
instructions in entry and exit functions are either logging functions or
|
||||
depend on the runtime settings and thus might never run). However,
|
||||
depend on the runtime settings and thus might never run). However,
|
||||
due to the number of potential instructions in the entry and exit snippets,
|
||||
the default behavior of ``omnitrace-instrument`` is to only instrument functions
|
||||
the default behavior of ``rocprof-sys-instrument`` is to only instrument functions
|
||||
which contain at least 1024 instructions.
|
||||
|
||||
However, recording every single invocation of the function can be extremely
|
||||
However, recording every single invocation of the function can be extremely
|
||||
useful for detecting anomalies, such as profiles that show minimum or maximum values much smaller or larger
|
||||
than the average or a high standard deviation. In this case, the traces help you
|
||||
than the average or a high standard deviation. In this case, the traces help you
|
||||
identify exactly when and where those instances deviated from the norm.
|
||||
Compare the level of detail in the following traces. In the top image,
|
||||
Compare the level of detail in the following traces. In the top image,
|
||||
every instance of the ``fib`` function is instrumented, while in the bottom image,
|
||||
the ``fib`` call-stack is derived via sampling.
|
||||
|
||||
|
||||
@@ -1,14 +1,14 @@
|
||||
.. meta::
|
||||
:description: Omnitrace documentation and reference
|
||||
:keywords: Omnitrace, ROCm, profiler, tracking, visualization, tool, Instinct, accelerator, AMD
|
||||
:description: ROCm Systems Profiler feature set documentation and reference
|
||||
:keywords: rocprof-sys, rocprofiler-systems, Omnitrace, ROCm, profiler, feature set, use cases, tracking, visualization, tool, Instinct, accelerator, AMD
|
||||
|
||||
***************************************
|
||||
The Omnitrace feature set and use cases
|
||||
The ROCm Systems Profiler feature set and use cases
|
||||
***************************************
|
||||
|
||||
`Omnitrace <https://github.com/ROCm/omnitrace>`_ is designed to be highly extensible.
|
||||
Internally, it leverages the `Timemory performance analysis toolkit <https://github.com/NERSC/timemory>`_
|
||||
to manage extensions, resources, data, and other items. It supports the following features,
|
||||
`ROCm Systems Profiler <https://github.com/ROCm/rocprofiler-systems>`_ is designed to be highly extensible.
|
||||
Internally, it leverages the `Timemory performance analysis toolkit <https://github.com/NERSC/timemory>`_
|
||||
to manage extensions, resources, data, and other items. It supports the following features,
|
||||
modes, metrics, and APIs.
|
||||
|
||||
Data collection modes
|
||||
@@ -22,11 +22,6 @@ Data collection modes
|
||||
* Statistical sampling: Periodic software interrupts per-thread
|
||||
* Process-level sampling: A background thread records process-, system- and device-level metrics while the application runs
|
||||
* Causal profiling: Quantifies the potential impact of optimizations in parallel code
|
||||
|
||||
.. note::
|
||||
|
||||
Critical trace support was removed in Omnitrace v1.11.0.
|
||||
It was replaced by the causal profiling feature.
|
||||
|
||||
Data analysis
|
||||
========================================
|
||||
@@ -98,40 +93,40 @@ Third-party API support
|
||||
* NVTX
|
||||
* ROCTX
|
||||
|
||||
Omnitrace use cases
|
||||
ROCm Systems Profiler use cases
|
||||
========================================
|
||||
|
||||
When analyzing the performance of an application, do NOT
|
||||
When analyzing the performance of an application, do NOT
|
||||
assume you know where the performance bottlenecks are
|
||||
and why they are happening. Omnitrace is a tool for analyzing the entire
|
||||
and why they are happening. ROCm Systems Profiler is a tool for analyzing the entire
|
||||
application and its performance. It is
|
||||
ideal for characterizing where optimization would have the greatest impact
|
||||
ideal for characterizing where optimization would have the greatest impact
|
||||
on an end-to-end run of the application and for
|
||||
viewing what else is happening on the system during a performance bottleneck.
|
||||
|
||||
When GPUs are involved, there is a tendency to assume that
|
||||
When GPUs are involved, there is a tendency to assume that
|
||||
the quickest path to performance improvement is minimizing
|
||||
the runtime of the GPU kernels. This is a highly flawed assumption.
|
||||
the runtime of the GPU kernels. This is a highly flawed assumption.
|
||||
If you optimize the runtime of a kernel from one millisecond
|
||||
to 1 microsecond (1000x speed-up) but the original application never
|
||||
to 1 microsecond (1000x speed-up) but the original application never
|
||||
spent time waiting for kernels to complete,
|
||||
there would be no statistically significant reduction in the end-to-end
|
||||
there would be no statistically significant reduction in the end-to-end
|
||||
runtime of your application. In other words, it does not matter
|
||||
how fast or slow the code on GPU is if the application has a
|
||||
how fast or slow the code on GPU is if the application has a
|
||||
bottleneck on waiting on the GPU.
|
||||
|
||||
Use Omnitrace to obtain a high-level view of the entire application. Use it
|
||||
Use ROCm Systems Profiler to obtain a high-level view of the entire application. Use it
|
||||
to determine where the performance bottlenecks are and
|
||||
obtain clues to why these bottlenecks are happening. Rather than worrying about kernel
|
||||
performance, start your investigation with Omnitrace, which characterizes the
|
||||
performance, start your investigation with ROCm Systems Profiler, which characterizes the
|
||||
broad picture.
|
||||
|
||||
.. note::
|
||||
|
||||
For insight into the execution of individual kernels on the GPU,
|
||||
use `Omniperf <https://github.com/rocm/omniperf>`_.
|
||||
For insight into the execution of individual kernels on the GPU,
|
||||
use `ROCm Compute Profiler <https://github.com/rocm/rocprofiler-compute>`_.
|
||||
|
||||
In terms of CPU analysis, Omnitrace does not target any specific vendor.
|
||||
In terms of CPU analysis, ROCm Systems Profiler does not target any specific vendor.
|
||||
It works just as well on AMD and non-AMD CPUs.
|
||||
With regard to the GPU, Omnitrace is currently restricted to HIP and HSA APIs
|
||||
With regard to the GPU, ROCm Systems Profiler is currently restricted to HIP and HSA APIs
|
||||
and kernels running on AMD GPUs.
|
||||
@@ -36,14 +36,14 @@ with open("../VERSION", encoding="utf-8") as f:
|
||||
raise ValueError("VERSION not found!")
|
||||
version_number = match[1]
|
||||
|
||||
external_projects_current_project = "omnitrace"
|
||||
external_projects_current_project = "rocprofiler-systems"
|
||||
|
||||
project = "omnitrace"
|
||||
project = "rocprofiler-systems"
|
||||
author = "Advanced Micro Devices, Inc."
|
||||
copyright = "Copyright (c) 2024 Advanced Micro Devices, Inc. All rights reserved."
|
||||
version = version_number
|
||||
release = version_number
|
||||
html_title = f"Omnitrace {version} documentation"
|
||||
html_title = f"ROCm Systems Profiler {version} documentation"
|
||||
|
||||
external_toc_path = "./sphinx/_toc.yml"
|
||||
|
||||
|
||||
|
До Ширина: | Высота: | Размер: 313 KiB После Ширина: | Высота: | Размер: 313 KiB |
|
До Ширина: | Высота: | Размер: 195 KiB После Ширина: | Высота: | Размер: 195 KiB |
|
До Ширина: | Высота: | Размер: 230 KiB После Ширина: | Высота: | Размер: 230 KiB |
|
До Ширина: | Высота: | Размер: 277 KiB После Ширина: | Высота: | Размер: 277 KiB |
@@ -4,7 +4,7 @@
|
||||
# Project related configuration options
|
||||
#---------------------------------------------------------------------------
|
||||
DOXYFILE_ENCODING = UTF-8
|
||||
PROJECT_NAME = omnitrace
|
||||
PROJECT_NAME = rocprofiler-systems
|
||||
PROJECT_NUMBER = 1.11.3
|
||||
PROJECT_BRIEF = "High-level and comprehensive application tracing and profiling on both the CPU and GPU"
|
||||
PROJECT_LOGO =
|
||||
@@ -19,8 +19,8 @@ ABBREVIATE_BRIEF =
|
||||
ALWAYS_DETAILED_SEC = YES
|
||||
INLINE_INHERITED_MEMB = YES
|
||||
FULL_PATH_NAMES = YES
|
||||
STRIP_FROM_PATH = /home/docs/checkouts/readthedocs.org/user_builds/advanced-micro-devices-omnitrace/checkouts/
|
||||
STRIP_FROM_INC_PATH = /home/docs/checkouts/readthedocs.org/user_builds/advanced-micro-devices-omnitrace/checkouts/
|
||||
STRIP_FROM_PATH = /home/docs/checkouts/readthedocs.org/user_builds/advanced-micro-devices-rocprofiler-systems/checkouts/
|
||||
STRIP_FROM_INC_PATH = /home/docs/checkouts/readthedocs.org/user_builds/advanced-micro-devices-rocprofiler-systems/checkouts/
|
||||
SHORT_NAMES = NO
|
||||
JAVADOC_AUTOBRIEF = NO
|
||||
JAVADOC_BANNER = NO
|
||||
@@ -114,10 +114,10 @@ WARN_LOGFILE = doc/warnings.log
|
||||
# Configuration options related to the input files
|
||||
#---------------------------------------------------------------------------
|
||||
INPUT = ../../README.md \
|
||||
../../source/lib/omnitrace-user/omnitrace/types.h \
|
||||
../../source/lib/omnitrace-user/omnitrace/categories.h \
|
||||
../../source/lib/omnitrace-user/omnitrace/user.h \
|
||||
../../source/lib/omnitrace-user/omnitrace/causal.h
|
||||
../../source/lib/rocprof-sys-user/rocprofiler-systems/types.h \
|
||||
../../source/lib/rocprof-sys-user/rocprofiler-systems/categories.h \
|
||||
../../source/lib/rocprof-sys-user/rocprofiler-systems/user.h \
|
||||
../../source/lib/rocprof-sys-user/rocprofiler-systems/causal.h
|
||||
INPUT_ENCODING = UTF-8
|
||||
FILE_PATTERNS = *.h \
|
||||
*.hh \
|
||||
@@ -198,9 +198,9 @@ HTML_DYNAMIC_SECTIONS = YES
|
||||
HTML_INDEX_NUM_ENTRIES = 1000
|
||||
GENERATE_DOCSET = NO
|
||||
DOCSET_FEEDNAME = "Doxygen generated docs"
|
||||
DOCSET_BUNDLE_ID = org.doxygen.omnitrace
|
||||
DOCSET_PUBLISHER_ID = org.doxygen.amdresearch
|
||||
DOCSET_PUBLISHER_NAME = "Audacious Software Group"
|
||||
DOCSET_BUNDLE_ID = org.doxygen.rocprofiler-systems
|
||||
DOCSET_PUBLISHER_ID = org.doxygen.amd
|
||||
DOCSET_PUBLISHER_NAME = "Advanced Micro Devices, Inc."
|
||||
GENERATE_HTMLHELP = NO
|
||||
CHM_FILE =
|
||||
HHC_LOCATION =
|
||||
@@ -217,7 +217,7 @@ QHP_CUST_FILTER_ATTRS =
|
||||
QHP_SECT_FILTER_ATTRS =
|
||||
QHG_LOCATION =
|
||||
GENERATE_ECLIPSEHELP = NO
|
||||
ECLIPSE_DOC_ID = org.doxygen.omnitrace
|
||||
ECLIPSE_DOC_ID = org.doxygen.rocprofiler-systems
|
||||
DISABLE_INDEX = NO
|
||||
GENERATE_TREEVIEW = NO
|
||||
ENUM_VALUES_PER_LINE = 1
|
||||
@@ -311,7 +311,7 @@ ENABLE_PREPROCESSING = YES
|
||||
MACRO_EXPANSION = YES
|
||||
EXPAND_ONLY_PREDEF = NO
|
||||
SEARCH_INCLUDES = YES
|
||||
INCLUDE_PATH = ../../source/lib/omnitrace-user
|
||||
INCLUDE_PATH = ../../source/lib/rocprof-sys-user
|
||||
INCLUDE_FILE_PATTERNS = *.h \
|
||||
*.hpp
|
||||
PREDEFINED = ROCPROFSYS_PUBLIC_API= \
|
||||
|
||||
@@ -1,131 +1,133 @@
|
||||
.. meta::
|
||||
:description: Omnitrace documentation and reference
|
||||
:keywords: Omnitrace, ROCm, profiler, tracking, visualization, tool, Instinct, accelerator, AMD
|
||||
:description: ROCm Systems Profiler runtime options documentation and reference
|
||||
:keywords: rocprof-sys, rocprofiler-systems, Omnitrace, ROCm, runtime options, profiler, tracking, visualization, tool, Instinct, accelerator, AMD
|
||||
|
||||
****************************************************
|
||||
Configuring runtime options
|
||||
****************************************************
|
||||
|
||||
The ``omnitrace.cfg`` file maintains a list of the `Omnitrace <https://github.com/ROCm/omnitrace>`_ runtime options. To create this configuration
|
||||
file and view the current runtime options, use the ``omnitrace-avail`` executable.
|
||||
The ``rocprof-sys.cfg`` file maintains a list of the
|
||||
`ROCm Systems Profiler <https://github.com/ROCm/rocprofiler-systems>`_ runtime
|
||||
options. To create this configuration
|
||||
file and view the current runtime options, use the ``rocprof-sys-avail`` executable.
|
||||
|
||||
The omnitrace-avail executable
|
||||
The rocprof-sys-avail executable
|
||||
========================================
|
||||
|
||||
The ``omnitrace-avail`` executable provides information about the runtime settings,
|
||||
The ``rocprof-sys-avail`` executable provides information about the runtime settings,
|
||||
data collection capabilities, and, when built with PAPI support, the
|
||||
available hardware counters. The executable is effectively
|
||||
self-updating. As new capabilities and settings are added to the Omnitrace source code, they are
|
||||
propagated to ``omnitrace-avail``. ``omnitrace-avail`` should be viewed as the ultimate authority
|
||||
self-updating. As new capabilities and settings are added to the ROCm Systems Profiler source code, they are
|
||||
propagated to ``rocprof-sys-avail``. ``rocprof-sys-avail`` should be viewed as the ultimate authority
|
||||
in the event of any conflicts with this documentation.
|
||||
|
||||
It is recommended that you create a default configuration file in
|
||||
``${HOME}/.omnitrace.cfg``. This can be done by
|
||||
running the command ``omnitrace-avail -G ~/.omnitrace.cfg``. Alternatively,
|
||||
use the ``omnitrace-avail -G ~/.omnitrace.cfg --all`` option
|
||||
It is recommended that you create a default configuration file in
|
||||
``${HOME}/.rocprof-sys.cfg``. This can be done by
|
||||
running the command ``rocprof-sys-avail -G ~/.rocprof-sys.cfg``. Alternatively,
|
||||
use the ``rocprof-sys-avail -G ~/.rocprof-sys.cfg --all`` option
|
||||
for a verbose configuration file with descriptions, categories, and additional information.
|
||||
|
||||
Modify ``${HOME}/.omnitrace.cfg`` as required. For example, enable `Perfetto <https://perfetto.dev/>`_,
|
||||
Modify ``${HOME}/.rocprof-sys.cfg`` as required. For example, enable `Perfetto <https://perfetto.dev/>`_,
|
||||
`Timemory <https://github.com/NERSC/timemory>`_, sampling, and process-level sampling by default
|
||||
and tweak the default sampling values.
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
# ...
|
||||
OMNITRACE_TRACE = true
|
||||
OMNITRACE_PROFILE = true
|
||||
OMNITRACE_USE_SAMPLING = true
|
||||
OMNITRACE_USE_PROCESS_SAMPLING = true
|
||||
ROCPROFSYS_TRACE = true
|
||||
ROCPROFSYS_PROFILE = true
|
||||
ROCPROFSYS_USE_SAMPLING = true
|
||||
ROCPROFSYS_USE_PROCESS_SAMPLING = true
|
||||
# ...
|
||||
OMNITRACE_SAMPLING_FREQ = 50
|
||||
OMNITRACE_SAMPLING_CPUS = all
|
||||
OMNITRACE_SAMPLING_GPUS = $env:HIP_VISIBLE_DEVICES
|
||||
ROCPROFSYS_SAMPLING_FREQ = 50
|
||||
ROCPROFSYS_SAMPLING_CPUS = all
|
||||
ROCPROFSYS_SAMPLING_GPUS = $env:HIP_VISIBLE_DEVICES
|
||||
|
||||
Exploring runtime settings
|
||||
-----------------------------------
|
||||
|
||||
Use the following command to view the list of the available runtime settings, their current values, and descriptions
|
||||
Use the following command to view the list of the available runtime settings, their current values, and descriptions
|
||||
for each setting:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
omnitrace-avail --description
|
||||
rocprof-sys-avail --description
|
||||
|
||||
.. note::
|
||||
|
||||
Use ``--brief`` to suppress printing the current value and/or ``-c 0`` to suppress truncation of the descriptions.
|
||||
|
||||
Any Boolean setting (``omnitrace-avail --settings --value --brief --filter bool``)
|
||||
accepts a case insensitive match for nearly all common Boolean logic expressions:
|
||||
Any Boolean setting (``rocprof-sys-avail --settings --value --brief --filter bool``)
|
||||
accepts a case insensitive match for nearly all common Boolean logic expressions:
|
||||
``ON``, ``OFF``, ``YES``, ``NO``, ``TRUE``, ``FALSE``, ``0``, ``1``, etc.
|
||||
|
||||
Exploring components
|
||||
-----------------------------------
|
||||
|
||||
Omnitrace uses `Timemory <https://github.com/NERSC/timemory>`_ extensively to provide
|
||||
ROCm Systems Profiler uses `Timemory <https://github.com/NERSC/timemory>`_ extensively to provide
|
||||
various capabilities and manage
|
||||
data and resources. By default, with ``OMNITRACE_PROFILE=ON``, Omnitrace only collects wall-clock
|
||||
timing values. However, by modifying the ``OMNITRACE_TIMEMORY_COMPONENTS`` setting,
|
||||
Omnitrace can be configured to
|
||||
data and resources. By default, with ``ROCPROFSYS_PROFILE=ON``, ROCm Systems Profiler only collects wall-clock
|
||||
timing values. However, by modifying the ``ROCPROFSYS_TIMEMORY_COMPONENTS`` setting,
|
||||
ROCm Systems Profiler can be configured to
|
||||
collect hardware counters, CPU-clock timers, memory usage, context switches, page faults, network statistics,
|
||||
and much more. Omnitrace can even be used as a dynamic instrumentation vehicle
|
||||
and much more. ROCm Systems Profiler can even be used as a dynamic instrumentation vehicle
|
||||
for other third-party profiling
|
||||
APIs such as `Caliper <https://github.com/LLNL/Caliper>`_ and `LIKWID <https://github.com/RRZE-HPC/likwid>`_.
|
||||
To leverage this capability, build Omnitrace from source with the CMake
|
||||
To leverage this capability, build ROCm Systems Profiler from source with the CMake
|
||||
options ``TIMEMORY_USE_CALIPER=ON`` or ``TIMEMORY_USE_LIKWID=ON`` and then add
|
||||
``caliper_marker``, ``likwid_marker``, or both to ``OMNITRACE_TIMEMORY_COMPONENTS``.
|
||||
``caliper_marker``, ``likwid_marker``, or both to ``ROCPROFSYS_TIMEMORY_COMPONENTS``.
|
||||
|
||||
To view all possible components and their descriptions:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
omnitrace-avail --components --description
|
||||
rocprof-sys-avail --components --description
|
||||
|
||||
To restrict the output to available components and view the string identifiers for ``OMNITRACE_TIMEMORY_COMPONENTS``:
|
||||
To restrict the output to available components and view the string identifiers for ``ROCPROFSYS_TIMEMORY_COMPONENTS``:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
omnitrace-avail --components --available --string --brief
|
||||
rocprof-sys-avail --components --available --string --brief
|
||||
|
||||
Exploring hardware counters
|
||||
-----------------------------------
|
||||
|
||||
Omnitrace supports hardware counter collection via PAPI and ROCm.
|
||||
ROCm Systems Profiler supports hardware counter collection via PAPI and ROCm.
|
||||
Generally, PAPI is used to collect CPU-based hardware counters and ROCm is used to collect GPU-based hardware
|
||||
counters. Although it is possible to install PAPI with ROCm support and use it to
|
||||
collect GPU-based hardware counters, this is not recommended because PAPI
|
||||
counters. Although it is possible to install PAPI with ROCm support and use it to
|
||||
collect GPU-based hardware counters, this is not recommended because PAPI
|
||||
cannot simultaneously collect CPU and GPU hardware counters.
|
||||
|
||||
To view all possible hardware counters and their descriptions, use the following command:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
omnitrace-avail --hw-counters --description
|
||||
rocprof-sys-avail --hw-counters --description
|
||||
|
||||
Appending the ``-c CPU`` option restricts the list of hardware counters to
|
||||
Appending the ``-c CPU`` option restricts the list of hardware counters to
|
||||
those available through PAPI, while ``-c GPU`` limits the list to those available from ROCm.
|
||||
|
||||
Enabling hardware counters
|
||||
-----------------------------------
|
||||
|
||||
PAPI Hardware counters are configured with the ``OMNITRACE_PAPI_EVENTS`` configuration variable.
|
||||
ROCm Hardware counters are configured with the ``OMNITRACE_ROCM_EVENTS`` configuration variable.
|
||||
ROCm hardware counters also require the ``OMNITRACE_USE_ROCPROFILER`` configuration
|
||||
variable to be enabled using ``OMNITRACE_USE_ROCPROFILER=ON``.
|
||||
PAPI Hardware counters are configured with the ``ROCPROFSYS_PAPI_EVENTS`` configuration variable.
|
||||
ROCm Hardware counters are configured with the ``ROCPROFSYS_ROCM_EVENTS`` configuration variable.
|
||||
ROCm hardware counters also require the ``ROCPROFSYS_USE_ROCPROFILER`` configuration
|
||||
variable to be enabled using ``ROCPROFSYS_USE_ROCPROFILER=ON``.
|
||||
|
||||
Here is a sample configuration for hardware counters:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
# using papi identifiers
|
||||
OMNITRACE_PAPI_EVENTS = PAPI_TOT_CYC PAPI_TOT_INS
|
||||
ROCPROFSYS_PAPI_EVENTS = PAPI_TOT_CYC PAPI_TOT_INS
|
||||
|
||||
# using perf identifiers
|
||||
OMNITRACE_PAPI_EVENTS = perf::INSTRUCTIONS perf::CACHE-REFERENCES perf::CACHE-MISSES
|
||||
ROCPROFSYS_PAPI_EVENTS = perf::INSTRUCTIONS perf::CACHE-REFERENCES perf::CACHE-MISSES
|
||||
|
||||
.. _omnitrace_papi_events:
|
||||
.. _rocprof-sys_papi_events:
|
||||
|
||||
OMNITRACE_PAPI_EVENTS
|
||||
ROCPROFSYS_PAPI_EVENTS
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
In order to collect the majority of hardware counters via PAPI, ensure the ``/proc/sys/kernel/perf_event_paranoid``
|
||||
@@ -135,18 +137,18 @@ has a value <= 2. If you have ``sudo`` access, use the following command to modi
|
||||
|
||||
echo 0 | sudo tee /proc/sys/kernel/perf_event_paranoid
|
||||
|
||||
However this value is not retained upon reboot.
|
||||
However this value is not retained upon reboot.
|
||||
Use the following command to preserve this setting after a reboot:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
echo 'kernel.perf_event_paranoid=0' | sudo tee -a /etc/sysctl.conf
|
||||
|
||||
PAPI events use a concept similar to a namespace. All specified hardware
|
||||
PAPI events use a concept similar to a namespace. All specified hardware
|
||||
counters must be from the same namespace.
|
||||
For hardware counters starting with the ``PAPI_`` prefix, these are high-level
|
||||
For hardware counters starting with the ``PAPI_`` prefix, these are high-level
|
||||
aggregates of multiple hardware counters.
|
||||
Otherwise, most events use two or three colons (``::`` or ``:::``) between the
|
||||
Otherwise, most events use two or three colons (``::`` or ``:::``) between the
|
||||
component name and the counter name, for example,
|
||||
``amd64_rapl::RAPL_ENERGY_PKG`` and ``perf::PERF_COUNT_HW_CPU_CYCLES``.
|
||||
|
||||
@@ -154,33 +156,33 @@ For example, the following is a valid configuration:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
OMNITRACE_PAPI_EVENTS = perf::INSTRUCTIONS perf::CACHE-REFERENCES perf::CACHE-MISSES
|
||||
ROCPROFSYS_PAPI_EVENTS = perf::INSTRUCTIONS perf::CACHE-REFERENCES perf::CACHE-MISSES
|
||||
|
||||
However, the following specification of a roughly equivalent set of hardware counters is an incorrect configuration because it mixes
|
||||
PAPI components from different namespaces:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
OMNITRACE_PAPI_EVENTS = PAPI_TOT_INS perf::CACHE-REFERENCES perf::CACHE-MISSES
|
||||
ROCPROFSYS_PAPI_EVENTS = PAPI_TOT_INS perf::CACHE-REFERENCES perf::CACHE-MISSES
|
||||
|
||||
.. note::
|
||||
|
||||
If Omnitrace was configured with the default ``OMNITRACE_BUILD_PAPI=ON`` setting,
|
||||
If ROCm Systems Profiler was configured with the default ``ROCPROFSYS_BUILD_PAPI=ON`` setting,
|
||||
standard PAPI command-line tools such as
|
||||
``papi_avail`` and ``papi_event_chooser`` are not able to provide information
|
||||
about the PAPI library used by Omnitrace
|
||||
(because Omnitrace statically links to ``libpapi``). However, all of these tools are
|
||||
installed with the prefix ``omnitrace-`` with
|
||||
underscores replaced with hypens, for example ``papi_avail`` becomes ``omnitrace-papi-avail``.
|
||||
``papi_avail`` and ``papi_event_chooser`` are not able to provide information
|
||||
about the PAPI library used by ROCm Systems Profiler
|
||||
(because ROCm Systems Profiler statically links to ``libpapi``). However, all of these tools are
|
||||
installed with the prefix ``rocprof-sys-`` with
|
||||
underscores replaced with hypens, for example ``papi_avail`` becomes ``rocprof-sys-papi-avail``.
|
||||
|
||||
OMNITRACE_ROCM_EVENTS
|
||||
ROCPROFSYS_ROCM_EVENTS
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Omnitrace reads the ROCm events from the ``${ROCM_PATH}/lib/rocprofiler/metrics.xml``
|
||||
ROCm Systems Profiler reads the ROCm events from the ``${ROCM_PATH}/lib/rocprofiler/metrics.xml``
|
||||
file. Use the ``ROCP_METRICS`` environment
|
||||
variable to point Omnitrace to a different XML metrics file, for example,
|
||||
variable to point ROCm Systems Profiler to a different XML metrics file, for example,
|
||||
``export ROCP_METRICS=${PWD}/custom_metrics.xml``.
|
||||
``omnitrace-avail -H -c GPU`` shows event names with a suffix of ``:device=N``
|
||||
``rocprof-sys-avail -H -c GPU`` shows event names with a suffix of ``:device=N``
|
||||
where ``N`` is the device number.
|
||||
For example, if you have two devices, the output is:
|
||||
|
||||
@@ -190,7 +192,7 @@ For example, if you have two devices, the output is:
|
||||
...
|
||||
| Wavefronts:device=1 | Derived counter: SQ_WAVES |
|
||||
|
||||
To collect the event on all devices, specify the event,
|
||||
To collect the event on all devices, specify the event,
|
||||
such as ``Wavefronts``, without the ``:device=`` suffix.
|
||||
To collect the event only on specific devices, use the ``:device=`` suffix.
|
||||
|
||||
@@ -202,12 +204,12 @@ The following example:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
OMNITRACE_ROCM_EVENTS = GPUBusy SQ_WAVES:device=0 SQ_INSTS_VALU:device=1
|
||||
ROCPROFSYS_ROCM_EVENTS = GPUBusy SQ_WAVES:device=0 SQ_INSTS_VALU:device=1
|
||||
|
||||
omnitrace-avail examples
|
||||
rocprof-sys-avail examples
|
||||
-----------------------------------
|
||||
|
||||
The following examples demonstrate how to use ``omnitrace-avail`` to perform several common
|
||||
The following examples demonstrate how to use ``rocprof-sys-avail`` to perform several common
|
||||
configuration tasks.
|
||||
|
||||
Generating a default configuration file
|
||||
@@ -215,96 +217,96 @@ Generating a default configuration file
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
$ omnitrace-avail -G ~/.omnitrace.cfg
|
||||
[omnitrace-avail] Outputting text configuration file '/home/user/.omnitrace.cfg'...
|
||||
$ cat ~/.omnitrace.cfg
|
||||
# auto-generated by omnitrace-avail (version 1.2.0) on 2022-06-27 @ 19:15
|
||||
$ rocprof-sys-avail -G ~/.rocprof-sys.cfg
|
||||
[rocprof-sys-avail] Outputting text configuration file '/home/user/.rocprof-sys.cfg'...
|
||||
$ cat ~/.rocprof-sys.cfg
|
||||
# auto-generated by rocprof-sys-avail (version 1.2.0) on 2022-06-27 @ 19:15
|
||||
|
||||
OMNITRACE_CONFIG_FILE =
|
||||
OMNITRACE_MODE = trace
|
||||
OMNITRACE_TRACE = true
|
||||
OMNITRACE_PROFILE = false
|
||||
OMNITRACE_USE_SAMPLING = false
|
||||
OMNITRACE_USE_PROCESS_SAMPLING = true
|
||||
OMNITRACE_USE_ROCTRACER = true
|
||||
OMNITRACE_USE_ROCM_SMI = true
|
||||
OMNITRACE_USE_KOKKOSP = false
|
||||
OMNITRACE_USE_CODE_COVERAGE = false
|
||||
OMNITRACE_USE_PID = true
|
||||
OMNITRACE_OUTPUT_PATH = omnitrace-%tag%-output
|
||||
OMNITRACE_OUTPUT_PREFIX =
|
||||
OMNITRACE_CI = false
|
||||
OMNITRACE_THREAD_POOL_SIZE = 8
|
||||
OMNITRACE_DEBUG = false
|
||||
OMNITRACE_DL_VERBOSE = 0
|
||||
OMNITRACE_INSTRUMENTATION_INTERVAL = 1
|
||||
OMNITRACE_KOKKOSP_KERNEL_LOGGER = false
|
||||
OMNITRACE_PAPI_EVENTS = PAPI_TOT_CYC
|
||||
OMNITRACE_PERFETTO_BACKEND = inprocess
|
||||
OMNITRACE_PERFETTO_BUFFER_SIZE_KB = 1024000
|
||||
OMNITRACE_PERFETTO_COMBINE_TRACES = false
|
||||
OMNITRACE_PERFETTO_FILE = perfetto-trace.proto
|
||||
OMNITRACE_PERFETTO_FILL_POLICY = discard
|
||||
OMNITRACE_PERFETTO_SHMEM_SIZE_HINT_KB = 4096
|
||||
OMNITRACE_ROCTRACER_HSA_ACTIVITY = false
|
||||
OMNITRACE_ROCTRACER_HSA_API = false
|
||||
OMNITRACE_ROCTRACER_HSA_API_TYPES =
|
||||
OMNITRACE_SAMPLING_CPUS =
|
||||
OMNITRACE_SAMPLING_DELAY = 0.5
|
||||
OMNITRACE_SAMPLING_FREQ = 10
|
||||
OMNITRACE_SAMPLING_GPUS = all
|
||||
OMNITRACE_TIME_OUTPUT = true
|
||||
OMNITRACE_TIMEMORY_COMPONENTS = wall_clock
|
||||
OMNITRACE_TRACE_THREAD_LOCKS = false
|
||||
OMNITRACE_VERBOSE = 0
|
||||
OMNITRACE_COLLAPSE_PROCESSES = false
|
||||
OMNITRACE_COLLAPSE_THREADS = false
|
||||
OMNITRACE_COUT_OUTPUT = false
|
||||
OMNITRACE_CPU_AFFINITY = false
|
||||
OMNITRACE_DIFF_OUTPUT = false
|
||||
OMNITRACE_ENABLE_SIGNAL_HANDLER = true
|
||||
OMNITRACE_ENABLED = true
|
||||
OMNITRACE_FILE_OUTPUT = true
|
||||
OMNITRACE_FLAT_PROFILE = false
|
||||
OMNITRACE_INPUT_EXTENSIONS = json,xml
|
||||
OMNITRACE_INPUT_PATH =
|
||||
OMNITRACE_INPUT_PREFIX =
|
||||
OMNITRACE_JSON_OUTPUT = true
|
||||
OMNITRACE_MAX_DEPTH = 65535
|
||||
OMNITRACE_MAX_WIDTH = 120
|
||||
OMNITRACE_MEMORY_PRECISION = -1
|
||||
OMNITRACE_MEMORY_SCIENTIFIC = false
|
||||
OMNITRACE_MEMORY_UNITS = MB
|
||||
OMNITRACE_MEMORY_WIDTH = -1
|
||||
OMNITRACE_NETWORK_INTERFACE =
|
||||
OMNITRACE_NODE_COUNT = 0
|
||||
OMNITRACE_PAPI_FAIL_ON_ERROR = false
|
||||
OMNITRACE_PAPI_MULTIPLEXING = false
|
||||
OMNITRACE_PAPI_OVERFLOW = 0
|
||||
OMNITRACE_PAPI_QUIET = false
|
||||
OMNITRACE_PAPI_THREADING = true
|
||||
OMNITRACE_PRECISION = -1
|
||||
OMNITRACE_SCIENTIFIC = false
|
||||
OMNITRACE_STRICT_CONFIG = true
|
||||
OMNITRACE_SUPPRESS_CONFIG = true
|
||||
OMNITRACE_SUPPRESS_PARSING = true
|
||||
OMNITRACE_TEXT_OUTPUT = true
|
||||
OMNITRACE_TIME_FORMAT = %F_%H.%M
|
||||
OMNITRACE_TIMELINE_PROFILE = false
|
||||
OMNITRACE_TIMING_PRECISION = 6
|
||||
OMNITRACE_TIMING_SCIENTIFIC = false
|
||||
OMNITRACE_TIMING_UNITS = sec
|
||||
OMNITRACE_TIMING_WIDTH = -1
|
||||
OMNITRACE_TREE_OUTPUT = true
|
||||
OMNITRACE_WIDTH = -1
|
||||
ROCPROFSYS_CONFIG_FILE =
|
||||
ROCPROFSYS_MODE = trace
|
||||
ROCPROFSYS_TRACE = true
|
||||
ROCPROFSYS_PROFILE = false
|
||||
ROCPROFSYS_USE_SAMPLING = false
|
||||
ROCPROFSYS_USE_PROCESS_SAMPLING = true
|
||||
ROCPROFSYS_USE_ROCTRACER = true
|
||||
ROCPROFSYS_USE_ROCM_SMI = true
|
||||
ROCPROFSYS_USE_KOKKOSP = false
|
||||
ROCPROFSYS_USE_CODE_COVERAGE = false
|
||||
ROCPROFSYS_USE_PID = true
|
||||
ROCPROFSYS_OUTPUT_PATH = rocprof-sys-%tag%-output
|
||||
ROCPROFSYS_OUTPUT_PREFIX =
|
||||
ROCPROFSYS_CI = false
|
||||
ROCPROFSYS_THREAD_POOL_SIZE = 8
|
||||
ROCPROFSYS_DEBUG = false
|
||||
ROCPROFSYS_DL_VERBOSE = 0
|
||||
ROCPROFSYS_INSTRUMENTATION_INTERVAL = 1
|
||||
ROCPROFSYS_KOKKOSP_KERNEL_LOGGER = false
|
||||
ROCPROFSYS_PAPI_EVENTS = PAPI_TOT_CYC
|
||||
ROCPROFSYS_PERFETTO_BACKEND = inprocess
|
||||
ROCPROFSYS_PERFETTO_BUFFER_SIZE_KB = 1024000
|
||||
ROCPROFSYS_PERFETTO_COMBINE_TRACES = false
|
||||
ROCPROFSYS_PERFETTO_FILE = perfetto-trace.proto
|
||||
ROCPROFSYS_PERFETTO_FILL_POLICY = discard
|
||||
ROCPROFSYS_PERFETTO_SHMEM_SIZE_HINT_KB = 4096
|
||||
ROCPROFSYS_ROCTRACER_HSA_ACTIVITY = false
|
||||
ROCPROFSYS_ROCTRACER_HSA_API = false
|
||||
ROCPROFSYS_ROCTRACER_HSA_API_TYPES =
|
||||
ROCPROFSYS_SAMPLING_CPUS =
|
||||
ROCPROFSYS_SAMPLING_DELAY = 0.5
|
||||
ROCPROFSYS_SAMPLING_FREQ = 10
|
||||
ROCPROFSYS_SAMPLING_GPUS = all
|
||||
ROCPROFSYS_TIME_OUTPUT = true
|
||||
ROCPROFSYS_TIMEMORY_COMPONENTS = wall_clock
|
||||
ROCPROFSYS_TRACE_THREAD_LOCKS = false
|
||||
ROCPROFSYS_VERBOSE = 0
|
||||
ROCPROFSYS_COLLAPSE_PROCESSES = false
|
||||
ROCPROFSYS_COLLAPSE_THREADS = false
|
||||
ROCPROFSYS_COUT_OUTPUT = false
|
||||
ROCPROFSYS_CPU_AFFINITY = false
|
||||
ROCPROFSYS_DIFF_OUTPUT = false
|
||||
ROCPROFSYS_ENABLE_SIGNAL_HANDLER = true
|
||||
ROCPROFSYS_ENABLED = true
|
||||
ROCPROFSYS_FILE_OUTPUT = true
|
||||
ROCPROFSYS_FLAT_PROFILE = false
|
||||
ROCPROFSYS_INPUT_EXTENSIONS = json,xml
|
||||
ROCPROFSYS_INPUT_PATH =
|
||||
ROCPROFSYS_INPUT_PREFIX =
|
||||
ROCPROFSYS_JSON_OUTPUT = true
|
||||
ROCPROFSYS_MAX_DEPTH = 65535
|
||||
ROCPROFSYS_MAX_WIDTH = 120
|
||||
ROCPROFSYS_MEMORY_PRECISION = -1
|
||||
ROCPROFSYS_MEMORY_SCIENTIFIC = false
|
||||
ROCPROFSYS_MEMORY_UNITS = MB
|
||||
ROCPROFSYS_MEMORY_WIDTH = -1
|
||||
ROCPROFSYS_NETWORK_INTERFACE =
|
||||
ROCPROFSYS_NODE_COUNT = 0
|
||||
ROCPROFSYS_PAPI_FAIL_ON_ERROR = false
|
||||
ROCPROFSYS_PAPI_MULTIPLEXING = false
|
||||
ROCPROFSYS_PAPI_OVERFLOW = 0
|
||||
ROCPROFSYS_PAPI_QUIET = false
|
||||
ROCPROFSYS_PAPI_THREADING = true
|
||||
ROCPROFSYS_PRECISION = -1
|
||||
ROCPROFSYS_SCIENTIFIC = false
|
||||
ROCPROFSYS_STRICT_CONFIG = true
|
||||
ROCPROFSYS_SUPPRESS_CONFIG = true
|
||||
ROCPROFSYS_SUPPRESS_PARSING = true
|
||||
ROCPROFSYS_TEXT_OUTPUT = true
|
||||
ROCPROFSYS_TIME_FORMAT = %F_%H.%M
|
||||
ROCPROFSYS_TIMELINE_PROFILE = false
|
||||
ROCPROFSYS_TIMING_PRECISION = 6
|
||||
ROCPROFSYS_TIMING_SCIENTIFIC = false
|
||||
ROCPROFSYS_TIMING_UNITS = sec
|
||||
ROCPROFSYS_TIMING_WIDTH = -1
|
||||
ROCPROFSYS_TREE_OUTPUT = true
|
||||
ROCPROFSYS_WIDTH = -1
|
||||
|
||||
When creating a new configuration file, the following recommendations apply:
|
||||
|
||||
* Use the ``--all`` option to view all descriptions, choices, and other information in the configuration file.
|
||||
* To create a new configuration without inheriting from an existing ``${HOME}/.omnitrace.cfg`` file,
|
||||
set ``OMNITRACE_SUPPRESS_CONFIG=ON`` in the environment beforehand.
|
||||
* To create a new configuration without inheriting from an existing ``${HOME}/.rocprof-sys.cfg`` file,
|
||||
set ``ROCPROFSYS_SUPPRESS_CONFIG=ON`` in the environment beforehand.
|
||||
* To create a new configuration that makes minor changes to an existing configuration,
|
||||
set ``OMNITRACE_CONFIG_FILE=/path/to/existing/file`` and define the changes as environment
|
||||
set ``ROCPROFSYS_CONFIG_FILE=/path/to/existing/file`` and define the changes as environment
|
||||
variables before generating it.
|
||||
|
||||
Viewing the setting descriptions
|
||||
@@ -312,89 +314,89 @@ Viewing the setting descriptions
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
$ omnitrace-avail -S -bd
|
||||
$ rocprof-sys-avail -S -bd
|
||||
|-----------------------------------------|-----------------------------------------|
|
||||
| ENVIRONMENT VARIABLE | DESCRIPTION |
|
||||
|-----------------------------------------|-----------------------------------------|
|
||||
| OMNITRACE_CI | Enable some runtime validation check... |
|
||||
| OMNITRACE_ADD_SECONDARY | Enable/disable components adding sec... |
|
||||
| OMNITRACE_COLLAPSE_PROCESSES | Enable/disable combining process-spe... |
|
||||
| OMNITRACE_COLLAPSE_THREADS | Enable/disable combining thread-spec... |
|
||||
| OMNITRACE_CONFIG_FILE | Configuration file for omnitrace |
|
||||
| OMNITRACE_COUT_OUTPUT | Write output to stdout |
|
||||
| OMNITRACE_CPU_AFFINITY | Enable pinning threads to CPUs (Linu... |
|
||||
| OMNITRACE_THREAD_POOL_SIZE | Number of threads to use when genera... |
|
||||
| OMNITRACE_DEBUG | Enable debug output |
|
||||
| OMNITRACE_DIFF_OUTPUT | Generate a difference output vs. a p... |
|
||||
| OMNITRACE_DL_VERBOSE | Verbosity within the omnitrace-dl li... |
|
||||
| OMNITRACE_ENABLED | Activation state of timemory |
|
||||
| OMNITRACE_ENABLE_SIGNAL_HANDLER | Enable signals in timemory_init |
|
||||
| OMNITRACE_FILE_OUTPUT | Write output to files |
|
||||
| OMNITRACE_FLAT_PROFILE | Set the label hierarchy mode to defa... |
|
||||
| OMNITRACE_INPUT_EXTENSIONS | File extensions used when searching ... |
|
||||
| OMNITRACE_INPUT_PATH | Explicitly specify the input folder ... |
|
||||
| OMNITRACE_INPUT_PREFIX | Explicitly specify the prefix for in... |
|
||||
| OMNITRACE_INSTRUMENTATION_INTERVAL | Instrumentation only takes measureme... |
|
||||
| OMNITRACE_JSON_OUTPUT | Write json output files |
|
||||
| OMNITRACE_KOKKOSP_KERNEL_LOGGER | Enables kernel logging |
|
||||
| OMNITRACE_MAX_DEPTH | Set the maximum depth of label hiera... |
|
||||
| OMNITRACE_MAX_THREAD_BOOKMARKS | Maximum number of times a worker thr... |
|
||||
| OMNITRACE_MAX_WIDTH | Set the maximum width for component ... |
|
||||
| OMNITRACE_MEMORY_PRECISION | Set the precision for components wit... |
|
||||
| OMNITRACE_MEMORY_SCIENTIFIC | Set the numerical reporting format f... |
|
||||
| OMNITRACE_MEMORY_UNITS | Set the units for components with u... |
|
||||
| OMNITRACE_MEMORY_WIDTH | Set the output width for components ... |
|
||||
| OMNITRACE_NETWORK_INTERFACE | Default network interface |
|
||||
| OMNITRACE_NODE_COUNT | Total number of nodes used in applic... |
|
||||
| OMNITRACE_OUTPUT_FILE | Perfetto filename |
|
||||
| OMNITRACE_OUTPUT_PATH | Explicitly specify the output folder... |
|
||||
| OMNITRACE_OUTPUT_PREFIX | Explicitly specify a prefix for all ... |
|
||||
| OMNITRACE_PAPI_EVENTS | PAPI presets and events to collect (... |
|
||||
| OMNITRACE_PAPI_FAIL_ON_ERROR | Configure PAPI errors to trigger a r... |
|
||||
| OMNITRACE_PAPI_MULTIPLEXING | Enable multiplexing when using PAPI |
|
||||
| OMNITRACE_PAPI_OVERFLOW | Value at which PAPI hw counters trig... |
|
||||
| OMNITRACE_PAPI_QUIET | Configure suppression of reporting P... |
|
||||
| OMNITRACE_PAPI_THREADING | Enable multithreading support when u... |
|
||||
| OMNITRACE_PERFETTO_BACKEND | Specify the perfetto backend to acti... |
|
||||
| OMNITRACE_PERFETTO_BUFFER_SIZE_KB | Size of perfetto buffer (in KB) |
|
||||
| OMNITRACE_PERFETTO_COMBINE_TRACES | Combine Perfetto traces. If not expl... |
|
||||
| OMNITRACE_PERFETTO_FILL_POLICY | Behavior when perfetto buffer is ful... |
|
||||
| OMNITRACE_PERFETTO_SHMEM_SIZE_HINT_KB | Hint for shared-memory buffer size i... |
|
||||
| OMNITRACE_PRECISION | Set the global output precision for ... |
|
||||
| OMNITRACE_ROCTRACER_HSA_ACTIVITY | Enable HSA activity tracing support |
|
||||
| OMNITRACE_ROCTRACER_HSA_API | Enable HSA API tracing support |
|
||||
| OMNITRACE_ROCTRACER_HSA_API_TYPES | HSA API type to collect |
|
||||
| OMNITRACE_SAMPLING_CPUS | CPUs to collect frequency informatio... |
|
||||
| OMNITRACE_SAMPLING_DELAY | Number of seconds to wait before the... |
|
||||
| OMNITRACE_SAMPLING_FREQ | Number of software interrupts per se... |
|
||||
| OMNITRACE_SAMPLING_GPUS | Devices to query when OMNITRACE_USE_... |
|
||||
| OMNITRACE_SCIENTIFIC | Set the global numerical reporting t... |
|
||||
| OMNITRACE_STRICT_CONFIG | Throw errors for unknown setting nam... |
|
||||
| OMNITRACE_SUPPRESS_CONFIG | Disable processing of setting config... |
|
||||
| OMNITRACE_SUPPRESS_PARSING | Disable parsing environment |
|
||||
| OMNITRACE_TEXT_OUTPUT | Write text output files |
|
||||
| OMNITRACE_TIMELINE_PROFILE | Set the label hierarchy mode to defa... |
|
||||
| OMNITRACE_TIMEMORY_COMPONENTS | List of components to collect via ti... |
|
||||
| OMNITRACE_TIME_FORMAT | Customize the folder generation when... |
|
||||
| OMNITRACE_TIME_OUTPUT | Output data to subfolder w/ a timest... |
|
||||
| OMNITRACE_TIMING_PRECISION | Set the precision for components wit... |
|
||||
| OMNITRACE_TIMING_SCIENTIFIC | Set the numerical reporting format f... |
|
||||
| OMNITRACE_TIMING_UNITS | Set the units for components with u... |
|
||||
| OMNITRACE_TIMING_WIDTH | Set the output width for components ... |
|
||||
| OMNITRACE_TRACE_THREAD_LOCKS | Enable tracking calls to pthread_mut... |
|
||||
| OMNITRACE_TREE_OUTPUT | Write hierarchical json output files |
|
||||
| OMNITRACE_USE_CODE_COVERAGE | Enable support for code coverage |
|
||||
| OMNITRACE_USE_KOKKOSP | Enable support for Kokkos Tools |
|
||||
| OMNITRACE_USE_OMPT | Enable support for OpenMP-Tools |
|
||||
| OMNITRACE_TRACE | Enable perfetto backend |
|
||||
| OMNITRACE_USE_PID | Enable tagging filenames with proces... |
|
||||
| OMNITRACE_USE_ROCM_SMI | Enable sampling GPU power, temp, uti... |
|
||||
| OMNITRACE_USE_ROCTRACER | Enable ROCM tracing |
|
||||
| OMNITRACE_USE_SAMPLING | Enable statistical sampling of call-... |
|
||||
| OMNITRACE_USE_PROCESS_SAMPLING | Enable a background thread which sam... |
|
||||
| OMNITRACE_PROFILE | Enable timemory backend |
|
||||
| OMNITRACE_VERBOSE | Verbosity level |
|
||||
| OMNITRACE_WIDTH | Set the global output width for comp... |
|
||||
| ROCPROFSYS_CI | Enable some runtime validation check... |
|
||||
| ROCPROFSYS_ADD_SECONDARY | Enable/disable components adding sec... |
|
||||
| ROCPROFSYS_COLLAPSE_PROCESSES | Enable/disable combining process-spe... |
|
||||
| ROCPROFSYS_COLLAPSE_THREADS | Enable/disable combining thread-spec... |
|
||||
| ROCPROFSYS_CONFIG_FILE | Configuration file for rocprof-sys |
|
||||
| ROCPROFSYS_COUT_OUTPUT | Write output to stdout |
|
||||
| ROCPROFSYS_CPU_AFFINITY | Enable pinning threads to CPUs (Linu... |
|
||||
| ROCPROFSYS_THREAD_POOL_SIZE | Number of threads to use when genera... |
|
||||
| ROCPROFSYS_DEBUG | Enable debug output |
|
||||
| ROCPROFSYS_DIFF_OUTPUT | Generate a difference output vs. a p... |
|
||||
| ROCPROFSYS_DL_VERBOSE | Verbosity within the rocprof-sys-dl ... |
|
||||
| ROCPROFSYS_ENABLED | Activation state of timemory |
|
||||
| ROCPROFSYS_ENABLE_SIGNAL_HANDLER | Enable signals in timemory_init |
|
||||
| ROCPROFSYS_FILE_OUTPUT | Write output to files |
|
||||
| ROCPROFSYS_FLAT_PROFILE | Set the label hierarchy mode to defa... |
|
||||
| ROCPROFSYS_INPUT_EXTENSIONS | File extensions used when searching ... |
|
||||
| ROCPROFSYS_INPUT_PATH | Explicitly specify the input folder ... |
|
||||
| ROCPROFSYS_INPUT_PREFIX | Explicitly specify the prefix for in... |
|
||||
| ROCPROFSYS_INSTRUMENTATION_INTERVAL | Instrumentation only takes measureme... |
|
||||
| ROCPROFSYS_JSON_OUTPUT | Write json output files |
|
||||
| ROCPROFSYS_KOKKOSP_KERNEL_LOGGER | Enables kernel logging |
|
||||
| ROCPROFSYS_MAX_DEPTH | Set the maximum depth of label hiera... |
|
||||
| ROCPROFSYS_MAX_THREAD_BOOKMARKS | Maximum number of times a worker thr... |
|
||||
| ROCPROFSYS_MAX_WIDTH | Set the maximum width for component ... |
|
||||
| ROCPROFSYS_MEMORY_PRECISION | Set the precision for components wit... |
|
||||
| ROCPROFSYS_MEMORY_SCIENTIFIC | Set the numerical reporting format f... |
|
||||
| ROCPROFSYS_MEMORY_UNITS | Set the units for components with u... |
|
||||
| ROCPROFSYS_MEMORY_WIDTH | Set the output width for components ... |
|
||||
| ROCPROFSYS_NETWORK_INTERFACE | Default network interface |
|
||||
| ROCPROFSYS_NODE_COUNT | Total number of nodes used in applic... |
|
||||
| ROCPROFSYS_OUTPUT_FILE | Perfetto filename |
|
||||
| ROCPROFSYS_OUTPUT_PATH | Explicitly specify the output folder... |
|
||||
| ROCPROFSYS_OUTPUT_PREFIX | Explicitly specify a prefix for all ... |
|
||||
| ROCPROFSYS_PAPI_EVENTS | PAPI presets and events to collect (... |
|
||||
| ROCPROFSYS_PAPI_FAIL_ON_ERROR | Configure PAPI errors to trigger a r... |
|
||||
| ROCPROFSYS_PAPI_MULTIPLEXING | Enable multiplexing when using PAPI |
|
||||
| ROCPROFSYS_PAPI_OVERFLOW | Value at which PAPI hw counters trig... |
|
||||
| ROCPROFSYS_PAPI_QUIET | Configure suppression of reporting P... |
|
||||
| ROCPROFSYS_PAPI_THREADING | Enable multithreading support when u... |
|
||||
| ROCPROFSYS_PERFETTO_BACKEND | Specify the perfetto backend to acti... |
|
||||
| ROCPROFSYS_PERFETTO_BUFFER_SIZE_KB | Size of perfetto buffer (in KB) |
|
||||
| ROCPROFSYS_PERFETTO_COMBINE_TRACES | Combine Perfetto traces. If not expl... |
|
||||
| ROCPROFSYS_PERFETTO_FILL_POLICY | Behavior when perfetto buffer is ful... |
|
||||
| ROCPROFSYS_PERFETTO_SHMEM_SIZE_HINT_KB | Hint for shared-memory buffer size i... |
|
||||
| ROCPROFSYS_PRECISION | Set the global output precision for ... |
|
||||
| ROCPROFSYS_ROCTRACER_HSA_ACTIVITY | Enable HSA activity tracing support |
|
||||
| ROCPROFSYS_ROCTRACER_HSA_API | Enable HSA API tracing support |
|
||||
| ROCPROFSYS_ROCTRACER_HSA_API_TYPES | HSA API type to collect |
|
||||
| ROCPROFSYS_SAMPLING_CPUS | CPUs to collect frequency informatio... |
|
||||
| ROCPROFSYS_SAMPLING_DELAY | Number of seconds to wait before the... |
|
||||
| ROCPROFSYS_SAMPLING_FREQ | Number of software interrupts per se... |
|
||||
| ROCPROFSYS_SAMPLING_GPUS | Devices to query when ROCPROFSYS_USE_... |
|
||||
| ROCPROFSYS_SCIENTIFIC | Set the global numerical reporting t... |
|
||||
| ROCPROFSYS_STRICT_CONFIG | Throw errors for unknown setting nam... |
|
||||
| ROCPROFSYS_SUPPRESS_CONFIG | Disable processing of setting config... |
|
||||
| ROCPROFSYS_SUPPRESS_PARSING | Disable parsing environment |
|
||||
| ROCPROFSYS_TEXT_OUTPUT | Write text output files |
|
||||
| ROCPROFSYS_TIMELINE_PROFILE | Set the label hierarchy mode to defa... |
|
||||
| ROCPROFSYS_TIMEMORY_COMPONENTS | List of components to collect via ti... |
|
||||
| ROCPROFSYS_TIME_FORMAT | Customize the folder generation when... |
|
||||
| ROCPROFSYS_TIME_OUTPUT | Output data to subfolder w/ a timest... |
|
||||
| ROCPROFSYS_TIMING_PRECISION | Set the precision for components wit... |
|
||||
| ROCPROFSYS_TIMING_SCIENTIFIC | Set the numerical reporting format f... |
|
||||
| ROCPROFSYS_TIMING_UNITS | Set the units for components with u... |
|
||||
| ROCPROFSYS_TIMING_WIDTH | Set the output width for components ... |
|
||||
| ROCPROFSYS_TRACE_THREAD_LOCKS | Enable tracking calls to pthread_mut... |
|
||||
| ROCPROFSYS_TREE_OUTPUT | Write hierarchical json output files |
|
||||
| ROCPROFSYS_USE_CODE_COVERAGE | Enable support for code coverage |
|
||||
| ROCPROFSYS_USE_KOKKOSP | Enable support for Kokkos Tools |
|
||||
| ROCPROFSYS_USE_OMPT | Enable support for OpenMP-Tools |
|
||||
| ROCPROFSYS_TRACE | Enable perfetto backend |
|
||||
| ROCPROFSYS_USE_PID | Enable tagging filenames with proces... |
|
||||
| ROCPROFSYS_USE_ROCM_SMI | Enable sampling GPU power, temp, uti... |
|
||||
| ROCPROFSYS_USE_ROCTRACER | Enable ROCM tracing |
|
||||
| ROCPROFSYS_USE_SAMPLING | Enable statistical sampling of call-... |
|
||||
| ROCPROFSYS_USE_PROCESS_SAMPLING | Enable a background thread which sam... |
|
||||
| ROCPROFSYS_PROFILE | Enable timemory backend |
|
||||
| ROCPROFSYS_VERBOSE | Verbosity level |
|
||||
| ROCPROFSYS_WIDTH | Set the global output width for comp... |
|
||||
|-----------------------------------------|-----------------------------------------|
|
||||
|
||||
Viewing components
|
||||
@@ -402,7 +404,7 @@ Viewing components
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
$ omnitrace-avail -C -bd
|
||||
$ rocprof-sys-avail -C -bd
|
||||
|-----------------------------------|----------------------------------------------|
|
||||
| COMPONENT | DESCRIPTION |
|
||||
|-----------------------------------|----------------------------------------------|
|
||||
@@ -460,7 +462,7 @@ Viewing components
|
||||
| wall_clock | Real-clock timer (i.e. wall-clock timer). |
|
||||
| written_bytes | Number of bytes sent to the storage layer. |
|
||||
| written_char | Number of bytes which this task has cause... |
|
||||
| omnitrace | Invokes instrumentation functions omnitr... |
|
||||
| rocprof-sys | Invokes instrumentation functions rocprof... |
|
||||
| roctracer | High-precision ROCm API and kernel tracing. |
|
||||
| sampling_wall_clock | Wall-clock timing. Derived from statistic... |
|
||||
| sampling_cpu_clock | CPU-clock timing. Derived from statistica... |
|
||||
@@ -476,7 +478,7 @@ Viewing hardware counters
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
$ omnitrace-avail -H -bd
|
||||
$ rocprof-sys-avail -H -bd
|
||||
|---------------------------------------|---------------------------------------|
|
||||
| HARDWARE COUNTER | DESCRIPTION |
|
||||
|---------------------------------------|---------------------------------------|
|
||||
@@ -1197,17 +1199,17 @@ Viewing hardware counters
|
||||
Creating a configuration file
|
||||
========================================
|
||||
|
||||
Omnitrace supports three configuration file formats: JSON, XML, and plain text.
|
||||
Use ``omnitrace-avail -G <filename> -F txt json xml`` to generate default
|
||||
ROCm Systems Profiler supports three configuration file formats: JSON, XML, and plain text.
|
||||
Use ``rocprof-sys-avail -G <filename> -F txt json xml`` to generate default
|
||||
configuration files in each format. Optionally
|
||||
include the ``--all`` flag to include full descriptions and other information.
|
||||
Configuration files are specified by the ``OMNITRACE_CONFIG_FILE`` environment variable
|
||||
which by default looks for ``${HOME}/.omnitrace.cfg`` and ``${HOME}/.omnitrace.json``.
|
||||
Configuration files are specified by the ``ROCPROFSYS_CONFIG_FILE`` environment variable
|
||||
which by default looks for ``${HOME}/.rocprof-sys.cfg`` and ``${HOME}/.rocprof-sys.json``.
|
||||
Multiple configuration files can be concatenated using the ``:`` symbol, for example:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
export OMNITRACE_CONFIG_FILE=~/.config/omnitrace.cfg:~/.config/omnitrace.json
|
||||
export ROCPROFSYS_CONFIG_FILE=~/.config/rocprof-sys.cfg:~/.config/rocprof-sys.json
|
||||
|
||||
If a configuration variable is specified in both a configuration file and in the environment,
|
||||
the environment variable takes precedence.
|
||||
@@ -1220,7 +1222,7 @@ Variables are created when an lvalue starts with a ``$`` and are
|
||||
de-referenced when they appear as rvalues.
|
||||
|
||||
Entries in the text configuration file which do not match a known setting
|
||||
in ``omnitrace-avail`` but are prefixed with ``OMNITRACE_`` are interpreted as
|
||||
in ``rocprof-sys-avail`` but are prefixed with ``ROCPROFSYS_`` are interpreted as
|
||||
environment variables. They are exported via ``setenv``
|
||||
but do not override an existing value for the environment variable.
|
||||
|
||||
@@ -1231,35 +1233,35 @@ but do not override an existing value for the environment variable.
|
||||
$SAMPLE = OFF
|
||||
|
||||
# use fields
|
||||
OMNITRACE_TRACE = $ENABLE
|
||||
OMNITRACE_PROFILE = $ENABLE
|
||||
OMNITRACE_USE_SAMPLING = $SAMPLE
|
||||
OMNITRACE_USE_PROCESS_SAMPLING = $SAMPLE
|
||||
ROCPROFSYS_TRACE = $ENABLE
|
||||
ROCPROFSYS_PROFILE = $ENABLE
|
||||
ROCPROFSYS_USE_SAMPLING = $SAMPLE
|
||||
ROCPROFSYS_USE_PROCESS_SAMPLING = $SAMPLE
|
||||
|
||||
# debug
|
||||
OMNITRACE_DEBUG = OFF
|
||||
OMNITRACE_VERBOSE = 1
|
||||
ROCPROFSYS_DEBUG = OFF
|
||||
ROCPROFSYS_VERBOSE = 1
|
||||
|
||||
# output fields
|
||||
OMNITRACE_OUTPUT_PATH = omnitrace-output
|
||||
OMNITRACE_OUTPUT_PREFIX = %tag%/
|
||||
OMNITRACE_TIME_OUTPUT = OFF
|
||||
OMNITRACE_USE_PID = OFF
|
||||
ROCPROFSYS_OUTPUT_PATH = rocprof-sys-output
|
||||
ROCPROFSYS_OUTPUT_PREFIX = %tag%/
|
||||
ROCPROFSYS_TIME_OUTPUT = OFF
|
||||
ROCPROFSYS_USE_PID = OFF
|
||||
|
||||
# timemory fields
|
||||
OMNITRACE_PAPI_EVENTS = PAPI_TOT_INS PAPI_FP_INS
|
||||
OMNITRACE_TIMEMORY_COMPONENTS = wall_clock peak_rss trip_count
|
||||
OMNITRACE_MEMORY_UNITS = MB
|
||||
OMNITRACE_TIMING_UNITS = sec
|
||||
ROCPROFSYS_PAPI_EVENTS = PAPI_TOT_INS PAPI_FP_INS
|
||||
ROCPROFSYS_TIMEMORY_COMPONENTS = wall_clock peak_rss trip_count
|
||||
ROCPROFSYS_MEMORY_UNITS = MB
|
||||
ROCPROFSYS_TIMING_UNITS = sec
|
||||
|
||||
# sampling fields
|
||||
OMNITRACE_SAMPLING_FREQ = 50
|
||||
OMNITRACE_SAMPLING_DELAY = 0.1
|
||||
OMNITRACE_SAMPLING_CPUS = 0-3
|
||||
OMNITRACE_SAMPLING_GPUS = $env:HIP_VISIBLE_DEVICES
|
||||
ROCPROFSYS_SAMPLING_FREQ = 50
|
||||
ROCPROFSYS_SAMPLING_DELAY = 0.1
|
||||
ROCPROFSYS_SAMPLING_CPUS = 0-3
|
||||
ROCPROFSYS_SAMPLING_GPUS = $env:HIP_VISIBLE_DEVICES
|
||||
|
||||
# misc env variables (see metadata JSON file after run)
|
||||
$env:OMNITRACE_SAMPLING_KEEP_DYNINST_SUFFIX = OFF
|
||||
$env:ROCPROFSYS_SAMPLING_KEEP_DYNINST_SUFFIX = OFF
|
||||
|
||||
Sample JSON configuration file
|
||||
-----------------------------------
|
||||
@@ -1269,9 +1271,9 @@ The full JSON specification for a configuration value contains a lot of informat
|
||||
.. code-block:: json
|
||||
|
||||
{
|
||||
"omnitrace": {
|
||||
"rocprof-sys": {
|
||||
"settings": {
|
||||
"OMNITRACE_ADD_SECONDARY": {
|
||||
"ROCPROFSYS_ADD_SECONDARY": {
|
||||
"count": -1,
|
||||
"name": "add_secondary",
|
||||
"data_type": "bool",
|
||||
@@ -1279,9 +1281,9 @@ The full JSON specification for a configuration value contains a lot of informat
|
||||
"value": true,
|
||||
"max_count": 1,
|
||||
"cmdline": [
|
||||
"--omnitrace-add-secondary"
|
||||
"--rocprof-sys-add-secondary"
|
||||
],
|
||||
"environ": "OMNITRACE_ADD_SECONDARY",
|
||||
"environ": "ROCPROFSYS_ADD_SECONDARY",
|
||||
"cereal_class_version": 1,
|
||||
"categories": [
|
||||
"component",
|
||||
@@ -1294,15 +1296,15 @@ The full JSON specification for a configuration value contains a lot of informat
|
||||
}
|
||||
}
|
||||
|
||||
However when writing an JSON configuration file, the following example is minimally acceptable
|
||||
for ``OMNITRACE_ADD_SECONDARY``:
|
||||
However when writing an JSON configuration file, the following example is minimally acceptable
|
||||
for ``ROCPROFSYS_ADD_SECONDARY``:
|
||||
|
||||
.. code-block:: json
|
||||
|
||||
{
|
||||
"omnitrace": {
|
||||
"rocprof-sys": {
|
||||
"settings": {
|
||||
"OMNITRACE_ADD_SECONDARY": {
|
||||
"ROCPROFSYS_ADD_SECONDARY": {
|
||||
"value": true
|
||||
}
|
||||
}
|
||||
@@ -1318,19 +1320,19 @@ The full XML specification for a configuration value contains the same informati
|
||||
|
||||
<?xml version="1.0" encoding="utf-8"?>
|
||||
<timemory_xml>
|
||||
<omnitrace>
|
||||
<rocprofiler-systems>
|
||||
<settings>
|
||||
<cereal_class_version>2</cereal_class_version>
|
||||
<!-- Full setting specification -->
|
||||
<OMNITRACE_ADD_SECONDARY>
|
||||
<ROCPROFSYS_ADD_SECONDARY>
|
||||
<cereal_class_version>1</cereal_class_version>
|
||||
<name>add_secondary</name>
|
||||
<environ>OMNITRACE_ADD_SECONDARY</environ>
|
||||
<environ>ROCPROFSYS_ADD_SECONDARY</environ>
|
||||
<description>...</description>
|
||||
<count>-1</count>
|
||||
<max_count>1</max_count>
|
||||
<cmdline>
|
||||
<value0>--omnitrace-add-secondary</value0>
|
||||
<value0>--rocprof-sys-add-secondary</value0>
|
||||
</cmdline>
|
||||
<categories>
|
||||
<value0>component</value0>
|
||||
@@ -1340,24 +1342,24 @@ The full XML specification for a configuration value contains the same informati
|
||||
<data_type>bool</data_type>
|
||||
<initial>true</initial>
|
||||
<value>true</value>
|
||||
</OMNITRACE_ADD_SECONDARY>
|
||||
</ROCPROFSYS_ADD_SECONDARY>
|
||||
<!-- etc. -->
|
||||
</settings>
|
||||
</omnitrace>
|
||||
</rocprofiler-systems>
|
||||
</timemory_xml>
|
||||
|
||||
However, when writing an XML configuration file, it is minimally acceptable
|
||||
to set ``OMNITRACE_ADD_SECONDARY=false``:
|
||||
However, when writing an XML configuration file, it is minimally acceptable
|
||||
to set ``ROCPROFSYS_ADD_SECONDARY=false``:
|
||||
|
||||
.. code-block:: xml
|
||||
|
||||
<?xml version="1.0" encoding="utf-8"?>
|
||||
<timemory_xml>
|
||||
<omnitrace>
|
||||
<rocprofiler-systems>
|
||||
<settings>
|
||||
<OMNITRACE_ADD_SECONDARY>
|
||||
<ROCPROFSYS_ADD_SECONDARY>
|
||||
<value>false</value>
|
||||
</OMNITRACE_ADD_SECONDARY>
|
||||
</ROCPROFSYS_ADD_SECONDARY>
|
||||
</settings>
|
||||
</omnitrace>
|
||||
</rocprofiler-systems>
|
||||
</timemory_xml>
|
||||
|
||||
@@ -1,47 +1,47 @@
|
||||
.. meta::
|
||||
:description: Omnitrace documentation and reference
|
||||
:keywords: Omnitrace, ROCm, profiler, tracking, visualization, tool, Instinct, accelerator, AMD
|
||||
:description: ROCm Systems Profiler environment validation documentation and reference
|
||||
:keywords: rocprof-sys, rocprofiler-systems, Omnitrace, ROCm, profiler, environment, tracking, visualization, tool, Instinct, accelerator, AMD
|
||||
|
||||
****************************************************
|
||||
Configuring and validating the environment
|
||||
****************************************************
|
||||
|
||||
After installing `Omnitrace <https://github.com/ROCm/omnitrace>`_, additional steps are required to set up
|
||||
After installing `ROCm Systems Profiler <https://github.com/ROCm/rocprofiler-systems>`_, additional steps are required to set up
|
||||
and validate the environment.
|
||||
|
||||
.. note::
|
||||
|
||||
The following instructions use the installation path ``/opt/omnitrace``. If
|
||||
Omnitrace is installed elsewhere, substitute the actual installation path.
|
||||
The following instructions use the installation path ``/opt/rocprofiler-systems``. If
|
||||
ROCm Systems Profiler is installed elsewhere, substitute the actual installation path.
|
||||
|
||||
Configuring the environment
|
||||
========================================
|
||||
|
||||
After Omnitrace is installed, source the ``setup-env.sh`` script to prefix the
|
||||
After ROCm Systems Profiler is installed, source the ``setup-env.sh`` script to prefix the
|
||||
``PATH``, ``LD_LIBRARY_PATH``, and other environment variables:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
source /opt/omnitrace/share/omnitrace/setup-env.sh
|
||||
source /opt/rocprofiler-systems/share/rocprofiler-systems/setup-env.sh
|
||||
|
||||
Alternatively, if environment modules are supported, add the ``<prefix>/share/modulefiles`` directory
|
||||
to ``MODULEPATH``:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
module use /opt/omnitrace/share/modulefiles
|
||||
module use /opt/rocprofiler-systems/share/modulefiles
|
||||
|
||||
.. note::
|
||||
|
||||
|
||||
As an alternative, the above line can be added to the ``${HOME}/.modulerc`` file.
|
||||
|
||||
After Omnitrace has been added to the ``MODULEPATH``, it can be loaded
|
||||
using ``module load omnitrace/<VERSION>`` and unloaded using ``module unload omnitrace/<VERSION>``.
|
||||
After ROCm Systems Profiler has been added to the ``MODULEPATH``, it can be loaded
|
||||
using ``module load rocprofiler-systems/<VERSION>`` and unloaded using ``module unload rocprofiler-systems/<VERSION>``.
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
module load omnitrace/1.0.0
|
||||
module unload omnitrace/1.0.0
|
||||
module load rocprofiler-systems/1.0.0
|
||||
module unload rocprofiler-systems/1.0.0
|
||||
|
||||
.. note::
|
||||
|
||||
@@ -51,21 +51,21 @@ using ``module load omnitrace/<VERSION>`` and unloaded using ``module unload omn
|
||||
Validating the environment configuration
|
||||
========================================
|
||||
|
||||
If the following commands all run successfully with the expected output,
|
||||
then you are ready to use Omnitrace:
|
||||
If the following commands all run successfully with the expected output,
|
||||
then you are ready to use ROCm Systems Profiler:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
which omnitrace
|
||||
which omnitrace-avail
|
||||
which omnitrace-sample
|
||||
omnitrace-instrument --help
|
||||
omnitrace-avail --all
|
||||
omnitrace-sample --help
|
||||
which rocprof-sys
|
||||
which rocprof-sys-avail
|
||||
which rocprof-sys-sample
|
||||
rocprof-sys-instrument --help
|
||||
rocprof-sys-avail --all
|
||||
rocprof-sys-sample --help
|
||||
|
||||
If Omnitrace was built with Python support, validate these additional commands:
|
||||
If ROCm Systems Profiler was built with Python support, validate these additional commands:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
which omnitrace-python
|
||||
omnitrace-python --help
|
||||
which rocprof-sys-python
|
||||
rocprof-sys-python --help
|
||||
|
||||
@@ -1,19 +1,19 @@
|
||||
.. meta::
|
||||
:description: Omnitrace documentation and reference
|
||||
:keywords: Omnitrace, ROCm, profiler, tracking, visualization, tool, Instinct, accelerator, AMD
|
||||
:description: ROCm Systems Profiler general tips and usage documentation and reference
|
||||
:keywords: rocprof-sys, rocprofiler-systems, Omnitrace, ROCm, tips, how to, profiler, tracking, visualization, tool, Instinct, accelerator, AMD
|
||||
|
||||
**********************************
|
||||
General tips for using Omnitrace
|
||||
General tips for using ROCm Systems Profiler
|
||||
**********************************
|
||||
|
||||
Follow these general guidelines when using Omnitrace. For an explanation of the terms used in this topic, see
|
||||
the :doc:`Omnitrace glossary <../reference/omnitrace-glossary>`.
|
||||
Follow these general guidelines when using ROCm Systems Profiler. For an explanation of the terms used in this topic, see
|
||||
the :doc:`ROCm Systems Profiler glossary <../reference/rocprof-sys-glossary>`.
|
||||
|
||||
* Use ``omnitrace-avail`` to look up configuration settings, hardware counters, and data collection components
|
||||
* Use ``rocprof-sys-avail`` to look up configuration settings, hardware counters, and data collection components
|
||||
|
||||
* Use the ``-d`` flag for descriptions
|
||||
|
||||
* Generate a default configuration with ``omnitrace-avail -G ${HOME}/.omnitrace.cfg`` and adjust it
|
||||
* Generate a default configuration with ``rocprof-sys-avail -G ${HOME}/.rocprof-sys.cfg`` and adjust it
|
||||
to the desired default behavior
|
||||
* **Decide whether binary instrumentation, statistical sampling, or both** provides the desired performance data (for non-Python applications)
|
||||
* Compile code with optimization enabled (``-O2`` or higher), disable asserts (i.e. ``-DNDEBUG``), and include debug info (for instance, ``-g1`` at a minimum)
|
||||
@@ -24,26 +24,26 @@ the :doc:`Omnitrace glossary <../reference/omnitrace-glossary>`.
|
||||
* **Use binary instrumentation for characterizing the performance of every invocation of specific functions**
|
||||
* **Use statistical sampling to characterize the performance of the entire application while minimizing overhead**
|
||||
* Enable statistical sampling after binary instrumentation to help "fill in the gaps" between instrumented regions
|
||||
* Use the user API to create custom regions and enable/disable Omnitrace for specific processes, threads, and regions
|
||||
* Use the user API to create custom regions and enable/disable ROCm Systems Profiler for specific processes, threads, and regions
|
||||
* Dynamic symbol interception, callback APIs, and the user API are always available with binary instrumentation and sampling
|
||||
|
||||
* Dynamic symbol interception and callback APIs are (generally) controlled through ``OMNITRACE_USE_<API>``
|
||||
options, for example, ``OMNITRACE_USE_KOKKOSP`` and ``OMNITRACE_USE_OMPT`` enable Kokkos-Tools and OpenMP-Tools
|
||||
* Dynamic symbol interception and callback APIs are (generally) controlled through ``ROCPROFSYS_USE_<API>``
|
||||
options, for example, ``ROCPROFSYS_USE_KOKKOSP`` and ``ROCPROFSYS_USE_OMPT`` enable Kokkos-Tools and OpenMP-Tools
|
||||
callbacks, respectively
|
||||
|
||||
* When generically seeking regions for performance improvement:
|
||||
|
||||
* **Start off by collecting a flat profile**
|
||||
* Look for functions with high call counts, large cumulative runtimes/values, or large standard deviations
|
||||
|
||||
|
||||
* When call counts are high, improving the performance of this function or "inlining" the function can result in quick and easy performance improvements
|
||||
* When the standard deviation is high, collect a hierarchical profile and see if the high variation can be attributable to the calling context.
|
||||
* When the standard deviation is high, collect a hierarchical profile and see if the high variation can be attributable to the calling context.
|
||||
In this scenario, consider creating a specialized version of the function for the longer-running contexts
|
||||
|
||||
* **Collect a hierarchical profile** and verify the functions that are part of the "critical path" of your
|
||||
* **Collect a hierarchical profile** and verify the functions that are part of the "critical path" of your
|
||||
application, as indicated in the flat profile
|
||||
|
||||
* For example, functions with high call counts but which are part of a "setup" or "post-processing"
|
||||
* For example, functions with high call counts but which are part of a "setup" or "post-processing"
|
||||
phase that does not consume much time relative to the overall time are generally a lower priority for optimization
|
||||
|
||||
* **Use the information from the profiles when analyzing detailed traces**
|
||||
@@ -54,7 +54,7 @@ the :doc:`Omnitrace glossary <../reference/omnitrace-glossary>`.
|
||||
* When using binary instrumentation with MPI, avoid runtime instrumentation
|
||||
|
||||
* Runtime instrumentation requires a fork and a ``ptrace``, which is generally incompatible with how MPI applications spawn processes
|
||||
* Perform a binary rewrite of the executable (and optionally, libraries used by the executable) using MPI and run
|
||||
the generated instrumented executable using ``omnitrace-run`` instead of the original.
|
||||
For example, instead of ``mpirun -n 2 ./myexe``, use ``mpirun -n 2 omnitrace-run -- ./myexe.inst``, where
|
||||
* Perform a binary rewrite of the executable (and optionally, libraries used by the executable) using MPI and run
|
||||
the generated instrumented executable using ``rocprof-sys-run`` instead of the original.
|
||||
For example, instead of ``mpirun -n 2 ./myexe``, use ``mpirun -n 2 rocprof-sys-run -- ./myexe.inst``, where
|
||||
``myexe.inst`` is the instrumented ``myexe`` executable that was generated.
|
||||
@@ -1,12 +1,12 @@
|
||||
.. meta::
|
||||
:description: Omnitrace documentation and reference
|
||||
:keywords: Omnitrace, ROCm, profiler, tracking, visualization, tool, Instinct, accelerator, AMD
|
||||
:description: ROCm Systems Profiler binary instrumentation and rewrite documentation and reference
|
||||
:keywords: rocprof-sys, rocprofiler-systems, Omnitrace, ROCm, binary instrumentation, binary rewrite, profiler, tracking, visualization, tool, Instinct, accelerator, AMD
|
||||
|
||||
****************************************************
|
||||
Instrumenting and rewriting a binary application
|
||||
****************************************************
|
||||
|
||||
There are three ways to perform instrumentation with the ``omnitrace-instrument`` executable:
|
||||
There are three ways to perform instrumentation with the ``rocprof-sys-instrument`` executable:
|
||||
|
||||
* Runtime instrumentation
|
||||
* Attaching to an already running process
|
||||
@@ -14,11 +14,11 @@ There are three ways to perform instrumentation with the ``omnitrace-instrument`
|
||||
|
||||
Here is a comparison of the three modes:
|
||||
|
||||
* Runtime instrumentation of the application using the ``omnitrace-instrument`` executable
|
||||
* Runtime instrumentation of the application using the ``rocprof-sys-instrument`` executable
|
||||
(analogous to ``gdb --args <program> <args>``)
|
||||
|
||||
* This mode is the default if neither the ``-p`` nor ``-o`` command-line options are used
|
||||
* Runtime instrumentation supports instrumenting not only the target executable but also
|
||||
* Runtime instrumentation supports instrumenting not only the target executable but also
|
||||
the shared libraries loaded by the target executable. Consequently, this mode consumes more memory,
|
||||
takes longer to perform the instrumentation, and tends to add more significant overhead to the
|
||||
runtime of the application.
|
||||
@@ -26,7 +26,7 @@ Here is a comparison of the three modes:
|
||||
libraries but also the performance of the library dependencies
|
||||
|
||||
* Attaching to a process that is currently running (analogous to ``gdb -p <PID>``)
|
||||
|
||||
|
||||
* This mode is activated using ``-p <PID>``
|
||||
* The same caveats from the first example apply with respect to memory and overhead
|
||||
|
||||
@@ -39,25 +39,25 @@ Here is a comparison of the three modes:
|
||||
|
||||
* This mode is activated through the ``-o <output-file>`` option
|
||||
* Binary rewriting is limited to the text section of the target executable or library. It does not instrument
|
||||
the dynamically-linked libraries. Consequently, this mode performs the
|
||||
the dynamically-linked libraries. Consequently, this mode performs the
|
||||
instrumentation significantly faster
|
||||
and has a much lower overhead when running the instrumented executable and libraries.
|
||||
* Binary rewriting is the recommended mode when the target executable uses
|
||||
* Binary rewriting is the recommended mode when the target executable uses
|
||||
process-level parallelism (for example, MPI)
|
||||
* If the target executable has a minimal ``main`` routine and the bulk of your
|
||||
* If the target executable has a minimal ``main`` routine and the bulk of your
|
||||
application is in one specific dynamic library,
|
||||
see :ref:`binary-rewriting-library-label` for help
|
||||
|
||||
The omnitrace-instrument executable
|
||||
The rocprof-sys-instrument executable
|
||||
========================================
|
||||
|
||||
Instrumentation is performed with the ``omnitrace-instrument`` executable. For more details, use the ``-h`` or ``--help`` option to
|
||||
Instrumentation is performed with the ``rocprof-sys-instrument`` executable. For more details, use the ``-h`` or ``--help`` option to
|
||||
view the help menu.
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
$ omnitrace-instrument --help
|
||||
[omnitrace-instrument] Usage: omnitrace-instrument [ --help (count: 0, dtype: bool)
|
||||
$ rocprof-sys-instrument --help
|
||||
[rocprof-sys-instrument] Usage: rocprof-sys-instrument [ --help (count: 0, dtype: bool)
|
||||
--version (count: 0, dtype: bool)
|
||||
--verbose (max: 1, dtype: bool)
|
||||
--error (max: 1, dtype: boolean)
|
||||
@@ -161,8 +161,8 @@ view the help menu.
|
||||
[MODE OPTIONS]
|
||||
|
||||
-o, --output Enable generation of a new executable (binary-rewrite). If a filename is not provided,
|
||||
omnitrace will use the basename and output to the cwd, unless the target binary is in the
|
||||
cwd. In the latter case, omnitrace will either use ${PWD}/<basename>.inst (non-libraries)
|
||||
rocprof-sys will use the basename and output to the cwd, unless the target binary is in the
|
||||
cwd. In the latter case, rocprof-sys will either use ${PWD}/<basename>.inst (non-libraries)
|
||||
or ${PWD}/instrumented/<basename> (libraries)
|
||||
-p, --pid Connect to running process
|
||||
-M, --mode [ coverage | sampling | trace ]
|
||||
@@ -177,7 +177,7 @@ view the help menu.
|
||||
[LIBRARY OPTIONS]
|
||||
|
||||
--prefer [ shared | static ] Prefer this library types when available
|
||||
-L, --library Libraries with instrumentation routines (default: "libomnitrace-dl")
|
||||
-L, --library Libraries with instrumentation routines (default: "librocprof-sys-dl")
|
||||
-m, --main-function The primary function to instrument around, e.g. \'main\'
|
||||
--load Supplemental instrumentation library names w/o extension (e.g. \'libinstr\' for
|
||||
\'libinstr.so\' or \'libinstr.a\')
|
||||
@@ -200,17 +200,17 @@ view the help menu.
|
||||
-ME, --module-exclude Regex(es) for excluding modules/files/libraries (always applied)
|
||||
-MR, --module-restrict Regex(es) for restricting modules/files/libraries only to those that match the provided
|
||||
regular-expressions
|
||||
--internal-function-include Regex(es) for including functions which are (likely) utilized by omnitrace itself. Use
|
||||
--internal-function-include Regex(es) for including functions which are (likely) utilized by rocprof-sys itself. Use
|
||||
this option with care.
|
||||
--internal-module-include Regex(es) for including modules/libraries which are (likely) utilized by omnitrace
|
||||
--internal-module-include Regex(es) for including modules/libraries which are (likely) utilized by rocprof-sys
|
||||
itself. Use this option with care.
|
||||
--instruction-exclude Regex(es) for excluding functions containing certain instructions
|
||||
--internal-library-deps Treat the libraries linked to the internal libraries as internal libraries. This increase
|
||||
the internal library processing time and consume more memory (so use with care) but may
|
||||
be useful when the application uses Boost libraries and Dyninst is dynamically linked
|
||||
against the same boost libraries
|
||||
--internal-library-append Append to the list of libraries which omnitrace treats as being used internally, e.g.
|
||||
OmniTrace will find all the symbols in this library and prevent them from being
|
||||
--internal-library-append Append to the list of libraries which rocprof-sys treats as being used internally, e.g.
|
||||
ROCm Systems Profiler will find all the symbols in this library and prevent them from being
|
||||
instrumented.
|
||||
--internal-library-remove [ ld-linux-x86-64.so.2
|
||||
libBrokenLocale.so.1
|
||||
@@ -272,7 +272,7 @@ view the help menu.
|
||||
libz.so
|
||||
libzstd.so ]
|
||||
Remove the specified libraries from being treated as being used internally, e.g.
|
||||
OmniTrace will permit all the symbols in these libraries to be eligible for
|
||||
ROCm System Profiler will permit all the symbols in these libraries to be eligible for
|
||||
instrumentation.
|
||||
--linkage [ global | local | unique | unknown | weak ]
|
||||
Only instrument functions with specified linkage (default: global, local, unique)
|
||||
@@ -287,11 +287,11 @@ view the help menu.
|
||||
options to gain more information about the function signature or location of the
|
||||
functions
|
||||
-C, --config Read in a configuration file and encode these values as the defaults in the executable
|
||||
-d, --default-components Default components to instrument (only useful when timemory is enabled in omnitrace
|
||||
-d, --default-components Default components to instrument (only useful when timemory is enabled in rocprof-sys
|
||||
library)
|
||||
--env Environment variables to add to the runtime in form VARIABLE=VALUE. E.g. use \'--env
|
||||
OMNITRACE_PROFILE=ON\' to default to using timemory instead of perfetto
|
||||
--mpi Enable MPI support (requires omnitrace built w/ full or partial MPI support). NOTE: this
|
||||
ROCPROFSYS_PROFILE=ON\' to default to using timemory instead of perfetto
|
||||
--mpi Enable MPI support (requires rocprof-sys built w/ full or partial MPI support). NOTE: this
|
||||
will automatically be activated if MPI_Init, MPI_Init_thread, MPI_Finalize,
|
||||
MPI_Comm_rank, or MPI_Comm_size are found in the symbol table of target
|
||||
|
||||
@@ -322,8 +322,8 @@ view the help menu.
|
||||
--allow-overlapping Allow dyninst to instrument either multiple functions which overlap (share part of same
|
||||
function body) or single functions with multiple entry points. For more info, see Section
|
||||
2 of the DyninstAPI documentation.
|
||||
--parse-all-modules By default, omnitrace simply requests Dyninst to provide all the procedures in the
|
||||
application image. If this option is enabled, omnitrace will iterate over all the modules
|
||||
--parse-all-modules By default, rocprof-sys simply requests Dyninst to provide all the procedures in the
|
||||
application image. If this option is enabled, rocprof-sys will iterate over all the modules
|
||||
and extract the functions. Theoretically, it should be the same but the data is slightly
|
||||
different, possibly due to weak binding scopes. In general, enabling option will probably
|
||||
have no visible effect
|
||||
@@ -344,17 +344,17 @@ view the help menu.
|
||||
TypeChecking ]
|
||||
Advanced dyninst options: BPatch::set<OPTION>(bool), e.g. bpatch->setTrampRecursive(true)
|
||||
|
||||
``omnitrace-instrument`` uses a similar syntax as LLVM to separate command-line arguments from the
|
||||
application's arguments. It uses a standalone
|
||||
double-hyphen (``--``) as a separator.
|
||||
``rocprof-sys-instrument`` uses a similar syntax as LLVM to separate command-line arguments from the
|
||||
application's arguments. It uses a standalone
|
||||
double-hyphen (``--``) as a separator.
|
||||
All arguments preceding the double-hyphen
|
||||
are interpreted as belonging to Omnitrace and all arguments following the
|
||||
are interpreted as belonging to ROCm Systems Profiler and all arguments following the
|
||||
double-hyphen are interpreted as being part of the
|
||||
application and its arguments. In binary rewrite mode, all application arguments after the first argument
|
||||
are ignored. As an example, ``./omnitrace-instrument -o ls.inst -- ls -l`` interprets ``ls`` as
|
||||
are ignored. As an example, ``./rocprof-sys-instrument -o ls.inst -- ls -l`` interprets ``ls`` as
|
||||
the target to instrument, ignoring the ``-l`` argument,
|
||||
and generates a ``ls.inst`` executable that you can subsequently run using the
|
||||
``omnitrace-run -- ls.inst -l`` command.
|
||||
and generates a ``ls.inst`` executable that you can subsequently run using the
|
||||
``rocprof-sys-run -- ls.inst -l`` command.
|
||||
|
||||
Runtime instrumentation example
|
||||
========================================
|
||||
@@ -363,7 +363,7 @@ The following example shows how to enable runtime instrumentation.
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
omnitrace-instrument <omnitrace-options> -- <exe> [<exe-options>...]
|
||||
rocprof-sys-instrument <rocprof-sys-options> -- <exe> [<exe-options>...]
|
||||
|
||||
Attaching to a running process
|
||||
========================================
|
||||
@@ -372,7 +372,7 @@ Use the following command to attach to an active process.
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
omnitrace-instrument <omnitrace-options> -p <PID> -- <exe-name>
|
||||
rocprof-sys-instrument <rocprof-sys-options> -p <PID> -- <exe-name>
|
||||
|
||||
Binary rewrite
|
||||
========================================
|
||||
@@ -381,24 +381,24 @@ This example demonstrates how to rewrite a binary.
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
omnitrace-instrument <omnitrace-options> -o <name-of-new-exe-or-library> -- <exe-or-library>
|
||||
rocprof-sys-instrument <rocprof-sys-options> -o <name-of-new-exe-or-library> -- <exe-or-library>
|
||||
|
||||
.. _binary-rewriting-library-label:
|
||||
|
||||
Binary rewrite of a library
|
||||
-----------------------------------
|
||||
|
||||
Many applications bundle the bulk of their functionality into one or more
|
||||
Many applications bundle the bulk of their functionality into one or more
|
||||
dynamic libraries and have a relatively simple ``main``
|
||||
which links to these libraries and serves as the "driver" for
|
||||
which links to these libraries and serves as the "driver" for
|
||||
setting up the workflow. If you perform a binary rewrite of an
|
||||
executable like this and find there is insufficient information, you
|
||||
executable like this and find there is insufficient information, you
|
||||
can either switch to runtime instrumentation or perform a
|
||||
binary rewrite on the relevant libraries.
|
||||
|
||||
Support for stand-alone binary rewriting of a dynamic library without a binary rewrite of
|
||||
Support for stand-alone binary rewriting of a dynamic library without a binary rewrite of
|
||||
the executable is a beta feature.
|
||||
In general, it is supported as long as the library contains the ``_init`` and
|
||||
In general, it is supported as long as the library contains the ``_init`` and
|
||||
``_fini`` symbols but these symbols are not
|
||||
standardized to the extent of ``main`` in an executable.
|
||||
|
||||
@@ -406,8 +406,8 @@ Here is the recommended workflow for the binary rewrite of a library:
|
||||
|
||||
#. Determine the names of the dynamically linked libraries of interest using ``ldd``
|
||||
#. Generate a binary rewrite of the executable
|
||||
#. Generate a binary rewrite of the desired libraries with the same base name as the
|
||||
original library, for example, ``libfoo.so.2`` instead of ``libfoo.so``, and output the instrumented
|
||||
#. Generate a binary rewrite of the desired libraries with the same base name as the
|
||||
original library, for example, ``libfoo.so.2`` instead of ``libfoo.so``, and output the instrumented
|
||||
library into a different folder than the original library.
|
||||
|
||||
#. Prefix the ``LD_LIBRARY_PATH`` executable with the output folder from the previous step
|
||||
@@ -433,10 +433,10 @@ Generate binary rewrites of ``foo`` and ``libfoo.so.2``:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
omnitrace-instrument -o ./foo.inst -- foo
|
||||
omnitrace-instrument -o ./libfoo.so.2 -- /usr/local/lib/libfoo.so.2
|
||||
rocprof-sys-instrument -o ./foo.inst -- foo
|
||||
rocprof-sys-instrument -o ./libfoo.so.2 -- /usr/local/lib/libfoo.so.2
|
||||
|
||||
At this point, the instrumented ``foo.inst`` executable still dynamically loads the
|
||||
At this point, the instrumented ``foo.inst`` executable still dynamically loads the
|
||||
original ``libfoo.so.2`` in ``/usr/local/lib``:
|
||||
|
||||
.. code-block:: shell
|
||||
@@ -446,7 +446,7 @@ original ``libfoo.so.2`` in ``/usr/local/lib``:
|
||||
libfoo.so.2 => /usr/local/lib/libfoo.so.2 (...)
|
||||
...
|
||||
|
||||
Prefix the ``LD_LIBRARY_PATH`` environment variable with the folder containing
|
||||
Prefix the ``LD_LIBRARY_PATH`` environment variable with the folder containing
|
||||
the instrumented ``libfoo.so.2``:
|
||||
|
||||
.. code-block:: shell
|
||||
@@ -465,90 +465,90 @@ the instrumented ``libfoo.so.2``:
|
||||
Selective instrumentation
|
||||
========================================
|
||||
|
||||
The default behavior of ``omnitrace-instrument`` does not instrument every symbol in the binary.
|
||||
The default behavior of ``rocprof-sys-instrument`` does not instrument every symbol in the binary.
|
||||
The default rules are:
|
||||
|
||||
* Skip instrumenting dynamic call-sites (such as function pointers)
|
||||
|
||||
* The ``--dynamic-callsites`` option forces instrumentation for all dynamic call-sites
|
||||
|
||||
* The cost of a function can be loosely approximated by the number of
|
||||
instructions. By default, ``omnitrace-instrument`` only instruments functions
|
||||
* The cost of a function can be loosely approximated by the number of
|
||||
instructions. By default, ``rocprof-sys-instrument`` only instruments functions
|
||||
with at least 1024 instructions
|
||||
|
||||
* The ``--min-instructions`` option modifies this heuristic for all functions which do not contain loops
|
||||
* The ``--min-instructions-loop`` option modifies this heuristic for functions which contain loops.
|
||||
|
||||
* The cost of a function can be also be loosely approximated by the size of the function
|
||||
in the binary so this heuristic can be used in lieu of or in addition to the
|
||||
* The cost of a function can be also be loosely approximated by the size of the function
|
||||
in the binary so this heuristic can be used in lieu of or in addition to the
|
||||
minimum number of instructions
|
||||
|
||||
* The ``--min-address-range`` option modifies this heuristic for all functions which do not contain loops
|
||||
* The ``--min-address-range-loop`` option modifies this heuristic for functions which contain loops
|
||||
* The ``--min-address-range-loop`` option modifies this heuristic for functions which contain loops
|
||||
|
||||
* Skip instrumentation points which require using a trap
|
||||
|
||||
|
||||
* See the description for the ``--traps`` and ``--loop-traps`` options for more information
|
||||
|
||||
* Skip instrumenting loops within the body of a function
|
||||
|
||||
* The ``--instrument-loops`` option enables this behavior
|
||||
|
||||
* Skip instrumenting functions with overlapping function bodies and single
|
||||
* Skip instrumenting functions with overlapping function bodies and single
|
||||
functions with multiple entry point
|
||||
|
||||
* These behaviors arise from various optimizations. Enable instrumenting for these functions
|
||||
* These behaviors arise from various optimizations. Enable instrumenting for these functions
|
||||
by using the ``--allow-overlapping`` option
|
||||
|
||||
.. note::
|
||||
|
||||
The separate loop options ``--min-instructions-loop`` and ``--min-address-range-loop``
|
||||
The separate loop options ``--min-instructions-loop`` and ``--min-address-range-loop``
|
||||
are provided because functions with loops can be compact in the binary while also being costly
|
||||
|
||||
Viewing the available, instrumented, excluded, and overlapping functions
|
||||
-------------------------------------------------------------------------
|
||||
|
||||
Whenever ``omnitrace-instrument`` runs with a verbosity of zero or higher,
|
||||
it generates files that detail which functions
|
||||
were available for instrumentation (along with the module they were defined in), actually instrumented,
|
||||
Whenever ``rocprof-sys-instrument`` runs with a verbosity of zero or higher,
|
||||
it generates files that detail which functions
|
||||
were available for instrumentation (along with the module they were defined in), actually instrumented,
|
||||
excluded, and which contained overlapping function bodies.
|
||||
By default, these files are saved to the ``omnitrace-<NAME>-output`` folder
|
||||
By default, these files are saved to the ``rocprof-sys-<NAME>-output`` folder
|
||||
where ``<NAME>`` is the base name of the targeted binary (or
|
||||
the base name of the resulting executable in the case of binary rewrite). For example,
|
||||
``omnitrace-instrument -- ls`` outputs these files to ``omnitrace-ls-output``
|
||||
whereas ``omnitrace-instrument -o ls.inst -- ls`` places them in ``omnitrace-ls.inst-output``.
|
||||
``rocprof-sys-instrument -- ls`` outputs these files to ``rocprof-sys-ls-output``
|
||||
whereas ``rocprof-sys-instrument -o ls.inst -- ls`` places them in ``rocprof-sys-ls.inst-output``.
|
||||
|
||||
To generate these files without running or generating an
|
||||
To generate these files without running or generating an
|
||||
executable, use the ``--simulate`` option:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
omnitrace-instrument --simulate -- foo
|
||||
omnitrace-instrument --simulate -o foo.inst -- foo
|
||||
rocprof-sys-instrument --simulate -- foo
|
||||
rocprof-sys-instrument --simulate -o foo.inst -- foo
|
||||
|
||||
Excluding and including modules and functions
|
||||
----------------------------------------------
|
||||
|
||||
Omnitrace has a set of six command-line options which each accept one or more
|
||||
ROCm Systems Profiler has a set of six command-line options which each accept one or more
|
||||
regular expressions for customizing the scope of which module and/or functions are
|
||||
instrumented. Multiple regex patterns per option are treated as an OR operation,
|
||||
instrumented. Multiple regex patterns per option are treated as an OR operation,
|
||||
for example, ``--module-include libfoo libbar`` is effectively the same as ``--module-include 'libfoo|libbar'``.
|
||||
|
||||
To force the inclusion of certain modules and/or function
|
||||
To force the inclusion of certain modules and/or function
|
||||
without changing any of the heuristics, use the ``--module-include`` and/or ``--function-include`` options.
|
||||
These options do not exclude modules or functions which do
|
||||
These options do not exclude modules or functions which do
|
||||
not satisfy their regular expression.
|
||||
|
||||
To narrow the scope of the instrumentation to a specific set
|
||||
To narrow the scope of the instrumentation to a specific set
|
||||
of libraries and/or functions, use the ``--module-restrict`` and ``--function-restrict`` options.
|
||||
These options let you exclusively select the union of one or more
|
||||
These options let you exclusively select the union of one or more
|
||||
regular expressions, regardless of whether or not the functions satisfy the
|
||||
previously-mentioned default heuristics. Any function or module that is not within
|
||||
previously-mentioned default heuristics. Any function or module that is not within
|
||||
the union of these regular expressions is excluded from instrumentation.
|
||||
|
||||
To avoid instrumenting a set of modules and/or functions,
|
||||
To avoid instrumenting a set of modules and/or functions,
|
||||
use the ``--module-exclude`` and ``--function-exclude`` options.
|
||||
These options are always applied, even if the module or function
|
||||
These options are always applied, even if the module or function
|
||||
satisfies the "restrict" or "include" regular expression.
|
||||
|
||||
.. _available-module-function-output:
|
||||
@@ -558,7 +558,7 @@ An example of the available module and function info output
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
omnitrace-instrument -o lulesh.inst --label file line args --simulate -- lulesh
|
||||
rocprof-sys-instrument -o lulesh.inst --label file line args --simulate -- lulesh
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
@@ -779,7 +779,7 @@ An example of instrumented module and function info output
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
omnitrace-instrument -o lulesh.inst --label file line args --simulate -- lulesh
|
||||
rocprof-sys-instrument -o lulesh.inst --label file line args --simulate -- lulesh
|
||||
|
||||
After the heuristics are applied based on the pattern in :ref:`available-module-function-output`,
|
||||
the selected module and functions are:
|
||||
@@ -850,15 +850,15 @@ Sampling
|
||||
|
||||
This capability has been deprecated in favor of :doc:`Call stack sampling <./sampling-call-stack>`.
|
||||
|
||||
By default, ``omnitrace-instrument`` uses ``--mode trace`` for instrumentation. The ``--mode sampling`` option
|
||||
By default, ``rocprof-sys-instrument`` uses ``--mode trace`` for instrumentation. The ``--mode sampling`` option
|
||||
only instruments ``main`` in an executable. It activates both CPU call-stack sampling and
|
||||
background system-level thread sampling by default.
|
||||
Tracing capabilities which do not rely on instrumentation, such as the HIP API and kernel tracing
|
||||
(which is collected by roctracer), are still available.
|
||||
|
||||
The Omnitrace sampling capabilities are always available, even in trace mode, but are deactivated by default.
|
||||
To activate sampling in trace mode, set ``OMNITRACE_USE_SAMPLING=ON`` in the environment
|
||||
or in an Omnitrace configuration file.
|
||||
The ROCm Systems Profiler sampling capabilities are always available, even in trace mode, but are deactivated by default.
|
||||
To activate sampling in trace mode, set ``ROCPROFSYS_USE_SAMPLING=ON`` in the environment
|
||||
or in an ROCm Systems Profiler configuration file.
|
||||
|
||||
Embedding a default configuration
|
||||
========================================
|
||||
@@ -872,31 +872,31 @@ the configuration settings are not be preserved for subsequent sessions:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
omnitrace-instrument -o ./foo.inst -- ./foo
|
||||
export OMNITRACE_USE_SAMPLING=ON
|
||||
export OMNITRACE_SAMPLING_FREQ=5
|
||||
omnitrace-run -- ./foo.inst
|
||||
rocprof-sys-instrument -o ./foo.inst -- ./foo
|
||||
export ROCPROFSYS_USE_SAMPLING=ON
|
||||
export ROCPROFSYS_SAMPLING_FREQ=5
|
||||
rocprof-sys-run -- ./foo.inst
|
||||
|
||||
Whereas the following command preserves those environment variables:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
omnitrace-instrument -o ./foo.samp --env OMNITRACE_USE_SAMPLING=ON OMNITRACE_SAMPLING_FREQ=5 -- ./foo
|
||||
rocprof-sys-instrument -o ./foo.samp --env ROCPROFSYS_USE_SAMPLING=ON ROCPROFSYS_SAMPLING_FREQ=5 -- ./foo
|
||||
|
||||
They can now be used in future sessions.
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
# will sample 5x per second
|
||||
omnitrace-run -- ./foo.samp
|
||||
rocprof-sys-run -- ./foo.samp
|
||||
|
||||
Even though the environment variables are preserved, subsequent sessions can still override those defaults:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
# will sample 100x per second
|
||||
export OMNITRACE_SAMPLING_FREQ=100
|
||||
omnitrace-run -- ./foo.samp
|
||||
export ROCPROFSYS_SAMPLING_FREQ=100
|
||||
rocprof-sys-run -- ./foo.samp
|
||||
|
||||
.. _rpath-troubleshooting:
|
||||
|
||||
@@ -906,10 +906,10 @@ Troubleshooting
|
||||
Checking for RPATH
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
If ``ldd ./foo.inst`` from the :ref:`binary-rewriting-library-label`
|
||||
section still returns ``/usr/local/lib/libfoo.so.2``, the executable could have
|
||||
If ``ldd ./foo.inst`` from the :ref:`binary-rewriting-library-label`
|
||||
section still returns ``/usr/local/lib/libfoo.so.2``, the executable could have
|
||||
an rpath encoded in the binary.
|
||||
This ELF entry results in the dynamic linker ignoring ``LD_LIBRARY_PATH`` if
|
||||
This ELF entry results in the dynamic linker ignoring ``LD_LIBRARY_PATH`` if
|
||||
it finds ``libfoo.so.2`` in the rpath.
|
||||
Using the ``objdump`` tool, perform the following query:
|
||||
|
||||
@@ -923,13 +923,13 @@ If this produces output that appears similar to this output.:
|
||||
|
||||
RUNPATH $ORIGIN:$ORIGIN/../lib
|
||||
|
||||
Remove or modify the rpath to get ``foo.inst`` to resolve
|
||||
Remove or modify the rpath to get ``foo.inst`` to resolve
|
||||
to the instrumented ``libfoo.so.2`` as explained in the next section.
|
||||
|
||||
Modifying an RPATH
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
This code snippet uses the ``patchelf`` tool to modify the rpath of the given executable
|
||||
This code snippet uses the ``patchelf`` tool to modify the rpath of the given executable
|
||||
or library to ``/home/user``, which is where the instrumented libraries are located.
|
||||
|
||||
.. note::
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
.. meta::
|
||||
:description: Omnitrace documentation and reference
|
||||
:keywords: Omnitrace, ROCm, profiler, tracking, visualization, tool, Instinct, accelerator, AMD
|
||||
:description: ROCm Systems Profiler causal profiling documentation and reference
|
||||
:keywords: rocprof-sys, rocprofiler-systems, Omnitrace, ROCm, causal profiling, profiler, tracking, visualization, tool, Instinct, accelerator, AMD
|
||||
|
||||
****************************************************
|
||||
Performing causal profiling
|
||||
@@ -18,10 +18,6 @@ Thus, causal profiling works by performing experiments on blocks of code during
|
||||
insert pauses to slow down all other concurrently running code. During post-processing, these experiments
|
||||
are translated into calculations for the potential impact of speeding up this block of code.
|
||||
|
||||
.. note::
|
||||
|
||||
Causal profiling supersedes the original critical trace feature, which was removed in Omnitrace v1.11.0.
|
||||
|
||||
Consider the following C++ code executing ``foo`` and ``bar`` concurrently in two different threads
|
||||
where ``foo`` is ideally 30% faster than ``bar``:
|
||||
|
||||
@@ -51,52 +47,52 @@ where ``foo`` is ideally 30% faster than ``bar``:
|
||||
itr.join();
|
||||
}
|
||||
|
||||
No matter how many optimizations are applied to ``foo``, the application will always
|
||||
No matter how many optimizations are applied to ``foo``, the application will always
|
||||
require the same amount of time
|
||||
because the end-to-end performance is limited by ``bar``. However, a 5% speed-up
|
||||
because the end-to-end performance is limited by ``bar``. However, a 5% speed-up
|
||||
in ``bar`` results in the
|
||||
end-to-end performance improving by 5%. This trend continues linearly, with a 10% speed-up
|
||||
end-to-end performance improving by 5%. This trend continues linearly, with a 10% speed-up
|
||||
in ``bar`` yielding a 10% speed-up in
|
||||
end-to-end performance, and so on, up to a 30% speed-up, at which point ``bar`` runs as fast as ``foo``.
|
||||
Any speed-up to ``bar`` beyond 30% still only yields an end-to-end performance
|
||||
Any speed-up to ``bar`` beyond 30% still only yields an end-to-end performance
|
||||
improvement of 30% because the application
|
||||
is now limited by performance of ``foo``, as demonstrated below in the causal
|
||||
is now limited by performance of ``foo``, as demonstrated below in the causal
|
||||
profiling visualization:
|
||||
|
||||
.. image:: ../data/causal-foobar.png
|
||||
:alt: Visualization of the performance improvements for two functions with causal profiling
|
||||
|
||||
The full details of the causal profiling methodology can be found in the paper
|
||||
The full details of the causal profiling methodology can be found in the paper
|
||||
`Coz: Finding Code that Counts with Causal Profiling <http://arxiv.org/pdf/1608.03676v1.pdf>`_.
|
||||
The author's implementation is publicly available on `GitHub <https://github.com/plasma-umass/coz>`_.
|
||||
|
||||
Getting started
|
||||
========================================
|
||||
|
||||
To effectively use causal profiling, it is important to understand a few key
|
||||
To effectively use causal profiling, it is important to understand a few key
|
||||
concepts, such as progress points.
|
||||
|
||||
Progress points
|
||||
-----------------------------------
|
||||
|
||||
Causal profiling requires "progress points" to track progress through the code
|
||||
Causal profiling requires "progress points" to track progress through the code
|
||||
in between samples. Progress points must be triggered in a deterministic manner via instrumentation.
|
||||
This can happen in three different ways:
|
||||
|
||||
* `Omnitrace <https://github.com/ROCm/omnitrace>`_ can leverage the callbacks from
|
||||
Kokkos-Tools, OpenMP-Tools, roctracer, etc. and the wrappers around functions for
|
||||
* `ROCm Systems Profiler <https://github.com/ROCm/rocprofiler-systems>`_ can leverage the callbacks from
|
||||
Kokkos-Tools, OpenMP-Tools, roctracer, etc. and the wrappers around functions for
|
||||
MPI, NUMA, RCCL, etc. to act as progress points
|
||||
* Users can leverage the :doc:`runtime instrumentation capabilities <./instrumenting-rewriting-binary-application>`
|
||||
* Users can leverage the :doc:`runtime instrumentation capabilities <./instrumenting-rewriting-binary-application>`
|
||||
to insert progress points
|
||||
* Users can leverage :doc:`User APIs <../how-to/using-omnitrace-api>`,
|
||||
such as ``OMNITRACE_CAUSAL_PROGRESS``
|
||||
* Users can leverage :doc:`User APIs <../how-to/using-rocprof-sys-api>`,
|
||||
such as ``ROCPROFSYS_CAUSAL_PROGRESS``
|
||||
|
||||
.. note::
|
||||
|
||||
Binary rewrite to insert progress points is not supported. When a rewritten binary
|
||||
runs, Dyninst translates the instruction pointer address in order to perform
|
||||
the instrumentation. As a result, call stack samples never return instruction
|
||||
pointer addresses within the valid Omnitrace range.
|
||||
Binary rewrite to insert progress points is not supported. When a rewritten binary
|
||||
runs, Dyninst translates the instruction pointer address in order to perform
|
||||
the instrumentation. As a result, call stack samples never return instruction
|
||||
pointer addresses within the valid ROCm Systems Profiler range.
|
||||
|
||||
Key concepts
|
||||
-----------------------------------
|
||||
@@ -104,26 +100,26 @@ Key concepts
|
||||
+------------------+-------------------------------------+----------------------------------+--------------------------------------------+
|
||||
| Concept | Setting | Options | Description |
|
||||
+==================+=====================================+==================================+============================================+
|
||||
| Backend | ``OMNITRACE_CAUSAL_BACKEND`` | ``perf``, ``timer`` | Backend for recording samples required |
|
||||
| Backend | ``ROCPROFSYS_CAUSAL_BACKEND`` | ``perf``, ``timer`` | Backend for recording samples required |
|
||||
| | | | to calculate the virtual speed-up |
|
||||
+------------------+-------------------------------------+----------------------------------+--------------------------------------------+
|
||||
| Mode | ``OMNITRACE_CAUSAL_MODE`` | ``function``, ``line`` | Select an entire function or individual |
|
||||
| Mode | ``ROCPROFSYS_CAUSAL_MODE`` | ``function``, ``line`` | Select an entire function or individual |
|
||||
| | | | line of code for causal experiments |
|
||||
+------------------+-------------------------------------+----------------------------------+--------------------------------------------+
|
||||
| End-to-end | ``OMNITRACE_CAUSAL_END_TO_END`` | Boolean | Perform a single experiment during the |
|
||||
| End-to-end | ``ROCPROFSYS_CAUSAL_END_TO_END`` | Boolean | Perform a single experiment during the |
|
||||
| | | | entire run (does not require |
|
||||
| | | | progress points) |
|
||||
+------------------+-------------------------------------+----------------------------------+--------------------------------------------+
|
||||
| Fixed speed-up | ``OMNITRACE_CAUSAL_FIXED_SPEEDUP`` | one or more values from [0, 100] | Virtual speed-up or pool of virtual |
|
||||
| Fixed speed-up | ``ROCPROFSYS_CAUSAL_FIXED_SPEEDUP`` | one or more values from [0, 100] | Virtual speed-up or pool of virtual |
|
||||
| | | | speed-ups to randomly select |
|
||||
+------------------+-------------------------------------+----------------------------------+--------------------------------------------+
|
||||
| Binary scope | ``OMNITRACE_CAUSAL_BINARY_SCOPE`` | regular expression(s) | Dynamic binaries containing code for |
|
||||
| Binary scope | ``ROCPROFSYS_CAUSAL_BINARY_SCOPE`` | regular expression(s) | Dynamic binaries containing code for |
|
||||
| | | | experiments |
|
||||
+------------------+-------------------------------------+----------------------------------+--------------------------------------------+
|
||||
| Source scope | ``OMNITRACE_CAUSAL_SOURCE_SCOPE`` | regular expression(s) | ``<file>`` and/or ``<file>:<line>`` |
|
||||
| Source scope | ``ROCPROFSYS_CAUSAL_SOURCE_SCOPE`` | regular expression(s) | ``<file>`` and/or ``<file>:<line>`` |
|
||||
| | | | containing code to include in experiments |
|
||||
+------------------+-------------------------------------+----------------------------------+--------------------------------------------+
|
||||
| Function scope | ``OMNITRACE_CAUSAL_FUNCTION_SCOPE`` | regular expression(s) | Restricts experiments to matching |
|
||||
| Function scope | ``ROCPROFSYS_CAUSAL_FUNCTION_SCOPE`` | regular expression(s) | Restricts experiments to matching |
|
||||
| | | | functions (function mode) or lines of |
|
||||
| | | | code within matching functions (line mode) |
|
||||
+------------------+-------------------------------------+----------------------------------+--------------------------------------------+
|
||||
@@ -137,30 +133,30 @@ Key concepts
|
||||
Backends
|
||||
-----------------------------------
|
||||
|
||||
There are two backends to choose from: ``perf`` and ``timer``.
|
||||
They are used to record the samples required to calculate the virtual speedup.
|
||||
There are two backends to choose from: ``perf`` and ``timer``.
|
||||
They are used to record the samples required to calculate the virtual speedup.
|
||||
Both backends interrupt each thread 1000 times per second (of CPU-time) to apply the virtual speed-ups.
|
||||
The difference between each backend is how the samples are recorded.
|
||||
There are three key differences between the two backends:
|
||||
|
||||
* the ``perf`` backend requires Linux Perf and elevated security priviledges
|
||||
* the ``perf`` backend interrupts the application less frequently whereas the ``timer`` backend
|
||||
* the ``perf`` backend interrupts the application less frequently whereas the ``timer`` backend
|
||||
interrupts the application 1000 times per second of realtime
|
||||
* the ``timer`` backend has less accurate call stacks due to instruction pointer skid
|
||||
|
||||
In general, the ``perf`` backend is preferred over the ``timer`` backend when sufficient
|
||||
In general, the ``perf`` backend is preferred over the ``timer`` backend when sufficient
|
||||
security priviledges permit its usage.
|
||||
If ``OMNITRACE_CAUSAL_BACKEND`` is set to ``auto``, Omnitrace falls back
|
||||
If ``ROCPROFSYS_CAUSAL_BACKEND`` is set to ``auto``, ROCm Systems Profiler falls back
|
||||
to using the ``timer`` backend only if
|
||||
the ``perf`` backend fails. If ``OMNITRACE_CAUSAL_BACKEND`` is
|
||||
set to ``perf`` and using this backend fails, Omnitrace aborts.
|
||||
the ``perf`` backend fails. If ``ROCPROFSYS_CAUSAL_BACKEND`` is
|
||||
set to ``perf`` and using this backend fails, ROCm Systems Profiler aborts.
|
||||
|
||||
Instruction pointer skid
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Instruction pointer (IP) skid measures how many instructions run after the event of interest
|
||||
before the program actually stops. The IP skid is calculated by subtracting
|
||||
the location of the IP at the point of interest from the location of the IP
|
||||
the location of the IP at the point of interest from the location of the IP
|
||||
when the kernel finally stops the application.
|
||||
For the ``timer`` backend, this translates to the
|
||||
difference in the IP between when the timer generated a signal and when the
|
||||
@@ -172,9 +168,9 @@ especially in ``line`` mode.
|
||||
Installing Linux Perf
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Linux Perf is built into the kernel and may already be installed
|
||||
Linux Perf is built into the kernel and may already be installed
|
||||
(for instance, it is included in the default kernel for OpenSUSE).
|
||||
The official method of checking whether Linux Perf is installed is
|
||||
The official method of checking whether Linux Perf is installed is
|
||||
checking for the existence of the file
|
||||
``/proc/sys/kernel/perf_event_paranoid``. If the file exists, the kernel has Perf installed.
|
||||
|
||||
@@ -184,12 +180,12 @@ If this file does not exist, as with Debian-based systems like Ubuntu, run the f
|
||||
|
||||
apt-get install linux-tools-common linux-tools-generic linux-tools-$(uname -r)
|
||||
|
||||
and reboot your computer. In order to use the ``perf`` backend, the value
|
||||
and reboot your computer. In order to use the ``perf`` backend, the value
|
||||
of ``/proc/sys/kernel/perf_event_paranoid``
|
||||
should be less than or equal to 2. If the value in this file is greater than 2, you can't
|
||||
should be less than or equal to 2. If the value in this file is greater than 2, you can't
|
||||
use the ``perf`` backend.
|
||||
|
||||
To update the paranoid level temporarily until the system is rebooted, run
|
||||
To update the paranoid level temporarily until the system is rebooted, run
|
||||
one of the following commands
|
||||
as a superuser (where ``PARANOID_LEVEL=<N>`` has a value of ``<N>`` in the range ``[-1, 2]``):
|
||||
|
||||
@@ -206,18 +202,18 @@ or
|
||||
To make the paranoid level persistent after a reboot, add ``kernel.perf_event_paranoid=<N>``
|
||||
(where ``<N>`` is the desired paranoid level) to the ``/etc/sysctl.conf`` file.
|
||||
|
||||
Speed-up prediction variability and the omnitrace-causal executable
|
||||
Speed-up prediction variability and the rocprof-sys-causal executable
|
||||
-----------------------------------------------------------------------
|
||||
|
||||
Causal profiling typically requires running the application several times in
|
||||
order to adequately sample all the code domains, experiment
|
||||
Causal profiling typically requires running the application several times in
|
||||
order to adequately sample all the code domains, experiment
|
||||
with speed-ups and other techniques, and resolve statistical fluctuations.
|
||||
The ``omnitrace-causal`` executable is designed to simplify this procedure:
|
||||
The ``rocprof-sys-causal`` executable is designed to simplify this procedure:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
$ omnitrace-causal --help
|
||||
[omnitrace-causal] Usage: ./bin/omnitrace-causal [ --help (count: 0, dtype: bool)
|
||||
$ rocprof-sys-causal --help
|
||||
[rocprof-sys-causal] Usage: ./bin/rocprof-sys-causal [ --help (count: 0, dtype: bool)
|
||||
--version (count: 0, dtype: bool)
|
||||
--monochrome (max: 1, dtype: bool)
|
||||
--debug (max: 1, dtype: bool)
|
||||
@@ -246,21 +242,21 @@ The ``omnitrace-causal`` executable is designed to simplify this procedure:
|
||||
This executable is designed to streamline that process.
|
||||
For example (assume all commands end with \'-- <exe> <args>\'):
|
||||
|
||||
omnitrace-causal -n 5 -- <exe> # runs <exe> 5x with causal profiling enabled
|
||||
rocprof-sys-causal -n 5 -- <exe> # runs <exe> 5x with causal profiling enabled
|
||||
|
||||
omnitrace-causal -s 0 5,10,15,20 # runs <exe> 2x with virtual speedups:
|
||||
rocprof-sys-causal -s 0 5,10,15,20 # runs <exe> 2x with virtual speedups:
|
||||
# - 0
|
||||
# - randomly selected from 5, 10, 15, and 20
|
||||
|
||||
omnitrace-causal -F func_A func_B func_(A|B) # runs <exe> 3x with the function scope limited to:
|
||||
rocprof-sys-causal -F func_A func_B func_(A|B) # runs <exe> 3x with the function scope limited to:
|
||||
# 1. func_A
|
||||
# 2. func_B
|
||||
# 3. func_A or func_B
|
||||
General tips:
|
||||
- Insert progress points at hotspots in your code or use omnitrace\'s runtime instrumentation
|
||||
- Insert progress points at hotspots in your code or use rocprof-sys\'s runtime instrumentation
|
||||
- Note: binary rewrite will produce a incompatible new binary
|
||||
- Run omnitrace-causal in "function" mode first (does not require debug info)
|
||||
- Run omnitrace-causal in "line" mode when you are targeting one function (requires debug info)
|
||||
- Run rocprof-sys-causal in "function" mode first (does not require debug info)
|
||||
- Run rocprof-sys-causal in "line" mode when you are targeting one function (requires debug info)
|
||||
- Preferably, use predictions from the "function" mode to determine which function to target
|
||||
- Limit the virtual speedups to a smaller pool, e.g., 0,5,10,25,50, to get reliable predictions quicker
|
||||
- Make use of the binary, source, and function scope to limit the functions/lines selected for experiments
|
||||
@@ -280,15 +276,15 @@ The ``omnitrace-causal`` executable is designed to simplify this procedure:
|
||||
[GENERAL OPTIONS]
|
||||
|
||||
-c, --config Base configuration file
|
||||
-l, --launcher When running MPI jobs, omnitrace-causal needs to be *before* the executable which launches the MPI processes (i.e.
|
||||
-l, --launcher When running MPI jobs, rocprof-sys-causal needs to be *before* the executable which launches the MPI processes (i.e.
|
||||
before `mpirun`, `srun`, etc.). Pass the name of the target executable (or a regex for matching to the name of the
|
||||
target) for causal profiling, e.g., `omnitrace-causal -l foo -- mpirun -n 4 foo`. This ensures that the omnitrace
|
||||
target) for causal profiling, e.g., `rocprof-sys-causal -l foo -- mpirun -n 4 foo`. This ensures that the rocprof-sys
|
||||
library is LD_PRELOADed on the proper target
|
||||
-g, --generate-configs Generate config files instead of passing environment variables directly. If no arguments are provided, the config files
|
||||
will be placed in ${PWD}/omnitrace-causal-config folder
|
||||
will be placed in ${PWD}/rocprof-sys-causal-config folder
|
||||
--no-defaults Do not activate default features which are recommended for causal profiling. For example: PID-tagging of output files
|
||||
and timestamped subdirectories are disabled by default. Kokkos tools support is added by default
|
||||
(OMNITRACE_USE_KOKKOSP=ON) because, for Kokkos applications, the Kokkos-Tools callbacks are used for progress points.
|
||||
(ROCPROFSYS_USE_KOKKOSP=ON) because, for Kokkos applications, the Kokkos-Tools callbacks are used for progress points.
|
||||
Activation of OpenMP tools support is similar
|
||||
|
||||
[CAUSAL PROFILING OPTIONS (General)]
|
||||
@@ -335,20 +331,20 @@ Examples
|
||||
|
||||
#!/bin/bash -e
|
||||
|
||||
module load omnitrace
|
||||
module load rocprofiler-systems
|
||||
|
||||
N=20
|
||||
I=3
|
||||
|
||||
# when providing speedups to omnitrace-causal, speedup
|
||||
# when providing speedups to rocprof-sys-causal, speedup
|
||||
# groups are separated by a space so "0,10" results in
|
||||
# one speedup group where omnitrace samples from
|
||||
# one speedup group where rocprof-sys samples from
|
||||
# the speedup set of {0, 10}. Passing "0 10" (without
|
||||
# quotes to omnitrace-causal multiplies the
|
||||
# quotes to rocprof-sys-causal multiplies the
|
||||
# number of runs by 2, where the first half of the
|
||||
# runs instruct omnitrace to only use 0 as the
|
||||
# runs instruct rocprof-sys to only use 0 as the
|
||||
# speedup and the second half of the runs instruct
|
||||
# omnitrace to only use 10 as the speedup.
|
||||
# rocprof-sys to only use 10 as the speedup.
|
||||
SPEEDUPS="0,0,0,10,20,30,40,50,50,75,75,75,90,90,90"
|
||||
# thus, -s ${SPEEDUPS} only multiplies the number
|
||||
# of runs by 1 whereas -S ${SPEEDUPS_E2E} multiplies
|
||||
@@ -370,14 +366,14 @@ Examples
|
||||
#
|
||||
# total executions: 20
|
||||
#
|
||||
omnitrace-causal \
|
||||
rocprof-sys-causal \
|
||||
-n ${N} \
|
||||
-s ${SPEEDUPS} \
|
||||
-m function \
|
||||
-o experiments.func \
|
||||
-S ".*\\.cpp" \
|
||||
-- \
|
||||
./causal-omni-cpu "${@}"
|
||||
./causal-rocprofsys-cpu "${@}"
|
||||
|
||||
|
||||
# 20 iterations in line mode with 1 speedup group
|
||||
@@ -390,14 +386,14 @@ Examples
|
||||
#
|
||||
# total executions: 20
|
||||
#
|
||||
omnitrace-causal \
|
||||
rocprof-sys-causal \
|
||||
-n ${N} \
|
||||
-s ${SPEEDUPS} \
|
||||
-m line \
|
||||
-o experiments.line \
|
||||
-S "causal\\.cpp:(100|110)" \
|
||||
-- \
|
||||
./causal-omni-cpu "${@}"
|
||||
./causal-rocprofsys-cpu "${@}"
|
||||
|
||||
|
||||
# 3 iterations in function mode of 15 singular speedups
|
||||
@@ -411,7 +407,7 @@ Examples
|
||||
#
|
||||
# total executions: 90
|
||||
#
|
||||
omnitrace-causal \
|
||||
rocprof-sys-causal \
|
||||
-n ${I} \
|
||||
-s ${SPEEDUPS_E2E} \
|
||||
-m func \
|
||||
@@ -420,7 +416,7 @@ Examples
|
||||
-F "cpu_slow_func" \
|
||||
"cpu_fast_func" \
|
||||
-- \
|
||||
./causal-omni-cpu "${@}"
|
||||
./causal-rocprofsys-cpu "${@}"
|
||||
|
||||
# 3 iterations in line mode of 15 singular speedups
|
||||
# in end-to-end mode with 2 different source scopes
|
||||
@@ -433,7 +429,7 @@ Examples
|
||||
#
|
||||
# total executions: 90
|
||||
#
|
||||
omnitrace-causal \
|
||||
rocprof-sys-causal \
|
||||
-n ${I} \
|
||||
-s ${SPEEDUPS_E2E} \
|
||||
-m line \
|
||||
@@ -442,7 +438,7 @@ Examples
|
||||
-S "causal\\.cpp:100" \
|
||||
"causal\\.cpp:110" \
|
||||
-- \
|
||||
./causal-omni-cpu "${@}"
|
||||
./causal-rocprofsys-cpu "${@}"
|
||||
|
||||
|
||||
export OMP_NUM_THREADS=8
|
||||
@@ -468,7 +464,7 @@ Examples
|
||||
# existing causal/experiments.func.(coz|json)
|
||||
# file due to "--reset" argument
|
||||
#
|
||||
omnitrace-causal \
|
||||
rocprof-sys-causal \
|
||||
--reset \
|
||||
-n ${N} \
|
||||
-s ${SPEEDUPS} \
|
||||
@@ -477,7 +473,7 @@ Examples
|
||||
-S "lulesh.*" \
|
||||
-FE "^(Kokkos::|std::enable_if)" \
|
||||
-- \
|
||||
./lulesh-omni -i 50 -s 200 -r 20 -b 5 -c 5 -p
|
||||
./lulesh-rocprofsys -i 50 -s 200 -r 20 -b 5 -c 5 -p
|
||||
|
||||
|
||||
# 5 iterations in line mode of 1 speedup
|
||||
@@ -498,7 +494,7 @@ Examples
|
||||
# existing causal/experiments.line.(coz|json)
|
||||
# file due to "--reset" argument
|
||||
#
|
||||
omnitrace-causal \
|
||||
rocprof-sys-causal \
|
||||
--reset \
|
||||
-n ${N} \
|
||||
-s ${SPEEDUPS} \
|
||||
@@ -507,7 +503,7 @@ Examples
|
||||
-S "lulesh.*" \
|
||||
-FE "^(exec_range|execute);construct_shared_allocation;\\._omp_fn\\." \
|
||||
-- \
|
||||
./lulesh-omni -i 50 -s 200 -r 20 -b 5 -c 5 -p
|
||||
./lulesh-rocprofsys -i 50 -s 200 -r 20 -b 5 -c 5 -p
|
||||
|
||||
|
||||
# 5 iterations in line mode of 1 speedup
|
||||
@@ -528,7 +524,7 @@ Examples
|
||||
# existing causal/experiments.line.(coz|json)
|
||||
# file due to "--reset" argument
|
||||
#
|
||||
omnitrace-causal \
|
||||
rocprof-sys-causal \
|
||||
--reset \
|
||||
-n ${N} \
|
||||
-s ${SPEEDUPS} \
|
||||
@@ -539,30 +535,30 @@ Examples
|
||||
"CalcVolumeForceForElems" \
|
||||
-S "lulesh\\.cc" \
|
||||
-- \
|
||||
./lulesh-omni -i 50 -s 200 -r 20 -b 5 -c 5 -p
|
||||
./lulesh-rocprofsys -i 50 -s 200 -r 20 -b 5 -c 5 -p
|
||||
|
||||
Using omnitrace-causal with other launchers like mpirun
|
||||
Using rocprof-sys-causal with other launchers like mpirun
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
The ``omnitrace-causal`` executable is intended to assist with application replay
|
||||
The ``rocprof-sys-causal`` executable is intended to assist with application replay
|
||||
and is designed to always be at the start of the command line as the primary process.
|
||||
``omnitrace-causal`` typically adds a ``LD_PRELOAD`` of the Omnitrace libraries
|
||||
``rocprof-sys-causal`` typically adds a ``LD_PRELOAD`` of the ROCm Systems Profiler libraries
|
||||
into the environment before launching the command to inject the functionality
|
||||
required to start the causal profiling tooling. However, this is problematic
|
||||
when the target application for causal profiling uses a launcher, in which case
|
||||
it is listed as an argument rather than as the main application. For example,
|
||||
``foo`` is the target application for profiling, but the command to run it is
|
||||
``mpirun -n 2 foo``. Running the command ``omnitrace-causal -- mpirun -n 2 foo``
|
||||
applies the causal profiling to ``mpirun`` instead of ``foo``.
|
||||
required to start the causal profiling tooling. However, this is problematic
|
||||
when the target application for causal profiling uses a launcher, in which case
|
||||
it is listed as an argument rather than as the main application. For example,
|
||||
``foo`` is the target application for profiling, but the command to run it is
|
||||
``mpirun -n 2 foo``. Running the command ``rocprof-sys-causal -- mpirun -n 2 foo``
|
||||
applies the causal profiling to ``mpirun`` instead of ``foo``.
|
||||
|
||||
``omnitrace-causal`` remedies this by providing a command-line option ``-l` / `--launcher``
|
||||
to indicate the target application is using a launcher script/executable. The
|
||||
``rocprof-sys-causal`` remedies this by providing a command-line option ``-l` / `--launcher``
|
||||
to indicate the target application is using a launcher script/executable. The
|
||||
argument to the command-line option is the name of, or regular expression for, the target application
|
||||
on the command line. When ``--launcher`` is used, ``omnitrace-causal`` generates
|
||||
on the command line. When ``--launcher`` is used, ``rocprof-sys-causal`` generates
|
||||
all the replay configurations and runs them but delays adding the ``LD_PRELOAD``. Instead it
|
||||
inserts a call to itself into the command line right before the target
|
||||
inserts a call to itself into the command line right before the target
|
||||
application. This recursive call inherits the configuration from
|
||||
the parent ``omnitrace-causal`` executable, inserts an ``LD_PRELOAD`` into the environment,
|
||||
the parent ``rocprof-sys-causal`` executable, inserts an ``LD_PRELOAD`` into the environment,
|
||||
and calls ``execv`` to replace itself with the new process launched by the target
|
||||
application.
|
||||
|
||||
@@ -570,32 +566,32 @@ In other words, the following command:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
omnitrace-causal -l foo -n 3 -- mpirun -n 2 foo`
|
||||
rocprof-sys-causal -l foo -n 3 -- mpirun -n 2 foo`
|
||||
|
||||
Effectively results in:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
mpirun -n 2 omnitrace-causal -- foo
|
||||
mpirun -n 2 omnitrace-causal -- foo
|
||||
mpirun -n 2 omnitrace-causal -- foo
|
||||
mpirun -n 2 rocprof-sys-causal -- foo
|
||||
mpirun -n 2 rocprof-sys-causal -- foo
|
||||
mpirun -n 2 rocprof-sys-causal -- foo
|
||||
|
||||
Visualizing the causal output
|
||||
-------------------------------------------------------------------------
|
||||
|
||||
Omnitrace generates ``causal/experiments.json`` and ``causal/experiments.coz`` in
|
||||
``${OMNITRACE_OUTPUT_PATH}/${OMNITRACE_OUTPUT_PREFIX}``. Visit
|
||||
ROCm Systems Profiler generates ``causal/experiments.json`` and ``causal/experiments.coz`` in
|
||||
``${ROCPROFSYS_OUTPUT_PATH}/${ROCPROFSYS_OUTPUT_PREFIX}``. Visit
|
||||
`plasma-umass.org/coz <https://plasma-umass.org/coz/>`_ to open the ``*.coz`` file.
|
||||
|
||||
Omnitrace versus Coz
|
||||
ROCm Systems Profiler versus Coz
|
||||
=======================================
|
||||
|
||||
This comparison is intended for readers who are familiar with the
|
||||
This comparison is intended for readers who are familiar with the
|
||||
`Coz profiler <https://github.com/plasma-umass/coz>`_.
|
||||
Omnitrace provides several additional features and utilities for causal profiling:
|
||||
ROCm Systems Profiler provides several additional features and utilities for causal profiling:
|
||||
|
||||
.. csv-table::
|
||||
:header: "Feature", "Coz", "Omnitrace", "Notes"
|
||||
.. csv-table::
|
||||
:header: "Feature", "Coz", "ROCm Systems Profiler", "Notes"
|
||||
:widths: 20, 60, 60, 30
|
||||
|
||||
"Debug info", "requires debug info in DWARF v3 format (``-gdwarf-3``)", "optional, supports any DWARF format version", "See Note #1 below"
|
||||
@@ -608,23 +604,23 @@ Omnitrace provides several additional features and utilities for causal profilin
|
||||
|
||||
.. note::
|
||||
|
||||
#. Omnitrace supports a "function" mode which does not require debug info.
|
||||
#. Omnitrace supports selecting an entire range of instruction pointers for a function instead
|
||||
#. ROCm Systems Profiler supports a "function" mode which does not require debug info.
|
||||
#. ROCm Systems Profiler supports selecting an entire range of instruction pointers for a function instead
|
||||
of an instruction pointer for one line. In large code bases, "function" mode
|
||||
can resolve in fewer iterations. After a target function is identified, you can
|
||||
can resolve in fewer iterations. After a target function is identified, you can
|
||||
switch to line mode and limit the function scope to the target function.
|
||||
#. Omnitrace supports randomly sampling from subsets, e.g. { 0, 0, 5, 10 }
|
||||
#. ROCm Systems Profiler supports randomly sampling from subsets, e.g. { 0, 0, 5, 10 }
|
||||
where 0% is randomly selected 50% of time and 5% and 10% are randomly selected 25% of the time.
|
||||
#. Omnitrace and COZ have the same definition for binary scope, which is the binaries
|
||||
#. ROCm Systems Profiler and COZ have the same definition for binary scope, which is the binaries
|
||||
loaded at runtime (the executable and linked libraries).
|
||||
#. Omnitrace "source scope" supports both ``<file>`` and ``<file>:<line>`` formats
|
||||
#. ROCm Systems Profiler "source scope" supports both ``<file>`` and ``<file>:<line>`` formats
|
||||
in contrast to the COZ "source scope" which requires ``<file>:<line>`` format.
|
||||
#. Omnitrace supports a "function" scope which narrows the function and lines
|
||||
#. ROCm Systems Profiler supports a "function" scope which narrows the function and lines
|
||||
which are eligible for causal experiments to those within the matching functions.
|
||||
#. Omnitrace supports a second filter on scopes for removing binary/source/function
|
||||
#. ROCm Systems Profiler supports a second filter on scopes for removing binary/source/function
|
||||
caught by an inclusive match. For example ``BINARY_SCOPE=.*`` and ``BINARY_EXCLUDE=libmpi.*``
|
||||
initially includes all binaries but exclude regex removes MPI libraries.
|
||||
#. In Omnitrace, the Linux Perf backend is preferred over use libunwind. However,
|
||||
#. In ROCm Systems Profiler, the Linux Perf backend is preferred over use libunwind. However,
|
||||
Linux Perf usage can be restricted for security reasons.
|
||||
Omnitrace falls back to using a second POSIX timer and libunwind if
|
||||
ROCm Systems Profiler falls back to using a second POSIX timer and libunwind if
|
||||
Linux Perf is not available.
|
||||
|
||||
@@ -1,80 +1,80 @@
|
||||
.. meta::
|
||||
:description: Omnitrace documentation and reference
|
||||
:keywords: Omnitrace, ROCm, profiler, tracking, visualization, tool, Instinct, accelerator, AMD
|
||||
:description: ROCm Systems Profiler Python profiling documentation and reference
|
||||
:keywords: rocprof-sys, rocprofiler-systems, Omnitrace, ROCm, Python, profiling Python, profiler, tracking, visualization, tool, Instinct, accelerator, AMD
|
||||
|
||||
****************************************************
|
||||
Profiling Python scripts
|
||||
****************************************************
|
||||
|
||||
`Omnitrace <https://github.com/ROCm/omnitrace>`_ supports profiling Python code at the
|
||||
`ROCm Systems Profiler <https://github.com/ROCm/rocprofiler-systems>`_ supports profiling Python code at the
|
||||
source level and the script level.
|
||||
Python support is enabled via the ``OMNITRACE_USE_PYTHON`` and the
|
||||
``OMNITRACE_PYTHON_VERSIONS="<MAJOR>.<MINOR>`` CMake options.
|
||||
Alternatively, to build multiple Python versions, use
|
||||
``OMNITRACE_PYTHON_VERSIONS="<MAJOR>.<MINOR>;[<MAJOR>.<MINOR>]"``,
|
||||
and ``OMNITRACE_PYTHON_ROOT_DIRS="/path/to/version;[/path/to/version]"`` instead of ``OMNITRACE_PYTHON_VERSION``.
|
||||
When building multiple Python versions, the length of the ``OMNITRACE_PYTHON_VERSIONS``
|
||||
and ``OMNITRACE_PYTHON_ROOT_DIRS`` lists must
|
||||
Python support is enabled via the ``ROCPROFSYS_USE_PYTHON`` and the
|
||||
``ROCPROFSYS_PYTHON_VERSIONS="<MAJOR>.<MINOR>`` CMake options.
|
||||
Alternatively, to build multiple Python versions, use
|
||||
``ROCPROFSYS_PYTHON_VERSIONS="<MAJOR>.<MINOR>;[<MAJOR>.<MINOR>]"``,
|
||||
and ``ROCPROFSYS_PYTHON_ROOT_DIRS="/path/to/version;[/path/to/version]"`` instead of ``ROCPROFSYS_PYTHON_VERSION``.
|
||||
When building multiple Python versions, the length of the ``ROCPROFSYS_PYTHON_VERSIONS``
|
||||
and ``ROCPROFSYS_PYTHON_ROOT_DIRS`` lists must
|
||||
be the same size.
|
||||
|
||||
.. note::
|
||||
|
||||
When using Omnitrace with Python programs, the Python interpreter major and minor version (e.g. 3.7)
|
||||
When using ROCm Systems Profiler with Python programs, the Python interpreter major and minor version (e.g. 3.7)
|
||||
must match the interpreter major and minor version
|
||||
used when compiling the Python bindings. When building Omnitrace,
|
||||
the shared object file ``libpyomnitrace.<IMPL>-<VERSION>-<ARCH>-<OS>-<ABI>.so`` is generated
|
||||
where ``IMPL`` is the Python implementation, ``VERSION`` is the major and minor
|
||||
used when compiling the Python bindings. When building ROCm Systems Profiler,
|
||||
the shared object file ``libpyrocprofsys.<IMPL>-<VERSION>-<ARCH>-<OS>-<ABI>.so`` is generated
|
||||
where ``IMPL`` is the Python implementation, ``VERSION`` is the major and minor
|
||||
version, ``ARCH`` is the architecture,
|
||||
``OS`` is the operating system, and ``ABI`` is the application binary interface,
|
||||
for example, ``libpyomnitrace.cpython-38-x86_64-linux-gnu.so``.
|
||||
``OS`` is the operating system, and ``ABI`` is the application binary interface,
|
||||
for example, ``libpyrocprofsys.cpython-38-x86_64-linux-gnu.so``.
|
||||
|
||||
Getting Started
|
||||
========================================
|
||||
|
||||
The Omnitrace Python package is installed in ``lib/pythonX.Y/site-packages/omnitrace``.
|
||||
To ensure the Python interpreter can find the Omnitrace package,
|
||||
The ROCm Systems Profiler Python package is installed in ``lib/pythonX.Y/site-packages/rocprofsys``.
|
||||
To ensure the Python interpreter can find the ROCm Systems Profiler package,
|
||||
add this path to the ``PYTHONPATH`` environment variable, as in the following example:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
export PYTHONPATH=/opt/omnitrace/lib/python3.8/site-packages:${PYTHONPATH}
|
||||
export PYTHONPATH=/opt/rocprofiler-systems/lib/python3.8/site-packages:${PYTHONPATH}
|
||||
|
||||
Both the ``share/omnitrace/setup-env.sh`` script and the module file in
|
||||
``share/modulefiles/omnitrace`` automatically handle the prefixing of the ``PYTHONPATH``
|
||||
Both the ``share/rocprofiler-systems/setup-env.sh`` script and the module file in
|
||||
``share/modulefiles/rocprofiler-systems`` automatically handle the prefixing of the ``PYTHONPATH``
|
||||
environment variable.
|
||||
|
||||
Running Omnitrace on a Python script
|
||||
Running ROCm Systems Profiler on a Python script
|
||||
========================================
|
||||
|
||||
Omnitrace provides an ``omnitrace-python`` helper bash script which
|
||||
ROCm Systems Profiler provides an ``rocprof-sys-python`` helper bash script which
|
||||
ensures ``PYTHONPATH`` is properly set and the correct Python interpreter is used.
|
||||
This means the following commands are effectively equivalent:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
omnitrace-python --help
|
||||
rocprof-sys-python --help
|
||||
|
||||
and
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
export PYTHONPATH=/opt/omnitrace/lib/python3.8/site-packages:${PYTHONPATH}
|
||||
python3.8 -m omnitrace --help
|
||||
export PYTHONPATH=/opt/rocprofiler-systems/lib/python3.8/site-packages:${PYTHONPATH}
|
||||
python3.8 -m rocprofsys --help
|
||||
|
||||
.. note::
|
||||
|
||||
``omnitrace-python`` and ``python -m omnitrace`` use the same command-line syntax
|
||||
as the other ``omnitrace`` executables (``omnitrace-python <OMNITRACE_ARGS> -- <SCRIPT> <SCRIPT_ARGS>``)
|
||||
``rocprof-sys-python`` and ``python -m rocprofsys`` use the same command-line syntax
|
||||
as the other ``rocprof-sys`` executables (``rocprof-sys-python <ROCPROFSYS_ARGS> -- <SCRIPT> <SCRIPT_ARGS>``)
|
||||
and has similar options.
|
||||
|
||||
Command line options
|
||||
-----------------------------------
|
||||
|
||||
Use ``omnitrace-python --help`` to view the available options:
|
||||
Use ``rocprof-sys-python --help`` to view the available options:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
usage: omnitrace [-h] [-v VERBOSITY] [-b] [-c FILE] [-s FILE] [-F [BOOL]] [--label [{args,file,line} [{args,file,line} ...]]] [-I FUNC [FUNC ...]] [-E FUNC [FUNC ...]] [-R FUNC [FUNC ...]] [-MI FILE [FILE ...]] [-ME FILE [FILE ...]] [-MR FILE [FILE ...]] [--trace-c [BOOL]]
|
||||
usage: rocprof-sys [-h] [-v VERBOSITY] [-b] [-c FILE] [-s FILE] [-F [BOOL]] [--label [{args,file,line} [{args,file,line} ...]]] [-I FUNC [FUNC ...]] [-E FUNC [FUNC ...]] [-R FUNC [FUNC ...]] [-MI FILE [FILE ...]] [-ME FILE [FILE ...]] [-MR FILE [FILE ...]] [--trace-c [BOOL]]
|
||||
|
||||
optional arguments:
|
||||
-h, --help show this help message and exit
|
||||
@@ -82,7 +82,7 @@ Use ``omnitrace-python --help`` to view the available options:
|
||||
Logging verbosity
|
||||
-b, --builtin Put 'profile' in the builtins. Use '@profile' to decorate a single function, or 'with profile:' to profile a single section of code.
|
||||
-c FILE, --config FILE
|
||||
OmniTrace configuration file
|
||||
ROCm Systems Profiler configuration file
|
||||
-s FILE, --setup FILE
|
||||
Code to execute before the code to profile
|
||||
-F [BOOL], --full-filepath [BOOL]
|
||||
@@ -103,19 +103,19 @@ Use ``omnitrace-python --help`` to view the available options:
|
||||
Select only entries from these files
|
||||
--trace-c [BOOL] Enable profiling C functions
|
||||
|
||||
usage: python3 -m omnitrace <OMNITRACE_ARGS> -- <SCRIPT> <SCRIPT_ARGS>
|
||||
usage: python3 -m rocprofsys <ROCPROFSYS_ARGS> -- <SCRIPT> <SCRIPT_ARGS>
|
||||
|
||||
.. note::
|
||||
|
||||
The ``--trace-c`` option does not incorporate Omnitrace's dynamic instrumentation support.
|
||||
The ``--trace-c`` option does not incorporate ROCm Systems Profiler's dynamic instrumentation support.
|
||||
It only enables profiling the underlying C function call within the Python interpreter.
|
||||
|
||||
Selective instrumentation
|
||||
-----------------------------------
|
||||
|
||||
Similar to the ``omnitrace-instrument`` executable, command-line options exist for restricting,
|
||||
Similar to the ``rocprof-sys-instrument`` executable, command-line options exist for restricting,
|
||||
including, and excluding certain functions and modules, for example, ``--function-exclude "^__init__$"``.
|
||||
Alternatively, add the ``@profile`` decorator to the primary function of interest
|
||||
Alternatively, add the ``@profile`` decorator to the primary function of interest
|
||||
in your program and use the ``-b`` / ``--builtin`` command-line option to narrow the scope of the
|
||||
instrumentation to this function and its children.
|
||||
|
||||
@@ -145,8 +145,8 @@ Consider the following Python code (``example.py``):
|
||||
if __name__ == "__main__":
|
||||
run(20)
|
||||
|
||||
Running ``omnitrace-python ./example.py`` with ``OMNITRACE_PROFILE=ON`` and
|
||||
``OMNITRACE_TIMEMORY_COMPONENTS=trip_count`` produces the following:
|
||||
Running ``rocprof-sys-python ./example.py`` with ``ROCPROFSYS_PROFILE=ON`` and
|
||||
``ROCPROFSYS_TIMEMORY_COMPONENTS=trip_count`` produces the following:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
@@ -187,7 +187,7 @@ If the ``inefficient`` function is decorated with ``@profile`` as follows:
|
||||
def inefficient(n):
|
||||
# ...
|
||||
|
||||
And then run using the command ``omnitrace-python -b -- ./example.py``, Omnitrace produces this output:
|
||||
And then run using the command ``rocprof-sys-python -b -- ./example.py``, ROCm Systems Profiler produces this output:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
@@ -199,37 +199,37 @@ And then run using the command ``omnitrace-python -b -- ./example.py``, Omnitrac
|
||||
| |0>>> inefficient | 1 | 0 | trip_count | 1 |
|
||||
|-----------------------------------------------------------|
|
||||
|
||||
Omnitrace Python source instrumentation
|
||||
ROCm Systems Profiler Python source instrumentation
|
||||
========================================
|
||||
|
||||
Starting with the unmodified ``example.py`` script above, import the ``omnitrace`` module:
|
||||
Starting with the unmodified ``example.py`` script above, import the ``rocprofsys`` module:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
import sys
|
||||
import omnitrace # import omnitrace
|
||||
import rocprofsys # import rocprofsys
|
||||
|
||||
def fib(n):
|
||||
# ... etc. ...
|
||||
|
||||
Next, add ``@omnitrace.profile()`` to the ``run`` function:
|
||||
Next, add ``@rocprofsys.profile()`` to the ``run`` function:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
@omnitrace.profile()
|
||||
@rocprofsys.profile()
|
||||
def run(n):
|
||||
# ...
|
||||
|
||||
Alternatively, use ``omnitrace.profile()`` as a context-manager around ``run(20)``:
|
||||
Alternatively, use ``rocprofsys.profile()`` as a context-manager around ``run(20)``:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
if __name__ == "__main__":
|
||||
with omnitrace.profile():
|
||||
with rocprofsys.profile():
|
||||
run(20)
|
||||
|
||||
The results for both of the source-level instrumentation modes are identical to the
|
||||
original ``omnitrace-python ./example.py`` results:
|
||||
The results for both of the source-level instrumentation modes are identical to the
|
||||
original ``rocprofsys-python ./example.py`` results:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
@@ -264,14 +264,14 @@ original ``omnitrace-python ./example.py`` results:
|
||||
|
||||
.. note::
|
||||
|
||||
When ``omnitrace-python`` is used without built-ins, the profiling results can be cluttered by the
|
||||
When ``rocprof-sys-python`` is used without built-ins, the profiling results can be cluttered by the
|
||||
numerous functions called when more complex modules are imported, such as ``import numpy``.
|
||||
|
||||
Omnitrace Python source instrumentation configuration
|
||||
ROCm Systems Profiler Python source instrumentation configuration
|
||||
-------------------------------------------------------------
|
||||
|
||||
Within the Python source code, the profiler can be configured by directly
|
||||
modifying the ``omnitrace.profiler.config`` data fields.
|
||||
Within the Python source code, the profiler can be configured by directly
|
||||
modifying the ``rocprof-sys.profiler.config`` data fields.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
@@ -295,8 +295,8 @@ modifying the ``omnitrace.profiler.config`` data fields.
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
from omnitrace.profiler import config
|
||||
from omnitrace import profile
|
||||
from rocprofsys.profiler import config
|
||||
from rocprofsys import profile
|
||||
|
||||
config.include_args = True
|
||||
config.include_filename = False
|
||||
|
||||
@@ -1,77 +1,77 @@
|
||||
.. meta::
|
||||
:description: Omnitrace documentation and reference
|
||||
:keywords: Omnitrace, ROCm, profiler, tracking, visualization, tool, Instinct, accelerator, AMD
|
||||
:description: ROCm Systems Profiler call stack sampling documentation and reference
|
||||
:keywords: rocprofiler-systems,rocprofsys, ROCm, profiler, sampling, call stack, tracking, visualization, tool, Instinct, accelerator, AMD
|
||||
|
||||
****************************************************
|
||||
Sampling the call stack
|
||||
****************************************************
|
||||
|
||||
`Omnitrace <https://github.com/ROCm/omnitrace>`_ can use call-stack sampling
|
||||
on a binary instrumented with either the ``omnitrace`` executable
|
||||
or the ``omnitrace-sample`` executable.
|
||||
`ROCm Systems Profiler <https://github.com/ROCm/rocprofiler-systems>`_ can use call-stack sampling
|
||||
on a binary instrumented with either the ``rocprof-sys`` executable
|
||||
or the ``rocprof-sys-sample`` executable.
|
||||
For example, all of the following commands are effectively equivalent:
|
||||
|
||||
* Binary rewrite with only the instrumentation necessary to start and stop sampling
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
omnitrace-instrument -M sampling -o foo.inst -- foo
|
||||
omnitrace-run -- ./foo.inst
|
||||
rocprof-sys-instrument -M sampling -o foo.inst -- foo
|
||||
rocprof-sys-run -- ./foo.inst
|
||||
|
||||
* Runtime instrumentation with only the instrumentation necessary to start and stop sampling
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
omnitrace-instrument -M sampling -- foo
|
||||
rocprof-sys-instrument -M sampling -- foo
|
||||
|
||||
* No instrumentation required
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
omnitrace-sample -- foo
|
||||
rocprof-sys-sample -- foo
|
||||
|
||||
.. note::
|
||||
|
||||
Set ``OMNITRACE_USE_SAMPLING=ON`` to activate call-stack sampling when executing an instrumented binary.
|
||||
Set ``ROCPROFSYS_USE_SAMPLING=ON`` to activate call-stack sampling when executing an instrumented binary.
|
||||
|
||||
All ``omnitrace-instrument -M sampling`` (subsequently referred to as "instrumented-sampling")
|
||||
All ``rocprof-sys-instrument -M sampling`` (subsequently referred to as "instrumented-sampling")
|
||||
does is wrap the ``main`` of the executable with initialization
|
||||
before ``main`` starts and finalization after ``main`` ends.
|
||||
This can be accomplished without instrumentation through a ``LD_PRELOAD``
|
||||
This can be accomplished without instrumentation through a ``LD_PRELOAD``
|
||||
of a library containing a dynamic symbol wrapper around ``__libc_start_main``.
|
||||
|
||||
The use of ``omnitrace-sample`` is **recommended** over
|
||||
``omnitrace-instrument -M sampling`` when binary instrumentation
|
||||
The use of ``rocprof-sys-sample`` is **recommended** over
|
||||
``rocprof-sys-instrument -M sampling`` when binary instrumentation
|
||||
is not necessary. This is for a number of reasons:
|
||||
|
||||
* ``omnitrace-sample`` provides command-line options for controlling the Omnitrace feature set instead of
|
||||
* ``rocprof-sys-sample`` provides command-line options for controlling the ROCm Systems Profiler feature set instead of
|
||||
requiring configuration files or environment variables
|
||||
* Despite the fact that instrumented-sampling only requires inserting snippets
|
||||
* Despite the fact that instrumented-sampling only requires inserting snippets
|
||||
around one function (``main``), Dyninst
|
||||
does not have a feature for specifying that parsing and processing all the
|
||||
does not have a feature for specifying that parsing and processing all the
|
||||
other symbols in the binary is unnecessary.
|
||||
In the best-case scenario when the target binary is relatively small,
|
||||
In the best-case scenario when the target binary is relatively small,
|
||||
instrumented-sampling has a slightly slower launch time,
|
||||
but in the worst case scenarios it requires a significant amount of time and memory to launch.
|
||||
* ``omnitrace-sample`` is fully compatible with MPI. For example,
|
||||
the command ``mpirun -n 2 omnitrace-sample -- foo`` is valid,
|
||||
whereas ``mpirun -n 2 omnitrace-instrument -M sampling -- foo``
|
||||
* ``rocprof-sys-sample`` is fully compatible with MPI. For example,
|
||||
the command ``mpirun -n 2 rocprof-sys-sample -- foo`` is valid,
|
||||
whereas ``mpirun -n 2 rocprof-sys-instrument -M sampling -- foo``
|
||||
is incompatible with some MPI distributions (particularly OpenMPI). This is because
|
||||
MPI prohibits forking within an MPI rank.
|
||||
|
||||
* When MPI and binary instrumentation are both involved, two steps are required:
|
||||
performing a binary rewrite of the executable and then using the instrumented executable
|
||||
in lieu of the original executable. ``omnitrace-sample`` is therefore much easier to use with MPI.
|
||||
performing a binary rewrite of the executable and then using the instrumented executable
|
||||
in lieu of the original executable. ``rocprof-sys-sample`` is therefore much easier to use with MPI.
|
||||
|
||||
The omnitrace-sample executable
|
||||
The rocprof-sys-sample executable
|
||||
========================================
|
||||
|
||||
View the help menu of ``omnitrace-sample`` with the ``-h`` / ``--help`` option:
|
||||
View the help menu of ``rocprof-sys-sample`` with the ``-h`` / ``--help`` option:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
$ omnitrace-sample --help
|
||||
[omnitrace-sample] Usage: omnitrace-sample [ --help (count: 0, dtype: bool)
|
||||
$ rocprof-sys-sample --help
|
||||
[rocprof-sys-sample] Usage: rocprof-sys-sample [ --help (count: 0, dtype: bool)
|
||||
--version (count: 0, dtype: bool)
|
||||
--monochrome (max: 1, dtype: bool)
|
||||
--debug (max: 1, dtype: bool)
|
||||
@@ -111,47 +111,47 @@ View the help menu of ``omnitrace-sample`` with the ``-h`` / ``--help`` option:
|
||||
--gpu-events (count: unlimited)
|
||||
--inlines (max: 1, dtype: bool)
|
||||
--hsa-interrupt (count: 1, dtype: int)
|
||||
]
|
||||
]
|
||||
Options:
|
||||
-h, -?, --help Shows this page (count: 0, dtype: bool)
|
||||
--version Prints the version and exit (count: 0, dtype: bool)
|
||||
|
||||
[DEBUG OPTIONS]
|
||||
|
||||
--monochrome Disable colorized output (max: 1, dtype: bool)
|
||||
--debug Debug output (max: 1, dtype: bool)
|
||||
-v, --verbose Verbose output (count: 1)
|
||||
|
||||
[GENERAL OPTIONS] These are options which are ubiquitously applied
|
||||
|
||||
-c, --config Configuration file (min: 0, dtype: filepath)
|
||||
-o, --output Output path. Accepts 1-2 parameters corresponding to the output path and the output prefix (min: 1)
|
||||
-T, --trace Generate a detailed trace (perfetto output) (max: 1, dtype: bool)
|
||||
-P, --profile Generate a call-stack-based profile (conflicts with --flat-profile) (max: 1, dtype: bool)
|
||||
-F, --flat-profile Generate a flat profile (conflicts with --profile) (max: 1, dtype: bool)
|
||||
-H, --host Enable sampling host-based metrics for the process. E.g. CPU frequency, memory usage, etc. (max: 1, dtype: bool)
|
||||
-D, --device Enable sampling device-based metrics for the process. E.g. GPU temperature, memory usage, etc. (max: 1, dtype: bool)
|
||||
-w, --wait This option is a combination of '--trace-wait' and '--sampling-wait'. See the descriptions for those two options.
|
||||
(count: 1)
|
||||
-d, --duration This option is a combination of '--trace-duration' and '--sampling-duration'. See the descriptions for those two
|
||||
options. (count: 1)
|
||||
|
||||
[TRACING OPTIONS] Specific options controlling tracing (i.e. deterministic measurements of every event)
|
||||
|
||||
--trace-file Specify the trace output filename. Relative filepath will be with respect to output path and output prefix. (count: 1,
|
||||
dtype: filepath)
|
||||
--trace-buffer-size Size limit for the trace output (in KB) (count: 1, dtype: KB)
|
||||
-h, -?, --help Shows this page (count: 0, dtype: bool)
|
||||
--version Prints the version and exit (count: 0, dtype: bool)
|
||||
|
||||
[DEBUG OPTIONS]
|
||||
|
||||
--monochrome Disable colorized output (max: 1, dtype: bool)
|
||||
--debug Debug output (max: 1, dtype: bool)
|
||||
-v, --verbose Verbose output (count: 1)
|
||||
|
||||
[GENERAL OPTIONS] These are options which are ubiquitously applied
|
||||
|
||||
-c, --config Configuration file (min: 0, dtype: filepath)
|
||||
-o, --output Output path. Accepts 1-2 parameters corresponding to the output path and the output prefix (min: 1)
|
||||
-T, --trace Generate a detailed trace (perfetto output) (max: 1, dtype: bool)
|
||||
-P, --profile Generate a call-stack-based profile (conflicts with --flat-profile) (max: 1, dtype: bool)
|
||||
-F, --flat-profile Generate a flat profile (conflicts with --profile) (max: 1, dtype: bool)
|
||||
-H, --host Enable sampling host-based metrics for the process. E.g. CPU frequency, memory usage, etc. (max: 1, dtype: bool)
|
||||
-D, --device Enable sampling device-based metrics for the process. E.g. GPU temperature, memory usage, etc. (max: 1, dtype: bool)
|
||||
-w, --wait This option is a combination of '--trace-wait' and '--sampling-wait'. See the descriptions for those two options.
|
||||
(count: 1)
|
||||
-d, --duration This option is a combination of '--trace-duration' and '--sampling-duration'. See the descriptions for those two
|
||||
options. (count: 1)
|
||||
|
||||
[TRACING OPTIONS] Specific options controlling tracing (i.e. deterministic measurements of every event)
|
||||
|
||||
--trace-file Specify the trace output filename. Relative filepath will be with respect to output path and output prefix. (count: 1,
|
||||
dtype: filepath)
|
||||
--trace-buffer-size Size limit for the trace output (in KB) (count: 1, dtype: KB)
|
||||
--trace-fill-policy [ discard | ring_buffer ]
|
||||
|
||||
|
||||
Policy for new data when the buffer size limit is reached:
|
||||
- discard : new data is ignored
|
||||
- ring_buffer : new data overwrites oldest data (count: 1)
|
||||
--trace-wait Set the wait time (in seconds) before collecting trace and/or profiling data(in seconds). By default, the duration is
|
||||
in seconds of realtime but that can changed via --trace-clock-id. (count: 1)
|
||||
--trace-duration Set the duration of the trace and/or profile data collection (in seconds). By default, the duration is in seconds of
|
||||
realtime but that can changed via --trace-clock-id. (count: 1)
|
||||
--trace-periods More powerful version of specifying trace delay and/or duration. Format is one or more groups of: <DELAY>:<DURATION>,
|
||||
<DELAY>:<DURATION>:<REPEAT>, and/or <DELAY>:<DURATION>:<REPEAT>:<CLOCK_ID>. (min: 1)
|
||||
--trace-wait Set the wait time (in seconds) before collecting trace and/or profiling data(in seconds). By default, the duration is
|
||||
in seconds of realtime but that can changed via --trace-clock-id. (count: 1)
|
||||
--trace-duration Set the duration of the trace and/or profile data collection (in seconds). By default, the duration is in seconds of
|
||||
realtime but that can changed via --trace-clock-id. (count: 1)
|
||||
--trace-periods More powerful version of specifying trace delay and/or duration. Format is one or more groups of: <DELAY>:<DURATION>,
|
||||
<DELAY>:<DURATION>:<REPEAT>, and/or <DELAY>:<DURATION>:<REPEAT>:<CLOCK_ID>. (min: 1)
|
||||
--trace-clock-id [ 0 (realtime|CLOCK_REALTIME)
|
||||
1 (monotonic|CLOCK_MONOTONIC)
|
||||
2 (cputime|CLOCK_PROCESS_CPUTIME_ID)
|
||||
@@ -159,40 +159,40 @@ View the help menu of ``omnitrace-sample`` with the ``-h`` / ``--help`` option:
|
||||
5 (realtime_coarse|CLOCK_REALTIME_COARSE)
|
||||
6 (monotonic_coarse|CLOCK_MONOTONIC_COARSE)
|
||||
7 (boottime|CLOCK_BOOTTIME) ]
|
||||
Set the default clock ID for for trace delay/duration. Note: "cputime" is the *process* CPU time and might need to be
|
||||
scaled based on the number of threads, i.e. 4 seconds of CPU-time for an application with 4 fully active threads would
|
||||
equate to ~1 second of realtime. If this proves to be difficult to handle in practice, please file a feature request
|
||||
for omnitrace to auto-scale based on the number of threads. (count: 1)
|
||||
|
||||
[PROFILE OPTIONS] Specific options controlling profiling (i.e. deterministic measurements which are aggregated into a summary)
|
||||
|
||||
Set the default clock ID for for trace delay/duration. Note: "cputime" is the *process* CPU time and might need to be
|
||||
scaled based on the number of threads, i.e. 4 seconds of CPU-time for an application with 4 fully active threads would
|
||||
equate to ~1 second of realtime. If this proves to be difficult to handle in practice, please file a feature request
|
||||
for rocprof-sys to auto-scale based on the number of threads. (count: 1)
|
||||
|
||||
[PROFILE OPTIONS] Specific options controlling profiling (i.e. deterministic measurements which are aggregated into a summary)
|
||||
|
||||
--profile-format [ console | json | text ]
|
||||
Data formats for profiling results (min: 1)
|
||||
--profile-diff Generate a diff output b/t the profile collected and an existing profile from another run Accepts 1-2 parameters
|
||||
corresponding to the input path and the input prefix (min: 1)
|
||||
|
||||
Data formats for profiling results (min: 1)
|
||||
--profile-diff Generate a diff output b/t the profile collected and an existing profile from another run Accepts 1-2 parameters
|
||||
corresponding to the input path and the input prefix (min: 1)
|
||||
|
||||
[HOST/DEVICE (PROCESS SAMPLING) OPTIONS]
|
||||
Process sampling is background measurements for resources available to the entire process. These samples are not tied
|
||||
to specific lines/regions of code
|
||||
|
||||
--process-freq Set the default host/device sampling frequency (number of interrupts per second) (count: 1)
|
||||
--process-wait Set the default wait time (i.e. delay) before taking first host/device sample (in seconds of realtime) (count: 1)
|
||||
--process-duration Set the duration of the host/device sampling (in seconds of realtime) (count: 1)
|
||||
--cpus CPU IDs for frequency sampling. Supports integers and/or ranges (count: unlimited, dtype: int or range)
|
||||
--gpus GPU IDs for SMI queries. Supports integers and/or ranges (count: unlimited, dtype: int or range)
|
||||
|
||||
[GENERAL SAMPLING OPTIONS] General options for timer-based sampling per-thread
|
||||
|
||||
-f, --freq Set the default sampling frequency (number of interrupts per second) (count: 1)
|
||||
--sampling-wait Set the default wait time (i.e. delay) before taking first sample (in seconds). This delay time is based on the clock
|
||||
of the sampler, i.e., a delay of 1 second for CPU-clock sampler may not equal 1 second of realtime (count: 1)
|
||||
--sampling-duration Set the duration of the sampling (in seconds of realtime). I.e., it is possible (currently) to set a CPU-clock time
|
||||
delay that exceeds the real-time duration... resulting in zero samples being taken (count: 1)
|
||||
-t, --tids Specify the default thread IDs for sampling, where 0 (zero) is the main thread and each thread created by the target
|
||||
application is assigned an atomically incrementing value. (min: 1)
|
||||
|
||||
[SAMPLING TIMER OPTIONS] These options determine the heuristic for deciding when to take a sample
|
||||
|
||||
Process sampling is background measurements for resources available to the entire process. These samples are not tied
|
||||
to specific lines/regions of code
|
||||
|
||||
--process-freq Set the default host/device sampling frequency (number of interrupts per second) (count: 1)
|
||||
--process-wait Set the default wait time (i.e. delay) before taking first host/device sample (in seconds of realtime) (count: 1)
|
||||
--process-duration Set the duration of the host/device sampling (in seconds of realtime) (count: 1)
|
||||
--cpus CPU IDs for frequency sampling. Supports integers and/or ranges (count: unlimited, dtype: int or range)
|
||||
--gpus GPU IDs for SMI queries. Supports integers and/or ranges (count: unlimited, dtype: int or range)
|
||||
|
||||
[GENERAL SAMPLING OPTIONS] General options for timer-based sampling per-thread
|
||||
|
||||
-f, --freq Set the default sampling frequency (number of interrupts per second) (count: 1)
|
||||
--sampling-wait Set the default wait time (i.e. delay) before taking first sample (in seconds). This delay time is based on the clock
|
||||
of the sampler, i.e., a delay of 1 second for CPU-clock sampler may not equal 1 second of realtime (count: 1)
|
||||
--sampling-duration Set the duration of the sampling (in seconds of realtime). I.e., it is possible (currently) to set a CPU-clock time
|
||||
delay that exceeds the real-time duration... resulting in zero samples being taken (count: 1)
|
||||
-t, --tids Specify the default thread IDs for sampling, where 0 (zero) is the main thread and each thread created by the target
|
||||
application is assigned an atomically incrementing value. (min: 1)
|
||||
|
||||
[SAMPLING TIMER OPTIONS] These options determine the heuristic for deciding when to take a sample
|
||||
|
||||
--cputime Sample based on a CPU-clock timer (default). Accepts zero or more arguments:
|
||||
0. Enables sampling based on CPU-clock timer.
|
||||
1. Interrupts per second. E.g., 100 == sample every 10 milliseconds of CPU-time.
|
||||
@@ -210,22 +210,22 @@ View the help menu of ``omnitrace-sample`` with the ``-h`` / ``--help`` option:
|
||||
When sampling with a real-clock timer, please note that enabling this will cause threads which are typically "idle"
|
||||
to consume more resources since, while idle, the real-clock time increases (and therefore triggers taking samples)
|
||||
whereas the CPU-clock time does not. (min: 0)
|
||||
|
||||
[BACKEND OPTIONS] These options control region information captured w/o sampling or instrumentation
|
||||
|
||||
|
||||
[BACKEND OPTIONS] These options control region information captured w/o sampling or instrumentation
|
||||
|
||||
-I, --include [ all | kokkosp | mpip | mutex-locks | ompt | rcclp | rocm-smi | rocprofiler | roctracer | roctx | rw-locks | spin-locks ]
|
||||
Include data from these backends (count: unlimited)
|
||||
Include data from these backends (count: unlimited)
|
||||
-E, --exclude [ all | kokkosp | mpip | mutex-locks | ompt | rcclp | rocm-smi | rocprofiler | roctracer | roctx | rw-locks | spin-locks ]
|
||||
Exclude data from these backends (count: unlimited)
|
||||
|
||||
[HARDWARE COUNTER OPTIONS] See also: omnitrace-avail -H
|
||||
|
||||
-C, --cpu-events Set the CPU hardware counter events to record (ref: `omnitrace-avail -H -c CPU`) (count: unlimited)
|
||||
-G, --gpu-events Set the GPU hardware counter events to record (ref: `omnitrace-avail -H -c GPU`) (count: unlimited)
|
||||
|
||||
[MISCELLANEOUS OPTIONS]
|
||||
|
||||
-i, --inlines Include inline info in output when available (max: 1, dtype: bool)
|
||||
Exclude data from these backends (count: unlimited)
|
||||
|
||||
[HARDWARE COUNTER OPTIONS] See also: rocprof-sys-avail -H
|
||||
|
||||
-C, --cpu-events Set the CPU hardware counter events to record (ref: `rocprof-sys-avail -H -c CPU`) (count: unlimited)
|
||||
-G, --gpu-events Set the GPU hardware counter events to record (ref: `rocprof-sys-avail -H -c GPU`) (count: unlimited)
|
||||
|
||||
[MISCELLANEOUS OPTIONS]
|
||||
|
||||
-i, --inlines Include inline info in output when available (max: 1, dtype: bool)
|
||||
--hsa-interrupt [ 0 | 1 ] Set the value of the HSA_ENABLE_INTERRUPT environment variable.
|
||||
ROCm version 5.2 and older have a bug which will cause a deadlock if a sample is taken while waiting for the signal
|
||||
that a kernel completed -- which happens when sampling with a real-clock timer. We require this option to be set to
|
||||
@@ -235,147 +235,148 @@ View the help menu of ``omnitrace-sample`` with the ``-h`` / ``--help`` option:
|
||||
0 avoid triggering the bug, potentially at the cost of reduced performance
|
||||
1 do not modify how ROCm is notified about kernel completion (count: 1, dtype: int)
|
||||
|
||||
The general syntax for separating Omnitrace command-line arguments from the
|
||||
following application arguments
|
||||
is consistent with the LLVM style of using a stand-alone double hyphen (``--``).
|
||||
The general syntax for separating ROCm Systems Profiler command-line arguments from the
|
||||
following application arguments
|
||||
is consistent with the LLVM style of using a stand-alone double hyphen (``--``).
|
||||
All arguments preceding the double hyphen
|
||||
are interpreted as belonging to Omnitrace and all arguments following it
|
||||
are interpreted as belonging to ROCm Systems Profiler and all arguments following it
|
||||
are interpreted as the
|
||||
application and its arguments. The double hyphen is only necessary when passing
|
||||
application and its arguments. The double hyphen is only necessary when passing
|
||||
command-line arguments to a target
|
||||
which also uses hyphens. For example, you can run ``omnitrace-sample ls``, but
|
||||
to run ``ls -la``, use ``omnitrace-sample -- ls -la``.
|
||||
which also uses hyphens. For example, you can run ``rocprof-sys-sample ls``, but
|
||||
to run ``ls -la``, use ``rocprof-sys-sample -- ls -la``.
|
||||
|
||||
:doc:`Configuring the Omnitrace runtime options <./configuring-runtime-options>`
|
||||
establishes the precedence of environment variable values over values specified
|
||||
:doc:`Configuring the ROCm Systems Profiler runtime options <./configuring-runtime-options>`
|
||||
establishes the precedence of environment variable values over values specified
|
||||
in the configuration files. This enables
|
||||
you to configure the Omnitrace runtime to your preferred default behavior
|
||||
in a file such as ``~/.omnitrace.cfg`` and then easily override
|
||||
those settings in the command line, for example, ``OMNITRACE_ENABLED=OFF omnitrace-sample -- foo``.
|
||||
Similarly, the command-line arguments passed to ``omnitrace-sample`` take precedence
|
||||
you to configure the ROCm Systems Profiler runtime to your preferred default behavior
|
||||
in a file such as ``~/.rocprof-sys.cfg`` and then easily override
|
||||
those settings in the command line, for example, ``ROCPROFSYS_ENABLED=OFF rocprof-sys-sample -- foo``.
|
||||
Similarly, the command-line arguments passed to ``rocprof-sys-sample`` take precedence
|
||||
over environment variables.
|
||||
|
||||
All of the command-line options above correlate to one or more configuration
|
||||
settings, for example, ``--cpu-events`` correlates to the ``OMNITRACE_PAPI_EVENTS`` configuration variable.
|
||||
``omnitrace-sample`` processes the arguments and outputs a summary of its configuration
|
||||
before running the target application.
|
||||
All of the command-line options above correlate to one or more configuration
|
||||
settings, for example, ``--cpu-events`` correlates to the ``ROCPROFSYS_PAPI_EVENTS`` configuration variable.
|
||||
``rocprof-sys-sample`` processes the arguments and outputs a summary of its configuration
|
||||
before running the target application.
|
||||
|
||||
The following snippets show how ``omnitrace-sample`` runs with various environment updates.
|
||||
The following snippets show how ``rocprof-sys-sample`` runs with various environment updates.
|
||||
|
||||
* This snippet shows the environment updates when ``omnitrace-sample`` is invoked with no arguments:
|
||||
* This snippet shows the environment updates when ``rocprof-sys-sample`` is invoked with no arguments:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
$ omnitrace-sample -- ./parallel-overhead-locks 30 4 100
|
||||
$ rocprof-sys-sample -- ./parallel-overhead-locks 30 4 100
|
||||
|
||||
HSA_TOOLS_LIB=/opt/omnitrace/lib/libomnitrace-dl.so.1.7.1
|
||||
HSA_TOOLS_LIB=/opt/rocprofiler-systems/lib/librocprof-sys-dl.so.1.7.1
|
||||
HSA_TOOLS_REPORT_LOAD_FAILURE=1
|
||||
LD_PRELOAD=/opt/omnitrace/lib/libomnitrace-dl.so.1.7.1
|
||||
OMNITRACE_USE_PROCESS_SAMPLING=false
|
||||
OMNITRACE_USE_SAMPLING=true
|
||||
OMP_TOOL_LIBRARIES=/opt/omnitrace/lib/libomnitrace-dl.so.1.7.1
|
||||
ROCP_TOOL_LIB=/opt/omnitrace/lib/libomnitrace.so.1.7.1
|
||||
LD_PRELOAD=/opt/rocprofiler-systems/lib/librocprof-sys-dl.so.1.7.1
|
||||
ROCPROFSYS_USE_PROCESS_SAMPLING=false
|
||||
ROCPROFSYS_USE_SAMPLING=true
|
||||
OMP_TOOL_LIBRARIES=/opt/rocprofiler-systems/lib/librocprof-sys-dl.so.1.7.1
|
||||
ROCP_TOOL_LIB=/opt/rocprofiler-systems/lib/librocprof-sys.so.1.7.1
|
||||
|
||||
* The next snippet shows the environment updates when ``omnitrace-sample`` enables
|
||||
* The next snippet shows the environment updates when ``rocprof-sys-sample`` enables
|
||||
profiling, tracing, host process-sampling, device process-sampling, and all the available backends:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
$ omnitrace-sample -PTDH -I all -- ./parallel-overhead-locks 30 4 100
|
||||
$ rocprof-sys-sample -PTDH -I all -- ./parallel-overhead-locks 30 4 100
|
||||
|
||||
HSA_TOOLS_LIB=/opt/omnitrace/lib/libomnitrace-dl.so.1.7.1
|
||||
HSA_TOOLS_LIB=/opt/rocprofiler-systems/lib/librocprof-sys-dl.so.1.7.1
|
||||
HSA_TOOLS_REPORT_LOAD_FAILURE=1
|
||||
KOKKOS_PROFILE_LIBRARY=/opt/omnitrace/lib/libomnitrace.so.1.7.1
|
||||
LD_PRELOAD=/opt/omnitrace/lib/libomnitrace-dl.so.1.7.1
|
||||
OMNITRACE_CPU_FREQ_ENABLED=true
|
||||
OMNITRACE_TRACE_THREAD_LOCKS=true
|
||||
OMNITRACE_TRACE_THREAD_RW_LOCKS=true
|
||||
OMNITRACE_TRACE_THREAD_SPIN_LOCKS=true
|
||||
OMNITRACE_USE_KOKKOSP=true
|
||||
OMNITRACE_USE_MPIP=true
|
||||
OMNITRACE_USE_OMPT=true
|
||||
OMNITRACE_TRACE=true
|
||||
OMNITRACE_USE_PROCESS_SAMPLING=true
|
||||
OMNITRACE_USE_RCCLP=true
|
||||
OMNITRACE_USE_ROCM_SMI=true
|
||||
OMNITRACE_USE_ROCPROFILER=true
|
||||
OMNITRACE_USE_ROCTRACER=true
|
||||
OMNITRACE_USE_ROCTX=true
|
||||
OMNITRACE_USE_SAMPLING=true
|
||||
OMNITRACE_PROFILE=true
|
||||
OMP_TOOL_LIBRARIES=/opt/omnitrace/lib/libomnitrace-dl.so.1.7.1
|
||||
ROCP_TOOL_LIB=/opt/omnitrace/lib/libomnitrace.so.1.7.1
|
||||
KOKKOS_PROFILE_LIBRARY=/opt/rocprofiler-systems/lib/librocprof-sys.so.1.7.1
|
||||
LD_PRELOAD=/opt/rocprofiler-systems/lib/librocprof-sys-dl.so.1.7.1
|
||||
ROCPROFSYS_CPU_FREQ_ENABLED=true
|
||||
ROCPROFSYS_TRACE_THREAD_LOCKS=true
|
||||
ROCPROFSYS_TRACE_THREAD_RW_LOCKS=true
|
||||
ROCPROFSYS_TRACE_THREAD_SPIN_LOCKS=true
|
||||
ROCPROFSYS_USE_KOKKOSP=true
|
||||
ROCPROFSYS_USE_MPIP=true
|
||||
ROCPROFSYS_USE_OMPT=true
|
||||
ROCPROFSYS_TRACE=true
|
||||
ROCPROFSYS_USE_PROCESS_SAMPLING=true
|
||||
ROCPROFSYS_USE_RCCLP=true
|
||||
ROCPROFSYS_USE_ROCM_SMI=true
|
||||
ROCPROFSYS_USE_ROCPROFILER=true
|
||||
ROCPROFSYS_USE_ROCTRACER=true
|
||||
ROCPROFSYS_USE_ROCTX=true
|
||||
ROCPROFSYS_USE_SAMPLING=true
|
||||
ROCPROFSYS_PROFILE=true
|
||||
OMP_TOOL_LIBRARIES=/opt/rocprofiler-systems/lib/librocprof-sys-dl.so.1.7.1
|
||||
ROCP_TOOL_LIB=/opt/rocprofiler-systems/lib/librocprof-sys.so.1.7.1
|
||||
...
|
||||
|
||||
* The final snippet shows the environment updates when ``omnitrace-sample`` enables
|
||||
* The final snippet shows the environment updates when ``rocprof-sys-sample`` enables
|
||||
profiling, tracing, host process-sampling, and device process-sampling,
|
||||
sets the output path to ``omnitrace-output`` and the output prefix to ``%tag%``, and disables
|
||||
sets the output path to ``rocprof-sys-output`` and the output prefix to ``%tag%``, and disables
|
||||
all the available backends:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
$ omnitrace-sample -PTDH -E all -o omnitrace-output %tag% -- ./parallel-overhead-locks 30 4 100
|
||||
$ rocprof-sys-sample -PTDH -E all -o rocprof-sys-output %tag% -- ./parallel-overhead-locks 30 4 100
|
||||
|
||||
LD_PRELOAD=/opt/omnitrace/lib/libomnitrace-dl.so.1.7.1
|
||||
OMNITRACE_CPU_FREQ_ENABLED=true
|
||||
OMNITRACE_OUTPUT_PATH=omnitrace-output
|
||||
OMNITRACE_OUTPUT_PREFIX=%tag%
|
||||
OMNITRACE_TRACE_THREAD_LOCKS=false
|
||||
OMNITRACE_TRACE_THREAD_RW_LOCKS=false
|
||||
OMNITRACE_TRACE_THREAD_SPIN_LOCKS=false
|
||||
OMNITRACE_USE_KOKKOSP=false
|
||||
OMNITRACE_USE_MPIP=false
|
||||
OMNITRACE_USE_OMPT=false
|
||||
OMNITRACE_TRACE=true
|
||||
OMNITRACE_USE_PROCESS_SAMPLING=true
|
||||
OMNITRACE_USE_RCCLP=false
|
||||
OMNITRACE_USE_ROCM_SMI=false
|
||||
OMNITRACE_USE_ROCPROFILER=false
|
||||
OMNITRACE_USE_ROCTRACER=false
|
||||
OMNITRACE_USE_ROCTX=false
|
||||
OMNITRACE_USE_SAMPLING=true
|
||||
OMNITRACE_PROFILE=true
|
||||
LD_PRELOAD=/opt/rocprofiler-systems/lib/librocprof-sys-dl.so.1.7.1
|
||||
ROCPROFSYS_CPU_FREQ_ENABLED=true
|
||||
ROCPROFSYS_OUTPUT_PATH=rocprof-sys-output
|
||||
ROCPROFSYS_OUTPUT_PREFIX=%tag%
|
||||
ROCPROFSYS_TRACE_THREAD_LOCKS=false
|
||||
ROCPROFSYS_TRACE_THREAD_RW_LOCKS=false
|
||||
ROCPROFSYS_TRACE_THREAD_SPIN_LOCKS=false
|
||||
ROCPROFSYS_USE_KOKKOSP=false
|
||||
ROCPROFSYS_USE_MPIP=false
|
||||
ROCPROFSYS_USE_OMPT=false
|
||||
ROCPROFSYS_TRACE=true
|
||||
ROCPROFSYS_USE_PROCESS_SAMPLING=true
|
||||
ROCPROFSYS_USE_RCCLP=false
|
||||
ROCPROFSYS_USE_ROCM_SMI=false
|
||||
ROCPROFSYS_USE_ROCPROFILER=false
|
||||
ROCPROFSYS_USE_ROCTRACER=false
|
||||
ROCPROFSYS_USE_ROCTX=false
|
||||
ROCPROFSYS_USE_SAMPLING=true
|
||||
ROCPROFSYS_PROFILE=true
|
||||
...
|
||||
|
||||
An omnitrace-sample example
|
||||
An rocprof-sys-sample example
|
||||
========================================
|
||||
|
||||
Here is the full output from the previous
|
||||
``omnitrace-sample -PTDH -E all -o omnitrace-output %tag% -- ./parallel-overhead-locks 30 4 100`` command:
|
||||
Here is the full output from the previous
|
||||
``rocprof-sys-sample -PTDH -E all -o rocprof-sys-output %tag% -- ./parallel-overhead-locks 30 4 100`` command:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
$ omnitrace-sample -PTDH -E all -o omnitrace-output %tag% -c -- ./parallel-overhead-locks 30 4 100
|
||||
$ rocprof-sys-sample -PTDH -E all -o rocprof-sys-output %tag% -c -- ./parallel-overhead-locks 30 4 100
|
||||
|
||||
LD_PRELOAD=/opt/omnitrace/lib/libomnitrace-dl.so.1.11.3
|
||||
OMNITRACE_CONFIG_FILE=
|
||||
OMNITRACE_CPU_FREQ_ENABLED=true
|
||||
OMNITRACE_OUTPUT_PATH=omnitrace-output
|
||||
OMNITRACE_OUTPUT_PREFIX=%tag%
|
||||
OMNITRACE_PROFILE=true
|
||||
OMNITRACE_TRACE=true
|
||||
OMNITRACE_TRACE_THREAD_LOCKS=false
|
||||
OMNITRACE_TRACE_THREAD_RW_LOCKS=false
|
||||
OMNITRACE_TRACE_THREAD_SPIN_LOCKS=false
|
||||
OMNITRACE_USE_KOKKOSP=false
|
||||
OMNITRACE_USE_MPIP=false
|
||||
OMNITRACE_USE_OMPT=false
|
||||
OMNITRACE_USE_PROCESS_SAMPLING=true
|
||||
OMNITRACE_USE_RCCLP=false
|
||||
OMNITRACE_USE_ROCM_SMI=false
|
||||
OMNITRACE_USE_ROCPROFILER=false
|
||||
OMNITRACE_USE_ROCTRACER=false
|
||||
OMNITRACE_USE_ROCTX=false
|
||||
OMNITRACE_USE_SAMPLING=true
|
||||
[omnitrace][dl][1785877] omnitrace_main
|
||||
[omnitrace][1785877][omnitrace_init_tooling] Instrumentation mode: Sampling
|
||||
______ .___ ___. .__ __. __ .___________..______ ___ ______ _______
|
||||
/ __ \ | \/ | | \ | | | | | || _ \ / \ / || ____|
|
||||
| | | | | \ / | | \| | | | `---| |----`| |_) | / ^ \ | ,----'| |__
|
||||
| | | | | |\/| | | . ` | | | | | | / / /_\ \ | | | __|
|
||||
| `--' | | | | | | |\ | | | | | | |\ \----./ _____ \ | `----.| |____
|
||||
\______/ |__| |__| |__| \__| |__| |__| | _| `._____/__/ \__\ \______||_______|
|
||||
omnitrace v1.11.2 (rev: 2586b74db8bf335742600010b8d9f1ce8da9cf89, compiler: GNU v11.4.1, rocm: v6.1.x)
|
||||
LD_PRELOAD=/opt/rocprofiler-systems/lib/librocprof-sys-dl.so.1.11.3
|
||||
ROCPROFSYS_CONFIG_FILE=
|
||||
ROCPROFSYS_CPU_FREQ_ENABLED=true
|
||||
ROCPROFSYS_OUTPUT_PATH=rocprof-sys-output
|
||||
ROCPROFSYS_OUTPUT_PREFIX=%tag%
|
||||
ROCPROFSYS_PROFILE=true
|
||||
ROCPROFSYS_TRACE=true
|
||||
ROCPROFSYS_TRACE_THREAD_LOCKS=false
|
||||
ROCPROFSYS_TRACE_THREAD_RW_LOCKS=false
|
||||
ROCPROFSYS_TRACE_THREAD_SPIN_LOCKS=false
|
||||
ROCPROFSYS_USE_KOKKOSP=false
|
||||
ROCPROFSYS_USE_MPIP=false
|
||||
ROCPROFSYS_USE_OMPT=false
|
||||
ROCPROFSYS_USE_PROCESS_SAMPLING=true
|
||||
ROCPROFSYS_USE_RCCLP=false
|
||||
ROCPROFSYS_USE_ROCM_SMI=false
|
||||
ROCPROFSYS_USE_ROCPROFILER=false
|
||||
ROCPROFSYS_USE_ROCTRACER=false
|
||||
ROCPROFSYS_USE_ROCTX=false
|
||||
ROCPROFSYS_USE_SAMPLING=true
|
||||
[rocprof-sys][dl][1785877] rocprofsys_main
|
||||
[rocprof-sys][1785877][rocprofsys_init_tooling] Instrumentation mode: Sampling
|
||||
__
|
||||
_ __ ___ ___ _ __ _ __ ___ / _| ___ _ _ ___
|
||||
| '__| / _ \ / __| | '_ \ | '__| / _ \ | |_ _____ / __| | | | | / __|
|
||||
| | | (_) | | (__ | |_) | | | | (_) | | _| |_____| \__ \ | |_| | \__ \
|
||||
|_| \___/ \___| | .__/ |_| \___/ |_| |___/ \__, | |___/
|
||||
|_| |___/
|
||||
|
||||
rocprof-sys v1.11.2 (rev: 2586b74db8bf335742600010b8d9f1ce8da9cf89, compiler: GNU v11.4.1, rocm: v6.1.x)
|
||||
[988.958] perfetto.cc:58649 Configured tracing session 1, #sources:1, duration:0 ms, #buffers:1, total buffer size:1024000 KB, total sessions:1, uid:0 session name: ""
|
||||
[parallel-overhead-locks] Threads: 4
|
||||
[parallel-overhead-locks] Iterations: 100
|
||||
@@ -386,19 +387,19 @@ Here is the full output from the previous
|
||||
[4] number of iterations: 100
|
||||
[parallel-overhead-locks] fibonacci(30) x 4 = 409221992
|
||||
[parallel-overhead-locks] number of mutex locks = 400
|
||||
[omnitrace][1785877][0][omnitrace_finalize] finalizing...
|
||||
[omnitrace][1785877][0][omnitrace_finalize]
|
||||
[omnitrace][1785877][0][omnitrace_finalize] omnitrace/process/1785877 : 0.294342 sec wall_clock, 4.776 MB peak_rss, 3.170 MB page_rss, 0.990000 sec cpu_clock, 336.3 % cpu_util [laps: 1]
|
||||
[omnitrace][1785877][0][omnitrace_finalize] omnitrace/process/1785877/thread/0 : 0.291535 sec wall_clock, 0.002619 sec thread_cpu_clock, 0.9 % thread_cpu_util, 4.776 MB peak_rss [laps: 1]
|
||||
[omnitrace][1785877][0][omnitrace_finalize] omnitrace/process/1785877/thread/1 : 0.271353 sec wall_clock, 0.222572 sec thread_cpu_clock, 82.0 % thread_cpu_util, 4.200 MB peak_rss [laps: 1]
|
||||
[omnitrace][1785877][0][omnitrace_finalize] omnitrace/process/1785877/thread/2 : 0.238218 sec wall_clock, 0.206405 sec thread_cpu_clock, 86.6 % thread_cpu_util, 3.432 MB peak_rss [laps: 1]
|
||||
[omnitrace][1785877][0][omnitrace_finalize] omnitrace/process/1785877/thread/3 : 0.209459 sec wall_clock, 0.193415 sec thread_cpu_clock, 92.3 % thread_cpu_util, 2.472 MB peak_rss [laps: 1]
|
||||
[omnitrace][1785877][0][omnitrace_finalize] omnitrace/process/1785877/thread/4 : 0.212029 sec wall_clock, 0.211694 sec thread_cpu_clock, 99.8 % thread_cpu_util, 1.152 MB peak_rss [laps: 1]
|
||||
[omnitrace][1785877][0][omnitrace_finalize]
|
||||
[omnitrace][1785877][0][omnitrace_finalize] Finalizing perfetto...
|
||||
[omnitrace][1785877][perfetto]> Outputting '/home/user/code/omnitrace/build-release/omnitrace-output/2024-07-15_16.21/parallel-overhead-locksperfetto-trace-1785877.proto' (39.12 KB / 0.04 MB / 0.00 GB)... Done
|
||||
[omnitrace][1785877][wall_clock]> Outputting 'omnitrace-output/2024-07-15_16.21/parallel-overhead-lockswall_clock-1785877.json'
|
||||
[omnitrace][1785877][wall_clock]> Outputting 'omnitrace-output/2024-07-15_16.21/parallel-overhead-lockswall_clock-1785877.txt'
|
||||
[omnitrace][1785877][metadata]> Outputting 'omnitrace-output/2024-07-15_16.21/parallel-overhead-locksmetadata-1785877.json' and 'omnitrace-output/2024-07-15_16.21/parallel-overhead-locksfunctions-1785877.json'
|
||||
[omnitrace][1785877][0][omnitrace_finalize] Finalized: 0.054582 sec wall_clock, 0.000 MB peak_rss, -1.798 MB page_rss, 0.040000 sec cpu_clock, 73.3 % cpu_util
|
||||
[rocprof-sys][1785877][0][rocprofsys_finalize] finalizing...
|
||||
[rocprof-sys][1785877][0][rocprofsys_finalize]
|
||||
[rocprof-sys][1785877][0][rocprofsys_finalize] rocprof-sys/process/1785877 : 0.294342 sec wall_clock, 4.776 MB peak_rss, 3.170 MB page_rss, 0.990000 sec cpu_clock, 336.3 % cpu_util [laps: 1]
|
||||
[rocprof-sys][1785877][0][rocprofsys_finalize] rocprof-sys/process/1785877/thread/0 : 0.291535 sec wall_clock, 0.002619 sec thread_cpu_clock, 0.9 % thread_cpu_util, 4.776 MB peak_rss [laps: 1]
|
||||
[rocprof-sys][1785877][0][rocprofsys_finalize] rocprof-sys/process/1785877/thread/1 : 0.271353 sec wall_clock, 0.222572 sec thread_cpu_clock, 82.0 % thread_cpu_util, 4.200 MB peak_rss [laps: 1]
|
||||
[rocprof-sys][1785877][0][rocprofsys_finalize] rocprof-sys/process/1785877/thread/2 : 0.238218 sec wall_clock, 0.206405 sec thread_cpu_clock, 86.6 % thread_cpu_util, 3.432 MB peak_rss [laps: 1]
|
||||
[rocprof-sys][1785877][0][rocprofsys_finalize] rocprof-sys/process/1785877/thread/3 : 0.209459 sec wall_clock, 0.193415 sec thread_cpu_clock, 92.3 % thread_cpu_util, 2.472 MB peak_rss [laps: 1]
|
||||
[rocprof-sys][1785877][0][rocprofsys_finalize] rocprof-sys/process/1785877/thread/4 : 0.212029 sec wall_clock, 0.211694 sec thread_cpu_clock, 99.8 % thread_cpu_util, 1.152 MB peak_rss [laps: 1]
|
||||
[rocprof-sys][1785877][0][rocprofsys_finalize]
|
||||
[rocprof-sys][1785877][0][rocprofsys_finalize] Finalizing perfetto...
|
||||
[rocprof-sys][1785877][perfetto]> Outputting '/home/user/code/rocprofiler-systems/build-release/rocprofiler-systems-output/2024-07-15_16.21/parallel-overhead-locksperfetto-trace-1785877.proto' (39.12 KB / 0.04 MB / 0.00 GB)... Done
|
||||
[rocprof-sys][1785877][wall_clock]> Outputting 'rocprof-sys-output/2024-07-15_16.21/parallel-overhead-lockswall_clock-1785877.json'
|
||||
[rocprof-sys][1785877][wall_clock]> Outputting 'rocprof-sys-output/2024-07-15_16.21/parallel-overhead-lockswall_clock-1785877.txt'
|
||||
[rocprof-sys][1785877][metadata]> Outputting 'rocprof-sys-output/2024-07-15_16.21/parallel-overhead-locksmetadata-1785877.json' and 'rocprof-sys-output/2024-07-15_16.21/parallel-overhead-locksfunctions-1785877.json'
|
||||
[rocprof-sys][1785877][0][rocprofsys_finalize] Finalized: 0.054582 sec wall_clock, 0.000 MB peak_rss, -1.798 MB page_rss, 0.040000 sec cpu_clock, 73.3 % cpu_util
|
||||
[989.312] perfetto.cc:60128 Tracing session 1 ended, total sessions:0
|
||||
|
||||
@@ -1,63 +1,63 @@
|
||||
.. meta::
|
||||
:description: Omnitrace documentation and reference
|
||||
:keywords: Omnitrace, ROCm, profiler, tracking, visualization, tool, Instinct, accelerator, AMD
|
||||
:description: ROCm Systems Profiler system output documentation and reference
|
||||
:keywords: rocprof-sys, rocprofiler-systems, Omnitrace, ROCm, profiler, system output, tracking, visualization, tool, Instinct, accelerator, AMD
|
||||
|
||||
****************************************************
|
||||
Understanding the Omnitrace output
|
||||
Understanding the Systems Profiler output
|
||||
****************************************************
|
||||
|
||||
The general output form of `Omnitrace <https://github.com/ROCm/omnitrace>`_ is
|
||||
The general output form of `ROCm Systems Profiler <https://github.com/ROCm/rocprofiler-systems>`_ is
|
||||
``<OUTPUT_PATH>[/<TIMESTAMP>]/[<PREFIX>]<DATA_NAME>[-<OUTPUT_SUFFIX>].<EXT>``.
|
||||
|
||||
For example, starting with the following base configuration:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
export OMNITRACE_OUTPUT_PATH=omnitrace-example-output
|
||||
export OMNITRACE_TIME_OUTPUT=ON
|
||||
export OMNITRACE_USE_PID=OFF
|
||||
export OMNITRACE_PROFILE=ON
|
||||
export OMNITRACE_TRACE=ON
|
||||
export ROCPROFSYS_OUTPUT_PATH=rocprof-sys-example-output
|
||||
export ROCPROFSYS_TIME_OUTPUT=ON
|
||||
export ROCPROFSYS_USE_PID=OFF
|
||||
export ROCPROFSYS_PROFILE=ON
|
||||
export ROCPROFSYS_TRACE=ON
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
$ omnitrace-instrument -- ./foo
|
||||
$ rocprof-sys-instrument -- ./foo
|
||||
...
|
||||
[omnitrace] Outputting 'omnitrace-example-output/perfetto-trace.proto'...
|
||||
[rocprof-sys] Outputting 'rocprof-sys-example-output/perfetto-trace.proto'...
|
||||
|
||||
[omnitrace] Outputting 'omnitrace-example-output/wall-clock.txt'...
|
||||
[omnitrace] Outputting 'omnitrace-example-output/wall-clock.json'...
|
||||
[rocprof-sys] Outputting 'rocprof-sys-example-output/wall-clock.txt'...
|
||||
[rocprof-sys] Outputting 'rocprof-sys-example-output/wall-clock.json'...
|
||||
|
||||
If the ``OMNITRACE_USE_PID`` option is enabled, then running a non-MPI executable
|
||||
If the ``ROCPROFSYS_USE_PID`` option is enabled, then running a non-MPI executable
|
||||
with a PID of ``63453`` results in the following output:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
$ export OMNITRACE_USE_PID=ON
|
||||
$ omnitrace-instrument -- ./foo
|
||||
$ export ROCPROFSYS_USE_PID=ON
|
||||
$ rocprof-sys-instrument -- ./foo
|
||||
...
|
||||
[omnitrace] Outputting 'omnitrace-example-output/perfetto-trace-63453.proto'...
|
||||
[rocprof-sys] Outputting 'rocprof-sys-example-output/perfetto-trace-63453.proto'...
|
||||
|
||||
[omnitrace] Outputting 'omnitrace-example-output/wall-clock-63453.txt'...
|
||||
[omnitrace] Outputting 'omnitrace-example-output/wall-clock-63453.json'...
|
||||
[rocprof-sys] Outputting 'rocprof-sys-example-output/wall-clock-63453.txt'...
|
||||
[rocprof-sys] Outputting 'rocprof-sys-example-output/wall-clock-63453.json'...
|
||||
|
||||
If ``OMNITRACE_TIME_OUTPUT`` is enabled, then a job that started on January 31, 2022 at 12:30 PM
|
||||
If ``ROCPROFSYS_TIME_OUTPUT`` is enabled, then a job that started on January 31, 2022 at 12:30 PM
|
||||
generates the following:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
$ export OMNITRACE_TIME_OUTPUT=ON
|
||||
$ omnitrace-instrument -- ./foo
|
||||
$ export ROCPROFSYS_TIME_OUTPUT=ON
|
||||
$ rocprof-sys-instrument -- ./foo
|
||||
...
|
||||
[omnitrace] Outputting 'omnitrace-example-output/2022-01-31_12.30_PM/perfetto-trace-63453.proto'...
|
||||
[rocprof-sys] Outputting 'rocprof-sys-example-output/2022-01-31_12.30_PM/perfetto-trace-63453.proto'...
|
||||
|
||||
[omnitrace] Outputting 'omnitrace-example-output/2022-01-31_12.30_PM/wall-clock-63453.txt'...
|
||||
[omnitrace] Outputting 'omnitrace-example-output/2022-01-31_12.30_PM/wall-clock-63453.json'...
|
||||
[rocprof-sys] Outputting 'rocprof-sys-example-output/2022-01-31_12.30_PM/wall-clock-63453.txt'...
|
||||
[rocprof-sys] Outputting 'rocprof-sys-example-output/2022-01-31_12.30_PM/wall-clock-63453.json'...
|
||||
|
||||
Metadata
|
||||
========================================
|
||||
|
||||
Omnitrace outputs a ``metadata.json`` file. This metadata file contains
|
||||
ROCm Systems Profiler outputs a ``metadata.json`` file. This metadata file contains
|
||||
information about the settings, environment variables, output files, and info
|
||||
about the system and the run, as follows:
|
||||
|
||||
@@ -77,7 +77,7 @@ Metadata JSON Sample
|
||||
.. code-block:: json
|
||||
|
||||
{
|
||||
"omnitrace": {
|
||||
"rocprof-sys": {
|
||||
"metadata": {
|
||||
"info": {
|
||||
"HW_L1_CACHE_SIZE": 32768,
|
||||
@@ -161,13 +161,13 @@ Metadata JSON Sample
|
||||
"text": [
|
||||
{
|
||||
"value": [
|
||||
"omnitrace-tests-output/parallel-overhead-binary-rewrite/roctracer.txt"
|
||||
"rocprof-sys-tests-output/parallel-overhead-binary-rewrite/roctracer.txt"
|
||||
],
|
||||
"key": "roctracer"
|
||||
},
|
||||
{
|
||||
"value": [
|
||||
"omnitrace-tests-output/parallel-overhead-binary-rewrite/wall_clock.txt"
|
||||
"rocprof-sys-tests-output/parallel-overhead-binary-rewrite/wall_clock.txt"
|
||||
],
|
||||
"key": "wall_clock"
|
||||
}
|
||||
@@ -175,15 +175,15 @@ Metadata JSON Sample
|
||||
"json": [
|
||||
{
|
||||
"value": [
|
||||
"omnitrace-tests-output/parallel-overhead-binary-rewrite/roctracer.json",
|
||||
"omnitrace-tests-output/parallel-overhead-binary-rewrite/roctracer.tree.json"
|
||||
"rocprof-sys-tests-output/parallel-overhead-binary-rewrite/roctracer.json",
|
||||
"rocprof-sys-tests-output/parallel-overhead-binary-rewrite/roctracer.tree.json"
|
||||
],
|
||||
"key": "roctracer"
|
||||
},
|
||||
{
|
||||
"value": [
|
||||
"omnitrace-tests-output/parallel-overhead-binary-rewrite/wall_clock.json",
|
||||
"omnitrace-tests-output/parallel-overhead-binary-rewrite/wall_clock.tree.json"
|
||||
"rocprof-sys-tests-output/parallel-overhead-binary-rewrite/wall_clock.json",
|
||||
"rocprof-sys-tests-output/parallel-overhead-binary-rewrite/wall_clock.tree.json"
|
||||
],
|
||||
"key": "wall_clock"
|
||||
}
|
||||
@@ -208,7 +208,7 @@ Metadata JSON Sample
|
||||
}
|
||||
],
|
||||
"settings": {
|
||||
"OMNITRACE_JSON_OUTPUT": {
|
||||
"ROCPROFSYS_JSON_OUTPUT": {
|
||||
"count": -1,
|
||||
"environ_updated": false,
|
||||
"name": "json_output",
|
||||
@@ -218,9 +218,9 @@ Metadata JSON Sample
|
||||
"value": true,
|
||||
"max_count": 1,
|
||||
"cmdline": [
|
||||
"--omnitrace-json-output"
|
||||
"--rocprof-sys-json-output"
|
||||
],
|
||||
"environ": "OMNITRACE_JSON_OUTPUT",
|
||||
"environ": "ROCPROFSYS_JSON_OUTPUT",
|
||||
"config_updated": false,
|
||||
"categories": [
|
||||
"io",
|
||||
@@ -237,10 +237,10 @@ Metadata JSON Sample
|
||||
}
|
||||
}
|
||||
|
||||
Configuring the Omnitrace output
|
||||
Configuring the ROCm Systems Profiler output
|
||||
========================================
|
||||
|
||||
Omnitrace includes a core set of options for controlling the format
|
||||
ROCm Systems Profiler includes a core set of options for controlling the format
|
||||
and contents of the output files. For additional information, see the guide on
|
||||
:doc:`configuring runtime options <./configuring-runtime-options>`.
|
||||
|
||||
@@ -251,19 +251,19 @@ Core configuration settings
|
||||
:header: "Setting", "Value", "Description"
|
||||
:widths: 30, 30, 100
|
||||
|
||||
"``OMNITRACE_OUTPUT_PATH``", "Any valid path", "Path to folder where output files should be placed"
|
||||
"``OMNITRACE_OUTPUT_PREFIX``", "String", "Useful for multiple runs with different arguments. See the next section on output prefix keys."
|
||||
"``OMNITRACE_OUTPUT_FILE``", "Any valid filepath", "Specific location for the Perfetto output file"
|
||||
"``OMNITRACE_TIME_OUTPUT``", "Boolean", "Place all output in a timestamped folder, timestamp format controlled via ``OMNITRACE_TIME_FORMAT``"
|
||||
"``OMNITRACE_TIME_FORMAT``", "String", "See ``strftime`` man pages for valid identifiers"
|
||||
"``OMNITRACE_USE_PID``", "Boolean", "Append either the PID or the MPI rank to all output files (before the extension)"
|
||||
"``ROCPROFSYS_OUTPUT_PATH``", "Any valid path", "Path to folder where output files should be placed"
|
||||
"``ROCPROFSYS_OUTPUT_PREFIX``", "String", "Useful for multiple runs with different arguments. See the next section on output prefix keys."
|
||||
"``ROCPROFSYS_OUTPUT_FILE``", "Any valid filepath", "Specific location for the Perfetto output file"
|
||||
"``ROCPROFSYS_TIME_OUTPUT``", "Boolean", "Place all output in a timestamped folder, timestamp format controlled via ``ROCPROFSYS_TIME_FORMAT``"
|
||||
"``ROCPROFSYS_TIME_FORMAT``", "String", "See ``strftime`` man pages for valid identifiers"
|
||||
"``ROCPROFSYS_USE_PID``", "Boolean", "Append either the PID or the MPI rank to all output files (before the extension)"
|
||||
|
||||
Output prefix keys
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Output prefix keys have many uses but are most helpful when dealing with multiple
|
||||
profiling runs or large MPI jobs.
|
||||
They are included in Omnitrace because they were introduced into Timemory
|
||||
They are included in ROCm Systems Profiler because they were introduced into Timemory
|
||||
for `compile-time-perf <https://github.com/jrmadsen/compile-time-perf>`_.
|
||||
They are needed to create different output files for a generic wrapper around
|
||||
compilation commands while still
|
||||
@@ -271,8 +271,8 @@ overwriting the output from the last time a file was compiled.
|
||||
|
||||
When doing scaling studies and specifying options via the command line,
|
||||
the recommended process is to
|
||||
use a common ``OMNITRACE_OUTPUT_PATH``, disable ``OMNITRACE_TIME_OUTPUT``,
|
||||
set ``OMNITRACE_OUTPUT_PREFIX="%argt%-"``, and let Omnitrace cleanly organize the output.
|
||||
use a common ``ROCPROFSYS_OUTPUT_PATH``, disable ``ROCPROFSYS_TIME_OUTPUT``,
|
||||
set ``ROCPROFSYS_OUTPUT_PREFIX="%argt%-"``, and let ROCm Systems Profiler cleanly organize the output.
|
||||
|
||||
.. csv-table::
|
||||
:header: "String", "Encoding"
|
||||
@@ -297,9 +297,9 @@ set ``OMNITRACE_OUTPUT_PREFIX="%argt%-"``, and let Omnitrace cleanly organize th
|
||||
"``%rank%``", "Value of ``SLURM_PROCID`` environment variable if exists, else ``MPI_Comm_rank`` (or ``0`` non-mpi)"
|
||||
"``%size%``", "``MPI_Comm_size`` or ``1`` if non-mpi"
|
||||
"``%nid%``", "``%rank%`` if possible, otherwise ``%pid%``"
|
||||
"``%launch_time%``", "Launch date and time (uses ``OMNITRACE_TIME_FORMAT``)"
|
||||
"``%launch_time%``", "Launch date and time (uses ``ROCPROFSYS_TIME_FORMAT``)"
|
||||
"``%env{NAME}%``", "Value of environment variable ``NAME`` (i.e. ``getenv(NAME)``)"
|
||||
"``%cfg{NAME}%``", "Value of configuration variable ``NAME`` (e.g. ``%cfg{OMNITRACE_SAMPLING_FREQ}%`` would resolve to sampling frequency)"
|
||||
"``%cfg{NAME}%``", "Value of configuration variable ``NAME`` (e.g. ``%cfg{ROCPROFSYS_SAMPLING_FREQ}%`` would resolve to sampling frequency)"
|
||||
"``$env{NAME}``", "Alternative syntax to ``%env{NAME}%``"
|
||||
"``$cfg{NAME}``", "Alternative syntax to ``%cfg{NAME}%``"
|
||||
"``%m``", "Shorthand for ``%argt_hash%``"
|
||||
@@ -318,8 +318,8 @@ set ``OMNITRACE_OUTPUT_PREFIX="%argt%-"``, and let Omnitrace cleanly organize th
|
||||
Perfetto output
|
||||
========================================
|
||||
|
||||
Use the ``OMNITRACE_OUTPUT_FILE`` to specify a specific location. If this is an
|
||||
absolute path, then all ``OMNITRACE_OUTPUT_PATH`` and similar
|
||||
Use the ``ROCPROFSYS_OUTPUT_FILE`` to specify a specific location. If this is an
|
||||
absolute path, then all ``ROCPROFSYS_OUTPUT_PATH`` and similar
|
||||
settings are ignored. Visit `ui.perfetto.dev <https://ui.perfetto.dev>`_ and open
|
||||
this file.
|
||||
|
||||
@@ -328,26 +328,26 @@ this file.
|
||||
If you are experiencing problems viewing your trace in the latest version of `Perfetto <http://ui.perfetto.dev>`_,
|
||||
then try using `Perfetto UI v46.0 <https://ui.perfetto.dev/v46.0-35b3d9845/#!/>`_.
|
||||
|
||||
.. image:: ../data/omnitrace-perfetto.png
|
||||
.. image:: ../data/rocprof-sys-perfetto.png
|
||||
:alt: Visualization of a performance graph in Perfetto
|
||||
|
||||
.. image:: ../data/omnitrace-rocm.png
|
||||
.. image:: ../data/rocprof-sys-rocm.png
|
||||
:alt: Visualization of ROCm data in Perfetto
|
||||
|
||||
.. image:: ../data/omnitrace-rocm-flow.png
|
||||
.. image:: ../data/rocprof-sys-rocm-flow.png
|
||||
:alt: Visualization of ROCm flow data in Perfetto
|
||||
|
||||
.. image:: ../data/omnitrace-user-api.png
|
||||
.. image:: ../data/rocprof-sys-user-api.png
|
||||
:alt: Visualization of ROCm API calls in Perfetto
|
||||
|
||||
Timemory output
|
||||
========================================
|
||||
|
||||
Use ``omnitrace-avail --components --filename`` to view the base filename for each component, as follows
|
||||
Use ``rocprof-sys-avail --components --filename`` to view the base filename for each component, as follows
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
$ omnitrace-avail wall_clock -C -f
|
||||
$ rocprof-sys-avail wall_clock -C -f
|
||||
|---------------------------------|---------------|------------------------|
|
||||
| COMPONENT | AVAILABLE | FILENAME |
|
||||
|---------------------------------|---------------|------------------------|
|
||||
@@ -355,16 +355,16 @@ Use ``omnitrace-avail --components --filename`` to view the base filename for ea
|
||||
| sampling_wall_clock | true | sampling_wall_clock |
|
||||
|---------------------------------|---------------|------------------------|
|
||||
|
||||
The ``OMNITRACE_COLLAPSE_THREADS`` and ``OMNITRACE_COLLAPSE_PROCESSES`` settings are
|
||||
only valid when full `MPI support is enabled <../install/install.html#mpi-support-within-omnitrace>`_.
|
||||
The ``ROCPROFSYS_COLLAPSE_THREADS`` and ``ROCPROFSYS_COLLAPSE_PROCESSES`` settings are
|
||||
only valid when full `MPI support is enabled <../install/install.html#mpi-support-within-rocprof-sys>`_.
|
||||
When they are set, Timemory combines the per-thread and per-rank data (respectively) of
|
||||
identical call stacks.
|
||||
|
||||
The ``OMNITRACE_FLAT_PROFILE`` setting removes all call stack hierarchy.
|
||||
Using ``OMNITRACE_FLAT_PROFILE=ON`` in combination
|
||||
with ``OMNITRACE_COLLAPSE_THREADS=ON`` is a useful configuration for identifying
|
||||
The ``ROCPROFSYS_FLAT_PROFILE`` setting removes all call stack hierarchy.
|
||||
Using ``ROCPROFSYS_FLAT_PROFILE=ON`` in combination
|
||||
with ``ROCPROFSYS_COLLAPSE_THREADS=ON`` is a useful configuration for identifying
|
||||
min/max measurements regardless of the calling context.
|
||||
The ``OMNITRACE_TIMELINE_PROFILE`` setting (with ``OMNITRACE_FLAT_PROFILE=OFF``) effectively
|
||||
The ``ROCPROFSYS_TIMELINE_PROFILE`` setting (with ``ROCPROFSYS_FLAT_PROFILE=OFF``) effectively
|
||||
generates similar data to that found
|
||||
in Perfetto. Enabling timeline and flat profiling effectively generates
|
||||
similar data to ``strace``. However, while Timemory generally
|
||||
@@ -376,11 +376,11 @@ Timemory text output
|
||||
|
||||
Timemory text output files are meant for human consumption (while JSON formats are for analysis),
|
||||
so some fields such as the ``LABEL`` might be truncated for readability.
|
||||
The truncation settings be changed through the ``OMNITRACE_MAX_WIDTH`` setting.
|
||||
The truncation settings be changed through the ``ROCPROFSYS_MAX_WIDTH`` setting.
|
||||
|
||||
.. note::
|
||||
|
||||
The generation of text output is configurable via ``OMNITRACE_TEXT_OUTPUT``.
|
||||
The generation of text output is configurable via ``ROCPROFSYS_TEXT_OUTPUT``.
|
||||
|
||||
.. _text-output-example-label:
|
||||
|
||||
@@ -389,7 +389,7 @@ Timemory text output example
|
||||
|
||||
In the following example, the ``NN`` field in ``|NN>>>`` is the thread ID. If MPI support is enabled,
|
||||
this becomes ``|MM|NN>>>`` where ``MM`` is the rank.
|
||||
If ``OMNITRACE_COLLAPSE_THREADS=ON`` and ``OMNITRACE_COLLAPSE_PROCESSES=ON`` are configured,
|
||||
If ``ROCPROFSYS_COLLAPSE_THREADS=ON`` and ``ROCPROFSYS_COLLAPSE_PROCESSES=ON`` are configured,
|
||||
neither the ``MM`` nor the ``NN`` are present unless the
|
||||
component explicitly sets type traits. Type traits specify that the data is only
|
||||
relevant per-thread or per-process, such as the ``thread_cpu_clock`` clock component.
|
||||
@@ -592,8 +592,8 @@ write a simple Python script for post-processing using this format than with the
|
||||
|
||||
.. note::
|
||||
|
||||
The generation of flat JSON output is configurable via ``OMNITRACE_JSON_OUTPUT``.
|
||||
The generation of hierarchical JSON data is configurable via ``OMNITRACE_TREE_OUTPUT``
|
||||
The generation of flat JSON output is configurable via ``ROCPROFSYS_JSON_OUTPUT``.
|
||||
The generation of hierarchical JSON data is configurable via ``ROCPROFSYS_TREE_OUTPUT``
|
||||
|
||||
Timemory JSON output sample
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
@@ -1,37 +1,38 @@
|
||||
.. meta::
|
||||
:description: Omnitrace documentation and reference
|
||||
:keywords: Omnitrace, ROCm, profiler, tracking, visualization, tool, Instinct, accelerator, AMD
|
||||
:description: ROCm Systems Profiler documentation and reference
|
||||
:keywords: rocprof-sys, rocprofiler-systems, Omnitrace, ROCm, profiler, tracking, visualization, tool, Instinct, accelerator, AMD
|
||||
|
||||
****************************************************
|
||||
Using the Omnitrace API
|
||||
Using the ROCm Systems Profiler API
|
||||
****************************************************
|
||||
|
||||
The following example shows how a program can use the Omnitrace API for run-time analysis.
|
||||
The following example shows how a program can use the ROCm Systems Profiler API
|
||||
for run-time analysis.
|
||||
|
||||
Omnitrace user API example program
|
||||
ROCm Systems Profiler user API example program
|
||||
========================================
|
||||
|
||||
You can use the Omnitrace API to define custom regions to profile and trace.
|
||||
The following C++ program demonstrates this technique by calling several functions from the
|
||||
Omnitrace API, such as ``omnitrace_user_push_region`` and
|
||||
``omnitrace_user_stop_thread_trace``.
|
||||
You can use the ROCm Systems Profiler API to define custom regions to profile and trace.
|
||||
The following C++ program demonstrates this technique by calling several functions from the
|
||||
ROCm Systems Profiler API, such as ``rocprofsys_user_push_region`` and
|
||||
``rocprofsys_user_stop_thread_trace``.
|
||||
|
||||
.. note::
|
||||
|
||||
By default, when Omnitrace detects any ``omnitrace_user_start_*`` or
|
||||
``omnitrace_user_stop_*`` function, instrumentation
|
||||
is disabled at start up, which means ``omnitrace_user_stop_trace()`` is not
|
||||
By default, when ROCm Systems Profiler detects any ``rocprofsys_user_start_*`` or
|
||||
``rocprofsys_user_stop_*`` function, instrumentation
|
||||
is disabled at start up, which means ``rocprofsys_user_stop_trace()`` is not
|
||||
required at the beginning of ``main``. This behavior
|
||||
can be manually controlled by using the ``OMNITRACE_INIT_ENABLED`` environment variable.
|
||||
can be manually controlled by using the ``ROCPROFSYS_INIT_ENABLED`` environment variable.
|
||||
User-defined regions are always
|
||||
recorded, regardless of whether ``omnitrace_user_start_*`` or
|
||||
``omnitrace_user_stop_*`` has been called.
|
||||
recorded, regardless of whether ``rocprofsys_user_start_*`` or
|
||||
``rocprofsys_user_stop_*`` has been called.
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
#include <omnitrace/categories.h>
|
||||
#include <omnitrace/types.h>
|
||||
#include <omnitrace/user.h>
|
||||
#include <rocprofiler-systems/categories.h>
|
||||
#include <rocprofiler-systems/types.h>
|
||||
#include <rocprofiler-systems/user.h>
|
||||
|
||||
#include <atomic>
|
||||
#include <cassert>
|
||||
@@ -56,52 +57,52 @@ Omnitrace API, such as ``omnitrace_user_push_region`` and
|
||||
|
||||
namespace
|
||||
{
|
||||
omnitrace_user_callbacks_t custom_callbacks = OMNITRACE_USER_CALLBACKS_INIT;
|
||||
omnitrace_user_callbacks_t original_callbacks = OMNITRACE_USER_CALLBACKS_INIT;
|
||||
rocprofsys_user_callbacks_t custom_callbacks = ROCPROFSYS_USER_CALLBACKS_INIT;
|
||||
rocprofsys_user_callbacks_t original_callbacks = ROCPROFSYS_USER_CALLBACKS_INIT;
|
||||
} // namespace
|
||||
|
||||
int
|
||||
main(int argc, char** argv)
|
||||
{
|
||||
custom_callbacks.push_region = &custom_push_region;
|
||||
omnitrace_user_configure(OMNITRACE_USER_UNION_CONFIG, custom_callbacks,
|
||||
rocprofsys_user_configure(ROCPROFSYS_USER_UNION_CONFIG, custom_callbacks,
|
||||
&original_callbacks);
|
||||
|
||||
omnitrace_user_push_region(argv[0]);
|
||||
omnitrace_user_push_region("initialization");
|
||||
rocprofsys_user_push_region(argv[0]);
|
||||
rocprofsys_user_push_region("initialization");
|
||||
size_t nthread = std::min<size_t>(16, std::thread::hardware_concurrency());
|
||||
size_t nitr = 50000;
|
||||
long nfib = 10;
|
||||
if(argc > 1) nfib = atol(argv[1]);
|
||||
if(argc > 2) nthread = atol(argv[2]);
|
||||
if(argc > 3) nitr = atol(argv[3]);
|
||||
omnitrace_user_pop_region("initialization");
|
||||
rocprofsys_user_pop_region("initialization");
|
||||
|
||||
printf("[%s] Threads: %zu\n[%s] Iterations: %zu\n[%s] fibonacci(%li)...\n", argv[0],
|
||||
nthread, argv[0], nitr, argv[0], nfib);
|
||||
|
||||
omnitrace_user_push_region("thread_creation");
|
||||
rocprofsys_user_push_region("thread_creation");
|
||||
std::vector<std::thread> threads{};
|
||||
threads.reserve(nthread);
|
||||
// disable instrumentation for child threads
|
||||
omnitrace_user_stop_thread_trace();
|
||||
rocprofsys_user_stop_thread_trace();
|
||||
for(size_t i = 0; i < nthread; ++i)
|
||||
{
|
||||
threads.emplace_back(&run, nitr, nfib);
|
||||
}
|
||||
// re-enable instrumentation
|
||||
omnitrace_user_start_thread_trace();
|
||||
omnitrace_user_pop_region("thread_creation");
|
||||
rocprofsys_user_start_thread_trace();
|
||||
rocprofsys_user_pop_region("thread_creation");
|
||||
|
||||
omnitrace_user_push_region("thread_wait");
|
||||
rocprofsys_user_push_region("thread_wait");
|
||||
for(auto& itr : threads)
|
||||
itr.join();
|
||||
omnitrace_user_pop_region("thread_wait");
|
||||
rocprofsys_user_pop_region("thread_wait");
|
||||
|
||||
run(nitr, nfib);
|
||||
|
||||
printf("[%s] fibonacci(%li) x %lu = %li\n", argv[0], nfib, nthread, total.load());
|
||||
omnitrace_user_pop_region(argv[0]);
|
||||
rocprofsys_user_pop_region(argv[0]);
|
||||
|
||||
return 0;
|
||||
}
|
||||
@@ -120,19 +121,19 @@ Omnitrace API, such as ``omnitrace_user_push_region`` and
|
||||
void
|
||||
run(size_t nitr, long n)
|
||||
{
|
||||
omnitrace_user_push_region(RUN_LABEL);
|
||||
rocprofsys_user_push_region(RUN_LABEL);
|
||||
long local = 0;
|
||||
for(size_t i = 0; i < nitr; ++i)
|
||||
local += fib(n);
|
||||
total += local;
|
||||
omnitrace_user_pop_region(RUN_LABEL);
|
||||
rocprofsys_user_pop_region(RUN_LABEL);
|
||||
}
|
||||
|
||||
int
|
||||
custom_push_region(const char* name)
|
||||
{
|
||||
if(!original_callbacks.push_region || !original_callbacks.push_annotated_region)
|
||||
return OMNITRACE_USER_ERROR_NO_BINDING;
|
||||
return ROCPROFSYS_USER_ERROR_NO_BINDING;
|
||||
|
||||
printf("Pushing custom region :: %s\n", name);
|
||||
|
||||
@@ -143,22 +144,22 @@ Omnitrace API, such as ``omnitrace_user_push_region`` and
|
||||
char _buff[1024];
|
||||
if(_err != 0) _msg = strerror_r(_err, _buff, sizeof(_buff));
|
||||
|
||||
omnitrace_annotation_t _annotations[] = {
|
||||
{ "errno", OMNITRACE_INT32, &_err }, { "strerror", OMNITRACE_STRING, _msg }
|
||||
rocprofsys_annotation_t _annotations[] = {
|
||||
{ "errno", ROCPROFSYS_INT32, &_err }, { "strerror", ROCPROFSYS_STRING, _msg }
|
||||
};
|
||||
|
||||
errno = 0; // reset errno
|
||||
return (*original_callbacks.push_annotated_region)(
|
||||
name, _annotations, sizeof(_annotations) / sizeof(omnitrace_annotation_t));
|
||||
name, _annotations, sizeof(_annotations) / sizeof(rocprofsys_annotation_t));
|
||||
}
|
||||
|
||||
return (*original_callbacks.push_region)(name);
|
||||
}
|
||||
|
||||
Linking the Omnitrace libraries to another program
|
||||
Linking the ROCm Systems Profiler libraries to another program
|
||||
=======================================================
|
||||
|
||||
To link the ``omnitrace-user-library`` to another program,
|
||||
To link the ``rocprofiler-systems-user-library`` to another program,
|
||||
use the following CMake and ``g++`` directives.
|
||||
|
||||
CMake
|
||||
@@ -166,19 +167,19 @@ CMake
|
||||
|
||||
.. code-block:: cmake
|
||||
|
||||
find_package(omnitrace REQUIRED COMPONENTS user)
|
||||
find_package(rocprofiler-systems REQUIRED COMPONENTS user)
|
||||
add_executable(foo foo.cpp)
|
||||
target_link_libraries(foo PRIVATE omnitrace::omnitrace-user-library)
|
||||
target_link_libraries(foo PRIVATE rocprofiler-systems::rocprofiler-systems-user-library)
|
||||
|
||||
g++ compilation
|
||||
-------------------------------------------------------
|
||||
|
||||
Assuming Omnitrace is installed in ``/opt/omnitrace``, use the ``g++`` compiler
|
||||
Assuming ROCm Systems Profiler is installed in ``/opt/rocprofiler-systems``, use the ``g++`` compiler
|
||||
to build the application.
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
g++ -I/opt/omnitrace foo.cpp -o foo -lomnitrace-user
|
||||
g++ -I/opt/rocprofiler-systems foo.cpp -o foo -lrocprofiler-systems-user
|
||||
|
||||
Output from the API example program
|
||||
========================================
|
||||
@@ -187,19 +188,19 @@ First, instrument and run the program.
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
$ omnitrace-instrument -l --min-instructions=8 -E custom_push_region -o -- ./user-api
|
||||
$ rocprof-sys-instrument -l --min-instructions=8 -E custom_push_region -o -- ./user-api
|
||||
...
|
||||
$ omnitrace-run --profile --use-pid off --time-output off -- ./user-api.inst 20 4 100
|
||||
$ rocprof-sys-run --profile --use-pid off --time-output off -- ./user-api.inst 20 4 100
|
||||
Pushing custom region :: ./user-api.inst
|
||||
[omnitrace][omnitrace_init_tooling] Instrumentation mode: Trace
|
||||
[rocprof-sys][rocprofsys_init_tooling] Instrumentation mode: Trace
|
||||
|
||||
|
||||
______ .___ ___. .__ __. __ .___________..______ ___ ______ _______
|
||||
/ __ \ | \/ | | \ | | | | | || _ \ / \ / || ____|
|
||||
| | | | | \ / | | \| | | | `---| |----`| |_) | / ^ \ | ,----'| |__
|
||||
| | | | | |\/| | | . ` | | | | | | / / /_\ \ | | | __|
|
||||
| `--' | | | | | | |\ | | | | | | |\ \----./ _____ \ | `----.| |____
|
||||
\______/ |__| |__| |__| \__| |__| |__| | _| `._____/__/ \__\ \______||_______|
|
||||
__
|
||||
_ __ ___ ___ _ __ _ __ ___ / _| ___ _ _ ___
|
||||
| '__| / _ \ / __| | '_ \ | '__| / _ \ | |_ _____ / __| | | | | / __|
|
||||
| | | (_) | | (__ | |_) | | | | (_) | | _| |_____| \__ \ | |_| | \__ \
|
||||
|_| \___/ \___| | .__/ |_| \___/ |_| |___/ \__, | |___/
|
||||
|_| |___/
|
||||
|
||||
|
||||
|
||||
@@ -215,29 +216,29 @@ First, instrument and run the program.
|
||||
Pushing custom region :: run(20) x 100
|
||||
Pushing custom region :: run(20) x 100
|
||||
[./user-api.inst] fibonacci(20) x 4 = 3382500
|
||||
[omnitrace][86267][0][omnitrace_finalize] finalizing...
|
||||
[rocprof-sys][86267][0][rocprofsys_finalize] finalizing...
|
||||
|
||||
|
||||
[omnitrace][86267][0] omnitrace : 5.190895 sec wall_clock, 2.748 mb peak_rss, 6.330000 sec cpu_clock, 121.9 % cpu_util [laps: 1]
|
||||
[omnitrace][86267][0] user-api.inst/thread-0 : 5.078713 sec wall_clock, 4.722415 sec thread_cpu_clock, 93.0 % thread_cpu_util, 1.276 mb peak_rss [laps: 1]
|
||||
[omnitrace][86267][0] user-api.inst/thread-1 : 0.322248 sec wall_clock, 0.322191 sec thread_cpu_clock, 100.0 % thread_cpu_util, 1.000 mb peak_rss [laps: 1]
|
||||
[omnitrace][86267][0] user-api.inst/thread-2 : 0.323255 sec wall_clock, 0.323194 sec thread_cpu_clock, 100.0 % thread_cpu_util, 0.000 mb peak_rss [laps: 1]
|
||||
[omnitrace][86267][0] user-api.inst/thread-3 : 0.323569 sec wall_clock, 0.323484 sec thread_cpu_clock, 100.0 % thread_cpu_util, 1.092 mb peak_rss [laps: 1]
|
||||
[omnitrace][86267][0] user-api.inst/thread-4 : 0.324178 sec wall_clock, 0.324057 sec thread_cpu_clock, 100.0 % thread_cpu_util, 1.184 mb peak_rss [laps: 1]
|
||||
[omnitrace][86267][0] Post-processing 51 cpu frequency and memory usage entries...
|
||||
[rocprof-sys][86267][0] rocprof-sys : 5.190895 sec wall_clock, 2.748 mb peak_rss, 6.330000 sec cpu_clock, 121.9 % cpu_util [laps: 1]
|
||||
[rocprof-sys][86267][0] user-api.inst/thread-0 : 5.078713 sec wall_clock, 4.722415 sec thread_cpu_clock, 93.0 % thread_cpu_util, 1.276 mb peak_rss [laps: 1]
|
||||
[rocprof-sys][86267][0] user-api.inst/thread-1 : 0.322248 sec wall_clock, 0.322191 sec thread_cpu_clock, 100.0 % thread_cpu_util, 1.000 mb peak_rss [laps: 1]
|
||||
[rocprof-sys][86267][0] user-api.inst/thread-2 : 0.323255 sec wall_clock, 0.323194 sec thread_cpu_clock, 100.0 % thread_cpu_util, 0.000 mb peak_rss [laps: 1]
|
||||
[rocprof-sys][86267][0] user-api.inst/thread-3 : 0.323569 sec wall_clock, 0.323484 sec thread_cpu_clock, 100.0 % thread_cpu_util, 1.092 mb peak_rss [laps: 1]
|
||||
[rocprof-sys][86267][0] user-api.inst/thread-4 : 0.324178 sec wall_clock, 0.324057 sec thread_cpu_clock, 100.0 % thread_cpu_util, 1.184 mb peak_rss [laps: 1]
|
||||
[rocprof-sys][86267][0] Post-processing 51 cpu frequency and memory usage entries...
|
||||
|
||||
[omnitrace][wall_clock]|0> Outputting 'omnitrace-user-api.inst-output/wall_clock.json'...
|
||||
[omnitrace][wall_clock]|0> Outputting 'omnitrace-user-api.inst-output/wall_clock.tree.json'...
|
||||
[omnitrace][wall_clock]|0> Outputting 'omnitrace-user-api.inst-output/wall_clock.txt'...
|
||||
[rocprof-sys][wall_clock]|0> Outputting 'rocprof-sys-user-api.inst-output/wall_clock.json'...
|
||||
[rocprof-sys][wall_clock]|0> Outputting 'rocprof-sys-user-api.inst-output/wall_clock.tree.json'...
|
||||
[rocprof-sys][wall_clock]|0> Outputting 'rocprof-sys-user-api.inst-output/wall_clock.txt'...
|
||||
|
||||
[omnitrace][manager::finalize][metadata]> Outputting 'omnitrace-user-api.inst-output/metadata.json' and 'omnitrace-user-api.inst-output/functions.json'...
|
||||
[omnitrace][86267][0][omnitrace_finalize] Finalized
|
||||
[rocprof-sys][manager::finalize][metadata]> Outputting 'rocprof-sys-user-api.inst-output/metadata.json' and 'rocprof-sys-user-api.inst-output/functions.json'...
|
||||
[rocprof-sys][86267][0][rocprofsys_finalize] Finalized
|
||||
|
||||
Then review the output.
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
$ cat omnitrace-example-output/wall_clock.txt
|
||||
$ cat rocprof-sys-example-output/wall_clock.txt
|
||||
|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
||||
| REAL-CLOCK TIMER (I.E. WALL-CLOCK TIMER) |
|
||||
|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
||||
@@ -1,17 +1,17 @@
|
||||
.. meta::
|
||||
:description: Omnitrace documentation and reference
|
||||
:keywords: Omnitrace, ROCm, profiler, tracking, visualization, tool, Instinct, accelerator, AMD
|
||||
:description: ROCm Systems Profiler documentation and reference
|
||||
:keywords: rocprof-sys, rocprofiler-systems, Omnitrace, ROCm, profiler, tracking, visualization, tool, Instinct, accelerator, AMD
|
||||
|
||||
***********************
|
||||
Omnitrace documentation
|
||||
ROCm Systems Profiler documentation
|
||||
***********************
|
||||
|
||||
Omnitrace is designed for the high-level profiling and comprehensive tracing
|
||||
ROCm Systems Profiler, formerly known as "Omnitrace", is designed for the high-level profiling and comprehensive tracing
|
||||
of applications running on the CPU or the CPU and GPU. It supports dynamic binary
|
||||
instrumentation, call-stack sampling, and various other features for determining
|
||||
which function and line number are currently executing. To learn more, see :doc:`what-is-omnitrace`
|
||||
which function and line number are currently executing. To learn more, see :doc:`what-is-rocprof-sys`
|
||||
|
||||
The code is open and hosted at `<https://github.com/ROCm/omnitrace>`_.
|
||||
The code is open and hosted at `<https://github.com/ROCm/rocprofiler-systems>`_.
|
||||
|
||||
|
||||
.. grid:: 2
|
||||
@@ -20,7 +20,7 @@ The code is open and hosted at `<https://github.com/ROCm/omnitrace>`_.
|
||||
.. grid-item-card:: Install
|
||||
|
||||
* :doc:`Quick start <./install/quick-start>`
|
||||
* :doc:`Omnitrace installation <./install/install>`
|
||||
* :doc:`ROCm Systems Profiler installation <./install/install>`
|
||||
|
||||
|
||||
The documentation is structured as follows:
|
||||
@@ -30,31 +30,30 @@ The documentation is structured as follows:
|
||||
|
||||
.. grid-item-card:: Tutorials
|
||||
|
||||
* `GitHub examples <https://github.com/ROCm/omnitrace/tree/amd-mainline/examples>`_
|
||||
* `GitHub examples <https://github.com/ROCm/rocprofiler-systems/tree/amd-mainline/examples>`_
|
||||
* :doc:`Video tutorials <./tutorials/video-tutorials>`
|
||||
|
||||
.. grid-item-card:: How to
|
||||
|
||||
* :doc:`Configuring and validating the Omnitrace environment <./how-to/configuring-validating-environment>`
|
||||
* :doc:`Configuring and validating the ROCm Systems Profiler environment <./how-to/configuring-validating-environment>`
|
||||
* :doc:`Configuring runtime options <./how-to/configuring-runtime-options>`
|
||||
* :doc:`Sampling the call stack <./how-to/sampling-call-stack>`
|
||||
* :doc:`Instrumenting and rewriting a binary application <./how-to/instrumenting-rewriting-binary-application>`
|
||||
* :doc:`Performing causal profiling <./how-to/performing-causal-profiling>`
|
||||
* :doc:`Understanding the Omnitrace output <./how-to/understanding-omnitrace-output>`
|
||||
* :doc:`Understanding the ROCm Systems Profiler output <./how-to/understanding-rocprof-sys-output>`
|
||||
* :doc:`Profiling Python scripts <./how-to/profiling-python-scripts>`
|
||||
* :doc:`Using the Omnitrace API <./how-to/using-omnitrace-api>`
|
||||
* :doc:`General tips for using Omnitrace <./how-to/general-tips-using-omnitrace>`
|
||||
|
||||
* :doc:`Using the ROCm Systems Profiler API <./how-to/using-rocprof-sys-api>`
|
||||
* :doc:`General tips for using ROCm Systems Profiler <./how-to/general-tips-using-rocprof-sys>`
|
||||
|
||||
.. grid-item-card:: Conceptual
|
||||
|
||||
* :doc:`Data collection modes <./conceptual/data-collection-modes>`
|
||||
* :doc:`The Omnitrace feature set <./conceptual/omnitrace-feature-set>`
|
||||
|
||||
* :doc:`The ROCm Systems Profiler feature set <./conceptual/rocprof-sys-feature-set>`
|
||||
|
||||
.. grid-item-card:: Reference
|
||||
|
||||
* :doc:`Development guide <./reference/development-guide>`
|
||||
* :doc:`Omnitrace glossary <./reference/omnitrace-glossary>`
|
||||
* :doc:`ROCm Systems Profiler glossary <./reference/rocprof-sys-glossary>`
|
||||
* :doc:`API library <./doxygen/html/files>`
|
||||
* :doc:`Class member functions <./doxygen/html/functions>`
|
||||
* :doc:`Globals <./doxygen/html/globals>`
|
||||
|
||||
@@ -1,38 +1,41 @@
|
||||
.. meta::
|
||||
:description: Omnitrace documentation and reference
|
||||
:keywords: Omnitrace, ROCm, profiler, tracking, visualization, tool, Instinct, accelerator, AMD
|
||||
:description: ROCm Systems Profiler installation documentation and reference
|
||||
:keywords: rocprof-sys, rocprofiler-systems, Omnitrace, ROCm, installation, installer, profiler, tracking, visualization, tool, Instinct, accelerator, AMD
|
||||
|
||||
*************************************
|
||||
Omnitrace installation
|
||||
ROCm Systems Profiler installation
|
||||
*************************************
|
||||
|
||||
The following information builds on the guidelines in the :doc:`Quick start <./quick-start>` guide.
|
||||
It covers how to install `Omnitrace <https://github.com/ROCm/omnitrace>`_ from source or a binary distribution,
|
||||
as well as the :ref:`post-installation-steps`.
|
||||
It covers how to install `ROCm Systems Profiler <https://github.com/ROCm/rocprofiler-systems>`_ from
|
||||
source or a binary distribution, as well as the :ref:`post-installation-steps`.
|
||||
|
||||
If you have problems using Omnitrace after installation,
|
||||
If you have problems using ROCm Systems Profiler after installation,
|
||||
consult the :ref:`post-installation-troubleshooting` section.
|
||||
|
||||
Release links
|
||||
========================================
|
||||
|
||||
To review and install either the current Omnitrace release or earlier releases, use these links:
|
||||
To review and install either the current ROCm Systems Profiler release or earlier releases, use these links:
|
||||
|
||||
* Latest Omnitrace Release: `<https://github.com/ROCm/omnitrace/releases/latest>`_
|
||||
* All Omnitrace Releases: `<https://github.com/ROCm/omnitrace/releases>`_
|
||||
* Latest ROCm Systems Profiler Release: `<https://github.com/ROCm/rocprofiler-systems/releases/latest>`_
|
||||
* All ROCm Systems Profiler Releases: `<https://github.com/ROCm/rocprofiler-systems/releases>`_
|
||||
|
||||
Operating system support
|
||||
========================================
|
||||
|
||||
Omnitrace is only supported on Linux. The following distributions are tested in the Omnitrace GitHub workflows:
|
||||
ROCm Systems Profiler is only supported on Linux. The following distributions are tested in the ROCm Systems Profiler GitHub workflows:
|
||||
|
||||
* Ubuntu 20.04
|
||||
* Ubuntu 22.04
|
||||
* OpenSUSE 15.3
|
||||
* OpenSUSE 15.4
|
||||
* Red Hat 8.7
|
||||
* Red Hat 9.0
|
||||
* Red Hat 9.1
|
||||
* OpenSUSE 15.5
|
||||
* OpenSUSE 15.6
|
||||
* Red Hat 8.8
|
||||
* Red Hat 8.9
|
||||
* Red Hat 8.10
|
||||
* Red Hat 9.2
|
||||
* Red Hat 9.3
|
||||
* Red Hat 9.4
|
||||
|
||||
Other OS distributions might function but are not supported or tested.
|
||||
|
||||
@@ -61,58 +64,58 @@ Architecture
|
||||
========================================
|
||||
|
||||
With regards to instrumentation, at present only AMD64 (x86_64) architectures are tested. However,
|
||||
Dyninst supports several more architectures and Omnitrace instrumentation may support other
|
||||
Dyninst supports several more architectures and ROCm Systems Profiler instrumentation may support other
|
||||
CPU architectures such as aarch64 and ppc64.
|
||||
Other modes of use, such as sampling and causal profiling, are not dependent on Dyninst and therefore
|
||||
might be more portable.
|
||||
|
||||
Installing Omnitrace from binary distributions
|
||||
Installing ROCm Systems Profiler from binary distributions
|
||||
================================================
|
||||
|
||||
Every Omnitrace release provides binary installer scripts of the form:
|
||||
Every ROCm Systems Profiler release provides binary installer scripts of the form:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
omnitrace-{VERSION}-{OS_DISTRIB}-{OS_VERSION}[-ROCm-{ROCM_VERSION}[-{EXTRA}]].sh
|
||||
rocprof-sys-{VERSION}-{OS_DISTRIB}-{OS_VERSION}[-ROCm-{ROCM_VERSION}[-{EXTRA}]].sh
|
||||
|
||||
For example,
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
omnitrace-1.0.0-ubuntu-18.04-OMPT-PAPI-Python3.sh
|
||||
omnitrace-1.0.0-ubuntu-18.04-ROCm-405000-OMPT-PAPI-Python3.sh
|
||||
rocprof-sys-1.0.0-ubuntu-18.04-OMPT-PAPI-Python3.sh
|
||||
rocprof-sys-1.0.0-ubuntu-18.04-ROCm-405000-OMPT-PAPI-Python3.sh
|
||||
...
|
||||
omnitrace-1.0.0-ubuntu-20.04-ROCm-50000-OMPT-PAPI-Python3.sh
|
||||
rocprof-sys-1.0.0-ubuntu-20.04-ROCm-50000-OMPT-PAPI-Python3.sh
|
||||
|
||||
Any of the ``EXTRA`` fields with a CMake build option
|
||||
(for example, PAPI, as referenced in a following section) or
|
||||
with no link requirements (such as OMPT) have
|
||||
self-contained support for these packages.
|
||||
|
||||
To install Omnitrace using a binary installer script, follow these steps:
|
||||
To install ROCm Systems Profiler using a binary installer script, follow these steps:
|
||||
|
||||
#. Download the appropriate binary distribution
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
wget https://github.com/ROCm/omnitrace/releases/download/v<VERSION>/<SCRIPT>
|
||||
wget https://github.com/ROCm/rocprofiler-systems/releases/download/v<VERSION>/<SCRIPT>
|
||||
|
||||
#. Create the target installation directory
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
mkdir /opt/omnitrace
|
||||
mkdir /opt/rocprofiler-systems
|
||||
|
||||
#. Run the installer script
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
./omnitrace-1.0.0-ubuntu-18.04-ROCm-405000-OMPT-PAPI.sh --prefix=/opt/omnitrace --exclude-subdir
|
||||
./rocprofiler-systems-1.0.0-ubuntu-18.04-ROCm-405000-OMPT-PAPI.sh --prefix=/opt/rocprofiler-systems --exclude-subdir
|
||||
|
||||
Installing Omnitrace from source
|
||||
Installing ROCm Systems Profiler from source
|
||||
========================================
|
||||
|
||||
Omnitrace needs a GCC compiler with full support for C++17 and CMake v3.16 or higher.
|
||||
ROCm Systems Profiler needs a GCC compiler with full support for C++17 and CMake v3.16 or higher.
|
||||
The Clang compiler may be used in lieu of the GCC compiler if `Dyninst <https://github.com/dyninst/dyninst>`_
|
||||
is already installed.
|
||||
|
||||
@@ -122,7 +125,7 @@ Build requirements
|
||||
* GCC compiler v7+
|
||||
|
||||
* Older GCC compilers may be supported but are not tested
|
||||
* Clang compilers are generally supported for Omnitrace but not Dyninst
|
||||
* Clang compilers are generally supported for ROCm Systems Profiler but not Dyninst
|
||||
|
||||
* `CMake <https://cmake.org/>`_ v3.16+
|
||||
|
||||
@@ -151,16 +154,16 @@ Required third-party packages
|
||||
* `libunwind <https://www.nongnu.org/libunwind/>`_ for call-stack sampling
|
||||
|
||||
Any of the third-party packages required by Dyninst, along with Dyninst itself, can be built and installed
|
||||
during the Omnitrace build. The following list indicates the package, the version,
|
||||
the application that requires the package (for example, Omnitrace requires Dyninst
|
||||
while Dyninst requires TBB), and the CMake option to build the package alongside Omnitrace:
|
||||
during the ROCm Systems Profiler build. The following list indicates the package, the version,
|
||||
the application that requires the package (for example, ROCm Systems Profiler requires Dyninst
|
||||
while Dyninst requires TBB), and the CMake option to build the package alongside ROCm Systems Profiler:
|
||||
|
||||
.. csv-table::
|
||||
:header: "Third-Party Library", "Minimum Version", "Required By", "CMake Option"
|
||||
:widths: 15, 10, 12, 40
|
||||
|
||||
"Dyninst", "12.0", "Omnitrace", "``OMNITRACE_BUILD_DYNINST`` (default: OFF)"
|
||||
"Libunwind", "", "Omnitrace", "``OMNITRACE_BUILD_LIBUNWIND`` (default: ON)"
|
||||
"Dyninst", "12.0", "ROCm Systems Profiler", "``ROCPROFSYS_BUILD_DYNINST`` (default: OFF)"
|
||||
"Libunwind", "", "ROCm Systems Profiler", "``ROCPROFSYS_BUILD_LIBUNWIND`` (default: ON)"
|
||||
"TBB", "2018.6", "Dyninst", "``DYNINST_BUILD_TBB`` (default: OFF)"
|
||||
"ElfUtils", "0.178", "Dyninst", "``DYNINST_BUILD_ELFUTILS`` (default: OFF)"
|
||||
"LibIberty", "", "Dyninst", "``DYNINST_BUILD_LIBIBERTY`` (default: OFF)"
|
||||
@@ -180,9 +183,9 @@ Optional third-party packages
|
||||
* `PAPI <https://icl.utk.edu/papi/>`_
|
||||
* MPI
|
||||
|
||||
* ``OMNITRACE_USE_MPI`` enables full MPI support
|
||||
* ``OMNITRACE_USE_MPI_HEADERS`` enables wrapping of the dynamically-linked MPI C function calls.
|
||||
(By default, if Omnitrace cannot find an OpenMPI MPI distribution, it uses a local copy
|
||||
* ``ROCPROFSYS_USE_MPI`` enables full MPI support
|
||||
* ``ROCPROFSYS_USE_MPI_HEADERS`` enables wrapping of the dynamically-linked MPI C function calls.
|
||||
(By default, if ROCm Systems Profiler cannot find an OpenMPI MPI distribution, it uses a local copy
|
||||
of the OpenMPI ``mpi.h``.)
|
||||
|
||||
* Several optional third-party profiling tools supported by Timemory
|
||||
@@ -192,19 +195,19 @@ Optional third-party packages
|
||||
:header: "Third-Party Library", "CMake Enable Option", "CMake Build Option"
|
||||
:widths: 15, 45, 40
|
||||
|
||||
"PAPI", "``OMNITRACE_USE_PAPI`` (default: ON)", "``OMNITRACE_BUILD_PAPI`` (default: ON)"
|
||||
"MPI", "``OMNITRACE_USE_MPI`` (default: OFF)", ""
|
||||
"MPI (header-only)", "``OMNITRACE_USE_MPI_HEADERS`` (default: ON)", ""
|
||||
"PAPI", "``ROCPROFSYS_USE_PAPI`` (default: ON)", "``ROCPROFSYS_BUILD_PAPI`` (default: ON)"
|
||||
"MPI", "``ROCPROFSYS_USE_MPI`` (default: OFF)", ""
|
||||
"MPI (header-only)", "``ROCPROFSYS_USE_MPI_HEADERS`` (default: ON)", ""
|
||||
|
||||
Installing Dyninst
|
||||
-----------------------------------
|
||||
|
||||
The easiest way to install Dyninst is alongside Omnitrace, but it can also be installed using Spack.
|
||||
The easiest way to install Dyninst is alongside ROCm Systems Profiler, but it can also be installed using Spack.
|
||||
|
||||
Building Dyninst alongside Omnitrace
|
||||
Building Dyninst alongside ROCm Systems Profiler
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
To install Dyninst alongside Omnitrace, configure Omnitrace with ``OMNITRACE_BUILD_DYNINST=ON``.
|
||||
To install Dyninst alongside ROCm Systems Profiler, configure ROCm Systems Profiler with ``ROCPROFSYS_BUILD_DYNINST=ON``.
|
||||
Depending on the version of Ubuntu, the ``apt`` package manager might have current enough
|
||||
versions of the Dyninst Boost, TBB, and LibIberty dependencies
|
||||
(use ``apt-get install libtbb-dev libiberty-dev libboost-dev``).
|
||||
@@ -213,8 +216,8 @@ its dependencies via ``DYNINST_BUILD_<DEP>=ON``, as follows:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
git clone https://github.com/ROCm/omnitrace.git omnitrace-source
|
||||
cmake -B omnitrace-build -DOMNITRACE_BUILD_DYNINST=ON -DDYNINST_BUILD_{TBB,ELFUTILS,BOOST,LIBIBERTY}=ON omnitrace-source
|
||||
git clone https://github.com/ROCm/rocprofiler-systems.git rocprof-sys-source
|
||||
cmake -B rocprof-sys-build -DROCPROFSYS_BUILD_DYNINST=ON -DDYNINST_BUILD_{TBB,ELFUTILS,BOOST,LIBIBERTY}=ON rocprof-sys-source
|
||||
|
||||
where ``-DDYNINST_BUILD_{TBB,BOOST,ELFUTILS,LIBIBERTY}=ON`` is expanded by
|
||||
the shell to ``-DDYNINST_BUILD_TBB=ON -DDYNINST_BUILD_BOOST=ON ...``
|
||||
@@ -234,18 +237,18 @@ Installing Dyninst via Spack
|
||||
spack install --reuse dyninst
|
||||
spack load -r dyninst
|
||||
|
||||
Installing Omnitrace
|
||||
Installing ROCm Systems Profiler
|
||||
-----------------------------------
|
||||
|
||||
Omnitrace has CMake configuration options for MPI support (``OMNITRACE_USE_MPI`` or
|
||||
``OMNITRACE_USE_MPI_HEADERS``), HIP kernel tracing (``OMNITRACE_USE_ROCTRACER``),
|
||||
ROCm device sampling (``OMNITRACE_USE_ROCM_SMI``), OpenMP-Tools (``OMNITRACE_USE_OMPT``),
|
||||
hardware counters via PAPI (``OMNITRACE_USE_PAPI``), among other features.
|
||||
ROCm Systems Profiler has CMake configuration options for MPI support (``ROCPROFSYS_USE_MPI`` or
|
||||
``ROCPROFSYS_USE_MPI_HEADERS``), HIP kernel tracing (``ROCPROFSYS_USE_ROCTRACER``),
|
||||
ROCm device sampling (``ROCPROFSYS_USE_ROCM_SMI``), OpenMP-Tools (``ROCPROFSYS_USE_OMPT``),
|
||||
hardware counters via PAPI (``ROCPROFSYS_USE_PAPI``), among other features.
|
||||
Various additional features can be enabled via the
|
||||
``TIMEMORY_USE_*`` `CMake options <https://timemory.readthedocs.io/en/develop/installation.html#cmake-options>`_.
|
||||
Any ``OMNITRACE_USE_<VAL>`` option which has a corresponding ``TIMEMORY_USE_<VAL>``
|
||||
Any ``ROCPROFSYS_USE_<VAL>`` option which has a corresponding ``TIMEMORY_USE_<VAL>``
|
||||
option means that the Timemory support for this feature has been integrated
|
||||
into Perfetto support for Omnitrace, for example, ``OMNITRACE_USE_PAPI=<VAL>`` also configures
|
||||
into Perfetto support for ROCm Systems Profiler, for example, ``ROCPROFSYS_USE_PAPI=<VAL>`` also configures
|
||||
``TIMEMORY_USE_PAPI=<VAL>``. This means the data that Timemory is able to collect via this package
|
||||
is passed along to Perfetto and is displayed when the ``.proto`` file is visualized
|
||||
in `the Perfetto UI <https://ui.perfetto.dev>`_.
|
||||
@@ -257,39 +260,39 @@ in `the Perfetto UI <https://ui.perfetto.dev>`_.
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
git clone https://github.com/ROCm/omnitrace.git omnitrace-source
|
||||
git clone https://github.com/ROCm/rocprofiler-systems.git rocprof-sys-source
|
||||
cmake \
|
||||
-B omnitrace-build \
|
||||
-D CMAKE_INSTALL_PREFIX=/opt/omnitrace \
|
||||
-D OMNITRACE_USE_HIP=ON \
|
||||
-D OMNITRACE_USE_ROCM_SMI=ON \
|
||||
-D OMNITRACE_USE_ROCTRACER=ON \
|
||||
-D OMNITRACE_USE_PYTHON=ON \
|
||||
-D OMNITRACE_USE_OMPT=ON \
|
||||
-D OMNITRACE_USE_MPI_HEADERS=ON \
|
||||
-D OMNITRACE_BUILD_PAPI=ON \
|
||||
-D OMNITRACE_BUILD_LIBUNWIND=ON \
|
||||
-D OMNITRACE_BUILD_DYNINST=ON \
|
||||
-B rocprof-sys-build \
|
||||
-D CMAKE_INSTALL_PREFIX=/opt/rocprofiler-systems \
|
||||
-D ROCPROFSYS_USE_HIP=ON \
|
||||
-D ROCPROFSYS_USE_ROCM_SMI=ON \
|
||||
-D ROCPROFSYS_USE_ROCTRACER=ON \
|
||||
-D ROCPROFSYS_USE_PYTHON=ON \
|
||||
-D ROCPROFSYS_USE_OMPT=ON \
|
||||
-D ROCPROFSYS_USE_MPI_HEADERS=ON \
|
||||
-D ROCPROFSYS_BUILD_PAPI=ON \
|
||||
-D ROCPROFSYS_BUILD_LIBUNWIND=ON \
|
||||
-D ROCPROFSYS_BUILD_DYNINST=ON \
|
||||
-D DYNINST_BUILD_TBB=ON \
|
||||
-D DYNINST_BUILD_BOOST=ON \
|
||||
-D DYNINST_BUILD_ELFUTILS=ON \
|
||||
-D DYNINST_BUILD_LIBIBERTY=ON \
|
||||
omnitrace-source
|
||||
cmake --build omnitrace-build --target all --parallel 8
|
||||
cmake --build omnitrace-build --target install
|
||||
source /opt/omnitrace/share/omnitrace/setup-env.sh
|
||||
rocprof-sys-source
|
||||
cmake --build rocprof-sys-build --target all --parallel 8
|
||||
cmake --build rocprof-sys-build --target install
|
||||
source /opt/rocprofiler-systems/share/rocprofiler-systems/setup-env.sh
|
||||
|
||||
.. _mpi-support-omnitrace:
|
||||
.. _mpi-support-rocprof-sys:
|
||||
|
||||
MPI support within Omnitrace
|
||||
MPI support within ROCm Systems Profiler
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Omnitrace can have full (``OMNITRACE_USE_MPI=ON``) or partial (``OMNITRACE_USE_MPI_HEADERS=ON``) MPI support.
|
||||
ROCm Systems Profiler can have full (``ROCPROFSYS_USE_MPI=ON``) or partial (``ROCPROFSYS_USE_MPI_HEADERS=ON``) MPI support.
|
||||
The only difference between these two modes is whether or not the results collected
|
||||
via Timemory and/or Perfetto can be aggregated into a single
|
||||
output file during finalization. When full MPI support is enabled, combining the
|
||||
Timemory results always occurs, whereas combining the Perfetto
|
||||
results is configurable via the ``OMNITRACE_PERFETTO_COMBINE_TRACES`` setting.
|
||||
results is configurable via the ``ROCPROFSYS_PERFETTO_COMBINE_TRACES`` setting.
|
||||
|
||||
The primary benefits of partial or full MPI support are the automatic wrapping
|
||||
of MPI functions and the ability
|
||||
@@ -298,13 +301,13 @@ instead of having to use the system process identifier (i.e. ``PID``).
|
||||
In general, it's recommended to use partial MPI support with the OpenMPI
|
||||
headers as this is the most portable configuration.
|
||||
If full MPI support is selected, make sure your target application is built
|
||||
against the same MPI distribution as Omnitrace.
|
||||
For example, do not build Omnitrace with MPICH and use it on a target application built against OpenMPI.
|
||||
against the same MPI distribution as ROCm Systems Profiler.
|
||||
For example, do not build ROCm Systems Profiler with MPICH and use it on a target application built against OpenMPI.
|
||||
If partial support is selected, the reason the OpenMPI headers are recommended instead of the MPICH headers is
|
||||
because the ``MPI_COMM_WORLD`` in OpenMPI is a pointer to ``ompi_communicator_t`` (8 bytes),
|
||||
whereas ``MPI_COMM_WORLD`` in MPICH is an ``int`` (4 bytes). Building Omnitrace with partial MPI support
|
||||
whereas ``MPI_COMM_WORLD`` in MPICH is an ``int`` (4 bytes). Building ROCm Systems Profiler with partial MPI support
|
||||
and the MPICH headers and then using
|
||||
Omnitrace on an application built against OpenMPI causes a segmentation fault.
|
||||
ROCm Systems Profiler on an application built against OpenMPI causes a segmentation fault.
|
||||
This happens because the value of the ``MPI_COMM_WORLD`` is truncated
|
||||
during the function wrapping before being passed along to the underlying MPI function.
|
||||
|
||||
@@ -313,8 +316,8 @@ during the function wrapping before being passed along to the underlying MPI fun
|
||||
Post-installation steps
|
||||
========================================
|
||||
|
||||
After installation, you can optionally configure the Omnitrace environment.
|
||||
You should also test the executables to confirm Omnitrace is correctly installed.
|
||||
After installation, you can optionally configure the ROCm Systems Profiler environment.
|
||||
You should also test the executables to confirm ROCm Systems Profiler is correctly installed.
|
||||
|
||||
Configure the environment
|
||||
-----------------------------------
|
||||
@@ -323,14 +326,14 @@ If environment modules are available and preferred, add them using these command
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
module use /opt/omnitrace/share/modulefiles
|
||||
module load omnitrace/1.0.0
|
||||
module use /opt/rocprofiler-systems/share/modulefiles
|
||||
module load rocprofiler-systems/1.0.0
|
||||
|
||||
Alternatively, you can directly source the ``setup-env.sh`` script:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
source /opt/omnitrace/share/omnitrace/setup-env.sh
|
||||
source /opt/rocprofiler-systems/share/rocprofiler-systems/setup-env.sh
|
||||
|
||||
Test the executables
|
||||
-----------------------------------
|
||||
@@ -340,8 +343,8 @@ issues locating the installed libraries:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
omnitrace-instrument --help
|
||||
omnitrace-avail --help
|
||||
rocprof-sys-instrument --help
|
||||
rocprof-sys-avail --help
|
||||
|
||||
.. note::
|
||||
|
||||
@@ -353,27 +356,27 @@ issues locating the installed libraries:
|
||||
Post-installation troubleshooting
|
||||
========================================
|
||||
|
||||
This section explains how to resolve certain issues that might happen when you first use Omnitrace.
|
||||
This section explains how to resolve certain issues that might happen when you first use ROCm Systems Profiler.
|
||||
|
||||
Issues with RHEL and SELinux
|
||||
----------------------------------------------------
|
||||
|
||||
RHEL (Red Hat Enterprise Linux) and related distributions of Linux automatically enable a security feature
|
||||
named SELinux (Security-Enhanced Linux) that prevents Omnitrace from running.
|
||||
named SELinux (Security-Enhanced Linux) that prevents ROCm Systems Profiler from running.
|
||||
This issue applies to any Linux distribution with SELinux installed, including RHEL,
|
||||
CentOS, Fedora, and Rocky Linux. The problem can happen with any GPU, or even without a GPU.
|
||||
|
||||
The problem occurs after you instrument a program and try to
|
||||
run ``omnitrace-run`` with the instrumented program.
|
||||
run ``rocprof-sys-run`` with the instrumented program.
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
g++ hello.cpp -o hello
|
||||
omniperf-instrument -M sampling -o hello.instr -- ./hello
|
||||
omnitrace-run -- ./hello.instr
|
||||
rocprof-sys-instrument -M sampling -o hello.instr -- ./hello
|
||||
rocprof-sys-run -- ./hello.instr
|
||||
|
||||
Instead of successfully running the binary with call-stack sampling,
|
||||
Omnitrace crashes with a segmentation fault.
|
||||
ROCm Systems Profiler crashes with a segmentation fault.
|
||||
|
||||
.. note::
|
||||
|
||||
@@ -412,4 +415,4 @@ Configuring PAPI to collect hardware counters
|
||||
|
||||
To use PAPI to collect the majority of hardware counters, ensure
|
||||
the ``/proc/sys/kernel/perf_event_paranoid`` setting has a value less than or equal to ``2``.
|
||||
For more information, see the :ref:`omnitrace_papi_events` section.
|
||||
For more information, see the :ref:`rocprof-sys_papi_events` section.
|
||||
@@ -1,21 +1,22 @@
|
||||
.. meta::
|
||||
:description: Omnitrace documentation and reference
|
||||
:keywords: Omnitrace, ROCm, profiler, tracking, visualization, tool, Instinct, accelerator, AMD
|
||||
:description: ROCm Systems Profiler quick start documentation and reference
|
||||
:keywords: rocprof-sys, rocprofiler-systems, Omnitrace, ROCm, profiler, quick start, getting started, quick install, tracking, visualization, tool, Instinct, accelerator, AMD
|
||||
|
||||
*************************************
|
||||
Omnitrace quick start
|
||||
ROCm Systems Profiler quick start
|
||||
*************************************
|
||||
|
||||
To install Omnitrace, download the `Omnitrace installer <https://github.com/ROCm/omnitrace/releases/latest/download/omnitrace-install.py>`_
|
||||
and specify ``--prefix <install-directory>``. The script attempts to auto-detect
|
||||
the appropriate OS distribution and version. To include AMD ROCm Software support,
|
||||
To install ROCm Systems Profiler, download the
|
||||
`ROCm Systems Profiler installer <https://github.com/ROCm/rocprofiler-systems/releases/latest/download/rocprofiler-systems-install.py>`_
|
||||
and specify ``--prefix <install-directory>``. The script attempts to auto-detect
|
||||
the appropriate OS distribution and version. To include AMD ROCm Software support,
|
||||
specify ``--rocm X.Y``, where ``X`` is the ROCm major
|
||||
version and ``Y`` is the ROCm minor version, for example, ``--rocm 6.2``.
|
||||
version and ``Y`` is the ROCm minor version, for example, ``--rocm 6.3``.
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
wget https://github.com/ROCm/omnitrace/releases/latest/download/omnitrace-install.py
|
||||
python3 ./omnitrace-install.py --prefix /opt/omnitrace --rocm 6.2
|
||||
wget https://github.com/ROCm/rocprofiler-systems/releases/latest/download/rocprofiler-systems-install.py
|
||||
python3 ./rocprofiler-systems-install.py --prefix /opt/rocprofiler-systems --rocm 6.3
|
||||
|
||||
This script supports installation on Ubuntu, OpenSUSE, Red Hat, Debian, CentOS, and Fedora.
|
||||
If the target OS is compatible with one of the operating system versions listed in
|
||||
@@ -23,8 +24,28 @@ the comprehensive :doc:`Installation guidelines <./install>`,
|
||||
specify ``-d <DISTRO> -v <VERSION>``. For example, if the OS is compatible with Ubuntu 22.04, pass
|
||||
``-d ubuntu -v 22.04`` to the script.
|
||||
|
||||
.. note::
|
||||
Install via package manager
|
||||
============================
|
||||
|
||||
If you have ROCm version 6.2 or higher installed, you can use the
|
||||
package manager to install a pre-built copy of Omnitrace using
|
||||
``apt install omnitrace`` or ``dnf install omnitrace``.
|
||||
If you have ROCm version 6.3 or higher installed, you can use the
|
||||
package manager to install a pre-built copy of ROCm Systems Profiler.
|
||||
|
||||
.. tab-set::
|
||||
|
||||
.. tab-item:: Ubuntu
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
$ sudo apt install rocprofiler-systems
|
||||
|
||||
.. tab-item:: Red Hat Enterprise Linux
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
$ sudo dnf install rocprofiler-systems
|
||||
|
||||
.. tab-item:: SUSE Linux Enterprise Server
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
$ sudo zypper install rocprofiler-systems
|
||||
|
||||
@@ -1,122 +1,122 @@
|
||||
.. meta::
|
||||
:description: Omnitrace documentation and reference
|
||||
:keywords: Omnitrace, ROCm, profiler, tracking, visualization, tool, Instinct, accelerator, AMD
|
||||
:description: ROCm Systems Profiler development documentation and reference
|
||||
:keywords: rocprof-sys, rocprofiler-systems, Omnitrace, ROCm, development, developers guide, profiler, tracking, visualization, tool, Instinct, accelerator, AMD
|
||||
|
||||
****************************************************
|
||||
Development guide
|
||||
****************************************************
|
||||
|
||||
This guide discusses the `Omnitrace <https://github.com/ROCm/omnitrace>`_ design.
|
||||
It includes a list of the executables and libraries, along with a discussion of the application's
|
||||
This guide discusses the `ROCm Systems Profiler <https://github.com/ROCm/rocprofiler-systems>`_ design.
|
||||
It includes a list of the executables and libraries, along with a discussion of the application's
|
||||
memory, sampling, and time-window constraint models.
|
||||
|
||||
Executables
|
||||
========================================
|
||||
|
||||
This section lists the Omnitrace executables.
|
||||
This section lists the ROCm Systems Profiler executables.
|
||||
|
||||
omnitrace-avail: `source/bin/omnitrace-avail <https://github.com/ROCm/omnitrace/tree/amd-mainline/source/bin/omnitrace-avail>`_
|
||||
rocprof-sys-avail: `source/bin/rocprof-sys-avail <https://github.com/ROCm/rocprofiler-systems/tree/amd-mainline/source/bin/rocprof-sys-avail>`_
|
||||
-------------------------------------------------------------------------------------------------------------------------------
|
||||
|
||||
The ``main`` routine of ``omnitrace-avail`` has three important sections:
|
||||
The ``main`` routine of ``rocprof-sys-avail`` has three important sections:
|
||||
|
||||
* Printing components
|
||||
* Printing options
|
||||
* Printing hardware counters
|
||||
|
||||
omnitrace-sample: `source/bin/omnitrace-sample <https://github.com/ROCm/omnitrace/tree/amd-mainline/source/bin/omnitrace-sample>`_
|
||||
rocprof-sys-sample: `source/bin/rocprof-sys-sample <https://github.com/ROCm/rocprofiler-systems/tree/amd-mainline/source/bin/rocprof-sys-sample>`_
|
||||
----------------------------------------------------------------------------------------------------------------------------------
|
||||
|
||||
* Requires a command-line format of ``omnitrace-sample <options> -- <command> <command-args>``
|
||||
* Requires a command-line format of ``rocprof-sys-sample <options> -- <command> <command-args>``
|
||||
* Translates command-line options into environment variables
|
||||
* Adds ``libomnitrace-dl.so`` to ``LD_PRELOAD``
|
||||
* Adds ``librocprof-sys-dl.so`` to ``LD_PRELOAD``
|
||||
* Is launched by using ``execvpe`` with ``<command> <command-args>`` and a modified environment
|
||||
|
||||
omnitrace-casual: `source/bin/omnitrace-causal <https://github.com/ROCm/omnitrace/tree/amd-mainline/source/bin/omnitrace-causal>`_
|
||||
rocprof-sys-casual: `source/bin/rocprof-sys-causal <https://github.com/ROCm/rocprofiler-systems/tree/amd-mainline/source/bin/rocprof-sys-causal>`_
|
||||
----------------------------------------------------------------------------------------------------------------------------------
|
||||
|
||||
When there is exactly one causal profiling configuration variant (which enables debugging),
|
||||
``omnitrace-casual`` has a nearly identical design to ``omnitrace-sample``
|
||||
``rocprof-sys-casual`` has a nearly identical design to ``rocprof-sys-sample``
|
||||
|
||||
When the command-line options produce more than one causal profiling configuration variant,
|
||||
the following actions take place for each variant:
|
||||
|
||||
* ``omnitrace-causal`` calls ``fork()``
|
||||
* ``rocprof-sys-causal`` calls ``fork()``
|
||||
* the child process launches ``<command> <command-args>`` using ``execvpe``, which modifies the environment for the variant
|
||||
* the parent process waits for the child process to finish
|
||||
|
||||
omnitrace-instrument: `source/bin/omnitrace-instrument <https://github.com/ROCm/omnitrace/tree/amd-mainline/source/bin/omnitrace-instrument>`_
|
||||
rocprof-sys-instrument: `source/bin/rocprof-sys-instrument <https://github.com/ROCm/rocprofiler-systems/tree/amd-mainline/source/bin/rocprof-sys-instrument>`_
|
||||
----------------------------------------------------------------------------------------------------------------------------------------------
|
||||
|
||||
* Requires a command-line format of ``omnitrace-instrument <options> -- <command> <command-args>``
|
||||
* Allows the user to provide options specifying whether to perform runtime instrumentation, use binary rewrite, or
|
||||
* Requires a command-line format of ``rocprof-sys-instrument <options> -- <command> <command-args>``
|
||||
* Allows the user to provide options specifying whether to perform runtime instrumentation, use binary rewrite, or
|
||||
attach to process
|
||||
* Either opens the instrumentation target (for binary rewrite), launches the target and stops it
|
||||
before it starts executing ``main``, or attaches to a running executable and pauses it
|
||||
* Finds all functions in the targets
|
||||
* Finds ``libomnitrace-dl`` and locates the functions
|
||||
* Iterates over and instruments all the functions, provided they satisfy the
|
||||
* Finds ``librocprof-sys-dl`` and locates the functions
|
||||
* Iterates over and instruments all the functions, provided they satisfy the
|
||||
defined criteria (such as a minimum number of instructions)
|
||||
|
||||
* See the ``module_function`` class
|
||||
|
||||
* Until this point, the workflow has been the same for the different options,
|
||||
* Until this point, the workflow has been the same for the different options,
|
||||
but it diverges after instrumentation is complete:
|
||||
|
||||
* For a binary rewrite: it produces a new instrumented binary and exits
|
||||
* For runtime instrumentation or attaching to a process: it instructs the application
|
||||
* For runtime instrumentation or attaching to a process: it instructs the application
|
||||
to resume and then waits for it to exit
|
||||
|
||||
Libraries
|
||||
========================================
|
||||
|
||||
Common library: `source/lib/common <https://github.com/ROCm/omnitrace/tree/amd-mainline/source/lib/common>`_
|
||||
Common library: `source/lib/common <https://github.com/ROCm/rocprofiler-systems/tree/amd-mainline/source/lib/common>`_
|
||||
--------------------------------------------------------------------------------------------------------------------------------
|
||||
|
||||
* General header-only functionality used in multiple executables and/or libraries.
|
||||
* General header-only functionality used in multiple executables and/or libraries.
|
||||
* Not installed or exported outside of the build tree.
|
||||
|
||||
Core library: `source/lib/core <https://github.com/ROCm/omnitrace/tree/amd-mainline/source/lib/core>`_
|
||||
Core library: `source/lib/core <https://github.com/ROCm/rocprofiler-systems/tree/amd-mainline/source/lib/core>`_
|
||||
--------------------------------------------------------------------------------------------------------------------------------
|
||||
|
||||
* Static PIC library with functionality that does not depend on any components.
|
||||
* Static PIC library with functionality that does not depend on any components.
|
||||
* Not installed or exported outside of the build tree.
|
||||
|
||||
Binary library: `source/lib/binary <https://github.com/ROCm/omnitrace/tree/amd-mainline/source/lib/binary>`_
|
||||
Binary library: `source/lib/binary <https://github.com/ROCm/rocprofiler-systems/tree/amd-mainline/source/lib/binary>`_
|
||||
--------------------------------------------------------------------------------------------------------------------------------
|
||||
|
||||
* Static PIC library with functionality for reading/analyzing binary info.
|
||||
* Mostly used by the causal profiling sections of ``libomnitrace``.
|
||||
* Mostly used by the causal profiling sections of ``librocprof-sys``.
|
||||
* Not installed or exported outside of the build tree.
|
||||
|
||||
libomnitrace: `source/lib/omnitrace <https://github.com/ROCm/omnitrace/tree/amd-mainline/source/lib/omnitrace>`_
|
||||
librocprof-sys: `source/lib/rocprof-sys <https://github.com/ROCm/rocprofiler-systems/tree/amd-mainline/source/lib/rocprof-sys>`_
|
||||
--------------------------------------------------------------------------------------------------------------------------------
|
||||
|
||||
This is the main library encapsulating all the capabilities.
|
||||
|
||||
libomnitrace-dl: `source/lib/omnitrace-dl <https://github.com/ROCm/omnitrace/tree/amd-mainline/source/lib/omnitrace-dl>`_
|
||||
librocprof-sys-dl: `source/lib/rocprof-sys-dl <https://github.com/ROCm/rocprofiler-systems/tree/amd-mainline/source/lib/rocprof-sys-dl>`_
|
||||
--------------------------------------------------------------------------------------------------------------------------------
|
||||
|
||||
This is a lightweight, front-end library for ``libomnitrace`` which serves three primary purposes:
|
||||
This is a lightweight, front-end library for ``librocprof-sys`` which serves three primary purposes:
|
||||
|
||||
* Dramatically speeds up instrumentation time compared to using ``libomnitrace`` directly because
|
||||
Dyninst must parse the entire library in order to find the instrumentation functions
|
||||
(a ``dlopen`` call is made on ``libomnitrace`` when the instrumentation functions get called)
|
||||
* Prevents re-entry if ``libomnitrace`` calls an instrumented function internally
|
||||
* Coordinates communication between ``libomnitrace-user`` and ``libomnitrace``
|
||||
* Dramatically speeds up instrumentation time compared to using ``librocprof-sys`` directly because
|
||||
Dyninst must parse the entire library in order to find the instrumentation functions
|
||||
(a ``dlopen`` call is made on ``librocprof-sys`` when the instrumentation functions get called)
|
||||
* Prevents re-entry if ``librocprof-sys`` calls an instrumented function internally
|
||||
* Coordinates communication between ``librocprof-sys-user`` and ``librocprof-sys``
|
||||
|
||||
libomnitrace-user: `source/lib/omnitrace-user <https://github.com/ROCm/omnitrace/tree/amd-mainline/source/lib/omnitrace-user>`_
|
||||
librocprof-sys-user: `source/lib/rocprof-sys-user <https://github.com/ROCm/rocprofiler-systems/tree/amd-mainline/source/lib/rocprof-sys-user>`_
|
||||
--------------------------------------------------------------------------------------------------------------------------------
|
||||
|
||||
* Provides a set of functions and types for the users to add to their code, for example,
|
||||
disabling data collection globally or on a specific thread or
|
||||
user-defined region
|
||||
* If ``libomnitrace-dl`` is not loaded, the user API is effectively a set of no-op function calls.
|
||||
* If ``librocprof-sys-dl`` is not loaded, the user API is effectively a set of no-op function calls.
|
||||
|
||||
Testing tools
|
||||
========================================
|
||||
|
||||
* `CDash Testing Dashboard <https://my.cdash.org/index.php?project=Omnitrace>`_ (requires a login)
|
||||
* `CDash Testing Dashboard <https://my.cdash.org/index.php?project=rocprofiler-systems>`_ (requires a login)
|
||||
|
||||
Components
|
||||
========================================
|
||||
@@ -124,34 +124,34 @@ Components
|
||||
Most measurements and capabilities are encapsulated into a "component" with the following definitions:
|
||||
|
||||
Measurement
|
||||
A recording of some data relevant to performance, for instance, the current call-stack,
|
||||
A recording of some data relevant to performance, for instance, the current call-stack,
|
||||
hardware counter values, current memory usage, or timestamp
|
||||
|
||||
Capability
|
||||
Handles the implementation or orchestration of some feature which is used
|
||||
to collect measurements, for example, a component which handles setting up function wrappers
|
||||
Handles the implementation or orchestration of some feature which is used
|
||||
to collect measurements, for example, a component which handles setting up function wrappers
|
||||
around various functions such as ``pthread_create`` or ``MPI_Init``.
|
||||
|
||||
Components are designed to either hold no data at all or only the data for both an instantaneous
|
||||
Components are designed to either hold no data at all or only the data for both an instantaneous
|
||||
measurement and a phase measurement.
|
||||
|
||||
Components which store data typically implement a static ``record()`` function
|
||||
Components which store data typically implement a static ``record()`` function
|
||||
for getting a record of the measurement,
|
||||
``start()`` and ``stop()`` member functions for calculating a phase measurement,
|
||||
``start()`` and ``stop()`` member functions for calculating a phase measurement,
|
||||
and a ``sample()`` member function for storing an
|
||||
instantaneous measurement. In reality, there are several more "standard" functions
|
||||
instantaneous measurement. In reality, there are several more "standard" functions
|
||||
but these are the most commonly-used ones.
|
||||
|
||||
Components which do not store data might also have ``start()``, ``stop()``, and ``sample()``
|
||||
Components which do not store data might also have ``start()``, ``stop()``, and ``sample()``
|
||||
functions. However, components which
|
||||
implement function wrappers typically provide a call operator or ``audit(...)``
|
||||
implement function wrappers typically provide a call operator or ``audit(...)``
|
||||
functions. These are invoked with the
|
||||
wrapped function's arguments before the wrapped function gets called and with the return value
|
||||
wrapped function's arguments before the wrapped function gets called and with the return value
|
||||
after the wrapped function gets called.
|
||||
|
||||
.. note::
|
||||
|
||||
The goal of this design is to provide relatively small and resuable lightweight objects
|
||||
The goal of this design is to provide relatively small and resuable lightweight objects
|
||||
for recording measurements and implementing capabilities.
|
||||
|
||||
Wall-clock component example
|
||||
@@ -195,7 +195,7 @@ A component for computing the elapsed wall-clock time looks like this:
|
||||
Function wrapper component example
|
||||
--------------------------------------
|
||||
|
||||
A component which implements wrappers around ``fork()`` and ``exit(int)`` (and stores no data)
|
||||
A component which implements wrappers around ``fork()`` and ``exit(int)`` (and stores no data)
|
||||
could look like this:
|
||||
|
||||
.. code-block:: cpp
|
||||
@@ -219,7 +219,7 @@ could look like this:
|
||||
void operator()(const gotcha_data&, void (*real_exit)(int), int _exit_code)
|
||||
{
|
||||
// catch the call to exit and finalize before truly exiting
|
||||
omnitrace_finalize();
|
||||
rocprofsys_finalize();
|
||||
|
||||
real_exit(_exit_code);
|
||||
}
|
||||
@@ -298,22 +298,22 @@ Collected data is generally handled in one of the three following ways:
|
||||
* It is managed implicitly by Timemory and accessed as needed
|
||||
* As thread-local data
|
||||
|
||||
In general, only instrumentation for relatively simple data is directly passed to
|
||||
In general, only instrumentation for relatively simple data is directly passed to
|
||||
Perfetto and/or Timemory during runtime.
|
||||
For example, the callbacks from binary instrumentation, user API instrumentation,
|
||||
For example, the callbacks from binary instrumentation, user API instrumentation,
|
||||
and roctracer directly invoke
|
||||
calls to Perfetto or Timemory's storage model. Otherwise, the data is stored
|
||||
by Omnitrace in the thread-data model
|
||||
calls to Perfetto or Timemory's storage model. Otherwise, the data is stored
|
||||
by ROCm Systems Profiler in the thread-data model
|
||||
which is more persistent than simply using ``thread_local`` static data, which gets deleted
|
||||
when the thread stops.
|
||||
|
||||
Thread identification
|
||||
--------------------------------------
|
||||
|
||||
Each CPU thread is assigned two integral identifiers. One identifier, the ``internal_value``, is
|
||||
Each CPU thread is assigned two integral identifiers. One identifier, the ``internal_value``, is
|
||||
atomically incremented every time a new thread is created.
|
||||
The other identifier, known as the ``sequent_value``, tries to account for the fact that Omnitrace, Perfetto, ROCm, and other applications
|
||||
start background threads. When a thread is created as a by-product of Omnitrace,
|
||||
The other identifier, known as the ``sequent_value``, tries to account for the fact that ROCm Systems Profiler, Perfetto, ROCm, and other applications
|
||||
start background threads. When a thread is created as a by-product of ROCm Systems Profiler,
|
||||
the index is offset by a large value. This serves
|
||||
two purposes:
|
||||
|
||||
@@ -325,88 +325,88 @@ The ``sequent_value`` identifier is typically used to access the thread-data.
|
||||
Thread-data class
|
||||
--------------------------------------
|
||||
|
||||
Currently, most thread data is effectively stored in a static
|
||||
``std::array<std::unique_ptr<T>, OMNITRACE_MAX_THREADS>`` instance.
|
||||
``OMNITRACE_MAX_THREADS`` is a value defined a compile-time and set to ``2048``
|
||||
Currently, most thread data is effectively stored in a static
|
||||
``std::array<std::unique_ptr<T>, ROCPROFSYS_MAX_THREADS>`` instance.
|
||||
``ROCPROFSYS_MAX_THREADS`` is a value defined a compile-time and set to ``2048``
|
||||
for release builds. During finalization,
|
||||
Omnitrace iterates through the thread-data and transforms that data
|
||||
ROCm Systems Profiler iterates through the thread-data and transforms that data
|
||||
into something that can be passed along to Perfetto and/or Timemory.
|
||||
The downside of the current model is that if the user exceeds ``OMNITRACE_MAX_THREADS``,
|
||||
The downside of the current model is that if the user exceeds ``ROCPROFSYS_MAX_THREADS``,
|
||||
a segmentation fault occurs. To fix this issue,
|
||||
a new model is being adopted which has all the benefits of this model
|
||||
a new model is being adopted which has all the benefits of this model
|
||||
but permits dynamic expansion.
|
||||
|
||||
Sampling model
|
||||
========================================
|
||||
|
||||
The general structure for the sampling is within Timemory (``source/timemory/sampling``).
|
||||
The general structure for the sampling is within Timemory (``source/timemory/sampling``).
|
||||
Currently, all sampling is done per-thread
|
||||
via POSIX timers. Omnitrace supports both a real-time timer and a CPU-time timer.
|
||||
via POSIX timers. ROCm Systems Profiler supports both a real-time timer and a CPU-time timer.
|
||||
Both have adjustable frequencies, delays, and durations.
|
||||
By default, only CPU-time sampling is enabled. Initial settings are inherited from
|
||||
the settings starting with ``OMNITRACE_SAMPLING_``.
|
||||
By default, only CPU-time sampling is enabled. Initial settings are inherited from
|
||||
the settings starting with ``ROCPROFSYS_SAMPLING_``.
|
||||
|
||||
For each type of timer, timer-specific settings can be used to
|
||||
override the common and inherited timer settings.
|
||||
These settings begin with ``OMNITRACE_SAMPLING_CPUTIME`` for the CPU-time sampler
|
||||
and ``OMNITRACE_SAMPLING_REALTIME`` for
|
||||
the real-time sampler. For example, ``OMNITRACE_SAMPLING_FREQ=500`` initially sets the
|
||||
sampling frequency to 500 interrupts per second. Adding the setting ``OMNITRACE_SAMPLING_REALTIME_FREQ=10``
|
||||
For each type of timer, timer-specific settings can be used to
|
||||
override the common and inherited timer settings.
|
||||
These settings begin with ``ROCPROFSYS_SAMPLING_CPUTIME`` for the CPU-time sampler
|
||||
and ``ROCPROFSYS_SAMPLING_REALTIME`` for
|
||||
the real-time sampler. For example, ``ROCPROFSYS_SAMPLING_FREQ=500`` initially sets the
|
||||
sampling frequency to 500 interrupts per second. Adding the setting ``ROCPROFSYS_SAMPLING_REALTIME_FREQ=10``
|
||||
lowers the sampling frequency for the real-time sampler
|
||||
to 10 interrupts per second of real-time.
|
||||
|
||||
The Omnitrace-specific implementation can be found in
|
||||
`source/lib/omnitrace/library/sampling.cpp <https://github.com/ROCm/omnitrace/blob/main/source/lib/omnitrace/library/sampling.cpp>`_.
|
||||
Within `sampling.cpp <https://github.com/ROCm/omnitrace/blob/main/source/lib/omnitrace/library/sampling.cpp>`_,
|
||||
The ROCm Systems Profiler-specific implementation can be found in
|
||||
`source/lib/rocprof-sys/library/sampling.cpp <https://github.com/ROCm/rocprofiler-systems/blob/main/source/lib/rocprof-sys/library/sampling.cpp>`_.
|
||||
Within `sampling.cpp <https://github.com/ROCm/rocprofiler-systems/blob/main/source/lib/rocprof-sys/library/sampling.cpp>`_,
|
||||
there is a bundle of three sampling components:
|
||||
|
||||
* `backtrace_timestamp <https://github.com/ROCm/omnitrace/blob/main/source/lib/omnitrace/library/components/backtrace_timestamp.hpp>`_ simply
|
||||
* `backtrace_timestamp <https://github.com/ROCm/rocprofiler-systems/blob/main/source/lib/rocprof-sys/library/components/backtrace_timestamp.hpp>`_ simply
|
||||
records the wall-clock time of the sample.
|
||||
* `backtrace <https://github.com/ROCm/omnitrace/blob/main/source/lib/omnitrace/library/components/backtrace.hpp>`_
|
||||
* `backtrace <https://github.com/ROCm/rocprofiler-systems/blob/main/source/lib/rocprof-sys/library/components/backtrace.hpp>`_
|
||||
records the call-stack via libunwind.
|
||||
* `backtrace_metrics <https://github.com/ROCm/omnitrace/blob/main/source/lib/omnitrace/library/components/backtrace_metrics.hpp>`_
|
||||
* `backtrace_metrics <https://github.com/ROCm/rocprofiler-systems/blob/main/source/lib/rocprof-sys/library/components/backtrace_metrics.hpp>`_
|
||||
records the sample metrics, such as peak RSS and the hardware counters.
|
||||
|
||||
These three components are bundled together in
|
||||
These three components are bundled together in
|
||||
a tuple-like ``struct`` (``tuple<backtrace_timestamp, backtrace, backtrace_metrics>``).
|
||||
A buffer of at least 1024 instances of this tuple is mapped using ``mmap``
|
||||
per-thread. When this buffer is full,
|
||||
A buffer of at least 1024 instances of this tuple is mapped using ``mmap``
|
||||
per-thread. When this buffer is full,
|
||||
the sampler hands the buffer off to its allocator thread and maps a new buffer with ``mmap``
|
||||
before taking the next sample. The allocator thread takes this data
|
||||
and either dynamically stores it in memory or writes it to a file depending on the
|
||||
value of ``OMNITRACE_USE_TEMPORARY_FILES``.
|
||||
This schema avoids all allocations in the signal handler, lets the data grow
|
||||
dynamically, avoids potentially slow I/O within the signal handler, and also enables
|
||||
before taking the next sample. The allocator thread takes this data
|
||||
and either dynamically stores it in memory or writes it to a file depending on the
|
||||
value of ``ROCPROFSYS_USE_TEMPORARY_FILES``.
|
||||
This schema avoids all allocations in the signal handler, lets the data grow
|
||||
dynamically, avoids potentially slow I/O within the signal handler, and also enables
|
||||
the capability of avoiding I/O altogether.
|
||||
The maximum number of samplers handled by each allocator is governed by the
|
||||
``OMNITRACE_SAMPLING_ALLOCATOR_SIZE`` setting (the default is eight). Whenever an allocator
|
||||
The maximum number of samplers handled by each allocator is governed by the
|
||||
``ROCPROFSYS_SAMPLING_ALLOCATOR_SIZE`` setting (the default is eight). Whenever an allocator
|
||||
has reached its limit,
|
||||
a new internal thread is created to handle the new samplers.
|
||||
|
||||
Time-window constraint model
|
||||
========================================
|
||||
|
||||
With the recent introduction of tracing delay and duration, the
|
||||
`constraint namespace <https://github.com/ROCm/omnitrace/blob/main/source/lib/core/constraint.hpp>`_
|
||||
was introduced to improve the management of delays and duration limits for
|
||||
With the recent introduction of tracing delay and duration, the
|
||||
`constraint namespace <https://github.com/ROCm/rocprofiler-systems/blob/main/source/lib/core/constraint.hpp>`_
|
||||
was introduced to improve the management of delays and duration limits for
|
||||
data collection. The ``spec`` class accepts a clock identifier, a delay value, a duration value, and an
|
||||
integer indicating how many times to repeat the delay and duration cycle. It is therefore
|
||||
integer indicating how many times to repeat the delay and duration cycle. It is therefore
|
||||
possible to perform tasks such as periodically enabling tracing for brief periods
|
||||
of time in between long periods without data collection while the application runs. The
|
||||
syntax follows the format ``clock_identifier:delay:capture_duration:cycles``, so a value of
|
||||
syntax follows the format ``clock_identifier:delay:capture_duration:cycles``, so a value of
|
||||
``10:1:3`` for the last three parameters represents the following sequence of operations:
|
||||
|
||||
* Ten seconds where no data is collected, then one second where it is
|
||||
* Ten seconds where no data is collected, then one second where it is
|
||||
* Ten seconds where no data is collected, then one second where it is
|
||||
* Ten seconds where no data is collected, then one second where it is
|
||||
* Ten seconds where no data is collected, then one second where it is
|
||||
* Stop
|
||||
|
||||
As another example, ``OMNITRACE_TRACE_PERIODS = realtime:10:1:5 process_cputime:10:2:20`` translates
|
||||
As another example, ``ROCPROFSYS_TRACE_PERIODS = realtime:10:1:5 process_cputime:10:2:20`` translates
|
||||
to this sequence:
|
||||
|
||||
* Five cycles of: no data collection for ten seconds of real-time followed by one second of data collection
|
||||
* Twenty cycles of: no data collection for ten seconds of process CPU time followed by two CPU-time seconds of data collection
|
||||
|
||||
Eventually, the goal is to migrate all subsets of data collection which currently support
|
||||
Eventually, the goal is to migrate all subsets of data collection which currently support
|
||||
more rudimentary models of time window constraints, such as process sampling and causal profiling,
|
||||
to this model.
|
||||
|
||||
@@ -1,40 +1,40 @@
|
||||
.. meta::
|
||||
:description: Omnitrace documentation and reference
|
||||
:keywords: Omnitrace, ROCm, profiler, tracking, visualization, tool, Instinct, accelerator, AMD
|
||||
:description: ROCm Systems Profiler glossary and reference
|
||||
:keywords: rocprof-sys, rocprofiler-systems, Omnitrace, ROCm, glossary, terminology, profiler, tracking, visualization, tool, Instinct, accelerator, AMD
|
||||
|
||||
*******************
|
||||
Omnitrace Glossary
|
||||
ROCm Systems Profiler Glossary
|
||||
*******************
|
||||
|
||||
This topic explains the terminology necessary to use Omnitrace.
|
||||
The list below provides a basic glossary for those who
|
||||
are new to binary instrumentation. It also clarifies ambiguities
|
||||
when certain terms have different
|
||||
contextual meanings, for example, the Omnitrace meaning of the term "module"
|
||||
This topic explains the terminology necessary to use ROCm Systems Profiler.
|
||||
The list below provides a basic glossary for those who
|
||||
are new to binary instrumentation. It also clarifies ambiguities
|
||||
when certain terms have different
|
||||
contextual meanings, for example, the ROCm Systems Profiler meaning of the term "module"
|
||||
when instrumenting Python.
|
||||
|
||||
**Binary**
|
||||
A file written in the Executable and Linkable Format (ELF). This is the standard file
|
||||
A file written in the Executable and Linkable Format (ELF). This is the standard file
|
||||
format for executable files, shared libraries, etc.
|
||||
|
||||
**Binary instrumentation**
|
||||
Inserting callbacks to instrumentation into an existing binary. This can be performed
|
||||
Inserting callbacks to instrumentation into an existing binary. This can be performed
|
||||
statically or dynamically.
|
||||
|
||||
**Static binary instrumentation**
|
||||
Loads an existing binary, determines instrumentation points, and generates a new binary
|
||||
with instrumentation directly embedded. It is applicable to executables and libraries but
|
||||
Loads an existing binary, determines instrumentation points, and generates a new binary
|
||||
with instrumentation directly embedded. It is applicable to executables and libraries but
|
||||
limited to only the functions defined in the binary. This is also known as **Binary rewrite**.
|
||||
|
||||
**Dynamic binary instrumentation**
|
||||
Loads an existing binary into memory, inserts instrumentation, and runs the binary.
|
||||
It is limited to executables but is capable of instrumenting linked libraries.
|
||||
Loads an existing binary into memory, inserts instrumentation, and runs the binary.
|
||||
It is limited to executables but is capable of instrumenting linked libraries.
|
||||
This is also known as **Runtime instrumentation**.
|
||||
|
||||
**Statistical sampling**
|
||||
At periodic intervals, the application is paused and the current call-stack of the CPU
|
||||
is recorded along with various other metrics. It uses timers that measure either
|
||||
(A) real clock time or (B) the CPU time used by the current thread and the CPU time
|
||||
**Statistical sampling**
|
||||
At periodic intervals, the application is paused and the current call-stack of the CPU
|
||||
is recorded along with various other metrics. It uses timers that measure either
|
||||
(A) real clock time or (B) the CPU time used by the current thread and the CPU time
|
||||
expended on behalf of the thread by the system. This is also known as simply **sampling**.
|
||||
|
||||
**Sampling rate**
|
||||
@@ -45,12 +45,12 @@ when instrumenting Python.
|
||||
* How long to wait before (A) and (B) begin triggering at their designated rate
|
||||
|
||||
**Sampling duration**
|
||||
* The amount of time (in real-time) after the start of the application to record samples.
|
||||
* The amount of time (in real-time) after the start of the application to record samples.
|
||||
* After this time limit has been reached, no more samples are recorded.
|
||||
|
||||
**Process sampling**
|
||||
At periodic (real-time) intervals, a background thread records global metrics without
|
||||
interrupting the current process. These metrics include, but are not limited to:
|
||||
At periodic (real-time) intervals, a background thread records global metrics without
|
||||
interrupting the current process. These metrics include, but are not limited to:
|
||||
CPU frequency, CPU memory high-water mark (i.e. peak memory usage), GPU temperature,
|
||||
and GPU power usage.
|
||||
|
||||
@@ -62,41 +62,41 @@ when instrumenting Python.
|
||||
* How long to wait (in real-time) before recording samples
|
||||
|
||||
**Sampling duration**
|
||||
* The amount of time (in real-time) after the start of the application to record samples.
|
||||
* The amount of time (in real-time) after the start of the application to record samples.
|
||||
* After this time limit has been reached, no more samples are recorded.
|
||||
|
||||
**Module**
|
||||
With respect to binary instrumentation, a module is defined as either the filename
|
||||
(such as ``foo.c``) or library name (``libfoo.so``) which contains the definition
|
||||
With respect to binary instrumentation, a module is defined as either the filename
|
||||
(such as ``foo.c``) or library name (``libfoo.so``) which contains the definition
|
||||
of one or more functions.
|
||||
|
||||
With respect to Python instrumentation, a module is defined as the **file** which contains
|
||||
the definition of one or more functions. The full path to this file typically contains the
|
||||
With respect to Python instrumentation, a module is defined as the **file** which contains
|
||||
the definition of one or more functions. The full path to this file typically contains the
|
||||
name of the "Python module".
|
||||
|
||||
**Basic block**
|
||||
A straight-line code sequence with no branches in (except for the entry) and
|
||||
A straight-line code sequence with no branches in (except for the entry) and
|
||||
no branches out (except for the exit).
|
||||
|
||||
**Address range**
|
||||
The instructions for a function in a binary start at certain address with the ELF file
|
||||
The instructions for a function in a binary start at certain address with the ELF file
|
||||
and end at a certain address. The range is ``end - start``.
|
||||
|
||||
The address range is a decent approximation for the "cost" of a function.
|
||||
The address range is a decent approximation for the "cost" of a function.
|
||||
For example, a larger address range approximately equates to more instructions.
|
||||
|
||||
**Instrumentation traps**
|
||||
On the x86 architecture, because instructions are of variable size, an instruction
|
||||
might be too small for Dyninst to replace it with the normal code sequence
|
||||
used to call instrumentation. When instrumentation is placed at points other
|
||||
than subroutine entry, exit, or call points, traps may be used to ensure
|
||||
the instrumentation fits. (By default, ``omnitrace-instrument`` avoids instrumentation
|
||||
On the x86 architecture, because instructions are of variable size, an instruction
|
||||
might be too small for Dyninst to replace it with the normal code sequence
|
||||
used to call instrumentation. When instrumentation is placed at points other
|
||||
than subroutine entry, exit, or call points, traps may be used to ensure
|
||||
the instrumentation fits. (By default, ``rocprof-sys-instrument`` avoids instrumentation
|
||||
which requires a trap.)
|
||||
|
||||
**Overlapping functions**
|
||||
Due to language constructs or compiler optimizations, it might be possible for
|
||||
multiple functions to overlap (that is, share part of the same function body)
|
||||
or for a single function to have multiple entry points. In practice, it's
|
||||
impossible to determine the difference between multiple overlapping functions
|
||||
and a single function with multiple entry points. (By default, ``omnitrace-instrument``
|
||||
Due to language constructs or compiler optimizations, it might be possible for
|
||||
multiple functions to overlap (that is, share part of the same function body)
|
||||
or for a single function to have multiple entry points. In practice, it's
|
||||
impossible to determine the difference between multiple overlapping functions
|
||||
and a single function with multiple entry points. (By default, ``rocprof-sys-instrument``
|
||||
avoids instrumenting overlapping functions.)
|
||||
@@ -6,18 +6,18 @@ defaults:
|
||||
root: index
|
||||
subtrees:
|
||||
- entries:
|
||||
- file: what-is-omnitrace.rst
|
||||
- file: what-is-rocprof-sys.rst
|
||||
|
||||
- caption: Install
|
||||
entries:
|
||||
- file: install/quick-start.rst
|
||||
title: Omnitrace quick start
|
||||
title: ROCm Systems Profiler quick start
|
||||
- file: install/install.rst
|
||||
title: Omnitrace installation guide
|
||||
title: ROCm Systems Profiler installation guide
|
||||
|
||||
- caption: Tutorials
|
||||
entries:
|
||||
- url: https://github.com/ROCm/omnitrace/tree/amd-mainline/examples
|
||||
- url: https://github.com/ROCm/rocprofiler-systems/tree/amd-mainline/examples
|
||||
title: GitHub examples
|
||||
- file: tutorials/video-tutorials.rst
|
||||
title: Video tutorials
|
||||
@@ -25,37 +25,37 @@ subtrees:
|
||||
- caption: How to
|
||||
entries:
|
||||
- file: how-to/configuring-validating-environment.rst
|
||||
title: Configuring and validating the environment
|
||||
title: Configuring and validating the environment
|
||||
- file: how-to/configuring-runtime-options.rst
|
||||
title: Configuring runtime options
|
||||
title: Configuring runtime options
|
||||
- file: how-to/sampling-call-stack.rst
|
||||
title: Sampling the call stack
|
||||
title: Sampling the call stack
|
||||
- file: how-to/instrumenting-rewriting-binary-application.rst
|
||||
title: Instrumenting and rewriting a binary application
|
||||
- file: how-to/performing-causal-profiling.rst
|
||||
title: Performing causal profiling
|
||||
- file: how-to/understanding-omnitrace-output.rst
|
||||
title: Understanding the Omnitrace output
|
||||
title: Performing causal profiling
|
||||
- file: how-to/understanding-rocprof-sys-output.rst
|
||||
title: Understanding the ROCm Systems Profiler output
|
||||
- file: how-to/profiling-python-scripts.rst
|
||||
title: Profiling Python scripts
|
||||
- file: how-to/using-omnitrace-api.rst
|
||||
title: Using the Omnitrace API
|
||||
- file: how-to/general-tips-using-omnitrace.rst
|
||||
title: General tips for using Omnitrace
|
||||
title: Profiling Python scripts
|
||||
- file: how-to/using-rocprof-sys-api.rst
|
||||
title: Using the ROCm Systems Profiler API
|
||||
- file: how-to/general-tips-using-rocprof-sys.rst
|
||||
title: General tips for using ROCm Systems Profiler
|
||||
|
||||
- caption: Conceptual
|
||||
entries:
|
||||
- file: conceptual/data-collection-modes.rst
|
||||
title: Data collection modes
|
||||
- file: conceptual/omnitrace-feature-set.rst
|
||||
title: The Omnitrace feature set and use cases
|
||||
- file: conceptual/rocprof-sys-feature-set.rst
|
||||
title: The ROCm Systems Profiler feature set and use cases
|
||||
|
||||
- caption: Reference
|
||||
entries:
|
||||
- file: reference/development-guide.rst
|
||||
title: Development guide
|
||||
- file: reference/omnitrace-glossary.rst
|
||||
title: Omnitrace glossary
|
||||
- file: reference/rocprof-sys-glossary.rst
|
||||
title: ROCm Systems Profiler glossary
|
||||
- file: doxygen/html/files
|
||||
title: API library
|
||||
- file: doxygen/html/functions
|
||||
|
||||
@@ -1,11 +1,14 @@
|
||||
.. meta::
|
||||
:description: Omnitrace documentation and reference
|
||||
:keywords: Omnitrace, ROCm, profiler, tracking, visualization, tool, Instinct, accelerator, AMD
|
||||
:description: ROCm Systems Profiler video documentation and reference
|
||||
:keywords: rocprof-sys, rocprofiler-systems, Omnitrace, ROCm, profiler, video, tutorial, demonstration, tracking, visualization, tool, Instinct, accelerator, AMD
|
||||
|
||||
****************************************************
|
||||
Video tutorials
|
||||
****************************************************
|
||||
|
||||
The following video tutorials provide a visual guide to using ROCm Systems Profiler.
|
||||
They were recorded using the former name of the tool, Omnitrace, but the content is still applicable.
|
||||
|
||||
Installing a binary release
|
||||
========================================
|
||||
|
||||
@@ -20,7 +23,7 @@ Instrumenting a binary
|
||||
|
||||
<p align="center"><iframe width="560" height="315" src="https://www.youtube.com/embed/2B0gRr3FygQ?modestbranding=1" title="YouTube video player" frameborder="0" allow="accelerometer; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe></p>
|
||||
|
||||
Writing an Omnitrace configuration file
|
||||
Writing an ROCm Systems Profiler configuration file
|
||||
========================================
|
||||
|
||||
.. raw:: html
|
||||
|
||||
@@ -1,18 +1,18 @@
|
||||
.. meta::
|
||||
:description: Omnitrace documentation and reference
|
||||
:keywords: Omnitrace, ROCm, profiler, tracking, visualization, tool, Instinct, accelerator, AMD
|
||||
:description: ROCm Systems Profiler introduction, explanation, and reference
|
||||
:keywords: rocprof-sys, rocprofiler-systems, Omnitrace, ROCm, profiler, explanation, introduction, what is, tracking, visualization, tool, Instinct, accelerator, AMD
|
||||
|
||||
******************
|
||||
What is Omnitrace?
|
||||
What is ROCm Systems Profiler?
|
||||
******************
|
||||
|
||||
Omnitrace is designed for the high-level profiling and comprehensive tracing
|
||||
ROCm Systems Profiler is designed for the high-level profiling and comprehensive tracing
|
||||
of applications running on the CPU or the CPU and GPU. It supports dynamic binary
|
||||
instrumentation, call-stack sampling, and various other features for determining
|
||||
which function and line number are currently executing.
|
||||
|
||||
A visualization of the comprehensive Omnitrace results can be observed in any modern
|
||||
web browser. Upload the Perfetto (``.proto``) output files produced by Omnitrace at
|
||||
A visualization of the comprehensive ROCm Systems Profiler results can be observed in any modern
|
||||
web browser. Upload the Perfetto (``.proto``) output files produced by ROCm Systems Profiler at
|
||||
`ui.perfetto.dev <https://ui.perfetto.dev/>`_ to see the details.
|
||||
|
||||
.. important::
|
||||
@@ -26,7 +26,7 @@ JSON files for programmatic analysis. The JSON output files are compatible with
|
||||
the performance data into pandas data frames and facilitates multi-run comparisons, filtering,
|
||||
and visualization in Jupyter notebooks.
|
||||
|
||||
To use Omnitrace for instrumentation, follow these two configuration steps:
|
||||
To use ROCm Systems Profiler for instrumentation, follow these two configuration steps:
|
||||
|
||||
#. Indicate the functions and modules to :doc:`instrument <./how-to/instrumenting-rewriting-binary-application>` in the target binaries, including the executable and any libraries
|
||||
#. Specify the :doc:`instrumentation parameters <./how-to/configuring-runtime-options>` to use when the instrumented binaries are launched
|
||||