diff --git a/projects/rocprofiler-systems/.cmake-format.yaml b/projects/rocprofiler-systems/.cmake-format.yaml index d319f4c3de..d4dfc49456 100644 --- a/projects/rocprofiler-systems/.cmake-format.yaml +++ b/projects/rocprofiler-systems/.cmake-format.yaml @@ -65,7 +65,12 @@ parse: RUN_ARGS: '*' ENVIRONMENT: '*' LABELS: '*' + DEPENDS: '*' + COMMAND: '*' PROPERTIES: '*' + PASS_REGULAR_EXPRESSION: '*' + FAIL_REGULAR_EXPRESSION: '*' + SKIP_REGULAR_EXPRESSION: '*' rocm_version_message: flags: - STATUS diff --git a/projects/rocprofiler-systems/.github/workflows/cancelling.yml b/projects/rocprofiler-systems/.github/workflows/cancelling.yml index 52ea7de330..29f77b3e24 100644 --- a/projects/rocprofiler-systems/.github/workflows/cancelling.yml +++ b/projects/rocprofiler-systems/.github/workflows/cancelling.yml @@ -1,20 +1,20 @@ name: cancel-builds-on-update on: workflow_run: - workflows: ['formatting', 'ubuntu-bionic', 'ubuntu-focal-dyninst-package', 'ubuntu-focal-external-rocm', 'ubuntu-focal-external', 'ubuntu-focal', 'cpack-bionic', 'cpack-bionic-rocm', 'cpack-focal', cpack-focal-rocm'] + workflows: ['formatting', 'ubuntu-bionic', 'ubuntu-focal-external-rocm', 'ubuntu-focal-external', 'ubuntu-focal', 'cpack-bionic', 'cpack-bionic-rocm', 'cpack-focal', cpack-focal-rocm'] types: ['requested'] jobs: cancel-duplicate-workflow-runs: - name: "Cancel duplicate workflow runs" + name: "Cancel duplicate ${{ github.event.workflow_run.id }} workflow runs" runs-on: ubuntu-latest steps: - uses: potiuk/cancel-workflow-runs@master - name: "Cancel duplicate workflow runs" + name: "Cancel duplicate ${{ github.event.workflow_run.id }} workflow runs" with: cancelMode: duplicates cancelFutureDuplicates: true token: ${{ secrets.GITHUB_TOKEN }} sourceRunId: ${{ github.event.workflow_run.id }} - notifyPRCancel: true + notifyPRCancel: false skipEventTypes: '["schedule"]' diff --git a/projects/rocprofiler-systems/README.md b/projects/rocprofiler-systems/README.md index ee2e330859..7f8c7cd4bb 100755 --- a/projects/rocprofiler-systems/README.md +++ b/projects/rocprofiler-systems/README.md @@ -5,7 +5,9 @@ [![Ubuntu 20.04 (GCC 9, MPICH, OpenMPI)](https://github.com/AMDResearch/omnitrace/actions/workflows/ubuntu-focal.yml/badge.svg)](https://github.com/AMDResearch/omnitrace/actions/workflows/ubuntu-focal.yml) [![Ubuntu 20.04 (GCC 9, MPICH, OpenMPI, ROCm 4.3, 4.5, 5.0)](https://github.com/AMDResearch/omnitrace/actions/workflows/ubuntu-focal-external-rocm.yml/badge.svg)](https://github.com/AMDResearch/omnitrace/actions/workflows/ubuntu-focal-external-rocm.yml) -Omnitrace is an AMD research project and should not be treated as an offical part of the ROCm software stack. +> ***[Omnitrace](https://github.com/AMDResearch/omnitrace) is an AMD research project and should*** +> ***not be treated as an offical part of the ROCm software stack.*** + The documentation for omnitrace is available at [amdresearch.github.io/omnitrace](https://amdresearch.github.io/omnitrace/). ## Using Omnitrace Executable @@ -19,7 +21,7 @@ omnitrace -- `omnitrace-avail -Sd` will provide a list of all the possible omnitrace settings, their current value, and a description of the setting. -> NOTE: Some settings may only affect the timemory backend. +> ***Some settings may only affect the timemory backend.*** These settings can be set via environment variables or placed in a config file and specified via `OMNITRACE_CONFIG_FILE=/path/to/config/file`. The config file can be a text, JSON, or XML file. Some of the most relevant settings are provided below: @@ -91,8 +93,8 @@ omnitrace -R '^hip' -o ./lib/libamdhip64.so.4 -- /opt/rocm/lib/libamdhip64.so.4 export LD_LIBRARY_PATH=${PWD}/lib:${LD_LIBRARY_PATH} ``` -> NOTE: Verify via `ldd` that your executable will load the instrumented library -- if you built your executable with -> an RPATH to the original library's directory, then prefixing `LD_LIBRARY_PATH` will have no effect. +> ***Verify via `ldd` that your executable will load the instrumented library -- if you built your executable with*** +> ***an RPATH to the original library's directory, then prefixing `LD_LIBRARY_PATH` will have no effect.*** Once you have rewritten your executable and/or libraries with instrumentation, you can just run the (instrumented) executable or exectuable which loads the instrumented libraries normally, e.g.: @@ -186,10 +188,10 @@ Once Julia is installed, install the necessary packages (this operation only nee julia -e 'using Pkg; for name in ["JSON", "DataFrames", "Dates", "CSV", "Chain", "PrettyTables"]; Pkg.add(name); end' ``` -> NOTE: Using `rocprof` externally for tracing is deprecated. The current version has built-in support for -> recording the GPU activity and HIP API calls. If you want to use an external rocprof, either -> configure CMake with `-DOMNITRACE_USE_ROCTRACER=OFF` or explicitly set `OMNITRACE_ROCTRACER_ENABLED=OFF` in the -> environment. +> ***Using `rocprof` externally for tracing is deprecated. The current version has built-in support for*** +> ***recording the GPU activity and HIP API calls. If you want to use an external rocprof, either*** +> ***configure CMake with `-DOMNITRACE_USE_ROCTRACER=OFF` or explicitly set `OMNITRACE_ROCTRACER_ENABLED=OFF` in the*** +> ***environment.*** Use the `omnitrace-merge.jl` Julia script to merge rocprof and perfetto traces. diff --git a/projects/rocprofiler-systems/examples/python/source.py b/projects/rocprofiler-systems/examples/python/source.py index 276efebccc..85b495fd75 100755 --- a/projects/rocprofiler-systems/examples/python/source.py +++ b/projects/rocprofiler-systems/examples/python/source.py @@ -30,9 +30,7 @@ def run(nfib): if __name__ == "__main__": - omnitrace.initialize(__file__) nfib = int(sys.argv[1]) if len(sys.argv) > 1 else 20 for i in range(5): ans = run(nfib) print(f"[{i}] fibonacci({nfib}) = {ans}") - omnitrace.finalize() diff --git a/projects/rocprofiler-systems/source/docs/about.md b/projects/rocprofiler-systems/source/docs/about.md index a20f89d9db..d85ef8be4c 100644 --- a/projects/rocprofiler-systems/source/docs/about.md +++ b/projects/rocprofiler-systems/source/docs/about.md @@ -6,10 +6,10 @@ :maxdepth: 4 ``` -[Browse Omnitrace source code on Github](https://github.com/AMDResearch/omnitrace) +> ***[Omnitrace](https://github.com/AMDResearch/omnitrace) is an AMD research project and should*** +> ***not be treated as an offical part of the ROCm software stack.*** -> [Omnitrace](https://github.com/AMDResearch/omnitrace) is an AMD research project and should -> not be treated as an offical part of the ROCm software stack. +[Browse Omnitrace source code on Github](https://github.com/AMDResearch/omnitrace) [Omnitrace](https://github.com/AMDResearch/omnitrace) is designed for both high-level and comprehensive application tracing and profiling on both the CPU and GPU. diff --git a/projects/rocprofiler-systems/source/docs/conf.py b/projects/rocprofiler-systems/source/docs/conf.py index 2762450dfd..47fcbb7593 100644 --- a/projects/rocprofiler-systems/source/docs/conf.py +++ b/projects/rocprofiler-systems/source/docs/conf.py @@ -36,7 +36,7 @@ project = "omnitrace" copyright = "2022, Advanced Micro Devices, Inc." author = "Audacious Software Group" -version = open(os.path.join("..", "VERSION")).read().strip() +version = open(os.path.join("..", "..", "VERSION")).read().strip() # The full version, including alpha/beta/rc tags release = version diff --git a/projects/rocprofiler-systems/source/docs/generate-doxyfile.cmake b/projects/rocprofiler-systems/source/docs/generate-doxyfile.cmake index 9cce9f437e..95b89c667e 100644 --- a/projects/rocprofiler-systems/source/docs/generate-doxyfile.cmake +++ b/projects/rocprofiler-systems/source/docs/generate-doxyfile.cmake @@ -15,5 +15,5 @@ string(REGEX REPLACE "(\n|\r)" "" FULL_VERSION_STRING "${FULL_VERSION_STRING}") string(REGEX REPLACE "([0-9]+)\\.([0-9]+)\\.([0-9]+)(.*)" "\\1.\\2.\\3" OMNITRACE_VERSION "${FULL_VERSION_STRING}") -configure_file(${SOURCE_DIR}/docs-source/omnitrace.dox.in - ${SOURCE_DIR}/docs-source/omnitrace.dox @ONLY) +configure_file(${SOURCE_DIR}/source/docs/omnitrace.dox.in + ${SOURCE_DIR}/source/docs/omnitrace.dox @ONLY) diff --git a/projects/rocprofiler-systems/source/docs/getting_started.md b/projects/rocprofiler-systems/source/docs/getting_started.md index 027bbf1a69..d31738ee58 100644 --- a/projects/rocprofiler-systems/source/docs/getting_started.md +++ b/projects/rocprofiler-systems/source/docs/getting_started.md @@ -9,3 +9,40 @@ runtime critical_trace ``` + +## Configuring Environment + +Source the `setup-env.sh` script to prefix the `PATH`, `LD_LIBRARY_PATH`, etc. environment variables: + +```bash +source /opt/omnitrace/share/omnitrace/setup-env.sh +``` + +Alternatively, if environment modules are supported, add the `/share/modulefiles` directory to `MODULEPATH` via: + +```bash +module use /opt/omnitrace/share/modulefiles +``` + +> ***Alternatively, the above line can be added to the `${HOME}/.modulerc` file.*** + +Once omnitrace is in the `MODULEPATH`, omnitrace can be loaded via `module load omnitrace/` and unloaded via `module unload omnitrace/`, e.g.: + + +```bash +module load omnitrace/1.0.0 +module unload omnitrace/1.0.0 +``` + +> ***You may need to also add the path to the ROCm libraries to `LD_LIBRARY_PATH`, e.g. `export LD_LIBRARY_PATH=/opt/rocm/lib:${LD_LIBRARY_PATH}`*** + +### Validating Environment Configuration + +If all the following commands execute successfully with output, then you are ready to use omnitrace: + +```bash +which omnitrace +which omnitrace-avail +omnitrace --help +omnitrace-avail --all +``` diff --git a/projects/rocprofiler-systems/source/docs/index.md b/projects/rocprofiler-systems/source/docs/index.md index 9a6316f964..b8b8bbf2ec 100644 --- a/projects/rocprofiler-systems/source/docs/index.md +++ b/projects/rocprofiler-systems/source/docs/index.md @@ -12,4 +12,5 @@ getting_started output user_api + python ``` diff --git a/projects/rocprofiler-systems/source/docs/instrumenting.md b/projects/rocprofiler-systems/source/docs/instrumenting.md index 15e375404d..a128b3eff2 100644 --- a/projects/rocprofiler-systems/source/docs/instrumenting.md +++ b/projects/rocprofiler-systems/source/docs/instrumenting.md @@ -180,8 +180,8 @@ There are three ways to perform instrumentation: see [Binary Rewriting a Library](#binary-rewriting-a-library) for help -> NOTE: Attaching to a running process is an alpha feature and support for detaching from the target process -> without ending the target process is not currently supported. +> ***Attaching to a running process is an alpha feature and support for detaching from the target process*** +> ***without ending the target process is not currently supported.*** The general syntax for separating omnitrace command line arguments from the application arguments follows the is consistent with the LLVM style of using a standalone double-hyphen (`--`). All arguments preceding the double-hyphen @@ -322,7 +322,7 @@ These options are always applied regardless of whether the module or function sa #### Example Available Module and Function Info Output -> `omnitrace -o lulesh.inst --label file line args --simulate -- lulesh` +> ***`omnitrace -o lulesh.inst --label file line args --simulate -- lulesh`*** ```console AddressRange Module Function FunctionSignature @@ -540,7 +540,7 @@ These options are always applied regardless of whether the module or function sa #### Example Instrumented Module and Function Info Output -> `omnitrace -o lulesh.inst --label file line args --simulate -- lulesh` +> ***`omnitrace -o lulesh.inst --label file line args --simulate -- lulesh`*** After the heuristics are applied in [Example Available Module and Function Info Output](#example-available-module-and-function-info-output), the selected module/functions are: @@ -674,7 +674,7 @@ You will have to remove or modify the rpath in order to get `foo.inst` to resolv #### Modifying RPATH -> Requires `patchelf` package +> ***Requires `patchelf` package*** ```shell patchelf --remove-rpath diff --git a/projects/rocprofiler-systems/source/docs/output.md b/projects/rocprofiler-systems/source/docs/output.md index 601bf69565..e752c230a4 100644 --- a/projects/rocprofiler-systems/source/docs/output.md +++ b/projects/rocprofiler-systems/source/docs/output.md @@ -61,7 +61,7 @@ $ omnitrace -- ./foo ### Core Configuration Settings -> See also: [Customizing Omnitrace Runtime](runtime.md) +> ***See also: [Customizing Omnitrace Runtime](runtime.md)*** | Setting | Value | Description | |---------------------------|--------------------|---------------------------------------------------------------------------------------------------| @@ -105,10 +105,10 @@ set `OMNITRACE_OUTPUT_PREFIX="%argt%-"` and let omnitrace cleanly organize the o | `%r` | Shorthand for `%rank%` | | `%s` | Shorthand for `%size%` | -> NOTE: any output prefix key which contain a '/' will have the `/` characters -> replaced with `_` and any leading underscores will be stripped, e.g. if `%arg0%` is `/usr/bin/foo`, this -> will translate to `usr_bin_foo`. Additionally, any `%arg%` keys which do not have a command line argument -> at position `` will be ignored. +> ***Any output prefix key which contain a `/` will have the `/` characters*** +> ***replaced with `_` and any leading underscores will be stripped, e.g. if `%arg0%` is `/usr/bin/foo`, this*** +> ***will translate to `usr_bin_foo`. Additionally, any `%arg%` keys which do not have a command line argument*** +> ***at position `` will be ignored.*** ## Perfetto Output @@ -140,7 +140,7 @@ requires significantly less memory than perfetto, this is not the case in timeli ### Timemory Text Output -> Hint: the generation of text output is configurable via `OMNITRACE_TEXT_OUTPUT` +> ***Hint: the generation of text output is configurable via `OMNITRACE_TEXT_OUTPUT`*** Timemory text output files are meant for human-consumption (use JSON formats for analysis) and as such, some fields such as the `LABEL` fields may be truncated for readability. @@ -336,7 +336,7 @@ component explicitly sets type-traits which specify that the data is only releva ### Timemory Flat JSON Output -> Hint: the generation of flat JSON output is configurable via `OMNITRACE_JSON_OUTPUT` +> ***Hint: the generation of flat JSON output is configurable via `OMNITRACE_JSON_OUTPUT`*** Timemory provides two JSON output formats. The flat JSON output files are similar to the text files: the hierarchical information is represented by the indentation of the `"prefix"` field and the `"depth"` field. All the data entries are in a single JSON array, @@ -521,7 +521,7 @@ This script applied to the corresponding JSON output from [Text Output Example]( ### Timemory Hierarchical JSON Output -> Hint: the generation of hierarchical JSON output is configurable via `OMNITRACE_TREE_OUTPUT` +> ***Hint: the generation of hierarchical JSON output is configurable via `OMNITRACE_TREE_OUTPUT`*** The hierarchical JSON output (extension: `.tree.json`) contains the very similar data to the flat JSON output, however, it's structure requires processing through recursion. The main use of these files are their analysis support diff --git a/projects/rocprofiler-systems/source/docs/python.md b/projects/rocprofiler-systems/source/docs/python.md new file mode 100644 index 0000000000..bed1e9966f --- /dev/null +++ b/projects/rocprofiler-systems/source/docs/python.md @@ -0,0 +1,297 @@ +# Python Support + +```eval_rst +.. toctree:: + :glob: + :maxdepth: 3 +``` + +[Omnitrace](https://github.com/AMDResearch/omnitrace) supports profiling Python code at the source-level and/or the script-level. +Python support is enabled via the `OMNITRACE_USE_PYTHON` and `OMNITRACE_PYTHON_VERSION=.` CMake options. + +> ***When using omnitrace for Python, the Python interpreter major and minor version (e.g. 3.7) must match the interpreter major and minor version*** +> ***used when compiling the Python bindings, i.e. when building omnitrace, a `libpyomnitrace.----.so` will be generated*** +> ***where `IMPL` is the Python implementation, `VERSION` is the major and minor version, `ARCH` is the architecture,*** +> ***`OS` is the operating system, and `ABI` is the application binary interface; Example: `libpyomnitrace.cpython-38-x86_64-linux-gnu.so`.*** + +## Getting Started + +The omnitrace Python package is installed in `lib/pythonX.Y/site-packages/omnitrace`. In order to ensure the Python interpreter can find the omnitrace package, +add this path to the `PYTHONPATH` environment variable, e.g.: + +```bash +export PYTHONPATH=/opt/omnitrace/lib/python3.8/site-packages:${PYTHONPATH} +``` + +If using either the `share/omnitrace/setup-env.sh` script or the modulefile in `share/modulefiles/omnitrace`, prefixing the `PYTHONPATH` +environment variable is automatically handled. + +## Running Omnitrace on a Python Script + +Omnitrace provides an `omnitrace-python` helper bash script which effectively handles ensuring `PYTHONPATH` is properly set and the correct python interpreter is used. +Thus the following are effectively equivalent: + +```bash +omnitrace-python --help + +export PYTHONPATH=/opt/omnitrace/lib/python3.8/site-packages:${PYTHONPATH} +python3.8 -m omnitrace --help +``` + +> ***`omnitrace-python` / `python -m omnitrace` uses the same command-line syntax as the `omnitrace` executable (i.e. `omnitrace-python --