AI/rocm-systems
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

omnitrace-avail generate config (#69)

Browse Source 
* Config updates

- See PR #69 for details

- change type of OMNITRACE_DL_VERBOSE
- add "deprecated" category to OMNITRACE_ROCM_SMI_DEVICES
- reduce size of perfetto shared memory size hint
- deprecate OMNITRACE_OUTPUT_FILE in favor of OMNITRACE_PERFETTO_FILE
- set papi event choices
- read config file after reading command line
- fix update of OMNITRACE_DL_VERBOSE
- mark several settings as hidden
- timemory update support hidden attribute for settings
- rework get_perfetto_output_filename()
- Hide settings from not available backends

* Rework omnitrace-avail to support dumping configurations

* Overwrite query, tests, output flag

- Support using -O flag when dumping config
- Support checking before overwriting existing config
- Support --force to overwrite existing config
- Fix get_component_info not including omnitrace components
- Testing for dumping config

* Update documentation on omnitrace-avail

* Fix issue with timemory format + "/__w/"

* Update output prefix keys docs

* Rename --dump-config to --generate-config

* Hide MPI related options

- OMNITRACE_PERFETTO_COMBINE_TRACES and OMNITRACE_COLLAPSE_PROCESSES are hidden w/o MPI support
This commit is contained in:
Jonathan R. Madsen
2022-06-28 01:36:04 -05:00
committed by GitHub
parent efe1edd253
commit 1877ebf47b
22 changed files with 2067 additions and 870 deletions
Download Patch File Download Diff File Expand all files Collapse all files
source/docs/output.md
+27 -21
View File
@@ -259,27 +259,33 @@ If you are ever doing scaling studies and specifying options via the command lin
use a common `OMNITRACE_OUTPUT_PATH`, disable `OMNITRACE_TIME_OUTPUT`,
set `OMNITRACE_OUTPUT_PREFIX="%argt%-"` and let omnitrace cleanly organize the output.
| String | Encoding |
|-----------------|-----------------------------------------------------------------------------------------------|
| `%arg<N>%` | Command line argument at position `<N>` (zero indexed), e.g. `%arg0%` for first argument. |
| `%arg<N>_hash%` | MD5 sum of `%arg<N>%` |
| `%argv%` | Entire command-line condensed into a single string |
| `%argv_hash%` | MD5 sum of `%argv%` |
| `%argt%` | Similar to `%argv%` except basename of first command line argument |
| `%argt_hash%` | MD5 sum if `%argt%` |
| `%args%` | All command line arguments condensed into a single string |
| `%args_hash%` | MD5 sum of `%args%` |
| `%tag%` | Basename of first command line argument |
| `%tag_hash%` | MD5 sum of `%tag%` |
| `%pid%` | Process identifier (i.e. `getpid()`) |
| `%job%` | Value of `SLURM_JOB_ID` environment variable if exists, else `0` |
| `%rank%` | Value of `SLURM_PROCID` environment variable if exists, else `MPI_Comm_rank` (or `0` non-mpi) |
| `%size%` | `MPI_Comm_size` or `1` if non-mpi |
| `%m` | Shorthand for `%argt_hash%` |
| `%p` | Shorthand for `%pid%` |
| `%j` | Shorthand for `%job%` |
| `%r` | Shorthand for `%rank%` |
| `%s` | Shorthand for `%size%` |
| String | Encoding |
|-----------------|--------------------------------------------------------------------------------------------------------------------|
| `%arg<N>%` | Command line argument at position `<N>` (zero indexed), e.g. `%arg0%` for first argument. |
| `%arg<N>_hash%` | MD5 sum of `%arg<N>%` |
| `%argv%` | Entire command-line condensed into a single string |
| `%argv_hash%` | MD5 sum of `%argv%` |
| `%argt%` | Similar to `%argv%` except basename of first command line argument |
| `%argt_hash%` | MD5 sum if `%argt%` |
| `%args%` | All command line arguments condensed into a single string |
| `%args_hash%` | MD5 sum of `%args%` |
| `%tag%` | Basename of first command line argument |
| `%tag_hash%` | MD5 sum of `%tag%` |
| `%pid%` | Process identifier (i.e. `getpid()`) |
| `%ppid%` | Parent process identifier (i.e. `getppid()`) |
| `%job%` | Value of `SLURM_JOB_ID` environment variable if exists, else `0` |
| `%rank%` | Value of `SLURM_PROCID` environment variable if exists, else `MPI_Comm_rank` (or `0` non-mpi) |
| `%size%` | `MPI_Comm_size` or `1` if non-mpi |
| `%launch_time%` | Launch date and time (uses `OMNITRACE_TIME_FORMAT`) |
| `%env{NAME}%` | Value of environment variable `NAME` (i.e. `getenv(NAME)`) |
| `%cfg{NAME}%` | Value of configuration variable `NAME` (e.g. `%cfg{OMNITRACE_SAMPLING_FREQ}%` would resolve to sampling frequency) |
| `$env{NAME}` | Alternative syntax to `%env{NAME}%` |
| `$cfg{NAME}` | Alternative syntax to `%cfg{NAME}%` |
| `%m` | Shorthand for `%argt_hash%` |
| `%p` | Shorthand for `%pid%` |
| `%j` | Shorthand for `%job%` |
| `%r` | Shorthand for `%rank%` |
| `%s` | Shorthand for `%size%` |
> ***Any output prefix key which contain a `/` will have the `/` characters***
> ***replaced with `_` and any leading underscores will be stripped, e.g. if `%arg0%` is `/usr/bin/foo`, this***
source/docs/runtime.md
+129 -3
View File
@@ -14,6 +14,26 @@ self-updating: when new capabilities and settings are added to the omnitrace sou
propagated to `omnitrace-avail`, thus it should be viewed as the single source of truth if any conflicting
information or missing feature is found in this documentation.
It is recommended to create a default configuration file in `${HOME}/.omnitrace.cfg`. This can be done via
executing `omnitrace-avail -D ~/.omnitrace.cfg`, or optionally, use `omnitrace-avail -D ~/.omnitrace.cfg --all`
for a verbose configuration file with descriptions, categories, etc.
Modify `${HOME}/.omnitrace.cfg` as desired, e.g. enable [perfetto](https://perfetto.dev/),
[timemory](https://github.com/NERSC/timemory), sampling, and process-level sampling by default
and tweak some sampling default values:
```console
# ...
OMNITRACE_USE_PERFETTO = true
OMNITRACE_USE_TIMEMORY = true
OMNITRACE_USE_SAMPLING = true
OMNITRACE_USE_PROCESS_SAMPLING = true
# ...
OMNITRACE_SAMPLING_FREQ = 50
OMNITRACE_SAMPLING_CPUS = all
OMNITRACE_SAMPLING_GPUS = $env:HIP_VISIBLE_DEVICES
```
### Exploring Runtime Settings
In order to view the list of the available runtime settings, their current value, and descriptions for each setting:
@@ -62,7 +82,111 @@ omnitrace-avail --hw-counters --description
### omnitrace-avail Examples
#### Settings
#### Generating Default Configuration
```console
$ omnitrace-avail -D ~/.omnitrace.cfg
[omnitrace-avail] Outputting text configuration file '/home/user/.omnitrace.cfg'...
$ cat ~/.omnitrace.cfg
# auto-generated by omnitrace-avail (version 1.2.0) on 2022-06-27 @ 19:15
OMNITRACE_CONFIG_FILE =
OMNITRACE_MODE = trace
OMNITRACE_USE_PERFETTO = true
OMNITRACE_USE_TIMEMORY = false
OMNITRACE_USE_SAMPLING = false
OMNITRACE_USE_PROCESS_SAMPLING = true
OMNITRACE_USE_ROCTRACER = true
OMNITRACE_USE_ROCM_SMI = true
OMNITRACE_USE_KOKKOSP = false
OMNITRACE_USE_CODE_COVERAGE = false
OMNITRACE_USE_PID = true
OMNITRACE_OUTPUT_PATH = omnitrace-%tag%-output
OMNITRACE_OUTPUT_PREFIX =
OMNITRACE_CI = false
OMNITRACE_CRITICAL_TRACE = false
OMNITRACE_CRITICAL_TRACE_BUFFER_COUNT = 2000
OMNITRACE_CRITICAL_TRACE_COUNT = 0
OMNITRACE_CRITICAL_TRACE_DEBUG = false
OMNITRACE_CRITICAL_TRACE_NUM_THREADS = 8
OMNITRACE_CRITICAL_TRACE_PER_ROW = 0
OMNITRACE_CRITICAL_TRACE_SERIALIZE_NAMES = false
OMNITRACE_DEBUG = false
OMNITRACE_DL_VERBOSE = 0
OMNITRACE_FLAT_SAMPLING = false
OMNITRACE_INSTRUMENTATION_INTERVAL = 1
OMNITRACE_KOKKOS_KERNEL_LOGGER = false
OMNITRACE_PAPI_EVENTS = PAPI_TOT_CYC
OMNITRACE_PERFETTO_BACKEND = inprocess
OMNITRACE_PERFETTO_BUFFER_SIZE_KB = 1024000
OMNITRACE_PERFETTO_COMBINE_TRACES = true
OMNITRACE_PERFETTO_FILE = perfetto-trace.proto
OMNITRACE_PERFETTO_FILL_POLICY = discard
OMNITRACE_PERFETTO_SHMEM_SIZE_HINT_KB = 4096
OMNITRACE_ROCTRACER_FLAT_PROFILE = false
OMNITRACE_ROCTRACER_HSA_ACTIVITY = false
OMNITRACE_ROCTRACER_HSA_API = false
OMNITRACE_ROCTRACER_HSA_API_TYPES =
OMNITRACE_ROCTRACER_TIMELINE_PROFILE = false
OMNITRACE_SAMPLING_CPUS =
OMNITRACE_SAMPLING_DELAY = 0.5
OMNITRACE_SAMPLING_FREQ = 10
OMNITRACE_SAMPLING_GPUS = all
OMNITRACE_TIME_OUTPUT = true
OMNITRACE_TIMELINE_SAMPLING = false
OMNITRACE_TIMEMORY_COMPONENTS = wall_clock
OMNITRACE_TRACE_THREAD_LOCKS = false
OMNITRACE_VERBOSE = 0
OMNITRACE_COLLAPSE_PROCESSES = false
OMNITRACE_COLLAPSE_THREADS = false
OMNITRACE_COUT_OUTPUT = false
OMNITRACE_CPU_AFFINITY = false
OMNITRACE_DIFF_OUTPUT = false
OMNITRACE_ENABLE_SIGNAL_HANDLER = true
OMNITRACE_ENABLED = true
OMNITRACE_FILE_OUTPUT = true
OMNITRACE_FLAT_PROFILE = false
OMNITRACE_INPUT_EXTENSIONS = json,xml
OMNITRACE_INPUT_PATH =
OMNITRACE_INPUT_PREFIX =
OMNITRACE_JSON_OUTPUT = true
OMNITRACE_MAX_DEPTH = 65535
OMNITRACE_MAX_WIDTH = 120
OMNITRACE_MEMORY_PRECISION = -1
OMNITRACE_MEMORY_SCIENTIFIC = false
OMNITRACE_MEMORY_UNITS = MB
OMNITRACE_MEMORY_WIDTH = -1
OMNITRACE_NETWORK_INTERFACE =
OMNITRACE_NODE_COUNT = 0
OMNITRACE_PAPI_FAIL_ON_ERROR = false
OMNITRACE_PAPI_MULTIPLEXING = false
OMNITRACE_PAPI_OVERFLOW = 0
OMNITRACE_PAPI_QUIET = false
OMNITRACE_PAPI_THREADING = true
OMNITRACE_PRECISION = -1
OMNITRACE_SCIENTIFIC = false
OMNITRACE_STRICT_CONFIG = true
OMNITRACE_SUPPRESS_CONFIG = true
OMNITRACE_SUPPRESS_PARSING = true
OMNITRACE_TEXT_OUTPUT = true
OMNITRACE_TIME_FORMAT = %F_%I.%M_%p
OMNITRACE_TIMELINE_PROFILE = false
OMNITRACE_TIMING_PRECISION = 6
OMNITRACE_TIMING_SCIENTIFIC = false
OMNITRACE_TIMING_UNITS = sec
OMNITRACE_TIMING_WIDTH = -1
OMNITRACE_TREE_OUTPUT = true
OMNITRACE_WIDTH = -1
```
- Recommendations
- Use the `--all` option for descriptions, choices, etc. in the configuration file.
- If you want to create a new configuration without inheriting from an existing `${HOME}/.omnitrace.cfg`,
set `OMNITRACE_SUPPRESS_CONFIG=ON` in the environment before executing.
- If you want to create a new configuration with some minor tweaks to an existing config,
set `OMNITRACE_CONFIG_FILE=/path/to/existing/file` and define the tweaks as environment variables before generating.
#### Viewing Setting Descriptions
```console
$ omnitrace-avail -S -bd
@@ -161,7 +285,7 @@ $ omnitrace-avail -S -bd
|-----------------------------------------|-----------------------------------------|
```
#### Components
#### Viewing Components
```console
$ omnitrace-avail -C -bd
@@ -234,7 +358,7 @@ $ omnitrace-avail -C -bd
|-----------------------------------|----------------------------------------------|
```
#### Hardware Counters
#### Viewing Hardware Counters
```console
$ omnitrace-avail -H -bd
@@ -638,6 +762,8 @@ $ omnitrace-avail -H -bd
## Creating a Configuration File
[Omnitrace](https://github.com/AMDResearch/omnitrace) supports 3 configuration file formats: JSON, XML, and plain text.
Use `omnitrace-avail -D <filename> -F txt json xml` to generate default configuration files of each format and, optionally,
include the `--all` flag for descriptions, etc.
Configuration files are specified via the `OMNITRACE_CONFIG_FILE` environment variable
and by default will look for `${HOME}/.omnitrace.cfg` and `${HOME}/.omnitrace.json`.
Multiple configuration files can be concatenated via `:`, e.g.: