importing 1.x docs from main branch

Signed-off-by: Karl W. Schulz <karl.schulz@amd.com>
2024-03-01 20:10:28 -06:00
commit fc24ad1d7d
@@ -0,0 +1,5 @@
+/build*
+/_build
+/_doxygen
+/.gitinfo
+/omniperf.dox
@@ -0,0 +1,20 @@
+# Minimal makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line, and also
+# from the environment for the first two.
+SPHINXOPTS    ?=
+SPHINXBUILD   ?= sphinx-build
+SOURCEDIR     = .
+BUILDDIR      = _build
+
+# Put it first so that "make" without argument is like "make help".
+help:
+	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+.PHONY: help Makefile
+
+# Catch-all target: route all unknown targets to Sphinx using the new
+# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
+%: Makefile
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
@@ -0,0 +1,6 @@
+This subdirectory houses the input markup for Omniperf documentation using
+Sphinx. Changes committed here on the main branch will automatically be built
+and pushed live using a Github action.
+
+You can build a local copy of the documentation in this directory using
+"make html" assuming you have the necessary sphinx dependencies installed.
@@ -0,0 +1,770 @@
+# Analyze Mode
+
+```eval_rst
+.. toctree::
+   :glob:
+   :maxdepth: 4
+```
+Omniperf offers several ways to interact with the metrics it generates from profiling. The option you choose will likey be influnced by your familiarity with the profiled application, computing enviroment, and experience with Omniperf.
+
+While analyzing with the CLI offers quick and straightforward access to Omniperf metrics from terminal, the GUI adds an extra layer of styling and interactiveness some users may prefer.
+
+See sections below for more information on each.
+
+## CLI Analysis
+> Profiling results from the [aforementioned vcopy workload](https://amdresearch.github.io/omniperf/profiling.html#workload-compilation) will be used in the following sections to demonstrate the use of Omniperf in MI GPU performance analysis. Unless otherwise noted, the performance analysis is done on the MI200 platform.
+
+### Features
+
+- All Omniperf built-in metrics.
+- Multiple runs base line comparison.
+- Metrics customization: pick up subset of build-in metrics or build your own profiling configuration.
+- Kernel, gpu-id, dispatch-id filters.
+
+Run `omniperf analyze -h` for more details.
+
+### Recommended workflow
+
+1) To begin, generate a comprehensive analysis report with Omniperf CLI.
+```shell-session
+$ omniperf analyze -p workloads/vcopy/mi200/
+
+--------
+Analyze
+--------
+
+
+--------------------------------------------------------------------------------
+0. Top Stat
+╒════╤══════════════════════════════════════════╤═════════╤═══════════╤════════════╤══════════════╤════════╕
+│    │ KernelName                               │   Count │   Sum(ns) │   Mean(ns) │   Median(ns) │    Pct │
+╞════╪══════════════════════════════════════════╪═════════╪═══════════╪════════════╪══════════════╪════════╡
+│  0 │ vecCopy(double*, double*, double*, int,  │       1 │  20000.00 │   20000.00 │     20000.00 │ 100.00 │
+│    │ int) [clone .kd]                         │         │           │            │              │        │
+╘════╧══════════════════════════════════════════╧═════════╧═══════════╧════════════╧══════════════╧════════╛
+
+
+--------------------------------------------------------------------------------
+1. System Info
+╒══════════════════╤═══════════════════════════════════════════════╕
+│                  │ Info                                          │
+╞══════════════════╪═══════════════════════════════════════════════╡
+│ workload_name    │ vcopy                                         │
+├──────────────────┼───────────────────────────────────────────────┤
+│ command          │ /home/colramos/vcopy 1048576 256              │
+├──────────────────┼───────────────────────────────────────────────┤
+│ host_name        │ sv-pdp-2                                      │
+├──────────────────┼───────────────────────────────────────────────┤
+│ host_cpu         │ AMD EPYC 7282 16-Core Processor               │
+├──────────────────┼───────────────────────────────────────────────┤
+│ host_distro      │ Ubuntu 20.04.3 LTS                            │
+├──────────────────┼───────────────────────────────────────────────┤
+│ host_kernel      │ 5.15.0-43-generic                             │
+├──────────────────┼───────────────────────────────────────────────┤
+│ host_rocmver     │ 5.2.1-79                                      │
+├──────────────────┼───────────────────────────────────────────────┤
+│ date             │ Fri Jan 20 11:22:20 2023 (CST)                │
+├──────────────────┼───────────────────────────────────────────────┤
+│ gpu_soc          │ gfx90a                                        │
+├──────────────────┼───────────────────────────────────────────────┤
+│ numSE            │ 8                                             │
+├──────────────────┼───────────────────────────────────────────────┤
+│ numCU            │ 104                                           │
+├──────────────────┼───────────────────────────────────────────────┤
+│ numSIMD          │ 4                                             │
+├──────────────────┼───────────────────────────────────────────────┤
+│ waveSize         │ 64                                            │
+├──────────────────┼───────────────────────────────────────────────┤
+│ maxWavesPerCU    │ 32                                            │
+├──────────────────┼───────────────────────────────────────────────┤
+│ maxWorkgroupSize │ 1024                                          │
+├──────────────────┼───────────────────────────────────────────────┤
+│ L1               │ 16                                            │
+├──────────────────┼───────────────────────────────────────────────┤
+│ L2               │ 8192                                          │
+├──────────────────┼───────────────────────────────────────────────┤
+│ sclk             │ 1700                                          │
+├──────────────────┼───────────────────────────────────────────────┤
+│ mclk             │ 1600                                          │
+├──────────────────┼───────────────────────────────────────────────┤
+│ cur_sclk         │ 800                                           │
+├──────────────────┼───────────────────────────────────────────────┤
+│ cur_mclk         │ 1600                                          │
+├──────────────────┼───────────────────────────────────────────────┤
+│ L2Banks          │ 32                                            │
+├──────────────────┼───────────────────────────────────────────────┤
+│ name             │ mi200                                         │
+├──────────────────┼───────────────────────────────────────────────┤
+│ numSQC           │ 56                                            │
+├──────────────────┼───────────────────────────────────────────────┤
+│ hbmBW            │ 1638.4                                        │
+├──────────────────┼───────────────────────────────────────────────┤
+│ ip_blocks        │ roofline|SQ|LDS|SQC|TA|TD|TCP|TCC|SPI|CPC|CPF │
+╘══════════════════╧═══════════════════════════════════════════════╛
+
+
+--------------------------------------------------------------------------------
+2. System Speed-of-Light
+....
+```
+ 2. Use `--list-metrics` to generate a list of availible metrics for inspection
+ ```shell-session
+$ omniperf analyze -p workloads/vcopy/mi200/ --list-metrics gfx90a
+╒═════════╤═════════════════════════════╕
+│         │ Metric                      │
+╞═════════╪═════════════════════════════╡
+│ 0       │ Top Stat                    │
+├─────────┼─────────────────────────────┤
+│ 1       │ System Info                 │
+├─────────┼─────────────────────────────┤
+│ 2.1.0   │ VALU_FLOPs                  │
+├─────────┼─────────────────────────────┤
+│ 2.1.1   │ VALU_IOPs                   │
+├─────────┼─────────────────────────────┤
+│ 2.1.2   │ MFMA_FLOPs_(BF16)           │
+├─────────┼─────────────────────────────┤
+│ 2.1.3   │ MFMA_FLOPs_(F16)            │
+├─────────┼─────────────────────────────┤
+│ 2.1.4   │ MFMA_FLOPs_(F32)            │
+├─────────┼─────────────────────────────┤
+│ 2.1.5   │ MFMA_FLOPs_(F64)            │
+├─────────┼─────────────────────────────┤
+│ 2.1.6   │ MFMA_IOPs_(Int8)            │
+├─────────┼─────────────────────────────┤
+│ 2.1.7   │ Active_CUs                  │
+├─────────┼─────────────────────────────┤
+│ 2.1.8   │ SALU_Util                   │
+├─────────┼─────────────────────────────┤
+│ 2.1.9   │ VALU_Util                   │
+├─────────┼─────────────────────────────┤
+│ 2.1.10  │ MFMA_Util                   │
+├─────────┼─────────────────────────────┤
+│ 2.1.11  │ VALU_Active_Threads/Wave    │
+├─────────┼─────────────────────────────┤
+│ 2.1.12  │ IPC_-_Issue                 │
+├─────────┼─────────────────────────────┤
+│ 2.1.13  │ LDS_BW                      │
+├─────────┼─────────────────────────────┤
+│ 2.1.14  │ LDS_Bank_Conflict           │
+├─────────┼─────────────────────────────┤
+│ 2.1.15  │ Instr_Cache_Hit_Rate        │
+├─────────┼─────────────────────────────┤
+│ 2.1.16  │ Instr_Cache_BW              │
+├─────────┼─────────────────────────────┤
+│ 2.1.17  │ Scalar_L1D_Cache_Hit_Rate   │
+├─────────┼─────────────────────────────┤
+│ 2.1.18  │ Scalar_L1D_Cache_BW         │
+├─────────┼─────────────────────────────┤
+│ 2.1.19  │ Vector_L1D_Cache_Hit_Rate   │
+├─────────┼─────────────────────────────┤
+│ 2.1.20  │ Vector_L1D_Cache_BW         │
+├─────────┼─────────────────────────────┤
+│ 2.1.21  │ L2_Cache_Hit_Rate           │
+├─────────┼─────────────────────────────┤
+│ 2.1.22  │ L2-Fabric_Read_BW           │
+├─────────┼─────────────────────────────┤
+│ 2.1.23  │ L2-Fabric_Write_BW          │
+├─────────┼─────────────────────────────┤
+│ 2.1.24  │ L2-Fabric_Read_Latency      │
+├─────────┼─────────────────────────────┤
+│ 2.1.25  │ L2-Fabric_Write_Latency     │
+├─────────┼─────────────────────────────┤
+...
+ ```
+ 2. Choose your own customized subset of metrics with `-b` (a.k.a. `--metric`), or build your own config following [config_template](https://github.com/AMDResearch/omniperf/blob/main/src/omniperf_analyze/configs/panel_config_template.yaml). Below shows how to generate a report containing only metric 2 (a.k.a. System Speed-of-Light).
+```shell-session
+$ omniperf analyze -p workloads/vcopy/mi200/ -b 2
+--------
+Analyze
+--------
+
+--------------------------------------------------------------------------------
+0. Top Stat
+╒════╤══════════════════════════════════════════╤═════════╤═══════════╤════════════╤══════════════╤════════╕
+│    │ KernelName                               │   Count │   Sum(ns) │   Mean(ns) │   Median(ns) │    Pct │
+╞════╪══════════════════════════════════════════╪═════════╪═══════════╪════════════╪══════════════╪════════╡
+│  0 │ vecCopy(double*, double*, double*, int,  │       1 │  20000.00 │   20000.00 │     20000.00 │ 100.00 │
+│    │ int) [clone .kd]                         │         │           │            │              │        │
+╘════╧══════════════════════════════════════════╧═════════╧═══════════╧════════════╧══════════════╧════════╛
+
+
+--------------------------------------------------------------------------------
+2. System Speed-of-Light
+╒═════════╤═══════════════════════════╤═══════════════════════╤══════════════════╤════════════════════╤════════════════════════╕
+│ Index   │ Metric                    │ Value                 │ Unit             │ Peak               │ PoP                    │
+╞═════════╪═══════════════════════════╪═══════════════════════╪══════════════════╪════════════════════╪════════════════════════╡
+│ 2.1.0   │ VALU FLOPs                │ 0.0                   │ Gflop            │ 22630.4            │ 0.0                    │
+├─────────┼───────────────────────────┼───────────────────────┼──────────────────┼────────────────────┼────────────────────────┤
+│ 2.1.1   │ VALU IOPs                 │ 367.0016              │ Giop             │ 22630.4            │ 1.6217194570135745     │
+├─────────┼───────────────────────────┼───────────────────────┼──────────────────┼────────────────────┼────────────────────────┤
+│ 2.1.2   │ MFMA FLOPs (BF16)         │ 0.0                   │ Gflop            │ 90521.6            │ 0.0                    │
+├─────────┼───────────────────────────┼───────────────────────┼──────────────────┼────────────────────┼────────────────────────┤
+│ 2.1.3   │ MFMA FLOPs (F16)          │ 0.0                   │ Gflop            │ 181043.2           │ 0.0                    │
+├─────────┼───────────────────────────┼───────────────────────┼──────────────────┼────────────────────┼────────────────────────┤
+│ 2.1.4   │ MFMA FLOPs (F32)          │ 0.0                   │ Gflop            │ 45260.8            │ 0.0                    │
+├─────────┼───────────────────────────┼───────────────────────┼──────────────────┼────────────────────┼────────────────────────┤
+│ 2.1.5   │ MFMA FLOPs (F64)          │ 0.0                   │ Gflop            │ 45260.8            │ 0.0                    │
+├─────────┼───────────────────────────┼───────────────────────┼──────────────────┼────────────────────┼────────────────────────┤
+│ 2.1.6   │ MFMA IOPs (Int8)          │ 0.0                   │ Giop             │ 181043.2           │ 0.0                    │
+├─────────┼───────────────────────────┼───────────────────────┼──────────────────┼────────────────────┼────────────────────────┤
+│ 2.1.7   │ Active CUs                │ 74                    │ Cus              │ 104                │ 71.15384615384616      │
+├─────────┼───────────────────────────┼───────────────────────┼──────────────────┼────────────────────┼────────────────────────┤
+│ 2.1.8   │ SALU Util                 │ 4.016057506716307     │ Pct              │ 100                │ 4.016057506716307      │
+├─────────┼───────────────────────────┼───────────────────────┼──────────────────┼────────────────────┼────────────────────────┤
+│ 2.1.9   │ VALU Util                 │ 5.737225009594725     │ Pct              │ 100                │ 5.737225009594725      │
+├─────────┼───────────────────────────┼───────────────────────┼──────────────────┼────────────────────┼────────────────────────┤
+│ 2.1.10  │ MFMA Util                 │ 0.0                   │ Pct              │ 100                │ 0.0                    │
+├─────────┼───────────────────────────┼───────────────────────┼──────────────────┼────────────────────┼────────────────────────┤
+│ 2.1.11  │ VALU Active Threads/Wave  │ 64.0                  │ Threads          │ 64                 │ 100.0                  │
+├─────────┼───────────────────────────┼───────────────────────┼──────────────────┼────────────────────┼────────────────────────┤
+│ 2.1.12  │ IPC - Issue               │ 1.0                   │ Instr/cycle      │ 5                  │ 20.0                   │
+├─────────┼───────────────────────────┼───────────────────────┼──────────────────┼────────────────────┼────────────────────────┤
+│ 2.1.13  │ LDS BW                    │ 0.0                   │ Gb/sec           │ 22630.4            │ 0.0                    │
+├─────────┼───────────────────────────┼───────────────────────┼──────────────────┼────────────────────┼────────────────────────┤
+│ 2.1.14  │ LDS Bank Conflict         │                       │ Conflicts/access │ 32                 │                        │
+├─────────┼───────────────────────────┼───────────────────────┼──────────────────┼────────────────────┼────────────────────────┤
+│ 2.1.15  │ Instr Cache Hit Rate      │ 99.91306912556854     │ Pct              │ 100                │ 99.91306912556854      │
+├─────────┼───────────────────────────┼───────────────────────┼──────────────────┼────────────────────┼────────────────────────┤
+│ 2.1.16  │ Instr Cache BW            │ 209.7152              │ Gb/s             │ 6092.8             │ 3.442016806722689      │
+├─────────┼───────────────────────────┼───────────────────────┼──────────────────┼────────────────────┼────────────────────────┤
+│ 2.1.17  │ Scalar L1D Cache Hit Rate │ 99.81986908342313     │ Pct              │ 100                │ 99.81986908342313      │
+├─────────┼───────────────────────────┼───────────────────────┼──────────────────┼────────────────────┼────────────────────────┤
+│ 2.1.18  │ Scalar L1D Cache BW       │ 209.7152              │ Gb/s             │ 6092.8             │ 3.442016806722689      │
+├─────────┼───────────────────────────┼───────────────────────┼──────────────────┼────────────────────┼────────────────────────┤
+│ 2.1.19  │ Vector L1D Cache Hit Rate │ 50.0                  │ Pct              │ 100                │ 50.0                   │
+├─────────┼───────────────────────────┼───────────────────────┼──────────────────┼────────────────────┼────────────────────────┤
+│ 2.1.20  │ Vector L1D Cache BW       │ 1677.7216             │ Gb/s             │ 11315.199999999999 │ 14.82714932126697      │
+├─────────┼───────────────────────────┼───────────────────────┼──────────────────┼────────────────────┼────────────────────────┤
+│ 2.1.21  │ L2 Cache Hit Rate         │ 35.55067615693325     │ Pct              │ 100                │ 35.55067615693325      │
+├─────────┼───────────────────────────┼───────────────────────┼──────────────────┼────────────────────┼────────────────────────┤
+│ 2.1.22  │ L2-Fabric Read BW         │ 419.8496              │ Gb/s             │ 1638.4             │ 25.6255859375          │
+├─────────┼───────────────────────────┼───────────────────────┼──────────────────┼────────────────────┼────────────────────────┤
+│ 2.1.23  │ L2-Fabric Write BW        │ 293.9456              │ Gb/s             │ 1638.4             │ 17.941015625           │
+├─────────┼───────────────────────────┼───────────────────────┼──────────────────┼────────────────────┼────────────────────────┤
+│ 2.1.24  │ L2-Fabric Read Latency    │ 256.6482321288385     │ Cycles           │                    │                        │
+├─────────┼───────────────────────────┼───────────────────────┼──────────────────┼────────────────────┼────────────────────────┤
+│ 2.1.25  │ L2-Fabric Write Latency   │ 317.2264255699014     │ Cycles           │                    │                        │
+├─────────┼───────────────────────────┼───────────────────────┼──────────────────┼────────────────────┼────────────────────────┤
+│ 2.1.26  │ Wave Occupancy            │ 1821.723057333852     │ Wavefronts       │ 3328               │ 54.73927455931046      │
+├─────────┼───────────────────────────┼───────────────────────┼──────────────────┼────────────────────┼────────────────────────┤
+│ 2.1.27  │ Instr Fetch BW            │ 4.174722306564298e-08 │ Gb/s             │ 3046.4             │ 1.3703789084047721e-09 │
+├─────────┼───────────────────────────┼───────────────────────┼──────────────────┼────────────────────┼────────────────────────┤
+│ 2.1.28  │ Instr Fetch Latency       │ 21.729248046875       │ Cycles           │                    │                        │
+╘═════════╧═══════════════════════════╧═══════════════════════╧══════════════════╧════════════════════╧════════════════════════╛
+```
+> **Note:** Some cells may be blank indicating a missing/unavailable hardware counter or NULL value
+
+3. Optimizatize application, iterate, and re-profile to inspect performance changes.
+4. Redo a comprehensive analysis with Omniperf CLI at any milestone or at the end.
+
+### Demo
+
+- Single run
+  ```shell
+  $ omniperf analyze -p workloads/vcopy/mi200/
+  ```
+
+- List top kernels
+  ```shell
+  $ omniperf analyze -p workloads/vcopy/mi200/  --list-kernels
+  ```
+
+- List metrics
+
+  ```shell
+  $ omniperf analyze -p workloads/vcopy/mi200/  --list-metrics gfx90a
+  ```
+
+- Customized profiling "System Speed-of-Light" and "CS_Busy" only
+  
+  ```shell
+  $ omniperf analyze -p workloads/vcopy/mi200/  -b 2  5.1.0
+  ```
+
+  > Note: Users can filter single metric or the whole IP block by its id. In this case, 1 is the id for "system speed of light" and 5.1.0 the id for metric "GPU Busy Cycles".
+
+- Filter kernels
+
+  First, list the top kernels in your application using `--list-kernels`.
+  ```shell-session
+  $ omniperf analyze -p workloads/vcopy/mi200/ --list-kernels
+  
+  --------
+  Analyze
+  --------
+
+
+  --------------------------------------------------------------------------------
+  Detected Kernels
+  ╒════╤══════════════════════════════════════════════════════════╕
+  │    │ KernelName                                               │
+  ╞════╪══════════════════════════════════════════════════════════╡
+  │  0 │ vecCopy(double*, double*, double*, int, int) [clone .kd] │
+  ╘════╧══════════════════════════════════════════════════════════╛
+
+  ```
+
+  Second, select the index of the kernel you'd like to filter (i.e. __vecCopy(double*, double*, double*, int, int) [clone .kd]__ at index __0__). Then, use this index to apply the filter via `-k/--kernels`.
+
+  ```shell-session
+  $ omniperf -p workloads/vcopy/mi200/ -k 0
+  
+  --------
+  Analyze
+  --------
+
+
+  --------------------------------------------------------------------------------
+  0. Top Stat
+  ╒════╤══════════════════════════════════════════╤═════════╤═══════════╤════════════╤══════════════╤════════╤═════╕
+  │    │ KernelName                               │   Count │   Sum(ns) │   Mean(ns) │   Median(ns) │    Pct │ S   │
+  ╞════╪══════════════════════════════════════════╪═════════╪═══════════╪════════════╪══════════════╪════════╪═════╡
+  │  0 │ vecCopy(double*, double*, double*, int,  │       1 │  20800.00 │   20800.00 │     20800.00 │ 100.00 │ *   │
+  │    │ int) [clone .kd]                         │         │           │            │              │        │     │
+  ╘════╧══════════════════════════════════════════╧═════════╧═══════════╧════════════╧══════════════╧════════╧═════╛
+  ... ...
+  ```
+  
+  > Note: You'll see your filtered kernel(s) indicated by a asterisk in the Top Stats table
+
+
+- Baseline comparison
+  
+  ```shell
+  omniperf analyze -p workload1/path/  -p workload2/path/
+  ```
+  > Note: You can also apply diffrent filters to each workload.
+  
+  OR
+  ```shell
+  omniperf analyze -p workload1/path/ -k 0  -p workload2/path/ -k 1
+  ```
+
+## GUI Analysis
+
+### Web-based GUI
+
+#### Features
+
+Omniperf's standalone GUI analyzer is a lightweight web page that can
+be generated directly from the command-line. This option is provided
+as an alternative for users wanting to explore profiling results
+graphically, but without the additional setup requirements or
+server-side overhead of Omniperf's detailed [Grafana
+interface](https://amdresearch.github.io/omniperf/analysis.html#grafana-based-gui)
+option.  The standalone GUI analyzer is provided as simple
+[Flask](https://flask.palletsprojects.com/en/2.2.x/) application
+allowing users to view results from within a web browser.
+
+```{admonition} Port forwarding
+
+Note that the standalone GUI analyzer publishes a web interface on port 8050 by default.
+On production HPC systems where profiling jobs run
+under the auspices of a resource manager, additional SSH tunneling
+between the desired web browser host (e.g. login node or remote workstation) and compute host may be
+required. Alternatively, users may find it more convenient to download
+profiled workloads to perform analysis on their local system.
+
+See [FAQ](https://amdresearch.github.io/omniperf/faq.html) for more details on SSH tunneling.
+```
+
+#### Usage
+
+To launch the standalone GUI, include the `--gui` flag with your desired analysis command. For example:
+
+```shell-session
+$ omniperf analyze -p workloads/vcopy/mi200/ --gui
+
+--------
+Analyze
+--------
+
+Dash is running on http://0.0.0.0:8050/
+
+ * Serving Flask app 'omniperf_analyze.omniperf_analyze' (lazy loading)
+ * Environment: production
+   WARNING: This is a development server. Do not use it in a production deployment.
+   Use a production WSGI server instead.
+ * Debug mode: off
+ * Running on all addresses (0.0.0.0)
+   WARNING: This is a development server. Do not use it in a production deployment.
+ * Running on http://127.0.0.1:8050
+ * Running on http://10.228.32.139:8050 (Press CTRL+C to quit)
+```
+
+At this point, users can then launch their web browser of choice and
+go to http://localhost:8050/ to see an analysis page.
+
+
+
+![Standalone GUI Homepage](images/standalone_gui.png)
+
+```{tip}
+To launch the web application on a port other than 8050, include an optional port argument:  
+`--gui <desired port>`
+```
+
+When no filters are applied, users will see five basic sections derived from their application's profiling data:
+
+1. Memory Chart Analysis
+2. Empirical Roofline Analysis
+3. Top Stats (Top Kernel Statistics)
+4. System Info
+5. System Speed-of-Light
+
+To dive deeper, use the top drop down menus to isolate particular
+kernel(s) or dispatch(s). You will then see the web page update with
+metrics specific to the filter you've applied.
+
+Once you have applied a filter, you will also see several additional
+sections become available with detailed metrics specific to that area
+of AMD hardware. These detailed sections mirror the data displayed in
+Omniperf's [Grafana
+interface](https://amdresearch.github.io/omniperf/analysis.html#grafana-based-gui).
+
+### Grafana-based GUI
+
+#### Features
+The Omniperf Grafana GUI Analyzer supports the following features to facilitate MI GPU performance profiling and analysis:
+
+- System and IP-Block Speed-of-Light (SOL)
+- Multiple normalization options, including per-cycle, per-wave, per-kernel and per-second.
+- Baseline comparisons 
+- Regex based Dispatch ID filtering
+- Roofline Analysis
+- Detailed per IP Block performance counters and metrics
+  - CPC/CPF
+  - SPI
+  - SQ
+  - SQC
+  - TA/TD
+  - TCP
+  - TCC (both aggregated and per-channel perf info)
+
+##### Speed-of-Light
+Speed-of-light panels are provided at both the system and per IP block level to help diagnosis performance bottlenecks. The performance numbers of the workload under testing are compared to the theoretical maximum, (e.g. floating point operations, bandwidth, cache hit rate, etc.), to indicate the available room to further utilize the hardware capability.
+
+##### Multi Normalization
+
+Multiple performance number normalizations are provided to allow performance inspection within both HW and SW context. The following normalizations are permitted:
+- per cycle
+- per wave
+- per kernel
+- per second
+
+##### Baseline Comparison
+Omniperf enables baseline comparison to allow checking A/B effect. The current release limits the baseline comparison to the same SoC. Cross comparison between SoCs is in development.
+
+For both the Current Workload and the Baseline Workload, one can independently setup the following filters to allow fine grained comparions:
+- Workload Name 
+- GPU ID filtering (multi selection)
+- Kernel Name filtering (multi selection)
+- Dispatch ID filtering (Regex filtering)
+- Omniperf Panels (multi selection)
+
+##### Regex based Dispatch ID filtering
+This release enables regex based dispatch ID filtering to flexibly choose the kernel invocations. One may refer to [Regex Numeric Range Generator](https://3widgets.com/), to generate typical number ranges. 
+
+For example, if one wants to inspect Dispatch Range from 17 to 48, inclusive, the corresponding regex is : **(1[7-9]|[23]\d|4[0-8])**. The generated express can be copied over for filtering.
+
+##### Incremental Profiling
+Omniperf supports incremental profiling to significantly speed up performance analysis.
+
+> Refer to [*IP Block profiling*](https://amdresearch.github.io/omniperf/profiling.html#ip-block-profiling) section for this command. 
+
+By default, the entire application is profiled to collect perfmon counter for all IP blocks, giving a system level view of where the workload stands in terms of performance optimization opportunities and bottlenecks. 
+
+After that one may focus on only a few IP blocks, (e.g., L1 Cache or LDS) to closely check the effect of software optimizations, without performing application replay for all other IP Blocks. This saves lots of compute time. In addition, the prior profiling results for other IP blocks are not overwritten. Instead, they can be merged during the import to piece together the system view. 
+
+##### Color Coding
+The uniform color coding is applied to most visualizations (bars, table, diagrams etc). Typically, Yellow color means over 50%, while Red color mean over 90% percent, for easy inspection.
+
+##### Global Variables and Configurations
+
+![Grafana GUI Global Variables](images/global_variables.png)
+
+#### Grafana GUI Import
+The omniperf database `--import` option imports the raw profiling data to Grafana's backend MongoDB database. This step is only required for Grafana GUI based performance analysis. 
+
+Default username and password for MongoDB (to be used in database mode) are as follows:
+
+ - Username: **temp**
+ - Password: **temp123**
+
+Each workload is imported to a separate database with the following naming convention:
+
+    omniperf_<team>_<database>_<soc>
+
+e.g., omniperf_asw_vcopy_mi200.
+
+When using database mode, be sure to tailor the connection options to the machine hosting your [sever-side instance](./installation.md). Below is the sample command to import the *vcopy* profiling data, lets assuming our host machine is called "dummybox".
+
+```shell-session
+$ omniperf database --help
+ROC Profiler:  /usr/bin/rocprof
+
+usage: 
+                                        
+omniperf database <interaction type> [connection options]
+
+                                        
+
+-------------------------------------------------------------------------------
+                                        
+Examples:
+                                        
+        omniperf database --import -H pavii1 -u temp -t asw -w workloads/vcopy/mi200/
+                                        
+        omniperf database --remove -H pavii1 -u temp -w omniperf_asw_sample_mi200
+                                        
+-------------------------------------------------------------------------------
+
+                                        
+
+Help:
+  -h, --help             show this help message and exit
+
+General Options:
+  -v, --version          show program's version number and exit
+  -V, --verbose          Increase output verbosity
+
+Interaction Type:
+  -i, --import                                          Import workload to Omniperf DB
+  -r, --remove                                          Remove a workload from Omniperf DB
+
+Connection Options:
+  -H , --host                                           Name or IP address of the server host.
+  -P , --port                                           TCP/IP Port. (DEFAULT: 27018)
+  -u , --username                                       Username for authentication.
+  -p , --password                                       The user's password. (will be requested later if it's not set)
+  -t , --team                                           Specify Team prefix.
+  -w , --workload                                       Specify name of workload (to remove) or path to workload (to import)
+  -k , --kernelVerbose                                  Specify Kernel Name verbose level 1-5. 
+                                                        Lower the level, shorter the kernel name. (DEFAULT: 2) (DISABLE: 5)
+```
+
+**omniperf import for vcopy:**
+```shell-session
+$ omniperf database --import -H dummybox -u temp -t asw -w workloads/vcopy/mi200/
+ROC Profiler:  /usr/bin/rocprof
+ 
+--------
+Import Profiling Results
+--------
+ 
+Pulling data from  /home/amd/xlu/test/workloads/vcopy/mi200
+The directory exists
+Found sysinfo file
+KernelName shortening enabled
+Kernel name verbose level: 2
+Password:
+Password recieved
+-- Conversion & Upload in Progress --
+  0%|                                                                                                                                                                                                             | 0/11 [00:00<?, ?it/s]/home/amd/xlu/test/workloads/vcopy/mi200/SQ_IFETCH_LEVEL.csv
+  9%|█████████████████▉                                                                                                                                                                                   | 1/11 [00:00<00:01,  8.53it/s]/home/amd/xlu/test/workloads/vcopy/mi200/pmc_perf.csv
+ 18%|███████████████████████████████████▊                                                                                                                                                                 | 2/11 [00:00<00:01,  6.99it/s]/home/amd/xlu/test/workloads/vcopy/mi200/SQ_INST_LEVEL_SMEM.csv
+ 27%|█████████████████████████████████████████████████████▋                                                                                                                                               | 3/11 [00:00<00:01,  7.90it/s]/home/amd/xlu/test/workloads/vcopy/mi200/SQ_LEVEL_WAVES.csv
+ 36%|███████████████████████████████████████████████████████████████████████▋                                                                                                                             | 4/11 [00:00<00:00,  8.56it/s]/home/amd/xlu/test/workloads/vcopy/mi200/SQ_INST_LEVEL_LDS.csv
+ 45%|█████████████████████████████████████████████████████████████████████████████████████████▌                                                                                                           | 5/11 [00:00<00:00,  9.00it/s]/home/amd/xlu/test/workloads/vcopy/mi200/SQ_INST_LEVEL_VMEM.csv
+ 55%|███████████████████████████████████████████████████████████████████████████████████████████████████████████▍                                                                                         | 6/11 [00:00<00:00,  9.24it/s]/home/amd/xlu/test/workloads/vcopy/mi200/sysinfo.csv
+ 64%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▎                                                                       | 7/11 [00:00<00:00,  9.37it/s]/home/amd/xlu/test/workloads/vcopy/mi200/roofline.csv
+ 82%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏                                   | 9/11 [00:00<00:00, 12.60it/s]/home/amd/xlu/test/workloads/vcopy/mi200/timestamps.csv
+100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 11/11 [00:00<00:00, 11.05it/s]
+9 collections added.
+Workload name uploaded
+-- Complete! --
+```
+
+#### Omniperf Panels
+
+##### Overview
+
+There are currently 18 main panel categories available for analyzing the compute workload performance. Each category contains several panels for close inspection of the system performance.
+
+- Kernel Statistics
+  - Kernel time histogram
+  - Top Ten bottleneck kernels
+- System Speed-of-Light
+  - Speed-of-Light
+  - System Info table
+- Memory Chart Analysis
+- Roofline Analysis
+  - FP32/FP64
+  - FP16/INT8
+- Command Processor
+  - Command Processor - Fetch (CPF)
+  - Command Processor - Controller (CPC)
+- Shader Processing Input (SPI)
+  - SPI Stats
+  - SPI Resource Allocations
+- Wavefront Launch
+  - Wavefront Launch Stats
+  - Wavefront runtime stats
+  - per-SE Wavefront Scheduling performance
+- Wavefront Lifetime
+  - Wavefront lifetime breakdown
+  - per-SE wavefront life (average)
+  - per-SE wavefront life (histogram)
+- Wavefront Occupancy
+  - per-SE wavefront occupancy
+  - per-CU wavefront occupancy
+- Compute Unit - Instruction Mix
+  - per-wave Instruction mix
+  - per-wave VALU Arithmetic instruction mix
+  - per-wave MFMA Arithmetic instruction mix
+- Compute Unit - Compute Pipeline
+  - Speed-of-Light: Compute Pipeline
+  - Arithmetic OPs count
+  - Compute pipeline stats
+  - Memory latencies
+- Local Data Share (LDS)
+  - Speed-of-Light: LDS
+  - LDS stats
+- Instruction Cache
+  - Speed-of-Light: Instruction Cache
+  - Instruction Cache Accesses
+- Constant Cache
+  - Speed-of-Light: Constant Cache
+  - Constant Cache Accesses
+  - Constant Cache - L2 Interface stats
+- Texture Address and Texture Data
+  - Texture Address (TA)
+  - Texture Data (TD)
+- L1 Cache
+  - Speed-of-Light: L1 Cache
+  - L1 Cache Accesses
+  - L1 Cache Stalls
+  - L1 - L2 Transactions
+  - L1 - UTCL1 Interface stats
+- L2 Cache
+  - Speed-of-Light: L2 Cache
+  - L2 Cache Accesses
+  - L2 - EA Transactions
+  - L2 - EA Stalls
+- L2 Cache Per Channel Performance
+  - Per-channel L2 Hit rate
+  - Per-channel L1-L2 Read requests
+  - Per-channel L1-L2 Write Requests
+  - Per-channel L1-L2 Atomic Requests
+  - Per-channel L2-EA Read requests
+  - Per-channel L2-EA Write requests
+  - Per-channel L2-EA Atomic requests
+  - Per-channel L2-EA Read latency
+  - Per-channel L2-EA Write latency
+  - Per-channel L2-EA Atomic  latency
+  - Per-channel L2-EA Read stall (I/O, GMI, HBM)
+  - Per-channel L2-EA Write stall (I/O, GMI, HBM, Starve)
+
+Most panels are designed around a specific IP block to thoroughly understand its behavior. Additional panels, including custom panels, could also be added to aid the performance analysis.
+
+##### System Info Panel
+![System Info Panel](images/System_info_panel.png)
+##### Kernel Statistics
+
+###### Kernel Time Histogram
+![Kernel Time Histogram](images/Kernel_time_histogram.png)
+###### Top Bottleneck Kernels
+![Top Bottleneck Kernels](images/Top_bottleneck_kernels.png)
+###### Top Bottleneck Dispatches
+![Top Bottleneck Dispatches](images/Top_bottleneck_dispatches.png)
+###### Current and Baseline Dispatch IDs (Filtered)
+![Current and Baseline Dispatch IDs](images/Current_and_baseline_dispatch_ids.png)
+
+##### System Speed-of-Light
+![System Speed-of-Light](images/System_speed_of_light.png)
+
+##### Memory Chart Analysis
+> Note: The Memory Chart Analysis support multiple normalizations. Due to the space limit, all transactions, when normalized to per-sec, default to unit of Billion transactions per second.
+
+![Memory Chart Analysis](images/Memory_chart_analysis.png)
+
+##### Roofline Analysis
+![Roofline Analysis](images/Roofline_analysis.png)
+##### Command Processor
+![Command Processor](images/Command_processor.png)
+##### Shader Processing Input (SPI)
+![Shader Processing Input](images/Shader_processing_input.png)
+##### Wavefront Launch
+![Wavefront Launch](images/Wavefront_launch.png)
+
+##### Compute Unit - Instruction Mix
+###### Instruction Mix
+![Instruction Mix](images/Instruction_mix.png)
+###### VALU Arithmetic Instruction Mix
+![VALU Arithmetic Instruction Mix](images/VALU_arithmetic_instruction_mix.png)
+###### MFMA Arithmetic Instruction Mix
+![MFMA Arithmetic Instruction Mix](images/MFMA_arithmetic_instruction_mix.png)
+###### VMEM Arithmetic Instruction Mix
+![VMEM Arithmetic Instruction Mix](images/VMEM_arithmetic_intensity_mix.png)
+
+##### Compute Unit - Compute Pipeline
+###### Speed-of-Light
+![Speed-of-Light](images/Comp_pipe_sol.png)
+###### Compute Pipeline Stats
+![Compute Pipeline Stats](images/Compute_pipeline_stats.png)
+###### Arithmetic Operations
+![Arithmetic Operations](images/Arithmetic_operations.png)
+###### Memory Latencies
+![Memory Latencies](images/Memory_latencies.png)
+
+##### Local Data Share (LDS)
+###### Speed-of-Light
+![Speed-of-Light](images/LDS_sol.png)
+###### LDS Stats
+![LDS Stats](images/LDS_stats.png)
+
+##### Instruction Cache
+###### Speed-of-Light
+![Speed-of-Light](images/Instruc_cache_sol.png)
+###### Instruction Cache Stats
+![Instruction Cache Stats](images/Instruction_cache_stats.png)
+
+##### Scalar L1D Cache
+###### Speed-of-Light
+![](images/L1D_sol.png)
+###### Constant Cache Stats
+![Constant Cache Stats](images/Vec_L1D_cache_accesses.png)
+###### Constant Cache - L2 Interface
+![Constant Cache - L2 Interface](images/Constant_cache_l2_interface.png)
+
+##### Texture Address and Texture Data
+###### Texture Address (TA)
+![Texture Address](images/Texture_address.png)
+###### Texture Data (TD)
+![Texture Data](images/Texture_data.png)
+
+##### Vector L1D Cache
+###### Speed-of-Light
+![Speed-of-Light](images/Vec_L1D_cache_sol.png)
+###### Vector L1D Cache Accesses
+![Vector L1D Cache Accesses](images/Vec_L1D_cache_accesses.png)
+###### L1 Cache Stalls
+![L1 Cache Stalls](images/L1_cache_stalls.png)
+###### L1 - L2 Transactions
+![L1 - L2 Transactions](images/L1_l2_transactions.png)
+###### L1 - UTCL1 Interface Stats
+![L1 - UTCL1 Interface Stats](images/L1_utcl1_transactions.png)
+
+##### L2 Cache
+###### Speed-of-Light
+![Speed-of-Light](images/L2_cache_sol.png)
+###### L2 Cache Accesses
+![L2 Cache Accesses](images/L2_cache_accesses.png)
+###### L2 - EA Transactions
+![L2 - EA Transactions](images/L2_ea_transactions.png)
+###### L2 - EA Stalls
+![L2 - EA Stalls](images/L2_ea_stalls.png)
+
+##### L2 Cache Per Channel Performance
+###### L1-L2 Transactions
+![L1-L2 Transactions](images/L1_l2_transactions_per_channel.png)
+###### L2-EA Transactions
+![L2-EA Transactions](images/L2_ea_transactions_per_channel.png)
+###### L2-EA Latencies
+![L2-EA Latencies](images/L2_ea_latencies_per_channel.png)
+###### L2-EA Stalls
+![L2-EA Stalls](images/L2_ea_stalls_per_channel.png)
+###### L2-EA Write Stalls
+![L2-EA Write Stalls](images/L2_ea_write_stalls_per_channel.png)
+###### L2-EA Write Starvation
+![L2-EA Write Starvation](images/L2_ea_write_starvation_per_channel.png)
@@ -0,0 +1,177 @@
+# -*- coding: utf-8 -*-
+#
+# Configuration file for the Sphinx documentation builder.
+#
+# This file does only contain a selection of the most common options. For a
+# full list see the documentation:
+# http://www.sphinx-doc.org/en/master/config
+
+# -- Path setup --------------------------------------------------------------
+
+# If extensions (or modules to document with autodoc) are in another directory,
+# add these directories to sys.path here. If the directory is relative to the
+# documentation root, use os.path.abspath to make it absolute, like shown here.
+#
+import os
+import sys
+import subprocess as sp
+
+sys.path.insert(0, os.path.abspath(".."))
+
+repo_version = "unknown"
+# Determine short version by file in repo
+if os.path.isfile("../../VERSION"):
+    with open("../../VERSION") as f:
+        repo_version = f.readline().strip()
+
+
+def install(package):
+    sp.call([sys.executable, "-m", "pip", "install", package])
+
+
+# -- Project information -----------------------------------------------------
+
+project = "Omniperf"
+copyright = "2022, Audacious Software Group"
+author = "Audacious Software Group"
+
+# The short X.Y version
+version = repo_version
+# The full version, including alpha/beta/rc tags
+release = ""
+
+# -- General configuration ---------------------------------------------------
+
+install("sphinx_rtd_theme")
+
+# Add any Sphinx extension module names here, as strings. They can be
+# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
+# ones.
+extensions = [
+    "sphinx.ext.githubpages",
+    "myst_parser",
+]
+
+myst_heading_anchors = 2
+# enable replacement of (tm) & friends
+myst_enable_extensions = ["replacements"]
+
+# Add any paths that contain templates here, relative to this directory.
+templates_path = ["_templates"]
+
+# The suffix(es) of source filenames.
+# You can specify multiple suffix as a list of string:
+source_suffix = {
+    ".rst": "restructuredtext",
+    ".txt": "markdown",
+    ".md": "markdown",
+}
+
+from recommonmark.parser import CommonMarkParser
+
+source_parsers = {".md": CommonMarkParser}
+
+# The master toctree document.
+master_doc = "index"
+
+# The language for content autogenerated by Sphinx. Refer to documentation
+# for a list of supported languages.
+#
+# This is also used if you do content translation via gettext catalogs.
+# Usually you set "language" from the command line for these cases.
+language = "en"
+
+# List of patterns, relative to source directory, that match files and
+# directories to ignore when looking for source files.
+# This pattern also affects html_static_path and html_extra_path.
+exclude_patterns = ["_build", "Thumbs.db", ".DS_Store"]
+
+# The name of the Pygments (syntax highlighting) style to use.
+pygments_style = None
+
+# options for latex output
+latex_engine = "lualatex"
+latex_show_urls = "footnote"
+
+
+# -- Options for HTML output -------------------------------------------------
+
+# The theme to use for HTML and HTML Help pages.  See the documentation for
+# a list of builtin themes.
+#
+html_theme = "sphinx_rtd_theme"
+
+# Theme options are theme-specific and customize the look and feel of a theme
+# further.  For a list of options available for each theme, see the
+# documentation.
+#
+# html_theme_options = {}
+
+# Add any paths that contain custom static files (such as style sheets) here,
+# relative to this directory. They are copied after the builtin static files,
+# so a file named "default.css" will overwrite the builtin "default.css".
+html_static_path = ["_static"]
+
+
+# -- Options for HTMLHelp output ---------------------------------------------
+
+# Output file base name for HTML help builder.
+htmlhelp_basename = "Omniperfdoc"
+
+html_theme_options = {
+    "analytics_id": "G-C5DYLCE9ED",  #  Provided by Google in your dashboard
+    "analytics_anonymize_ip": False,
+    "logo_only": False,
+    "display_version": True,
+    "prev_next_buttons_location": "bottom",
+    "style_external_links": False,
+    "vcs_pageview_mode": "",
+    # 'style_nav_header_background': 'white',
+    # Toc options
+    "collapse_navigation": True,
+    "sticky_navigation": True,
+    "navigation_depth": 4,
+    "includehidden": True,
+    "titles_only": False,
+}
+
+from pygments.styles import get_all_styles
+
+# The name of the Pygments (syntax highlighting) style to use.
+styles = list(get_all_styles())
+preferences = ("emacs", "pastie", "colorful")
+for pref in preferences:
+    if pref in styles:
+        pygments_style = pref
+        break
+
+from recommonmark.transform import AutoStructify
+
+
+# app setup hook
+def setup(app):
+    app.add_config_value(
+        "recommonmark_config",
+        {
+            "auto_toc_tree_section": "Contents",
+            "enable_eval_rst": True,
+            "enable_auto_doc_ref": False,
+        },
+        True,
+    )
+    app.add_transform(AutoStructify)
+    app.add_config_value("docstring_replacements", {}, True)
+    app.connect("source-read", replaceString)
+
+
+# function to replace version string througout documentation
+
+
+def replaceString(app, docname, source):
+    result = source[0]
+    for key in app.config.docstring_replacements:
+        result = result.replace(key, app.config.docstring_replacements[key])
+    source[0] = result
+
+
+docstring_replacements = {"{__VERSION__}": version}
@@ -0,0 +1,55 @@
+# FAQ
+
+```eval_rst
+.. toctree::
+   :glob:
+   :maxdepth: 4
+```
+
+**1. How do I export profiling data I've already generated using Omniperf?**
+
+In order to interact with the Grafana GUI you must sync data with the MongoDB backend. This interaction is done through ***database*** mode.
+
+Simply pass the directory of your desired workload like so,
+```shell
+$ omniperf database --import -w <path-to-results> -H <hostname> -u <username> -t <team-name> 
+```
+**2. python ast error: 'Constant' object has no attribute 'kind'**
+
+This comes from a bug in the default astunparse 1.6.3 with python 3.8. Seems good with python 3.7 and 3.9.
+
+Workaround:
+```shell
+$ pip3 uninstall astunparse
+$ pip3 astunparse
+```
+
+**3. tabulate doesn't print properly**
+Workaround:
+```shell
+$ export LC_ALL=C.UTF-8
+$ export LANG=C.UTF-8
+```
+
+**3. How can I SSH Tunnel in MobaXterm?**
+
+1. Open MobaXterm
+2. In the top ribbon, select `Tunneling`
+![Tunnel Button](images/tunnel_demo1.png)
+This pop up will appear
+![Pop up](images/tunnel_demo2.png)
+3. Press `New SSH tunnel`
+![Pop up](images/tunnel_demo3.png)
+4. Configure tunnel accordingly
+
+   Local clients
+   - Forwarded Port: [PORT]
+   
+   Remote Server
+   - Remote Server: localhost
+   - Remote Port: [PORT]
+   
+   SSH Server
+   - SSH server: Name of the server one is connecting to
+   - SSH login: Username to login to the server
+   - SSH port: 22
@@ -0,0 +1,93 @@
+# Getting Started
+
+```eval_rst
+.. toctree::
+   :glob:
+   :maxdepth: 4
+```
+
+## Quickstart
+
+1. **Launch & Profile the target application with the command line profiler**
+   
+    The command line profiler launches the target application, calls the rocProfiler API, and collects profile results for the specified kernels, dispatches, and/or IP blocks.  If not specified, Omniperf will default to collecting all available counters for all kernels/dispatches launched by the user's executable.
+
+    To collect the default set of data for all kernels in the target application, launch, e.g.:
+    ```shell
+    $ omniperf profile -n vcopy_data -- ./vcopy 1048576 256
+    ```
+    The app runs, each kernel is launched, and profiling results are generated. By default, results are written to (e.g.,) ./workloads/vcopy_data (configurable via the `-n` argument). To collect all requested profile information, it may be required to replay kernels multiple times.
+
+2. **Customize data collection**
+    
+    Options are available to specify for which kernels/metrics data should be collected.
+    Note that filtering can be applied either in the profiling or analysis stage, however filtering at during profiling collection will often speed up your overall profiling run time.
+
+    Some common filters include:
+
+    - `-k`/`--kernel` enables filtering kernels by name. `-d`/`--dispatch` enables filtering based on dispatch ID
+    - `-b`/`--ipblocks` enables collects metrics for only the specified (one or more) IP Blocks.
+
+    To view available metrics by IP Block you can use the `--list-metrics` argument to view a list of all available metrics organized by IP Block.
+    ```shell
+    $ omniperf analyze --list-metrics <sys_arch>
+    ```
+
+3. **Analyze at the command line**
+   
+   After generating a local output folder (./workloads/\<name>), the command line tool can also be used to quickly interface with profiling results. View different metrics derived from your profiled results and get immediate access all metrics organized by IP block.
+
+   If no kernel, dispatch, or ipblock filters are applied at this stage, analysis will be reflective of the entirety of the profiling data.
+
+   To interact with profiling results from a different session, users just provide the workload path.  `-p`/`--path` enables users to analyze existing profiling data in the Omniperf CLI.
+
+4. **Analyze in the Grafana GUI**
+   
+   To conduct a more in-depth analysis of profiling results we recommend users utilize the Omniperf Grafana GUI. To interact with profiling results, users must import their data to the MongoDB instance included in the Omniperf dockerfile.
+
+    To interact with Grafana GUI data, stored in the Omniperf DB, users can enter ***database*** mode. For example:
+   ```shell
+    $ omniperf database --import [CONNECTION OPTIONS]
+   ```
+
+## Usage
+
+### Modes
+Modes change the fundamental behavior of the Omniperf command line tool. Depending on which mode is chosen, different command line options become available.
+
+- **Profile**: Target application is launched on the local system utilizing AMD’s [ROC Profiler](https://github.com/ROCm-Developer-Tools/rocprofiler). Depending on the profiling options chosen, selected kernels, dispatches, and/or IP Blocks in the application are profiled and results are stored locally in an output folder (./workloads/\<name>).
+
+    ```shell
+    $ omniperf profile --help
+    ```
+
+- **Analyze**: Profiling data from `-p`/`--path` directory is loaded into the Omniperf CLI analyzer where users have immediate access to profiling results and generated metrics. Metrics are quickly generated from the entirety of your profiled application or a subset you’ve identified through the Omniperf CLI analysis filters.
+
+    To gererate a lightweight GUI interface users can add the `--gui` flag to their analysis command.
+
+    This mode is designed to be a middle ground to the highly detailed Omniperf Grafana GUI and is great for users who want immediate access to an IP Block they’re already familiar with.
+
+    ```shell
+    $ omniperf analyze --help
+    ```
+
+- **Database**: Our detailed Grafana GUI is built on a MongoDB database. `--import` profiling results to the DB to interact with the workload in Grafana or `--remove` the workload from the DB.
+
+    Connection options will need to be specified. See the [*Grafana
+    Analysis*](https://amdresearch.github.io/omniperf/analysis.html#grafana-gui-import) import section
+    for more details on this.
+
+    ```shell
+    $ omniperf database --help
+    ```
+
+## Basic Operations
+
+Operation | Mode | Required Arguments
+:--|:--|:--
+Profile a workload | profile | `--name`, `-- <profile_cmd>`
+Standalone roofline analysis | profile | `--name`, `--roof-only`, `-- <profile_cmd>`
+Import a workload to database | database | `--import`, `--host`, `--username`, `--workload`, `--team`
+Remove a workload from database | database | `--remove`, `--host`, `--username`, `--workload`, `--team`
+Launch standalone GUI from CLI | analyze | `--path`, `--gui`
+Interact with profiling results from CLI | analyze | `--path`
@@ -0,0 +1,21 @@
+# High Level Design
+
+```eval_rst
+.. toctree::
+   :glob:
+   :maxdepth: 4
+```
+
+The [Omniperf](https://github.com/AMDResearch/omniperf) Tool is architecturally composed of three major components, as shown in the following figure.
+
+- **Omniperf Profiling**: Acquire raw performance counters via application replay based on the [rocProfiler](https://rocm.docs.amd.com/projects/rocprofiler/en/latest/rocprof.html).  The counters are stored in a comma-seperated value, for further analyis. A set of MI200 specific micro benchmarks are also run to acquire the hierarchical roofline data. The roofline model is not available on earlier accelerators.
+
+- **Omniperf Grafana Analyzer**: 
+  - *Grafana database import*: All raw performance counters are imported into the backend MongoDB database for Grafana GUI analysis and visualization. Compatibility of previously generated data between Omniperf versions is not necessarily guarenteed.
+  - *Grafana GUI Analyzer*: A Grafana dashboard is designed to retrieve the raw counters info from the backend database. It also creates the relevant performance metrics and visualization.
+- **Omniperf Standalone GUI Analyzer**: A standalone GUI is provided to enable performance analysis without importing data into the backend database.
+
+![Omniperf Architectual Diagram](images/omniperf_server_vs_client_install.png)
+
+> Note: To learn more about the client vs. server model of Omniperf and our install process please see the [Deployment section](./installation.md) of the docs.
+
@@ -0,0 +1,16 @@
+# Welcome to the [Omniperf](https://github.com/AMDResearch/omniperf) Documentation!
+
+```eval_rst
+.. toctree::
+   :glob:
+   :maxdepth: 4
+   :caption: Table of Contents
+
+   introduction
+   high_level_design
+   installation
+   getting_started
+   profiling
+   analysis
+   faq
+```
@@ -0,0 +1,243 @@
+# Deployment
+
+```eval_rst
+.. toctree::
+   :glob:
+   :maxdepth: 4
+```
+
+Omniperf is broken into two installation components:
+
+1. **Omniperf Client-side (_Required_)**
+   - Provides core application profiling capability
+   - Allows collection of performance counters, filtering by IP block, dispatch, kernel, etc
+   - CLI based analysis mode
+   - Stand alone web interface for importing analysis metrics
+2. **Omniperf Server-side (_Optional_)**
+   - Mongo DB backend + Grafana instance
+   - Packaged in a Docker container for easy setup
+
+Determine what you need to install based on how you'd like to interact with Omniperf. See the decision tree below to help determine what installation is right for you.
+
+![Omniperf Installtion Decision Tree](images/install_decision_tree.png)
+
+---
+
+## Client-side Installation
+
+Omniperf client-side requires the following basic software dependencies prior to usage:
+
+* Python (>=3.7)
+* CMake (>= 3.19)
+* ROCm (>= 5.2.0)
+
+In addition, Omniperf leverages a number of Python packages that are
+documented in the top-level `requirements.txt` file.  These must be
+installed prior to Omniperf configuration.  
+
+The recommended procedure for Omniperf usage is to install into a shared file system so that multiple users can access the final installation.  The following steps illustrate how to install the necessary python dependencies using [pip](https://packaging.python.org/en/latest/) and Omniperf into a shared location controlled by the `INSTALL_DIR` environment variable.
+
+```{admonition} Configuration variables
+The following installation example leverages several
+[CMake](https://cmake.org/cmake/help/latest/) project variables
+defined as follows:
+| Variable             | Description                                                          |
+| -------------------- | -------------------------------------------------------------------- |
+| CMAKE_INSTALL_PREFIX | controls install path for Omniperf files                             |
+| PYTHON_DEPS          | provides optional path to resolve Python package dependencies        |
+| MOD_INSTALL_PATH     | provides optional path for separate Omniperf modulefile installation |
+
+```
+
+A typical install will begin by downloading the latest release tarball
+available from the
+[Releases](https://github.com/AMDResearch/omniperf/releases) section
+of the Omniperf development site. From there, untar and descend into
+the top-level directory as follows:
+
+```shell
+$ tar xfz omniperf-v{__VERSION__}.tar.gz
+$ cd omniperf-v{__VERSION__}
+```
+
+Next, install Python dependencies and complete the Omniperf configuration/install process as follows:
+
+```shell
+# define top-level install path
+$ export INSTALL_DIR=<your-top-level-desired-install-path>
+
+# install python deps
+$ python3 -m pip install -t ${INSTALL_DIR}/python-libs -r requirements.txt
+
+# configure Omniperf for shared install
+$ mkdir build
+$ cd build
+$ cmake -DCMAKE_INSTALL_PREFIX=${INSTALL_DIR}/{__VERSION__} \
+        -DPYTHON_DEPS=${INSTALL_DIR}/python-libs \
+        -DMOD_INSTALL_PATH=${INSTALL_DIR}/modulefiles ..
+
+# install
+$ make install
+```
+
+```{tip}
+You may require `sudo` during the final install step if you
+do not have write access to the chosen install path.
+```
+
+
+After completing these steps, a successful top-level installation directory looks as follows:
+```shell
+$ ls $INSTALL_DIR
+modulefiles  {__VERSION__}  python-libs
+```
+
+### Execution using modulefiles
+
+The installation process includes creation of an environment
+modulefile for use with [Lmod](https://lmod.readthedocs.io). On
+systems that support Lmod, a user can register the Omniperf modulefile
+directory and setup their environment for execution of Omniperf as
+follows:
+
+
+
+```shell
+$ module use $INSTALL_DIR/modulefiles
+$ module load omniperf
+$ which omniperf
+/opt/apps/omniperf/{__VERSION__}/bin/omniperf
+
+$ omniperf --version
+ROC Profiler:   /opt/rocm-5.1.0/bin/rocprof
+
+omniperf (v{__VERSION__})
+```
+
+```{tip} Users relying on an Lmod Python module locally may wish to
+customize the resulting Omniperf modulefile post-installation to
+include additional module dependencies.
+```
+
+### Execution without modulefiles
+
+To use Omniperf without the companion modulefile, update your `PATH`
+settings to enable access to the command-line binary. If you installed Python
+dependencies in a shared location, update your `PYTHONPATH` config as well:
+
+```shell
+export PATH=$INSTALL_DIR/{__VERSION__}/bin:$PATH
+export PYTHONPATH=$INSTALL_DIR/python-libs
+```
+
+### rocProf
+
+Omniperf relies on a rocprof binary during the profiling
+process. Normally the path to this binary will be detected
+automatically, but it can also be overridden via the setting the
+optional `ROCPROF` environment variable to the path of the binary the user
+wishes to use instead.
+
+
+
+
+
+%%% ### Generate Packaging
+%%% ```console
+%%% cd build
+%%% cpack -G STGZ
+%%% cpack -G DEB -D CPACK_PACKAGING_INSTALL_PREFIX=/opt/omniperf
+%%% cpack -G RPM -D CPACK_PACKAGING_INSTALL_PREFIX=/opt/omniperf
+%%% ```
+
+---
+
+## Server-side Setup
+
+> Note: Server-side setup is not required to profile or analyze performance data from the CLI. It is provided as an additional mechanism to import performance data for examination within a detailed [Grafana](https://github.com/grafana/grafana) GUI.
+
+Omniperf server-side requires the following basic software dependencies prior to usage:
+
+* [Docker Engine](https://docs.docker.com/engine/install/)
+
+The recommended process for enabling the server-side of Omniperf is to use the provided Docker file to build the Grafana and MongoDB instance.
+
+Once you've decided which machine you'd like to use to host the Grafana and MongoDB instance, please follow the set up instructions below.
+
+### 1) Install MongoDB Utils
+Omniperf uses [mongoimport](https://www.mongodb.com/docs/database-tools/mongoimport/) to upload data to Grafana's backend database. Install for Ubuntu 20.04 is as follows:
+
+```bash 
+$ wget https://fastdl.mongodb.org/tools/db/mongodb-database-tools-ubuntu2004-x86_64-100.6.1.deb
+$ sudo apt install ./mongodb-database-tools-ubuntu2004-x86_64-100.6.1.deb
+```
+> Installation instructions for alternative distributions can be found [here](https://www.mongodb.com/download-center/database-tools/releases/archive)
+
+### 2) Persistent Storage
+
+The user will also bind MongoDB to a directory on the host OS to create a local backup in case of a crash or reset. In the Docker world, this is known as "creating a persistent volume":
+
+```bash
+$ sudo mkdir -p /usr/local/persist && cd /usr/local/persist/
+$ sudo mkdir -p grafana-storage mongodb
+$ sudo docker volume create --driver local --opt type=none --opt device=/usr/local/persist/grafana-storage --opt o=bind grafana-storage
+$ sudo docker volume create --driver local --opt type=none --opt device=/usr/local/persist/mongodb --opt o=bind grafana-mongo-db
+```
+
+### 3) Build and Launch
+
+We're now ready to build our Docker file. Navigate to your Omniperf install directory to begin.
+```bash
+$ sudo docker-compose build
+$ sudo docker-compose up -d
+```
+> Note that TCP ports for Grafana (4000) and MongoDB (27017) in the docker container are mapped to 14000 and 27018, respectively, on the host side.
+
+### 4) Setup Grafana Instance
+Once you've launced your docker container you should be able to reach Grafana at **http://\<host-ip>:14000**. The default login credentials for the first-time Grafana setup are:
+
+- Username: **admin**
+- Password: **admin**
+
+![Grafana Welcome Page](images/grafana_welcome.png)
+
+MongoDB Datasource Configuration
+
+The MongoDB Datasource must be configured prior to the first-time use. Navigate to Grafana's Configuration page (shown below) to add the **Omniperf Data** connection.
+
+![Omniperf Datasource Config](images/datasource_config.jpg)
+
+Configure the following fields in the datasource settings:
+
+- HTTP URL: set to *http://localhost:3333*
+- MongoDB URL: set to *mongodb://temp:temp123@\<host-ip>:27018/admin?authSource=admin*
+- Database Name: set to *admin*
+
+After properly configuring these fields click **Save & Test** (as shown below) to make sure your connection is successful.
+
+> Note to avoid potential DNS issue, one may need to use the actual IP address for the host node in the MongoDB URL.
+
+![Datasource Settings](images/datasource_settings.jpg)
+
+Omniperf Dashboard Import
+
+From *Create* → *Import*, (as shown below) upload the dashboard file, `/dashboards/Omniperf_v{__VERSION__}_pub.json`, from the Omniperf tarball.
+
+Edit both the Dashboard Name and the Unique Identifier (UID) to uniquely identify the dashboard he/she will use. Click Import to finish the process.
+
+![Import Dashboard](images/import_dashboard.png)
+
+Using your dashboard
+
+Once you've imported a dashboard you're ready to begin! Start by browsing availible dashboards and selecting the dashboard you've just imported.
+
+![Opening your dashboard](images/opening_dashboard.png)
+
+Remeber, you'll need to upload workload data to the DB backend before analyzing in your Grafana interface. We provide a detailed example of this in our [Analysis section](./analysis.md#grafana-gui-import).
+
+After a workload has been successfully uploaded, you should be able to select it from the workload dropdown located at the top of your Grafana dashboard.
+
+![Selecting Grafana workload](images/grafana_workload_selection.png)
+
+For more information on how to use the Grafana interface for anlysis please see the [Grafana section](./analysis.md#grafana-based-gui) in the Analyze Mode tab.
+
@@ -0,0 +1,57 @@
+# Introduction
+
+```eval_rst
+.. toctree::
+   :glob:
+   :maxdepth: 4
+```
+
+[Browse Omniperf source code on Github](https://github.com/AMDResearch/omniperf)
+
+## Scope
+
+MI Performance Profiler ([Omniperf](https://github.com/AMDResearch/omniperf)) is a system performance profiling tool for Machine Learning/HPC workloads running on AMD Instinct (tm) Accelerators. It is currently built on top of the [rocProfiler](https://rocm.docs.amd.com/projects/rocprofiler/en/latest/rocprof.html) to monitor hardware performance counters. The Omniperf tool primarily targets accelerators in the MI100 and MI200 families. Development is in progress to support MI300 and Radeon (tm) RDNA (tm) GPUs.
+
+## Features
+
+The Omniperf tool performs system profiling based on all available hardware counters for the target accelerator. It provides high level performance analysis features including System Speed-of-Light, IP block Speed-of-Light, Memory Chart Analysis, Roofline Analysis, Baseline Comparisons, and more...
+  
+Both command line analysis and GUI analysis are supported. 
+
+Detailed Feature List:
+- MI100 support
+- MI200 support
+- Standalone GUI Analyzer
+- Grafana/MongoDB GUI Analyzer
+- Dispatch Filtering
+- Kernel Filtering
+- GPU ID Filtering
+- Baseline Comparison
+- Multi-Normalizations
+- System Info Panel
+- System Speed-of-Light Panel
+- Kernel Statistic Panel
+- Memory Chart Analysis Panel
+- Roofline Analysis Panel (*Supported on MI200 only, SLES 15 SP3 or RHEL8*)
+- Command Processor (CP) Panel
+- Shader Processing Input (SPI) Panel
+- Wavefront Launch Panel
+- Compute Unit - Instruction Mix Panel
+- Compute Unit - Pipeline Panel
+- Local Data Share (LDS) Panel
+- Instruction Cache Panel
+- Scalar L1D Cache Panel
+- Texture Addresser and Data Panel
+- Vector L1D Cache Panel
+- L2 Cache Panel
+- L2 Cache (per-Channel) Panel
+
+## Compatible SOCs
+
+| Platform | Status         |
+| :------- | :------------- |
+| Vega 20 (MI-50/60)  | No  |
+| MI100    | Supported      |
+| MI200    | Supported      |
+| MI300    | In development |
+
@@ -0,0 +1,35 @@
+@ECHO OFF
+
+pushd %~dp0
+
+REM Command file for Sphinx documentation
+
+if "%SPHINXBUILD%" == "" (
+	set SPHINXBUILD=sphinx-build
+)
+set SOURCEDIR=.
+set BUILDDIR=_build
+
+if "%1" == "" goto help
+
+%SPHINXBUILD% >NUL 2>NUL
+if errorlevel 9009 (
+	echo.
+	echo.The 'sphinx-build' command was not found. Make sure you have Sphinx
+	echo.installed, then set the SPHINXBUILD environment variable to point
+	echo.to the full path of the 'sphinx-build' executable. Alternatively you
+	echo.may add the Sphinx directory to PATH.
+	echo.
+	echo.If you don't have Sphinx installed, grab it from
+	echo.http://sphinx-doc.org/
+	exit /b 1
+)
+
+%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS%
+goto end
+
+:help
+%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS%
+
+:end
+popd
@@ -0,0 +1,425 @@
+# Profile Mode
+
+```eval_rst
+.. toctree::
+   :glob:
+   :maxdepth: 5
+```
+
+The [Omniperf](https://github.com/AMDResearch/omniperf) repository
+includes source code for a sample GPU compute workload,
+__vcopy.cpp__. A copy of this file is available in the `share/sample`
+subdirectory after a normal Omniperf installation, or via the
+`$OMNIPERF_SHARE/sample` directory when using the supplied modulefile.
+
+A compiled version of this workload is used throughout the following
+sections to demonstrate the use of Omniperf in MI GPU performance
+analysis. Unless otherwise noted, the performance analysis is done on
+the MI200 platform.
+
+## Workload Compilation
+**vcopy compilation:**
+```shell-session
+$ hipcc vcopy.cpp -o vcopy
+$ ls
+vcopy   vcopy.cpp
+$ ./vcopy 1048576 256
+Finished allocating vectors on the CPU
+Finished allocating vectors on the GPU
+Finished copying vectors to the GPU
+sw thinks it moved 1.000000 KB per wave
+Total threads: 1048576, Grid Size: 4096 block Size:256, Wavefronts:16384:
+Launching the  kernel on the GPU
+Finished executing kernel
+Finished copying the output vector from the GPU to the CPU
+Releasing GPU memory
+Releasing CPU memory
+```
+
+## Omniperf Profiling
+The *omniperf* script, availible through the [Omniperf](https://github.com/AMDResearch/omniperf) repository, is used to aquire all necessary perfmon data through analysis of compute workloads.
+
+**omniperf help:**
+```shell-session
+$ omniperf profile --help
+ROC Profiler:  /usr/bin/rocprof
+
+usage:
+
+omniperf profile --name <workload_name> [profile options] [roofline options] -- <profile_cmd>
+
+
+
+-------------------------------------------------------------------------------
+
+Examples:
+
+        omniperf profile -n vcopy_all -- ./vcopy 1048576 256
+
+        omniperf profile -n vcopy_SPI_TCC -b SQ TCC -- ./vcopy 1048576 256
+
+        omniperf profile -n vcopy_kernel -k vecCopy -- ./vcopy 1048576 256
+
+        omniperf profile -n vcopy_disp -d 0 -- ./vcopy 1048576 256
+
+        omniperf profile -n vcopy_roof --roof-only -- ./vcopy 1048576 256
+
+-------------------------------------------------------------------------------
+
+
+
+Help:
+  -h, --help                      show this help message and exit
+
+General Options:
+  -v, --version                   show program's version number and exit
+  -V, --verbose                   Increase output verbosity
+
+Profile Options:
+  -n , --name                                           Assign a name to workload.
+  -p , --path                                           Specify path to save workload.
+                                                        (DEFAULT: /home/colramos/GitHub/omniperf/workloads/<name>)
+  -k  [ ...], --kernel  [ ...]                          Kernel filtering.
+  -b  [ ...], --ipblocks  [ ...]                        IP block filtering:
+                                                           SQ
+                                                           SQC
+                                                           TA
+                                                           TD
+                                                           TCP
+                                                           TCC
+                                                           SPI
+                                                           CPC
+                                                           CPF
+  -d  [ ...], --dispatch  [ ...]                        Dispatch ID filtering.
+  --no-roof                                             Profile without collecting roofline data.
+  -- [ ...]                                             Provide command for profiling after double dash.
+
+Standalone Roofline Options:
+  --roof-only                                           Profile roofline data only.
+  --sort                                                Overlay top kernels or top dispatches: (DEFAULT: kernels)
+                                                           kernels
+                                                           dispatches
+  -m , --mem-level                                      Filter by memory level: (DEFAULT: ALL)
+                                                           HBM
+                                                           L2
+                                                           vL1D
+                                                           LDS
+  --device                                              GPU device ID. (DEFAULT: ALL)
+  --kernel-names                                        Include kernel names in roofline plot.
+```
+
+The following sample command profiles the *vcopy* workload.
+
+**vcopy profiling:**
+```shell-session
+$ omniperf profile --name vcopy -- ./vcopy 1048576 256
+Resolving rocprof
+ROC Profiler:  /usr/bin/rocprof
+
+
+-------------
+Profile only
+-------------
+
+omniperf ver:  1.0.8-PR1
+Path:  /home/colramos/GitHub/omniperf-pub/workloads
+Target:  mi200
+Command:  /home/colramos/vcopy 1048576 256
+Kernel Selection:  None
+Dispatch Selection:  None
+IP Blocks: All
+Log:  /home/colramos/GitHub/omniperf-pub/workloads/vcopy/mi200/log.txt
+
+/home/colramos/GitHub/omniperf-pub/workloads/vcopy/mi200/perfmon/SQ_INST_LEVEL_SMEM.txt
+RPL: on '230411_165021' from '/opt/rocm-5.2.1' in '/home/colramos/GitHub/omniperf-pub'
+RPL: profiling '""/home/colramos/vcopy 1048576 256""'
+RPL: input file '/home/colramos/GitHub/omniperf-pub/workloads/vcopy/mi200/perfmon/SQ_INST_LEVEL_SMEM.txt'
+RPL: output dir '/tmp/rpl_data_230411_165021_26406'
+RPL: result dir '/tmp/rpl_data_230411_165021_26406/input0_results_230411_165021'
+Finished allocating vectors on the CPU
+ROCProfiler: input from "/tmp/rpl_data_230411_165021_26406/input0.xml"
+  gpu_index = 
+  kernel = 
+  range = 
+  3 metrics
+    SQ_INSTS_SMEM, SQ_INST_LEVEL_SMEM, SQ_ACCUM_PREV_HIRES
+Finished allocating vectors on the GPU
+Finished copying vectors to the GPU
+sw thinks it moved 1.000000 KB per wave 
+Total threads: 1048576, Grid Size: 4096 block Size:256, Wavefronts:16384:
+Launching the  kernel on the GPU
+Finished executing kernel
+Finished copying the output vector from the GPU to the CPU
+Releasing GPU memory
+Releasing CPU memory
+ 
+... ...
+ROCPRofiler: 1 contexts collected, output directory /tmp/rpl_data_220527_130317_1787038/input_results_220527_130317
+File 'workloads/vcopy/mi200/timestamps.csv' is generating
+Total detected GPU devices: 2
+GPU Device 0: Profiling...
+ 99% [||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ]
+HBM BW, GPU ID: 0, workgroupSize:256, workgroups:2097152, experiments:100, traffic:8589934592 bytes, duration:6.2 ms, mean:1382.7 GB/sec, stdev=2.4 GB/sec
+ 99% [||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ]
+L2 BW, GPU ID: 0, workgroupSize:256, workgroups:8192, experiments:100, traffic:687194767360 bytes, duration:157.9 ms, mean:4358.7 GB/sec, stdev=4.7 GB/sec
+ 99% [||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ]
+L1 BW, GPU ID: 0, workgroupSize:256, workgroups:16384, experiments:100, traffic:26843545600 bytes, duration:3.3 ms, mean:8247.1 GB/sec, stdev=5.1 GB/sec
+ 99% [||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ]
+LDS BW, GPU ID: 0, workgroupSize:256, workgroups:16384, experiments:100, traffic:33554432000 bytes, duration:2.4 ms, mean:14246.3 GB/sec, stdev=29.5 GB/sec
+ 99% [||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ]
+Peak FLOPs (FP32), GPU ID: 0, workgroupSize:256, workgroups:16384, experiments:100, FLOP:274877906944, duration:14.507 ms, mean:18949.6 GFLOPS, stdev=4.5 GFLOPS
+ 99% [||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ]
+Peak FLOPs (FP64), GPU ID: 0, workgroupSize:256, workgroups:16384, experiments:100, FLOP:137438953472, duration:7.5 ms, mean:18308.197266.1 GFLOPS, stdev=3.6 GFLOPS
+ 99% [||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ]
+Peak MFMA FLOPs (BF16), GPU ID: 0, workgroupSize:256, workgroups:16384, experiments:100, FLOP:2147483648000, duration:14.0 ms, mean:153574.8 GFLOPS, stdev=79.9 GFLOPS
+ 99% [||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ]
+Peak MFMA FLOPs (F16), GPU ID: 0, workgroupSize:256, workgroups:16384, experiments:100, FLOP:2147483648000, duration:14.5 ms, mean:147680.1 GFLOPS, stdev=34.7 GFLOPS
+ 99% [||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ]
+Peak MFMA FLOPs (F32), GPU ID: 0, workgroupSize:256, workgroups:16384, experiments:100, FLOP:536870912000, duration:14.5 ms, mean:37142.1 GFLOPS, stdev=8.4 GFLOPS
+ 99% [||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ]
+Peak MFMA FLOPs (F64), GPU ID: 0, workgroupSize:256, workgroups:16384, experiments:100, FLOP:268435456000, duration:7.3 ms, mean:36919.5 GFLOPS, stdev=14.1 GFLOPS
+ 99% [||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ]
+Peak MFMA IOPs (I8), GPU ID: 0, workgroupSize:256, workgroups:16384, experiments:100, IOP:2147483648000, duration:14.4 ms, mean:149570.6 GOPS, stdev=41.7 GOPS
+GPU Device 1: Profiling...
+ 99% [||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ]
+HBM BW, GPU ID: 1, workgroupSize:256, workgroups:2097152, experiments:100, traffic:8589934592 bytes, duration:6.2 ms, mean:1382.7 GB/sec, stdev=2.9 GB/sec
+ 99% [||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ]
+L2 BW, GPU ID: 1, workgroupSize:256, workgroups:8192, experiments:100, traffic:687194767360 bytes, duration:157.6 ms, mean:4371.0 GB/sec, stdev=4.1 GB/sec
+ 99% [||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ]
+L1 BW, GPU ID: 1, workgroupSize:256, workgroups:16384, experiments:100, traffic:26843545600 bytes, duration:3.2 ms, mean:8297.4 GB/sec, stdev=11.6 GB/sec
+ 99% [||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ]
+LDS BW, GPU ID: 1, workgroupSize:256, workgroups:16384, experiments:100, traffic:33554432000 bytes, duration:1.8 ms, mean:18839.2 GB/sec, stdev=44.5 GB/sec
+ 99% [||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ]
+Peak FLOPs (FP32), GPU ID: 1, workgroupSize:256, workgroups:16384, experiments:100, FLOP:274877906944, duration:14.441 ms, mean:19037.6 GFLOPS, stdev=2.7 GFLOPS
+ 99% [||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ]
+Peak FLOPs (FP64), GPU ID: 1, workgroupSize:256, workgroups:16384, experiments:100, FLOP:137438953472, duration:7.5 ms, mean:18402.255859.1 GFLOPS, stdev=20.1 GFLOPS
+ 99% [||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ]
+Peak MFMA FLOPs (BF16), GPU ID: 1, workgroupSize:256, workgroups:16384, experiments:100, FLOP:2147483648000, duration:13.9 ms, mean:154240.3 GFLOPS, stdev=119.3 GFLOPS
+ 99% [||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ]
+Peak MFMA FLOPs (F16), GPU ID: 1, workgroupSize:256, workgroups:16384, experiments:100, FLOP:2147483648000, duration:14.5 ms, mean:148450.1 GFLOPS, stdev=112.6 GFLOPS
+ 99% [||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ]
+Peak MFMA FLOPs (F32), GPU ID: 1, workgroupSize:256, workgroups:16384, experiments:100, FLOP:536870912000, duration:14.4 ms, mean:37335.2 GFLOPS, stdev=43.1 GFLOPS
+ 99% [||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ]
+Peak MFMA FLOPs (F64), GPU ID: 1, workgroupSize:256, workgroups:16384, experiments:100, FLOP:268435456000, duration:7.2 ms, mean:37105.3 GFLOPS, stdev=39.5 GFLOPS
+ 99% [||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ]
+Peak MFMA IOPs (I8), GPU ID: 1, workgroupSize:256, workgroups:16384, experiments:100, IOP:2147483648000, duration:14.3 ms, mean:150317.8 GOPS, stdev=203.5 GOPS
+```
+You'll notice two stages in *default* Omniperf profiling. The first stage collects all the counters needed for Omniperf analysis (omitting any filters you've provided). The second stage collects data for the roofline analysis (this stage can be disabled using `--no-roof`)
+
+At the end of the profiling, all resulting csv files should be located in a SOC specific target directory, e.g.:
+  - "mi200" for the AMD Instinct (tm) MI-200 family of accelerators
+  - "mi100" for the AMD Instinct (tm) MI-100 family of accelerators
+etc.  The SOC names are generated as a part of Omniperf, and do not necessarily distinguish between different accelerators in the same family (e.g., an AMD Instinct (tm) MI-210 vs an MI-250)
+
+> Note: Additionally, you'll notice a few extra files. An SoC parameters file, *sysinfo.csv*, is created to reflect the target device settings. All profiling output is stored in *log.txt*. Roofline specific benchmark results are stored in *roofline.csv*.
+
+```shell
+$ ls workloads/vcopy/mi200/
+total 112
+drwxrwxr-x 3 colramos colramos  4096 Apr 11 16:42 .
+drwxrwxr-x 3 colramos colramos  4096 Apr 11 16:42 ..
+-rw-rw-r-- 1 colramos colramos 40750 Apr 11 16:44 log.txt
+drwxrwxr-x 2 colramos colramos  4096 Apr 11 16:42 perfmon
+-rw-rw-r-- 1 colramos colramos 25877 Apr 11 16:42 pmc_perf.csv
+-rw-rw-r-- 1 colramos colramos  1716 Apr 11 16:44 roofline.csv
+-rw-rw-r-- 1 colramos colramos   429 Apr 11 16:42 SQ_IFETCH_LEVEL.csv
+-rw-rw-r-- 1 colramos colramos   366 Apr 11 16:42 SQ_INST_LEVEL_LDS.csv
+-rw-rw-r-- 1 colramos colramos   391 Apr 11 16:42 SQ_INST_LEVEL_SMEM.csv
+-rw-rw-r-- 1 colramos colramos   384 Apr 11 16:42 SQ_INST_LEVEL_VMEM.csv
+-rw-rw-r-- 1 colramos colramos   509 Apr 11 16:42 SQ_LEVEL_WAVES.csv
+-rw-rw-r-- 1 colramos colramos   498 Apr 11 16:42 sysinfo.csv
+-rw-rw-r-- 1 colramos colramos   309 Apr 11 16:42 timestamps.csv
+```
+
+### Filtering
+To reduce profiling time and the counters collected one may use profiling filters. Profiling filters and their functionality depend on the underlying profiler being used. While Omniperf is profiler agnostic, we've provided a detailed description of profiling filters available when using Omniperf with [rocProfiler](https://rocm.docs.amd.com/projects/rocprofiler/en/latest/rocprof.html) below.
+
+
+
+Filtering Options:
+
+- The `-k` \<kernel> flag allows for kernel filtering. Useage is equivalent with the current rocprof utility ([see details below](#kernel-filtering)).
+
+- The `-d` \<dispatch> flag allows for dispatch ID filtering. Useage is equivalent with the current rocprof utility ([see details below](#dispatch-filtering)).
+
+- The `-b` \<ipblocks> allows system profiling on one or more selected IP blocks to speed up the profiling process. One can gradually incorporate more IP blocks, without overwriting performance data acquired on other IP blocks.
+
+```{note}
+Be cautious while combining different profiling filters in the same call. Conflicting filters may result in error.
+
+i.e. filtering dispatch X, but dispatch X does not match your kernel name filter
+```
+
+#### IP Block Filtering
+One can profile a selected IP Block to speed up the profiling process. All profiling results are accumulated in the same target directory, without overwriting those for other IP blocks, hence enabling the incremental profiling and analysis.
+
+The following example only gathers hardware counters for SQ and TCC, skipping all other IP Blocks:
+```shell
+$ omniperf profile --name vcopy -b SQ TCC -- ./sample/vcopy 1048576 256
+Resolving rocprof
+ROC Profiler:  /usr/bin/rocprof
+
+
+-------------
+Profile only
+-------------
+
+omniperf ver:  1.0.8-PR1
+Path:  /home/colramos/GitHub/omniperf-pub/workloads
+Target:  mi200
+Command:  /home/colramos/vcopy 1048576 256
+Kernel Selection:  None
+Dispatch Selection:  None
+IP Blocks:  ['SQ', 'TCC']
+fname: pmc_sq_perf2: Added
+fname: pmc_td_perf: Skipped
+fname: pmc_tcc2_perf: Skipped
+fname: pmc_tcp_perf: Skipped
+fname: pmc_spi_perf: Skipped
+fname: pmc_sq_perf4: Added
+fname: pmc_sqc_perf1: Skipped
+fname: pmc_tcc_perf: Added
+fname: pmc_cpf_perf: Skipped
+fname: pmc_sq_perf8: Added
+fname: pmc_cpc_perf: Skipped
+fname: pmc_sq_perf1: Added
+fname: pmc_ta_perf: Skipped
+fname: pmc_sq_perf3: Added
+fname: pmc_sq_perf6: Added
+Log:  /home/colramos/GitHub/omniperf-pub/workloads/vcopy/mi200/log.txt
+...
+```
+
+#### Kernel Filtering
+Kernel filtering is based on the name of the kernel(s) you'd like to isolate. Use a kernel name substring list to isolate desired kernels.
+
+The following example demonstrates profiling isolating the kernel matching substring "vecCopy":
+```shell
+$ omniperf profile --name vcopy -k vecCopy -- ./vcopy 1048576 256
+Resolving rocprof
+ROC Profiler:  /usr/bin/rocprof
+
+
+-------------
+Profile only
+-------------
+
+omniperf ver:  1.0.8-PR1
+Path:  /home/colramos/GitHub/omniperf-pub/workloads
+Target:  mi200
+Command:  /home/colramos/vcopy 1048576 256
+Kernel Selection:  ['vecCopy']
+Dispatch Selection:  None
+IP Blocks: All
+Log:  /home/colramos/GitHub/omniperf-pub/workloads/vcopy/mi200/log.txt
+
+/home/colramos/GitHub/omniperf-pub/workloads/vcopy/mi200/perfmon/SQ_INST_LEVEL_SMEM.txt
+RPL: on '230411_170300' from '/opt/rocm-5.2.1' in '/home/colramos/GitHub/omniperf-pub'
+RPL: profiling '""/home/colramos/vcopy 1048576 256""'
+RPL: input file '/home/colramos/GitHub/omniperf-pub/workloads/vcopy/mi200/perfmon/SQ_INST_LEVEL_SMEM.txt'
+RPL: output dir '/tmp/rpl_data_230411_170300_29696'
+RPL: result dir '/tmp/rpl_data_230411_170300_29696/input0_results_230411_170300'
+Finished allocating vectors on the CPU
+ROCProfiler: input from "/tmp/rpl_data_230411_170300_29696/input0.xml"
+  gpu_index = 
+  kernel = vecCopy
+ 
+... ...
+```
+
+#### Dispatch Filtering
+Dispatch filtering is based on the *global* dispatch index of kernels in a run. 
+
+The following example profiles only the 0th dispatched kernel in execution of the application:
+```shell-session
+$ omniperf profile --name vcopy -d 0 -- ./vcopy 1048576 256
+Resolving rocprof
+ROC Profiler:  /usr/bin/rocprof
+
+
+-------------
+Profile only
+-------------
+
+omniperf ver:  1.0.8-PR1
+Path:  /home/colramos/GitHub/omniperf-pub/workloads
+Target:  mi200
+Command:  /home/colramos/vcopy 1048576 256
+Kernel Selection:  None
+Dispatch Selection:  ['0']
+IP Blocks: All
+Log:  /home/colramos/GitHub/omniperf-pub/workloads/vcopy/mi200/log.txt
+
+/home/colramos/GitHub/omniperf-pub/workloads/vcopy/mi200/perfmon/SQ_INST_LEVEL_SMEM.txt
+RPL: on '230411_170356' from '/opt/rocm-5.2.1' in '/home/colramos/GitHub/omniperf-pub'
+RPL: profiling '""/home/colramos/vcopy 1048576 256""'
+RPL: input file '/home/colramos/GitHub/omniperf-pub/workloads/vcopy/mi200/perfmon/SQ_INST_LEVEL_SMEM.txt'
+RPL: output dir '/tmp/rpl_data_230411_170356_30314'
+RPL: result dir '/tmp/rpl_data_230411_170356_30314/input0_results_230411_170356'
+Finished allocating vectors on the CPU
+ROCProfiler: input from "/tmp/rpl_data_230411_170356_30314/input0.xml"
+  gpu_index = 
+  kernel = 
+  range = 0
+...
+```
+
+
+
+### Standalone Roofline
+If you're only interested in generating roofline analysis data try using `--roof-only`. This will only collect counters relevent to roofline, as well as generate a standalone .pdf output of your roofline plot. 
+
+Standalone Roofline Options:
+
+- The `--sort` \<desired_sort> allows you to specify whether you'd like to overlay top kernel or top dispatch data in your roofline plot.
+
+- The `-m` \<cache_level> allows you to specify specific level(s) of cache you'd like to include in your roofline plot.
+
+- The `--device` \<gpu_id> allows you to specify a device id to collect performace data from when running our roofline benchmark on your system.
+
+- If you'd like to distinguish different kernels in your .pdf roofline plot use `--kernel-names`. This will give each kernel a unique marker identifiable from the plot's key.
+
+
+#### Roofline Only
+The following example demonstrates profiling roofline data only:
+```shell-session
+$ omniperf profile --name vcopy --roof-only -- ./vcopy 1048576 256
+Resolving rocprof
+ROC Profiler:  /usr/bin/rocprof
+
+
+--------
+Roofline only
+--------
+
+Checking for roofline.csv in  /home/colramos/GitHub/omniperf-pub/workloads/vcopy/mi200
+No roofline data found. Generating...
+Empirical Roofline Calculation
+Copyright © 2022  Advanced Micro Devices, Inc. All rights reserved.
+Total detected GPU devices: 4
+GPU Device 0: Profiling...
+ 99% [||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ]
+ ... ...
+Checking for roofline.csv in  /home/colramos/GitHub/omniperf-pub/workloads/mix/mi200
+Checking for sysinfo.csv in  /home/colramos/GitHub/omniperf-pub/workloads/mix/mi200
+Checking for pmc_perf.csv in  /home/colramos/GitHub/omniperf-pub/workloads/mix/mi200
+Empirical Roofline PDFs saved!
+```
+An inspection of our workload output folder shows .pdf plots were generated successfully
+```shell-session
+$ ls workloads/vcopy/mi200/
+total 176
+drwxrwxr-x 3 colramos colramos  4096 Apr 11 17:18 .
+drwxrwxr-x 3 colramos colramos  4096 Apr 11 17:15 ..
+-rw-rw-r-- 1 colramos colramos 13271 Apr 11 17:18 empirRoof_gpu-ALL_fp32.pdf
+-rw-rw-r-- 1 colramos colramos 13175 Apr 11 17:18 empirRoof_gpu-ALL_int8_fp16.pdf
+-rw-rw-r-- 1 colramos colramos 26560 Apr 11 17:16 log.txt
+drwxrwxr-x 2 colramos colramos  4096 Apr 11 17:16 perfmon
+-rw-rw-r-- 1 colramos colramos 54031 Apr 11 17:16 pmc_perf.csv
+-rw-rw-r-- 1 colramos colramos  1714 Apr 11 17:16 roofline.csv
+-rw-rw-r-- 1 colramos colramos   457 Apr 11 17:16 sysinfo.csv
+-rw-rw-r-- 1 colramos colramos 37521 Apr 11 17:16 timestamps.csv
+```
+A sample *empirRoof_gpu-ALL_fp32.pdf* looks something like this:
+
+![Sample Standalone Roof Plot](images/sample-roof-plot.png)