Sync staging with mainline (#524)
* External CI: rename pipeline to rocprofiler-compute (#463) Signed-off-by: Daniel Su <danielsu@amd.com> * Update webui branding (#459) * Update name and icon for browser tab to rocprofiler-compute. Signed-off-by: xuchen-amd <xuchen@amd.com> * Update name and icon for browser tab to rocprofiler-compute. Signed-off-by: xuchen-amd <xuchen@amd.com> --------- Signed-off-by: xuchen-amd <xuchen@amd.com> * Update branding in documentation (#442) * find/replace Omniperf to ROCm Compute Profiler Signed-off-by: Peter Park <peter.park@amd.com> * update name in Sphinx conf Signed-off-by: Peter Park <peter.park@amd.com> * mv what-is-omniperf.rst -> what-is-rocprof-compute.rst Signed-off-by: Peter Park <peter.park@amd.com> * update Tutorials section Signed-off-by: Peter Park <peter.park@amd.com> * add Omniperf as keyword to Conceptual section for internal search Signed-off-by: Peter Park <peter.park@amd.com> * update Reference section Signed-off-by: Peter Park <peter.park@amd.com> * black fmt conf.py Signed-off-by: Peter Park <peter.park@amd.com> * update profile mode and basic usage subsections Signed-off-by: Peter Park <peter.park@amd.com> * update how to use analyze mode subsection Signed-off-by: Peter Park <peter.park@amd.com> * update install section Signed-off-by: Peter Park <peter.park@amd.com> * fix sphinx warnings Signed-off-by: Peter Park <peter.park@amd.com> * fix cmd line examples in profile/mode.rst Signed-off-by: Peter Park <peter.park@amd.com> * update install decision tree image Signed-off-by: Peter Park <peter.park@amd.com> * fix TOC and index Signed-off-by: Peter Park <peter.park@amd.com> fix weird wording * fix cli text: deriving rocprofiler-compute metrics... Signed-off-by: Peter Park <peter.park@amd.com> * update standalone-gui.rst Signed-off-by: Peter Park <peter.park@amd.com> * restore removed doc updates from #428 Signed-off-by: Peter Park <peter.park@amd.com> * update ref to Omniperf in index.rst Signed-off-by: Peter Park <peter.park@amd.com> * fix grafana connection name to match image Signed-off-by: Peter Park <peter.park@amd.com> * update cmds in tutorials Signed-off-by: Peter Park <peter.park@amd.com> --------- Signed-off-by: Peter Park <peter.park@amd.com> * MI300 roofline enablement in rocprofiler-compute (#470) * MI300 roofline enablement in rocprofiler-compute requirements.txt - running some modules complained about numpy version too new, adding extra requirement that numpy be 1.x pmc_roof_perf.txt - adding TCC_BUBBLE_sum counter to profile soc_gfx940.py soc_gfx941.py soc_gfx942.py - remove console logs reading that roofline is temporarily disabled, uncommenting blocks that check for roofline csv and run roofline post-processing roofline_calc.py - add mi300 to supported soc - add new calculation for hbm_data for MI300 using tcc_bubble_sum, checks if counter > 0 to use - add to a few comments roofline-ubuntu-20_04-mi300-rocm6 - binary for the ubuntu systems to enable mi300 roofline calculations from rocm-amdgpu-bench Note- other distros will get roofline bins to enable mi300, but need to be further tested before putting into branch. Signed-off-by: Carrie Fallows <carrie.fallows@amd.com> * Reformatting roofline_calc.py Signed-off-by: Carrie Fallows <carrie.fallows@amd.com> --------- Signed-off-by: Carrie Fallows <carrie.fallows@amd.com> * Update Python format checker (#471) * Add pre commit hook for Python formatting Signed-off-by: coleramos425 <colramos@amd.com> * Update formatting workflow to run on latest Python and add isort formatter Signed-off-by: coleramos425 <colramos@amd.com> * Fix caught yaml formatting issues * Update pyproject file * Add pre-commit hook instruction to CONTRIBUTING guide * Remove target-version from black pyproject.toml * Fixed formatting errors found with black and isort Signed-off-by: David Galiffi <David.Galiffi@amd.com> * Run hook: Whitespaces, fix end of file spaces --------- Signed-off-by: coleramos425 <colramos@amd.com> Signed-off-by: David Galiffi <David.Galiffi@amd.com> Co-authored-by: David Galiffi <David.Galiffi@amd.com> * Bump cryptography from 43.0.0 to 43.0.1 in /docs/sphinx (#473) Bumps [cryptography](https://github.com/pyca/cryptography) from 43.0.0 to 43.0.1. - [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst) - [Commits](https://github.com/pyca/cryptography/compare/43.0.0...43.0.1) --- updated-dependencies: - dependency-name: cryptography dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Fix file permission on MI300 roofline binary (#477) Signed-off-by: David Galiffi <David.Galiffi@amd.com> * Removing numpy requirements of <2 (#478) Checks are failing if version too high and no need for lower version Signed-off-by: Carrie Fallows <Carrie.Fallows@amd.com> * Fix crash when loading web UI roofline for gfx942 (#479) * Fix crash when loading web UI roofline for gfx942 * Fix formatting Signed-off-by: benrichard-amd <ben.richard@amd.com> * Make same changs for gfx940, gfx942. Signed-off-by: benrichard-amd <ben.richard@amd.com> * Fix formatting in soc_gfx940 and soc_gfx941. Signed-off-by: benrichard-amd <ben.richard@amd.com> --------- Signed-off-by: benrichard-amd <ben.richard@amd.com> * Rebranding name change patch (#469) * Patch in missed name change for rebranding. Signed-off-by: xuchen-amd <xuchen@amd.com> * Patch in missed name change for rebranding. Signed-off-by: xuchen-amd <xuchen@amd.com> --------- Signed-off-by: xuchen-amd <xuchen@amd.com> * Move dependabot.yml to .github/ and bump rocm-docs-core (#481) * Move dependabot.yml to .github/ * Bump rocm-docs-core to 1.8.5 * Bump rocm-docs-core to 1.9.0 * Fix packaging for upgrading (#486) Specify that "rocprofiler-compute" replaces / obsoletes the "omniperf" package. * Renamed extension path from omniperf to rocprofiler_compute (#487) Signed-off-by: Tim Gu <Tim.Gu@amd.com> * MI300 rhel and sles roofline binaries (#480) * Roofline bins for MI300 on rhel and sles distributions Built from rocm-amdgpu-bench, tested on respective distro systems with MI300 hardware. Signed-off-by: Carrie Fallows <Carrie.Fallows@amd.com> * Minor modifications removing hardcoded variables in roofline files. Signed-off-by: Carrie Fallows <Carrie.Fallows@amd.com> --------- Signed-off-by: Carrie Fallows <Carrie.Fallows@amd.com> * Modify test_profile_general.py ctest to include MI300 enablement (#498) Signed-off-by: Carrie Fallows <Carrie.Fallows@amd.com> * part 1 to support rocprofv3 (#492) * rocprofv3 support initial commit -Can run rocprofv3 but ultimately fails. rocprofv3 says the counter capacity is exceeded and the output CSV file format is different from v1/v2. * Add rocprofv3 detection so v2 can still be used It's hacky but it'll do for now. * Add code path to convert rocprofv3 JSON output into CSV * Grab correct value for Queue ID * Use _sum suffix to sum TCC counters Previously we were specifying each channel for TCC counters. rocprofv3 does not support specifing each TCC channel, and instead will auto sum given the TCC counter name. The counter name with the _sum suffix is also supported and is also supported in v1 and v2. So we will use the TCC counter name with the _sum suffix. * Fix incorrect counter outputs when using rocprofv3 In the JSON output some counters appear multime times and must be summed to get the correct value. These summed values match the rocprofv3 output in CSV mode and also match the rocprofv2 output. * Remove duplicate Correlation_ID and Wave_Size in output * Handle json output that does not contain any dispatches Omniperf was assuming each JSON output from rocprofv3 would always contain dispatches. This is not the case. For example, in a multi-process workload where one of the processes does not dispatch any kernels. A JSON file will still be output for this process but it will not contain any dispatches. * Code cleanup * Update search path for rocprofv3 results Rocprofv3 was updated to include the hostname in the path where it outputs results. * Handle accumulate counters In v1/v2 rocprof uses the SQ_ACCUM_PREV_HIRES counter for the accumualte counters. v3 does not have this. So we need to define our own counters in counter_defs.yaml. For this we use the counter name + _ACCUM, for example SQ_INSTR_LEVEL_SMEM_ACCUM. To use rocprofv3 you will need to update counter_defs.yaml to include these new counter definitions. * Use correct GPU ID When converting JSON -> CSV we were assigning node_id to GPU_ID. Since the JSON contains non-GPU devices, the node_id for GPUs might not start at 0 as expected. This commit maps the agent ID to the appropriate GPU ID. * Parse scratch memory per work item from JSON * Support rocprofv3 CSV parsing JSON decoding is very slow for large files. Include support for parsing rocprofv3 CSV output and make that the default. CSV/JSON can be toggled via the ROCPROF_OUTPUT_FORMAT environment variable e.g. ROCPROF_OUTPUT_FORMAT=csv or ROCPROF_OUTPUT_FORMAT=json * black format after merge * format isort * change return of rocprof_cmd to try to resolve test's error * hack to pick last part of rocminfo's name * debug log of hacks * Modify test_profile_general.py ctest to include MI300 enablement. Currently failing because of explicitly excluded roofline files for the soc and autofailed asserts for roof-only tests- originally in place because roofline was not enabled on mi300 yet. Signed-off-by: Carrie Fallows <Carrie.Fallows@amd.com> * black and isort formated * corrected line of copyright --------- Signed-off-by: Carrie Fallows <Carrie.Fallows@amd.com> Co-authored-by: benrichard-amd <ben.richard@amd.com> Co-authored-by: YANG WANG <ywang@ywang-ubuntu.amd.com> Co-authored-by: Carrie Fallows <Carrie.Fallows@amd.com> * fix for crash of timestamp of part 1 for rocprofv3 (#499) * fix the error caused by ignoring the lack of counter csv file from rocprofv3 for timestamp * isort and black formated * quick fix for gfx906 roofline (#505) * Multi node support (#503) * [CTest] Pipeline failures for MI300 (#483) * Propagate new chip_id logic to testing workflow Signed-off-by: coleramos425 <colramos@amd.com> * Add a debug line to tests Signed-off-by: coleramos425 <colramos@amd.com> * Trying to set rocprofv2 generally in CTest module Signed-off-by: coleramos425 <colramos@amd.com> * Remove temp debugging lines from CI Signed-off-by: coleramos425 <colramos@amd.com> * Add roofline entry for MI300 expected files in CI tests Signed-off-by: coleramos425 <colramos@amd.com> * Make num_devices modifier global in scope Signed-off-by: coleramos425 <colramos@amd.com> * Change kernel name in PyTest to confirm rocprofv2 bug Related to https://ontrack-internal.amd.com/browse/SWDEV-503453 Signed-off-by: coleramos425 <colramos@amd.com> --------- Signed-off-by: coleramos425 <colramos@amd.com> * Spatial-multiplexing: part 1 profiling stage (#465) * rocprofv3 support initial commit -Can run rocprofv3 but ultimately fails. rocprofv3 says the counter capacity is exceeded and the output CSV file format is different from v1/v2. * Add rocprofv3 detection so v2 can still be used It's hacky but it'll do for now. * Add code path to convert rocprofv3 JSON output into CSV * Grab correct value for Queue ID * Use _sum suffix to sum TCC counters Previously we were specifying each channel for TCC counters. rocprofv3 does not support specifing each TCC channel, and instead will auto sum given the TCC counter name. The counter name with the _sum suffix is also supported and is also supported in v1 and v2. So we will use the TCC counter name with the _sum suffix. * Fix incorrect counter outputs when using rocprofv3 In the JSON output some counters appear multime times and must be summed to get the correct value. These summed values match the rocprofv3 output in CSV mode and also match the rocprofv2 output. * Remove duplicate Correlation_ID and Wave_Size in output * Handle json output that does not contain any dispatches Omniperf was assuming each JSON output from rocprofv3 would always contain dispatches. This is not the case. For example, in a multi-process workload where one of the processes does not dispatch any kernels. A JSON file will still be output for this process but it will not contain any dispatches. * Code cleanup * Update search path for rocprofv3 results Rocprofv3 was updated to include the hostname in the path where it outputs results. * Handle accumulate counters In v1/v2 rocprof uses the SQ_ACCUM_PREV_HIRES counter for the accumualte counters. v3 does not have this. So we need to define our own counters in counter_defs.yaml. For this we use the counter name + _ACCUM, for example SQ_INSTR_LEVEL_SMEM_ACCUM. To use rocprofv3 you will need to update counter_defs.yaml to include these new counter definitions. * debug code * add logic code for multiplexing * minor fix * more fixes * rocprofv3 support initial commit -Can run rocprofv3 but ultimately fails. rocprofv3 says the counter capacity is exceeded and the output CSV file format is different from v1/v2. * Add rocprofv3 detection so v2 can still be used It's hacky but it'll do for now. * Add code path to convert rocprofv3 JSON output into CSV * Grab correct value for Queue ID * Use _sum suffix to sum TCC counters Previously we were specifying each channel for TCC counters. rocprofv3 does not support specifing each TCC channel, and instead will auto sum given the TCC counter name. The counter name with the _sum suffix is also supported and is also supported in v1 and v2. So we will use the TCC counter name with the _sum suffix. * Fix incorrect counter outputs when using rocprofv3 In the JSON output some counters appear multime times and must be summed to get the correct value. These summed values match the rocprofv3 output in CSV mode and also match the rocprofv2 output. * Remove duplicate Correlation_ID and Wave_Size in output * Handle json output that does not contain any dispatches Omniperf was assuming each JSON output from rocprofv3 would always contain dispatches. This is not the case. For example, in a multi-process workload where one of the processes does not dispatch any kernels. A JSON file will still be output for this process but it will not contain any dispatches. * Code cleanup * Update search path for rocprofv3 results Rocprofv3 was updated to include the hostname in the path where it outputs results. * Handle accumulate counters In v1/v2 rocprof uses the SQ_ACCUM_PREV_HIRES counter for the accumualte counters. v3 does not have this. So we need to define our own counters in counter_defs.yaml. For this we use the counter name + _ACCUM, for example SQ_INSTR_LEVEL_SMEM_ACCUM. To use rocprofv3 you will need to update counter_defs.yaml to include these new counter definitions. * count accu files as well * Use correct GPU ID When converting JSON -> CSV we were assigning node_id to GPU_ID. Since the JSON contains non-GPU devices, the node_id for GPUs might not start at 0 as expected. This commit maps the agent ID to the appropriate GPU ID. * fix error with csv file parse from json and merge during post-processing * implemented parsing of csv files from v3 output for optimization * Parse scratch memory per work item from JSON * Support rocprofv3 CSV parsing JSON decoding is very slow for large files. Include support for parsing rocprofv3 CSV output and make that the default. CSV/JSON can be toggled via the ROCPROF_OUTPUT_FORMAT environment variable e.g. ROCPROF_OUTPUT_FORMAT=csv or ROCPROF_OUTPUT_FORMAT=json * black format after merge * format isort * change return of rocprof_cmd to try to resolve test's error * hack to pick last part of rocminfo's name * debug log of hacks * Modify test_profile_general.py ctest to include MI300 enablement. Currently failing because of explicitly excluded roofline files for the soc and autofailed asserts for roof-only tests- originally in place because roofline was not enabled on mi300 yet. Signed-off-by: Carrie Fallows <Carrie.Fallows@amd.com> * black and isort formated * formated by isort and black * change default rocprof's output to csv * repaired crash caused by missing csv counter file when running for timestamp * change name to spatial-multiplexing from multiplexing * make necessary modification for review * set the value of spatial_multiplexing in argument defautly to None * repair the part that blocks regular pmc files' generation --------- Signed-off-by: Carrie Fallows <Carrie.Fallows@amd.com> Co-authored-by: benrichard-amd <ben.richard@amd.com> Co-authored-by: fei.zheng <fei.zheng@amd.com> Co-authored-by: YANG WANG <ywang@ywang-ubuntu.amd.com> Co-authored-by: Carrie Fallows <Carrie.Fallows@amd.com> * Simple fix for gpu model value. (#508) Signed-off-by: xuchen-amd <xuchen@amd.com> * Add FP64 to plot adhering to pdf name (#507) * Replacing FP32-only plot with an FP32&FP64 combo plot. Results will likely be negligible but the plot name indicates both should be graphed. Signed-off-by: Carrie Fallows <Carrie.Fallows@amd.com> * Remove duplicate AI plot to clean up fp32 fp64 graph Signed-off-by: Carrie Fallows <Carrie.Fallows@amd.com> --------- Signed-off-by: Carrie Fallows <Carrie.Fallows@amd.com> * Add gpu series for roofline (#510) * Add gpu_series for roofline. * Use gpu_series in path names for roofline. * Fix TCC on MI200 when introduce rocprofv3 (#509) * quick fix for v2 * one more fix * revert a bit --------- Co-authored-by: ywang103-amd <ywang103@amd.com> * Bump rocm-docs-core from 1.9.0 to 1.12.0 in /docs/sphinx (#511) Bumps [rocm-docs-core](https://github.com/ROCm/rocm-docs-core) from 1.9.0 to 1.12.0. - [Release notes](https://github.com/ROCm/rocm-docs-core/releases) - [Changelog](https://github.com/ROCm/rocm-docs-core/blob/develop/CHANGELOG.md) - [Commits](https://github.com/ROCm/rocm-docs-core/compare/v1.9.0...v1.12.0) --- updated-dependencies: - dependency-name: rocm-docs-core dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Update sample roofline plot img (#516) * Modify path to use gpu_model instead of gpu_series to match other workload directory path creation/search points. Affects manual testing, does not seem to affect ctests. (#513) Signed-off-by: Carrie Fallows <Carrie.Fallows@amd.com> * Improve formatting when displaying rocprof command. (#476) * Improve formatting when displaying rocprof command. Signed-off-by: xuchen-amd <xuchen@amd.com> * Fix python formatting. Signed-off-by: xuchen-amd <xuchen@amd.com> * Strip unwanted characters (rocprofv1 specific) from rocprof commands. Signed-off-by: xuchen-amd <xuchen@amd.com> * Strip unwanted characters (rocprofv1 specific) from rocprof commands. Signed-off-by: xuchen-amd <xuchen@amd.com> * Save the unmodified arguments for rocprof for debug message display. Signed-off-by: xuchen-amd <xuchen@amd.com> --------- Signed-off-by: xuchen-amd <xuchen@amd.com> * quick fix for mpi_support (#518) * Pass accumulate counters to rocprofv3 using -E option (#522) rocprofv3 has a new -E option where extra counters can be passed (see accum_counters.yaml) instead of defining them in counter_defs.yaml. * Unify all file handling with pathlib (#512) * Replace occurences of os.path functions with equivalent functions from pathlib library * Remove unwanted imports of os.path and os * Add coding guidelines for using pathlib instead of os.path * Auto sync staging and mainline on a weekly cadence (#517) Signed-off-by: coleramos425 <colramos@amd.com> --------- Signed-off-by: Daniel Su <danielsu@amd.com> Signed-off-by: xuchen-amd <xuchen@amd.com> Signed-off-by: Peter Park <peter.park@amd.com> Signed-off-by: Carrie Fallows <carrie.fallows@amd.com> Signed-off-by: coleramos425 <colramos@amd.com> Signed-off-by: David Galiffi <David.Galiffi@amd.com> Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Carrie Fallows <Carrie.Fallows@amd.com> Signed-off-by: benrichard-amd <ben.richard@amd.com> Signed-off-by: Tim Gu <Tim.Gu@amd.com> Co-authored-by: Daniel Su <danielsu@amd.com> Co-authored-by: xuchen-amd <xuchen@amd.com> Co-authored-by: Peter Park <peter.park@amd.com> Co-authored-by: cfallows-amd <Carrie.Fallows@amd.com> Co-authored-by: David Galiffi <David.Galiffi@amd.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ben Richard <143630488+benrichard-amd@users.noreply.github.com> Co-authored-by: Tim Gu <Tim.Gu@amd.com> Co-authored-by: ywang103-amd <ywang103@amd.com> Co-authored-by: benrichard-amd <ben.richard@amd.com> Co-authored-by: YANG WANG <ywang@ywang-ubuntu.amd.com> Co-authored-by: Fei Zheng <44449748+feizheng10@users.noreply.github.com> Co-authored-by: fei.zheng <fei.zheng@amd.com> Co-authored-by: vedithal-amd <Vignesh.Edithal@amd.com>
This commit is contained in:
Cole Ramos
GitHub
+17
-15
@@ -22,22 +22,24 @@
|
||||
# SOFTWARE.
|
||||
##############################################################################el
|
||||
|
||||
import getpass
|
||||
import os
|
||||
from abc import ABC, abstractmethod
|
||||
from pathlib import Path
|
||||
|
||||
import pandas as pd
|
||||
from pymongo import MongoClient
|
||||
from tqdm import tqdm
|
||||
|
||||
from utils.kernel_name_shortener import kernel_name_shortener
|
||||
from utils.utils import (
|
||||
is_workload_empty,
|
||||
demarcate,
|
||||
console_debug,
|
||||
console_error,
|
||||
console_log,
|
||||
console_warning,
|
||||
console_debug,
|
||||
demarcate,
|
||||
is_workload_empty,
|
||||
)
|
||||
from pymongo import MongoClient
|
||||
from tqdm import tqdm
|
||||
from utils.kernel_name_shortener import kernel_name_shortener
|
||||
|
||||
import os
|
||||
import getpass
|
||||
import pandas as pd
|
||||
|
||||
MAX_SERVER_SEL_DELAY = 5000 # 5 sec connection timeout
|
||||
|
||||
@@ -63,8 +65,8 @@ class DatabaseConnector:
|
||||
@demarcate
|
||||
def prep_import(self):
|
||||
# Extract SoC and workload name from sysinfo.csv
|
||||
sys_info = os.path.join(self.connection_info["workload"], "sysinfo.csv")
|
||||
if os.path.isfile(sys_info):
|
||||
sys_info = str(Path(self.connection_info["workload"]).joinpath("sysinfo.csv"))
|
||||
if Path(sys_info).is_file():
|
||||
sys_info = pd.read_csv(sys_info)
|
||||
try:
|
||||
soc = sys_info["gpu_model"][0].strip()
|
||||
@@ -192,7 +194,7 @@ class DatabaseConnector:
|
||||
% self.interaction_type
|
||||
)
|
||||
|
||||
if os.path.isdir(os.path.abspath(self.connection_info["workload"])):
|
||||
if Path(self.connection_info["workload"]).absolute().is_dir():
|
||||
is_workload_empty(self.connection_info["workload"])
|
||||
else:
|
||||
console_error(
|
||||
@@ -203,8 +205,8 @@ class DatabaseConnector:
|
||||
console_error("--team exceeds 13 character limit. Try again.")
|
||||
|
||||
# format path properly
|
||||
self.connection_info["workload"] = os.path.abspath(
|
||||
self.connection_info["workload"]
|
||||
self.connection_info["workload"] = str(
|
||||
Path(self.connection_info["workload"]).absolute().resolve()
|
||||
)
|
||||
|
||||
# Detect password
|
||||
|
||||
+108
-38
@@ -22,20 +22,21 @@
|
||||
# SOFTWARE.
|
||||
##############################################################################el
|
||||
|
||||
import os
|
||||
import sys
|
||||
import pandas as pd
|
||||
import re
|
||||
import yaml
|
||||
|
||||
import glob
|
||||
import collections
|
||||
import glob
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
from collections import OrderedDict
|
||||
from pathlib import Path
|
||||
from utils import schema
|
||||
from utils.utils import console_debug, console_error, demarcate
|
||||
from utils.kernel_name_shortener import kernel_name_shortener
|
||||
|
||||
import pandas as pd
|
||||
import yaml
|
||||
|
||||
import config
|
||||
from utils import schema
|
||||
from utils.kernel_name_shortener import kernel_name_shortener
|
||||
from utils.utils import console_debug, console_error, demarcate
|
||||
|
||||
# TODO: use pandas chunksize or dask to read really large csv file
|
||||
# from dask import dataframe as dd
|
||||
@@ -72,7 +73,7 @@ def load_panel_configs(dir):
|
||||
for root, dirs, files in os.walk(dir):
|
||||
for f in files:
|
||||
if f.endswith(".yaml"):
|
||||
with open(os.path.join(root, f)) as file:
|
||||
with open(str(Path(root).joinpath(f))) as file:
|
||||
config = yaml.safe_load(file)
|
||||
d[config["Panel Config"]["id"]] = config["Panel Config"]
|
||||
|
||||
@@ -86,9 +87,11 @@ def load_panel_configs(dir):
|
||||
|
||||
@demarcate
|
||||
def create_df_kernel_top_stats(
|
||||
df_in,
|
||||
raw_data_dir,
|
||||
filter_gpu_ids,
|
||||
filter_dispatch_ids,
|
||||
filter_nodes,
|
||||
time_unit,
|
||||
max_stat_num,
|
||||
kernel_verbose,
|
||||
@@ -97,14 +100,17 @@ def create_df_kernel_top_stats(
|
||||
"""
|
||||
Create top stats info by grouping kernels with user's filters.
|
||||
"""
|
||||
# NB:
|
||||
# We even don't have to create pmc_kernel_top.csv explictly
|
||||
df = pd.read_csv(os.path.join(raw_data_dir, schema.pmc_perf_file_prefix + ".csv"))
|
||||
|
||||
df = df_in["pmc_perf"]
|
||||
# Demangle original KernelNames
|
||||
kernel_name_shortener(df, kernel_verbose)
|
||||
|
||||
# The logic below for filters are the same as in parser.apply_filters(),
|
||||
# which can be merged together if need it.
|
||||
|
||||
if filter_nodes:
|
||||
df = df.loc[df["Node"].astype(str).isin([filter_nodes])]
|
||||
|
||||
if filter_gpu_ids:
|
||||
df = df.loc[df["GPU_ID"].astype(str).isin([filter_gpu_ids])]
|
||||
|
||||
@@ -118,8 +124,14 @@ def create_df_kernel_top_stats(
|
||||
df = df.loc[df["Dispatch_ID"].astype(str).isin(filter_dispatch_ids)]
|
||||
|
||||
# First, create a dispatches file used to populate global vars
|
||||
dispatch_info = df.loc[:, ["Dispatch_ID", "Kernel_Name", "GPU_ID"]]
|
||||
dispatch_info.to_csv(os.path.join(raw_data_dir, "pmc_dispatch_info.csv"), index=False)
|
||||
dispatch_info = (
|
||||
df.loc[:, ["Node", "Dispatch_ID", "Kernel_Name", "GPU_ID"]]
|
||||
if "Node" in df.columns
|
||||
else df.loc[:, ["Dispatch_ID", "Kernel_Name", "GPU_ID"]]
|
||||
)
|
||||
dispatch_info.to_csv(
|
||||
str(Path(raw_data_dir).joinpath("pmc_dispatch_info.csv")), index=False
|
||||
)
|
||||
|
||||
time_stats = pd.concat(
|
||||
[df["Kernel_Name"], (df["End_Timestamp"] - df["Start_Timestamp"])],
|
||||
@@ -153,37 +165,80 @@ def create_df_kernel_top_stats(
|
||||
# Sort by total time as default.
|
||||
if sortby == "sum":
|
||||
grouped = grouped.sort_values(by=("Sum" + time_unit_str), ascending=False)
|
||||
grouped.to_csv(os.path.join(raw_data_dir, "pmc_kernel_top.csv"), index=False)
|
||||
grouped.to_csv(
|
||||
str(Path(raw_data_dir).joinpath("pmc_kernel_top.csv")), index=False
|
||||
)
|
||||
elif sortby == "kernel":
|
||||
grouped = grouped.sort_values("Kernel_Name")
|
||||
grouped.to_csv(os.path.join(raw_data_dir, "pmc_kernel_top.csv"), index=False)
|
||||
grouped.to_csv(
|
||||
str(Path(raw_data_dir).joinpath("pmc_kernel_top.csv")), index=False
|
||||
)
|
||||
|
||||
|
||||
@demarcate
|
||||
def create_df_pmc(raw_data_dir, kernel_verbose, verbose):
|
||||
def create_df_pmc(raw_data_root_dir, nodes, kernel_verbose, verbose):
|
||||
"""
|
||||
Load all raw pmc counters and join into one df.
|
||||
"""
|
||||
dfs = []
|
||||
coll_levels = []
|
||||
|
||||
df = pd.DataFrame()
|
||||
new_df = pd.DataFrame()
|
||||
for root, dirs, files in os.walk(raw_data_dir):
|
||||
for f in files:
|
||||
# print("file ", f)
|
||||
if (f.endswith(".csv") and f.startswith("SQ")) or (
|
||||
f == schema.pmc_perf_file_prefix + ".csv"
|
||||
):
|
||||
tmp_df = pd.read_csv(os.path.join(root, f))
|
||||
# Demangle original KernelNames
|
||||
kernel_name_shortener(tmp_df, kernel_verbose)
|
||||
dfs.append(tmp_df)
|
||||
coll_levels.append(f[:-4])
|
||||
final_df = pd.concat(dfs, keys=coll_levels, axis=1, copy=False)
|
||||
if verbose >= 2:
|
||||
console_debug("pmc_raw_data final_df %s" % final_df.info)
|
||||
return final_df
|
||||
def create_single_df_pmc(raw_data_dir, node_name, kernel_verbose, verbose):
|
||||
dfs = []
|
||||
coll_levels = []
|
||||
|
||||
df = pd.DataFrame()
|
||||
new_df = pd.DataFrame()
|
||||
for root, dirs, files in os.walk(raw_data_dir):
|
||||
for f in files:
|
||||
# print("file ", f)
|
||||
if (f.endswith(".csv") and f.startswith("SQ")) or (
|
||||
f == schema.pmc_perf_file_prefix + ".csv"
|
||||
):
|
||||
tmp_df = pd.read_csv(str(Path(root).joinpath(f)))
|
||||
# Demangle original KernelNames
|
||||
kernel_name_shortener(tmp_df, kernel_verbose)
|
||||
|
||||
# NB:
|
||||
# Idealy, the Node column should be added out of
|
||||
# multiindexing level. Here, we add it into pmc_perf
|
||||
# as it is the main sub-df which can be handled easily
|
||||
# later.
|
||||
if f == "pmc_perf.csv" and node_name != None:
|
||||
tmp_df.insert(0, "Node", node_name)
|
||||
dfs.append(tmp_df)
|
||||
coll_levels.append(f[:-4])
|
||||
|
||||
final_df = pd.concat(dfs, keys=coll_levels, axis=1, copy=False)
|
||||
if verbose >= 2:
|
||||
console_debug("pmc_raw_data final_single_df %s" % final_df.info)
|
||||
return final_df
|
||||
|
||||
# regular single node case
|
||||
if nodes is None:
|
||||
return create_single_df_pmc(raw_data_root_dir, None, kernel_verbose, verbose)
|
||||
|
||||
# "empty list" means all nodes
|
||||
elif not nodes:
|
||||
df = pd.DataFrame()
|
||||
# todo: more err check
|
||||
for subdir in Path(raw_data_root_dir).iterdir():
|
||||
if subdir.is_dir():
|
||||
new_df = create_single_df_pmc(
|
||||
subdir, str(subdir.name), kernel_verbose, verbose
|
||||
)
|
||||
df = pd.concat([df, new_df])
|
||||
return df
|
||||
|
||||
# specified node list
|
||||
else:
|
||||
df = pd.DataFrame()
|
||||
# todo: more err check
|
||||
for subdir in nodes:
|
||||
p = Path(raw_data_root_dir)
|
||||
new_df = create_single_df_pmc(
|
||||
p.joinpath(subdir), subdir, kernel_verbose, verbose
|
||||
)
|
||||
df = pd.concat([df, new_df])
|
||||
return df
|
||||
|
||||
|
||||
def collect_wave_occu_per_cu(in_dir, out_dir, numSE):
|
||||
@@ -239,3 +294,18 @@ def is_single_panel_config(root_dir, supported_archs):
|
||||
return False
|
||||
else:
|
||||
console_error("Found multiple panel config sets but incomplete for all archs.")
|
||||
|
||||
|
||||
def find_1st_sub_dir(directory):
|
||||
"""
|
||||
Find the first sub dir in a directory
|
||||
"""
|
||||
dir_path = Path(directory)
|
||||
try:
|
||||
# Iterate over entries in the directory
|
||||
for entry in dir_path.iterdir():
|
||||
if entry.is_dir(): # Check if it's a directory
|
||||
return entry
|
||||
except FileNotFoundError:
|
||||
print(f"The directory '{directory}' does not exist.")
|
||||
return None
|
||||
|
||||
@@ -22,24 +22,23 @@
|
||||
# SOFTWARE.
|
||||
##############################################################################el
|
||||
|
||||
import sys
|
||||
|
||||
from lxml import html
|
||||
from PyQt5.QtGui import QStandardItem, QStandardItemModel
|
||||
from PyQt5.QtWidgets import (
|
||||
QMainWindow,
|
||||
QAbstractItemView,
|
||||
QAction,
|
||||
QApplication,
|
||||
QTreeView,
|
||||
QFileDialog,
|
||||
QHBoxLayout,
|
||||
QMainWindow,
|
||||
QTableWidget,
|
||||
QTableWidgetItem,
|
||||
)
|
||||
from PyQt5.QtWidgets import (
|
||||
QHBoxLayout,
|
||||
QTreeView,
|
||||
QWidget,
|
||||
QAction,
|
||||
QFileDialog,
|
||||
QAbstractItemView,
|
||||
qApp,
|
||||
)
|
||||
from PyQt5.QtGui import QStandardItemModel, QStandardItem
|
||||
from lxml import html
|
||||
import sys
|
||||
|
||||
|
||||
# class view(QWidget):
|
||||
|
||||
+3
-3
@@ -22,10 +22,10 @@
|
||||
# SOFTWARE.
|
||||
##############################################################################el
|
||||
|
||||
import pandas as pd
|
||||
from dash import html, dash_table
|
||||
import plotly.express as px
|
||||
import colorlover
|
||||
import pandas as pd
|
||||
import plotly.express as px
|
||||
from dash import dash_table, html
|
||||
|
||||
from utils import schema
|
||||
from utils.utils import console_error
|
||||
|
||||
@@ -22,8 +22,8 @@
|
||||
# SOFTWARE.
|
||||
##############################################################################el
|
||||
|
||||
from dash import html, dcc
|
||||
import dash_bootstrap_components as dbc
|
||||
from dash import dcc, html
|
||||
|
||||
from utils import schema
|
||||
|
||||
|
||||
@@ -23,7 +23,8 @@
|
||||
##############################################################################el
|
||||
|
||||
from dash import html
|
||||
from dash_svg import Svg, G, Path, Rect, Text
|
||||
from dash_svg import G, Path, Rect, Svg, Text
|
||||
|
||||
from utils.utils import console_error
|
||||
|
||||
hidden_columns = ["Tips", "coll_level"]
|
||||
|
||||
@@ -22,12 +22,13 @@
|
||||
# SOFTWARE.
|
||||
##############################################################################el
|
||||
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
|
||||
import pandas as pd
|
||||
|
||||
from utils.utils import console_error, console_log, console_debug
|
||||
from utils.utils import console_debug, console_error, console_log
|
||||
|
||||
cache = dict()
|
||||
|
||||
@@ -118,8 +119,8 @@ def kernel_name_shortener(df, level):
|
||||
|
||||
# Only shorten if valid shortening level
|
||||
if level < 5:
|
||||
cpp_filt = os.path.join("/usr", "bin", "c++filt")
|
||||
if not os.path.isfile(cpp_filt):
|
||||
cpp_filt = str(Path("/usr").joinpath("bin", "c++filt"))
|
||||
if not Path(cpp_filt).is_file():
|
||||
console_error(
|
||||
"Could not resolve c++filt in expected directory: %s" % cpp_filt
|
||||
)
|
||||
|
||||
+3
-1
@@ -25,6 +25,8 @@
|
||||
import logging
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
from utils.utils import trace_logger
|
||||
|
||||
# Define the colors
|
||||
@@ -110,7 +112,7 @@ def setup_console_handler():
|
||||
|
||||
# Setup file handler - enabled in profile mode
|
||||
def setup_file_handler(loglevel, workload_dir):
|
||||
filename = os.path.join(workload_dir, "log.txt")
|
||||
filename = str(Path(workload_dir).joinpath("log.txt"))
|
||||
file_handler = logging.FileHandler(filename, "w")
|
||||
file_loglevel = min([loglevel, logging.INFO])
|
||||
file_handler.setLevel(file_loglevel)
|
||||
|
||||
@@ -23,8 +23,9 @@
|
||||
###############################################################################el
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Dict, List, Mapping, Generator
|
||||
from types import SimpleNamespace as NS
|
||||
from typing import Dict, Generator, List, Mapping
|
||||
|
||||
from plotille import Canvas
|
||||
|
||||
|
||||
|
||||
+17
-7
@@ -23,16 +23,17 @@
|
||||
##############################################################################el
|
||||
|
||||
import ast
|
||||
import astunparse
|
||||
import re
|
||||
import os
|
||||
import warnings
|
||||
import pandas as pd
|
||||
import numpy as np
|
||||
from utils import schema
|
||||
from utils.utils import console_warning, console_error, demarcate
|
||||
from pathlib import Path
|
||||
|
||||
import astunparse
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
|
||||
from utils import schema
|
||||
from utils.utils import console_error, console_warning, demarcate
|
||||
|
||||
# ------------------------------------------------------------------------------
|
||||
# Internal global definitions
|
||||
|
||||
@@ -844,6 +845,15 @@ def apply_filters(workload, dir, is_gui, debug):
|
||||
# TODO: error out properly if filters out of bound
|
||||
ret_df = workload.raw_pmc
|
||||
|
||||
if workload.filter_nodes:
|
||||
ret_df = ret_df.loc[
|
||||
ret_df[schema.pmc_perf_file_prefix]["Node"]
|
||||
.astype(str)
|
||||
.isin([workload.filter_gpu_ids])
|
||||
]
|
||||
if ret_df.empty:
|
||||
console_error("analysis", "{} is invalid".format(workload.filter_nodes))
|
||||
|
||||
if workload.filter_gpu_ids:
|
||||
ret_df = ret_df.loc[
|
||||
ret_df[schema.pmc_perf_file_prefix]["GPU_ID"]
|
||||
@@ -862,7 +872,7 @@ def apply_filters(workload, dir, is_gui, debug):
|
||||
if workload.filter_kernel_ids:
|
||||
if all(type(kid) == int for kid in workload.filter_kernel_ids):
|
||||
# Verify valid kernel filter
|
||||
kernels_df = pd.read_csv(os.path.join(dir, "pmc_kernel_top.csv"))
|
||||
kernels_df = pd.read_csv(str(Path(dir).joinpath("pmc_kernel_top.csv")))
|
||||
for kernel_id in workload.filter_kernel_ids:
|
||||
if kernel_id >= len(kernels_df["Kernel_Name"]):
|
||||
console_error(
|
||||
|
||||
+44
-17
@@ -22,11 +22,11 @@
|
||||
# SOFTWARE.
|
||||
##############################################################################el
|
||||
|
||||
import os
|
||||
|
||||
from dataclasses import dataclass
|
||||
from utils.utils import console_debug
|
||||
import csv
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
|
||||
from utils.utils import console_debug
|
||||
|
||||
################################################
|
||||
# Global vars
|
||||
@@ -34,8 +34,6 @@ import csv
|
||||
|
||||
IMGNAME = "empirRoof"
|
||||
|
||||
L2_BANKS = 32 # default assuming mi200
|
||||
|
||||
XMIN = 0.01
|
||||
XMAX = 1000
|
||||
|
||||
@@ -43,7 +41,7 @@ FONT_SIZE = 16
|
||||
FONT_COLOR = "black"
|
||||
FONT_WEIGHT = "bold"
|
||||
|
||||
SUPPORTED_SOC = ["mi200"]
|
||||
SUPPORTED_SOC = ["mi200", "mi300"]
|
||||
|
||||
TOP_N = 10
|
||||
|
||||
@@ -188,7 +186,7 @@ def calc_ceilings(roofline_parameters, dtype, benchmark_data):
|
||||
# Overlay application performance
|
||||
# -------------------------------------------------------------------------------------
|
||||
# Calculate relevant metrics for ai calculation
|
||||
def calc_ai(sort_type, ret_df):
|
||||
def calc_ai(mspec, sort_type, ret_df):
|
||||
"""Given counter data, calculate arithmetic intensity for each kernel in the application."""
|
||||
df = ret_df["pmc_perf"]
|
||||
# Sort by top kernels or top dispatches?
|
||||
@@ -306,8 +304,8 @@ def calc_ai(sort_type, ret_df):
|
||||
lds_data += (
|
||||
(df["SQ_LDS_IDX_ACTIVE"][idx] - df["SQ_LDS_BANK_CONFLICT"][idx])
|
||||
* 4
|
||||
* L2_BANKS
|
||||
) # L2_BANKS = 32 (since assuming mi200)
|
||||
* (mspec.lds_banks_per_cu)
|
||||
)
|
||||
except KeyError:
|
||||
console_debug(
|
||||
"roofline",
|
||||
@@ -338,12 +336,39 @@ def calc_ai(sort_type, ret_df):
|
||||
)
|
||||
pass
|
||||
try:
|
||||
hbm_data += (
|
||||
(df["TCC_EA_RDREQ_32B_sum"][idx] * 32)
|
||||
+ ((df["TCC_EA_RDREQ_sum"][idx] - df["TCC_EA_RDREQ_32B_sum"][idx]) * 64)
|
||||
+ (df["TCC_EA_WRREQ_64B_sum"][idx] * 64)
|
||||
+ ((df["TCC_EA_WRREQ_sum"][idx] - df["TCC_EA_WRREQ_64B_sum"][idx]) * 32)
|
||||
)
|
||||
if mspec.gpu_series == "MI200":
|
||||
hbm_data += (
|
||||
(df["TCC_EA_RDREQ_32B_sum"][idx] * 32)
|
||||
+ (
|
||||
(df["TCC_EA_RDREQ_sum"][idx] - df["TCC_EA_RDREQ_32B_sum"][idx])
|
||||
* 64
|
||||
)
|
||||
+ (df["TCC_EA_WRREQ_64B_sum"][idx] * 64)
|
||||
+ (
|
||||
(df["TCC_EA_WRREQ_sum"][idx] - df["TCC_EA_WRREQ_64B_sum"][idx])
|
||||
* 32
|
||||
)
|
||||
)
|
||||
|
||||
else:
|
||||
# Use TCC_BUBBLE_sum to calculate hbm_data
|
||||
hbm_data += (
|
||||
(df["TCC_BUBBLE_sum"][idx] * 128)
|
||||
+ (df["TCC_EA0_RDREQ_32B_sum"][idx] * 32)
|
||||
+ (
|
||||
(
|
||||
df["TCC_EA0_RDREQ_sum"][idx]
|
||||
- df["TCC_BUBBLE_sum"][idx]
|
||||
- df["TCC_EA0_RDREQ_32B_sum"][idx]
|
||||
)
|
||||
* 64
|
||||
)
|
||||
+ (
|
||||
(df["TCC_EA0_WRREQ_sum"][idx] - df["TCC_EA0_WRREQ_64B_sum"][idx])
|
||||
* 32
|
||||
)
|
||||
+ (df["TCC_EA0_WRREQ_64B_sum"][idx] * 64)
|
||||
)
|
||||
except KeyError:
|
||||
console_debug(
|
||||
"roofline",
|
||||
@@ -475,7 +500,9 @@ def calc_ai(sort_type, ret_df):
|
||||
|
||||
|
||||
def constuct_roof(roofline_parameters, dtype):
|
||||
benchmark_results = os.path.join(roofline_parameters["workload_dir"], "roofline.csv")
|
||||
benchmark_results = str(
|
||||
Path(roofline_parameters["workload_dir"]).joinpath("roofline.csv")
|
||||
)
|
||||
# -----------------------------------------------------
|
||||
# Initialize roofline data dictionary from roofline.csv
|
||||
# -----------------------------------------------------
|
||||
|
||||
BIN
File binario non mostrato.
BIN
File binario non mostrato.
BIN
File binario non mostrato.
+5
-3
@@ -27,10 +27,11 @@
|
||||
# predifned dict and global functions.
|
||||
#
|
||||
|
||||
import pandas as pd
|
||||
from typing import Dict, List, Mapping, Generator
|
||||
from dataclasses import dataclass, field
|
||||
from collections import OrderedDict
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Dict, Generator, List, Mapping
|
||||
|
||||
import pandas as pd
|
||||
|
||||
|
||||
@dataclass
|
||||
@@ -65,6 +66,7 @@ class Workload:
|
||||
filter_kernel_ids: List[int] = field(default_factory=list)
|
||||
filter_gpu_ids: List[int] = field(default_factory=list)
|
||||
filter_dispatch_ids: List[int] = field(default_factory=list)
|
||||
filter_nodes: List[str] = field(default_factory=list)
|
||||
avail_ips: List[int] = field(default_factory=list)
|
||||
|
||||
|
||||
|
||||
+23
-15
@@ -24,26 +24,27 @@
|
||||
# SOFTWARE.
|
||||
##############################################################################el
|
||||
|
||||
import importlib
|
||||
import os
|
||||
import re
|
||||
import socket
|
||||
import subprocess
|
||||
import importlib
|
||||
import config
|
||||
import pandas as pd
|
||||
|
||||
from dataclasses import dataclass, field, fields
|
||||
from datetime import datetime
|
||||
from math import ceil
|
||||
from dataclasses import dataclass, field, fields
|
||||
from pathlib import Path as path
|
||||
from utils.utils import (
|
||||
total_xcds,
|
||||
get_version,
|
||||
console_error,
|
||||
console_warning,
|
||||
console_log,
|
||||
)
|
||||
|
||||
import pandas as pd
|
||||
|
||||
import config
|
||||
from utils.tty import get_table_string
|
||||
from utils.utils import (
|
||||
console_error,
|
||||
console_log,
|
||||
console_warning,
|
||||
get_version,
|
||||
total_xcds,
|
||||
)
|
||||
|
||||
VERSION_LOC = [
|
||||
"version",
|
||||
@@ -336,6 +337,13 @@ class MachineSpecs:
|
||||
##########################################
|
||||
## B. SoC Specs
|
||||
##########################################
|
||||
gpu_series: str = field(
|
||||
default=None,
|
||||
metadata={
|
||||
"doc": "The series of the accelerators/GPUs in the system.",
|
||||
"name": "GPU Series",
|
||||
},
|
||||
)
|
||||
gpu_model: str = field(
|
||||
default=None,
|
||||
metadata={
|
||||
@@ -407,7 +415,7 @@ class MachineSpecs:
|
||||
chip_id: str = field(
|
||||
default=None,
|
||||
metadata={
|
||||
"doc": "<>",
|
||||
"doc": "The Chip ID of the accelerators/GPUs in the system.",
|
||||
"name": "Chip ID",
|
||||
},
|
||||
)
|
||||
@@ -586,8 +594,8 @@ class MachineSpecs:
|
||||
def get_rocm_ver():
|
||||
rocm_found = False
|
||||
for itr in VERSION_LOC:
|
||||
_path = os.path.join(os.getenv("ROCM_PATH", "/opt/rocm"), ".info", itr)
|
||||
if os.path.exists(_path):
|
||||
_path = str(path(os.getenv("ROCM_PATH", "/opt/rocm")).joinpath(".info", itr))
|
||||
if path(_path).exists():
|
||||
rocm_ver = path(_path).read_text()
|
||||
rocm_found = True
|
||||
break
|
||||
|
||||
+5
-4
@@ -22,13 +22,14 @@
|
||||
# SOFTWARE.
|
||||
##############################################################################el
|
||||
|
||||
import pandas as pd
|
||||
from pathlib import Path
|
||||
from tabulate import tabulate
|
||||
import copy
|
||||
from pathlib import Path
|
||||
|
||||
import pandas as pd
|
||||
from tabulate import tabulate
|
||||
|
||||
from utils import parser
|
||||
from utils.utils import console_warning, console_log
|
||||
from utils.utils import console_log, console_warning
|
||||
|
||||
hidden_columns = ["Tips", "coll_level"]
|
||||
hidden_sections = [1900, 2000]
|
||||
|
||||
+429
-25
@@ -22,21 +22,29 @@
|
||||
# SOFTWARE.
|
||||
##############################################################################el
|
||||
|
||||
import glob
|
||||
import io
|
||||
import json
|
||||
import locale
|
||||
import logging
|
||||
import sys
|
||||
import os
|
||||
import io
|
||||
import pathlib
|
||||
import re
|
||||
import selectors
|
||||
import subprocess
|
||||
import shutil
|
||||
import pandas as pd
|
||||
import glob
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from collections import OrderedDict
|
||||
from itertools import product
|
||||
from pathlib import Path as path
|
||||
|
||||
import pandas as pd
|
||||
|
||||
import config
|
||||
|
||||
rocprof_cmd = ""
|
||||
rocprof_args = ""
|
||||
|
||||
|
||||
def demarcate(function):
|
||||
@@ -97,7 +105,7 @@ def get_version(rocprof_compute_home) -> dict:
|
||||
versionDir = None
|
||||
|
||||
for dir in searchDirs:
|
||||
version = os.path.join(dir, "VERSION")
|
||||
version = str(path(dir).joinpath("VERSION"))
|
||||
try:
|
||||
with open(version, "r") as file:
|
||||
VER = file.read().replace("\n", "")
|
||||
@@ -110,8 +118,8 @@ def get_version(rocprof_compute_home) -> dict:
|
||||
console_error("Cannot find VERSION file at {}".format(searchDirs))
|
||||
|
||||
# git version info
|
||||
gitDir = os.path.join(rocprof_compute_home.parent, ".git")
|
||||
if (shutil.which("git") is not None) and os.path.exists(gitDir):
|
||||
gitDir = str(path(rocprof_compute_home.parent).joinpath(".git"))
|
||||
if (shutil.which("git") is not None) and path(gitDir).exists():
|
||||
gitQuery = subprocess.run(
|
||||
["git", "log", "--pretty=format:%h", "-n", "1"],
|
||||
stdout=subprocess.PIPE,
|
||||
@@ -124,7 +132,7 @@ def get_version(rocprof_compute_home) -> dict:
|
||||
SHA = gitQuery.stdout.decode("utf-8")
|
||||
MODE = "dev"
|
||||
else:
|
||||
shaFile = os.path.join(versionDir, "VERSION.sha")
|
||||
shaFile = str(path(versionDir).joinpath("VERSION.sha"))
|
||||
try:
|
||||
with open(shaFile, "r") as file:
|
||||
SHA = file.read().replace("\n", "")
|
||||
@@ -172,13 +180,23 @@ def detect_rocprof():
|
||||
)
|
||||
else:
|
||||
# Resolve any sym links in file path
|
||||
rocprof_path = os.path.realpath(rocprof_path.rstrip("\n"))
|
||||
rocprof_path = str(path(rocprof_path.rstrip("\n")).resolve())
|
||||
console_debug("ROC Profiler: " + str(rocprof_path))
|
||||
return rocprof_cmd # TODO: Do we still need to return this? It's not being used in the function call
|
||||
|
||||
console_debug("rocprof_cmd is {}".format(str(rocprof_cmd)))
|
||||
return rocprof_cmd # TODO: Do we still need to return this? It's not being used in the function call
|
||||
|
||||
|
||||
def store_app_cmd(args):
|
||||
global rocprof_args
|
||||
rocprof_args = args
|
||||
|
||||
|
||||
def capture_subprocess_output(subprocess_args, new_env=None, profileMode=False):
|
||||
console_debug("subprocess", subprocess_args)
|
||||
global rocprof_args
|
||||
# Format command for debug messages, formatting for rocprofv1 and rocprofv2
|
||||
command = " ".join(rocprof_args)
|
||||
console_debug("subprocess", "Running: " + command)
|
||||
# Start subprocess
|
||||
# bufsize = 1 means output is line buffered
|
||||
# universal_newlines = True is required for line buffering
|
||||
@@ -243,11 +261,319 @@ def capture_subprocess_output(subprocess_args, new_env=None, profileMode=False):
|
||||
return (success, output)
|
||||
|
||||
|
||||
def run_prof(fname, profiler_options, workload_dir, mspec, loglevel):
|
||||
# Create a dictionary that maps agent ID to agent objects
|
||||
def get_agent_dict(data):
|
||||
agents = data["rocprofiler-sdk-tool"][0]["agents"]
|
||||
|
||||
fbase = os.path.splitext(os.path.basename(fname))[0]
|
||||
agent_map = {}
|
||||
|
||||
console_debug("pmc file: %s" % str(os.path.basename(fname)))
|
||||
for agent in agents:
|
||||
agent_id = agent["id"]["handle"]
|
||||
agent_map[agent_id] = agent
|
||||
|
||||
return agent_map
|
||||
|
||||
|
||||
# Returns a dictionary that maps agent ID to GPU ID
|
||||
# starting at 0.
|
||||
def get_gpuid_dict(data):
|
||||
|
||||
agents = data["rocprofiler-sdk-tool"][0]["agents"]
|
||||
|
||||
agent_list = []
|
||||
|
||||
# Get agent ID and node_id for GPU agents only
|
||||
for agent in agents:
|
||||
|
||||
if agent["type"] == 2:
|
||||
agent_id = agent["id"]["handle"]
|
||||
node_id = agent["node_id"]
|
||||
agent_list.append((agent_id, node_id))
|
||||
|
||||
# Sort by node ID
|
||||
agent_list.sort(key=lambda x: x[1])
|
||||
|
||||
# Map agent ID to node id
|
||||
map = {}
|
||||
gpu_id = 0
|
||||
for agent in agent_list:
|
||||
map[agent[0]] = gpu_id
|
||||
gpu_id = gpu_id + 1
|
||||
|
||||
return map
|
||||
|
||||
|
||||
# Create a dictionary that maps counter ID to counter objects
|
||||
def v3_json_get_counters(data):
|
||||
counters = data["rocprofiler-sdk-tool"][0]["counters"]
|
||||
|
||||
counter_map = {}
|
||||
|
||||
for counter in counters:
|
||||
counter_id = counter["id"]["handle"]
|
||||
agent_id = counter["agent_id"]["handle"]
|
||||
|
||||
counter_map[(agent_id, counter_id)] = counter
|
||||
|
||||
return counter_map
|
||||
|
||||
|
||||
def v3_json_get_dispatches(data):
|
||||
records = data["rocprofiler-sdk-tool"][0]["buffer_records"]
|
||||
|
||||
records_map = {}
|
||||
|
||||
for rec in records["kernel_dispatch"]:
|
||||
id = rec["correlation_id"]["internal"]
|
||||
|
||||
records_map[id] = rec
|
||||
|
||||
return records_map
|
||||
|
||||
|
||||
def v3_json_to_csv(json_file_path, csv_file_path):
|
||||
|
||||
f = open(json_file_path, "rt")
|
||||
data = json.load(f)
|
||||
|
||||
dispatch_records = v3_json_get_dispatches(data)
|
||||
dispatches = data["rocprofiler-sdk-tool"][0]["callback_records"]["counter_collection"]
|
||||
kernel_symbols = data["rocprofiler-sdk-tool"][0]["kernel_symbols"]
|
||||
agents = get_agent_dict(data)
|
||||
pid = data["rocprofiler-sdk-tool"][0]["metadata"]["pid"]
|
||||
|
||||
gpuid_map = get_gpuid_dict(data)
|
||||
|
||||
counter_info = v3_json_get_counters(data)
|
||||
|
||||
# CSV headers. If there are no dispatches we still end up with a valid CSV file.
|
||||
csv_data = dict.fromkeys(
|
||||
[
|
||||
"Dispatch_ID",
|
||||
"GPU_ID",
|
||||
"Queue_ID",
|
||||
"PID",
|
||||
"TID",
|
||||
"Grid_Size",
|
||||
"Workgroup_Size",
|
||||
"LDS_Per_Workgroup",
|
||||
"Scratch_Per_Workitem",
|
||||
"Arch_VGPR",
|
||||
"Accum_VGPR",
|
||||
"SGPR",
|
||||
"Wave_Size",
|
||||
"Kernel_Name",
|
||||
"Start_Timestamp",
|
||||
"End_Timestamp",
|
||||
"Correlation_ID",
|
||||
]
|
||||
)
|
||||
|
||||
for key in csv_data:
|
||||
csv_data[key] = []
|
||||
|
||||
for d in dispatches:
|
||||
|
||||
dispatch_info = d["dispatch_data"]["dispatch_info"]
|
||||
|
||||
agent_id = dispatch_info["agent_id"]["handle"]
|
||||
|
||||
kernel_id = dispatch_info["kernel_id"]
|
||||
|
||||
row = {}
|
||||
|
||||
row["Dispatch_ID"] = dispatch_info["dispatch_id"]
|
||||
|
||||
row["GPU_ID"] = gpuid_map[agent_id]
|
||||
|
||||
row["Queue_ID"] = dispatch_info["queue_id"]["handle"]
|
||||
row["PID"] = pid
|
||||
row["TID"] = d["thread_id"]
|
||||
|
||||
grid_size = dispatch_info["grid_size"]
|
||||
row["Grid_Size"] = grid_size["x"] * grid_size["y"] * grid_size["z"]
|
||||
|
||||
wg = dispatch_info["workgroup_size"]
|
||||
row["Workgroup_Size"] = wg["x"] * wg["y"] * wg["z"]
|
||||
|
||||
row["LDS_Per_Workgroup"] = d["lds_block_size_v"]
|
||||
|
||||
row["Scratch_Per_Workitem"] = kernel_symbols[kernel_id]["private_segment_size"]
|
||||
row["Arch_VGPR"] = d["arch_vgpr_count"]
|
||||
|
||||
# TODO: Accum VGPR is missing from rocprofv3 output.
|
||||
row["Accum_VGPR"] = 0
|
||||
|
||||
row["SGPR"] = d["sgpr_count"]
|
||||
row["Wave_Size"] = agents[agent_id]["wave_front_size"]
|
||||
|
||||
row["Kernel_Name"] = kernel_symbols[kernel_id]["formatted_kernel_name"]
|
||||
|
||||
id = d["dispatch_data"]["correlation_id"]["internal"]
|
||||
rec = dispatch_records[id]
|
||||
|
||||
row["Start_Timestamp"] = rec["start_timestamp"]
|
||||
row["End_Timestamp"] = rec["end_timestamp"]
|
||||
row["Correlation_ID"] = d["dispatch_data"]["correlation_id"]["external"]
|
||||
|
||||
# Get counters
|
||||
ctrs = {}
|
||||
|
||||
records = d["records"]
|
||||
for r in records:
|
||||
ctr_id = r["counter_id"]["handle"]
|
||||
value = r["value"]
|
||||
|
||||
name = counter_info[(agent_id, ctr_id)]["name"]
|
||||
|
||||
if name.endswith("_ACCUM"):
|
||||
# It's an accumulate counter. Omniperf expects the accumulated value
|
||||
# to be in SQ_ACCUM_PREV_HIRES.
|
||||
name = "SQ_ACCUM_PREV_HIRES"
|
||||
|
||||
# Some counters appear multiple times and need to be summed
|
||||
if name in ctrs:
|
||||
ctrs[name] += value
|
||||
else:
|
||||
ctrs[name] = value
|
||||
|
||||
# Append counter values
|
||||
for ctr, value in ctrs.items():
|
||||
row[ctr] = value
|
||||
|
||||
# Add row to CSV data
|
||||
for col_name, value in row.items():
|
||||
if col_name not in csv_data:
|
||||
csv_data[col_name] = []
|
||||
|
||||
csv_data[col_name].append(value)
|
||||
|
||||
df = pd.DataFrame(csv_data)
|
||||
|
||||
df.to_csv(csv_file_path, index=False)
|
||||
|
||||
|
||||
def v3_counter_csv_to_v2_csv(counter_file, agent_info_filepath, converted_csv_file):
|
||||
"""
|
||||
Convert the counter file of csv output for a certain csv from rocprofv3 format to rocprfv2 format.
|
||||
This function is not for use of other csv out file such as kernel trace file.
|
||||
"""
|
||||
pd_counter_collections = pd.read_csv(counter_file)
|
||||
pd_agent_info = pd.read_csv(agent_info_filepath)
|
||||
result = pd_counter_collections.pivot_table(
|
||||
index=[
|
||||
"Correlation_Id",
|
||||
"Dispatch_Id",
|
||||
"Agent_Id",
|
||||
"Queue_Id",
|
||||
"Process_Id",
|
||||
"Thread_Id",
|
||||
"Grid_Size",
|
||||
"Kernel_Id",
|
||||
"Kernel_Name",
|
||||
"Workgroup_Size",
|
||||
"LDS_Block_Size",
|
||||
"Scratch_Size",
|
||||
"VGPR_Count",
|
||||
"SGPR_Count",
|
||||
"Start_Timestamp",
|
||||
"End_Timestamp",
|
||||
],
|
||||
columns="Counter_Name",
|
||||
values="Counter_Value",
|
||||
).reset_index()
|
||||
|
||||
# Grab the Wave_Front_Size column from agent info
|
||||
result = result.merge(
|
||||
pd_agent_info[["Node_Id", "Wave_Front_Size"]],
|
||||
left_on="Agent_Id",
|
||||
right_on="Node_Id",
|
||||
how="left",
|
||||
)
|
||||
|
||||
# Map agent ID (Node_Id) to GPU_ID
|
||||
gpu_id_map = {}
|
||||
gpu_id = 0
|
||||
for idx, row in pd_agent_info.iterrows():
|
||||
if row["Agent_Type"] == "GPU":
|
||||
agent_id = row["Node_Id"]
|
||||
gpu_id_map[agent_id] = gpu_id
|
||||
gpu_id = gpu_id + 1
|
||||
|
||||
# Update Agent_Id for each record to match GPU ID
|
||||
for idx, row in result["Agent_Id"].items():
|
||||
agent_id = result.at[idx, "Agent_Id"]
|
||||
result.at[idx, "Agent_Id"] = gpu_id_map[agent_id]
|
||||
|
||||
# Accum_VGPR is currently missing in rocprofv3 output
|
||||
result["Accum_VGPR"] = 0
|
||||
|
||||
# Drop the 'Node_Id' column if you don't need it in the final DataFrame
|
||||
result.drop(columns="Node_Id", inplace=True)
|
||||
result["Accum_VGPR"] = 0
|
||||
|
||||
name_mapping = {
|
||||
"Dispatch_Id": "Dispatch_ID",
|
||||
"Agent_Id": "GPU_ID",
|
||||
"Queue_Id": "Queue_ID",
|
||||
"Process_Id": "PID",
|
||||
"Thread_Id": "TID",
|
||||
"Grid_Size": "Grid_Size",
|
||||
"Workgroup_Size": "Workgroup_Size",
|
||||
"LDS_Block_Size": "LDS_Per_Workgroup",
|
||||
"Scratch_Size": "Scratch_Per_Workitem",
|
||||
"VGPR_Count": "Arch_VGPR",
|
||||
# "":"Accum_VGPR",
|
||||
"SGPR_Count": "SGPR",
|
||||
"Wave_Front_Size": "Wave_Size",
|
||||
"Kernel_Name": "Kernel_Name",
|
||||
"Start_Timestamp": "Start_Timestamp",
|
||||
"End_Timestamp": "End_Timestamp",
|
||||
"Correlation_Id": "Correlation_ID",
|
||||
"Kernel_Id": "Kernel_ID",
|
||||
}
|
||||
result.rename(columns=name_mapping, inplace=True)
|
||||
|
||||
index = [
|
||||
"Dispatch_ID",
|
||||
"GPU_ID",
|
||||
"Queue_ID",
|
||||
"PID",
|
||||
"TID",
|
||||
"Grid_Size",
|
||||
"Workgroup_Size",
|
||||
"LDS_Per_Workgroup",
|
||||
"Scratch_Per_Workitem",
|
||||
"Arch_VGPR",
|
||||
"Accum_VGPR",
|
||||
"SGPR",
|
||||
"Wave_Size",
|
||||
"Kernel_Name",
|
||||
"Start_Timestamp",
|
||||
"End_Timestamp",
|
||||
"Correlation_ID",
|
||||
"Kernel_ID",
|
||||
]
|
||||
|
||||
remaining_column_names = [col for col in result.columns if col not in index]
|
||||
index = index + remaining_column_names
|
||||
result = result.reindex(columns=index)
|
||||
|
||||
# Rename the accumulate counter to SQ_ACCUM_PREV_HIRES.
|
||||
for col in result.columns:
|
||||
if col.endswith("_ACCUM"):
|
||||
result.rename(columns={col: "SQ_ACCUM_PREV_HIRES"}, inplace=True)
|
||||
|
||||
result.to_csv(converted_csv_file, index=False)
|
||||
|
||||
|
||||
def run_prof(
|
||||
fname, profiler_options, workload_dir, mspec, loglevel, format_rocprof_output
|
||||
):
|
||||
time_0 = time.time()
|
||||
fbase = path(fname).stem
|
||||
|
||||
console_debug("pmc file: %s" % path(fname).name)
|
||||
|
||||
# standard rocprof options
|
||||
default_options = ["-i", fname]
|
||||
@@ -261,15 +587,17 @@ def run_prof(fname, profiler_options, workload_dir, mspec, loglevel):
|
||||
or mspec.gpu_model.lower() == "mi300a_a0"
|
||||
or mspec.gpu_model.lower() == "mi300a_a1"
|
||||
) and (
|
||||
os.path.basename(fname) == "pmc_perf_13.txt"
|
||||
or os.path.basename(fname) == "pmc_perf_14.txt"
|
||||
or os.path.basename(fname) == "pmc_perf_15.txt"
|
||||
or os.path.basename(fname) == "pmc_perf_16.txt"
|
||||
or os.path.basename(fname) == "pmc_perf_17.txt"
|
||||
path(fname).name == "pmc_perf_13.txt"
|
||||
or path(fname).name == "pmc_perf_14.txt"
|
||||
or path(fname).name == "pmc_perf_15.txt"
|
||||
or path(fname).name == "pmc_perf_16.txt"
|
||||
or path(fname).name == "pmc_perf_17.txt"
|
||||
):
|
||||
new_env = os.environ.copy()
|
||||
new_env["ROCPROFILER_INDIVIDUAL_XCC_MODE"] = "1"
|
||||
|
||||
time_1 = time.time()
|
||||
|
||||
# profile the app
|
||||
if new_env:
|
||||
success, output = capture_subprocess_output(
|
||||
@@ -280,6 +608,13 @@ def run_prof(fname, profiler_options, workload_dir, mspec, loglevel):
|
||||
[rocprof_cmd] + options, profileMode=True
|
||||
)
|
||||
|
||||
time_2 = time.time()
|
||||
console_debug(
|
||||
"Finishing subprocess of fname {}, the time it takes was {} m {} sec ".format(
|
||||
fname, int((time_2 - time_1) / 60), str((time_2 - time_1) % 60)
|
||||
)
|
||||
)
|
||||
|
||||
if not success:
|
||||
if loglevel > logging.INFO:
|
||||
for line in output.splitlines():
|
||||
@@ -302,6 +637,75 @@ def run_prof(fname, profiler_options, workload_dir, mspec, loglevel):
|
||||
workload_dir + "/out/pmc_1/results_" + fbase + ".csv", index=False
|
||||
)
|
||||
|
||||
if rocprof_cmd.endswith("v3"):
|
||||
results_files_csv = {}
|
||||
if format_rocprof_output == "json":
|
||||
results_files_json = glob.glob(workload_dir + "/out/pmc_1/*/*.json")
|
||||
|
||||
for json_file in results_files_json:
|
||||
csv_file = pathlib.Path(json_file).with_suffix(".csv")
|
||||
v3_json_to_csv(json_file, csv_file)
|
||||
results_files_csv = glob.glob(workload_dir + "/out/pmc_1/*/*.csv")
|
||||
elif format_rocprof_output == "csv":
|
||||
counter_info_csvs = glob.glob(
|
||||
workload_dir + "/out/pmc_1/*/*_counter_collection.csv"
|
||||
)
|
||||
existing_counter_files_csv = [
|
||||
d for d in counter_info_csvs if path(d).is_file()
|
||||
]
|
||||
|
||||
if len(existing_counter_files_csv) > 0:
|
||||
for counter_file in existing_counter_files_csv:
|
||||
current_dir = str(path(counter_file).parent)
|
||||
agent_info_filepath = str(
|
||||
path(current_dir).joinpath(
|
||||
path(counter_file).name.replace(
|
||||
"_counter_collection", "_agent_info"
|
||||
)
|
||||
)
|
||||
)
|
||||
if not path(agent_info_filepath).is_file():
|
||||
raise ValueError(
|
||||
'{} has no coresponding "agent info" file'.format(
|
||||
counter_file
|
||||
)
|
||||
)
|
||||
|
||||
converted_csv_file = str(
|
||||
path(current_dir).joinpath(
|
||||
path(counter_file).name.replace(
|
||||
"_counter_collection", "_converted"
|
||||
)
|
||||
)
|
||||
)
|
||||
|
||||
v3_counter_csv_to_v2_csv(
|
||||
counter_file, agent_info_filepath, converted_csv_file
|
||||
)
|
||||
|
||||
results_files_csv = glob.glob(
|
||||
workload_dir + "/out/pmc_1/*/*_converted.csv"
|
||||
)
|
||||
else:
|
||||
results_files_csv = glob.glob(
|
||||
workload_dir + "/out/pmc_1/*/*_kernel_trace.csv"
|
||||
)
|
||||
|
||||
else:
|
||||
console_error("The output file of rocprofv3 can only support json or csv!!!")
|
||||
|
||||
# Combine results into single CSV file
|
||||
combined_results = pd.concat(
|
||||
[pd.read_csv(f) for f in results_files_csv], ignore_index=True
|
||||
)
|
||||
|
||||
# Overwrite column to ensure unique IDs.
|
||||
combined_results["Dispatch_ID"] = range(0, len(combined_results))
|
||||
|
||||
combined_results.to_csv(
|
||||
workload_dir + "/out/pmc_1/results_" + fbase + ".csv", index=False
|
||||
)
|
||||
|
||||
if new_env:
|
||||
# flatten tcc for applicable mi300 input
|
||||
f = path(workload_dir + "/out/pmc_1/results_" + fbase + ".csv")
|
||||
@@ -309,7 +713,7 @@ def run_prof(fname, profiler_options, workload_dir, mspec, loglevel):
|
||||
df = flatten_tcc_info_across_xcds(f, xcds, int(mspec._l2_banks))
|
||||
df.to_csv(f, index=False)
|
||||
|
||||
if os.path.exists(workload_dir + "/out"):
|
||||
if path(workload_dir + "/out").exists():
|
||||
# copy and remove out directory if needed
|
||||
shutil.copyfile(
|
||||
workload_dir + "/out/pmc_1/results_" + fbase + ".csv",
|
||||
@@ -398,7 +802,7 @@ def detect_roofline(mspec):
|
||||
|
||||
if "ROOFLINE_BIN" in os.environ.keys():
|
||||
rooflineBinary = os.environ["ROOFLINE_BIN"]
|
||||
if os.path.exists(rooflineBinary):
|
||||
if path(rooflineBinary).exists():
|
||||
console_warning("roofline", "Detected user-supplied binary")
|
||||
return {
|
||||
"rocm_ver": "override",
|
||||
@@ -484,7 +888,7 @@ def mibench(args, mspec):
|
||||
+ "-"
|
||||
+ distro_map[target_binary["distro"]]
|
||||
+ "-"
|
||||
+ mspec.gpu_model.lower()
|
||||
+ mspec.gpu_series.lower()
|
||||
+ "-rocm"
|
||||
+ target_binary["rocm_ver"]
|
||||
)
|
||||
@@ -493,7 +897,7 @@ def mibench(args, mspec):
|
||||
# Distro is valid but cant find rocm ver
|
||||
found = False
|
||||
for path in binary_paths:
|
||||
if os.path.exists(path):
|
||||
if pathlib.Path(path).exists():
|
||||
found = True
|
||||
path_to_binary = path
|
||||
break
|
||||
@@ -647,7 +1051,7 @@ def get_submodules(package_name):
|
||||
def is_workload_empty(path):
|
||||
"""Peek workload directory to verify valid profiling output"""
|
||||
pmc_perf_path = path + "/pmc_perf.csv"
|
||||
if os.path.isfile(pmc_perf_path):
|
||||
if pathlib.Path(pmc_perf_path).is_file():
|
||||
temp_df = pd.read_csv(pmc_perf_path)
|
||||
if temp_df.dropna().empty:
|
||||
console_error(
|
||||
|
||||
Fai riferimento in un nuovo problema
Block a user