Pytest (#212)
* Initial commit for pytest Signed-off-by: JoseSantosAMD <Jose.Santos@amd.com> * Check if csvs populated Signed-off-by: JoseSantosAMD <Jose.Santos@amd.com> * pytest now checks soc for soc specific testing Signed-off-by: JoseSantosAMD <Jose.Santos@amd.com> * Roof only csvs updated - Removes dir if workload dir already present Signed-off-by: JoseSantosAMD <Jose.Santos@amd.com> * Tests create difference logs calculated from a baseline Signed-off-by: JoseSantosAMD <Jose.Santos@amd.com> * pytest scripts updated to revamp test cases Signed-off-by: JoseSantosAMD <Jose.Santos@amd.com> * Fix dispatch notation Signed-off-by: JoseSantosAMD <Jose.Santos@amd.com> * Introduce a cmake option to enable python code coverage. It now defaults to being disabled and can be enabled via a -DENABLE_COVERAGE=ON option (#194). Introduce CI on mi100 that leverages code coverage and publishes results along with a testing report. Signed-off-by: Karl W Schulz <karl.schulz@amd.com> Signed-off-by: JoseSantosAMD <Jose.Santos@amd.com> * update branch for CI -> 2.x Signed-off-by: Karl W Schulz <karl.schulz@amd.com> Signed-off-by: JoseSantosAMD <Jose.Santos@amd.com> * add profile execution test in cmake and enable in CI Signed-off-by: Karl W Schulz <karl.schulz@amd.com> Signed-off-by: JoseSantosAMD <Jose.Santos@amd.com> * run all ci tests Signed-off-by: Karl W. Schulz <karl.schulz@amd.com> Signed-off-by: JoseSantosAMD <Jose.Santos@amd.com> * Initial overhaul of Profile mode Signed-off-by: colramos-amd <colramos@amd.com> Signed-off-by: JoseSantosAMD <Jose.Santos@amd.com> * Modify vcopy to enable multi iteration and multi kernel for CI Signed-off-by: colramos-amd <colramos@amd.com> Signed-off-by: JoseSantosAMD <Jose.Santos@amd.com> * Update docs and --help menu to reflect new vcopy usage Signed-off-by: colramos-amd <colramos@amd.com> Signed-off-by: JoseSantosAMD <Jose.Santos@amd.com> * Fix typo in IPBlock logging Signed-off-by: colramos-amd <colramos@amd.com> Signed-off-by: JoseSantosAMD <Jose.Santos@amd.com> * Cache ascii art in source and remove ascii dependency Signed-off-by: colramos-amd <colramos@amd.com> Signed-off-by: JoseSantosAMD <Jose.Santos@amd.com> * Update testing collateral to reflect name change of kernel-verbose option Signed-off-by: colramos-amd <colramos@amd.com> Signed-off-by: JoseSantosAMD <Jose.Santos@amd.com> * Small typo corrections in docs Signed-off-by: colramos-amd <colramos@amd.com> Signed-off-by: JoseSantosAMD <Jose.Santos@amd.com> * Fixing more typos... Signed-off-by: colramos-amd <colramos@amd.com> Signed-off-by: JoseSantosAMD <Jose.Santos@amd.com> * run profiler tests through queue Signed-off-by: Karl W Schulz <karl.schulz@amd.com> Signed-off-by: JoseSantosAMD <Jose.Santos@amd.com> * bump wallclock runlimit for test Signed-off-by: Karl W Schulz <karl.schulz@amd.com> Signed-off-by: JoseSantosAMD <Jose.Santos@amd.com> * add mi100 badge Signed-off-by: Karl W. Schulz <karl.schulz@amd.com> Signed-off-by: JoseSantosAMD <Jose.Santos@amd.com> * include step to build hip binary Signed-off-by: Karl W Schulz <karl.schulz@amd.com> Signed-off-by: JoseSantosAMD <Jose.Santos@amd.com> * add additional top-level vars for number of kernels expected and dispatch id; tests updated to use these vars so that vcopy binary can be used instead of mixbench Signed-off-by: Karl W Schulz <karl.schulz@amd.com> Signed-off-by: JoseSantosAMD <Jose.Santos@amd.com> * add requirements-test.txt file to organize python testing dependencies (#205) * add a requirements-test.txt file to organize additional python packages needed to support testing; update CI to install python packages from dependency files Signed-off-by: Karl W Schulz <karl.schulz@amd.com> Signed-off-by: JoseSantosAMD <Jose.Santos@amd.com> * Dynamically load dependency checker from requirements.txt Signed-off-by: colramos-amd <colramos@amd.com> Signed-off-by: JoseSantosAMD <Jose.Santos@amd.com> * Add comments Signed-off-by: colramos-amd <colramos@amd.com> Signed-off-by: JoseSantosAMD <Jose.Santos@amd.com> * Replace any - with _ for checker logic Signed-off-by: colramos-amd <colramos@amd.com> Signed-off-by: JoseSantosAMD <Jose.Santos@amd.com> * Update cmake config with Fix for pyyaml package installation check Signed-off-by: Karl W Schulz <karl.schulz@amd.com> Signed-off-by: JoseSantosAMD <Jose.Santos@amd.com> * update mi100 CI config Signed-off-by: Karl W Schulz <karl.schulz@amd.com> Signed-off-by: JoseSantosAMD <Jose.Santos@amd.com> * upgrade jsdom version (#206) Signed-off-by: JoseSantosAMD <Jose.Santos@amd.com> * Adding the log_metric differ function Signed-off-by: JoseSantosAMD <Jose.Santos@amd.com> * Added missing hasattr checker Signed-off-by: colramos-amd <colramos@amd.com> Signed-off-by: JoseSantosAMD <Jose.Santos@amd.com> * Enable running on mi100 and fixed typos - Also enables RHEL9 roofline Signed-off-by: JoseSantosAMD <Jose.Santos@amd.com> * update CMakeLists.txt Signed-off-by: JoseSantosAMD <Jose.Santos@amd.com> * Remove pytest raises SystemExit Add default thresholds Signed-off-by: JoseSantosAMD <Jose.Santos@amd.com> * Initial overhaul of Analyze mode. Basic CLI is enabled. Signed-off-by: colramos-amd <colramos@amd.com> Signed-off-by: JoseSantosAMD <Jose.Santos@amd.com> * Revert "Remove pytest raises SystemExit" This reverts commit 9762728eb37e740c39ced87ced7dd7a0d423824d. Signed-off-by: JoseSantosAMD <Jose.Santos@amd.com> * restore formal call to sys.exit() in top-level main; this is needed for companion testing collateral which expects SystemExit exceptions to be raised. Signed-off-by: Karl W Schulz <karl.schulz@amd.com> Signed-off-by: JoseSantosAMD <Jose.Santos@amd.com> * Update return code error checks for roofline-only analysis on MI100 Signed-off-by: Karl W Schulz <karl.schulz@amd.com> Signed-off-by: JoseSantosAMD <Jose.Santos@amd.com> * disable coverage comment Signed-off-by: Karl W Schulz <karl.schulz@amd.com> Signed-off-by: JoseSantosAMD <Jose.Santos@amd.com> * fix comment typo Signed-off-by: Karl W Schulz <karl.schulz@amd.com> Signed-off-by: JoseSantosAMD <Jose.Santos@amd.com> * Move error logging to util.py Signed-off-by: colramos-amd <colramos@amd.com> Signed-off-by: JoseSantosAMD <Jose.Santos@amd.com> * Move perfmon_configs dir into omniperf_soc dir. Rename config dirs for clarity Signed-off-by: colramos-amd <colramos@amd.com> Signed-off-by: JoseSantosAMD <Jose.Santos@amd.com> * Add a supported_archs property to Omniperf base class Signed-off-by: colramos-amd <colramos@amd.com> Signed-off-by: JoseSantosAMD <Jose.Santos@amd.com> * Remove pytest raises SystemExit Add default thresholds Signed-off-by: JoseSantosAMD <Jose.Santos@amd.com> * Revert "Remove pytest raises SystemExit" This reverts commit 9762728eb37e740c39ced87ced7dd7a0d423824d. Signed-off-by: JoseSantosAMD <Jose.Santos@amd.com> * Add Absolute Difference to report-diff output - Pytest regex updated to read Absolute Difference - Pytest creates Baseline (Assumes vcopy binary present) - Adding Metric logging Signed-off-by: JoseSantosAMD <Jose.Santos@amd.com> * vcopy binary created before running tests Signed-off-by: JoseSantosAMD <Jose.Santos@amd.com> * Adding the log_metric differ function Signed-off-by: JoseSantosAMD <Jose.Santos@amd.com> * Update test_profile_general.py remove extra log_metric function Signed-off-by: JoseSantosAMD <87447437+JoseSantosAMD@users.noreply.github.com> * Update CMakeLists.txt soc_params was merged into a class, it isn't a standalone file anymore Signed-off-by: JoseSantosAMD <87447437+JoseSantosAMD@users.noreply.github.com> --------- Signed-off-by: JoseSantosAMD <Jose.Santos@amd.com> Signed-off-by: Karl W Schulz <karl.schulz@amd.com> Signed-off-by: Karl W. Schulz <karl.schulz@amd.com> Signed-off-by: colramos-amd <colramos@amd.com> Signed-off-by: JoseSantosAMD <87447437+JoseSantosAMD@users.noreply.github.com> Co-authored-by: Karl W Schulz <karl.schulz@amd.com> Co-authored-by: colramos-amd <colramos@amd.com> Co-authored-by: Karl W. Schulz <koomie@users.noreply.github.com>
This commit is contained in:
JoseSantosAMD
GitHub
+16
-7
@@ -119,8 +119,12 @@ def show_all(args, runs, archConfigs, output):
|
||||
],
|
||||
axis=1,
|
||||
)
|
||||
diff = t_df.iloc[:, 1] - t_df.iloc[:, 0]
|
||||
t_df = diff / t_df.iloc[:, 0].replace(0, 1)
|
||||
absolute_diff = (
|
||||
t_df.iloc[:, 1] - t_df.iloc[:, 0]
|
||||
).round(args.decimal)
|
||||
t_df = absolute_diff / t_df.iloc[:, 0].replace(
|
||||
0, 1
|
||||
)
|
||||
if args.verbose >= 2:
|
||||
print("---------", header, t_df)
|
||||
|
||||
@@ -131,6 +135,7 @@ def show_all(args, runs, archConfigs, output):
|
||||
)
|
||||
# show value + percentage
|
||||
# TODO: better alignment
|
||||
|
||||
t_df = (
|
||||
cur_df[header]
|
||||
.astype(float)
|
||||
@@ -141,24 +146,28 @@ def show_all(args, runs, archConfigs, output):
|
||||
+ "%)"
|
||||
)
|
||||
df = pd.concat([df, t_df], axis=1)
|
||||
|
||||
df["Abs Diff"] = absolute_diff
|
||||
|
||||
# DEBUG: When in a CI setting and flag is set,
|
||||
# then verify metrics meet threshold requirement
|
||||
if args.report_diff:
|
||||
if (
|
||||
t_df_pretty.abs()
|
||||
header in ["Value", "Count", "Avg"]
|
||||
and t_df_pretty.abs()
|
||||
.gt(args.report_diff)
|
||||
.any()
|
||||
):
|
||||
violation_idx = t_df_pretty.index[t_df_pretty.abs() > args.report_diff]
|
||||
violation_idx = t_df_pretty.index[
|
||||
t_df_pretty.abs() > args.report_diff
|
||||
]
|
||||
print(
|
||||
"DEBUG ERROR: Dataframe diff exceeds {} threshold requirement\nSee metric {}".format(
|
||||
str(args.report_diff) + "%",
|
||||
violation_idx.to_numpy()
|
||||
violation_idx.to_numpy(),
|
||||
)
|
||||
)
|
||||
print(df)
|
||||
sys.exit(1)
|
||||
|
||||
else:
|
||||
cur_df_copy = copy.deepcopy(cur_df)
|
||||
cur_df_copy[header] = [
|
||||
|
||||
Reference in New Issue
Block a user