[rocprofiler-compute] Improve iteration multiplexing code and documentation (#2080)
* Improve Iteration multiplexing
* Improve iteration multiplexing documentation by adding usage note and
listing caveats
* Bugfixes for iteration mulitplexing
* Use merge iteration multiplexing in analysis webui and db mode
* Do not remove Dispatch_ID column in merge iteration multiplexing
since it is needed for analysis of top dispatches based on
duration
* Bugfixes for analysis logic
* Graceful handling of missing counters in case of iteration
multiplexing
* Improved warnings when metrics could not be calculated due to
missing counter data
* Fix the check to prevent showing table when a column is full of
N/A
* Improve detection of empty values when metric evaludation fails
due to missing counter data
* Bugfixes for profile logic
* Fix kernel filtering during roofline benchmark phase
* Update changelog for bugfixes
* Remove unnecessary columns when merging dispatches for iteration multiplexing
* bugfix
* Better analysis warnings
* fix to_std() in parser
* Use median in merge iteration multiplex
* Address review comments
* Fix cmake formatting
* fix None handling of parser util functions
* Enable stochastic counter accuracy test
* fix cmake formatting
This commit is contained in:
vedithal-amd
committad av
GitHub
GitHub
förälder
bd6c6852fc
incheckning
e4abee4f7d
@@ -13,11 +13,7 @@ Full documentation for ROCm Compute Profiler is available at [https://rocm.docs.
|
||||
* ``--no-native-tool`` option is provided, forcing usage of the default profiler.
|
||||
* When performing a dynamic attach to a process for profiling.
|
||||
|
||||
* Iteration multiplexing to collect counters in single application run:
|
||||
* Is incompatible with --no-native-tool
|
||||
* Two options:
|
||||
* kernel: Counters are collected in a round robin fashion for unique kernels.
|
||||
* kernel_launch_params: Counters are collected in a round robin fashion for unique kernels having the exact same launch parameters.
|
||||
* Iteration multiplexing to collect counters in single application run
|
||||
|
||||
* Runtime compilation of Roofline benchmarking:
|
||||
* GPU kernels from [rocm-amdgpu-bench](https://github.com/ROCm/rocm-amdgpu-bench) repository are moved into the ROCm Compute Profiler and are compiled at runtime using local HIP and HIPRTC Python wrappers.
|
||||
@@ -44,6 +40,11 @@ Full documentation for ROCm Compute Profiler is available at [https://rocm.docs.
|
||||
* Fixed issue where detected max memory clock from amd-smi interface was using max gfx clock
|
||||
* Fixed issue where values detected from amd-smi were wrong when some GPU devices were hidden using ROCR or HIP environment variables
|
||||
|
||||
* Analysis mode bugfixes
|
||||
* Improved warnings when metrics could not be calculated due to missing counter data
|
||||
* Fix the check to prevent showing table where a column is full of N/A
|
||||
* Improve detection of empty values when metric evalulation fails due to counter data missing
|
||||
|
||||
### Removed
|
||||
|
||||
* Removed "VL1 Lat" metric for AMD Instinct MI300 series GPUs, due to MI300 series not supporting TCP_TCP_LATENCY_sum counter.
|
||||
|
||||
@@ -157,7 +157,11 @@ endif()
|
||||
# For build-time checks, use ${Python3_EXECUTABLE} (absolute path from find_package)
|
||||
# For runtime tests (add_test), use portable command that resolves from PATH
|
||||
# This allows tests to work when build/install folders are moved to different machines
|
||||
set(PYTHON_TEST_COMMAND "python3" CACHE STRING "Python command for running tests (portable)")
|
||||
set(PYTHON_TEST_COMMAND
|
||||
"python3"
|
||||
CACHE STRING
|
||||
"Python command for running tests (portable)"
|
||||
)
|
||||
message(STATUS "Python test command: ${PYTHON_TEST_COMMAND}")
|
||||
|
||||
# ----------------------
|
||||
@@ -376,6 +380,15 @@ add_test(
|
||||
tests/test_profile_general.py ${WORKING_DIR_OPTION}
|
||||
)
|
||||
|
||||
add_test(
|
||||
NAME test_profile_iteration_multiplexing_stochastic
|
||||
COMMAND
|
||||
${PYTHON_TEST_COMMAND} -m pytest -s -m
|
||||
test_profile_iteration_multiplexing_stochastic
|
||||
--junitxml=tests/test_profile_iteration_multiplexing_stochastic.xml ${COV_OPTION}
|
||||
tests/test_profile_general.py ${WORKING_DIR_OPTION}
|
||||
)
|
||||
|
||||
set_tests_properties(
|
||||
test_profile_kernel_execution
|
||||
test_profile_dispatch
|
||||
@@ -392,6 +405,7 @@ set_tests_properties(
|
||||
test_profile_live_attach_detach
|
||||
test_profile_iteration_multiplexing_1
|
||||
test_profile_iteration_multiplexing_2
|
||||
test_profile_iteration_multiplexing_stochastic
|
||||
PROPERTIES LABELS "profile" RESOURCE_GROUPS gpus:1 TIMEOUT 1800
|
||||
)
|
||||
|
||||
|
||||
@@ -668,9 +668,17 @@ Iteration Multiplexing
|
||||
To reduce profiling overhead when collecting a large number of performance counters,
|
||||
ROCm Compute Profiler supports iteration multiplexing. This technique divides the
|
||||
total set of requested performance counters into smaller subsets that can be collected
|
||||
over multiple iterations of the kernel execution. Each iteration collects a different
|
||||
subset of counters, and the results are later combined to provide a comprehensive view
|
||||
of the performance metrics.
|
||||
over multiple iterations of the kernel execution, thereby preventing the need for
|
||||
application replay. Each iteration collects a different subset of counters, and the
|
||||
results are later combined to provide a comprehensive view of the performance metrics.
|
||||
|
||||
.. note::
|
||||
|
||||
Iteration multiplexing is most beneficial for large workloads that take a long time to run,
|
||||
as it helps reduce profiling overhead by eliminating the need for application replay while
|
||||
spreading counter collection across iterations. For small workloads with few kernel dispatches,
|
||||
iteration multiplexing may result in incomplete metric calculations due to insufficient kernel
|
||||
dispatch counts to cover all counter subsets.
|
||||
|
||||
Usage
|
||||
-----
|
||||
@@ -691,21 +699,22 @@ By default, if no policy is specified, ROCm Compute Profiler uses the ``kernel_l
|
||||
|
||||
.. note::
|
||||
|
||||
* Do not use ``--no-native-tool`` with ``--iteration-multiplexing``.
|
||||
Iteration multiplexing is only supported when using ROCm Compute Profiler with
|
||||
the native counter collection tool. Ensure that ``--no-native-tool`` is not used in your profiling command.
|
||||
* Do not use ``--no-native-tool`` with ``--iteration-multiplexing``.
|
||||
Iteration multiplexing is only supported when using ROCm Compute Profiler with
|
||||
the native counter collection tool. Ensure that ``--no-native-tool`` is not used in your profiling command.
|
||||
|
||||
* Ensure that your workload runs for enough iterations to cover all counter subsets.
|
||||
When using iteration multiplexing, the total number of iterations, for each kernel (for ``kernel`` policy)
|
||||
or for each unique kernel and launch parameters combination (for ``kernel_launch_params`` policy),
|
||||
specified in the workload should be sufficient to cover all subsets of counters. If the number of iterations
|
||||
is too low, some counters may not be collected.
|
||||
* Ensure that your workload runs for enough iterations to cover all counter subsets.
|
||||
When using iteration multiplexing, the total number of iterations, for each kernel (for ``kernel`` policy)
|
||||
or for each unique kernel and launch parameters combination (for ``kernel_launch_params`` policy),
|
||||
specified in the workload should be sufficient to cover all subsets of counters. If the number of iterations
|
||||
is too low, some counters may not be collected.
|
||||
|
||||
* Launch paramaters for ``kernel_launch_params`` policy.
|
||||
Launch parameters refer to the following paramaters.
|
||||
- Grid size
|
||||
- Workgroup size
|
||||
- LDS size
|
||||
* Launch paramaters for ``kernel_launch_params`` policy.
|
||||
Launch parameters refer to the following paramaters:
|
||||
|
||||
- Grid size
|
||||
- Workgroup size
|
||||
- LDS size
|
||||
|
||||
The following example demonstrates how to use iteration multiplexing with the
|
||||
``vcopy`` workload:
|
||||
@@ -759,3 +768,25 @@ The following example demonstrates how to use iteration multiplexing with the
|
||||
[INFO] |-> [rocprofiler-sdk] vcopy testing on GCD 0
|
||||
[INFO] |-> [rocprofiler-sdk] Finished allocating vectors on the CPU
|
||||
...
|
||||
|
||||
|
||||
Caveats
|
||||
------
|
||||
|
||||
Iteration multiplexing feature comes with some caveats to be considered when profiling any workload:
|
||||
|
||||
* **Accuracy vs speed trade-off**
|
||||
|
||||
Iteration multiplexing provides a trade-off with decreased profiling time by eliminating application replay while sacrificing accuracy since only a handful of counters can be collected per kernel dispatch; while we test for closeness in metric values with and without iteration multiplexing in our automatic test suite, more accurate results can be obtained by not using iteration multiplexing.
|
||||
|
||||
* **Minimum number of kernel dispatches required**
|
||||
|
||||
When using iteration multiplexing it is recommended to filter by kernel(s) of interest and make sure these kernels are dispatched enough times (50 recommended) to cover all counter subsets (currently around 15); a warning is thrown for kernels with insufficient dispatch counts to warn the user about missing counter data for those kernels, and it is not possible to calculate some metrics for these kernels.
|
||||
|
||||
* **Non-deterministic workloads**
|
||||
|
||||
Workloads which dispatch kernels with non-deterministic names and launch parameters may trigger warnings for insufficient dispatch counts because iteration multiplexing identifies unique kernels by their names and optionally by their launch parameters; this is especially true of large AI workloads that dispatch kernels non-deterministically based on the model layers being used for the current input, and in such cases kernel filtering of common kernels is recommended.
|
||||
|
||||
* **Cannot use with dispatch filtering**
|
||||
|
||||
It is not possible to use dispatch filtering mentioned in :ref:`Filtering <filtering>` with iteration multiplexing, because iteration multiplexing merges counters across dispatches, making it impossible to isolate specific dispatches for profiling and analysis, so attempting to combine them will result in an error.
|
||||
@@ -60,6 +60,7 @@ from utils.roofline_calc import (
|
||||
SUPPORTED_DATATYPES,
|
||||
)
|
||||
from utils.utils import get_uuid, get_version
|
||||
import numpy as np
|
||||
|
||||
|
||||
class db_analysis(OmniAnalyze_Base):
|
||||
@@ -77,22 +78,7 @@ class db_analysis(OmniAnalyze_Base):
|
||||
)
|
||||
self._roofline_ceilings_per_workload = self.calc_roofline_ceilings()
|
||||
self._pc_sampling_data_per_workload = self.calc_pc_sampling_data()
|
||||
self._pmc_df_per_workload = {
|
||||
workload_path: rocpd_data.process_rocpd_csv(
|
||||
pd.read_csv(Path(workload_path) / "pmc_perf.csv")
|
||||
)
|
||||
for workload_path in self._runs.keys()
|
||||
}
|
||||
self._top_kernels_per_workload = {
|
||||
workload_path: pmc_df.assign(
|
||||
duration=pmc_df["End_Timestamp"] - pmc_df["Start_Timestamp"]
|
||||
)
|
||||
.sort_values(by="duration", ascending=False)
|
||||
.drop_duplicates("Kernel_Name")["Kernel_Name"]
|
||||
.to_list()
|
||||
for workload_path, pmc_df in self._pmc_df_per_workload.items()
|
||||
}
|
||||
console_debug("Collected dispatch data")
|
||||
self._pmc_df_per_workload = self.calc_pmc_df_data()
|
||||
self._pmc_df_per_workload = self.apply_pmc_filters()
|
||||
self._dispatch_data_per_workload = self.calc_dispatch_data()
|
||||
self._metrics_info_data_per_workload, self._values_data_per_workload = (
|
||||
@@ -233,6 +219,34 @@ class db_analysis(OmniAnalyze_Base):
|
||||
console_debug("Completed writing database")
|
||||
console_warning(f"Created file: {db_name}")
|
||||
|
||||
def calc_pmc_df_data(self) -> dict[str, pd.DataFrame]:
|
||||
pmc_df_per_workload: dict[str, pd.DataFrame] = {}
|
||||
args = self.get_args()
|
||||
|
||||
for workload_path in self._runs.keys():
|
||||
pmc_df = rocpd_data.process_rocpd_csv(
|
||||
pd.read_csv(Path(workload_path) / "pmc_perf.csv")
|
||||
)
|
||||
|
||||
# Create multi index df with collection level as pmc_perf
|
||||
raw_pmc = pd.concat([pmc_df], keys=["pmc_perf"], axis=1, copy=False)
|
||||
|
||||
if args.spatial_multiplexing:
|
||||
raw_pmc = self.spatial_multiplex_merge_counters(
|
||||
raw_pmc
|
||||
)
|
||||
|
||||
if self._profiling_config.get("iteration_multiplexing") is not None:
|
||||
raw_pmc = self.iteration_multiplex_merge_counters(
|
||||
raw_pmc,
|
||||
policy=self._profiling_config["iteration_multiplexing"],
|
||||
)
|
||||
|
||||
pmc_df_per_workload[workload_path] = raw_pmc["pmc_perf"]
|
||||
|
||||
console_debug("Collected dispatch data")
|
||||
return pmc_df_per_workload
|
||||
|
||||
def calc_roofline_ceilings(self) -> dict[str, dict[str, Any]]:
|
||||
roofline_ceilings_per_workload: dict[str, dict[str, Any]] = {}
|
||||
|
||||
@@ -404,7 +418,7 @@ class db_analysis(OmniAnalyze_Base):
|
||||
value,
|
||||
)
|
||||
try:
|
||||
return eval(
|
||||
eval_result = eval(
|
||||
compile(value, "<string>", "eval"),
|
||||
{}, # no globals
|
||||
{
|
||||
@@ -424,6 +438,28 @@ class db_analysis(OmniAnalyze_Base):
|
||||
"to_sum": to_sum,
|
||||
},
|
||||
)
|
||||
|
||||
# eval_result can be None if expression has None explicitly specified
|
||||
# Do not give warning for this case and simply return None
|
||||
if eval_result is None or "None" in value:
|
||||
return None
|
||||
|
||||
# Only return None for scalar NA values
|
||||
# For vectors/Series, return as-is to preserve shape for downstream operations
|
||||
# Note: pd.NA is not detected as scalar by np.isscalar()
|
||||
is_scalar_na = (
|
||||
eval_result is pd.NA
|
||||
or (np.isscalar(eval_result) and pd.isna(eval_result))
|
||||
)
|
||||
|
||||
if is_scalar_na:
|
||||
console_warning(
|
||||
f"Could not evaluate expression for {name}: {value} - likely due to missing "
|
||||
"counter data."
|
||||
)
|
||||
return None
|
||||
else:
|
||||
return eval_result
|
||||
except Exception as e:
|
||||
console_warning(f"Failed to evaluate expression for {name}: {value} - {e}")
|
||||
return None
|
||||
@@ -587,6 +623,14 @@ class db_analysis(OmniAnalyze_Base):
|
||||
pmc_df_per_workload = self._pmc_df_per_workload.copy()
|
||||
|
||||
for workload_path, pmc_df in pmc_df_per_workload.items():
|
||||
top_kernels = (
|
||||
pmc_df.assign(
|
||||
duration=pmc_df["End_Timestamp"] - pmc_df["Start_Timestamp"]
|
||||
)
|
||||
.sort_values(by="duration", ascending=False)
|
||||
.drop_duplicates("Kernel_Name")["Kernel_Name"]
|
||||
.to_list()
|
||||
)
|
||||
# Filter gpu_ids
|
||||
if self._runs[workload_path].filter_gpu_ids:
|
||||
pmc_df = pmc_df.loc[
|
||||
@@ -598,7 +642,7 @@ class db_analysis(OmniAnalyze_Base):
|
||||
if self._runs[workload_path].filter_kernel_ids:
|
||||
pmc_df = pmc_df.loc[
|
||||
pmc_df["Kernel_Name"].isin([
|
||||
self._top_kernels_per_workload[workload_path][id]
|
||||
top_kernels[id]
|
||||
for id in self._runs[workload_path].filter_kernel_ids
|
||||
])
|
||||
]
|
||||
@@ -639,6 +683,14 @@ class db_analysis(OmniAnalyze_Base):
|
||||
"l2_cache_data": roofline_data_expressions.get("AI L2", ""),
|
||||
"hbm_cache_data": roofline_data_expressions.get("AI HBM", ""),
|
||||
}
|
||||
top_kernels = (
|
||||
pmc_df.assign(
|
||||
duration=pmc_df["End_Timestamp"] - pmc_df["Start_Timestamp"]
|
||||
)
|
||||
.sort_values(by="duration", ascending=False)
|
||||
.drop_duplicates("Kernel_Name")["Kernel_Name"]
|
||||
.to_list()
|
||||
)
|
||||
|
||||
roofline_df = pd.DataFrame([
|
||||
{
|
||||
@@ -653,7 +705,7 @@ class db_analysis(OmniAnalyze_Base):
|
||||
for metric_name in roofline_data_expressions
|
||||
},
|
||||
}
|
||||
for kernel_name in self._top_kernels_per_workload[workload_path][
|
||||
for kernel_name in top_kernels[
|
||||
: self.get_args().max_stat_num
|
||||
]
|
||||
])
|
||||
|
||||
@@ -354,6 +354,12 @@ class webui_analysis(OmniAnalyze_Base):
|
||||
self._runs[self.dest_dir].raw_pmc
|
||||
)
|
||||
|
||||
if self._profiling_config.get("iteration_multiplexing") is not None:
|
||||
self._runs[self.dest_dir].raw_pmc = self.iteration_multiplex_merge_counters(
|
||||
self._runs[self.dest_dir].raw_pmc,
|
||||
policy=self._profiling_config["iteration_multiplexing"],
|
||||
)
|
||||
|
||||
file_io.create_df_kernel_top_stats(
|
||||
df_in=self._runs[self.dest_dir].raw_pmc,
|
||||
raw_data_dir=self.dest_dir,
|
||||
|
||||
@@ -184,7 +184,8 @@ class Roofline:
|
||||
df_list = df_pmc["Kernel_Name"].tolist()
|
||||
|
||||
for idx in range(len(df_list)):
|
||||
if df_list[idx].split("(")[0] not in args.kernel:
|
||||
# If there is no any kernel match, drop the row
|
||||
if not any([kernel in df_list[idx] for kernel in args.kernel]):
|
||||
df_filtered.drop(index=idx, inplace=True)
|
||||
|
||||
# Verify that final filtered kernel df matches the kernel list requested
|
||||
|
||||
@@ -118,7 +118,7 @@ PC_SAMPLING_NOT_ISSUE_PREFIX = "ROCPROFILER_PC_SAMPLING_INSTRUCTION_NOT_ISSUED_R
|
||||
# ------------------------------------------------------------------------------
|
||||
|
||||
|
||||
def to_min(*args: Any) -> Union[float, None]:
|
||||
def to_min(*args: Any) -> float:
|
||||
if len(args) == 1 and isinstance(args[0], pd.Series):
|
||||
return args[0].min()
|
||||
elif min(args) is None:
|
||||
@@ -127,7 +127,7 @@ def to_min(*args: Any) -> Union[float, None]:
|
||||
return min(args)
|
||||
|
||||
|
||||
def to_max(*args: Any) -> Union[float, np.ndarray, None]:
|
||||
def to_max(*args: Any) -> Union[float, np.ndarray]:
|
||||
if len(args) == 1 and isinstance(args[0], pd.Series):
|
||||
return args[0].max()
|
||||
elif len(args) == 2 and (
|
||||
@@ -142,9 +142,11 @@ def to_max(*args: Any) -> Union[float, np.ndarray, None]:
|
||||
|
||||
def to_avg(
|
||||
a: Union[pd.Series, np.ndarray, list, int, float, str, np.number, None],
|
||||
) -> Union[float, np.floating, None]:
|
||||
) -> Union[float, np.floating]:
|
||||
if a is None:
|
||||
return np.nan
|
||||
if np.isscalar(a) and pd.isna(a):
|
||||
return np.nan
|
||||
elif isinstance(a, pd.Series):
|
||||
if a.empty:
|
||||
return np.nan
|
||||
@@ -173,9 +175,9 @@ def to_avg(
|
||||
raise Exception(f"to_avg: unsupported type: {type(a)}")
|
||||
|
||||
|
||||
def to_median(a: Union[pd.Series, None]) -> Union[float, None]:
|
||||
def to_median(a: Union[pd.Series, None]) -> float:
|
||||
if a is None:
|
||||
return None
|
||||
return np.nan
|
||||
elif isinstance(a, pd.Series):
|
||||
with warnings.catch_warnings():
|
||||
warnings.simplefilter("ignore", category=RuntimeWarning)
|
||||
@@ -186,6 +188,9 @@ def to_median(a: Union[pd.Series, None]) -> Union[float, None]:
|
||||
|
||||
def to_std(a: pd.Series) -> float:
|
||||
if isinstance(a, pd.Series):
|
||||
# Define std as 0.0 if there is only one element
|
||||
if len(a) <= 1:
|
||||
return 0.0
|
||||
return a.std()
|
||||
else:
|
||||
raise Exception("to_std: unsupported type.")
|
||||
@@ -193,20 +198,23 @@ def to_std(a: pd.Series) -> float:
|
||||
|
||||
def to_int(
|
||||
a: Union[int, float, str, np.integer, pd.Series, None],
|
||||
) -> Union[int, pd.Series, None]:
|
||||
) -> Union[int, float, pd.Series]:
|
||||
if a is None:
|
||||
return None
|
||||
return np.nan
|
||||
if np.isscalar(a) and pd.isna(a):
|
||||
return np.nan
|
||||
elif isinstance(a, (int, float, np.integer)):
|
||||
return int(a)
|
||||
elif isinstance(a, pd.Series):
|
||||
return a.astype(int)
|
||||
# "Int64" handles null values
|
||||
return a.astype("Int64")
|
||||
elif isinstance(a, str):
|
||||
return int(a)
|
||||
else:
|
||||
raise Exception("to_int: unsupported type.")
|
||||
|
||||
|
||||
def to_sum(a: Union[pd.Series, None]) -> Union[float, None]:
|
||||
def to_sum(a: Union[pd.Series, None]) -> float:
|
||||
if a is None:
|
||||
return np.nan
|
||||
elif np.isnan(a).all():
|
||||
@@ -226,9 +234,9 @@ def to_round(a: Union[pd.Series, float], b: int) -> Union[pd.Series, float]:
|
||||
return round(a, b)
|
||||
|
||||
|
||||
def to_quantile(a: Union[pd.Series, None], b: float) -> Union[float, None]:
|
||||
def to_quantile(a: Union[pd.Series, None], b: float) -> float:
|
||||
if a is None:
|
||||
return None
|
||||
return np.nan
|
||||
elif isinstance(a, pd.Series):
|
||||
return a.quantile(b)
|
||||
else:
|
||||
@@ -346,7 +354,26 @@ class MetricEvaluator:
|
||||
local_expr_context,
|
||||
)
|
||||
|
||||
if eval_result is None or np.isnan(eval_result).any():
|
||||
# Only return "N/A" for scalar NA values
|
||||
# For vectors/Series, return as-is to preserve shape for
|
||||
# downstream operations
|
||||
# Note: None and pd.NA are not detected as scalar by np.isscalar()
|
||||
if (
|
||||
eval_result is None
|
||||
or eval_result is pd.NA
|
||||
or (np.isscalar(eval_result) and pd.isna(eval_result))
|
||||
):
|
||||
# Do not give warning if None is explicitly specified in expression
|
||||
if "None" not in expr:
|
||||
console_warning(
|
||||
f"Could not evaluate expression '{expr}' - likely "
|
||||
"due to missing counter data."
|
||||
)
|
||||
else:
|
||||
console_debug(
|
||||
f"Expression '{expr}' evaluated to None - likely "
|
||||
"explicitly specified."
|
||||
)
|
||||
return "N/A"
|
||||
else:
|
||||
return eval_result
|
||||
@@ -360,18 +387,18 @@ class MetricEvaluator:
|
||||
return "N/A"
|
||||
|
||||
except AttributeError as attribute_error:
|
||||
if str(attribute_error) == "'NoneType' object has no attribute 'get'":
|
||||
console_warning(
|
||||
f"Failed to evaluate expression '{expr}': {attribute_error}."
|
||||
)
|
||||
return "N/A"
|
||||
else:
|
||||
console_error("analysis", str(attribute_error))
|
||||
return "N/A"
|
||||
console_warning(
|
||||
f"Failed to evaluate expression '{expr}': {attribute_error}."
|
||||
)
|
||||
return "N/A"
|
||||
|
||||
except pd.errors.IntCastingNaNError as exception:
|
||||
console_warning(f"Missing data: {exception}. Using empty value.")
|
||||
return ""
|
||||
console_warning(f"Failed to evaluate expression '{expr}': {exception}.")
|
||||
return "N/A"
|
||||
|
||||
except ValueError as value_error:
|
||||
console_warning(f"Failed to evaluate expression '{expr}': {value_error}.")
|
||||
return "N/A"
|
||||
|
||||
|
||||
def build_eval_string(equation: str, coll_level: str, config: dict) -> str:
|
||||
@@ -929,9 +956,12 @@ def calc_builtin_vars(
|
||||
# Pass sys_vars so that $num_xcd and other system variables are available
|
||||
temporary_evaluator = MetricEvaluator(raw_pmc_df, sys_vars, {})
|
||||
calculation_result = temporary_evaluator.eval_expression(eval_string)
|
||||
# Convert "N/A" string to np.nan to maintain numeric type for calculations
|
||||
if np.isscalar(calculation_result) and calculation_result == "N/A":
|
||||
calculation_result = np.nan
|
||||
builtin_vars_collection[f"ammolite__{variable_key}"] = calculation_result
|
||||
except (TypeError, NameError, KeyError, AttributeError):
|
||||
builtin_vars_collection[f"ammolite__{variable_key}"] = None
|
||||
builtin_vars_collection[f"ammolite__{variable_key}"] = np.nan
|
||||
|
||||
# Second pass: calculate remaining variables that depend on per-XCD values
|
||||
for variable_key, variable_value in BUILD_IN_VARS.items():
|
||||
@@ -946,9 +976,12 @@ def calc_builtin_vars(
|
||||
combined_vars = {**sys_vars, **builtin_vars_collection}
|
||||
temporary_evaluator = MetricEvaluator(raw_pmc_df, combined_vars, {})
|
||||
calculation_result = temporary_evaluator.eval_expression(eval_string)
|
||||
# Convert "N/A" string to np.nan to maintain numeric type for calculations
|
||||
if np.isscalar(calculation_result) and calculation_result == "N/A":
|
||||
calculation_result = np.nan
|
||||
builtin_vars_collection[f"ammolite__{variable_key}"] = calculation_result
|
||||
except (TypeError, NameError, KeyError, AttributeError):
|
||||
builtin_vars_collection[f"ammolite__{variable_key}"] = None
|
||||
builtin_vars_collection[f"ammolite__{variable_key}"] = np.nan
|
||||
|
||||
return builtin_vars_collection
|
||||
|
||||
@@ -1653,9 +1686,7 @@ def load_pc_sampling_data(
|
||||
csv_kernel_trace_file_path = Path(dir_path) / f"{file_prefix}_kernel_trace.csv"
|
||||
|
||||
if not csv_kernel_trace_file_path.exists():
|
||||
console_error(
|
||||
f"PC sampling: can not read {csv_kernel_trace_file_path}", exit=False
|
||||
)
|
||||
console_warning(f"PC sampling: can not read {csv_kernel_trace_file_path}")
|
||||
return pd.DataFrame()
|
||||
|
||||
if stochastic_path.exists():
|
||||
@@ -1722,7 +1753,7 @@ def load_pc_sampling_data(
|
||||
|
||||
elif len(workload.filter_kernel_ids) == 1:
|
||||
if not json_file_path.exists():
|
||||
console_error(f"PC sampling: can not read {json_file_path}", exit=False)
|
||||
console_warning(f"PC sampling: can not read {json_file_path}")
|
||||
return pd.DataFrame()
|
||||
else:
|
||||
# NB:
|
||||
|
||||
@@ -382,7 +382,7 @@ def format_table_output(
|
||||
|
||||
# Check if any column in df is empty
|
||||
is_empty_columns_exist = any(
|
||||
df.replace("", None).iloc[:, col_idx].isnull().all()
|
||||
df.replace(["", "N/A"], None).iloc[:, col_idx].isnull().all()
|
||||
for col_idx in range(len(df.columns))
|
||||
)
|
||||
|
||||
|
||||
@@ -1384,10 +1384,6 @@ def merge_counters_iteration_multiplex(
|
||||
"Kernel_ID",
|
||||
]
|
||||
|
||||
expired_column_index = [
|
||||
"Dispatch_ID",
|
||||
]
|
||||
|
||||
result_dfs: list[pd.DataFrame] = []
|
||||
|
||||
# TODO: will need to optimize to avoid this conversion to single index format
|
||||
@@ -1419,30 +1415,32 @@ def merge_counters_iteration_multiplex(
|
||||
|
||||
pd.set_option("display.max_columns", None)
|
||||
|
||||
# Reset Dispatch_ID
|
||||
dispatch_id_counter = 0
|
||||
|
||||
for name, group in unique_occurences:
|
||||
# Create a dictionary to store the merged row for the current group
|
||||
merged_row: dict[str, Any] = {}
|
||||
|
||||
# Process non-counter columns
|
||||
for col in [
|
||||
col
|
||||
for col in non_counter_column_index
|
||||
if col not in expired_column_index
|
||||
]:
|
||||
for col in non_counter_column_index:
|
||||
if col == "End_Timestamp":
|
||||
# For End_Timestamp, calculate the median delta time
|
||||
delta_time = group["End_Timestamp"] - group["Start_Timestamp"]
|
||||
median_delta_time = delta_time.median()
|
||||
merged_row[col] = merged_row["Start_Timestamp"] + median_delta_time
|
||||
merged_row["Median_Time"] = median_delta_time
|
||||
merged_row["Mean_Time"] = delta_time.mean()
|
||||
delta_time = group[col] - group["Start_Timestamp"]
|
||||
merged_row[col] = group["Start_Timestamp"] + delta_time.median()
|
||||
if col == "Dispatch_ID":
|
||||
# Assign new Dispatch_ID
|
||||
merged_row[col] = dispatch_id_counter
|
||||
dispatch_id_counter += 1
|
||||
elif pd.api.types.is_numeric_dtype(group[col]):
|
||||
# For other non-counter numeric columns, take the median value
|
||||
merged_row[col] = group[col].median()
|
||||
if pd.api.types.is_integer_dtype(group[col]):
|
||||
merged_row[col] = merged_row[col].astype(int)
|
||||
else:
|
||||
# For other non-counter columns, take the first occurrence (0th row)
|
||||
# For other non-counter non-numeric columns,
|
||||
# take the first occurrence (0th row)
|
||||
# Only Kernel_Name should be non-numeric here
|
||||
merged_row[col] = group.iloc[0][col]
|
||||
|
||||
# Process counter columns (assumed to be all columns not in
|
||||
@@ -1451,16 +1449,19 @@ def merge_counters_iteration_multiplex(
|
||||
col for col in group.columns if col not in non_counter_column_index
|
||||
]
|
||||
for counter_col in counter_columns:
|
||||
# for counter columns, take the first non-none (or non-nan) value
|
||||
current_valid_counter_group = group[group[counter_col].notna()]
|
||||
first_valid_value = (
|
||||
current_valid_counter_group.iloc[0][counter_col]
|
||||
if len(current_valid_counter_group) > 0
|
||||
else None
|
||||
)
|
||||
merged_row[counter_col] = first_valid_value
|
||||
|
||||
merged_row["Count"] = group["Dispatch_ID"].nunique()
|
||||
# For counter columns, calculate median only across non-NaN values
|
||||
# Preserve original data type
|
||||
valid_values = group[counter_col].dropna()
|
||||
if not valid_values.empty:
|
||||
median_value = valid_values.median()
|
||||
# Preserve original data type - check if all
|
||||
# non-null values are integers
|
||||
if (valid_values == valid_values.astype(int)).all():
|
||||
merged_row[counter_col] = int(median_value)
|
||||
else:
|
||||
merged_row[counter_col] = median_value
|
||||
else:
|
||||
merged_row[counter_col] = None
|
||||
|
||||
# Append the merged row to the result list
|
||||
result_data.append(merged_row)
|
||||
@@ -1543,9 +1544,8 @@ def merge_counters_spatial_multiplex(df_multi_index: pd.DataFrame) -> pd.DataFra
|
||||
merged_row[col] = group["Start_Timestamp"].median()
|
||||
elif col == "End_Timestamp":
|
||||
# For End_Timestamp, calculate the median delta time
|
||||
delta_time = group["End_Timestamp"] - group["Start_Timestamp"]
|
||||
median_delta_time = delta_time.median()
|
||||
merged_row[col] = merged_row["Start_Timestamp"] + median_delta_time
|
||||
delta_time = group[col] - group["Start_Timestamp"]
|
||||
merged_row[col] = group["Start_Timestamp"] + delta_time.median()
|
||||
else:
|
||||
# For other non-counter columns, take the first occurrence (0th row)
|
||||
merged_row[col] = group.iloc[0][col]
|
||||
|
||||
@@ -993,7 +993,7 @@ def test_parser_utility_functions():
|
||||
assert result == 9, "to_max should return maximum value"
|
||||
|
||||
result = to_median(None)
|
||||
assert result is None, "to_median should return None for None input"
|
||||
assert np.isnan(result), "to_median should return np.nan for None input"
|
||||
|
||||
try:
|
||||
to_median("invalid_string")
|
||||
@@ -1008,7 +1008,7 @@ def test_parser_utility_functions():
|
||||
assert "unsupported type" in str(e)
|
||||
|
||||
result = to_int(None)
|
||||
assert result is None, "to_int should return None for None input"
|
||||
assert np.isnan(result), "to_int should return np.nan for None input"
|
||||
|
||||
try:
|
||||
to_int(["list", "not", "supported"])
|
||||
@@ -1017,7 +1017,7 @@ def test_parser_utility_functions():
|
||||
assert "unsupported type" in str(e)
|
||||
|
||||
result = to_quantile(None, 0.5)
|
||||
assert result is None, "to_quantile should return None for None input"
|
||||
assert np.isnan(result), "to_quantile should return np.nan for None input"
|
||||
|
||||
try:
|
||||
to_quantile("invalid_string", 0.5)
|
||||
|
||||
@@ -2621,7 +2621,7 @@ def test_iteration_multiplexing_kernel_launch_params(
|
||||
def test_iteration_multiplexing_deterministic_counter_accuracy(
|
||||
binary_handler_profile_rocprof_compute,
|
||||
):
|
||||
workload_dir = test_utils.get_output_dir()
|
||||
workload_dir = test_utils.get_output_dir(param_id="no_iter_mplx")
|
||||
_ = binary_handler_profile_rocprof_compute(
|
||||
config, workload_dir, check_success=True, roof=False, app_name="app_laplace_eqn"
|
||||
)
|
||||
@@ -2630,7 +2630,7 @@ def test_iteration_multiplexing_deterministic_counter_accuracy(
|
||||
)["pmc_perf.csv"]
|
||||
|
||||
options = ["--iteration-multiplexing", "kernel"]
|
||||
workload_dir = test_utils.get_output_dir()
|
||||
workload_dir = test_utils.get_output_dir(param_id="iter_mplx_kernel")
|
||||
_ = binary_handler_profile_rocprof_compute(
|
||||
config,
|
||||
workload_dir,
|
||||
@@ -2644,7 +2644,7 @@ def test_iteration_multiplexing_deterministic_counter_accuracy(
|
||||
)["pmc_perf.csv"]
|
||||
|
||||
options = ["--iteration-multiplexing", "kernel_launch_params"]
|
||||
workload_dir = test_utils.get_output_dir()
|
||||
workload_dir = test_utils.get_output_dir(param_id="iter_mplx_params")
|
||||
_ = binary_handler_profile_rocprof_compute(
|
||||
config,
|
||||
workload_dir,
|
||||
@@ -2668,7 +2668,7 @@ def test_iteration_multiplexing_deterministic_counter_accuracy(
|
||||
def test_iteration_multiplexing_stochastic_counter_accuracy(
|
||||
binary_handler_profile_rocprof_compute,
|
||||
):
|
||||
workload_dir = test_utils.get_output_dir()
|
||||
workload_dir = test_utils.get_output_dir(param_id="no_mplx")
|
||||
_ = binary_handler_profile_rocprof_compute(
|
||||
config, workload_dir, check_success=True, roof=False, app_name="app_laplace_eqn"
|
||||
)
|
||||
@@ -2677,7 +2677,7 @@ def test_iteration_multiplexing_stochastic_counter_accuracy(
|
||||
)["pmc_perf.csv"]
|
||||
|
||||
options = ["--iteration-multiplexing", "kernel"]
|
||||
workload_dir = test_utils.get_output_dir()
|
||||
workload_dir = test_utils.get_output_dir(param_id="iter_mplx_kernel")
|
||||
_ = binary_handler_profile_rocprof_compute(
|
||||
config,
|
||||
workload_dir,
|
||||
@@ -2691,7 +2691,7 @@ def test_iteration_multiplexing_stochastic_counter_accuracy(
|
||||
)["pmc_perf.csv"]
|
||||
|
||||
options = ["--iteration-multiplexing", "kernel_launch_params"]
|
||||
workload_dir = test_utils.get_output_dir()
|
||||
workload_dir = test_utils.get_output_dir(param_id="iter_mplx_params")
|
||||
_ = binary_handler_profile_rocprof_compute(
|
||||
config,
|
||||
workload_dir,
|
||||
|
||||
Referens i nytt ärende
Block a user