diff --git a/projects/amdsmi/CHANGELOG.md b/projects/amdsmi/CHANGELOG.md index c065447491..226ff81be2 100644 --- a/projects/amdsmi/CHANGELOG.md +++ b/projects/amdsmi/CHANGELOG.md @@ -27,6 +27,9 @@ Added `AMDSMI_EVT_NOTIF_RING_HANG` to the possible events in the `amdsmi_evt_not ### Optimizations +- **Updated naming for `amdsmi_set_gpu_clear_sram_data()` to `amdsmi_set_gpu_run_cleaner_shader()`**. +Changed the naming to be more accurate to what the function was doing. This change also extends to the CLI where we changed the `clear-sram-data` command to `run-shader` that accepts a shader name to run. + - **Updated `amdsmi_clk_info_t` struct in amdsmi.h and amdsmi_interface.py to align with host/guest**. Changed cur_clk to clk, changed sleep_clk to clk_deep_sleep, and added clk_locked value. New struct will be in the following format: @@ -66,7 +69,7 @@ The file structure referenced to dpm_policy changed to soc_pstate and we have ch Previously on devices without a FRU we would not populate the product name in the `amdsmi_board_info_t` structure, now we will fallback to using the name listed according to the pciids file if available. - **Updated CLI voltage curve command output**. -The output for `amd-smi metric --voltage-curve` now splits the frequency and voltage output by curve point or outputs N/A if not applicable +The output for `amd-smi metric --voltage-curve` now splits the frequency and voltage output by curve point or outputs N/A for each curve point if not applicable ```shell GPU: 0 @@ -85,6 +88,43 @@ ASIC products. This requires users to update any ABIs using this structure. ### Fixes +- **Fixed multiple processes not being registered in `amd-smi process` with json and csv format**. +Multiple process outputs in the CLI tool were not being registered correctly. The json output did not handle multiple processes and is now in a new valid json format: + +```shell +[ + { + "gpu": 0, + "process_list": [ + { + "process_info": { + "name": "TransferBench", + "pid": 420157, + "mem_usage": { + "value": 0, + "unit": "B" + } + } + }, + { + "process_info": { + "name": "rvs", + "pid": 420315, + "mem_usage": { + "value": 0, + "unit": "B" + } + } + } + ] + } +] +``` + + +- **Removed `throttle-status` from `amd-smi monitor` as it is no longer reliably supported**. +Throttle status may work for older ASICs, but will be replaced with PVIOL and TVIOL metrics for future ASIC support. It remains a field in the gpu_metrics API and in `amd-smi metric --power`. + - **`amdsmi_get_gpu_board_info()` no longer returns junk char strings**. Previously if there was a partial failure to retrieve character strings, we would return garbage output to users using the API. This fix intends to populate as many values as possible. @@ -96,7 +136,7 @@ The parsing of `pp_od_clk_voltage` was not dynamic enough to work with the dropp ### Known Issues -- N/A +- **Process isolation and run shader commands do no currently work and will be supported in a future release**. ## amd_smi_lib for ROCm 6.1.2 @@ -106,7 +146,7 @@ The parsing of `pp_od_clk_voltage` was not dynamic enough to work with the dropp Added APIs CLI and APIs to address LeftoverLocals security issues. Allowing clearing the sram data and setting process isolation on a per GPU basis. New APIs: - `amdsmi_get_gpu_process_isolation()` - `amdsmi_set_gpu_process_isolation()` - - `amdsmi_set_gpu_run_cleaner_shader()` + - `amdsmi_set_gpu_clear_sram_data()` - **Added `MIN_POWER` to output of `amd-smi static --limit`**. This change helps users identify the range to which they can change the power cap of the GPU. The change is added to simplify why a device supports (or does not support) power capping (also known as overdrive). See `amd-smi set -g all --power-cap ` or `amd-smi reset -g all --power-cap`. diff --git a/projects/amdsmi/amdsmi_cli/README.md b/projects/amdsmi/amdsmi_cli/README.md index c4fc361ec0..d87ec235b6 100644 --- a/projects/amdsmi/amdsmi_cli/README.md +++ b/projects/amdsmi/amdsmi_cli/README.md @@ -594,7 +594,7 @@ Command Modifiers: usage: amd-smi monitor [-h] [--json | --csv] [--file FILE] [--loglevel LEVEL] [-g GPU [GPU ...] | -U CPU [CPU ...] | -O CORE [CORE ...]] [-w INTERVAL] [-W TIME] [-i ITERATIONS] [-p] [-t] [-u] [-m] [-n] - [-d] [-s] [-e] [-v] [-r] + [-d] [-e] [-v] [-r] Monitor a target device for the specified arguments. If no arguments are provided, all arguments will be enabled. @@ -626,7 +626,6 @@ Monitor Arguments: -m, --mem Monitor memory utilization (%) and clock (MHz) -n, --encoder Monitor encoder utilization (%) and clock (MHz) -d, --decoder Monitor decoder utilization (%) and clock (MHz) - -s, --throttle-status Monitor thermal throttle status -e, --ecc Monitor ECC single bit, ECC double bit, and PCIe replay error counts -v, --vram-usage Monitor memory usage in MB -r, --pcie Monitor PCIe bandwidth in Mb/s diff --git a/projects/amdsmi/amdsmi_cli/amdsmi_commands.py b/projects/amdsmi/amdsmi_cli/amdsmi_commands.py index 109d7daae4..d063698921 100644 --- a/projects/amdsmi/amdsmi_cli/amdsmi_commands.py +++ b/projects/amdsmi/amdsmi_cli/amdsmi_commands.py @@ -1328,8 +1328,8 @@ class AMDSMICommands(): 'gfx_voltage': "N/A", 'soc_voltage': "N/A", 'mem_voltage': "N/A", - 'power_management': "N/A", - 'throttle_status': "N/A"} + 'throttle_status': "N/A", + 'power_management': "N/A"} try: voltage_unit = "mV" @@ -1792,46 +1792,46 @@ class AMDSMICommands(): values_dict["fan"] = fan_dict if "voltage_curve" in current_platform_args: if args.voltage_curve: + # Populate N/A values per voltage point + voltage_point_dict = {} + for point in range(amdsmi_interface.AMDSMI_NUM_VOLTAGE_CURVE_POINTS): + voltage_point_dict[f'point_{point}_frequency'] = "N/A" + voltage_point_dict[f'point_{point}_voltage'] = "N/A" + try: od_volt = amdsmi_interface.amdsmi_get_gpu_od_volt_info(args.gpu) logging.debug(f"OD Voltage info: {od_volt}") - - # Populate N/A values per voltage point - voltage_point_dict = {} - for point in range(amdsmi_interface.AMDSMI_NUM_VOLTAGE_CURVE_POINTS): - voltage_point_dict[f'point_{point}_frequency'] = "N/A" - voltage_point_dict[f'point_{point}_voltage'] = "N/A" - - # Populate voltage point values - for point in range(amdsmi_interface.AMDSMI_NUM_VOLTAGE_CURVE_POINTS): - if isinstance(od_volt, dict): - logging.debug(f"point_{point} frequency: {od_volt['curve.vc_points'][point].frequency}") - logging.debug(f"point_{point} voltage: {od_volt['curve.vc_points'][point].voltage}") - frequency = int(od_volt["curve.vc_points"][point].frequency / 1000000) - voltage = int(od_volt["curve.vc_points"][point].voltage) - else: - frequency = "N/A" - voltage = "N/A" - - if frequency == 0: - frequency = "N/A" - - if voltage == 0: - voltage = "N/A" - - if frequency != "N/A": - frequency = self.helpers.unit_format(self.logger, frequency, "Mhz") - - if voltage != "N/A": - voltage = self.helpers.unit_format(self.logger, voltage, "mV") - - voltage_point_dict[f'point_{point}_frequency'] = frequency - voltage_point_dict[f'point_{point}_voltage'] = voltage - - values_dict['voltage_curve'] = voltage_point_dict except amdsmi_exception.AmdSmiLibraryException as e: - values_dict['voltage_curve'] = "N/A" + od_volt = "N/A" # Value not used, but needs to not be a dict logging.debug("Failed to get voltage curve for gpu %s | %s", gpu_id, e.get_error_info()) + + # Populate voltage point values + for point in range(amdsmi_interface.AMDSMI_NUM_VOLTAGE_CURVE_POINTS): + if isinstance(od_volt, dict): + logging.debug(f"point_{point} frequency: {od_volt['curve.vc_points'][point].frequency}") + logging.debug(f"point_{point} voltage: {od_volt['curve.vc_points'][point].voltage}") + frequency = int(od_volt["curve.vc_points"][point].frequency / 1000000) + voltage = int(od_volt["curve.vc_points"][point].voltage) + else: + frequency = "N/A" + voltage = "N/A" + + if frequency == 0: + frequency = "N/A" + + if voltage == 0: + voltage = "N/A" + + if frequency != "N/A": + frequency = self.helpers.unit_format(self.logger, frequency, "Mhz") + + if voltage != "N/A": + voltage = self.helpers.unit_format(self.logger, voltage, "mV") + + voltage_point_dict[f'point_{point}_frequency'] = frequency + voltage_point_dict[f'point_{point}_voltage'] = voltage + + values_dict['voltage_curve'] = voltage_point_dict if "overdrive" in current_platform_args: if args.overdrive: try: @@ -2706,47 +2706,47 @@ class AMDSMICommands(): process_names.append(process_info) filtered_process_values = process_names + # If the name or pid args filter processes out then insert an N/A placeholder + if not filtered_process_values: + filtered_process_values.append({'process_info': "N/A"}) + logging.debug(f"Process Info for GPU {gpu_id} | {filtered_process_values}") - multiple_devices_csv_override = False - # Convert and store output by pid for csv format - if self.logger.is_csv_format(): - # Check for empty list first - if not filtered_process_values: - self.logger.store_output(args.gpu, 'process_info', 'No running processes detected') - else: - for process_info in filtered_process_values: - if process_info['process_info'] == "N/A": - self.logger.store_output(args.gpu, 'process_info', 'No running processes detected') - else: - for key, value in process_info['process_info'].items(): - multiple_devices_csv_override = True - if watching_output: - self.logger.store_output(args.gpu, 'timestamp', int(time.time())) - self.logger.store_output(args.gpu, key, value) + for index, process in enumerate(filtered_process_values): + if process['process_info'] == "N/A": + filtered_process_values[index]['process_info'] = "No running processes detected" - self.logger.store_multiple_device_output() - else: + if self.logger.is_json_format(): if watching_output: self.logger.store_output(args.gpu, 'timestamp', int(time.time())) + self.logger.store_output(args.gpu, 'process_list', filtered_process_values) - # Store values in logger.output - if not filtered_process_values: - self.logger.store_output(args.gpu, 'process_info', 'No running processes detected') - else: - for process_info in filtered_process_values: - if process_info['process_info'] == "N/A": - process_info['process_info'] = 'No running processes detected' - self.logger.store_output(args.gpu, 'process_info', process_info['process_info']) + if self.logger.is_human_readable_format(): + if watching_output: + self.logger.store_output(args.gpu, 'timestamp', int(time.time())) + # When we print out process_info we remove the index + # The removal is needed only for human readable process format to align with Host + for index, process in enumerate(filtered_process_values): + self.logger.store_output(args.gpu, f'process_info_{index}', process['process_info']) + + multiple_devices_csv_override = False + if self.logger.is_csv_format(): + multiple_devices_csv_override = True + for process in filtered_process_values: + if watching_output: + self.logger.store_output(args.gpu, 'timestamp', int(time.time())) + self.logger.store_output(args.gpu, 'process_info', process['process_info']) + self.logger.store_multiple_device_output() if multiple_devices: self.logger.store_multiple_device_output() return # Skip printing when there are multiple devices - self.logger.print_output(multiple_device_enabled=multiple_devices_csv_override, watching_output=watching_output) + multiple_devices = multiple_devices or multiple_devices_csv_override + self.logger.print_output(multiple_device_enabled=multiple_devices, watching_output=watching_output) if watching_output: # End of single gpu add to watch_output - self.logger.store_watch_output(multiple_device_enabled=multiple_devices_csv_override) + self.logger.store_watch_output(multiple_device_enabled=multiple_devices) def profile(self, args): @@ -3968,7 +3968,7 @@ class AMDSMICommands(): def monitor(self, args, multiple_devices=False, watching_output=False, gpu=None, watch=None, watch_time=None, iterations=None, power_usage=None, temperature=None, gfx_util=None, mem_util=None, encoder=None, decoder=None, - throttle_status=None, ecc=None, vram_usage=None, pcie=None): + ecc=None, vram_usage=None, pcie=None): """ Populate a table with each GPU as an index to rows of targeted data Args: @@ -3984,7 +3984,6 @@ class AMDSMICommands(): mem (bool, optional): Value override for args.mem. Defaults to None. encoder (bool, optional): Value override for args.encoder. Defaults to None. decoder (bool, optional): Value override for args.decoder. Defaults to None. - throttle_status (bool, optional): Value override for args.throttle_status. Defaults to None. ecc (bool, optional): Value override for args.ecc. Defaults to None. vram_usage (bool, optional): Value override for args.vram_usage. Defaults to None. pcie (bool, optional): Value override for args.pcie. Defaults to None. @@ -4019,8 +4018,6 @@ class AMDSMICommands(): args.encoder = encoder if decoder: args.decoder = decoder - if throttle_status: - args.throttle_status = throttle_status if ecc: args.ecc = ecc if vram_usage: @@ -4034,10 +4031,10 @@ class AMDSMICommands(): # If all arguments are False, the print all values if not any([args.power_usage, args.temperature, args.gfx, args.mem, - args.encoder, args.decoder, args.throttle_status, args.ecc, + args.encoder, args.decoder, args.ecc, args.vram_usage, args.pcie]): args.power_usage = args.temperature = args.gfx = args.mem = \ - args.encoder = args.decoder = args.throttle_status = args.ecc = \ + args.encoder = args.decoder = args.ecc = \ args.vram_usage = args.pcie = True # Handle watch logic, will only enter this block once @@ -4282,20 +4279,6 @@ class AMDSMICommands(): logging.debug("Failed to get decoder clock on gpu %s | %s", gpu_id, e.get_error_info()) self.logger.table_header += 'DEC_CLOCK'.rjust(11) - if args.throttle_status: - try: - throttle_status = amdsmi_interface.amdsmi_get_gpu_metrics_info(args.gpu)['throttle_status'] - if throttle_status != "N/A": - if throttle_status: - throttle_status = "THROTTLED" - else: - throttle_status = "UNTHROTTLED" - monitor_values['throttle_status'] = throttle_status - except amdsmi_exception.AmdSmiLibraryException as e: - monitor_values['throttle_status'] = "N/A" - logging.debug("Failed to get throttle status on gpu %s | %s", gpu_id, e.get_error_info()) - - self.logger.table_header += 'THROTTLE'.rjust(13) if args.ecc: try: ecc = amdsmi_interface.amdsmi_get_gpu_total_ecc_count(args.gpu) diff --git a/projects/amdsmi/amdsmi_cli/amdsmi_helpers.py b/projects/amdsmi/amdsmi_cli/amdsmi_helpers.py index f1bc229e21..7db9be1cb1 100644 --- a/projects/amdsmi/amdsmi_cli/amdsmi_helpers.py +++ b/projects/amdsmi/amdsmi_cli/amdsmi_helpers.py @@ -104,8 +104,8 @@ class AMDSMIHelpers(): if string_format: return f"{operating_system} {operating_system_type}" - else: - return (operating_system, operating_system_type) + + return (operating_system, operating_system_type) def is_virtual_os(self): diff --git a/projects/amdsmi/amdsmi_cli/amdsmi_logger.py b/projects/amdsmi/amdsmi_cli/amdsmi_logger.py index 473235b44c..5b6e35457b 100644 --- a/projects/amdsmi/amdsmi_cli/amdsmi_logger.py +++ b/projects/amdsmi/amdsmi_cli/amdsmi_logger.py @@ -72,9 +72,11 @@ class AMDSMILogger(): def is_human_readable_format(self): return self.format == self.LoggerFormat.human_readable.value + def clear_multiple_devices_ouput(self): self.multiple_device_output.clear() + def _capitalize_keys(self, input_dict): output_dict = {} for key in input_dict.keys(): @@ -116,7 +118,7 @@ class AMDSMILogger(): table_values += value.rjust(11) elif key == 'vram_total' or 'ecc' in key: table_values += value.rjust(12) - elif key in ('throttle_status', 'pcie_replay'): + elif key in ['pcie_replay']: table_values += value.rjust(13) # Only for handling topology tables elif 'gpu_' in key: @@ -163,6 +165,9 @@ class AMDSMILogger(): yaml_output = yaml_output.replace("AMDSMI_SPACING_REMOVAL:\n", "") yaml_output = yaml_output.replace("'", "") # Remove '' + # Remove process_info indicies for Host parity: + yaml_output = re.sub(r'PROCESS_INFO_[0-9]+:', 'PROCESS_INFO:', yaml_output) + clean_yaml_output = '' for line in yaml_output.splitlines(): line = line.split(':') diff --git a/projects/amdsmi/amdsmi_cli/amdsmi_parser.py b/projects/amdsmi/amdsmi_cli/amdsmi_parser.py index edc188b737..278f594df8 100644 --- a/projects/amdsmi/amdsmi_cli/amdsmi_parser.py +++ b/projects/amdsmi/amdsmi_cli/amdsmi_parser.py @@ -148,6 +148,16 @@ class AMDSMIParser(argparse.ArgumentParser): raise amdsmi_cli_exceptions.AmdSmiInvalidParameterValueException(int_value, outputformat) + def _is_valid_string(self, string_value): + # Argument type validator + # This is for triggering a cli exception if an empty string is detected + if string_value: + return string_value + + outputformat = self.helpers.get_output_format() + raise amdsmi_cli_exceptions.AmdSmiInvalidParameterValueException(string_value, outputformat) + + def _check_output_file_path(self): """ Argument action validator: Returns a path to a file from the output file path provided. @@ -863,7 +873,7 @@ class AMDSMIParser(argparse.ArgumentParser): process_parser.add_argument('-G', '--general', action='store_true', required=False, help=general_help) process_parser.add_argument('-e', '--engine', action='store_true', required=False, help=engine_help) process_parser.add_argument('-p', '--pid', action='store', type=self._not_negative_int, required=False, help=pid_help) - process_parser.add_argument('-n', '--name', action='store', required=False, help=name_help) + process_parser.add_argument('-n', '--name', action='store', type=self._is_valid_string, required=False, help=name_help) def _add_profile_parser(self, subparsers, func): @@ -1113,7 +1123,6 @@ class AMDSMIParser(argparse.ArgumentParser): mem_util_help = "Monitor memory utilization (%%) and clock (MHz)" encoder_util_help = "Monitor encoder utilization (%%) and clock (MHz)" decoder_util_help = "Monitor decoder utilization (%%) and clock (MHz)" - throttle_help = "Monitor thermal throttle status" ecc_help = "Monitor ECC single bit, ECC double bit, and PCIe replay error counts" mem_usage_help = "Monitor memory usage in MB" pcie_bandwidth_help = "Monitor PCIe bandwidth in Mb/s" @@ -1136,7 +1145,6 @@ class AMDSMIParser(argparse.ArgumentParser): monitor_parser.add_argument('-m', '--mem', action='store_true', required=False, help=mem_util_help) monitor_parser.add_argument('-n', '--encoder', action='store_true', required=False, help=encoder_util_help) monitor_parser.add_argument('-d', '--decoder', action='store_true', required=False, help=decoder_util_help) - monitor_parser.add_argument('-s', '--throttle-status', action='store_true', required=False, help=throttle_help) monitor_parser.add_argument('-e', '--ecc', action='store_true', required=False, help=ecc_help) monitor_parser.add_argument('-v', '--vram-usage', action='store_true', required=False, help=mem_usage_help) monitor_parser.add_argument('-r', '--pcie', action='store_true', required=False, help=pcie_bandwidth_help) diff --git a/projects/amdsmi/py-interface/README.md b/projects/amdsmi/py-interface/README.md index 1863a4abff..21fb77d338 100644 --- a/projects/amdsmi/py-interface/README.md +++ b/projects/amdsmi/py-interface/README.md @@ -1588,8 +1588,12 @@ try: print("No GPUs on machine") else: for device in devices: - memory = amdsmi_get_gpu_memory_total(device) - print(memory) + vram_memory_total = amdsmi_get_gpu_memory_total(device, amdsmi_interface.AmdSmiMemoryType.VRAM) + print(vram_memory_total) + vis_vram_memory_total = amdsmi_get_gpu_memory_total(device, amdsmi_interface.AmdSmiMemoryType.VIS_VRAM) + print(vis_vram_memory_total) + gtt_memory_total = amdsmi_get_gpu_memory_total(device, amdsmi_interface.AmdSmiMemoryType.GTT) + print(gtt_memory_total) except AmdSmiException as e: print(e) ``` @@ -1660,8 +1664,12 @@ try: print("No GPUs on machine") else: for device in devices: - memory = amdsmi_get_gpu_memory_usage(device) - print(memory) + vram_memory_usage = amdsmi_get_gpu_memory_usage(device, amdsmi_interface.AmdSmiMemoryType.VRAM) + print(vram_memory_usage) + vis_vram_memory_usage = amdsmi_get_gpu_memory_usage(device, amdsmi_interface.AmdSmiMemoryType.VIS_VRAM) + print(vis_vram_memory_usage) + gtt_memory_usage = amdsmi_get_gpu_memory_usage(device, amdsmi_interface.AmdSmiMemoryType.GTT) + print(gtt_memory_usage) except AmdSmiException as e: print(e) ``` @@ -2257,7 +2265,7 @@ Output: Dictionary with fields `current_dclk0` | Current dclk0 | MHz `current_vclk1` | Current vclk1 | MHz `current_dclk1` | Current dclk1 | MHz -`throttle_status` | Current throttle status | MHz +`throttle_status` | Current throttle status | bool `current_fan_speed` | Current fan speed | RPM `pcie_link_width` | PCIe link width (number of lanes) | lanes `pcie_link_speed` | PCIe link speed in 0.1 GT/s (Giga Transfers per second) | GT/s diff --git a/projects/amdsmi/src/amd_smi/amd_smi_gpu_device.cc b/projects/amdsmi/src/amd_smi/amd_smi_gpu_device.cc index 72c5cc4d3c..650227d9fd 100644 --- a/projects/amdsmi/src/amd_smi/amd_smi_gpu_device.cc +++ b/projects/amdsmi/src/amd_smi/amd_smi_gpu_device.cc @@ -158,6 +158,12 @@ amdsmi_status_t AMDSmiGPUDevice::amdgpu_query_vbios(void *info) const { int32_t AMDSmiGPUDevice::get_compute_process_list_impl(GPUComputeProcessList_t& compute_process_list, ComputeProcessListType_t list_type) { + + /** + * Clear the compute_process_list before starting. + */ + compute_process_list.clear(); + /** * The first call to GetProcessInfo() helps to find the size it needs, * so we can create a tailored size list.