AI/rocm-systems
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge amd-staging into amd-master 20240612

Browse Source 
Signed-off-by: Jingyu1 Li <Jingyu1.Li@amd.com>
Change-Id: Ie220a097143ff4efd6149078869aff9cd82da826


[ROCm/amdsmi commit: 00d19f491e]
This commit is contained in:
Jingyu1 Li
2024-06-12 11:26:19 +08:00
parent 5e10a07fe2 807ca0ad89
commit ea7f3c164d
8 changed files with 150 additions and 101 deletions
Download Patch File Download Diff File Expand all files Collapse all files
projects/amdsmi/CHANGELOG.md
+43 -3
View File
@@ -27,6 +27,9 @@ Added `AMDSMI_EVT_NOTIF_RING_HANG` to the possible events in the `amdsmi_evt_not
### Optimizations
- **Updated naming for `amdsmi_set_gpu_clear_sram_data()` to `amdsmi_set_gpu_run_cleaner_shader()`**.
Changed the naming to be more accurate to what the function was doing. This change also extends to the CLI where we changed the `clear-sram-data` command to `run-shader` that accepts a shader name to run.
- **Updated `amdsmi_clk_info_t` struct in amdsmi.h and amdsmi_interface.py to align with host/guest**.
Changed cur_clk to clk, changed sleep_clk to clk_deep_sleep, and added clk_locked value. New struct will be in the following format:
@@ -66,7 +69,7 @@ The file structure referenced to dpm_policy changed to soc_pstate and we have ch
Previously on devices without a FRU we would not populate the product name in the `amdsmi_board_info_t` structure, now we will fallback to using the name listed according to the pciids file if available.
- **Updated CLI voltage curve command output**.
The output for `amd-smi metric --voltage-curve` now splits the frequency and voltage output by curve point or outputs N/A if not applicable
The output for `amd-smi metric --voltage-curve` now splits the frequency and voltage output by curve point or outputs N/A for each curve point if not applicable
```shell
GPU: 0
@@ -85,6 +88,43 @@ ASIC products. This requires users to update any ABIs using this structure.
### Fixes
- **Fixed multiple processes not being registered in `amd-smi process` with json and csv format**.
Multiple process outputs in the CLI tool were not being registered correctly. The json output did not handle multiple processes and is now in a new valid json format:
```shell
[
{
"gpu": 0,
"process_list": [
{
"process_info": {
"name": "TransferBench",
"pid": 420157,
"mem_usage": {
"value": 0,
"unit": "B"
}
}
},
{
"process_info": {
"name": "rvs",
"pid": 420315,
"mem_usage": {
"value": 0,
"unit": "B"
}
}
}
]
}
]
```
- **Removed `throttle-status` from `amd-smi monitor` as it is no longer reliably supported**.
Throttle status may work for older ASICs, but will be replaced with PVIOL and TVIOL metrics for future ASIC support. It remains a field in the gpu_metrics API and in `amd-smi metric --power`.
- **`amdsmi_get_gpu_board_info()` no longer returns junk char strings**.
Previously if there was a partial failure to retrieve character strings, we would return
garbage output to users using the API. This fix intends to populate as many values as possible.
@@ -96,7 +136,7 @@ The parsing of `pp_od_clk_voltage` was not dynamic enough to work with the dropp
### Known Issues
- N/A
- **Process isolation and run shader commands do no currently work and will be supported in a future release**.
## amd_smi_lib for ROCm 6.1.2
@@ -106,7 +146,7 @@ The parsing of `pp_od_clk_voltage` was not dynamic enough to work with the dropp
Added APIs CLI and APIs to address LeftoverLocals security issues. Allowing clearing the sram data and setting process isolation on a per GPU basis. New APIs:
- `amdsmi_get_gpu_process_isolation()`
- `amdsmi_set_gpu_process_isolation()`
- `amdsmi_set_gpu_run_cleaner_shader()`
- `amdsmi_set_gpu_clear_sram_data()`
- **Added `MIN_POWER` to output of `amd-smi static --limit`**.
This change helps users identify the range to which they can change the power cap of the GPU. The change is added to simplify why a device supports (or does not support) power capping (also known as overdrive). See `amd-smi set -g all --power-cap <value in W>` or `amd-smi reset -g all --power-cap`.
projects/amdsmi/amdsmi_cli/README.md
+1 -2
View File
@@ -594,7 +594,7 @@ Command Modifiers:
usage: amd-smi monitor [-h] [--json | --csv] [--file FILE] [--loglevel LEVEL]
[-g GPU [GPU ...] | -U CPU [CPU ...] | -O CORE [CORE ...]]
[-w INTERVAL] [-W TIME] [-i ITERATIONS] [-p] [-t] [-u] [-m] [-n]
[-d] [-s] [-e] [-v] [-r]
[-d] [-e] [-v] [-r]
Monitor a target device for the specified arguments.
If no arguments are provided, all arguments will be enabled.
@@ -626,7 +626,6 @@ Monitor Arguments:
-m, --mem Monitor memory utilization (%) and clock (MHz)
-n, --encoder Monitor encoder utilization (%) and clock (MHz)
-d, --decoder Monitor decoder utilization (%) and clock (MHz)
-s, --throttle-status Monitor thermal throttle status
-e, --ecc Monitor ECC single bit, ECC double bit, and PCIe replay error counts
-v, --vram-usage Monitor memory usage in MB
-r, --pcie Monitor PCIe bandwidth in Mb/s
projects/amdsmi/amdsmi_cli/amdsmi_commands.py
+68 -85
View File
@@ -1328,8 +1328,8 @@ class AMDSMICommands():
'gfx_voltage': "N/A",
'soc_voltage': "N/A",
'mem_voltage': "N/A",
'power_management': "N/A",
'throttle_status': "N/A"}
'throttle_status': "N/A",
'power_management': "N/A"}
try:
voltage_unit = "mV"
@@ -1792,46 +1792,46 @@ class AMDSMICommands():
values_dict["fan"] = fan_dict
if "voltage_curve" in current_platform_args:
if args.voltage_curve:
# Populate N/A values per voltage point
voltage_point_dict = {}
for point in range(amdsmi_interface.AMDSMI_NUM_VOLTAGE_CURVE_POINTS):
voltage_point_dict[f'point_{point}_frequency'] = "N/A"
voltage_point_dict[f'point_{point}_voltage'] = "N/A"
try:
od_volt = amdsmi_interface.amdsmi_get_gpu_od_volt_info(args.gpu)
logging.debug(f"OD Voltage info: {od_volt}")
# Populate N/A values per voltage point
voltage_point_dict = {}
for point in range(amdsmi_interface.AMDSMI_NUM_VOLTAGE_CURVE_POINTS):
voltage_point_dict[f'point_{point}_frequency'] = "N/A"
voltage_point_dict[f'point_{point}_voltage'] = "N/A"
# Populate voltage point values
for point in range(amdsmi_interface.AMDSMI_NUM_VOLTAGE_CURVE_POINTS):
if isinstance(od_volt, dict):
logging.debug(f"point_{point} frequency: {od_volt['curve.vc_points'][point].frequency}")
logging.debug(f"point_{point} voltage: {od_volt['curve.vc_points'][point].voltage}")
frequency = int(od_volt["curve.vc_points"][point].frequency / 1000000)
voltage = int(od_volt["curve.vc_points"][point].voltage)
else:
frequency = "N/A"
voltage = "N/A"
if frequency == 0:
frequency = "N/A"
if voltage == 0:
voltage = "N/A"
if frequency != "N/A":
frequency = self.helpers.unit_format(self.logger, frequency, "Mhz")
if voltage != "N/A":
voltage = self.helpers.unit_format(self.logger, voltage, "mV")
voltage_point_dict[f'point_{point}_frequency'] = frequency
voltage_point_dict[f'point_{point}_voltage'] = voltage
values_dict['voltage_curve'] = voltage_point_dict
except amdsmi_exception.AmdSmiLibraryException as e:
values_dict['voltage_curve'] = "N/A"
od_volt = "N/A" # Value not used, but needs to not be a dict
logging.debug("Failed to get voltage curve for gpu %s | %s", gpu_id, e.get_error_info())
# Populate voltage point values
for point in range(amdsmi_interface.AMDSMI_NUM_VOLTAGE_CURVE_POINTS):
if isinstance(od_volt, dict):
logging.debug(f"point_{point} frequency: {od_volt['curve.vc_points'][point].frequency}")
logging.debug(f"point_{point} voltage: {od_volt['curve.vc_points'][point].voltage}")
frequency = int(od_volt["curve.vc_points"][point].frequency / 1000000)
voltage = int(od_volt["curve.vc_points"][point].voltage)
else:
frequency = "N/A"
voltage = "N/A"
if frequency == 0:
frequency = "N/A"
if voltage == 0:
voltage = "N/A"
if frequency != "N/A":
frequency = self.helpers.unit_format(self.logger, frequency, "Mhz")
if voltage != "N/A":
voltage = self.helpers.unit_format(self.logger, voltage, "mV")
voltage_point_dict[f'point_{point}_frequency'] = frequency
voltage_point_dict[f'point_{point}_voltage'] = voltage
values_dict['voltage_curve'] = voltage_point_dict
if "overdrive" in current_platform_args:
if args.overdrive:
try:
@@ -2706,47 +2706,47 @@ class AMDSMICommands():
process_names.append(process_info)
filtered_process_values = process_names
# If the name or pid args filter processes out then insert an N/A placeholder
if not filtered_process_values:
filtered_process_values.append({'process_info': "N/A"})
logging.debug(f"Process Info for GPU {gpu_id} | {filtered_process_values}")
multiple_devices_csv_override = False
# Convert and store output by pid for csv format
if self.logger.is_csv_format():
# Check for empty list first
if not filtered_process_values:
self.logger.store_output(args.gpu, 'process_info', 'No running processes detected')
else:
for process_info in filtered_process_values:
if process_info['process_info'] == "N/A":
self.logger.store_output(args.gpu, 'process_info', 'No running processes detected')
else:
for key, value in process_info['process_info'].items():
multiple_devices_csv_override = True
if watching_output:
self.logger.store_output(args.gpu, 'timestamp', int(time.time()))
self.logger.store_output(args.gpu, key, value)
for index, process in enumerate(filtered_process_values):
if process['process_info'] == "N/A":
filtered_process_values[index]['process_info'] = "No running processes detected"
self.logger.store_multiple_device_output()
else:
if self.logger.is_json_format():
if watching_output:
self.logger.store_output(args.gpu, 'timestamp', int(time.time()))
self.logger.store_output(args.gpu, 'process_list', filtered_process_values)
# Store values in logger.output
if not filtered_process_values:
self.logger.store_output(args.gpu, 'process_info', 'No running processes detected')
else:
for process_info in filtered_process_values:
if process_info['process_info'] == "N/A":
process_info['process_info'] = 'No running processes detected'
self.logger.store_output(args.gpu, 'process_info', process_info['process_info'])
if self.logger.is_human_readable_format():
if watching_output:
self.logger.store_output(args.gpu, 'timestamp', int(time.time()))
# When we print out process_info we remove the index
# The removal is needed only for human readable process format to align with Host
for index, process in enumerate(filtered_process_values):
self.logger.store_output(args.gpu, f'process_info_{index}', process['process_info'])
multiple_devices_csv_override = False
if self.logger.is_csv_format():
multiple_devices_csv_override = True
for process in filtered_process_values:
if watching_output:
self.logger.store_output(args.gpu, 'timestamp', int(time.time()))
self.logger.store_output(args.gpu, 'process_info', process['process_info'])
self.logger.store_multiple_device_output()
if multiple_devices:
self.logger.store_multiple_device_output()
return # Skip printing when there are multiple devices
self.logger.print_output(multiple_device_enabled=multiple_devices_csv_override, watching_output=watching_output)
multiple_devices = multiple_devices or multiple_devices_csv_override
self.logger.print_output(multiple_device_enabled=multiple_devices, watching_output=watching_output)
if watching_output: # End of single gpu add to watch_output
self.logger.store_watch_output(multiple_device_enabled=multiple_devices_csv_override)
self.logger.store_watch_output(multiple_device_enabled=multiple_devices)
def profile(self, args):
@@ -3968,7 +3968,7 @@ class AMDSMICommands():
def monitor(self, args, multiple_devices=False, watching_output=False, gpu=None,
watch=None, watch_time=None, iterations=None, power_usage=None,
temperature=None, gfx_util=None, mem_util=None, encoder=None, decoder=None,
throttle_status=None, ecc=None, vram_usage=None, pcie=None):
ecc=None, vram_usage=None, pcie=None):
""" Populate a table with each GPU as an index to rows of targeted data
Args:
@@ -3984,7 +3984,6 @@ class AMDSMICommands():
mem (bool, optional): Value override for args.mem. Defaults to None.
encoder (bool, optional): Value override for args.encoder. Defaults to None.
decoder (bool, optional): Value override for args.decoder. Defaults to None.
throttle_status (bool, optional): Value override for args.throttle_status. Defaults to None.
ecc (bool, optional): Value override for args.ecc. Defaults to None.
vram_usage (bool, optional): Value override for args.vram_usage. Defaults to None.
pcie (bool, optional): Value override for args.pcie. Defaults to None.
@@ -4019,8 +4018,6 @@ class AMDSMICommands():
args.encoder = encoder
if decoder:
args.decoder = decoder
if throttle_status:
args.throttle_status = throttle_status
if ecc:
args.ecc = ecc
if vram_usage:
@@ -4034,10 +4031,10 @@ class AMDSMICommands():
# If all arguments are False, the print all values
if not any([args.power_usage, args.temperature, args.gfx, args.mem,
args.encoder, args.decoder, args.throttle_status, args.ecc,
args.encoder, args.decoder, args.ecc,
args.vram_usage, args.pcie]):
args.power_usage = args.temperature = args.gfx = args.mem = \
args.encoder = args.decoder = args.throttle_status = args.ecc = \
args.encoder = args.decoder = args.ecc = \
args.vram_usage = args.pcie = True
# Handle watch logic, will only enter this block once
@@ -4282,20 +4279,6 @@ class AMDSMICommands():
logging.debug("Failed to get decoder clock on gpu %s | %s", gpu_id, e.get_error_info())
self.logger.table_header += 'DEC_CLOCK'.rjust(11)
if args.throttle_status:
try:
throttle_status = amdsmi_interface.amdsmi_get_gpu_metrics_info(args.gpu)['throttle_status']
if throttle_status != "N/A":
if throttle_status:
throttle_status = "THROTTLED"
else:
throttle_status = "UNTHROTTLED"
monitor_values['throttle_status'] = throttle_status
except amdsmi_exception.AmdSmiLibraryException as e:
monitor_values['throttle_status'] = "N/A"
logging.debug("Failed to get throttle status on gpu %s | %s", gpu_id, e.get_error_info())
self.logger.table_header += 'THROTTLE'.rjust(13)
if args.ecc:
try:
ecc = amdsmi_interface.amdsmi_get_gpu_total_ecc_count(args.gpu)
projects/amdsmi/amdsmi_cli/amdsmi_helpers.py
+2 -2
View File
@@ -104,8 +104,8 @@ class AMDSMIHelpers():
if string_format:
return f"{operating_system} {operating_system_type}"
else:
return (operating_system, operating_system_type)
return (operating_system, operating_system_type)
def is_virtual_os(self):
projects/amdsmi/amdsmi_cli/amdsmi_logger.py
+6 -1
View File
@@ -72,9 +72,11 @@ class AMDSMILogger():
def is_human_readable_format(self):
return self.format == self.LoggerFormat.human_readable.value
def clear_multiple_devices_ouput(self):
self.multiple_device_output.clear()
def _capitalize_keys(self, input_dict):
output_dict = {}
for key in input_dict.keys():
@@ -116,7 +118,7 @@ class AMDSMILogger():
table_values += value.rjust(11)
elif key == 'vram_total' or 'ecc' in key:
table_values += value.rjust(12)
elif key in ('throttle_status', 'pcie_replay'):
elif key in ['pcie_replay']:
table_values += value.rjust(13)
# Only for handling topology tables
elif 'gpu_' in key:
@@ -163,6 +165,9 @@ class AMDSMILogger():
yaml_output = yaml_output.replace("AMDSMI_SPACING_REMOVAL:\n", "")
yaml_output = yaml_output.replace("'", "") # Remove ''
# Remove process_info indicies for Host parity:
yaml_output = re.sub(r'PROCESS_INFO_[0-9]+:', 'PROCESS_INFO:', yaml_output)
clean_yaml_output = ''
for line in yaml_output.splitlines():
line = line.split(':')
projects/amdsmi/amdsmi_cli/amdsmi_parser.py
+11 -3
View File
@@ -148,6 +148,16 @@ class AMDSMIParser(argparse.ArgumentParser):
raise amdsmi_cli_exceptions.AmdSmiInvalidParameterValueException(int_value, outputformat)
def _is_valid_string(self, string_value):
# Argument type validator
# This is for triggering a cli exception if an empty string is detected
if string_value:
return string_value
outputformat = self.helpers.get_output_format()
raise amdsmi_cli_exceptions.AmdSmiInvalidParameterValueException(string_value, outputformat)
def _check_output_file_path(self):
""" Argument action validator:
Returns a path to a file from the output file path provided.
@@ -863,7 +873,7 @@ class AMDSMIParser(argparse.ArgumentParser):
process_parser.add_argument('-G', '--general', action='store_true', required=False, help=general_help)
process_parser.add_argument('-e', '--engine', action='store_true', required=False, help=engine_help)
process_parser.add_argument('-p', '--pid', action='store', type=self._not_negative_int, required=False, help=pid_help)
process_parser.add_argument('-n', '--name', action='store', required=False, help=name_help)
process_parser.add_argument('-n', '--name', action='store', type=self._is_valid_string, required=False, help=name_help)
def _add_profile_parser(self, subparsers, func):
@@ -1113,7 +1123,6 @@ class AMDSMIParser(argparse.ArgumentParser):
mem_util_help = "Monitor memory utilization (%%) and clock (MHz)"
encoder_util_help = "Monitor encoder utilization (%%) and clock (MHz)"
decoder_util_help = "Monitor decoder utilization (%%) and clock (MHz)"
throttle_help = "Monitor thermal throttle status"
ecc_help = "Monitor ECC single bit, ECC double bit, and PCIe replay error counts"
mem_usage_help = "Monitor memory usage in MB"
pcie_bandwidth_help = "Monitor PCIe bandwidth in Mb/s"
@@ -1136,7 +1145,6 @@ class AMDSMIParser(argparse.ArgumentParser):
monitor_parser.add_argument('-m', '--mem', action='store_true', required=False, help=mem_util_help)
monitor_parser.add_argument('-n', '--encoder', action='store_true', required=False, help=encoder_util_help)
monitor_parser.add_argument('-d', '--decoder', action='store_true', required=False, help=decoder_util_help)
monitor_parser.add_argument('-s', '--throttle-status', action='store_true', required=False, help=throttle_help)
monitor_parser.add_argument('-e', '--ecc', action='store_true', required=False, help=ecc_help)
monitor_parser.add_argument('-v', '--vram-usage', action='store_true', required=False, help=mem_usage_help)
monitor_parser.add_argument('-r', '--pcie', action='store_true', required=False, help=pcie_bandwidth_help)
projects/amdsmi/py-interface/README.md
+13 -5
View File
@@ -1588,8 +1588,12 @@ try:
print("No GPUs on machine")
else:
for device in devices:
memory = amdsmi_get_gpu_memory_total(device)
print(memory)
vram_memory_total = amdsmi_get_gpu_memory_total(device, amdsmi_interface.AmdSmiMemoryType.VRAM)
print(vram_memory_total)
vis_vram_memory_total = amdsmi_get_gpu_memory_total(device, amdsmi_interface.AmdSmiMemoryType.VIS_VRAM)
print(vis_vram_memory_total)
gtt_memory_total = amdsmi_get_gpu_memory_total(device, amdsmi_interface.AmdSmiMemoryType.GTT)
print(gtt_memory_total)
except AmdSmiException as e:
print(e)
```
@@ -1660,8 +1664,12 @@ try:
print("No GPUs on machine")
else:
for device in devices:
memory = amdsmi_get_gpu_memory_usage(device)
print(memory)
vram_memory_usage = amdsmi_get_gpu_memory_usage(device, amdsmi_interface.AmdSmiMemoryType.VRAM)
print(vram_memory_usage)
vis_vram_memory_usage = amdsmi_get_gpu_memory_usage(device, amdsmi_interface.AmdSmiMemoryType.VIS_VRAM)
print(vis_vram_memory_usage)
gtt_memory_usage = amdsmi_get_gpu_memory_usage(device, amdsmi_interface.AmdSmiMemoryType.GTT)
print(gtt_memory_usage)
except AmdSmiException as e:
print(e)
```
@@ -2257,7 +2265,7 @@ Output: Dictionary with fields
`current_dclk0` | Current dclk0 | MHz
`current_vclk1` | Current vclk1 | MHz
`current_dclk1` | Current dclk1 | MHz
`throttle_status` | Current throttle status | MHz
`throttle_status` | Current throttle status | bool
`current_fan_speed` | Current fan speed | RPM
`pcie_link_width` | PCIe link width (number of lanes) | lanes
`pcie_link_speed` | PCIe link speed in 0.1 GT/s (Giga Transfers per second) | GT/s
projects/amdsmi/src/amd_smi/amd_smi_gpu_device.cc
+6
View File
@@ -158,6 +158,12 @@ amdsmi_status_t AMDSmiGPUDevice::amdgpu_query_vbios(void *info) const {
int32_t AMDSmiGPUDevice::get_compute_process_list_impl(GPUComputeProcessList_t& compute_process_list,
ComputeProcessListType_t list_type)
{
/**
* Clear the compute_process_list before starting.
*/
compute_process_list.clear();
/**
* The first call to GetProcessInfo() helps to find the size it needs,
* so we can create a tailored size list.