Partition EBUSY with RSMI_STATUS_BUSY & invalid GPU Metrics check

* Updates:
   - [API/CLI] rsmi_dev_*_partition_set &
     rsmi_dev_*_partition_reset - exposed RSMI_STATUS_BUSY for
     EBUSY writes + cleaned up accidental map insertions
     (maplookup[] can insert values that are not in the map,
     map.at(key) fixes this potential issue)
   - [API] rsmi_dev_gpu_metrics_info_get() - returns
     RSMI_STATUS_NOT_SUPPORTED for unsupported metric tables
     outside of 1v1/1v2/1v3
   - [API] writeDevInfoStr() - exposes RSMI_STATUS_BUSY for
     EBUSY write errors; kept backward compatibility
     for other writes which do not care about these states
   - [API] rsmi_dev_od_volt_info_get()
      & rsmi_dev_od_volt_curve_regions_get() have better logging
     + Expose more details on why they are erroring
   - [Utils/logs/example] Expose AMD GPU gfx target version to aid in
     system troubleshooting
   - [Utils] Added test methods that look at od volt
     freq & regions into here - for easier access across
     several tests
   - [Utils] Updated getRSMIStatusString(new argument - fullstatus;
     default to true for backwards compatibility)
     -> true shows shortened RSMI STATUS response
   - [Utils] Added splitString to cut out noisy return responses
     (used in getRSMIStatusString(), when fullstatus = true)
   - [Utils] Added getFileCreationDate() to expose build date
     of the library - helpful for local builds or experimental builds
   - [Utils] Macro cleanup
   - [Example] Added a few gpu_metric checks - helpful for upcoming
     updates
   - [Device] SYSFS/DebugFS - now have better r/w displayed in logs
   - [LOGS] Expose library build date - see above for details
   - [Tests] Add more warnings/errors to test builds
   - [Tests] Moved up Partition tests for ordered test runs - helped
     identify issues with GPU BUSY writes
   - [Tests] compute_partition_read_write - handles RSMI_STATUS_BUSY
     with waits for busy status found & cleaned up how we checked
     for partition changes - with RSMI responses exposed more clearly
   - [Tests] perf_determinism - multi gpu now properly runs through
     with full resets as needed
   - [Tests] volt_freq_curv_read - better error handling with more
     verbose output

Change-Id: Ie94c6abb6a9aab95c345996d3ad3843cf6734977
Signed-off-by: Charis Poag <Charis.Poag@amd.com>


[ROCm/amdsmi commit: 57b6135e54]

projects/amdsmi/python_smi_tools/rocm_smi.py

@@ -966,6 +966,9 @@ def resetComputePartition(deviceList):
             printLog(device, 'Permission denied', None)
         elif ret == rsmi_status_t.RSMI_STATUS_NOT_SUPPORTED:
             printLog(device, 'Not supported on the given system', None)
+        elif ret == rsmi_status_t.RSMI_STATUS_BUSY:
+            printLog(device, 'Device is currently busy, try again later',
+                     None)
         else:
             rsmi_ret_ok(ret, device, 'reset_compute_partition')
             printErrLog(device, 'Failed to reset the compute partition to boot state')
@@ -1002,6 +1005,9 @@ def resetMemoryPartition(deviceList):
             printLog(device, 'Permission denied', None, addExtraLine)
         elif ret == rsmi_status_t.RSMI_STATUS_NOT_SUPPORTED:
             printLog(device, 'Not supported on the given system', None, addExtraLine)
+        elif ret == rsmi_status_t.RSMI_STATUS_BUSY:
+            printLog(device, 'Device is currently busy, try again later',
+                     None)
         else:
             rsmi_ret_ok(ret, device, 'reset_memory_partition')
             printErrLog(device, 'Failed to reset memory partition to boot state')
@@ -1603,6 +1609,9 @@ def setComputePartition(deviceList, computePartitionType):
                      %computePartitionType, None)
         elif ret == rsmi_status_t.RSMI_STATUS_NOT_SUPPORTED:
             printLog(device, 'Not supported on the given system', None)
+        elif ret == rsmi_status_t.RSMI_STATUS_BUSY:
+            printLog(device, 'Device is currently busy, try again later',
+                     None)
         else:
             rsmi_ret_ok(ret, device, 'set_compute_partition')
             printErrLog(device, 'Failed to retrieve compute partition, even though device supports it.')
@@ -1673,6 +1682,9 @@ def setMemoryPartition(deviceList, memoryPartition):
             printLog(device, 'Permission denied', None, addExtraLine)
         elif ret == rsmi_status_t.RSMI_STATUS_NOT_SUPPORTED:
             printLog(device, 'Not supported on the given system', None, addExtraLine)
+        elif ret == rsmi_status_t.RSMI_STATUS_BUSY:
+            printLog(device, 'Device is currently busy, try again later',
+                     None, addExtraLine)
         else:
             rsmi_ret_ok(ret, device, 'set_memory_partition')
             printErrLog(device, 'Failed to retrieve memory partition, even though device supports it.')