[SWDEV-512693] Iteration based counter multiplexing (#272)

Adds iteration based multiplexing to counter collection. Counter groups can now be specified. These counter groups are collected on a device individually until a specified interval period is reached. When the interval is reached, the next counter group is set to be collected on subsequent kernel executions.

Supplies two new argument types that can be included in YAML/JSON inputs:

pmc_groups: an array of arrays containing the counter groups to run (i.e. [ ["SQ_WAVES", "GRBM_COUNT"], ["GRBM_GUI_ACTIVE"])
pmc_group_interval: the number of kernel invocations on a GPU of a group before rotating to the next group

Note: originally there was a random_seed_generator proposed in the linked ticket, that was not implemented since there are very few instances where you would want the selection of the groups to be randomly generated (and if you do, you can randomly generate the pattern and place it as a large list of groups in pmc_group).

All existing counter functionality should be preserved (selection of counters on specific devices only, profiling of only specific kernels, etc).

---------

Co-authored-by: Benjamin Welton <bewelton@amd.com>

source/docs/how-to/using-rocprofv3.rst

@@ -861,6 +861,47 @@ Here are the contents of ``counter_collection.csv`` file:
 
 For the description of the fields in the output file, see :ref:`output-file-fields`.
 
+Iteration based counter multiplexing
+++++++++++++++++++++++++++++++++++++
+
+Counter multiplexing allows a single run of the program to collect groups of counters. This is useful when the counters you want to collect exceed the hardware limits and you cannot run the program multiple times for collection. 
+
+This feature is available when using YAML (.yaml/.yml) or JSON (.json) input formats. Two new fields are introduced,  ``pmc_groups`` and ``pmc_group_interval``. The ``pmc_groups`` field is used to specify the groups of counters to be collected in each run. The ``pmc_group_interval`` field is used to specify the interval between each group of counters. Interval is per-device and increments per dispatch on the device (i.e. dispatch_id). When the interval is reached the next group is selected.
+
+Here is a sample input.yaml file for specifying counter multiplexing:
+
+.. code-block:: yaml
+   
+   jobs:
+   - pmc_groups: [["SQ_WAVES", "GRBM_COUNT"], ["GRBM_GUI_ACTIVE"]]
+      pmc_group_interval: 4
+
+This sample input will collect the first group of counters (``SQ_WAVES``, ``GRBM_COUNT``) for the first 4 kernel executions on the device, then the second group of counters (``GRBM_GUI_ACTIVE``) for the next 4 kernel executions on the device, and so on.
+
+An example of the interval period for this input is given below:
+
+.. code-block:: shell
+    
+    Device 1, <Kernel A>, Collect SQ_WAVES, GRBM_COUNT
+    Device 1, <Kernel A>, Collect SQ_WAVES, GRBM_COUNT
+    Device 1, <Kernel B>, Collect SQ_WAVES, GRBM_COUNT
+    Device 1, <Kernel C>, Collect SQ_WAVES, GRBM_COUNT
+    <Interval reached on Device 1, Swtiching Counters>
+    Device 1, <Kernel D>, Collect GRBM_GUI_ACTIVE
+
+Here is the same sample in JSON format:
+
+.. code-block:: shell
+
+   {
+      "jobs": [
+         {
+               "pmc_groups": [["SQ_WAVES", "GRBM_COUNT"], ["GRBM_GUI_ACTIVE"]],
+               "pmc_group_interval": 4
+         }
+      ]
+   }
+
 Agent info
 ++++++++++++
 

source/docs/rocprofv3_input_schema.json

@@ -15,7 +15,16 @@
                       "type" : "array",
                       "description": "list of counters to collect"
                      },
-
+                    
+                    "pmc_groups": {
+                         "type" : "array",
+                         "description": "An array containing lists of PMC counters to collect in a multiplexing fashion (e.x. [[counter1, counter2], [counter3, counter4]])"
+                     },
+                    "pmc_group_interval": {
+                         "type" : "integer",
+                          "description": "Number of kernel launches between selecting the next group of counters to collect"
+                     },
+                     
                      "kernel_include_regex":{
                         "type": "string",
                         "description": "Include the kernels matching this filter"