Agent, Counters, and AQL (#55)

* Migrate XML counter defs and reader from v1/v2 * Current Working Set * Modified parser * Evaluate AST Start * Update lib/common/xml - move definitions out of class declaration * Update lib/rocprofiler/counters/parser - update build of bison and flex build - reproducible generation - add ROCPROFILER_REGENERATE_COUNTERS_PARSER option - fix namespacing * Update lib/rocprofiler/counters/xml - change location of XML files and install them * Update lib/rocprofiler/counter/tests - normalize the test names - improve test failures (more clear about where failure is) * Update lib/rocprofiler/counters - fix namespace - update to new XML metrics directory * Update lib/rocprofiler/CMakeLists.txt - link to object library * Update lib/rocprofiler/hsa/types.hpp - reorganize includes * Add metric loading class/printers * Agent Implementation * Queue Implementation (#79) * Queue Implementation * API Implementation For Counters (part 1) (#80) * API Implementation For Counters * Bewelton/counter collection 3 (#84) * Added counter sample * More changes * More changes * Update samples/counter_collection - mostly formatting * Update include/rocprofiler/counters.h - formatting * Add lib.common/synchronized.hpp - Synchronized struct * Update lib/rocprofiler/counters/xml/basic_counters.xml - whitespace * Update scripts/patch-parser.cmake - tweaks for consistency * Update lib/rocprofiler/counters/parser/tests/parser_tests.cpp - formatting * Update lib/rocprofiler/counters/parser - improve consistency in rocprofiler-expr-parser-patch - update parser.{h,cpp} and scanner.cpp - formatting + regenerated * Update lib/rocprofiler/aql - formatting - clang-tidy fixes - guard against memory pool access errors * Update lib/rocprofiler/aql/tests - formatting - update use of get_val - normalize test names * Update lib/rocprofiler/counters/tests - formatting - patch basic_counters and derived_counters - normalize test names * Update lib/rocprofiler/aql/tests - set_tests_properties * Update test labels - fix minor issue with gtest labels * Update lib/rocprofiler/counters - formatting - clang-tidy fixes * Update lib/rocprofiler/hsa - fix includes - formatting - clang-tidy fixes - tweak to queue_controller_init interface * Update lib/rocprofiler - include fixes - namespace fixes - clang-tidy fixes - formatting * Update scripts/run-ci.py - exclude counters/parser from code coverage (generated files) * Update include/rocprofiler/counters.h - fix doxygen comment * Update lib/rocprofiler/aql/packet_construct.cpp - guard against HSA_AMD_MEMORY_POOL_ACCESS_DISALLOWED_BY_DEFAULT and HSA_AMD_MEMORY_POOL_ACCESS_NEVER_ALLOWED * Update lib/rocprofiler/counters/parser/raw_ast.hpp - clang-tidy fixes * Update lib/rocprofiler/counters/evaluate_ast.hpp - clang-tidy fixes * Update lib/rocprofiler/aql/tests - disable packet_generation_single and packet_generation_multi tests - the entire implementation rocprofiler::get_ext_table() is incorrect * Minor fixes before cleanup * More changes * More fixes * More fixes * source formatting (clang-format v11) (#99) Co-authored-by: bwelton <bwelton@users.noreply.github.com> * Revert PTL submodule * Update scripts/run-ci.py - exclude counters/parser from code coverage (generated files) * Migrating counters state to context * Linting * source formatting (clang-format v11) (#101) Co-authored-by: bwelton <bwelton@users.noreply.github.com> * revert run-ci * Testing fixes * More test changes * Fix minor typo * Small queue change * Small queue change * source formatting (clang-format v11) (#102) Co-authored-by: bwelton <bwelton@users.noreply.github.com> * source formatting (clang-format v11) (#105) Co-authored-by: bwelton <bwelton@users.noreply.github.com> * Documentation Change * More documentation fixes * source formatting (clang-format v11) (#106) Co-authored-by: bwelton <bwelton@users.noreply.github.com> * Threading fixes * Threading fixes * source formatting (clang-format v11) (#107) Co-authored-by: bwelton <bwelton@users.noreply.github.com> * Threading fixes * More test fixes * More agent fixes * More build fixes * source formatting (clang-format v11) (#109) Co-authored-by: bwelton <bwelton@users.noreply.github.com> * changed test timeouts * Build fix * Build fix * Updates to agent * source formatting (clang-format v11) (#114) Co-authored-by: bwelton <bwelton@users.noreply.github.com> * cmake formatting (cmake-format) (#113) Co-authored-by: bwelton <bwelton@users.noreply.github.com> * remove git worktree folder * Doc update * testing fix * Another test fix * More test changes * Rebase * source formatting (clang-format v11) (#116) Co-authored-by: bwelton <bwelton@users.noreply.github.com> * Documentation * source formatting (clang-format v11) (#119) Co-authored-by: bwelton <bwelton@users.noreply.github.com> * PTL Changes * Minor agent fix for empty labels * source formatting (clang-format v11) (#120) Co-authored-by: bwelton <bwelton@users.noreply.github.com> * Minor agent fix for empty labels * Refactor read_map * source formatting (clang-format v11) (#121) Co-authored-by: bwelton <bwelton@users.noreply.github.com> * Refactor read_map * Cache fixes * source formatting (clang-format v11) (#122) Co-authored-by: bwelton <bwelton@users.noreply.github.com> --------- Co-authored-by: Jonathan R. Madsen <jonathanrmadsen@gmail.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: bwelton <bwelton@users.noreply.github.com> [ROCm/rocprofiler-sdk commit: 010693b795]
2023-10-16 13:41:40 -07:00
@@ -270,7 +270,7 @@ jobs:
        git config --global --add safe.directory '*'

    - name: Configure, Build, and Test
-      timeout-minutes: 30
+      timeout-minutes: 45
      shell: bash
      run:
        python3 ./source/scripts/run-ci.py -B build
@@ -78,6 +78,9 @@ rocprofiler_add_option(ROCPROFILER_BUILD_STACK_PROTECTOR "Build with -fstack-pro
                       ON ADVANCED)
 rocprofiler_add_option(ROCPROFILER_UNSAFE_NO_VERSION_CHECK
                       "Disable HSA version checking (for development only)" OFF ADVANCED)
+rocprofiler_add_option(
+    ROCPROFILER_REGENERATE_COUNTERS_PARSER
+    "Regenerate the counter parser (requires bison and flex)" OFF ADVANCED)

 # In the future, we will do this even with clang-tidy enabled
 if(ROCPROFILER_BUILD_CI AND NOT ROCPROFILER_BUILD_WERROR)
@@ -18,3 +18,4 @@ endif()
 add_subdirectory(pc_sampling)
 add_subdirectory(api_callback_tracing)
 add_subdirectory(api_buffered_tracing)
+add_subdirectory(counter_collection)
@@ -0,0 +1,55 @@
+#
+#
+#
+cmake_minimum_required(VERSION 3.21.0 FATAL_ERROR)
+
+if(NOT CMAKE_HIP_COMPILER)
+    find_program(
+        amdclangpp_EXECUTABLE
+        NAMES amdclang++
+        HINTS ${ROCM_PATH} ENV ROCM_PATH /opt/rocm
+        PATHS ${ROCM_PATH} ENV ROCM_PATH /opt/rocm
+        PATH_SUFFIXES bin llvm/bin NO_CACHE)
+    mark_as_advanced(amdclangpp_EXECUTABLE)
+
+    if(amdclangpp_EXECUTABLE)
+        set(CMAKE_HIP_COMPILER "${amdclangpp_EXECUTABLE}")
+    endif()
+endif()
+
+project(rocprofiler-samples-counter-collection LANGUAGES CXX HIP)
+
+foreach(_TYPE DEBUG MINSIZEREL RELEASE RELWITHDEBINFO)
+    if("${CMAKE_HIP_FLAGS_${_TYPE}}" STREQUAL "")
+        set(CMAKE_HIP_FLAGS_${_TYPE} "${CMAKE_CXX_FLAGS_${_TYPE}}")
+    endif()
+endforeach()
+
+if(NOT TARGET rocprofiler::rocprofiler)
+    find_package(rocprofiler REQUIRED)
+endif()
+
+add_library(counter-collection-client SHARED)
+target_sources(counter-collection-client PRIVATE client.cpp client.hpp)
+target_link_libraries(counter-collection-client PRIVATE rocprofiler::rocprofiler)
+
+set_source_files_properties(main.cpp PROPERTIES LANGUAGE HIP)
+find_package(Threads REQUIRED)
+
+add_executable(counter-collection)
+target_sources(counter-collection PRIVATE main.cpp)
+target_link_libraries(counter-collection PRIVATE counter-collection-client
+                                                 Threads::Threads)
+
+add_test(NAME counter-collection COMMAND $<TARGET_FILE:counter-collection>)
+
+set_tests_properties(
+    counter-collection
+    PROPERTIES
+        TIMEOUT
+        45
+        LABELS
+        "samples"
+        ENVIRONMENT
+        "${ROCPROFILER_MEMCHECK_PRELOAD_ENV};HSA_TOOLS_LIB=$<TARGET_FILE:rocprofiler::rocprofiler-library>"
+    )
@@ -0,0 +1,164 @@
+#include "client.hpp"
+
+#include <set>
+#include <sstream>  // std::stringstream
+#include <vector>
+
+#include <rocprofiler/registration.h>
+#include <rocprofiler/rocprofiler.h>
+
+#define ROCPROFILER_CALL(result, msg)                                                              \
+    {                                                                                              \
+        rocprofiler_status_t CHECKSTATUS = result;                                                 \
+        if(CHECKSTATUS != ROCPROFILER_STATUS_SUCCESS)                                              \
+        {                                                                                          \
+            std::cerr << #result << " failed with error code " << CHECKSTATUS << std::endl;        \
+            throw std::runtime_error(#result " failure");                                          \
+        }                                                                                          \
+    }
+
+int
+start()
+{
+    return 1;
+}
+
+namespace
+{
+rocprofiler_context_id_t&
+get_client_ctx()
+{
+    static rocprofiler_context_id_t ctx;
+    return ctx;
+}
+
+void
+test_callback(rocprofiler_queue_id_t       queue_id,
+              rocprofiler_agent_t          agent_id,
+              rocprofiler_correlation_id_t corr_id,
+              const hsa_kernel_dispatch_packet_t*,
+              void*,
+              rocprofiler_dispatch_profile_counting_record_t**,
+              size_t,
+              rocprofiler_profile_config_id_t)
+{
+    // Callback containing counter data.
+    std::clog << "[" << __FUNCTION__ << "] " << queue_id.handle << " | " << agent_id.id.handle
+              << " | " << corr_id.id << "\n";
+}
+
+int
+tool_init(rocprofiler_client_finalize_t, void*)
+{
+    std::set<std::string> counters_to_collect = {"SQ_WAVES"};
+
+    std::vector<rocprofiler_agent_t> gpu_agents;
+    auto agent_query = [](const rocprofiler_agent_t** agents, size_t num_agents, void* user_data) {
+        std::vector<rocprofiler_agent_t>* vec =
+            static_cast<std::vector<rocprofiler_agent_t>*>(user_data);
+        for(size_t i = 0; i < num_agents; i++)
+        {
+            const rocprofiler_agent_t* agent = agents[i];
+            if(agent->type == ROCPROFILER_AGENT_TYPE_GPU)
+            {
+                vec->push_back(*agent);
+            }
+        }
+        return ROCPROFILER_STATUS_SUCCESS;
+    };
+
+    ROCPROFILER_CALL(rocprofiler_query_available_agents(
+                         agent_query, sizeof(rocprofiler_agent_t), static_cast<void*>(&gpu_agents)),
+                     "Could not query agents");
+
+    ROCPROFILER_CALL(rocprofiler_create_context(&get_client_ctx()), "context creation failed");
+
+    std::vector<rocprofiler_profile_config_id_t> profile_configs;
+
+    for(auto& agent : gpu_agents)
+    {
+        std::vector<rocprofiler_counter_id_t> collect_counters;
+        std::vector<rocprofiler_counter_id_t> gpu_counters;
+        std::clog << agent.name << "\n";
+        ROCPROFILER_CALL(
+            rocprofiler_iterate_agent_supported_counters(
+                agent,
+                [](rocprofiler_counter_id_t* counters, size_t num_counters, void* user_data) {
+                    std::vector<rocprofiler_counter_id_t>* vec =
+                        static_cast<std::vector<rocprofiler_counter_id_t>*>(user_data);
+                    for(size_t i = 0; i < num_counters; i++)
+                    {
+                        vec->push_back(counters[i]);
+                    }
+                    return ROCPROFILER_STATUS_SUCCESS;
+                },
+                static_cast<void*>(&gpu_counters)),
+            "Could not fetch supported counters");
+
+        for(auto& counter : gpu_counters)
+        {
+            const char* name;
+            size_t      size;
+            ROCPROFILER_CALL(rocprofiler_query_counter_name(counter, &name, &size),
+                             "Could not query name");
+            if(counters_to_collect.count(std::string(name)) > 0)
+            {
+                std::clog << "Counter: " << counter.handle << " " << name << "\n";
+                collect_counters.push_back(counter);
+            }
+        }
+        rocprofiler_profile_config_id_t profile;
+        ROCPROFILER_CALL(rocprofiler_create_profile_config(
+                             agent, collect_counters.data(), collect_counters.size(), &profile),
+                         "Could not construct profile cfg");
+        ROCPROFILER_CALL(rocprofiler_configure_dispatch_profile_counting_service(
+                             get_client_ctx(), profile, test_callback, nullptr),
+                         "Could not setup dispatch service");
+        profile_configs.push_back(profile);
+    }
+    rocprofiler_start_context(get_client_ctx());
+
+    // no errors
+    return 0;
+}
+
+void
+tool_fini(void*)
+{
+    rocprofiler_stop_context(get_client_ctx());
+    std::clog << "In tool fini\n";
+}
+
+}  // namespace
+
+extern "C" rocprofiler_tool_configure_result_t*
+rocprofiler_configure(uint32_t    version,
+                      const char* runtime_version,
+                      uint32_t,
+                      rocprofiler_client_id_t* id)
+{
+    // set the client name
+    id->name = "CounterClientSample";
+
+    // compute major/minor/patch version info
+    uint32_t major = version / 10000;
+    uint32_t minor = (version % 10000) / 100;
+    uint32_t patch = version % 100;
+
+    // generate info string
+    auto info = std::stringstream{};
+    info << id->name << " is using rocprofiler v" << major << "." << minor << "." << patch << " ("
+         << runtime_version << ")";
+
+    std::clog << info.str() << std::endl;
+
+    // create configure data
+    static auto cfg =
+        rocprofiler_tool_configure_result_t{sizeof(rocprofiler_tool_configure_result_t),
+                                            &tool_init,
+                                            &tool_fini,
+                                            static_cast<void*>(nullptr)};
+
+    // return pointer to configure data
+    return &cfg;
+}
@@ -0,0 +1,8 @@
+#pragma once
+
+#include <iostream>
+
+#define CLIENT_API __attribute__((visibility("default")))
+
+int
+start() CLIENT_API;
@@ -0,0 +1,92 @@
+#include <hip/hip_runtime.h>
+
+#include "client.hpp"
+
+#define HIP_CALL(call)                                                                             \
+    do                                                                                             \
+    {                                                                                              \
+        hipError_t err = call;                                                                     \
+        if(err != hipSuccess)                                                                      \
+        {                                                                                          \
+            fprintf(stderr, "%s\n", hipGetErrorString(err));                                       \
+            abort();                                                                               \
+        }                                                                                          \
+    } while(0)
+
+__global__ void
+kernelA(int x, int y)
+{
+    x = x + y;
+}
+
+__global__ void
+kernelB(int x, int y)
+{
+    x = x + y;
+}
+
+template <typename T>
+__global__ void
+kernelC(T* C_d, const T* A_d, size_t N)
+{
+    size_t offset = (blockIdx.x * blockDim.x + threadIdx.x);
+    size_t stride = blockDim.x * gridDim.x;
+    for(size_t i = offset; i < N; i += stride)
+    {
+        C_d[i] = A_d[i] * A_d[i];
+    }
+}
+
+void
+launchKernals()
+{
+    const int NUM_LAUNCH = 1000;
+    // Normal HIP Calls
+    int*                             gpuMem;
+    [[maybe_unused]] hipDeviceProp_t devProp;
+    HIP_CALL(hipGetDeviceProperties(&devProp, 0));
+    HIP_CALL(hipMalloc((void**) &gpuMem, 1 * sizeof(int)));
+
+    for(int i = 0; i < NUM_LAUNCH; i++)
+    {
+        // KernelA and KernelB to be profiled as part of the session
+        hipLaunchKernelGGL(kernelA, dim3(1), dim3(1), 0, 0, 1, 2);
+        hipLaunchKernelGGL(kernelB, dim3(1), dim3(1), 0, 0, 1, 2);
+    }
+
+    const int NElems = 512 * 512;
+    const int Nbytes = NElems * 2;
+    int *     A_d, *C_d;
+    int       A_h[NElems], C_h[NElems];
+
+    for(int i = 0; i < NElems; i++)
+    {
+        A_h[i] = i;
+    }
+
+    HIP_CALL(hipDeviceSynchronize());
+
+    HIP_CALL(hipMalloc(&A_d, Nbytes));
+    HIP_CALL(hipMalloc(&C_d, Nbytes));
+    HIP_CALL(hipMemcpy(A_d, A_h, Nbytes, hipMemcpyHostToDevice));
+    HIP_CALL(hipDeviceSynchronize());
+    const unsigned blocks          = 512;
+    const unsigned threadsPerBlock = 256;
+    for(int i = 0; i < NUM_LAUNCH; i++)
+    {
+        hipLaunchKernelGGL(kernelC, dim3(blocks), dim3(threadsPerBlock), 0, 0, C_d, A_d, NElems);
+    }
+    HIP_CALL(hipMemcpy(C_h, C_d, Nbytes, hipMemcpyDeviceToHost));
+    HIP_CALL(hipDeviceSynchronize());
+    HIP_CALL(hipFree(gpuMem));
+    HIP_CALL(hipFree(A_d));
+    HIP_CALL(hipFree(C_d));
+    std::cerr << "Run complete\n";
+}
+
+int
+main()
+{
+    start();
+    launchKernals();
+}
@@ -38,37 +38,44 @@ ROCPROFILER_EXTERN_C_INIT
 * @brief Query Counter name.
 *
 * @param [in] counter_id
- * @param [out] name if nullptr, size will be returned
- * @param [out] size
+ * @param [out] name returns a pointer to the name of the counter
+ * @param [out] size returns the size of the name returned
 * @return ::rocprofiler_status_t
 */
 rocprofiler_status_t ROCPROFILER_API
-rocprofiler_query_counter_name(rocprofiler_counter_id_t counter_id, const char* name, size_t* size)
-    ROCPROFILER_NONNULL(3);
+rocprofiler_query_counter_name(rocprofiler_counter_id_t counter_id, const char** name, size_t* size)
+    ROCPROFILER_NONNULL(2, 3);

 /**
 * @brief Query Counter Instances Count.
 *
- * @param [in] counter_id
- * @param [out] instance_count
+ * @param [in] agent rocprofiler agent
+ * @param [in] counter_id counter id (obtained from iterate_agent_supported_counters)
+ * @param [out] instance_count number of instances the counter has
 * @return rocprofiler_status_t
 */
 rocprofiler_status_t ROCPROFILER_API
-rocprofiler_query_counter_instance_count(rocprofiler_counter_id_t counter_id,
-                                         size_t* instance_count) ROCPROFILER_NONNULL(2);
+rocprofiler_query_counter_instance_count(rocprofiler_agent_t      agent,
+                                         rocprofiler_counter_id_t counter_id,
+                                         size_t* instance_count) ROCPROFILER_NONNULL(3);
+
+typedef rocprofiler_status_t (*rocprofiler_available_counters_cb_t)(
+    rocprofiler_counter_id_t* counters,
+    size_t                    num_counters,
+    void*                     user_data);

 /**
 * @brief Query Agent Counters Availability.
 *
 * @param [in] agent
- * @param [out] counters_list
- * @param [out] counters_count
+ * @param [in] cb callback to caller to get counters
+ * @param [in] user_data data to pass into the callback
 * @return ::rocprofiler_status_t
 */
 rocprofiler_status_t ROCPROFILER_API
-rocprofiler_query_agent_supported_counters(rocprofiler_agent_t       agent,
-                                           rocprofiler_counter_id_t* counters_list,
-                                           size_t* counters_count) ROCPROFILER_NONNULL(2, 3);
+rocprofiler_iterate_agent_supported_counters(rocprofiler_agent_t                 agent,
+                                             rocprofiler_available_counters_cb_t cb,
+                                             void* user_data) ROCPROFILER_NONNULL(2);

 /** @} */

@@ -69,28 +69,28 @@ typedef struct
 * @param [in] config
 */
 typedef void (*rocprofiler_profile_counting_dispatch_callback_t)(
-    rocprofiler_queue_id_t              queue_id,
-    rocprofiler_agent_t                 agent_id,
-    rocprofiler_correlation_id_t        correlation_id,
-    const hsa_kernel_dispatch_packet_t* dispatch_packet,
-    void*                               callback_data_args,
-    rocprofiler_profile_config_id_t*    config);
+    rocprofiler_queue_id_t                           queue_id,
+    rocprofiler_agent_t                              agent_id,
+    rocprofiler_correlation_id_t                     correlation_id,
+    const hsa_kernel_dispatch_packet_t*              dispatch_packet,
+    void*                                            callback_data_args,
+    rocprofiler_dispatch_profile_counting_record_t** records,
+    size_t                                           record_count,
+    rocprofiler_profile_config_id_t                  config);

 /**
 * @brief Configure Dispatch Profile Counting Service.
 *
- * @param [in] context_id
- * @param [in] agent_id
- * @param [in] buffer_id
- * @param [in] callback
- * @param [in] callback_data_args
+ * @param [in] context_id context id
+ * @param [in] profile profile config to use for dispatch
+ * @param [in] callback callback
+ * @param [in] callback_data_args callback data
 * @return ::rocprofiler_status_t
 */
 rocprofiler_status_t ROCPROFILER_API
 rocprofiler_configure_dispatch_profile_counting_service(
    rocprofiler_context_id_t                         context_id,
-    rocprofiler_agent_t                              agent_id,
-    rocprofiler_buffer_id_t                          buffer_id,
+    rocprofiler_profile_config_id_t                  profile,
    rocprofiler_profile_counting_dispatch_callback_t callback,
    void*                                            callback_data_args);

@@ -69,6 +69,8 @@ typedef enum  // NOLINT(performance-enum-size)
    ROCPROFILER_STATUS_ERROR_NOT_IMPLEMENTED,             ///< Function is not implemented
    ROCPROFILER_STATUS_ERROR_INCOMPATIBLE_ABI,  ///< Data structure provided by user is incompatible
                                                ///< with current version of rocprofiler
+    ROCPROFILER_STATUS_ERROR_AGENT_NOT_FOUND,   ///< Agent not found
+    ROCPROFILER_STATUS_ERROR_COUNTER_NOT_FOUND,  ///< Counter does not exist
    ROCPROFILER_STATUS_LAST,
 } rocprofiler_status_t;

@@ -3,9 +3,9 @@
 #
 rocprofiler_activate_clang_tidy()

-set(common_sources config.cpp environment.cpp demangle.cpp utility.cpp)
+set(common_sources config.cpp environment.cpp demangle.cpp utility.cpp xml.cpp)
 set(common_headers config.hpp defines.hpp environment.hpp demangle.hpp mpl.hpp
-                   utility.hpp xml.hpp)
+                   synchronized.hpp utility.hpp xml.hpp)

 add_library(rocprofiler-common-library STATIC)
 add_library(rocprofiler::rocprofiler-common-library ALIAS rocprofiler-common-library)
@@ -1,22 +1,22 @@
-/* Copyright (c) 2022 Advanced Micro Devices, Inc.
-
- Permission is hereby granted, free of charge, to any person obtaining a copy
- of this software and associated documentation files (the "Software"), to deal
- in the Software without restriction, including without limitation the rights
- to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
- copies of the Software, and to permit persons to whom the Software is
- furnished to do so, subject to the following conditions:
-
- The above copyright notice and this permission notice shall be included in
- all copies or substantial portions of the Software.
-
- THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
- IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
- FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
- AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
- LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
- OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
- THE SOFTWARE. */
+// Copyright (c) 2023 Advanced Micro Devices, Inc.
+//
+// Permission is hereby granted, free of charge, to any person obtaining a copy
+// of this software and associated documentation files (the "Software"), to deal
+// in the Software without restriction, including without limitation the rights
+// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+// copies of the Software, and to permit persons to whom the Software is
+// furnished to do so, subject to the following conditions:
+//
+// The above copyright notice and this permission notice shall be included in
+// all copies or substantial portions of the Software.
+//
+// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+// THE SOFTWARE.

 #pragma once

@@ -0,0 +1,96 @@
+// MIT License
+//
+// Copyright (c) 2023 ROCm Developer Tools
+//
+// Permission is hereby granted, free of charge, to any person obtaining a copy
+// of this software and associated documentation files (the "Software"), to deal
+// in the Software without restriction, including without limitation the rights
+// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+// copies of the Software, and to permit persons to whom the Software is
+// furnished to do so, subject to the following conditions:
+//
+// The above copyright notice and this permission notice shall be included in all
+// copies or substantial portions of the Software.
+//
+// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+// SOFTWARE.
+
+#pragma once
+
+#include <cstddef>
+#include <functional>
+#include <mutex>
+#include <shared_mutex>
+
+namespace rocprofiler
+{
+namespace common
+{
+/**
+ * Sychronized is a wrapper that adds lock based write/read
+ * protection around a datatype. The protected data is accessed
+ * only by rlock/wlock. rlock(lambda) gets a reader lock of the
+ * protected value, passing the protected value to the lambda as a
+ * const. wlock(lambda) gets a writer lock on the protective value
+ * and does the same. The reason for this class is to make it less
+ * error prone to access shared data and more obvious when a lock
+ * is being held.
+ *
+ * Example usage:
+ *
+ * Synchronized<int> x(9);
+ * x.rlock([](const auto& data){
+ *  // data = 9
+ * });
+ *
+ * x.wlock([](auto& data){
+ *  // set data to new value
+ * });
+ */
+template <typename LockedType>
+class Synchronized
+{
+public:
+    Synchronized() = default;
+    Synchronized(LockedType&& data)
+    : data_(std::move(data))
+    {}
+    // Do not allow this data structure to be copied, std::move only.
+    Synchronized(const Synchronized&) = delete;
+
+    void rlock(std::function<void(const LockedType&)> lambda) const
+    {
+        std::shared_lock lock(mutex_);
+        lambda(data_);
+    }
+
+    void wlock(std::function<void(LockedType&)> lambda)
+    {
+        std::unique_lock lock(mutex_);
+        lambda(data_);
+    }
+
+    // Upgradable lock. If read returns false, write will be called with a unique_lock.
+    // Essentially a helper function that does .rlock() followed by .wlock().
+    void ulock(std::function<bool(const LockedType&)> read, std::function<bool(LockedType&)> write)
+    {
+        {
+            std::shared_lock lock(mutex_);
+            if(read(data_)) return;
+        }
+
+        std::unique_lock lock(mutex_);
+        write(data_);
+    }
+
+private:
+    mutable std::shared_mutex mutex_;
+    LockedType                data_;
+};
+}  // namespace common
+}  // namespace rocprofiler
@@ -66,5 +66,36 @@ get_val(Container& map, const Key& key)
    auto pos = map.find(key);
    return (pos != map.end() ? &pos->second : nullptr);
 }
+
+/**
+ * A simple wrapper that will call a function when the
+ * wrapper is being destroyed. This is primarily useful
+ * for static variables where we want to run some destruction
+ * operations when the program exits.
+ */
+template <typename T, typename L>
+class static_cleanup_wrapper
+{
+public:
+    static_cleanup_wrapper(T&& data, L&& destroy_func)
+    : _data(std::move(data))
+    , _destroy_func(destroy_func)
+    {}
+
+    static_cleanup_wrapper(L&& destroy_func)
+    : _destroy_func(destroy_func)
+    {}
+
+    ~static_cleanup_wrapper() { _destroy_func(_data); }
+
+    void destroy() { _destroy_func(_data); }
+
+    T& get() { return _data; }
+
+private:
+    T _data;
+    L _destroy_func;
+};
+
 }  // namespace common
 }  // namespace rocprofiler
@@ -0,0 +1,529 @@
+// Copyright (c) 2023 Advanced Micro Devices, Inc.
+//
+// Permission is hereby granted, free of charge, to any person obtaining a copy
+// of this software and associated documentation files (the "Software"), to deal
+// in the Software without restriction, including without limitation the rights
+// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+// copies of the Software, and to permit persons to whom the Software is
+// furnished to do so, subject to the following conditions:
+//
+// The above copyright notice and this permission notice shall be included in
+// all copies or substantial portions of the Software.
+//
+// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+// THE SOFTWARE.
+
+#include "lib/common/xml.hpp"
+
+#include <glog/logging.h>
+
+namespace rocprofiler
+{
+namespace common
+{
+Xml::Xml(std::string file_name, const Xml* obj)
+: file_name_(std::move(file_name))
+, state_(BODY_STATE)
+{
+    if(obj != nullptr)
+    {
+        map_      = obj->map_;
+        level_    = obj->level_;
+        included_ = true;
+    }
+}
+
+Xml::~Xml()
+{
+    for(auto& x : stack_)
+    {
+        x->nodes.clear();
+        x->copy.reset();
+    }
+    if(!map_) return;
+    for(auto& [_, nodes] : *map_)
+    {
+        for(auto& node : nodes)
+        {
+            node->nodes.clear();
+            node->copy.reset();
+        }
+    }
+}
+
+std::shared_ptr<Xml>
+Xml::Create(const std::string& file_name, const Xml* obj)
+{
+    auto xml = std::make_shared<Xml>(file_name, obj);
+    if(xml != nullptr)
+    {
+        if(xml->Init() != false)
+        {
+            const std::size_t pos  = file_name.rfind('/');
+            const std::string path = (pos != std::string::npos) ? file_name.substr(0, pos + 1) : "";
+
+            xml->PreProcess();
+            nodes_t incl_nodes;
+            for(const auto& node : xml->GetNodes("top.include"))
+            {
+                if(node->opts.find("touch") == node->opts.end())
+                {
+                    node->opts["touch"] = "";
+                    incl_nodes.push_back(node);
+                }
+            }
+            for(const auto& incl : incl_nodes)
+            {
+                const std::string& incl_name = path + incl->opts["file"];
+                auto               ixml      = Create(incl_name, xml.get());
+                if(!ixml)
+                {
+                    xml.reset();
+                    break;
+                }
+            }
+            if(xml)
+            {
+                xml->Process();
+            }
+        }
+    }
+
+    return xml;
+}
+
+void
+Xml::AddExpr(const std::string& full_tag, const std::string& name, const std::string& expr)
+{
+    const std::size_t pos       = full_tag.rfind('.');
+    const std::size_t pos1      = (pos == std::string::npos) ? 0 : pos + 1;
+    const std::string level_tag = full_tag.substr(pos1);
+    auto              level     = std::make_shared<level_t>();
+    (*map_)[full_tag].push_back(level);
+    level->tag          = level_tag;
+    level->opts["name"] = name;
+    level->opts["expr"] = expr;
+}
+
+void
+Xml::AddConst(const std::string& full_tag, const std::string& name, const uint64_t& val)
+{
+    std::ostringstream oss;
+    oss << val;
+    AddExpr(full_tag, name, oss.str());
+}
+
+bool
+Xml::print_func::operator()(const std::string& global_tag, const std::shared_ptr<level_t>& node)
+{
+    std::cout << global_tag << ":\n";
+    for(auto& opt : node->opts)
+    {
+        std::cout << global_tag << "." << opt.first << " = " << opt.second << "\n";
+    }
+    return true;
+}
+
+void
+Xml::Print() const
+{
+    std::cout << "XML file '" << file_name_ << "':\n";
+    ForEach(print_func{});
+}
+
+bool
+Xml::Init()
+{
+    fd_ = open(file_name_.c_str(), O_RDONLY);
+    if(fd_ == -1)
+    {
+        // perror((std::string("open XML file ") + file_name_).c_str());
+        return false;
+    }
+
+    if(map_ == nullptr)
+    {
+        map_ = std::make_unique<map_t>();
+        AddLevel("top");
+    }
+
+    return true;
+}
+
+void
+Xml::PreProcess()
+{
+    uint32_t ind = 0;
+    char     buf[kBufSize];
+    bool     error = false;
+
+    while(true)
+    {
+        const uint32_t pos  = lseek(fd_, 0, SEEK_CUR);
+        uint32_t       size = read(fd_, buf, kBufSize);
+        if(size <= 0) break;
+        buf[size - 1] = '\0';
+
+        if(strncmp(buf, "#include \"", 10) == 0)
+        {
+            for(ind = 0; (ind < size) && (buf[ind] != '\n'); ++ind)
+            {}
+            if(ind < size)
+            {
+                buf[ind] = '\0';
+                size     = ind;
+                lseek(fd_, pos + ind + 1, SEEK_SET);
+            }
+
+            for(ind = 10; (ind < size) && (buf[ind] != '"'); ++ind)
+            {}
+            if(ind == size)
+            {
+                error = true;
+                break;
+            }
+            buf[ind] = '\0';
+
+            AddLevel("include");
+            AddOption("file", &buf[10]);
+            UpLevel();
+        }
+    }
+
+    if(error)
+    {
+        fprintf(stderr, "XML PreProcess failed, line '%s'\n", buf);
+        abort();
+    }
+
+    lseek(fd_, 0, SEEK_SET);
+}
+
+void
+Xml::Process()
+{
+    token_t remainder;
+
+    while(true)
+    {
+        token_t token = (!remainder.empty()) ? remainder : NextToken();
+        remainder.clear();
+
+        // token_t token1 = token;
+        // token1.push_back('\0');
+        // std::cout << ">>> " << &token1[0] << std::endl;
+
+        // End of file
+        if(token.empty()) break;
+
+        switch(state_)
+        {
+            case BODY_STATE:
+                if(token[0] == '<')
+                {
+                    bool     node_begin = true;
+                    unsigned ind        = 1;
+                    if(token[1] == '/')
+                    {
+                        node_begin = false;
+                        ++ind;
+                    }
+
+                    unsigned i = ind;
+                    while(i < token.size())
+                    {
+                        if(token[i] == '>') break;
+                        ++i;
+                    }
+                    for(unsigned j = i + 1; j < token.size(); ++j)
+                        remainder.push_back(token[j]);
+
+                    if(i == token.size())
+                    {
+                        if(node_begin)
+                            state_ = DECL_STATE;
+                        else
+                            BadFormat(token);
+                        token.push_back('\0');
+                    }
+                    else
+                    {
+                        token[i] = '\0';
+                    }
+
+                    const char* tag = &token[ind];
+                    if(node_begin)
+                    {
+                        AddLevel(tag);
+                    }
+                    else
+                    {
+                        Inherit(GetOption("base"));
+
+                        if(strncmp(CurrentLevel().c_str(), tag, strlen(tag)) != 0)
+                        {
+                            token.back() = '>';
+                            BadFormat(token);
+                        }
+                        UpLevel();
+                    }
+                }
+                else
+                {
+                    BadFormat(token);
+                }
+                break;
+            case DECL_STATE:
+                if(token[0] == '>')
+                {
+                    state_ = BODY_STATE;
+                    for(unsigned j = 1; j < token.size(); ++j)
+                        remainder.push_back(token[j]);
+                    continue;
+                }
+                else
+                {
+                    token.push_back('\0');
+                    unsigned j = 0;
+                    for(j = 0; j < token.size(); ++j)
+                        if(token[j] == '=') break;
+                    if(j == token.size()) BadFormat(token);
+                    token[j]                = '\0';
+                    const std::string key   = token.data();
+                    const std::string value = &token[j + 1];
+                    AddOption(key, value);
+                }
+                break;
+            default:
+            {
+                LOG(ERROR) << "XML parser error: wrong state: " << state_;
+                abort();
+            }
+        }
+    }
+}
+
+bool
+Xml::SpaceCheck() const
+{
+    bool cond = ((buffer_[index_] == ' ') || (buffer_[index_] == '\t'));
+    return cond;
+}
+
+bool
+Xml::LineEndCheck()
+{
+    bool found = false;
+    if(buffer_[index_] == '\n')
+    {
+        buffer_[index_] = ' ';
+        ++file_line_;
+        found    = true;
+        comment_ = false;
+    }
+    else if(comment_ || (buffer_[index_] == '#'))
+    {
+        found    = true;
+        comment_ = true;
+    }
+    return found;
+}
+
+Xml::token_t
+Xml::NextToken()
+{
+    token_t token;
+    bool    in_string    = false;
+    bool    special_symb = false;
+
+    while(true)
+    {
+        if(data_size_ == 0)
+        {
+            data_size_ = read(fd_, buffer_, kBufSize);
+            if(data_size_ <= 0) break;
+        }
+
+        if(token.empty())
+        {
+            while((index_ < data_size_) && (SpaceCheck() || LineEndCheck()))
+            {
+                ++index_;
+            }
+        }
+        while((index_ < data_size_) && (in_string || !(SpaceCheck() || LineEndCheck())))
+        {
+            const char symb      = buffer_[index_];
+            bool       skip_symb = false;
+
+            switch(symb)
+            {
+                case '\\':
+                    if(special_symb)
+                    {
+                        special_symb = false;
+                    }
+                    else
+                    {
+                        special_symb = true;
+                        skip_symb    = true;
+                    }
+                    break;
+                case '"':
+                    if(special_symb)
+                    {
+                        special_symb = false;
+                    }
+                    else
+                    {
+                        in_string = !in_string;
+                        if(!in_string)
+                        {
+                            buffer_[index_] = ' ';
+                            --index_;
+                        }
+                        skip_symb = true;
+                    }
+                    break;
+            }
+
+            if(!skip_symb) token.push_back(symb);
+            ++index_;
+        }
+
+        if(index_ == data_size_)
+        {
+            index_     = 0;
+            data_size_ = 0;
+        }
+        else
+        {
+            if(special_symb || in_string) BadFormat(token);
+            break;
+        }
+    }
+
+    return token;
+}
+
+void
+Xml::BadFormat(token_t token)
+{
+    token.push_back('\0');
+    LOG(ERROR) << "Error: " << file_name_ << ", line " << file_line_ << ", bad XML token '"
+               << token.data() << "'";
+    abort();
+}
+
+void
+Xml::AddLevel(const std::string& tag)
+{
+    auto level = std::make_shared<level_t>();
+    level->tag = tag;
+    if(level_)
+    {
+        level_->nodes.push_back(level);
+        stack_.push_back(level_);
+    }
+    level_ = level;
+
+    std::string global_tag = GlobalTag(tag);
+    (*map_)[global_tag].push_back(level_);
+}
+
+void
+Xml::UpLevel()
+{
+    level_ = stack_.back();
+    stack_.pop_back();
+}
+
+void
+Xml::Copy(const std::shared_ptr<level_t>& from, const std::shared_ptr<level_t>& to)
+{
+    auto level = to;
+    if(level == nullptr)
+    {
+        AddLevel(from->tag);
+        level = level_;
+    }
+    level->copy = from;
+    level->opts = from->opts;
+
+    for(const auto& node : from->nodes)
+    {
+        bool              found      = false;
+        const std::string name       = GetOption("name", node);
+        const std::string global_tag = GlobalTag(level->tag) + "." + node->tag;
+        for(const auto& item : (*map_)[global_tag])
+        {
+            if((name == GetOption("name", item)) || (node == item->copy))
+            {
+                found = true;
+                break;
+            }
+        }
+        if(found == false) Copy(node, nullptr);
+    }
+
+    if(to == nullptr) UpLevel();
+}
+
+void
+Xml::Inherit(const std::string& tag)
+{
+    if(!tag.empty())
+    {
+        const std::string global_tag = GlobalTag(tag);
+        auto              it         = map_->find(global_tag);
+        if(it == map_->end())
+        {
+            fprintf(
+                stderr, "Node \"%s\": Base not found \"%s\"\n", level_->tag.c_str(), tag.c_str());
+            abort();
+        }
+        for(const auto& node : it->second)
+        {
+            Copy(node, level_);
+        }
+    }
+}
+
+std::string
+Xml::CurrentLevel() const
+{
+    return level_->tag;
+}
+
+std::string
+Xml::GlobalTag(const std::string& tag) const
+{
+    std::string global_tag;
+    for(const auto& level : stack_)
+    {
+        global_tag += level->tag + ".";
+    }
+    global_tag += tag;
+    return global_tag;
+}
+
+void
+Xml::AddOption(const std::string& key, const std::string& value)
+{
+    level_->opts[key] = value;
+}
+
+std::string
+Xml::GetOption(const std::string& key, std::shared_ptr<const level_t> level)
+{
+    level   = (level != nullptr) ? level : level_;
+    auto it = level->opts.find(key);
+    return (it != level->opts.end()) ? it->second : "";
+}
+}  // namespace common
+}  // namespace rocprofiler
@@ -1,24 +1,22 @@
-/******************************************************************************
-Copyright (c) 2023 Advanced Micro Devices, Inc. All rights reserved.
-
-Permission is hereby granted, free of charge, to any person obtaining a copy
-of this software and associated documentation files (the "Software"), to deal
-in the Software without restriction, including without limitation the rights
-to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-copies of the Software, and to permit persons to whom the Software is
-furnished to do so, subject to the following conditions:
-
-The above copyright notice and this permission notice shall be included in
-all copies or substantial portions of the Software.
-
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL THE
-AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
-THE SOFTWARE.
-*******************************************************************************/
+// Copyright (c) 2023 Advanced Micro Devices, Inc.
+//
+// Permission is hereby granted, free of charge, to any person obtaining a copy
+// of this software and associated documentation files (the "Software"), to deal
+// in the Software without restriction, including without limitation the rights
+// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+// copies of the Software, and to permit persons to whom the Software is
+// furnished to do so, subject to the following conditions:
+//
+// The above copyright notice and this permission notice shall be included in
+// all copies or substantial portions of the Software.
+//
+// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+// THE SOFTWARE.

 #pragma once

@@ -34,10 +32,13 @@ THE SOFTWARE.
 #include <list>
 #include <map>
 #include <memory>
+#include <sstream>
 #include <string>
 #include <vector>

-namespace xml
+namespace rocprofiler
+{
+namespace common
 {
 class Xml
 {
@@ -66,525 +67,45 @@ public:
        BODY_STATE
    };

-    static std::shared_ptr<Xml> Create(const std::string& file_name, const Xml* obj = nullptr)
-    {
-        auto xml = std::make_shared<Xml>(file_name, obj);
-        if(xml != nullptr)
-        {
-            if(xml->Init() != false)
-            {
-                const std::size_t pos = file_name.rfind('/');
-                const std::string path =
-                    (pos != std::string::npos) ? file_name.substr(0, pos + 1) : "";
-
-                xml->PreProcess();
-                nodes_t incl_nodes;
-                for(const auto& node : xml->GetNodes("top.include"))
-                {
-                    if(node->opts.find("touch") == node->opts.end())
-                    {
-                        node->opts["touch"] = "";
-                        incl_nodes.push_back(node);
-                    }
-                }
-                for(const auto& incl : incl_nodes)
-                {
-                    const std::string& incl_name = path + incl->opts["file"];
-                    auto               ixml      = Create(incl_name, xml.get());
-                    if(!ixml)
-                    {
-                        xml.reset();
-                        break;
-                    }
-                }
-                if(xml)
-                {
-                    xml->Process();
-                }
-            }
-        }
-
-        return xml;
-    }
+    static std::shared_ptr<Xml> Create(const std::string& file_name, const Xml* obj = nullptr);

    std::string GetName() { return file_name_; }

-    // clang-tidy incorrectly marks these functions as being staticable. They are not.
-    // NOLINTBEGIN
-    void AddExpr(const std::string& full_tag, const std::string& name, const std::string& expr)
-    {
-        const std::size_t pos       = full_tag.rfind('.');
-        const std::size_t pos1      = (pos == std::string::npos) ? 0 : pos + 1;
-        const std::string level_tag = full_tag.substr(pos1);
-        auto              level     = std::make_shared<level_t>();
-        (*map_)[full_tag].push_back(level);
-        level->tag          = level_tag;
-        level->opts["name"] = name;
-        level->opts["expr"] = expr;
-    }
-
-    void AddConst(const std::string& full_tag, const std::string& name, const uint64_t& val)
-    {
-        std::ostringstream oss;
-        oss << val;
-        AddExpr(full_tag, name, oss.str());
-    }
-    // NOLINTEND
+    void AddExpr(const std::string& full_tag, const std::string& name, const std::string& expr);
+    void AddConst(const std::string& full_tag, const std::string& name, const uint64_t& val);

    nodes_t      GetNodes(const std::string& global_tag) { return (*map_)[global_tag]; }
    const map_t& GetAllNodes() { return (*map_); }

-    template <class F>
-    F ForEach(const F& f_i)
-    {
-        F f = f_i;
-        if(map_)
-        {
-            for(auto& entry : *map_)
-            {
-                for(auto node : entry.second)
-                {
-                    if(f.fun(entry.first, node) == false) break;
-                }
-            }
-        }
-        return f;
-    }
-
-    template <class F>
-    F ForEach(const F& f_i) const
-    {
-        F f = f_i;
-        if(map_)
-        {
-            for(auto& entry : *map_)
-            {
-                for(const auto& node : entry.second)
-                {
-                    if(f.fun(entry.first, node) == false) break;
-                }
-            }
-        }
-        return f;
-    }
+    template <typename Tp>
+    Tp ForEach(const Tp& v_i) const;

    struct print_func
    {
-        static bool fun(const std::string& global_tag, const std::shared_ptr<level_t>& node)
-        {
-            std::cout << global_tag << ":" << std::endl;
-            for(auto& opt : node->opts)
-            {
-                std::cout << global_tag << "." << opt.first << " = " << opt.second << std::endl;
-            }
-            return true;
-        }
+        bool operator()(const std::string& global_tag, const std::shared_ptr<level_t>& node);
    };

-    void Print() const
-    {
-        std::cout << "XML file '" << file_name_ << "':" << std::endl;
-        ForEach(print_func());
-    }
+    void Print() const;

-    Xml(std::string file_name, const Xml* obj)
-    : file_name_(std::move(file_name))
-    , state_(BODY_STATE)
-    {
-        if(obj != nullptr)
-        {
-            map_      = obj->map_;
-            level_    = obj->level_;
-            included_ = true;
-        }
-    }
-
-    ~Xml()
-    {
-        for(auto& x : stack_)
-        {
-            x->nodes.clear();
-            x->copy.reset();
-        }
-        if(!map_) return;
-        for(auto& [_, nodes] : *map_)
-        {
-            for(auto& node : nodes)
-            {
-                node->nodes.clear();
-                node->copy.reset();
-            }
-        }
-    }
+    Xml(std::string file_name, const Xml* obj);
+    ~Xml();

 private:
-    bool Init()
-    {
-        fd_ = open(file_name_.c_str(), O_RDONLY);
-        if(fd_ == -1)
-        {
-            // perror((std::string("open XML file ") + file_name_).c_str());
-            return false;
-        }
-
-        if(map_ == nullptr)
-        {
-            map_ = std::make_unique<map_t>();
-            AddLevel("top");
-        }
-
-        return true;
-    }
-
-    void PreProcess()
-    {
-        uint32_t ind = 0;
-        char     buf[kBufSize];
-        bool     error = false;
-
-        while(true)
-        {
-            const uint32_t pos  = lseek(fd_, 0, SEEK_CUR);
-            uint32_t       size = read(fd_, buf, kBufSize);
-            if(size <= 0) break;
-            buf[size - 1] = '\0';
-
-            if(strncmp(buf, "#include \"", 10) == 0)
-            {
-                for(ind = 0; (ind < size) && (buf[ind] != '\n'); ++ind)
-                {}
-                if(ind < size)
-                {
-                    buf[ind] = '\0';
-                    size     = ind;
-                    lseek(fd_, pos + ind + 1, SEEK_SET);
-                }
-
-                for(ind = 10; (ind < size) && (buf[ind] != '"'); ++ind)
-                {}
-                if(ind == size)
-                {
-                    error = true;
-                    break;
-                }
-                buf[ind] = '\0';
-
-                AddLevel("include");
-                AddOption("file", &buf[10]);
-                UpLevel();
-            }
-        }
-
-        if(error)
-        {
-            fprintf(stderr, "XML PreProcess failed, line '%s'\n", buf);
-            abort();
-        }
-
-        lseek(fd_, 0, SEEK_SET);
-    }
-
-    void Process()
-    {
-        token_t remainder;
-
-        while(true)
-        {
-            token_t token = (!remainder.empty()) ? remainder : NextToken();
-            remainder.clear();
-
-            // token_t token1 = token;
-            // token1.push_back('\0');
-            // std::cout << ">>> " << &token1[0] << std::endl;
-
-            // End of file
-            if(token.empty()) break;
-
-            switch(state_)
-            {
-                case BODY_STATE:
-                    if(token[0] == '<')
-                    {
-                        bool     node_begin = true;
-                        unsigned ind        = 1;
-                        if(token[1] == '/')
-                        {
-                            node_begin = false;
-                            ++ind;
-                        }
-
-                        unsigned i = ind;
-                        while(i < token.size())
-                        {
-                            if(token[i] == '>') break;
-                            ++i;
-                        }
-                        for(unsigned j = i + 1; j < token.size(); ++j)
-                            remainder.push_back(token[j]);
-
-                        if(i == token.size())
-                        {
-                            if(node_begin)
-                                state_ = DECL_STATE;
-                            else
-                                BadFormat(token);
-                            token.push_back('\0');
-                        }
-                        else
-                        {
-                            token[i] = '\0';
-                        }
-
-                        const char* tag = &token[ind];
-                        if(node_begin)
-                        {
-                            AddLevel(tag);
-                        }
-                        else
-                        {
-                            Inherit(GetOption("base"));
-
-                            if(strncmp(CurrentLevel().c_str(), tag, strlen(tag)) != 0)
-                            {
-                                token.back() = '>';
-                                BadFormat(token);
-                            }
-                            UpLevel();
-                        }
-                    }
-                    else
-                    {
-                        BadFormat(token);
-                    }
-                    break;
-                case DECL_STATE:
-                    if(token[0] == '>')
-                    {
-                        state_ = BODY_STATE;
-                        for(unsigned j = 1; j < token.size(); ++j)
-                            remainder.push_back(token[j]);
-                        continue;
-                    }
-                    else
-                    {
-                        token.push_back('\0');
-                        unsigned j = 0;
-                        for(j = 0; j < token.size(); ++j)
-                            if(token[j] == '=') break;
-                        if(j == token.size()) BadFormat(token);
-                        token[j]                = '\0';
-                        const std::string key   = token.data();
-                        const std::string value = &token[j + 1];
-                        AddOption(key, value);
-                    }
-                    break;
-                default:
-                    std::cout << "XML parser error: wrong state: " << state_ << std::endl;
-                    abort();
-            }
-        }
-    }
-
-    bool SpaceCheck() const
-    {
-        bool cond = ((buffer_[index_] == ' ') || (buffer_[index_] == '\t'));
-        return cond;
-    }
-
-    bool LineEndCheck()
-    {
-        bool found = false;
-        if(buffer_[index_] == '\n')
-        {
-            buffer_[index_] = ' ';
-            ++file_line_;
-            found    = true;
-            comment_ = false;
-        }
-        else if(comment_ || (buffer_[index_] == '#'))
-        {
-            found    = true;
-            comment_ = true;
-        }
-        return found;
-    }
-
-    token_t NextToken()
-    {
-        token_t token;
-        bool    in_string    = false;
-        bool    special_symb = false;
-
-        while(true)
-        {
-            if(data_size_ == 0)
-            {
-                data_size_ = read(fd_, buffer_, kBufSize);
-                if(data_size_ <= 0) break;
-            }
-
-            if(token.empty())
-            {
-                while((index_ < data_size_) && (SpaceCheck() || LineEndCheck()))
-                {
-                    ++index_;
-                }
-            }
-            while((index_ < data_size_) && (in_string || !(SpaceCheck() || LineEndCheck())))
-            {
-                const char symb      = buffer_[index_];
-                bool       skip_symb = false;
-
-                switch(symb)
-                {
-                    case '\\':
-                        if(special_symb)
-                        {
-                            special_symb = false;
-                        }
-                        else
-                        {
-                            special_symb = true;
-                            skip_symb    = true;
-                        }
-                        break;
-                    case '"':
-                        if(special_symb)
-                        {
-                            special_symb = false;
-                        }
-                        else
-                        {
-                            in_string = !in_string;
-                            if(!in_string)
-                            {
-                                buffer_[index_] = ' ';
-                                --index_;
-                            }
-                            skip_symb = true;
-                        }
-                        break;
-                }
-
-                if(!skip_symb) token.push_back(symb);
-                ++index_;
-            }
-
-            if(index_ == data_size_)
-            {
-                index_     = 0;
-                data_size_ = 0;
-            }
-            else
-            {
-                if(special_symb || in_string) BadFormat(token);
-                break;
-            }
-        }
-
-        return token;
-    }
-
-    void BadFormat(token_t token)
-    {
-        token.push_back('\0');
-        std::cout << "Error: " << file_name_ << ", line " << file_line_ << ", bad XML token '"
-                  << token.data() << "'" << std::endl;
-        abort();
-    }
-
-    void AddLevel(const std::string& tag)
-    {
-        auto level = std::make_shared<level_t>();
-        level->tag = tag;
-        if(level_)
-        {
-            level_->nodes.push_back(level);
-            stack_.push_back(level_);
-        }
-        level_ = level;
-
-        std::string global_tag = GlobalTag(tag);
-        (*map_)[global_tag].push_back(level_);
-    }
-
-    void UpLevel()
-    {
-        level_ = stack_.back();
-        stack_.pop_back();
-    }
-
-    void Copy(const std::shared_ptr<level_t>& from, const std::shared_ptr<level_t>& to)
-    {
-        auto level = to;
-        if(level == nullptr)
-        {
-            AddLevel(from->tag);
-            level = level_;
-        }
-        level->copy = from;
-        level->opts = from->opts;
-
-        for(const auto& node : from->nodes)
-        {
-            bool              found      = false;
-            const std::string name       = GetOption("name", node);
-            const std::string global_tag = GlobalTag(level->tag) + "." + node->tag;
-            for(const auto& item : (*map_)[global_tag])
-            {
-                if((name == GetOption("name", item)) || (node == item->copy))
-                {
-                    found = true;
-                    break;
-                }
-            }
-            if(found == false) Copy(node, nullptr);
-        }
-
-        if(to == nullptr) UpLevel();
-    }
-
-    void Inherit(const std::string& tag)
-    {
-        if(!tag.empty())
-        {
-            const std::string global_tag = GlobalTag(tag);
-            auto              it         = map_->find(global_tag);
-            if(it == map_->end())
-            {
-                fprintf(stderr,
-                        "Node \"%s\": Base not found \"%s\"\n",
-                        level_->tag.c_str(),
-                        tag.c_str());
-                abort();
-            }
-            for(const auto& node : it->second)
-            {
-                Copy(node, level_);
-            }
-        }
-    }
-
-    std::string CurrentLevel() const { return level_->tag; }
-
-    std::string GlobalTag(const std::string& tag) const
-    {
-        std::string global_tag;
-        for(const auto& level : stack_)
-        {
-            global_tag += level->tag + ".";
-        }
-        global_tag += tag;
-        return global_tag;
-    }
-
-    void AddOption(const std::string& key, const std::string& value) { level_->opts[key] = value; }
-    std::string GetOption(const std::string& key, std::shared_ptr<const level_t> level = nullptr)
-    {
-        level   = (level != nullptr) ? level : level_;
-        auto it = level->opts.find(key);
-        return (it != level->opts.end()) ? it->second : "";
-    }
+    bool        Init();
+    void        PreProcess();
+    void        Process();
+    bool        SpaceCheck() const;
+    bool        LineEndCheck();
+    token_t     NextToken();
+    void        BadFormat(token_t token);
+    void        AddLevel(const std::string& tag);
+    void        UpLevel();
+    void        Copy(const std::shared_ptr<level_t>& from, const std::shared_ptr<level_t>& to);
+    void        Inherit(const std::string& tag);
+    std::string CurrentLevel() const;
+    std::string GlobalTag(const std::string& tag) const;
+    void        AddOption(const std::string& key, const std::string& value);
+    std::string GetOption(const std::string& key, std::shared_ptr<const level_t> level = nullptr);

    const std::string file_name_;
    unsigned          file_line_{0};
@@ -603,4 +124,22 @@ private:
    std::shared_ptr<map_t>                map_;
 };

-}  // namespace xml
+template <typename Tp>
+Tp
+Xml::ForEach(const Tp& v_i) const
+{
+    Tp v = v_i;
+    if(map_)
+    {
+        for(auto& entry : *map_)
+        {
+            for(const auto& node : entry.second)
+            {
+                if(Tp{}(entry.first, node) == false) break;
+            }
+        }
+    }
+    return v;
+}
+}  // namespace common
+}  // namespace rocprofiler
@@ -10,8 +10,11 @@ set(ROCPROFILER_LIB_SOURCES
    buffer_tracing.cpp
    callback_tracing.cpp
    context.cpp
+    counters.cpp
+    dispatch_profile.cpp
    internal_threading.cpp
    pc_sampling.cpp
+    profile_config.cpp
    rocprofiler.cpp
    registration.cpp)

@@ -31,6 +34,7 @@ target_sources(rocprofiler-object-library PRIVATE ${ROCPROFILER_LIB_SOURCES}
 add_subdirectory(hsa)
 add_subdirectory(context)
 add_subdirectory(counters)
+add_subdirectory(aql)

 target_link_libraries(
    rocprofiler-object-library
@@ -59,9 +63,13 @@ target_link_libraries(
    rocprofiler-shared-library
    INTERFACE rocprofiler::rocprofiler-headers rocprofiler::rocprofiler-hsa-runtime
              rocprofiler::rocprofiler-hip
-    PRIVATE rocprofiler::rocprofiler-build-flags rocprofiler::rocprofiler-memcheck
-            rocprofiler::rocprofiler-common-library rocprofiler::rocprofiler-stdcxxfs
-            rocprofiler::rocprofiler-dl rocprofiler::rocprofiler-amd-comgr)
+    PRIVATE rocprofiler::rocprofiler-build-flags
+            rocprofiler::rocprofiler-memcheck
+            rocprofiler::rocprofiler-common-library
+            rocprofiler::rocprofiler-stdcxxfs
+            rocprofiler::rocprofiler-dl
+            rocprofiler::rocprofiler-amd-comgr
+            rocprofiler::rocprofiler-object-library)

 set_target_properties(
    rocprofiler-shared-library
@@ -97,7 +105,8 @@ target_link_libraries(
    rocprofiler-static-library
    PUBLIC rocprofiler::rocprofiler-headers rocprofiler::rocprofiler-hsa-runtime
           rocprofiler::rocprofiler-hip
-    PRIVATE rocprofiler::rocprofiler-common-library)
+    PRIVATE rocprofiler::rocprofiler-common-library
+            rocprofiler::rocprofiler-object-library)

 set_target_properties(
    rocprofiler-static-library PROPERTIES OUTPUT_NAME rocprofiler64 DEFINE_SYMBOL
@@ -0,0 +1,9 @@
+set(ROCPROFILER_LIB_AQL_SOURCES helpers.cpp packet_construct.cpp)
+set(ROCPROFILER_LIB_AQL_HEADERS helpers.hpp packet_construct.hpp)
+
+target_sources(rocprofiler-object-library PRIVATE ${ROCPROFILER_LIB_AQL_SOURCES}
+                                                  ${ROCPROFILER_LIB_AQL_HEADERS})
+
+if(ROCPROFILER_BUILD_TESTS)
+    add_subdirectory(tests)
+endif()
@@ -0,0 +1,43 @@
+#include "lib/rocprofiler/aql/helpers.hpp"
+
+#include <fmt/core.h>
+#include <glog/logging.h>
+
+namespace rocprofiler
+{
+namespace aql
+{
+hsa_ven_amd_aqlprofile_id_query_t
+get_query_info(hsa_agent_t agent, const counters::Metric& metric)
+{
+    DLOG(WARNING) << fmt::format("Querying HSA for Counter: {}", metric);
+
+    hsa_ven_amd_aqlprofile_profile_t  profile{.agent = agent};
+    hsa_ven_amd_aqlprofile_id_query_t query = {metric.block().c_str(), 0, 0};
+    if(hsa_ven_amd_aqlprofile_get_info(&profile, HSA_VEN_AMD_AQLPROFILE_INFO_BLOCK_ID, &query) !=
+       HSA_STATUS_SUCCESS)
+    {
+        throw std::runtime_error(fmt::format("AQL failed to query info for counter {}", metric));
+    }
+    return query;
+}
+
+uint32_t
+get_block_counters(hsa_agent_t agent, const hsa_ven_amd_aqlprofile_event_t& event)
+{
+    hsa_ven_amd_aqlprofile_profile_t query              = {.agent       = agent,
+                                              .type        = HSA_VEN_AMD_AQLPROFILE_EVENT_TYPE_PMC,
+                                              .events      = &event,
+                                              .event_count = 1};
+    uint32_t                         max_block_counters = 0;
+    if(hsa_ven_amd_aqlprofile_get_info(&query,
+                                       HSA_VEN_AMD_AQLPROFILE_INFO_BLOCK_COUNTERS,
+                                       &max_block_counters) != HSA_STATUS_SUCCESS)
+    {
+        throw std::runtime_error(fmt::format("AQL failed to max block info for counter {}",
+                                             static_cast<int64_t>(event.block_name)));
+    }
+    return max_block_counters;
+}
+}  // namespace aql
+}  // namespace rocprofiler
@@ -0,0 +1,22 @@
+
+#pragma once
+
+#include <functional>
+
+#include <hsa/hsa_ven_amd_aqlprofile.h>
+
+#include "lib/rocprofiler/counters/metrics.hpp"
+
+namespace rocprofiler
+{
+namespace aql
+{
+// Query HSA_VEN_AMD_AQLPROFILE_INFO_BLOCK_ID from aqlprofile
+hsa_ven_amd_aqlprofile_id_query_t
+get_query_info(hsa_agent_t agent, const counters::Metric& metric);
+
+// Query HSA_VEN_AMD_AQLPROFILE_INFO_BLOCK_COUNTERS from aqlprofiler
+uint32_t
+get_block_counters(hsa_agent_t agent, const hsa_ven_amd_aqlprofile_event_t& event);
+}  // namespace aql
+}  // namespace rocprofiler
@@ -0,0 +1,23 @@
+#include "lib/rocprofiler/aql/intercept.hpp"
+
+#include "lib/rocprofiler/hsa/hsa.hpp"
+
+namespace rocprofiler
+{
+namespace aql
+{
+std::shared_ptr<const Intercept>
+Intercept::create(const std::function<void(HsaApiTable&)>& mod_cb)
+{
+    return std::make_shared<const Intercept>(mod_cb);
+}
+
+Intercept::Intercept(const std::function<void(HsaApiTable&)>& mod_cb)
+: _original(rocprofiler::hsa::get_table())
+, _modified(rocprofiler::hsa::get_table())
+{
+    mod_cb(_modified);
+};
+
+}  // namespace aql
+}  // namespace rocprofiler
@@ -0,0 +1,27 @@
+#pragma once
+
+#include <functional>
+#include <memory>
+
+#include <hsa/hsa.h>
+#include <hsa/hsa_api_trace.h>
+#include <hsa/hsa_ven_amd_aqlprofile.h>
+
+namespace rocprofiler
+{
+namespace aql
+{
+class Intercept
+{
+public:
+    static std::shared_ptr<const Intercept> create(const std::function<void(HsaApiTable&)>& mod_cb);
+
+    explicit Intercept(const std::function<void(HsaApiTable&)>& mod_cb);
+
+private:
+    HsaApiTable  _original;
+    HsaApiTable& _modified;
+};
+
+}  // namespace aql
+}  // namespace rocprofiler
@@ -0,0 +1,189 @@
+#include "lib/rocprofiler/aql/packet_construct.hpp"
+
+#include <fmt/core.h>
+#include <hsa/hsa_ext_amd.h>
+#include "glog/logging.h"
+
+namespace rocprofiler
+{
+namespace aql
+{
+AQLPacketConstruct::AQLPacketConstruct(const hsa::AgentCache&               agent,
+                                       const std::vector<counters::Metric>& metrics)
+: _agent(agent)
+{
+    if(metrics.empty())
+    {
+        throw std::runtime_error("No metrics supplied");
+    }
+
+    // Validate that the counter exists and construct the block instances
+    // for the counter.
+    for(const auto& x : metrics)
+    {
+        auto query_info                = get_query_info(_agent.get_agent(), x);
+        _metrics.emplace_back().metric = x;
+        uint32_t event_id              = std::atoi(x.event().c_str());
+        for(unsigned block_index = 0; block_index < query_info.instance_count; ++block_index)
+        {
+            _metrics.back().instances.push_back(
+                {static_cast<hsa_ven_amd_aqlprofile_block_name_t>(query_info.id),
+                 block_index,
+                 event_id});
+            bool validate_event_result;
+            LOG_IF(FATAL,
+                   hsa_ven_amd_aqlprofile_validate_event(_agent.get_agent(),
+                                                         &_metrics.back().instances.back(),
+                                                         &validate_event_result) !=
+                       HSA_STATUS_SUCCESS);
+            LOG_IF(FATAL, !validate_event_result)
+                << "Invalid Metric: " << block_index << " " << event_id;
+        }
+    }
+    // Check that we can collect all of the metrics in a single execution
+    // with a single AQL packet
+    can_collect();
+    _events = get_all_events();
+}
+
+std::unique_ptr<hsa::AQLPacket>
+AQLPacketConstruct::construct_packet(const AmdExtTable& ext) const
+{
+    const size_t MEM_PAGE_MASK = 0x1000 - 1;
+    auto         pkt_ptr       = std::make_unique<hsa::AQLPacket>(ext.hsa_amd_memory_pool_free_fn);
+    auto&        pkt           = *pkt_ptr;
+    if(_events.empty())
+    {
+        throw std::runtime_error("Constructing packet with no events");
+    }
+
+    pkt.profile = hsa_ven_amd_aqlprofile_profile_t{
+        _agent.get_agent(),
+        HSA_VEN_AMD_AQLPROFILE_EVENT_TYPE_PMC,  // SPM?
+        _events.data(),
+        static_cast<uint32_t>(_events.size()),
+        nullptr,
+        0u,
+        hsa_ven_amd_aqlprofile_descriptor_t{.ptr = nullptr, .size = 0},
+        hsa_ven_amd_aqlprofile_descriptor_t{.ptr = nullptr, .size = 0}};
+    auto& profile = pkt.profile;
+
+    hsa_amd_memory_pool_access_t _access = HSA_AMD_MEMORY_POOL_ACCESS_NEVER_ALLOWED;
+    ext.hsa_amd_agent_memory_pool_get_info_fn(_agent.get_agent(),
+                                              _agent.kernarg_pool(),
+                                              HSA_AMD_AGENT_MEMORY_POOL_INFO_ACCESS,
+                                              static_cast<void*>(&_access));
+    // Memory is accessable by both the GPU and CPU, unlock the command buffer for
+    // sharing.
+    if(_access == HSA_AMD_MEMORY_POOL_ACCESS_NEVER_ALLOWED)
+    {
+        throw std::runtime_error(
+            fmt::format("Agent {} does not allow memory pool access for counter collection",
+                        _agent.get_agent().handle));
+    }
+
+    auto throw_if_failed = [](auto status, auto& message) {
+        if(status != HSA_STATUS_SUCCESS)
+        {
+            throw std::runtime_error(message);
+        }
+    };
+
+    throw_if_failed(hsa_ven_amd_aqlprofile_start(&profile, nullptr),
+                    "could not generate packet sizes");
+
+    if(profile.command_buffer.size == 0 || profile.output_buffer.size == 0)
+    {
+        throw std::runtime_error(
+            fmt::format("No command or output buffer size set. CMD_BUF={} PROFILE_BUF={}",
+                        profile.command_buffer.size,
+                        profile.output_buffer.size));
+    }
+
+    // Allocate buffers and check the results
+    auto alloc_and_check = [&](auto& pool, auto** mem_loc, auto size) -> bool {
+        bool   malloced     = false;
+        size_t page_aligned = (size + MEM_PAGE_MASK) & ~MEM_PAGE_MASK;
+        if(ext.hsa_amd_memory_pool_allocate_fn(
+               pool, page_aligned, 0, static_cast<void**>(mem_loc)) != HSA_STATUS_SUCCESS)
+        {
+            *mem_loc = malloc(page_aligned);
+            malloced = true;
+        }
+        else
+        {
+            CHECK(*mem_loc);
+            hsa_agent_t agent = _agent.get_agent();
+            // Memory is accessable by both the GPU and CPU, unlock the command buffer for
+            // sharing.
+            LOG_IF(FATAL,
+                   ext.hsa_amd_agents_allow_access_fn(1, &agent, nullptr, *mem_loc) !=
+                       HSA_STATUS_SUCCESS)
+                << "Error: Allowing access to Command Buffer";
+        }
+        return malloced;
+    };
+
+    // Build command and output buffers
+    pkt.command_buf_mallocd = alloc_and_check(
+        _agent.cpu_pool(), &profile.command_buffer.ptr, profile.command_buffer.size);
+    pkt.output_buffer_malloced = alloc_and_check(
+        _agent.kernarg_pool(), &profile.output_buffer.ptr, profile.output_buffer.size);
+    memset(profile.output_buffer.ptr, 0x0, profile.output_buffer.size);
+
+    // throw if we do not construct the packets correctly.
+    throw_if_failed(hsa_ven_amd_aqlprofile_start(&profile, &pkt.start),
+                    "could not generate start packet");
+    throw_if_failed(hsa_ven_amd_aqlprofile_stop(&profile, &pkt.stop),
+                    "could not generate stop packet");
+    throw_if_failed(hsa_ven_amd_aqlprofile_read(&profile, &pkt.read),
+                    "could not generate read packet");
+    return pkt_ptr;
+}
+
+std::vector<hsa_ven_amd_aqlprofile_event_t>
+AQLPacketConstruct::get_all_events() const
+{
+    std::vector<hsa_ven_amd_aqlprofile_event_t> ret;
+    for(const auto& metric : _metrics)
+    {
+        ret.insert(ret.end(), metric.instances.begin(), metric.instances.end());
+    }
+    return ret;
+}
+
+void
+AQLPacketConstruct::can_collect()
+{
+    // Verify that the counters fit within harrdware limits
+    std::map<std::pair<hsa_ven_amd_aqlprofile_block_name_t, uint32_t>, int64_t> counter_count;
+    std::map<std::pair<hsa_ven_amd_aqlprofile_block_name_t, uint32_t>, int64_t> max_allowed;
+    for(auto& metric : _metrics)
+    {
+        for(auto& instance : metric.instances)
+        {
+            auto block_pair       = std::make_pair(instance.block_name, instance.block_index);
+            auto [iter, inserted] = counter_count.emplace(block_pair, 0);
+            iter->second++;
+            if(inserted)
+            {
+                max_allowed.emplace(block_pair, get_block_counters(_agent.get_agent(), instance));
+            }
+        }
+    }
+
+    // Check if the block count > max count
+    for(auto& [block_name, count] : counter_count)
+    {
+        if(auto* max = CHECK_NOTNULL(common::get_val(max_allowed, block_name)); count > *max)
+        {
+            throw std::runtime_error(
+                fmt::format("Block {} exceeds max number of hardware counters ({} > {})",
+                            static_cast<int64_t>(block_name.first),
+                            count,
+                            *max));
+        }
+    }
+}
+}  // namespace aql
+}  // namespace rocprofiler
@@ -0,0 +1,50 @@
+#pragma once
+
+#include <functional>
+#include <map>
+#include <vector>
+
+#include <hsa/hsa.h>
+#include <hsa/hsa_api_trace.h>
+#include <hsa/hsa_ven_amd_aqlprofile.h>
+
+#include "lib/rocprofiler/aql/helpers.hpp"
+#include "lib/rocprofiler/counters/metrics.hpp"
+#include "lib/rocprofiler/hsa/agent_cache.hpp"
+#include "lib/rocprofiler/hsa/queue.hpp"
+
+namespace rocprofiler
+{
+namespace aql
+{
+/**
+ * Class to construct AQL Packets for a specific agent and metric set.
+ * Thie class checks that the counters supplied are collectable on the
+ * agent in question (including making sure that they stay within block
+ * limits). construct_packet returns an AQLPacket class containing the
+ * consturcted start/stop/read packets along with allocated buffers needed
+ * to collect the counter data.
+ */
+class AQLPacketConstruct
+{
+public:
+    AQLPacketConstruct(const hsa::AgentCache& agent, const std::vector<counters::Metric>& metrics);
+    std::unique_ptr<hsa::AQLPacket> construct_packet(const AmdExtTable&) const;
+
+private:
+    struct AQLProfileMetric
+    {
+        counters::Metric                            metric;
+        std::vector<hsa_ven_amd_aqlprofile_event_t> instances;
+    };
+
+    std::vector<hsa_ven_amd_aqlprofile_event_t> get_all_events() const;
+    void                                        can_collect();
+
+    const hsa::AgentCache&                      _agent;
+    std::vector<AQLProfileMetric>               _metrics;
+    std::vector<hsa_ven_amd_aqlprofile_event_t> _events;
+};
+
+}  // namespace aql
+}  // namespace rocprofiler
@@ -0,0 +1,23 @@
+rocprofiler_deactivate_clang_tidy()
+
+include(GoogleTest)
+
+set(ROCPROFILER_LIB_AQL_TEST_SOURCES "aql_test.cpp")
+
+add_executable(aql-test)
+
+target_sources(aql-test PRIVATE ${ROCPROFILER_LIB_AQL_TEST_SOURCES})
+
+target_link_libraries(
+    aql-test
+    PRIVATE rocprofiler::rocprofiler-static-library rocprofiler::rocprofiler-glog
+            rocprofiler::rocprofiler-hip rocprofiler::rocprofiler-common-library
+            GTest::gtest GTest::gtest_main)
+
+gtest_add_tests(
+    TARGET aql-test
+    SOURCES ${ROCPROFILER_LIB_AQL_TEST_SOURCES}
+    TEST_LIST aql-test_TESTS
+    WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR})
+
+set_tests_properties(${aql-test_TESTS} PROPERTIES TIMEOUT 45 LABELS "unittests")
@@ -0,0 +1,150 @@
+#include <glog/logging.h>
+#include <gtest/gtest.h>
+
+#include <functional>
+#include <map>
+#include <unordered_set>
+
+#include <hsa/hsa.h>
+#include <hsa/hsa_api_trace.h>
+#include <hsa/hsa_ven_amd_aqlprofile.h>
+
+#include "lib/rocprofiler/aql/helpers.hpp"
+#include "lib/rocprofiler/aql/packet_construct.hpp"
+#include "lib/rocprofiler/counters/metrics.hpp"
+#include "lib/rocprofiler/hsa/agent_cache.hpp"
+#include "lib/rocprofiler/hsa/queue.hpp"
+
+namespace rocprofiler
+{
+AmdExtTable
+get_ext_table()
+{
+    return {.hsa_amd_memory_pool_get_info_fn       = hsa_amd_memory_pool_get_info,
+            .hsa_amd_agent_iterate_memory_pools_fn = hsa_amd_agent_iterate_memory_pools,
+            .hsa_amd_memory_pool_allocate_fn       = hsa_amd_memory_pool_allocate,
+            .hsa_amd_memory_pool_free_fn           = hsa_amd_memory_pool_free,
+            .hsa_amd_agent_memory_pool_get_info_fn = hsa_amd_agent_memory_pool_get_info,
+            .hsa_amd_agents_allow_access_fn        = hsa_amd_agents_allow_access};
+}
+
+auto
+findDeviceMetrics(const hsa::AgentCache& agent, const std::unordered_set<std::string>& metrics)
+{
+    std::vector<counters::Metric> ret;
+    auto                          all_counters = counters::getBaseHardwareMetrics();
+
+    auto gfx_metrics = common::get_val(all_counters, std::string(agent.name()));
+    if(!gfx_metrics) return ret;
+
+    for(auto& counter : *gfx_metrics)
+    {
+        if(metrics.count(counter.name()) > 0 || metrics.empty())
+        {
+            ret.push_back(counter);
+        }
+    }
+    return ret;
+}
+
+}  // namespace rocprofiler
+
+using namespace rocprofiler::aql;
+
+TEST(aql_profile, construct_packets)
+{
+    hsa_init();
+    try
+    {
+        auto agents = rocprofiler::hsa::get_queue_controller().get_supported_agents();
+        for(const auto& [_, agent] : agents)
+        {
+            LOG(WARNING) << fmt::format("Found Agent: {}", agent.get_agent().handle);
+            auto metrics = rocprofiler::findDeviceMetrics(agent, {"SQ_WAVES"});
+            ASSERT_EQ(metrics.size(), 1);
+            AQLPacketConstruct(agent, metrics);
+        }
+    } catch(std::runtime_error&)
+    {
+        LOG(WARNING) << "Could not fetch agents on host, skipping test";
+        return;
+    }
+    hsa_shut_down();
+}
+
+TEST(aql_profile, too_many_counters)
+{
+    hsa_init();
+    try
+    {
+        auto agents = rocprofiler::hsa::get_queue_controller().get_supported_agents();
+
+        for(const auto& [_, agent] : agents)
+        {
+            LOG(WARNING) << fmt::format("Found Agent: {}", agent.get_agent().handle);
+
+            auto metrics = rocprofiler::findDeviceMetrics(agent, {});
+            EXPECT_THROW(
+                {
+                    try
+                    {
+                        AQLPacketConstruct(agent, metrics);
+                    } catch(const std::exception& e)
+                    {
+                        EXPECT_NE(e.what(), nullptr) << e.what();
+                        throw;
+                    }
+                },
+                std::runtime_error);
+        }
+    } catch(std::runtime_error&)
+    {
+        LOG(WARNING) << "Could not fetch agents on host, skipping test";
+        return;
+    }
+    hsa_shut_down();
+}
+
+TEST(aql_profile, packet_generation_single)
+{
+    hsa_init();
+    try
+    {
+        auto agents = rocprofiler::hsa::get_queue_controller().get_supported_agents();
+        for(const auto& [_, agent] : agents)
+        {
+            auto               metrics = rocprofiler::findDeviceMetrics(agent, {"SQ_WAVES"});
+            AQLPacketConstruct pkt(agent, metrics);
+            auto               test_pkt = pkt.construct_packet(rocprofiler::get_ext_table());
+            EXPECT_TRUE(test_pkt);
+        }
+    } catch(std::runtime_error&)
+    {
+        LOG(WARNING) << "Could not fetch agents on host, skipping test";
+        return;
+    }
+
+    hsa_shut_down();
+}
+
+TEST(aql_profile, packet_generation_multi)
+{
+    hsa_init();
+    try
+    {
+        auto agents = rocprofiler::hsa::get_queue_controller().get_supported_agents();
+        for(const auto& [_, agent] : agents)
+        {
+            auto metrics =
+                rocprofiler::findDeviceMetrics(agent, {"SQ_WAVES", "TA_FLAT_READ_WAVEFRONTS"});
+            AQLPacketConstruct pkt(agent, metrics);
+            auto               test_pkt = pkt.construct_packet(rocprofiler::get_ext_table());
+            EXPECT_TRUE(test_pkt);
+        }
+    } catch(std::runtime_error&)
+    {
+        LOG(WARNING) << "Could not fetch agents on host, skipping test";
+        return;
+    }
+    hsa_shut_down();
+}
@@ -26,6 +26,7 @@
 #include "lib/common/container/stable_vector.hpp"
 #include "lib/rocprofiler/buffer.hpp"
 #include "lib/rocprofiler/context/context.hpp"
+#include "lib/rocprofiler/counters/core.hpp"

 #include <glog/logging.h>

@@ -206,6 +207,8 @@ start_context(rocprofiler_context_id_t context_id)

    if(!success) return ROCPROFILER_STATUS_ERROR_CONTEXT_NOT_STARTED;

+    rocprofiler::counters::start_context(context_id);
+
    return ROCPROFILER_STATUS_SUCCESS;
 }

@@ -221,6 +224,7 @@ stop_context(rocprofiler_context_id_t idx)
        {
            bool success = itr.compare_exchange_strong(_expected, nullptr);

+            rocprofiler::counters::stop_context(idx);
            if(success) return ROCPROFILER_STATUS_SUCCESS;
        }
    }
@@ -28,6 +28,7 @@

 #include "lib/common/container/stable_vector.hpp"
 #include "lib/rocprofiler/context/domain.hpp"
+#include "lib/rocprofiler/counters/core.hpp"

 #include <array>
 #include <atomic>
@@ -78,15 +79,29 @@ struct buffer_tracing_service
    buffer_array_t           buffer_data = {};
 };

+struct counter_collection_service
+{
+    // Contains a vector of counter collection instances associated with this context.
+    // Each instance is assocated with an agent and a counter collection profile.
+    // Contains callback information along with other data needed to collect/process
+    // counters.
+    std::vector<std::shared_ptr<rocprofiler::counters::counter_callback_info>> callbacks{};
+    // A flag to state wether or not the counter set is currently enabled. This is primarily
+    // to protect against multithreaded calls to enable a context (and enabling already enabled
+    // counters).
+    rocprofiler::common::Synchronized<bool> enabled{false};
+};
+
 struct context
 {
    // size is used to ensure that we never read past the end of the version
-    size_t                                    size               = 0;
-    uint64_t                                  context_idx        = 0;  // context id
-    uint32_t                                  client_idx         = 0;  // tool id
-    correlation_tracing_service               correlation_tracer = {};
-    std::unique_ptr<callback_tracing_service> callback_tracer    = {};
-    std::unique_ptr<buffer_tracing_service>   buffered_tracer    = {};
+    size_t                                      size               = 0;
+    uint64_t                                    context_idx        = 0;  // context id
+    uint32_t                                    client_idx         = 0;  // tool id
+    correlation_tracing_service                 correlation_tracer = {};
+    std::unique_ptr<callback_tracing_service>   callback_tracer    = {};
+    std::unique_ptr<buffer_tracing_service>     buffered_tracer    = {};
+    std::unique_ptr<counter_collection_service> counter_collection = {};
 };

 // set the client index needs to be called before allocate_context()
@@ -0,0 +1,112 @@
+#include <rocprofiler/rocprofiler.h>
+
+#include "lib/common/synchronized.hpp"
+#include "lib/rocprofiler/aql/helpers.hpp"
+#include "lib/rocprofiler/counters/evaluate_ast.hpp"
+#include "lib/rocprofiler/counters/metrics.hpp"
+#include "lib/rocprofiler/hsa/agent_cache.hpp"
+#include "lib/rocprofiler/hsa/queue.hpp"
+
+extern "C" {
+/**
+ * @brief Query Counter name.
+ *
+ * @param [in] counter_id
+ * @param [out] name if nullptr, size will be returned
+ * @param [out] size
+ * @return ::rocprofiler_status_t
+ */
+rocprofiler_status_t ROCPROFILER_API
+rocprofiler_query_counter_name(rocprofiler_counter_id_t counter_id, const char** name, size_t* size)
+{
+    const auto& id_map = rocprofiler::counters::getMetricIdMap();
+
+    if(const auto* metric_ptr = rocprofiler::common::get_val(id_map, counter_id.handle))
+    {
+        *name = metric_ptr->name().c_str();
+        *size = metric_ptr->name().size();
+        return ROCPROFILER_STATUS_SUCCESS;
+    }
+
+    return ROCPROFILER_STATUS_ERROR_COUNTER_NOT_FOUND;
+}
+
+/**
+ * @brief Query Counter Instances Count.
+ *
+ * @param [in] counter_id
+ * @param [out] instance_count
+ * @return rocprofiler_status_t
+ */
+rocprofiler_status_t ROCPROFILER_API
+rocprofiler_query_counter_instance_count(rocprofiler_agent_t      agent,
+                                         rocprofiler_counter_id_t counter_id,
+                                         size_t*                  instance_count)
+{
+    const auto& id_map     = rocprofiler::counters::getMetricIdMap();
+    const auto* metric_ptr = rocprofiler::common::get_val(id_map, counter_id.handle);
+    if(!metric_ptr) return ROCPROFILER_STATUS_ERROR_COUNTER_NOT_FOUND;
+
+    *instance_count = 0;
+    // Special counters like KERNEL_DURATION are not real counters and wont
+    // have any query info.
+    if(!metric_ptr->special().empty())
+    {
+        *instance_count = 1;
+        return ROCPROFILER_STATUS_SUCCESS;
+    }
+
+    // Returns the set of hardware counters needed to evaluate the metric.
+    // For derived metrics, this can be more than one counter. In that case,
+    // we return the maximum instance count among all underlying counters.
+    auto req_counters =
+        rocprofiler::counters::get_required_hardware_counters(std::string(agent.name), *metric_ptr);
+    if(!req_counters) return ROCPROFILER_STATUS_ERROR_COUNTER_NOT_FOUND;
+
+    // NOTE: to look up instance information, we require HSA be init'd. Reason
+    // for this is the call to get instance information is an HSA call.
+    const auto* maybe_agent = rocprofiler::common::get_val(
+        rocprofiler::hsa::get_queue_controller().get_supported_agents(), agent.id.handle);
+    if(!maybe_agent)
+    {
+        LOG(ERROR) << "HSA must be loaded to obtain instance information.";
+        return ROCPROFILER_STATUS_ERROR;
+    }
+
+    for(const auto& counter : *req_counters)
+    {
+        if(!counter.special().empty())
+        {
+            *instance_count = std::max(size_t(1), *instance_count);
+            continue;
+        }
+        auto query_info = rocprofiler::aql::get_query_info(maybe_agent->get_agent(), counter);
+        *instance_count = std::max(static_cast<size_t>(query_info.instance_count), *instance_count);
+    }
+
+    return ROCPROFILER_STATUS_SUCCESS;
+}
+/**
+ * @brief Query Agent Counters Availability.
+ *
+ * @param [in] agent
+ * @param [out] counters_list
+ * @param [out] counters_count
+ * @return ::rocprofiler_status_t
+ */
+rocprofiler_status_t ROCPROFILER_API
+rocprofiler_iterate_agent_supported_counters(rocprofiler_agent_t                 agent,
+                                             rocprofiler_available_counters_cb_t cb,
+                                             void*                               user_data)
+{
+    const auto& metrics = rocprofiler::counters::getMetricsForAgent(std::string(agent.name));
+    std::vector<rocprofiler_counter_id_t> ids;
+    ids.reserve(metrics.size());
+    for(const auto& metric : metrics)
+    {
+        ids.push_back({.handle = metric.id()});
+    }
+
+    return cb(ids.data(), ids.size(), user_data);
+}
+}
@@ -1,10 +1,10 @@
-set(ROCPROFILER_LIB_COUNTERS_SOURCES metrics.cpp)
-set(ROCPROFILER_LIB_COUNTERS_HEADERS metrics.hpp)
-
+set(ROCPROFILER_LIB_COUNTERS_SOURCES metrics.cpp evaluate_ast.cpp core.cpp)
+set(ROCPROFILER_LIB_COUNTERS_HEADERS metrics.hpp evaluate_ast.hpp core.hpp)
 target_sources(rocprofiler-object-library PRIVATE ${ROCPROFILER_LIB_COUNTERS_SOURCES}
                                                  ${ROCPROFILER_LIB_COUNTERS_HEADERS})

 add_subdirectory(xml)
+add_subdirectory(parser)

 if(ROCPROFILER_BUILD_TESTS)
    add_subdirectory(tests)
@@ -0,0 +1,236 @@
+#include "lib/rocprofiler/counters/core.hpp"
+
+#include "lib/common/synchronized.hpp"
+#include "lib/rocprofiler/aql/helpers.hpp"
+#include "lib/rocprofiler/aql/packet_construct.hpp"
+#include "lib/rocprofiler/context/context.hpp"
+#include "lib/rocprofiler/registration.hpp"
+
+#include <rocprofiler/rocprofiler.h>
+
+namespace rocprofiler
+{
+namespace counters
+{
+/**
+ * Callback we get from HSA interceptor when a kernel packet is being enqueued.
+ *
+ * We return an AQLPacket containing the start/stop/read packets for injection.
+ */
+std::unique_ptr<rocprofiler::hsa::AQLPacket>
+queue_cb(const std::shared_ptr<rocprofiler::counters::counter_callback_info>& info,
+         const hsa::Queue&                                                    queue,
+         hsa::ClientID,
+         const hsa_ext_amd_aql_pm4_packet_t&)
+{
+    if(!info) return nullptr;
+
+    std::unique_ptr<rocprofiler::hsa::AQLPacket> ret_pkt;
+
+    // Check packet cache
+    info->packets.wlock([&](auto& pkt_vector) {
+        // Delay packet generator construction until first HSA packet is processed
+        // This ensures that HSA exists
+        if(!info->pkt_generator)
+        {
+            info->pkt_generator = std::make_unique<rocprofiler::aql::AQLPacketConstruct>(
+                queue.get_agent(),
+                std::vector<counters::Metric>{info->profile_cfg.reqired_hw_counters.begin(),
+                                              info->profile_cfg.reqired_hw_counters.end()});
+        }
+
+        if(!pkt_vector.empty())
+        {
+            ret_pkt = std::move(pkt_vector.back());
+            pkt_vector.pop_back();
+        }
+    });
+
+    if(!ret_pkt)
+    {
+        // If we do not have a packet in the cache, create one.
+        ret_pkt =
+            info->pkt_generator->construct_packet(hsa::get_queue_controller().get_ext_table());
+    }
+    return ret_pkt;
+}
+
+/**
+ * Callback called by HSA interceptor when the kernel has completed processing.
+ */
+void
+completed_cb(const std::shared_ptr<rocprofiler::counters::counter_callback_info>& info,
+             const hsa::Queue&                                                    queue,
+             hsa::ClientID,
+             const hsa_ext_amd_aql_pm4_packet_t&          kernel,
+             std::unique_ptr<rocprofiler::hsa::AQLPacket> pkt)
+{
+    if(!info) return;
+
+    // auto out_buf = pkt->profile.output_buffer.ptr;
+    // Read data and create user return....
+
+    // return AQL packet for reuse.
+
+    info->packets.wlock([&](auto& pkt_vector) {
+        if(pkt)
+        {
+            pkt_vector.emplace_back(std::move(pkt));
+        }
+    });
+
+    if(!info->user_cb) return;
+
+    info->user_cb(queue.get_id(),
+                  info->profile_cfg.agent,
+                  rocprofiler_correlation_id_t{},
+                  reinterpret_cast<const hsa_kernel_dispatch_packet_t*>(&kernel),
+                  info->callback_args,
+                  nullptr,  // Date pointer does here.
+                  0,        // Number of objects
+                  info->profile_cfg.id);
+}
+
+class CounterController
+{
+public:
+    // Adds a counter collection profile to our global cache.
+    // Note: these profiles can be used across multiple contexts
+    //       and are independent of the context.
+    uint64_t add_profile(profile_config&& config)
+    {
+        static std::atomic<uint64_t> profile_val = 1;
+        uint64_t                     ret         = 0;
+        _configs.wlock([&](auto& data) {
+            config.id = rocprofiler_profile_config_id_t{.handle = profile_val};
+            data.emplace(profile_val, std::move(config));
+            ret = profile_val;
+            profile_val++;
+        });
+        return ret;
+    }
+
+    void destroy_profile(uint64_t id)
+    {
+        _configs.wlock([&](auto& data) { data.erase(id); });
+    }
+
+    // Setup the counter collection service. counter_callback_info is created here
+    // to contain the counters that need to be collected (specified in profile_id) and
+    // the AQL packet generator for injecting packets. Note: the service is created
+    // in the stop state.
+    bool configure_dispatch(rocprofiler_context_id_t                         context_id,
+                            uint64_t                                         profile_id,
+                            rocprofiler_profile_counting_dispatch_callback_t callback,
+                            void*                                            callback_args) const
+    {
+        auto& ctx = *rocprofiler::context::get_registered_contexts().at(context_id.handle);
+
+        // Note: A single profile config could be used on multiple contexts
+        profile_config cfg;
+        _configs.rlock([&](const auto& map) { cfg = map.at(profile_id); });
+
+        if(!ctx.counter_collection)
+        {
+            ctx.counter_collection =
+                std::make_unique<rocprofiler::context::counter_collection_service>();
+        }
+
+        auto& cb = *ctx.counter_collection->callbacks.emplace_back(
+            std::make_shared<rocprofiler::counters::counter_callback_info>());
+
+        cb.user_cb = callback;
+
+        // Secondary copy of the config to be shared with async callback
+        cb.profile_cfg   = cfg;
+        cb.callback_args = callback_args;
+        cb.context       = context_id;
+        return true;
+    }
+
+private:
+    rocprofiler::common::Synchronized<std::unordered_map<uint64_t, profile_config>> _configs;
+};
+
+CounterController&
+get_controller()
+{
+    static CounterController controller;
+    return controller;
+}
+
+uint64_t
+create_counter_profile(profile_config&& config)
+{
+    return get_controller().add_profile(std::move(config));
+}
+
+void
+destroy_counter_profile(uint64_t id)
+{
+    get_controller().destroy_profile(id);
+}
+
+void
+start_context(rocprofiler_context_id_t context_id)
+{
+    auto& ctx        = *rocprofiler::context::get_registered_contexts().at(context_id.handle);
+    auto& controller = hsa::get_queue_controller();
+    if(!ctx.counter_collection) return;
+
+    // Only one thread should be attempting to enable/disable this context
+    ctx.counter_collection->enabled.wlock([&](auto& enabled) {
+        if(enabled) return;
+        for(auto& cb : ctx.counter_collection->callbacks)
+        {
+            // Insert our callbacks into HSA Interceptor. This
+            // turns on counter instrumentation.
+            cb->queue_id = controller.add_callback(
+                cb->profile_cfg.agent,
+                [=](const hsa::Queue&                   q,
+                    hsa::ClientID                       c,
+                    const hsa_ext_amd_aql_pm4_packet_t& kern_pkt) {
+                    return queue_cb(cb, q, c, kern_pkt);
+                },
+                // Completion CB
+                [=](const hsa::Queue&                   q,
+                    hsa::ClientID                       c,
+                    const hsa_ext_amd_aql_pm4_packet_t& kern_pkt,
+                    std::unique_ptr<hsa::AQLPacket>     aql) {
+                    completed_cb(cb, q, c, kern_pkt, std::move(aql));
+                });
+        }
+        enabled = true;
+    });
+}
+
+void
+stop_context(rocprofiler_context_id_t context_id)
+{
+    auto& controller = hsa::get_queue_controller();
+    auto& ctx        = *rocprofiler::context::get_registered_contexts().at(context_id.handle);
+    if(!ctx.counter_collection) return;
+
+    ctx.counter_collection->enabled.wlock([&](auto& enabled) {
+        if(!enabled) return;
+        for(auto& cb : ctx.counter_collection->callbacks)
+        {
+            // Remove our callbacks from HSA's queue controller
+            controller.remove_callback(cb->queue_id);
+            cb->queue_id = -1;
+        }
+        enabled = false;
+    });
+}
+
+bool
+configure_dispatch(rocprofiler_context_id_t                         context_id,
+                   uint64_t                                         profile_id,
+                   rocprofiler_profile_counting_dispatch_callback_t callback,
+                   void*                                            callback_args)
+{
+    return get_controller().configure_dispatch(context_id, profile_id, callback, callback_args);
+}
+
+}  // namespace counters
+}  // namespace rocprofiler
@@ -0,0 +1,69 @@
+#pragma once
+
+#include <rocprofiler/agent.h>
+#include <rocprofiler/dispatch_profile.h>
+
+#include "lib/rocprofiler/aql/helpers.hpp"
+#include "lib/rocprofiler/aql/packet_construct.hpp"
+#include "lib/rocprofiler/counters/evaluate_ast.hpp"
+#include "lib/rocprofiler/counters/metrics.hpp"
+#include "lib/rocprofiler/hsa/agent_cache.hpp"
+
+namespace rocprofiler
+{
+namespace counters
+{
+// Stores counter profiling information such as the agent
+// to collect counters on, the metrics to collect, the hw
+// counters needed to evaluate the metrics, and the ASTs.
+// This profile can be shared among many rocprof contexts.
+struct profile_config
+{
+    rocprofiler_agent_t           agent{};
+    std::vector<counters::Metric> metrics{};
+    // HW counters that must be collected to compute the above
+    // metrics (derived metrics are broken down into hw counters
+    // in this vector).
+    std::set<counters::Metric> reqired_hw_counters{};
+    // ASTs to evaluate
+    std::vector<counters::EvaluateAST> asts{};
+    rocprofiler_profile_config_id_t    id{.handle = 0};
+};
+
+// Internal counter struct that stores the state needed to handle an intercepted
+// HSA kernel packet.
+struct counter_callback_info
+{
+    // Packet generator to create AQL packets for insertion
+    std::unique_ptr<rocprofiler::aql::AQLPacketConstruct> pkt_generator{nullptr};
+    // A packet cache of AQL packets. This allows reuse of AQL packets (preventing costly
+    // allocation of new packets/destruction).
+    rocprofiler::common::Synchronized<std::vector<std::unique_ptr<rocprofiler::hsa::AQLPacket>>>
+        packets{};
+    // User callback
+    rocprofiler_profile_counting_dispatch_callback_t user_cb{nullptr};
+    // Profile configuration used for this callback containing the counters
+    // to collect and the evaluation ASTs
+    profile_config profile_cfg{};
+    // User id
+    void* callback_args{nullptr};
+    // Link to the context this is associated with
+    rocprofiler_context_id_t context{.handle = 0};
+    // HSA Queue ClientID. This is an ID we get when we insert a callback into the
+    // HSA queue interceptor. This ID can be used to disable the callback.
+    rocprofiler::hsa::ClientID queue_id{-1};
+};
+
+uint64_t
+     create_counter_profile(profile_config&& config);
+void destroy_counter_profile(uint64_t);
+bool
+     configure_dispatch(rocprofiler_context_id_t                         context_id,
+                        uint64_t                                         profile_id,
+                        rocprofiler_profile_counting_dispatch_callback_t callback,
+                        void*                                            callback_args);
+void start_context(rocprofiler_context_id_t);
+
+void stop_context(rocprofiler_context_id_t);
+}  // namespace counters
+}  // namespace rocprofiler
@@ -0,0 +1,79 @@
+#include "lib/rocprofiler/counters/evaluate_ast.hpp"
+
+#include <optional>
+
+#include "lib/common/synchronized.hpp"
+#include "lib/common/utility.hpp"
+#include "lib/rocprofiler/counters/parser/reader.hpp"
+
+namespace rocprofiler
+{
+namespace counters
+{
+const std::unordered_map<std::string, EvaluateASTMap>&
+get_ast_map()
+{
+    static std::unordered_map<std::string, EvaluateASTMap> ast_map = []() {
+        std::unordered_map<std::string, EvaluateASTMap> data;
+        const auto&                                     metric_map = counters::getMetricMap();
+        for(const auto& [gfx, metrics] : metric_map)
+        {
+            // TODO: Remove global XML from derrived counters...
+            if(gfx == "global") continue;
+
+            std::unordered_map<std::string, Metric> by_name;
+            for(const auto& metric : metrics)
+            {
+                by_name.emplace(metric.name(), metric);
+            }
+
+            auto& eval_map = data.emplace(gfx, EvaluateASTMap{}).first->second;
+            for(auto& [_, metric] : by_name)
+            {
+                RawAST* ast = nullptr;
+                auto*   buf =
+                    yy_scan_string(metric.expression().empty() ? metric.name().c_str()
+                                                               : metric.expression().c_str());
+                yyparse(&ast);
+                if(!ast)
+                {
+                    LOG(ERROR) << fmt::format("Unable to parse metric {}", metric);
+                    throw std::runtime_error(fmt::format("Unable to parse metric {}", metric));
+                }
+                try
+                {
+                    eval_map.emplace(metric.name(), EvaluateAST(by_name, *ast));
+                } catch(std::out_of_range& e)
+                {
+                    throw std::runtime_error(
+                        fmt::format("AST was not generated for {}:{}, Counter will be unavailable. "
+                                    "Likely cause is a base counter not being defined used in a "
+                                    "derrived counter.",
+                                    gfx,
+                                    metric.name()));
+                }
+                yy_delete_buffer(buf);
+                delete ast;
+            }
+        }
+        return data;
+    }();
+    return ast_map;
+}
+
+std::optional<std::set<Metric>>
+get_required_hardware_counters(const std::string& agent, const Metric& metric)
+{
+    const auto& asts      = get_ast_map();
+    const auto* agent_map = rocprofiler::common::get_val(asts, agent);
+    if(!agent_map) return std::nullopt;
+    const auto* counter_ast = rocprofiler::common::get_val(*agent_map, metric.name());
+    if(!counter_ast) return std::nullopt;
+
+    std::set<Metric> required_counters;
+    counter_ast->get_required_counters(*agent_map, required_counters);
+    return required_counters;
+}
+
+}  // namespace counters
+}  // namespace rocprofiler
@@ -0,0 +1,93 @@
+#pragma once
+
+#include <iostream>
+#include <set>
+#include <unordered_map>
+
+#include "lib/common/utility.hpp"
+#include "lib/rocprofiler/counters/metrics.hpp"
+#include "lib/rocprofiler/counters/parser/raw_ast.hpp"
+
+namespace rocprofiler
+{
+namespace counters
+{
+class EvaluateAST
+{
+public:
+    EvaluateAST(const std::unordered_map<std::string, Metric>& metrics, const RawAST& ast)
+    : _type(ast.type)
+    , _op(ast.operation)
+    {
+        if(_type == NodeType::REFERENCE_NODE)
+        {
+            _metric = metrics.at(std::get<std::string>(ast.value));
+            // LOG(ERROR) << fmt::format("CHILD METRIC {}", _metric);
+        }
+
+        if(_type == NodeType::NUMBER_NODE)
+        {
+            _raw_value = std::get<int64_t>(ast.value);
+        }
+
+        for(const auto& nextAst : ast.counter_set)
+        {
+            _children.emplace_back(metrics, *nextAst);
+        }
+    }
+
+    void get_required_counters(const std::unordered_map<std::string, EvaluateAST>& asts,
+                               std::set<Metric>&                                   counters) const
+    {
+        if(!_metric.empty() && children().empty() && _type != NodeType::NUMBER_NODE)
+        {
+            // Base counter
+            if(_metric.expression().empty())
+            {
+                counters.insert(_metric);
+                return;
+            }
+
+            // Derrived Counter
+            const auto* expr_ptr = rocprofiler::common::get_val(asts, _metric.name());
+            if(!expr_ptr) throw std::runtime_error("could not find derived counter");
+            expr_ptr->get_required_counters(asts, counters);
+            return;
+        }
+
+        for(const auto& child : children())
+        {
+            child.get_required_counters(asts, counters);
+        }
+    }
+
+    NodeType                        type() const { return _type; }
+    NodeType                        op() const { return _op; }
+    const std::vector<EvaluateAST>& children() const { return _children; }
+    const Metric&                   metric() const { return _metric; }
+
+private:
+    NodeType                 _type{NONE};
+    NodeType                 _op{NONE};
+    Metric                   _metric;
+    double                   _raw_value{0};
+    std::vector<EvaluateAST> _children;
+};
+
+using EvaluateASTMap = std::unordered_map<std::string, EvaluateAST>;
+
+/**
+ * Construct the ASTs for all counters appearing in basic/derrived counter
+ * definition files.
+ */
+const std::unordered_map<std::string, EvaluateASTMap>&
+get_ast_map();
+
+/**
+ * Get the required basic/hardware counters needed to evaluate a
+ * specific metric (may be multiple HW counters if a derrived metric).
+ */
+std::optional<std::set<Metric>>
+get_required_hardware_counters(const std::string& agent, const Metric& metric);
+}  // namespace counters
+}  // namespace rocprofiler
@@ -22,15 +22,22 @@ THE SOFTWARE.

 #include "metrics.hpp"

+#include <rocprofiler/rocprofiler.h>
+
+#include "lib/common/synchronized.hpp"
+#include "lib/common/utility.hpp"
+#include "lib/common/xml.hpp"
+
+#include "glog/logging.h"
+
 #include <dlfcn.h>  // for dladdr
+#include <atomic>
 #include <cstdlib>
 #include <filesystem>
 #include <optional>

-#include "glog/logging.h"
-#include "lib/common/xml.hpp"
-#include "rocprofiler/rocprofiler.h"
-
+namespace rocprofiler
+{
 namespace counters
 {
 namespace
@@ -38,10 +45,11 @@ namespace
 MetricMap
 loadXml(const std::string& filename)
 {
-    MetricMap ret;
+    static std::atomic<uint64_t> id = 0;
+    MetricMap                    ret;
    DLOG(INFO) << "Loading Counter Config: " << filename;
    // todo: return unique_ptr....
-    auto xml = xml::Xml::Create(filename);
+    auto xml = common::Xml::Create(filename);
    LOG_IF(FATAL, !xml)
        << "Could not open XML Counter Config File (set env ROCPROFILER_METRICS_PATH)";

@@ -68,23 +76,25 @@ loadXml(const std::string& filename)
                                   node->opts["block"],
                                   node->opts["event"],
                                   node->opts["descr"],
-                                   node->opts["expr"]);
+                                   node->opts["expr"],
+                                   node->opts["special"],
+                                   id);
+            id++;
        }
    }

-    DLOG(INFO) << fmt::format("{}", ret);
    return ret;
 }

 std::string
 findViaInstallPath(const std::string& filename)
 {
-    Dl_info dl_info;
+    Dl_info dl_info = {};
    DLOG(INFO) << filename << " is being looked up via install path";
    if(dladdr(reinterpret_cast<const void*>(rocprofiler_query_available_agents), &dl_info) != 0)
    {
-        return std::filesystem::path{dl_info.dli_fname}.remove_filename() /
-               fmt::format("../lib/{}", filename);
+        return std::filesystem::path{dl_info.dli_fname}.parent_path().parent_path() /
+               fmt::format("share/rocprofiler/{}", filename);
    }
    return filename;
 }
@@ -92,10 +102,10 @@ findViaInstallPath(const std::string& filename)
 std::string
 findViaEnvironment(const std::string& filename)
 {
-    if(getenv("ROCPROFILER_METRICS_PATH"))
+    if(const char* metrics_path = nullptr; (metrics_path = getenv("ROCPROFILER_METRICS_PATH")))
    {
        DLOG(INFO) << filename << " is being looked up via env variable ROCPROFILER_METRICS_PATH";
-        return std::filesystem::path{std::string(getenv("ROCPROFILER_METRICS_PATH"))} / filename;
+        return std::filesystem::path{std::string{metrics_path}} / filename;
    }
    // No environment variable, lookup via install path
    return findViaInstallPath(filename);
@@ -115,4 +125,58 @@ getBaseHardwareMetrics()
    return loadXml(findViaEnvironment("basic_counters.xml"));
 }

-};  // namespace counters
+const MetricIdMap&
+getMetricIdMap()
+{
+    static MetricIdMap id_map = []() {
+        MetricIdMap map;
+        for(const auto& [_, val] : getMetricMap())
+        {
+            for(const auto& metric : val)
+            {
+                map.emplace(metric.id(), metric);
+            }
+        }
+        return map;
+    }();
+    return id_map;
+}
+
+const MetricMap&
+getMetricMap()
+{
+    static MetricMap map = []() {
+        MetricMap ret = getBaseHardwareMetrics();
+        for(auto& [key, val] : getDerivedHardwareMetrics())
+        {
+            auto [iter, inserted] = ret.emplace(key, val);
+            if(!inserted)
+            {
+                iter->second.insert(iter->second.end(), val.begin(), val.end());
+            }
+        }
+        return ret;
+    }();
+    return map;
+}
+
+const std::vector<Metric>&
+getMetricsForAgent(const std::string& agent)
+{
+    static const std::vector<Metric> empty;
+    const auto&                      map = getMetricMap();
+    if(const auto* metric_ptr = rocprofiler::common::get_val(map, agent))
+    {
+        return *metric_ptr;
+    }
+
+    return empty;
+}
+
+bool
+operator<(Metric const& lhs, Metric const& rhs)
+{
+    return lhs.id() < rhs.id();
+}
+}  // namespace counters
+}  // namespace rocprofiler
@@ -10,22 +10,29 @@
 #include "fmt/core.h"
 #include "fmt/ranges.h"

+namespace rocprofiler
+{
 namespace counters
 {
 // Base metrics (w/o instance information) defined in gfx_metrics/derrived.xml
 class Metric
 {
 public:
+    Metric() = default;
    Metric(std::string name,
           std::string block,
           std::string event,
           std::string dsc,
-           std::string expr)
+           std::string expr,
+           std::string special,
+           uint64_t    id)
    : name_(std::move(name))
    , block_(std::move(block))
    , event_(std::move(event))
    , description_(std::move(dsc))
    , expression_(std::move(expr))
+    , special_(std::move(special))
+    , id_(id)
    {}

    const std::string& name() const { return name_; }
@@ -33,30 +40,64 @@ public:
    const std::string& event() const { return event_; }
    const std::string& description() const { return description_; }
    const std::string& expression() const { return expression_; }
+    const std::string& special() const { return special_; }
+    uint64_t           id() const { return id_; }
+    bool               empty() const { return empty_; }
+
+    friend bool operator<(Metric const& lhs, Metric const& rhs);

 private:
-    std::string name_;
-    std::string block_;
-    std::string event_;
-    std::string description_;
-    std::string expression_;
+    std::string name_        = {};
+    std::string block_       = {};
+    std::string event_       = {};
+    std::string description_ = {};
+    std::string expression_  = {};
+    std::string special_     = {};
+    int64_t     id_          = -1;
+    bool        empty_       = false;
 };

-using MetricMap = std::unordered_map<std::string, std::vector<Metric>>;
+using MetricMap   = std::unordered_map<std::string, std::vector<Metric>>;
+using MetricIdMap = std::unordered_map<uint64_t, Metric>;

+/**
+ * Get base hardware counters for all GFXs Map<GFX Name, Counters>
+ */
 MetricMap
 getBaseHardwareMetrics();

+/**
+ * Get derived hardware metrics for all GFXs  Map<GFX Name, Counters>
+ */
 MetricMap
 getDerivedHardwareMetrics();

+/**
+ * Combined map containing both base and derived counters
+ */
+const MetricMap&
+getMetricMap();
+
+/**
+ * Get the metrics that apply to a specific agent. Supplied parameter
+ * is the GFXIP of the agent.
+ */
+const std::vector<Metric>&
+getMetricsForAgent(const std::string&);
+
+/**
+ * Get a map of metric::id() -> metric
+ */
+const MetricIdMap&
+getMetricIdMap();
 }  // namespace counters
+}  // namespace rocprofiler

 namespace fmt
 {
 // fmt::format support for metric
 template <>
-struct formatter<counters::Metric>
+struct formatter<rocprofiler::counters::Metric>
 {
    template <typename ParseContext>
    constexpr auto parse(ParseContext& ctx)
@@ -65,21 +106,23 @@ struct formatter<counters::Metric>
    }

    template <typename Ctx>
-    auto format(counters::Metric const& metric, Ctx& ctx) const
+    auto format(rocprofiler::counters::Metric const& metric, Ctx& ctx) const
    {
-        return fmt::format_to(ctx.out(),
-                              "Metric: {} [Block: {}, Event: {}, Expression: {}, Description: {}]",
-                              metric.name(),
-                              metric.block(),
-                              metric.event(),
-                              metric.expression().empty() ? "<None>" : metric.expression(),
-                              metric.description());
+        return fmt::format_to(
+            ctx.out(),
+            "Metric: {} [Block: {}, Event: {}, Expression: {}, Description: {}, id: {}]",
+            metric.name(),
+            metric.block(),
+            metric.event(),
+            metric.expression().empty() ? "<None>" : metric.expression(),
+            metric.description(),
+            metric.id());
    }
 };

 // fmt::format support for MetricMap
 template <>
-struct formatter<counters::MetricMap>
+struct formatter<rocprofiler::counters::MetricMap>
 {
    template <typename ParseContext>
    constexpr auto parse(ParseContext& ctx)
@@ -88,7 +131,7 @@ struct formatter<counters::MetricMap>
    }

    template <typename Ctx>
-    auto format(counters::MetricMap const& map, Ctx& ctx) const
+    auto format(rocprofiler::counters::MetricMap const& map, Ctx& ctx) const
    {
        std::string out;
        for(const auto& [gfxName, counters] : map)
@@ -0,0 +1,50 @@
+rocprofiler_deactivate_clang_tidy()
+
+set(expr_parser_sources parser.cpp parser.h scanner.cpp raw_ast.hpp reader.hpp)
+
+add_library(rocprofiler-expr-parser OBJECT)
+
+if(ROCPROFILER_REGENERATE_COUNTERS_PARSER)
+    find_package(FLEX REQUIRED)
+    find_package(BISON REQUIRED)
+
+    configure_file(${CMAKE_CURRENT_SOURCE_DIR}/parser.h
+                   ${CMAKE_CURRENT_BINARY_DIR}/parser.h COPYONLY)
+
+    bison_target(
+        ExprBison parser.y ${CMAKE_CURRENT_BINARY_DIR}/parser.cpp
+        COMPILE_FLAGS "-t"
+        DEFINES_FILE ${CMAKE_CURRENT_BINARY_DIR}/parser.h)
+    flex_target(ExprFlex scanner.l ${CMAKE_CURRENT_BINARY_DIR}/scanner.cpp)
+    add_flex_bison_dependency(ExprFlex ExprBison)
+
+    set_source_files_properties(${expr_parser_sources} PROPERTIES COMPILE_DEFINITIONS
+                                                                  YYDEBUG=1)
+
+    add_custom_target(
+        rocprofiler-expr-parser-patch
+        COMMAND
+            ${CMAKE_COMMAND} -DPROJECT_SRC_DIR=${PROJECT_SOURCE_DIR}
+            -DPROJECT_BLD_DIR=${PROJECT_BINARY_DIR}
+            -DSOURCE_DIR=${CMAKE_CURRENT_SOURCE_DIR}
+            -DBINARY_DIR=${CMAKE_CURRENT_BINARY_DIR}
+            -DFORMAT_EXE=${ROCPROFILER_CLANG_FORMAT_EXE} -P
+            ${PROJECT_SOURCE_DIR}/source/scripts/patch-parser.cmake
+        DEPENDS ${CMAKE_CURRENT_BINARY_DIR}/parser.cpp
+                ${CMAKE_CURRENT_BINARY_DIR}/scanner.cpp
+        VERBATIM)
+
+    # ensure gets applied when rocprofiler-expr-parser is built
+    add_dependencies(rocprofiler-expr-parser rocprofiler-expr-parser-patch)
+endif()
+
+target_sources(rocprofiler-expr-parser PRIVATE ${expr_parser_sources})
+target_include_directories(rocprofiler-expr-parser PRIVATE ${CMAKE_CURRENT_SOURCE_DIR})
+target_link_libraries(rocprofiler-expr-parser
+                      PRIVATE rocprofiler::rocprofiler-common-library)
+target_sources(rocprofiler-object-library
+               PUBLIC $<TARGET_OBJECTS:rocprofiler-expr-parser>)
+
+if(ROCPROFILER_BUILD_TESTS)
+    add_subdirectory(tests)
+endif()
@@ -0,0 +1,106 @@
+/* A Bison parser, made by GNU Bison 3.5.1.  */
+
+/* Bison interface for Yacc-like parsers in C
+
+   Copyright (C) 1984, 1989-1990, 2000-2015, 2018-2020 Free Software Foundation,
+   Inc.
+
+   This program is free software: you can redistribute it and/or modify
+   it under the terms of the GNU General Public License as published by
+   the Free Software Foundation, either version 3 of the License, or
+   (at your option) any later version.
+
+   This program is distributed in the hope that it will be useful,
+   but WITHOUT ANY WARRANTY; without even the implied warranty of
+   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+   GNU General Public License for more details.
+
+   You should have received a copy of the GNU General Public License
+   along with this program.  If not, see <http://www.gnu.org/licenses/>.  */
+
+/* As a special exception, you may create a larger work that contains
+   part or all of the Bison parser skeleton and distribute that work
+   under terms of your choice, so long as that work isn't itself a
+   parser generator using the skeleton or a modified version thereof
+   as a parser skeleton.  Alternatively, if you modify or redistribute
+   the parser skeleton itself, you may (at your option) remove this
+   special exception, which will cause the skeleton and the resulting
+   Bison output files to be licensed under the GNU General Public
+   License without this special exception.
+
+   This special exception was added by the Free Software Foundation in
+   version 2.2 of Bison.  */
+
+/* Undocumented macros, especially those whose name start with YY_,
+   are private implementation details.  Do not rely on them.  */
+
+#ifndef YY_YY_ROCPROFILER_SOURCE_LIB_ROCPROFILER_COUNTERS_PARSER_PARSER_H_INCLUDED
+#define YY_YY_ROCPROFILER_SOURCE_LIB_ROCPROFILER_COUNTERS_PARSER_PARSER_H_INCLUDED
+/* Debug traces.  */
+#ifndef YYDEBUG
+#    define YYDEBUG 1
+#endif
+#if YYDEBUG
+extern int yydebug;
+#endif
+/* "%code requires" blocks.  */
+#line 2 "parser.y"
+
+#include "raw_ast.hpp"
+using namespace rocprofiler::counters;
+#define YYDEBUG 1
+
+#line 54 "parser.h"
+
+/* Token type.  */
+#ifndef YYTOKENTYPE
+#    define YYTOKENTYPE
+enum yytokentype
+{
+    ADD             = 258,
+    SUB             = 259,
+    MUL             = 260,
+    DIV             = 261,
+    ABS             = 262,
+    EQUALS          = 263,
+    OP              = 264,
+    CP              = 265,
+    O_SQ            = 266,
+    C_SQ            = 267,
+    COLON           = 268,
+    EOL             = 269,
+    UMINUS          = 270,
+    CM              = 271,
+    NUMBER          = 272,
+    RANGE           = 273,
+    NAME            = 274,
+    REDUCE          = 275,
+    SELECT          = 276,
+    LOWER_THAN_ELSE = 277,
+    ELSE            = 278
+};
+#endif
+
+/* Value type.  */
+#if !defined YYSTYPE && !defined YYSTYPE_IS_DECLARED
+union YYSTYPE
+{
+#    line 34 "parser.y"
+
+    RawAST* a; /* For ast node */
+    int64_t d;
+    char*   s;
+
+#    line 95 "parser.h"
+};
+typedef union YYSTYPE YYSTYPE;
+#    define YYSTYPE_IS_TRIVIAL  1
+#    define YYSTYPE_IS_DECLARED 1
+#endif
+
+extern YYSTYPE yylval;
+
+int
+yyparse(RawAST** result);
+
+#endif /* !YY_YY_ROCPROFILER_SOURCE_LIB_ROCPROFILER_COUNTERS_PARSER_PARSER_H_INCLUDED  */
@@ -0,0 +1,103 @@
+%parse-param {RawAST** result}
+%code requires {
+#include "raw_ast.hpp"
+using namespace rocprofiler::counters;
+#define YYDEBUG 1
+}
+
+%{
+#include <stdexcept>
+#include <stdio.h>
+#include <string>
+
+#include <glog/logging.h>
+
+#include "raw_ast.hpp"
+
+int yyparse(rocprofiler::counters::RawAST** result);
+int yylex(void);
+void yyerror(rocprofiler::counters::RawAST**, const char *s) { LOG(ERROR) << s; }
+%}
+
+/* declare tokens */
+%token ADD SUB MUL DIV ABS EQUALS
+%token OP CP O_SQ C_SQ COLON
+%token EOL
+
+/* set associativity rules for operand tokens */
+%right EQUALS
+%left ADD SUB
+%left MUL DIV
+%nonassoc '|' UMINUS CM
+
+/*declare data types*/
+%union {
+ RawAST* a;          /* For ast node */
+ int64_t d;
+ char* s;
+}
+
+%token NUMBER RANGE              /* set data type for numbers */
+%token NAME                      /* set data type for variables and user-defined functions */
+%token REDUCE SELECT             /* set data type for special functions */
+%type <a> exp                    /* set data type for expressions */
+%type <s> NAME
+%type <d> NUMBER
+
+%nonassoc LOWER_THAN_ELSE
+%nonassoc ELSE
+
+// %token <pos_int> POS_INTEGER
+
+%%
+
+top:
+  exp { *result = $1;};
+
+// line: /* nothing */
+//  | line exp EOL {
+//     // TODO
+//     //printf("= %g\n", eval($2)); //evaluate and print the AST
+//     //printf("> ");
+//    }
+//  | line EOL { printf("> "); } /* blank line or a comment */
+//  ;
+
+exp: NUMBER                               { $$ = new RawAST(NUMBER_NODE, $1); }
+  | exp ADD exp                           { $$ = new RawAST(ADDITION_NODE, {$1, $3}); }
+  | exp SUB exp                           { $$ = new RawAST(SUBTRACTION_NODE, {$1, $3}); }
+  | exp MUL exp                           { $$ = new RawAST(MULTIPLY_NODE, {$1, $3}); }
+  | exp DIV exp                           { $$ = new RawAST(DIVIDE_NODE, {$1, $3}); }
+  | OP exp CP                             { $$ = $2; }
+  | O_SQ exp COLON exp C_SQ               { $$ = new RawAST(RANGE_NODE, {$2, $4}); }
+  | NAME                                  { $$ = new RawAST(REFERENCE_NODE, $1);
+                                            free($1);
+                                          }
+  | NAME EQUALS exp                       { $$ = new RawAST(REFERENCE_SET, $1, $3);
+                                            free($1);
+                                          }
+  | NAME EQUALS exp CM exp                { $$ = new RawAST(REFERENCE_SET, $1, $3, $5);
+                                            free($1);
+                                          }
+  | REDUCE OP exp CM NAME CP              { $$ = new RawAST(REDUCE_NODE, $3, $5);
+                                            free($5);
+                                          }
+  | REDUCE OP exp CM NAME CM exp CP       { $$ = new RawAST(REDUCE_NODE, $3, $5, $7);
+                                            free($5);
+                                          }
+  | SELECT OP exp CM NAME CP              { $$ = new RawAST(SELECT_NODE, $3, $5);
+                                            free($5);
+                                          }
+  | SELECT OP exp CM NAME CM exp CP       { $$ = new RawAST(SELECT_NODE, $3, $5, $7);
+                                            free($5);
+                                          }
+  // | NAME O_SQ POS_INTEGER C_SQ            { $$ = create_index_access_node($1, $3); }
+  ;
+
+
+%%
+
+// void yyerror(char const *s)
+// {
+//   fprintf(stderr, "check error saurabh: %s\n", s);
+// }
@@ -0,0 +1,189 @@
+#pragma once
+
+#include <map>
+#include <optional>
+#include <string>
+#include <utility>
+#include <variant>
+#include <vector>
+
+#include <fmt/core.h>
+#include <fmt/ranges.h>
+#include <glog/logging.h>
+
+namespace rocprofiler
+{
+namespace counters
+{
+enum NodeType
+{
+    NONE = 0,
+    ADDITION_NODE,
+    DIVIDE_NODE,
+    MULTIPLY_NODE,
+    NUMBER_NODE,
+    RANGE_NODE,
+    REDUCE_NODE,
+    REFERENCE_NODE,
+    REFERENCE_SET,
+    SELECT_NODE,
+    SUBTRACTION_NODE,
+};
+
+struct RawAST
+{
+    // Node type
+    NodeType type{NONE};  // Operation to perform on the counter set
+    NodeType operation{NONE};
+
+    // Stores either the name or digit dependening on whether this
+    // is a name or number
+    std::variant<std::monostate, std::string, int64_t> value{std::monostate{}};
+
+    // Counter set of ASTs needed to compute this counter.
+    // Operation is applied to all counters in this set.
+    std::vector<RawAST*> counter_set;
+
+    // Reference set to remove dimensions (such as shader)
+    // from the result. This is a future looking change and
+    // will be unsupported in 6.0.
+    std::vector<RawAST*> reference_set;
+
+    // Range restriction on this node
+    RawAST* range{nullptr};
+
+    ~RawAST()
+    {
+        auto deleteVec = [](auto& vec) {
+            for(auto val : vec)
+            {
+                delete val;
+            }
+        };
+
+        deleteVec(reference_set);
+        deleteVec(counter_set);
+        delete range;
+    }
+
+    // Constructors for raw value types
+    RawAST(NodeType t, const char* v)
+    : type(t)
+    , value(std::string{CHECK_NOTNULL(v)})
+    {}
+
+    RawAST(NodeType t, int64_t v)
+    : type(t)
+    , value(v)
+    {}
+
+    // Reduce/Select operation constructor. Counter is the counter AST
+    // to use for the reduce/select op, op is how to reduce (i.e. SUM,AVG,etc),
+    // refs is the reference set AST. This reference set is copied to flatten
+    // the AST.
+    RawAST(NodeType t, RawAST* counter, const char* op, RawAST* refs = nullptr)
+    : type(t)
+    , value(std::string{CHECK_NOTNULL(op)})
+    , counter_set({counter})
+    {
+        copy_reference_set(refs);
+    }
+
+    RawAST(NodeType t, std::vector<RawAST*> c)
+    : type(t)
+    , counter_set(std::move(c))
+    {}
+
+    // Following two calls are for future reference set settings
+    // for select/reduce ops.
+
+    // Referene set constructor, refs is a pointer to an existing
+    // reference set when multiple references are given (i.e.
+    // shader=X,anotherRef=Y,....).
+    RawAST(NodeType t, const char* v, RawAST* r, RawAST* refs = nullptr)
+    : type(t)
+    , value(std::string{CHECK_NOTNULL(v)})
+    , range(r)
+    {
+        LOG(ERROR) << "BUilding bad ast";
+        copy_reference_set(refs);
+    }
+
+    // Flattens reference set tree into this node.
+    void copy_reference_set(RawAST* ast)
+    {
+        if(!ast) return;
+        reference_set.push_back(ast);
+        reference_set.insert(
+            reference_set.end(), ast->reference_set.begin(), ast->reference_set.end());
+        ast->reference_set.clear();
+    }
+};
+}  // namespace counters
+}  // namespace rocprofiler
+
+namespace fmt
+{
+// fmt::format support for RawAST
+template <>
+struct formatter<rocprofiler::counters::RawAST>
+{
+    template <typename ParseContext>
+    constexpr auto parse(ParseContext& ctx)
+    {
+        return ctx.begin();
+    }
+
+    template <typename Ctx>
+    auto format(rocprofiler::counters::RawAST const& ast, Ctx& ctx) const
+    {
+        static const std::map<rocprofiler::counters::NodeType, std::string> NodeTypeToString = {
+            {rocprofiler::counters::NONE, "NONE"},
+            {rocprofiler::counters::ADDITION_NODE, "ADDITION_NODE"},
+            {rocprofiler::counters::DIVIDE_NODE, "DIVIDE_NODE"},
+            {rocprofiler::counters::MULTIPLY_NODE, "MULTIPLY_NODE"},
+            {rocprofiler::counters::NUMBER_NODE, "NUMBER_NODE"},
+            {rocprofiler::counters::RANGE_NODE, "RANGE_NODE"},
+            {rocprofiler::counters::REDUCE_NODE, "REDUCE_NODE"},
+            {rocprofiler::counters::REFERENCE_NODE, "REFERENCE_NODE"},
+            {rocprofiler::counters::REFERENCE_SET, "REFERENCE_SET"},
+            {rocprofiler::counters::SELECT_NODE, "SELECT_NODE"},
+            {rocprofiler::counters::SUBTRACTION_NODE, "SUBTRACTION_NODE"},
+        };
+
+        auto out = fmt::format_to(ctx.out(),
+                                  "{{\"Type\":\"{}\", \"Operation\":\"{}\",",
+                                  NodeTypeToString.at(ast.type),
+                                  NodeTypeToString.at(ast.operation));
+
+        if(const auto* string_val = std::get_if<std::string>(&ast.value))
+        {
+            out = fmt::format_to(out, " \"Value\":\"{}\",", *string_val);
+        }
+        else if(const auto* int_val = std::get_if<int64_t>(&ast.value))
+        {
+            out = fmt::format_to(out, " \"Value\":{},", *int_val);
+        }
+
+        if(ast.range)
+        {
+            out = fmt::format_to(out, " \"Range\":{},", *ast.range);
+        }
+
+        out = fmt::format_to(out, "\"ReferenceSet\":[");
+        for(const auto& ref : ast.reference_set)
+        {
+            out = fmt::format_to(
+                out, "{}{}", *CHECK_NOTNULL(ref), ref == ast.reference_set.back() ? "" : ",");
+        }
+
+        out = fmt::format_to(out, "], \"CounterSet\":[");
+        for(const auto& ref : ast.counter_set)
+        {
+            out = fmt::format_to(
+                out, "{}{}", *CHECK_NOTNULL(ref), ref == ast.counter_set.back() ? "" : ",");
+        }
+        return fmt::format_to(out, "]}}");
+    }
+};
+}  // namespace fmt
@@ -0,0 +1,12 @@
+#pragma ONCE
+
+#include "parser.h"
+
+// Bison functions for parsers
+typedef struct yy_buffer_state* YY_BUFFER_STATE;
+extern int
+yyparse(rocprofiler::counters::RawAST** result);
+extern YY_BUFFER_STATE
+yy_scan_string(const char* str);
+extern void
+yy_delete_buffer(YY_BUFFER_STATE buffer);
@@ -0,0 +1,47 @@
+%option noyywrap nodefault yylineno nounput
+
+%{
+#include <fmt/core.h>
+
+#include "raw_ast.hpp"
+#include "parser.h"
+using namespace std;
+#define YYDEBUG 1
+%}
+
+/* float exponent */
+EXP	([Ee][-+]?[0-9]+)
+
+%%
+"+"	{ return ADD; }
+"-"	{ return SUB; }
+"*"	{ return MUL; }
+"/"	{ return DIV; }
+"|" { return ABS; }
+"(" { return OP; }
+")" { return CP; }
+"=" { return EQUALS; }
+"," { return CM; }
+":" { return COLON; }
+\[  { return O_SQ; }
+\]  { return C_SQ; }
+
+[0-9]+"."[0-9]*{EXP}? | 
+"."?[0-9]+{EXP}? { 
+    yylval.d = atoi(yytext);
+    return NUMBER; }
+
+"reduce" { return REDUCE; }
+"select" { return SELECT; }
+
+[a-z_A-Z][a-z_A-Z0-9]*  { 
+    yylval.s = strdup(yytext); 
+    return NAME; }
+
+
+\n      { return EOL; }
+"//".*  
+[ \t]   { /* ignore white space */ }
+.	{ throw std::runtime_error(fmt::format("Mystery character {}", *yytext)); }
+%%
+
@@ -0,0 +1,25 @@
+#
+#
+#
+rocprofiler_deactivate_clang_tidy()
+
+include(GoogleTest)
+
+set(ROCPROFILER_LIB_PARSER_TEST_SOURCES "parser_test.cpp")
+
+add_executable(parser-test)
+
+target_sources(parser-test PRIVATE ${ROCPROFILER_LIB_PARSER_TEST_SOURCES})
+
+target_link_libraries(
+    parser-test
+    PRIVATE rocprofiler::rocprofiler-common-library
+            rocprofiler::rocprofiler-static-library GTest::gtest GTest::gtest_main)
+
+gtest_add_tests(
+    TARGET parser-test
+    SOURCES ${ROCPROFILER_LIB_PARSER_TEST_SOURCES}
+    TEST_LIST parser-tests_TESTS
+    WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR})
+
+set_tests_properties(${parser-tests_TESTS} PROPERTIES TIMEOUT 45 LABELS "unittests")
@@ -0,0 +1,192 @@
+#include <map>
+#include <string>
+
+#include <gtest/gtest.h>
+
+#include "lib/rocprofiler/counters/metrics.hpp"
+#include "lib/rocprofiler/counters/parser/reader.hpp"
+
+TEST(parser, base_ops)
+{
+    std::map<std::string, std::string> expressionToExpected = {
+        {"AB + BA",
+         "{\"Type\":\"ADDITION_NODE\", \"Operation\":\"NONE\",\"ReferenceSet\":[], "
+         "\"CounterSet\":[{\"Type\":\"REFERENCE_NODE\", \"Operation\":\"NONE\", "
+         "\"Value\":\"AB\",\"ReferenceSet\":[], \"CounterSet\":[]},{\"Type\":\"REFERENCE_NODE\", "
+         "\"Operation\":\"NONE\", \"Value\":\"BA\",\"ReferenceSet\":[], \"CounterSet\":[]}]}"},
+        {"CD - ZX",
+         "{\"Type\":\"SUBTRACTION_NODE\", \"Operation\":\"NONE\",\"ReferenceSet\":[], "
+         "\"CounterSet\":[{\"Type\":\"REFERENCE_NODE\", \"Operation\":\"NONE\", "
+         "\"Value\":\"CD\",\"ReferenceSet\":[], \"CounterSet\":[]},{\"Type\":\"REFERENCE_NODE\", "
+         "\"Operation\":\"NONE\", \"Value\":\"ZX\",\"ReferenceSet\":[], \"CounterSet\":[]}]}"},
+        {"NM / DB",
+         "{\"Type\":\"DIVIDE_NODE\", \"Operation\":\"NONE\",\"ReferenceSet\":[], "
+         "\"CounterSet\":[{\"Type\":\"REFERENCE_NODE\", \"Operation\":\"NONE\", "
+         "\"Value\":\"NM\",\"ReferenceSet\":[], \"CounterSet\":[]},{\"Type\":\"REFERENCE_NODE\", "
+         "\"Operation\":\"NONE\", \"Value\":\"DB\",\"ReferenceSet\":[], \"CounterSet\":[]}]}"},
+        {"AB * BA",
+         "{\"Type\":\"MULTIPLY_NODE\", \"Operation\":\"NONE\",\"ReferenceSet\":[], "
+         "\"CounterSet\":[{\"Type\":\"REFERENCE_NODE\", \"Operation\":\"NONE\", "
+         "\"Value\":\"AB\",\"ReferenceSet\":[], \"CounterSet\":[]},{\"Type\":\"REFERENCE_NODE\", "
+         "\"Operation\":\"NONE\", \"Value\":\"BA\",\"ReferenceSet\":[], \"CounterSet\":[]}]}"}};
+
+    for(auto [op, expected] : expressionToExpected)
+    {
+        RawAST* ast = nullptr;
+        auto*   buf = yy_scan_string(op.c_str());
+        yyparse(&ast);
+        ASSERT_TRUE(ast);
+        EXPECT_EQ(fmt::format("{}", *ast), expected);
+        yy_delete_buffer(buf);
+        delete ast;
+    }
+}
+
+TEST(parser, order_of_ops)
+{
+    std::map<std::string, std::string> expressionToExpected = {
+        {"(AB + BA) / CD",
+         "{\"Type\":\"DIVIDE_NODE\", \"Operation\":\"NONE\",\"ReferenceSet\":[], "
+         "\"CounterSet\":[{\"Type\":\"ADDITION_NODE\", \"Operation\":\"NONE\",\"ReferenceSet\":[], "
+         "\"CounterSet\":[{\"Type\":\"REFERENCE_NODE\", \"Operation\":\"NONE\", "
+         "\"Value\":\"AB\",\"ReferenceSet\":[], \"CounterSet\":[]},{\"Type\":\"REFERENCE_NODE\", "
+         "\"Operation\":\"NONE\", \"Value\":\"BA\",\"ReferenceSet\":[], "
+         "\"CounterSet\":[]}]},{\"Type\":\"REFERENCE_NODE\", \"Operation\":\"NONE\", "
+         "\"Value\":\"CD\",\"ReferenceSet\":[], \"CounterSet\":[]}]}"},
+        {"AD / (CD - ZX)",
+         "{\"Type\":\"DIVIDE_NODE\", \"Operation\":\"NONE\",\"ReferenceSet\":[], "
+         "\"CounterSet\":[{\"Type\":\"REFERENCE_NODE\", \"Operation\":\"NONE\", "
+         "\"Value\":\"AD\",\"ReferenceSet\":[], \"CounterSet\":[]},{\"Type\":\"SUBTRACTION_NODE\", "
+         "\"Operation\":\"NONE\",\"ReferenceSet\":[], "
+         "\"CounterSet\":[{\"Type\":\"REFERENCE_NODE\", \"Operation\":\"NONE\", "
+         "\"Value\":\"CD\",\"ReferenceSet\":[], \"CounterSet\":[]},{\"Type\":\"REFERENCE_NODE\", "
+         "\"Operation\":\"NONE\", \"Value\":\"ZX\",\"ReferenceSet\":[], \"CounterSet\":[]}]}]}"},
+        {"MN * (NM / DB)",
+         "{\"Type\":\"MULTIPLY_NODE\", \"Operation\":\"NONE\",\"ReferenceSet\":[], "
+         "\"CounterSet\":[{\"Type\":\"REFERENCE_NODE\", \"Operation\":\"NONE\", "
+         "\"Value\":\"MN\",\"ReferenceSet\":[], \"CounterSet\":[]},{\"Type\":\"DIVIDE_NODE\", "
+         "\"Operation\":\"NONE\",\"ReferenceSet\":[], "
+         "\"CounterSet\":[{\"Type\":\"REFERENCE_NODE\", \"Operation\":\"NONE\", "
+         "\"Value\":\"NM\",\"ReferenceSet\":[], \"CounterSet\":[]},{\"Type\":\"REFERENCE_NODE\", "
+         "\"Operation\":\"NONE\", \"Value\":\"DB\",\"ReferenceSet\":[], \"CounterSet\":[]}]}]}"},
+        {"(AB / BA) - BN",
+         "{\"Type\":\"SUBTRACTION_NODE\", \"Operation\":\"NONE\",\"ReferenceSet\":[], "
+         "\"CounterSet\":[{\"Type\":\"DIVIDE_NODE\", \"Operation\":\"NONE\",\"ReferenceSet\":[], "
+         "\"CounterSet\":[{\"Type\":\"REFERENCE_NODE\", \"Operation\":\"NONE\", "
+         "\"Value\":\"AB\",\"ReferenceSet\":[], \"CounterSet\":[]},{\"Type\":\"REFERENCE_NODE\", "
+         "\"Operation\":\"NONE\", \"Value\":\"BA\",\"ReferenceSet\":[], "
+         "\"CounterSet\":[]}]},{\"Type\":\"REFERENCE_NODE\", \"Operation\":\"NONE\", "
+         "\"Value\":\"BN\",\"ReferenceSet\":[], \"CounterSet\":[]}]}"}};
+
+    for(auto [op, expected] : expressionToExpected)
+    {
+        RawAST* ast = nullptr;
+        auto*   buf = yy_scan_string(op.c_str());
+        yyparse(&ast);
+        ASSERT_TRUE(ast);
+        EXPECT_EQ(fmt::format("{}", *ast), expected);
+        yy_delete_buffer(buf);
+        delete ast;
+    }
+}
+
+TEST(parser, reduction)
+{
+    std::map<std::string, std::string> expressionToExpected = {
+        {"reduce(AB, SUM)",
+         "{\"Type\":\"REDUCE_NODE\", \"Operation\":\"NONE\", "
+         "\"Value\":\"SUM\",\"ReferenceSet\":[], \"CounterSet\":[{\"Type\":\"REFERENCE_NODE\", "
+         "\"Operation\":\"NONE\", \"Value\":\"AB\",\"ReferenceSet\":[], \"CounterSet\":[]}]}"},
+        {"reduce(AB+CD, SUM)",
+         "{\"Type\":\"REDUCE_NODE\", \"Operation\":\"NONE\", "
+         "\"Value\":\"SUM\",\"ReferenceSet\":[], \"CounterSet\":[{\"Type\":\"ADDITION_NODE\", "
+         "\"Operation\":\"NONE\",\"ReferenceSet\":[], "
+         "\"CounterSet\":[{\"Type\":\"REFERENCE_NODE\", \"Operation\":\"NONE\", "
+         "\"Value\":\"AB\",\"ReferenceSet\":[], \"CounterSet\":[]},{\"Type\":\"REFERENCE_NODE\", "
+         "\"Operation\":\"NONE\", \"Value\":\"CD\",\"ReferenceSet\":[], \"CounterSet\":[]}]}]}"},
+        {"reduce(AB,DIV)+reduce(DC,SUM)",
+         "{\"Type\":\"ADDITION_NODE\", \"Operation\":\"NONE\",\"ReferenceSet\":[], "
+         "\"CounterSet\":[{\"Type\":\"REDUCE_NODE\", \"Operation\":\"NONE\", "
+         "\"Value\":\"DIV\",\"ReferenceSet\":[], \"CounterSet\":[{\"Type\":\"REFERENCE_NODE\", "
+         "\"Operation\":\"NONE\", \"Value\":\"AB\",\"ReferenceSet\":[], "
+         "\"CounterSet\":[]}]},{\"Type\":\"REDUCE_NODE\", \"Operation\":\"NONE\", "
+         "\"Value\":\"SUM\",\"ReferenceSet\":[], \"CounterSet\":[{\"Type\":\"REFERENCE_NODE\", "
+         "\"Operation\":\"NONE\", \"Value\":\"DC\",\"ReferenceSet\":[], \"CounterSet\":[]}]}]}"},
+        {"reduce(AB, SUM, shader)",
+         "{\"Type\":\"REDUCE_NODE\", \"Operation\":\"NONE\", "
+         "\"Value\":\"SUM\",\"ReferenceSet\":[{\"Type\":\"REFERENCE_NODE\", "
+         "\"Operation\":\"NONE\", \"Value\":\"shader\",\"ReferenceSet\":[], \"CounterSet\":[]}], "
+         "\"CounterSet\":[{\"Type\":\"REFERENCE_NODE\", \"Operation\":\"NONE\", "
+         "\"Value\":\"AB\",\"ReferenceSet\":[], \"CounterSet\":[]}]}"}};
+
+    for(auto [op, expected] : expressionToExpected)
+    {
+        RawAST* ast = nullptr;
+        auto*   buf = yy_scan_string(op.c_str());
+        yyparse(&ast);
+        ASSERT_TRUE(ast);
+        EXPECT_EQ(fmt::format("{}", *ast), expected);
+        yy_delete_buffer(buf);
+        delete ast;
+    }
+}
+
+TEST(parser, selection)
+{
+    std::map<std::string, std::string> expressionToExpected = {
+        {"select(AB, SUM)",
+         "{\"Type\":\"SELECT_NODE\", \"Operation\":\"NONE\", "
+         "\"Value\":\"SUM\",\"ReferenceSet\":[], \"CounterSet\":[{\"Type\":\"REFERENCE_NODE\", "
+         "\"Operation\":\"NONE\", \"Value\":\"AB\",\"ReferenceSet\":[], \"CounterSet\":[]}]}"},
+        {"select(AB+CD, SUM)",
+         "{\"Type\":\"SELECT_NODE\", \"Operation\":\"NONE\", "
+         "\"Value\":\"SUM\",\"ReferenceSet\":[], \"CounterSet\":[{\"Type\":\"ADDITION_NODE\", "
+         "\"Operation\":\"NONE\",\"ReferenceSet\":[], "
+         "\"CounterSet\":[{\"Type\":\"REFERENCE_NODE\", \"Operation\":\"NONE\", "
+         "\"Value\":\"AB\",\"ReferenceSet\":[], \"CounterSet\":[]},{\"Type\":\"REFERENCE_NODE\", "
+         "\"Operation\":\"NONE\", \"Value\":\"CD\",\"ReferenceSet\":[], \"CounterSet\":[]}]}]}"},
+        {"select(AB,DIV)+select(DC,SUM)",
+         "{\"Type\":\"ADDITION_NODE\", \"Operation\":\"NONE\",\"ReferenceSet\":[], "
+         "\"CounterSet\":[{\"Type\":\"SELECT_NODE\", \"Operation\":\"NONE\", "
+         "\"Value\":\"DIV\",\"ReferenceSet\":[], \"CounterSet\":[{\"Type\":\"REFERENCE_NODE\", "
+         "\"Operation\":\"NONE\", \"Value\":\"AB\",\"ReferenceSet\":[], "
+         "\"CounterSet\":[]}]},{\"Type\":\"SELECT_NODE\", \"Operation\":\"NONE\", "
+         "\"Value\":\"SUM\",\"ReferenceSet\":[], \"CounterSet\":[{\"Type\":\"REFERENCE_NODE\", "
+         "\"Operation\":\"NONE\", \"Value\":\"DC\",\"ReferenceSet\":[], \"CounterSet\":[]}]}]}"},
+        {"select(AB, SUM, shader)",
+         "{\"Type\":\"SELECT_NODE\", \"Operation\":\"NONE\", "
+         "\"Value\":\"SUM\",\"ReferenceSet\":[{\"Type\":\"REFERENCE_NODE\", "
+         "\"Operation\":\"NONE\", \"Value\":\"shader\",\"ReferenceSet\":[], \"CounterSet\":[]}], "
+         "\"CounterSet\":[{\"Type\":\"REFERENCE_NODE\", \"Operation\":\"NONE\", "
+         "\"Value\":\"AB\",\"ReferenceSet\":[], \"CounterSet\":[]}]}"}};
+
+    for(auto [op, expected] : expressionToExpected)
+    {
+        RawAST* ast = nullptr;
+        auto*   buf = yy_scan_string(op.c_str());
+        yyparse(&ast);
+        ASSERT_TRUE(ast);
+        EXPECT_EQ(fmt::format("{}", *ast), expected);
+        yy_delete_buffer(buf);
+        delete ast;
+    }
+}
+
+TEST(parser, parse_derived_counters)
+{
+    // Checks that ASTs are properly formed from derived counters defined in XML
+    // Does not check accuracy, only parseability
+    auto derived_counters = rocprofiler::counters::getDerivedHardwareMetrics();
+    for(auto& [gfx, counter_list] : derived_counters)
+    {
+        for(const auto& v : counter_list)
+        {
+            RawAST* ast = nullptr;
+            auto*   buf = yy_scan_string(v.expression().c_str());
+            yyparse(&ast);
+            ASSERT_TRUE(ast);
+            yy_delete_buffer(buf);
+            delete ast;
+        }
+    }
+}
@@ -2,7 +2,7 @@ rocprofiler_deactivate_clang_tidy()

 include(GoogleTest)

-set(ROCPROFILER_LIB_COUNTER_TEST_SOURCES "metrics_test.cpp")
+set(ROCPROFILER_LIB_COUNTER_TEST_SOURCES metrics_test.cpp evaluate_ast_test.cpp)

 add_executable(counter-test)

@@ -0,0 +1,190 @@
+#include <tuple>
+
+#include <gtest/gtest.h>
+
+#include "lib/rocprofiler/counters/evaluate_ast.hpp"
+#include "lib/rocprofiler/counters/parser/reader.hpp"
+
+namespace
+{
+bool
+isIdentical(const EvaluateAST& eval_ast, const RawAST& raw_ast)
+{
+    if(raw_ast.counter_set.size() != eval_ast.children().size() ||
+       raw_ast.type != eval_ast.type() || raw_ast.operation != eval_ast.op())
+    {
+        return false;
+    }
+
+    for(size_t i = 0; i < raw_ast.counter_set.size(); i++)
+    {
+        if(!isIdentical(eval_ast.children()[i], *raw_ast.counter_set[i]))
+        {
+            return false;
+        }
+    }
+    return true;
+}
+}  // namespace
+
+TEST(evaluate_ast, basic_copy)
+{
+    std::unordered_map<std::string, Metric> metrics = {
+        {"SQ_WAVES", Metric("a", "a", "a", "a", "a", "", 0)},
+        {"TCC_HIT", Metric("b", "b", "b", "b", "b", "", 1)}};
+
+    RawAST* ast = nullptr;
+    auto*   buf = yy_scan_string("SQ_WAVES + TCC_HIT");
+    yyparse(&ast);
+    ASSERT_TRUE(ast);
+
+    auto eval_ast = EvaluateAST(metrics, *ast);
+
+    EXPECT_TRUE(isIdentical(eval_ast, *ast));
+    yy_delete_buffer(buf);
+    delete ast;
+}
+
+TEST(evaluate_ast, counter_expansion)
+{
+    std::unordered_map<std::string, Metric> metrics = {
+        {"SQ_WAVES", Metric("SQ_WAVES", "a", "a", "a", "", "", 0)},
+        {"TCC_HIT", Metric("TCC_HIT", "b", "b", "b", "", "", 1)},
+        {"TEST_DERRIVED", Metric("TEST_DERRIVED", "C", "C", "C", "SQ_WAVES+TCC_HIT", "", 2)}};
+
+    std::unordered_map<std::string, EvaluateAST> asts;
+    for(auto [val, metric] : metrics)
+    {
+        RawAST* ast = nullptr;
+        auto    buf = yy_scan_string(metric.expression().empty() ? metric.name().c_str()
+                                                                 : metric.expression().c_str());
+        yyparse(&ast);
+        ASSERT_TRUE(ast);
+        asts.emplace(val, std::move(EvaluateAST(metrics, *ast)));
+        yy_delete_buffer(buf);
+        delete ast;
+    }
+
+    std::set<Metric> required_counters;
+    asts.at("TEST_DERRIVED").get_required_counters(asts, required_counters);
+    EXPECT_EQ(required_counters.size(), 2);
+    auto expected = std::set<Metric>{{Metric("TCC_HIT", "b", "b", "b", "", "", 1),
+                                      Metric("SQ_WAVES", "a", "a", "a", "", "", 0)}};
+
+    for(auto& counter_found : required_counters)
+    {
+        EXPECT_NE(expected.find(counter_found), expected.end());
+    }
+}
+
+TEST(evaluate_ast, counter_expansion_multi_derived)
+{
+    std::unordered_map<std::string, Metric> metrics = {
+        {"SQ_WAVES", Metric("SQ_WAVES", "a", "a", "a", "", "", 0)},
+        {"TCC_HIT", Metric("TCC_HIT", "b", "b", "b", "", "", 1)},
+        {"TEST_DERRIVED", Metric("TEST_DERRIVED", "C", "C", "C", "SQ_WAVES+TCC_HIT", "", 2)},
+        {"TEST_DERRIVED3",
+         Metric("TEST_DERRIVED3", "C", "C", "C", "TEST_DERRIVED+SQ_WAVES+TCC_HIT", "", 3)}};
+
+    std::unordered_map<std::string, EvaluateAST> asts;
+    for(auto [val, metric] : metrics)
+    {
+        RawAST* ast = nullptr;
+        auto    buf = yy_scan_string(metric.expression().empty() ? metric.name().c_str()
+                                                                 : metric.expression().c_str());
+        yyparse(&ast);
+        ASSERT_TRUE(ast);
+        asts.emplace(val, std::move(EvaluateAST(metrics, *ast)));
+        yy_delete_buffer(buf);
+        delete ast;
+    }
+
+    std::set<Metric> required_counters;
+    asts.at("TEST_DERRIVED3").get_required_counters(asts, required_counters);
+    EXPECT_EQ(required_counters.size(), 2);
+    auto expected = std::set<Metric>{{Metric("TCC_HIT", "b", "b", "b", "", "", 1),
+                                      Metric("SQ_WAVES", "a", "a", "a", "", "", 0)}};
+
+    for(auto& counter_found : required_counters)
+    {
+        EXPECT_NE(expected.find(counter_found), expected.end());
+    }
+}
+
+TEST(evaluate_ast, counter_expansion_order)
+{
+    std::unordered_map<std::string, Metric> metrics = {
+        {"SQ_WAVES", Metric("SQ_WAVES", "a", "a", "a", "", "", 0)},
+        {"TCC_HIT", Metric("TCC_HIT", "b", "b", "b", "", "", 1)},
+        {"VLL", Metric("VLL", "b", "b", "b", "", "", 4)},
+        {"TEST_DERRIVED", Metric("TEST_DERRIVED", "C", "C", "C", "SQ_WAVES+VLL", "", 2)},
+        {"TEST_DERRIVED3",
+         Metric("TEST_DERRIVED3", "C", "C", "C", "TEST_DERRIVED+SQ_WAVES+TCC_HIT", "", 3)}};
+
+    std::unordered_map<std::string, EvaluateAST> asts;
+    for(auto [val, metric] : metrics)
+    {
+        RawAST* ast = nullptr;
+        auto    buf = yy_scan_string(metric.expression().empty() ? metric.name().c_str()
+                                                                 : metric.expression().c_str());
+        yyparse(&ast);
+        ASSERT_TRUE(ast);
+        asts.emplace(val, std::move(EvaluateAST(metrics, *ast)));
+        yy_delete_buffer(buf);
+        delete ast;
+    }
+
+    std::set<Metric> required_counters;
+    asts.at("TEST_DERRIVED3").get_required_counters(asts, required_counters);
+    EXPECT_EQ(required_counters.size(), 3);
+    auto expected = std::set<Metric>{{Metric("VLL", "b", "b", "b", "", "", 4),
+                                      Metric("TCC_HIT", "b", "b", "b", "", "", 1),
+                                      Metric("SQ_WAVES", "a", "a", "a", "", "", 0)}};
+
+    for(auto& counter_found : required_counters)
+    {
+        EXPECT_NE(expected.find(counter_found), expected.end());
+    }
+}
+
+TEST(evaluate_ast, counter_expansion_function)
+{
+    std::unordered_map<std::string, Metric> metrics = {
+        {"SQ_WAVES", Metric("SQ_WAVES", "a", "a", "a", "", "", 0)},
+        {"TCC_HIT", Metric("TCC_HIT", "b", "b", "b", "", "", 1)},
+        {"VLL", Metric("VLL", "b", "b", "b", "", "", 4)},
+        {"TEST_DERRIVED", Metric("TEST_DERRIVED", "C", "C", "C", "SQ_WAVES+VLL", "", 2)},
+        {"TEST_DERRIVED3",
+         Metric("TEST_DERRIVED3",
+                "C",
+                "C",
+                "C",
+                "reduce(TEST_DERRIVED,max)+SQ_WAVES+TCC_HIT",
+                "",
+                3)}};
+
+    std::unordered_map<std::string, EvaluateAST> asts;
+    for(auto [val, metric] : metrics)
+    {
+        RawAST* ast = nullptr;
+        auto    buf = yy_scan_string(metric.expression().empty() ? metric.name().c_str()
+                                                                 : metric.expression().c_str());
+        yyparse(&ast);
+        ASSERT_TRUE(ast);
+        asts.emplace(val, std::move(EvaluateAST(metrics, *ast)));
+        yy_delete_buffer(buf);
+        delete ast;
+    }
+
+    std::set<Metric> required_counters;
+    asts.at("TEST_DERRIVED3").get_required_counters(asts, required_counters);
+    EXPECT_EQ(required_counters.size(), 3);
+    auto expected = std::set<Metric>{{Metric("VLL", "b", "b", "b", "", "", 4),
+                                      Metric("TCC_HIT", "b", "b", "b", "", "", 1),
+                                      Metric("SQ_WAVES", "a", "a", "a", "", "", 0)}};
+
+    for(auto& counter_found : required_counters)
+    {
+        EXPECT_NE(expected.find(counter_found), expected.end());
+    }
+}
@@ -6,6 +6,8 @@

 namespace
 {
+namespace counters = ::rocprofiler::counters;
+
 auto
 loadTestData(const std::unordered_map<std::string, std::vector<std::vector<std::string>>>& map)
 {
@@ -15,8 +17,13 @@ loadTestData(const std::unordered_map<std::string, std::vector<std::vector<std::
        auto& metric_vec = ret.emplace(gfx, std::vector<counters::Metric>{}).first->second;
        for(const auto& data_vec : dataMap)
        {
-            metric_vec.emplace_back(
-                data_vec.at(0), data_vec.at(1), data_vec.at(2), data_vec.at(4), data_vec.at(3));
+            metric_vec.emplace_back(data_vec.at(0),
+                                    data_vec.at(1),
+                                    data_vec.at(2),
+                                    data_vec.at(4),
+                                    data_vec.at(3),
+                                    "",
+                                    0);
        }
    }
    return ret;
@@ -25,18 +32,61 @@ loadTestData(const std::unordered_map<std::string, std::vector<std::vector<std::

 TEST(metrics, base_load)
 {
-    auto x         = counters::getBaseHardwareMetrics();
+    auto rocp_data = counters::getBaseHardwareMetrics();
    auto test_data = loadTestData(basic_gfx908);
-    ASSERT_EQ(x.count("gfx908"), 1);
+    ASSERT_EQ(rocp_data.count("gfx908"), 1);
    ASSERT_EQ(test_data.count("gfx908"), 1);
-    EXPECT_EQ(fmt::format("{}", x["gfx908"]), fmt::format("{}", test_data["gfx908"]));
+    auto rocp_data_v = rocp_data.at("gfx908");
+    auto test_data_v = test_data.at("gfx908");
+    EXPECT_EQ(rocp_data_v.size(), test_data_v.size());
+    auto find = [&rocp_data_v](const auto& v) -> std::optional<counters::Metric> {
+        for(const auto& ditr : rocp_data_v)
+            if(ditr.name() == v.name()) return ditr;
+        return std::nullopt;
+    };
+    auto equal = [](const auto& lhs, const auto& rhs) {
+        return std::tie(lhs.name(), lhs.block(), lhs.event(), lhs.description()) ==
+               std::tie(rhs.name(), rhs.block(), rhs.event(), rhs.description());
+    };
+    for(const auto& itr : test_data_v)
+    {
+        auto val = find(itr);
+        if(!val)
+        {
+            EXPECT_TRUE(val) << "failed to find " << fmt::format("{}", itr);
+            continue;
+        }
+        EXPECT_TRUE(equal(itr, *val)) << fmt::format("\n\t{} \n\t\t!= \n\t{}", itr, *val);
+    }
 }

 TEST(metrics, derived_load)
 {
-    auto x         = counters::getDerivedHardwareMetrics();
-    auto test_data = loadTestData(derrived_gfx908);
-    ASSERT_EQ(x.count("gfx908"), 1);
+    auto rocp_data = counters::getDerivedHardwareMetrics();
+    auto test_data = loadTestData(derived_gfx908);
+    ASSERT_EQ(rocp_data.count("gfx908"), 1);
    ASSERT_EQ(test_data.count("gfx908"), 1);
-    EXPECT_EQ(fmt::format("{}", x["gfx908"]), fmt::format("{}", test_data["gfx908"]));
+    auto rocp_data_v = rocp_data.at("gfx908");
+    auto test_data_v = test_data.at("gfx908");
+    EXPECT_EQ(rocp_data_v.size(), test_data_v.size());
+    auto find = [&rocp_data_v](const auto& v) -> std::optional<counters::Metric> {
+        for(const auto& ditr : rocp_data_v)
+            if(ditr.name() == v.name()) return ditr;
+        return std::nullopt;
+    };
+    auto equal = [](const auto& lhs, const auto& rhs) {
+        return std::tie(
+                   lhs.name(), lhs.block(), lhs.event(), lhs.description(), lhs.expression()) ==
+               std::tie(rhs.name(), rhs.block(), rhs.event(), rhs.description(), rhs.expression());
+    };
+    for(const auto& itr : test_data_v)
+    {
+        auto val = find(itr);
+        if(!val)
+        {
+            EXPECT_TRUE(val) << "failed to find " << fmt::format("{}", itr);
+            continue;
+        }
+        EXPECT_TRUE(equal(itr, *val)) << fmt::format("\n\t{} \n\t\t!= \n\t{}", itr, *val);
+    }
 }
@@ -9,7 +9,12 @@
 // Layout is: {name, block, event, expression, description}.
 static const std::unordered_map<std::string, std::vector<std::vector<std::string>>> basic_gfx908 = {
    {"gfx908",
-     {{"SQ_INSTS_VMEM_WR",
+     {{"MAX_WAVE_SIZE", "", "", "1", "Max wave size constant"},
+      {"KERNEL_DURATION", "", "", "1", "The duration of the kernel dispatch"},
+      {"SE_NUM", "", "", "1", "SE_NUM"},
+      {"SIMD_NUM", "", "", "1", "SIMD Number"},
+      {"CU_NUM", "", "", "1", "CU_NUM"},
+      {"SQ_INSTS_VMEM_WR",
       "SQ",
       "28",
       "<None>",
@@ -138,186 +143,192 @@ static const std::unordered_map<std::string, std::vector<std::vector<std::string
       "<None>",
       "TCP stalls TA data interface. Now Windowed."}}}};

-static const std::unordered_map<std::string, std::vector<std::vector<std::string>>>
-    derrived_gfx908 = {
-        {"gfx908",
-         {{"TCC_HIT_sum",
-           "",
-           "",
-           "sum(TCC_HIT,32)",
-           "Number of cache hits. Sum over TCC instances."},
-          {"TCC_MISS_sum",
-           "",
-           "",
-           "sum(TCC_MISS,32)",
-           "Number of cache misses. Sum over TCC instances."},
-          {"TCC_EA_RDREQ_32B_sum",
-           "",
-           "",
-           "sum(TCC_EA_RDREQ_32B,32)",
-           "Number of 32-byte TCC/EA read requests. Sum over TCC instances."},
-          {"TCC_EA_RDREQ_sum",
-           "",
-           "",
-           "sum(TCC_EA_RDREQ,32)",
-           "Number of TCC/EA read requests (either 32-byte or 64-byte). Sum over TCC instances."},
-          {"TCC_EA_WRREQ_sum",
-           "",
-           "",
-           "sum(TCC_EA_WRREQ,32)",
-           "Number of transactions (either 32-byte or 64-byte) going over the TC_EA_wrreq "
-           "interface. Sum over TCC instances."},
-          {"TCC_EA_WRREQ_64B_sum",
-           "",
-           "",
-           "sum(TCC_EA_WRREQ_64B,32)",
-           "Number of 64-byte transactions going (64-byte write or CMPSWAP) over the TC_EA_wrreq "
-           "interface. Sum over TCC instances."},
-          {"TCC_WRREQ_STALL_max",
-           "",
-           "",
-           "max(TCC_EA_WRREQ_STALL,32)",
-           "Number of cycles a write request was stalled. Max over TCC instances."},
-          {"CU_UTILIZATION",
-           "",
-           "",
-           "GRBM_GUI_ACTIVE/GRBM_COUNT",
-           "The total number of active cycles divided by total number of elapsed cycles"},
-          {"KERNEL_DURATION", "", "", "1", "The duration of the kernel dispatch"},
-          {"TA_BUSY_avr",
-           "",
-           "",
-           "avr(TA_TA_BUSY,16)",
-           "TA block is busy. Average over TA instances."},
-          {"TA_BUSY_max", "", "", "max(TA_TA_BUSY,16)", "TA block is busy. Max over TA instances."},
-          {"TA_BUSY_min", "", "", "min(TA_TA_BUSY,16)", "TA block is busy. Min over TA instances."},
-          {"TA_FLAT_READ_WAVEFRONTS_sum",
-           "",
-           "",
-           "sum(TA_FLAT_READ_WAVEFRONTS,16)",
-           "Number of flat opcode reads processed by the TA. Sum over TA instances."},
-          {"TA_FLAT_WRITE_WAVEFRONTS_sum",
-           "",
-           "",
-           "sum(TA_FLAT_WRITE_WAVEFRONTS,16)",
-           "Number of flat opcode writes processed by the TA. Sum over TA instances."},
-          {"TCP_TCP_TA_DATA_STALL_CYCLES_sum",
-           "",
-           "",
-           "sum(TCP_TCP_TA_DATA_STALL_CYCLES,16)",
-           "Total number of TCP stalls TA data interface."},
-          {"TCP_TCP_TA_DATA_STALL_CYCLES_max",
-           "",
-           "",
-           "max(TCP_TCP_TA_DATA_STALL_CYCLES,16)",
-           "Maximum number of TCP stalls TA data interface."},
-          {"FETCH_SIZE",
-           "",
-           "",
-           "(TCC_EA_RDREQ_32B_sum*32+(TCC_EA_RDREQ_sum-TCC_EA_RDREQ_32B_sum)*64)/1024",
-           "The total kilobytes fetched from the video memory. This is measured with all extra "
-           "fetches and any cache or memory effects taken into account."},
-          {"WRITE_SIZE",
-           "",
-           "",
-           "((TCC_EA_WRREQ_sum-TCC_EA_WRREQ_64B_sum)*32+TCC_EA_WRREQ_64B_sum*64)/1024",
-           "The total kilobytes written to the video memory. This is measured with all extra "
-           "fetches and any cache or memory effects taken into account."},
-          {"WRITE_REQ_32B",
-           "",
-           "",
-           "TCC_EA_WRREQ_64B_sum*2+(TCC_EA_WRREQ_sum-TCC_EA_WRREQ_64B_sum)",
-           "The total number of 32-byte effective memory writes."},
-          {"VFetchInsts",
-           "",
-           "",
-           "(SQ_INSTS_VMEM_RD-TA_FLAT_READ_WAVEFRONTS_sum)/SQ_WAVES",
-           "The average number of vector fetch instructions from the video memory executed per "
-           "work-item (affected by flow control). Excludes FLAT instructions that fetch from video "
-           "memory."},
-          {"VWriteInsts",
-           "",
-           "",
-           "(SQ_INSTS_VMEM_WR-TA_FLAT_WRITE_WAVEFRONTS_sum)/SQ_WAVES",
-           "The average number of vector write instructions to the video memory executed per "
-           "work-item (affected by flow control). Excludes FLAT instructions that write to video "
-           "memory."},
-          {"FlatVMemInsts",
-           "",
-           "",
-           "(SQ_INSTS_FLAT-SQ_INSTS_FLAT_LDS_ONLY)/SQ_WAVES",
-           "The average number of FLAT instructions that read from or write to the video memory "
-           "executed per work item (affected by flow control). Includes FLAT instructions that "
-           "read from or write to scratch."},
-          {"LDSInsts",
-           "",
-           "",
-           "(SQ_INSTS_LDS-SQ_INSTS_FLAT_LDS_ONLY)/SQ_WAVES",
-           "The average number of LDS read or LDS write instructions executed per work item "
-           "(affected by flow control).  Excludes FLAT instructions that read from or write to "
-           "LDS."},
-          {"FlatLDSInsts",
-           "",
-           "",
-           "SQ_INSTS_FLAT_LDS_ONLY/SQ_WAVES",
-           "The average number of FLAT instructions that read or write to LDS executed per work "
-           "item (affected by flow control)."},
-          {"VALUUtilization",
-           "",
-           "",
-           "100*SQ_THREAD_CYCLES_VALU/(SQ_ACTIVE_INST_VALU*MAX_WAVE_SIZE)",
-           "The percentage of active vector ALU threads in a wave. A lower number can mean either "
-           "more thread divergence in a wave or that the work-group size is not a multiple of 64. "
-           "Value range: 0\% (bad), 100\% (ideal - no thread divergence)."},
-          {"VALUBusy",
-           "",
-           "",
-           "100*SQ_ACTIVE_INST_VALU*4/SIMD_NUM/GRBM_GUI_ACTIVE",
-           "The percentage of GPUTime vector ALU instructions are processed. Value range: 0\% "
-           "(bad) to 100\% (optimal)."},
-          {"SALUBusy",
-           "",
-           "",
-           "100*SQ_INST_CYCLES_SALU*4/SIMD_NUM/GRBM_GUI_ACTIVE",
-           "The percentage of GPUTime scalar ALU instructions are processed. Value range: 0% (bad) "
-           "to 100% (optimal)."},
-          {"FetchSize",
-           "",
-           "",
-           "FETCH_SIZE",
-           "The total kilobytes fetched from the video memory. This is measured with all extra "
-           "fetches and any cache or memory effects taken into account."},
-          {"WriteSize",
-           "",
-           "",
-           "WRITE_SIZE",
-           "The total kilobytes written to the video memory. This is measured with all extra "
-           "fetches and any cache or memory effects taken into account."},
-          {"MemWrites32B",
-           "",
-           "",
-           "WRITE_REQ_32B",
-           "The total number of effective 32B write transactions to the memory"},
-          {"L2CacheHit",
-           "",
-           "",
-           "100*sum(TCC_HIT,16)/(sum(TCC_HIT,16)+sum(TCC_MISS,16))",
-           "The percentage of fetch, write, atomic, and other instructions that hit the data in L2 "
-           "cache. Value range: 0\% (no hit) to 100\% (optimal)."},
-          {"MemUnitStalled",
-           "",
-           "",
-           "100*TCP_TCP_TA_DATA_STALL_CYCLES_max/GRBM_GUI_ACTIVE/SE_NUM",
-           "The percentage of GPUTime the memory unit is stalled. Try reducing the number or size "
-           "of fetches and writes if possible. Value range: 0\% (optimal) to 100\% (bad)."},
-          {"WriteUnitStalled",
-           "",
-           "",
-           "100*TCC_WRREQ_STALL_max/GRBM_GUI_ACTIVE",
-           "The percentage of GPUTime the Write unit is stalled. Value range: 0\% to 100\% (bad)."},
-          {"LDSBankConflict",
-           "",
-           "",
-           "100*SQ_LDS_BANK_CONFLICT/GRBM_GUI_ACTIVE/CU_NUM",
-           "The percentage of GPUTime LDS is stalled by bank conflicts. Value range: 0\% (optimal) "
-           "to 100\% (bad)."}}}};
+static const std::unordered_map<std::string, std::vector<std::vector<std::string>>> derived_gfx908 =
+    {{"gfx908",
+      {{"TCC_HIT_sum",
+        "",
+        "",
+        "reduce(TCC_HIT,sum)",
+        "Number of cache hits. Sum over TCC instances."},
+       {"TCC_MISS_sum",
+        "",
+        "",
+        "reduce(TCC_MISS,sum)",
+        "Number of cache misses. Sum over TCC instances."},
+       {"TCC_EA_RDREQ_32B_sum",
+        "",
+        "",
+        "reduce(TCC_EA_RDREQ_32B,sum)",
+        "Number of 32-byte TCC/EA read requests. Sum over TCC instances."},
+       {"TCC_EA_RDREQ_sum",
+        "",
+        "",
+        "reduce(TCC_EA_RDREQ,sum)",
+        "Number of TCC/EA read requests (either 32-byte or 64-byte). Sum over TCC instances."},
+       {"TCC_EA_WRREQ_sum",
+        "",
+        "",
+        "reduce(TCC_EA_WRREQ,sum)",
+        "Number of transactions (either 32-byte or 64-byte) going over the TC_EA_wrreq "
+        "interface. Sum over TCC instances."},
+       {"TCC_EA_WRREQ_64B_sum",
+        "",
+        "",
+        "reduce(TCC_EA_WRREQ_64B,sum)",
+        "Number of 64-byte transactions going (64-byte write or CMPSWAP) over the TC_EA_wrreq "
+        "interface. Sum over TCC instances."},
+       {"TCC_WRREQ_STALL_max",
+        "",
+        "",
+        "reduce(TCC_EA_WRREQ_STALL,max)",
+        "Number of cycles a write request was stalled. Max over TCC instances."},
+       {"CU_UTILIZATION",
+        "",
+        "",
+        "GRBM_GUI_ACTIVE/GRBM_COUNT",
+        "The total number of active cycles divided by total number of elapsed cycles"},
+       {"TA_BUSY_avr",
+        "",
+        "",
+        "reduce(TA_TA_BUSY,average)",
+        "TA block is busy. Average over TA instances."},
+       {"TA_BUSY_max",
+        "",
+        "",
+        "reduce(TA_TA_BUSY,max)",
+        "TA block is busy. Max over TA instances."},
+       {"TA_BUSY_min",
+        "",
+        "",
+        "reduce(TA_TA_BUSY,min)",
+        "TA block is busy. Min over TA instances."},
+       {"TA_FLAT_READ_WAVEFRONTS_sum",
+        "",
+        "",
+        "reduce(TA_FLAT_READ_WAVEFRONTS,sum)",
+        "Number of flat opcode reads processed by the TA. Sum over TA instances."},
+       {"TA_FLAT_WRITE_WAVEFRONTS_sum",
+        "",
+        "",
+        "reduce(TA_FLAT_WRITE_WAVEFRONTS,sum)",
+        "Number of flat opcode writes processed by the TA. Sum over TA instances."},
+       {"TCP_TCP_TA_DATA_STALL_CYCLES_sum",
+        "",
+        "",
+        "reduce(TCP_TCP_TA_DATA_STALL_CYCLES,sum)",
+        "Total number of TCP stalls TA data interface."},
+       {"TCP_TCP_TA_DATA_STALL_CYCLES_max",
+        "",
+        "",
+        "reduce(TCP_TCP_TA_DATA_STALL_CYCLES,max)",
+        "Maximum number of TCP stalls TA data interface."},
+       {"FETCH_SIZE",
+        "",
+        "",
+        "(TCC_EA_RDREQ_32B_sum*32+(TCC_EA_RDREQ_sum-TCC_EA_RDREQ_32B_sum)*64)/1024",
+        "The total kilobytes fetched from the video memory. This is measured with all extra "
+        "fetches and any cache or memory effects taken into account."},
+       {"WRITE_SIZE",
+        "",
+        "",
+        "((TCC_EA_WRREQ_sum-TCC_EA_WRREQ_64B_sum)*32+TCC_EA_WRREQ_64B_sum*64)/1024",
+        "The total kilobytes written to the video memory. This is measured with all extra "
+        "fetches and any cache or memory effects taken into account."},
+       {"WRITE_REQ_32B",
+        "",
+        "",
+        "TCC_EA_WRREQ_64B_sum*2+(TCC_EA_WRREQ_sum-TCC_EA_WRREQ_64B_sum)",
+        "The total number of 32-byte effective memory writes."},
+       {"VFetchInsts",
+        "",
+        "",
+        "(SQ_INSTS_VMEM_RD-TA_FLAT_READ_WAVEFRONTS_sum)/SQ_WAVES",
+        "The average number of vector fetch instructions from the video memory executed per "
+        "work-item (affected by flow control). Excludes FLAT instructions that fetch from video "
+        "memory."},
+       {"VWriteInsts",
+        "",
+        "",
+        "(SQ_INSTS_VMEM_WR-TA_FLAT_WRITE_WAVEFRONTS_sum)/SQ_WAVES",
+        "The average number of vector write instructions to the video memory executed per "
+        "work-item (affected by flow control). Excludes FLAT instructions that write to video "
+        "memory."},
+       {"FlatVMemInsts",
+        "",
+        "",
+        "(SQ_INSTS_FLAT-SQ_INSTS_FLAT_LDS_ONLY)/SQ_WAVES",
+        "The average number of FLAT instructions that read from or write to the video memory "
+        "executed per work item (affected by flow control). Includes FLAT instructions that "
+        "read from or write to scratch."},
+       {"LDSInsts",
+        "",
+        "",
+        "(SQ_INSTS_LDS-SQ_INSTS_FLAT_LDS_ONLY)/SQ_WAVES",
+        "The average number of LDS read or LDS write instructions executed per work item "
+        "(affected by flow control).  Excludes FLAT instructions that read from or write to "
+        "LDS."},
+       {"FlatLDSInsts",
+        "",
+        "",
+        "SQ_INSTS_FLAT_LDS_ONLY/SQ_WAVES",
+        "The average number of FLAT instructions that read or write to LDS executed per work "
+        "item (affected by flow control)."},
+       {"VALUUtilization",
+        "",
+        "",
+        "100*SQ_THREAD_CYCLES_VALU/(SQ_ACTIVE_INST_VALU*MAX_WAVE_SIZE)",
+        "The percentage of active vector ALU threads in a wave. A lower number can mean either "
+        "more thread divergence in a wave or that the work-group size is not a multiple of 64. "
+        "Value range: 0\% (bad), 100\% (ideal - no thread divergence)."},
+       {"VALUBusy",
+        "",
+        "",
+        "100*SQ_ACTIVE_INST_VALU*4/SIMD_NUM/GRBM_GUI_ACTIVE",
+        "The percentage of GPUTime vector ALU instructions are processed. Value range: 0\% "
+        "(bad) to 100\% (optimal)."},
+       {"SALUBusy",
+        "",
+        "",
+        "100*SQ_INST_CYCLES_SALU*4/SIMD_NUM/GRBM_GUI_ACTIVE",
+        "The percentage of GPUTime scalar ALU instructions are processed. Value range: 0% (bad) "
+        "to 100% (optimal)."},
+       {"FetchSize",
+        "",
+        "",
+        "FETCH_SIZE",
+        "The total kilobytes fetched from the video memory. This is measured with all extra "
+        "fetches and any cache or memory effects taken into account."},
+       {"WriteSize",
+        "",
+        "",
+        "WRITE_SIZE",
+        "The total kilobytes written to the video memory. This is measured with all extra "
+        "fetches and any cache or memory effects taken into account."},
+       {"MemWrites32B",
+        "",
+        "",
+        "WRITE_REQ_32B",
+        "The total number of effective 32B write transactions to the memory"},
+       {"L2CacheHit",
+        "",
+        "",
+        "100*reduce(TCC_HIT,sum)/(reduce(TCC_HIT,sum)+reduce(TCC_MISS,sum))",
+        "The percentage of fetch, write, atomic, and other instructions that hit the data in L2 "
+        "cache. Value range: 0\% (no hit) to 100\% (optimal)."},
+       {"MemUnitStalled",
+        "",
+        "",
+        "100*TCP_TCP_TA_DATA_STALL_CYCLES_max/GRBM_GUI_ACTIVE/SE_NUM",
+        "The percentage of GPUTime the memory unit is stalled. Try reducing the number or size "
+        "of fetches and writes if possible. Value range: 0\% (optimal) to 100\% (bad)."},
+       {"WriteUnitStalled",
+        "",
+        "",
+        "100*TCC_WRREQ_STALL_max/GRBM_GUI_ACTIVE",
+        "The percentage of GPUTime the Write unit is stalled. Value range: 0\% to 100\% (bad)."},
+       {"LDSBankConflict",
+        "",
+        "",
+        "100*SQ_LDS_BANK_CONFLICT/GRBM_GUI_ACTIVE/CU_NUM",
+        "The percentage of GPUTime LDS is stalled by bank conflicts. Value range: 0\% (optimal) "
+        "to 100\% (bad)."}}}};
@@ -1,3 +1,10 @@
-configure_file(basic_counters.xml ${PROJECT_BINARY_DIR}/lib/basic_counters.xml COPYONLY)
-configure_file(derived_counters.xml ${PROJECT_BINARY_DIR}/lib/derived_counters.xml
-               COPYONLY)
+configure_file(basic_counters.xml
+               ${PROJECT_BINARY_DIR}/share/rocprofiler/basic_counters.xml COPYONLY)
+configure_file(derived_counters.xml
+               ${PROJECT_BINARY_DIR}/share/rocprofiler/derived_counters.xml COPYONLY)
+
+install(
+    FILES ${PROJECT_BINARY_DIR}/share/rocprofiler/basic_counters.xml
+          ${PROJECT_BINARY_DIR}/share/rocprofiler/derived_counters.xml
+    DESTINATION share/rocprofiler
+    COMPONENT core)
@@ -1,4 +1,9 @@
 <gfx8 base="gfx8">
+  <metric name="MAX_WAVE_SIZE"  descr="Max wave size constant" special="yes"></metric>
+  <metric name="KERNEL_DURATION" descr="The duration of the kernel dispatch" special="yes"></metric>
+  <metric name="SE_NUM" descr="SE_NUM" special="yes"></metric>
+  <metric name="SIMD_NUM" descr="SIMD Number" special="yes"></metric>
+  <metric name="CU_NUM" descr="CU_NUM" special="yes"></metric>
  <metric name="GRBM_COUNT" block=GRBM event=0 descr="Tie High - Count Number of Clocks"></metric>
  <metric name="GRBM_GUI_ACTIVE" block=GRBM event=2 descr="The GUI is Active"></metric>

@@ -33,6 +38,11 @@
 </gfx8>

 <gfx9>
+  <metric name="MAX_WAVE_SIZE"  descr="Max wave size constant" special="yes"></metric>
+  <metric name="KERNEL_DURATION" descr="The duration of the kernel dispatch" special="yes"></metric>
+  <metric name="SE_NUM" descr="SE_NUM" special="yes"></metric>
+  <metric name="SIMD_NUM" descr="SIMD Number" special="yes"></metric>
+  <metric name="CU_NUM" descr="CU_NUM" special="yes"></metric>
  <metric name="GRBM_COUNT" block=GRBM event=0 descr="Tie High - Count Number of Clocks"></metric>
  <metric name="GRBM_GUI_ACTIVE" block=GRBM event=2 descr="The GUI is Active"></metric>

@@ -70,6 +80,11 @@

 <gfx906 base="gfx9">
  # EA1
+  <metric name="MAX_WAVE_SIZE"  descr="Max wave size constant" special="yes"></metric>
+  <metric name="KERNEL_DURATION" descr="The duration of the kernel dispatch" special="yes"></metric>
+  <metric name="SE_NUM" descr="SE_NUM" special="yes"></metric>
+  <metric name="SIMD_NUM" descr="SIMD Number" special="yes"></metric>
+  <metric name="CU_NUM" descr="CU_NUM" special="yes"></metric>
  <metric name="TCC_EA1_WRREQ" block=TCC event=256 descr="Number of transactions (either 32-byte or 64-byte) going over the TC_EA_wrreq interface. Atomics may travel over the same interface and are generally classified as write requests. This does not include probe commands."></metric>
  <metric name="TCC_EA1_WRREQ_64B" block=TCC event=257 descr="Number of 64-byte transactions going (64-byte write or CMPSWAP) over the TC_EA_wrreq interface."></metric>
  <metric name="TCC_EA1_WRREQ_STALL" block=TCC event=260 descr="Number of cycles a write request was stalled."></metric>
@@ -103,6 +118,11 @@
 </gfx908>

 <gfx90a>
+  <metric name="MAX_WAVE_SIZE"  descr="Max wave size constant" special="yes"></metric>
+  <metric name="KERNEL_DURATION" descr="The duration of the kernel dispatch" special="yes"></metric>
+  <metric name="SE_NUM" descr="SE_NUM" special="yes"></metric>
+  <metric name="SIMD_NUM" descr="SIMD Number" special="yes"></metric>
+  <metric name="CU_NUM" descr="CU_NUM" special="yes"></metric>
  <metric name="SQ_WAIT_INST_LDS" block=SQ event=91 descr="Number of wave-cycles spent waiting for LDS instruction issue. In units of 4 cycles. (per-simd, nondeterministic)"></metric>
  <metric name="TCP_TCP_TA_DATA_STALL_CYCLES" block=TCP event=6 descr="TCP stalls TA data interface. Now Windowed."></metric>
  <metric name="GRBM_COUNT" block=GRBM event=0 descr="Tie High - Count Number of Clocks"></metric>
@@ -366,6 +386,11 @@
 </gfx90a>

 <gfx940>
+  <metric name="MAX_WAVE_SIZE"  descr="Max wave size constant" special="yes"></metric>
+  <metric name="KERNEL_DURATION" descr="The duration of the kernel dispatch" special="yes"></metric>
+  <metric name="SE_NUM" descr="SE_NUM" special="yes"></metric>
+  <metric name="SIMD_NUM" descr="SIMD Number" special="yes"></metric>
+  <metric name="CU_NUM" descr="CU_NUM" special="yes"></metric>
  <metric name="SQ_WAIT_INST_LDS" block=SQ event=96 descr="Number of wave-cycles spent waiting for LDS instruction issue. In units of 4 cycles. (per-simd, nondeterministic)"></metric>
  <metric name="TCP_TCP_TA_DATA_STALL_CYCLES" block=TCP event=6 descr="TCP stalls TA data interface. Now Windowed."></metric>
  <metric name="GRBM_COUNT" block=GRBM event=0 descr="Tie High - Count Number of Clocks"></metric>
@@ -634,6 +659,11 @@
 <gfx942 base="gfx940"></gfx942>

 <gfx10>
+  <metric name="MAX_WAVE_SIZE"  descr="Max wave size constant" special="yes"></metric>
+  <metric name="KERNEL_DURATION" descr="The duration of the kernel dispatch" special="yes"></metric>
+  <metric name="SE_NUM" descr="SE_NUM" special="yes"></metric>
+  <metric name="SIMD_NUM" descr="SIMD Number" special="yes"></metric>
+  <metric name="CU_NUM" descr="CU_NUM" special="yes"></metric>
  <metric name="GRBM_COUNT" block=GRBM event=0 descr="Tie High - Count Number of Clocks"></metric>
  <metric name="GRBM_GUI_ACTIVE" block=GRBM event=2 descr="The GUI is Active"></metric>
  <metric name="GRBM_CP_BUSY" block=GRBM event=3 descr="Any of the Command Processor (CPG/CPC/CPF) blocks are busy."></metric>
@@ -678,7 +708,7 @@
  <metric name="SQ_INSTS_LDS" block=SQ event=59 descr="Number of LDS indexed instructions issued. {emulated, C1}"></metric>
  <metric name="SQ_INSTS_GDS" block=SQ event=55 descr="Number of GDS instructions issued. {emulated, C1}"></metric>
  <metric name="SQ_WAIT_INST_LDS" block=SQ event=31 descr="Number of clock cycles spent waiting for LDS (indexed) instruction issue. In units of cycles. {nondeterministic, C1}"></metric>
-  
+
  <metric name="TA_TA_BUSY" block=TA event=15 descr="TA block is busy. Perf_Windowing not supported for this counter."></metric>
  <metric name="TA_FLAT_LOAD_WAVEFRONTS" block=TA event=101 descr=" Number of flat load vec32 packets processed by TA, same as flat_read_wavefronts in earlier IP"></metric>
  <metric name="TA_FLAT_STORE_WAVEFRONTS" block=TA event=102 descr="Number of flat store vec32 packets processed by TA, same as flat_write_wavefronts in earlier IP"></metric>
@@ -694,6 +724,11 @@
 </gfx1032>

 <gfx11>
+  <metric name="MAX_WAVE_SIZE"  descr="Max wave size constant" special="yes"></metric>
+  <metric name="KERNEL_DURATION" descr="The duration of the kernel dispatch" special="yes"></metric>
+  <metric name="SE_NUM" descr="SE_NUM" special="yes"></metric>
+  <metric name="SIMD_NUM" descr="SIMD Number" special="yes"></metric>
+  <metric name="CU_NUM" descr="CU_NUM" special="yes"></metric>
  <metric name="GRBM_COUNT" block=GRBM event=0 descr="Tie High - Count Number of Clocks"></metric>
  <metric name="GRBM_GUI_ACTIVE" block=GRBM event=2 descr="The GUI is Active"></metric>
  <metric name="GL2C_HIT" block=GL2C event=42 descr="Number of cache hits"></metric>
@@ -741,4 +776,3 @@

 <gfx1101 base="gfx11">
 </gfx1101>
-
@@ -1,16 +1,15 @@
-<gfx8_expr>
-  <metric name="KERNEL_DURATION" expr=1 descr="The duration of the kernel dispatch"></metric>
-  <metric name="TA_BUSY_avr" expr=avr(TA_TA_BUSY,16) descr="TA block is busy. Average over TA instances."></metric>
-  <metric name="TA_BUSY_max" expr=max(TA_TA_BUSY,16) descr="TA block is busy. Max over TA instances."></metric>
-  <metric name="TA_BUSY_min" expr=min(TA_TA_BUSY,16) descr="TA block is busy. Min over TA instances."></metric>
-  <metric name="TA_FLAT_READ_WAVEFRONTS_sum" expr=sum(TA_FLAT_READ_WAVEFRONTS,16) descr="Number of flat opcode reads processed by the TA. Sum over TA instances."></metric>
-  <metric name="TA_FLAT_WRITE_WAVEFRONTS_sum" expr=sum(TA_FLAT_WRITE_WAVEFRONTS,16) descr="Number of flat opcode writes processed by the TA. Sum over TA instances."></metric>
+<gfx8>
+  <metric name="TA_BUSY_avr" expr=reduce(TA_TA_BUSY,average) descr="TA block is busy. Average over TA instances."></metric>
+  <metric name="TA_BUSY_max" expr=reduce(TA_TA_BUSY,max) descr="TA block is busy. Max over TA instances."></metric>
+  <metric name="TA_BUSY_min" expr=reduce(TA_TA_BUSY,min) descr="TA block is busy. Min over TA instances."></metric>
+  <metric name="TA_FLAT_READ_WAVEFRONTS_sum" expr=reduce(TA_FLAT_READ_WAVEFRONTS,sum) descr="Number of flat opcode reads processed by the TA. Sum over TA instances."></metric>
+  <metric name="TA_FLAT_WRITE_WAVEFRONTS_sum" expr=reduce(TA_FLAT_WRITE_WAVEFRONTS,sum) descr="Number of flat opcode writes processed by the TA. Sum over TA instances."></metric>

-  <metric name="TCC_HIT_sum" expr=sum(TCC_HIT,16) descr="Number of cache hits. Sum over TCC instances."></metric>
-  <metric name="TCC_MISS_sum" expr=sum(TCC_MISS,16) descr="Number of cache misses. Sum over TCC instances."></metric>
-  <metric name="TCC_MC_RDREQ_sum" expr=sum(TCC_MC_RDREQ,16) descr="Number of 32-byte reads. Sum over TCC instaces."></metric>
-  <metric name="TCC_MC_WRREQ_sum" expr=sum(TCC_MC_WRREQ,16) descr="Number of 32-byte transactions going over the TC_MC_wrreq interface. Sum over TCC instaces."></metric>
-  <metric name="TCC_WRREQ_STALL_max" expr=max(TCC_MC_WRREQ_STALL,16) descr="Number of cycles a write request was stalled. Max over TCC instances."></metric>
+  <metric name="TCC_HIT_sum" expr=reduce(TCC_HIT,sum) descr="Number of cache hits. Sum over TCC instances."></metric>
+  <metric name="TCC_MISS_sum" expr=reduce(TCC_MISS,sum) descr="Number of cache misses. Sum over TCC instances."></metric>
+  <metric name="TCC_MC_RDREQ_sum" expr=reduce(TCC_MC_RDREQ,sum) descr="Number of 32-byte reads. Sum over TCC instaces."></metric>
+  <metric name="TCC_MC_WRREQ_sum" expr=reduce(TCC_MC_WRREQ,sum) descr="Number of 32-byte transactions going over the TC_MC_wrreq interface. Sum over TCC instaces."></metric>
+  <metric name="TCC_WRREQ_STALL_max" expr=reduce(TCC_MC_WRREQ_STALL,max) descr="Number of cycles a write request was stalled. Max over TCC instances."></metric>

  <metric name="FETCH_SIZE" expr=(TCC_MC_RDREQ_sum*32)/1024 descr="The total kilobytes fetched from the video memory. This is measured with all extra fetches and any cache or memory effects taken into account."></metric>
  <metric name="WRITE_SIZE" expr=(TCC_MC_WRREQ_sum*32)/1024 descr="The total kilobytes written to the video memory. This is measured with all extra fetches and any cache or memory effects taken into account."></metric>
@@ -26,31 +25,30 @@
  <metric name="FetchSize" expr=FETCH_SIZE descr="The total kilobytes fetched from the video memory. This is measured with all extra fetches and any cache or memory effects taken into account."></metric>
  <metric name="WriteSize" expr=WRITE_SIZE descr="The total kilobytes written to the video memory. This is measured with all extra fetches and any cache or memory effects taken into account."></metric>
  <metric name="MemWrites32B" expr=WRITE_REQ_32B descr="The total number of effective 32B write transactions to the memory"></metric>
-  <metric name="L2CacheHit" expr=100*sum(TCC_HIT,16)/(sum(TCC_HIT,16)+sum(TCC_MISS,16)) descr="The percentage of fetch, write, atomic, and other instructions that hit the data in L2 cache. Value range: 0% (no hit) to 100% (optimal)."></metric>
-  <metric name="MemUnitStalled" expr=100*max(TCP_TCP_TA_DATA_STALL_CYCLES,16)/GRBM_GUI_ACTIVE/SE_NUM descr="The percentage of GPUTime the memory unit is stalled. Try reducing the number or size of fetches and writes if possible. Value range: 0% (optimal) to 100% (bad)."></metric>
+  <metric name="L2CacheHit" expr=100*reduce(TCC_HIT,sum)/(reduce(TCC_HIT,sum)+reduce(TCC_MISS,sum)) descr="The percentage of fetch, write, atomic, and other instructions that hit the data in L2 cache. Value range: 0% (no hit) to 100% (optimal)."></metric>
+  <metric name="MemUnitStalled" expr=100*reduce(TCP_TCP_TA_DATA_STALL_CYCLES,max)/GRBM_GUI_ACTIVE/SE_NUM descr="The percentage of GPUTime the memory unit is stalled. Try reducing the number or size of fetches and writes if possible. Value range: 0% (optimal) to 100% (bad)."></metric>
  <metric name="WriteUnitStalled" expr=100*TCC_WRREQ_STALL_max/GRBM_GUI_ACTIVE descr="The percentage of GPUTime the Write unit is stalled. Value range: 0% to 100% (bad)."></metric>
  # LDSBankConflict The percentage of GPUTime LDS is stalled by bank conflicts. Value range: 0% (optimal) to 100% (bad).
  <metric name="LDSBankConflict" expr=100*SQ_LDS_BANK_CONFLICT/GRBM_GUI_ACTIVE/CU_NUM descr="The percentage of GPUTime LDS is stalled by bank conflicts. Value range: 0% (optimal) to 100% (bad)."></metric>
-</gfx8_expr>
+</gfx8>

-<gfx9_expr>
-  <metric name="KERNEL_DURATION" expr=1 descr="The duration of the kernel dispatch"></metric>
-  <metric name="TA_BUSY_avr" expr=avr(TA_TA_BUSY,16) descr="TA block is busy. Average over TA instances."></metric>
-  <metric name="TA_BUSY_max" expr=max(TA_TA_BUSY,16) descr="TA block is busy. Max over TA instances."></metric>
-  <metric name="TA_BUSY_min" expr=min(TA_TA_BUSY,16) descr="TA block is busy. Min over TA instances."></metric>
-  <metric name="TA_FLAT_READ_WAVEFRONTS_sum" expr=sum(TA_FLAT_READ_WAVEFRONTS,16) descr="Number of flat opcode reads processed by the TA. Sum over TA instances."></metric>
-  <metric name="TA_FLAT_WRITE_WAVEFRONTS_sum" expr=sum(TA_FLAT_WRITE_WAVEFRONTS,16) descr="Number of flat opcode writes processed by the TA. Sum over TA instances."></metric>
+<gfx9>
+  <metric name="TA_BUSY_avr" expr=reduce(TA_TA_BUSY,average) descr="TA block is busy. Average over TA instances."></metric>
+  <metric name="TA_BUSY_max" expr=reduce(TA_TA_BUSY,max) descr="TA block is busy. Max over TA instances."></metric>
+  <metric name="TA_BUSY_min" expr=reduce(TA_TA_BUSY,min) descr="TA block is busy. Min over TA instances."></metric>
+  <metric name="TA_FLAT_READ_WAVEFRONTS_sum" expr=reduce(TA_FLAT_READ_WAVEFRONTS,sum) descr="Number of flat opcode reads processed by the TA. Sum over TA instances."></metric>
+  <metric name="TA_FLAT_WRITE_WAVEFRONTS_sum" expr=reduce(TA_FLAT_WRITE_WAVEFRONTS,sum) descr="Number of flat opcode writes processed by the TA. Sum over TA instances."></metric>

-  <metric name="TCC_HIT_sum" expr=sum(TCC_HIT,16) descr="Number of cache hits. Sum over TCC instances."></metric>
-  <metric name="TCC_MISS_sum" expr=sum(TCC_MISS,16) descr="Number of cache misses. Sum over TCC instances."></metric>
-  <metric name="TCC_EA_RDREQ_32B_sum" expr=sum(TCC_EA_RDREQ_32B,16) descr="Number of 32-byte TCC/EA read requests. Sum over TCC instances."></metric>
-  <metric name="TCC_EA_RDREQ_sum" expr=sum(TCC_EA_RDREQ,16) descr="Number of TCC/EA read requests (either 32-byte or 64-byte). Sum over TCC instances."></metric>
-  <metric name="TCC_EA_WRREQ_sum" expr=sum(TCC_EA_WRREQ,16) descr="Number of transactions (either 32-byte or 64-byte) going over the TC_EA_wrreq interface. Sum over TCC instances."></metric>
-  <metric name="TCC_EA_WRREQ_64B_sum" expr=sum(TCC_EA_WRREQ_64B,16) descr="Number of 64-byte transactions going (64-byte write or CMPSWAP) over the TC_EA_wrreq interface. Sum over TCC instances."></metric>
-  <metric name="TCC_WRREQ_STALL_max" expr=max(TCC_EA_WRREQ_STALL,16) descr="Number of cycles a write request was stalled. Max over TCC instances."></metric>
+  <metric name="TCC_HIT_sum" expr=reduce(TCC_HIT,sum) descr="Number of cache hits. Sum over TCC instances."></metric>
+  <metric name="TCC_MISS_sum" expr=reduce(TCC_MISS,sum) descr="Number of cache misses. Sum over TCC instances."></metric>
+  <metric name="TCC_EA_RDREQ_32B_sum" expr=reduce(TCC_EA_RDREQ_32B,sum) descr="Number of 32-byte TCC/EA read requests. Sum over TCC instances."></metric>
+  <metric name="TCC_EA_RDREQ_sum" expr=reduce(TCC_EA_RDREQ,sum) descr="Number of TCC/EA read requests (either 32-byte or 64-byte). Sum over TCC instances."></metric>
+  <metric name="TCC_EA_WRREQ_sum" expr=reduce(TCC_EA_WRREQ,sum) descr="Number of transactions (either 32-byte or 64-byte) going over the TC_EA_wrreq interface. Sum over TCC instances."></metric>
+  <metric name="TCC_EA_WRREQ_64B_sum" expr=reduce(TCC_EA_WRREQ_64B,sum) descr="Number of 64-byte transactions going (64-byte write or CMPSWAP) over the TC_EA_wrreq interface. Sum over TCC instances."></metric>
+  <metric name="TCC_WRREQ_STALL_max" expr=reduce(TCC_EA_WRREQ_STALL,max) descr="Number of cycles a write request was stalled. Max over TCC instances."></metric>

-  <metric name="TCP_TCP_TA_DATA_STALL_CYCLES_sum" expr=sum(TCP_TCP_TA_DATA_STALL_CYCLES,16) descr="Total number of TCP stalls TA data interface."></metric>
-  <metric name="TCP_TCP_TA_DATA_STALL_CYCLES_max" expr=max(TCP_TCP_TA_DATA_STALL_CYCLES,16) descr="Maximum number of TCP stalls TA data interface."></metric>
+  <metric name="TCP_TCP_TA_DATA_STALL_CYCLES_sum" expr=reduce(TCP_TCP_TA_DATA_STALL_CYCLES,sum) descr="Total number of TCP stalls TA data interface."></metric>
+  <metric name="TCP_TCP_TA_DATA_STALL_CYCLES_max" expr=reduce(TCP_TCP_TA_DATA_STALL_CYCLES,max) descr="Maximum number of TCP stalls TA data interface."></metric>

  <metric name="FETCH_SIZE" expr=(TCC_EA_RDREQ_32B_sum*32+(TCC_EA_RDREQ_sum-TCC_EA_RDREQ_32B_sum)*64)/1024 descr="The total kilobytes fetched from the video memory. This is measured with all extra fetches and any cache or memory effects taken into account."></metric>
  <metric name="WRITE_SIZE" expr=((TCC_EA_WRREQ_sum-TCC_EA_WRREQ_64B_sum)*32+TCC_EA_WRREQ_64B_sum*64)/1024 descr="The total kilobytes written to the video memory. This is measured with all extra fetches and any cache or memory effects taken into account."></metric>
@@ -66,20 +64,20 @@
  <metric name="FetchSize" expr=FETCH_SIZE descr="The total kilobytes fetched from the video memory. This is measured with all extra fetches and any cache or memory effects taken into account."></metric>
  <metric name="WriteSize" expr=WRITE_SIZE descr="The total kilobytes written to the video memory. This is measured with all extra fetches and any cache or memory effects taken into account."></metric>
  <metric name="MemWrites32B" expr=WRITE_REQ_32B descr="The total number of effective 32B write transactions to the memory"></metric>
-  <metric name="L2CacheHit" expr=100*sum(TCC_HIT,16)/(sum(TCC_HIT,16)+sum(TCC_MISS,16)) descr="The percentage of fetch, write, atomic, and other instructions that hit the data in L2 cache. Value range: 0% (no hit) to 100% (optimal)."></metric>
+  <metric name="L2CacheHit" expr=100*reduce(TCC_HIT,sum)/(reduce(TCC_HIT,sum)+reduce(TCC_MISS,sum)) descr="The percentage of fetch, write, atomic, and other instructions that hit the data in L2 cache. Value range: 0% (no hit) to 100% (optimal)."></metric>
  <metric name="MemUnitStalled" expr=100*TCP_TCP_TA_DATA_STALL_CYCLES_max/GRBM_GUI_ACTIVE/SE_NUM descr="The percentage of GPUTime the memory unit is stalled. Try reducing the number or size of fetches and writes if possible. Value range: 0% (optimal) to 100% (bad)."></metric>
  <metric name="WriteUnitStalled" expr=100*TCC_WRREQ_STALL_max/GRBM_GUI_ACTIVE descr="The percentage of GPUTime the Write unit is stalled. Value range: 0% to 100% (bad)."></metric>
  # LDSBankConflict The percentage of GPUTime LDS is stalled by bank conflicts. Value range: 0% (optimal) to 100% (bad).
  <metric name="LDSBankConflict" expr=100*SQ_LDS_BANK_CONFLICT/GRBM_GUI_ACTIVE/CU_NUM descr="The percentage of GPUTime LDS is stalled by bank conflicts. Value range: 0% (optimal) to 100% (bad)."></metric>
-</gfx9_expr>
+</gfx9>

-<gfx906_expr base="gfx9_expr">
+<gfx906 base="gfx9">
  # EA1
-  <metric name="TCC_EA1_RDREQ_32B_sum" expr=sum(TCC_EA1_RDREQ_32B,16) descr="Number of 32-byte TCC/EA read requests. Sum over TCC EA1s."></metric>
-  <metric name="TCC_EA1_RDREQ_sum" expr=sum(TCC_EA1_RDREQ,16) descr="Number of TCC/EA read requests (either 32-byte or 64-byte). Sum over TCC EA1s."></metric>
-  <metric name="TCC_EA1_WRREQ_sum" expr=sum(TCC_EA1_WRREQ,16) descr="Number of transactions (either 32-byte or 64-byte) going over the TC_EA_wrreq interface. Sum over TCC EA1s."></metric>
-  <metric name="TCC_EA1_WRREQ_64B_sum" expr=sum(TCC_EA1_WRREQ_64B,16) descr="Number of 64-byte transactions going (64-byte write or CMPSWAP) over the TC_EA_wrreq interface. Sum over TCC EA1s."></metric>
-  <metric name="TCC_WRREQ1_STALL_max" expr=max(TCC_EA1_WRREQ_STALL,16) descr="Number of cycles a write request was stalled. Max over TCC instances."></metric>
+  <metric name="TCC_EA1_RDREQ_32B_sum" expr=reduce(TCC_EA1_RDREQ_32B,sum) descr="Number of 32-byte TCC/EA read requests. Sum over TCC EA1s."></metric>
+  <metric name="TCC_EA1_RDREQ_sum" expr=reduce(TCC_EA1_RDREQ,sum) descr="Number of TCC/EA read requests (either 32-byte or 64-byte). Sum over TCC EA1s."></metric>
+  <metric name="TCC_EA1_WRREQ_sum" expr=reduce(TCC_EA1_WRREQ,sum) descr="Number of transactions (either 32-byte or 64-byte) going over the TC_EA_wrreq interface. Sum over TCC EA1s."></metric>
+  <metric name="TCC_EA1_WRREQ_64B_sum" expr=reduce(TCC_EA1_WRREQ_64B,sum) descr="Number of 64-byte transactions going (64-byte write or CMPSWAP) over the TC_EA_wrreq interface. Sum over TCC EA1s."></metric>
+  <metric name="TCC_WRREQ1_STALL_max" expr=reduce(TCC_EA1_WRREQ_STALL,max) descr="Number of cycles a write request was stalled. Max over TCC instances."></metric>

  <metric name="RDATA1_SIZE" expr=(TCC_EA1_RDREQ_32B_sum*32+(TCC_EA1_RDREQ_sum-TCC_EA1_RDREQ_32B_sum)*64) descr="The total kilobytes fetched from the video memory. This is measured on EA1s."></metric>
  <metric name="WDATA1_SIZE" expr=((TCC_EA1_WRREQ_sum-TCC_EA1_WRREQ_64B_sum)*32+TCC_EA1_WRREQ_64B_sum*64) descr="The total kilobytes written to the video memory. This is measured on EA1s."></metric>
@@ -88,134 +86,134 @@
  <metric name="FETCH_SIZE" expr=(TCC_EA_RDREQ_32B_sum*32+(TCC_EA_RDREQ_sum-TCC_EA_RDREQ_32B_sum)*64+RDATA1_SIZE)/1024 descr="The total kilobytes fetched from the video memory. This is measured with all extra fetches and any cache or memory effects taken into account."></metric>
  <metric name="WRITE_SIZE" expr=((TCC_EA_WRREQ_sum-TCC_EA_WRREQ_64B_sum)*32+TCC_EA_WRREQ_64B_sum*64+WDATA1_SIZE)/1024 descr="The total kilobytes written to the video memory. This is measured with all extra fetches and any cache or memory effects taken into account."></metric>
  <metric name="WRITE_REQ_32B" expr=(TCC_EA_WRREQ_sum-TCC_EA_WRREQ_64B_sum)+(TCC_EA1_WRREQ_sum-TCC_EA1_WRREQ_64B_sum)+(TCC_EA_WRREQ_64B_sum+TCC_EA1_WRREQ_64B_sum)*2 descr="The total number of 32-byte effective memory writes."></metric>
-</gfx906_expr>
+</gfx906>

-<gfx908_expr base="gfx9_expr">
-  <metric name="TCC_HIT_sum" expr=sum(TCC_HIT,32) descr="Number of cache hits. Sum over TCC instances."></metric>
-  <metric name="TCC_MISS_sum" expr=sum(TCC_MISS,32) descr="Number of cache misses. Sum over TCC instances."></metric>
-  <metric name="TCC_EA_RDREQ_32B_sum" expr=sum(TCC_EA_RDREQ_32B,32) descr="Number of 32-byte TCC/EA read requests. Sum over TCC instances."></metric>
-  <metric name="TCC_EA_RDREQ_sum" expr=sum(TCC_EA_RDREQ,32) descr="Number of TCC/EA read requests (either 32-byte or 64-byte). Sum over TCC instances."></metric>
-  <metric name="TCC_EA_WRREQ_sum" expr=sum(TCC_EA_WRREQ,32) descr="Number of transactions (either 32-byte or 64-byte) going over the TC_EA_wrreq interface. Sum over TCC instances."></metric>
-  <metric name="TCC_EA_WRREQ_64B_sum" expr=sum(TCC_EA_WRREQ_64B,32) descr="Number of 64-byte transactions going (64-byte write or CMPSWAP) over the TC_EA_wrreq interface. Sum over TCC instances."></metric>
-  <metric name="TCC_WRREQ_STALL_max" expr=max(TCC_EA_WRREQ_STALL,32) descr="Number of cycles a write request was stalled. Max over TCC instances."></metric>
+<gfx908 base="gfx9">
+  <metric name="TCC_HIT_sum" expr=reduce(TCC_HIT,sum) descr="Number of cache hits. Sum over TCC instances."></metric>
+  <metric name="TCC_MISS_sum" expr=reduce(TCC_MISS,sum) descr="Number of cache misses. Sum over TCC instances."></metric>
+  <metric name="TCC_EA_RDREQ_32B_sum" expr=reduce(TCC_EA_RDREQ_32B,sum) descr="Number of 32-byte TCC/EA read requests. Sum over TCC instances."></metric>
+  <metric name="TCC_EA_RDREQ_sum" expr=reduce(TCC_EA_RDREQ,sum) descr="Number of TCC/EA read requests (either 32-byte or 64-byte). Sum over TCC instances."></metric>
+  <metric name="TCC_EA_WRREQ_sum" expr=reduce(TCC_EA_WRREQ,sum) descr="Number of transactions (either 32-byte or 64-byte) going over the TC_EA_wrreq interface. Sum over TCC instances."></metric>
+  <metric name="TCC_EA_WRREQ_64B_sum" expr=reduce(TCC_EA_WRREQ_64B,sum) descr="Number of 64-byte transactions going (64-byte write or CMPSWAP) over the TC_EA_wrreq interface. Sum over TCC instances."></metric>
+  <metric name="TCC_WRREQ_STALL_max" expr=reduce(TCC_EA_WRREQ_STALL,max) descr="Number of cycles a write request was stalled. Max over TCC instances."></metric>

  <metric name="CU_UTILIZATION"                                         expr=GRBM_GUI_ACTIVE/GRBM_COUNT descr="The total number of active cycles divided by total number of elapsed cycles"></metric>
-</gfx908_expr>
+</gfx908>

-<gfx90a_expr base="gfx9_expr">
+<gfx90a base="gfx9">
  <metric name="MeanOccupancyPerCU" expr=SQ_LEVEL_WAVES*0+SQ_ACCUM_PREV_HIRES/GRBM_GUI_ACTIVE/CU_NUM descr="Mean occupancy per compute unit."></metric>
  <metric name="MeanOccupancyPerActiveCU" expr=SQ_LEVEL_WAVES*0+SQ_ACCUM_PREV_HIRES*4/SQ_BUSY_CYCLES/CU_NUM descr="Mean occupancy per active compute unit."></metric>
-  <metric name="TA_BUSY_avr"                                            expr=avr(TA_TA_BUSY,16) descr="TA block is busy. Average over TA instances."></metric>
-  <metric name="TA_BUSY_max"                                            expr=max(TA_TA_BUSY,16) descr="TA block is busy. Max over TA instances."></metric>
-  <metric name="TA_BUSY_min"                                            expr=min(TA_TA_BUSY,16) descr="TA block is busy. Min over TA instances."></metric>
-  <metric name="TA_TA_BUSY_sum"                                         expr=sum(TA_TA_BUSY,16) descr="TA block is busy. Perf_Windowing not supported for this counter. Sum over TA instances."></metric>
-  <metric name="TA_TOTAL_WAVEFRONTS_sum"                                expr=sum(TA_TOTAL_WAVEFRONTS,16) descr="Total number of wavefronts processed by TA. Sum over TA instances."></metric>
-  <metric name="TA_ADDR_STALLED_BY_TC_CYCLES_sum"                       expr=sum(TA_ADDR_STALLED_BY_TC_CYCLES,16) descr="Number of cycles addr path stalled by TC. Perf_Windowing not supported for this counter. Sum over TA instances."></metric>
-  <metric name="TA_ADDR_STALLED_BY_TD_CYCLES_sum"                       expr=sum(TA_ADDR_STALLED_BY_TD_CYCLES,16) descr="Number of cycles addr path stalled by TD. Perf_Windowing not supported for this counter. Sum over TA instances."></metric>
-  <metric name="TA_DATA_STALLED_BY_TC_CYCLES_sum"                       expr=sum(TA_DATA_STALLED_BY_TC_CYCLES,16) descr="Number of cycles data path stalled by TC. Perf_Windowing not supported for this counter. Sum over TA instances."></metric>
-  <metric name="TA_FLAT_WAVEFRONTS_sum"                                 expr=sum(TA_FLAT_WAVEFRONTS,16) descr="Number of flat opcode wavfronts processed by the TA. Sum over TA instances."></metric>
-  <metric name="TA_FLAT_READ_WAVEFRONTS_sum"                            expr=sum(TA_FLAT_READ_WAVEFRONTS,16) descr="Number of flat opcode reads processed by the TA. Sum over TA instances."></metric>
-  <metric name="TA_FLAT_WRITE_WAVEFRONTS_sum"                           expr=sum(TA_FLAT_WRITE_WAVEFRONTS,16) descr="Number of flat opcode writes processed by the TA. Sum over TA instances."></metric>
-  <metric name="TA_FLAT_ATOMIC_WAVEFRONTS_sum"                          expr=sum(TA_FLAT_ATOMIC_WAVEFRONTS,16) descr="Number of flat opcode atomics processed by the TA. Sum over TA instances."></metric>
-  <metric name="TA_BUFFER_WAVEFRONTS_sum"                               expr=sum(TA_BUFFER_WAVEFRONTS,16) descr="Number of buffer wavefronts processed by TA. Sum over TA instances."></metric>
-  <metric name="TA_BUFFER_READ_WAVEFRONTS_sum"                          expr=sum(TA_BUFFER_READ_WAVEFRONTS,16) descr="Number of buffer read wavefronts processed by TA. Sum over TA instances."></metric>
-  <metric name="TA_BUFFER_WRITE_WAVEFRONTS_sum"                         expr=sum(TA_BUFFER_WRITE_WAVEFRONTS,16) descr="Number of buffer write wavefronts processed by TA. Sum over TA instances."></metric>
-  <metric name="TA_BUFFER_ATOMIC_WAVEFRONTS_sum"                        expr=sum(TA_BUFFER_ATOMIC_WAVEFRONTS,16) descr="Number of buffer atomic wavefronts processed by TA. Sum over TA instances."></metric>
-  <metric name="TA_BUFFER_TOTAL_CYCLES_sum"                             expr=sum(TA_BUFFER_TOTAL_CYCLES,16) descr="Number of buffer cycles issued to TC. Sum over TA instances."></metric>
-  <metric name="TA_BUFFER_COALESCED_READ_CYCLES_sum"                    expr=sum(TA_BUFFER_COALESCED_READ_CYCLES,16) descr="Number of buffer coalesced read cycles issued to TC. Sum over TA instances."></metric>
-  <metric name="TA_BUFFER_COALESCED_WRITE_CYCLES_sum"                   expr=sum(TA_BUFFER_COALESCED_WRITE_CYCLES,16) descr="Number of buffer coalesced write cycles issued to TC. Sum over TA instances."></metric>
-  <metric name="TD_TD_BUSY_sum"                                         expr=sum(TD_TD_BUSY,16) descr="TD is processing or waiting for data. Perf_Windowing not supported for this counter. Sum over TD instances."></metric>
-  <metric name="TD_TC_STALL_sum"                                        expr=sum(TD_TC_STALL,16) descr="TD is stalled waiting for TC data. Sum over TD instances."></metric>
-  <metric name="TD_LOAD_WAVEFRONT_sum"                                  expr=sum(TD_LOAD_WAVEFRONT,16) descr="Count the wavefronts with opcode = load, include atomics and store. Sum over TD instances."></metric>
-  <metric name="TD_ATOMIC_WAVEFRONT_sum"                                expr=sum(TD_ATOMIC_WAVEFRONT,16) descr="Count the wavefronts with opcode = atomic. Sum over TD instances."></metric>
-  <metric name="TD_STORE_WAVEFRONT_sum"                                 expr=sum(TD_STORE_WAVEFRONT,16) descr="Count the wavefronts with opcode = store. Sum over TD instances."></metric>
-  <metric name="TD_COALESCABLE_WAVEFRONT_sum"                           expr=sum(TD_COALESCABLE_WAVEFRONT,16) descr="Count wavefronts that TA finds coalescable. Sum over TD instances."></metric>
-  <metric name="TD_SPI_STALL_sum"                                       expr=sum(TD_SPI_STALL,16) descr="TD is stalled SPI vinit, sum of TCP instances"></metric>
-  <metric name="TCP_GATE_EN1_sum"                                       expr=sum(TCP_GATE_EN1,16) descr="TCP interface clocks are turned on. Not Windowed. Sum over TCP instances."></metric>
-  <metric name="TCP_GATE_EN2_sum"                                       expr=sum(TCP_GATE_EN2,16) descr="TCP core clocks are turned on. Not Windowed. Sum over TCP instances."></metric>
-  <metric name="TCP_TD_TCP_STALL_CYCLES_sum"                            expr=sum(TCP_TD_TCP_STALL_CYCLES,16) descr="TD stalls TCP. Sum over TCP instances."></metric>
-  <metric name="TCP_TCR_TCP_STALL_CYCLES_sum"                           expr=sum(TCP_TCR_TCP_STALL_CYCLES,16) descr="TCR stalls TCP_TCR_req interface. Sum over TCP instances."></metric>
-  <metric name="TCP_READ_TAGCONFLICT_STALL_CYCLES_sum"                  expr=sum(TCP_READ_TAGCONFLICT_STALL_CYCLES,16) descr="Tagram conflict stall on a read. Sum over TCP instances."></metric>
-  <metric name="TCP_WRITE_TAGCONFLICT_STALL_CYCLES_sum"                 expr=sum(TCP_WRITE_TAGCONFLICT_STALL_CYCLES,16) descr="Tagram conflict stall on a write. Sum over TCP instances."></metric>
-  <metric name="TCP_ATOMIC_TAGCONFLICT_STALL_CYCLES_sum"                expr=sum(TCP_ATOMIC_TAGCONFLICT_STALL_CYCLES,16) descr="Tagram conflict stall on an atomic. Sum over TCP instances."></metric>
-  <metric name="TCP_VOLATILE_sum"                                       expr=sum(TCP_VOLATILE,16) descr="Total number of L1 volatile pixels/buffers from TA. Sum over TCP instances."></metric>
-  <metric name="TCP_TOTAL_ACCESSES_sum"                                 expr=sum(TCP_TOTAL_ACCESSES,16) descr="Total number of pixels/buffers from TA. Equals TCP_PERF_SEL_TOTAL_READ+TCP_PERF_SEL_TOTAL_NONREAD. Sum over TCP instances."></metric>
-  <metric name="TCP_TOTAL_READ_sum"                                     expr=sum(TCP_TOTAL_READ,16) descr="Total number of read pixels/buffers from TA. Equals TCP_PERF_SEL_TOTAL_HIT_LRU_READ + TCP_PERF_SEL_TOTAL_MISS_LRU_READ + TCP_PERF_SEL_TOTAL_MISS_EVICT_READ. Sum over TCP instances."></metric>
-  <metric name="TCP_TOTAL_WRITE_sum"                                    expr=sum(TCP_TOTAL_WRITE,16) descr="Total number of local write pixels/buffers from TA. Equals TCP_PERF_SEL_TOTAL_MISS_LRU_WRITE+ TCP_PERF_SEL_TOTAL_MISS_EVICT_WRITE. Sum over TCP instances."></metric>
-  <metric name="TCP_TOTAL_ATOMIC_WITH_RET_sum"                          expr=sum(TCP_TOTAL_ATOMIC_WITH_RET,16) descr="Total number of atomic with return pixels/buffers from TA. Sum over TCP instances."></metric>
-  <metric name="TCP_TOTAL_ATOMIC_WITHOUT_RET_sum"                       expr=sum(TCP_TOTAL_ATOMIC_WITHOUT_RET,16) descr="Total number of atomic without return pixels/buffers from TA Sum over TCP instances."></metric>
-  <metric name="TCP_TOTAL_WRITEBACK_INVALIDATES_sum"                    expr=sum(TCP_TOTAL_WRITEBACK_INVALIDATES,16) descr="Total number of cache invalidates. Equals TCP_PERF_SEL_TOTAL_WBINVL1+ TCP_PERF_SEL_TOTAL_WBINVL1_VOL+ TCP_PERF_SEL_CP_TCP_INVALIDATE+ TCP_PERF_SEL_SQ_TCP_INVALIDATE_VOL. Not Windowed. Sum over TCP instances."></metric>
-  <metric name="TCP_UTCL1_REQUEST_sum"                                  expr=sum(TCP_UTCL1_REQUEST,16) descr="Total CLIENT_UTCL1 NORMAL requests Sum over TCP instances."></metric>
-  <metric name="TCP_UTCL1_TRANSLATION_MISS_sum"                         expr=sum(TCP_UTCL1_TRANSLATION_MISS,16) descr="Total utcl1 translation misses Sum over TCP instances."></metric>
-  <metric name="TCP_UTCL1_TRANSLATION_HIT_sum"                          expr=sum(TCP_UTCL1_TRANSLATION_HIT,16) descr="Total utcl1 translation hits Sum over TCP instances."></metric>
-  <metric name="TCP_UTCL1_PERMISSION_MISS_sum"                          expr=sum(TCP_UTCL1_PERMISSION_MISS,16) descr="Total utcl1 permission misses Sum over TCP instances."></metric>
-  <metric name="TCP_TOTAL_CACHE_ACCESSES_sum"                           expr=sum(TCP_TOTAL_CACHE_ACCESSES,16) descr="Count of total cache line (tag) accesses (includes hits and misses). Sum over TCP instances."></metric>
-  <metric name="TCP_TCP_LATENCY_sum"                                    expr=sum(TCP_TCP_LATENCY,16) descr="Total TCP wave latency (from first clock of wave entering to first clock of wave leaving), divide by TA_TCP_STATE_READ to avg wave latency Sum over TCP instances."></metric>
-  <metric name="TCP_TA_TCP_STATE_READ_sum"                              expr=sum(TCP_TA_TCP_STATE_READ,16) descr="Number of state reads Sum over TCP instances."></metric>
-  <metric name="TCP_TCC_READ_REQ_LATENCY_sum"                           expr=sum(TCP_TCC_READ_REQ_LATENCY,16) descr="Total TCP->TCC request latency for reads and atomics with return. Not Windowed. Sum over TCP instances."></metric>
-  <metric name="TCP_TCC_WRITE_REQ_LATENCY_sum"                          expr=sum(TCP_TCC_WRITE_REQ_LATENCY,16) descr="Total TCP->TCC request latency for writes and atomics without return. Not Windowed. Sum over TCP instances."></metric>
-  <metric name="TCP_TCC_READ_REQ_sum"                                   expr=sum(TCP_TCC_READ_REQ,16) descr="Total read requests from TCP to all TCCs Sum over TCP instances."></metric>
-  <metric name="TCP_TCC_WRITE_REQ_sum"                                  expr=sum(TCP_TCC_WRITE_REQ,16) descr="Total write requests from TCP to all TCCs Sum over TCP instances."></metric>
-  <metric name="TCP_TCC_ATOMIC_WITH_RET_REQ_sum"                        expr=sum(TCP_TCC_ATOMIC_WITH_RET_REQ,16) descr="Total atomic with return requests from TCP to all TCCs Sum over TCP instances."></metric>
-  <metric name="TCP_TCC_ATOMIC_WITHOUT_RET_REQ_sum"                     expr=sum(TCP_TCC_ATOMIC_WITHOUT_RET_REQ,16) descr="Total atomic without return requests from TCP to all TCCs Sum over TCP instances."></metric>
-  <metric name="TCP_TCC_NC_READ_REQ_sum"                                expr=sum(TCP_TCC_NC_READ_REQ,16) descr="Total read requests with NC mtype from this TCP to all TCCs Sum over TCP instances."></metric>
-  <metric name="TCP_TCC_NC_WRITE_REQ_sum"                               expr=sum(TCP_TCC_NC_WRITE_REQ,16) descr="Total write requests with NC mtype from this TCP to all TCCs Sum over TCP instances."></metric>
-  <metric name="TCP_TCC_NC_ATOMIC_REQ_sum"                              expr=sum(TCP_TCC_NC_ATOMIC_REQ,16) descr="Total atomic requests with NC mtype from this TCP to all TCCs Sum over TCP instances."></metric>
-  <metric name="TCP_TCC_UC_READ_REQ_sum"                                expr=sum(TCP_TCC_UC_READ_REQ,16) descr="Total read requests with UC mtype from this TCP to all TCCs Sum over TCP instances."></metric>
-  <metric name="TCP_TCC_UC_WRITE_REQ_sum"                               expr=sum(TCP_TCC_UC_WRITE_REQ,16) descr="Total write requests with UC mtype from this TCP to all TCCs Sum over TCP instances."></metric>
-  <metric name="TCP_TCC_UC_ATOMIC_REQ_sum"                              expr=sum(TCP_TCC_UC_ATOMIC_REQ,16) descr="Total atomic requests with UC mtype from this TCP to all TCCs Sum over TCP instances."></metric>
-  <metric name="TCP_TCC_CC_READ_REQ_sum"                                expr=sum(TCP_TCC_CC_READ_REQ,16) descr="Total write requests with CC mtype from this TCP to all TCCs Sum over TCP instances."></metric>
-  <metric name="TCP_TCC_CC_WRITE_REQ_sum"                               expr=sum(TCP_TCC_CC_WRITE_REQ,16) descr="Total write requests with CC mtype from this TCP to all TCCs Sum over TCP instances."></metric>
-  <metric name="TCP_TCC_CC_ATOMIC_REQ_sum"                              expr=sum(TCP_TCC_CC_ATOMIC_REQ,16) descr="Total atomic requests with CC mtype from this TCP to all TCCs Sum over TCP instances."></metric>
-  <metric name="TCP_TCC_RW_READ_REQ_sum"                                expr=sum(TCP_TCC_RW_READ_REQ,16) descr="Total write requests with RW mtype from this TCP to all TCCs. Sum over TCP instances."></metric>
-  <metric name="TCP_TCC_RW_WRITE_REQ_sum"                               expr=sum(TCP_TCC_RW_WRITE_REQ,16) descr="Total write requests with RW mtype from this TCP to all TCCs. Sum over TCP instances."></metric>
-  <metric name="TCP_TCC_RW_ATOMIC_REQ_sum"                              expr=sum(TCP_TCC_RW_ATOMIC_REQ,16) descr="Total atomic requests with RW mtype from this TCP to all TCCs. Sum over TCP instances."></metric>
-  <metric name="TCP_PENDING_STALL_CYCLES_sum"                           expr=sum(TCP_PENDING_STALL_CYCLES,16) descr="Stall due to data pending from L2. Sum over TCP instances."></metric>
-  <metric name="TCA_CYCLE_sum"                                          expr=sum(TCA_CYCLE,16)  descr="Number of cycles. Sum over all TCA instances "></metric>
-  <metric name="TCA_BUSY_sum"                                           expr=sum(TCA_BUSY,16)   descr="Number of cycles we have a request pending. Sum over all TCA instances."></metric>
-  <metric name="TCC_BUSY_avr"                                           expr=avr(TCC_BUSY,32) descr="TCC_BUSY avr over all memory channels."></metric>
-  <metric name="TCC_WRREQ_STALL_max"                                    expr=max(TCC_EA_WRREQ_STALL,32) descr="Number of cycles a write request was stalled. Max over TCC instances."></metric>
-  <metric name="TCC_CYCLE_sum"                                          expr=sum(TCC_CYCLE,32) descr="Number of cycles. Not windowable. Sum over TCC instances."></metric>
-  <metric name="TCC_BUSY_sum"                                           expr=sum(TCC_BUSY,32) descr="Number of cycles we have a request pending. Not windowable. Sum over TCC instances."></metric>
-  <metric name="TCC_REQ_sum"                                            expr=sum(TCC_REQ,32) descr="Number of requests of all types. This is measured at the tag block. This may be more than the number of requests arriving at the TCC, but it is a good indication of the total amount of work that needs to be performed. Sum over TCC instances."></metric>
-  <metric name="TCC_STREAMING_REQ_sum"                                  expr=sum(TCC_STREAMING_REQ,32) descr="Number of streaming requests. This is measured at the tag block. Sum over TCC instances."></metric>
-  <metric name="TCC_NC_REQ_sum"                                         expr=sum(TCC_NC_REQ,32) descr="The number of noncoherently cached requests. This is measured at the tag block. Sum over TCC instances."></metric>
-  <metric name="TCC_UC_REQ_sum"                                         expr=sum(TCC_UC_REQ,32) descr="The number of uncached requests. This is measured at the tag block. Sum over TCC instances."></metric>
-  <metric name="TCC_CC_REQ_sum"                                         expr=sum(TCC_CC_REQ,32) descr="The number of coherently cached requests. This is measured at the tag block. Sum over TCC instances."></metric>
-  <metric name="TCC_RW_REQ_sum"                                         expr=sum(TCC_RW_REQ,32) descr="The number of RW requests. This is measured at the tag block. Sum over TCC instances."></metric>
-  <metric name="TCC_PROBE_sum"                                          expr=sum(TCC_PROBE,32) descr="Number of probe requests. Not windowable. Sum over TCC instances."></metric>
-  <metric name="TCC_PROBE_ALL_sum"                                      expr=sum(TCC_PROBE_ALL,32) descr="Number of external probe requests with with EA_TCC_preq_all== 1. Not windowable. Sum over TCC instances."></metric>
-  <metric name="TCC_READ_sum"                                           expr=sum(TCC_READ,32) descr="Number of read requests. Compressed reads are included in this, but metadata reads are not included. Sum over TCC instances."></metric>
-  <metric name="TCC_WRITE_sum"                                          expr=sum(TCC_WRITE,32) descr="Number of write requests. Sum over TCC instances."></metric>
-  <metric name="TCC_ATOMIC_sum"                                         expr=sum(TCC_ATOMIC,32) descr="Number of atomic requests of all types. Sum over TCC instances."></metric>
-  <metric name="TCC_HIT_sum"                                            expr=sum(TCC_HIT,32) descr="Number of cache hits. Sum over TCC instances."></metric>
-  <metric name="TCC_MISS_sum"                                           expr=sum(TCC_MISS,32) descr="Number of cache misses. UC reads count as misses. Sum over TCC instances."></metric>
-  <metric name="TCC_WRITEBACK_sum"                                      expr=sum(TCC_WRITEBACK,32) descr="Number of lines written back to main memory. This includes writebacks of dirty lines and uncached write/atomic requests. Sum over TCC instances."></metric>
-  <metric name="TCC_EA_WRREQ_sum"                                       expr=sum(TCC_EA_WRREQ,32) descr="Number of transactions (either 32-byte or 64-byte) going over the TC_EA_wrreq interface. Atomics may travel over the same interface and are generally classified as write requests. This does not include probe commands. Sum over TCC instances."></metric>
-  <metric name="TCC_EA_WRREQ_64B_sum"                                   expr=sum(TCC_EA_WRREQ_64B,32) descr="Number of 64-byte transactions going (64-byte write or CMPSWAP) over the TC_EA_wrreq interface. Sum over TCC instances."></metric>
-  <metric name="TCC_EA_WR_UNCACHED_32B_sum"                             expr=sum(TCC_EA_WR_UNCACHED_32B,32) descr="Number of 32-byte write/atomic going over the TC_EA_wrreq interface due to uncached traffic. Note that CC mtypes can produce uncached requests, and those are included in this. A 64-byte request will be counted as 2. Sum over TCC instances."></metric>
-  <metric name="TCC_EA_WRREQ_STALL_sum"                                 expr=sum(TCC_EA_WRREQ_STALL,32) descr="Number of cycles a write request was stalled. Sum over TCC instances."></metric>
-  <metric name="TCC_EA_WRREQ_IO_CREDIT_STALL_sum"                       expr=sum(TCC_EA_WRREQ_IO_CREDIT_STALL,32) descr="Number of cycles a EA write request was stalled because the interface was out of IO credits. Sum over TCC instances."></metric>
-  <metric name="TCC_EA_WRREQ_GMI_CREDIT_STALL_sum"                      expr=sum(TCC_EA_WRREQ_GMI_CREDIT_STALL,32) descr="Number of cycles a EA write request was stalled because the interface was out of GMI credits. Sum over TCC instances."></metric>
-  <metric name="TCC_EA_WRREQ_DRAM_CREDIT_STALL_sum"                     expr=sum(TCC_EA_WRREQ_DRAM_CREDIT_STALL,32) descr="Number of cycles a EA write request was stalled because the interface was out of DRAM credits. Sum over TCC instances."></metric>
-  <metric name="TCC_TOO_MANY_EA_WRREQS_STALL_sum"                       expr=sum(TCC_TOO_MANY_EA_WRREQS_STALL,32) descr="Number of cycles the TCC could not send a EA write request because it already reached its maximum number of pending EA write requests. Sum over TCC instances."></metric>
-  <metric name="TCC_EA_WRREQ_LEVEL_sum"                                 expr=sum(TCC_EA_WRREQ_LEVEL,32) descr="The sum of the number of EA write requests in flight. This is primarily meant for measure average EA write latency. Average write latency = TCC_PERF_SEL_EA_WRREQ_LEVEL/TCC_PERF_SEL_EA_WRREQ. Sum over TCC instances."></metric>
-  <metric name="TCC_EA_RDREQ_LEVEL_sum"                                 expr=sum(TCC_EA_RDREQ_LEVEL,32) descr="The sum of the number of TCC/EA read requests in flight. This is primarily meant for measure average EA read latency. Average read latency = TCC_PERF_SEL_EA_RDREQ_LEVEL/TCC_PERF_SEL_EA_RDREQ. Sum over TCC instances."></metric>
-  <metric name="TCC_EA_ATOMIC_sum"                                      expr=sum(TCC_EA_ATOMIC,32) descr="Number of transactions going over the TC_EA_wrreq interface that are actually atomic requests. Sum over TCC instances."></metric>
-  <metric name="TCC_EA_ATOMIC_LEVEL_sum"                                expr=sum(TCC_EA_ATOMIC_LEVEL,32) descr="The sum of the number of EA atomics in flight. This is primarily meant for measure average EA atomic latency. Average atomic latency = TCC_PERF_SEL_EA_WRREQ_ATOMIC_LEVEL/TCC_PERF_SEL_EA_WRREQ_ATOMIC. Sum over TCC instances."></metric>
-  <metric name="TCC_EA_RDREQ_sum"                                       expr=sum(TCC_EA_RDREQ,32) descr="Number of TCC/EA read requests (either 32-byte or 64-byte) Sum over TCC instances."></metric>
-  <metric name="TCC_EA_RDREQ_32B_sum"                                   expr=sum(TCC_EA_RDREQ_32B,32) descr="Number of 32-byte TCC/EA read requests Sum over TCC instances."></metric>
-  <metric name="TCC_EA_RD_UNCACHED_32B_sum"                             expr=sum(TCC_EA_RD_UNCACHED_32B,32) descr="Number of 32-byte TCC/EA read due to uncached traffic. A 64-byte request will be counted as 2 Sum over TCC instances."></metric>
-  <metric name="TCC_EA_RDREQ_IO_CREDIT_STALL_sum"                       expr=sum(TCC_EA_RDREQ_IO_CREDIT_STALL,32) descr="Number of cycles there was a stall because the read request interface was out of IO credits. Stalls occur regardless of whether a read needed to be performed or not. Sum over TCC instances."></metric>
-  <metric name="TCC_EA_RDREQ_GMI_CREDIT_STALL_sum"                      expr=sum(TCC_EA_RDREQ_GMI_CREDIT_STALL,32) descr="Number of cycles there was a stall because the read request interface was out of GMI credits. Stalls occur regardless of whether a read needed to be performed or not. Sum over TCC instances."></metric>
-  <metric name="TCC_EA_RDREQ_DRAM_CREDIT_STALL_sum"                     expr=sum(TCC_EA_RDREQ_DRAM_CREDIT_STALL,32) descr="Number of cycles there was a stall because the read request interface was out of DRAM credits. Stalls occur regardless of whether a read needed to be performed or not. Sum over TCC instances."></metric>
-  <metric name="TCC_TAG_STALL_sum"                                      expr=sum(TCC_TAG_STALL,32) descr="."></metric>
-  <metric name="TCC_NORMAL_WRITEBACK_sum"                               expr=sum(TCC_NORMAL_WRITEBACK,32) descr="Number of writebacks due to requests that are not writeback requests. Sum over TCC instances."></metric>
-  <metric name="TCC_ALL_TC_OP_WB_WRITEBACK_sum"                         expr=sum(TCC_ALL_TC_OP_WB_WRITEBACK,32) descr="Number of writebacks due to all TC_OP writeback requests. Sum over TCC instances."></metric>
-  <metric name="TCC_NORMAL_EVICT_sum"                                   expr=sum(TCC_NORMAL_EVICT,32) descr="Number of evictions due to requests that are not invalidate or probe requests. Sum over TCC instances."></metric>
-  <metric name="TCC_ALL_TC_OP_INV_EVICT_sum"                            expr=sum(TCC_ALL_TC_OP_INV_EVICT,32) descr="Number of evictions due to all TC_OP invalidate requests. Sum over TCC instances."></metric>
-  <metric name="TCC_EA_RDREQ_DRAM_sum"                                  expr=sum(TCC_EA_RDREQ_DRAM,32) descr="Number of TCC/EA read requests (either 32-byte or 64-byte) destined for DRAM (MC). Sum over TCC instances."></metric>
-  <metric name="TCC_EA_WRREQ_DRAM_sum"                                  expr=sum(TCC_EA_WRREQ_DRAM,32) descr="Number of TCC/EA write requests (either 32-byte of 64-byte) destined for DRAM (MC). Sum over TCC instances."></metric>
+  <metric name="TA_BUSY_avr"                                            expr=reduce(TA_TA_BUSY,average) descr="TA block is busy. Average over TA instances."></metric>
+  <metric name="TA_BUSY_max"                                            expr=reduce(TA_TA_BUSY,max) descr="TA block is busy. Max over TA instances."></metric>
+  <metric name="TA_BUSY_min"                                            expr=reduce(TA_TA_BUSY,min) descr="TA block is busy. Min over TA instances."></metric>
+  <metric name="TA_TA_BUSY_sum"                                         expr=reduce(TA_TA_BUSY,sum) descr="TA block is busy. Perf_Windowing not supported for this counter. Sum over TA instances."></metric>
+  <metric name="TA_TOTAL_WAVEFRONTS_sum"                                expr=reduce(TA_TOTAL_WAVEFRONTS,sum) descr="Total number of wavefronts processed by TA. Sum over TA instances."></metric>
+  <metric name="TA_ADDR_STALLED_BY_TC_CYCLES_sum"                       expr=reduce(TA_ADDR_STALLED_BY_TC_CYCLES,sum) descr="Number of cycles addr path stalled by TC. Perf_Windowing not supported for this counter. Sum over TA instances."></metric>
+  <metric name="TA_ADDR_STALLED_BY_TD_CYCLES_sum"                       expr=reduce(TA_ADDR_STALLED_BY_TD_CYCLES,sum) descr="Number of cycles addr path stalled by TD. Perf_Windowing not supported for this counter. Sum over TA instances."></metric>
+  <metric name="TA_DATA_STALLED_BY_TC_CYCLES_sum"                       expr=reduce(TA_DATA_STALLED_BY_TC_CYCLES,sum) descr="Number of cycles data path stalled by TC. Perf_Windowing not supported for this counter. Sum over TA instances."></metric>
+  <metric name="TA_FLAT_WAVEFRONTS_sum"                                 expr=reduce(TA_FLAT_WAVEFRONTS,sum) descr="Number of flat opcode wavfronts processed by the TA. Sum over TA instances."></metric>
+  <metric name="TA_FLAT_READ_WAVEFRONTS_sum"                            expr=reduce(TA_FLAT_READ_WAVEFRONTS,sum) descr="Number of flat opcode reads processed by the TA. Sum over TA instances."></metric>
+  <metric name="TA_FLAT_WRITE_WAVEFRONTS_sum"                           expr=reduce(TA_FLAT_WRITE_WAVEFRONTS,sum) descr="Number of flat opcode writes processed by the TA. Sum over TA instances."></metric>
+  <metric name="TA_FLAT_ATOMIC_WAVEFRONTS_sum"                          expr=reduce(TA_FLAT_ATOMIC_WAVEFRONTS,sum) descr="Number of flat opcode atomics processed by the TA. Sum over TA instances."></metric>
+  <metric name="TA_BUFFER_WAVEFRONTS_sum"                               expr=reduce(TA_BUFFER_WAVEFRONTS,sum) descr="Number of buffer wavefronts processed by TA. Sum over TA instances."></metric>
+  <metric name="TA_BUFFER_READ_WAVEFRONTS_sum"                          expr=reduce(TA_BUFFER_READ_WAVEFRONTS,sum) descr="Number of buffer read wavefronts processed by TA. Sum over TA instances."></metric>
+  <metric name="TA_BUFFER_WRITE_WAVEFRONTS_sum"                         expr=reduce(TA_BUFFER_WRITE_WAVEFRONTS,sum) descr="Number of buffer write wavefronts processed by TA. Sum over TA instances."></metric>
+  <metric name="TA_BUFFER_ATOMIC_WAVEFRONTS_sum"                        expr=reduce(TA_BUFFER_ATOMIC_WAVEFRONTS,sum) descr="Number of buffer atomic wavefronts processed by TA. Sum over TA instances."></metric>
+  <metric name="TA_BUFFER_TOTAL_CYCLES_sum"                             expr=reduce(TA_BUFFER_TOTAL_CYCLES,sum) descr="Number of buffer cycles issued to TC. Sum over TA instances."></metric>
+  <metric name="TA_BUFFER_COALESCED_READ_CYCLES_sum"                    expr=reduce(TA_BUFFER_COALESCED_READ_CYCLES,sum) descr="Number of buffer coalesced read cycles issued to TC. Sum over TA instances."></metric>
+  <metric name="TA_BUFFER_COALESCED_WRITE_CYCLES_sum"                   expr=reduce(TA_BUFFER_COALESCED_WRITE_CYCLES,sum) descr="Number of buffer coalesced write cycles issued to TC. Sum over TA instances."></metric>
+  <metric name="TD_TD_BUSY_sum"                                         expr=reduce(TD_TD_BUSY,sum) descr="TD is processing or waiting for data. Perf_Windowing not supported for this counter. Sum over TD instances."></metric>
+  <metric name="TD_TC_STALL_sum"                                        expr=reduce(TD_TC_STALL,sum) descr="TD is stalled waiting for TC data. Sum over TD instances."></metric>
+  <metric name="TD_LOAD_WAVEFRONT_sum"                                  expr=reduce(TD_LOAD_WAVEFRONT,sum) descr="Count the wavefronts with opcode = load, include atomics and store. Sum over TD instances."></metric>
+  <metric name="TD_ATOMIC_WAVEFRONT_sum"                                expr=reduce(TD_ATOMIC_WAVEFRONT,sum) descr="Count the wavefronts with opcode = atomic. Sum over TD instances."></metric>
+  <metric name="TD_STORE_WAVEFRONT_sum"                                 expr=reduce(TD_STORE_WAVEFRONT,sum) descr="Count the wavefronts with opcode = store. Sum over TD instances."></metric>
+  <metric name="TD_COALESCABLE_WAVEFRONT_sum"                           expr=reduce(TD_COALESCABLE_WAVEFRONT,sum) descr="Count wavefronts that TA finds coalescable. Sum over TD instances."></metric>
+  <metric name="TD_SPI_STALL_sum"                                       expr=reduce(TD_SPI_STALL,sum) descr="TD is stalled SPI vinit, sum of TCP instances"></metric>
+  <metric name="TCP_GATE_EN1_sum"                                       expr=reduce(TCP_GATE_EN1,sum) descr="TCP interface clocks are turned on. Not Windowed. Sum over TCP instances."></metric>
+  <metric name="TCP_GATE_EN2_sum"                                       expr=reduce(TCP_GATE_EN2,sum) descr="TCP core clocks are turned on. Not Windowed. Sum over TCP instances."></metric>
+  <metric name="TCP_TD_TCP_STALL_CYCLES_sum"                            expr=reduce(TCP_TD_TCP_STALL_CYCLES,sum) descr="TD stalls TCP. Sum over TCP instances."></metric>
+  <metric name="TCP_TCR_TCP_STALL_CYCLES_sum"                           expr=reduce(TCP_TCR_TCP_STALL_CYCLES,sum) descr="TCR stalls TCP_TCR_req interface. Sum over TCP instances."></metric>
+  <metric name="TCP_READ_TAGCONFLICT_STALL_CYCLES_sum"                  expr=reduce(TCP_READ_TAGCONFLICT_STALL_CYCLES,sum) descr="Tagram conflict stall on a read. Sum over TCP instances."></metric>
+  <metric name="TCP_WRITE_TAGCONFLICT_STALL_CYCLES_sum"                 expr=reduce(TCP_WRITE_TAGCONFLICT_STALL_CYCLES,sum) descr="Tagram conflict stall on a write. Sum over TCP instances."></metric>
+  <metric name="TCP_ATOMIC_TAGCONFLICT_STALL_CYCLES_sum"                expr=reduce(TCP_ATOMIC_TAGCONFLICT_STALL_CYCLES,sum) descr="Tagram conflict stall on an atomic. Sum over TCP instances."></metric>
+  <metric name="TCP_VOLATILE_sum"                                       expr=reduce(TCP_VOLATILE,sum) descr="Total number of L1 volatile pixels/buffers from TA. Sum over TCP instances."></metric>
+  <metric name="TCP_TOTAL_ACCESSES_sum"                                 expr=reduce(TCP_TOTAL_ACCESSES,sum) descr="Total number of pixels/buffers from TA. Equals TCP_PERF_SEL_TOTAL_READ+TCP_PERF_SEL_TOTAL_NONREAD. Sum over TCP instances."></metric>
+  <metric name="TCP_TOTAL_READ_sum"                                     expr=reduce(TCP_TOTAL_READ,sum) descr="Total number of read pixels/buffers from TA. Equals TCP_PERF_SEL_TOTAL_HIT_LRU_READ + TCP_PERF_SEL_TOTAL_MISS_LRU_READ + TCP_PERF_SEL_TOTAL_MISS_EVICT_READ. Sum over TCP instances."></metric>
+  <metric name="TCP_TOTAL_WRITE_sum"                                    expr=reduce(TCP_TOTAL_WRITE,sum) descr="Total number of local write pixels/buffers from TA. Equals TCP_PERF_SEL_TOTAL_MISS_LRU_WRITE+ TCP_PERF_SEL_TOTAL_MISS_EVICT_WRITE. Sum over TCP instances."></metric>
+  <metric name="TCP_TOTAL_ATOMIC_WITH_RET_sum"                          expr=reduce(TCP_TOTAL_ATOMIC_WITH_RET,sum) descr="Total number of atomic with return pixels/buffers from TA. Sum over TCP instances."></metric>
+  <metric name="TCP_TOTAL_ATOMIC_WITHOUT_RET_sum"                       expr=reduce(TCP_TOTAL_ATOMIC_WITHOUT_RET,sum) descr="Total number of atomic without return pixels/buffers from TA Sum over TCP instances."></metric>
+  <metric name="TCP_TOTAL_WRITEBACK_INVALIDATES_sum"                    expr=reduce(TCP_TOTAL_WRITEBACK_INVALIDATES,sum) descr="Total number of cache invalidates. Equals TCP_PERF_SEL_TOTAL_WBINVL1+ TCP_PERF_SEL_TOTAL_WBINVL1_VOL+ TCP_PERF_SEL_CP_TCP_INVALIDATE+ TCP_PERF_SEL_SQ_TCP_INVALIDATE_VOL. Not Windowed. Sum over TCP instances."></metric>
+  <metric name="TCP_UTCL1_REQUEST_sum"                                  expr=reduce(TCP_UTCL1_REQUEST,sum) descr="Total CLIENT_UTCL1 NORMAL requests Sum over TCP instances."></metric>
+  <metric name="TCP_UTCL1_TRANSLATION_MISS_sum"                         expr=reduce(TCP_UTCL1_TRANSLATION_MISS,sum) descr="Total utcl1 translation misses Sum over TCP instances."></metric>
+  <metric name="TCP_UTCL1_TRANSLATION_HIT_sum"                          expr=reduce(TCP_UTCL1_TRANSLATION_HIT,sum) descr="Total utcl1 translation hits Sum over TCP instances."></metric>
+  <metric name="TCP_UTCL1_PERMISSION_MISS_sum"                          expr=reduce(TCP_UTCL1_PERMISSION_MISS,sum) descr="Total utcl1 permission misses Sum over TCP instances."></metric>
+  <metric name="TCP_TOTAL_CACHE_ACCESSES_sum"                           expr=reduce(TCP_TOTAL_CACHE_ACCESSES,sum) descr="Count of total cache line (tag) accesses (includes hits and misses). Sum over TCP instances."></metric>
+  <metric name="TCP_TCP_LATENCY_sum"                                    expr=reduce(TCP_TCP_LATENCY,sum) descr="Total TCP wave latency (from first clock of wave entering to first clock of wave leaving), divide by TA_TCP_STATE_READ to avg wave latency Sum over TCP instances."></metric>
+  <metric name="TCP_TA_TCP_STATE_READ_sum"                              expr=reduce(TCP_TA_TCP_STATE_READ,sum) descr="Number of state reads Sum over TCP instances."></metric>
+  <metric name="TCP_TCC_READ_REQ_LATENCY_sum"                           expr=reduce(TCP_TCC_READ_REQ_LATENCY,sum) descr="Total TCP->TCC request latency for reads and atomics with return. Not Windowed. Sum over TCP instances."></metric>
+  <metric name="TCP_TCC_WRITE_REQ_LATENCY_sum"                          expr=reduce(TCP_TCC_WRITE_REQ_LATENCY,sum) descr="Total TCP->TCC request latency for writes and atomics without return. Not Windowed. Sum over TCP instances."></metric>
+  <metric name="TCP_TCC_READ_REQ_sum"                                   expr=reduce(TCP_TCC_READ_REQ,sum) descr="Total read requests from TCP to all TCCs Sum over TCP instances."></metric>
+  <metric name="TCP_TCC_WRITE_REQ_sum"                                  expr=reduce(TCP_TCC_WRITE_REQ,sum) descr="Total write requests from TCP to all TCCs Sum over TCP instances."></metric>
+  <metric name="TCP_TCC_ATOMIC_WITH_RET_REQ_sum"                        expr=reduce(TCP_TCC_ATOMIC_WITH_RET_REQ,sum) descr="Total atomic with return requests from TCP to all TCCs Sum over TCP instances."></metric>
+  <metric name="TCP_TCC_ATOMIC_WITHOUT_RET_REQ_sum"                     expr=reduce(TCP_TCC_ATOMIC_WITHOUT_RET_REQ,sum) descr="Total atomic without return requests from TCP to all TCCs Sum over TCP instances."></metric>
+  <metric name="TCP_TCC_NC_READ_REQ_sum"                                expr=reduce(TCP_TCC_NC_READ_REQ,sum) descr="Total read requests with NC mtype from this TCP to all TCCs Sum over TCP instances."></metric>
+  <metric name="TCP_TCC_NC_WRITE_REQ_sum"                               expr=reduce(TCP_TCC_NC_WRITE_REQ,sum) descr="Total write requests with NC mtype from this TCP to all TCCs Sum over TCP instances."></metric>
+  <metric name="TCP_TCC_NC_ATOMIC_REQ_sum"                              expr=reduce(TCP_TCC_NC_ATOMIC_REQ,sum) descr="Total atomic requests with NC mtype from this TCP to all TCCs Sum over TCP instances."></metric>
+  <metric name="TCP_TCC_UC_READ_REQ_sum"                                expr=reduce(TCP_TCC_UC_READ_REQ,sum) descr="Total read requests with UC mtype from this TCP to all TCCs Sum over TCP instances."></metric>
+  <metric name="TCP_TCC_UC_WRITE_REQ_sum"                               expr=reduce(TCP_TCC_UC_WRITE_REQ,sum) descr="Total write requests with UC mtype from this TCP to all TCCs Sum over TCP instances."></metric>
+  <metric name="TCP_TCC_UC_ATOMIC_REQ_sum"                              expr=reduce(TCP_TCC_UC_ATOMIC_REQ,sum) descr="Total atomic requests with UC mtype from this TCP to all TCCs Sum over TCP instances."></metric>
+  <metric name="TCP_TCC_CC_READ_REQ_sum"                                expr=reduce(TCP_TCC_CC_READ_REQ,sum) descr="Total write requests with CC mtype from this TCP to all TCCs Sum over TCP instances."></metric>
+  <metric name="TCP_TCC_CC_WRITE_REQ_sum"                               expr=reduce(TCP_TCC_CC_WRITE_REQ,sum) descr="Total write requests with CC mtype from this TCP to all TCCs Sum over TCP instances."></metric>
+  <metric name="TCP_TCC_CC_ATOMIC_REQ_sum"                              expr=reduce(TCP_TCC_CC_ATOMIC_REQ,sum) descr="Total atomic requests with CC mtype from this TCP to all TCCs Sum over TCP instances."></metric>
+  <metric name="TCP_TCC_RW_READ_REQ_sum"                                expr=reduce(TCP_TCC_RW_READ_REQ,sum) descr="Total write requests with RW mtype from this TCP to all TCCs. Sum over TCP instances."></metric>
+  <metric name="TCP_TCC_RW_WRITE_REQ_sum"                               expr=reduce(TCP_TCC_RW_WRITE_REQ,sum) descr="Total write requests with RW mtype from this TCP to all TCCs. Sum over TCP instances."></metric>
+  <metric name="TCP_TCC_RW_ATOMIC_REQ_sum"                              expr=reduce(TCP_TCC_RW_ATOMIC_REQ,sum) descr="Total atomic requests with RW mtype from this TCP to all TCCs. Sum over TCP instances."></metric>
+  <metric name="TCP_PENDING_STALL_CYCLES_sum"                           expr=reduce(TCP_PENDING_STALL_CYCLES,sum) descr="Stall due to data pending from L2. Sum over TCP instances."></metric>
+  <metric name="TCA_CYCLE_sum"                                          expr=reduce(TCA_CYCLE,sum)  descr="Number of cycles. Sum over all TCA instances "></metric>
+  <metric name="TCA_BUSY_sum"                                           expr=reduce(TCA_BUSY,sum)   descr="Number of cycles we have a request pending. Sum over all TCA instances."></metric>
+  <metric name="TCC_BUSY_avr"                                           expr=reduce(TCC_BUSY,average) descr="TCC_BUSY avr over all memory channels."></metric>
+  <metric name="TCC_WRREQ_STALL_max"                                    expr=reduce(TCC_EA_WRREQ_STALL,max) descr="Number of cycles a write request was stalled. Max over TCC instances."></metric>
+  <metric name="TCC_CYCLE_sum"                                          expr=reduce(TCC_CYCLE,sum) descr="Number of cycles. Not windowable. Sum over TCC instances."></metric>
+  <metric name="TCC_BUSY_sum"                                           expr=reduce(TCC_BUSY,sum) descr="Number of cycles we have a request pending. Not windowable. Sum over TCC instances."></metric>
+  <metric name="TCC_REQ_sum"                                            expr=reduce(TCC_REQ,sum) descr="Number of requests of all types. This is measured at the tag block. This may be more than the number of requests arriving at the TCC, but it is a good indication of the total amount of work that needs to be performed. Sum over TCC instances."></metric>
+  <metric name="TCC_STREAMING_REQ_sum"                                  expr=reduce(TCC_STREAMING_REQ,sum) descr="Number of streaming requests. This is measured at the tag block. Sum over TCC instances."></metric>
+  <metric name="TCC_NC_REQ_sum"                                         expr=reduce(TCC_NC_REQ,sum) descr="The number of noncoherently cached requests. This is measured at the tag block. Sum over TCC instances."></metric>
+  <metric name="TCC_UC_REQ_sum"                                         expr=reduce(TCC_UC_REQ,sum) descr="The number of uncached requests. This is measured at the tag block. Sum over TCC instances."></metric>
+  <metric name="TCC_CC_REQ_sum"                                         expr=reduce(TCC_CC_REQ,sum) descr="The number of coherently cached requests. This is measured at the tag block. Sum over TCC instances."></metric>
+  <metric name="TCC_RW_REQ_sum"                                         expr=reduce(TCC_RW_REQ,sum) descr="The number of RW requests. This is measured at the tag block. Sum over TCC instances."></metric>
+  <metric name="TCC_PROBE_sum"                                          expr=reduce(TCC_PROBE,sum) descr="Number of probe requests. Not windowable. Sum over TCC instances."></metric>
+  <metric name="TCC_PROBE_ALL_sum"                                      expr=reduce(TCC_PROBE_ALL,sum) descr="Number of external probe requests with with EA_TCC_preq_all== 1. Not windowable. Sum over TCC instances."></metric>
+  <metric name="TCC_READ_sum"                                           expr=reduce(TCC_READ,sum) descr="Number of read requests. Compressed reads are included in this, but metadata reads are not included. Sum over TCC instances."></metric>
+  <metric name="TCC_WRITE_sum"                                          expr=reduce(TCC_WRITE,sum) descr="Number of write requests. Sum over TCC instances."></metric>
+  <metric name="TCC_ATOMIC_sum"                                         expr=reduce(TCC_ATOMIC,sum) descr="Number of atomic requests of all types. Sum over TCC instances."></metric>
+  <metric name="TCC_HIT_sum"                                            expr=reduce(TCC_HIT,sum) descr="Number of cache hits. Sum over TCC instances."></metric>
+  <metric name="TCC_MISS_sum"                                           expr=reduce(TCC_MISS,sum) descr="Number of cache misses. UC reads count as misses. Sum over TCC instances."></metric>
+  <metric name="TCC_WRITEBACK_sum"                                      expr=reduce(TCC_WRITEBACK,sum) descr="Number of lines written back to main memory. This includes writebacks of dirty lines and uncached write/atomic requests. Sum over TCC instances."></metric>
+  <metric name="TCC_EA_WRREQ_sum"                                       expr=reduce(TCC_EA_WRREQ,sum) descr="Number of transactions (either 32-byte or 64-byte) going over the TC_EA_wrreq interface. Atomics may travel over the same interface and are generally classified as write requests. This does not include probe commands. Sum over TCC instances."></metric>
+  <metric name="TCC_EA_WRREQ_64B_sum"                                   expr=reduce(TCC_EA_WRREQ_64B,sum) descr="Number of 64-byte transactions going (64-byte write or CMPSWAP) over the TC_EA_wrreq interface. Sum over TCC instances."></metric>
+  <metric name="TCC_EA_WR_UNCACHED_32B_sum"                             expr=reduce(TCC_EA_WR_UNCACHED_32B,sum) descr="Number of 32-byte write/atomic going over the TC_EA_wrreq interface due to uncached traffic. Note that CC mtypes can produce uncached requests, and those are included in this. A 64-byte request will be counted as 2. Sum over TCC instances."></metric>
+  <metric name="TCC_EA_WRREQ_STALL_sum"                                 expr=reduce(TCC_EA_WRREQ_STALL,sum) descr="Number of cycles a write request was stalled. Sum over TCC instances."></metric>
+  <metric name="TCC_EA_WRREQ_IO_CREDIT_STALL_sum"                       expr=reduce(TCC_EA_WRREQ_IO_CREDIT_STALL,sum) descr="Number of cycles a EA write request was stalled because the interface was out of IO credits. Sum over TCC instances."></metric>
+  <metric name="TCC_EA_WRREQ_GMI_CREDIT_STALL_sum"                      expr=reduce(TCC_EA_WRREQ_GMI_CREDIT_STALL,sum) descr="Number of cycles a EA write request was stalled because the interface was out of GMI credits. Sum over TCC instances."></metric>
+  <metric name="TCC_EA_WRREQ_DRAM_CREDIT_STALL_sum"                     expr=reduce(TCC_EA_WRREQ_DRAM_CREDIT_STALL,sum) descr="Number of cycles a EA write request was stalled because the interface was out of DRAM credits. Sum over TCC instances."></metric>
+  <metric name="TCC_TOO_MANY_EA_WRREQS_STALL_sum"                       expr=reduce(TCC_TOO_MANY_EA_WRREQS_STALL,sum) descr="Number of cycles the TCC could not send a EA write request because it already reached its maximum number of pending EA write requests. Sum over TCC instances."></metric>
+  <metric name="TCC_EA_WRREQ_LEVEL_sum"                                 expr=reduce(TCC_EA_WRREQ_LEVEL,sum) descr="The sum of the number of EA write requests in flight. This is primarily meant for measure average EA write latency. Average write latency = TCC_PERF_SEL_EA_WRREQ_LEVEL/TCC_PERF_SEL_EA_WRREQ. Sum over TCC instances."></metric>
+  <metric name="TCC_EA_RDREQ_LEVEL_sum"                                 expr=reduce(TCC_EA_RDREQ_LEVEL,sum) descr="The sum of the number of TCC/EA read requests in flight. This is primarily meant for measure average EA read latency. Average read latency = TCC_PERF_SEL_EA_RDREQ_LEVEL/TCC_PERF_SEL_EA_RDREQ. Sum over TCC instances."></metric>
+  <metric name="TCC_EA_ATOMIC_sum"                                      expr=reduce(TCC_EA_ATOMIC,sum) descr="Number of transactions going over the TC_EA_wrreq interface that are actually atomic requests. Sum over TCC instances."></metric>
+  <metric name="TCC_EA_ATOMIC_LEVEL_sum"                                expr=reduce(TCC_EA_ATOMIC_LEVEL,sum) descr="The sum of the number of EA atomics in flight. This is primarily meant for measure average EA atomic latency. Average atomic latency = TCC_PERF_SEL_EA_WRREQ_ATOMIC_LEVEL/TCC_PERF_SEL_EA_WRREQ_ATOMIC. Sum over TCC instances."></metric>
+  <metric name="TCC_EA_RDREQ_sum"                                       expr=reduce(TCC_EA_RDREQ,sum) descr="Number of TCC/EA read requests (either 32-byte or 64-byte) Sum over TCC instances."></metric>
+  <metric name="TCC_EA_RDREQ_32B_sum"                                   expr=reduce(TCC_EA_RDREQ_32B,sum) descr="Number of 32-byte TCC/EA read requests Sum over TCC instances."></metric>
+  <metric name="TCC_EA_RD_UNCACHED_32B_sum"                             expr=reduce(TCC_EA_RD_UNCACHED_32B,sum) descr="Number of 32-byte TCC/EA read due to uncached traffic. A 64-byte request will be counted as 2 Sum over TCC instances."></metric>
+  <metric name="TCC_EA_RDREQ_IO_CREDIT_STALL_sum"                       expr=reduce(TCC_EA_RDREQ_IO_CREDIT_STALL,sum) descr="Number of cycles there was a stall because the read request interface was out of IO credits. Stalls occur regardless of whether a read needed to be performed or not. Sum over TCC instances."></metric>
+  <metric name="TCC_EA_RDREQ_GMI_CREDIT_STALL_sum"                      expr=reduce(TCC_EA_RDREQ_GMI_CREDIT_STALL,sum) descr="Number of cycles there was a stall because the read request interface was out of GMI credits. Stalls occur regardless of whether a read needed to be performed or not. Sum over TCC instances."></metric>
+  <metric name="TCC_EA_RDREQ_DRAM_CREDIT_STALL_sum"                     expr=reduce(TCC_EA_RDREQ_DRAM_CREDIT_STALL,sum) descr="Number of cycles there was a stall because the read request interface was out of DRAM credits. Stalls occur regardless of whether a read needed to be performed or not. Sum over TCC instances."></metric>
+  <metric name="TCC_TAG_STALL_sum"                                      expr=reduce(TCC_TAG_STALL,sum) descr="."></metric>
+  <metric name="TCC_NORMAL_WRITEBACK_sum"                               expr=reduce(TCC_NORMAL_WRITEBACK,sum) descr="Number of writebacks due to requests that are not writeback requests. Sum over TCC instances."></metric>
+  <metric name="TCC_ALL_TC_OP_WB_WRITEBACK_sum"                         expr=reduce(TCC_ALL_TC_OP_WB_WRITEBACK,sum) descr="Number of writebacks due to all TC_OP writeback requests. Sum over TCC instances."></metric>
+  <metric name="TCC_NORMAL_EVICT_sum"                                   expr=reduce(TCC_NORMAL_EVICT,sum) descr="Number of evictions due to requests that are not invalidate or probe requests. Sum over TCC instances."></metric>
+  <metric name="TCC_ALL_TC_OP_INV_EVICT_sum"                            expr=reduce(TCC_ALL_TC_OP_INV_EVICT,sum) descr="Number of evictions due to all TC_OP invalidate requests. Sum over TCC instances."></metric>
+  <metric name="TCC_EA_RDREQ_DRAM_sum"                                  expr=reduce(TCC_EA_RDREQ_DRAM,sum) descr="Number of TCC/EA read requests (either 32-byte or 64-byte) destined for DRAM (MC). Sum over TCC instances."></metric>
+  <metric name="TCC_EA_WRREQ_DRAM_sum"                                  expr=reduce(TCC_EA_WRREQ_DRAM,sum) descr="Number of TCC/EA write requests (either 32-byte of 64-byte) destined for DRAM (MC). Sum over TCC instances."></metric>

  <metric name="FETCH_SIZE"                                             expr=(TCC_EA_RDREQ_32B_sum*32+(TCC_EA_RDREQ_sum-TCC_EA_RDREQ_32B_sum)*64)/1024 descr="The total kilobytes fetched from the video memory. This is measured with all extra fetches and any cache or memory effects taken into account."></metric>
  <metric name="WRITE_SIZE"                                             expr=((TCC_EA_WRREQ_sum-TCC_EA_WRREQ_64B_sum)*32+TCC_EA_WRREQ_64B_sum*64)/1024 descr="The total kilobytes written to the video memory. This is measured with all extra fetches and any cache or memory effects taken into account."></metric>
@@ -290,11 +288,11 @@
  <metric name="EaWrDramStallRate"                                      expr=100*TCC_EA_WRREQ_DRAM_CREDIT_STALL_sum/TCC_BUSY_sum              descr="Unit: percent"></metric>
  <metric name="EaWrStarveRate"                                         expr=100*TCC_TOO_MANY_EA_WRREQS_STALL_sum/TCC_BUSY_sum                descr="Unit: percent"></metric>
  <metric name="EaAtomicLatency"                                        expr=TCC_EA_ATOMIC_LEVEL_sum/TCC_EA_ATOMIC_sum                        descr="Unit: cycles"></metric>
-</gfx90a_expr>
+</gfx90a>

-<gfx940_expr>
-  <metric name="TCP_TCP_TA_DATA_STALL_CYCLES_sum" expr=sum(TCP_TCP_TA_DATA_STALL_CYCLES,16) descr="Total number of TCP stalls TA data interface."></metric>
-  <metric name="TCP_TCP_TA_DATA_STALL_CYCLES_max" expr=max(TCP_TCP_TA_DATA_STALL_CYCLES,16) descr="Maximum number of TCP stalls TA data interface."></metric>
+<gfx940>
+  <metric name="TCP_TCP_TA_DATA_STALL_CYCLES_sum" expr=reduce(TCP_TCP_TA_DATA_STALL_CYCLES,sum) descr="Total number of TCP stalls TA data interface."></metric>
+  <metric name="TCP_TCP_TA_DATA_STALL_CYCLES_max" expr=reduce(TCP_TCP_TA_DATA_STALL_CYCLES,max) descr="Maximum number of TCP stalls TA data interface."></metric>

  <metric name="MeanOccupancyPerCU" expr=SQ_LEVEL_WAVES*0+SQ_ACCUM_PREV_HIRES/GRBM_GUI_ACTIVE/CU_NUM descr="Mean occupancy per compute unit."></metric>
  <metric name="MeanOccupancyPerActiveCU" expr=SQ_LEVEL_WAVES*0+SQ_ACCUM_PREV_HIRES*4/SQ_BUSY_CYCLES/CU_NUM descr="Mean occupancy per active compute unit."></metric>
@@ -307,114 +305,114 @@
  <metric name="WriteSize" expr=WRITE_SIZE descr="The total kilobytes written to the video memory. This is measured with all extra fetches and any cache or memory effects taken into account."></metric>
  <metric name="MemWrites32B" expr=WRITE_REQ_32B descr="The total number of effective 32B write transactions to the memory"></metric>
  <metric name="MemUnitStalled" expr=100*TCP_TCP_TA_DATA_STALL_CYCLES_max/GRBM_GUI_ACTIVE/SE_NUM descr="The percentage of GPUTime the memory unit is stalled. Try reducing the number or size of fetches and writes if possible. Value range: 0% (optimal) to 100% (bad)."></metric>
-  <metric name="TA_BUSY_avr"                                            expr=avr(TA_TA_BUSY,16) descr="TA block is busy. Average over TA instances."></metric>
-  <metric name="TA_BUSY_max"                                            expr=max(TA_TA_BUSY,16) descr="TA block is busy. Max over TA instances."></metric>
-  <metric name="TA_BUSY_min"                                            expr=min(TA_TA_BUSY,16) descr="TA block is busy. Min over TA instances."></metric>
-  <metric name="TA_TA_BUSY_sum"                                         expr=sum(TA_TA_BUSY,16) descr="TA block is busy. Perf_Windowing not supported for this counter. Sum over TA instances."></metric>
-  <metric name="TA_TOTAL_WAVEFRONTS_sum"                                expr=sum(TA_TOTAL_WAVEFRONTS,16) descr="Total number of wavefronts processed by TA. Sum over TA instances."></metric>
-  <metric name="TA_ADDR_STALLED_BY_TC_CYCLES_sum"                       expr=sum(TA_ADDR_STALLED_BY_TC_CYCLES,16) descr="Number of cycles addr path stalled by TC. Perf_Windowing not supported for this counter. Sum over TA instances."></metric>
-  <metric name="TA_ADDR_STALLED_BY_TD_CYCLES_sum"                       expr=sum(TA_ADDR_STALLED_BY_TD_CYCLES,16) descr="Number of cycles addr path stalled by TD. Perf_Windowing not supported for this counter. Sum over TA instances."></metric>
-  <metric name="TA_DATA_STALLED_BY_TC_CYCLES_sum"                       expr=sum(TA_DATA_STALLED_BY_TC_CYCLES,16) descr="Number of cycles data path stalled by TC. Perf_Windowing not supported for this counter. Sum over TA instances."></metric>
-  <metric name="TA_FLAT_WAVEFRONTS_sum"                                 expr=sum(TA_FLAT_WAVEFRONTS,16) descr="Number of flat opcode wavfronts processed by the TA. Sum over TA instances."></metric>
-  <metric name="TA_FLAT_READ_WAVEFRONTS_sum"                            expr=sum(TA_FLAT_READ_WAVEFRONTS,16) descr="Number of flat opcode reads processed by the TA. Sum over TA instances."></metric>
-  <metric name="TA_FLAT_WRITE_WAVEFRONTS_sum"                           expr=sum(TA_FLAT_WRITE_WAVEFRONTS,16) descr="Number of flat opcode writes processed by the TA. Sum over TA instances."></metric>
-  <metric name="TA_FLAT_ATOMIC_WAVEFRONTS_sum"                          expr=sum(TA_FLAT_ATOMIC_WAVEFRONTS,16) descr="Number of flat opcode atomics processed by the TA. Sum over TA instances."></metric>
-  <metric name="TA_BUFFER_WAVEFRONTS_sum"                               expr=sum(TA_BUFFER_WAVEFRONTS,16) descr="Number of buffer wavefronts processed by TA. Sum over TA instances."></metric>
-  <metric name="TA_BUFFER_READ_WAVEFRONTS_sum"                          expr=sum(TA_BUFFER_READ_WAVEFRONTS,16) descr="Number of buffer read wavefronts processed by TA. Sum over TA instances."></metric>
-  <metric name="TA_BUFFER_WRITE_WAVEFRONTS_sum"                         expr=sum(TA_BUFFER_WRITE_WAVEFRONTS,16) descr="Number of buffer write wavefronts processed by TA. Sum over TA instances."></metric>
-  <metric name="TA_BUFFER_ATOMIC_WAVEFRONTS_sum"                        expr=sum(TA_BUFFER_ATOMIC_WAVEFRONTS,16) descr="Number of buffer atomic wavefronts processed by TA. Sum over TA instances."></metric>
-  <metric name="TA_BUFFER_TOTAL_CYCLES_sum"                             expr=sum(TA_BUFFER_TOTAL_CYCLES,16) descr="Number of buffer cycles issued to TC. Sum over TA instances."></metric>
-  <metric name="TA_BUFFER_COALESCED_READ_CYCLES_sum"                    expr=sum(TA_BUFFER_COALESCED_READ_CYCLES,16) descr="Number of buffer coalesced read cycles issued to TC. Sum over TA instances."></metric>
-  <metric name="TA_BUFFER_COALESCED_WRITE_CYCLES_sum"                   expr=sum(TA_BUFFER_COALESCED_WRITE_CYCLES,16) descr="Number of buffer coalesced write cycles issued to TC. Sum over TA instances."></metric>
-  <metric name="TD_TD_BUSY_sum"                                         expr=sum(TD_TD_BUSY,16) descr="TD is processing or waiting for data. Perf_Windowing not supported for this counter. Sum over TD instances."></metric>
-  <metric name="TD_TC_STALL_sum"                                        expr=sum(TD_TC_STALL,16) descr="TD is stalled waiting for TC data. Sum over TD instances."></metric>
-  <metric name="TD_LOAD_WAVEFRONT_sum"                                  expr=sum(TD_LOAD_WAVEFRONT,16) descr="Count the wavefronts with opcode = load, include atomics and store. Sum over TD instances."></metric>
-  <metric name="TD_ATOMIC_WAVEFRONT_sum"                                expr=sum(TD_ATOMIC_WAVEFRONT,16) descr="Count the wavefronts with opcode = atomic. Sum over TD instances."></metric>
-  <metric name="TD_STORE_WAVEFRONT_sum"                                 expr=sum(TD_STORE_WAVEFRONT,16) descr="Count the wavefronts with opcode = store. Sum over TD instances."></metric>
-  <metric name="TD_COALESCABLE_WAVEFRONT_sum"                           expr=sum(TD_COALESCABLE_WAVEFRONT,16) descr="Count wavefronts that TA finds coalescable. Sum over TD instances."></metric>
-  <metric name="TD_SPI_STALL_sum"                                       expr=sum(TD_SPI_STALL,16) descr="TD is stalled SPI vinit, sum of TCP instances"></metric>
-  <metric name="TCP_GATE_EN1_sum"                                       expr=sum(TCP_GATE_EN1,16) descr="TCP interface clocks are turned on. Not Windowed. Sum over TCP instances."></metric>
-  <metric name="TCP_GATE_EN2_sum"                                       expr=sum(TCP_GATE_EN2,16) descr="TCP core clocks are turned on. Not Windowed. Sum over TCP instances."></metric>
-  <metric name="TCP_TD_TCP_STALL_CYCLES_sum"                            expr=sum(TCP_TD_TCP_STALL_CYCLES,16) descr="TD stalls TCP. Sum over TCP instances."></metric>
-  <metric name="TCP_TCR_TCP_STALL_CYCLES_sum"                           expr=sum(TCP_TCR_TCP_STALL_CYCLES,16) descr="TCR stalls TCP_TCR_req interface. Sum over TCP instances."></metric>
-  <metric name="TCP_READ_TAGCONFLICT_STALL_CYCLES_sum"                  expr=sum(TCP_READ_TAGCONFLICT_STALL_CYCLES,16) descr="Tagram conflict stall on a read. Sum over TCP instances."></metric>
-  <metric name="TCP_WRITE_TAGCONFLICT_STALL_CYCLES_sum"                 expr=sum(TCP_WRITE_TAGCONFLICT_STALL_CYCLES,16) descr="Tagram conflict stall on a write. Sum over TCP instances."></metric>
-  <metric name="TCP_ATOMIC_TAGCONFLICT_STALL_CYCLES_sum"                expr=sum(TCP_ATOMIC_TAGCONFLICT_STALL_CYCLES,16) descr="Tagram conflict stall on an atomic. Sum over TCP instances."></metric>
-  <metric name="TCP_VOLATILE_sum"                                       expr=sum(TCP_VOLATILE,16) descr="Total number of L1 volatile pixels/buffers from TA. Sum over TCP instances."></metric>
-  <metric name="TCP_TOTAL_ACCESSES_sum"                                 expr=sum(TCP_TOTAL_ACCESSES,16) descr="Total number of pixels/buffers from TA. Equals TCP_PERF_SEL_TOTAL_READ+TCP_PERF_SEL_TOTAL_NONREAD. Sum over TCP instances."></metric>
-  <metric name="TCP_TOTAL_READ_sum"                                     expr=sum(TCP_TOTAL_READ,16) descr="Total number of read pixels/buffers from TA. Equals TCP_PERF_SEL_TOTAL_HIT_LRU_READ + TCP_PERF_SEL_TOTAL_MISS_LRU_READ + TCP_PERF_SEL_TOTAL_MISS_EVICT_READ. Sum over TCP instances."></metric>
-  <metric name="TCP_TOTAL_WRITE_sum"                                    expr=sum(TCP_TOTAL_WRITE,16) descr="Total number of local write pixels/buffers from TA. Equals TCP_PERF_SEL_TOTAL_MISS_LRU_WRITE+ TCP_PERF_SEL_TOTAL_MISS_EVICT_WRITE. Sum over TCP instances."></metric>
-  <metric name="TCP_TOTAL_ATOMIC_WITH_RET_sum"                          expr=sum(TCP_TOTAL_ATOMIC_WITH_RET,16) descr="Total number of atomic with return pixels/buffers from TA. Sum over TCP instances."></metric>
-  <metric name="TCP_TOTAL_ATOMIC_WITHOUT_RET_sum"                       expr=sum(TCP_TOTAL_ATOMIC_WITHOUT_RET,16) descr="Total number of atomic without return pixels/buffers from TA Sum over TCP instances."></metric>
-  <metric name="TCP_TOTAL_WRITEBACK_INVALIDATES_sum"                    expr=sum(TCP_TOTAL_WRITEBACK_INVALIDATES,16) descr="Total number of cache invalidates. Equals TCP_PERF_SEL_TOTAL_WBINVL1+ TCP_PERF_SEL_TOTAL_WBINVL1_VOL+ TCP_PERF_SEL_CP_TCP_INVALIDATE+ TCP_PERF_SEL_SQ_TCP_INVALIDATE_VOL. Not Windowed. Sum over TCP instances."></metric>
-  <metric name="TCP_UTCL1_REQUEST_sum"                                  expr=sum(TCP_UTCL1_REQUEST,16) descr="Total CLIENT_UTCL1 NORMAL requests Sum over TCP instances."></metric>
-  <metric name="TCP_UTCL1_TRANSLATION_MISS_sum"                         expr=sum(TCP_UTCL1_TRANSLATION_MISS,16) descr="Total utcl1 translation misses Sum over TCP instances."></metric>
-  <metric name="TCP_UTCL1_TRANSLATION_HIT_sum"                          expr=sum(TCP_UTCL1_TRANSLATION_HIT,16) descr="Total utcl1 translation hits Sum over TCP instances."></metric>
-  <metric name="TCP_UTCL1_PERMISSION_MISS_sum"                          expr=sum(TCP_UTCL1_PERMISSION_MISS,16) descr="Total utcl1 permission misses Sum over TCP instances."></metric>
-  <metric name="TCP_TOTAL_CACHE_ACCESSES_sum"                           expr=sum(TCP_TOTAL_CACHE_ACCESSES,16) descr="Count of total cache line (tag) accesses (includes hits and misses). Sum over TCP instances."></metric>
-  <metric name="TCP_TA_TCP_STATE_READ_sum"                              expr=sum(TCP_TA_TCP_STATE_READ,16) descr="Number of state reads Sum over TCP instances."></metric>
-  <metric name="TCP_TCC_READ_REQ_sum"                                   expr=sum(TCP_TCC_READ_REQ,16) descr="Total read requests from TCP to all TCCs Sum over TCP instances."></metric>
-  <metric name="TCP_TCC_WRITE_REQ_sum"                                  expr=sum(TCP_TCC_WRITE_REQ,16) descr="Total write requests from TCP to all TCCs Sum over TCP instances."></metric>
-  <metric name="TCP_TCC_ATOMIC_WITH_RET_REQ_sum"                        expr=sum(TCP_TCC_ATOMIC_WITH_RET_REQ,16) descr="Total atomic with return requests from TCP to all TCCs Sum over TCP instances."></metric>
-  <metric name="TCP_TCC_ATOMIC_WITHOUT_RET_REQ_sum"                     expr=sum(TCP_TCC_ATOMIC_WITHOUT_RET_REQ,16) descr="Total atomic without return requests from TCP to all TCCs Sum over TCP instances."></metric>
-  <metric name="TCP_TCC_NC_READ_REQ_sum"                                expr=sum(TCP_TCC_NC_READ_REQ,16) descr="Total read requests with NC mtype from this TCP to all TCCs Sum over TCP instances."></metric>
-  <metric name="TCP_TCC_NC_WRITE_REQ_sum"                               expr=sum(TCP_TCC_NC_WRITE_REQ,16) descr="Total write requests with NC mtype from this TCP to all TCCs Sum over TCP instances."></metric>
-  <metric name="TCP_TCC_NC_ATOMIC_REQ_sum"                              expr=sum(TCP_TCC_NC_ATOMIC_REQ,16) descr="Total atomic requests with NC mtype from this TCP to all TCCs Sum over TCP instances."></metric>
-  <metric name="TCP_TCC_UC_READ_REQ_sum"                                expr=sum(TCP_TCC_UC_READ_REQ,16) descr="Total read requests with UC mtype from this TCP to all TCCs Sum over TCP instances."></metric>
-  <metric name="TCP_TCC_UC_WRITE_REQ_sum"                               expr=sum(TCP_TCC_UC_WRITE_REQ,16) descr="Total write requests with UC mtype from this TCP to all TCCs Sum over TCP instances."></metric>
-  <metric name="TCP_TCC_UC_ATOMIC_REQ_sum"                              expr=sum(TCP_TCC_UC_ATOMIC_REQ,16) descr="Total atomic requests with UC mtype from this TCP to all TCCs Sum over TCP instances."></metric>
-  <metric name="TCP_TCC_CC_READ_REQ_sum"                                expr=sum(TCP_TCC_CC_READ_REQ,16) descr="Total write requests with CC mtype from this TCP to all TCCs Sum over TCP instances."></metric>
-  <metric name="TCP_TCC_CC_WRITE_REQ_sum"                               expr=sum(TCP_TCC_CC_WRITE_REQ,16) descr="Total write requests with CC mtype from this TCP to all TCCs Sum over TCP instances."></metric>
-  <metric name="TCP_TCC_CC_ATOMIC_REQ_sum"                              expr=sum(TCP_TCC_CC_ATOMIC_REQ,16) descr="Total atomic requests with CC mtype from this TCP to all TCCs Sum over TCP instances."></metric>
-  <metric name="TCP_TCC_RW_READ_REQ_sum"                                expr=sum(TCP_TCC_RW_READ_REQ,16) descr="Total write requests with RW mtype from this TCP to all TCCs. Sum over TCP instances."></metric>
-  <metric name="TCP_TCC_RW_WRITE_REQ_sum"                               expr=sum(TCP_TCC_RW_WRITE_REQ,16) descr="Total write requests with RW mtype from this TCP to all TCCs. Sum over TCP instances."></metric>
-  <metric name="TCP_TCC_RW_ATOMIC_REQ_sum"                              expr=sum(TCP_TCC_RW_ATOMIC_REQ,16) descr="Total atomic requests with RW mtype from this TCP to all TCCs. Sum over TCP instances."></metric>
-  <metric name="TCP_PENDING_STALL_CYCLES_sum"                           expr=sum(TCP_PENDING_STALL_CYCLES,16) descr="Stall due to data pending from L2. Sum over TCP instances."></metric>
-  <metric name="TCA_CYCLE_sum"                                          expr=sum(TCA_CYCLE,16)  descr="Number of cycles. Sum over all TCA instances "></metric>
-  <metric name="TCA_BUSY_sum"                                           expr=sum(TCA_BUSY,16)   descr="Number of cycles we have a request pending. Sum over all TCA instances."></metric>
-  <metric name="TCC_BUSY_avr"                                           expr=avr(TCC_BUSY,16) descr="TCC_BUSY avr over all memory channels."></metric>
-  <metric name="TCC_WRREQ_STALL_max"                                    expr=max(TCC_EA0_WRREQ_STALL,16) descr="Number of cycles a write request was stalled. Max over TCC instances."></metric>
-  <metric name="TCC_CYCLE_sum"                                          expr=sum(TCC_CYCLE,16) descr="Number of cycles. Not windowable. Sum over TCC instances."></metric>
-  <metric name="TCC_BUSY_sum"                                           expr=sum(TCC_BUSY,16) descr="Number of cycles we have a request pending. Not windowable. Sum over TCC instances."></metric>
-  <metric name="TCC_REQ_sum"                                            expr=sum(TCC_REQ,16) descr="Number of requests of all types. This is measured at the tag block. This may be more than the number of requests arriving at the TCC, but it is a good indication of the total amount of work that needs to be performed. Sum over TCC instances."></metric>
-  <metric name="TCC_STREAMING_REQ_sum"                                  expr=sum(TCC_STREAMING_REQ,16) descr="Number of streaming requests. This is measured at the tag block. Sum over TCC instances."></metric>
-  <metric name="TCC_NC_REQ_sum"                                         expr=sum(TCC_NC_REQ,16) descr="The number of noncoherently cached requests. This is measured at the tag block. Sum over TCC instances."></metric>
-  <metric name="TCC_UC_REQ_sum"                                         expr=sum(TCC_UC_REQ,16) descr="The number of uncached requests. This is measured at the tag block. Sum over TCC instances."></metric>
-  <metric name="TCC_CC_REQ_sum"                                         expr=sum(TCC_CC_REQ,16) descr="The number of coherently cached requests. This is measured at the tag block. Sum over TCC instances."></metric>
-  <metric name="TCC_RW_REQ_sum"                                         expr=sum(TCC_RW_REQ,16) descr="The number of RW requests. This is measured at the tag block. Sum over TCC instances."></metric>
-  <metric name="TCC_PROBE_sum"                                          expr=sum(TCC_PROBE,16) descr="Number of probe requests. Not windowable. Sum over TCC instances."></metric>
-  <metric name="TCC_PROBE_ALL_sum"                                      expr=sum(TCC_PROBE_ALL,16) descr="Number of external probe requests with with EA_TCC_preq_all== 1. Not windowable. Sum over TCC instances."></metric>
-  <metric name="TCC_READ_sum"                                           expr=sum(TCC_READ,16) descr="Number of read requests. Compressed reads are included in this, but metadata reads are not included. Sum over TCC instances."></metric>
-  <metric name="TCC_WRITE_sum"                                          expr=sum(TCC_WRITE,16) descr="Number of write requests. Sum over TCC instances."></metric>
-  <metric name="TCC_ATOMIC_sum"                                         expr=sum(TCC_ATOMIC,16) descr="Number of atomic requests of all types. Sum over TCC instances."></metric>
-  <metric name="TCC_HIT_sum"                                            expr=sum(TCC_HIT,16) descr="Number of cache hits. Sum over TCC instances."></metric>
-  <metric name="TCC_MISS_sum"                                           expr=sum(TCC_MISS,16) descr="Number of cache misses. UC reads count as misses. Sum over TCC instances."></metric>
-  <metric name="TCC_WRITEBACK_sum"                                      expr=sum(TCC_WRITEBACK,16) descr="Number of lines written back to main memory. This includes writebacks of dirty lines and uncached write/atomic requests. Sum over TCC instances."></metric>
-  <metric name="TCC_EA0_WRREQ_sum"                                       expr=sum(TCC_EA0_WRREQ,16) descr="Number of transactions (either 32-byte or 64-byte) going over the TC_EA_wrreq interface. Atomics may travel over the same interface and are generally classified as write requests. This does not include probe commands. Sum over TCC instances."></metric>
-  <metric name="TCC_EA0_WRREQ_64B_sum"                                   expr=sum(TCC_EA0_WRREQ_64B,16) descr="Number of 64-byte transactions going (64-byte write or CMPSWAP) over the TC_EA_wrreq interface. Sum over TCC instances."></metric>
-  <metric name="TCC_EA0_WR_UNCACHED_32B_sum"                             expr=sum(TCC_EA0_WR_UNCACHED_32B,16) descr="Number of 32-byte write/atomic going over the TC_EA_wrreq interface due to uncached traffic. Note that CC mtypes can produce uncached requests, and those are included in this. A 64-byte request will be counted as 2. Sum over TCC instances."></metric>
-  <metric name="TCC_EA0_WRREQ_STALL_sum"                                 expr=sum(TCC_EA0_WRREQ_STALL,16) descr="Number of cycles a write request was stalled. Sum over TCC instances."></metric>
-  <metric name="TCC_EA0_WRREQ_IO_CREDIT_STALL_sum"                       expr=sum(TCC_EA0_WRREQ_IO_CREDIT_STALL,16) descr="Number of cycles a EA write request was stalled because the interface was out of IO credits. Sum over TCC instances."></metric>
-  <metric name="TCC_EA0_WRREQ_GMI_CREDIT_STALL_sum"                      expr=sum(TCC_EA0_WRREQ_GMI_CREDIT_STALL,16) descr="Number of cycles a EA write request was stalled because the interface was out of GMI credits. Sum over TCC instances."></metric>
-  <metric name="TCC_EA0_WRREQ_DRAM_CREDIT_STALL_sum"                     expr=sum(TCC_EA0_WRREQ_DRAM_CREDIT_STALL,16) descr="Number of cycles a EA write request was stalled because the interface was out of DRAM credits. Sum over TCC instances."></metric>
-  <metric name="TCC_TOO_MANY_EA_WRREQS_STALL_sum"                       expr=sum(TCC_TOO_MANY_EA_WRREQS_STALL,16) descr="Number of cycles the TCC could not send a EA write request because it already reached its maximum number of pending EA write requests. Sum over TCC instances."></metric>
-  <metric name="TCC_EA0_WRREQ_LEVEL_sum"                                 expr=sum(TCC_EA0_WRREQ_LEVEL,16) descr="The sum of the number of EA write requests in flight. This is primarily meant for measure average EA write latency. Average write latency = TCC_PERF_SEL_EA_WRREQ_LEVEL/TCC_PERF_SEL_EA_WRREQ. Sum over TCC instances."></metric>
-  <metric name="TCC_EA0_RDREQ_LEVEL_sum"                                 expr=sum(TCC_EA0_RDREQ_LEVEL,16) descr="The sum of the number of TCC/EA read requests in flight. This is primarily meant for measure average EA read latency. Average read latency = TCC_PERF_SEL_EA_RDREQ_LEVEL/TCC_PERF_SEL_EA_RDREQ. Sum over TCC instances."></metric>
-  <metric name="TCC_EA0_ATOMIC_sum"                                      expr=sum(TCC_EA0_ATOMIC,16) descr="Number of transactions going over the TC_EA_wrreq interface that are actually atomic requests. Sum over TCC instances."></metric>
-  <metric name="TCC_EA0_ATOMIC_LEVEL_sum"                                expr=sum(TCC_EA0_ATOMIC_LEVEL,16) descr="The sum of the number of EA atomics in flight. This is primarily meant for measure average EA atomic latency. Average atomic latency = TCC_PERF_SEL_EA_WRREQ_ATOMIC_LEVEL/TCC_PERF_SEL_EA_WRREQ_ATOMIC. Sum over TCC instances."></metric>
-  <metric name="TCC_EA0_RDREQ_sum"                                       expr=sum(TCC_EA0_RDREQ,16) descr="Number of TCC/EA read requests (either 32-byte or 64-byte) Sum over TCC instances."></metric>
-  <metric name="TCC_EA0_RDREQ_32B_sum"                                   expr=sum(TCC_EA0_RDREQ_32B,16) descr="Number of 32-byte TCC/EA read requests Sum over TCC instances."></metric>
-  <metric name="TCC_EA0_RD_UNCACHED_32B_sum"                             expr=sum(TCC_EA0_RD_UNCACHED_32B,16) descr="Number of 32-byte TCC/EA read due to uncached traffic. A 64-byte request will be counted as 2 Sum over TCC instances."></metric>
-  <metric name="TCC_EA0_RDREQ_IO_CREDIT_STALL_sum"                       expr=sum(TCC_EA0_RDREQ_IO_CREDIT_STALL,16) descr="Number of cycles there was a stall because the read request interface was out of IO credits. Stalls occur regardless of whether a read needed to be performed or not. Sum over TCC instances."></metric>
-  <metric name="TCC_EA0_RDREQ_GMI_CREDIT_STALL_sum"                      expr=sum(TCC_EA0_RDREQ_GMI_CREDIT_STALL,16) descr="Number of cycles there was a stall because the read request interface was out of GMI credits. Stalls occur regardless of whether a read needed to be performed or not. Sum over TCC instances."></metric>
-  <metric name="TCC_EA0_RDREQ_DRAM_CREDIT_STALL_sum"                     expr=sum(TCC_EA0_RDREQ_DRAM_CREDIT_STALL,16) descr="Number of cycles there was a stall because the read request interface was out of DRAM credits. Stalls occur regardless of whether a read needed to be performed or not. Sum over TCC instances."></metric>
-  <metric name="TCC_TAG_STALL_sum"                                      expr=sum(TCC_TAG_STALL,16) descr="."></metric>
-  <metric name="TCC_NORMAL_WRITEBACK_sum"                               expr=sum(TCC_NORMAL_WRITEBACK,16) descr="Number of writebacks due to requests that are not writeback requests. Sum over TCC instances."></metric>
-  <metric name="TCC_ALL_TC_OP_WB_WRITEBACK_sum"                         expr=sum(TCC_ALL_TC_OP_WB_WRITEBACK,16) descr="Number of writebacks due to all TC_OP writeback requests. Sum over TCC instances."></metric>
-  <metric name="TCC_NORMAL_EVICT_sum"                                   expr=sum(TCC_NORMAL_EVICT,16) descr="Number of evictions due to requests that are not invalidate or probe requests. Sum over TCC instances."></metric>
-  <metric name="TCC_ALL_TC_OP_INV_EVICT_sum"                            expr=sum(TCC_ALL_TC_OP_INV_EVICT,16) descr="Number of evictions due to all TC_OP invalidate requests. Sum over TCC instances."></metric>
-  <metric name="TCC_EA0_RDREQ_DRAM_sum"                                  expr=sum(TCC_EA0_RDREQ_DRAM,16) descr="Number of TCC/EA read requests (either 32-byte or 64-byte) destined for DRAM (MC). Sum over TCC instances."></metric>
-  <metric name="TCC_EA0_WRREQ_DRAM_sum"                                  expr=sum(TCC_EA0_WRREQ_DRAM,16) descr="Number of TCC/EA write requests (either 32-byte of 64-byte) destined for DRAM (MC). Sum over TCC instances."></metric>
+  <metric name="TA_BUSY_avr"                                            expr=reduce(TA_TA_BUSY,average) descr="TA block is busy. Average over TA instances."></metric>
+  <metric name="TA_BUSY_max"                                            expr=reduce(TA_TA_BUSY,max) descr="TA block is busy. Max over TA instances."></metric>
+  <metric name="TA_BUSY_min"                                            expr=reduce(TA_TA_BUSY,min) descr="TA block is busy. Min over TA instances."></metric>
+  <metric name="TA_TA_BUSY_sum"                                         expr=reduce(TA_TA_BUSY,sum) descr="TA block is busy. Perf_Windowing not supported for this counter. Sum over TA instances."></metric>
+  <metric name="TA_TOTAL_WAVEFRONTS_sum"                                expr=reduce(TA_TOTAL_WAVEFRONTS,sum) descr="Total number of wavefronts processed by TA. Sum over TA instances."></metric>
+  <metric name="TA_ADDR_STALLED_BY_TC_CYCLES_sum"                       expr=reduce(TA_ADDR_STALLED_BY_TC_CYCLES,sum) descr="Number of cycles addr path stalled by TC. Perf_Windowing not supported for this counter. Sum over TA instances."></metric>
+  <metric name="TA_ADDR_STALLED_BY_TD_CYCLES_sum"                       expr=reduce(TA_ADDR_STALLED_BY_TD_CYCLES,sum) descr="Number of cycles addr path stalled by TD. Perf_Windowing not supported for this counter. Sum over TA instances."></metric>
+  <metric name="TA_DATA_STALLED_BY_TC_CYCLES_sum"                       expr=reduce(TA_DATA_STALLED_BY_TC_CYCLES,sum) descr="Number of cycles data path stalled by TC. Perf_Windowing not supported for this counter. Sum over TA instances."></metric>
+  <metric name="TA_FLAT_WAVEFRONTS_sum"                                 expr=reduce(TA_FLAT_WAVEFRONTS,sum) descr="Number of flat opcode wavfronts processed by the TA. Sum over TA instances."></metric>
+  <metric name="TA_FLAT_READ_WAVEFRONTS_sum"                            expr=reduce(TA_FLAT_READ_WAVEFRONTS,sum) descr="Number of flat opcode reads processed by the TA. Sum over TA instances."></metric>
+  <metric name="TA_FLAT_WRITE_WAVEFRONTS_sum"                           expr=reduce(TA_FLAT_WRITE_WAVEFRONTS,sum) descr="Number of flat opcode writes processed by the TA. Sum over TA instances."></metric>
+  <metric name="TA_FLAT_ATOMIC_WAVEFRONTS_sum"                          expr=reduce(TA_FLAT_ATOMIC_WAVEFRONTS,sum) descr="Number of flat opcode atomics processed by the TA. Sum over TA instances."></metric>
+  <metric name="TA_BUFFER_WAVEFRONTS_sum"                               expr=reduce(TA_BUFFER_WAVEFRONTS,sum) descr="Number of buffer wavefronts processed by TA. Sum over TA instances."></metric>
+  <metric name="TA_BUFFER_READ_WAVEFRONTS_sum"                          expr=reduce(TA_BUFFER_READ_WAVEFRONTS,sum) descr="Number of buffer read wavefronts processed by TA. Sum over TA instances."></metric>
+  <metric name="TA_BUFFER_WRITE_WAVEFRONTS_sum"                         expr=reduce(TA_BUFFER_WRITE_WAVEFRONTS,sum) descr="Number of buffer write wavefronts processed by TA. Sum over TA instances."></metric>
+  <metric name="TA_BUFFER_ATOMIC_WAVEFRONTS_sum"                        expr=reduce(TA_BUFFER_ATOMIC_WAVEFRONTS,sum) descr="Number of buffer atomic wavefronts processed by TA. Sum over TA instances."></metric>
+  <metric name="TA_BUFFER_TOTAL_CYCLES_sum"                             expr=reduce(TA_BUFFER_TOTAL_CYCLES,sum) descr="Number of buffer cycles issued to TC. Sum over TA instances."></metric>
+  <metric name="TA_BUFFER_COALESCED_READ_CYCLES_sum"                    expr=reduce(TA_BUFFER_COALESCED_READ_CYCLES,sum) descr="Number of buffer coalesced read cycles issued to TC. Sum over TA instances."></metric>
+  <metric name="TA_BUFFER_COALESCED_WRITE_CYCLES_sum"                   expr=reduce(TA_BUFFER_COALESCED_WRITE_CYCLES,sum) descr="Number of buffer coalesced write cycles issued to TC. Sum over TA instances."></metric>
+  <metric name="TD_TD_BUSY_sum"                                         expr=reduce(TD_TD_BUSY,sum) descr="TD is processing or waiting for data. Perf_Windowing not supported for this counter. Sum over TD instances."></metric>
+  <metric name="TD_TC_STALL_sum"                                        expr=reduce(TD_TC_STALL,sum) descr="TD is stalled waiting for TC data. Sum over TD instances."></metric>
+  <metric name="TD_LOAD_WAVEFRONT_sum"                                  expr=reduce(TD_LOAD_WAVEFRONT,sum) descr="Count the wavefronts with opcode = load, include atomics and store. Sum over TD instances."></metric>
+  <metric name="TD_ATOMIC_WAVEFRONT_sum"                                expr=reduce(TD_ATOMIC_WAVEFRONT,sum) descr="Count the wavefronts with opcode = atomic. Sum over TD instances."></metric>
+  <metric name="TD_STORE_WAVEFRONT_sum"                                 expr=reduce(TD_STORE_WAVEFRONT,sum) descr="Count the wavefronts with opcode = store. Sum over TD instances."></metric>
+  <metric name="TD_COALESCABLE_WAVEFRONT_sum"                           expr=reduce(TD_COALESCABLE_WAVEFRONT,sum) descr="Count wavefronts that TA finds coalescable. Sum over TD instances."></metric>
+  <metric name="TD_SPI_STALL_sum"                                       expr=reduce(TD_SPI_STALL,sum) descr="TD is stalled SPI vinit, sum of TCP instances"></metric>
+  <metric name="TCP_GATE_EN1_sum"                                       expr=reduce(TCP_GATE_EN1,sum) descr="TCP interface clocks are turned on. Not Windowed. Sum over TCP instances."></metric>
+  <metric name="TCP_GATE_EN2_sum"                                       expr=reduce(TCP_GATE_EN2,sum) descr="TCP core clocks are turned on. Not Windowed. Sum over TCP instances."></metric>
+  <metric name="TCP_TD_TCP_STALL_CYCLES_sum"                            expr=reduce(TCP_TD_TCP_STALL_CYCLES,sum) descr="TD stalls TCP. Sum over TCP instances."></metric>
+  <metric name="TCP_TCR_TCP_STALL_CYCLES_sum"                           expr=reduce(TCP_TCR_TCP_STALL_CYCLES,sum) descr="TCR stalls TCP_TCR_req interface. Sum over TCP instances."></metric>
+  <metric name="TCP_READ_TAGCONFLICT_STALL_CYCLES_sum"                  expr=reduce(TCP_READ_TAGCONFLICT_STALL_CYCLES,sum) descr="Tagram conflict stall on a read. Sum over TCP instances."></metric>
+  <metric name="TCP_WRITE_TAGCONFLICT_STALL_CYCLES_sum"                 expr=reduce(TCP_WRITE_TAGCONFLICT_STALL_CYCLES,sum) descr="Tagram conflict stall on a write. Sum over TCP instances."></metric>
+  <metric name="TCP_ATOMIC_TAGCONFLICT_STALL_CYCLES_sum"                expr=reduce(TCP_ATOMIC_TAGCONFLICT_STALL_CYCLES,sum) descr="Tagram conflict stall on an atomic. Sum over TCP instances."></metric>
+  <metric name="TCP_VOLATILE_sum"                                       expr=reduce(TCP_VOLATILE,sum) descr="Total number of L1 volatile pixels/buffers from TA. Sum over TCP instances."></metric>
+  <metric name="TCP_TOTAL_ACCESSES_sum"                                 expr=reduce(TCP_TOTAL_ACCESSES,sum) descr="Total number of pixels/buffers from TA. Equals TCP_PERF_SEL_TOTAL_READ+TCP_PERF_SEL_TOTAL_NONREAD. Sum over TCP instances."></metric>
+  <metric name="TCP_TOTAL_READ_sum"                                     expr=reduce(TCP_TOTAL_READ,sum) descr="Total number of read pixels/buffers from TA. Equals TCP_PERF_SEL_TOTAL_HIT_LRU_READ + TCP_PERF_SEL_TOTAL_MISS_LRU_READ + TCP_PERF_SEL_TOTAL_MISS_EVICT_READ. Sum over TCP instances."></metric>
+  <metric name="TCP_TOTAL_WRITE_sum"                                    expr=reduce(TCP_TOTAL_WRITE,sum) descr="Total number of local write pixels/buffers from TA. Equals TCP_PERF_SEL_TOTAL_MISS_LRU_WRITE+ TCP_PERF_SEL_TOTAL_MISS_EVICT_WRITE. Sum over TCP instances."></metric>
+  <metric name="TCP_TOTAL_ATOMIC_WITH_RET_sum"                          expr=reduce(TCP_TOTAL_ATOMIC_WITH_RET,sum) descr="Total number of atomic with return pixels/buffers from TA. Sum over TCP instances."></metric>
+  <metric name="TCP_TOTAL_ATOMIC_WITHOUT_RET_sum"                       expr=reduce(TCP_TOTAL_ATOMIC_WITHOUT_RET,sum) descr="Total number of atomic without return pixels/buffers from TA Sum over TCP instances."></metric>
+  <metric name="TCP_TOTAL_WRITEBACK_INVALIDATES_sum"                    expr=reduce(TCP_TOTAL_WRITEBACK_INVALIDATES,sum) descr="Total number of cache invalidates. Equals TCP_PERF_SEL_TOTAL_WBINVL1+ TCP_PERF_SEL_TOTAL_WBINVL1_VOL+ TCP_PERF_SEL_CP_TCP_INVALIDATE+ TCP_PERF_SEL_SQ_TCP_INVALIDATE_VOL. Not Windowed. Sum over TCP instances."></metric>
+  <metric name="TCP_UTCL1_REQUEST_sum"                                  expr=reduce(TCP_UTCL1_REQUEST,sum) descr="Total CLIENT_UTCL1 NORMAL requests Sum over TCP instances."></metric>
+  <metric name="TCP_UTCL1_TRANSLATION_MISS_sum"                         expr=reduce(TCP_UTCL1_TRANSLATION_MISS,sum) descr="Total utcl1 translation misses Sum over TCP instances."></metric>
+  <metric name="TCP_UTCL1_TRANSLATION_HIT_sum"                          expr=reduce(TCP_UTCL1_TRANSLATION_HIT,sum) descr="Total utcl1 translation hits Sum over TCP instances."></metric>
+  <metric name="TCP_UTCL1_PERMISSION_MISS_sum"                          expr=reduce(TCP_UTCL1_PERMISSION_MISS,sum) descr="Total utcl1 permission misses Sum over TCP instances."></metric>
+  <metric name="TCP_TOTAL_CACHE_ACCESSES_sum"                           expr=reduce(TCP_TOTAL_CACHE_ACCESSES,sum) descr="Count of total cache line (tag) accesses (includes hits and misses). Sum over TCP instances."></metric>
+  <metric name="TCP_TA_TCP_STATE_READ_sum"                              expr=reduce(TCP_TA_TCP_STATE_READ,sum) descr="Number of state reads Sum over TCP instances."></metric>
+  <metric name="TCP_TCC_READ_REQ_sum"                                   expr=reduce(TCP_TCC_READ_REQ,sum) descr="Total read requests from TCP to all TCCs Sum over TCP instances."></metric>
+  <metric name="TCP_TCC_WRITE_REQ_sum"                                  expr=reduce(TCP_TCC_WRITE_REQ,sum) descr="Total write requests from TCP to all TCCs Sum over TCP instances."></metric>
+  <metric name="TCP_TCC_ATOMIC_WITH_RET_REQ_sum"                        expr=reduce(TCP_TCC_ATOMIC_WITH_RET_REQ,sum) descr="Total atomic with return requests from TCP to all TCCs Sum over TCP instances."></metric>
+  <metric name="TCP_TCC_ATOMIC_WITHOUT_RET_REQ_sum"                     expr=reduce(TCP_TCC_ATOMIC_WITHOUT_RET_REQ,sum) descr="Total atomic without return requests from TCP to all TCCs Sum over TCP instances."></metric>
+  <metric name="TCP_TCC_NC_READ_REQ_sum"                                expr=reduce(TCP_TCC_NC_READ_REQ,sum) descr="Total read requests with NC mtype from this TCP to all TCCs Sum over TCP instances."></metric>
+  <metric name="TCP_TCC_NC_WRITE_REQ_sum"                               expr=reduce(TCP_TCC_NC_WRITE_REQ,sum) descr="Total write requests with NC mtype from this TCP to all TCCs Sum over TCP instances."></metric>
+  <metric name="TCP_TCC_NC_ATOMIC_REQ_sum"                              expr=reduce(TCP_TCC_NC_ATOMIC_REQ,sum) descr="Total atomic requests with NC mtype from this TCP to all TCCs Sum over TCP instances."></metric>
+  <metric name="TCP_TCC_UC_READ_REQ_sum"                                expr=reduce(TCP_TCC_UC_READ_REQ,sum) descr="Total read requests with UC mtype from this TCP to all TCCs Sum over TCP instances."></metric>
+  <metric name="TCP_TCC_UC_WRITE_REQ_sum"                               expr=reduce(TCP_TCC_UC_WRITE_REQ,sum) descr="Total write requests with UC mtype from this TCP to all TCCs Sum over TCP instances."></metric>
+  <metric name="TCP_TCC_UC_ATOMIC_REQ_sum"                              expr=reduce(TCP_TCC_UC_ATOMIC_REQ,sum) descr="Total atomic requests with UC mtype from this TCP to all TCCs Sum over TCP instances."></metric>
+  <metric name="TCP_TCC_CC_READ_REQ_sum"                                expr=reduce(TCP_TCC_CC_READ_REQ,sum) descr="Total write requests with CC mtype from this TCP to all TCCs Sum over TCP instances."></metric>
+  <metric name="TCP_TCC_CC_WRITE_REQ_sum"                               expr=reduce(TCP_TCC_CC_WRITE_REQ,sum) descr="Total write requests with CC mtype from this TCP to all TCCs Sum over TCP instances."></metric>
+  <metric name="TCP_TCC_CC_ATOMIC_REQ_sum"                              expr=reduce(TCP_TCC_CC_ATOMIC_REQ,sum) descr="Total atomic requests with CC mtype from this TCP to all TCCs Sum over TCP instances."></metric>
+  <metric name="TCP_TCC_RW_READ_REQ_sum"                                expr=reduce(TCP_TCC_RW_READ_REQ,sum) descr="Total write requests with RW mtype from this TCP to all TCCs. Sum over TCP instances."></metric>
+  <metric name="TCP_TCC_RW_WRITE_REQ_sum"                               expr=reduce(TCP_TCC_RW_WRITE_REQ,sum) descr="Total write requests with RW mtype from this TCP to all TCCs. Sum over TCP instances."></metric>
+  <metric name="TCP_TCC_RW_ATOMIC_REQ_sum"                              expr=reduce(TCP_TCC_RW_ATOMIC_REQ,sum) descr="Total atomic requests with RW mtype from this TCP to all TCCs. Sum over TCP instances."></metric>
+  <metric name="TCP_PENDING_STALL_CYCLES_sum"                           expr=reduce(TCP_PENDING_STALL_CYCLES,sum) descr="Stall due to data pending from L2. Sum over TCP instances."></metric>
+  <metric name="TCA_CYCLE_sum"                                          expr=reduce(TCA_CYCLE,sum)  descr="Number of cycles. Sum over all TCA instances "></metric>
+  <metric name="TCA_BUSY_sum"                                           expr=reduce(TCA_BUSY,sum)   descr="Number of cycles we have a request pending. Sum over all TCA instances."></metric>
+  <metric name="TCC_BUSY_avr"                                           expr=reduce(TCC_BUSY,average) descr="TCC_BUSY avr over all memory channels."></metric>
+  <metric name="TCC_WRREQ_STALL_max"                                    expr=reduce(TCC_EA0_WRREQ_STALL,max) descr="Number of cycles a write request was stalled. Max over TCC instances."></metric>
+  <metric name="TCC_CYCLE_sum"                                          expr=reduce(TCC_CYCLE,sum) descr="Number of cycles. Not windowable. Sum over TCC instances."></metric>
+  <metric name="TCC_BUSY_sum"                                           expr=reduce(TCC_BUSY,sum) descr="Number of cycles we have a request pending. Not windowable. Sum over TCC instances."></metric>
+  <metric name="TCC_REQ_sum"                                            expr=reduce(TCC_REQ,sum) descr="Number of requests of all types. This is measured at the tag block. This may be more than the number of requests arriving at the TCC, but it is a good indication of the total amount of work that needs to be performed. Sum over TCC instances."></metric>
+  <metric name="TCC_STREAMING_REQ_sum"                                  expr=reduce(TCC_STREAMING_REQ,sum) descr="Number of streaming requests. This is measured at the tag block. Sum over TCC instances."></metric>
+  <metric name="TCC_NC_REQ_sum"                                         expr=reduce(TCC_NC_REQ,sum) descr="The number of noncoherently cached requests. This is measured at the tag block. Sum over TCC instances."></metric>
+  <metric name="TCC_UC_REQ_sum"                                         expr=reduce(TCC_UC_REQ,sum) descr="The number of uncached requests. This is measured at the tag block. Sum over TCC instances."></metric>
+  <metric name="TCC_CC_REQ_sum"                                         expr=reduce(TCC_CC_REQ,sum) descr="The number of coherently cached requests. This is measured at the tag block. Sum over TCC instances."></metric>
+  <metric name="TCC_RW_REQ_sum"                                         expr=reduce(TCC_RW_REQ,sum) descr="The number of RW requests. This is measured at the tag block. Sum over TCC instances."></metric>
+  <metric name="TCC_PROBE_sum"                                          expr=reduce(TCC_PROBE,sum) descr="Number of probe requests. Not windowable. Sum over TCC instances."></metric>
+  <metric name="TCC_PROBE_ALL_sum"                                      expr=reduce(TCC_PROBE_ALL,sum) descr="Number of external probe requests with with EA_TCC_preq_all== 1. Not windowable. Sum over TCC instances."></metric>
+  <metric name="TCC_READ_sum"                                           expr=reduce(TCC_READ,sum) descr="Number of read requests. Compressed reads are included in this, but metadata reads are not included. Sum over TCC instances."></metric>
+  <metric name="TCC_WRITE_sum"                                          expr=reduce(TCC_WRITE,sum) descr="Number of write requests. Sum over TCC instances."></metric>
+  <metric name="TCC_ATOMIC_sum"                                         expr=reduce(TCC_ATOMIC,sum) descr="Number of atomic requests of all types. Sum over TCC instances."></metric>
+  <metric name="TCC_HIT_sum"                                            expr=reduce(TCC_HIT,sum) descr="Number of cache hits. Sum over TCC instances."></metric>
+  <metric name="TCC_MISS_sum"                                           expr=reduce(TCC_MISS,sum) descr="Number of cache misses. UC reads count as misses. Sum over TCC instances."></metric>
+  <metric name="TCC_WRITEBACK_sum"                                      expr=reduce(TCC_WRITEBACK,sum) descr="Number of lines written back to main memory. This includes writebacks of dirty lines and uncached write/atomic requests. Sum over TCC instances."></metric>
+  <metric name="TCC_EA0_WRREQ_sum"                                       expr=reduce(TCC_EA0_WRREQ,sum) descr="Number of transactions (either 32-byte or 64-byte) going over the TC_EA_wrreq interface. Atomics may travel over the same interface and are generally classified as write requests. This does not include probe commands. Sum over TCC instances."></metric>
+  <metric name="TCC_EA0_WRREQ_64B_sum"                                   expr=reduce(TCC_EA0_WRREQ_64B,sum) descr="Number of 64-byte transactions going (64-byte write or CMPSWAP) over the TC_EA_wrreq interface. Sum over TCC instances."></metric>
+  <metric name="TCC_EA0_WR_UNCACHED_32B_sum"                             expr=reduce(TCC_EA0_WR_UNCACHED_32B,sum) descr="Number of 32-byte write/atomic going over the TC_EA_wrreq interface due to uncached traffic. Note that CC mtypes can produce uncached requests, and those are included in this. A 64-byte request will be counted as 2. Sum over TCC instances."></metric>
+  <metric name="TCC_EA0_WRREQ_STALL_sum"                                 expr=reduce(TCC_EA0_WRREQ_STALL,sum) descr="Number of cycles a write request was stalled. Sum over TCC instances."></metric>
+  <metric name="TCC_EA0_WRREQ_IO_CREDIT_STALL_sum"                       expr=reduce(TCC_EA0_WRREQ_IO_CREDIT_STALL,sum) descr="Number of cycles a EA write request was stalled because the interface was out of IO credits. Sum over TCC instances."></metric>
+  <metric name="TCC_EA0_WRREQ_GMI_CREDIT_STALL_sum"                      expr=reduce(TCC_EA0_WRREQ_GMI_CREDIT_STALL,sum) descr="Number of cycles a EA write request was stalled because the interface was out of GMI credits. Sum over TCC instances."></metric>
+  <metric name="TCC_EA0_WRREQ_DRAM_CREDIT_STALL_sum"                     expr=reduce(TCC_EA0_WRREQ_DRAM_CREDIT_STALL,sum) descr="Number of cycles a EA write request was stalled because the interface was out of DRAM credits. Sum over TCC instances."></metric>
+  <metric name="TCC_TOO_MANY_EA_WRREQS_STALL_sum"                       expr=reduce(TCC_TOO_MANY_EA_WRREQS_STALL,sum) descr="Number of cycles the TCC could not send a EA write request because it already reached its maximum number of pending EA write requests. Sum over TCC instances."></metric>
+  <metric name="TCC_EA0_WRREQ_LEVEL_sum"                                 expr=reduce(TCC_EA0_WRREQ_LEVEL,sum) descr="The sum of the number of EA write requests in flight. This is primarily meant for measure average EA write latency. Average write latency = TCC_PERF_SEL_EA_WRREQ_LEVEL/TCC_PERF_SEL_EA_WRREQ. Sum over TCC instances."></metric>
+  <metric name="TCC_EA0_RDREQ_LEVEL_sum"                                 expr=reduce(TCC_EA0_RDREQ_LEVEL,sum) descr="The sum of the number of TCC/EA read requests in flight. This is primarily meant for measure average EA read latency. Average read latency = TCC_PERF_SEL_EA_RDREQ_LEVEL/TCC_PERF_SEL_EA_RDREQ. Sum over TCC instances."></metric>
+  <metric name="TCC_EA0_ATOMIC_sum"                                      expr=reduce(TCC_EA0_ATOMIC,sum) descr="Number of transactions going over the TC_EA_wrreq interface that are actually atomic requests. Sum over TCC instances."></metric>
+  <metric name="TCC_EA0_ATOMIC_LEVEL_sum"                                expr=reduce(TCC_EA0_ATOMIC_LEVEL,sum) descr="The sum of the number of EA atomics in flight. This is primarily meant for measure average EA atomic latency. Average atomic latency = TCC_PERF_SEL_EA_WRREQ_ATOMIC_LEVEL/TCC_PERF_SEL_EA_WRREQ_ATOMIC. Sum over TCC instances."></metric>
+  <metric name="TCC_EA0_RDREQ_sum"                                       expr=reduce(TCC_EA0_RDREQ,sum) descr="Number of TCC/EA read requests (either 32-byte or 64-byte) Sum over TCC instances."></metric>
+  <metric name="TCC_EA0_RDREQ_32B_sum"                                   expr=reduce(TCC_EA0_RDREQ_32B,sum) descr="Number of 32-byte TCC/EA read requests Sum over TCC instances."></metric>
+  <metric name="TCC_EA0_RD_UNCACHED_32B_sum"                             expr=reduce(TCC_EA0_RD_UNCACHED_32B,sum) descr="Number of 32-byte TCC/EA read due to uncached traffic. A 64-byte request will be counted as 2 Sum over TCC instances."></metric>
+  <metric name="TCC_EA0_RDREQ_IO_CREDIT_STALL_sum"                       expr=reduce(TCC_EA0_RDREQ_IO_CREDIT_STALL,sum) descr="Number of cycles there was a stall because the read request interface was out of IO credits. Stalls occur regardless of whether a read needed to be performed or not. Sum over TCC instances."></metric>
+  <metric name="TCC_EA0_RDREQ_GMI_CREDIT_STALL_sum"                      expr=reduce(TCC_EA0_RDREQ_GMI_CREDIT_STALL,sum) descr="Number of cycles there was a stall because the read request interface was out of GMI credits. Stalls occur regardless of whether a read needed to be performed or not. Sum over TCC instances."></metric>
+  <metric name="TCC_EA0_RDREQ_DRAM_CREDIT_STALL_sum"                     expr=reduce(TCC_EA0_RDREQ_DRAM_CREDIT_STALL,sum) descr="Number of cycles there was a stall because the read request interface was out of DRAM credits. Stalls occur regardless of whether a read needed to be performed or not. Sum over TCC instances."></metric>
+  <metric name="TCC_TAG_STALL_sum"                                      expr=reduce(TCC_TAG_STALL,sum) descr="."></metric>
+  <metric name="TCC_NORMAL_WRITEBACK_sum"                               expr=reduce(TCC_NORMAL_WRITEBACK,sum) descr="Number of writebacks due to requests that are not writeback requests. Sum over TCC instances."></metric>
+  <metric name="TCC_ALL_TC_OP_WB_WRITEBACK_sum"                         expr=reduce(TCC_ALL_TC_OP_WB_WRITEBACK,sum) descr="Number of writebacks due to all TC_OP writeback requests. Sum over TCC instances."></metric>
+  <metric name="TCC_NORMAL_EVICT_sum"                                   expr=reduce(TCC_NORMAL_EVICT,sum) descr="Number of evictions due to requests that are not invalidate or probe requests. Sum over TCC instances."></metric>
+  <metric name="TCC_ALL_TC_OP_INV_EVICT_sum"                            expr=reduce(TCC_ALL_TC_OP_INV_EVICT,sum) descr="Number of evictions due to all TC_OP invalidate requests. Sum over TCC instances."></metric>
+  <metric name="TCC_EA0_RDREQ_DRAM_sum"                                  expr=reduce(TCC_EA0_RDREQ_DRAM,sum) descr="Number of TCC/EA read requests (either 32-byte or 64-byte) destined for DRAM (MC). Sum over TCC instances."></metric>
+  <metric name="TCC_EA0_WRREQ_DRAM_sum"                                  expr=reduce(TCC_EA0_WRREQ_DRAM,sum) descr="Number of TCC/EA write requests (either 32-byte of 64-byte) destined for DRAM (MC). Sum over TCC instances."></metric>
  <metric name="FETCH_SIZE"                                             expr=(TCC_EA0_RDREQ_32B_sum*32+(TCC_EA0_RDREQ_sum-TCC_EA0_RDREQ_32B_sum)*64)/1024 descr="The total kilobytes fetched from the video memory. This is measured with all extra fetches and any cache or memory effects taken into account."></metric>
  <metric name="WRITE_SIZE"                                             expr=((TCC_EA0_WRREQ_sum-TCC_EA0_WRREQ_64B_sum)*32+TCC_EA0_WRREQ_64B_sum*64)/1024 descr="The total kilobytes written to the video memory. This is measured with all extra fetches and any cache or memory effects taken into account."></metric>
  <metric name="WRITE_REQ_32B"                                          expr=TCC_EA0_WRREQ_64B_sum*2+(TCC_EA0_WRREQ_sum-TCC_EA0_WRREQ_64B_sum) descr="The total number of 32-byte effective memory writes."></metric>
@@ -423,10 +421,9 @@
  <metric name="TOTAL_16_OPS"                                           expr=(SQ_INSTS_VALU_FMA_F16*2+SQ_INSTS_VALU_ADD_F16+SQ_INSTS_VALU_MUL_F16+SQ_INSTS_VALU_TRANS_F16)*64+((SQ_INSTS_VALU_MFMA_MOPS_F16+SQ_INSTS_VALU_MFMA_MOPS_BF16)*512) descr="The number of 16 bits OPS executed"></metric>
  <metric name="TOTAL_32_OPS"                                           expr=(SQ_INSTS_VALU_FMA_F32*2+SQ_INSTS_VALU_INT32+SQ_INSTS_VALU_ADD_F32+SQ_INSTS_VALU_MUL_F32+SQ_INSTS_VALU_TRANS_F32)*64+(SQ_INSTS_VALU_MFMA_MOPS_F32*512) descr="The number of 32 bits OPS executed"></metric>
  <metric name="TOTAL_64_OPS"                                           expr=(SQ_INSTS_VALU_FMA_F64*2+SQ_INSTS_VALU_INT64+SQ_INSTS_VALU_ADD_F64+SQ_INSTS_VALU_MUL_F64)*64+(SQ_INSTS_VALU_MFMA_MOPS_F64*512) descr="The number of 64 bits OPS executed"></metric>
-</gfx940_expr>
+</gfx940>

-<gfx10_expr>
-  <metric name="KERNEL_DURATION" expr=1 descr="The duration of the kernel dispatch"></metric>
+<gfx10>
  <metric name="MeanOccupancyPerCU" expr=GRBM_COUNT*0+SQ_LEVEL_WAVES*0+SQ_ACCUM_PREV/GRBM_GUI_ACTIVE/CU_NUM descr="Mean occupancy per compute unit."></metric>
  <metric name="MeanOccupancyPerActiveCU" expr=GRBM_COUNT*0+SQ_LEVEL_WAVES*0+SQ_ACCUM_PREV*4/SQ_BUSY_CYCLES/CU_NUM descr="Mean occupancy per active compute unit."></metric>

@@ -439,90 +436,89 @@
  <metric name="WAVE_DEP_WAIT" expr=100*SQ_WAIT_ANY/SQ_WAVE_CYCLES descr="Percentage of the SQ_WAVE_CYCLE time spent waiting for anything."></metric>
  <metric name="WAVE_ISSUE_WAIT" expr=100*SQ_WAIT_INST_ANY/SQ_WAVE_CYCLES descr="Percentage of the SQ_WAVE_CYCLE time spent waiting for any instruction issue."></metric>

-  <metric name="TA_BUSY_avr" expr=avr(TA_TA_BUSY,16) descr="TA block is busy. Average over TA instances."></metric>
-  <metric name="TA_BUSY_max" expr=max(TA_TA_BUSY,16) descr="TA block is busy. Max over TA instances."></metric>
-  <metric name="TA_BUSY_min" expr=min(TA_TA_BUSY,16) descr="TA block is busy. Min over TA instances."></metric>
-  <metric name="TA_FLAT_LOAD_WAVEFRONTS_sum" expr=sum(TA_FLAT_LOAD_WAVEFRONTS,16) descr="Number of flat load vec32 packets processed by the TA. Sum over TA instances."></metric>
-  <metric name="TA_FLAT_STORE_WAVEFRONTS_sum" expr=sum(TA_FLAT_STORE_WAVEFRONTS,16) descr="Number of flat store vec32 packets processed by the TA. Sum over TA instances."></metric>
+  <metric name="TA_BUSY_avr" expr=reduce(TA_TA_BUSY,average) descr="TA block is busy. Average over TA instances."></metric>
+  <metric name="TA_BUSY_max" expr=reduce(TA_TA_BUSY,max) descr="TA block is busy. Max over TA instances."></metric>
+  <metric name="TA_BUSY_min" expr=reduce(TA_TA_BUSY,min) descr="TA block is busy. Min over TA instances."></metric>
+  <metric name="TA_FLAT_LOAD_WAVEFRONTS_sum" expr=reduce(TA_FLAT_LOAD_WAVEFRONTS,sum) descr="Number of flat load vec32 packets processed by the TA. Sum over TA instances."></metric>
+  <metric name="TA_FLAT_STORE_WAVEFRONTS_sum" expr=reduce(TA_FLAT_STORE_WAVEFRONTS,sum) descr="Number of flat store vec32 packets processed by the TA. Sum over TA instances."></metric>

-  <metric name="GL2C_HIT_sum" expr=sum(GL2C_HIT,16) descr="Number of cache hits. Sum over GL2C instances."></metric>
-  <metric name="GL2C_MISS_sum" expr=sum(GL2C_MISS,16) descr="Number of cache misses. Sum over GL2C instances."></metric>
-  <metric name="GL2C_EA_RDREQ_32B_sum" expr=sum(GL2C_EA_RDREQ_32B,16) descr="Number of 32-byte GL2C/EA read requests. Sum over GL2C instances."></metric>
-  <metric name="GL2C_EA_RDREQ_64B_sum" expr=sum(GL2C_EA_RDREQ_64B,16) descr="Number of 64-byte GL2C/EA read requests. Sum over GL2C instances."></metric>
-  <metric name="GL2C_EA_RDREQ_96B_sum" expr=sum(GL2C_EA_RDREQ_96B,16) descr="Number of 96-byte GL2C/EA read requests. Sum over GL2C instances."></metric>
-  <metric name="GL2C_EA_RDREQ_128B_sum" expr=sum(GL2C_EA_RDREQ_128B,16) descr="Number of 128-byte GL2C/EA read requests. Sum over GL2C instances."></metric>
-  <metric name="GL2C_MC_RDREQ_sum" expr=sum(GL2C_MC_RDREQ,16) descr="Number of GL2C/EA read requests (either 32-byte or 64-byte or 128-byte). Sum over GL2C instances."></metric>
-  <metric name="GL2C_MC_WRREQ_sum" expr=sum(GL2C_MC_WRREQ,16) descr="Number of transactions (either 32-byte or 64-byte) going over the GL2C_MC_wrreq interface. Sum over GL2C instances."></metric>
-  <metric name="GL2C_EA_WRREQ_64B_sum" expr=sum(GL2C_EA_WRREQ_64B,16) descr="Number of 64-byte transactions going (64-byte write or CMPSWAP) over the GL2C_EA_wrreq interface. Sum over GL2C instances."></metric>
-  <metric name="GL2C_WRREQ_STALL_max" expr=max(GL2C_MC_WRREQ_STALL,16) descr="Number of cycles a write request was stalled. Max over GL2C instances."></metric>
-  <metric name="L2CacheHit" expr=100*sum(GL2C_HIT,16)/(sum(GL2C_HIT,16)+sum(GL2C_MISS,16)) descr="The percentage of fetch, write, atomic, and other instructions that hit the data in L2 cache. Value range: 0% (no hit) to 100% (optimal)."></metric>
+  <metric name="GL2C_HIT_sum" expr=reduce(GL2C_HIT,sum) descr="Number of cache hits. Sum over GL2C instances."></metric>
+  <metric name="GL2C_MISS_sum" expr=reduce(GL2C_MISS,sum) descr="Number of cache misses. Sum over GL2C instances."></metric>
+  <metric name="GL2C_EA_RDREQ_32B_sum" expr=reduce(GL2C_EA_RDREQ_32B,sum) descr="Number of 32-byte GL2C/EA read requests. Sum over GL2C instances."></metric>
+  <metric name="GL2C_EA_RDREQ_64B_sum" expr=reduce(GL2C_EA_RDREQ_64B,sum) descr="Number of 64-byte GL2C/EA read requests. Sum over GL2C instances."></metric>
+  <metric name="GL2C_EA_RDREQ_96B_sum" expr=reduce(GL2C_EA_RDREQ_96B,sum) descr="Number of 96-byte GL2C/EA read requests. Sum over GL2C instances."></metric>
+  <metric name="GL2C_EA_RDREQ_128B_sum" expr=reduce(GL2C_EA_RDREQ_128B,sum) descr="Number of 128-byte GL2C/EA read requests. Sum over GL2C instances."></metric>
+  <metric name="GL2C_MC_RDREQ_sum" expr=reduce(GL2C_MC_RDREQ,sum) descr="Number of GL2C/EA read requests (either 32-byte or 64-byte or 128-byte). Sum over GL2C instances."></metric>
+  <metric name="GL2C_MC_WRREQ_sum" expr=reduce(GL2C_MC_WRREQ,sum) descr="Number of transactions (either 32-byte or 64-byte) going over the GL2C_MC_wrreq interface. Sum over GL2C instances."></metric>
+  <metric name="GL2C_EA_WRREQ_64B_sum" expr=reduce(GL2C_EA_WRREQ_64B,sum) descr="Number of 64-byte transactions going (64-byte write or CMPSWAP) over the GL2C_EA_wrreq interface. Sum over GL2C instances."></metric>
+  <metric name="GL2C_WRREQ_STALL_max" expr=reduce(GL2C_MC_WRREQ_STALL,max) descr="Number of cycles a write request was stalled. Max over GL2C instances."></metric>
+  <metric name="L2CacheHit" expr=100*reduce(GL2C_HIT,sum)/(reduce(GL2C_HIT,sum)+reduce(GL2C_MISS,sum)) descr="The percentage of fetch, write, atomic, and other instructions that hit the data in L2 cache. Value range: 0% (no hit) to 100% (optimal)."></metric>
  <metric name="FETCH_SIZE" expr=(GL2C_EA_RDREQ_32B_sum*32+GL2C_EA_RDREQ_64B_sum*64+GL2C_EA_RDREQ_96B_sum*96+GL2C_EA_RDREQ_128B_sum*128)/1024 descr="The total kilobytes fetched from the video memory. This is measured with all extra fetches and any cache or memory effects taken into account."></metric>
  <metric name="WriteUnitStalled" expr=100*GL2C_WRREQ_STALL_max/GRBM_GUI_ACTIVE descr="The percentage of GPUTime the Write unit is stalled. Value range: 0% to 100% (bad)."></metric>
  <metric name="LDSBankConflict" expr=100*SQC_LDS_BANK_CONFLICT/SQC_LDS_IDX_ACTIVE descr="The percentage of GPUTime LDS is stalled by bank conflicts. Value range: 0% (optimal) to 100% (bad)."></metric>
-</gfx10_expr>
+</gfx10>

-<gfx1030_expr base="gfx10_expr">
-</gfx1030_expr>
+<gfx1030 base="gfx10">
+</gfx1030>

-<gfx1031_expr base="gfx10_expr">
-</gfx1031_expr>
+<gfx1031 base="gfx10">
+</gfx1031>

-<gfx1032_expr base="gfx10_expr">
-</gfx1032_expr>
+<gfx1032 base="gfx10">
+</gfx1032>

-<gfx11_expr>
-  <metric name="KERNEL_DURATION" expr=1 descr="The duration of the kernel dispatch"></metric>
+<gfx11>
  <metric name="GPU_UTIL" expr=100*GRBM_GUI_ACTIVE/GRBM_COUNT descr="Percentage of the time that GUI is active"></metric>
  <metric name="WAVE_DEP_WAIT" expr=100*SQ_WAIT_ANY/SQ_WAVE_CYCLES descr="Percentage of the SQ_WAVE_CYCLE time spent waiting for anything."></metric>
  <metric name="WAVE_ISSUE_WAIT" expr=100*SQ_WAIT_INST_ANY/SQ_WAVE_CYCLES descr="Percentage of the SQ_WAVE_CYCLE time spent waiting for any instruction issue."></metric>
-  <metric name="TA_BUSY_avr" expr=avr(TA_TA_BUSY,16) descr="TA block is busy. Average over TA instances."></metric>
-  <metric name="TA_BUSY_max" expr=max(TA_TA_BUSY,16) descr="TA block is busy. Max over TA instances."></metric>
-  <metric name="TA_BUSY_min" expr=min(TA_TA_BUSY,16) descr="TA block is busy. Min over TA instances."></metric>
-  <metric name="TA_BUFFER_LOAD_WAVEFRONTS_sum" expr=sum(TA_BUFFER_LOAD_WAVEFRONTS,16) descr="Number of buffer load vec32 packets processed by the TA. Sum over TA instances."></metric>
-  <metric name="TA_BUFFER_STORE_WAVEFRONTS_sum" expr=sum(TA_BUFFER_STORE_WAVEFRONTS,16) descr="Number of buffer store vec32 packets processed by the TA. Sum over TA instances."></metric>
-  <metric name="GL2C_HIT_sum" expr=sum(GL2C_HIT,16) descr="Number of cache hits. Sum over GL2C instances."></metric>
-  <metric name="GL2C_MISS_sum" expr=sum(GL2C_MISS,16) descr="Number of cache misses. Sum over GL2C instances."></metric>
-  <metric name="GL2C_EA_RDREQ_32B_sum" expr=sum(GL2C_EA_RDREQ_32B,16) descr="Number of 32-byte GL2C/EA read requests. Sum over GL2C instances."></metric>
-  <metric name="GL2C_EA_RDREQ_64B_sum" expr=sum(GL2C_EA_RDREQ_64B,16) descr="Number of 64-byte GL2C/EA read requests. Sum over GL2C instances."></metric>
-  <metric name="GL2C_EA_RDREQ_96B_sum" expr=sum(GL2C_EA_RDREQ_96B,16) descr="Number of 96-byte GL2C/EA read requests. Sum over GL2C instances."></metric>
-  <metric name="GL2C_EA_RDREQ_128B_sum" expr=sum(GL2C_EA_RDREQ_128B,16) descr="Number of 128-byte GL2C/EA read requests. Sum over GL2C instances."></metric>
-  <metric name="GL2C_MC_RDREQ_sum" expr=sum(GL2C_MC_RDREQ,16) descr="Number of GL2C/EA read requests (either 32-byte or 64-byte or 128-byte). Sum over GL2C instances."></metric>
-  <metric name="GL2C_MC_WRREQ_sum" expr=sum(GL2C_MC_WRREQ,16) descr="Number of transactions (either 32-byte or 64-byte) going over the GL2C_MC_wrreq interface. Sum over GL2C instances."></metric>
-  <metric name="GL2C_EA_WRREQ_64B_sum" expr=sum(GL2C_EA_WRREQ_64B,16) descr="Number of 64-byte transactions going (64-byte write or CMPSWAP) over the GL2C_EA_wrreq interface. Sum over GL2C instances."></metric>
-  <metric name="GL2C_WRREQ_STALL_max" expr=max(GL2C_MC_WRREQ_STALL,16) descr="Number of cycles a write request was stalled. Max over GL2C instances."></metric>
-  <metric name="L2CacheHit" expr=100*sum(GL2C_HIT,16)/(sum(GL2C_HIT,16)+sum(GL2C_MISS,16)) descr="The percentage of fetch, write, atomic, and other instructions that hit the data in L2 cache. Value range: 0% (no hit) to 100% (optimal)."></metric>
+  <metric name="TA_BUSY_avr" expr=reduce(TA_TA_BUSY,average) descr="TA block is busy. Average over TA instances."></metric>
+  <metric name="TA_BUSY_max" expr=reduce(TA_TA_BUSY,max) descr="TA block is busy. Max over TA instances."></metric>
+  <metric name="TA_BUSY_min" expr=reduce(TA_TA_BUSY,min) descr="TA block is busy. Min over TA instances."></metric>
+  <metric name="TA_BUFFER_LOAD_WAVEFRONTS_sum" expr=reduce(TA_BUFFER_LOAD_WAVEFRONTS,sum) descr="Number of buffer load vec32 packets processed by the TA. Sum over TA instances."></metric>
+  <metric name="TA_BUFFER_STORE_WAVEFRONTS_sum" expr=reduce(TA_BUFFER_STORE_WAVEFRONTS,sum) descr="Number of buffer store vec32 packets processed by the TA. Sum over TA instances."></metric>
+  <metric name="GL2C_HIT_sum" expr=reduce(GL2C_HIT,sum) descr="Number of cache hits. Sum over GL2C instances."></metric>
+  <metric name="GL2C_MISS_sum" expr=reduce(GL2C_MISS,sum) descr="Number of cache misses. Sum over GL2C instances."></metric>
+  <metric name="GL2C_EA_RDREQ_32B_sum" expr=reduce(GL2C_EA_RDREQ_32B,sum) descr="Number of 32-byte GL2C/EA read requests. Sum over GL2C instances."></metric>
+  <metric name="GL2C_EA_RDREQ_64B_sum" expr=reduce(GL2C_EA_RDREQ_64B,sum) descr="Number of 64-byte GL2C/EA read requests. Sum over GL2C instances."></metric>
+  <metric name="GL2C_EA_RDREQ_96B_sum" expr=reduce(GL2C_EA_RDREQ_96B,sum) descr="Number of 96-byte GL2C/EA read requests. Sum over GL2C instances."></metric>
+  <metric name="GL2C_EA_RDREQ_128B_sum" expr=reduce(GL2C_EA_RDREQ_128B,sum) descr="Number of 128-byte GL2C/EA read requests. Sum over GL2C instances."></metric>
+  <metric name="GL2C_MC_RDREQ_sum" expr=reduce(GL2C_MC_RDREQ,sum) descr="Number of GL2C/EA read requests (either 32-byte or 64-byte or 128-byte). Sum over GL2C instances."></metric>
+  <metric name="GL2C_MC_WRREQ_sum" expr=reduce(GL2C_MC_WRREQ,sum) descr="Number of transactions (either 32-byte or 64-byte) going over the GL2C_MC_wrreq interface. Sum over GL2C instances."></metric>
+  <metric name="GL2C_EA_WRREQ_64B_sum" expr=reduce(GL2C_EA_WRREQ_64B,sum) descr="Number of 64-byte transactions going (64-byte write or CMPSWAP) over the GL2C_EA_wrreq interface. Sum over GL2C instances."></metric>
+  <metric name="GL2C_WRREQ_STALL_max" expr=reduce(GL2C_MC_WRREQ_STALL,max) descr="Number of cycles a write request was stalled. Max over GL2C instances."></metric>
+  <metric name="L2CacheHit" expr=100*reduce(GL2C_HIT,sum)/(reduce(GL2C_HIT,sum)+reduce(GL2C_MISS,sum)) descr="The percentage of fetch, write, atomic, and other instructions that hit the data in L2 cache. Value range: 0% (no hit) to 100% (optimal)."></metric>
  <metric name="FETCH_SIZE" expr=(GL2C_EA_RDREQ_32B_sum*32+GL2C_EA_RDREQ_64B_sum*64+GL2C_EA_RDREQ_96B_sum*96+GL2C_EA_RDREQ_128B_sum*128)/1024 descr="The total kilobytes fetched from the video memory. This is measured with all extra fetches and any cache or memory effects taken into account."></metric>
  <metric name="WriteUnitStalled" expr=100*GL2C_WRREQ_STALL_max/GRBM_GUI_ACTIVE descr="The percentage of GPUTime the Write unit is stalled. Value range: 0% to 100% (bad)."></metric>
  <metric name="LDSBankConflict" expr=100*SQC_LDS_BANK_CONFLICT/SQC_LDS_IDX_ACTIVE descr="The percentage of GPUTime LDS is stalled by bank conflicts. Value range: 0% (optimal) to 100% (bad)."></metric>
-</gfx11_expr>
+</gfx11>

-<gfx1100_expr base="gfx11_expr">
-</gfx1100_expr>
+<gfx1100 base="gfx11">
+</gfx1100>

-<gfx1101_expr base="gfx11_expr">
-</gfx1101_expr>
+<gfx1101 base="gfx11">
+</gfx1101>

-<gfx8 base="gfx8_expr"></gfx8>
-<gfx9 base="gfx9_expr"></gfx9>
-<gfx10 base="gfx10_expr"></gfx10>
-<gfx11 base="gfx11_expr"></gfx11>
+<gfx8 base="gfx8"></gfx8>
+<gfx9 base="gfx9"></gfx9>
+<gfx10 base="gfx10"></gfx10>
+<gfx11 base="gfx11"></gfx11>
 # Vega20
-<gfx906 base="gfx906_expr"></gfx906>
+<gfx906 base="gfx906"></gfx906>
 # Arcturus
-<gfx908 base="gfx908_expr"></gfx908>
+<gfx908 base="gfx908"></gfx908>
 # Aldebaran
-<gfx90a base="gfx90a_expr"></gfx90a>
+<gfx90a base="gfx90a"></gfx90a>
 #Mi300
-<gfx940 base="gfx940_expr"></gfx940>
-<gfx941 base="gfx940_expr"></gfx941>
-<gfx942 base="gfx940_expr"></gfx942>
+<gfx940 base="gfx940"></gfx940>
+<gfx941 base="gfx940"></gfx941>
+<gfx942 base="gfx940"></gfx942>
 #Navi21
-<gfx1030 base="gfx1030_expr"></gfx1030>
-<gfx1031 base="gfx1031_expr"></gfx1031>
-<gfx1032 base="gfx1032_expr"></gfx1032>
+<gfx1030 base="gfx1030"></gfx1030>
+<gfx1031 base="gfx1031"></gfx1031>
+<gfx1032 base="gfx1032"></gfx1032>
 #Navi31
-<gfx1100 base="gfx1100_expr"></gfx1100>
-<gfx1101 base="gfx1101_expr"></gfx1101>
+<gfx1100 base="gfx1100"></gfx1100>
+<gfx1101 base="gfx1101"></gfx1101>


 <global>
@@ -572,7 +568,7 @@
  <metric
    name="MemUnitBusy"
    descr="The percentage of GPUTime the memory unit is active. The result includes the stall time (MemUnitStalled). This is measured with all extra fetches and writes and any cache or memory effects taken into account. Value range: 0% to 100% (fetch-bound)."
-    expr=100*max(TA_TA_BUSY,16)/GRBM_GUI_ACTIVE/SE_NUM
+    expr=100*reduce(TA_TA_BUSY,max)/GRBM_GUI_ACTIVE/SE_NUM
  ></metric>

  # ALUStalledByLDS The percentage of GPUTime ALU units are stalled by the LDS input queue being full or the output queue being not ready. If there are LDS bank conflicts, reduce them. Otherwise, try reducing the number of LDS accesses if possible. Value range: 0% (optimal) to 100% (bad).
@@ -0,0 +1,32 @@
+#include <rocprofiler/rocprofiler.h>
+
+#include "lib/rocprofiler/aql/helpers.hpp"
+#include "lib/rocprofiler/counters/core.hpp"
+#include "lib/rocprofiler/counters/evaluate_ast.hpp"
+#include "lib/rocprofiler/counters/metrics.hpp"
+#include "lib/rocprofiler/hsa/agent_cache.hpp"
+
+extern "C" {
+/**
+ * @brief Configure Dispatch Profile Counting Service.
+ *
+ * @param [in] context_id
+ * @param [in] agent_id
+ * @param [in] buffer_id
+ * @param [in] callback
+ * @param [in] callback_data_args
+ * @return ::rocprofiler_status_t
+ */
+rocprofiler_status_t ROCPROFILER_API
+rocprofiler_configure_dispatch_profile_counting_service(
+    rocprofiler_context_id_t                         context_id,
+    rocprofiler_profile_config_id_t                  profile,
+    rocprofiler_profile_counting_dispatch_callback_t callback,
+    void*                                            callback_data_args)
+{
+    return rocprofiler::counters::configure_dispatch(
+               context_id, profile.handle, callback, callback_data_args)
+               ? ROCPROFILER_STATUS_SUCCESS
+               : ROCPROFILER_STATUS_ERROR;
+}
+}
@@ -1,5 +1,6 @@
-set(ROCPROFILER_LIB_HSA_SOURCES hsa.cpp)
-set(ROCPROFILER_LIB_HSA_HEADERS hsa.hpp defines.hpp types.hpp utils.hpp)
+set(ROCPROFILER_LIB_HSA_SOURCES hsa.cpp queue.cpp agent_cache.cpp)
+set(ROCPROFILER_LIB_HSA_HEADERS hsa.hpp defines.hpp types.hpp utils.hpp queue.hpp
+                                agent_cache.hpp)
 target_sources(rocprofiler-object-library PRIVATE ${ROCPROFILER_LIB_HSA_SOURCES}
                                                  ${ROCPROFILER_LIB_HSA_HEADERS})

@@ -0,0 +1,196 @@
+// Copyright (c) 2018-2023 Advanced Micro Devices, Inc.
+//
+// Permission is hereby granted, free of charge, to any person obtaining a copy
+// of this software and associated documentation files (the "Software"), to deal
+// in the Software without restriction, including without limitation the rights
+// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+// copies of the Software, and to permit persons to whom the Software is
+// furnished to do so, subject to the following conditions:
+//
+// The above copyright notice and this permission notice shall be included in
+// all copies or substantial portions of the Software.
+//
+// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL THE
+// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+// THE SOFTWARE.
+
+#include "agent_cache.hpp"
+
+#include <glog/logging.h>
+#include <filesystem>
+#include <fstream>
+#include <optional>
+
+#include "lib/common/synchronized.hpp"
+#include "lib/common/utility.hpp"
+
+namespace
+{
+// This function checks to see if the provided
+// pool has the HSA_AMD_SEGMENT_GLOBAL property. If the kern_arg flag is true,
+// the function adds an additional requirement that the pool have the
+// HSA_AMD_MEMORY_POOL_GLOBAL_FLAG_KERNARG_INIT property. If kern_arg is false,
+// pools must NOT have this property.
+// Upon finding a pool that meets these conditions, HSA_STATUS_INFO_BREAK is
+// returned. HSA_STATUS_SUCCESS is returned if no errors were encountered, but
+// no pool was found meeting the requirements. If an error is encountered, we
+// return that error.
+hsa_status_t
+FindGlobalPool(hsa_amd_memory_pool_t pool, void* data, bool kern_arg)
+{
+    if(!data) return HSA_STATUS_ERROR_INVALID_ARGUMENT;
+
+    auto [api_ptr, pool_ptr] =
+        *static_cast<std::pair<const AmdExtTable*, hsa_amd_memory_pool_t*>*>(data);
+    hsa_amd_segment_t segment;
+    LOG_IF(FATAL,
+           api_ptr->hsa_amd_memory_pool_get_info_fn(
+               pool, HSA_AMD_MEMORY_POOL_INFO_SEGMENT, &segment) == HSA_STATUS_ERROR)
+        << "Could not get pool segment";
+    if(HSA_AMD_SEGMENT_GLOBAL != segment) return HSA_STATUS_SUCCESS;
+
+    uint32_t flag;
+    LOG_IF(FATAL,
+           api_ptr->hsa_amd_memory_pool_get_info_fn(
+               pool, HSA_AMD_MEMORY_POOL_INFO_GLOBAL_FLAGS, &flag) == HSA_STATUS_ERROR)
+        << "Could not get flag value";
+    uint32_t karg_st = flag & HSA_AMD_MEMORY_POOL_GLOBAL_FLAG_KERNARG_INIT;
+    if((karg_st == 0 && kern_arg) || (karg_st != 0 && !kern_arg))
+    {
+        return HSA_STATUS_SUCCESS;
+    }
+    *(pool_ptr) = pool;
+    return HSA_STATUS_INFO_BREAK;
+}
+
+// This is the call-back function for hsa_amd_agent_iterate_memory_pools() that
+// finds a pool with the properties of HSA_AMD_SEGMENT_GLOBAL and that is NOT
+// HSA_AMD_MEMORY_POOL_GLOBAL_FLAG_KERNARG_INIT
+hsa_status_t
+FindStandardPool(hsa_amd_memory_pool_t pool, void* data)
+{
+    return FindGlobalPool(pool, data, false);
+}
+
+// This is the call-back function for hsa_amd_agent_iterate_memory_pools() that
+// finds a pool with the properties of HSA_AMD_SEGMENT_GLOBAL and that IS
+// HSA_AMD_MEMORY_POOL_GLOBAL_FLAG_KERNARG_INIT
+hsa_status_t
+FindKernArgPool(hsa_amd_memory_pool_t pool, void* data)
+{
+    return FindGlobalPool(pool, data, true);
+}
+
+void
+init_cpu_pool(const AmdExtTable& api, rocprofiler::hsa::AgentCache& agent)
+{
+    std::pair<const AmdExtTable*, hsa_amd_memory_pool_t*> params =
+        std::make_pair(&api, &agent.cpu_pool());
+
+    auto status =
+        api.hsa_amd_agent_iterate_memory_pools_fn(agent.near_cpu(), FindStandardPool, &params);
+    LOG_IF(FATAL, status != HSA_STATUS_SUCCESS && status != HSA_STATUS_INFO_BREAK)
+        << "Error: Command Buffer Pool is not initialized";
+
+    params.second = &agent.kernarg_pool();
+    status =
+        api.hsa_amd_agent_iterate_memory_pools_fn(agent.near_cpu(), FindKernArgPool, &(params));
+    LOG_IF(FATAL, status != HSA_STATUS_SUCCESS && status != HSA_STATUS_INFO_BREAK)
+        << "Error: Output Buffer Pool is not initialized";
+}
+
+void
+init_gpu_pool(const AmdExtTable& api, rocprofiler::hsa::AgentCache& agent)
+{
+    std::pair<const AmdExtTable*, hsa_amd_memory_pool_t*> params =
+        std::make_pair(&api, &agent.gpu_pool());
+    auto status =
+        api.hsa_amd_agent_iterate_memory_pools_fn(agent.get_agent(), FindStandardPool, &params);
+
+    LOG_IF(FATAL, status != HSA_STATUS_SUCCESS && status != HSA_STATUS_INFO_BREAK)
+        << "Error: GPU Pool is not initialized";
+}
+
+}  // namespace
+
+namespace rocprofiler
+{
+namespace hsa
+{
+AgentCache::AgentCache(rocprofiler_agent_t   agent_t,
+                       size_t                index,
+                       const ::CoreApiTable& table,
+                       const AmdExtTable&    ext)
+: _agent_t(agent_t)
+, _index(index)
+, _name(agent_t.name)
+{
+    // Get HSA Agents
+    std::vector<hsa_agent_t> agents;
+    table.hsa_iterate_agents_fn(
+        [](hsa_agent_t agent, void* data) {
+            CHECK_NOTNULL(static_cast<std::vector<hsa_agent_t>*>(data))->emplace_back(agent);
+            return HSA_STATUS_SUCCESS;
+        },
+        &agents);
+
+    // In case HSA_AMD_AGENT_INFO_NEAREST_CPU is non-functional, default to original v1 behavior
+    // of last CPU agent being nearest.
+    std::optional<hsa_agent_t> last_cpu;
+
+    bool found = false;
+    // Find the HSA agent that is represented by rocprofiler_agent_t
+    for(const auto& agent : agents)
+    {
+        hsa_device_type_t type = HSA_DEVICE_TYPE_CPU;
+        if(table.hsa_agent_get_info_fn(agent, HSA_AGENT_INFO_DEVICE, &type) != HSA_STATUS_SUCCESS)
+        {
+            throw std::runtime_error("hsa_agent_get_info failed to find device");
+        }
+
+        if(type != HSA_DEVICE_TYPE_GPU)
+        {
+            if(type == HSA_DEVICE_TYPE_CPU && !last_cpu) last_cpu = agent;
+            continue;
+        }
+
+        uint32_t node_id = 0;
+        if(table.hsa_agent_get_info_fn(
+               agent, static_cast<hsa_agent_info_t>(HSA_AMD_AGENT_INFO_DRIVER_NODE_ID), &node_id) !=
+           HSA_STATUS_SUCCESS)
+        {
+            throw std::runtime_error("hsa_agent_get_info failed to find driver id");
+        }
+
+        // Match rocprofiler_agent_t to hsa_agent for GPU agents
+        if(_index != node_id) continue;
+
+        if(table.hsa_agent_get_info_fn(
+               agent,
+               static_cast<hsa_agent_info_t>(HSA_AMD_AGENT_INFO_NEAREST_CPU),
+               &_nearest_cpu) != HSA_STATUS_SUCCESS)
+        {
+            if(!last_cpu) throw std::runtime_error("HSA_AMD_AGENT_INFO_NEAREST_CPU failed!");
+            _nearest_cpu = *last_cpu;
+        }
+
+        found  = true;
+        _agent = agent;
+    }
+
+    if(!found)
+    {
+        throw std::runtime_error(fmt::format("Could not find GPU id = {}", agent_t.id.handle));
+    }
+
+    // Construct CPU/GPU pools
+    init_cpu_pool(ext, *this);
+    init_gpu_pool(ext, *this);
+}
+
+}  // namespace hsa
+}  // namespace rocprofiler
@@ -0,0 +1,86 @@
+// Copyright (c) 2018-2023 Advanced Micro Devices, Inc.
+//
+// Permission is hereby granted, free of charge, to any person obtaining a copy
+// of this software and associated documentation files (the "Software"), to deal
+// in the Software without restriction, including without limitation the rights
+// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+// copies of the Software, and to permit persons to whom the Software is
+// furnished to do so, subject to the following conditions:
+//
+// The above copyright notice and this permission notice shall be included in
+// all copies or substantial portions of the Software.
+//
+// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL THE
+// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+// THE SOFTWARE.
+
+#pragma once
+
+#include <hsa/hsa.h>
+#include <hsa/hsa_api_trace.h>
+#include <hsa/hsa_ext_amd.h>
+
+#include "fmt/core.h"
+#include "fmt/ranges.h"
+
+#include <string>
+#include <string_view>
+#include <unordered_map>
+#include <vector>
+
+#include <rocprofiler/agent.h>
+#include "lib/common/utility.hpp"
+
+// Construct const and non-const accessor functions
+#define CONST_NONCONST_ACCESSOR(RTYPE, NAME, VAL)                                                  \
+    const RTYPE& NAME() const { return VAL; }                                                      \
+    RTYPE&       NAME() { return VAL; }
+
+namespace rocprofiler
+{
+namespace hsa
+{
+static const uint32_t LDS_BLOCK_SIZE = 128 * 4;
+
+// Stores per-agent HSA information such as GPU and Kernel pools
+// along with nearest CPU agent and its pool. Links rocprofiler_agent_t
+// to its HSA agent. Note this class is only valid when HSA is
+// init'd
+class AgentCache
+{
+public:
+    AgentCache(rocprofiler_agent_t, size_t index, const ::CoreApiTable&, const AmdExtTable&);
+
+    // Provides const and a non-const accessor functions.
+    CONST_NONCONST_ACCESSOR(hsa_amd_memory_pool_t, cpu_pool, _cpu_pool);
+    CONST_NONCONST_ACCESSOR(hsa_amd_memory_pool_t, kernarg_pool, _kernarg_pool);
+    CONST_NONCONST_ACCESSOR(hsa_amd_memory_pool_t, gpu_pool, _gpu_pool);
+    CONST_NONCONST_ACCESSOR(rocprofiler_agent_t, agent_t, _agent_t);
+    CONST_NONCONST_ACCESSOR(hsa_agent_t, get_agent, _agent);
+    CONST_NONCONST_ACCESSOR(hsa_agent_t, near_cpu, _nearest_cpu);
+
+    const std::string& name() const { return _name; }
+
+private:
+    // Agent info
+    rocprofiler_agent_t _agent_t;
+    size_t              _index{0};  // rocprofiler_agent index
+
+    // GPU Agent
+    hsa_agent_t _agent{.handle = 0};
+    hsa_agent_t _nearest_cpu{.handle = 0};
+
+    // memory pools
+    hsa_amd_memory_pool_t _cpu_pool{.handle = 0};
+    hsa_amd_memory_pool_t _kernarg_pool{.handle = 0};
+    hsa_amd_memory_pool_t _gpu_pool{.handle = 0};
+
+    std::string _name;
+};
+
+}  // namespace hsa
+}  // namespace rocprofiler
@@ -0,0 +1,471 @@
+/* Copyright (c) 2022 Advanced Micro Devices, Inc.
+
+ Permission is hereby granted, free of charge, to any person obtaining a copy
+ of this software and associated documentation files (the "Software"), to deal
+ in the Software without restriction, including without limitation the rights
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+ copies of the Software, and to permit persons to whom the Software is
+ furnished to do so, subject to the following conditions:
+
+ The above copyright notice and this permission notice shall be included in
+ all copies or substantial portions of the Software.
+
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+ THE SOFTWARE. */
+
+#include "lib/rocprofiler/hsa/queue.hpp"
+
+#include <glog/logging.h>
+
+namespace rocprofiler
+{
+namespace hsa
+{
+namespace
+{
+bool
+AsyncSignalHandler(hsa_signal_value_t, void* data)
+{
+    if(!data) return true;
+    auto& queue_info_session = *static_cast<Queue::queue_info_session_t*>(data);
+
+    // Calls our internal callbacks to callers who need to be notified post
+    // kernel execution.
+    queue_info_session.queue.signal_callback([&](const auto& map) {
+        for(const auto& [client_id, cb_pair] : map)
+        {
+            // If this is the client that gave us the AQLPacket,
+            // return it to that client otherwise notify.
+            if(queue_info_session.inst_pkt_id == client_id)
+            {
+                cb_pair.second(queue_info_session.queue,
+                               client_id,
+                               queue_info_session.kernel_pkt,
+                               std::move(queue_info_session.inst_pkt));
+            }
+            else
+            {
+                cb_pair.second(
+                    queue_info_session.queue, client_id, queue_info_session.kernel_pkt, nullptr);
+            }
+        }
+    });
+
+    // Delete signals and packets, signal we have completed.
+    if(queue_info_session.interrupt_signal.handle != 0u)
+        queue_info_session.queue.core_api().hsa_signal_destroy_fn(
+            queue_info_session.interrupt_signal);
+    if(queue_info_session.kernel_pkt.completion_signal.handle != 0u)
+    {
+        queue_info_session.queue.core_api().hsa_signal_destroy_fn(
+            queue_info_session.kernel_pkt.completion_signal);
+    }
+    queue_info_session.queue.async_complete();
+
+    delete static_cast<Queue::queue_info_session_t*>(data);
+    return false;
+}
+
+void
+CreateBarrierPacket(const hsa_signal_t&                        packet_completion_signal,
+                    std::vector<hsa_ext_amd_aql_pm4_packet_t>& transformed_packets)
+{
+    hsa_barrier_and_packet_t barrier{};
+    barrier.header        = HSA_PACKET_TYPE_BARRIER_AND << HSA_PACKET_HEADER_TYPE;
+    barrier.dep_signal[0] = packet_completion_signal;
+    void* barrier_ptr     = &barrier;
+    transformed_packets.emplace_back(*reinterpret_cast<hsa_ext_amd_aql_pm4_packet_t*>(barrier_ptr));
+}
+
+void
+AddVendorSpecificPacket(const hsa_ext_amd_aql_pm4_packet_t&        packet,
+                        std::vector<hsa_ext_amd_aql_pm4_packet_t>& transformed_packets,
+                        const hsa_signal_t&                        packet_completion_signal)
+{
+    transformed_packets.emplace_back(packet).completion_signal = packet_completion_signal;
+}
+}  // namespace
+
+void
+Queue::signal_async_handler(const hsa_signal_t& signal, Queue::queue_info_session_t* data) const
+{
+    hsa_status_t status = _ext_api.hsa_amd_signal_async_handler_fn(
+        signal, HSA_SIGNAL_CONDITION_EQ, 0, AsyncSignalHandler, static_cast<void*>(data));
+    LOG_IF(FATAL, status != HSA_STATUS_SUCCESS && status != HSA_STATUS_INFO_BREAK)
+        << "Error: hsa_amd_signal_async_handler failed";
+}
+
+void
+Queue::create_signal(uint32_t attribute, hsa_signal_t* signal) const
+{
+    hsa_status_t status = _ext_api.hsa_amd_signal_create_fn(1, 0, nullptr, attribute, signal);
+    LOG_IF(FATAL, status != HSA_STATUS_SUCCESS && status != HSA_STATUS_INFO_BREAK)
+        << "Error: hsa_amd_signal_create failed";
+}
+
+template <typename Integral = uint64_t>
+constexpr Integral
+bit_mask(int first, int last)
+{
+    assert(last >= first && "Error: hsa_support::bit_mask -> invalid argument");
+    size_t num_bits = last - first + 1;
+    return ((num_bits >= sizeof(Integral) * 8) ? ~Integral{0}
+                                               /* num_bits exceed the size of Integral */
+                                               : ((Integral{1} << num_bits) - 1))
+           << first;
+}
+
+/* Extract bits [last:first] from t.  */
+template <typename Integral>
+constexpr Integral
+bit_extract(Integral x, int first, int last)
+{
+    return (x >> first) & bit_mask<Integral>(0, last - first);
+}
+
+/**
+ * @brief This function is a queue write interceptor. It intercepts the
+ * packet write function. Creates an instance of packet class with the raw
+ * pointer. invoke the populate function of the packet class which returns a
+ * pointer to the packet. This packet is written into the queue by this
+ * interceptor by invoking the writer function.
+ */
+void
+WriteInterceptor(const void* packets,
+                 uint64_t    pkt_count,
+                 uint64_t,
+                 void*                                 data,
+                 hsa_amd_queue_intercept_packet_writer writer)
+{
+    Queue& queue_info = *static_cast<Queue*>(data);
+
+    // We have no packets or no one who needs to be notified, do nothing.
+    if(pkt_count == 0 || queue_info.get_notifiers() == 0)
+    {
+        writer(packets, pkt_count);
+        return;
+    }
+
+    // hsa_ext_amd_aql_pm4_packet_t
+    const hsa_ext_amd_aql_pm4_packet_t* packets_arr =
+        static_cast<const hsa_ext_amd_aql_pm4_packet_t*>(packets);
+    std::vector<hsa_ext_amd_aql_pm4_packet_t> transformed_packets;
+
+    // Searching accross all the packets given during this write
+    for(size_t i = 0; i < pkt_count; ++i)
+    {
+        const auto& original_packet = static_cast<const hsa_barrier_and_packet_t*>(packets)[i];
+        if(bit_extract(original_packet.header,
+                       HSA_PACKET_HEADER_TYPE,
+                       HSA_PACKET_HEADER_TYPE + HSA_PACKET_HEADER_WIDTH_TYPE - 1) !=
+           HSA_PACKET_TYPE_KERNEL_DISPATCH)
+        {
+            transformed_packets.emplace_back(packets_arr[i]);
+            continue;
+        }
+
+        // Copy kernel pkt, copy is to allow for signal to be modified
+        hsa_ext_amd_aql_pm4_packet_t kernel_pkt = packets_arr[i];
+        queue_info.create_signal(HSA_AMD_SIGNAL_AMD_GPU_ONLY, &kernel_pkt.completion_signal);
+
+        // Stores the instrumentation pkt (i.e. AQL packets for counter collection)
+        // along with an ID of the client we got the packet from (this will be returned via
+        // CompletedCB)
+        ClientID                   inst_pkt_id = -1;
+        std::unique_ptr<AQLPacket> inst_pkt;
+
+        // Signal callbacks that a kernel_pkt is being enqueued
+        queue_info.signal_callback([&](const auto& map) {
+            for(const auto& [client_id, cb_pair] : map)
+            {
+                if(auto maybe_pkt = cb_pair.first(queue_info, client_id, kernel_pkt))
+                {
+                    LOG_IF(FATAL, inst_pkt)
+                        << "We do not support two injections into the HSA queue";
+                    inst_pkt    = std::move(maybe_pkt);
+                    inst_pkt_id = client_id;
+                }
+            }
+        });
+
+        // Write instrumentation start packet (if one exists)
+        if(inst_pkt)
+        {
+            hsa_signal_t dummy_signal{};
+            dummy_signal.handle    = 0;
+            inst_pkt->start.header = HSA_PACKET_TYPE_VENDOR_SPECIFIC << HSA_PACKET_HEADER_TYPE;
+            AddVendorSpecificPacket(inst_pkt->start, transformed_packets, dummy_signal);
+
+            CreateBarrierPacket(inst_pkt->start.completion_signal, transformed_packets);
+        }
+
+        transformed_packets.emplace_back(kernel_pkt);
+
+        // Make a copy of the original packet, adding its signal to a barrier
+        // packet and create a new signal for it to get timestamps
+        if(original_packet.completion_signal.handle != 0u)
+        {
+            hsa_barrier_and_packet_t barrier{};
+            barrier.header = HSA_PACKET_TYPE_BARRIER_AND << HSA_PACKET_HEADER_TYPE;
+            hsa_ext_amd_aql_pm4_packet_t* __attribute__((__may_alias__)) pkt =
+                (reinterpret_cast<hsa_ext_amd_aql_pm4_packet_t*>(&barrier));
+            transformed_packets.emplace_back(*pkt).completion_signal =
+                original_packet.completion_signal;
+        }
+
+        hsa_signal_t interrupt_signal{};
+        // Adding a barrier packet with the original packet's completion signal.
+        queue_info.create_signal(0, &interrupt_signal);
+
+        if(inst_pkt)
+        {
+            hsa_signal_t dummy_signal{};
+            dummy_signal.handle   = 0;
+            inst_pkt->stop.header = HSA_PACKET_TYPE_VENDOR_SPECIFIC << HSA_PACKET_HEADER_TYPE;
+            AddVendorSpecificPacket(inst_pkt->stop, transformed_packets, dummy_signal);
+            inst_pkt->read.header = HSA_PACKET_TYPE_VENDOR_SPECIFIC << HSA_PACKET_HEADER_TYPE;
+            AddVendorSpecificPacket(inst_pkt->read, transformed_packets, interrupt_signal);
+
+            // Added Interrupt Signal with barrier and provided handler for it
+            CreateBarrierPacket(interrupt_signal, transformed_packets);
+        }
+        else
+        {
+            hsa_barrier_and_packet_t barrier{};
+            barrier.header            = HSA_PACKET_TYPE_BARRIER_AND << HSA_PACKET_HEADER_TYPE;
+            barrier.completion_signal = interrupt_signal;
+            hsa_ext_amd_aql_pm4_packet_t* __attribute__((__may_alias__)) pkt =
+                (reinterpret_cast<hsa_ext_amd_aql_pm4_packet_t*>(&barrier));
+            transformed_packets.emplace_back(*pkt);
+        }
+
+        // Enqueue the signal into the handler. Will call completed_cb when
+        // signal completes.
+        queue_info.async_started();
+        queue_info.signal_async_handler(
+            interrupt_signal,
+            new Queue::queue_info_session_t{.queue            = queue_info,
+                                            .inst_pkt         = std::move(inst_pkt),
+                                            .inst_pkt_id      = inst_pkt_id,
+                                            .kernel_pkt       = kernel_pkt,
+                                            .interrupt_signal = interrupt_signal});
+    }
+
+    writer(transformed_packets.data(), transformed_packets.size());
+}
+
+Queue::Queue(const AgentCache&  agent,
+             uint32_t           size,
+             hsa_queue_type32_t type,
+             void (*callback)(hsa_status_t status, hsa_queue_t* source, void* data),
+             void*         data,
+             uint32_t      private_segment_size,
+             uint32_t      group_segment_size,
+             CoreApiTable  core_api,
+             AmdExtTable   ext_api,
+             hsa_queue_t** queue)
+: _core_api(core_api)
+, _ext_api(ext_api)
+, _agent(agent)
+
+{
+    LOG_IF(FATAL,
+           _ext_api.hsa_amd_queue_intercept_create_fn(_agent.get_agent(),
+                                                      size,
+                                                      type,
+                                                      callback,
+                                                      data,
+                                                      private_segment_size,
+                                                      group_segment_size,
+                                                      &_intercept_queue) != HSA_STATUS_SUCCESS)
+        << "Could not create intercept queue";
+
+    LOG_IF(FATAL,
+           _ext_api.hsa_amd_profiling_set_profiler_enabled_fn(_intercept_queue, true) !=
+               HSA_STATUS_SUCCESS)
+        << "Could not setup intercept profiler";
+
+    LOG_IF(FATAL,
+           _ext_api.hsa_amd_queue_intercept_register_fn(_intercept_queue, WriteInterceptor, this))
+        << "Could not register interceptor";
+    *queue = _intercept_queue;
+}
+
+void
+Queue::register_callback(ClientID id, QueueCB enqueue_cb, CompletedCB complete_cb)
+{
+    _callbacks.wlock([&](auto& map) {
+        LOG_IF(FATAL, rocprofiler::common::get_val(map, id)) << "ID already exists!";
+        _notifiers++;
+        map[id] = std::make_pair(enqueue_cb, complete_cb);
+    });
+}
+
+void
+Queue::remove_callback(ClientID id)
+{
+    _callbacks.wlock([&](auto& map) {
+        if(map.erase(id) == 1) _notifiers--;
+    });
+}
+
+void
+QueueController::add_queue(hsa_queue_t* id, std::unique_ptr<Queue> queue)
+{
+    CHECK(queue);
+    _callback_cache.wlock([&](auto& callbacks) {
+        _queues.wlock([&](auto& map) {
+            const auto agent_id = queue->get_agent().agent_t().id.handle;
+            map[id]             = std::move(queue);
+            for(const auto& [cbid, cb_tuple] : callbacks)
+            {
+                auto& [agent, qcb, ccb] = cb_tuple;
+                if(agent.id.handle == agent_id)
+                {
+                    map[id]->register_callback(cbid, qcb, ccb);
+                }
+            }
+        });
+    });
+}
+
+void
+QueueController::destory_queue(hsa_queue_t* id)
+{
+    _queues.wlock([&](auto& map) { map.erase(id); });
+}
+
+ClientID
+QueueController::add_callback(const rocprofiler_agent_t& agent,
+                              Queue::QueueCB             qcb,
+                              Queue::CompletedCB         ccb)
+{
+    static std::atomic<ClientID> client_id = 1;
+    ClientID                     return_id;
+    _callback_cache.wlock([&](auto& cb_cache) {
+        return_id           = client_id;
+        cb_cache[client_id] = std::tuple(agent, qcb, ccb);
+        client_id++;
+        _queues.wlock([&](auto& map) {
+            for(auto& [_, queue] : map)
+            {
+                if(queue->get_agent().agent_t().id.handle == agent.id.handle)
+                {
+                    queue->register_callback(return_id, qcb, ccb);
+                }
+            }
+        });
+    });
+    return return_id;
+}
+
+void
+QueueController::remove_callback(ClientID id)
+{
+    _callback_cache.wlock([&](auto& cb_cache) {
+        cb_cache.erase(id);
+        _queues.wlock([&](auto& map) {
+            for(auto& [_, queue] : map)
+            {
+                queue->remove_callback(id);
+            }
+        });
+    });
+}
+
+// HSA Intercept Functions (create_queue/destroy_queue)
+hsa_status_t
+create_queue(hsa_agent_t        agent,
+             uint32_t           size,
+             hsa_queue_type32_t type,
+             void (*callback)(hsa_status_t status, hsa_queue_t* source, void* data),
+             void*         data,
+             uint32_t      private_segment_size,
+             uint32_t      group_segment_size,
+             hsa_queue_t** queue)
+{
+    for(const auto& [_, agent_info] : get_queue_controller().get_supported_agents())
+    {
+        if(agent_info.get_agent().handle == agent.handle)
+        {
+            auto new_queue = std::make_unique<Queue>(agent_info,
+                                                     size,
+                                                     type,
+                                                     callback,
+                                                     data,
+                                                     private_segment_size,
+                                                     group_segment_size,
+                                                     get_queue_controller().get_core_table(),
+                                                     get_queue_controller().get_ext_table(),
+                                                     queue);
+            get_queue_controller().add_queue(*queue, std::move(new_queue));
+            return HSA_STATUS_SUCCESS;
+        }
+    }
+    LOG(FATAL) << "Could not find agent - " << agent.handle;
+    return HSA_STATUS_ERROR_FATAL;
+}
+
+hsa_status_t
+destroy_queue(hsa_queue_t* hsa_queue)
+{
+    get_queue_controller().destory_queue(hsa_queue);
+    return HSA_STATUS_SUCCESS;
+}
+
+void
+QueueController::Init(CoreApiTable& core_table, AmdExtTable& ext_table)
+{
+    _core_table = core_table;
+    _ext_table  = ext_table;
+
+    core_table.hsa_queue_create_fn  = create_queue;
+    core_table.hsa_queue_destroy_fn = destroy_queue;
+
+    // Generate supported agents
+    rocprofiler_query_available_agents(
+        [](const rocprofiler_agent_t** agents, size_t num_agents, void* user_data) {
+            CHECK(user_data);
+            QueueController& queue = *reinterpret_cast<QueueController*>(user_data);
+            for(size_t i = 0; i < num_agents; i++)
+            {
+                const auto& agent = *agents[i];
+                if(agent.type != ROCPROFILER_AGENT_TYPE_GPU) continue;
+                try
+                {
+                    queue.get_supported_agents().emplace(
+                        i, AgentCache{agent, i, queue.get_core_table(), queue.get_ext_table()});
+                } catch(std::runtime_error& error)
+                {
+                    LOG(ERROR) << fmt::format("GPU Agent Construction Failed (HSA queue will not "
+                                              "be intercepted): {} ({})",
+                                              agent.id.handle,
+                                              error.what());
+                }
+            }
+            return ROCPROFILER_STATUS_SUCCESS;
+        },
+        sizeof(rocprofiler_agent_t),
+        this);
+}
+
+QueueController&
+get_queue_controller()
+{
+    static QueueController controller;
+    return controller;
+}
+
+void
+queue_controller_init(HsaApiTable* table)
+{
+    get_queue_controller().Init(*table->core_, *table->amd_ext_);
+}
+
+}  // namespace hsa
+}  // namespace rocprofiler
@@ -0,0 +1,237 @@
+/* Copyright (c) 2022 Advanced Micro Devices, Inc.
+
+ Permission is hereby granted, free of charge, to any person obtaining a copy
+ of this software and associated documentation files (the "Software"), to deal
+ in the Software without restriction, including without limitation the rights
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+ copies of the Software, and to permit persons to whom the Software is
+ furnished to do so, subject to the following conditions:
+
+ The above copyright notice and this permission notice shall be included in
+ all copies or substantial portions of the Software.
+
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+ THE SOFTWARE. */
+
+#pragma once
+
+#include <atomic>
+#include <functional>
+#include <iostream>
+#include <memory>
+#include <string>
+#include <unordered_map>
+#include <vector>
+
+#include <hsa/amd_hsa_kernel_code.h>
+#include <hsa/hsa.h>
+#include <hsa/hsa_api_trace.h>
+#include <hsa/hsa_ext_amd.h>
+#include <hsa/hsa_ven_amd_aqlprofile.h>
+#include <hsa/hsa_ven_amd_loader.h>
+
+#include <rocprofiler/fwd.h>
+#include "lib/common/synchronized.hpp"
+#include "lib/rocprofiler/hsa/agent_cache.hpp"
+
+namespace rocprofiler
+{
+namespace hsa
+{
+/**
+ * Struct containing AQL packet information. Including start/stop/read
+ * packets along with allocated buffers
+ */
+struct AQLPacket
+{
+    hsa_ven_amd_aqlprofile_profile_t                  profile;
+    hsa_ext_amd_aql_pm4_packet_t                      start{.header            = 0,
+                                       .pm4_command       = {0},
+                                       .completion_signal = {.handle = 0}};
+    hsa_ext_amd_aql_pm4_packet_t                      stop{.header            = 0,
+                                      .pm4_command       = {0},
+                                      .completion_signal = {.handle = 0}};
+    hsa_ext_amd_aql_pm4_packet_t                      read{.header            = 0,
+                                      .pm4_command       = {0},
+                                      .completion_signal = {.handle = 0}};
+    bool                                              command_buf_mallocd{false};
+    bool                                              output_buffer_malloced{false};
+    std::function<decltype(hsa_amd_memory_pool_free)> free_func;
+    AQLPacket(std::function<decltype(hsa_amd_memory_pool_free)> func)
+    : free_func(std::move(func))
+    {}
+
+    ~AQLPacket()
+    {
+        if(!command_buf_mallocd)
+        {
+            free_func(profile.command_buffer.ptr);
+        }
+        else
+        {
+            free(profile.command_buffer.ptr);
+        }
+
+        if(!output_buffer_malloced)
+        {
+            free_func(profile.output_buffer.ptr);
+        }
+        else
+        {
+            free(profile.output_buffer.ptr);
+        }
+    }
+
+    // Keep move constuctors (i.e. std::move())
+    AQLPacket(AQLPacket&& other) = default;
+    AQLPacket& operator=(AQLPacket&& other) = default;
+
+    // Do not allow copying this class
+    AQLPacket(const AQLPacket&) = delete;
+    AQLPacket& operator=(const AQLPacket&) = delete;
+};
+
+using ClientID = int64_t;
+
+// Interceptor for a single specific queue
+class Queue
+{
+public:
+    // Internal session information that is used by write interceptor
+    // to track state of the intercepted kernel.
+    struct queue_info_session_t
+    {
+        Queue&                       queue;
+        std::unique_ptr<AQLPacket>   inst_pkt;
+        ClientID                     inst_pkt_id;
+        hsa_ext_amd_aql_pm4_packet_t kernel_pkt;
+        hsa_signal_t                 interrupt_signal;
+    };
+
+    Queue(const AgentCache&  agent,
+          uint32_t           size,
+          hsa_queue_type32_t type,
+          void (*callback)(hsa_status_t status, hsa_queue_t* source, void* data),
+          void*         data,
+          uint32_t      private_segment_size,
+          uint32_t      group_segment_size,
+          CoreApiTable  core_api,
+          AmdExtTable   ext_api,
+          hsa_queue_t** queue);
+
+    const hsa_queue_t* intercept_queue() const { return _intercept_queue; };
+    const AgentCache&  get_agent() const { return _agent; }
+
+    void create_signal(uint32_t attribute, hsa_signal_t* signal) const;
+    void signal_async_handler(const hsa_signal_t& signal, Queue::queue_info_session_t* data) const;
+
+    rocprofiler_queue_id_t get_id() const
+    {
+        return {.handle = reinterpret_cast<uint64_t>(intercept_queue())};
+    };
+
+    template <class Func>
+    void signal_callback(Func&& func) const
+    {
+        _callbacks.rlock([&func](const auto& data) { func(data); });
+    }
+
+    // Fast check to see if we have any callbacks we need to notify
+    int get_notifiers() const { return _notifiers; }
+
+    // Tracks the number of in flight kernel executions we
+    // are waiting on. We cannot destroy Queue until all kernels
+    // have comleted.
+    void async_started() { _active_async_packets++; }
+    void async_complete() { _active_async_packets--; }
+
+    ~Queue()
+    {
+        // Potentially replace with condition variable at some point
+        // but performance may not matter here.
+        while(_active_async_packets > 0)
+        {}
+    }
+
+    // Function prototype used to notify consumers that a kernel has been
+    // enqueued. An AQL packet can be returned that will be injected into
+    // the queue.
+    using QueueCB = std::function<
+        std::unique_ptr<AQLPacket>(const Queue&, ClientID, const hsa_ext_amd_aql_pm4_packet_t&)>;
+    // Signals the completion of the kernel packet.
+    using CompletedCB = std::function<void(const Queue&,
+                                           ClientID,
+                                           const hsa_ext_amd_aql_pm4_packet_t&,
+                                           std::unique_ptr<AQLPacket>)>;
+
+    void register_callback(ClientID id, QueueCB enqueue_cb, CompletedCB complete_cb);
+    void remove_callback(ClientID id);
+
+    const CoreApiTable& core_api() const { return _core_api; }
+    const AmdExtTable&  ext_api() const { return _ext_api; }
+
+private:
+    std::atomic<int64_t> _active_async_packets{0};
+    CoreApiTable         _core_api;
+    AmdExtTable          _ext_api;
+    const AgentCache&    _agent;
+    std::atomic<int>     _notifiers;
+    rocprofiler::common::Synchronized<std::unordered_map<ClientID, std::pair<QueueCB, CompletedCB>>>
+                 _callbacks;
+    hsa_queue_t* _intercept_queue;
+};
+
+// Tracks and manages HSA queues
+class QueueController
+{
+public:
+    QueueController() = default;
+    // Initializes the QueueInterceptor. This must be delayed until
+    // HSA has been inited.
+    void Init(CoreApiTable& core_table, AmdExtTable& ext_table);
+    // Called to add a queue that was created by the user program
+    void add_queue(hsa_queue_t*, std::unique_ptr<Queue>);
+    void destory_queue(hsa_queue_t*);
+
+    // Add callback to queues associated with the agent. Returns a client
+    // id that can be used by callers to remove the callback.
+    ClientID add_callback(const rocprofiler_agent_t&, Queue::QueueCB, Queue::CompletedCB);
+    void     remove_callback(ClientID);
+
+    const CoreApiTable& get_core_table() const { return _core_table; }
+    const AmdExtTable&  get_ext_table() const { return _ext_table; }
+
+    // Gets the list of supported HSA agents that can be intercepted
+    const std::unordered_map<uint32_t, AgentCache>& get_supported_agents() const
+    {
+        return _supported_agents;
+    }
+
+    std::unordered_map<uint32_t, AgentCache>& get_supported_agents() { return _supported_agents; }
+
+private:
+    CoreApiTable _core_table;
+    AmdExtTable  _ext_table;
+    rocprofiler::common::Synchronized<std::unordered_map<hsa_queue_t*, std::unique_ptr<Queue>>>
+        _queues;
+    rocprofiler::common::Synchronized<
+        std::unordered_map<ClientID,
+                           std::tuple<rocprofiler_agent_t, Queue::QueueCB, Queue::CompletedCB>>>
+        _callback_cache;
+
+    std::unordered_map<uint32_t, AgentCache> _supported_agents;
+};
+
+QueueController&
+get_queue_controller();
+
+void
+queue_controller_init(HsaApiTable* table);
+
+}  // namespace hsa
+}  // namespace rocprofiler
@@ -20,9 +20,10 @@

 #pragma once

+#include <rocprofiler/hsa.h>
+#include <rocprofiler/version.h>
+
 #include "lib/common/defines.hpp"
-#include "rocprofiler/hsa.h"
-#include "rocprofiler/version.h"

 #ifndef ROCPROFILER_UNSAFE_NO_VERSION_CHECK
 #    if defined(ROCPROFILER_CI) && ROCPROFILER_CI > 0
@@ -25,6 +25,7 @@
 #include <rocprofiler/rocprofiler.h>

 #include "lib/common/container/stable_vector.hpp"
+#include "lib/common/utility.hpp"
 #include "lib/rocprofiler/buffer.hpp"
 #include "lib/rocprofiler/context/context.hpp"
 #include "lib/rocprofiler/internal_threading.hpp"
@@ -141,21 +142,25 @@ execute_creation_notifiers(rocprofiler_internal_thread_library_t libs,
    (execute(get_creation_notifier<Idx>()), ...);
 }

-auto*&
+// using thread_pool_vec_t = std::vector<std::unique_ptr<thread_pool_t>>;
+// using task_group_vec_t  = std::vector<std::unique_ptr<task_group_t>>;
+
+auto&
 get_thread_pools()
 {
-    // use raw pointers here because of the access to this variable via atexit function call.
-    // if not a raw pointer, this may be destroyed automatically when the atexit handler is invoked
-    static auto* _v = new thread_pool_vec_t{};
+    static auto _v = thread_pool_vec_t{};
    return _v;
 }

-auto*&
+auto&
 get_task_groups()
 {
-    // use raw pointers here because of the access to this variable via atexit function call.
-    // if not a raw pointer, this may be destroyed automatically when the atexit handler is invoked
-    static auto* _v = new task_group_vec_t{};
+    static auto _v = task_group_vec_t([](auto& data) {
+        for(auto& itr : data)
+            itr.first->join();
+        data.clear();
+    });
+
    return _v;
 }
 }  // namespace
@@ -166,33 +171,21 @@ initialize()
 {
    static auto _once = std::once_flag{};
    std::call_once(_once, []() {
-        atexit(&registration::finalize);
+        // Note: create_callback_thread() must occur before atexit
+        // registration or else the static objects it is pointing to
+        // will be destroyed before finalize is invoked.
        create_callback_thread();
+        atexit(&registration::finalize);
    });
 }

-// sync all the task groups and destroy the thread pools
 void
 finalize()
 {
-    if(get_task_groups())
-    {
-        for(auto& itr : *get_task_groups())
-            if(itr) itr->join();
-        get_task_groups()->clear();
-    }
-
-    if(get_thread_pools())
-    {
-        for(auto& itr : *get_thread_pools())
-            if(itr) itr->destroy_threadpool();
-        get_thread_pools()->clear();
-    }
-
-    delete get_task_groups();
-    delete get_thread_pools();
-    get_task_groups()  = nullptr;
-    get_thread_pools() = nullptr;
+    // PLT::ThreadPool::f_thread_ids() is not destruction order safe
+    // if it does become safe, these two calls could be removed.
+    get_task_groups().destroy();
+    get_thread_pools().clear();
 }

 void
@@ -210,20 +203,19 @@ notify_post_internal_thread_create(rocprofiler_internal_thread_library_t libs)
 rocprofiler_callback_thread_t
 create_callback_thread()
 {
-    if(!get_thread_pools()) throw std::runtime_error{"thread pools already deleted"};
-    if(!get_task_groups()) throw std::runtime_error{"task groups already deleted"};
-
    // notify that rocprofiler library is about to create an inernal thread
    notify_pre_internal_thread_create(ROCPROFILER_LIBRARY);

    // this will be index after emplace_back
-    auto idx = get_thread_pools()->size();
+    auto idx = get_thread_pools().size();

-    auto& thr_pool =
-        get_thread_pools()->emplace_back(new thread_pool_t{thread_pool_config_t{.pool_size = 1}});
+    auto& thr_pool = get_thread_pools().emplace_back(std::make_shared<thread_pool_cleanup_t>(
+        std::make_unique<thread_pool_t>(thread_pool_config_t{.pool_size = 1}),
+        [](auto& tp) { tp->destroy_threadpool(); }));

    // construct the task group to use the newly created thread pool
-    get_task_groups()->emplace_back(new task_group_t{thr_pool.get()});
+    get_task_groups().get().emplace_back(std::make_unique<task_group_t>(thr_pool->get().get()),
+                                         thr_pool);

    // notify that rocprofiler library finished creating an internal thread
    notify_post_internal_thread_create(ROCPROFILER_LIBRARY);
@@ -235,7 +227,9 @@ create_callback_thread()
 task_group_t*
 get_task_group(rocprofiler_callback_thread_t cb_tid)
 {
-    return (get_task_groups()) ? get_task_groups()->at(cb_tid.handle).get() : nullptr;
+    return (!get_task_groups().get().empty())
+               ? get_task_groups().get().at(cb_tid.handle).first.get()
+               : nullptr;
 }
 }  // namespace internal_threading
 }  // namespace rocprofiler
@@ -275,8 +269,7 @@ rocprofiler_status_t ROCPROFILER_API
 rocprofiler_assign_callback_thread(rocprofiler_buffer_id_t       buffer_id,
                                   rocprofiler_callback_thread_t cb_thread_id)
 {
-    if(!rocprofiler::internal_threading::get_task_groups() ||
-       cb_thread_id.handle >= rocprofiler::internal_threading::get_task_groups()->size())
+    if(cb_thread_id.handle >= rocprofiler::internal_threading::get_task_groups().get().size())
        return ROCPROFILER_STATUS_ERROR_THREAD_NOT_FOUND;

    for(auto& bitr : rocprofiler::buffer::get_buffers())
@@ -26,6 +26,7 @@

 #include "lib/common/container/stable_vector.hpp"
 #include "lib/common/defines.hpp"
+#include "lib/common/utility.hpp"

 #include <PTL/TaskGroup.hh>
 #include <PTL/ThreadPool.hh>
@@ -38,10 +39,20 @@ namespace rocprofiler
 {
 namespace internal_threading
 {
-using thread_pool_t     = PTL::ThreadPool;
-using task_group_t      = PTL::TaskGroup<void>;
-using thread_pool_vec_t = std::vector<std::unique_ptr<thread_pool_t>>;
-using task_group_vec_t  = std::vector<std::unique_ptr<task_group_t>>;
+using thread_pool_t         = PTL::ThreadPool;
+using task_group_t          = PTL::TaskGroup<void>;
+using thread_pool_cleanup_t = rocprofiler::common::static_cleanup_wrapper<
+    std::unique_ptr<thread_pool_t>,
+    std::function<void(std::unique_ptr<thread_pool_t>&)>>;
+using task_group_cleanup_t =
+    std::pair<std::unique_ptr<task_group_t>, std::shared_ptr<thread_pool_cleanup_t>>;
+using thread_pool_vec_t = std::vector<std::shared_ptr<thread_pool_cleanup_t>>;
+
+// Note: task_group maintains a shared_ptr copy to thread_pool to ensure it is not destroyed
+// before the task can be sync'd.
+using task_group_vec_t = rocprofiler::common::static_cleanup_wrapper<
+    std::vector<task_group_cleanup_t>,
+    std::function<void(std::vector<task_group_cleanup_t>&)>>;

 void notify_pre_internal_thread_create(rocprofiler_internal_thread_library_t);
 void notify_post_internal_thread_create(rocprofiler_internal_thread_library_t);
@@ -0,0 +1,64 @@
+#include <rocprofiler/rocprofiler.h>
+
+#include "lib/common/synchronized.hpp"
+#include "lib/common/utility.hpp"
+#include "lib/rocprofiler/aql/helpers.hpp"
+#include "lib/rocprofiler/counters/core.hpp"
+#include "lib/rocprofiler/counters/evaluate_ast.hpp"
+#include "lib/rocprofiler/counters/metrics.hpp"
+#include "lib/rocprofiler/hsa/agent_cache.hpp"
+
+extern "C" {
+/**
+ * @brief Create Profile Configuration.
+ *
+ * @param [in] agent Agent identifier
+ * @param [in] counters_list List of GPU counters
+ * @param [in] counters_count Size of counters list
+ * @param [out] config_id Identifier for GPU counters group
+ * @return ::rocprofiler_status_t
+ */
+rocprofiler_status_t ROCPROFILER_API
+rocprofiler_create_profile_config(rocprofiler_agent_t              agent,
+                                  rocprofiler_counter_id_t*        counters_list,
+                                  size_t                           counters_count,
+                                  rocprofiler_profile_config_id_t* config_id)
+{
+    rocprofiler::counters::profile_config config;
+    const auto&                           id_map = rocprofiler::counters::getMetricIdMap();
+
+    for(size_t i = 0; i < counters_count; i++)
+    {
+        auto& counter_id = counters_list[i];
+
+        const auto* metric_ptr = rocprofiler::common::get_val(id_map, counter_id.handle);
+        if(!metric_ptr) return ROCPROFILER_STATUS_ERROR_COUNTER_NOT_FOUND;
+        config.metrics.push_back(*metric_ptr);
+
+        auto agent_name = std::string(agent.name);
+        auto req_counters =
+            rocprofiler::counters::get_required_hardware_counters(agent_name, *metric_ptr);
+        if(!req_counters) return ROCPROFILER_STATUS_ERROR_COUNTER_NOT_FOUND;
+        config.reqired_hw_counters.insert(req_counters->begin(), req_counters->end());
+
+        const auto& asts      = rocprofiler::counters::get_ast_map();
+        const auto* agent_map = rocprofiler::common::get_val(asts, agent_name);
+        if(!agent_map) return ROCPROFILER_STATUS_ERROR_COUNTER_NOT_FOUND;
+        const auto* counter_ast = rocprofiler::common::get_val(*agent_map, metric_ptr->name());
+        if(!counter_ast) return ROCPROFILER_STATUS_ERROR_COUNTER_NOT_FOUND;
+        config.asts.push_back(*counter_ast);
+    }
+
+    config.agent      = agent;
+    config_id->handle = rocprofiler::counters::create_counter_profile(std::move(config));
+
+    return ROCPROFILER_STATUS_SUCCESS;
+}
+
+rocprofiler_status_t ROCPROFILER_API
+rocprofiler_destroy_profile_config(rocprofiler_profile_config_id_t config_id)
+{
+    rocprofiler::counters::destroy_counter_profile(config_id.handle);
+    return ROCPROFILER_STATUS_SUCCESS;
+}
+}
@@ -23,6 +23,7 @@
 #include "lib/rocprofiler/registration.hpp"
 #include "lib/rocprofiler/context/context.hpp"
 #include "lib/rocprofiler/hsa/hsa.hpp"
+#include "lib/rocprofiler/hsa/queue.hpp"
 #include "lib/rocprofiler/internal_threading.hpp"

 #include <rocprofiler/context.h>
@@ -542,6 +543,7 @@ rocprofiler_set_api_table(const char* name,
        auto& saved_hsa_api_table = rocprofiler::hsa::get_table();
        ::copyTables(hsa_api_table, &saved_hsa_api_table);

+        rocprofiler::hsa::queue_controller_init(hsa_api_table);
        rocprofiler::hsa::update_table(hsa_api_table);
    }
    else if(std::string_view{name} == "roctx")
@@ -51,5 +51,5 @@ gtest_add_tests(

 set_tests_properties(
    ${shared_lib_TESTS}
-    PROPERTIES TIMEOUT 45 LABELS "unittests" ENVIRONMENT
+    PROPERTIES TIMEOUT 360 LABELS "unittests" ENVIRONMENT
               "HSA_TOOLS_LIB=$<TARGET_FILE:rocprofiler-shared-library>")
@@ -18,8 +18,7 @@
 // OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
 // THE SOFTWARE.

-#include "agent.hpp"
-
+#include "lib/rocprofiler/tests/details/agent.hpp"
 #include "lib/common/utility.hpp"

 #include <glog/logging.h>
@@ -20,4 +20,4 @@ gtest_add_tests(
    TEST_LIST buffering-tests_TESTS
    WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR})

-set_tests_properties(${buffering-tests_TESTS} PROPERTIES TIMEOUT 120 LABELS "unittests")
+set_tests_properties(${buffering-tests_TESTS} PROPERTIES TIMEOUT 360 LABELS "unittests")
@@ -0,0 +1,28 @@
+cmake_minimum_required(VERSION 3.16 FATAL_ERROR)
+
+foreach(BINARY_OUTPUT ${BINARY_DIR}/parser.h ${BINARY_DIR}/parser.cpp
+                      ${BINARY_DIR}/scanner.cpp)
+    string(REPLACE "${BINARY_DIR}" "${SOURCE_DIR}" SOURCE_OUTPUT "${BINARY_OUTPUT}")
+    foreach(VAR PROJECT_SRC_DIR PROJECT_BLD_DIR)
+        string(REPLACE "/" "_" ${VAR} "${${VAR}}")
+        string(REPLACE "-" "_" ${VAR} "${${VAR}}")
+        string(REPLACE "+" "" ${VAR} "${${VAR}}")
+        string(TOUPPER "${${VAR}}" ${VAR})
+    endforeach()
+
+    # remove absolute path from file
+    if(NOT SOURCE_OUTPUT STREQUAL BINARY_OUTPUT)
+        file(READ ${BINARY_OUTPUT} OUTPUT_DATA)
+        string(REPLACE "${SOURCE_DIR}/" "" OUTPUT_DATA "${OUTPUT_DATA}")
+        string(REPLACE "${BINARY_DIR}/" "" OUTPUT_DATA "${OUTPUT_DATA}")
+        string(REPLACE "${PROJECT_BLD_DIR}" "_ROCPROFILER" OUTPUT_DATA "${OUTPUT_DATA}")
+        string(REPLACE "${PROJECT_SRC_DIR}" "_ROCPROFILER" OUTPUT_DATA "${OUTPUT_DATA}")
+        file(WRITE ${BINARY_OUTPUT} "${OUTPUT_DATA}")
+
+        if(FORMAT_EXE)
+            execute_process(COMMAND ${FORMAT_EXE} -i ${BINARY_OUTPUT})
+        endif()
+
+        configure_file(${BINARY_OUTPUT} ${SOURCE_OUTPUT} COPYONLY)
+    endif()
+endforeach()
@@ -92,6 +92,7 @@ def generate_custom(args, cmake_args, ctest_args):
        "samples/.*",
        "tests/.*",
        ".*/details/.*",
+        "*/counters/parser/.*",
    ]
    if args.coverage == "samples":
        codecov_exclude += [".*/lib/common/.*"]