Buffered Counter Collection API (#179)
* Added buffer counter collection API. Initial testing added into counter-collection sample. Added support for constant metrics in counter collection (#194) * Added support for constant metrics in counter collection Adds support and test cases for constant metrics (such as max wave size) and adds the metric kernel duration (though this is still not yet calculated). * Minor doc updates * Simple counter unit tests (#199) * Simple counter unit tests Unit tests and some minor fixes for simple and derived counter evaluation * Added unit tests for reduction operations (#200) * Added unit tests for reduction operations * added tests for combo (constant+regular) counters (#201) source formatting (clang-format v11) (#202) Co-authored-by: bwelton <bwelton@users.noreply.github.com> source formatting (clang-format v11) (#203) Co-authored-by: bwelton <bwelton@users.noreply.github.com> Local changes source formatting (clang-format v11) (#205) Co-authored-by: bwelton <bwelton@users.noreply.github.com> Minor doc fix Remove kernel_duration, migrate over set_dimensions to after HSA init source formatting (clang-format v11) (#207) Co-authored-by: bwelton <bwelton@users.noreply.github.com> Added output to ROCPROFILER_SAMPLE_OUTPUT_FILE: * Remove integer based counter in return struct This casues a lot of complications and seems to provide limit benefit of just treating all counters as doubles. For ease of use, drop the integer based counter. * source formatting (clang-format v11) (#217) Co-authored-by: bwelton <bwelton@users.noreply.github.com> * Add correlation id support to counters (#218) Adds correlation id support to counter collection. Requires tracing to be enabled to return any useful value currently (since we do not have HIP kernel tracing yet). * source formatting (clang-format v11) (#223) Co-authored-by: bwelton <bwelton@users.noreply.github.com> * Add sample that attempts to fetch all counters On whatever machine this test is run on, all counters available on the platform will attempted to be fetched from a kernel execution. Each counter will be fetched one time to check that the counter can be fetched on the platform and that the counter is returning the correct instance count (however due to the lack of transparency from AQL profiler this check is not functional for some counters). We do not do any implicit reduction on any counter, the result is that we see more counters than the number of events being requested. Below is the status of all counters on MI210. All counters appear functional with the changes in this PR. However, the instance count retruned will be greater than that returned by rocprofiler_query_counter_instance_count. Got 516 counters collected Counter ID: 0 (size) expected 1 instances and got 1 Counter ID: 1 (processor_id_low) expected 1 instances and got 1 Counter ID: 2 (capability) expected 1 instances and got 1 Counter ID: 3 (local_mem_size) expected 1 instances and got 1 Counter ID: 4 (min_latency) expected 1 instances and got 1 Counter ID: 5 (weight) expected 1 instances and got 1 Counter ID: 6 (node_from) expected 1 instances and got 1 Counter ID: 7 (version_major) expected 1 instances and got 1 Counter ID: 8 (version_minor) expected 1 instances and got 1 Counter ID: 9 (mem_clk_max) expected 1 instances and got 1 Counter ID: 10 (num_xcc) expected 1 instances and got 1 Counter ID: 11 (width) expected 1 instances and got 1 Counter ID: 12 (flags) expected 1 instances and got 1 Counter ID: 13 (size_in_bytes) expected 1 instances and got 1 Counter ID: 14 (array_count) expected 1 instances and got 1 Counter ID: 15 (num_gws) expected 1 instances and got 1 Counter ID: 16 (simd_id_base) expected 1 instances and got 1 Counter ID: 17 (max_waves_per_simd) expected 1 instances and got 1 Counter ID: 18 (sdma_fw_version) expected 1 instances and got 1 Counter ID: 19 (gfx_target_version) expected 1 instances and got 1 Counter ID: 20 (max_bandwidth) expected 1 instances and got 1 Counter ID: 21 (cpu_core_id_base) expected 1 instances and got 1 Counter ID: 22 (cache_line_size) expected 1 instances and got 1 Counter ID: 23 (level) expected 1 instances and got 1 Counter ID: 24 (min_bandwidth) expected 1 instances and got 1 Counter ID: 25 (location_id) expected 1 instances and got 1 Counter ID: 26 (wave_front_size) expected 1 instances and got 1 Counter ID: 27 (lds_size_in_kb) expected 1 instances and got 1 Counter ID: 28 (simd_count) expected 1 instances and got 1 Counter ID: 29 (fw_version) expected 1 instances and got 1 Counter ID: 30 (recommended_transfer_size) expected 1 instances and got 1 Counter ID: 31 (simd_per_cu) expected 1 instances and got 1 Counter ID: 32 (association) expected 1 instances and got 1 Counter ID: 33 (mem_banks_count) expected 1 instances and got 1 Counter ID: 34 (latency) expected 1 instances and got 1 Counter ID: 35 (max_latency) expected 1 instances and got 1 Counter ID: 36 (cpu_cores_count) expected 1 instances and got 1 Counter ID: 37 (io_links_count) expected 1 instances and got 1 Counter ID: 38 (domain) expected 1 instances and got 1 Counter ID: 39 (max_engine_clk_fcompute) expected 1 instances and got 1 Counter ID: 40 (caches_count) expected 1 instances and got 1 Counter ID: 41 (simd_arrays_per_engine) expected 1 instances and got 1 Counter ID: 42 (cache_lines_per_tag) expected 1 instances and got 1 Counter ID: 43 (gds_size_in_kb) expected 1 instances and got 1 Counter ID: 44 (cu_per_simd_array) expected 1 instances and got 1 Counter ID: 45 (type) expected 1 instances and got 1 Counter ID: 46 (max_slots_scratch_cu) expected 1 instances and got 1 Counter ID: 47 (vendor_id) expected 1 instances and got 1 Counter ID: 48 (device_id) expected 1 instances and got 1 Counter ID: 49 (heap_type) expected 1 instances and got 1 Counter ID: 50 (drm_render_minor) expected 1 instances and got 1 Counter ID: 51 (num_sdma_engines) expected 1 instances and got 1 Counter ID: 52 (node_to) expected 1 instances and got 1 Counter ID: 53 (num_sdma_xgmi_engines) expected 1 instances and got 1 Counter ID: 54 (num_sdma_queues_per_engine) expected 1 instances and got 1 Counter ID: 55 (hive_id) expected 1 instances and got 1 Counter ID: 56 (num_cp_queues) expected 1 instances and got 1 Counter ID: 57 (max_engine_clk_ccompute) expected 1 instances and got 1 Counter ID: 517 (MAX_WAVE_SIZE) expected 1 instances and got 1 Counter ID: 518 (SE_NUM) expected 1 instances and got 1 Counter ID: 519 (SIMD_NUM) expected 1 instances and got 1 Counter ID: 520 (CU_NUM) expected 1 instances and got 1 [ERROR]Counter ID: 521 (SQ_WAIT_INST_LDS) expected 1 instances and got 8 [ERROR]Counter ID: 522 (TCP_TCP_TA_DATA_STALL_CYCLES) expected 16 instances and got 128 Counter ID: 523 (GRBM_COUNT) expected 1 instances and got 1 Counter ID: 524 (GRBM_GUI_ACTIVE) expected 1 instances and got 1 Counter ID: 525 (GRBM_CP_BUSY) expected 1 instances and got 1 Counter ID: 526 (GRBM_SPI_BUSY) expected 1 instances and got 1 Counter ID: 527 (GRBM_TA_BUSY) expected 1 instances and got 1 Counter ID: 528 (GRBM_TC_BUSY) expected 1 instances and got 1 Counter ID: 529 (GRBM_CPC_BUSY) expected 1 instances and got 1 Counter ID: 530 (GRBM_CPF_BUSY) expected 1 instances and got 1 Counter ID: 531 (GRBM_UTCL2_BUSY) expected 1 instances and got 1 Counter ID: 532 (GRBM_EA_BUSY) expected 1 instances and got 1 Counter ID: 533 (CPC_ME1_BUSY_FOR_PACKET_DECODE) expected 1 instances and got 1 Counter ID: 534 (CPC_UTCL1_STALL_ON_TRANSLATION) expected 1 instances and got 1 Counter ID: 535 (CPC_CPC_STAT_BUSY) expected 1 instances and got 1 Counter ID: 536 (CPC_CPC_STAT_IDLE) expected 1 instances and got 1 Counter ID: 537 (CPC_CPC_STAT_STALL) expected 1 instances and got 1 Counter ID: 538 (CPC_CPC_TCIU_BUSY) expected 1 instances and got 1 Counter ID: 539 (CPC_CPC_TCIU_IDLE) expected 1 instances and got 1 Counter ID: 540 (CPC_CPC_UTCL2IU_BUSY) expected 1 instances and got 1 Counter ID: 541 (CPC_CPC_UTCL2IU_IDLE) expected 1 instances and got 1 Counter ID: 542 (CPC_CPC_UTCL2IU_STALL) expected 1 instances and got 1 Counter ID: 543 (CPC_ME1_DC0_SPI_BUSY) expected 1 instances and got 1 Counter ID: 544 (CPF_CMP_UTCL1_STALL_ON_TRANSLATION) expected 1 instances and got 1 Counter ID: 545 (CPF_CPF_STAT_BUSY) expected 1 instances and got 1 Counter ID: 546 (CPF_CPF_STAT_IDLE) expected 1 instances and got 1 Counter ID: 547 (CPF_CPF_STAT_STALL) expected 1 instances and got 1 Counter ID: 548 (CPF_CPF_TCIU_BUSY) expected 1 instances and got 1 Counter ID: 549 (CPF_CPF_TCIU_IDLE) expected 1 instances and got 1 Counter ID: 550 (CPF_CPF_TCIU_STALL) expected 1 instances and got 1 [ERROR]Counter ID: 551 (SPI_CSN_WINDOW_VALID) expected 1 instances and got 8 [ERROR]Counter ID: 552 (SPI_CSN_BUSY) expected 1 instances and got 8 [ERROR]Counter ID: 553 (SPI_CSN_NUM_THREADGROUPS) expected 1 instances and got 8 [ERROR]Counter ID: 554 (SPI_CSN_WAVE) expected 1 instances and got 8 [ERROR]Counter ID: 555 (SPI_RA_REQ_NO_ALLOC) expected 1 instances and got 8 [ERROR]Counter ID: 556 (SPI_RA_REQ_NO_ALLOC_CSN) expected 1 instances and got 8 [ERROR]Counter ID: 557 (SPI_RA_RES_STALL_CSN) expected 1 instances and got 8 [ERROR]Counter ID: 558 (SPI_RA_TMP_STALL_CSN) expected 1 instances and got 8 [ERROR]Counter ID: 559 (SPI_RA_WAVE_SIMD_FULL_CSN) expected 1 instances and got 8 [ERROR]Counter ID: 560 (SPI_RA_VGPR_SIMD_FULL_CSN) expected 1 instances and got 8 [ERROR]Counter ID: 561 (SPI_RA_SGPR_SIMD_FULL_CSN) expected 1 instances and got 8 [ERROR]Counter ID: 562 (SPI_RA_LDS_CU_FULL_CSN) expected 1 instances and got 8 [ERROR]Counter ID: 563 (SPI_RA_BAR_CU_FULL_CSN) expected 1 instances and got 8 [ERROR]Counter ID: 564 (SPI_RA_BULKY_CU_FULL_CSN) expected 1 instances and got 8 [ERROR]Counter ID: 565 (SPI_RA_TGLIM_CU_FULL_CSN) expected 1 instances and got 8 [ERROR]Counter ID: 566 (SPI_RA_WVLIM_STALL_CSN) expected 1 instances and got 8 [ERROR]Counter ID: 567 (SPI_SWC_CSC_WR) expected 1 instances and got 8 [ERROR]Counter ID: 568 (SPI_VWC_CSC_WR) expected 1 instances and got 8 [ERROR]Counter ID: 569 (SQ_ACCUM_PREV) expected 1 instances and got 8 [ERROR]Counter ID: 570 (SQ_CYCLES) expected 1 instances and got 8 [ERROR]Counter ID: 571 (SQ_BUSY_CYCLES) expected 1 instances and got 8 [ERROR]Counter ID: 572 (SQ_WAVES) expected 1 instances and got 8 [ERROR]Counter ID: 573 (SQ_LEVEL_WAVES) expected 1 instances and got 8 [ERROR]Counter ID: 574 (SQ_WAVES_EQ_64) expected 1 instances and got 8 [ERROR]Counter ID: 575 (SQ_WAVES_LT_64) expected 1 instances and got 8 [ERROR]Counter ID: 576 (SQ_WAVES_LT_48) expected 1 instances and got 8 [ERROR]Counter ID: 577 (SQ_WAVES_LT_32) expected 1 instances and got 8 [ERROR]Counter ID: 578 (SQ_WAVES_LT_16) expected 1 instances and got 8 [ERROR]Counter ID: 579 (SQ_BUSY_CU_CYCLES) expected 1 instances and got 8 [ERROR]Counter ID: 580 (SQ_ITEMS) expected 1 instances and got 8 [ERROR]Counter ID: 581 (SQ_INSTS) expected 1 instances and got 8 [ERROR]Counter ID: 582 (SQ_INSTS_VALU) expected 1 instances and got 8 [ERROR]Counter ID: 583 (SQ_INSTS_VALU_ADD_F16) expected 1 instances and got 8 [ERROR]Counter ID: 584 (SQ_INSTS_VALU_MUL_F16) expected 1 instances and got 8 [ERROR]Counter ID: 585 (SQ_INSTS_VALU_FMA_F16) expected 1 instances and got 8 [ERROR]Counter ID: 586 (SQ_INSTS_VALU_TRANS_F16) expected 1 instances and got 8 [ERROR]Counter ID: 587 (SQ_INSTS_VALU_ADD_F32) expected 1 instances and got 8 [ERROR]Counter ID: 588 (SQ_INSTS_VALU_MUL_F32) expected 1 instances and got 8 [ERROR]Counter ID: 589 (SQ_INSTS_VALU_FMA_F32) expected 1 instances and got 8 [ERROR]Counter ID: 590 (SQ_INSTS_VALU_TRANS_F32) expected 1 instances and got 8 [ERROR]Counter ID: 591 (SQ_INSTS_VALU_ADD_F64) expected 1 instances and got 8 [ERROR]Counter ID: 592 (SQ_INSTS_VALU_MUL_F64) expected 1 instances and got 8 [ERROR]Counter ID: 593 (SQ_INSTS_VALU_FMA_F64) expected 1 instances and got 8 [ERROR]Counter ID: 594 (SQ_INSTS_VALU_TRANS_F64) expected 1 instances and got 8 [ERROR]Counter ID: 595 (SQ_INSTS_VALU_INT32) expected 1 instances and got 8 [ERROR]Counter ID: 596 (SQ_INSTS_VALU_INT64) expected 1 instances and got 8 [ERROR]Counter ID: 597 (SQ_INSTS_VALU_CVT) expected 1 instances and got 8 [ERROR]Counter ID: 598 (SQ_INSTS_VALU_MFMA_I8) expected 1 instances and got 8 [ERROR]Counter ID: 599 (SQ_INSTS_VALU_MFMA_F16) expected 1 instances and got 8 [ERROR]Counter ID: 600 (SQ_INSTS_VALU_MFMA_BF16) expected 1 instances and got 8 [ERROR]Counter ID: 601 (SQ_INSTS_VALU_MFMA_F32) expected 1 instances and got 8 [ERROR]Counter ID: 602 (SQ_INSTS_VALU_MFMA_F64) expected 1 instances and got 8 [ERROR]Counter ID: 603 (SQ_INSTS_VALU_MFMA_MOPS_I8) expected 1 instances and got 8 [ERROR]Counter ID: 604 (SQ_INSTS_VALU_MFMA_MOPS_F16) expected 1 instances and got 8 [ERROR]Counter ID: 605 (SQ_INSTS_VALU_MFMA_MOPS_BF16) expected 1 instances and got 8 [ERROR]Counter ID: 606 (SQ_INSTS_VALU_MFMA_MOPS_F32) expected 1 instances and got 8 [ERROR]Counter ID: 607 (SQ_INSTS_VALU_MFMA_MOPS_F64) expected 1 instances and got 8 [ERROR]Counter ID: 608 (SQ_INSTS_MFMA) expected 1 instances and got 8 [ERROR]Counter ID: 609 (SQ_INSTS_VMEM_WR) expected 1 instances and got 8 [ERROR]Counter ID: 610 (SQ_INSTS_VMEM_RD) expected 1 instances and got 8 [ERROR]Counter ID: 611 (SQ_INSTS_VMEM) expected 1 instances and got 8 [ERROR]Counter ID: 612 (SQ_INSTS_SALU) expected 1 instances and got 8 [ERROR]Counter ID: 613 (SQ_INSTS_SMEM) expected 1 instances and got 8 [ERROR]Counter ID: 614 (SQ_INSTS_FLAT) expected 1 instances and got 8 [ERROR]Counter ID: 615 (SQ_INSTS_FLAT_LDS_ONLY) expected 1 instances and got 8 [ERROR]Counter ID: 616 (SQ_INSTS_LDS) expected 1 instances and got 8 [ERROR]Counter ID: 617 (SQ_INSTS_GDS) expected 1 instances and got 8 [ERROR]Counter ID: 618 (SQ_INSTS_EXP_GDS) expected 1 instances and got 8 [ERROR]Counter ID: 619 (SQ_INSTS_BRANCH) expected 1 instances and got 8 [ERROR]Counter ID: 620 (SQ_INSTS_SENDMSG) expected 1 instances and got 8 [ERROR]Counter ID: 621 (SQ_INSTS_VSKIPPED) expected 1 instances and got 8 [ERROR]Counter ID: 622 (SQ_INST_LEVEL_VMEM) expected 1 instances and got 8 [ERROR]Counter ID: 623 (SQ_INST_LEVEL_SMEM) expected 1 instances and got 8 [ERROR]Counter ID: 624 (SQ_INST_LEVEL_LDS) expected 1 instances and got 8 [ERROR]Counter ID: 625 (SQ_VALU_MFMA_BUSY_CYCLES) expected 1 instances and got 8 [ERROR]Counter ID: 626 (SQ_WAVE_CYCLES) expected 1 instances and got 8 [ERROR]Counter ID: 627 (SQ_WAIT_ANY) expected 1 instances and got 8 [ERROR]Counter ID: 628 (SQ_WAIT_INST_ANY) expected 1 instances and got 8 [ERROR]Counter ID: 629 (SQ_ACTIVE_INST_ANY) expected 1 instances and got 8 [ERROR]Counter ID: 630 (SQ_ACTIVE_INST_VMEM) expected 1 instances and got 8 [ERROR]Counter ID: 631 (SQ_ACTIVE_INST_LDS) expected 1 instances and got 8 [ERROR]Counter ID: 632 (SQ_ACTIVE_INST_VALU) expected 1 instances and got 8 [ERROR]Counter ID: 633 (SQ_ACTIVE_INST_SCA) expected 1 instances and got 8 [ERROR]Counter ID: 634 (SQ_ACTIVE_INST_EXP_GDS) expected 1 instances and got 8 [ERROR]Counter ID: 635 (SQ_ACTIVE_INST_MISC) expected 1 instances and got 8 [ERROR]Counter ID: 636 (SQ_ACTIVE_INST_FLAT) expected 1 instances and got 8 [ERROR]Counter ID: 637 (SQ_INST_CYCLES_VMEM_WR) expected 1 instances and got 8 [ERROR]Counter ID: 638 (SQ_INST_CYCLES_VMEM_RD) expected 1 instances and got 8 [ERROR]Counter ID: 639 (SQ_INST_CYCLES_SMEM) expected 1 instances and got 8 [ERROR]Counter ID: 640 (SQ_INST_CYCLES_SALU) expected 1 instances and got 8 [ERROR]Counter ID: 641 (SQ_THREAD_CYCLES_VALU) expected 1 instances and got 8 [ERROR]Counter ID: 642 (SQ_IFETCH) expected 1 instances and got 8 [ERROR]Counter ID: 643 (SQ_IFETCH_LEVEL) expected 1 instances and got 8 [ERROR]Counter ID: 644 (SQ_LDS_BANK_CONFLICT) expected 1 instances and got 8 [ERROR]Counter ID: 645 (SQ_LDS_ADDR_CONFLICT) expected 1 instances and got 8 [ERROR]Counter ID: 646 (SQ_LDS_UNALIGNED_STALL) expected 1 instances and got 8 [ERROR]Counter ID: 647 (SQ_LDS_MEM_VIOLATIONS) expected 1 instances and got 8 [ERROR]Counter ID: 648 (SQ_LDS_ATOMIC_RETURN) expected 1 instances and got 8 [ERROR]Counter ID: 649 (SQ_LDS_IDX_ACTIVE) expected 1 instances and got 8 [ERROR]Counter ID: 650 (SQ_ACCUM_PREV_HIRES) expected 1 instances and got 8 [ERROR]Counter ID: 651 (SQ_WAVES_RESTORED) expected 1 instances and got 8 [ERROR]Counter ID: 652 (SQ_WAVES_SAVED) expected 1 instances and got 8 [ERROR]Counter ID: 653 (SQ_INSTS_SMEM_NORM) expected 1 instances and got 8 [ERROR]Counter ID: 654 (SQC_DCACHE_INPUT_VALID_READYB) expected 1 instances and got 8 [ERROR]Counter ID: 655 (SQC_TC_REQ) expected 1 instances and got 8 [ERROR]Counter ID: 656 (SQC_TC_INST_REQ) expected 1 instances and got 8 [ERROR]Counter ID: 657 (SQC_TC_DATA_READ_REQ) expected 1 instances and got 8 [ERROR]Counter ID: 658 (SQC_TC_DATA_WRITE_REQ) expected 1 instances and got 8 [ERROR]Counter ID: 659 (SQC_TC_DATA_ATOMIC_REQ) expected 1 instances and got 8 [ERROR]Counter ID: 660 (SQC_TC_STALL) expected 1 instances and got 8 [ERROR]Counter ID: 661 (SQC_ICACHE_REQ) expected 1 instances and got 8 [ERROR]Counter ID: 662 (SQC_ICACHE_HITS) expected 1 instances and got 8 [ERROR]Counter ID: 663 (SQC_ICACHE_MISSES) expected 1 instances and got 8 [ERROR]Counter ID: 664 (SQC_ICACHE_MISSES_DUPLICATE) expected 1 instances and got 8 [ERROR]Counter ID: 665 (SQC_DCACHE_REQ) expected 1 instances and got 8 [ERROR]Counter ID: 666 (SQC_DCACHE_HITS) expected 1 instances and got 8 [ERROR]Counter ID: 667 (SQC_DCACHE_MISSES) expected 1 instances and got 8 [ERROR]Counter ID: 668 (SQC_DCACHE_MISSES_DUPLICATE) expected 1 instances and got 8 [ERROR]Counter ID: 669 (SQC_DCACHE_ATOMIC) expected 1 instances and got 8 [ERROR]Counter ID: 670 (SQC_DCACHE_REQ_READ_1) expected 1 instances and got 8 [ERROR]Counter ID: 671 (SQC_DCACHE_REQ_READ_2) expected 1 instances and got 8 [ERROR]Counter ID: 672 (SQC_DCACHE_REQ_READ_4) expected 1 instances and got 8 [ERROR]Counter ID: 673 (SQC_DCACHE_REQ_READ_8) expected 1 instances and got 8 [ERROR]Counter ID: 674 (SQC_DCACHE_REQ_READ_16) expected 1 instances and got 8 [ERROR]Counter ID: 675 (TA_TA_BUSY) expected 16 instances and got 128 [ERROR]Counter ID: 676 (TA_TOTAL_WAVEFRONTS) expected 16 instances and got 128 [ERROR]Counter ID: 677 (TA_BUFFER_WAVEFRONTS) expected 16 instances and got 128 [ERROR]Counter ID: 678 (TA_BUFFER_READ_WAVEFRONTS) expected 16 instances and got 128 [ERROR]Counter ID: 679 (TA_BUFFER_WRITE_WAVEFRONTS) expected 16 instances and got 128 [ERROR]Counter ID: 680 (TA_BUFFER_ATOMIC_WAVEFRONTS) expected 16 instances and got 128 [ERROR]Counter ID: 681 (TA_BUFFER_TOTAL_CYCLES) expected 16 instances and got 128 [ERROR]Counter ID: 682 (TA_BUFFER_COALESCED_READ_CYCLES) expected 16 instances and got 128 [ERROR]Counter ID: 683 (TA_BUFFER_COALESCED_WRITE_CYCLES) expected 16 instances and got 128 [ERROR]Counter ID: 684 (TA_ADDR_STALLED_BY_TC_CYCLES) expected 16 instances and got 128 [ERROR]Counter ID: 685 (TA_ADDR_STALLED_BY_TD_CYCLES) expected 16 instances and got 128 [ERROR]Counter ID: 686 (TA_DATA_STALLED_BY_TC_CYCLES) expected 16 instances and got 128 [ERROR]Counter ID: 687 (TA_FLAT_WAVEFRONTS) expected 16 instances and got 128 [ERROR]Counter ID: 688 (TA_FLAT_READ_WAVEFRONTS) expected 16 instances and got 128 [ERROR]Counter ID: 689 (TA_FLAT_WRITE_WAVEFRONTS) expected 16 instances and got 128 [ERROR]Counter ID: 690 (TA_FLAT_ATOMIC_WAVEFRONTS) expected 16 instances and got 128 [ERROR]Counter ID: 691 (TD_TD_BUSY) expected 16 instances and got 128 [ERROR]Counter ID: 692 (TD_TC_STALL) expected 16 instances and got 128 [ERROR]Counter ID: 693 (TD_SPI_STALL) expected 16 instances and got 128 [ERROR]Counter ID: 694 (TD_LOAD_WAVEFRONT) expected 16 instances and got 128 [ERROR]Counter ID: 695 (TD_ATOMIC_WAVEFRONT) expected 16 instances and got 128 [ERROR]Counter ID: 696 (TD_STORE_WAVEFRONT) expected 16 instances and got 128 [ERROR]Counter ID: 697 (TD_COALESCABLE_WAVEFRONT) expected 16 instances and got 128 [ERROR]Counter ID: 698 (TCP_GATE_EN1) expected 16 instances and got 128 [ERROR]Counter ID: 699 (TCP_GATE_EN2) expected 16 instances and got 128 [ERROR]Counter ID: 700 (TCP_TD_TCP_STALL_CYCLES) expected 16 instances and got 128 [ERROR]Counter ID: 701 (TCP_TCR_TCP_STALL_CYCLES) expected 16 instances and got 128 [ERROR]Counter ID: 702 (TCP_READ_TAGCONFLICT_STALL_CYCLES) expected 16 instances and got 128 [ERROR]Counter ID: 703 (TCP_WRITE_TAGCONFLICT_STALL_CYCLES) expected 16 instances and got 128 [ERROR]Counter ID: 704 (TCP_ATOMIC_TAGCONFLICT_STALL_CYCLES) expected 16 instances and got 128 [ERROR]Counter ID: 705 (TCP_PENDING_STALL_CYCLES) expected 16 instances and got 128 [ERROR]Counter ID: 706 (TCP_TA_TCP_STATE_READ) expected 16 instances and got 128 [ERROR]Counter ID: 707 (TCP_VOLATILE) expected 16 instances and got 128 [ERROR]Counter ID: 708 (TCP_TOTAL_ACCESSES) expected 16 instances and got 128 [ERROR]Counter ID: 709 (TCP_TOTAL_READ) expected 16 instances and got 128 [ERROR]Counter ID: 710 (TCP_TOTAL_WRITE) expected 16 instances and got 128 [ERROR]Counter ID: 711 (TCP_TOTAL_ATOMIC_WITH_RET) expected 16 instances and got 128 [ERROR]Counter ID: 712 (TCP_TOTAL_ATOMIC_WITHOUT_RET) expected 16 instances and got 128 [ERROR]Counter ID: 713 (TCP_TOTAL_WRITEBACK_INVALIDATES) expected 16 instances and got 128 [ERROR]Counter ID: 714 (TCP_UTCL1_REQUEST) expected 16 instances and got 128 [ERROR]Counter ID: 715 (TCP_UTCL1_TRANSLATION_MISS) expected 16 instances and got 128 [ERROR]Counter ID: 716 (TCP_UTCL1_TRANSLATION_HIT) expected 16 instances and got 128 [ERROR]Counter ID: 717 (TCP_UTCL1_PERMISSION_MISS) expected 16 instances and got 128 [ERROR]Counter ID: 718 (TCP_TOTAL_CACHE_ACCESSES) expected 16 instances and got 128 [ERROR]Counter ID: 719 (TCP_TCP_LATENCY) expected 16 instances and got 128 [ERROR]Counter ID: 720 (TCP_TCC_READ_REQ_LATENCY) expected 16 instances and got 128 [ERROR]Counter ID: 721 (TCP_TCC_WRITE_REQ_LATENCY) expected 16 instances and got 128 [ERROR]Counter ID: 722 (TCP_TCC_READ_REQ) expected 16 instances and got 128 [ERROR]Counter ID: 723 (TCP_TCC_WRITE_REQ) expected 16 instances and got 128 [ERROR]Counter ID: 724 (TCP_TCC_ATOMIC_WITH_RET_REQ) expected 16 instances and got 128 [ERROR]Counter ID: 725 (TCP_TCC_ATOMIC_WITHOUT_RET_REQ) expected 16 instances and got 128 [ERROR]Counter ID: 726 (TCP_TCC_NC_READ_REQ) expected 16 instances and got 128 [ERROR]Counter ID: 727 (TCP_TCC_NC_WRITE_REQ) expected 16 instances and got 128 [ERROR]Counter ID: 728 (TCP_TCC_NC_ATOMIC_REQ) expected 16 instances and got 128 [ERROR]Counter ID: 729 (TCP_TCC_UC_READ_REQ) expected 16 instances and got 128 [ERROR]Counter ID: 730 (TCP_TCC_UC_WRITE_REQ) expected 16 instances and got 128 [ERROR]Counter ID: 731 (TCP_TCC_UC_ATOMIC_REQ) expected 16 instances and got 128 [ERROR]Counter ID: 732 (TCP_TCC_CC_READ_REQ) expected 16 instances and got 128 [ERROR]Counter ID: 733 (TCP_TCC_CC_WRITE_REQ) expected 16 instances and got 128 [ERROR]Counter ID: 734 (TCP_TCC_CC_ATOMIC_REQ) expected 16 instances and got 128 [ERROR]Counter ID: 735 (TCP_TCC_RW_READ_REQ) expected 16 instances and got 128 [ERROR]Counter ID: 736 (TCP_TCC_RW_WRITE_REQ) expected 16 instances and got 128 [ERROR]Counter ID: 737 (TCP_TCC_RW_ATOMIC_REQ) expected 16 instances and got 128 Counter ID: 738 (TCA_CYCLE) expected 32 instances and got 32 Counter ID: 739 (TCA_BUSY) expected 32 instances and got 32 Counter ID: 740 (TCC_CYCLE) expected 32 instances and got 32 Counter ID: 741 (TCC_BUSY) expected 32 instances and got 32 Counter ID: 742 (TCC_REQ) expected 32 instances and got 32 Counter ID: 743 (TCC_STREAMING_REQ) expected 32 instances and got 32 Counter ID: 744 (TCC_NC_REQ) expected 32 instances and got 32 Counter ID: 745 (TCC_UC_REQ) expected 32 instances and got 32 Counter ID: 746 (TCC_CC_REQ) expected 32 instances and got 32 Counter ID: 747 (TCC_RW_REQ) expected 32 instances and got 32 Counter ID: 748 (TCC_PROBE) expected 32 instances and got 32 Counter ID: 749 (TCC_PROBE_ALL) expected 32 instances and got 32 Counter ID: 750 (TCC_READ) expected 32 instances and got 32 Counter ID: 751 (TCC_WRITE) expected 32 instances and got 32 Counter ID: 752 (TCC_ATOMIC) expected 32 instances and got 32 Counter ID: 753 (TCC_HIT) expected 32 instances and got 32 Counter ID: 754 (TCC_MISS) expected 32 instances and got 32 Counter ID: 755 (TCC_WRITEBACK) expected 32 instances and got 32 Counter ID: 756 (TCC_EA_WRREQ) expected 32 instances and got 32 Counter ID: 757 (TCC_EA_WRREQ_64B) expected 32 instances and got 32 Counter ID: 758 (TCC_EA_WR_UNCACHED_32B) expected 32 instances and got 32 Counter ID: 759 (TCC_EA_WRREQ_STALL) expected 32 instances and got 32 Counter ID: 760 (TCC_EA_WRREQ_IO_CREDIT_STALL) expected 32 instances and got 32 Counter ID: 761 (TCC_EA_WRREQ_GMI_CREDIT_STALL) expected 32 instances and got 32 Counter ID: 762 (TCC_EA_WRREQ_DRAM_CREDIT_STALL) expected 32 instances and got 32 Counter ID: 763 (TCC_TOO_MANY_EA_WRREQS_STALL) expected 32 instances and got 32 Counter ID: 764 (TCC_EA_WRREQ_LEVEL) expected 32 instances and got 32 Counter ID: 765 (TCC_EA_ATOMIC) expected 32 instances and got 32 Counter ID: 766 (TCC_EA_ATOMIC_LEVEL) expected 32 instances and got 32 Counter ID: 767 (TCC_EA_RDREQ) expected 32 instances and got 32 Counter ID: 768 (TCC_EA_RDREQ_32B) expected 32 instances and got 32 Counter ID: 769 (TCC_EA_RD_UNCACHED_32B) expected 32 instances and got 32 Counter ID: 770 (TCC_EA_RDREQ_IO_CREDIT_STALL) expected 32 instances and got 32 Counter ID: 771 (TCC_EA_RDREQ_GMI_CREDIT_STALL) expected 32 instances and got 32 Counter ID: 772 (TCC_EA_RDREQ_DRAM_CREDIT_STALL) expected 32 instances and got 32 Counter ID: 773 (TCC_EA_RDREQ_LEVEL) expected 32 instances and got 32 Counter ID: 774 (TCC_TAG_STALL) expected 32 instances and got 32 Counter ID: 775 (TCC_NORMAL_WRITEBACK) expected 32 instances and got 32 Counter ID: 776 (TCC_ALL_TC_OP_WB_WRITEBACK) expected 32 instances and got 32 Counter ID: 777 (TCC_NORMAL_EVICT) expected 32 instances and got 32 Counter ID: 778 (TCC_ALL_TC_OP_INV_EVICT) expected 32 instances and got 32 Counter ID: 779 (TCC_EA_RDREQ_DRAM) expected 32 instances and got 32 Counter ID: 780 (TCC_EA_WRREQ_DRAM) expected 32 instances and got 32 [ERROR]Counter ID: 1893 (MeanOccupancyPerCU) expected 1 instances and got 8 [ERROR]Counter ID: 1894 (MeanOccupancyPerActiveCU) expected 1 instances and got 8 [ERROR]Counter ID: 1895 (TA_BUSY_avr) expected 16 instances and got 1 [ERROR]Counter ID: 1896 (TA_BUSY_max) expected 16 instances and got 1 [ERROR]Counter ID: 1897 (TA_BUSY_min) expected 16 instances and got 1 [ERROR]Counter ID: 1898 (TA_TA_BUSY_sum) expected 16 instances and got 1 [ERROR]Counter ID: 1899 (TA_TOTAL_WAVEFRONTS_sum) expected 16 instances and got 1 [ERROR]Counter ID: 1900 (TA_ADDR_STALLED_BY_TC_CYCLES_sum) expected 16 instances and got 1 [ERROR]Counter ID: 1901 (TA_ADDR_STALLED_BY_TD_CYCLES_sum) expected 16 instances and got 1 [ERROR]Counter ID: 1902 (TA_DATA_STALLED_BY_TC_CYCLES_sum) expected 16 instances and got 1 [ERROR]Counter ID: 1903 (TA_FLAT_WAVEFRONTS_sum) expected 16 instances and got 1 [ERROR]Counter ID: 1904 (TA_FLAT_READ_WAVEFRONTS_sum) expected 16 instances and got 1 [ERROR]Counter ID: 1905 (TA_FLAT_WRITE_WAVEFRONTS_sum) expected 16 instances and got 1 [ERROR]Counter ID: 1906 (TA_FLAT_ATOMIC_WAVEFRONTS_sum) expected 16 instances and got 1 [ERROR]Counter ID: 1907 (TA_BUFFER_WAVEFRONTS_sum) expected 16 instances and got 1 [ERROR]Counter ID: 1908 (TA_BUFFER_READ_WAVEFRONTS_sum) expected 16 instances and got 1 [ERROR]Counter ID: 1909 (TA_BUFFER_WRITE_WAVEFRONTS_sum) expected 16 instances and got 1 [ERROR]Counter ID: 1910 (TA_BUFFER_ATOMIC_WAVEFRONTS_sum) expected 16 instances and got 1 [ERROR]Counter ID: 1911 (TA_BUFFER_TOTAL_CYCLES_sum) expected 16 instances and got 1 [ERROR]Counter ID: 1912 (TA_BUFFER_COALESCED_READ_CYCLES_sum) expected 16 instances and got 1 [ERROR]Counter ID: 1913 (TA_BUFFER_COALESCED_WRITE_CYCLES_sum) expected 16 instances and got 1 [ERROR]Counter ID: 1914 (TD_TD_BUSY_sum) expected 16 instances and got 1 [ERROR]Counter ID: 1915 (TD_TC_STALL_sum) expected 16 instances and got 1 [ERROR]Counter ID: 1916 (TD_LOAD_WAVEFRONT_sum) expected 16 instances and got 1 [ERROR]Counter ID: 1917 (TD_ATOMIC_WAVEFRONT_sum) expected 16 instances and got 1 [ERROR]Counter ID: 1918 (TD_STORE_WAVEFRONT_sum) expected 16 instances and got 1 [ERROR]Counter ID: 1919 (TD_COALESCABLE_WAVEFRONT_sum) expected 16 instances and got 1 [ERROR]Counter ID: 1920 (TD_SPI_STALL_sum) expected 16 instances and got 1 [ERROR]Counter ID: 1921 (TCP_GATE_EN1_sum) expected 16 instances and got 1 [ERROR]Counter ID: 1922 (TCP_GATE_EN2_sum) expected 16 instances and got 1 [ERROR]Counter ID: 1923 (TCP_TD_TCP_STALL_CYCLES_sum) expected 16 instances and got 1 [ERROR]Counter ID: 1924 (TCP_TCR_TCP_STALL_CYCLES_sum) expected 16 instances and got 1 [ERROR]Counter ID: 1925 (TCP_READ_TAGCONFLICT_STALL_CYCLES_sum) expected 16 instances and got 1 [ERROR]Counter ID: 1926 (TCP_WRITE_TAGCONFLICT_STALL_CYCLES_sum) expected 16 instances and got 1 [ERROR]Counter ID: 1927 (TCP_ATOMIC_TAGCONFLICT_STALL_CYCLES_sum) expected 16 instances and got 1 [ERROR]Counter ID: 1928 (TCP_VOLATILE_sum) expected 16 instances and got 1 [ERROR]Counter ID: 1929 (TCP_TOTAL_ACCESSES_sum) expected 16 instances and got 1 [ERROR]Counter ID: 1930 (TCP_TOTAL_READ_sum) expected 16 instances and got 1 [ERROR]Counter ID: 1931 (TCP_TOTAL_WRITE_sum) expected 16 instances and got 1 [ERROR]Counter ID: 1932 (TCP_TOTAL_ATOMIC_WITH_RET_sum) expected 16 instances and got 1 [ERROR]Counter ID: 1933 (TCP_TOTAL_ATOMIC_WITHOUT_RET_sum) expected 16 instances and got 1 [ERROR]Counter ID: 1934 (TCP_TOTAL_WRITEBACK_INVALIDATES_sum) expected 16 instances and got 1 [ERROR]Counter ID: 1935 (TCP_UTCL1_REQUEST_sum) expected 16 instances and got 1 [ERROR]Counter ID: 1936 (TCP_UTCL1_TRANSLATION_MISS_sum) expected 16 instances and got 1 [ERROR]Counter ID: 1937 (TCP_UTCL1_TRANSLATION_HIT_sum) expected 16 instances and got 1 [ERROR]Counter ID: 1938 (TCP_UTCL1_PERMISSION_MISS_sum) expected 16 instances and got 1 [ERROR]Counter ID: 1939 (TCP_TOTAL_CACHE_ACCESSES_sum) expected 16 instances and got 1 [ERROR]Counter ID: 1940 (TCP_TCP_LATENCY_sum) expected 16 instances and got 1 [ERROR]Counter ID: 1941 (TCP_TA_TCP_STATE_READ_sum) expected 16 instances and got 1 [ERROR]Counter ID: 1942 (TCP_TCC_READ_REQ_LATENCY_sum) expected 16 instances and got 1 [ERROR]Counter ID: 1943 (TCP_TCC_WRITE_REQ_LATENCY_sum) expected 16 instances and got 1 [ERROR]Counter ID: 1944 (TCP_TCC_READ_REQ_sum) expected 16 instances and got 1 [ERROR]Counter ID: 1945 (TCP_TCC_WRITE_REQ_sum) expected 16 instances and got 1 [ERROR]Counter ID: 1946 (TCP_TCC_ATOMIC_WITH_RET_REQ_sum) expected 16 instances and got 1 [ERROR]Counter ID: 1947 (TCP_TCC_ATOMIC_WITHOUT_RET_REQ_sum) expected 16 instances and got 1 [ERROR]Counter ID: 1948 (TCP_TCC_NC_READ_REQ_sum) expected 16 instances and got 1 [ERROR]Counter ID: 1949 (TCP_TCC_NC_WRITE_REQ_sum) expected 16 instances and got 1 [ERROR]Counter ID: 1950 (TCP_TCC_NC_ATOMIC_REQ_sum) expected 16 instances and got 1 [ERROR]Counter ID: 1951 (TCP_TCC_UC_READ_REQ_sum) expected 16 instances and got 1 [ERROR]Counter ID: 1952 (TCP_TCC_UC_WRITE_REQ_sum) expected 16 instances and got 1 [ERROR]Counter ID: 1953 (TCP_TCC_UC_ATOMIC_REQ_sum) expected 16 instances and got 1 [ERROR]Counter ID: 1954 (TCP_TCC_CC_READ_REQ_sum) expected 16 instances and got 1 [ERROR]Counter ID: 1955 (TCP_TCC_CC_WRITE_REQ_sum) expected 16 instances and got 1 [ERROR]Counter ID: 1956 (TCP_TCC_CC_ATOMIC_REQ_sum) expected 16 instances and got 1 [ERROR]Counter ID: 1957 (TCP_TCC_RW_READ_REQ_sum) expected 16 instances and got 1 [ERROR]Counter ID: 1958 (TCP_TCC_RW_WRITE_REQ_sum) expected 16 instances and got 1 [ERROR]Counter ID: 1959 (TCP_TCC_RW_ATOMIC_REQ_sum) expected 16 instances and got 1 [ERROR]Counter ID: 1960 (TCP_PENDING_STALL_CYCLES_sum) expected 16 instances and got 1 [ERROR]Counter ID: 1961 (TCA_CYCLE_sum) expected 32 instances and got 1 [ERROR]Counter ID: 1962 (TCA_BUSY_sum) expected 32 instances and got 1 [ERROR]Counter ID: 1963 (TCC_BUSY_avr) expected 32 instances and got 1 [ERROR]Counter ID: 1964 (TCC_WRREQ_STALL_max) expected 32 instances and got 1 [ERROR]Counter ID: 1965 (TCC_CYCLE_sum) expected 32 instances and got 1 [ERROR]Counter ID: 1966 (TCC_BUSY_sum) expected 32 instances and got 1 [ERROR]Counter ID: 1967 (TCC_REQ_sum) expected 32 instances and got 1 [ERROR]Counter ID: 1968 (TCC_STREAMING_REQ_sum) expected 32 instances and got 1 [ERROR]Counter ID: 1969 (TCC_NC_REQ_sum) expected 32 instances and got 1 [ERROR]Counter ID: 1970 (TCC_UC_REQ_sum) expected 32 instances and got 1 [ERROR]Counter ID: 1971 (TCC_CC_REQ_sum) expected 32 instances and got 1 [ERROR]Counter ID: 1972 (TCC_RW_REQ_sum) expected 32 instances and got 1 [ERROR]Counter ID: 1973 (TCC_PROBE_sum) expected 32 instances and got 1 [ERROR]Counter ID: 1974 (TCC_PROBE_ALL_sum) expected 32 instances and got 1 [ERROR]Counter ID: 1975 (TCC_READ_sum) expected 32 instances and got 1 [ERROR]Counter ID: 1976 (TCC_WRITE_sum) expected 32 instances and got 1 [ERROR]Counter ID: 1977 (TCC_ATOMIC_sum) expected 32 instances and got 1 [ERROR]Counter ID: 1978 (TCC_HIT_sum) expected 32 instances and got 1 [ERROR]Counter ID: 1979 (TCC_MISS_sum) expected 32 instances and got 1 [ERROR]Counter ID: 1980 (TCC_WRITEBACK_sum) expected 32 instances and got 1 [ERROR]Counter ID: 1981 (TCC_EA_WRREQ_sum) expected 32 instances and got 1 [ERROR]Counter ID: 1982 (TCC_EA_WRREQ_64B_sum) expected 32 instances and got 1 [ERROR]Counter ID: 1983 (TCC_EA_WR_UNCACHED_32B_sum) expected 32 instances and got 1 [ERROR]Counter ID: 1984 (TCC_EA_WRREQ_STALL_sum) expected 32 instances and got 1 [ERROR]Counter ID: 1985 (TCC_EA_WRREQ_IO_CREDIT_STALL_sum) expected 32 instances and got 1 [ERROR]Counter ID: 1986 (TCC_EA_WRREQ_GMI_CREDIT_STALL_sum) expected 32 instances and got 1 [ERROR]Counter ID: 1987 (TCC_EA_WRREQ_DRAM_CREDIT_STALL_sum) expected 32 instances and got 1 [ERROR]Counter ID: 1988 (TCC_TOO_MANY_EA_WRREQS_STALL_sum) expected 32 instances and got 1 [ERROR]Counter ID: 1989 (TCC_EA_WRREQ_LEVEL_sum) expected 32 instances and got 1 [ERROR]Counter ID: 1990 (TCC_EA_RDREQ_LEVEL_sum) expected 32 instances and got 1 [ERROR]Counter ID: 1991 (TCC_EA_ATOMIC_sum) expected 32 instances and got 1 [ERROR]Counter ID: 1992 (TCC_EA_ATOMIC_LEVEL_sum) expected 32 instances and got 1 [ERROR]Counter ID: 1993 (TCC_EA_RDREQ_sum) expected 32 instances and got 1 [ERROR]Counter ID: 1994 (TCC_EA_RDREQ_32B_sum) expected 32 instances and got 1 [ERROR]Counter ID: 1995 (TCC_EA_RD_UNCACHED_32B_sum) expected 32 instances and got 1 [ERROR]Counter ID: 1996 (TCC_EA_RDREQ_IO_CREDIT_STALL_sum) expected 32 instances and got 1 [ERROR]Counter ID: 1997 (TCC_EA_RDREQ_GMI_CREDIT_STALL_sum) expected 32 instances and got 1 [ERROR]Counter ID: 1998 (TCC_EA_RDREQ_DRAM_CREDIT_STALL_sum) expected 32 instances and got 1 [ERROR]Counter ID: 1999 (TCC_TAG_STALL_sum) expected 32 instances and got 1 [ERROR]Counter ID: 2000 (TCC_NORMAL_WRITEBACK_sum) expected 32 instances and got 1 [ERROR]Counter ID: 2001 (TCC_ALL_TC_OP_WB_WRITEBACK_sum) expected 32 instances and got 1 [ERROR]Counter ID: 2002 (TCC_NORMAL_EVICT_sum) expected 32 instances and got 1 [ERROR]Counter ID: 2003 (TCC_ALL_TC_OP_INV_EVICT_sum) expected 32 instances and got 1 [ERROR]Counter ID: 2004 (TCC_EA_RDREQ_DRAM_sum) expected 32 instances and got 1 [ERROR]Counter ID: 2005 (TCC_EA_WRREQ_DRAM_sum) expected 32 instances and got 1 [ERROR]Counter ID: 2006 (FETCH_SIZE) expected 32 instances and got 1 [ERROR]Counter ID: 2007 (WRITE_SIZE) expected 32 instances and got 1 [ERROR]Counter ID: 2008 (WRITE_REQ_32B) expected 32 instances and got 1 [ERROR]Counter ID: 2009 (CU_OCCUPANCY) expected 1 instances and got 8 Counter ID: 2010 (CU_UTILIZATION) expected 1 instances and got 1 [ERROR]Counter ID: 2011 (TOTAL_16_OPS) expected 1 instances and got 8 [ERROR]Counter ID: 2012 (TOTAL_32_OPS) expected 1 instances and got 8 [ERROR]Counter ID: 2013 (TOTAL_64_OPS) expected 1 instances and got 8 Counter ID: 2014 (AggSysCycles) expected 1 instances and got 1 Counter ID: 2015 (GpuUtil) expected 1 instances and got 1 Counter ID: 2016 (CpUtil) expected 1 instances and got 1 Counter ID: 2017 (SpiUtil) expected 1 instances and got 1 Counter ID: 2018 (TaUtil) expected 1 instances and got 1 Counter ID: 2019 (TcUtil) expected 1 instances and got 1 Counter ID: 2020 (EaUtil) expected 1 instances and got 1 [ERROR]Counter ID: 2021 (InstrFetchLatency) expected 1 instances and got 8 [ERROR]Counter ID: 2022 (WaveOccupancy) expected 1 instances and got 8 [ERROR]Counter ID: 2023 (WaveDuration) expected 1 instances and got 8 [ERROR]Counter ID: 2024 (WaveDepWait) expected 1 instances and got 8 [ERROR]Counter ID: 2025 (WaveIssueWait) expected 1 instances and got 8 [ERROR]Counter ID: 2026 (WaveExec) expected 1 instances and got 8 [ERROR]Counter ID: 2027 (ValuIops) expected 1 instances and got 8 [ERROR]Counter ID: 2028 (MfmaFlops) expected 1 instances and got 8 [ERROR]Counter ID: 2029 (MfmaFlopsF16) expected 1 instances and got 8 [ERROR]Counter ID: 2030 (MfmaFlopsBF16) expected 1 instances and got 8 [ERROR]Counter ID: 2031 (MfmaFlopsF32) expected 1 instances and got 8 [ERROR]Counter ID: 2032 (MfmaFlopsF64) expected 1 instances and got 8 [ERROR]Counter ID: 2033 (ScaPipeIssueUtil) expected 1 instances and got 8 [ERROR]Counter ID: 2034 (ValuPipeIssueUtil) expected 1 instances and got 8 [ERROR]Counter ID: 2035 (VmemPipeIssueUtil) expected 1 instances and got 8 [ERROR]Counter ID: 2036 (MfmaUtil) expected 1 instances and got 8 [ERROR]Counter ID: 2037 (AvgNumActiveThreads) expected 1 instances and got 8 [ERROR]Counter ID: 2038 (VmemLatency) expected 1 instances and got 8 [ERROR]Counter ID: 2039 (SmemLatency) expected 1 instances and got 8 [ERROR]Counter ID: 2040 (LdsUtil) expected 1 instances and got 8 [ERROR]Counter ID: 2041 (LdsPipeIssueUtil) expected 1 instances and got 8 [ERROR]Counter ID: 2042 (LdsLatency) expected 1 instances and got 8 [ERROR]Counter ID: 2043 (LdsBankConflict) expected 1 instances and got 8 [ERROR]Counter ID: 2044 (L1iCacheHitRate) expected 1 instances and got 8 [ERROR]Counter ID: 2045 (sL1dCacheHitRate) expected 1 instances and got 8 [ERROR]Counter ID: 2046 (vL1dBufCoalesceRate) expected 16 instances and got 1 [ERROR]Counter ID: 2047 (vL1dCacheUtil) expected 16 instances and got 1 [ERROR]Counter ID: 2048 (vL1dCacheTcbHitRate) expected 16 instances and got 1 [ERROR]Counter ID: 2049 (vL1dCacheWaveLatency) expected 16 instances and got 1 [ERROR]Counter ID: 2050 (vL1dReadFromL2Latency) expected 16 instances and got 1 [ERROR]Counter ID: 2051 (vL1dWriteToL2Latency) expected 16 instances and got 1 [ERROR]Counter ID: 2052 (vL1dRdTagConfStallRate) expected 16 instances and got 1 [ERROR]Counter ID: 2053 (vL1dWrTagConfStallRate) expected 16 instances and got 1 [ERROR]Counter ID: 2054 (vL1dAtomicTagConfStallRate) expected 16 instances and got 1 [ERROR]Counter ID: 2055 (vL1dMissReqStallRate) expected 16 instances and got 1 [ERROR]Counter ID: 2056 (vL1dDataPendRate) expected 16 instances and got 1 [ERROR]Counter ID: 2057 (vL1dDataRetStallRate) expected 16 instances and got 1 [ERROR]Counter ID: 2058 (L2CacheHitRate) expected 32 instances and got 1 [ERROR]Counter ID: 2059 (L2CacheTagRamStallRate) expected 32 instances and got 1 [ERROR]Counter ID: 2060 (EaRdLatency) expected 32 instances and got 1 [ERROR]Counter ID: 2061 (EaRdIoStallRate) expected 32 instances and got 1 [ERROR]Counter ID: 2062 (EaRdGmiStallRate) expected 32 instances and got 1 [ERROR]Counter ID: 2063 (EaRdDramStallRate) expected 32 instances and got 1 [ERROR]Counter ID: 2064 (EaWrLatency) expected 32 instances and got 1 [ERROR]Counter ID: 2065 (EaWrIoStallRate) expected 32 instances and got 1 [ERROR]Counter ID: 2066 (EaWrGmiStallRate) expected 32 instances and got 1 [ERROR]Counter ID: 2067 (EaWrDramStallRate) expected 32 instances and got 1 [ERROR]Counter ID: 2068 (EaWrStarveRate) expected 32 instances and got 1 [ERROR]Counter ID: 2069 (EaAtomicLatency) expected 32 instances and got 1 [ERROR]Counter ID: 2070 (TCP_TCP_TA_DATA_STALL_CYCLES_sum) expected 16 instances and got 1 [ERROR]Counter ID: 2071 (TCP_TCP_TA_DATA_STALL_CYCLES_max) expected 16 instances and got 1 [ERROR]Counter ID: 2072 (VFetchInsts) expected 16 instances and got 8 [ERROR]Counter ID: 2073 (VWriteInsts) expected 16 instances and got 8 [ERROR]Counter ID: 2074 (FlatVMemInsts) expected 1 instances and got 8 [ERROR]Counter ID: 2075 (LDSInsts) expected 1 instances and got 8 [ERROR]Counter ID: 2076 (FlatLDSInsts) expected 1 instances and got 8 [ERROR]Counter ID: 2077 (VALUUtilization) expected 1 instances and got 8 [ERROR]Counter ID: 2078 (VALUBusy) expected 1 instances and got 8 [ERROR]Counter ID: 2079 (SALUBusy) expected 1 instances and got 8 [ERROR]Counter ID: 2080 (FetchSize) expected 32 instances and got 1 [ERROR]Counter ID: 2081 (WriteSize) expected 32 instances and got 1 [ERROR]Counter ID: 2082 (MemWrites32B) expected 32 instances and got 1 [ERROR]Counter ID: 2083 (L2CacheHit) expected 32 instances and got 1 [ERROR]Counter ID: 2084 (MemUnitStalled) expected 16 instances and got 1 [ERROR]Counter ID: 2085 (WriteUnitStalled) expected 32 instances and got 1 [ERROR]Counter ID: 2086 (LDSBankConflict) expected 1 instances and got 8 * source formatting (clang-format v11) (#225) Co-authored-by: bwelton <bwelton@users.noreply.github.com> * cmake formatting (cmake-format) (#224) Co-authored-by: bwelton <bwelton@users.noreply.github.com> * Minor fixes * source formatting (clang-format v11) (#226) Co-authored-by: bwelton <bwelton@users.noreply.github.com> * Minor test change --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: bwelton <bwelton@users.noreply.github.com>
This commit is contained in:
Benjamin Welton
GitHub
@@ -29,22 +29,44 @@ if(NOT TARGET rocprofiler::rocprofiler)
|
||||
find_package(rocprofiler REQUIRED)
|
||||
endif()
|
||||
|
||||
add_library(counter-collection-client SHARED)
|
||||
target_sources(counter-collection-client PRIVATE client.cpp client.hpp)
|
||||
target_link_libraries(counter-collection-client PRIVATE rocprofiler::rocprofiler)
|
||||
|
||||
add_library(counter-collection-buffer-client SHARED)
|
||||
target_sources(counter-collection-buffer-client PRIVATE client.cpp client.hpp)
|
||||
target_link_libraries(counter-collection-buffer-client PRIVATE rocprofiler::rocprofiler)
|
||||
set_source_files_properties(main.cpp PROPERTIES LANGUAGE HIP)
|
||||
find_package(Threads REQUIRED)
|
||||
add_executable(counter-collection-buffer)
|
||||
target_sources(counter-collection-buffer PRIVATE main.cpp)
|
||||
target_link_libraries(counter-collection-buffer PRIVATE counter-collection-buffer-client
|
||||
Threads::Threads)
|
||||
|
||||
add_executable(counter-collection)
|
||||
target_sources(counter-collection PRIVATE main.cpp)
|
||||
target_link_libraries(counter-collection PRIVATE counter-collection-client
|
||||
Threads::Threads)
|
||||
|
||||
add_test(NAME counter-collection COMMAND $<TARGET_FILE:counter-collection>)
|
||||
add_test(NAME counter-collection-buffer COMMAND $<TARGET_FILE:counter-collection-buffer>)
|
||||
|
||||
set_tests_properties(
|
||||
counter-collection
|
||||
counter-collection-buffer
|
||||
PROPERTIES
|
||||
TIMEOUT
|
||||
300
|
||||
LABELS
|
||||
"samples"
|
||||
ENVIRONMENT
|
||||
"${ROCPROFILER_MEMCHECK_PRELOAD_ENV};HSA_TOOLS_LIB=$<TARGET_FILE:rocprofiler::rocprofiler-library>"
|
||||
FAIL_REGULAR_EXPRESSION
|
||||
"threw an exception")
|
||||
|
||||
add_library(counter-collection-test-counter-client SHARED)
|
||||
target_sources(counter-collection-test-counter-client PRIVATE test_counter_client.cpp
|
||||
client.hpp)
|
||||
target_link_libraries(counter-collection-test-counter-client
|
||||
PRIVATE rocprofiler::rocprofiler)
|
||||
add_executable(counter-collection-test-counters)
|
||||
target_sources(counter-collection-test-counters PRIVATE main.cpp)
|
||||
target_link_libraries(counter-collection-test-counters
|
||||
PRIVATE counter-collection-test-counter-client Threads::Threads)
|
||||
|
||||
add_test(NAME counter-collection-test-counters
|
||||
COMMAND $<TARGET_FILE:counter-collection-test-counters>)
|
||||
|
||||
set_tests_properties(
|
||||
counter-collection-test-counters
|
||||
PROPERTIES
|
||||
TIMEOUT
|
||||
300
|
||||
|
||||
@@ -22,8 +22,13 @@
|
||||
|
||||
#include "client.hpp"
|
||||
|
||||
#include <fstream>
|
||||
#include <iostream>
|
||||
#include <mutex>
|
||||
#include <set>
|
||||
#include <sstream> // std::stringstream
|
||||
#include <shared_mutex>
|
||||
#include <sstream>
|
||||
#include <unordered_map>
|
||||
#include <vector>
|
||||
|
||||
#include <rocprofiler/registration.h>
|
||||
@@ -60,101 +65,163 @@ get_client_ctx()
|
||||
return ctx;
|
||||
}
|
||||
|
||||
rocprofiler_buffer_id_t&
|
||||
get_buffer()
|
||||
{
|
||||
static rocprofiler_buffer_id_t buf = {};
|
||||
return buf;
|
||||
}
|
||||
|
||||
std::ostream*
|
||||
get_output_stream()
|
||||
{
|
||||
static std::ostream* isTerm = []() -> std::ostream* {
|
||||
if(auto* outfile = getenv("ROCPROFILER_SAMPLE_OUTPUT_FILE"))
|
||||
{
|
||||
if(outfile == "stdout")
|
||||
return static_cast<std::ostream*>(&std::cout);
|
||||
else if(outfile == "stderr")
|
||||
return &std::cerr;
|
||||
}
|
||||
return nullptr;
|
||||
}();
|
||||
static std::unique_ptr<std::ofstream> stream;
|
||||
|
||||
if(isTerm) return isTerm;
|
||||
if(stream) return stream.get();
|
||||
std::string filename = "counter_collection.log";
|
||||
if(auto* outfile = getenv("ROCPROFILER_SAMPLE_OUTPUT_FILE"))
|
||||
{
|
||||
filename = outfile;
|
||||
}
|
||||
stream = std::make_unique<std::ofstream>(filename);
|
||||
return stream.get();
|
||||
}
|
||||
|
||||
void
|
||||
test_callback(rocprofiler_queue_id_t queue_id,
|
||||
rocprofiler_agent_t agent_id,
|
||||
rocprofiler_correlation_id_t corr_id,
|
||||
const hsa_kernel_dispatch_packet_t*,
|
||||
void*,
|
||||
rocprofiler_record_counter_t* out_counters,
|
||||
size_t out_size,
|
||||
rocprofiler_profile_config_id_t)
|
||||
buffered_callback(rocprofiler_context_id_t,
|
||||
rocprofiler_buffer_id_t,
|
||||
rocprofiler_record_header_t** headers,
|
||||
size_t num_headers,
|
||||
void*,
|
||||
uint64_t)
|
||||
{
|
||||
static int enter_count = 0;
|
||||
enter_count++;
|
||||
// Limit output to avoid massive log size
|
||||
if(enter_count % 100 != 0) return;
|
||||
|
||||
std::stringstream ss;
|
||||
for(size_t i = 0; i < out_size; i++)
|
||||
for(size_t i = 0; i < num_headers; ++i)
|
||||
{
|
||||
ss << "(Id: " << out_counters[i].id << " Value [D]: " << out_counters[i].derived_counter
|
||||
<< ", Value [I]: " << out_counters[i].hw_counter << "),";
|
||||
auto* header = headers[i];
|
||||
if(header->category == ROCPROFILER_BUFFER_CATEGORY_COUNTERS && header->kind == 0)
|
||||
{
|
||||
auto* record = static_cast<rocprofiler_record_counter_t*>(header->payload);
|
||||
ss << "(Id: " << record->id << " Value [D]: " << record->counter_value
|
||||
<< " Corr_Id: " << record->corr_id.internal << "),";
|
||||
}
|
||||
}
|
||||
// Callback containing counter data.
|
||||
std::clog << "[" << __FUNCTION__ << "] " << queue_id.handle << " | " << agent_id.id.handle
|
||||
<< " | " << corr_id.internal << "|" << ss.str() << "\n";
|
||||
|
||||
*get_output_stream() << "[" << __FUNCTION__ << "] " << ss.str() << "\n";
|
||||
}
|
||||
|
||||
void
|
||||
dispatch_callback(rocprofiler_queue_id_t queue_id,
|
||||
const rocprofiler_agent_t* agent,
|
||||
rocprofiler_correlation_id_t correlation_id,
|
||||
const hsa_kernel_dispatch_packet_t* dispatch_packet,
|
||||
void* callback_data_args,
|
||||
rocprofiler_profile_config_id_t* config)
|
||||
{
|
||||
/**
|
||||
* This simple example uses the same profile counter set for all agents.
|
||||
* We store this in a cache to prevent constructing many identical profile counter
|
||||
* sets. We first check the cache to see if we have already constructed a counter"
|
||||
* set for the agent. If we have, return it. Otherwise, construct a new profile counter
|
||||
* set.
|
||||
*/
|
||||
static std::shared_mutex m_mutex = {};
|
||||
static std::unordered_map<uint64_t, rocprofiler_profile_config_id_t> profile_cache = {};
|
||||
|
||||
auto search_cache = [&]() {
|
||||
if(auto pos = profile_cache.find(agent->id.handle); pos != profile_cache.end())
|
||||
{
|
||||
*config = pos->second;
|
||||
return true;
|
||||
}
|
||||
return false;
|
||||
};
|
||||
|
||||
{
|
||||
auto rlock = std::shared_lock{m_mutex};
|
||||
if(search_cache()) return;
|
||||
}
|
||||
|
||||
auto wlock = std::unique_lock{m_mutex};
|
||||
if(search_cache()) return;
|
||||
|
||||
std::set<std::string> counters_to_collect = {"SQ_WAVES"};
|
||||
std::vector<rocprofiler_counter_id_t> gpu_counters;
|
||||
ROCPROFILER_CALL(
|
||||
rocprofiler_iterate_agent_supported_counters(
|
||||
*agent,
|
||||
[](rocprofiler_counter_id_t* counters, size_t num_counters, void* user_data) {
|
||||
std::vector<rocprofiler_counter_id_t>* vec =
|
||||
static_cast<std::vector<rocprofiler_counter_id_t>*>(user_data);
|
||||
for(size_t i = 0; i < num_counters; i++)
|
||||
{
|
||||
vec->push_back(counters[i]);
|
||||
}
|
||||
return ROCPROFILER_STATUS_SUCCESS;
|
||||
},
|
||||
static_cast<void*>(&gpu_counters)),
|
||||
"Could not fetch supported counters");
|
||||
|
||||
std::vector<rocprofiler_counter_id_t> collect_counters;
|
||||
for(auto& counter : gpu_counters)
|
||||
{
|
||||
const char* name;
|
||||
size_t size;
|
||||
ROCPROFILER_CALL(rocprofiler_query_counter_name(counter, &name, &size),
|
||||
"Could not query name");
|
||||
if(counters_to_collect.count(std::string(name)) > 0)
|
||||
{
|
||||
std::clog << "Counter: " << counter.handle << " " << name << "\n";
|
||||
collect_counters.push_back(counter);
|
||||
}
|
||||
}
|
||||
|
||||
rocprofiler_profile_config_id_t profile;
|
||||
ROCPROFILER_CALL(rocprofiler_create_profile_config(
|
||||
*agent, collect_counters.data(), collect_counters.size(), &profile),
|
||||
"Could not construct profile cfg");
|
||||
|
||||
profile_cache.emplace(agent->id.handle, profile);
|
||||
*config = profile;
|
||||
}
|
||||
|
||||
int
|
||||
tool_init(rocprofiler_client_finalize_t, void*)
|
||||
{
|
||||
std::set<std::string> counters_to_collect = {"SQ_WAVES"};
|
||||
|
||||
std::vector<rocprofiler_agent_t> gpu_agents;
|
||||
auto agent_query = [](const rocprofiler_agent_t** agents, size_t num_agents, void* user_data) {
|
||||
std::vector<rocprofiler_agent_t>* vec =
|
||||
static_cast<std::vector<rocprofiler_agent_t>*>(user_data);
|
||||
for(size_t i = 0; i < num_agents; i++)
|
||||
{
|
||||
const rocprofiler_agent_t* agent = agents[i];
|
||||
if(agent->type == ROCPROFILER_AGENT_TYPE_GPU)
|
||||
{
|
||||
vec->push_back(*agent);
|
||||
}
|
||||
}
|
||||
return ROCPROFILER_STATUS_SUCCESS;
|
||||
};
|
||||
|
||||
ROCPROFILER_CALL(rocprofiler_query_available_agents(
|
||||
agent_query, sizeof(rocprofiler_agent_t), static_cast<void*>(&gpu_agents)),
|
||||
"Could not query agents");
|
||||
|
||||
ROCPROFILER_CALL(rocprofiler_create_context(&get_client_ctx()), "context creation failed");
|
||||
|
||||
std::vector<rocprofiler_profile_config_id_t> profile_configs;
|
||||
ROCPROFILER_CALL(rocprofiler_create_buffer(get_client_ctx(),
|
||||
4096,
|
||||
2048,
|
||||
ROCPROFILER_BUFFER_POLICY_LOSSLESS,
|
||||
buffered_callback,
|
||||
nullptr,
|
||||
&get_buffer()),
|
||||
"buffer creation failed");
|
||||
|
||||
for(auto& agent : gpu_agents)
|
||||
{
|
||||
std::vector<rocprofiler_counter_id_t> collect_counters;
|
||||
std::vector<rocprofiler_counter_id_t> gpu_counters;
|
||||
std::clog << agent.name << "\n";
|
||||
ROCPROFILER_CALL(
|
||||
rocprofiler_iterate_agent_supported_counters(
|
||||
agent,
|
||||
[](rocprofiler_counter_id_t* counters, size_t num_counters, void* user_data) {
|
||||
std::vector<rocprofiler_counter_id_t>* vec =
|
||||
static_cast<std::vector<rocprofiler_counter_id_t>*>(user_data);
|
||||
for(size_t i = 0; i < num_counters; i++)
|
||||
{
|
||||
vec->push_back(counters[i]);
|
||||
}
|
||||
return ROCPROFILER_STATUS_SUCCESS;
|
||||
},
|
||||
static_cast<void*>(&gpu_counters)),
|
||||
"Could not fetch supported counters");
|
||||
|
||||
for(auto& counter : gpu_counters)
|
||||
{
|
||||
const char* name;
|
||||
size_t size;
|
||||
ROCPROFILER_CALL(rocprofiler_query_counter_name(counter, &name, &size),
|
||||
"Could not query name");
|
||||
if(counters_to_collect.count(std::string(name)) > 0)
|
||||
{
|
||||
std::clog << "Counter: " << counter.handle << " " << name << "\n";
|
||||
collect_counters.push_back(counter);
|
||||
}
|
||||
}
|
||||
rocprofiler_profile_config_id_t profile;
|
||||
ROCPROFILER_CALL(rocprofiler_create_profile_config(
|
||||
agent, collect_counters.data(), collect_counters.size(), &profile),
|
||||
"Could not construct profile cfg");
|
||||
ROCPROFILER_CALL(rocprofiler_configure_dispatch_profile_counting_service(
|
||||
get_client_ctx(), profile, test_callback, nullptr),
|
||||
"Could not setup dispatch service");
|
||||
profile_configs.push_back(profile);
|
||||
}
|
||||
auto client_thread = rocprofiler_callback_thread_t{};
|
||||
ROCPROFILER_CALL(rocprofiler_create_callback_thread(&client_thread),
|
||||
"failure creating callback thread");
|
||||
get_output_stream();
|
||||
ROCPROFILER_CALL(rocprofiler_assign_callback_thread(get_buffer(), client_thread),
|
||||
"failed to assign thread for buffer");
|
||||
ROCPROFILER_CALL(rocprofiler_configure_buffered_dispatch_profile_counting_service(
|
||||
get_client_ctx(), get_buffer(), dispatch_callback, nullptr),
|
||||
"Could not setup buffered service");
|
||||
rocprofiler_start_context(get_client_ctx());
|
||||
|
||||
// no errors
|
||||
|
||||
@@ -0,0 +1,277 @@
|
||||
#include "client.hpp"
|
||||
|
||||
#include <unistd.h>
|
||||
#include <map>
|
||||
#include <mutex>
|
||||
#include <optional>
|
||||
#include <set>
|
||||
#include <shared_mutex>
|
||||
#include <sstream>
|
||||
#include <unordered_map>
|
||||
#include <vector>
|
||||
|
||||
#include <rocprofiler/registration.h>
|
||||
#include <rocprofiler/rocprofiler.h>
|
||||
|
||||
/**
|
||||
* Tests the collection of all counters on the agent the test is run on.
|
||||
*/
|
||||
|
||||
#define ROCPROFILER_CALL(result, msg) \
|
||||
{ \
|
||||
rocprofiler_status_t CHECKSTATUS = result; \
|
||||
if(CHECKSTATUS != ROCPROFILER_STATUS_SUCCESS) \
|
||||
{ \
|
||||
std::string status_msg = rocprofiler_get_status_string(CHECKSTATUS); \
|
||||
std::cerr << "[" #result "][" << __FILE__ << ":" << __LINE__ << "] " << msg \
|
||||
<< " failed with error code " << CHECKSTATUS << ": " << status_msg \
|
||||
<< std::endl; \
|
||||
std::stringstream errmsg{}; \
|
||||
errmsg << "[" #result "][" << __FILE__ << ":" << __LINE__ << "] " << msg " failure (" \
|
||||
<< status_msg << ")"; \
|
||||
throw std::runtime_error(errmsg.str()); \
|
||||
} \
|
||||
}
|
||||
|
||||
int
|
||||
start()
|
||||
{
|
||||
return 1;
|
||||
}
|
||||
|
||||
namespace
|
||||
{
|
||||
rocprofiler_context_id_t&
|
||||
get_client_ctx()
|
||||
{
|
||||
static rocprofiler_context_id_t ctx;
|
||||
return ctx;
|
||||
}
|
||||
|
||||
rocprofiler_buffer_id_t&
|
||||
get_buffer()
|
||||
{
|
||||
static rocprofiler_buffer_id_t buf = {};
|
||||
return buf;
|
||||
}
|
||||
|
||||
struct CaptureRecords
|
||||
{
|
||||
std::shared_mutex m_mutex{};
|
||||
// <counter id handle, expected instances>
|
||||
std::map<uint64_t, size_t> expected{};
|
||||
std::map<uint64_t, std::string> expected_counter_names{};
|
||||
std::vector<rocprofiler_counter_id_t> remaining{};
|
||||
// <counter_id handle, instances seen>
|
||||
std::map<uint64_t, size_t> captured{};
|
||||
};
|
||||
|
||||
CaptureRecords* REC = new CaptureRecords;
|
||||
|
||||
CaptureRecords*
|
||||
get_capture()
|
||||
{
|
||||
return REC;
|
||||
}
|
||||
|
||||
void
|
||||
buffered_callback(rocprofiler_context_id_t,
|
||||
rocprofiler_buffer_id_t,
|
||||
rocprofiler_record_header_t** headers,
|
||||
size_t num_headers,
|
||||
void*,
|
||||
uint64_t)
|
||||
{
|
||||
auto& cap = *get_capture();
|
||||
auto wlock = std::unique_lock{cap.m_mutex};
|
||||
std::map<uint64_t, size_t> seen_counters;
|
||||
for(size_t i = 0; i < num_headers; ++i)
|
||||
{
|
||||
auto* header = headers[i];
|
||||
if(header->category == ROCPROFILER_BUFFER_CATEGORY_COUNTERS && header->kind == 0)
|
||||
{
|
||||
rocprofiler_counter_id_t counter;
|
||||
auto* record = static_cast<rocprofiler_record_counter_t*>(header->payload);
|
||||
rocprofiler_query_record_counter_id(record->id, &counter);
|
||||
if(counter.handle == 517)
|
||||
{
|
||||
std::clog << "HERE";
|
||||
}
|
||||
seen_counters.emplace(counter.handle, 0).first->second++;
|
||||
}
|
||||
}
|
||||
|
||||
for(const auto& [counter_id, instances] : seen_counters)
|
||||
{
|
||||
cap.captured.emplace(counter_id, 0).first->second += instances;
|
||||
}
|
||||
}
|
||||
|
||||
void
|
||||
dispatch_callback(rocprofiler_queue_id_t queue_id,
|
||||
const rocprofiler_agent_t* agent,
|
||||
rocprofiler_correlation_id_t correlation_id,
|
||||
const hsa_kernel_dispatch_packet_t* dispatch_packet,
|
||||
void* callback_data_args,
|
||||
rocprofiler_profile_config_id_t* config)
|
||||
{
|
||||
auto& cap = *get_capture();
|
||||
auto wlock = std::unique_lock{cap.m_mutex};
|
||||
|
||||
if(cap.expected.empty())
|
||||
{
|
||||
std::vector<rocprofiler_counter_id_t> counters_needed;
|
||||
ROCPROFILER_CALL(
|
||||
rocprofiler_iterate_agent_supported_counters(
|
||||
*agent,
|
||||
[](rocprofiler_counter_id_t* counters, size_t num_counters, void* user_data) {
|
||||
std::vector<rocprofiler_counter_id_t>* vec =
|
||||
static_cast<std::vector<rocprofiler_counter_id_t>*>(user_data);
|
||||
for(size_t i = 0; i < num_counters; i++)
|
||||
{
|
||||
vec->push_back(counters[i]);
|
||||
}
|
||||
return ROCPROFILER_STATUS_SUCCESS;
|
||||
},
|
||||
static_cast<void*>(&counters_needed)),
|
||||
"Could not fetch supported counters");
|
||||
|
||||
for(auto& found_counter : counters_needed)
|
||||
{
|
||||
size_t expected = 0;
|
||||
rocprofiler_query_counter_instance_count(*agent, found_counter, &expected);
|
||||
cap.remaining.push_back(found_counter);
|
||||
cap.expected.emplace(found_counter.handle, expected);
|
||||
const char* name;
|
||||
size_t name_size;
|
||||
ROCPROFILER_CALL(rocprofiler_query_counter_name(found_counter, &name, &name_size),
|
||||
"Could not query name");
|
||||
cap.expected_counter_names.emplace(found_counter.handle, std::string(name));
|
||||
}
|
||||
if(cap.expected.empty()) throw std::runtime_error("No counters found for agent");
|
||||
}
|
||||
if(cap.remaining.empty()) return;
|
||||
|
||||
rocprofiler_profile_config_id_t profile;
|
||||
|
||||
ROCPROFILER_CALL(
|
||||
rocprofiler_create_profile_config(*agent, &(cap.remaining.back()), 1, &profile),
|
||||
"Could not construct profile cfg");
|
||||
|
||||
cap.remaining.pop_back();
|
||||
*config = profile;
|
||||
}
|
||||
|
||||
int
|
||||
tool_init(rocprofiler_client_finalize_t, void*)
|
||||
{
|
||||
get_capture();
|
||||
ROCPROFILER_CALL(rocprofiler_create_context(&get_client_ctx()), "context creation failed");
|
||||
|
||||
ROCPROFILER_CALL(rocprofiler_create_buffer(get_client_ctx(),
|
||||
4096,
|
||||
2048,
|
||||
ROCPROFILER_BUFFER_POLICY_LOSSLESS,
|
||||
buffered_callback,
|
||||
nullptr,
|
||||
&get_buffer()),
|
||||
"buffer creation failed");
|
||||
|
||||
auto client_thread = rocprofiler_callback_thread_t{};
|
||||
ROCPROFILER_CALL(rocprofiler_create_callback_thread(&client_thread),
|
||||
"failure creating callback thread");
|
||||
|
||||
ROCPROFILER_CALL(rocprofiler_assign_callback_thread(get_buffer(), client_thread),
|
||||
"failed to assign thread for buffer");
|
||||
ROCPROFILER_CALL(rocprofiler_configure_buffered_dispatch_profile_counting_service(
|
||||
get_client_ctx(), get_buffer(), dispatch_callback, nullptr),
|
||||
"Could not setup buffered service");
|
||||
rocprofiler_start_context(get_client_ctx());
|
||||
|
||||
// no errors
|
||||
return 0;
|
||||
}
|
||||
|
||||
void
|
||||
tool_fini(void*)
|
||||
{
|
||||
rocprofiler_flush_buffer(get_buffer());
|
||||
rocprofiler_stop_context(get_client_ctx());
|
||||
// Flush buffer isn't waiting....
|
||||
sleep(2);
|
||||
|
||||
std::clog << "In tool fini\n";
|
||||
|
||||
auto cap_ptr = get_capture();
|
||||
auto& cap = *get_capture();
|
||||
auto wlock = std::unique_lock{cap.m_mutex};
|
||||
|
||||
if(cap.captured.size() != cap.expected.size())
|
||||
{
|
||||
std::clog << "[ERROR] Expected " << cap.expected.size() << " counters collected but got "
|
||||
<< cap.captured.size() << "\n";
|
||||
}
|
||||
|
||||
for(const auto& [counter_id, expected] : cap.expected)
|
||||
{
|
||||
std::string name = "UNKNOWN";
|
||||
if(auto pos = cap.expected_counter_names.find(counter_id);
|
||||
pos != cap.expected_counter_names.end())
|
||||
{
|
||||
name = pos->second;
|
||||
}
|
||||
|
||||
std::optional<size_t> actual_size;
|
||||
|
||||
if(auto pos = cap.captured.find(counter_id); pos != cap.captured.end())
|
||||
{
|
||||
actual_size = pos->second;
|
||||
}
|
||||
|
||||
if(actual_size && *actual_size != expected)
|
||||
{
|
||||
std::clog << (*actual_size == expected ? "" : "[ERROR]") << "Counter ID: " << counter_id
|
||||
<< " (" << name << ")"
|
||||
<< " expected " << expected << " instances and got " << *actual_size << "\n";
|
||||
}
|
||||
else
|
||||
{
|
||||
std::clog << "[ERROR] Counter ID: " << counter_id << " (" << name
|
||||
<< ") is missing from output\n";
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
extern "C" rocprofiler_tool_configure_result_t*
|
||||
rocprofiler_configure(uint32_t version,
|
||||
const char* runtime_version,
|
||||
uint32_t,
|
||||
rocprofiler_client_id_t* id)
|
||||
{
|
||||
// set the client name
|
||||
id->name = "CounterClientSample";
|
||||
|
||||
// compute major/minor/patch version info
|
||||
uint32_t major = version / 10000;
|
||||
uint32_t minor = (version % 10000) / 100;
|
||||
uint32_t patch = version % 100;
|
||||
|
||||
// generate info string
|
||||
auto info = std::stringstream{};
|
||||
info << id->name << " is using rocprofiler v" << major << "." << minor << "." << patch << " ("
|
||||
<< runtime_version << ")";
|
||||
|
||||
std::clog << info.str() << std::endl;
|
||||
|
||||
// create configure data
|
||||
static auto cfg =
|
||||
rocprofiler_tool_configure_result_t{sizeof(rocprofiler_tool_configure_result_t),
|
||||
&tool_init,
|
||||
&tool_fini,
|
||||
static_cast<void*>(nullptr)};
|
||||
|
||||
// return pointer to configure data
|
||||
return &cfg;
|
||||
}
|
||||
Reference in New Issue
Block a user