Adding JSON support (#860)
* Adding json support minor bugs Fixing tests Fixing formatting issues Fixing test test fix Misc testing fixes Use rocprofiler/cxx/name_info in rocprofiler-sdk-tool fixes to reduce the Json file size Update source/lib/rocprofiler-sdk-tool/generateJSON.cpp Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Update source/lib/rocprofiler-sdk-tool/generateJSON.cpp Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Update source/lib/rocprofiler-sdk-tool/tool.cpp Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Update source/lib/rocprofiler-sdk-tool/tool.cpp Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Update source/lib/rocprofiler-sdk-tool/helper.hpp Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Update source/lib/rocprofiler-sdk-tool/helper.hpp Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Update source/lib/rocprofiler-sdk-tool/generateJSON.hpp Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Update source/lib/rocprofiler-sdk-tool/generateJSON.cpp Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Update source/lib/rocprofiler-sdk-tool/generateJSON.cpp Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Update source/lib/rocprofiler-sdk-tool/tool.cpp Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Update source/lib/rocprofiler-sdk-tool/tool.cpp Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Update source/lib/rocprofiler-sdk-tool/helper.hpp Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Update source/lib/rocprofiler-sdk-tool/tool.cpp Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Update source/lib/rocprofiler-sdk-tool/generateJSON.cpp Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Update source/lib/rocprofiler-sdk-tool/generateJSON.cpp Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> misc fixes Removing int cast for JSON tests formatting removing a condition test on Navi3 adding debug info Misc fix * CSV updates - fix stats - numerical formatter support for customizing write_csv_entry - misc formatting - get_marker_stats_file * Misc tests/rocprofv3/counter-collection/input2 fixes - rocprofiler_configure_pytest_files in rocprofv3/counter-collection/input2 - removed state code from merge in rocprofv3/counter-collection/input2 * Tool: "Agent-id" -> "Agent_Id" - consistency * Tool update - remove rocprofiler_tool_marker_record_t - add marker_tracing_kind_conversion - fix memory leak in write_json - minor update to get_output_stream - rework handling of marker records * Update tests/pytest-packages/pytest_utils/__init__.py - add collapse_dict_list function for converting a dictionary value that is a list of length one into a directly mapped value * Update tests/rocprofv3/**/conftest.py - use collapse_dict_list when reading in JSONs * Update tests/rocprofv3/counter-collection/input1/validate.py - relax testing requirements gfx1102 (AQLProfile bugs) - in addition to relaxed testing requirements for gfx1101 * Update tests/rocprofv3/tracing/validate.py - fix removal of PID in every marker record * Update tests/rocprofv3/tracing-plus-cc - remove test design that relies on iterating subdirectories * Wrapper around __libc_start_main - Ensures finalization happens before main returns - Update tests/rocprofv3/tracing/validate.py - wrapper around __libc_start_main changed roctx calls * Combine include/rocprofiler-sdk/cxx/serialization.hpp and include/rocprofiler-sdk/external/serialization.hpp - tests/common/serialization.hpp simply includes include/rocprofiler-sdk/cxx/serialization.hpp now * Update lib/rocprofiler-sdk/hip/hip.cpp - tracing function immediately returns when fini_status is non-zero * Update lib/rocprofiler-sdk/hsa/hsa.cpp - remove logging of tracing function when fini_status is non-zero * Update lib/rocprofiler-sdk-tool/CMakeLists.txt - remove rocprofv3_trigger_list_metrics.cpp from TOOL_SOURCES * Update tests/rocprofv3/tracing-plus-cc/CMakeLists.txt - fix depends * Domain statistics * Update tests/rocprofv3/tracing-plus-cc/CMakeLists.txt - do not set ROCP_LOG_LEVEL in env * Remove erroneous <bits/utility.h> include * Restructure tool source + reduce tool table + support multiple formats - buffered_output struct for handling output - support multiple output formats, e.g. --output-format csv,json - rename buffer_type_t -> domain_type - simplified generation of CSV output files - removed rocprofiler_tool_marker_record_t * Update lib/common/container/ring_buffer.hpp - value_type alias in ring_buffer<Tp> * Remove all but one json-execute tests - generate CSV and JSON in same run * Fix include for domain_type.cpp * Update tests/rocprofv3/tracing-plus-cc/input.txt - only specify counters which can be found on gfx8, gfx9, gfx10, gfx11, etc. - use :device= syntax * Update lib/rocprofiler-sdk-tool/config.cpp - support :device=N syntax for counters file - improve stripping comments in PMC files - only read after pmc: * Rework tool library counter collection - fatal error if all requested counters for device are not found - support :device= syntax * Update tests/rocprofv3/tracing-plus-cc/input.txt - removed L2CacheHit (not supported on mi300) * Disable JSON tests in tests/rocprofv3 * Update include/rocprofiler-sdk/cxx/serialization.hpp - support rocprofiler_record_dimension_info_t * Update tool JSON schema - remove domain_type::CODE_OBJECT - rocprofiler_tool_agent_v0_t - rocprofiler_agent_v0_t + counters - rocprofiler_tool_counter_info_t - get_code_object_data() * Update JSON schema for tool * Update lib/rocprofiler-sdk-tool/tool.cpp - fix ROCP_WARNING_IF * rocprofv3 -> rocprofv3.sh - install rocprofv3.sh into sbin - configure_file <source-tree>/rocprofv3.sh -> <binary-tree>/bin/rocprofv3 * Update tool counter collection - rocprofiler_tool_record_counter_t - rocprofiler_tool_counter_collection_record_t * Update tests/rocprofv3/counter-collection/input1/CMakeLists.txt - use rocprofiler_configure_pytest_files for validate.py, conftest.py, and input.txt * Update tests/rocprofv3/counter-collection/input1/validate.py - re-enable test_validate_counter_collection_pmc1_json * Update tests/rocprofv3/counter-collection/input2/validate.py - remove unused code * Update tests/rocprofv3/counter-collection/input2/validate.py - remove unused code * Update tests/rocprofv3/hsa-queue-dependency/validate.py - re-enable JSON tests * Misc tests/rocprofv3 CMake updates * Update tests/rocprofv3/tracing/validate.py - re-enable JSON tests * Update tests/rocprofv3/tracing-hip-in-libraries/validate.py - re-enable JSON tests * Update tests/rocprofv3/tracing/validate.py - remove unused node_exists function * Update tests/rocprofv3/tracing/validate.py - fix test_marker_api_trace_json --------- Co-authored-by: Sriraksha Nagaraj <Sriraksha.Nagaraj@amd.com>
这个提交包含在：
Jonathan R. Madsen
GitHub
@@ -128,7 +128,7 @@ handle_special_chars(std::string& str)
|
||||
{
|
||||
// Iterate over the string and replace any special characters with a space.
|
||||
auto pos = std::string::npos;
|
||||
while((pos = str.find_first_of("!@#$%&(),*+-./;<=>?@{}^`~|:")) != std::string::npos)
|
||||
while((pos = str.find_first_of("!@#$%&(),*+-./;<>?@{}^`~|")) != std::string::npos)
|
||||
str.at(pos) = ' ';
|
||||
}
|
||||
|
||||
@@ -186,21 +186,36 @@ parse_counters(std::string line)
|
||||
|
||||
if(line.empty()) return counters;
|
||||
|
||||
// trim line for any white spaces
|
||||
// strip the comment
|
||||
if(auto pos = std::string::npos; (pos = line.find('#')) != std::string::npos)
|
||||
line = line.substr(0, pos);
|
||||
|
||||
// trim line for any white spaces after comment strip
|
||||
trim(line);
|
||||
|
||||
if(!(line[0] == '#' || line.find("pmc") == std::string::npos))
|
||||
// check to see if comment stripping + trim resulted in empty line
|
||||
if(line.empty()) return counters;
|
||||
|
||||
constexpr auto pmc_qualifier = std::string_view{"pmc:"};
|
||||
auto pos = std::string::npos;
|
||||
|
||||
// should we handle an "pmc:" not being present? Seems like it should be a fatal error
|
||||
if((pos = line.find(pmc_qualifier)) != std::string::npos)
|
||||
{
|
||||
// strip out pmc qualifier
|
||||
line = line.substr(pos + pmc_qualifier.length());
|
||||
|
||||
handle_special_chars(line);
|
||||
|
||||
std::stringstream input_line(line);
|
||||
std::string counter;
|
||||
while(getline(input_line, counter, ' '))
|
||||
auto input_ss = std::stringstream{line};
|
||||
while(true)
|
||||
{
|
||||
if(counter.substr(0, 3) != "pmc" && has_counter_format(counter))
|
||||
{
|
||||
auto counter = std::string{};
|
||||
input_ss >> counter;
|
||||
if(counter.empty())
|
||||
break;
|
||||
else if(counter != pmc_qualifier && has_counter_format(counter))
|
||||
counters.emplace(counter);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -227,7 +242,45 @@ get_mpi_rank()
|
||||
config::config()
|
||||
: kernel_names{parse_kernel_names(get_env("ROCPROF_KERNEL_NAMES", std::string{}))}
|
||||
, counters{parse_counters(get_env("ROCPROF_COUNTERS", std::string{}))}
|
||||
{}
|
||||
{
|
||||
auto output_format = get_env("ROCPROF_OUTPUT_FORMAT", "CSV");
|
||||
|
||||
for(auto& itr : output_format)
|
||||
itr = toupper(itr);
|
||||
|
||||
for(auto itr : {',', ';', ':'})
|
||||
{
|
||||
auto pos = std::string::npos;
|
||||
do
|
||||
{
|
||||
pos = output_format.find(itr);
|
||||
if(pos != std::string::npos) output_format.replace(pos, 1, " ");
|
||||
} while(pos != std::string::npos);
|
||||
}
|
||||
|
||||
auto entries = std::set<std::string>{};
|
||||
auto parser = std::stringstream{output_format};
|
||||
|
||||
while(true)
|
||||
{
|
||||
auto _val = std::string{};
|
||||
parser >> _val;
|
||||
if(!_val.empty())
|
||||
entries.emplace(_val);
|
||||
else
|
||||
break;
|
||||
}
|
||||
|
||||
csv_output = entries.count("CSV") > 0 || entries.empty();
|
||||
json_output = entries.count("JSON") > 0;
|
||||
|
||||
const auto supported_formats = std::set<std::string_view>{"CSV", "JSON"};
|
||||
for(const auto& itr : entries)
|
||||
{
|
||||
LOG_IF(FATAL, supported_formats.count(itr) == 0)
|
||||
<< "Unsupported output format type: " << itr;
|
||||
}
|
||||
}
|
||||
|
||||
std::vector<output_key>
|
||||
output_keys(std::string _tag)
|
||||
|
||||
在新工单中引用
屏蔽一个用户