Agent, Counters, and AQL (#55)
* Migrate XML counter defs and reader from v1/v2 * Current Working Set * Modified parser * Evaluate AST Start * Update lib/common/xml - move definitions out of class declaration * Update lib/rocprofiler/counters/parser - update build of bison and flex build - reproducible generation - add ROCPROFILER_REGENERATE_COUNTERS_PARSER option - fix namespacing * Update lib/rocprofiler/counters/xml - change location of XML files and install them * Update lib/rocprofiler/counter/tests - normalize the test names - improve test failures (more clear about where failure is) * Update lib/rocprofiler/counters - fix namespace - update to new XML metrics directory * Update lib/rocprofiler/CMakeLists.txt - link to object library * Update lib/rocprofiler/hsa/types.hpp - reorganize includes * Add metric loading class/printers * Agent Implementation * Queue Implementation (#79) * Queue Implementation * API Implementation For Counters (part 1) (#80) * API Implementation For Counters * Bewelton/counter collection 3 (#84) * Added counter sample * More changes * More changes * Update samples/counter_collection - mostly formatting * Update include/rocprofiler/counters.h - formatting * Add lib.common/synchronized.hpp - Synchronized struct * Update lib/rocprofiler/counters/xml/basic_counters.xml - whitespace * Update scripts/patch-parser.cmake - tweaks for consistency * Update lib/rocprofiler/counters/parser/tests/parser_tests.cpp - formatting * Update lib/rocprofiler/counters/parser - improve consistency in rocprofiler-expr-parser-patch - update parser.{h,cpp} and scanner.cpp - formatting + regenerated * Update lib/rocprofiler/aql - formatting - clang-tidy fixes - guard against memory pool access errors * Update lib/rocprofiler/aql/tests - formatting - update use of get_val - normalize test names * Update lib/rocprofiler/counters/tests - formatting - patch basic_counters and derived_counters - normalize test names * Update lib/rocprofiler/aql/tests - set_tests_properties * Update test labels - fix minor issue with gtest labels * Update lib/rocprofiler/counters - formatting - clang-tidy fixes * Update lib/rocprofiler/hsa - fix includes - formatting - clang-tidy fixes - tweak to queue_controller_init interface * Update lib/rocprofiler - include fixes - namespace fixes - clang-tidy fixes - formatting * Update scripts/run-ci.py - exclude counters/parser from code coverage (generated files) * Update include/rocprofiler/counters.h - fix doxygen comment * Update lib/rocprofiler/aql/packet_construct.cpp - guard against HSA_AMD_MEMORY_POOL_ACCESS_DISALLOWED_BY_DEFAULT and HSA_AMD_MEMORY_POOL_ACCESS_NEVER_ALLOWED * Update lib/rocprofiler/counters/parser/raw_ast.hpp - clang-tidy fixes * Update lib/rocprofiler/counters/evaluate_ast.hpp - clang-tidy fixes * Update lib/rocprofiler/aql/tests - disable packet_generation_single and packet_generation_multi tests - the entire implementation rocprofiler::get_ext_table() is incorrect * Minor fixes before cleanup * More changes * More fixes * More fixes * source formatting (clang-format v11) (#99) Co-authored-by: bwelton <bwelton@users.noreply.github.com> * Revert PTL submodule * Update scripts/run-ci.py - exclude counters/parser from code coverage (generated files) * Migrating counters state to context * Linting * source formatting (clang-format v11) (#101) Co-authored-by: bwelton <bwelton@users.noreply.github.com> * revert run-ci * Testing fixes * More test changes * Fix minor typo * Small queue change * Small queue change * source formatting (clang-format v11) (#102) Co-authored-by: bwelton <bwelton@users.noreply.github.com> * source formatting (clang-format v11) (#105) Co-authored-by: bwelton <bwelton@users.noreply.github.com> * Documentation Change * More documentation fixes * source formatting (clang-format v11) (#106) Co-authored-by: bwelton <bwelton@users.noreply.github.com> * Threading fixes * Threading fixes * source formatting (clang-format v11) (#107) Co-authored-by: bwelton <bwelton@users.noreply.github.com> * Threading fixes * More test fixes * More agent fixes * More build fixes * source formatting (clang-format v11) (#109) Co-authored-by: bwelton <bwelton@users.noreply.github.com> * changed test timeouts * Build fix * Build fix * Updates to agent * source formatting (clang-format v11) (#114) Co-authored-by: bwelton <bwelton@users.noreply.github.com> * cmake formatting (cmake-format) (#113) Co-authored-by: bwelton <bwelton@users.noreply.github.com> * remove git worktree folder * Doc update * testing fix * Another test fix * More test changes * Rebase * source formatting (clang-format v11) (#116) Co-authored-by: bwelton <bwelton@users.noreply.github.com> * Documentation * source formatting (clang-format v11) (#119) Co-authored-by: bwelton <bwelton@users.noreply.github.com> * PTL Changes * Minor agent fix for empty labels * source formatting (clang-format v11) (#120) Co-authored-by: bwelton <bwelton@users.noreply.github.com> * Minor agent fix for empty labels * Refactor read_map * source formatting (clang-format v11) (#121) Co-authored-by: bwelton <bwelton@users.noreply.github.com> * Refactor read_map * Cache fixes * source formatting (clang-format v11) (#122) Co-authored-by: bwelton <bwelton@users.noreply.github.com> --------- Co-authored-by: Jonathan R. Madsen <jonathanrmadsen@gmail.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: bwelton <bwelton@users.noreply.github.com>
This commit is contained in:
Benjamin Welton
GitHub
@@ -0,0 +1,529 @@
|
||||
// Copyright (c) 2023 Advanced Micro Devices, Inc.
|
||||
//
|
||||
// Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
// of this software and associated documentation files (the "Software"), to deal
|
||||
// in the Software without restriction, including without limitation the rights
|
||||
// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
// copies of the Software, and to permit persons to whom the Software is
|
||||
// furnished to do so, subject to the following conditions:
|
||||
//
|
||||
// The above copyright notice and this permission notice shall be included in
|
||||
// all copies or substantial portions of the Software.
|
||||
//
|
||||
// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
|
||||
// THE SOFTWARE.
|
||||
|
||||
#include "lib/common/xml.hpp"
|
||||
|
||||
#include <glog/logging.h>
|
||||
|
||||
namespace rocprofiler
|
||||
{
|
||||
namespace common
|
||||
{
|
||||
Xml::Xml(std::string file_name, const Xml* obj)
|
||||
: file_name_(std::move(file_name))
|
||||
, state_(BODY_STATE)
|
||||
{
|
||||
if(obj != nullptr)
|
||||
{
|
||||
map_ = obj->map_;
|
||||
level_ = obj->level_;
|
||||
included_ = true;
|
||||
}
|
||||
}
|
||||
|
||||
Xml::~Xml()
|
||||
{
|
||||
for(auto& x : stack_)
|
||||
{
|
||||
x->nodes.clear();
|
||||
x->copy.reset();
|
||||
}
|
||||
if(!map_) return;
|
||||
for(auto& [_, nodes] : *map_)
|
||||
{
|
||||
for(auto& node : nodes)
|
||||
{
|
||||
node->nodes.clear();
|
||||
node->copy.reset();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
std::shared_ptr<Xml>
|
||||
Xml::Create(const std::string& file_name, const Xml* obj)
|
||||
{
|
||||
auto xml = std::make_shared<Xml>(file_name, obj);
|
||||
if(xml != nullptr)
|
||||
{
|
||||
if(xml->Init() != false)
|
||||
{
|
||||
const std::size_t pos = file_name.rfind('/');
|
||||
const std::string path = (pos != std::string::npos) ? file_name.substr(0, pos + 1) : "";
|
||||
|
||||
xml->PreProcess();
|
||||
nodes_t incl_nodes;
|
||||
for(const auto& node : xml->GetNodes("top.include"))
|
||||
{
|
||||
if(node->opts.find("touch") == node->opts.end())
|
||||
{
|
||||
node->opts["touch"] = "";
|
||||
incl_nodes.push_back(node);
|
||||
}
|
||||
}
|
||||
for(const auto& incl : incl_nodes)
|
||||
{
|
||||
const std::string& incl_name = path + incl->opts["file"];
|
||||
auto ixml = Create(incl_name, xml.get());
|
||||
if(!ixml)
|
||||
{
|
||||
xml.reset();
|
||||
break;
|
||||
}
|
||||
}
|
||||
if(xml)
|
||||
{
|
||||
xml->Process();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return xml;
|
||||
}
|
||||
|
||||
void
|
||||
Xml::AddExpr(const std::string& full_tag, const std::string& name, const std::string& expr)
|
||||
{
|
||||
const std::size_t pos = full_tag.rfind('.');
|
||||
const std::size_t pos1 = (pos == std::string::npos) ? 0 : pos + 1;
|
||||
const std::string level_tag = full_tag.substr(pos1);
|
||||
auto level = std::make_shared<level_t>();
|
||||
(*map_)[full_tag].push_back(level);
|
||||
level->tag = level_tag;
|
||||
level->opts["name"] = name;
|
||||
level->opts["expr"] = expr;
|
||||
}
|
||||
|
||||
void
|
||||
Xml::AddConst(const std::string& full_tag, const std::string& name, const uint64_t& val)
|
||||
{
|
||||
std::ostringstream oss;
|
||||
oss << val;
|
||||
AddExpr(full_tag, name, oss.str());
|
||||
}
|
||||
|
||||
bool
|
||||
Xml::print_func::operator()(const std::string& global_tag, const std::shared_ptr<level_t>& node)
|
||||
{
|
||||
std::cout << global_tag << ":\n";
|
||||
for(auto& opt : node->opts)
|
||||
{
|
||||
std::cout << global_tag << "." << opt.first << " = " << opt.second << "\n";
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
void
|
||||
Xml::Print() const
|
||||
{
|
||||
std::cout << "XML file '" << file_name_ << "':\n";
|
||||
ForEach(print_func{});
|
||||
}
|
||||
|
||||
bool
|
||||
Xml::Init()
|
||||
{
|
||||
fd_ = open(file_name_.c_str(), O_RDONLY);
|
||||
if(fd_ == -1)
|
||||
{
|
||||
// perror((std::string("open XML file ") + file_name_).c_str());
|
||||
return false;
|
||||
}
|
||||
|
||||
if(map_ == nullptr)
|
||||
{
|
||||
map_ = std::make_unique<map_t>();
|
||||
AddLevel("top");
|
||||
}
|
||||
|
||||
return true;
|
||||
}
|
||||
|
||||
void
|
||||
Xml::PreProcess()
|
||||
{
|
||||
uint32_t ind = 0;
|
||||
char buf[kBufSize];
|
||||
bool error = false;
|
||||
|
||||
while(true)
|
||||
{
|
||||
const uint32_t pos = lseek(fd_, 0, SEEK_CUR);
|
||||
uint32_t size = read(fd_, buf, kBufSize);
|
||||
if(size <= 0) break;
|
||||
buf[size - 1] = '\0';
|
||||
|
||||
if(strncmp(buf, "#include \"", 10) == 0)
|
||||
{
|
||||
for(ind = 0; (ind < size) && (buf[ind] != '\n'); ++ind)
|
||||
{}
|
||||
if(ind < size)
|
||||
{
|
||||
buf[ind] = '\0';
|
||||
size = ind;
|
||||
lseek(fd_, pos + ind + 1, SEEK_SET);
|
||||
}
|
||||
|
||||
for(ind = 10; (ind < size) && (buf[ind] != '"'); ++ind)
|
||||
{}
|
||||
if(ind == size)
|
||||
{
|
||||
error = true;
|
||||
break;
|
||||
}
|
||||
buf[ind] = '\0';
|
||||
|
||||
AddLevel("include");
|
||||
AddOption("file", &buf[10]);
|
||||
UpLevel();
|
||||
}
|
||||
}
|
||||
|
||||
if(error)
|
||||
{
|
||||
fprintf(stderr, "XML PreProcess failed, line '%s'\n", buf);
|
||||
abort();
|
||||
}
|
||||
|
||||
lseek(fd_, 0, SEEK_SET);
|
||||
}
|
||||
|
||||
void
|
||||
Xml::Process()
|
||||
{
|
||||
token_t remainder;
|
||||
|
||||
while(true)
|
||||
{
|
||||
token_t token = (!remainder.empty()) ? remainder : NextToken();
|
||||
remainder.clear();
|
||||
|
||||
// token_t token1 = token;
|
||||
// token1.push_back('\0');
|
||||
// std::cout << ">>> " << &token1[0] << std::endl;
|
||||
|
||||
// End of file
|
||||
if(token.empty()) break;
|
||||
|
||||
switch(state_)
|
||||
{
|
||||
case BODY_STATE:
|
||||
if(token[0] == '<')
|
||||
{
|
||||
bool node_begin = true;
|
||||
unsigned ind = 1;
|
||||
if(token[1] == '/')
|
||||
{
|
||||
node_begin = false;
|
||||
++ind;
|
||||
}
|
||||
|
||||
unsigned i = ind;
|
||||
while(i < token.size())
|
||||
{
|
||||
if(token[i] == '>') break;
|
||||
++i;
|
||||
}
|
||||
for(unsigned j = i + 1; j < token.size(); ++j)
|
||||
remainder.push_back(token[j]);
|
||||
|
||||
if(i == token.size())
|
||||
{
|
||||
if(node_begin)
|
||||
state_ = DECL_STATE;
|
||||
else
|
||||
BadFormat(token);
|
||||
token.push_back('\0');
|
||||
}
|
||||
else
|
||||
{
|
||||
token[i] = '\0';
|
||||
}
|
||||
|
||||
const char* tag = &token[ind];
|
||||
if(node_begin)
|
||||
{
|
||||
AddLevel(tag);
|
||||
}
|
||||
else
|
||||
{
|
||||
Inherit(GetOption("base"));
|
||||
|
||||
if(strncmp(CurrentLevel().c_str(), tag, strlen(tag)) != 0)
|
||||
{
|
||||
token.back() = '>';
|
||||
BadFormat(token);
|
||||
}
|
||||
UpLevel();
|
||||
}
|
||||
}
|
||||
else
|
||||
{
|
||||
BadFormat(token);
|
||||
}
|
||||
break;
|
||||
case DECL_STATE:
|
||||
if(token[0] == '>')
|
||||
{
|
||||
state_ = BODY_STATE;
|
||||
for(unsigned j = 1; j < token.size(); ++j)
|
||||
remainder.push_back(token[j]);
|
||||
continue;
|
||||
}
|
||||
else
|
||||
{
|
||||
token.push_back('\0');
|
||||
unsigned j = 0;
|
||||
for(j = 0; j < token.size(); ++j)
|
||||
if(token[j] == '=') break;
|
||||
if(j == token.size()) BadFormat(token);
|
||||
token[j] = '\0';
|
||||
const std::string key = token.data();
|
||||
const std::string value = &token[j + 1];
|
||||
AddOption(key, value);
|
||||
}
|
||||
break;
|
||||
default:
|
||||
{
|
||||
LOG(ERROR) << "XML parser error: wrong state: " << state_;
|
||||
abort();
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
bool
|
||||
Xml::SpaceCheck() const
|
||||
{
|
||||
bool cond = ((buffer_[index_] == ' ') || (buffer_[index_] == '\t'));
|
||||
return cond;
|
||||
}
|
||||
|
||||
bool
|
||||
Xml::LineEndCheck()
|
||||
{
|
||||
bool found = false;
|
||||
if(buffer_[index_] == '\n')
|
||||
{
|
||||
buffer_[index_] = ' ';
|
||||
++file_line_;
|
||||
found = true;
|
||||
comment_ = false;
|
||||
}
|
||||
else if(comment_ || (buffer_[index_] == '#'))
|
||||
{
|
||||
found = true;
|
||||
comment_ = true;
|
||||
}
|
||||
return found;
|
||||
}
|
||||
|
||||
Xml::token_t
|
||||
Xml::NextToken()
|
||||
{
|
||||
token_t token;
|
||||
bool in_string = false;
|
||||
bool special_symb = false;
|
||||
|
||||
while(true)
|
||||
{
|
||||
if(data_size_ == 0)
|
||||
{
|
||||
data_size_ = read(fd_, buffer_, kBufSize);
|
||||
if(data_size_ <= 0) break;
|
||||
}
|
||||
|
||||
if(token.empty())
|
||||
{
|
||||
while((index_ < data_size_) && (SpaceCheck() || LineEndCheck()))
|
||||
{
|
||||
++index_;
|
||||
}
|
||||
}
|
||||
while((index_ < data_size_) && (in_string || !(SpaceCheck() || LineEndCheck())))
|
||||
{
|
||||
const char symb = buffer_[index_];
|
||||
bool skip_symb = false;
|
||||
|
||||
switch(symb)
|
||||
{
|
||||
case '\\':
|
||||
if(special_symb)
|
||||
{
|
||||
special_symb = false;
|
||||
}
|
||||
else
|
||||
{
|
||||
special_symb = true;
|
||||
skip_symb = true;
|
||||
}
|
||||
break;
|
||||
case '"':
|
||||
if(special_symb)
|
||||
{
|
||||
special_symb = false;
|
||||
}
|
||||
else
|
||||
{
|
||||
in_string = !in_string;
|
||||
if(!in_string)
|
||||
{
|
||||
buffer_[index_] = ' ';
|
||||
--index_;
|
||||
}
|
||||
skip_symb = true;
|
||||
}
|
||||
break;
|
||||
}
|
||||
|
||||
if(!skip_symb) token.push_back(symb);
|
||||
++index_;
|
||||
}
|
||||
|
||||
if(index_ == data_size_)
|
||||
{
|
||||
index_ = 0;
|
||||
data_size_ = 0;
|
||||
}
|
||||
else
|
||||
{
|
||||
if(special_symb || in_string) BadFormat(token);
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
return token;
|
||||
}
|
||||
|
||||
void
|
||||
Xml::BadFormat(token_t token)
|
||||
{
|
||||
token.push_back('\0');
|
||||
LOG(ERROR) << "Error: " << file_name_ << ", line " << file_line_ << ", bad XML token '"
|
||||
<< token.data() << "'";
|
||||
abort();
|
||||
}
|
||||
|
||||
void
|
||||
Xml::AddLevel(const std::string& tag)
|
||||
{
|
||||
auto level = std::make_shared<level_t>();
|
||||
level->tag = tag;
|
||||
if(level_)
|
||||
{
|
||||
level_->nodes.push_back(level);
|
||||
stack_.push_back(level_);
|
||||
}
|
||||
level_ = level;
|
||||
|
||||
std::string global_tag = GlobalTag(tag);
|
||||
(*map_)[global_tag].push_back(level_);
|
||||
}
|
||||
|
||||
void
|
||||
Xml::UpLevel()
|
||||
{
|
||||
level_ = stack_.back();
|
||||
stack_.pop_back();
|
||||
}
|
||||
|
||||
void
|
||||
Xml::Copy(const std::shared_ptr<level_t>& from, const std::shared_ptr<level_t>& to)
|
||||
{
|
||||
auto level = to;
|
||||
if(level == nullptr)
|
||||
{
|
||||
AddLevel(from->tag);
|
||||
level = level_;
|
||||
}
|
||||
level->copy = from;
|
||||
level->opts = from->opts;
|
||||
|
||||
for(const auto& node : from->nodes)
|
||||
{
|
||||
bool found = false;
|
||||
const std::string name = GetOption("name", node);
|
||||
const std::string global_tag = GlobalTag(level->tag) + "." + node->tag;
|
||||
for(const auto& item : (*map_)[global_tag])
|
||||
{
|
||||
if((name == GetOption("name", item)) || (node == item->copy))
|
||||
{
|
||||
found = true;
|
||||
break;
|
||||
}
|
||||
}
|
||||
if(found == false) Copy(node, nullptr);
|
||||
}
|
||||
|
||||
if(to == nullptr) UpLevel();
|
||||
}
|
||||
|
||||
void
|
||||
Xml::Inherit(const std::string& tag)
|
||||
{
|
||||
if(!tag.empty())
|
||||
{
|
||||
const std::string global_tag = GlobalTag(tag);
|
||||
auto it = map_->find(global_tag);
|
||||
if(it == map_->end())
|
||||
{
|
||||
fprintf(
|
||||
stderr, "Node \"%s\": Base not found \"%s\"\n", level_->tag.c_str(), tag.c_str());
|
||||
abort();
|
||||
}
|
||||
for(const auto& node : it->second)
|
||||
{
|
||||
Copy(node, level_);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
std::string
|
||||
Xml::CurrentLevel() const
|
||||
{
|
||||
return level_->tag;
|
||||
}
|
||||
|
||||
std::string
|
||||
Xml::GlobalTag(const std::string& tag) const
|
||||
{
|
||||
std::string global_tag;
|
||||
for(const auto& level : stack_)
|
||||
{
|
||||
global_tag += level->tag + ".";
|
||||
}
|
||||
global_tag += tag;
|
||||
return global_tag;
|
||||
}
|
||||
|
||||
void
|
||||
Xml::AddOption(const std::string& key, const std::string& value)
|
||||
{
|
||||
level_->opts[key] = value;
|
||||
}
|
||||
|
||||
std::string
|
||||
Xml::GetOption(const std::string& key, std::shared_ptr<const level_t> level)
|
||||
{
|
||||
level = (level != nullptr) ? level : level_;
|
||||
auto it = level->opts.find(key);
|
||||
return (it != level->opts.end()) ? it->second : "";
|
||||
}
|
||||
} // namespace common
|
||||
} // namespace rocprofiler
|
||||
Reference in New Issue
Block a user