AI/rocm-systems
2
0
Rozštěpit 0
Zdrojový kód Úkoly Pull requesty Akce Balíčky Projekty Vydání Wiki Aktivita

Shared Library Constructor (rocprofv3 deadlock fix) (#599)

Procházet zdrojové kódy 
* Moved tests/apps to tests/bin

* Renamed cmake project in tests/bin

* Update samples

- Use ROCPROFILER_DEFAULT_FAIL_REGEX
- tweaks to stdout messages

* Update tests

- Use ROCPROFILER_DEFAULT_FAIL_REGEX

* Add tests/lib

- libraries with HIP code

* Update PTL submodule

- remove atexit delete of thread_id_map

* Update cmake/rocprofiler_options.cmake

- Set ROCPROFILER_DEFAULT_FAIL_REGEX

* Update common lib: env + logging

- improved customization of logging settings
- default to disabling logging to files
- install failure handler for rocprofv3
- set_env support in environment.*

* Add lib/rocprofiler-sdk/shared_library.cpp

- shared library constructor

* Update lib/rocprofiler-sdk-tool/tool.cpp

- destructor thread safety
- convert callback_name_info and buffered_name_info to pointers
- install failure handler for logging

* Add tests/bin/hip-in-libraries

- hip-in-libraries is an exe which uses two shared libraries where each shared library contains HIP kernels
  - used for testing deadlocking within __hipRegisterFatBinary

* Update bin/rocprofv3

- reorganized the env variables
- use exec to launch command
- set ROCPROFILER_LIBRARY_CTOR=1

* Add tests/rocprofv3/tracing-hip-in-libraries

- uses hip-in-libraries exe for exe which uses shared libraries to launch HIP kernels

* Update bin/rocprofv3

- fix counter collection (no exec)

* Update lib/rocprofiler-sdk-tool/tool.cpp

- replace "Kernel-Name" with "Kernel_Name"

* Update lib/rocprofiler-sdk/registration.cpp

Use RTLD_LOCAL instead of RTLD_GLOBAL for env libraries

* Update tests/rocprofv3

- replace "Kernel-Name" with "Kernel_Name"

* Update tests

- vector-ops (bin) stream syncs + runs with 4 queues per device
- improve counter-collection/input1 validation
- rocprofv3/tracing-hip-in-libraries does not do sys-trace
- improved validation script for tracing-hip-in-libraries
- updated dispatch_callback in json-tool.cpp following reworking of prototypes for counter collection

* Update samples/counter_collection

- updated dispatch_callback(s) and record_callback(s) following reworking of prototypes

* Update bin/rocprofv3

- reorganized help menu
- added options for sub-HSA tables
- added --hip-runtime-trace
- changed --hip-trace to include --hip-compiler-trace

* Update lib/rocprofiler-sdk-tool

- improved kernel filtering
- removed arch_vgpr, accum_vgpr, sgpr code (in rocprofiler-sdk)
- fixed issue with counter-collection w/o tracing
- added support for fine grained HSA API tracing
- removed directly linking to HSA-runtime

* Update lib/rocprofiler-sdk/agent.cpp

- rocp_agents != hsa_agents is non-fatal when ROCPROFILER_BUILD_CI=OFF (CMake option)

* GPR (vector and scalar) info in kernel symbol data

- rocprofiler_callback_tracing_code_object_kernel_symbol_register_data_t contains general purpose register info

* Header include order fix

- Include repo headers first
- Third party library headers next
- standard library headers last

* Update dispatch profiling public API

- introduce rocprofiler_profile_counting_dispatch_data_t
- change signature of rocprofiler_profile_counting_dispatch_callback_t and rocprofiler_profile_counting_record_callback_t
- provide rocprofiler_user_data_t pointer in dispatch callback
- provide rocprofiler_user_data_t value (from dispatch cb) in record callback

* Update tests/bin/CMakeLists.txt

- fix add_subdirectory(hip-in-libraries) order

* Update VERSION

- bump to 0.2.0 in prep for AFAR
Tento commit je obsažen v:
Jonathan R. Madsen
2024-03-07 22:21:26 -06:00
odevzdal GitHub
rodič 665c546e65
revize 7b6d3c70bd
85 změnil soubory, kde provedl 2497 přidání a 856 odebrání
Stáhněte soubor záplaty Stáhněte rozdílový soubor Rozbalit všechny soubory Sbalit všechny soubory
source/lib/rocprofiler-sdk/counters/core.cpp
+46 -16
Zobrazit soubor
@@ -24,6 +24,7 @@
#include "lib/common/container/small_vector.hpp"
#include "lib/common/synchronized.hpp"
#include "lib/common/utility.hpp"
#include "lib/rocprofiler-sdk/agent.hpp"
#include "lib/rocprofiler-sdk/aql/helpers.hpp"
#include "lib/rocprofiler-sdk/aql/packet_construct.hpp"
@@ -32,6 +33,7 @@
#include "lib/rocprofiler-sdk/hsa/queue_controller.hpp"
#include "lib/rocprofiler-sdk/registration.hpp"
#include <rocprofiler-sdk/fwd.h>
#include <rocprofiler-sdk/rocprofiler.h>
namespace rocprofiler
@@ -255,6 +257,7 @@ queue_cb(const std::shared_ptr<counter_callback_info>& info,
const hsa::Queue& queue,
const hsa::rocprofiler_packet& pkt,
uint64_t kernel_id,
rocprofiler_user_data_t* user_data,
const hsa::Queue::queue_info_session_t::external_corr_id_map_t& extern_corr_ids,
const context::correlation_id* correlation_id)
{
@@ -272,14 +275,25 @@ queue_cb(const std::shared_ptr<counter_callback_info>& info,
}
}
rocprofiler_profile_config_id_t req_profile = {.handle = 0};
info->user_cb(queue.get_id(),
queue.get_agent().get_rocp_agent(),
_corr_id_v,
&pkt.kernel_dispatch,
kernel_id,
info->callback_args,
&req_profile);
auto req_profile = rocprofiler_profile_config_id_t{.handle = 0};
auto dispatch_data =
common::init_public_api_struct(rocprofiler_profile_counting_dispatch_data_t{});
dispatch_data.kernel_id = kernel_id;
dispatch_data.agent_id = CHECK_NOTNULL(queue.get_agent().get_rocp_agent())->id;
dispatch_data.queue_id = queue.get_id();
dispatch_data.correlation_id = _corr_id_v;
dispatch_data.private_segment_size = pkt.kernel_dispatch.private_segment_size;
dispatch_data.group_segment_size = pkt.kernel_dispatch.group_segment_size;
dispatch_data.workgroup_size = {pkt.kernel_dispatch.workgroup_size_x,
pkt.kernel_dispatch.workgroup_size_y,
pkt.kernel_dispatch.workgroup_size_z};
dispatch_data.grid_size = {pkt.kernel_dispatch.grid_size_x,
pkt.kernel_dispatch.grid_size_y,
pkt.kernel_dispatch.grid_size_z};
info->user_cb(dispatch_data, &req_profile, user_data, info->callback_args);
if(req_profile.handle == 0) return nullptr;
auto prof_config = get_controller().get_profile_cfg(req_profile);
@@ -407,13 +421,27 @@ completed_cb(const std::shared_ptr<counter_callback_info>& info,
if(!out.empty())
{
CHECK(info->record_callback);
info->record_callback(queue.get_id(),
queue.get_agent().get_rocp_agent()->id,
_corr_id_v,
session.kernel_id,
info->record_callback_args,
out.size(),
out.data());
auto dispatch_data =
common::init_public_api_struct(rocprofiler_profile_counting_dispatch_data_t{});
const auto& kernel_dispatch_pkt = session.kernel_pkt.kernel_dispatch;
dispatch_data.kernel_id = session.kernel_id;
dispatch_data.agent_id = CHECK_NOTNULL(queue.get_agent().get_rocp_agent())->id;
dispatch_data.queue_id = queue.get_id();
dispatch_data.correlation_id = _corr_id_v;
dispatch_data.private_segment_size = kernel_dispatch_pkt.private_segment_size;
dispatch_data.group_segment_size = kernel_dispatch_pkt.group_segment_size;
dispatch_data.workgroup_size = {kernel_dispatch_pkt.workgroup_size_x,
kernel_dispatch_pkt.workgroup_size_y,
kernel_dispatch_pkt.workgroup_size_z};
dispatch_data.grid_size = {kernel_dispatch_pkt.grid_size_x,
kernel_dispatch_pkt.grid_size_y,
kernel_dispatch_pkt.grid_size_z};
info->record_callback(
dispatch_data, out.data(), out.size(), session.user_data, info->record_callback_args);
}
}
@@ -436,9 +464,11 @@ start_context(const context::context* ctx)
[=](const hsa::Queue& q,
const hsa::rocprofiler_packet& kern_pkt,
uint64_t kernel_id,
rocprofiler_user_data_t* user_data,
const hsa::Queue::queue_info_session_t::external_corr_id_map_t& extern_corr_ids,
const context::correlation_id* correlation_id) {
return queue_cb(cb, q, kern_pkt, kernel_id, extern_corr_ids, correlation_id);
return queue_cb(
cb, q, kern_pkt, kernel_id, user_data, extern_corr_ids, correlation_id);
},
// Completion CB
[=](const hsa::Queue& q,
source/lib/rocprofiler-sdk/counters/core.hpp
+2 -1
Zobrazit soubor
@@ -24,6 +24,7 @@
#include <rocprofiler-sdk/agent.h>
#include <rocprofiler-sdk/dispatch_profile.h>
#include <rocprofiler-sdk/fwd.h>
#include "lib/rocprofiler-sdk/aql/helpers.hpp"
#include "lib/rocprofiler-sdk/aql/packet_construct.hpp"
@@ -80,7 +81,6 @@ struct counter_callback_info
// HSA Queue ClientID. This is an ID we get when we insert a callback into the
// HSA queue interceptor. This ID can be used to disable the callback.
rocprofiler::hsa::ClientID queue_id{-1};
// Buffer to use for storing counter data. Used if callback is not set.
std::optional<rocprofiler_buffer_id_t> buffer;
@@ -130,6 +130,7 @@ queue_cb(const std::shared_ptr<counter_callback_info>& info,
const hsa::Queue& queue,
const hsa::rocprofiler_packet& pkt,
uint64_t kernel_id,
rocprofiler_user_data_t* user_data,
const hsa::Queue::queue_info_session_t::external_corr_id_map_t& extern_corr_ids,
const context::correlation_id* correlation_id);
source/lib/rocprofiler-sdk/counters/tests/core.cpp
+88 -65
Zobrazit soubor
@@ -20,10 +20,23 @@
// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
// SOFTWARE.
#include <algorithm>
#include <cstdint>
#include <sstream>
#include <tuple>
#include "lib/rocprofiler-sdk/counters/core.hpp"
#include "lib/common/static_object.hpp"
#include "lib/common/utility.hpp"
#include "lib/rocprofiler-sdk/agent.hpp"
#include "lib/rocprofiler-sdk/buffer.hpp"
#include "lib/rocprofiler-sdk/context/context.hpp"
#include "lib/rocprofiler-sdk/counters/id_decode.hpp"
#include "lib/rocprofiler-sdk/counters/metrics.hpp"
#include "lib/rocprofiler-sdk/hsa/agent_cache.hpp"
#include "lib/rocprofiler-sdk/hsa/queue.hpp"
#include "lib/rocprofiler-sdk/hsa/queue_controller.hpp"
#include "lib/rocprofiler-sdk/registration.hpp"
#include <rocprofiler-sdk/dispatch_profile.h>
#include <rocprofiler-sdk/fwd.h>
#include <rocprofiler-sdk/registration.h>
#include <rocprofiler-sdk/rocprofiler.h>
#include <fmt/core.h>
#include <gtest/gtest.h>
@@ -31,21 +44,10 @@
#include <hsa/hsa_api_trace.h>
#include <hsa/hsa_ext_amd.h>
#include <rocprofiler-sdk/rocprofiler.h>
#include "lib/common/static_object.hpp"
#include "lib/common/utility.hpp"
#include "lib/rocprofiler-sdk/agent.hpp"
#include "lib/rocprofiler-sdk/buffer.hpp"
#include "lib/rocprofiler-sdk/context/context.hpp"
#include "lib/rocprofiler-sdk/counters/core.hpp"
#include "lib/rocprofiler-sdk/counters/id_decode.hpp"
#include "lib/rocprofiler-sdk/counters/metrics.hpp"
#include "lib/rocprofiler-sdk/hsa/agent_cache.hpp"
#include "lib/rocprofiler-sdk/hsa/queue.hpp"
#include "lib/rocprofiler-sdk/hsa/queue_controller.hpp"
#include "lib/rocprofiler-sdk/registration.hpp"
#include "rocprofiler-sdk/registration.h"
#include <algorithm>
#include <cstdint>
#include <sstream>
#include <tuple>
using namespace rocprofiler::counters;
using namespace rocprofiler;
@@ -204,13 +206,10 @@ buffered_callback(rocprofiler_context_id_t,
}
void
null_dispatch_callback(rocprofiler_queue_id_t,
const rocprofiler_agent_t*,
rocprofiler_correlation_id_t,
const hsa_kernel_dispatch_packet_t*,
uint64_t,
void*,
rocprofiler_profile_config_id_t*)
null_dispatch_callback(rocprofiler_profile_counting_dispatch_data_t,
rocprofiler_profile_config_id_t*,
rocprofiler_user_data_t*,
void*)
{}
void
@@ -223,13 +222,11 @@ null_buffered_callback(rocprofiler_context_id_t,
{}
void
null_record_callback(rocprofiler_queue_id_t,
rocprofiler_agent_id_t,
rocprofiler_correlation_id_t,
uint64_t,
void*,
null_record_callback(rocprofiler_profile_counting_dispatch_data_t,
rocprofiler_record_counter_t*,
size_t,
rocprofiler_record_counter_t*)
rocprofiler_user_data_t,
void*)
{}
} // namespace
@@ -326,10 +323,10 @@ public:
, _agent(a)
, _id(id)
{}
virtual const AgentCache& get_agent() const override final { return _agent; };
virtual rocprofiler_queue_id_t get_id() const override final { return _id; };
const AgentCache& get_agent() const final { return _agent; };
rocprofiler_queue_id_t get_id() const final { return _id; };
~FakeQueue() {}
~FakeQueue() override = default;
private:
const AgentCache& _agent;
@@ -339,39 +336,60 @@ private:
} // namespace hsa
} // namespace rocprofiler
bool
operator==(rocprofiler_dim3_t lhs, rocprofiler_dim3_t rhs)
{
return std::tie(lhs.x, lhs.y, lhs.z) == std::tie(rhs.x, rhs.y, rhs.z);
}
bool
operator==(rocprofiler_agent_id_t lhs, rocprofiler_agent_id_t rhs)
{
return (lhs.handle == rhs.handle);
}
namespace
{
struct expected_dispatch
{
// To pass back
rocprofiler_profile_config_id_t id;
rocprofiler_queue_id_t queue_id;
const rocprofiler_agent_t* agent;
rocprofiler_correlation_id_t correlation_id;
hsa_kernel_dispatch_packet_t* dispatch_packet;
uint64_t kernel_id;
rocprofiler_profile_config_id_t* config;
rocprofiler_profile_config_id_t id = {};
rocprofiler_queue_id_t queue_id = {.handle = 0};
rocprofiler_agent_id_t agent_id = {.handle = 0};
uint64_t kernel_id = 0;
rocprofiler_correlation_id_t correlation_id = {.internal = 0, .external = {.value = 0}};
rocprofiler_dim3_t workgroup_size = {0, 0, 0};
rocprofiler_dim3_t grid_size = {0, 0, 0};
rocprofiler_profile_config_id_t* config = nullptr;
};
void
user_dispatch_cb(rocprofiler_queue_id_t queue_id,
const rocprofiler_agent_t* agent,
rocprofiler_correlation_id_t correlation_id,
const hsa_kernel_dispatch_packet_t* dispatch_packet,
uint64_t kernel_id,
void* callback_data_args,
rocprofiler_profile_config_id_t* config)
user_dispatch_cb(rocprofiler_profile_counting_dispatch_data_t dispatch_data,
rocprofiler_profile_config_id_t* config,
rocprofiler_user_data_t* user_data,
void* callback_data_args)
{
expected_dispatch& expected = *static_cast<expected_dispatch*>(callback_data_args);
ASSERT_EQ(expected.agent, agent);
ASSERT_EQ(expected.queue_id.handle, queue_id.handle);
ASSERT_EQ(expected.correlation_id.internal, correlation_id.internal);
ASSERT_EQ(expected.correlation_id.external.ptr, correlation_id.external.ptr);
ASSERT_EQ(expected.correlation_id.external.value, correlation_id.external.value);
ASSERT_EQ(expected.dispatch_packet, dispatch_packet);
ASSERT_EQ(expected.kernel_id, kernel_id);
auto agent_id = dispatch_data.agent_id;
auto queue_id = dispatch_data.queue_id;
auto correlation_id = dispatch_data.correlation_id;
auto kernel_id = dispatch_data.kernel_id;
EXPECT_EQ(sizeof(rocprofiler_profile_counting_dispatch_data_t), dispatch_data.size);
EXPECT_EQ(expected.kernel_id, kernel_id);
EXPECT_EQ(expected.agent_id, agent_id);
EXPECT_EQ(expected.queue_id.handle, queue_id.handle);
EXPECT_EQ(expected.correlation_id.internal, correlation_id.internal);
EXPECT_EQ(expected.correlation_id.external.ptr, correlation_id.external.ptr);
EXPECT_EQ(expected.correlation_id.external.value, correlation_id.external.value);
EXPECT_EQ(expected.workgroup_size, dispatch_data.workgroup_size);
EXPECT_EQ(expected.grid_size, dispatch_data.grid_size);
ASSERT_NE(config, nullptr);
config->handle = expected.id.handle;
(void) user_data;
}
} // namespace
@@ -440,17 +458,22 @@ TEST(core, check_callbacks)
hsa::rocprofiler_packet pkt;
pkt.ext_amd_aql_pm4.header = count++;
expected.correlation_id = {.internal = corr_id.internal,
expected.correlation_id = {.internal = corr_id.internal,
.external = context::null_user_data};
expected.dispatch_packet = &pkt.kernel_dispatch;
expected.kernel_id = count++;
expected.queue_id = qid;
expected.agent = fq.get_agent().get_rocp_agent();
expected.workgroup_size = {pkt.kernel_dispatch.workgroup_size_x,
pkt.kernel_dispatch.workgroup_size_y,
pkt.kernel_dispatch.workgroup_size_z};
expected.grid_size = {pkt.kernel_dispatch.grid_size_x,
pkt.kernel_dispatch.grid_size_y,
pkt.kernel_dispatch.grid_size_z};
expected.kernel_id = count++;
expected.queue_id = qid;
expected.agent_id = fq.get_agent().get_rocp_agent()->id;
hsa::Queue::queue_info_session_t::external_corr_id_map_t extern_ids = {};
auto ret_pkt =
counters::queue_cb(cb_info, fq, pkt, expected.kernel_id, extern_ids, &corr_id);
auto user_data = rocprofiler_user_data_t{.value = corr_id.internal};
auto ret_pkt = counters::queue_cb(
cb_info, fq, pkt, expected.kernel_id, &user_data, extern_ids, &corr_id);
ASSERT_TRUE(ret_pkt) << fmt::format("Expected a packet to be generated for - {}",
metric.name());
source/lib/rocprofiler-sdk/counters/tests/dimension.cpp
+9 -9
Zobrazit soubor
@@ -20,14 +20,6 @@
// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
// SOFTWARE.
#include <gtest/gtest.h>
#include <fmt/core.h>
#include <hsa/hsa.h>
#include <hsa/hsa_api_trace.h>
#include <hsa/hsa_ext_amd.h>
#include <rocprofiler-sdk/rocprofiler.h>
#include "lib/common/static_object.hpp"
#include "lib/common/utility.hpp"
#include "lib/rocprofiler-sdk/agent.hpp"
@@ -42,7 +34,15 @@
#include "lib/rocprofiler-sdk/hsa/queue.hpp"
#include "lib/rocprofiler-sdk/hsa/queue_controller.hpp"
#include "lib/rocprofiler-sdk/registration.hpp"
#include "rocprofiler-sdk/registration.h"
#include <rocprofiler-sdk/registration.h>
#include <rocprofiler-sdk/rocprofiler.h>
#include <fmt/core.h>
#include <gtest/gtest.h>
#include <hsa/hsa.h>
#include <hsa/hsa_api_trace.h>
#include <hsa/hsa_ext_amd.h>
namespace
{
source/lib/rocprofiler-sdk/counters/tests/init_order.cpp
+16 -18
Zobrazit soubor
@@ -20,16 +20,6 @@
// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
// SOFTWARE.
#include <algorithm>
#include <cstdint>
#include <sstream>
#include <tuple>
#include <fmt/core.h>
#include <gtest/gtest.h>
#include <rocprofiler-sdk/rocprofiler.h>
#include "lib/common/static_object.hpp"
#include "lib/common/utility.hpp"
#include "lib/rocprofiler-sdk/buffer.hpp"
@@ -37,7 +27,18 @@
#include "lib/rocprofiler-sdk/counters/id_decode.hpp"
#include "lib/rocprofiler-sdk/counters/metrics.hpp"
#include "lib/rocprofiler-sdk/registration.hpp"
#include "rocprofiler-sdk/registration.h"
#include <rocprofiler-sdk/fwd.h>
#include <rocprofiler-sdk/registration.h>
#include <rocprofiler-sdk/rocprofiler.h>
#include <fmt/core.h>
#include <gtest/gtest.h>
#include <algorithm>
#include <cstdint>
#include <sstream>
#include <tuple>
using namespace rocprofiler::counters;
@@ -125,13 +126,10 @@ buffered_callback(rocprofiler_context_id_t,
{}
void
dispatch_callback(rocprofiler_queue_id_t,
const rocprofiler_agent_t*,
rocprofiler_correlation_id_t,
const hsa_kernel_dispatch_packet_t*,
uint64_t,
void*,
rocprofiler_profile_config_id_t*)
dispatch_callback(rocprofiler_profile_counting_dispatch_data_t,
rocprofiler_profile_config_id_t*,
rocprofiler_user_data_t*,
void*)
{}
rocprofiler_context_id_t&