AI/rocm-systems
2
0
Tükrözés 0
Kód Hibajegyek Egyesítési kérések Actions Packages Projektek Kiadások Wiki Tevékenység

Shared Library Constructor (rocprofv3 deadlock fix) (#599)

Forráskód böngészése 
* Moved tests/apps to tests/bin

* Renamed cmake project in tests/bin

* Update samples

- Use ROCPROFILER_DEFAULT_FAIL_REGEX
- tweaks to stdout messages

* Update tests

- Use ROCPROFILER_DEFAULT_FAIL_REGEX

* Add tests/lib

- libraries with HIP code

* Update PTL submodule

- remove atexit delete of thread_id_map

* Update cmake/rocprofiler_options.cmake

- Set ROCPROFILER_DEFAULT_FAIL_REGEX

* Update common lib: env + logging

- improved customization of logging settings
- default to disabling logging to files
- install failure handler for rocprofv3
- set_env support in environment.*

* Add lib/rocprofiler-sdk/shared_library.cpp

- shared library constructor

* Update lib/rocprofiler-sdk-tool/tool.cpp

- destructor thread safety
- convert callback_name_info and buffered_name_info to pointers
- install failure handler for logging

* Add tests/bin/hip-in-libraries

- hip-in-libraries is an exe which uses two shared libraries where each shared library contains HIP kernels
  - used for testing deadlocking within __hipRegisterFatBinary

* Update bin/rocprofv3

- reorganized the env variables
- use exec to launch command
- set ROCPROFILER_LIBRARY_CTOR=1

* Add tests/rocprofv3/tracing-hip-in-libraries

- uses hip-in-libraries exe for exe which uses shared libraries to launch HIP kernels

* Update bin/rocprofv3

- fix counter collection (no exec)

* Update lib/rocprofiler-sdk-tool/tool.cpp

- replace "Kernel-Name" with "Kernel_Name"

* Update lib/rocprofiler-sdk/registration.cpp

Use RTLD_LOCAL instead of RTLD_GLOBAL for env libraries

* Update tests/rocprofv3

- replace "Kernel-Name" with "Kernel_Name"

* Update tests

- vector-ops (bin) stream syncs + runs with 4 queues per device
- improve counter-collection/input1 validation
- rocprofv3/tracing-hip-in-libraries does not do sys-trace
- improved validation script for tracing-hip-in-libraries
- updated dispatch_callback in json-tool.cpp following reworking of prototypes for counter collection

* Update samples/counter_collection

- updated dispatch_callback(s) and record_callback(s) following reworking of prototypes

* Update bin/rocprofv3

- reorganized help menu
- added options for sub-HSA tables
- added --hip-runtime-trace
- changed --hip-trace to include --hip-compiler-trace

* Update lib/rocprofiler-sdk-tool

- improved kernel filtering
- removed arch_vgpr, accum_vgpr, sgpr code (in rocprofiler-sdk)
- fixed issue with counter-collection w/o tracing
- added support for fine grained HSA API tracing
- removed directly linking to HSA-runtime

* Update lib/rocprofiler-sdk/agent.cpp

- rocp_agents != hsa_agents is non-fatal when ROCPROFILER_BUILD_CI=OFF (CMake option)

* GPR (vector and scalar) info in kernel symbol data

- rocprofiler_callback_tracing_code_object_kernel_symbol_register_data_t contains general purpose register info

* Header include order fix

- Include repo headers first
- Third party library headers next
- standard library headers last

* Update dispatch profiling public API

- introduce rocprofiler_profile_counting_dispatch_data_t
- change signature of rocprofiler_profile_counting_dispatch_callback_t and rocprofiler_profile_counting_record_callback_t
- provide rocprofiler_user_data_t pointer in dispatch callback
- provide rocprofiler_user_data_t value (from dispatch cb) in record callback

* Update tests/bin/CMakeLists.txt

- fix add_subdirectory(hip-in-libraries) order

* Update VERSION

- bump to 0.2.0 in prep for AFAR
This commit is contained in:
Jonathan R. Madsen
2024-03-07 22:21:26 -06:00
committed by GitHub
szülő 665c546e65
commit 7b6d3c70bd
85 fájl változott, egészen pontosan 2497 új sor hozzáadva és 856 régi sor törölve
Download Patch File Download Diff File Expand all files Collapse all files
samples/api_buffered_tracing/CMakeLists.txt
+1 -1
Fájl megtekintése
@@ -55,4 +55,4 @@ set_tests_properties(
ENVIRONMENT
"${ROCPROFILER_MEMCHECK_PRELOAD_ENV};HSA_TOOLS_LIB=$<TARGET_FILE:rocprofiler::rocprofiler-shared-library>"
FAIL_REGULAR_EXPRESSION
"threw an exception")
"${ROCPROFILER_DEFAULT_FAIL_REGEX}")
samples/api_buffered_tracing/main.cpp
+6 -4
Fájl megtekintése
@@ -155,7 +155,7 @@ run(int rank, int tid, hipStream_t stream, int argc, char** argv)
if(argc > 3) nsync = atoll(argv[3]);
auto_lock_t _lk{print_lock};
std::cout << "[" << rank << "][" << tid << "] M: " << M << " N: " << N << std::endl;
std::cout << "[transpose][" << rank << "][" << tid << "] M: " << M << " N: " << N << std::endl;
_lk.unlock();
std::default_random_engine _engine{std::random_device{}() * (rank + 1) * (tid + 1)};
@@ -183,7 +183,7 @@ run(int rank, int tid, hipStream_t stream, int argc, char** argv)
dim3 block(32, 32, 1); // transpose_a
print_lock.lock();
printf("[%i][%i] grid=(%i,%i,%i), block=(%i,%i,%i)\n",
printf("[transpose][%i][%i] grid=(%i,%i,%i), block=(%i,%i,%i)\n",
rank,
tid,
grid.x,
@@ -208,8 +208,10 @@ run(int rank, int tid, hipStream_t stream, int argc, char** argv)
float GB = (float) size * nitr * 2 / (1 << 30);
print_lock.lock();
std::cout << "[" << rank << "][" << tid << "] Runtime of transpose is " << time << " sec\n"
<< "The average performance of transpose is " << GB / time << " GBytes/sec"
std::cout << "[transpose][" << rank << "][" << tid << "] Runtime of transpose is " << time
<< " sec\n";
std::cout << "[transpose][" << rank << "][" << tid
<< "] The average performance of transpose is " << GB / time << " GBytes/sec"
<< std::endl;
print_lock.unlock();
samples/api_callback_tracing/CMakeLists.txt
+1 -1
Fájl megtekintése
@@ -57,4 +57,4 @@ set(callback-api-tracing-env
set_tests_properties(
callback-api-tracing
PROPERTIES TIMEOUT 45 LABELS "samples" ENVIRONMENT "${callback-api-tracing-env}"
FAIL_REGULAR_EXPRESSION "threw an exception")
FAIL_REGULAR_EXPRESSION "${ROCPROFILER_DEFAULT_FAIL_REGEX}")
samples/api_callback_tracing/main.cpp
+5 -3
Fájl megtekintése
@@ -177,7 +177,7 @@ run(int rank, int tid, hipStream_t stream, int argc, char** argv)
if(argc > 3) nsync = atoll(argv[3]);
auto_lock_t _lk{print_lock};
std::cout << "[" << rank << "][" << tid << "] M: " << M << " N: " << N << std::endl;
std::cout << "[transpose][" << rank << "][" << tid << "] M: " << M << " N: " << N << std::endl;
_lk.unlock();
auto _seed = std::random_device{}() * (rank + 1) * (tid + 1);
@@ -219,8 +219,10 @@ run(int rank, int tid, hipStream_t stream, int argc, char** argv)
float GB = (float) size * nitr * 2 / (1 << 30);
print_lock.lock();
std::cout << "[" << rank << "][" << tid << "] Runtime of transpose is " << time << " sec\n"
<< "The average performance of transpose is " << GB / time << " GBytes/sec"
std::cout << "[transpose][" << rank << "][" << tid << "] Runtime of transpose is " << time
<< " sec\n";
std::cout << "[transpose][" << rank << "][" << tid
<< "] The average performance of transpose is " << GB / time << " GBytes/sec"
<< std::endl;
print_lock.unlock();
samples/code_object_tracing/CMakeLists.txt
+1 -1
Fájl megtekintése
@@ -55,4 +55,4 @@ set_tests_properties(
ENVIRONMENT
"${ROCPROFILER_MEMCHECK_PRELOAD_ENV};HSA_TOOLS_LIB=$<TARGET_FILE:rocprofiler::rocprofiler-shared-library>"
FAIL_REGULAR_EXPRESSION
"threw an exception")
"${ROCPROFILER_DEFAULT_FAIL_REGEX}")
samples/code_object_tracing/main.cpp
+6 -4
Fájl megtekintése
@@ -145,7 +145,7 @@ run(int rank, int tid, hipStream_t stream, int argc, char** argv)
if(argc > 3) nsync = atoll(argv[3]);
auto_lock_t _lk{print_lock};
std::cout << "[" << rank << "][" << tid << "] M: " << M << " N: " << N << std::endl;
std::cout << "[transpose][" << rank << "][" << tid << "] M: " << M << " N: " << N << std::endl;
_lk.unlock();
std::default_random_engine _engine{std::random_device{}() * (rank + 1) * (tid + 1)};
@@ -173,7 +173,7 @@ run(int rank, int tid, hipStream_t stream, int argc, char** argv)
dim3 block(32, 32, 1); // transpose_a
print_lock.lock();
printf("[%i][%i] grid=(%i,%i,%i), block=(%i,%i,%i)\n",
printf("[transpose][%i][%i] grid=(%i,%i,%i), block=(%i,%i,%i)\n",
rank,
tid,
grid.x,
@@ -198,8 +198,10 @@ run(int rank, int tid, hipStream_t stream, int argc, char** argv)
float GB = (float) size * nitr * 2 / (1 << 30);
print_lock.lock();
std::cout << "[" << rank << "][" << tid << "] Runtime of transpose is " << time << " sec\n"
<< "The average performance of transpose is " << GB / time << " GBytes/sec"
std::cout << "[transpose][" << rank << "][" << tid << "] Runtime of transpose is " << time
<< " sec\n";
std::cout << "[transpose][" << rank << "][" << tid
<< "] The average performance of transpose is " << GB / time << " GBytes/sec"
<< std::endl;
print_lock.unlock();
samples/common/CMakeLists.txt
+5
Fájl megtekintése
@@ -2,6 +2,11 @@
# common utilities for samples
#
# default FAIL_REGULAR_EXPRESSION for tests
set(ROCPROFILER_DEFAULT_FAIL_REGEX
"threw an exception|Permission denied|Could not create logging file"
CACHE STRING "Default FAIL_REGULAR_EXPRESSION for tests")
# build flags
add_library(rocprofiler-samples-build-flags INTERFACE)
add_library(rocprofiler::samples-build-flags ALIAS rocprofiler-samples-build-flags)
samples/counter_collection/CMakeLists.txt
+3 -3
Fájl megtekintése
@@ -52,7 +52,7 @@ set_tests_properties(
ENVIRONMENT
"${ROCPROFILER_MEMCHECK_PRELOAD_ENV};HSA_TOOLS_LIB=$<TARGET_FILE:rocprofiler::rocprofiler-shared-library>"
FAIL_REGULAR_EXPRESSION
"threw an exception")
"${ROCPROFILER_DEFAULT_FAIL_REGEX}")
add_library(counter-collection-callback-client SHARED)
target_sources(counter-collection-callback-client PRIVATE callback_client.cpp client.hpp)
@@ -80,7 +80,7 @@ set_tests_properties(
ENVIRONMENT
"${ROCPROFILER_MEMCHECK_PRELOAD_ENV};HSA_TOOLS_LIB=$<TARGET_FILE:rocprofiler::rocprofiler-shared-library>"
FAIL_REGULAR_EXPRESSION
"threw an exception")
"${ROCPROFILER_DEFAULT_FAIL_REGEX}")
add_library(counter-collection-functional-counter-client SHARED)
target_sources(counter-collection-functional-counter-client
@@ -109,4 +109,4 @@ set_tests_properties(
ENVIRONMENT
"${ROCPROFILER_MEMCHECK_PRELOAD_ENV};HSA_TOOLS_LIB=$<TARGET_FILE:rocprofiler::rocprofiler-shared-library>"
FAIL_REGULAR_EXPRESSION
"threw an exception")
"${ROCPROFILER_DEFAULT_FAIL_REGEX}")
samples/counter_collection/callback_client.cpp
+19 -20
Fájl megtekintése
@@ -67,15 +67,14 @@ get_client_ctx()
}
void
record_callback(rocprofiler_queue_id_t,
rocprofiler_agent_id_t,
rocprofiler_correlation_id_t,
uint64_t,
void* callback_data_args,
size_t record_count,
rocprofiler_record_counter_t* record_data)
record_callback(rocprofiler_profile_counting_dispatch_data_t dispatch_data,
rocprofiler_record_counter_t* record_data,
size_t record_count,
rocprofiler_user_data_t user_data,
void* callback_data_args)
{
std::stringstream ss;
ss << "Kernel_id " << dispatch_data.kernel_id << ": ";
for(size_t i = 0; i < record_count; ++i)
{
ss << "(Id: " << record_data[i].id << " Value [D]: " << record_data[i].counter_value
@@ -84,6 +83,8 @@ record_callback(rocprofiler_queue_id_t,
auto* output_stream = static_cast<std::ostream*>(callback_data_args);
if(!output_stream) throw std::runtime_error{"nullptr to output stream"};
*output_stream << "[" << __FUNCTION__ << "] " << ss.str() << "\n";
(void) user_data;
}
/**
@@ -93,13 +94,10 @@ record_callback(rocprofiler_queue_id_t,
* to collect the counter SQ_WAVES for all kernel dispatch packets.
*/
void
dispatch_callback(rocprofiler_queue_id_t /*queue_id*/,
const rocprofiler_agent_t* agent,
rocprofiler_correlation_id_t /*correlation_id*/,
const hsa_kernel_dispatch_packet_t* /*dispatch_packet*/,
uint64_t /*kernel_id*/,
void* /*callback_data_args*/,
rocprofiler_profile_config_id_t* config)
dispatch_callback(rocprofiler_profile_counting_dispatch_data_t dispatch_data,
rocprofiler_profile_config_id_t* config,
rocprofiler_user_data_t* /*user_data*/,
void* /*callback_data_args*/)
{
/**
* This simple example uses the same profile counter set for all agents.
@@ -112,7 +110,7 @@ dispatch_callback(rocprofiler_queue_id_t /*queue_id*/,
static std::unordered_map<uint64_t, rocprofiler_profile_config_id_t> profile_cache = {};
auto search_cache = [&]() {
if(auto pos = profile_cache.find(agent->id.handle); pos != profile_cache.end())
if(auto pos = profile_cache.find(dispatch_data.agent_id.handle); pos != profile_cache.end())
{
*config = pos->second;
return true;
@@ -135,7 +133,7 @@ dispatch_callback(rocprofiler_queue_id_t /*queue_id*/,
// Iterate through the agents and get the counters available on that agent
ROCPROFILER_CALL(rocprofiler_iterate_agent_supported_counters(
agent->id,
dispatch_data.agent_id,
[](rocprofiler_agent_id_t,
rocprofiler_counter_id_t* counters,
size_t num_counters,
@@ -169,11 +167,12 @@ dispatch_callback(rocprofiler_queue_id_t /*queue_id*/,
// Create a colleciton profile for the counters
rocprofiler_profile_config_id_t profile;
ROCPROFILER_CALL(rocprofiler_create_profile_config(
agent->id, collect_counters.data(), collect_counters.size(), &profile),
"Could not construct profile cfg");
ROCPROFILER_CALL(
rocprofiler_create_profile_config(
dispatch_data.agent_id, collect_counters.data(), collect_counters.size(), &profile),
"Could not construct profile cfg");
profile_cache.emplace(agent->id.handle, profile);
profile_cache.emplace(dispatch_data.agent_id.handle, profile);
// Return the profile to collect those counters for this dispatch
*config = profile;
}
samples/counter_collection/client.cpp
+12 -13
Fájl megtekintése
@@ -32,6 +32,7 @@
#include <unordered_map>
#include <vector>
#include <rocprofiler-sdk/fwd.h>
#include <rocprofiler-sdk/registration.h>
#include <rocprofiler-sdk/rocprofiler.h>
@@ -116,13 +117,10 @@ buffered_callback(rocprofiler_context_id_t,
* to collect the counter SQ_WAVES for all kernel dispatch packets.
*/
void
dispatch_callback(rocprofiler_queue_id_t /*queue_id*/,
const rocprofiler_agent_t* agent,
rocprofiler_correlation_id_t /*correlation_id*/,
const hsa_kernel_dispatch_packet_t* /*dispatch_packet*/,
uint64_t /*kernel_id*/,
void* /*callback_data_args*/,
rocprofiler_profile_config_id_t* config)
dispatch_callback(rocprofiler_profile_counting_dispatch_data_t dispatch_data,
rocprofiler_profile_config_id_t* config,
rocprofiler_user_data_t* /*user_data*/,
void* /*callback_data_args*/)
{
/**
* This simple example uses the same profile counter set for all agents.
@@ -135,7 +133,7 @@ dispatch_callback(rocprofiler_queue_id_t /*queue_id*/,
static std::unordered_map<uint64_t, rocprofiler_profile_config_id_t> profile_cache = {};
auto search_cache = [&]() {
if(auto pos = profile_cache.find(agent->id.handle); pos != profile_cache.end())
if(auto pos = profile_cache.find(dispatch_data.agent_id.handle); pos != profile_cache.end())
{
*config = pos->second;
return true;
@@ -158,7 +156,7 @@ dispatch_callback(rocprofiler_queue_id_t /*queue_id*/,
// Iterate through the agents and get the counters available on that agent
ROCPROFILER_CALL(rocprofiler_iterate_agent_supported_counters(
agent->id,
dispatch_data.agent_id,
[](rocprofiler_agent_id_t,
rocprofiler_counter_id_t* counters,
size_t num_counters,
@@ -192,11 +190,12 @@ dispatch_callback(rocprofiler_queue_id_t /*queue_id*/,
// Create a colleciton profile for the counters
rocprofiler_profile_config_id_t profile;
ROCPROFILER_CALL(rocprofiler_create_profile_config(
agent->id, collect_counters.data(), collect_counters.size(), &profile),
"Could not construct profile cfg");
ROCPROFILER_CALL(
rocprofiler_create_profile_config(
dispatch_data.agent_id, collect_counters.data(), collect_counters.size(), &profile),
"Could not construct profile cfg");
profile_cache.emplace(agent->id.handle, profile);
profile_cache.emplace(dispatch_data.agent_id.handle, profile);
// Return the profile to collect those counters for this dispatch
*config = profile;
}
samples/counter_collection/print_functional_counters.cpp
+48 -16
Fájl megtekintése
@@ -189,15 +189,46 @@ buffered_callback(rocprofiler_context_id_t,
}
}
void
dispatch_callback(rocprofiler_queue_id_t /*queue_id*/,
const rocprofiler_agent_t* agent,
rocprofiler_correlation_id_t /*correlation_id*/,
const hsa_kernel_dispatch_packet_t* /*dispatch_packet*/,
uint64_t /*kernel_id*/,
void* /*callback_data_args*/,
rocprofiler_profile_config_id_t* config)
using agent_map_t = std::map<uint64_t, const rocprofiler_agent_v0_t*>;
agent_map_t
get_agent_info()
{
auto iterate_cb = [](rocprofiler_agent_version_t agents_ver,
const void** agents_arr,
size_t num_agents,
void* user_data) {
if(agents_ver != ROCPROFILER_AGENT_INFO_VERSION_0)
throw std::runtime_error{"unexpected rocprofiler agent version"};
auto* agents_v = static_cast<agent_map_t*>(user_data);
for(size_t i = 0; i < num_agents; ++i)
{
const auto* itr = static_cast<const rocprofiler_agent_v0_t*>(agents_arr[i]);
agents_v->emplace(itr->id.handle, itr);
}
return ROCPROFILER_STATUS_SUCCESS;
};
auto _agents = agent_map_t{};
ROCPROFILER_CALL(
rocprofiler_query_available_agents(ROCPROFILER_AGENT_INFO_VERSION_0,
iterate_cb,
sizeof(rocprofiler_agent_t),
const_cast<void*>(static_cast<const void*>(&_agents))),
"query available agents");
return _agents;
}
void
dispatch_callback(rocprofiler_profile_counting_dispatch_data_t dispatch_data,
rocprofiler_profile_config_id_t* config,
rocprofiler_user_data_t* /*user_data*/,
void* /*callback_data_args*/)
{
static auto agents = get_agent_info();
auto& cap = *get_capture();
auto wlock = std::unique_lock{cap.m_mutex};
@@ -211,7 +242,7 @@ dispatch_callback(rocprofiler_queue_id_t /*queue_id*/,
{
std::vector<rocprofiler_counter_id_t> counters_needed;
ROCPROFILER_CALL(rocprofiler_iterate_agent_supported_counters(
agent->id,
dispatch_data.agent_id,
[](rocprofiler_agent_id_t,
rocprofiler_counter_id_t* counters,
size_t num_counters,
@@ -237,9 +268,9 @@ dispatch_callback(rocprofiler_queue_id_t /*queue_id*/,
"Could not query counter_id");
cap.expected_counter_names.emplace(found_counter.handle, std::string(version.name));
size_t expected = 0;
ROCPROFILER_CALL(
rocprofiler_query_counter_instance_count(agent->id, found_counter, &expected),
"COULD NOT QUERY INSTANCES");
ROCPROFILER_CALL(rocprofiler_query_counter_instance_count(
dispatch_data.agent_id, found_counter, &expected),
"COULD NOT QUERY INSTANCES");
cap.remaining.push_back(found_counter);
cap.expected.emplace(found_counter.handle, expected);
@@ -266,7 +297,8 @@ dispatch_callback(rocprofiler_queue_id_t /*queue_id*/,
}
if(cap.expected.empty())
{
std::clog << "No counters found for agent - " << agent->name;
std::clog << "No counters found for agent " << dispatch_data.agent_id.handle << " ("
<< agents.at(dispatch_data.agent_id.handle)->name << ")";
}
}
if(cap.remaining.empty()) return;
@@ -274,9 +306,9 @@ dispatch_callback(rocprofiler_queue_id_t /*queue_id*/,
rocprofiler_profile_config_id_t profile;
// Select the next counter to collect.
ROCPROFILER_CALL(
rocprofiler_create_profile_config(agent->id, &(cap.remaining.back()), 1, &profile),
"Could not construct profile cfg");
ROCPROFILER_CALL(rocprofiler_create_profile_config(
dispatch_data.agent_id, &(cap.remaining.back()), 1, &profile),
"Could not construct profile cfg");
cap.remaining.pop_back();
*config = profile;
samples/intercept_table/CMakeLists.txt
+1 -1
Fájl megtekintése
@@ -54,4 +54,4 @@ set_tests_properties(
ENVIRONMENT
"${ROCPROFILER_MEMCHECK_PRELOAD_ENV};HSA_TOOLS_LIB=$<TARGET_FILE:rocprofiler::rocprofiler-shared-library>"
FAIL_REGULAR_EXPRESSION
"threw an exception")
"${ROCPROFILER_DEFAULT_FAIL_REGEX}")
samples/intercept_table/main.cpp
+1 -1
Fájl megtekintése
@@ -145,7 +145,7 @@ run(int rank, int tid, hipStream_t stream, int argc, char** argv)
if(argc > 3) nsync = atoll(argv[3]);
auto_lock_t _lk{print_lock};
std::cout << "[" << rank << "][" << tid << "] M: " << M << " N: " << N << std::endl;
std::cout << "[transpose][" << rank << "][" << tid << "] M: " << M << " N: " << N << std::endl;
_lk.unlock();
std::default_random_engine _engine{std::random_device{}() * (rank + 1) * (tid + 1)};