rocm-smi and KokkosTools support (#23)

* renamed omnitrace_thread_data to thread_data

* initial implementation

* Numerous fixes and updates

- Updated timemory submodule
- Updated perfetto submodule (pulls in fixes for TRACE_EVENT)
- pthread_gotcha only after omnitrace_init_tooling
- omnitrace banner
- config settings for rocm-smi freq and devices
- critical_trace::get_entries
- OMNITRACE_BASIC_PRINT
- rocm_smi perfetto category
- redirect roctracer warnings for ROCm 4.5.0
- property specializations for rocm-smi components
- units fixes data_tracker types
- roctracer entries for pthread_create and start_thread
- omnitrace-avail defaults to settings, not components
- settings have conforming names
- settings warn about duplicates
- ptl named threads
- decreased max freq for sampler SIGALRM
- rocm-smi names thread
- rocm-smi avoids call to hipGetDeviceCount
- name roctracer activity callback threads
- fixed binary rewrite test output names

* Update lulesh example

- supports non-UVM GPU

* Lulesh tweaks + formatting

* KokkosP + Mode + Roctracer sampling deadlock fix

- kokkosp support
- omnitrace_init_library
- config::print_settings()
- config::get_mode()
- omnitrace::Mode
- omnitrace-avail improvements (removes settings)
- handle get_verbose() < 0
- disable dyninst InstrStackFrames by default
- handle perf_event_paranoid > 1 by disabling PAPI
- SIGALRM max freq to 5.0
- Name threads
- rocm-smi handles get_use_perfetto() and get_use_timemory()
- HSA_ENABLE_INTERRUPT=0 when roctracer + sampling (fixes deadlock)

* Tests, API renaming, roctracer

- disable renaming of thread 0
- verbprintf_bare
- enable dyninst merge tramp
- tweaked some omnitrace exe verbose levels
- reworked roctracer::setup and roctracer::shutdown
- rocm_smi::data::poll checks get_state()
- omnitrace_trace_finalize -> omnitrace_finalize
- omnitrace_trace_init -> omnitrace_init
- omnitrace_trace_set_env -> omnitrace_set_env
- omnitrace_trace_set_mpi -> omnitrace_set_mpi
- sampling mode does not disable timemory
- disable roctracer before shutting down rocm-smi
- lulesh tests w/ and w/o kokkosp
- lulesh tests for perfetto only
    - with --dynamic-callsites --traps --allow-overlapping
- lulesh tests for timemory only
    - with --stdlib --dynamic-callsites --traps --allow-overlapping

* Update timemory submodule

- fix for TIMEMORY_PROPERTY_SPECIALIZATION

* get_verbose() handling + timemory submodule update

- Findroctracer.cmake uses find_package(hsakmt)

* Stability fixes + rework roctracer + perfetto

- reworked roctracer start up
- critical_trace perfetto basic values
- perfetto sampling category
- sampler checks signals
- peak_rss in sampling
- pthread_gotcha::shutdown()
- rocm_smi::device_count()
- HSA_TOOLS_LIB is set
- HSA_ENABLE_INTERRUPT in omnitrace exe
- omnitrace exe verbosity level changes
- Avoid instrumenting Impl ns in Kokkos
- gpu::device_count prefers rocm_smi instead of hip
- ptl blocks signals
- fixed pthread_gotcha roctracer_data values
- removed runtime-instrument-sampling tests
- timemory submodule update

* cmake formatting

* timemory + roctracer updates

- fix timemory issue with papi_common
- fix timemory issue with units
- define roctracer::is_setup()

* Miscellaneous tweaks

- Disable sampling during runtime instrument
- Fixed warnings about dynamic callsites
- Fixed backtrace output when timemory disabled
- Test tweaks

* cmake-format

* omnitrace_target_compile_definitions

* timemory submodule update

* config, omnitrace, State, mpi_gotcha updates

- use OMNITRACE_THROW instead of direct throw
- is_attached()
- is_binary_rewrite()
- get_is_continuous_integration()
- get_debug_init()
- get_debug_finalize()
- max_thread_bookmarks default to 1
- State::Init
- app_thread oneTimeCode
- runtime instrumentation uses waitpid
- fixed init_names
- include main in MPI runs
- fixed sampling setup when disabled
- reworked mpi_gotcha
- disabled critical trace in transpose test

* cmake-format

* handle rocm_smi::device_count() exception

* CI timeouts

* Re-enable runtime-instrument + sampling
This commit is contained in:
Jonathan R. Madsen
2022-02-08 17:42:17 -06:00
committed by GitHub
parent eccba14f00
commit 39f17ae8b8
59 changed files with 5116 additions and 2592 deletions
-2
View File
@@ -55,8 +55,6 @@ add_critical_trace(int64_t _tid, size_t _cpu_cid, size_t _gpu_cid, size_t _paren
int64_t _ts_beg, int64_t _ts_val, size_t _hash, uint16_t _depth,
uint16_t _prio = 0)
{
if(!get_use_critical_trace()) return;
// clang-format off
// these are used to create unique type mutexes
struct critical_insert {};
+19 -6
View File
@@ -29,12 +29,25 @@
// forward decl of the API
extern "C"
{
/// handles configuration logic
void omnitrace_init_library(void) TIMEMORY_VISIBILITY("default");
/// starts gotcha wrappers
void omnitrace_init(const char*, bool, const char*) TIMEMORY_VISIBILITY("default");
/// shuts down all tooling and generates output
void omnitrace_finalize(void) TIMEMORY_VISIBILITY("default");
/// sets an environment variable
void omnitrace_set_env(const char* env_name, const char* env_val)
TIMEMORY_VISIBILITY("default");
/// sets whether MPI should be used
void omnitrace_set_mpi(bool use, bool attached) TIMEMORY_VISIBILITY("default");
/// starts an instrumentation region
void omnitrace_push_trace(const char* name) TIMEMORY_VISIBILITY("default");
/// stops an instrumentation region
void omnitrace_pop_trace(const char* name) TIMEMORY_VISIBILITY("default");
void omnitrace_trace_init(const char*, bool, const char*)
TIMEMORY_VISIBILITY("default");
void omnitrace_trace_finalize(void) TIMEMORY_VISIBILITY("default");
void omnitrace_trace_set_env(const char* env_name, const char* env_val)
TIMEMORY_VISIBILITY("default");
void omnitrace_trace_set_mpi(bool use, bool attached) TIMEMORY_VISIBILITY("default");
}
+1
View File
@@ -43,6 +43,7 @@
TIMEMORY_DEFINE_NS_API(api, omnitrace)
TIMEMORY_DEFINE_NS_API(api, sampling)
TIMEMORY_DEFINE_NS_API(api, rocm_smi)
namespace omnitrace
{
+1
View File
@@ -91,6 +91,7 @@ struct backtrace
private:
int64_t m_tid = 0;
int64_t m_thr_cpu_ts = 0;
int64_t m_mem_peak = 0;
size_t m_size = 0;
time_point_type m_ts = {};
data_t m_data = {};
+85 -12
View File
@@ -28,6 +28,7 @@
#include <timemory/components/macros.hpp>
#include <timemory/components/user_bundle/types.hpp>
#include <timemory/enum.h>
#include <timemory/mpl/concepts.hpp>
TIMEMORY_DECLARE_COMPONENT(roctracer)
@@ -46,9 +47,21 @@ struct backtrace_cpu_clock
{};
struct backtrace_fraction
{};
struct backtrace_gpu_busy
{};
struct backtrace_gpu_temp
{};
struct backtrace_gpu_power
{};
struct backtrace_gpu_memory
{};
using sampling_wall_clock = data_tracker<double, backtrace_wall_clock>;
using sampling_cpu_clock = data_tracker<double, backtrace_cpu_clock>;
using sampling_percent = data_tracker<double, backtrace_fraction>;
using sampling_gpu_busy = data_tracker<double, backtrace_gpu_busy>;
using sampling_gpu_temp = data_tracker<double, backtrace_gpu_temp>;
using sampling_gpu_power = data_tracker<double, backtrace_gpu_power>;
using sampling_gpu_memory = data_tracker<double, backtrace_gpu_memory>;
using roctracer = tim::component::roctracer;
} // namespace component
} // namespace omnitrace
@@ -59,7 +72,7 @@ TIMEMORY_DEFINE_CONCRETE_TRAIT(is_available, component::roctracer, false_type)
#if !defined(TIMEMORY_USE_LIBUNWIND)
TIMEMORY_DEFINE_CONCRETE_TRAIT(is_available, omnitrace::api::sampling, false_type)
TIMEMORY_DEFINE_CONCRETE_TRAIT(is_available, omnitrace::sampling::backtrace, false_type)
TIMEMORY_DEFINE_CONCRETE_TRAIT(is_available, omnitrace::component::backtrace, false_type)
TIMEMORY_DEFINE_CONCRETE_TRAIT(is_available, omnitrace::component::sampling_wall_clock,
false_type)
TIMEMORY_DEFINE_CONCRETE_TRAIT(is_available, omnitrace::component::sampling_cpu_clock,
@@ -68,6 +81,17 @@ TIMEMORY_DEFINE_CONCRETE_TRAIT(is_available, omnitrace::component::sampling_perc
false_type)
#endif
#if !defined(TIMEMORY_USE_LIBUNWIND) || !defined(OMNITRACE_USE_ROCM_SMI)
TIMEMORY_DEFINE_CONCRETE_TRAIT(is_available, omnitrace::component::sampling_gpu_busy,
false_type)
TIMEMORY_DEFINE_CONCRETE_TRAIT(is_available, omnitrace::component::sampling_gpu_temp,
false_type)
TIMEMORY_DEFINE_CONCRETE_TRAIT(is_available, omnitrace::component::sampling_gpu_power,
false_type)
TIMEMORY_DEFINE_CONCRETE_TRAIT(is_available, omnitrace::component::sampling_gpu_memory,
false_type)
#endif
TIMEMORY_PROPERTY_SPECIALIZATION(omnitrace::component::omnitrace, OMNITRACE_COMPONENT,
"omnitrace", "omnitrace_component")
TIMEMORY_PROPERTY_SPECIALIZATION(omnitrace::component::roctracer, OMNITRACE_ROCTRACER,
@@ -78,38 +102,87 @@ TIMEMORY_PROPERTY_SPECIALIZATION(omnitrace::component::sampling_cpu_clock,
OMNITRACE_SAMPLING_CPU_CLOCK, "sampling_cpu_clock", "")
TIMEMORY_PROPERTY_SPECIALIZATION(omnitrace::component::sampling_percent,
OMNITRACE_SAMPLING_PERCENT, "sampling_percent", "")
TIMEMORY_PROPERTY_SPECIALIZATION(omnitrace::component::sampling_gpu_busy,
OMNITRACE_SAMPLING_GPU_BUSY, "sampling_gpu_busy",
"sampling_gpu_util")
TIMEMORY_PROPERTY_SPECIALIZATION(omnitrace::component::sampling_gpu_memory,
OMNITRACE_SAMPLING_GPU_MEMORY_USAGE,
"sampling_gpu_memory_usage", "")
TIMEMORY_PROPERTY_SPECIALIZATION(omnitrace::component::sampling_gpu_power,
OMNITRACE_SAMPLING_GPU_POWER, "sampling_gpu_power", "")
TIMEMORY_PROPERTY_SPECIALIZATION(omnitrace::component::sampling_gpu_temp,
OMNITRACE_SAMPLING_GPU_TEMP, "sampling_gpu_temp", "")
TIMEMORY_METADATA_SPECIALIZATION(omnitrace::component::roctracer, "roctracer",
"High-precision ROCm API and kernel tracing", "")
TIMEMORY_METADATA_SPECIALIZATION(omnitrace::component::sampling_wall_clock,
"sampling_wall_clock", "Wall-clock timing",
"derived from statistical sampling")
"Derived from statistical sampling")
TIMEMORY_METADATA_SPECIALIZATION(omnitrace::component::sampling_cpu_clock,
"sampling_cpu_clock", "CPU-clock timing",
"derived from statistical sampling")
"Derived from statistical sampling")
TIMEMORY_METADATA_SPECIALIZATION(omnitrace::component::sampling_percent,
"sampling_percent",
"Fraction of wall-clock time spent in functions",
"derived from statistical sampling")
TIMEMORY_METADATA_SPECIALIZATION(omnitrace::component::roctracer, "roctracer",
"High-precision ROCm API and kernel tracing", "")
"Derived from statistical sampling")
TIMEMORY_METADATA_SPECIALIZATION(omnitrace::component::sampling_gpu_busy,
"sampling_gpu_busy",
"GPU Utilization (% busy) via ROCm-SMI",
"Derived from sampling")
TIMEMORY_METADATA_SPECIALIZATION(omnitrace::component::sampling_gpu_memory,
"sampling_gpu_memory_usage",
"GPU Memory Usage via ROCm-SMI", "Derived from sampling")
TIMEMORY_METADATA_SPECIALIZATION(omnitrace::component::sampling_gpu_power,
"sampling_gpu_power", "GPU Power Usage via ROCm-SMI",
"Derived from sampling")
TIMEMORY_METADATA_SPECIALIZATION(omnitrace::component::sampling_gpu_temp,
"sampling_gpu_temp", "GPU Temperature via ROCm-SMI",
"Derived from sampling")
// statistics type
TIMEMORY_STATISTICS_TYPE(omnitrace::component::sampling_wall_clock, double)
TIMEMORY_STATISTICS_TYPE(omnitrace::component::sampling_cpu_clock, double)
TIMEMORY_STATISTICS_TYPE(omnitrace::component::sampling_gpu_busy, double)
TIMEMORY_STATISTICS_TYPE(omnitrace::component::sampling_gpu_temp, double)
TIMEMORY_STATISTICS_TYPE(omnitrace::component::sampling_gpu_power, double)
TIMEMORY_STATISTICS_TYPE(omnitrace::component::sampling_gpu_memory, double)
// enable timing units
TIMEMORY_DEFINE_CONCRETE_TRAIT(is_timing_category,
omnitrace::component::sampling_wall_clock, true_type)
TIMEMORY_DEFINE_CONCRETE_TRAIT(uses_timing_units,
omnitrace::component::sampling_wall_clock, true_type)
TIMEMORY_DEFINE_CONCRETE_TRAIT(is_timing_category,
omnitrace::component::sampling_cpu_clock, true_type)
TIMEMORY_DEFINE_CONCRETE_TRAIT(uses_timing_units,
omnitrace::component::sampling_cpu_clock, true_type)
TIMEMORY_DEFINE_CONCRETE_TRAIT(is_timing_category, omnitrace::component::sampling_percent,
true_type)
TIMEMORY_DEFINE_CONCRETE_TRAIT(uses_timing_units,
omnitrace::component::sampling_wall_clock, true_type)
TIMEMORY_DEFINE_CONCRETE_TRAIT(uses_timing_units,
omnitrace::component::sampling_cpu_clock, true_type)
// enable percent units
TIMEMORY_DEFINE_CONCRETE_TRAIT(uses_percent_units,
omnitrace::component::sampling_gpu_busy, true_type)
// enable memory units
TIMEMORY_DEFINE_CONCRETE_TRAIT(is_memory_category,
omnitrace::component::sampling_gpu_memory, true_type)
TIMEMORY_DEFINE_CONCRETE_TRAIT(uses_memory_units,
omnitrace::component::sampling_gpu_memory, true_type)
// reporting categories (sum)
TIMEMORY_DEFINE_CONCRETE_TRAIT(report_sum, omnitrace::component::sampling_gpu_busy,
false_type)
TIMEMORY_DEFINE_CONCRETE_TRAIT(report_sum, omnitrace::component::sampling_gpu_temp,
false_type)
TIMEMORY_DEFINE_CONCRETE_TRAIT(report_sum, omnitrace::component::sampling_gpu_power,
false_type)
TIMEMORY_DEFINE_CONCRETE_TRAIT(report_sum, omnitrace::component::sampling_gpu_memory,
false_type)
// reporting categories (mean)
TIMEMORY_DEFINE_CONCRETE_TRAIT(report_mean, omnitrace::component::sampling_percent,
false_type)
TIMEMORY_DEFINE_CONCRETE_TRAIT(report_units, omnitrace::component::sampling_percent,
false_type)
// reporting categories (stats)
TIMEMORY_DEFINE_CONCRETE_TRAIT(report_statistics, omnitrace::component::sampling_percent,
false_type)
@@ -56,6 +56,7 @@ struct pthread_gotcha : tim::component::base<pthread_gotcha, void>
// generate the gotcha wrappers
static void configure();
static void shutdown();
// threads can set this to avoid starting sampling on child threads
static bool& enable_sampling_on_child_threads();
+165
View File
@@ -0,0 +1,165 @@
// Copyright (c) 2018 Advanced Micro Devices, Inc. All Rights Reserved.
//
// Permission is hereby granted, free of charge, to any person obtaining a copy
// of this software and associated documentation files (the "Software"), to deal
// with the Software without restriction, including without limitation the
// rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
// sell copies of the Software, and to permit persons to whom the Software is
// furnished to do so, subject to the following conditions:
//
// * Redistributions of source code must retain the above copyright notice,
// this list of conditions and the following disclaimers.
//
// * Redistributions in binary form must reproduce the above copyright
// notice, this list of conditions and the following disclaimers in the
// documentation and/or other materials provided with the distribution.
//
// * Neither the names of Advanced Micro Devices, Inc. nor the names of its
// contributors may be used to endorse or promote products derived from
// this Software without specific prior written permission.
//
// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
// CONTRIBUTORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS WITH
// THE SOFTWARE.
#pragma once
#include "library/common.hpp"
#include "library/components/fwd.hpp"
#include "library/defines.hpp"
#include "library/thread_data.hpp"
#include <chrono>
#include <cstdint>
#include <deque>
#include <future>
#include <limits>
#include <memory>
#include <ratio>
#include <thread>
#include <tuple>
#include <type_traits>
namespace omnitrace
{
namespace rocm_smi
{
void
setup();
void
shutdown();
uint32_t
device_count();
struct data
{
using msec_t = std::chrono::milliseconds;
using usec_t = std::chrono::microseconds;
using nsec_t = std::chrono::nanoseconds;
using promise_t = std::promise<void>;
using timestamp_t = int64_t;
using power_t = uint64_t;
using busy_perc_t = uint32_t;
using mem_usage_t = uint64_t;
using temp_t = int64_t;
TIMEMORY_DEFAULT_OBJECT(data)
explicit data(uint32_t _dev_id);
void sample(uint32_t _dev_id);
void print(std::ostream& _os) const;
template <typename Tp = nsec_t,
std::enable_if_t<!std::is_same_v<std::decay_t<Tp>, nsec_t>, int> = 0>
static void poll(std::atomic<State>* _state, Tp&& _interval, promise_t*);
static void set_state(State);
static void poll(std::atomic<State>* _state, nsec_t _interval, promise_t*);
static void post_process(uint32_t _dev_id);
uint32_t m_dev_id = std::numeric_limits<uint32_t>::max();
timestamp_t m_ts = 0;
busy_perc_t m_busy_perc = 0;
temp_t m_temp = 0;
power_t m_power = 0;
mem_usage_t m_mem_usage = 0;
friend std::ostream& operator<<(std::ostream& _os, const data& _v)
{
_v.print(_os);
return _os;
}
private:
friend void omnitrace::rocm_smi::setup();
friend void omnitrace::rocm_smi::shutdown();
static size_t device_count;
static std::set<uint32_t> device_list;
static std::unique_ptr<promise_t> polling_finished;
static std::vector<data>& get_initial();
static std::unique_ptr<std::thread>& get_thread();
static bool setup();
static bool shutdown();
};
template <
typename Tp,
std::enable_if_t<!std::is_same_v<std::decay_t<Tp>, std::chrono::nanoseconds>, int>>
void
data::poll(std::atomic<State>* _state, Tp&& _interval, promise_t* _prom)
{
poll(_state, std::chrono::duration_cast<nsec_t>(_interval), _prom);
}
using bundle_t = std::deque<data>;
using sampler_instances = thread_data<bundle_t, api::rocm_smi>;
#if !defined(OMNITRACE_USE_ROCM_SMI)
inline void
setup()
{}
inline void
shutdown()
{}
inline void data::post_process(uint32_t) {}
#endif
} // namespace rocm_smi
} // namespace omnitrace
#if defined(OMNITRACE_USE_ROCM_SMI)
# if !defined(OMNITRACE_EXTERN_COMPONENTS) || \
(defined(OMNITRACE_EXTERN_COMPONENTS) && OMNITRACE_EXTERN_COMPONENTS > 0)
# include <timemory/components/base.hpp>
# include <timemory/components/data_tracker/components.hpp>
# include <timemory/operations.hpp>
TIMEMORY_DECLARE_EXTERN_COMPONENT(
TIMEMORY_ESC(data_tracker<double, omnitrace::component::backtrace_gpu_busy>), true,
double)
TIMEMORY_DECLARE_EXTERN_COMPONENT(
TIMEMORY_ESC(data_tracker<double, omnitrace::component::backtrace_gpu_temp>), true,
double)
TIMEMORY_DECLARE_EXTERN_COMPONENT(
TIMEMORY_ESC(data_tracker<double, omnitrace::component::backtrace_gpu_power>), true,
double)
TIMEMORY_DECLARE_EXTERN_COMPONENT(
TIMEMORY_ESC(data_tracker<double, omnitrace::component::backtrace_gpu_memory>), true,
double)
# endif
#endif
+24 -8
View File
@@ -52,26 +52,42 @@ struct roctracer
static void preinit();
static void global_init() { setup(); }
static void global_finalize() { tear_down(); }
static void global_finalize() { shutdown(); }
static bool is_setup();
static void setup();
static void tear_down();
static void shutdown();
static void add_setup(const std::string&, std::function<void()>&&);
static void add_tear_down(const std::string&, std::function<void()>&&);
static void add_shutdown(const std::string&, std::function<void()>&&);
static void remove_setup(const std::string&);
static void remove_tear_down(const std::string&);
static void remove_shutdown(const std::string&);
void start();
void stop();
void set_prefix(const char* _v) { m_prefix = _v; }
private:
const char* m_prefix = nullptr;
};
#if !defined(OMNITRACE_USE_ROCTRACER)
inline void
roctracer::setup()
{}
inline void
roctracer::shutdown()
{}
inline bool
roctracer::is_setup()
{
return false;
}
#endif
} // namespace component
} // namespace tim
#if !defined(OMNITRACE_USE_ROCTRACER)
TIMEMORY_DEFINE_CONCRETE_TRAIT(is_available, component::roctracer_data, false_type)
#endif
TIMEMORY_SET_COMPONENT_API(component::roctracer_data, project::timemory, category::timing,
os::supports_unix)
TIMEMORY_DEFINE_CONCRETE_TRAIT(is_timing_category, component::roctracer_data, true_type)
@@ -90,5 +90,5 @@ roctracer_functions_t&
roctracer_setup_routines();
roctracer_functions_t&
roctracer_tear_down_routines();
roctracer_shutdown_routines();
} // namespace omnitrace
+57 -6
View File
@@ -31,6 +31,7 @@
#include "library/defines.hpp"
#include "library/state.hpp"
#include "library/timemory.hpp"
#include "timemory/macros/language.hpp"
#include <timemory/backends/threading.hpp>
@@ -43,10 +44,9 @@ namespace omnitrace
// bundle of components around omnitrace_init and omnitrace_finalize
using main_bundle_t =
tim::lightweight_tuple<comp::wall_clock, comp::peak_rss, comp::cpu_clock,
comp::cpu_util, comp::roctracer>;
comp::cpu_util, pthread_gotcha_t>;
using gotcha_bundle_t =
tim::lightweight_tuple<fork_gotcha_t, mpi_gotcha_t, pthread_gotcha_t>;
using gotcha_bundle_t = tim::lightweight_tuple<fork_gotcha_t, mpi_gotcha_t>;
// bundle of components used in instrumentation
using instrumentation_bundle_t =
@@ -69,26 +69,63 @@ using omnitrace_thread_bundle_t =
//
// Initialization routines
//
inline namespace config
{
void
configure_settings() TIMEMORY_VISIBILITY("default");
configure_settings();
void
print_config_settings(
print_banner(std::ostream& _os = std::cout);
void
print_settings(
std::ostream& _os,
std::function<bool(const std::string_view&, const std::set<std::string>&)>&& _filter);
void
print_settings();
std::string&
get_exe_name();
template <typename Tp>
bool
set_setting_value(const std::string& _name, Tp&& _v)
{
auto _instance = tim::settings::shared_instance();
auto _setting = _instance->find(_name);
if(_setting == _instance->end()) return false;
if(!_setting->second) return false;
return _setting->second->set(std::forward<Tp>(_v));
}
//
// User-configurable settings
//
std::string
get_config_file();
Mode
get_mode();
bool&
is_attached();
bool&
is_binary_rewrite();
bool
get_is_continuous_integration();
bool
get_debug_env();
bool
get_debug_init();
bool
get_debug_finalize();
bool
get_debug();
@@ -113,6 +150,9 @@ get_use_timemory();
bool&
get_use_roctracer();
bool&
get_use_rocm_smi();
bool&
get_use_sampling();
@@ -125,6 +165,9 @@ get_use_mpip();
bool&
get_use_critical_trace();
bool
get_use_kokkosp();
bool
get_timeline_sampling();
@@ -167,8 +210,9 @@ get_trace_hsa_api_types();
std::string&
get_backend();
// make this visible so omnitrace-avail can call it
std::string&
get_perfetto_output_filename();
get_perfetto_output_filename() TIMEMORY_VISIBILITY("default");
int64_t
get_critical_trace_count();
@@ -182,8 +226,15 @@ get_sampling_freq();
double&
get_sampling_delay();
double&
get_rocm_smi_freq();
std::string
get_rocm_smi_devices();
int64_t
get_critical_trace_per_row();
} // namespace config
//
// Runtime configuration data
+6 -1
View File
@@ -175,7 +175,7 @@ struct call_chain : private std::vector<entry>
}
template <Device DevT>
void generate_perfetto(std::set<entry>& _used) const;
void generate_perfetto(std::set<entry>& _used, bool _basic = false) const;
template <bool BoolV = true, typename FuncT>
bool query(FuncT&&) const;
@@ -201,6 +201,11 @@ update(int64_t _tid = threading::get_id());
void
compute(int64_t _tid = threading::get_id());
std::vector<std::pair<std::string, entry>>
get_entries(
int64_t _ts,
const std::function<bool(const entry&)>& _eval = [](const entry&) { return true; });
struct id
{};
+53
View File
@@ -33,6 +33,8 @@
namespace omnitrace
{
inline namespace config
{
bool
get_debug();
@@ -44,6 +46,7 @@ get_debug_pid();
bool
get_critical_trace_debug();
} // namespace config
} // namespace omnitrace
#if defined(TIMEMORY_USE_MPI)
@@ -80,8 +83,58 @@ get_critical_trace_debug();
fflush(stderr); \
}
#if defined(TIMEMORY_USE_MPI)
# define OMNITRACE_CONDITIONAL_THROW(COND, ...) \
if(COND) \
{ \
char _msg_buffer[2048]; \
snprintf(_msg_buffer, 2048, "[omnitrace][%i][%li] ", \
static_cast<int>(tim::dmp::rank()), tim::threading::get_id()); \
auto len = strlen(_msg_buffer); \
snprintf(_msg_buffer + len, 2048 - len, __VA_ARGS__); \
throw std::runtime_error(_msg_buffer); \
}
#else
# define OMNITRACE_CONDITIONAL_THROW(COND, ...) \
if(COND) \
{ \
char _msg_buffer[2048]; \
snprintf(_msg_buffer, 2048, "[omnitrace][%i][%li] ", \
static_cast<int>(tim::process::get_id()), \
tim::threading::get_id()); \
auto len = strlen(_msg_buffer); \
snprintf(_msg_buffer + len, 2048 - len, __VA_ARGS__); \
throw std::runtime_error(_msg_buffer); \
}
#endif
#define OMNITRACE_CONDITIONAL_BASIC_THROW(COND, ...) \
if(COND) \
{ \
char _msg_buffer[2048]; \
snprintf(_msg_buffer, 2048, "[omnitrace] " __VA_ARGS__); \
auto len = strlen(_msg_buffer); \
snprintf(_msg_buffer + len, 2048 - len, __VA_ARGS__); \
throw std::runtime_error(_msg_buffer); \
}
#define OMNITRACE_DEBUG(...) \
OMNITRACE_CONDITIONAL_PRINT(::omnitrace::get_debug(), __VA_ARGS__)
#define OMNITRACE_PRINT(...) OMNITRACE_CONDITIONAL_PRINT(true, __VA_ARGS__)
#define OMNITRACE_CT_DEBUG(...) \
OMNITRACE_CONDITIONAL_PRINT(::omnitrace::get_critical_trace_debug(), __VA_ARGS__)
#define OMNITRACE_BASIC_PRINT(...) OMNITRACE_CONDITIONAL_BASIC_PRINT(true, __VA_ARGS__)
#define OMNITRACE_THROW(...) OMNITRACE_CONDITIONAL_THROW(true, __VA_ARGS__)
#define OMNITRACE_BASIC_THROW(...) OMNITRACE_CONDITIONAL_BASIC_THROW(true, __VA_ARGS__)
#include <string>
namespace std
{
inline std::string
to_string(bool _v)
{
return (_v) ? "true" : "false";
}
} // namespace std
+13 -7
View File
@@ -36,11 +36,17 @@
// clang-format on
#define TIMEMORY_USER_COMPONENT_ENUM \
OMNITRACE_SAMPLING_WALL_CLOCK_idx, OMNITRACE_SAMPLING_CPU_CLOCK_idx, \
OMNITRACE_SAMPLING_PERCENT_idx, OMNITRACE_COMPONENT_idx, OMNITRACE_ROCTRACER_idx,
OMNITRACE_COMPONENT_idx, OMNITRACE_ROCTRACER_idx, OMNITRACE_SAMPLING_WALL_CLOCK_idx, \
OMNITRACE_SAMPLING_CPU_CLOCK_idx, OMNITRACE_SAMPLING_PERCENT_idx, \
OMNITRACE_SAMPLING_GPU_POWER_idx, OMNITRACE_SAMPLING_GPU_TEMP_idx, \
OMNITRACE_SAMPLING_GPU_BUSY_idx, OMNITRACE_SAMPLING_GPU_MEMORY_USAGE_idx,
#define OMNITRACE_COMPONENT OMNITRACE_COMPONENT_idx
#define OMNITRACE_ROCTRACER OMNITRACE_ROCTRACER_idx
#define OMNITRACE_SAMPLING_WALL_CLOCK OMNITRACE_SAMPLING_WALL_CLOCK_idx
#define OMNITRACE_SAMPLING_CPU_CLOCK OMNITRACE_SAMPLING_CPU_CLOCK_idx
#define OMNITRACE_SAMPLING_PERCENT OMNITRACE_SAMPLING_PERCENT_idx
#define OMNITRACE_COMPONENT OMNITRACE_COMPONENT_idx
#define OMNITRACE_ROCTRACER OMNITRACE_ROCTRACER_idx
#define OMNITRACE_SAMPLING_WALL_CLOCK OMNITRACE_SAMPLING_WALL_CLOCK_idx
#define OMNITRACE_SAMPLING_CPU_CLOCK OMNITRACE_SAMPLING_CPU_CLOCK_idx
#define OMNITRACE_SAMPLING_PERCENT OMNITRACE_SAMPLING_PERCENT_idx
#define OMNITRACE_SAMPLING_GPU_POWER OMNITRACE_SAMPLING_GPU_POWER_idx
#define OMNITRACE_SAMPLING_GPU_TEMP OMNITRACE_SAMPLING_GPU_TEMP_idx
#define OMNITRACE_SAMPLING_GPU_BUSY OMNITRACE_SAMPLING_GPU_BUSY_idx
#define OMNITRACE_SAMPLING_GPU_MEMORY_USAGE OMNITRACE_SAMPLING_GPU_MEMORY_USAGE_idx
+46
View File
@@ -33,6 +33,9 @@
# define PERFETTO_CATEGORIES \
perfetto::Category("host").SetDescription("Host-side function tracing"), \
perfetto::Category("device").SetDescription("Device-side function tracing"), \
perfetto::Category("rocm_smi").SetDescription("Device-level metrics"), \
perfetto::Category("sampling") \
.SetDescription("Metrics derived from sampling"), \
perfetto::Category("host-critical-trace") \
.SetDescription("Host-side critical traces"), \
perfetto::Category("device-critical-trace") \
@@ -41,6 +44,9 @@
# define PERFETTO_CATEGORIES \
perfetto::Category("host").SetDescription("Host-side function tracing"), \
perfetto::Category("device").SetDescription("Device-side function tracing"), \
perfetto::Category("rocm_smi").SetDescription("Device-level metrics"), \
perfetto::Category("sampling") \
.SetDescription("Metrics derived from sampling"), \
perfetto::Category("host-critical-trace") \
.SetDescription("Host-side critical traces"), \
perfetto::Category("device-critical-trace") \
@@ -87,4 +93,44 @@ public:
PERFETTO_DECLARE_DATA_SOURCE_STATIC_MEMBERS(CustomDataSource);
#endif
template <typename Tp>
struct perfetto_counter_track
{
using track_map_t = std::map<uint32_t, std::vector<perfetto::CounterTrack>>;
using name_map_t = std::map<uint32_t, std::vector<std::string>>;
using data_t = std::pair<name_map_t, track_map_t>;
static auto init() { (void) get_data(); }
static auto exists(size_t _idx, int64_t _n = -1)
{
bool _v = get_data().second.count(_idx) != 0;
if(_n < 0 || !_v) return _v;
return static_cast<size_t>(_n) < get_data().second.at(_idx).size();
}
static size_t size(size_t _idx)
{
bool _v = get_data().second.count(_idx) != 0;
if(!_v) return 0;
return get_data().second.at(_idx).size();
}
static auto emplace(size_t _idx, const std::string& _v, const char* _units)
{
get_data().first[_idx].emplace_back(_v);
get_data().second[_idx].emplace_back(get_data().first[_idx].back().c_str(),
_units);
}
static auto& at(size_t _idx, size_t _n) { return get_data().second.at(_idx).at(_n); }
private:
static data_t& get_data()
{
static auto* _v = new data_t{};
return *_v;
}
};
} // namespace omnitrace
+95
View File
@@ -0,0 +1,95 @@
// MIT License
//
// Copyright (c) 2022 Advanced Micro Devices, Inc. All Rights Reserved.
//
// Permission is hereby granted, free of charge, to any person obtaining a copy
// of this software and associated documentation files (the "Software"), to deal
// in the Software without restriction, including without limitation the rights
// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
// copies of the Software, and to permit persons to whom the Software is
// furnished to do so, subject to the following conditions:
//
// The above copyright notice and this permission notice shall be included in all
// copies or substantial portions of the Software.
//
// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
// SOFTWARE.
#pragma once
#include <iostream>
#include <ostream>
#include <sstream>
#include <streambuf>
#include <string>
namespace omnitrace
{
inline namespace config
{
bool
get_debug();
int
get_verbose();
} // namespace config
struct redirect
{
redirect(std::ostream& _os, std::string _expected)
: m_os{ _os }
, m_expected{ std::move(_expected) }
{
if(!get_debug())
{
// save stream buffer
m_strm_buffer = m_os.rdbuf();
// redirect to stringstream
_os.rdbuf(m_buffer.rdbuf());
}
}
~redirect()
{
if(!m_strm_buffer) return;
// restore stream buffer
m_os.rdbuf(m_strm_buffer);
auto _v = m_buffer.str();
_v = replace(m_buffer.str(), '\n');
auto _expect = replace(m_expected, '\n');
if(_v != _expect)
{
if(get_verbose() > 0)
std::cerr << "[omnitrace::redirect] Expected:\n[omnitrace::redirect] "
<< _expect
<< "\n[omnitrace::redirect] Found:\n[omnitrace::redirect] "
<< _v << "\n";
if(get_verbose() <= 0 || (&m_os != &std::cerr && &m_os != &std::cout))
m_os << m_buffer.str() << std::flush;
}
}
private:
template <typename Tp>
static std::string replace(std::string _v, Tp _c, const std::string& _s = " ")
{
while(true)
{
auto _pos = _v.find(_c);
if(_pos == std::string::npos) break;
_v = _v.replace(_pos, 1, _s);
}
return _v;
}
std::ostream& m_os;
std::string m_expected = {};
std::stringstream m_buffer{};
std::streambuf* m_strm_buffer = nullptr;
};
} // namespace omnitrace
+6 -2
View File
@@ -46,6 +46,10 @@ using component::backtrace_cpu_clock; // NOLINT
using component::backtrace_fraction; // NOLINT
using component::backtrace_wall_clock; // NOLINT
using component::sampling_cpu_clock;
using component::sampling_gpu_busy;
using component::sampling_gpu_memory;
using component::sampling_gpu_power;
using component::sampling_gpu_temp;
using component::sampling_percent;
using component::sampling_wall_clock;
@@ -64,7 +68,7 @@ void unblock_signals(std::set<int> = {});
using bundle_t = tim::lightweight_tuple<backtrace>;
using sampler_t = tim::sampling::sampler<bundle_t, tim::sampling::dynamic>;
using sampler_instances = omnitrace_thread_data<sampler_t, api::sampling>;
using sampler_instances = thread_data<sampler_t, api::sampling>;
std::unique_ptr<sampler_t>&
get_sampler(int64_t _tid = threading::get_id());
@@ -76,7 +80,7 @@ TIMEMORY_DEFINE_CONCRETE_TRAIT(prevent_reentry, omnitrace::sampling::sampler_t,
std::true_type)
TIMEMORY_DEFINE_CONCRETE_TRAIT(check_signals, omnitrace::sampling::sampler_t,
std::false_type)
std::true_type)
TIMEMORY_DEFINE_CONCRETE_TRAIT(buffer_size, omnitrace::sampling::sampler_t,
TIMEMORY_ESC(std::integral_constant<size_t, 256>))
+18
View File
@@ -31,7 +31,25 @@ enum class State : unsigned short
{
DelayedInit = 0,
PreInit,
Init,
Active,
Finalized
};
enum class Mode : unsigned short
{
Trace = 0,
Sampling
};
} // namespace omnitrace
#include <string>
namespace std
{
std::string
to_string(omnitrace::State _v);
std::string
to_string(omnitrace::Mode _v);
} // namespace std
+8 -8
View File
@@ -40,7 +40,7 @@ namespace omnitrace
static constexpr size_t max_supported_threads = OMNITRACE_MAX_THREADS;
template <typename Tp, typename Tag = void, size_t MaxThreads = max_supported_threads>
struct omnitrace_thread_data
struct thread_data
{
using instance_array_t = std::array<std::unique_ptr<Tp>, MaxThreads>;
using construct_on_init = std::true_type;
@@ -58,7 +58,7 @@ struct omnitrace_thread_data
template <typename Tp, typename Tag, size_t MaxThreads>
template <typename... Args>
void
omnitrace_thread_data<Tp, Tag, MaxThreads>::construct(Args&&... _args)
thread_data<Tp, Tag, MaxThreads>::construct(Args&&... _args)
{
// construct outside of lambda to prevent data-race
static auto& _instances = instances();
@@ -72,14 +72,14 @@ omnitrace_thread_data<Tp, Tag, MaxThreads>::construct(Args&&... _args)
template <typename Tp, typename Tag, size_t MaxThreads>
std::unique_ptr<Tp>&
omnitrace_thread_data<Tp, Tag, MaxThreads>::instance()
thread_data<Tp, Tag, MaxThreads>::instance()
{
return instances().at(threading::get_id());
}
template <typename Tp, typename Tag, size_t MaxThreads>
typename omnitrace_thread_data<Tp, Tag, MaxThreads>::instance_array_t&
omnitrace_thread_data<Tp, Tag, MaxThreads>::instances()
typename thread_data<Tp, Tag, MaxThreads>::instance_array_t&
thread_data<Tp, Tag, MaxThreads>::instances()
{
static auto _v = instance_array_t{};
return _v;
@@ -88,7 +88,7 @@ omnitrace_thread_data<Tp, Tag, MaxThreads>::instances()
template <typename Tp, typename Tag, size_t MaxThreads>
template <typename... Args>
std::unique_ptr<Tp>&
omnitrace_thread_data<Tp, Tag, MaxThreads>::instance(construct_on_init, Args&&... _args)
thread_data<Tp, Tag, MaxThreads>::instance(construct_on_init, Args&&... _args)
{
construct(std::forward<Args>(_args)...);
return instances().at(threading::get_id());
@@ -96,8 +96,8 @@ omnitrace_thread_data<Tp, Tag, MaxThreads>::instance(construct_on_init, Args&&..
template <typename Tp, typename Tag, size_t MaxThreads>
template <typename... Args>
typename omnitrace_thread_data<Tp, Tag, MaxThreads>::instance_array_t&
omnitrace_thread_data<Tp, Tag, MaxThreads>::instances(construct_on_init, Args&&... _args)
typename thread_data<Tp, Tag, MaxThreads>::instance_array_t&
thread_data<Tp, Tag, MaxThreads>::instances(construct_on_init, Args&&... _args)
{
static auto _v = [&]() {
auto _internal = instance_array_t{};
+8 -4
View File
@@ -160,6 +160,10 @@ static regexvec_t file_exclude = {};
if(verbose_level >= LEVEL) fprintf(stdout, "[omnitrace][exe] " __VA_ARGS__); \
fflush(stdout);
#define verbprintf_bare(LEVEL, ...) \
if(verbose_level >= LEVEL) fprintf(stdout, __VA_ARGS__); \
fflush(stdout);
//======================================================================================//
template <typename... T>
@@ -477,7 +481,7 @@ dump_info(const string_t& _oname, const fmodset_t& _data, int _level, bool _fail
{
verbprintf(_level, "Dumping '%s'... ", _oname.c_str());
dump_info(ofs, _data);
verbprintf(_level, "Done\n");
verbprintf_bare(_level, "Done\n");
}
else
{
@@ -597,7 +601,7 @@ omnitrace_get_address_space(patch_pointer_t& _bpatch, int _cmdc, char** _cmdv,
_name.c_str());
throw std::runtime_error("Failed to open binary");
}
verbprintf(1, "Done\n");
verbprintf_bare(1, "Done\n");
}
else if(_pid >= 0)
{
@@ -611,7 +615,7 @@ omnitrace_get_address_space(patch_pointer_t& _bpatch, int _cmdc, char** _cmdv,
(int) _pid);
throw std::runtime_error("Failed to attach to process");
}
verbprintf(1, "Done\n");
verbprintf_bare(1, "Done\n");
}
else
{
@@ -630,7 +634,7 @@ omnitrace_get_address_space(patch_pointer_t& _bpatch, int _cmdc, char** _cmdv,
ss.str().c_str());
throw std::runtime_error("Failed to create process");
}
verbprintf(1, "Done\n");
verbprintf_bare(1, "Done\n");
}
return mutatee;