Rework sampling and colorized logs (#140)
## Overview
This is a significant PR which has 3 very notable characteristics:
1. Omnitrace colorizes most of it's logging
2. Completely reworked the sampling
- Samples now record the current instruction pointers instead of strings
- This _dramatically_ decreases the overhead of taking a sample
- The collection of metrics during a sample are split out into another component, enabling that data collection to be disabled -- which decreases the sampling overhead even further
- When both `OMNITRACE_SAMPLING_CPUTIME` and `OMNITRACE_SAMPLING_REALTIME` are ON:
- `OMNITRACE_SAMPLING_CPUTIME_FREQ` and `OMNITRACE_SAMPLING_REALTIME_FREQ` can be used to individually control the sampling frequency
- `OMNITRACE_SAMPLING_CPUTIME_DELAY` and `OMNITRACE_SAMPLING_REALTIME_DELAY` can be used to individually control the delay time before starting
- Now, omnitrace does not start a real-time sampler on the main thread unless `OMNITRACE_SAMPLING_REALTIME` is ON
- In the future, an `OMNITRACE_SAMPLING_TIDS` (and real-time, cpu-time variants) configuration variable(s) will allow you to select which threads will be sampled
3. Files produced by `omnitrace` exe -- `available-instr.txt`, `instrumented-instr.txt`, etc. -- now no longer has `-instr` suffix and are placed in `instrumentation/` subfolder, i.e. `available-instr.txt` -> instrumentation/available.txt`
- This helped de-clutter the output folder
Most of the other edits were reorganization (e.g. internal namespace changes), cleanup, and splitting up functionality.
## Bug Fixes
There is a bug fix with respect to the HSA callbacks which disabled sampling on child threads when an HSA API call was made
## Details
- created thread_info struct for mapping different thread IDs
- reorganized file structure significantly
- added categories.hpp, concepts.hpp
- moved around name trait definitions
- moved all omnitrace components into `omnitrace::component` namespace
- there was a lot of inconsistency b/t using `tim::component` in some places and `omnitrace::component`
- added macros like OMNITRACE_DECLARE_COMPONENT in lieu of TIMEMORY_DECLARE_COMPONENT
- OMNITRACE_CRITICAL_TRACE_NUM_THREADS -> OMNITRACE_THREAD_POOL_SIZE
- roctracer and critical_trace use same thread pool
- critical_trace functions do not lock anymore bc of thread-local TaskGroup
- added `component::local_category_region` to support using `component::category_region` without explicitly passing in name
- removed `component::omnitrace` (unused)
- migrated KokkosP and OMPT to use `component::local_category_region`
- removed `component::user_region` as a result
- migrated omnitrace_{push,pop}_{trace,region}_hidden to use component::category_region
- removed `component::functors` as a result
- migrated some ppdefs
- `api::omnitrace` -> `project::omnitrace`
- `api::(...)` -> `category::(...)`
- improved recording the execution time of threads
- migrated this functionality out of pthread_create_gotcha and into thread_info
- moved mpi_gotcha, fork_gotcha, exit_gotcha, rcclp into omnitrace::component namespace
- split backtrace up into backtrace, backtrace_metrics, backtrace_timestamp components
- sampling.cpp handles setup and post-processing that was formerly in backtrace
- updated logging to use colors
- `OMNITRACE_COLORIZED_LOG` config variable
- updated docs on JSON output from timemory
- instrumentation info in instrumentation subfolder
- added testing for KokkosP entries
- added testing for ompt entries
- add_critical_trace function defined in critical_trace.hpp
- disable push_thread_state and pop_thread_state when thread state is Disabled or Completed
- add comp::page_rss to main bundle
- thread_data supports std::optional instead of std::unique_ptr
- thread_data supports tim::identity<T> to avoid unique_ptr or optional
- tracing::record_thread_start_time()
- tracing::push_timemory and tracing::pop_timemory are templated on CategoryT
- removed anonymous namespace from omnitrace::utility
- sampling backtrace stores instruction pointers instead of strings
- component::category_region updates
- handle disabled thread state
- handle finalized state
- fewer debug messages
- invoke thread_init()
- invoke thread_init_sampling()
- handle push/pop count based on category
- push/pop count only modified when used
- component::cpu_freq
- components/ensure_storage.hpp
- reworked the pthread_create replacement function
- updated parallel-overhead example to report # of times locked
- OMNITRACE_MAX_UNWIND_DEPTH build option
- update timemory submodule
This commit is contained in:
Jonathan R. Madsen
committed by
GitHub
GitHub
parent
a1dcd1bc4b
commit
808ea7dfa7
@@ -21,11 +21,20 @@
|
||||
// SOFTWARE.
|
||||
|
||||
#include "library/sampling.hpp"
|
||||
#include "library/common.hpp"
|
||||
#include "library/components/backtrace.hpp"
|
||||
#include "library/components/backtrace_metrics.hpp"
|
||||
#include "library/components/backtrace_timestamp.hpp"
|
||||
#include "library/components/fwd.hpp"
|
||||
#include "library/components/pthread_gotcha.hpp"
|
||||
#include "library/config.hpp"
|
||||
#include "library/debug.hpp"
|
||||
#include "library/ptl.hpp"
|
||||
#include "library/runtime.hpp"
|
||||
#include "library/thread_data.hpp"
|
||||
#include "library/thread_info.hpp"
|
||||
#include "library/tracing.hpp"
|
||||
#include "library/utility.hpp"
|
||||
|
||||
#include <timemory/backends/papi.hpp>
|
||||
#include <timemory/backends/threading.hpp>
|
||||
@@ -67,21 +76,30 @@ namespace omnitrace
|
||||
{
|
||||
namespace sampling
|
||||
{
|
||||
using bundle_t = tim::lightweight_tuple<backtrace>;
|
||||
using sampler_t = tim::sampling::sampler<bundle_t, tim::sampling::dynamic>;
|
||||
using hw_counters = typename component::backtrace_metrics::hw_counters;
|
||||
using signal_type_instances = thread_data<std::set<int>, category::sampling>;
|
||||
using sampler_running_instances = thread_data<bool, category::sampling>;
|
||||
using bundle_t =
|
||||
tim::lightweight_tuple<backtrace_timestamp, backtrace, backtrace_metrics>;
|
||||
using sampler_t = tim::sampling::sampler<bundle_t, tim::sampling::dynamic>;
|
||||
using sampler_instances = thread_data<sampler_t, category::sampling>;
|
||||
using sampler_init_instances = thread_data<bundle_t, category::sampling>;
|
||||
|
||||
using tim::sampling::timer;
|
||||
} // namespace sampling
|
||||
} // namespace omnitrace
|
||||
|
||||
OMNITRACE_DEFINE_CONCRETE_TRAIT(prevent_reentry, sampling::sampler_t, std::true_type)
|
||||
|
||||
OMNITRACE_DEFINE_CONCRETE_TRAIT(provide_backtrace, sampling::sampler_t, std::false_type)
|
||||
|
||||
OMNITRACE_DEFINE_CONCRETE_TRAIT(buffer_size, sampling::sampler_t,
|
||||
TIMEMORY_ESC(std::integral_constant<size_t, 1024>))
|
||||
|
||||
namespace omnitrace
|
||||
{
|
||||
namespace sampling
|
||||
{
|
||||
using hw_counters = typename component::backtrace::hw_counters;
|
||||
using signal_type_instances = thread_data<std::set<int>, api::sampling>;
|
||||
using backtrace_init_instances = thread_data<backtrace, api::sampling>;
|
||||
using sampler_running_instances = thread_data<bool, api::sampling>;
|
||||
using papi_vector_instances = thread_data<hw_counters, api::sampling>;
|
||||
|
||||
namespace
|
||||
{
|
||||
template <typename... Args>
|
||||
@@ -119,6 +137,166 @@ get_signal_names(Tp&& _v)
|
||||
" ";
|
||||
return _sig_names.substr(0, _sig_names.length() - 1);
|
||||
}
|
||||
|
||||
unique_ptr_t<sampler_t>&
|
||||
get_sampler(int64_t _tid = threading::get_id())
|
||||
{
|
||||
static auto& _v = sampler_instances::instances();
|
||||
return _v.at(_tid);
|
||||
}
|
||||
|
||||
unique_ptr_t<bundle_t>&
|
||||
get_sampler_init(int64_t _tid = threading::get_id())
|
||||
{
|
||||
static auto& _v = sampler_init_instances::instances();
|
||||
if(!_v.at(_tid)) _v.at(_tid) = unique_ptr_t<bundle_t>{ new bundle_t{} };
|
||||
return _v.at(_tid);
|
||||
}
|
||||
|
||||
unique_ptr_t<bool>&
|
||||
get_sampler_running(int64_t _tid)
|
||||
{
|
||||
static auto& _v = sampler_running_instances::instances(
|
||||
sampler_running_instances::construct_on_init{}, false);
|
||||
return _v.at(_tid);
|
||||
}
|
||||
|
||||
std::set<int>
|
||||
configure(bool _setup, int64_t _tid = threading::get_id())
|
||||
{
|
||||
const auto& _info = thread_info::get(_tid, InternalTID);
|
||||
auto& _sampler = sampling::get_sampler(_tid);
|
||||
auto& _running = get_sampler_running(_tid);
|
||||
bool _is_running = (!_running) ? false : *_running;
|
||||
auto& _signal_types = sampling::get_signal_types(_tid);
|
||||
|
||||
pthread_gotcha::push_enable_sampling_on_child_threads(false);
|
||||
auto _dtor = scope::destructor{ []() {
|
||||
pthread_gotcha::pop_enable_sampling_on_child_threads();
|
||||
} };
|
||||
|
||||
if(_setup && !_sampler && !_is_running && !_signal_types->empty())
|
||||
{
|
||||
// if this thread has an offset ID, that means it was created internally
|
||||
// and is probably here bc it called a function which was instrumented.
|
||||
// thus we should not start a sampler for it
|
||||
if(_tid > 0 && _info && _info->is_offset) return std::set<int>{};
|
||||
// if the thread state is disabled or completed, return
|
||||
if(_info && _info->index_data->internal_value == _tid &&
|
||||
get_thread_state() == ThreadState::Disabled)
|
||||
return std::set<int>{};
|
||||
|
||||
(void) get_debug_sampling(); // make sure query in sampler does not allocate
|
||||
assert(_tid == threading::get_id());
|
||||
|
||||
if(trait::runtime_enabled<backtrace_metrics>::get())
|
||||
backtrace_metrics::configure(_setup, _tid);
|
||||
|
||||
// NOTE: signals need to be unblocked by calling function
|
||||
sampling::block_signals(*_signal_types);
|
||||
|
||||
auto _verbose = std::min<int>(get_verbose() - 2, 2);
|
||||
if(get_debug_sampling()) _verbose = 2;
|
||||
|
||||
OMNITRACE_DEBUG("Configuring sampler for thread %lu...\n", _tid);
|
||||
sampling::sampler_instances::construct("omnitrace", _tid, _verbose);
|
||||
|
||||
_sampler->set_flags(SA_RESTART);
|
||||
_sampler->set_verbose(_verbose);
|
||||
|
||||
if(_signal_types->count(get_realtime_signal()) > 0)
|
||||
{
|
||||
_sampler->configure(timer{ get_realtime_signal(), CLOCK_REALTIME,
|
||||
SIGEV_THREAD_ID, get_sampling_real_freq(),
|
||||
get_sampling_real_delay(), _tid,
|
||||
threading::get_sys_tid() });
|
||||
}
|
||||
|
||||
if(_signal_types->count(get_cputime_signal()) > 0)
|
||||
{
|
||||
_sampler->configure(timer{ get_cputime_signal(), CLOCK_THREAD_CPUTIME_ID,
|
||||
SIGEV_THREAD_ID, get_sampling_cpu_freq(),
|
||||
get_sampling_cpu_delay(), _tid,
|
||||
threading::get_sys_tid() });
|
||||
}
|
||||
|
||||
static_assert(tim::trait::buffer_size<sampling::sampler_t>::value > 0,
|
||||
"Error! Zero buffer size");
|
||||
|
||||
OMNITRACE_CONDITIONAL_THROW(
|
||||
_sampler->get_buffer_size() !=
|
||||
tim::trait::buffer_size<sampling::sampler_t>::value,
|
||||
"dynamic sampler has a buffer size different from static trait: %zu instead "
|
||||
"of %zu",
|
||||
_sampler->get_buffer_size(),
|
||||
tim::trait::buffer_size<sampling::sampler_t>::value);
|
||||
|
||||
OMNITRACE_CONDITIONAL_THROW(
|
||||
_sampler->get_buffer_size() <= 0,
|
||||
"dynamic sampler requires a positive buffer size: %zu",
|
||||
_sampler->get_buffer_size());
|
||||
|
||||
for(auto itr : *_signal_types)
|
||||
{
|
||||
const char* _type = (itr == get_realtime_signal()) ? "wall" : "CPU";
|
||||
const auto* _timer = _sampler->get_timer(itr);
|
||||
if(_timer)
|
||||
{
|
||||
OMNITRACE_VERBOSE(
|
||||
1,
|
||||
"[SIG%i] Sampler for thread %lu will be triggered %.1fx per "
|
||||
"second of %s-time (every %.3e milliseconds)...\n",
|
||||
itr, _tid, _timer->get_frequency(units::sec), _type,
|
||||
_timer->get_period(units::msec));
|
||||
}
|
||||
}
|
||||
|
||||
*_running = true;
|
||||
sampling::get_sampler_init(_tid)->sample();
|
||||
_sampler->start();
|
||||
}
|
||||
else if(!_setup && _sampler && _is_running)
|
||||
{
|
||||
OMNITRACE_DEBUG("Destroying sampler for thread %lu...\n", _tid);
|
||||
*_running = false;
|
||||
|
||||
if(_tid == threading::get_id() && !_signal_types->empty())
|
||||
{
|
||||
sampling::block_signals(*_signal_types);
|
||||
}
|
||||
|
||||
if(_tid == 0)
|
||||
{
|
||||
// this propagates to all threads
|
||||
_sampler->ignore(*_signal_types);
|
||||
for(int64_t i = 1; i < OMNITRACE_MAX_THREADS; ++i)
|
||||
{
|
||||
if(sampling::get_sampler(i))
|
||||
{
|
||||
sampling::get_sampler(i)->stop();
|
||||
sampling::get_sampler(i)->reset();
|
||||
*get_sampler_running(i) = false;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
_sampler->stop();
|
||||
|
||||
if(trait::runtime_enabled<backtrace_metrics>::get())
|
||||
backtrace_metrics::configure(_setup, _tid);
|
||||
|
||||
OMNITRACE_DEBUG("Sampler destroyed for thread %lu\n", _tid);
|
||||
}
|
||||
|
||||
return (_signal_types) ? *_signal_types : std::set<int>{};
|
||||
}
|
||||
|
||||
void
|
||||
post_process_perfetto(int64_t _tid, const bundle_t* _init,
|
||||
const std::vector<bundle_t*>& _data);
|
||||
void
|
||||
post_process_timemory(int64_t _tid, const bundle_t* _init,
|
||||
const std::vector<bundle_t*>& _data);
|
||||
} // namespace
|
||||
|
||||
unique_ptr_t<std::set<int>>&
|
||||
@@ -133,13 +311,13 @@ std::set<int>
|
||||
setup()
|
||||
{
|
||||
if(!get_use_sampling()) return std::set<int>{};
|
||||
return backtrace::configure(true);
|
||||
return configure(true);
|
||||
}
|
||||
|
||||
std::set<int>
|
||||
shutdown()
|
||||
{
|
||||
return backtrace::configure(false);
|
||||
return configure(false);
|
||||
}
|
||||
|
||||
void
|
||||
@@ -176,11 +354,366 @@ unblock_signals(std::set<int> _signals)
|
||||
thread_sigmask(SIG_UNBLOCK, &_v, nullptr);
|
||||
}
|
||||
|
||||
unique_ptr_t<sampler_t>&
|
||||
get_sampler(int64_t _tid)
|
||||
void
|
||||
post_process()
|
||||
{
|
||||
static auto& _v = sampler_instances::instances();
|
||||
return _v.at(_tid);
|
||||
omnitrace::component::backtrace::stop();
|
||||
OMNITRACE_VERBOSE(2 || get_debug_sampling(), "Stopping backtrace metrics...\n");
|
||||
|
||||
for(size_t i = 0; i < max_supported_threads; ++i)
|
||||
backtrace_metrics::configure(false, i);
|
||||
|
||||
OMNITRACE_VERBOSE(1 || get_debug_sampling(), "Post-processing sampling data...\n");
|
||||
|
||||
for(size_t i = 0; i < max_supported_threads; ++i)
|
||||
{
|
||||
auto& _sampler = get_sampler(i);
|
||||
|
||||
if(!_sampler)
|
||||
{
|
||||
// this should be relatively common
|
||||
OMNITRACE_CONDITIONAL_PRINT(
|
||||
get_debug() && get_verbose() >= 2,
|
||||
"Post-processing sampling entries for thread %lu skipped (no sampler)\n",
|
||||
i);
|
||||
continue;
|
||||
}
|
||||
|
||||
auto* _init = get_sampler_init(i).get();
|
||||
|
||||
if(!_init)
|
||||
{
|
||||
// this is not common
|
||||
OMNITRACE_PRINT("Post-processing sampling entries for thread %lu skipped "
|
||||
"(not initialized)\n",
|
||||
i);
|
||||
continue;
|
||||
}
|
||||
|
||||
const auto& _thread_info = thread_info::get(i, InternalTID);
|
||||
|
||||
OMNITRACE_VERBOSE(3 || get_debug_sampling(),
|
||||
"Getting sampler data for thread %lu...\n", i);
|
||||
|
||||
_sampler->stop();
|
||||
auto& _raw_data = _sampler->get_data();
|
||||
|
||||
OMNITRACE_VERBOSE(0 || get_debug_sampling(),
|
||||
"Sampler data for thread %lu has %zu initial entries...\n", i,
|
||||
_raw_data.size());
|
||||
|
||||
OMNITRACE_CI_THROW(
|
||||
_sampler->get_sample_count() != _raw_data.size(),
|
||||
"Error! sampler recorded %zu samples but %zu samples were returned\n",
|
||||
_sampler->get_sample_count(), _raw_data.size());
|
||||
// single sample that is useless (backtrace to unblocking signals)
|
||||
if(_raw_data.size() == 1 && _raw_data.front().size() <= 1) _raw_data.clear();
|
||||
|
||||
std::vector<sampling::bundle_t*> _data{};
|
||||
for(auto& itr : _raw_data)
|
||||
{
|
||||
_data.reserve(_data.size() + itr.size());
|
||||
auto* _bt = itr.get<backtrace>();
|
||||
auto* _ts = itr.get<backtrace_timestamp>();
|
||||
if(!_bt || !_ts) continue;
|
||||
if(_bt->empty()) continue;
|
||||
if(!_thread_info->is_valid_time(_ts->get_timestamp())) continue;
|
||||
_data.emplace_back(&itr);
|
||||
}
|
||||
|
||||
if(_data.empty())
|
||||
{
|
||||
OMNITRACE_VERBOSE(
|
||||
3 || get_debug_sampling(),
|
||||
"Sampler data for thread %lu has %zu valid entries... (skipped)\n", i,
|
||||
_raw_data.size());
|
||||
continue;
|
||||
}
|
||||
|
||||
OMNITRACE_VERBOSE(0 || get_debug_sampling(),
|
||||
"Sampler data for thread %lu has %zu valid entries...\n", i,
|
||||
_raw_data.size());
|
||||
|
||||
if(get_use_perfetto()) post_process_perfetto(i, _init, _data);
|
||||
if(get_use_timemory()) post_process_timemory(i, _init, _data);
|
||||
}
|
||||
|
||||
OMNITRACE_VERBOSE(0 || get_debug_sampling(),
|
||||
"Post-processing sampling entries completed\n");
|
||||
|
||||
for(size_t i = 0; i < max_supported_threads; ++i)
|
||||
{
|
||||
get_sampler(i).reset();
|
||||
}
|
||||
|
||||
OMNITRACE_VERBOSE(0 || get_debug_sampling(), "Post-processing samplers destroyed\n");
|
||||
}
|
||||
|
||||
namespace
|
||||
{
|
||||
void
|
||||
post_process_perfetto(int64_t _tid, const bundle_t* _init,
|
||||
const std::vector<bundle_t*>& _data)
|
||||
{
|
||||
if(trait::runtime_enabled<backtrace_metrics>::get())
|
||||
{
|
||||
OMNITRACE_VERBOSE(3 || get_debug_sampling(),
|
||||
"[%li] Post-processing metrics for perfetto...\n", _tid);
|
||||
backtrace_metrics::init_perfetto(_tid);
|
||||
for(const auto& itr : _data)
|
||||
{
|
||||
const auto* _bt_metrics = itr->get<backtrace_metrics>();
|
||||
const auto* _bt_time = itr->get<backtrace_timestamp>();
|
||||
if(!_bt_metrics || !_bt_time) continue;
|
||||
if(_bt_time->get_tid() != _tid) continue;
|
||||
_bt_metrics->post_process_perfetto(_tid, _bt_time->get_timestamp());
|
||||
}
|
||||
|
||||
backtrace_metrics::fini_perfetto(_tid);
|
||||
}
|
||||
|
||||
auto _process_perfetto = [_tid,
|
||||
_init](const std::vector<sampling::bundle_t*>& _data) {
|
||||
OMNITRACE_VERBOSE(3 || get_debug_sampling(),
|
||||
"[%li] Post-processing backtraces for perfetto...\n", _tid);
|
||||
|
||||
const auto& _thread_info = thread_info::get(_tid, InternalTID);
|
||||
OMNITRACE_CI_THROW(!_thread_info, "No valid thread info for tid=%li\n", _tid);
|
||||
|
||||
if(!_thread_info) return;
|
||||
|
||||
uint64_t _beg_ns = _thread_info->get_start();
|
||||
uint64_t _end_ns = _thread_info->get_stop();
|
||||
uint64_t _last_ts = std::max<uint64_t>(
|
||||
_init->get<backtrace_timestamp>()->get_timestamp(), _beg_ns);
|
||||
|
||||
tracing::push_perfetto_ts(category::sampling{}, "samples [omnitrace]", _beg_ns,
|
||||
"begin_ns", _beg_ns);
|
||||
|
||||
for(const auto& itr : _data)
|
||||
{
|
||||
const auto* _bt_ts = itr->get<backtrace_timestamp>();
|
||||
const auto* _bt_cs = itr->get<backtrace>();
|
||||
|
||||
if(!_bt_ts || !_bt_cs) continue;
|
||||
if(_bt_ts->get_tid() != _tid) continue;
|
||||
|
||||
static std::set<std::string> _static_strings{};
|
||||
for(const auto& itr : backtrace::filter_and_patch(_bt_cs->get()))
|
||||
{
|
||||
const auto* _name = _static_strings.emplace(itr).first->c_str();
|
||||
uint64_t _beg = _last_ts;
|
||||
uint64_t _end = _bt_ts->get_timestamp();
|
||||
if(!_thread_info->is_valid_lifetime({ _beg, _end })) continue;
|
||||
|
||||
tracing::push_perfetto_ts(category::sampling{}, _name, _beg, "begin_ns",
|
||||
_beg);
|
||||
tracing::pop_perfetto_ts(category::sampling{}, _name, _end, "end_ns",
|
||||
_end);
|
||||
}
|
||||
_last_ts = _bt_ts->get_timestamp();
|
||||
}
|
||||
|
||||
tracing::pop_perfetto_ts(category::sampling{}, "samples [omnitrace]", _end_ns,
|
||||
"end_ns", _end_ns);
|
||||
};
|
||||
|
||||
auto _processing_thread = threading::get_tid();
|
||||
auto _process_perfetto_wrapper = [&]() {
|
||||
if(threading::get_tid() != _processing_thread)
|
||||
threading::set_thread_name(TIMEMORY_JOIN(" ", "Thread", _tid, "(S)").c_str());
|
||||
|
||||
try
|
||||
{
|
||||
_process_perfetto(_data);
|
||||
} catch(std::runtime_error& _e)
|
||||
{
|
||||
OMNITRACE_PRINT("[sampling][post_process_perfetto] Exception: %s\n",
|
||||
_e.what());
|
||||
OMNITRACE_CI_ABORT(true, "[sampling][post_process_perfetto] Exception: %s\n",
|
||||
_e.what());
|
||||
}
|
||||
};
|
||||
|
||||
if(_tid == 0 && config::get_mode() == Mode::Sampling &&
|
||||
config::get_perfetto_fill_policy() == "discard")
|
||||
{
|
||||
_process_perfetto(_data);
|
||||
}
|
||||
else
|
||||
{
|
||||
pthread_gotcha::push_enable_sampling_on_child_threads(false);
|
||||
std::thread{ _process_perfetto_wrapper }.join();
|
||||
pthread_gotcha::pop_enable_sampling_on_child_threads();
|
||||
}
|
||||
}
|
||||
|
||||
void
|
||||
post_process_timemory(int64_t _tid, const bundle_t* _init,
|
||||
const std::vector<bundle_t*>& _data)
|
||||
{
|
||||
std::map<int64_t, std::map<int64_t, int64_t>> _depth_sum = {};
|
||||
auto _scope = tim::scope::config{};
|
||||
if(get_timeline_sampling()) _scope += scope::timeline{};
|
||||
if(get_flat_sampling()) _scope += scope::flat{};
|
||||
|
||||
OMNITRACE_VERBOSE(3 || get_debug_sampling(),
|
||||
"[%li] Post-processing data for timemory...\n", _tid);
|
||||
|
||||
const auto* _last = _init;
|
||||
for(const auto& itr : _data)
|
||||
{
|
||||
using bundle_t = tim::lightweight_tuple<comp::trip_count, sampling_wall_clock,
|
||||
sampling_cpu_clock, hw_counters>;
|
||||
|
||||
auto* _bt_data = itr->get<backtrace>();
|
||||
auto* _bt_time = itr->get<backtrace_timestamp>();
|
||||
auto* _bt_metrics = itr->get<backtrace_metrics>();
|
||||
|
||||
if(!_bt_data || !_bt_time || !_bt_metrics) continue;
|
||||
|
||||
double _elapsed_wc = (_bt_time->get_timestamp() -
|
||||
_last->get<backtrace_timestamp>()->get_timestamp());
|
||||
double _elapsed_cc = (_bt_metrics->get_cpu_timestamp() -
|
||||
_last->get<backtrace_metrics>()->get_cpu_timestamp());
|
||||
|
||||
std::vector<bundle_t> _tc{};
|
||||
_tc.reserve(_bt_data->size());
|
||||
|
||||
// generate the instances of the tuple of components and start them
|
||||
for(const auto& itr : backtrace::filter_and_patch(_bt_data->get()))
|
||||
{
|
||||
_tc.emplace_back(tim::string_view_t{ itr }, _scope);
|
||||
_tc.back().push(_bt_time->get_tid());
|
||||
_tc.back().start();
|
||||
}
|
||||
|
||||
// stop the instances and update the values as needed
|
||||
for(size_t i = 0; i < _tc.size(); ++i)
|
||||
{
|
||||
auto& itr = _tc.at(_tc.size() - i - 1);
|
||||
size_t _depth = 0;
|
||||
_depth_sum[_bt_time->get_tid()][_depth] += 1;
|
||||
itr.stop();
|
||||
if constexpr(tim::trait::is_available<sampling_wall_clock>::value)
|
||||
{
|
||||
auto* _sc = itr.get<sampling_wall_clock>();
|
||||
if(_sc)
|
||||
{
|
||||
auto _value = _elapsed_wc / sampling_wall_clock::get_unit();
|
||||
_sc->set_value(_value);
|
||||
_sc->set_accum(_value);
|
||||
}
|
||||
}
|
||||
if constexpr(tim::trait::is_available<sampling_cpu_clock>::value)
|
||||
{
|
||||
auto* _cc = itr.get<sampling_cpu_clock>();
|
||||
if(_cc)
|
||||
{
|
||||
_cc->set_value(_elapsed_cc / sampling_cpu_clock::get_unit());
|
||||
_cc->set_accum(_elapsed_cc / sampling_cpu_clock::get_unit());
|
||||
}
|
||||
}
|
||||
if constexpr(tim::trait::is_available<hw_counters>::value)
|
||||
{
|
||||
auto _hw_cnt_vals = _bt_metrics->get_hw_counters();
|
||||
if(_last && _bt_metrics->get_hw_counters().size() ==
|
||||
_last->get<backtrace_metrics>()->get_hw_counters().size())
|
||||
{
|
||||
for(size_t k = 0; k < _bt_metrics->get_hw_counters().size(); ++k)
|
||||
{
|
||||
if(_last->get<backtrace_metrics>()->get_hw_counters()[k] >
|
||||
_hw_cnt_vals[k])
|
||||
_hw_cnt_vals[k] -=
|
||||
_last->get<backtrace_metrics>()->get_hw_counters()[k];
|
||||
}
|
||||
}
|
||||
auto* _hw_counter = itr.get<hw_counters>();
|
||||
if(_hw_counter)
|
||||
{
|
||||
_hw_counter->set_value(_hw_cnt_vals);
|
||||
_hw_counter->set_accum(_hw_cnt_vals);
|
||||
}
|
||||
}
|
||||
itr.pop();
|
||||
}
|
||||
_last = itr;
|
||||
}
|
||||
|
||||
for(auto&& itr : _data)
|
||||
{
|
||||
using bundle_t =
|
||||
tim::lightweight_tuple<sampling_percent, quirk::config<quirk::tree_scope>>;
|
||||
|
||||
auto* _bt_data = itr->get<backtrace>();
|
||||
auto* _bt_time = itr->get<backtrace_timestamp>();
|
||||
|
||||
if(!_bt_time || !_bt_data) continue;
|
||||
if(_depth_sum.find(_bt_time->get_tid()) == _depth_sum.end()) continue;
|
||||
|
||||
std::vector<bundle_t> _tc{};
|
||||
_tc.reserve(_bt_data->size());
|
||||
|
||||
// generate the instances of the tuple of components and start them
|
||||
for(const auto& itr : backtrace::filter_and_patch(_bt_data->get()))
|
||||
{
|
||||
_tc.emplace_back(tim::string_view_t{ itr });
|
||||
_tc.back().push(_bt_time->get_tid());
|
||||
_tc.back().start();
|
||||
}
|
||||
|
||||
// stop the instances and update the values as needed
|
||||
for(size_t i = 0; i < _tc.size(); ++i)
|
||||
{
|
||||
auto& itr = _tc.at(_tc.size() - i - 1);
|
||||
size_t _depth = 0;
|
||||
double _value = (1.0 / _depth_sum[_bt_time->get_tid()][_depth]) * 100.0;
|
||||
itr.store(std::plus<double>{}, _value);
|
||||
itr.stop();
|
||||
itr.pop();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
struct sampling_initialization
|
||||
{
|
||||
static void preinit()
|
||||
{
|
||||
sampling_wall_clock::label() = "sampling_wall_clock";
|
||||
sampling_wall_clock::description() = "Wall clock time (via sampling)";
|
||||
|
||||
sampling_cpu_clock::label() = "sampling_cpu_clock";
|
||||
sampling_cpu_clock::description() = "CPU clock time (via sampling)";
|
||||
|
||||
sampling_percent::label() = "sampling_percent";
|
||||
sampling_percent::description() = "Percentage of samples";
|
||||
|
||||
sampling_gpu_busy::label() = "sampling_gpu_busy_percent";
|
||||
sampling_gpu_busy::description() = "Utilization of GPU(s)";
|
||||
sampling_gpu_busy::set_precision(0);
|
||||
sampling_gpu_busy::set_format_flags(sampling_gpu_busy::get_format_flags() &
|
||||
std::ios_base::showpoint);
|
||||
|
||||
sampling_gpu_memory::label() = "sampling_gpu_memory_usage";
|
||||
sampling_gpu_memory::description() = "Memory usage of GPU(s)";
|
||||
|
||||
sampling_gpu_power::label() = "sampling_gpu_power";
|
||||
sampling_gpu_power::description() = "Power usage of GPU(s)";
|
||||
sampling_gpu_power::unit() = units::watt;
|
||||
sampling_gpu_power::display_unit() = "watts";
|
||||
sampling_gpu_power::set_precision(2);
|
||||
sampling_gpu_power::set_format_flags(sampling_gpu_power::get_format_flags());
|
||||
|
||||
sampling_gpu_temp::label() = "sampling_gpu_temperature";
|
||||
sampling_gpu_temp::description() = "Temperature of GPU(s)";
|
||||
sampling_gpu_temp::unit() = 1;
|
||||
sampling_gpu_temp::display_unit() = "degC";
|
||||
sampling_gpu_temp::set_precision(1);
|
||||
sampling_gpu_temp::set_format_flags(sampling_gpu_temp::get_format_flags());
|
||||
}
|
||||
};
|
||||
} // namespace
|
||||
} // namespace sampling
|
||||
} // namespace omnitrace
|
||||
|
||||
TIMEMORY_INVOKE_PREINIT(omnitrace::sampling::sampling_initialization)
|
||||
|
||||
Reference in New Issue
Block a user