Linux Perf Support + Causal Profiling Updates (#276)

* causal backtrace updates - fix initial causal sampling period value * causal delay updates - tweak handling of sleep_for_overhead * Fix experiment global scaling for prog pts - results in drastically improved predictions * pthread_mutex_gotcha updates - disable all wrappers during causal profiling * validate-causal-json.py updates - support decimal stddev - fix setting stddev from command-line * causal perform_experiment_impl update - handle start failing because finalizing * deprecate causal::component::sample_rate - appears to not help at all * Rework sample info * Increase causal unwind_depth - use OMNITRACE_MAX_UNWIND_DEPTH * validate-causal-json updates - min experiments - exclude reporting predictions with less than X experiments at a given speedup - percent samples - only print samples within X% of the peak (default: 95%) * Update timemory submodule - extensions to sampling for signals delivered via non-timer method - e.g. via HW counter overflow * dwarf_entry::operator< updates - sort via file * causal profiling docs updates - info about backends - info about installing/enabling perf * config updates: causal backend - CausalBackend enum - OMNITRACE_CAUSAL_BACKEND: perf, timer, auto - omnitrace-causal option: --backend * debug update - use spin_mutex instead of std::mutex * address_range::contains update - range from 0-100 contains range from 10-100 but was returning false because high was == 100 not < 100 * symbol::operator< update - handle load address differences * sampling updates (non-causal) - update get_timer to get_trigger + dynamic_cast * container::static_vector updates - support construction from container::c_array - update_size private member func for handling atomic m_size * Move perf files - moved library/causal/perf.{hpp,cpp} to library/perf.{hpp,cpp} * causal example update - created impl.hpp (forward decls) - renamed {cpu,rng}_func_impl to {cpu,rng}_impl_func - only create two threads which run N iterations instead of two threads each iteration * Update timemory submodule - updates to unwind::processed_entry - updates to procfs::maps * Updated causal documentation - fixed line numbers changed by modifications to causal example * omnitrace-causal exe updates - set OMNITRACE_THREAD_POOL_SIZE to zero by default * core/containers updates - static_vector: provide data() member function - c_array pop_front() and pop_back() member functions * core: config and argparse updates + perf - core/perf.{hpp,cpp} - forward decl of enums - config-related capabilities - argparse: --sample-overflow - renamed some config functions - e.g. get_sampling_cpu_freq -> get_sampling_cputime_freq - added config settings related to overflow sampling via perf - added timer_sampling and overflow_sampling categories * Update timemory submodule - sampling allocator flushing * binary updates - lookup_ipaddr_entry - use bfd_find_nearest_line instead of bfd_find_nearest_line_discriminator - discriminators are not used - explicit instantiations of inlined_symbol::serialize * Bump VERSION to 1.10.0 * sampling and perf updates - support overflow sampling via Linux Perf - update perf namespace - update perf::perf_event - update record ctor: pointer instead of const ref - update open member func: return optional string - add m_batch_size member variable - sampling updates - support overflow sampling - flush allocators - increase buffer size from 1024 to 2048 - restructure post-processing in light of perf overflow supports - improve offload memory usage only load buffers for thread - load_offload_buffer(tid) uses thread-specific filepos - component updates - backtrace_metrics::operator-= - backtrace_metrics::operator- - backtrace::sample does not record for overflow signal - callchain: perf overflow sample * core updates - component::sampling_percent does not report self + uses_percent_units * causal updates - tweak get_line_info - overloads for set_current_selection (uint64_t, c_array, std::array) - delay - use sampling::pause/sampling::resume - experiment - experiment::sample derives from unwind::processed_entry - experiment::samples is vector instead of set - fixed samples - overloads for is_selected (uint64_t, c_array, std::array) - scaling factor defaults to 100 instead of 50 - serialize updates follow change to experiment::sample - modify algorithm for increasing/decreasing experiment length - sample_data - use map<uintptr, uint64_t> instead of set<sample_data> - get_samples returns vector<sample_data> instead of set<sample_data> - sampling - support overflow via Linux Perf - update causal_offload_buffer - flush sampling allocator - backtrace - overflow component * libomnitrace-dl updates - handle dl::InstrumentMode::PythonProfile * testing updates (causal) - causal line 155 -> causal line 100 - causal line 165 -> causal line 110 * formatting * exit_gotcha updates - exit_info for abort() - message about non-zero exit code * testing updates - fail regex for causal tests - validate-causal-json: >= min_experiments instead of > min_experiments - handle OMNITRACE_DEBUG_SETTINGS in omnitrace_write_test_config * causal sampling updates - add new lines where appropriate * causal data updates - reorder diagnostic info when experiment fails to start * binary updates - symbol address range from address to address + symsize + 1 - add 1 based on debug info * causal data updates - sample_selection wait_ns defaults to 1,000 instead of 10,000 - sample_selection wait scaled by iteration number - save_line_info_impl verbosity - print latest_eligible_pc when experiment does not start * causal sampling + component updates - perf backend disables component::backtrace - ensure get_sampling_(realtime|cputime|overflow)_signal do not malloc * causal: remove period stats * validate-causal-json update - fix --help * causal data updates - improve eligible pc history reporting when experiment fails to start * causal data updates - fix compute_eligible_lines_impl - eligible address ranges returning too many ranges - occasionally, overwrite all *true* eligible address ranges * causal data updates - reduce scoped ranges to symbol ranges - is_eligible_address() returns true contains (not just coarse) - revert some sample_selection behavior * binary address_multirange updates - make coarse_range private - fix operator+=(pair<coarse, uintptr_t>) * causal example update - fix nsync to default to once per iteration * binary analysis updates - tweak header file includes * causal updates - remove factoring in sleep_for_overhead - invoke delay::process() even if experiment is not active * causal data updates - update latest_eligible_pc structure * update omnitrace-install.py.in - fix support for fedora - /etc/os-release does not have ID_LIKE - fallback to RHEL 8.7 if version not specified * update omnitrace-install.py.in - fix support for debian - /etc/os-release does not have ID_LIKE - version mapping * Update documentation - update docs on installation * causal data and experiment updates - data: reset_sample_selection * causal set_current_selection debugging - debug messages for failed e2e runs * causal data and backtrace component updates - data: set_current_selection returns the number of eligible addresses added - backtrace: if cputime signal has selected zero IPs > 5x, then realtime signal starts contributing call-stacks * core library updates - move config::parse_numeric_range to utility namespace - add core/utility.cpp - support range:increment, e.g. 5-25:10 expands to '5 15 25' instead of '5 10 15 20 25' * omnitrace-causal update - end-to-end expands all speedups - support range:increment in speedups * causal backtrace updates - remove select_ival (realtime signal always contributes when select_count == 0) * containers: static_vector update - explicit c_array constructor - explicit std::array constructor * causal data updates - remove set_current_selection(uint64_t) - remove set_current_selection(std::array) - sample_selection increase default wait time - report eligible PC candidates - move reset_sample_selection to perform_experiment_impl - decrease latest_eligible_pc array size - set_current_selection does not guard for experiment::active * core debug updates - OMNITRACE_PRINT_COLOR macros * causal data updates - tweak to experiment never started message * causal gotcha updates - remove unused code * critical trace updates - remove unused code * omnitrace-causal - OMNITRACE_LAUNCHER * causal data updates - don't fail on end-to-end + omnitrace-causal * causal backtrace updates - reintroduce select_ival behavior * causal data updates - tweak verbose messages about number of PC candidates * core mproc updates - utilities for waiting on child PID and diagnosing status - omnitrace::mproc::wait_pid - omnitrace::mproc::diagnose_status * omnitrace-run updates - support --fork argument for executing via fork in current process + execvpe on child instead of execvpe in current process * omnitrace-causal updates - wait_pid and diagnose_status just call equivalent functions in omnitrace::mproc * ubuntu-focal workflow update - attempt to launch ubuntu-focal-codecov job with CAP_SYS_ADMIN and use perf backend * tests reorg and updates - remove binary-rewrite-sampling and runtime-instrument-sampling tests - rename *-preload tests (which use omnitrace-sample exe) to *-sampling - split tests/CMakeLists.txt into several tests/omnitrace-<category>-tests.cmake files - tweak to causal-both-omni-func test - add args: -n 2 -b timer * update validate-causal-json.py - better reasoning info for adjusting tolerance - always apply tolerance adjustments in CI mode * causal e2e tests update - add label "causal-e2e" label - tweak params - old: 80 12 432525 500000000 - new: 80 50 432525 100000000 - disable processor affinity for slow-func/line-100 tests - artificially inflates some speedups with perf * unblocking_gotcha updates - overload operator() according to gotcha function index * blocking_gotcha updates - overload operator() according to gotcha function index - fix bug where potentially post block functors (e.g. pthread_mutex_trylock) throw error if lock is not acquired. * parse_numeric_range update - support unordered_set * config update - OMNITRACE_DEBUG_{TIDS,PIDS} use parse_numeric_range
2023-04-13 02:14:35 -05:00
Commit 9de3a6b0b4
@@ -21,6 +21,7 @@
 // SOFTWARE.

 #include "library/causal/sampling.hpp"
+#include "binary/analysis.hpp"
 #include "core/common.hpp"
 #include "core/concepts.hpp"
 #include "core/config.hpp"
@@ -30,6 +31,8 @@
 #include "core/utility.hpp"
 #include "library/causal/components/backtrace.hpp"
 #include "library/causal/data.hpp"
+#include "library/causal/sample_data.hpp"
+#include "library/perf.hpp"
 #include "library/ptl.hpp"
 #include "library/runtime.hpp"
 #include "library/sampling.hpp"
@@ -37,8 +40,11 @@
 #include "library/thread_info.hpp"

 #include <timemory/macros.hpp>
+#include <timemory/mpl/types.hpp>
 #include <timemory/sampling/allocator.hpp>
+#include <timemory/sampling/overflow.hpp>
 #include <timemory/sampling/sampler.hpp>
+#include <timemory/sampling/timer.hpp>
 #include <timemory/units.hpp>
 #include <timemory/utility/backtrace.hpp>
 #include <timemory/variadic.hpp>
@@ -46,6 +52,7 @@
 #include <csignal>
 #include <cstring>
 #include <ctime>
+#include <memory>
 #include <mutex>
 #include <sstream>
 #include <string>
@@ -58,11 +65,14 @@ namespace causal
 namespace sampling
 {
 using ::tim::sampling::dynamic;
+using ::tim::sampling::overflow;
 using ::tim::sampling::timer;

 using causal_bundle_t =
-    tim::lightweight_tuple<causal::component::sample_rate, causal::component::backtrace>;
-using causal_sampler_t = tim::sampling::sampler<causal_bundle_t, dynamic>;
+    tim::lightweight_tuple<causal::component::overflow, causal::component::backtrace>;
+using causal_sampler_t  = tim::sampling::sampler<causal_bundle_t, dynamic>;
+using backtrace_enabled = trait::runtime_enabled<component::backtrace>;
+using overflow_enabled  = trait::runtime_enabled<component::overflow>;
 }  // namespace sampling
 }  // namespace causal
 }  // namespace omnitrace
@@ -156,17 +166,34 @@ void
 causal_offload_buffer(int64_t, causal_sampler_buffer_t&& _buf)
 {
    auto _data      = std::move(_buf);
-    auto _processed = std::map<uint32_t, std::vector<uintptr_t>>{};
+    auto _processed = std::map<uint32_t, std::map<uintptr_t, uint64_t>>{};
    while(!_data.is_empty())
    {
        auto _bundle = causal_sampler_bundle_t{};
        _data.read(&_bundle);
-        auto* _bt_causal = _bundle.get<causal::component::backtrace>();
+
+        const auto* _bt_causal = _bundle.get<causal::component::backtrace>();
        if(_bt_causal)
        {
-            for(auto&& itr : _bt_causal->get_stack())
+            auto _stack = _bt_causal->get_stack();
+
+            for(auto itr : _stack)
            {
-                if(itr > 0) _processed[_bt_causal->get_index()].emplace_back(itr);
+                if(itr > 0) _processed[_bt_causal->get_index()][itr] += 1;
+            }
+        }
+
+        const auto* _of_causal = _bundle.get<causal::component::overflow>();
+        if(_of_causal)
+        {
+            const auto& _stack = _of_causal->get_stack();
+
+            for(const auto& ditr : _stack)
+            {
+                for(auto aitr : ditr)
+                {
+                    if(aitr > 0) _processed[_of_causal->get_index()][aitr] += 1;
+                }
            }
        }
    }
@@ -177,7 +204,9 @@ causal_offload_buffer(int64_t, causal_sampler_buffer_t&& _buf)
        static auto _mutex = locking::atomic_mutex{};
        auto        _lk    = locking::atomic_lock{ _mutex };
        for(const auto& itr : _processed)
+        {
            add_samples(itr.first, itr.second);
+        }
    }
 }

@@ -186,6 +215,7 @@ configure(bool _setup, int64_t _tid)
 {
    const auto& _info         = thread_info::get(_tid, SequentTID);
    auto&       _causal       = get_causal_sampler(_tid);
+    auto&       _causal_perf  = perf::get_instance(_tid);
    auto&       _running      = get_causal_sampler_running(_tid);
    auto&       _signal_types = get_causal_sampler_signals(_tid);

@@ -197,6 +227,19 @@ configure(bool _setup, int64_t _tid)

    if(_setup && _signal_types.empty()) _signal_types = get_sampling_signals(_tid);

+    // initialize
+    if(_setup)
+    {
+        using global_init_mode = operation::mode_constant<operation::init_mode::global>;
+        using thread_init_mode = operation::mode_constant<operation::init_mode::thread>;
+        // initialize backtrace
+        operation::init<component::backtrace>{}(global_init_mode{});
+        operation::init<component::backtrace>{}(thread_init_mode{});
+        // initialize overflow
+        operation::init<component::overflow>{}(global_init_mode{});
+        operation::init<component::overflow>{}(thread_init_mode{});
+    }
+
    if(_setup && !_causal && !_running && !_signal_types.empty())
    {
        auto _verbose = std::min<int>(get_verbose() - 2, 2);
@@ -218,21 +261,95 @@ configure(bool _setup, int64_t _tid)
        _causal = std::make_unique<causal_sampler_t>(_causal_alloc, "omnitrace", _tid,
                                                     _verbose);

+        auto _activate_perf_backend = [&_causal, &_causal_perf, &_info, &_tid]() {
+            _causal_perf = std::make_unique<perf::perf_event>();
+            auto _open_error =
+                _causal_perf->open(1000.0, 10, _info->index_data->system_value);
+            if(_open_error)
+            {
+                _causal_perf.reset();
+            }
+            else
+            {
+                overflow_enabled::set(true);
+                overflow_enabled::set(scope::thread_scope{}, true);
+                backtrace_enabled::set(false);
+                backtrace_enabled::set(scope::thread_scope{}, false);
+                _causal->configure(overflow{ get_sampling_overflow_signal(),
+                                             [](int, pid_t, long, int64_t) {
+                                                 // perf::get_instance(_idx)->set_ready_signal(_sig);
+                                                 return true;
+                                             },
+                                             [](int, pid_t, long, int64_t _idx) {
+                                                 return perf::get_instance(_idx)->start();
+                                             },
+                                             [](int, pid_t, long, int64_t _idx) {
+                                                 return perf::get_instance(_idx)->stop();
+                                             },
+                                             _tid, threading::get_sys_tid() });
+                if(_tid == 0) OMNITRACE_VERBOSE(1, "causal profiling backend: perf\n");
+            }
+
+            return _open_error;
+        };
+
+        auto _activate_timer_backend = [&_causal, &_tid]() {
+            backtrace_enabled::set(true);
+            backtrace_enabled::set(scope::thread_scope{}, true);
+            overflow_enabled::set(false);
+            overflow_enabled::set(scope::thread_scope{}, false);
+            _causal->configure(timer{ get_sampling_realtime_signal(), CLOCK_REALTIME,
+                                      SIGEV_THREAD_ID, 1000.0, 1.0e-6, _tid,
+                                      threading::get_sys_tid() });
+            if(_tid == 0) OMNITRACE_VERBOSE(1, "causal profiling backend: timer\n");
+            return true;
+        };
+
        TIMEMORY_REQUIRE(_causal) << "nullptr to causal profiling instance";

        _causal->set_flags(SA_RESTART);
        _causal->set_verbose(_verbose);
        _causal->set_offload(&causal_offload_buffer);

-        _causal->configure(timer{ get_realtime_signal(), CLOCK_REALTIME, SIGEV_THREAD_ID,
-                                  1000.0, 1.0e-6, _tid, threading::get_sys_tid() });
+        if(get_causal_backend() == CausalBackend::Perf)
+        {
+            auto _perf_error = _activate_perf_backend();
+            OMNITRACE_REQUIRE(!_perf_error)
+                << "perf backend for causal profiling failed to activate: "
+                << *_perf_error << "\n";
+        }
+        else if(get_causal_backend() == CausalBackend::Timer)
+        {
+            OMNITRACE_REQUIRE(_activate_timer_backend())
+                << "timer backend for causal profiling failed to activate\n";
+        }
+        else if(get_causal_backend() == CausalBackend::Auto)
+        {
+            auto _perf_error = _activate_perf_backend();
+            if(!_perf_error)
+            {
+                config::set_setting_value("OMNITRACE_CAUSAL_BACKEND",
+                                          std::string{ "perf" });
+            }
+            else
+            {
+                OMNITRACE_WARNING_F(
+                    0, "perf backend for causal profiling failed to activate: %s\n",
+                    _perf_error->c_str());

-        _causal->configure(timer{ get_cputime_signal(), CLOCK_THREAD_CPUTIME_ID,
+                OMNITRACE_REQUIRE(_activate_timer_backend())
+                    << "timer backend for causal profiling failed to activate\n";
+
+                config::set_setting_value("OMNITRACE_CAUSAL_BACKEND",
+                                          std::string{ "timer" });
+            }
+        }
+
+        _causal->configure(timer{ get_sampling_cputime_signal(), CLOCK_THREAD_CPUTIME_ID,
                                  SIGEV_THREAD_ID, 1000.0, 1.0e-6, _tid,
                                  threading::get_sys_tid() });

        _running = true;
-        if(_tid == 0) causal::component::backtrace::start();
        _causal->start();
    }
    else if(!_setup && _causal && _running)
@@ -257,12 +374,22 @@ configure(bool _setup, int64_t _tid)
                    get_causal_sampler(i)->stop();
                    get_causal_sampler(i)->reset();
                }
+
+                if(perf::get_instance(i))
+                {
+                    perf::get_instance(i).reset();
+                }
            }
        }

        _causal->stop();
        _causal->reset();

+        if(_causal_perf)
+        {
+            _causal_perf.reset();
+        }
+
        OMNITRACE_DEBUG("Causal sampler destroyed for thread %lu\n", _tid);
    }

@@ -311,17 +438,113 @@ unblock_samples()
 void
 block_backtrace_samples()
 {
-    trait::runtime_enabled<causal::component::backtrace>::set(scope::thread_scope{},
-                                                              false);
+    pause(scope::thread_scope{});
 }

 void
 unblock_backtrace_samples()
 {
-    trait::runtime_enabled<causal::component::backtrace>::set(scope::thread_scope{},
-                                                              true);
+    resume(scope::thread_scope{});
 }

+namespace
+{
+std::optional<bool>              _process_paused = {};
+thread_local std::optional<bool> _thread_paused  = {};
+namespace signals                                = ::tim::signals;
+
+const auto&
+sampling_signals()
+{
+    static thread_local auto _v = get_signal_types(threading::get_id());
+    return _v;
+}
+}  // namespace
+
+template <typename ScopeT>
+void pause(ScopeT)
+{
+    static_assert(
+        tim::is_one_of<ScopeT,
+                       type_list<scope::thread_scope, scope::process_scope>>::value,
+        "Unsupported scope");
+
+    if constexpr(std::is_same<ScopeT, scope::thread_scope>::value)
+    {
+        if(!_thread_paused) _thread_paused = false;
+
+        bool _paused_v = *_thread_paused;
+        if(!_paused_v)
+        {
+            auto& _causal_perf = perf::get_instance(threading::get_id());
+            if(_causal_perf) _causal_perf->stop();
+            signals::block_signals(sampling_signals(), signals::sigmask_scope::thread);
+            _thread_paused = true;
+        }
+    }
+    else
+    {
+        if(!_process_paused) _process_paused = false;
+
+        bool _paused_v = *_process_paused;
+        if(!_paused_v)
+        {
+            for(auto i = 0; i < OMNITRACE_MAX_THREADS; ++i)
+            {
+                auto& _causal_perf = perf::get_instance(i);
+                if(_causal_perf) _causal_perf->stop();
+            }
+            signals::block_signals(sampling_signals(), signals::sigmask_scope::process);
+            _process_paused = true;
+        }
+    }
+}
+
+template <typename ScopeT>
+void resume(ScopeT)
+{
+    static_assert(
+        tim::is_one_of<ScopeT,
+                       type_list<scope::thread_scope, scope::process_scope>>::value,
+        "Unsupported scope");
+
+    if constexpr(std::is_same<ScopeT, scope::thread_scope>::value)
+    {
+        if(!_thread_paused) _thread_paused = true;
+
+        bool _paused_v = *_thread_paused;
+        if(_paused_v)
+        {
+            auto& _causal_perf = perf::get_instance(threading::get_id());
+            if(_causal_perf) _causal_perf->start();
+            signals::unblock_signals(sampling_signals(), signals::sigmask_scope::thread);
+            _thread_paused = false;
+        }
+    }
+    else
+    {
+        if(!_process_paused) _process_paused = true;
+
+        bool _paused_v = *_process_paused;
+        if(_paused_v)
+        {
+            for(auto i = 0; i < OMNITRACE_MAX_THREADS; ++i)
+            {
+                auto& _causal_perf = perf::get_instance(i);
+                if(_causal_perf) _causal_perf->start();
+            }
+            signals::unblock_signals(sampling_signals(), signals::sigmask_scope::process);
+            _process_paused = false;
+        }
+    }
+}
+
+template void pause<scope::thread_scope>(scope::thread_scope);
+template void pause<scope::process_scope>(scope::process_scope);
+
+template void resume<scope::thread_scope>(scope::thread_scope);
+template void resume<scope::process_scope>(scope::process_scope);
+
 void
 block_signals(std::set<int> _signals)
 {
@@ -354,10 +577,15 @@ post_process()
    {
        auto& _causal = get_causal_sampler(i);
        if(_causal) _causal->stop();
+        auto& _causal_perf = perf::get_instance(i);
+        if(_causal_perf) _causal_perf->stop();
    }

    configure(false, 0);

+    auto _allocator = get_causal_sampler_allocator(false);
+    if(_allocator) _allocator->flush();
+
    for(size_t i = 0; i < max_supported_threads; ++i)
    {
        auto& _causal = get_causal_sampler(i);
@@ -370,12 +598,15 @@ post_process()
    for(size_t i = 0; i < max_supported_threads; ++i)
    {
        get_causal_sampler(i).reset();
+
+        auto& _causal_perf = perf::get_instance(i);
+        if(_causal_perf)
+        {
+            _causal_perf.reset();
+        }
    }

-    if(get_causal_sampler_allocator(false))
-    {
-        get_causal_sampler_allocator(false).reset();
-    }
+    if(_allocator) _allocator.reset();
 }

 namespace
@@ -386,9 +617,27 @@ post_process_causal(int64_t, const std::vector<causal_bundle_t>& _data)
    for(const auto& itr : _data)
    {
        const auto* _bt_causal = itr.get<causal::component::backtrace>();
-        for(auto&& ditr : _bt_causal->get_stack())
+        if(_bt_causal)
        {
-            if(ditr > 0) add_sample(_bt_causal->get_index(), ditr);
+            auto _stack = _bt_causal->get_stack();
+            for(auto&& ditr : _stack)
+            {
+                if(ditr > 0) add_sample(_bt_causal->get_index(), ditr);
+            }
+        }
+
+        const auto* _of_causal = itr.get<causal::component::overflow>();
+        if(_of_causal)
+        {
+            const auto& _stack = _of_causal->get_stack();
+
+            for(const auto& ditr : _stack)
+            {
+                for(auto aitr : ditr)
+                {
+                    if(aitr > 0) add_sample(_of_causal->get_index(), aitr);
+                }
+            }
        }
    }
 }