AI/rocm-systems
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

omnitrace-run executable - required for running binary writes (#257)

Browse Source 
* omnitrace-run exe

- ensure LD_PRELOAD for libomnitrace-dl.so
- convert config options into command-line options

* Update timemory submodule

- updates to tsettings
- updates to argparser

* common environment update

- throw error if get_env<bool> has empty string

* config updates

- minor tweaks to categories of settings

* core lib update

- add argparse for common handling of argument parsers

* omnitrace-sample update

- fix handling of --trace-file (OMNITRACE_PERFETTO_FILE)

* omnitrace-run update

- updated to use omnitrace::argparse functions

* Tests for omnitrace-run

* argparse core update

- remove choices for --cpu-events and --gpu-events

* remove some debugging prints

* fix timemory include in argparse.cpp

* always provide --hsa-interrupt option

* Update source/lib/core/argparse.cpp

- fix pedantic warning

* Update testing

- remove testing args that may not be there in some builds

* roctracer/pthread_create fix

- disable roctracer_data when roctracer not enabled

* omnitrace-causal tweak

* omnitrace-instrument: module_function tweak

- allow DEFAULT_MODULE and LIBRARY_MODULE

* common environment update

- support get_env for enums

* core: config update

- Add "mode" category to OMNITRACE_MODE

* Update timemory submodule

- remove debug print statement

* omnitrace-sample tweak

- change var init

* omnitrace-run testing update

- use --help instead of -?

* core: common.hpp

- tweak header include style

* core: argparser update

- add_ld_preload func
- launcher and command member variables in parser_data
- support launcher

* omnitrace-run update

- clean up and reworked

* libomnitrace-dl updates

- require LD_PRELOAD with binary rewrite
- dl::InstrumentMode
- dl::get_instrumented()
- verify_instrumented_preloaded()
- omnitrace_set_instrumented(int)
- relocated omnitrace_main from main.c to dl.cpp
- omnitrace_set_env does not dlopen libomnitrace
- omnitrace_set_main(func_ptr) [internal API]
- OMNITRACE_HIDDEN_API -> OMNITRACE_INTERNAL_API

* Update testing to new LD_PRELOAD requirements

* omnitrace-instrument updates

- adhere to LD_PRELOAD requirementsa
- invoke omnitrace_set_instrumented
- binary rewrite does not instrument main
- binary rewrite does not instrument call to omnitrace_init
- runtime instr does not instrument main
- runtime instr does not instrument call to omnitrace_init

* Bump to v1.9.0

- LD_PRELOAD requirement necessitates minor version increment

* common: environment

- fix ambiguous get_env calls

* omnitrace-instrument update

- fix issue with temporaries

* omnitrace-instrument and libomnitrace-dl updates

- runtime instrumentation does not work if libomnitrace-dl is preloaded

* libomnitrace-dl and libpyomnitrace updates

- define dl::InstrumentMode in dl.hpp
- handle instrumentation via setprofile libpyomnitrace
  - do not push trace in omnitrace_init

* omnitrace-instrument and libomnitrace-dl updates

- move header to dl subdirectory
- omnitrace::omnitrace-headers include omnitrace-dl folder
- use InstrumentMode in omnitrace-instrument

* Update workflows and scripts

- Use omnitrace-run on instrumented exes

* Update docs

- add omnitrace-run to examples of running binary rewritten exes
This commit is contained in:
Jonathan R. Madsen
2023-03-14 19:48:29 -05:00
committed by GitHub
parent ab0e5d9b44
commit abe35de43a
39 changed files with 2872 additions and 236 deletions
Download Patch File Download Diff File Expand all files Collapse all files
source/lib/common/environment.hpp
+29 -5
View File
@@ -26,8 +26,10 @@
#include <cstdlib>
#include <cstring>
#include <iostream>
#include <stdexcept>
#include <string>
#include <string_view>
#include <type_traits>
namespace omnitrace
{
@@ -36,7 +38,7 @@ inline namespace common
namespace
{
inline std::string
get_env(std::string_view env_id, std::string_view _default)
get_env_impl(std::string_view env_id, std::string_view _default)
{
if(env_id.empty()) return std::string{ _default };
char* env_var = ::std::getenv(env_id.data());
@@ -45,13 +47,13 @@ get_env(std::string_view env_id, std::string_view _default)
}
inline std::string
get_env(std::string_view env_id, const char* _default)
get_env_impl(std::string_view env_id, const char* _default)
{
return get_env(env_id, std::string_view{ _default });
return get_env_impl(env_id, std::string_view{ _default });
}
inline int
get_env(std::string_view env_id, int _default)
get_env_impl(std::string_view env_id, int _default)
{
if(env_id.empty()) return _default;
char* env_var = ::std::getenv(env_id.data());
@@ -73,15 +75,21 @@ get_env(std::string_view env_id, int _default)
}
inline bool
get_env(std::string_view env_id, bool _default)
get_env_impl(std::string_view env_id, bool _default)
{
if(env_id.empty()) return _default;
char* env_var = ::std::getenv(env_id.data());
if(env_var)
{
if(std::string_view{ env_var }.empty())
throw std::runtime_error(std::string{ "No boolean value provided for " } +
std::string{ env_id });
if(std::string_view{ env_var }.find_first_not_of("0123456789") ==
std::string_view::npos)
{
return static_cast<bool>(std::stoi(env_var));
}
else
{
for(size_t i = 0; i < strlen(env_var); ++i)
@@ -93,6 +101,22 @@ get_env(std::string_view env_id, bool _default)
}
return _default;
}
template <typename Tp>
inline auto
get_env(std::string_view env_id, Tp&& _default)
{
if constexpr(std::is_enum<Tp>::value)
{
using Up = std::underlying_type_t<Tp>;
// cast to underlying type -> get_env -> cast to enum type
return static_cast<Tp>(get_env_impl(env_id, static_cast<Up>(_default)));
}
else
{
return get_env_impl(env_id, std::forward<Tp>(_default));
}
}
} // namespace
} // namespace common
} // namespace omnitrace
source/lib/core/CMakeLists.txt
+2
View File
@@ -3,6 +3,7 @@ configure_file(${CMAKE_CURRENT_SOURCE_DIR}/defines.hpp.in
${CMAKE_CURRENT_BINARY_DIR}/defines.hpp @ONLY)
set(core_sources
${CMAKE_CURRENT_LIST_DIR}/argparse.cpp
${CMAKE_CURRENT_LIST_DIR}/categories.cpp
${CMAKE_CURRENT_LIST_DIR}/config.cpp
${CMAKE_CURRENT_LIST_DIR}/constraint.cpp
@@ -16,6 +17,7 @@ set(core_sources
${CMAKE_CURRENT_LIST_DIR}/timemory.cpp)
set(core_headers
${CMAKE_CURRENT_LIST_DIR}/argparse.hpp
${CMAKE_CURRENT_LIST_DIR}/categories.hpp
${CMAKE_CURRENT_LIST_DIR}/common.hpp
${CMAKE_CURRENT_LIST_DIR}/concepts.hpp
source/lib/core/argparse.cpp
+1418
View File
File diff suppressed because it is too large Load Diff
source/lib/core/argparse.hpp
+93
View File
@@ -0,0 +1,93 @@
// MIT License
//
// Copyright (c) 2022 Advanced Micro Devices, Inc. All Rights Reserved.
//
// Permission is hereby granted, free of charge, to any person obtaining a copy
// of this software and associated documentation files (the "Software"), to deal
// in the Software without restriction, including without limitation the rights
// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
// copies of the Software, and to permit persons to whom the Software is
// furnished to do so, subject to the following conditions:
//
// The above copyright notice and this permission notice shall be included in all
// copies or substantial portions of the Software.
//
// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
// SOFTWARE.
#pragma once
#include "defines.hpp"
#include <timemory/settings/vsettings.hpp>
#include <timemory/utility/argparse.hpp>
#include <functional>
#include <set>
#include <vector>
namespace omnitrace
{
namespace argparse
{
struct parser_data;
using parser_t = ::tim::argparse::argument_parser;
using vsetting_t = ::tim::vsettings;
using vsettings_set_t = std::set<vsetting_t*>;
using strset_t = std::set<std::string>;
using strvec_t = std::vector<std::string>;
using setting_filter_t = std::function<bool(vsetting_t*, const parser_data&)>;
using environ_filter_t = std::function<bool(std::string_view, const parser_data&)>;
using grouping_filter_t = std::function<bool(std::string_view, const parser_data&)>;
bool
default_setting_filter(vsetting_t*, const parser_data&);
bool
default_environ_filter(std::string_view, const parser_data&);
bool
default_grouping_filter(std::string_view, const parser_data&);
struct parser_data
{
bool monochrome = false;
bool debug = false;
int verbose = 0;
std::string dl_libpath = {};
std::string omni_libpath = {};
std::string launcher = {};
vsettings_set_t processed_settings = {};
std::set<std::string> processed_environs = {};
std::set<std::string> processed_groups = {};
std::vector<char*> current = {};
std::vector<char*> command = {};
std::set<std::string_view> updated = {};
std::set<std::string> initial = {};
grouping_filter_t grouping_filter = default_grouping_filter;
setting_filter_t setting_filter = default_setting_filter;
environ_filter_t environ_filter = default_environ_filter;
};
parser_data&
init_parser(parser_data&);
parser_data&
add_ld_preload(parser_data&);
parser_data&
add_core_arguments(parser_t&, parser_data&);
parser_data&
add_group_arguments(parser_t&, const std::string&, parser_data&, bool _add_group = false);
parser_data&
add_extended_arguments(parser_t&, parser_data&);
} // namespace argparse
} // namespace omnitrace
source/lib/core/common.hpp
+3 -3
View File
@@ -22,10 +22,10 @@
#pragma once
#include "categories.hpp"
#include "common/join.hpp"
#include "concepts.hpp"
#include "defines.hpp"
#include "core/categories.hpp"
#include "core/concepts.hpp"
#include "core/defines.hpp"
#include <timemory/api.hpp>
#include <timemory/api/macros.hpp>
source/lib/core/config.cpp
+29 -13
View File
@@ -281,7 +281,7 @@ configure_settings(bool _init)
std::string, "OMNITRACE_MODE",
"Data collection mode. Used to set default values for OMNITRACE_USE_* options. "
"Typically set by omnitrace binary instrumenter.",
std::string{ "trace" }, "backend", "advanced")
std::string{ "trace" }, "backend", "advanced", "mode")
->set_choices({ "trace", "sampling", "causal", "coverage" });
OMNITRACE_CONFIG_SETTING(bool, "OMNITRACE_CI",
@@ -308,7 +308,7 @@ configure_settings(bool _init)
"threads that get sampled, omnitrace can start all the background threads during "
"initialization",
get_env<size_t>("OMNITRACE_NUM_THREADS", 1), "threading", "performance",
"sampling", "debugging", "advanced");
"sampling", "parallelism", "advanced");
OMNITRACE_CONFIG_SETTING(bool, "OMNITRACE_USE_PERFETTO", "Enable perfetto backend",
_default_perfetto_v, "backend", "perfetto");
@@ -680,13 +680,13 @@ configure_settings(bool _init)
"Enable collecting profiling and trace data for these "
"categories and disable all other categories",
"", "trace", "profile", "perfetto", "timemory", "data",
"advanced")
"category", "advanced")
->set_choices(get_available_categories<std::vector<std::string>>());
OMNITRACE_CONFIG_SETTING(
std::string, "OMNITRACE_DISABLE_CATEGORIES",
"Disable collecting profiling and trace data for these categories", "", "trace",
"profile", "perfetto", "timemory", "data", "advanced")
"profile", "perfetto", "timemory", "data", "category", "advanced")
->set_choices(get_available_categories<std::vector<std::string>>());
OMNITRACE_CONFIG_SETTING(bool, "OMNITRACE_PERFETTO_ANNOTATIONS",
@@ -705,19 +705,18 @@ configure_settings(bool _init)
OMNITRACE_CONFIG_EXT_SETTING(int64_t, "OMNITRACE_CRITICAL_TRACE_COUNT",
"Number of critical trace to export (0 == all)",
int64_t{ 0 }, "data", "critical_trace",
"omnitrace-critical-trace", "perfetto", "advanced");
int64_t{ 0 }, "critical_trace",
"omnitrace-critical-trace", "advanced");
OMNITRACE_CONFIG_SETTING(uint64_t, "OMNITRACE_CRITICAL_TRACE_BUFFER_COUNT",
"Number of critical trace records to store in thread-local "
"memory before submitting to shared buffer",
uint64_t{ 2000 }, "data", "critical_trace", "advanced");
uint64_t{ 2000 }, "critical_trace", "advanced");
OMNITRACE_CONFIG_EXT_SETTING(
int64_t, "OMNITRACE_CRITICAL_TRACE_PER_ROW",
"How many critical traces per row in perfetto (0 == all in one row)",
int64_t{ 0 }, "io", "critical_trace", "omnitrace-critical-trace", "perfetto",
"advanced");
int64_t{ 0 }, "critical_trace", "omnitrace-critical-trace", "advanced");
OMNITRACE_CONFIG_SETTING(
std::string, "OMNITRACE_TIMEMORY_COMPONENTS",
@@ -1506,9 +1505,15 @@ print_banner(std::ostream& _os)
)banner";
auto _tag = std::string_view{ OMNITRACE_GIT_DESCRIBE };
auto _rev = std::string_view{ OMNITRACE_GIT_REVISION };
std::stringstream _version_info{};
#if OMNITRACE_HIP_VERSION_MAJOR > 0
auto _hip = JOIN('.', OMNITRACE_HIP_VERSION_MAJOR, OMNITRACE_HIP_VERSION_MINOR, "x");
#else
auto _hip = std::string_view{};
#endif
std::stringstream _version_info{};
_version_info << "omnitrace v" << OMNITRACE_VERSION_STRING;
if(!_tag.empty() || !_rev.empty())
if(!_tag.empty() || !_rev.empty() || !_hip.empty())
{
_version_info << " (";
if(!_tag.empty())
@@ -1516,10 +1521,21 @@ print_banner(std::ostream& _os)
_version_info << "tag: " << OMNITRACE_GIT_DESCRIBE;
if(!_rev.empty()) _version_info << ", ";
}
if(!_rev.empty()) _version_info << "rev: " << OMNITRACE_GIT_REVISION;
_version_info << ")";
if(!_rev.empty())
{
_version_info << "rev: " << OMNITRACE_GIT_REVISION;
if(!_hip.empty()) _version_info << ", ";
}
if(!_hip.empty())
{
_version_info << "rocm: " << _hip;
}
}
if(!_version_info.str().empty()) _version_info << ")";
tim::log::stream(_os, tim::log::color::info()) << _banner << _version_info.str();
_os << std::endl;
}
source/lib/omnitrace-dl/CMakeLists.txt
+4 -4
View File
@@ -17,8 +17,8 @@ add_library(omnitrace::omnitrace-dl-library ALIAS omnitrace-dl-library)
target_sources(
omnitrace-dl-library
PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}/dl.cpp ${CMAKE_CURRENT_SOURCE_DIR}/dl.hpp
${CMAKE_CURRENT_SOURCE_DIR}/main.c)
PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}/dl.cpp ${CMAKE_CURRENT_SOURCE_DIR}/main.c
${CMAKE_CURRENT_SOURCE_DIR}/dl/dl.hpp)
target_include_directories(
omnitrace-dl-library
PUBLIC $<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}>
@@ -44,7 +44,7 @@ set_target_properties(
omnitrace_strip_target(omnitrace-dl-library)
install(FILES ${CMAKE_CURRENT_SOURCE_DIR}/dl.hpp
DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}/omnitrace)
install(FILES ${CMAKE_CURRENT_SOURCE_DIR}/dl/dl.hpp
DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}/omnitrace/dl)
install(TARGETS omnitrace-dl-library DESTINATION ${CMAKE_INSTALL_LIBDIR})
source/lib/omnitrace-dl/dl.cpp
+364 -29
View File
@@ -38,12 +38,21 @@
#include "common/invoke.hpp"
#include "common/join.hpp"
#include "common/setup.hpp"
#include "dl.hpp"
#include "dl/dl.hpp"
#include "omnitrace/categories.h"
#include "omnitrace/types.h"
#include <timemory/utility/filepath.hpp>
#include <cassert>
#include <chrono>
#include <gnu/libc-version.h>
#include <link.h>
#include <linux/limits.h>
#include <sys/types.h>
#include <sys/wait.h>
#include <thread>
#include <unistd.h>
//--------------------------------------------------------------------------------------//
@@ -69,6 +78,8 @@
//--------------------------------------------------------------------------------------//
using main_func_t = int (*)(int, char**, char**);
std::ostream&
operator<<(std::ostream& _os, const SpaceHandle& _handle)
{
@@ -78,7 +89,7 @@ operator<<(std::ostream& _os, const SpaceHandle& _handle)
namespace omnitrace
{
inline namespace dl
namespace dl
{
namespace
{
@@ -97,6 +108,16 @@ get_omnitrace_dl_env()
: get_env("OMNITRACE_DL_VERBOSE", get_omnitrace_env());
}
inline bool&
get_omnitrace_is_preloaded()
{
static bool _v = []() {
auto&& _preload_libs = get_env("LD_PRELOAD", std::string{});
return (_preload_libs.find("libomnitrace-dl.so") != std::string::npos);
}();
return _v;
}
inline bool
get_omnitrace_preload()
{
@@ -136,6 +157,12 @@ get_omnitrace_root_pid()
return get_env("OMNITRACE_ROOT_PROCESS", _pid);
}
void
omnitrace_preinit() OMNITRACE_INTERNAL_API;
void
omnitrace_postinit(std::string exe = {}) OMNITRACE_INTERNAL_API;
pid_t _omnitrace_root_pid = get_omnitrace_root_pid();
// environment priority:
@@ -189,7 +216,7 @@ const char* _omnitrace_dl_dlopen_descr = "RTLD_LAZY | RTLD_LOCAL";
#endif
/// This class contains function pointers for omnitrace's instrumentation functions
struct OMNITRACE_HIDDEN_API indirect
struct OMNITRACE_INTERNAL_API indirect
{
OMNITRACE_INLINE indirect(const std::string& _omnilib, const std::string& _userlib,
const std::string& _dllib)
@@ -456,7 +483,7 @@ private:
};
inline indirect&
get_indirect() OMNITRACE_HIDDEN_API;
get_indirect() OMNITRACE_INTERNAL_API;
indirect&
get_indirect()
@@ -519,6 +546,13 @@ get_thread_status()
return _v;
}
InstrumentMode&
get_instrumented()
{
static auto _v = get_env("OMNITRACE_INSTRUMENT_MODE", InstrumentMode::None);
return _v;
}
// ensure finalization is called
bool _omnitrace_dl_fini = (std::atexit([]() {
if(get_active()) omnitrace_finalize();
@@ -555,7 +589,7 @@ bool _omnitrace_dl_fini = (std::atexit([]() {
fflush(stderr); \
}
using omnitrace::get_indirect;
using omnitrace::dl::get_indirect;
namespace dl = omnitrace::dl;
extern "C"
@@ -598,6 +632,9 @@ extern "C"
return;
}
if(dl::get_instrumented() < dl::InstrumentMode::PythonProfile)
dl::omnitrace_preinit();
bool _invoked = false;
OMNITRACE_DL_INVOKE_STATUS(_invoked, get_indirect().omnitrace_init_f, a, b, c);
if(_invoked)
@@ -605,6 +642,8 @@ extern "C"
dl::get_active() = true;
dl::get_inited() = true;
dl::_omnitrace_dl_verbose = dl::get_omnitrace_dl_env();
if(dl::get_instrumented() < dl::InstrumentMode::PythonProfile)
dl::omnitrace_postinit((c) ? std::string{ c } : std::string{});
}
}
@@ -727,8 +766,9 @@ extern "C"
OMNITRACE_DL_IGNORE(2, "already initialized and active", a, b);
return;
}
OMNITRACE_DL_LOG(2, "%s(%s, %s)\n", __FUNCTION__, a, b);
setenv(a, b, 0);
OMNITRACE_DL_INVOKE(get_indirect().omnitrace_set_env_f, a, b);
// OMNITRACE_DL_INVOKE(get_indirect().omnitrace_set_env_f, a, b);
}
void omnitrace_set_mpi(bool a, bool b)
@@ -840,6 +880,22 @@ extern "C"
_annotations, _annotation_count);
}
void omnitrace_set_instrumented(int _mode)
{
OMNITRACE_DL_LOG(2, "%s(%i)\n", __FUNCTION__, _mode);
auto _mode_v = static_cast<dl::InstrumentMode>(_mode);
if(_mode_v < dl::InstrumentMode::None || _mode_v >= dl::InstrumentMode::Last)
{
OMNITRACE_DL_LOG(-127,
"%s(mode=%i) invoked with invalid instrumentation mode. "
"mode should be %i >= mode < %i\n",
__FUNCTION__, _mode,
static_cast<int>(dl::InstrumentMode::None),
static_cast<int>(dl::InstrumentMode::Last));
}
dl::get_instrumented() = _mode_v;
}
//----------------------------------------------------------------------------------//
//
// KokkosP
@@ -1060,18 +1116,153 @@ extern "C"
namespace omnitrace
{
inline namespace dl
namespace dl
{
namespace
{
bool
omnitrace_preload() OMNITRACE_HIDDEN_API;
omnitrace_preload() OMNITRACE_INTERNAL_API;
std::vector<std::string>
get_link_map(const char*,
std::vector<int>&& = { (RTLD_LAZY | RTLD_NOLOAD) }) OMNITRACE_INTERNAL_API;
const char*
get_default_mode() OMNITRACE_INTERNAL_API;
void
verify_instrumented_preloaded() OMNITRACE_INTERNAL_API;
std::vector<std::string>
get_link_map(const char* _name, std::vector<int>&& _open_modes)
{
void* _handle = nullptr;
bool _noload = false;
for(auto _mode : _open_modes)
{
_handle = dlopen(_name, _mode);
_noload = (_mode & RTLD_NOLOAD) == RTLD_NOLOAD;
if(_handle) break;
}
auto _chain = std::vector<std::string>{};
if(_handle)
{
struct link_map* _link_map = nullptr;
dlinfo(_handle, RTLD_DI_LINKMAP, &_link_map);
struct link_map* _next = _link_map->l_next;
while(_next)
{
if(_next->l_name != nullptr && !std::string_view{ _next->l_name }.empty())
{
_chain.emplace_back(_next->l_name);
}
_next = _next->l_next;
}
if(_noload == false) dlclose(_handle);
}
return _chain;
}
const char*
get_default_mode()
{
if(get_env("OMNITRACE_USE_CAUSAL", false)) return "causal";
auto _link_map = get_link_map(nullptr);
for(const auto& itr : _link_map)
{
if(itr.find("libomnitrace-rt.so") != std::string::npos ||
itr.find("libdyninstAPI_RT.so") != std::string::npos)
return "trace";
}
return "sampling";
}
void
omnitrace_preinit()
{
switch(get_instrumented())
{
case InstrumentMode::None:
case InstrumentMode::BinaryRewrite:
case InstrumentMode::ProcessCreate:
case InstrumentMode::ProcessAttach:
{
auto _use_mpip = get_env("OMNITRACE_USE_MPIP", false);
auto _use_mpi = get_env("OMNITRACE_USE_MPI", _use_mpip);
auto _causal = get_env("OMNITRACE_USE_CAUSAL", false);
auto _mode = get_env("OMNITRACE_MODE", get_default_mode());
if(_use_mpi && !(_causal && _mode == "causal"))
{
// only make this call if true bc otherwise, if
// false, it will disable the MPIP component and
// we may intercept the MPI init call later.
// If _use_mpi defaults to true above, calling this
// will override can current env or config value for
// OMNITRACE_USE_PID.
omnitrace_set_mpi(_use_mpi, dl::get_instrumented() ==
dl::InstrumentMode::ProcessAttach);
}
break;
}
case InstrumentMode::PythonProfile:
case InstrumentMode::Last: break;
}
}
void
omnitrace_postinit(std::string _exe)
{
switch(get_instrumented())
{
case InstrumentMode::None:
case InstrumentMode::BinaryRewrite:
case InstrumentMode::ProcessCreate:
case InstrumentMode::ProcessAttach:
{
if(_exe.empty())
_exe = tim::filepath::readlink(join('/', "/proc", getpid(), "exe"));
omnitrace_init_tooling();
if(_exe.empty())
omnitrace_push_trace("main");
else
omnitrace_push_trace(basename(_exe.c_str()));
break;
}
case InstrumentMode::PythonProfile:
{
omnitrace_init_tooling();
break;
}
case InstrumentMode::Last: break;
}
}
bool
omnitrace_preload()
{
auto _preload = get_omnitrace_preload() && get_env("OMNITRACE_ENABLED", true);
auto _use_mpi = get_env("OMNITRACE_USE_MPI", get_env("OMNITRACE_USE_MPIP", false));
auto _preload = get_omnitrace_is_preloaded() && get_omnitrace_preload() &&
get_env("OMNITRACE_ENABLED", true);
auto _link_map = get_link_map(nullptr);
auto _instr_mode =
get_env("OMNITRACE_INSTRUMENT_MODE", dl::InstrumentMode::BinaryRewrite);
for(const auto& itr : _link_map)
{
if(itr.find("libomnitrace-rt.so") != std::string::npos ||
itr.find("libdyninstAPI_RT.so") != std::string::npos)
{
omnitrace_set_instrumented(static_cast<int>(_instr_mode));
break;
}
}
verify_instrumented_preloaded();
static bool _once = false;
if(_once) return _preload;
@@ -1081,30 +1272,174 @@ omnitrace_preload()
{
reset_omnitrace_preload();
omnitrace_preinit_library();
auto _causal = get_env("OMNITRACE_USE_CAUSAL", false);
auto _mode = get_env("OMNITRACE_MODE", (_causal) ? "causal" : "sampling");
OMNITRACE_DL_LOG(1, "[%s] invoking %s(%s)\n", __FUNCTION__, "omnitrace_init",
::omnitrace::join(::omnitrace::QuoteStrings{}, ", ", _mode,
false, "omnitrace")
.c_str());
if(_use_mpi && !(_causal && _mode == "causal"))
{
// only make this call if true bc otherwise, if
// false, it will disable the MPIP component and
// we may intercept the MPI init call later.
// If _use_mpi defaults to true above, calling this
// will override can current env or config value for
// OMNITRACE_USE_PID.
omnitrace_set_mpi(_use_mpi, false);
}
omnitrace_init(_mode.c_str(), false, nullptr);
omnitrace_init_tooling();
}
return _preload;
}
bool _handle_preload = omnitrace::dl::omnitrace_preload();
void
verify_instrumented_preloaded()
{
// if preloaded then we are fine
if(get_omnitrace_is_preloaded()) return;
// value returned by get_instrumented is set by either:
// - the search of the linked libraries
// - via the instrumenter
// if binary rewrite or runtime instrumentation, there is an opportunity for
// LD_PRELOAD
switch(dl::get_instrumented())
{
case dl::InstrumentMode::None:
case dl::InstrumentMode::ProcessAttach:
case dl::InstrumentMode::ProcessCreate:
{
return;
}
case dl::InstrumentMode::BinaryRewrite:
{
break;
}
case dl::InstrumentMode::Last:
{
throw std::runtime_error(
"Invalid instrumentation type: InstrumentMode::Last");
}
}
static const char* _notice = R"notice(
NNNNNNNN NNNNNNNN OOOOOOOOO TTTTTTTTTTTTTTTTTTTTTTTIIIIIIIIII CCCCCCCCCCCCCEEEEEEEEEEEEEEEEEEEEEE
N:::::::N N::::::N OO:::::::::OO T:::::::::::::::::::::TI::::::::I CCC::::::::::::CE::::::::::::::::::::E
N::::::::N N::::::N OO:::::::::::::OO T:::::::::::::::::::::TI::::::::I CC:::::::::::::::CE::::::::::::::::::::E
N:::::::::N N::::::NO:::::::OOO:::::::OT:::::TT:::::::TT:::::TII::::::IIC:::::CCCCCCCC::::CEE::::::EEEEEEEEE::::E
N::::::::::N N::::::NO::::::O O::::::OTTTTTT T:::::T TTTTTT I::::I C:::::C CCCCCC E:::::E EEEEEE
N:::::::::::N N::::::NO:::::O O:::::O T:::::T I::::IC:::::C E:::::E
N:::::::N::::N N::::::NO:::::O O:::::O T:::::T I::::IC:::::C E::::::EEEEEEEEEE
N::::::N N::::N N::::::NO:::::O O:::::O T:::::T I::::IC:::::C E:::::::::::::::E
N::::::N N::::N:::::::NO:::::O O:::::O T:::::T I::::IC:::::C E:::::::::::::::E
N::::::N N:::::::::::NO:::::O O:::::O T:::::T I::::IC:::::C E::::::EEEEEEEEEE
N::::::N N::::::::::NO:::::O O:::::O T:::::T I::::IC:::::C E:::::E
N::::::N N:::::::::NO::::::O O::::::O T:::::T I::::I C:::::C CCCCCC E:::::E EEEEEE
N::::::N N::::::::NO:::::::OOO:::::::O TT:::::::TT II::::::IIC:::::CCCCCCCC::::CEE::::::EEEEEEEE:::::E
N::::::N N:::::::N OO:::::::::::::OO T:::::::::T I::::::::I CC:::::::::::::::CE::::::::::::::::::::E
N::::::N N::::::N OO:::::::::OO T:::::::::T I::::::::I CCC::::::::::::CE::::::::::::::::::::E
NNNNNNNN NNNNNNN OOOOOOOOO TTTTTTTTTTT IIIIIIIIII CCCCCCCCCCCCCEEEEEEEEEEEEEEEEEEEEEE
_ _ _____ ______
| | | |/ ____| ____|
| | | | (___ | |__
| | | |\___ \| __|
| |__| |____) | |____
\____/|_____/|______|
____ __ __ _ _ _____ _______ _____ _____ ______ _____ _ _ _ _
/ __ \| \/ | \ | |_ _|__ __| __ \ /\ / ____| ____| | __ \| | | | \ | |
| | | | \ / | \| | | | | | | |__) | / \ | | | |__ ______| |__) | | | | \| |
| | | | |\/| | . ` | | | | | | _ / / /\ \| | | __|______| _ /| | | | . ` |
| |__| | | | | |\ |_| |_ | | | | \ \ / ____ \ |____| |____ | | \ \| |__| | |\ |
\____/|_| |_|_| \_|_____| |_| |_| \_\/_/ \_\_____|______| |_| \_\\____/|_| \_|
Due to a variety of edge cases we've encountered, OmniTrace now requires that binary rewritten executables and libraries be launched
with the 'omnitrace-run' executable.
In order to launch the executable with 'omnitrace-run', prefix the current command with 'omnitrace-run' and a standalone double hyphen ('--').
For MPI applications, place 'omnitrace-run --' after the MPI command.
E.g.:
<EXECUTABLE> <ARGS...>
mpirun -n 2 <EXECUTABLE> <ARGS...>
should be:
omnitrace-run -- <EXECUTABLE> <ARGS...>
mpirun -n 2 omnitrace-run -- <EXECUTABLE> <ARGS...>
Note: the command-line arguments passed to 'omnitrace-run' (which are specified before the double hyphen) will override configuration variables
and/or any configuration values specified to 'omnitrace-instrument' via the '--config' or '--env' options.
E.g.:
$ omnitrace-instrument -o ./sleep.inst --env OMNITRACE_SAMPLING_DELAY=5.0 -- sleep
$ echo "OMNITRACE_SAMPLING_FREQ = 500" > omnitrace.cfg
$ export OMNITRACE_CONFIG_FILE=omnitrace.cfg
$ omnitrace-run --sampling-freq=100 --sampling-delay=1.0 -- ./sleep.inst 10
In the first command, a default sampling delay of 5 seconds in embedded into the instrumented 'sleep.inst'.
In the second command, the sampling frequency will be set to 500 interrupts per second when OmniTrace reads the config file
In the fourth command, the sampling frequency and sampling delay are overridden to 100 interrupts per second and 1 second, respectively, when sleep.inst runs
Thanks for using OmniTrace and happy optimizing!
)notice";
// emit notice
std::cerr << _notice << std::endl;
std::quick_exit(EXIT_FAILURE);
}
bool _handle_preload = omnitrace_preload();
main_func_t main_real = nullptr;
} // namespace
} // namespace dl
} // namespace omnitrace
extern "C"
{
int omnitrace_main(int argc, char** argv, char** envp) OMNITRACE_INTERNAL_API;
void omnitrace_set_main(main_func_t) OMNITRACE_INTERNAL_API;
void omnitrace_set_main(main_func_t _main_real)
{
::omnitrace::dl::main_real = _main_real;
}
int omnitrace_main(int argc, char** argv, char** envp)
{
OMNITRACE_DL_LOG(0, "%s\n", __FUNCTION__);
using ::omnitrace::common::get_env;
using ::omnitrace::dl::get_default_mode;
// prevent re-entry
static int _reentry = 0;
if(_reentry > 0) return -1;
_reentry = 1;
if(!::omnitrace::dl::main_real)
throw std::runtime_error("[omnitrace][dl] Unsuccessful wrapping of main: "
"nullptr to real main function");
if(envp)
{
size_t _idx = 0;
while(envp[_idx] != nullptr)
{
auto _env_v = std::string_view{ envp[_idx++] };
if(_env_v.find("OMNITRACE") != 0 &&
_env_v.find("libomnitrace") == std::string_view::npos)
continue;
auto _pos = _env_v.find('=');
if(_pos < _env_v.length())
{
auto _var = std::string{ _env_v }.substr(0, _pos);
auto _val = std::string{ _env_v }.substr(_pos + 1);
OMNITRACE_DL_LOG(1, "%s(%s, %s)\n", "omnitrace_set_env", _var.c_str(),
_val.c_str());
setenv(_var.c_str(), _val.c_str(), 0);
}
}
}
auto _mode = get_env("OMNITRACE_MODE", get_default_mode());
omnitrace_init(_mode.c_str(),
dl::get_instrumented() == dl::InstrumentMode::BinaryRewrite,
argv[0]);
int ret = (*::omnitrace::dl::main_real)(argc, argv, envp);
omnitrace_pop_trace(basename(argv[0]));
omnitrace_finalize();
return ret;
}
}
source/lib/omnitrace-dl/dl.hpp → source/lib/omnitrace-dl/dl/dl.hpp
+17
View File
@@ -76,6 +76,7 @@ extern "C"
void omnitrace_set_env(const char* env_name,
const char* env_val) OMNITRACE_PUBLIC_API;
void omnitrace_set_mpi(bool use, bool attached) OMNITRACE_PUBLIC_API;
void omnitrace_set_instrumented(int) OMNITRACE_PUBLIC_API;
void omnitrace_push_trace(const char* name) OMNITRACE_PUBLIC_API;
void omnitrace_pop_trace(const char* name) OMNITRACE_PUBLIC_API;
int omnitrace_push_region(const char*) OMNITRACE_PUBLIC_API;
@@ -191,4 +192,20 @@ extern "C"
#endif
}
namespace omnitrace
{
namespace dl
{
enum class InstrumentMode : int
{
None = -1,
BinaryRewrite = 0,
ProcessCreate = 1, // runtime instrumentation at start of process
ProcessAttach = 2, // runtime instrumentation of running process
PythonProfile = 3, // python setprofile
Last,
};
}
} // namespace omnitrace
#endif // OMNITRACE_DL_HPP_ 1
source/lib/omnitrace-dl/main.c
+12 -38
View File
@@ -22,8 +22,9 @@
#define _GNU_SOURCE
#define OMNITRACE_PUBLIC_API __attribute__((visibility("default")));
#define OMNITRACE_HIDDEN_API __attribute__((visibility("hidden")));
#define OMNITRACE_PUBLIC_API __attribute__((visibility("default")));
#define OMNITRACE_HIDDEN_API __attribute__((visibility("hidden")));
#define OMNITRACE_INTERNAL_API __attribute__((visibility("internal")));
#include <dlfcn.h>
#include <stdbool.h>
@@ -35,25 +36,18 @@
//
// local type definitions
//
typedef int (*main_func_t)(int, char**, char**);
typedef int (*start_main_t)(int (*)(int, char**, char**), int, char**,
int (*)(int, char**, char**), void (*)(void), void (*)(void),
void*);
//
// local variables
//
static int (*main_real)(int, char**, char**); // Trampoline for the real main()
//
// local function declarations
//
int
omnitrace_main(int, char**, char**) OMNITRACE_HIDDEN_API;
int
omnitrace_libc_start_main(int (*)(int, char**, char**), int, char**,
int (*)(int, char**, char**), void (*)(void), void (*)(void),
void*) OMNITRACE_HIDDEN_API;
void*) OMNITRACE_INTERNAL_API;
int
__libc_start_main(int (*)(int, char**, char**), int, char**, int (*)(int, char**, char**),
@@ -80,32 +74,13 @@ omnitrace_init_tooling(void);
extern void
omnitrace_init(const char*, bool, const char*);
//
// local function definitions
//
int
omnitrace_main(int argc, char** argv, char** envp)
{
// prevent re-entry
static int _reentry = 0;
if(_reentry > 0) return -1;
_reentry = 1;
extern char*
basename(const char*);
// set the relevant environment variables
// omnitrace_update_env(&envp);
extern void omnitrace_set_main(main_func_t) OMNITRACE_INTERNAL_API;
const char* mode = getenv("OMNITRACE_MODE");
omnitrace_init(mode ? mode : "sampling", false, argv[0]);
omnitrace_init_tooling();
omnitrace_push_trace(basename(argv[0]));
int ret = main_real(argc, argv, envp);
omnitrace_pop_trace(basename(argv[0]));
omnitrace_finalize();
return ret;
}
extern int
omnitrace_main(int argc, char** argv, char** envp) OMNITRACE_INTERNAL_API;
int
omnitrace_libc_start_main(int (*_main)(int, char**, char**), int _argc, char** _argv,
@@ -123,7 +98,7 @@ omnitrace_libc_start_main(int (*_main)(int, char**, char**), int _argc, char** _
void* _this_func = __builtin_return_address(0);
// Save the real main function address
main_real = _main;
omnitrace_set_main(_main);
// Find the real __libc_start_main()
start_main_t user_main = dlsym(RTLD_NEXT, "__libc_start_main");
@@ -136,8 +111,7 @@ omnitrace_libc_start_main(int (*_main)(int, char**, char**), int _argc, char** _
if(_preload == 0)
{
// call original main
return user_main(main_real, _argc, _argv, _init, _fini, _rtld_fini,
_stack_end);
return user_main(_main, _argc, _argv, _init, _fini, _rtld_fini, _stack_end);
}
else
{
source/lib/omnitrace/library.cpp
+4 -3
View File
@@ -150,7 +150,7 @@ ensure_finalization(bool _static_init = false)
_tid->system_value);
}
if(get_env("OMNITRACE_MONOCHROME", false)) tim::log::monochrome() = true;
if(common::get_env("OMNITRACE_MONOCHROME", false)) tim::log::monochrome() = true;
(void) tim::manager::instance();
(void) tim::settings::shared_instance();
@@ -637,7 +637,7 @@ extern "C" void
omnitrace_reset_preload_hidden(void)
{
tim::set_env("OMNITRACE_PRELOAD", "0", 1);
auto&& _preload_libs = get_env("LD_PRELOAD", std::string{});
auto&& _preload_libs = common::get_env("LD_PRELOAD", std::string{});
if(_preload_libs.find("libomnitrace") != std::string::npos)
{
auto _modified_preload = std::string{};
@@ -732,7 +732,8 @@ omnitrace_finalize_hidden(void)
if(dmp::rank() == 0)
{
OMNITRACE_PRINT_F("\n");
config::print_settings(get_env<bool>("OMNITRACE_PRINT_ENV", get_debug()));
config::print_settings(
tim::get_env<bool>("OMNITRACE_PRINT_ENV", get_debug()));
}
}
source/lib/omnitrace/library/components/pthread_create_gotcha.cpp
+2
View File
@@ -36,6 +36,7 @@
#include <timemory/backends/threading.hpp>
#include <timemory/components/macros.hpp>
#include <timemory/components/timing/wall_clock.hpp>
#include <timemory/mpl/types.hpp>
#include <timemory/sampling/allocator.hpp>
#include <timemory/utility/types.hpp>
@@ -76,6 +77,7 @@ inline void
start_bundle(bundle_t& _bundle, Args&&... _args)
{
if(!get_use_timemory() && !get_use_perfetto()) return;
trait::runtime_enabled<comp::roctracer_data>::set(get_use_roctracer());
OMNITRACE_BASIC_VERBOSE_F(3, "starting bundle '%s'...\n", _bundle.key().c_str());
if constexpr(sizeof...(Args) > 0)
{