Linux Perf Support + Causal Profiling Updates (#276)
* causal backtrace updates
- fix initial causal sampling period value
* causal delay updates
- tweak handling of sleep_for_overhead
* Fix experiment global scaling for prog pts
- results in drastically improved predictions
* pthread_mutex_gotcha updates
- disable all wrappers during causal profiling
* validate-causal-json.py updates
- support decimal stddev
- fix setting stddev from command-line
* causal perform_experiment_impl update
- handle start failing because finalizing
* deprecate causal::component::sample_rate
- appears to not help at all
* Rework sample info
* Increase causal unwind_depth
- use OMNITRACE_MAX_UNWIND_DEPTH
* validate-causal-json updates
- min experiments
- exclude reporting predictions with less than X experiments at a given speedup
- percent samples
- only print samples within X% of the peak (default: 95%)
* Update timemory submodule
- extensions to sampling for signals delivered via non-timer method
- e.g. via HW counter overflow
* dwarf_entry::operator< updates
- sort via file
* causal profiling docs updates
- info about backends
- info about installing/enabling perf
* config updates: causal backend
- CausalBackend enum
- OMNITRACE_CAUSAL_BACKEND: perf, timer, auto
- omnitrace-causal option: --backend
* debug update
- use spin_mutex instead of std::mutex
* address_range::contains update
- range from 0-100 contains range from 10-100 but was returning false because high was == 100 not < 100
* symbol::operator< update
- handle load address differences
* sampling updates (non-causal)
- update get_timer to get_trigger + dynamic_cast
* container::static_vector updates
- support construction from container::c_array
- update_size private member func for handling atomic m_size
* Move perf files
- moved library/causal/perf.{hpp,cpp} to library/perf.{hpp,cpp}
* causal example update
- created impl.hpp (forward decls)
- renamed {cpu,rng}_func_impl to {cpu,rng}_impl_func
- only create two threads which run N iterations instead of two threads each iteration
* Update timemory submodule
- updates to unwind::processed_entry
- updates to procfs::maps
* Updated causal documentation
- fixed line numbers changed by modifications to causal example
* omnitrace-causal exe updates
- set OMNITRACE_THREAD_POOL_SIZE to zero by default
* core/containers updates
- static_vector: provide data() member function
- c_array pop_front() and pop_back() member functions
* core: config and argparse updates + perf
- core/perf.{hpp,cpp}
- forward decl of enums
- config-related capabilities
- argparse: --sample-overflow
- renamed some config functions
- e.g. get_sampling_cpu_freq -> get_sampling_cputime_freq
- added config settings related to overflow sampling via perf
- added timer_sampling and overflow_sampling categories
* Update timemory submodule
- sampling allocator flushing
* binary updates
- lookup_ipaddr_entry
- use bfd_find_nearest_line instead of bfd_find_nearest_line_discriminator
- discriminators are not used
- explicit instantiations of inlined_symbol::serialize
* Bump VERSION to 1.10.0
* sampling and perf updates
- support overflow sampling via Linux Perf
- update perf namespace
- update perf::perf_event
- update record ctor: pointer instead of const ref
- update open member func: return optional string
- add m_batch_size member variable
- sampling updates
- support overflow sampling
- flush allocators
- increase buffer size from 1024 to 2048
- restructure post-processing in light of perf overflow supports
- improve offload memory usage only load buffers for thread
- load_offload_buffer(tid) uses thread-specific filepos
- component updates
- backtrace_metrics::operator-=
- backtrace_metrics::operator-
- backtrace::sample does not record for overflow signal
- callchain: perf overflow sample
* core updates
- component::sampling_percent does not report self + uses_percent_units
* causal updates
- tweak get_line_info
- overloads for set_current_selection (uint64_t, c_array, std::array)
- delay
- use sampling::pause/sampling::resume
- experiment
- experiment::sample derives from unwind::processed_entry
- experiment::samples is vector instead of set
- fixed samples
- overloads for is_selected (uint64_t, c_array, std::array)
- scaling factor defaults to 100 instead of 50
- serialize updates follow change to experiment::sample
- modify algorithm for increasing/decreasing experiment length
- sample_data
- use map<uintptr, uint64_t> instead of set<sample_data>
- get_samples returns vector<sample_data> instead of set<sample_data>
- sampling
- support overflow via Linux Perf
- update causal_offload_buffer
- flush sampling allocator
- backtrace
- overflow component
* libomnitrace-dl updates
- handle dl::InstrumentMode::PythonProfile
* testing updates (causal)
- causal line 155 -> causal line 100
- causal line 165 -> causal line 110
* formatting
* exit_gotcha updates
- exit_info for abort()
- message about non-zero exit code
* testing updates
- fail regex for causal tests
- validate-causal-json: >= min_experiments instead of > min_experiments
- handle OMNITRACE_DEBUG_SETTINGS in omnitrace_write_test_config
* causal sampling updates
- add new lines where appropriate
* causal data updates
- reorder diagnostic info when experiment fails to start
* binary updates
- symbol address range from address to address + symsize + 1
- add 1 based on debug info
* causal data updates
- sample_selection wait_ns defaults to 1,000 instead of 10,000
- sample_selection wait scaled by iteration number
- save_line_info_impl verbosity
- print latest_eligible_pc when experiment does not start
* causal sampling + component updates
- perf backend disables component::backtrace
- ensure get_sampling_(realtime|cputime|overflow)_signal do not malloc
* causal: remove period stats
* validate-causal-json update
- fix --help
* causal data updates
- improve eligible pc history reporting when experiment fails to start
* causal data updates
- fix compute_eligible_lines_impl
- eligible address ranges returning too many ranges
- occasionally, overwrite all *true* eligible address ranges
* causal data updates
- reduce scoped ranges to symbol ranges
- is_eligible_address() returns true contains (not just coarse)
- revert some sample_selection behavior
* binary address_multirange updates
- make coarse_range private
- fix operator+=(pair<coarse, uintptr_t>)
* causal example update
- fix nsync to default to once per iteration
* binary analysis updates
- tweak header file includes
* causal updates
- remove factoring in sleep_for_overhead
- invoke delay::process() even if experiment is not active
* causal data updates
- update latest_eligible_pc structure
* update omnitrace-install.py.in
- fix support for fedora
- /etc/os-release does not have ID_LIKE
- fallback to RHEL 8.7 if version not specified
* update omnitrace-install.py.in
- fix support for debian
- /etc/os-release does not have ID_LIKE
- version mapping
* Update documentation
- update docs on installation
* causal data and experiment updates
- data: reset_sample_selection
* causal set_current_selection debugging
- debug messages for failed e2e runs
* causal data and backtrace component updates
- data: set_current_selection returns the number of eligible addresses added
- backtrace: if cputime signal has selected zero IPs > 5x, then realtime signal starts contributing call-stacks
* core library updates
- move config::parse_numeric_range to utility namespace
- add core/utility.cpp
- support range:increment, e.g. 5-25:10 expands to '5 15 25' instead of '5 10 15 20 25'
* omnitrace-causal update
- end-to-end expands all speedups
- support range:increment in speedups
* causal backtrace updates
- remove select_ival (realtime signal always contributes when select_count == 0)
* containers: static_vector update
- explicit c_array constructor
- explicit std::array constructor
* causal data updates
- remove set_current_selection(uint64_t)
- remove set_current_selection(std::array)
- sample_selection increase default wait time
- report eligible PC candidates
- move reset_sample_selection to perform_experiment_impl
- decrease latest_eligible_pc array size
- set_current_selection does not guard for experiment::active
* core debug updates
- OMNITRACE_PRINT_COLOR macros
* causal data updates
- tweak to experiment never started message
* causal gotcha updates
- remove unused code
* critical trace updates
- remove unused code
* omnitrace-causal
- OMNITRACE_LAUNCHER
* causal data updates
- don't fail on end-to-end + omnitrace-causal
* causal backtrace updates
- reintroduce select_ival behavior
* causal data updates
- tweak verbose messages about number of PC candidates
* core mproc updates
- utilities for waiting on child PID and diagnosing status
- omnitrace::mproc::wait_pid
- omnitrace::mproc::diagnose_status
* omnitrace-run updates
- support --fork argument for executing via fork in current process + execvpe on child instead of execvpe in current process
* omnitrace-causal updates
- wait_pid and diagnose_status just call equivalent functions in omnitrace::mproc
* ubuntu-focal workflow update
- attempt to launch ubuntu-focal-codecov job with CAP_SYS_ADMIN and use perf backend
* tests reorg and updates
- remove binary-rewrite-sampling and runtime-instrument-sampling tests
- rename *-preload tests (which use omnitrace-sample exe) to *-sampling
- split tests/CMakeLists.txt into several tests/omnitrace-<category>-tests.cmake files
- tweak to causal-both-omni-func test
- add args: -n 2 -b timer
* update validate-causal-json.py
- better reasoning info for adjusting tolerance
- always apply tolerance adjustments in CI mode
* causal e2e tests update
- add label "causal-e2e" label
- tweak params
- old: 80 12 432525 500000000
- new: 80 50 432525 100000000
- disable processor affinity for slow-func/line-100 tests
- artificially inflates some speedups with perf
* unblocking_gotcha updates
- overload operator() according to gotcha function index
* blocking_gotcha updates
- overload operator() according to gotcha function index
- fix bug where potentially post block functors (e.g. pthread_mutex_trylock) throw error if lock is not acquired.
* parse_numeric_range update
- support unordered_set
* config update
- OMNITRACE_DEBUG_{TIDS,PIDS} use parse_numeric_range
Цей коміт міститься в:
Jonathan R. Madsen authored and 
GitHub committed
1 джерело
cc14b52584
коміт
9de3a6b0b4
96 files changed
+5489
-3013
No files matched your search
+17
-1420
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,53 @@
|
||||
# -------------------------------------------------------------------------------------- #
|
||||
#
|
||||
# attach tests
|
||||
#
|
||||
# -------------------------------------------------------------------------------------- #
|
||||
|
||||
set(_VALID_PTRACE_SCOPE OFF)
|
||||
|
||||
if(EXISTS "/proc/sys/kernel/yama/ptrace_scope")
|
||||
file(READ "/proc/sys/kernel/yama/ptrace_scope" _PTRACE_SCOPE LIMIT 1)
|
||||
|
||||
if("${_PTRACE_SCOPE}" EQUAL 0)
|
||||
set(_VALID_PTRACE_SCOPE ON)
|
||||
endif()
|
||||
else()
|
||||
omnitrace_message(
|
||||
AUTHOR_WARNING
|
||||
"Disabling attach tests. Run 'echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope' to enable attaching to process"
|
||||
)
|
||||
endif()
|
||||
|
||||
if(NOT _VALID_PTRACE_SCOPE)
|
||||
return()
|
||||
endif()
|
||||
|
||||
if(NOT TARGET parallel-overhead)
|
||||
return()
|
||||
endif()
|
||||
|
||||
add_test(
|
||||
NAME parallel-overhead-attach
|
||||
COMMAND
|
||||
${CMAKE_CURRENT_LIST_DIR}/run-omnitrace-pid.sh $<TARGET_FILE:omnitrace-instrument>
|
||||
-ME "\.c$" -E fib -e -v 1 --label return args file -l --
|
||||
$<TARGET_FILE:parallel-overhead> 30 8 1000
|
||||
WORKING_DIRECTORY ${PROJECT_BINARY_DIR})
|
||||
|
||||
set(_parallel_overhead_attach_environ
|
||||
"${_attach_environment}" "OMNITRACE_OUTPUT_PATH=omnitrace-tests-output"
|
||||
"OMNITRACE_OUTPUT_PREFIX=parallel-overhead-attach/")
|
||||
|
||||
set_tests_properties(
|
||||
parallel-overhead-attach
|
||||
PROPERTIES ENVIRONMENT
|
||||
"${_parallel_overhead_attach_environ}"
|
||||
TIMEOUT
|
||||
300
|
||||
LABELS
|
||||
"parallel-overhead;attach"
|
||||
PASS_REGULAR_EXPRESSION
|
||||
"Outputting.*(perfetto-trace.proto).*Outputting.*(wall_clock.txt)"
|
||||
FAIL_REGULAR_EXPRESSION
|
||||
"Dyninst was unable to attach to the specified process")
|
||||
@@ -0,0 +1,180 @@
|
||||
# -------------------------------------------------------------------------------------- #
|
||||
#
|
||||
# causal profiling tests
|
||||
#
|
||||
# -------------------------------------------------------------------------------------- #
|
||||
|
||||
omnitrace_add_causal_test(
|
||||
NAME cpu-omni-func
|
||||
TARGET causal-cpu-omni
|
||||
RUN_ARGS 70 10 432525 1000000000
|
||||
CAUSAL_MODE "function"
|
||||
CAUSAL_PASS_REGEX
|
||||
"Starting causal experiment #1(.*)causal/experiments.json(.*)causal/experiments.coz"
|
||||
)
|
||||
|
||||
omnitrace_add_causal_test(
|
||||
SKIP_BASELINE
|
||||
NAME cpu-omni-func-ndebug
|
||||
TARGET causal-cpu-omni-ndebug
|
||||
RUN_ARGS 70 10 432525 1000000000
|
||||
CAUSAL_MODE "function"
|
||||
CAUSAL_PASS_REGEX
|
||||
"Starting causal experiment #1(.*)causal/experiments.json(.*)causal/experiments.coz"
|
||||
)
|
||||
|
||||
omnitrace_add_causal_test(
|
||||
SKIP_BASELINE
|
||||
NAME cpu-omni-line
|
||||
TARGET causal-cpu-omni
|
||||
RUN_ARGS 70 10 432525 1000000000
|
||||
CAUSAL_MODE "line"
|
||||
CAUSAL_PASS_REGEX
|
||||
"Starting causal experiment #1(.*)causal/experiments.json(.*)causal/experiments.coz"
|
||||
)
|
||||
|
||||
omnitrace_add_causal_test(
|
||||
NAME both-omni-func
|
||||
TARGET causal-both-omni
|
||||
RUN_ARGS 70 10 432525 400000000
|
||||
CAUSAL_MODE "function"
|
||||
CAUSAL_ARGS
|
||||
-n
|
||||
2
|
||||
-w
|
||||
1
|
||||
-d
|
||||
3
|
||||
--monochrome
|
||||
-g
|
||||
${CMAKE_BINARY_DIR}/omnitrace-tests-config/causal-both-omni-func
|
||||
-l
|
||||
causal-both-omni
|
||||
-v
|
||||
3
|
||||
-b
|
||||
timer
|
||||
ENVIRONMENT "OMNITRACE_STRICT_CONFIG=OFF"
|
||||
CAUSAL_PASS_REGEX
|
||||
"Starting causal experiment #1(.*)causal/experiments.json(.*)causal/experiments.coz"
|
||||
)
|
||||
|
||||
omnitrace_add_causal_test(
|
||||
NAME lulesh-func
|
||||
TARGET lulesh-omni
|
||||
RUN_ARGS -i 35 -s 50 -p
|
||||
CAUSAL_MODE "function"
|
||||
CAUSAL_ARGS -s 0,10,25,50,75
|
||||
CAUSAL_PASS_REGEX
|
||||
"Starting causal experiment #1(.*)causal/experiments.json(.*)causal/experiments.coz"
|
||||
)
|
||||
|
||||
omnitrace_add_causal_test(
|
||||
SKIP_BASELINE
|
||||
NAME lulesh-func-ndebug
|
||||
TARGET lulesh-omni-ndebug
|
||||
RUN_ARGS -i 35 -s 50 -p
|
||||
CAUSAL_MODE "function"
|
||||
CAUSAL_ARGS -s 0,10,25,50,75
|
||||
CAUSAL_PASS_REGEX
|
||||
"Starting causal experiment #1(.*)causal/experiments.json(.*)causal/experiments.coz"
|
||||
)
|
||||
|
||||
omnitrace_add_causal_test(
|
||||
SKIP_BASELINE
|
||||
NAME lulesh-line
|
||||
TARGET lulesh-omni
|
||||
RUN_ARGS -i 35 -s 50 -p
|
||||
CAUSAL_MODE "line"
|
||||
CAUSAL_ARGS -s 0,10,25,50,75 -S lulesh.cc
|
||||
CAUSAL_PASS_REGEX
|
||||
"Starting causal experiment #1(.*)causal/experiments.json(.*)causal/experiments.coz"
|
||||
)
|
||||
|
||||
# set(_causal_e2e_exe_args 80 100 432525 100000000) set(_causal_e2e_exe_args 80 12 432525
|
||||
# 500000000)
|
||||
set(_causal_e2e_exe_args 80 50 432525 100000000)
|
||||
set(_causal_common_args
|
||||
"-n 5 -e -s 0 10 20 30 -B $<TARGET_FILE_BASE_NAME:causal-cpu-omni>")
|
||||
|
||||
macro(
|
||||
causal_e2e_args_and_validation
|
||||
_NAME
|
||||
_TEST
|
||||
_MODE
|
||||
_EXPER
|
||||
_V10 # expected value for virtual speedup of 15
|
||||
_V20
|
||||
_V30
|
||||
_TOL # tolerance for virtual speedup
|
||||
)
|
||||
# arguments to omnitrace-causal
|
||||
set(${_NAME}_args "${_causal_common_args} ${_MODE} ${_EXPER}")
|
||||
|
||||
# arguments to validate-causal-json.py
|
||||
set(${_NAME}_valid
|
||||
"-n 0 -i omnitrace-tests-output/causal-cpu-omni-${_TEST}-e2e/causal/experiments.json -v ${_EXPER} $<TARGET_FILE_BASE_NAME:causal-cpu-omni> 10 ${_V10} ${_TOL} ${_EXPER} $<TARGET_FILE_BASE_NAME:causal-cpu-omni> 20 ${_V20} ${_TOL} ${_EXPER} $<TARGET_FILE_BASE_NAME:causal-cpu-omni> 30 ${_V30} ${_TOL}"
|
||||
)
|
||||
|
||||
# patch string for command-line
|
||||
string(REPLACE " " ";" ${_NAME}_args "${${_NAME}_args}")
|
||||
string(REPLACE " " ";" ${_NAME}_valid "${${_NAME}_valid}")
|
||||
endmacro()
|
||||
|
||||
causal_e2e_args_and_validation(_causal_slow_func slow-func "-F" "cpu_slow_func" 10 20 20
|
||||
5)
|
||||
causal_e2e_args_and_validation(_causal_fast_func fast-func "-F" "cpu_fast_func" 0 0 0 5)
|
||||
causal_e2e_args_and_validation(_causal_line_100 line-100 "-S" "causal.cpp:100" 10 20 20 5)
|
||||
causal_e2e_args_and_validation(_causal_line_110 line-110 "-S" "causal.cpp:110" 0 0 0 5)
|
||||
|
||||
omnitrace_add_causal_test(
|
||||
SKIP_BASELINE
|
||||
NAME cpu-omni-slow-func-e2e
|
||||
TARGET causal-cpu-omni
|
||||
LABELS "causal-e2e"
|
||||
RUN_ARGS ${_causal_e2e_exe_args}
|
||||
CAUSAL_MODE "func"
|
||||
CAUSAL_ARGS ${_causal_slow_func_args}
|
||||
CAUSAL_VALIDATE_ARGS ${_causal_slow_func_valid}
|
||||
CAUSAL_PASS_REGEX
|
||||
"Starting causal experiment #1(.*)causal/experiments.json(.*)causal/experiments.coz"
|
||||
PROPERTIES PROCESSORS 2 PROCESSOR_AFFINITY OFF)
|
||||
|
||||
omnitrace_add_causal_test(
|
||||
SKIP_BASELINE
|
||||
NAME cpu-omni-fast-func-e2e
|
||||
TARGET causal-cpu-omni
|
||||
LABELS "causal-e2e"
|
||||
RUN_ARGS ${_causal_e2e_exe_args}
|
||||
CAUSAL_MODE "func"
|
||||
CAUSAL_ARGS ${_causal_fast_func_args}
|
||||
CAUSAL_VALIDATE_ARGS ${_causal_fast_func_valid}
|
||||
CAUSAL_PASS_REGEX
|
||||
"Starting causal experiment #1(.*)causal/experiments.json(.*)causal/experiments.coz"
|
||||
PROPERTIES PROCESSORS 2 PROCESSOR_AFFINITY OFF)
|
||||
|
||||
omnitrace_add_causal_test(
|
||||
SKIP_BASELINE
|
||||
NAME cpu-omni-line-100-e2e
|
||||
TARGET causal-cpu-omni
|
||||
LABELS "causal-e2e"
|
||||
RUN_ARGS ${_causal_e2e_exe_args}
|
||||
CAUSAL_MODE "line"
|
||||
CAUSAL_ARGS ${_causal_line_100_args}
|
||||
CAUSAL_VALIDATE_ARGS ${_causal_line_100_valid}
|
||||
CAUSAL_PASS_REGEX
|
||||
"Starting causal experiment #1(.*)causal/experiments.json(.*)causal/experiments.coz"
|
||||
PROPERTIES PROCESSORS 2 PROCESSOR_AFFINITY OFF)
|
||||
|
||||
omnitrace_add_causal_test(
|
||||
SKIP_BASELINE
|
||||
NAME cpu-omni-line-110-e2e
|
||||
TARGET causal-cpu-omni
|
||||
LABELS "causal-e2e"
|
||||
RUN_ARGS ${_causal_e2e_exe_args}
|
||||
CAUSAL_MODE "line"
|
||||
CAUSAL_ARGS ${_causal_line_110_args}
|
||||
CAUSAL_VALIDATE_ARGS ${_causal_line_110_valid}
|
||||
CAUSAL_PASS_REGEX
|
||||
"Starting causal experiment #1(.*)causal/experiments.json(.*)causal/experiments.coz"
|
||||
PROPERTIES PROCESSORS 2 PROCESSOR_AFFINITY OFF)
|
||||
@@ -0,0 +1,137 @@
|
||||
# -------------------------------------------------------------------------------------- #
|
||||
#
|
||||
# code-coverage tests
|
||||
#
|
||||
# -------------------------------------------------------------------------------------- #
|
||||
|
||||
omnitrace_add_test(
|
||||
SKIP_BASELINE SKIP_SAMPLING
|
||||
NAME code-coverage
|
||||
TARGET code-coverage
|
||||
REWRITE_ARGS
|
||||
-e
|
||||
-v
|
||||
2
|
||||
--min-instructions=4
|
||||
-E
|
||||
^std::
|
||||
-M
|
||||
coverage
|
||||
--coverage
|
||||
function
|
||||
RUNTIME_ARGS
|
||||
-e
|
||||
-v
|
||||
1
|
||||
--min-instructions=4
|
||||
-E
|
||||
^std::
|
||||
--label
|
||||
file
|
||||
line
|
||||
return
|
||||
args
|
||||
-M
|
||||
coverage
|
||||
--coverage
|
||||
function
|
||||
--module-restrict
|
||||
code.coverage
|
||||
LABELS "coverage;function-coverage"
|
||||
RUN_ARGS 10 ${NUM_THREADS} 1000
|
||||
ENVIRONMENT "${_base_environment}"
|
||||
RUNTIME_PASS_REGEX "(\\\[[0-9]+\\\]) code coverage :: 66.67%"
|
||||
REWRITE_RUN_PASS_REGEX "(\\\[[0-9]+\\\]) code coverage :: 66.67%")
|
||||
|
||||
omnitrace_add_test(
|
||||
SKIP_BASELINE SKIP_SAMPLING
|
||||
NAME code-coverage-hybrid
|
||||
TARGET code-coverage
|
||||
REWRITE_ARGS -e -v 2 --min-instructions=4 -E ^std:: --coverage function
|
||||
RUNTIME_ARGS
|
||||
-e
|
||||
-v
|
||||
1
|
||||
--min-instructions=4
|
||||
-E
|
||||
^std::
|
||||
--label
|
||||
file
|
||||
line
|
||||
return
|
||||
args
|
||||
--coverage
|
||||
function
|
||||
--module-restrict
|
||||
code.coverage
|
||||
LABELS "coverage;function-coverage;hybrid-coverage"
|
||||
RUN_ARGS 10 ${NUM_THREADS} 1000
|
||||
ENVIRONMENT "${_base_environment}"
|
||||
RUNTIME_PASS_REGEX "(\\\[[0-9]+\\\]) code coverage :: 66.67%"
|
||||
REWRITE_RUN_PASS_REGEX "(\\\[[0-9]+\\\]) code coverage :: 66.67%")
|
||||
|
||||
omnitrace_add_test(
|
||||
SKIP_BASELINE SKIP_SAMPLING
|
||||
NAME code-coverage-basic-blocks
|
||||
TARGET code-coverage
|
||||
REWRITE_ARGS
|
||||
-e
|
||||
-v
|
||||
2
|
||||
--min-instructions=4
|
||||
-E
|
||||
^std::
|
||||
-M
|
||||
coverage
|
||||
--coverage
|
||||
basic_block
|
||||
RUNTIME_ARGS
|
||||
-e
|
||||
-v
|
||||
1
|
||||
--min-instructions=4
|
||||
-E
|
||||
^std::
|
||||
--label
|
||||
file
|
||||
line
|
||||
return
|
||||
args
|
||||
-M
|
||||
coverage
|
||||
--coverage
|
||||
basic_block
|
||||
--module-restrict
|
||||
code.coverage
|
||||
LABELS "coverage;bb-coverage"
|
||||
RUN_ARGS 10 ${NUM_THREADS} 1000
|
||||
ENVIRONMENT "${_base_environment}"
|
||||
RUNTIME_PASS_REGEX "(\\\[[0-9]+\\\]) function coverage :: 66.67%"
|
||||
REWRITE_RUN_PASS_REGEX "(\\\[[0-9]+\\\]) function coverage :: 66.67%")
|
||||
|
||||
omnitrace_add_test(
|
||||
SKIP_BASELINE SKIP_SAMPLING
|
||||
NAME code-coverage-basic-blocks-hybrid
|
||||
TARGET code-coverage
|
||||
REWRITE_ARGS -e -v 2 --min-instructions=4 -E ^std:: --coverage basic_block
|
||||
RUNTIME_ARGS
|
||||
-e
|
||||
-v
|
||||
1
|
||||
--min-instructions=4
|
||||
-E
|
||||
^std::
|
||||
--label
|
||||
file
|
||||
line
|
||||
return
|
||||
args
|
||||
--coverage
|
||||
basic_block
|
||||
--module-restrict
|
||||
code.coverage
|
||||
LABELS "coverage;bb-coverage;hybrid-coverage"
|
||||
RUN_ARGS 10 ${NUM_THREADS} 1000
|
||||
ENVIRONMENT "${_base_environment}"
|
||||
RUNTIME_PASS_REGEX "(\\\[[0-9]+\\\]) function coverage :: 66.67%"
|
||||
REWRITE_RUN_PASS_REGEX "(\\\[[0-9]+\\\]) function coverage :: 66.67%")
|
||||
@@ -0,0 +1,40 @@
|
||||
# -------------------------------------------------------------------------------------- #
|
||||
#
|
||||
# general config file tests
|
||||
#
|
||||
# -------------------------------------------------------------------------------------- #
|
||||
|
||||
file(
|
||||
WRITE ${CMAKE_CURRENT_BINARY_DIR}/invalid.cfg
|
||||
"
|
||||
OMNITRACE_CONFIG_FILE =
|
||||
FOOBAR = ON
|
||||
")
|
||||
|
||||
if(TARGET parallel-overhead)
|
||||
set(_CONFIG_TEST_EXE $<TARGET_FILE:parallel-overhead>)
|
||||
else()
|
||||
set(_CONFIG_TEST_EXE ls)
|
||||
endif()
|
||||
|
||||
add_test(
|
||||
NAME omnitrace-invalid-config
|
||||
COMMAND $<TARGET_FILE:omnitrace-instrument> -- ${_CONFIG_TEST_EXE}
|
||||
WORKING_DIRECTORY ${PROJECT_BINARY_DIR})
|
||||
|
||||
set_tests_properties(
|
||||
omnitrace-invalid-config
|
||||
PROPERTIES ENVIRONMENT
|
||||
"OMNITRACE_CONFIG_FILE=${CMAKE_CURRENT_BINARY_DIR}/invalid.cfg" TIMEOUT
|
||||
120 LABELS "config" WILL_FAIL ON)
|
||||
|
||||
add_test(
|
||||
NAME omnitrace-missing-config
|
||||
COMMAND $<TARGET_FILE:omnitrace-instrument> -- ${_CONFIG_TEST_EXE}
|
||||
WORKING_DIRECTORY ${PROJECT_BINARY_DIR})
|
||||
|
||||
set_tests_properties(
|
||||
omnitrace-missing-config
|
||||
PROPERTIES ENVIRONMENT
|
||||
"OMNITRACE_CONFIG_FILE=${CMAKE_CURRENT_BINARY_DIR}/missing.cfg" TIMEOUT
|
||||
120 LABELS "config" WILL_FAIL ON)
|
||||
@@ -0,0 +1,52 @@
|
||||
# -------------------------------------------------------------------------------------- #
|
||||
#
|
||||
# critical-trace tests
|
||||
#
|
||||
# -------------------------------------------------------------------------------------- #
|
||||
|
||||
omnitrace_add_test(
|
||||
SKIP_BASELINE SKIP_RUNTIME SKIP_SAMPLING
|
||||
NAME parallel-overhead-critical-trace
|
||||
TARGET parallel-overhead
|
||||
LABELS "critical-trace"
|
||||
REWRITE_ARGS
|
||||
-e
|
||||
-i
|
||||
8
|
||||
-E
|
||||
"^fib"
|
||||
-v
|
||||
2
|
||||
--print-instrumented
|
||||
functions
|
||||
RUN_ARGS 10 4 100
|
||||
ENVIRONMENT "${_critical_trace_environment}")
|
||||
|
||||
add_test(
|
||||
NAME parallel-overhead-process-critical-trace
|
||||
COMMAND
|
||||
$<TARGET_FILE:omnitrace-critical-trace>
|
||||
${PROJECT_BINARY_DIR}/omnitrace-tests-output/parallel-overhead-critical-trace-binary-rewrite/call-chain.json
|
||||
WORKING_DIRECTORY ${PROJECT_BINARY_DIR})
|
||||
|
||||
set(_parallel_overhead_critical_trace_environ
|
||||
"OMNITRACE_OUTPUT_PATH=omnitrace-tests-output"
|
||||
"OMNITRACE_OUTPUT_PREFIX=parallel-overhead-critical-trace/"
|
||||
"OMNITRACE_CRITICAL_TRACE_DEBUG=ON"
|
||||
"OMNITRACE_VERBOSE=4"
|
||||
"OMNITRACE_USE_PID=OFF"
|
||||
"OMNITRACE_TIME_OUTPUT=OFF")
|
||||
|
||||
set_tests_properties(
|
||||
parallel-overhead-process-critical-trace
|
||||
PROPERTIES
|
||||
ENVIRONMENT
|
||||
"${_parallel_overhead_critical_trace_environ}"
|
||||
TIMEOUT
|
||||
300
|
||||
LABELS
|
||||
"parallel-overhead;critical-trace"
|
||||
PASS_REGULAR_EXPRESSION
|
||||
"Outputting.*(critical-trace-cpu.json).*Outputting.*(critical-trace-any.json)"
|
||||
DEPENDS
|
||||
parallel-overhead-critical-trace-binary-rewrite-run)
|
||||
@@ -0,0 +1,23 @@
|
||||
# -------------------------------------------------------------------------------------- #
|
||||
#
|
||||
# fork tests
|
||||
#
|
||||
# -------------------------------------------------------------------------------------- #
|
||||
|
||||
omnitrace_add_test(
|
||||
NAME fork
|
||||
TARGET fork-example
|
||||
REWRITE_ARGS -e -v 2 --print-instrumented modules -i 16
|
||||
RUNTIME_ARGS -e -v 1 --label file -i 16
|
||||
ENVIRONMENT
|
||||
"${_base_environment};OMNITRACE_CRITICAL_TRACE=ON;OMNITRACE_SAMPLING_FREQ=250;OMNITRACE_SAMPLING_REALTIME=ON"
|
||||
SAMPLING_PASS_REGEX "fork.. called on PID"
|
||||
RUNTIME_PASS_REGEX "fork.. called on PID"
|
||||
REWRITE_RUN_PASS_REGEX "fork.. called on PID"
|
||||
SAMPLING_FAIL_REGEX
|
||||
"(terminate called after throwing an instance|calling abort.. in |Exit code: [1-9])"
|
||||
RUNTIME_FAIL_REGEX
|
||||
"(terminate called after throwing an instance|calling abort.. in |Exit code: [1-9])"
|
||||
REWRITE_RUN_FAIL_REGEX
|
||||
"(terminate called after throwing an instance|calling abort.. in |Exit code: [1-9])"
|
||||
)
|
||||
@@ -0,0 +1,54 @@
|
||||
# -------------------------------------------------------------------------------------- #
|
||||
#
|
||||
# binary-rewrite and runtime-instrumentation tests
|
||||
#
|
||||
# -------------------------------------------------------------------------------------- #
|
||||
|
||||
omnitrace_add_test(
|
||||
SKIP_SAMPLING SKIP_RUNTIME
|
||||
NAME rewrite-caller
|
||||
TARGET rewrite-caller
|
||||
LABELS "caller-include"
|
||||
REWRITE_ARGS
|
||||
-e
|
||||
-i
|
||||
256
|
||||
--caller-include
|
||||
"^inner"
|
||||
-v
|
||||
2
|
||||
--print-instrumented
|
||||
functions
|
||||
RUN_ARGS 17
|
||||
ENVIRONMENT "${_base_environment};OMNITRACE_COUT_OUTPUT=ON"
|
||||
BASELINE_PASS_REGEX "number of calls made = 17"
|
||||
REWRITE_PASS_REGEX "\\[function\\]\\[Forcing\\] caller-include-regex :: 'outer'"
|
||||
REWRITE_RUN_PASS_REGEX ">>> ._outer ([ \\|]+) 17")
|
||||
|
||||
omnitrace_add_test(
|
||||
NAME parallel-overhead
|
||||
TARGET parallel-overhead
|
||||
REWRITE_ARGS -e -v 2 --min-instructions=8
|
||||
RUNTIME_ARGS
|
||||
-e
|
||||
-v
|
||||
1
|
||||
--min-instructions=8
|
||||
--label
|
||||
file
|
||||
line
|
||||
return
|
||||
args
|
||||
RUN_ARGS 10 ${NUM_THREADS} 1000
|
||||
ENVIRONMENT "${_base_environment};OMNITRACE_CRITICAL_TRACE=OFF")
|
||||
|
||||
omnitrace_add_test(
|
||||
SKIP_BASELINE SKIP_RUNTIME
|
||||
NAME parallel-overhead-locks-perfetto
|
||||
TARGET parallel-overhead-locks
|
||||
LABELS "locks"
|
||||
REWRITE_ARGS -e -v 2 --min-instructions=8
|
||||
RUN_ARGS 10 4 1000
|
||||
ENVIRONMENT
|
||||
"${_lock_environment};OMNITRACE_FLAT_PROFILE=ON;OMNITRACE_USE_TIMEMORY=OFF;OMNITRACE_USE_PERFETTO=ON;OMNITRACE_SAMPLING_KEEP_INTERNAL=OFF"
|
||||
)
|
||||
@@ -0,0 +1,128 @@
|
||||
# -------------------------------------------------------------------------------------- #
|
||||
#
|
||||
# kokkos (lulesh) tests
|
||||
#
|
||||
# -------------------------------------------------------------------------------------- #
|
||||
|
||||
omnitrace_add_test(
|
||||
NAME lulesh
|
||||
TARGET lulesh
|
||||
MPI ${LULESH_USE_MPI}
|
||||
GPU ${LULESH_USE_GPU}
|
||||
NUM_PROCS 8
|
||||
LABELS "kokkos"
|
||||
REWRITE_ARGS -e -v 2 --label file line return args
|
||||
RUNTIME_ARGS
|
||||
-e
|
||||
-v
|
||||
1
|
||||
--label
|
||||
file
|
||||
line
|
||||
return
|
||||
args
|
||||
-ME
|
||||
[==[lib(gomp|m-)]==]
|
||||
LABELS "kokkos;kokkos-profile-library"
|
||||
RUN_ARGS -i 25 -s 20 -p
|
||||
ENVIRONMENT
|
||||
"${_base_environment};OMNITRACE_CRITICAL_TRACE=OFF;OMNITRACE_USE_KOKKOSP=ON;OMNITRACE_COUT_OUTPUT=ON;OMNITRACE_SAMPLING_FREQ=50;OMNITRACE_KOKKOSP_PREFIX=[kokkos];KOKKOS_PROFILE_LIBRARY=libomnitrace-dl.so"
|
||||
REWRITE_RUN_PASS_REGEX "\\|_\\[kokkos\\] [a-zA-Z]"
|
||||
RUNTIME_PASS_REGEX "\\|_\\[kokkos\\] [a-zA-Z]")
|
||||
|
||||
omnitrace_add_test(
|
||||
SKIP_RUNTIME SKIP_REWRITE
|
||||
NAME lulesh-baseline-kokkosp-libomnitrace
|
||||
TARGET lulesh
|
||||
MPI ${LULESH_USE_MPI}
|
||||
GPU ${LULESH_USE_GPU}
|
||||
NUM_PROCS 8
|
||||
LABELS "kokkos;kokkos-profile-library"
|
||||
RUN_ARGS -i 10 -s 20 -p
|
||||
ENVIRONMENT
|
||||
"${_base_environment};OMNITRACE_CRITICAL_TRACE=OFF;OMNITRACE_USE_KOKKOSP=ON;OMNITRACE_COUT_OUTPUT=ON;OMNITRACE_SAMPLING_FREQ=50;OMNITRACE_KOKKOSP_PREFIX=[kokkos];KOKKOS_PROFILE_LIBRARY=libomnitrace.so"
|
||||
BASELINE_PASS_REGEX "\\|_\\[kokkos\\] [a-zA-Z]")
|
||||
|
||||
omnitrace_add_test(
|
||||
SKIP_RUNTIME SKIP_REWRITE
|
||||
NAME lulesh-baseline-kokkosp-libomnitrace-dl
|
||||
TARGET lulesh
|
||||
MPI ${LULESH_USE_MPI}
|
||||
GPU ${LULESH_USE_GPU}
|
||||
NUM_PROCS 8
|
||||
LABELS "kokkos;kokkos-profile-library"
|
||||
RUN_ARGS -i 10 -s 20 -p
|
||||
ENVIRONMENT
|
||||
"${_base_environment};OMNITRACE_CRITICAL_TRACE=OFF;OMNITRACE_USE_KOKKOSP=ON;OMNITRACE_COUT_OUTPUT=ON;OMNITRACE_SAMPLING_FREQ=50;OMNITRACE_KOKKOSP_PREFIX=[kokkos];KOKKOS_PROFILE_LIBRARY=libomnitrace-dl.so"
|
||||
BASELINE_PASS_REGEX "\\|_\\[kokkos\\] [a-zA-Z]")
|
||||
|
||||
omnitrace_add_test(
|
||||
SKIP_BASELINE
|
||||
NAME lulesh-kokkosp
|
||||
TARGET lulesh
|
||||
MPI ${LULESH_USE_MPI}
|
||||
GPU ${LULESH_USE_GPU}
|
||||
NUM_PROCS 8
|
||||
LABELS "kokkos"
|
||||
REWRITE_ARGS -e -v 2
|
||||
RUNTIME_ARGS
|
||||
-e
|
||||
-v
|
||||
1
|
||||
--label
|
||||
file
|
||||
line
|
||||
return
|
||||
args
|
||||
-ME
|
||||
[==[lib(gomp|m-)]==]
|
||||
RUN_ARGS -i 10 -s 20 -p
|
||||
ENVIRONMENT
|
||||
"${_base_environment};OMNITRACE_CRITICAL_TRACE=OFF;OMNITRACE_USE_KOKKOSP=ON")
|
||||
|
||||
omnitrace_add_test(
|
||||
SKIP_BASELINE
|
||||
NAME lulesh-perfetto
|
||||
TARGET lulesh
|
||||
MPI ${LULESH_USE_MPI}
|
||||
GPU ${LULESH_USE_GPU}
|
||||
NUM_PROCS 8
|
||||
LABELS "kokkos;loops"
|
||||
REWRITE_ARGS -e -v 2
|
||||
RUNTIME_ARGS
|
||||
-e
|
||||
-v
|
||||
1
|
||||
-l
|
||||
--dynamic-callsites
|
||||
--traps
|
||||
--allow-overlapping
|
||||
-ME
|
||||
[==[libgomp]==]
|
||||
RUN_ARGS -i 10 -s 20 -p
|
||||
ENVIRONMENT
|
||||
"${_perfetto_environment};OMNITRACE_CRITICAL_TRACE=OFF;OMNITRACE_USE_KOKKOSP=OFF")
|
||||
|
||||
omnitrace_add_test(
|
||||
NAME lulesh-timemory
|
||||
TARGET lulesh
|
||||
MPI ${LULESH_USE_MPI}
|
||||
GPU ${LULESH_USE_GPU}
|
||||
NUM_PROCS 8
|
||||
LABELS "kokkos;loops"
|
||||
REWRITE_ARGS -e -v 2 -l --dynamic-callsites --traps --allow-overlapping
|
||||
RUNTIME_ARGS
|
||||
-e
|
||||
-v
|
||||
1
|
||||
-l
|
||||
--dynamic-callsites
|
||||
-ME
|
||||
[==[libgomp]==]
|
||||
-d
|
||||
wall_clock
|
||||
peak_rss
|
||||
RUN_ARGS -i 10 -s 20 -p
|
||||
ENVIRONMENT
|
||||
"${_timemory_environment};OMNITRACE_CRITICAL_TRACE=OFF;OMNITRACE_USE_KOKKOSP=OFF"
|
||||
REWRITE_FAIL_REGEX "0 instrumented loops in procedure")
|
||||
@@ -0,0 +1,122 @@
|
||||
# -------------------------------------------------------------------------------------- #
|
||||
#
|
||||
# MPI tests
|
||||
#
|
||||
# -------------------------------------------------------------------------------------- #
|
||||
|
||||
if(NOT OMNITRACE_USE_MPI AND NOT OMNITRACE_USE_MPI_HEADERS)
|
||||
return()
|
||||
endif()
|
||||
|
||||
omnitrace_add_test(
|
||||
SKIP_RUNTIME
|
||||
NAME "mpi"
|
||||
TARGET mpi-example
|
||||
MPI ON
|
||||
NUM_PROCS 4
|
||||
REWRITE_ARGS
|
||||
-e
|
||||
-v
|
||||
2
|
||||
--label
|
||||
file
|
||||
line
|
||||
return
|
||||
args
|
||||
--min-instructions
|
||||
0
|
||||
ENVIRONMENT "${_base_environment};OMNITRACE_VERBOSE=1"
|
||||
REWRITE_RUN_PASS_REGEX
|
||||
"(/[A-Za-z-]+/perfetto-trace-0.proto).*(/[A-Za-z-]+/wall_clock-0.txt')"
|
||||
REWRITE_RUN_FAIL_REGEX
|
||||
"(perfetto-trace|trip_count|sampling_percent|sampling_cpu_clock|sampling_wall_clock|wall_clock)-[0-9][0-9]+.(json|txt|proto)"
|
||||
)
|
||||
|
||||
omnitrace_add_test(
|
||||
SKIP_RUNTIME
|
||||
NAME "mpi-flat-mpip"
|
||||
TARGET mpi-example
|
||||
MPI ON
|
||||
NUM_PROCS 4
|
||||
LABELS "mpip"
|
||||
REWRITE_ARGS
|
||||
-e
|
||||
-v
|
||||
2
|
||||
--label
|
||||
file
|
||||
line
|
||||
args
|
||||
--min-instructions
|
||||
0
|
||||
ENVIRONMENT
|
||||
"${_flat_environment};OMNITRACE_USE_SAMPLING=OFF;OMNITRACE_STRICT_CONFIG=OFF;OMNITRACE_USE_MPIP=ON"
|
||||
REWRITE_RUN_PASS_REGEX
|
||||
">>> mpi-flat-mpip.inst(.*\n.*)>>> MPI_Init_thread(.*\n.*)>>> pthread_create(.*\n.*)>>> MPI_Comm_size(.*\n.*)>>> MPI_Comm_rank(.*\n.*)>>> MPI_Barrier(.*\n.*)>>> MPI_Alltoall"
|
||||
)
|
||||
|
||||
omnitrace_add_test(
|
||||
SKIP_RUNTIME
|
||||
NAME "mpi-flat"
|
||||
TARGET mpi-example
|
||||
MPI ON
|
||||
NUM_PROCS 4
|
||||
LABELS "mpip"
|
||||
REWRITE_ARGS
|
||||
-e
|
||||
-v
|
||||
2
|
||||
--label
|
||||
file
|
||||
line
|
||||
args
|
||||
--min-instructions
|
||||
0
|
||||
ENVIRONMENT "${_flat_environment};OMNITRACE_USE_SAMPLING=OFF"
|
||||
REWRITE_RUN_PASS_REGEX
|
||||
">>> mpi-flat.inst(.*\n.*)>>> MPI_Init_thread(.*\n.*)>>> pthread_create(.*\n.*)>>> MPI_Comm_size(.*\n.*)>>> MPI_Comm_rank(.*\n.*)>>> MPI_Barrier(.*\n.*)>>> MPI_Alltoall"
|
||||
)
|
||||
|
||||
set(_mpip_environment
|
||||
"OMNITRACE_USE_PERFETTO=ON"
|
||||
"OMNITRACE_USE_TIMEMORY=ON"
|
||||
"OMNITRACE_USE_SAMPLING=OFF"
|
||||
"OMNITRACE_USE_PROCESS_SAMPLING=OFF"
|
||||
"OMNITRACE_TIME_OUTPUT=OFF"
|
||||
"OMNITRACE_FILE_OUTPUT=ON"
|
||||
"OMNITRACE_USE_MPIP=ON"
|
||||
"OMNITRACE_DEBUG=OFF"
|
||||
"OMNITRACE_VERBOSE=2"
|
||||
"OMNITRACE_DL_VERBOSE=2"
|
||||
"${_test_openmp_env}"
|
||||
"${_test_library_path}")
|
||||
|
||||
set(_mpip_all2all_environment
|
||||
"OMNITRACE_USE_PERFETTO=ON"
|
||||
"OMNITRACE_USE_TIMEMORY=ON"
|
||||
"OMNITRACE_USE_SAMPLING=OFF"
|
||||
"OMNITRACE_USE_PROCESS_SAMPLING=OFF"
|
||||
"OMNITRACE_TIME_OUTPUT=OFF"
|
||||
"OMNITRACE_FILE_OUTPUT=ON"
|
||||
"OMNITRACE_USE_MPIP=ON"
|
||||
"OMNITRACE_DEBUG=ON"
|
||||
"OMNITRACE_VERBOSE=3"
|
||||
"OMNITRACE_DL_VERBOSE=3"
|
||||
"${_test_openmp_env}"
|
||||
"${_test_library_path}")
|
||||
|
||||
foreach(_EXAMPLE all2all allgather allreduce bcast reduce scatter-gather send-recv)
|
||||
if("${_mpip_${_EXAMPLE}_environment}" STREQUAL "")
|
||||
set(_mpip_${_EXAMPLE}_environment "${_mpip_environment}")
|
||||
endif()
|
||||
omnitrace_add_test(
|
||||
SKIP_RUNTIME SKIP_SAMPLING
|
||||
NAME "mpi-${_EXAMPLE}"
|
||||
TARGET mpi-${_EXAMPLE}
|
||||
MPI ON
|
||||
NUM_PROCS 2
|
||||
LABELS "mpip"
|
||||
REWRITE_ARGS -e -v 2 --label file line --min-instructions 0
|
||||
RUN_ARGS 30
|
||||
ENVIRONMENT "${_mpip_${_EXAMPLE}_environment}")
|
||||
endforeach()
|
||||
@@ -0,0 +1,99 @@
|
||||
# -------------------------------------------------------------------------------------- #
|
||||
#
|
||||
# openmp tests
|
||||
#
|
||||
# -------------------------------------------------------------------------------------- #
|
||||
|
||||
if(OMNITRACE_OPENMP_USING_LIBOMP_LIBRARY AND OMNITRACE_USE_OMPT)
|
||||
set(_OMPT_PASS_REGEX "\\|_ompt_")
|
||||
else()
|
||||
set(_OMPT_PASS_REGEX "")
|
||||
endif()
|
||||
|
||||
omnitrace_add_test(
|
||||
NAME openmp-cg
|
||||
TARGET openmp-cg
|
||||
LABELS "openmp"
|
||||
REWRITE_ARGS -e -v 2 --instrument-loops
|
||||
RUNTIME_ARGS -e -v 1 --label return args
|
||||
REWRITE_TIMEOUT 180
|
||||
RUNTIME_TIMEOUT 360
|
||||
ENVIRONMENT "${_ompt_environment};OMNITRACE_USE_SAMPLING=OFF;OMNITRACE_COUT_OUTPUT=ON"
|
||||
REWRITE_RUN_PASS_REGEX "${_OMPT_PASS_REGEX}"
|
||||
RUNTIME_PASS_REGEX "${_OMPT_PASS_REGEX}"
|
||||
REWRITE_FAIL_REGEX "0 instrumented loops in procedure")
|
||||
|
||||
omnitrace_add_test(
|
||||
SKIP_RUNTIME
|
||||
NAME openmp-lu
|
||||
TARGET openmp-lu
|
||||
LABELS "openmp"
|
||||
REWRITE_ARGS -e -v 2 --instrument-loops
|
||||
RUNTIME_ARGS -e -v 1 --label return args -E ^GOMP
|
||||
REWRITE_TIMEOUT 180
|
||||
RUNTIME_TIMEOUT 360
|
||||
ENVIRONMENT
|
||||
"${_ompt_environment};OMNITRACE_USE_SAMPLING=ON;OMNITRACE_SAMPLING_FREQ=50;OMNITRACE_COUT_OUTPUT=ON"
|
||||
REWRITE_RUN_PASS_REGEX "${_OMPT_PASS_REGEX}"
|
||||
REWRITE_FAIL_REGEX "0 instrumented loops in procedure")
|
||||
|
||||
set(_ompt_sampling_environ
|
||||
"${_ompt_environment}"
|
||||
"OMNITRACE_VERBOSE=2"
|
||||
"OMNITRACE_USE_OMPT=OFF"
|
||||
"OMNITRACE_USE_SAMPLING=ON"
|
||||
"OMNITRACE_USE_PROCESS_SAMPLING=OFF"
|
||||
"OMNITRACE_SAMPLING_FREQ=100"
|
||||
"OMNITRACE_SAMPLING_DELAY=0.1"
|
||||
"OMNITRACE_SAMPLING_DURATION=0.25"
|
||||
"OMNITRACE_SAMPLING_CPUTIME=ON"
|
||||
"OMNITRACE_SAMPLING_REALTIME=ON"
|
||||
"OMNITRACE_SAMPLING_CPUTIME_FREQ=1000"
|
||||
"OMNITRACE_SAMPLING_REALTIME_FREQ=500"
|
||||
"OMNITRACE_MONOCHROME=ON")
|
||||
|
||||
set(_ompt_sample_no_tmpfiles_environ
|
||||
"${_ompt_environment}"
|
||||
"OMNITRACE_VERBOSE=2"
|
||||
"OMNITRACE_USE_OMPT=OFF"
|
||||
"OMNITRACE_USE_SAMPLING=ON"
|
||||
"OMNITRACE_USE_PROCESS_SAMPLING=OFF"
|
||||
"OMNITRACE_SAMPLING_CPUTIME=ON"
|
||||
"OMNITRACE_SAMPLING_REALTIME=OFF"
|
||||
"OMNITRACE_SAMPLING_CPUTIME_FREQ=700"
|
||||
"OMNITRACE_USE_TEMPORARY_FILES=OFF"
|
||||
"OMNITRACE_MONOCHROME=ON")
|
||||
|
||||
set(_ompt_sampling_samp_regex
|
||||
"Sampler for thread 0 will be triggered 1000.0x per second of CPU-time(.*)Sampler for thread 0 will be triggered 500.0x per second of wall-time(.*)Sampling will be disabled after 0.250000 seconds(.*)Sampling duration of 0.250000 seconds has elapsed. Shutting down sampling"
|
||||
)
|
||||
set(_ompt_sampling_file_regex
|
||||
"sampling-duration-sampling/sampling_percent.(json|txt)(.*)sampling-duration-sampling/sampling_cpu_clock.(json|txt)(.*)sampling-duration-sampling/sampling_wall_clock.(json|txt)"
|
||||
)
|
||||
set(_notmp_sampling_file_regex
|
||||
"sampling-no-tmp-files-sampling/sampling_percent.(json|txt)(.*)sampling-no-tmp-files-sampling/sampling_cpu_clock.(json|txt)(.*)sampling-no-tmp-files-sampling/sampling_wall_clock.(json|txt)"
|
||||
)
|
||||
|
||||
omnitrace_add_test(
|
||||
SKIP_BASELINE SKIP_RUNTIME SKIP_REWRITE
|
||||
NAME openmp-cg-sampling-duration
|
||||
TARGET openmp-cg
|
||||
LABELS "openmp;sampling-duration"
|
||||
ENVIRONMENT "${_ompt_sampling_environ}"
|
||||
SAMPLING_PASS_REGEX "${_ompt_sampling_samp_regex}(.*)${_ompt_sampling_file_regex}")
|
||||
|
||||
omnitrace_add_test(
|
||||
SKIP_BASELINE SKIP_RUNTIME SKIP_REWRITE
|
||||
NAME openmp-lu-sampling-duration
|
||||
TARGET openmp-lu
|
||||
LABELS "openmp;sampling-duration"
|
||||
ENVIRONMENT "${_ompt_sampling_environ}"
|
||||
SAMPLING_PASS_REGEX "${_ompt_sampling_samp_regex}(.*)${_ompt_sampling_file_regex}")
|
||||
|
||||
omnitrace_add_test(
|
||||
SKIP_BASELINE SKIP_RUNTIME SKIP_REWRITE
|
||||
NAME openmp-cg-sampling-no-tmp-files
|
||||
TARGET openmp-cg
|
||||
LABELS "openmp;no-tmp-files"
|
||||
ENVIRONMENT "${_ompt_sample_no_tmpfiles_environ}"
|
||||
SAMPLING_PASS_REGEX "${_notmp_sampling_file_regex}")
|
||||
@@ -0,0 +1,34 @@
|
||||
# -------------------------------------------------------------------------------------- #
|
||||
#
|
||||
# binary-rewrite and runtime-instrumentation tests
|
||||
#
|
||||
# -------------------------------------------------------------------------------------- #
|
||||
|
||||
omnitrace_add_test(
|
||||
NAME parallel-overhead-locks
|
||||
TARGET parallel-overhead-locks
|
||||
LABELS "locks"
|
||||
REWRITE_ARGS -e -i 256
|
||||
RUNTIME_ARGS -e -i 256
|
||||
RUN_ARGS 30 4 1000
|
||||
ENVIRONMENT
|
||||
"${_lock_environment};OMNITRACE_USE_TIMEMORY=ON;OMNITRACE_USE_PERFETTO=ON;OMNITRACE_COLLAPSE_THREADS=OFF;OMNITRACE_SAMPLING_REALTIME=ON;OMNITRACE_SAMPLING_REALTIME_FREQ=10;OMNITRACE_SAMPLING_REALTIME_TIDS=0;OMNITRACE_SAMPLING_KEEP_INTERNAL=OFF"
|
||||
REWRITE_RUN_PASS_REGEX
|
||||
"wall_clock .*\\|_pthread_create .* 4 .*\\|_pthread_mutex_lock .* 1000 .*\\|_pthread_mutex_unlock .* 1000 .*\\|_pthread_mutex_lock .* 1000 .*\\|_pthread_mutex_unlock .* 1000 .*\\|_pthread_mutex_lock .* 1000 .*\\|_pthread_mutex_unlock .* 1000 .*\\|_pthread_mutex_lock .* 1000 .*\\|_pthread_mutex_unlock .* 1000"
|
||||
RUNTIME_PASS_REGEX
|
||||
"wall_clock .*\\|_pthread_create .* 4 .*\\|_pthread_mutex_lock .* 1000 .*\\|_pthread_mutex_unlock .* 1000 .*\\|_pthread_mutex_lock .* 1000 .*\\|_pthread_mutex_unlock .* 1000 .*\\|_pthread_mutex_lock .* 1000 .*\\|_pthread_mutex_unlock .* 1000 .*\\|_pthread_mutex_lock .* 1000 .*\\|_pthread_mutex_unlock .* 1000"
|
||||
)
|
||||
|
||||
omnitrace_add_test(
|
||||
SKIP_RUNTIME
|
||||
NAME parallel-overhead-locks-timemory
|
||||
TARGET parallel-overhead-locks
|
||||
LABELS "locks"
|
||||
REWRITE_ARGS -e -v 2 --min-instructions=32 --dyninst-options InstrStackFrames SaveFPR
|
||||
TrampRecursive
|
||||
RUN_ARGS 10 4 1000
|
||||
ENVIRONMENT
|
||||
"${_lock_environment};OMNITRACE_FLAT_PROFILE=ON;OMNITRACE_USE_TIMEMORY=ON;OMNITRACE_USE_PERFETTO=OFF;OMNITRACE_SAMPLING_KEEP_INTERNAL=OFF"
|
||||
REWRITE_RUN_PASS_REGEX
|
||||
"start_thread (.*) 4 (.*) pthread_mutex_lock (.*) 4000 (.*) pthread_mutex_unlock (.*) 4000"
|
||||
)
|
||||
@@ -0,0 +1,267 @@
|
||||
# -------------------------------------------------------------------------------------- #
|
||||
#
|
||||
# python tests
|
||||
#
|
||||
# -------------------------------------------------------------------------------------- #
|
||||
|
||||
set(_INDEX 0)
|
||||
|
||||
foreach(_VERSION ${OMNITRACE_PYTHON_VERSIONS})
|
||||
if(NOT OMNITRACE_USE_PYTHON)
|
||||
continue()
|
||||
endif()
|
||||
|
||||
list(GET OMNITRACE_PYTHON_ROOT_DIRS ${_INDEX} _PYTHON_ROOT_DIR)
|
||||
|
||||
omnitrace_find_python(
|
||||
_PYTHON
|
||||
ROOT_DIR "${_PYTHON_ROOT_DIR}"
|
||||
COMPONENTS Interpreter)
|
||||
|
||||
# ---------------------------------------------------------------------------------- #
|
||||
# python tests
|
||||
# ---------------------------------------------------------------------------------- #
|
||||
omnitrace_add_python_test(
|
||||
NAME python-external
|
||||
PYTHON_EXECUTABLE ${_PYTHON_EXECUTABLE}
|
||||
PYTHON_VERSION ${_VERSION}
|
||||
FILE ${CMAKE_SOURCE_DIR}/examples/python/external.py
|
||||
PROFILE_ARGS "--label" "file"
|
||||
RUN_ARGS -v 10 -n 5
|
||||
ENVIRONMENT "${_python_environment}")
|
||||
|
||||
omnitrace_add_python_test(
|
||||
NAME python-external-exclude-inefficient
|
||||
PYTHON_EXECUTABLE ${_PYTHON_EXECUTABLE}
|
||||
PYTHON_VERSION ${_VERSION}
|
||||
FILE ${CMAKE_SOURCE_DIR}/examples/python/external.py
|
||||
PROFILE_ARGS -E "^inefficient$"
|
||||
RUN_ARGS -v 10 -n 5
|
||||
ENVIRONMENT "${_python_environment}")
|
||||
|
||||
omnitrace_add_python_test(
|
||||
NAME python-builtin
|
||||
PYTHON_EXECUTABLE ${_PYTHON_EXECUTABLE}
|
||||
PYTHON_VERSION ${_VERSION}
|
||||
FILE ${CMAKE_SOURCE_DIR}/examples/python/builtin.py
|
||||
PROFILE_ARGS "-b" "--label" "file" "line"
|
||||
RUN_ARGS -v 10 -n 5
|
||||
ENVIRONMENT "${_python_environment}")
|
||||
|
||||
omnitrace_add_python_test(
|
||||
NAME python-builtin-noprofile
|
||||
PYTHON_EXECUTABLE ${_PYTHON_EXECUTABLE}
|
||||
PYTHON_VERSION ${_VERSION}
|
||||
FILE ${CMAKE_SOURCE_DIR}/examples/python/noprofile.py
|
||||
PROFILE_ARGS "-b" "--label" "file"
|
||||
RUN_ARGS -v 15 -n 5
|
||||
ENVIRONMENT "${_python_environment}")
|
||||
|
||||
omnitrace_add_python_test(
|
||||
STANDALONE
|
||||
NAME python-source
|
||||
PYTHON_EXECUTABLE ${_PYTHON_EXECUTABLE}
|
||||
PYTHON_VERSION ${_VERSION}
|
||||
FILE ${CMAKE_SOURCE_DIR}/examples/python/source.py
|
||||
RUN_ARGS -v 5 -n 5 -s 3
|
||||
ENVIRONMENT "${_python_environment}")
|
||||
|
||||
omnitrace_add_python_test(
|
||||
STANDALONE
|
||||
NAME python-code-coverage
|
||||
PYTHON_EXECUTABLE ${_PYTHON_EXECUTABLE}
|
||||
PYTHON_VERSION ${_VERSION}
|
||||
FILE ${CMAKE_SOURCE_DIR}/examples/code-coverage/code-coverage.py
|
||||
RUN_ARGS
|
||||
-i
|
||||
${PROJECT_BINARY_DIR}/omnitrace-tests-output/code-coverage-basic-blocks-binary-rewrite/coverage.json
|
||||
${PROJECT_BINARY_DIR}/omnitrace-tests-output/code-coverage-basic-blocks-hybrid-runtime-instrument/coverage.json
|
||||
-o
|
||||
${PROJECT_BINARY_DIR}/omnitrace-tests-output/code-coverage-basic-blocks-summary/coverage.json
|
||||
DEPENDS code-coverage-basic-blocks-binary-rewrite
|
||||
code-coverage-basic-blocks-binary-rewrite-run
|
||||
code-coverage-basic-blocks-hybrid-runtime-instrument
|
||||
LABELS "code-coverage"
|
||||
ENVIRONMENT "${_python_environment}")
|
||||
|
||||
# ---------------------------------------------------------------------------------- #
|
||||
# python output tests
|
||||
# ---------------------------------------------------------------------------------- #
|
||||
if(CMAKE_VERSION VERSION_LESS "3.18.0")
|
||||
find_program(
|
||||
OMNITRACE_CAT_EXE
|
||||
NAMES cat
|
||||
PATH_SUFFIXES bin)
|
||||
|
||||
if(OMNITRACE_CAT_EXE)
|
||||
set(OMNITRACE_CAT_COMMAND ${OMNITRACE_CAT_EXE})
|
||||
endif()
|
||||
else()
|
||||
set(OMNITRACE_CAT_COMMAND ${CMAKE_COMMAND} -E cat)
|
||||
endif()
|
||||
|
||||
if(OMNITRACE_CAT_COMMAND)
|
||||
omnitrace_add_python_test(
|
||||
NAME python-external-check
|
||||
COMMAND ${OMNITRACE_CAT_COMMAND}
|
||||
PYTHON_VERSION ${_VERSION}
|
||||
FILE omnitrace-tests-output/python-external/${_VERSION}/trip_count.txt
|
||||
PASS_REGEX
|
||||
"(\\\[compile\\\]).*(\\\| \\\|0>>> \\\[run\\\]\\\[external.py\\\]).*(\\\| \\\|0>>> \\\|_\\\[fib\\\]\\\[external.py\\\]).*(\\\| \\\|0>>> \\\|_\\\[inefficient\\\]\\\[external.py\\\])"
|
||||
DEPENDS python-external-${_VERSION}
|
||||
ENVIRONMENT "${_python_environment}")
|
||||
|
||||
omnitrace_add_python_test(
|
||||
NAME python-external-exclude-inefficient-check
|
||||
COMMAND ${OMNITRACE_CAT_COMMAND}
|
||||
PYTHON_VERSION ${_VERSION}
|
||||
FILE omnitrace-tests-output/python-external-exclude-inefficient/${_VERSION}/trip_count.txt
|
||||
FAIL_REGEX "(\\\|_inefficient).*(\\\|_sum)"
|
||||
DEPENDS python-external-exclude-inefficient-${_VERSION}
|
||||
ENVIRONMENT "${_python_environment}")
|
||||
|
||||
omnitrace_add_python_test(
|
||||
NAME python-builtin-check
|
||||
COMMAND ${OMNITRACE_CAT_COMMAND}
|
||||
PYTHON_VERSION ${_VERSION}
|
||||
FILE omnitrace-tests-output/python-builtin/${_VERSION}/trip_count.txt
|
||||
PASS_REGEX "\\\[inefficient\\\]\\\[builtin.py:14\\\]"
|
||||
DEPENDS python-builtin-${_VERSION}
|
||||
ENVIRONMENT "${_python_environment}")
|
||||
|
||||
omnitrace_add_python_test(
|
||||
NAME python-builtin-noprofile-check
|
||||
COMMAND ${OMNITRACE_CAT_COMMAND}
|
||||
PYTHON_VERSION ${_VERSION}
|
||||
FILE omnitrace-tests-output/python-builtin-noprofile/${_VERSION}/trip_count.txt
|
||||
PASS_REGEX ".(run)..(noprofile.py)."
|
||||
FAIL_REGEX ".(fib|inefficient)..(noprofile.py)."
|
||||
DEPENDS python-builtin-noprofile-${_VERSION}
|
||||
ENVIRONMENT "${_python_environment}")
|
||||
else()
|
||||
omnitrace_message(
|
||||
WARNING
|
||||
"Neither 'cat' nor 'cmake -E cat' are available. Python source checks are disabled"
|
||||
)
|
||||
endif()
|
||||
|
||||
function(OMNITRACE_ADD_PYTHON_VALIDATION_TEST)
|
||||
cmake_parse_arguments(
|
||||
TEST "" "NAME;TIMEMORY_METRIC;TIMEMORY_FILE;PERFETTO_METRIC;PERFETTO_FILE"
|
||||
"ARGS" ${ARGN})
|
||||
|
||||
omnitrace_add_python_test(
|
||||
NAME ${TEST_NAME}-validate-timemory
|
||||
COMMAND
|
||||
${_PYTHON_EXECUTABLE} ${CMAKE_CURRENT_LIST_DIR}/validate-timemory-json.py
|
||||
-m ${TEST_TIMEMORY_METRIC} ${TEST_ARGS} -i
|
||||
PYTHON_VERSION ${_VERSION}
|
||||
FILE omnitrace-tests-output/${TEST_NAME}/${_VERSION}/${TEST_TIMEMORY_FILE}
|
||||
DEPENDS ${TEST_NAME}-${_VERSION}
|
||||
PASS_REGEX
|
||||
"omnitrace-tests-output/${TEST_NAME}/${_VERSION}/${TEST_TIMEMORY_FILE} validated"
|
||||
ENVIRONMENT "${_python_environment}")
|
||||
|
||||
omnitrace_add_python_test(
|
||||
NAME ${TEST_NAME}-validate-perfetto
|
||||
COMMAND
|
||||
${_PYTHON_EXECUTABLE} ${CMAKE_CURRENT_LIST_DIR}/validate-perfetto-proto.py
|
||||
-m ${TEST_PERFETTO_METRIC} ${TEST_ARGS} -p -i
|
||||
PYTHON_VERSION ${_VERSION}
|
||||
FILE omnitrace-tests-output/${TEST_NAME}/${_VERSION}/${TEST_PERFETTO_FILE}
|
||||
DEPENDS ${TEST_NAME}-${_VERSION}
|
||||
PASS_REGEX
|
||||
"omnitrace-tests-output/${TEST_NAME}/${_VERSION}/${TEST_PERFETTO_FILE} validated"
|
||||
ENVIRONMENT "${_python_environment}")
|
||||
endfunction()
|
||||
|
||||
set(python_source_labels
|
||||
main_loop
|
||||
run
|
||||
fib
|
||||
fib
|
||||
fib
|
||||
fib
|
||||
fib
|
||||
inefficient
|
||||
_sum)
|
||||
set(python_source_count
|
||||
5
|
||||
3
|
||||
3
|
||||
6
|
||||
12
|
||||
18
|
||||
6
|
||||
3
|
||||
3)
|
||||
set(python_source_depth
|
||||
0
|
||||
1
|
||||
2
|
||||
3
|
||||
4
|
||||
5
|
||||
6
|
||||
2
|
||||
3)
|
||||
|
||||
omnitrace_add_python_validation_test(
|
||||
NAME python-source
|
||||
TIMEMORY_METRIC "trip_count"
|
||||
TIMEMORY_FILE "trip_count.json"
|
||||
PERFETTO_METRIC "host;user"
|
||||
PERFETTO_FILE "perfetto-trace.proto"
|
||||
ARGS -l ${python_source_labels} -c ${python_source_count} -d
|
||||
${python_source_depth})
|
||||
|
||||
set(python_builtin_labels
|
||||
[run][builtin.py:28]
|
||||
[fib][builtin.py:10]
|
||||
[fib][builtin.py:10]
|
||||
[fib][builtin.py:10]
|
||||
[fib][builtin.py:10]
|
||||
[fib][builtin.py:10]
|
||||
[fib][builtin.py:10]
|
||||
[fib][builtin.py:10]
|
||||
[fib][builtin.py:10]
|
||||
[fib][builtin.py:10]
|
||||
[fib][builtin.py:10]
|
||||
[inefficient][builtin.py:14])
|
||||
set(python_builtin_count
|
||||
5
|
||||
5
|
||||
10
|
||||
20
|
||||
40
|
||||
80
|
||||
160
|
||||
260
|
||||
220
|
||||
80
|
||||
10
|
||||
5)
|
||||
set(python_builtin_depth
|
||||
0
|
||||
1
|
||||
2
|
||||
3
|
||||
4
|
||||
5
|
||||
6
|
||||
7
|
||||
8
|
||||
9
|
||||
10
|
||||
1)
|
||||
|
||||
omnitrace_add_python_validation_test(
|
||||
NAME python-builtin
|
||||
TIMEMORY_METRIC "trip_count"
|
||||
TIMEMORY_FILE "trip_count.json"
|
||||
PERFETTO_METRIC "host;user"
|
||||
PERFETTO_FILE "perfetto-trace.proto"
|
||||
ARGS -l ${python_builtin_labels} -c ${python_builtin_count} -d
|
||||
${python_builtin_depth})
|
||||
math(EXPR _INDEX "${_INDEX} + 1")
|
||||
endforeach()
|
||||
@@ -0,0 +1,60 @@
|
||||
# -------------------------------------------------------------------------------------- #
|
||||
#
|
||||
# rccl tests
|
||||
#
|
||||
# -------------------------------------------------------------------------------------- #
|
||||
|
||||
foreach(_TARGET ${RCCL_TEST_TARGETS})
|
||||
string(REPLACE "rccl-tests::" "" _NAME "${_TARGET}")
|
||||
string(REPLACE "_" "-" _NAME "${_NAME}")
|
||||
omnitrace_add_test(
|
||||
NAME rccl-test-${_NAME}
|
||||
TARGET ${_TARGET}
|
||||
LABELS "rccl-tests;rcclp"
|
||||
MPI ON
|
||||
GPU ON
|
||||
NUM_PROCS 1
|
||||
REWRITE_ARGS
|
||||
-e
|
||||
-v
|
||||
2
|
||||
-i
|
||||
8
|
||||
--label
|
||||
file
|
||||
line
|
||||
return
|
||||
args
|
||||
RUNTIME_ARGS
|
||||
-e
|
||||
-v
|
||||
1
|
||||
-i
|
||||
8
|
||||
--label
|
||||
file
|
||||
line
|
||||
return
|
||||
args
|
||||
-ME
|
||||
sysdeps
|
||||
--log-file
|
||||
rccl-test-${_NAME}.log
|
||||
RUN_ARGS -t
|
||||
1
|
||||
-g
|
||||
1
|
||||
-i
|
||||
10
|
||||
-w
|
||||
2
|
||||
-m
|
||||
2
|
||||
-p
|
||||
-c
|
||||
1
|
||||
-z
|
||||
-s
|
||||
1
|
||||
ENVIRONMENT "${_rccl_environment}")
|
||||
endforeach()
|
||||
@@ -0,0 +1,85 @@
|
||||
# -------------------------------------------------------------------------------------- #
|
||||
#
|
||||
# ROCm tests
|
||||
#
|
||||
# -------------------------------------------------------------------------------------- #
|
||||
|
||||
set(OMNITRACE_ROCM_EVENTS_TEST
|
||||
"GRBM_COUNT,GPUBusy,SQ_WAVES,SQ_INSTS_VALU,VALUInsts,TCC_HIT_sum,TA_TA_BUSY[0]:device=0,TA_TA_BUSY[11]:device=0"
|
||||
)
|
||||
|
||||
omnitrace_add_test(
|
||||
NAME transpose
|
||||
TARGET transpose
|
||||
MPI ${TRANSPOSE_USE_MPI}
|
||||
GPU ON
|
||||
NUM_PROCS ${NUM_PROCS}
|
||||
REWRITE_ARGS -e -v 2 --print-instructions -E uniform_int_distribution
|
||||
RUNTIME_ARGS
|
||||
-e
|
||||
-v
|
||||
1
|
||||
--label
|
||||
file
|
||||
line
|
||||
return
|
||||
args
|
||||
-E
|
||||
uniform_int_distribution
|
||||
ENVIRONMENT "${_base_environment};OMNITRACE_CRITICAL_TRACE=ON")
|
||||
|
||||
omnitrace_add_test(
|
||||
SKIP_BASELINE SKIP_RUNTIME
|
||||
NAME transpose-loops
|
||||
TARGET transpose
|
||||
LABELS "loops"
|
||||
MPI ${TRANSPOSE_USE_MPI}
|
||||
GPU ON
|
||||
NUM_PROCS ${NUM_PROCS}
|
||||
REWRITE_ARGS
|
||||
-e
|
||||
-v
|
||||
2
|
||||
--label
|
||||
return
|
||||
args
|
||||
-l
|
||||
-i
|
||||
8
|
||||
-E
|
||||
uniform_int_distribution
|
||||
RUN_ARGS 2 100 50
|
||||
ENVIRONMENT "${_base_environment};OMNITRACE_CRITICAL_TRACE=OFF"
|
||||
REWRITE_FAIL_REGEX "0 instrumented loops in procedure transpose")
|
||||
|
||||
if(OMNITRACE_USE_ROCPROFILER)
|
||||
omnitrace_add_test(
|
||||
SKIP_BASELINE SKIP_RUNTIME
|
||||
NAME transpose-rocprofiler
|
||||
TARGET transpose
|
||||
LABELS "rocprofiler"
|
||||
MPI ${TRANSPOSE_USE_MPI}
|
||||
GPU ON
|
||||
NUM_PROCS ${NUM_PROCS}
|
||||
REWRITE_ARGS -e -v 2 -E uniform_int_distribution
|
||||
ENVIRONMENT
|
||||
"${_base_environment};OMNITRACE_CRITICAL_TRACE=OFF;OMNITRACE_ROCM_EVENTS=${OMNITRACE_ROCM_EVENTS_TEST}"
|
||||
REWRITE_RUN_PASS_REGEX
|
||||
"rocprof-device-0-GRBM_COUNT.txt(.*)rocprof-device-0-GPUBusy.txt(.*)rocprof-device-0-SQ_WAVES.txt(.*)rocprof-device-0-SQ_INSTS_VALU.txt(.*)rocprof-device-0-VALUInsts.txt(.*)rocprof-device-0-TCC_HIT_sum.txt(.*)rocprof-device-0-TA_TA_BUSY_0.txt(.*)rocprof-device-0-TA_TA_BUSY_11.txt"
|
||||
)
|
||||
|
||||
omnitrace_add_test(
|
||||
SKIP_BASELINE SKIP_RUNTIME
|
||||
NAME transpose-rocprofiler-no-roctracer
|
||||
TARGET transpose
|
||||
LABELS "rocprofiler"
|
||||
MPI ${TRANSPOSE_USE_MPI}
|
||||
GPU ON
|
||||
NUM_PROCS ${NUM_PROCS}
|
||||
REWRITE_ARGS -e -v 2 -E uniform_int_distribution
|
||||
ENVIRONMENT
|
||||
"${_base_environment};OMNITRACE_CRITICAL_TRACE=OFF;OMNITRACE_USE_ROCTRACER=OFF;OMNITRACE_ROCM_EVENTS=${OMNITRACE_ROCM_EVENTS_TEST}"
|
||||
REWRITE_RUN_PASS_REGEX
|
||||
"rocprof-device-0-GRBM_COUNT.txt(.*)rocprof-device-0-GPUBusy.txt(.*)rocprof-device-0-SQ_WAVES.txt(.*)rocprof-device-0-SQ_INSTS_VALU.txt(.*)rocprof-device-0-VALUInsts.txt(.*)rocprof-device-0-TCC_HIT_sum.txt(.*)rocprof-device-0-TA_TA_BUSY_0.txt(.*)rocprof-device-0-TA_TA_BUSY_11.txt"
|
||||
REWRITE_RUN_FAIL_REGEX "roctracer.txt")
|
||||
endif()
|
||||
@@ -262,7 +262,11 @@ function(OMNITRACE_WRITE_TEST_CONFIG _FILE _ENV)
|
||||
set(_FILE_CONTENTS)
|
||||
set(_ENV_CONTENTS)
|
||||
|
||||
set(_DEBUG_SETTINGS ON)
|
||||
foreach(_VAL ${${_ENV}})
|
||||
if("${_VAL}" MATCHES "^OMNITRACE_DEBUG_SETTINGS=")
|
||||
set(_DEBUG_SETTINGS OFF)
|
||||
endif()
|
||||
if("${_VAL}" MATCHES "^OMNITRACE_" AND NOT "${_VAL}" MATCHES "${_ENV_ONLY}")
|
||||
set(_FILE_CONTENTS "${_FILE_CONTENTS}${_VAL}\n")
|
||||
else()
|
||||
@@ -290,7 +294,9 @@ OMNITRACE_ROCTRACER_HSA_ACTIVITY = ON
|
||||
${_FILE_CONTENTS}
|
||||
")
|
||||
list(APPEND _ENV_CONTENTS "OMNITRACE_CONFIG_FILE=${_CONFIG_FILE}")
|
||||
list(APPEND _ENV_CONTENTS "OMNITRACE_DEBUG_SETTINGS=1")
|
||||
if(_DEBUG_SETTINGS)
|
||||
list(APPEND _ENV_CONTENTS "OMNITRACE_DEBUG_SETTINGS=1")
|
||||
endif()
|
||||
set(${_ENV}
|
||||
"${_ENV_CONTENTS}"
|
||||
PARENT_SCOPE)
|
||||
@@ -336,25 +342,24 @@ endmacro()
|
||||
# -------------------------------------------------------------------------------------- #
|
||||
|
||||
function(OMNITRACE_ADD_TEST)
|
||||
foreach(_PREFIX PRELOAD RUNTIME REWRITE REWRITE_RUN BASELINE)
|
||||
foreach(_PREFIX SAMPLING RUNTIME REWRITE REWRITE_RUN BASELINE)
|
||||
foreach(_TYPE PASS FAIL SKIP)
|
||||
list(APPEND _REGEX_OPTS "${_PREFIX}_${_TYPE}_REGEX")
|
||||
endforeach()
|
||||
endforeach()
|
||||
set(_KWARGS REWRITE_ARGS RUNTIME_ARGS RUN_ARGS ENVIRONMENT LABELS PROPERTIES
|
||||
${_REGEX_OPTS})
|
||||
set(_KWARGS REWRITE_ARGS RUNTIME_ARGS SAMPLING_ARGS RUN_ARGS ENVIRONMENT LABELS
|
||||
PROPERTIES ${_REGEX_OPTS})
|
||||
|
||||
cmake_parse_arguments(
|
||||
TEST
|
||||
"SKIP_BASELINE;SKIP_PRELOAD;SKIP_REWRITE;SKIP_RUNTIME;SKIP_SAMPLING;FORCE_SAMPLING"
|
||||
"NAME;TARGET;MPI;GPU;NUM_PROCS;REWRITE_TIMEOUT;RUNTIME_TIMEOUT;PRELOAD"
|
||||
"${_KWARGS}"
|
||||
TEST "SKIP_BASELINE;SKIP_SAMPLING;SKIP_REWRITE;SKIP_RUNTIME"
|
||||
"NAME;TARGET;MPI;GPU;NUM_PROCS;REWRITE_TIMEOUT;RUNTIME_TIMEOUT" "${_KWARGS}"
|
||||
${ARGN})
|
||||
|
||||
foreach(_PREFIX PRELOAD RUNTIME REWRITE REWRITE_RUN BASELINE)
|
||||
foreach(_PREFIX SAMPLING RUNTIME REWRITE REWRITE_RUN BASELINE)
|
||||
if("${${_PREFIX}_FAIL_REGEX}" STREQUAL "")
|
||||
set(${_PREFIX}_FAIL_REGEX
|
||||
"(### ERROR ###|address of faulting memory reference)")
|
||||
"(### ERROR ###|address of faulting memory reference|exiting with non-zero exit code)"
|
||||
)
|
||||
endif()
|
||||
endforeach()
|
||||
|
||||
@@ -387,8 +392,8 @@ function(OMNITRACE_ADD_TEST)
|
||||
set(TEST_RUNTIME_TIMEOUT 300)
|
||||
endif()
|
||||
|
||||
if(NOT TEST_PRELOAD_TIMEOUT)
|
||||
set(TEST_PRELOAD_TIMEOUT 120)
|
||||
if(NOT TEST_SAMPLING_TIMEOUT)
|
||||
set(TEST_SAMPLING_TIMEOUT 120)
|
||||
endif()
|
||||
|
||||
if(NOT DEFINED TEST_ENVIRONMENT OR "${TEST_ENVIRONMENT}" STREQUAL "")
|
||||
@@ -448,11 +453,12 @@ function(OMNITRACE_ADD_TEST)
|
||||
WORKING_DIRECTORY ${PROJECT_BINARY_DIR})
|
||||
endif()
|
||||
|
||||
if(NOT TEST_SKIP_PRELOAD)
|
||||
if(NOT TEST_SKIP_SAMPLING)
|
||||
add_test(
|
||||
NAME ${TEST_NAME}-preload
|
||||
COMMAND ${COMMAND_PREFIX} $<TARGET_FILE:omnitrace-sample> --
|
||||
$<TARGET_FILE:${TEST_TARGET}> ${TEST_RUN_ARGS}
|
||||
NAME ${TEST_NAME}-sampling
|
||||
COMMAND
|
||||
${COMMAND_PREFIX} $<TARGET_FILE:omnitrace-sample> ${TEST_SAMPLE_ARGS}
|
||||
-- $<TARGET_FILE:${TEST_TARGET}> ${TEST_RUN_ARGS}
|
||||
WORKING_DIRECTORY ${PROJECT_BINARY_DIR})
|
||||
endif()
|
||||
|
||||
@@ -473,23 +479,6 @@ function(OMNITRACE_ADD_TEST)
|
||||
WORKING_DIRECTORY ${PROJECT_BINARY_DIR})
|
||||
endif()
|
||||
|
||||
if(TEST_FORCE_SAMPLING OR (NOT TEST_SKIP_REWRITE AND NOT TEST_SKIP_SAMPLING))
|
||||
add_test(
|
||||
NAME ${TEST_NAME}-binary-rewrite-sampling
|
||||
COMMAND
|
||||
$<TARGET_FILE:omnitrace-instrument> -o
|
||||
$<TARGET_FILE_DIR:${TEST_TARGET}>/${TEST_NAME}.samp -M sampling
|
||||
${TEST_REWRITE_ARGS} -- $<TARGET_FILE:${TEST_TARGET}>
|
||||
WORKING_DIRECTORY ${PROJECT_BINARY_DIR})
|
||||
|
||||
add_test(
|
||||
NAME ${TEST_NAME}-binary-rewrite-sampling-run
|
||||
COMMAND
|
||||
${COMMAND_PREFIX} $<TARGET_FILE:omnitrace-run> --
|
||||
$<TARGET_FILE_DIR:${TEST_TARGET}>/${TEST_NAME}.samp ${TEST_RUN_ARGS}
|
||||
WORKING_DIRECTORY ${PROJECT_BINARY_DIR})
|
||||
endif()
|
||||
|
||||
if(NOT TEST_SKIP_RUNTIME AND NOT OMNITRACE_USE_SANITIZER)
|
||||
add_test(
|
||||
NAME ${TEST_NAME}-runtime-instrument
|
||||
@@ -498,34 +487,16 @@ function(OMNITRACE_ADD_TEST)
|
||||
WORKING_DIRECTORY ${PROJECT_BINARY_DIR})
|
||||
endif()
|
||||
|
||||
if((TEST_FORCE_SAMPLING OR (NOT TEST_SKIP_RUNTIME AND NOT TEST_SKIP_SAMPLING))
|
||||
AND NOT OMNITRACE_USE_SANITIZER)
|
||||
add_test(
|
||||
NAME ${TEST_NAME}-runtime-instrument-sampling
|
||||
COMMAND
|
||||
$<TARGET_FILE:omnitrace-instrument> -M sampling ${TEST_RUNTIME_ARGS}
|
||||
-- $<TARGET_FILE:${TEST_TARGET}> ${TEST_RUN_ARGS}
|
||||
WORKING_DIRECTORY ${PROJECT_BINARY_DIR})
|
||||
endif()
|
||||
|
||||
if(TEST ${TEST_NAME}-binary-rewrite-run)
|
||||
set_tests_properties(${TEST_NAME}-binary-rewrite-run
|
||||
PROPERTIES DEPENDS ${TEST_NAME}-binary-rewrite)
|
||||
endif()
|
||||
|
||||
if(TEST ${TEST_NAME}-binary-rewrite-sampling-run)
|
||||
set_tests_properties(${TEST_NAME}-binary-rewrite-sampling-run
|
||||
PROPERTIES DEPENDS ${TEST_NAME}-binary-rewrite-sampling)
|
||||
endif()
|
||||
|
||||
foreach(
|
||||
_TEST
|
||||
baseline preload binary-rewrite binary-rewrite-run binary-rewrite-sampling
|
||||
binary-rewrite-sampling-run runtime-instrument runtime-instrument-sampling)
|
||||
foreach(_TEST baseline sampling binary-rewrite binary-rewrite-run
|
||||
runtime-instrument)
|
||||
string(REGEX REPLACE "-run(-|/)" "\\1" _prefix "${TEST_NAME}-${_TEST}/")
|
||||
set(_labels "${_TEST}")
|
||||
string(REPLACE "-run" "" _labels "${_TEST}")
|
||||
string(REPLACE "-sampling" ";sampling" _labels "${_labels}")
|
||||
if(TEST_TARGET)
|
||||
list(APPEND _labels "${TEST_TARGET}")
|
||||
endif()
|
||||
@@ -539,14 +510,14 @@ function(OMNITRACE_ADD_TEST)
|
||||
"OMNITRACE_OUTPUT_PREFIX=${_prefix}")
|
||||
|
||||
set(_timeout ${TEST_REWRITE_TIMEOUT})
|
||||
if("${_TEST}" MATCHES "preload")
|
||||
set(_timeout ${TEST_PRELOAD_TIMEOUT})
|
||||
if("${_TEST}" MATCHES "sampling")
|
||||
set(_timeout ${TEST_SAMPLING_TIMEOUT})
|
||||
elseif("${_TEST}" MATCHES "runtime-instrument")
|
||||
set(_timeout ${TEST_RUNTIME_TIMEOUT})
|
||||
endif()
|
||||
|
||||
set(_props)
|
||||
if("${_TEST}" MATCHES "run|preload|baseline")
|
||||
if("${_TEST}" MATCHES "run|sampling|baseline")
|
||||
set(_props ${TEST_PROPERTIES})
|
||||
if(NOT "RUN_SERIAL" IN_LIST _props)
|
||||
list(APPEND _props RUN_SERIAL ON)
|
||||
@@ -561,13 +532,13 @@ function(OMNITRACE_ADD_TEST)
|
||||
set(_REGEX_VAR REWRITE)
|
||||
elseif("${_TEST}" MATCHES "baseline")
|
||||
set(_REGEX_VAR BASELINE)
|
||||
elseif("${_TEST}" MATCHES "preload")
|
||||
set(_REGEX_VAR PRELOAD)
|
||||
elseif("${_TEST}" MATCHES "sampling")
|
||||
set(_REGEX_VAR SAMPLING)
|
||||
else()
|
||||
set(_REGEX_VAR)
|
||||
endif()
|
||||
|
||||
if("${_TEST}" MATCHES "binary-rewrite-run|runtime-instrument|preload")
|
||||
if("${_TEST}" MATCHES "binary-rewrite-run|runtime-instrument|sampling")
|
||||
omnitrace_patch_sanitizer_environment(_environ)
|
||||
endif()
|
||||
|
||||
@@ -632,6 +603,12 @@ function(OMNITRACE_ADD_CAUSAL_TEST)
|
||||
set(TEST_CAUSAL_VALIDATE_TIMEOUT 60)
|
||||
endif()
|
||||
|
||||
if("${TEST_CAUSAL_FAIL_REGEX}" STREQUAL "")
|
||||
set(TEST_CAUSAL_FAIL_REGEX
|
||||
"(### ERROR ###|address of faulting memory reference|exiting with non-zero exit code)"
|
||||
)
|
||||
endif()
|
||||
|
||||
if(TARGET ${TEST_TARGET})
|
||||
set(COMMAND_PREFIX $<TARGET_FILE:omnitrace-causal> --reset -m ${TEST_CAUSAL_MODE}
|
||||
${TEST_CAUSAL_ARGS} --)
|
||||
@@ -692,7 +669,10 @@ function(OMNITRACE_ADD_CAUSAL_TEST)
|
||||
"OMNITRACE_OUTPUT_PREFIX=${_prefix}"
|
||||
"OMNITRACE_CI=ON"
|
||||
"OMNITRACE_USE_PID=OFF"
|
||||
"OMNITRACE_THREAD_POOL_SIZE=1"
|
||||
"OMNITRACE_THREAD_POOL_SIZE=0"
|
||||
"OMNITRACE_VERBOSE=1"
|
||||
"OMNITRACE_DL_VERBOSE=0"
|
||||
"OMNITRACE_DEBUG_SETTINGS=0"
|
||||
"${TEST_ENVIRONMENT}")
|
||||
|
||||
set(_timeout ${TEST_CAUSAL_TIMEOUT})
|
||||
@@ -954,6 +934,9 @@ function(OMNITRACE_ADD_VALIDATION_TEST)
|
||||
endforeach()
|
||||
|
||||
list(APPEND TEST_DEPENDS "${TEST_NAME}")
|
||||
if("${TEST_NAME}" MATCHES "-binary-rewrite")
|
||||
list(APPEND TEST_DEPENDS "${TEST_NAME}-run")
|
||||
endif()
|
||||
|
||||
if(NOT TEST_PASS_REGEX)
|
||||
set(TEST_PASS_REGEX
|
||||
|
||||
@@ -0,0 +1,114 @@
|
||||
# -------------------------------------------------------------------------------------- #
|
||||
#
|
||||
# time-window tests
|
||||
#
|
||||
# -------------------------------------------------------------------------------------- #
|
||||
|
||||
if(_OS_RELEASE STREQUAL "ubuntu-18.04")
|
||||
set(_TRACE_WINDOW_SKIP SKIP_RUNTIME)
|
||||
endif()
|
||||
|
||||
omnitrace_add_test(
|
||||
SKIP_BASELINE SKIP_SAMPLING ${_TRACE_WINDOW_SKIP}
|
||||
NAME trace-time-window
|
||||
TARGET trace-time-window
|
||||
REWRITE_ARGS -e -v 2 --caller-include inner -i 4096
|
||||
RUNTIME_ARGS -e -v 1 --caller-include inner -i 4096
|
||||
LABELS "time-window"
|
||||
ENVIRONMENT "${_window_environment};OMNITRACE_TRACE_DURATION=1.25")
|
||||
|
||||
omnitrace_add_validation_test(
|
||||
NAME trace-time-window-binary-rewrite
|
||||
TIMEMORY_METRIC "wall_clock"
|
||||
TIMEMORY_FILE "wall_clock.json"
|
||||
PERFETTO_METRIC "host"
|
||||
PERFETTO_FILE "perfetto-trace.proto"
|
||||
LABELS "time-window"
|
||||
FAIL_REGEX "outer_d"
|
||||
ARGS -l
|
||||
trace-time-window.inst
|
||||
outer_a
|
||||
outer_b
|
||||
outer_c
|
||||
-c
|
||||
1
|
||||
1
|
||||
1
|
||||
1
|
||||
-d
|
||||
0
|
||||
1
|
||||
1
|
||||
1
|
||||
-p)
|
||||
|
||||
omnitrace_add_validation_test(
|
||||
NAME trace-time-window-runtime-instrument
|
||||
TIMEMORY_METRIC "wall_clock"
|
||||
TIMEMORY_FILE "wall_clock.json"
|
||||
PERFETTO_METRIC "host"
|
||||
PERFETTO_FILE "perfetto-trace.proto"
|
||||
LABELS "time-window"
|
||||
FAIL_REGEX "outer_d"
|
||||
ARGS -l
|
||||
trace-time-window
|
||||
outer_a
|
||||
outer_b
|
||||
outer_c
|
||||
-c
|
||||
1
|
||||
1
|
||||
1
|
||||
1
|
||||
-d
|
||||
0
|
||||
1
|
||||
1
|
||||
1
|
||||
-p)
|
||||
|
||||
omnitrace_add_test(
|
||||
SKIP_BASELINE SKIP_SAMPLING ${_TRACE_WINDOW_SKIP}
|
||||
NAME trace-time-window-delay
|
||||
TARGET trace-time-window
|
||||
REWRITE_ARGS -e -v 2 --caller-include inner -i 4096
|
||||
RUNTIME_ARGS -e -v 1 --caller-include inner -i 4096
|
||||
LABELS "time-window"
|
||||
ENVIRONMENT
|
||||
"${_window_environment};OMNITRACE_TRACE_DELAY=0.75;OMNITRACE_TRACE_DURATION=0.75")
|
||||
|
||||
omnitrace_add_validation_test(
|
||||
NAME trace-time-window-delay-binary-rewrite
|
||||
TIMEMORY_METRIC "wall_clock"
|
||||
TIMEMORY_FILE "wall_clock.json"
|
||||
PERFETTO_METRIC "host"
|
||||
PERFETTO_FILE "perfetto-trace.proto"
|
||||
LABELS "time-window"
|
||||
ARGS -l
|
||||
outer_c
|
||||
outer_d
|
||||
-c
|
||||
1
|
||||
1
|
||||
-d
|
||||
0
|
||||
0
|
||||
-p)
|
||||
|
||||
omnitrace_add_validation_test(
|
||||
NAME trace-time-window-delay-runtime-instrument
|
||||
TIMEMORY_METRIC "wall_clock"
|
||||
TIMEMORY_FILE "wall_clock.json"
|
||||
PERFETTO_METRIC "host"
|
||||
PERFETTO_FILE "perfetto-trace.proto"
|
||||
LABELS "time-window"
|
||||
ARGS -l
|
||||
outer_c
|
||||
outer_d
|
||||
-c
|
||||
1
|
||||
1
|
||||
-d
|
||||
0
|
||||
0
|
||||
-p)
|
||||
@@ -0,0 +1,31 @@
|
||||
# -------------------------------------------------------------------------------------- #
|
||||
#
|
||||
# User API tests
|
||||
#
|
||||
# -------------------------------------------------------------------------------------- #
|
||||
|
||||
omnitrace_add_test(
|
||||
NAME user-api
|
||||
TARGET user-api
|
||||
LABELS "loops"
|
||||
REWRITE_ARGS -e -v 2 -l --min-instructions=8 -E custom_push_region
|
||||
RUNTIME_ARGS
|
||||
-e
|
||||
-v
|
||||
1
|
||||
-l
|
||||
--min-instructions=8
|
||||
-E
|
||||
custom_push_region
|
||||
--label
|
||||
file
|
||||
line
|
||||
return
|
||||
args
|
||||
RUN_ARGS 10 ${NUM_THREADS} 1000
|
||||
ENVIRONMENT "${_base_environment};OMNITRACE_CRITICAL_TRACE=OFF"
|
||||
REWRITE_RUN_PASS_REGEX "Pushing custom region :: run.10. x 1000"
|
||||
RUNTIME_PASS_REGEX "Pushing custom region :: run.10. x 1000"
|
||||
SAMPLING_PASS_REGEX "Pushing custom region :: run.10. x 1000"
|
||||
BASELINE_FAIL_REGEX "Pushing custom region"
|
||||
REWRITE_FAIL_REGEX "0 instrumented loops in procedure")
|
||||
@@ -9,7 +9,7 @@ import argparse
|
||||
from collections import OrderedDict
|
||||
|
||||
|
||||
num_stddev = 1
|
||||
num_stddev = 1.0
|
||||
|
||||
|
||||
def mean(_data):
|
||||
@@ -21,7 +21,7 @@ def stddev(_data):
|
||||
return 0.0
|
||||
_mean = mean(_data)
|
||||
_variance = sum([((x - _mean) ** 2) for x in _data]) / float(len(_data))
|
||||
return _variance**0.5
|
||||
return float(num_stddev) * math.sqrt(_variance)
|
||||
|
||||
|
||||
def simpsons_rule(a, b, fa, fb):
|
||||
@@ -66,21 +66,28 @@ class validation(object):
|
||||
return None
|
||||
|
||||
_tolerance = self.tolerance
|
||||
|
||||
if _ci is True and _virt_speedup > 10:
|
||||
"""On GitHub Action servers, you typically only get one core with two hyperthreads.
|
||||
The hyperthreading causes the speedup potential to drop off at higher virtual speedups
|
||||
so we consider
|
||||
_reason = "[unspecified reason]"
|
||||
if _ci is True:
|
||||
"""On GitHub Action servers, you typically only get two CPUs, which may be one
|
||||
core with two hyperthreads. The hyperthreading can causes the speedup potential
|
||||
to drop. Furthermore, these are typically shared resources so the runtime may
|
||||
vary significantly. Thus, always account for stddev to prevent failures due to
|
||||
these causes
|
||||
"""
|
||||
_tolerance += max([_base_speedup_stddev, _prog_speedup_stddev])
|
||||
_reason = "results obtained on a shared CI system... potentially artificially deflating speedup predictions"
|
||||
elif _base_speedup_stddev > self.tolerance:
|
||||
_tolerance += math.sqrt(_base_speedup_stddev)
|
||||
_reason = (
|
||||
f"large standard deviation of the baseline ({_base_speedup_stddev:.3f})"
|
||||
)
|
||||
elif _prog_speedup_stddev > 1.0:
|
||||
_tolerance += math.sqrt(_prog_speedup_stddev)
|
||||
_reason = f"large standard deviation of the program speedup ({_prog_speedup_stddev:.3f})"
|
||||
|
||||
if _tolerance > self.tolerance:
|
||||
sys.stderr.write(
|
||||
f" [{_exp_name}][{_pp_name}][{_virt_speedup}] Tolerance adjusted due to stddev or to account for hyperthreading on CI systems ({self.tolerance:.3f} increased to {_tolerance:.3f})...\n"
|
||||
f" [{_exp_name}][{_pp_name}][{_virt_speedup}] Tolerance increased: {_reason} ({self.tolerance:.3f} increased to {_tolerance:.3f})...\n"
|
||||
)
|
||||
|
||||
def _compute(_speedup_v, _tolerance_v):
|
||||
@@ -195,9 +202,7 @@ class line_speedup(object):
|
||||
if self.data is None or self.base is None:
|
||||
return f"{self.name}"
|
||||
_line_speedup = self.compute_speedup()
|
||||
_line_stddev = (
|
||||
float(num_stddev) * self.compute_speedup_stddev()
|
||||
) # 3 stddev == 99.87%
|
||||
_line_stddev = self.compute_speedup_stddev() # 3 stddev == 99.87%
|
||||
_name = self.get_name()
|
||||
return f"[{_name}][{self.prog}][{self.data.speedup:3}] speedup: {_line_speedup:6.1f} +/- {_line_stddev:6.2f} %"
|
||||
|
||||
@@ -345,7 +350,6 @@ def compute_speedups(_data, args):
|
||||
for selected, pitr in _data.items():
|
||||
for progpt, ditr in pitr.items():
|
||||
if 0 not in ditr.keys():
|
||||
# print(f"missing baseline data for {progpt} in {selected}...")
|
||||
continue
|
||||
_baseline = ditr[0].mean()
|
||||
for speedup, itr in ditr.items():
|
||||
@@ -353,8 +357,9 @@ def compute_speedups(_data, args):
|
||||
continue
|
||||
if speedup != itr.speedup:
|
||||
raise ValueError(f"in {selected}: {speedup} != {itr.speedup}")
|
||||
_val = line_speedup(selected, progpt, itr, ditr[0])
|
||||
ret.append(_val)
|
||||
if len(itr) >= args.min_experiments:
|
||||
_val = line_speedup(selected, progpt, itr, ditr[0])
|
||||
ret.append(_val)
|
||||
|
||||
ret.sort()
|
||||
_last_name = None
|
||||
@@ -400,6 +405,8 @@ def get_validations(args):
|
||||
def main():
|
||||
import argparse
|
||||
|
||||
global num_stddev
|
||||
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument(
|
||||
"-e", "--experiments", type=str, help="Regex for experiments", default=".*"
|
||||
@@ -414,6 +421,13 @@ def main():
|
||||
parser.add_argument(
|
||||
"-n", "--num-points", type=int, help="Minimum number of data points", default=5
|
||||
)
|
||||
parser.add_argument(
|
||||
"-m",
|
||||
"--min-experiments",
|
||||
type=int,
|
||||
help="Minimum number of experiments per speedup (e.g. do not display speedups when there are fewer than X experiments at this speedup)",
|
||||
default=2,
|
||||
)
|
||||
parser.add_argument(
|
||||
"-i", "--input", type=str, nargs="*", help="Input file(s)", required=True
|
||||
)
|
||||
@@ -428,9 +442,9 @@ def main():
|
||||
parser.add_argument(
|
||||
"-d",
|
||||
"--stddev",
|
||||
type=int,
|
||||
type=float,
|
||||
help="Number of standard deviations to report",
|
||||
default=1,
|
||||
default=1.0,
|
||||
)
|
||||
parser.add_argument(
|
||||
"-v",
|
||||
@@ -440,6 +454,12 @@ def main():
|
||||
help="Validate speedup: {experiment regex} {progress-point regex} {virtual-speedup} {expected-speedup} {tolerance}",
|
||||
default=[],
|
||||
)
|
||||
parser.add_argument(
|
||||
"--samples",
|
||||
type=float,
|
||||
help="Report samples within this percentage of the peak (0.0, 100.0] (default: 95 percent)",
|
||||
default=95.0,
|
||||
)
|
||||
parser.add_argument(
|
||||
"--ci",
|
||||
action="store_true",
|
||||
@@ -454,6 +474,13 @@ def main():
|
||||
num_stddev = args.stddev
|
||||
num_speedups = len(args.speedups)
|
||||
|
||||
percent_samples = args.samples
|
||||
if not percent_samples > 0.0 and not percent_samples <= 100.0:
|
||||
raise ValueError(
|
||||
f"Invalid samples value: {percent_samples}. Supported range: 0.0 < x <= 100.0"
|
||||
)
|
||||
percent_samples = 1.0 - (percent_samples / 100.0)
|
||||
|
||||
if num_speedups > 0 and args.num_points > num_speedups:
|
||||
args.num_points = num_speedups
|
||||
|
||||
@@ -466,9 +493,11 @@ def main():
|
||||
samp = process_samples(samp, inp_data)
|
||||
|
||||
print("Samples:")
|
||||
width = max([len(x) for x in samp.keys()])
|
||||
for name, count in sorted(samp.items()):
|
||||
print(f" {name:{width}} :: {count}")
|
||||
width = max([int(math.log10(x) + 1) for _, x in samp.items()])
|
||||
samp_peak = max([count for _, count in samp.items()])
|
||||
for name, count in sorted(samp.items(), key=lambda x: x[1], reverse=True):
|
||||
if count >= samp_peak * percent_samples:
|
||||
print(f" {count:{width}} :: {name}")
|
||||
|
||||
results = compute_speedups(data, args)
|
||||
print("")
|
||||
|
||||
Reference in new issue
Заблокувати користувача