AI/rocm-systems
2
0
Garpu 0
Kode Masalah Tarik Permintaan Actions Packages Projects Rilis Wiki Kegiatan

rocm-smi and KokkosTools support (#23)

Telusuri Sumber 
* renamed omnitrace_thread_data to thread_data

* initial implementation

* Numerous fixes and updates

- Updated timemory submodule
- Updated perfetto submodule (pulls in fixes for TRACE_EVENT)
- pthread_gotcha only after omnitrace_init_tooling
- omnitrace banner
- config settings for rocm-smi freq and devices
- critical_trace::get_entries
- OMNITRACE_BASIC_PRINT
- rocm_smi perfetto category
- redirect roctracer warnings for ROCm 4.5.0
- property specializations for rocm-smi components
- units fixes data_tracker types
- roctracer entries for pthread_create and start_thread
- omnitrace-avail defaults to settings, not components
- settings have conforming names
- settings warn about duplicates
- ptl named threads
- decreased max freq for sampler SIGALRM
- rocm-smi names thread
- rocm-smi avoids call to hipGetDeviceCount
- name roctracer activity callback threads
- fixed binary rewrite test output names

* Update lulesh example

- supports non-UVM GPU

* Lulesh tweaks + formatting

* KokkosP + Mode + Roctracer sampling deadlock fix

- kokkosp support
- omnitrace_init_library
- config::print_settings()
- config::get_mode()
- omnitrace::Mode
- omnitrace-avail improvements (removes settings)
- handle get_verbose() < 0
- disable dyninst InstrStackFrames by default
- handle perf_event_paranoid > 1 by disabling PAPI
- SIGALRM max freq to 5.0
- Name threads
- rocm-smi handles get_use_perfetto() and get_use_timemory()
- HSA_ENABLE_INTERRUPT=0 when roctracer + sampling (fixes deadlock)

* Tests, API renaming, roctracer

- disable renaming of thread 0
- verbprintf_bare
- enable dyninst merge tramp
- tweaked some omnitrace exe verbose levels
- reworked roctracer::setup and roctracer::shutdown
- rocm_smi::data::poll checks get_state()
- omnitrace_trace_finalize -> omnitrace_finalize
- omnitrace_trace_init -> omnitrace_init
- omnitrace_trace_set_env -> omnitrace_set_env
- omnitrace_trace_set_mpi -> omnitrace_set_mpi
- sampling mode does not disable timemory
- disable roctracer before shutting down rocm-smi
- lulesh tests w/ and w/o kokkosp
- lulesh tests for perfetto only
    - with --dynamic-callsites --traps --allow-overlapping
- lulesh tests for timemory only
    - with --stdlib --dynamic-callsites --traps --allow-overlapping

* Update timemory submodule

- fix for TIMEMORY_PROPERTY_SPECIALIZATION

* get_verbose() handling + timemory submodule update

- Findroctracer.cmake uses find_package(hsakmt)

* Stability fixes + rework roctracer + perfetto

- reworked roctracer start up
- critical_trace perfetto basic values
- perfetto sampling category
- sampler checks signals
- peak_rss in sampling
- pthread_gotcha::shutdown()
- rocm_smi::device_count()
- HSA_TOOLS_LIB is set
- HSA_ENABLE_INTERRUPT in omnitrace exe
- omnitrace exe verbosity level changes
- Avoid instrumenting Impl ns in Kokkos
- gpu::device_count prefers rocm_smi instead of hip
- ptl blocks signals
- fixed pthread_gotcha roctracer_data values
- removed runtime-instrument-sampling tests
- timemory submodule update

* cmake formatting

* timemory + roctracer updates

- fix timemory issue with papi_common
- fix timemory issue with units
- define roctracer::is_setup()

* Miscellaneous tweaks

- Disable sampling during runtime instrument
- Fixed warnings about dynamic callsites
- Fixed backtrace output when timemory disabled
- Test tweaks

* cmake-format

* omnitrace_target_compile_definitions

* timemory submodule update

* config, omnitrace, State, mpi_gotcha updates

- use OMNITRACE_THROW instead of direct throw
- is_attached()
- is_binary_rewrite()
- get_is_continuous_integration()
- get_debug_init()
- get_debug_finalize()
- max_thread_bookmarks default to 1
- State::Init
- app_thread oneTimeCode
- runtime instrumentation uses waitpid
- fixed init_names
- include main in MPI runs
- fixed sampling setup when disabled
- reworked mpi_gotcha
- disabled critical trace in transpose test

* cmake-format

* handle rocm_smi::device_count() exception

* CI timeouts

* Re-enable runtime-instrument + sampling

[ROCm/rocprofiler-systems commit: 39f17ae8b8]
This commit is contained in:
Jonathan R. Madsen
2022-02-08 17:42:17 -06:00
committed by GitHub
orang tua b4a82711d1
melakukan 4ae26e2d08
59 mengubah file dengan 5116 tambahan dan 2592 penghapusan
Download Patch File Download Diff File Expand all files Collapse all files
projects/rocprofiler-systems/tests/CMakeLists.txt
+144 -32
Melihat File
@@ -26,7 +26,29 @@ endif()
set(_base_environment
"OMNITRACE_USE_PERFETTO=ON"
"OMNITRACE_USE_TIMEMORY=ON"
"OMNITRACE_USE_SAMPLING=OFF"
"OMNITRACE_USE_SAMPLING=ON"
"OMNITRACE_TIME_OUTPUT=OFF"
"OMP_PROC_BIND=spread"
"OMP_PLACES=threads"
"OMP_NUM_THREADS=2"
"LD_LIBRARY_PATH=${PROJECT_BINARY_DIR}:${OMNITRACE_DYNINST_API_RT_DIR}:$ENV{LD_LIBRARY_PATH}"
)
set(_perfetto_environment
"OMNITRACE_USE_PERFETTO=ON"
"OMNITRACE_USE_TIMEMORY=OFF"
"OMNITRACE_USE_SAMPLING=ON"
"OMNITRACE_TIME_OUTPUT=OFF"
"OMP_PROC_BIND=spread"
"OMP_PLACES=threads"
"OMP_NUM_THREADS=2"
"LD_LIBRARY_PATH=${PROJECT_BINARY_DIR}:${OMNITRACE_DYNINST_API_RT_DIR}:$ENV{LD_LIBRARY_PATH}"
)
set(_timemory_environment
"OMNITRACE_USE_PERFETTO=OFF"
"OMNITRACE_USE_TIMEMORY=ON"
"OMNITRACE_USE_SAMPLING=ON"
"OMNITRACE_TIME_OUTPUT=OFF"
"OMP_PROC_BIND=spread"
"OMP_PLACES=threads"
@@ -48,12 +70,15 @@ set(_fast_environment
"LD_LIBRARY_PATH=${PROJECT_BINARY_DIR}:${OMNITRACE_DYNINST_API_RT_DIR}:$ENV{LD_LIBRARY_PATH}"
)
# -------------------------------------------------------------------------------------- #
function(OMNITRACE_ADD_TEST)
cmake_parse_arguments(
TEST
"" # options
"NAME;TARGET;MPI;NUM_PROCS" # single value args
"REWRITE_ARGS;RUNTIME_ARGS;RUN_ARGS;ENVIRONMENT;LABELS" # multiple value args
"NAME;TARGET;MPI;NUM_PROCS;REWRITE_TIMEOUT;RUNTIME_TIMEOUT" # single value args
"REWRITE_ARGS;RUNTIME_ARGS;RUN_ARGS;ENVIRONMENT;LABELS;PROPERTIES" # multiple
# value args
${ARGN})
if("${TEST_MPI}" STREQUAL "")
@@ -68,10 +93,20 @@ function(OMNITRACE_ADD_TEST)
set(TEST_NUM_PROCS 0)
endif()
if(NOT TEST_REWRITE_TIMEOUT)
set(TEST_REWRITE_TIMEOUT 120)
endif()
if(NOT TEST_RUNTIME_TIMEOUT)
set(TEST_RUNTIME_TIMEOUT 300)
endif()
if(NOT DEFINED TEST_ENVIRONMENT OR "${TEST_ENVIRONMENT}" STREQUAL "")
set(TEST_ENVIRONMENT "${_test_environment}")
endif()
list(APPEND TEST_ENVIRONMENT "OMNITRACE_CI=ON")
if(TARGET ${TEST_TARGET})
if(DEFINED TEST_MPI
AND ${TEST_MPI}
@@ -88,14 +123,14 @@ function(OMNITRACE_ADD_TEST)
add_test(
NAME ${TEST_NAME}-baseline
COMMAND $<TARGET_FILE:${TEST_TARGET}> ${TEST_RUN_ARGS}
COMMAND ${COMMAND_PREFIX} $<TARGET_FILE:${TEST_TARGET}> ${TEST_RUN_ARGS}
WORKING_DIRECTORY $<TARGET_FILE_DIR:${TEST_TARGET}>)
add_test(
NAME ${TEST_NAME}-binary-rewrite
COMMAND
$<TARGET_FILE:omnitrace-exe> -o
$<TARGET_FILE_DIR:${TEST_TARGET}>/${TEST_TARGET}.inst ${TEST_REWRITE_ARGS}
$<TARGET_FILE_DIR:${TEST_TARGET}>/${TEST_NAME}.inst ${TEST_REWRITE_ARGS}
-- $<TARGET_FILE:${TEST_TARGET}>
WORKING_DIRECTORY $<TARGET_FILE_DIR:${TEST_TARGET}>)
@@ -103,20 +138,20 @@ function(OMNITRACE_ADD_TEST)
NAME ${TEST_NAME}-binary-rewrite-sampling
COMMAND
$<TARGET_FILE:omnitrace-exe> -o
$<TARGET_FILE_DIR:${TEST_TARGET}>/${TEST_TARGET}.samp -M sampling
$<TARGET_FILE_DIR:${TEST_TARGET}>/${TEST_NAME}.samp -M sampling
${TEST_REWRITE_ARGS} -- $<TARGET_FILE:${TEST_TARGET}>
WORKING_DIRECTORY $<TARGET_FILE_DIR:${TEST_TARGET}>)
add_test(
NAME ${TEST_NAME}-binary-rewrite-run
COMMAND ${COMMAND_PREFIX}
$<TARGET_FILE_DIR:${TEST_TARGET}>/${TEST_TARGET}.inst ${TEST_RUN_ARGS}
COMMAND ${COMMAND_PREFIX} $<TARGET_FILE_DIR:${TEST_TARGET}>/${TEST_NAME}.inst
${TEST_RUN_ARGS}
WORKING_DIRECTORY $<TARGET_FILE_DIR:${TEST_TARGET}>)
add_test(
NAME ${TEST_NAME}-binary-rewrite-run-sampling
COMMAND ${COMMAND_PREFIX}
$<TARGET_FILE_DIR:${TEST_TARGET}>/${TEST_TARGET}.samp ${TEST_RUN_ARGS}
COMMAND ${COMMAND_PREFIX} $<TARGET_FILE_DIR:${TEST_TARGET}>/${TEST_NAME}.samp
${TEST_RUN_ARGS}
WORKING_DIRECTORY $<TARGET_FILE_DIR:${TEST_TARGET}>)
add_test(
@@ -127,10 +162,8 @@ function(OMNITRACE_ADD_TEST)
add_test(
NAME ${TEST_NAME}-runtime-instrument-sampling
COMMAND
$<TARGET_FILE:omnitrace-exe> -M sampling --env
OMNITRACE_OUTPUT_PREFIX=sampling- ${TEST_RUNTIME_ARGS} --
$<TARGET_FILE:${TEST_TARGET}> ${TEST_RUN_ARGS}
COMMAND $<TARGET_FILE:omnitrace-exe> -M sampling ${TEST_RUNTIME_ARGS} --
$<TARGET_FILE:${TEST_TARGET}> ${TEST_RUN_ARGS}
WORKING_DIRECTORY $<TARGET_FILE_DIR:${TEST_TARGET}>)
set_tests_properties(${TEST_NAME}-binary-rewrite-run
@@ -143,36 +176,52 @@ function(OMNITRACE_ADD_TEST)
_TEST
baseline binary-rewrite binary-rewrite-run binary-rewrite-sampling
binary-rewrite-run-sampling runtime-instrument runtime-instrument-sampling)
string(REPLACE "-run-" "-" _PREFIX "${TEST_NAME}-${_TEST}-")
string(REPLACE "-run-" "-" _prefix "${TEST_NAME}-${_TEST}-")
set(_environ "${TEST_ENVIRONMENT}")
set(_labels "${_TEST}")
set(_timeout ${TEST_REWRITE_TIMEOUT})
list(APPEND _environ "OMNITRACE_OUTPUT_PATH=omnitrace-tests-output"
"OMNITRACE_OUTPUT_PREFIX=${_PREFIX}")
set(_LABELS "${_TEST}")
string(REPLACE "-run" "" _LABELS "${_TEST}")
string(REPLACE "-sampling" ";sampling" _LABELS "${_LABELS}")
set_tests_properties(
${TEST_NAME}-${_TEST} PROPERTIES ENVIRONMENT "${_environ}" TIMEOUT 600
LABELS "${_LABELS};${TEST_LABELS}")
"OMNITRACE_OUTPUT_PREFIX=${_prefix}")
string(REPLACE "-run" "" _labels "${_TEST}")
string(REPLACE "-sampling" ";sampling" _labels "${_labels}")
if("${_TEST}" MATCHES "runtime-instrument")
set(_timeout ${TEST_RUNTIME_TIMEOUT})
endif()
set(_props)
if("${_TEST}" MATCHES "run|baseline")
set(_props ${TEST_PROPERTIES})
if(NOT "RUN_SERIAL" IN_LIST _props)
list(APPEND _props RUN_SERIAL ON)
endif()
endif()
if(TEST ${TEST_NAME}-${_TEST})
set_tests_properties(
${TEST_NAME}-${_TEST}
PROPERTIES ENVIRONMENT "${_environ}" TIMEOUT ${_timeout} LABELS
"${_labels};${TEST_LABELS}" ${_props})
endif()
endforeach()
endif()
endfunction()
# -------------------------------------------------------------------------------------- #
omnitrace_add_test(
NAME transpose
TARGET transpose
MPI ${TRANSPOSE_USE_MPI}
NUM_PROCS ${NUM_PROCS}
REWRITE_ARGS -e -v 1
REWRITE_ARGS -e -v 2
RUNTIME_ARGS -e -v 1 --label file line return args
RUN_ARGS ""
ENVIRONMENT "${_base_environment};OMNITRACE_CRITICAL_TRACE=ON")
ENVIRONMENT "${_base_environment};OMNITRACE_CRITICAL_TRACE=OFF")
omnitrace_add_test(
NAME transpose-no-save-fpr
TARGET transpose
MPI ${TRANSPOSE_USE_MPI}
NUM_PROCS ${NUM_PROCS}
REWRITE_ARGS -e -v 1 --dyninst-options DelayedParsing TypeChecking
REWRITE_ARGS -e -v 2 --dyninst-options DelayedParsing TypeChecking
RUNTIME_ARGS
-e
-v
@@ -191,7 +240,7 @@ omnitrace_add_test(
omnitrace_add_test(
NAME parallel-overhead
TARGET parallel-overhead
REWRITE_ARGS -e -v 1 --min-address-range-loop=64
REWRITE_ARGS -e -v 2 --min-address-range-loop=64
RUNTIME_ARGS
-e
-v
@@ -202,13 +251,13 @@ omnitrace_add_test(
line
return
args
RUN_ARGS 10 ${NUM_THREADS}
RUN_ARGS 10 ${NUM_THREADS} 1000
ENVIRONMENT "${_base_environment};OMNITRACE_CRITICAL_TRACE=OFF")
omnitrace_add_test(
NAME parallel-overhead-no-save-fpr
TARGET parallel-overhead
REWRITE_ARGS -e -v 1 --min-address-range-loop=32 --dyninst-options DelayedParsing
REWRITE_ARGS -e -v 2 --min-address-range-loop=32 --dyninst-options DelayedParsing
TypeChecking
RUNTIME_ARGS
-e
@@ -223,7 +272,7 @@ omnitrace_add_test(
--dyninst-options
DelayedParsing
TypeChecking
RUN_ARGS 10 ${NUM_THREADS}
RUN_ARGS 10 ${NUM_THREADS} 1000
ENVIRONMENT "${_fast_environment}")
omnitrace_add_test(
@@ -231,7 +280,7 @@ omnitrace_add_test(
TARGET lulesh
MPI ${LULESH_USE_MPI}
NUM_PROCS 8
REWRITE_ARGS -e -v 1
REWRITE_ARGS -e -v 2
RUNTIME_ARGS
-e
-v
@@ -242,6 +291,69 @@ omnitrace_add_test(
return
args
-ME
[==['lib(gomp|m-)']==]
[==[lib(gomp|m-)]==]
RUN_ARGS -i 10 -s 20 -p
ENVIRONMENT "${_base_environment};OMNITRACE_CRITICAL_TRACE=OFF")
ENVIRONMENT
"${_base_environment};OMNITRACE_CRITICAL_TRACE=OFF;OMNITRACE_USE_KOKKOSP=OFF")
omnitrace_add_test(
NAME lulesh-kokkosp
TARGET lulesh
MPI ${LULESH_USE_MPI}
NUM_PROCS 8
REWRITE_ARGS -e -v 2
RUNTIME_ARGS
-e
-v
1
--label
file
line
return
args
-ME
[==[lib(gomp|m-)]==]
RUN_ARGS -i 10 -s 20 -p
ENVIRONMENT
"${_base_environment};OMNITRACE_CRITICAL_TRACE=OFF;OMNITRACE_USE_KOKKOSP=ON")
omnitrace_add_test(
NAME lulesh-perfetto
TARGET lulesh
MPI ${LULESH_USE_MPI}
NUM_PROCS 8
REWRITE_ARGS -e -v 2
RUNTIME_ARGS
-e
-v
1
-l
--dynamic-callsites
--traps
--allow-overlapping
-ME
[==[libgomp]==]
RUN_ARGS -i 10 -s 20 -p
ENVIRONMENT
"${_perfetto_environment};OMNITRACE_CRITICAL_TRACE=OFF;OMNITRACE_USE_KOKKOSP=OFF")
omnitrace_add_test(
NAME lulesh-timemory
TARGET lulesh
MPI ${LULESH_USE_MPI}
NUM_PROCS 8
REWRITE_ARGS -e -v 2 -l --dynamic-callsites --traps --allow-overlapping
RUNTIME_ARGS
-e
-v
1
-l
--dynamic-callsites
-ME
[==[libgomp]==]
-d
wall_clock
peak_rss
RUN_ARGS -i 10 -s 20 -p
ENVIRONMENT
"${_timemory_environment};OMNITRACE_CRITICAL_TRACE=OFF;OMNITRACE_USE_KOKKOSP=OFF")