AI/rocm-systems
2
0
Derivar 0
Código Questões Pedidos de integração Operações Pacotes Planeamentos Lançamentos Wiki Trabalho

[rocprofv3] SQLite3 database output (rocpd) support + rocprofiler-sdk-rocpd (#403)

Ver a proveniência 
* [rocprofv3] rocpd SQLite3 database output support

* Move counters xml and yaml to source/share/rocprofiler-sdk

- more representative of install hierarchy

* Add share/rocprofiler-sdk/rocpd SQL files

* Experimental rocprofiler-sdk SQL API

* rocprofv3 default output format is rocpd

* Fix rocpd event ids for counter collection w/o kernel dispatch

* Remove fktable entries from rocpd_tables.sql

* Fix rocpd schema path

* Fix install component for roctx python bindings

* rocprofiler-sdk-rocpd

- create include/rocprofiler-sdk-rocpd
- create rocprofiler-sdk-rocpd library, package, etc.
- default all "guid" fields to "{{guid}}" in tables
- remove "{{view_uuid}}" support (always unused)

* Migrate rocprofv3 to use rocprofiler-sdk-rocpd

* Fix missing foreign key reference

* Revert change

* Fix cmake comment

* Fix maybe-uninitialized compiler warning

* Fix maybe-uninitialized compiler warning

* Add logging to rocpd_sql_load_schema

* Improve string sanitization when inserting json strings

* Initialize rocpd logging on rocprofiler-sdk-rocpd library load

* Revert lib/output/generatePerfetto.cpp changes

* [temporary] Tweak rocprofv3-test-list-avail-trace-execute test log level

* Update get_install_path for lib/rocprofiler-sdk-rocpd/sql.cpp

- try to resolve issues on RHEL/SLES for dladdr

* Update lib/common/logging.cpp

- enable environ overrides

* dlsym for rocpd_sql_load_schema

* Make dl_info.dli_fname lexically normal

* Implement node_info alternatives if /etc/machine-id does not exist

* Misc include fixes

* SHA256 and UUIDv7 support

* Implement UUIDv7 in generateRocpd.cpp

* Support push/pop environment variables

* Minor tweak

* Fix glog segfaults when unsetting glog env

* Updated CHANGELOG

* Updates tests/pytest-packages

- rocpd_reader.py: RocpdReader

* Update tests / marker_views.sql

- add test_rocpd_data

* Update rocpd_tables.sql

- Use AUTOINCREMENT
- insert "uuid" and "guid" into rocpd_metadata

* Minor updates to generateRocpd.cpp

- don't quote GUID
- use sqlite3_open_v2
- use sqlite3_close_v2

* Update execute_raw_sql_statements_impl

- uses sqlite3_last_insert_rowid for autoincrement

* Update SQL deferred_transaction

- CI check for nullptr to connection

* Apply suggestions from code review

Co-authored-by: Welton, Benjamin <Benjamin.Welton@amd.com>

* Code review updates

- formatting
- replace if with switch
- remove loop for {{uuid}}

* Fix pmc_groups handling in rocprofv3

* Address code review feedback

- Include rocm_version in rocprofv3 version info
- Note `--version` option for `rocprofv3` in CHANGELOG.md
- remove commented out code

* Fix packaging dependencies

* Fix install package step of CI workflow

* Fix install package step of CI workflow

---------

Co-authored-by: Jonathan R. Madsen <jonathanrmadsen@gmail.com>
Co-authored-by: Welton, Benjamin <Benjamin.Welton@amd.com>
Este cometimento está contido em:
Madsen, Jonathan
2025-05-30 00:13:19 -05:00
cometido por GitHub
ascendente dbb2e52216
cometimento 7afedc63be
81 ficheiros modificados com 7725 adições e 993 eliminações
Descarregar ficheiro patch Descarregar ficheiro diff Expandir todos os ficheiros Colapsar todos os ficheiros
tests/rocprofv3/counter-collection/input3/CMakeLists.txt
+2 -2
Ver ficheiro
@@ -19,14 +19,14 @@ add_test(
COMMAND
$<TARGET_FILE:rocprofiler-sdk::rocprofv3> -i
${CMAKE_CURRENT_BINARY_DIR}/input.json -d ${CMAKE_CURRENT_BINARY_DIR}/%argt%-cc
-o out_json -- $<TARGET_FILE:simple-transpose>)
-o out_json -f csv -- $<TARGET_FILE:simple-transpose>)
add_test(
NAME rocprofv3-test-counter-collection-yaml-pmc1-execute
COMMAND
$<TARGET_FILE:rocprofiler-sdk::rocprofv3> -i
${CMAKE_CURRENT_BINARY_DIR}/input.yml -d ${CMAKE_CURRENT_BINARY_DIR}/%argt%-cc -o
out_yaml -- $<TARGET_FILE:simple-transpose>)
out_yaml -f csv -- $<TARGET_FILE:simple-transpose>)
string(REPLACE "LD_PRELOAD=" "ROCPROF_PRELOAD=" PRELOAD_ENV
"${ROCPROFILER_MEMCHECK_PRELOAD_ENV}")
tests/rocprofv3/counter-collection/list_metrics/CMakeLists.txt
+2 -2
Ver ficheiro
@@ -30,12 +30,12 @@ add_test(
COMMAND
$<TARGET_FILE:rocprofiler-sdk::rocprofv3> -d
${CMAKE_CURRENT_BINARY_DIR}/out_counter_collection_2 -o metrics --list-avail
--sys-trace -- $<TARGET_FILE:transpose>)
--sys-trace --log-level warning -- $<TARGET_FILE:transpose>)
string(REPLACE "LD_PRELOAD=" "ROCPROF_PRELOAD=" PRELOAD_ENV
"${ROCPROFILER_MEMCHECK_PRELOAD_ENV}")
set(cc-env-list-metrics "${PRELOAD_ENV}")
set(cc-env-list-metrics "${PRELOAD_ENV}" "ROCPD_LOG_LEVEL=trace")
set_tests_properties(
rocprofv3-test-list-avail-execute
tests/rocprofv3/counter-collection/multiplex/CMakeLists.txt
+2 -2
Ver ficheiro
@@ -41,14 +41,14 @@ add_test(
COMMAND
$<TARGET_FILE:rocprofiler-sdk::rocprofv3> -i
${CMAKE_CURRENT_BINARY_DIR}/input.json -d ${CMAKE_CURRENT_BINARY_DIR}/%argt%-cc
-o out_json -- $<TARGET_FILE:simple-transpose>)
-o out_json -f csv -- $<TARGET_FILE:simple-transpose>)
add_test(
NAME rocprofv3-test-counter-collection-multiple-yaml-execute
COMMAND
$<TARGET_FILE:rocprofiler-sdk::rocprofv3> -i
${CMAKE_CURRENT_BINARY_DIR}/input.yml -d ${CMAKE_CURRENT_BINARY_DIR}/%argt%-cc -o
out_yaml -- $<TARGET_FILE:simple-transpose>)
out_yaml -f csv -- $<TARGET_FILE:simple-transpose>)
string(REPLACE "LD_PRELOAD=" "ROCPROF_PRELOAD=" PRELOAD_ENV
"${ROCPROFILER_MEMCHECK_PRELOAD_ENV}")
tests/rocprofv3/tracing/CMakeLists.txt
+14 -6
Ver ficheiro
@@ -29,7 +29,7 @@ add_test(
COMMAND
$<TARGET_FILE:rocprofiler-sdk::rocprofv3> -M --hsa-trace --kernel-trace
--memory-copy-trace --marker-trace -d ${CMAKE_CURRENT_BINARY_DIR}/%argt%-trace -o
cmdl-input/out --output-format pftrace csv json --log-level ${LOG_LEVEL} --
cmdl-input/out --output-format pftrace csv json rocpd --log-level ${LOG_LEVEL} --
$<TARGET_FILE:simple-transpose>)
set_tests_properties(
@@ -82,7 +82,9 @@ add_test(
--json-input
${CMAKE_CURRENT_BINARY_DIR}/simple-transpose-trace/cmdl-input/out_results.json
--pftrace-input
${CMAKE_CURRENT_BINARY_DIR}/simple-transpose-trace/cmdl-input/out_results.pftrace)
${CMAKE_CURRENT_BINARY_DIR}/simple-transpose-trace/cmdl-input/out_results.pftrace
--rocpd-input
${CMAKE_CURRENT_BINARY_DIR}/simple-transpose-trace/cmdl-input/out_results.db)
add_test(
NAME rocprofv3-test-trace-input-json-validate
@@ -100,9 +102,12 @@ add_test(
--json-input
${CMAKE_CURRENT_BINARY_DIR}/simple-transpose-trace/json-input/out_results.json
--pftrace-input
${CMAKE_CURRENT_BINARY_DIR}/simple-transpose-trace/json-input/out_results.pftrace)
${CMAKE_CURRENT_BINARY_DIR}/simple-transpose-trace/json-input/out_results.pftrace
--rocpd-input
${CMAKE_CURRENT_BINARY_DIR}/simple-transpose-trace/json-input/out_results.db)
set(VALIDATION_FILES
${CMAKE_CURRENT_BINARY_DIR}/simple-transpose-trace/cmdl-input/out_results.db
${CMAKE_CURRENT_BINARY_DIR}/simple-transpose-trace/cmdl-input/out_results.pftrace
${CMAKE_CURRENT_BINARY_DIR}/simple-transpose-trace/cmdl-input/out_results.json
${CMAKE_CURRENT_BINARY_DIR}/simple-transpose-trace/cmdl-input/out_memory_copy_trace.csv
@@ -152,7 +157,7 @@ add_test(
COMMAND
$<TARGET_FILE:rocprofiler-sdk::rocprofv3> --sys-trace -d
${CMAKE_CURRENT_BINARY_DIR}/%argt%-systrace -o cmdl-input/out --output-format
pftrace csv json -- $<TARGET_FILE:simple-transpose>)
pftrace csv json rocpd -- $<TARGET_FILE:simple-transpose>)
set_tests_properties(
rocprofv3-test-systrace-execute
@@ -204,7 +209,8 @@ add_test(
${CMAKE_CURRENT_BINARY_DIR}/simple-transpose-systrace/cmdl-input/out_results.json
--pftrace-input
${CMAKE_CURRENT_BINARY_DIR}/simple-transpose-systrace/cmdl-input/out_results.pftrace
)
--rocpd-input
${CMAKE_CURRENT_BINARY_DIR}/simple-transpose-systrace/cmdl-input/out_results.db)
add_test(
NAME rocprofv3-test-systrace-input-json-validate
@@ -224,9 +230,11 @@ add_test(
${CMAKE_CURRENT_BINARY_DIR}/simple-transpose-systrace/json-input/out_results.json
--pftrace-input
${CMAKE_CURRENT_BINARY_DIR}/simple-transpose-systrace/json-input/out_results.pftrace
)
--rocpd-input
${CMAKE_CURRENT_BINARY_DIR}/simple-transpose-systrace/json-input/out_results.db)
set(SYS_VALIDATION_FILES
${CMAKE_CURRENT_BINARY_DIR}/simple-transpose-systrace/cmdl-input/out_results.db
${CMAKE_CURRENT_BINARY_DIR}/simple-transpose-systrace/cmdl-input/out_results.pftrace
${CMAKE_CURRENT_BINARY_DIR}/simple-transpose-systrace/cmdl-input/out_results.json
${CMAKE_CURRENT_BINARY_DIR}/simple-transpose-systrace/cmdl-input/out_memory_copy_trace.csv
tests/rocprofv3/tracing/conftest.py
+12
Ver ficheiro
@@ -29,6 +29,7 @@ import json
from rocprofiler_sdk.pytest_utils.dotdict import dotdict
from rocprofiler_sdk.pytest_utils import collapse_dict_list
from rocprofiler_sdk.pytest_utils.perfetto_reader import PerfettoReader
from rocprofiler_sdk.pytest_utils.rocpd_reader import RocpdReader
def pytest_addoption(parser):
@@ -72,6 +73,11 @@ def pytest_addoption(parser):
action="store",
help="Path to Perfetto trace file.",
)
parser.addoption(
"--rocpd-input",
action="store",
help="Path to rocpd SQLite3 database file.",
)
@pytest.fixture
@@ -157,3 +163,9 @@ def json_data(request):
def pftrace_data(request):
filename = request.config.getoption("--pftrace-input")
return PerfettoReader(filename).read()[0]
@pytest.fixture
def rocpd_data(request):
filename = request.config.getoption("--rocpd-input")
return RocpdReader(filename).read()[0]
tests/rocprofv3/tracing/input_systrace.json
+2 -1
Ver ficheiro
@@ -6,7 +6,8 @@
"output_format": [
"csv",
"json",
"pftrace"
"pftrace",
"rocpd"
]
}
]
tests/rocprofv3/tracing/input_trace.json
+2 -1
Ver ficheiro
@@ -9,7 +9,8 @@
"output_format": [
"csv",
"json",
"pftrace"
"pftrace",
"rocpd"
]
}
]
tests/rocprofv3/tracing/validate.py
+8
Ver ficheiro
@@ -323,6 +323,14 @@ def test_perfetto_data(pftrace_data, json_data):
)
def test_rocpd_data(rocpd_data, json_data):
import rocprofiler_sdk.tests.rocprofv3 as rocprofv3
rocprofv3.test_rocpd_data(
rocpd_data, json_data, ("hip", "hsa", "marker", "kernel", "memory_copy")
)
if __name__ == "__main__":
exit_code = pytest.main(["-x", __file__] + sys.argv[1:])
sys.exit(exit_code)