e0901eba2876cd3cf8ab085765851ad5da54565e
1 Υποβολές
| Συγγραφέας | SHA1 | Μήνυμα | Ημερομηνία | |
|---|---|---|---|---|
|
|
dff1e9acb5 |
[rocprofv3] rocpd Python package (#384)
* Squashed commit of the following: commit f764eb6f4a45baa25eb8f1b50b1035c84578c200 Author: Jonathan R. Madsen <jonathanrmadsen@gmail.com> Date: Thu May 1 09:09:37 2025 -0500 Misc post rebase fixes commit 447418b0765819eb2fb5c8b5c3ca9128a091d37e Author: Jonathan R. Madsen <jonathanrmadsen@gmail.com> Date: Thu May 1 07:22:07 2025 -0500 Formatting commit 975661f5e498cde99f8c3ce5486c47db03856d1b Author: Young Hui <young.hui@amd.com> Date: Wed Apr 30 21:19:30 2025 -0400 Reorganize rocpd command line and grouped Required Arguments together - had to add --input to each output.py file again since it was moved out of output_config.py - ran formatter commit 9322328611a332c3979f040b652a9e9a9482200e Author: acanadas <acanadas@amd.com> Date: Wed Apr 30 17:00:58 2025 +0000 corrected indices on some operation, and kind for kernel dispatch commit 6c146cd0c508dca6f2453e3844e09e1ed3f9978a Author: acanadas <acanadas@amd.com> Date: Wed Apr 30 16:01:26 2025 +0000 some corrections on pf trace output: added categories commit 4e02d3f8617324c95e4a449243ab9ab3f4695471 Author: acanadas <acanadas@amd.com> Date: Wed Apr 30 14:35:19 2025 +0000 fixed perfetto cpp with adding stack id and parent stack id to views:tests commit d7efd9334361cd7d6a842d083a3f8ca51efe72d3 Author: acanadas <acanadas@amd.com> Date: Wed Apr 30 14:16:32 2025 +0000 fixed perfetto cpp with adding stack id and parent stack id to views commit cdd0e2ec0788d44fdf2d5833822e055c43cddec6 Author: acanadas <acanadas@amd.com> Date: Wed Apr 30 09:47:07 2025 -0500 restore output_config, add output_file arg for generate csv commit 5f9b7d93dcbefd55e0ba6e2674602e809aa61632 Author: JIn Tao <jintao12@amd.com> Date: Wed Apr 30 09:30:32 2025 +0000 add ROCDecode and ROCJpeg API calls commit 7724de1263c5f960cc64c5b0e7afb3834d797f87 Author: JIn Tao <jintao12@amd.com> Date: Wed Apr 30 09:05:14 2025 +0000 Json output: add counters_collection commit a13930d6d2b87605ca1ece58291172f79d81d91f Author: JIn Tao <jintao12@amd.com> Date: Wed Apr 30 09:01:17 2025 +0000 Json output: add scratch_memory commit 54e62e25c6d89e718324ce3bc51eb80c25756c48 Author: JIn Tao <jintao12@amd.com> Date: Wed Apr 30 08:00:08 2025 +0000 Json output: add marker_api commit ab920196c7ddb68a9a1fdc121f43528e513d5a67 Author: root <root@smc300x-ccs-aus-GPUF2C5.cs-aus.dcgpu> Date: Tue Apr 29 10:48:53 2025 -0500 csv refactor, fix output-format argument for script commit e033d18356f397e3a684e255dcffd0c0d64ec19e Author: JIn Tao <jintao12@amd.com> Date: Tue Apr 29 11:30:16 2025 +0000 minor revison commit 748f6754ac0238eca63bb12b26f62b514de65a0d Author: Jin Tao <jintao12@amd.com> Date: Tue Apr 29 10:09:54 2025 +0000 Json output: complete structures commit 52c8d77e0eeb8dca7476814ff03b5cdf88055fd6 Author: oshkarav_amdeng <oshkarav@amd.com> Date: Mon Apr 28 15:44:47 2025 +0000 forced tests upd commit 7fabc80d3b8db7d137b05a958c633ad5bf8dbae9 Author: oshkarav_amdeng <oshkarav@amd.com> Date: Mon Apr 28 15:43:34 2025 +0000 fixed the relative-type index issue (missing load) for agent info and related parameter adjustmets stuff for python commit f8f5bffc010ad6d43a9f8fee90a79e4342fb9562 Author: oshkarav_amdeng <oshkarav@amd.com> Date: Mon Apr 28 12:28:35 2025 +0000 added 3 convenience views, some refactoring and fixing kernel-rename option commit 831cd336115153d1e73f01c9120a67c904478f89 Author: Jin Tao <jintao12@amd.com> Date: Mon Apr 28 14:45:31 2025 +0000 Json output: add kernel_dispatch, hip_api, hsa_api commit 4c414a1abce51fbdd6d5856b2e36e6272279c671 Author: Jin Tao <jintao12@amd.com> Date: Fri Apr 25 13:30:01 2025 +0000 optimize the json output code, certain problems need to be done, e.g., empty counters and strings commit ceecd7cc5b81f014766199c0a57645386ade23dd Author: oshkarav_amdeng <oshkarav@amd.com> Date: Fri Apr 25 10:51:16 2025 +0000 removed unused variable session from write_otf commit 29fdb2db4fe0cc930cd6b3172092604ee5409242 Author: oshkarav_amdeng <oshkarav@amd.com> Date: Fri Apr 25 10:14:48 2025 +0000 added tests for generated csv and otf2 commit 5091d2d51e7e4d68fcdc95a97a82a0df41f28350 Author: oshkarav_amdeng <oshkarav@amd.com> Date: Fri Apr 25 10:12:30 2025 +0000 run formating from command line commit abbb7637b1704ea904540c5ff717102bf450c76d Author: oshkarav_amdeng <oshkarav@amd.com> Date: Fri Apr 25 10:09:25 2025 +0000 added check up if cvs os not broken after other chnages commit 9ff614d6d8e87fc3647d8f3b0120425c24213f3b Author: oshkarav_amdeng <oshkarav@amd.com> Date: Thu Apr 24 15:09:40 2025 +0000 updated new csv.cpp and otf2.cpp to fit the string_view fix commit e94ea0f3668c9b972f2dd4144cb4152c1b202f93 Author: oshkarav_amdeng <oshkarav@amd.com> Date: Thu Apr 24 14:51:55 2025 +0000 fixed string view reporting which works at least forgenerateOTF2.cpp commit 5c5ea532279fba0b7ef5abcd1916d20d0b7fb7b8 Author: Jin Tao <jintao12@amd.com> Date: Thu Apr 24 14:32:10 2025 +0000 Json output: add strings.marker_api commit d28d1c18c9693421f7676d6de82c2c20af11eaa0 Author: oshkarav_amdeng <oshkarav@amd.com> Date: Thu Apr 24 13:10:13 2025 +0000 small upd on cmake for tests commit 325cb3719517ad514291ab620dd85fb04daeb906 Author: oshkarav_amdeng <oshkarav@amd.com> Date: Thu Apr 24 13:08:58 2025 +0000 fixed abs_index for connecting data and handles for otf2 location reporting commit e9a648ade545795646f6aca61fdbece5a39fea5c Author: oshkarav_amdeng <oshkarav@amd.com> Date: Tue Apr 22 14:00:12 2025 +0000 commit after fixing warnings, executabel name, validate works commit 1cd2d63501b8f951996c9dfcd1d0af6d6f16c006 Author: Jin Tao <jintao12@amd.com> Date: Thu Apr 24 12:38:03 2025 +0000 Json output: add marker_api in buffer_records commit f01ab23d2f8f6c524568e2c453fe26a2e4320a1c Author: Young Hui <young.hui@amd.com> Date: Wed Apr 23 21:37:36 2025 -0400 Add python binding for agent-index-value to output_config - command line passes correctly for csv, perfetto needs to be fixed commit a92dd0c060dd398db365ec37af905dcca25c8a7e Author: acanadas <acanadas@amd.com> Date: Wed Apr 23 09:08:43 2025 -0500 provisional fix in json.cpp - For now using absolut index for agents commit fddacacbb54f5678a40d552ec8a3a2f9de65381b Author: acanadas <acanadas@amd.com> Date: Wed Apr 23 08:55:49 2025 -0500 adding agent indexes (abs, log, type) to types for agent_index_value options - Fixing agent_id populated in rocpd_memory_allocate table commit 3b0414ba5271d25996564eddc6d757b1afc637af Author: Jin Tao <jintao12@amd.com> Date: Wed Apr 23 11:22:59 2025 +0000 minor revision regarding json output commit a84bc3d0f7dd9bf1014d024ef15eeb7c7ec990c5 Author: Jin Tao <jintao12@amd.com> Date: Wed Apr 23 11:16:48 2025 +0000 add json output commit e6c0dd98de0b5f24492ac4396cf8d59bd20d58ad Author: Young Hui <young.hui@amd.com> Date: Tue Apr 22 17:22:45 2025 -0400 Add rocpd commandline input param file check, to ensure DB exists - added OTF2 script - added placeholder JSON script commit 1d482257b8f23bf4d64d57d8bd36775b38254026 Author: Young Hui <young.hui@amd.com> Date: Mon Apr 21 12:47:52 2025 -0400 Clean up some rocpd python files - removed some unused files - cleanup __main__.py imports and duplicate main commit c15af2aac9935ffce92d9d6ce35ab5e9eabed57c Author: Young Hui <young.hui@amd.com> Date: Thu Apr 17 18:40:13 2025 -0400 Add rocpd command line support - right now pftrace and csv are supported - also removed some otf2 test files, to fix cmake configure - formatting edits commit 10bee3bcf496edd8e1ad9521498c926915a33f07 Author: oshkarav_amdeng <oshkarav@amd.com> Date: Thu Apr 17 15:17:15 2025 +0000 experimented with roctxA, formatted commit 11ff50882bcbedbe516c6461c5f8d65e38d0aae5 Author: oshkarav_amdeng <oshkarav@amd.com> Date: Thu Apr 17 15:17:05 2025 +0000 experimented with roctxA, formatted commit 7cc87cbf56f6d7117df10dc2cfe45174bedff22a Author: oshkarav_amdeng <oshkarav@amd.com> Date: Thu Apr 17 14:16:16 2025 +0000 small refactoring on api-calls preprocessing commit 421bb11d5b97a4b888c0f9a0b46fca229e4abf25 Author: oshkarav_amdeng <oshkarav@amd.com> Date: Thu Apr 17 14:03:39 2025 +0000 added rccl, rocdecode, rocjpeg, corrected markers commit 71c1122ec001ce2548aa1e6d7b0d4bbd5ac16d79 Author: oshkarav_amdeng <oshkarav@amd.com> Date: Wed Apr 16 15:59:39 2025 +0000 integration tests for otf2 generation will stay till otf2 gen becomes stable commit b8ff32bb269a4efec001804eb0064a8c6c7f8f6d Author: oshkarav_amdeng <oshkarav@amd.com> Date: Wed Apr 16 15:07:19 2025 +0000 intermediate local commit of functioningotf2 output after refactoring writer code commit 4d6140fbad00a713aed20b72d41fa62219f9aed7 Author: oshkarav_amdeng <oshkarav@amd.com> Date: Wed Apr 16 09:18:40 2025 +0000 intermediate local commit of functioningotf2 output commit 96f40ebce93ff3a27c01d5e5267eda67c3ab68ec Author: oshkarav_amdeng <oshkarav@amd.com> Date: Fri Apr 11 17:58:24 2025 +0000 first working commit on generating otf2 commit 75ddceb4bd3dc6c32cd8a60450bd1c70bf4d3193 Author: acanadas <acanadas@amd.com> Date: Wed Apr 16 11:24:30 2025 +0000 Add CSV export functionality to ROCm Profiler Data Implement complete CSV export pipeline with the following components: - Add write_csv function to libpyrocpd.cpp for core CSV generation - Create csv.hpp and csv.cpp for CSV formatting and management - Implement stats_summary.hpp and stats_summary.cpp for performance metrics - Add comprehensive type definitions for markers, counters and statistics - Create SQL views in schema_data for efficient data extraction - Add csv.py module similar to pftrace.py for Python API consistency - Implement convert.py script with multiple format support (CSV, Perfetto) This change enables exporting profiling data in CSV format for easier analysis and integration with external tools. commit 953223e32faa862e79bd1f61e28a55874efa0589 Author: Jonathan R. Madsen <jonathanrmadsen@gmail.com> Date: Tue Apr 1 19:58:18 2025 -0500 [output] Update generateRocpd.cpp to new schema + misc - support guid, {{upid}}, {{view_upid}}, etc. - improve env support for MPI in format_path commit 54bd3b0def48d91a81045676fb2f5f549b813880 Author: Jonathan R. Madsen <jonathanrmadsen@gmail.com> Date: Tue Apr 1 19:54:41 2025 -0500 [rocpd] Updates + cleanup - remove stale {autograd,call_stacks,strings,subclass}.py - reorganize the SQL schema files - add source/functions.{hpp,cpp} - custom SQL function - update interop for defining functions - misc improvements to "write_perfetto" function - added rocpd.pftrace - added rocpd.output_config commit 32f668b019c961f0797eec9f613cf5dfea0aa377 Author: Jonathan R. Madsen <jonathanrmadsen@gmail.com> Date: Tue Apr 1 14:35:31 2025 -0500 [common] md5sum calculator commit b6ae75ba270ea92661f6cfe75647531a4d6202f3 Author: Jonathan R. Madsen <jonathanrmadsen@gmail.com> Date: Sun Mar 30 01:57:50 2025 -0500 Optimize sql_generator when query requires ORDER BY commit 51d6f33b0b1f80dd09de70c91592f928e31a730f Author: Jonathan R. Madsen <jonathanrmadsen@gmail.com> Date: Sat Mar 29 01:49:52 2025 -0500 Minimal support for merged pftrace from multiple database commit 90c4add9001cad2af85d14783ac1fb35c89c7770 Author: Jonathan R. Madsen <jonathanrmadsen@gmail.com> Date: Sat Mar 29 00:46:59 2025 -0500 Update cereal submodule - fix recursive include commit a5d75dcb5de9c0667af03ec7a34ac484ff864bac Author: Ian Trowbridge <Ian.Trowbridge@amd.com> Date: Fri Mar 28 14:30:32 2025 -0500 Formatting commit 7345810d5ea5d76b9a4ed9bd548399cb8df1feda Author: Ian Trowbridge <Ian.Trowbridge@amd.com> Date: Fri Mar 28 13:44:24 2025 -0500 Fixed interpolation bug commit 289739669a84ac83b920f417ef94310dd9ee40c6 Author: Ian Trowbridge <Ian.Trowbridge@amd.com> Date: Fri Mar 28 13:32:29 2025 -0500 Initial memory copy implementation created commit 91416d784bae05984b8e9670d6cd22231fbc8bed Author: Ian Trowbridge <Ian.Trowbridge@amd.com> Date: Fri Mar 28 13:12:43 2025 -0500 Added memory allocation counter tracks, removed midpoint interpolation temporarily due to strange outputs commit 7e733e393d06fecc198ae4f7891edecb90882136 Author: Jonathan R. Madsen <jonathanrmadsen@gmail.com> Date: Fri Mar 28 13:22:34 2025 -0500 Support multiple database connections commit e82d7b3ee24a855258dee50ea3a4ee8e52f70509 Author: Mark Meserve <Mark.Meserve@amd.com> Date: Fri Mar 28 17:44:05 2025 +0000 fix tracing_session init commit d29ac6270e7466794ca43ffdf061b7514a29ad94 Author: Mark Meserve <Mark.Meserve@amd.com> Date: Fri Mar 28 17:00:08 2025 +0000 seperate perfetto init to RAII class commit 0538261898fdc83a05dc3835ec07d90c4b8dd937 Author: Ian Trowbridge <Ian.Trowbridge@amd.com> Date: Fri Mar 28 11:03:45 2025 -0500 Kernel trace in perfetto now functional commit 3eb63de679e8dfbc3bc551302ca097356f17de7d Author: Ian Trowbridge <Ian.Trowbridge@amd.com> Date: Thu Mar 27 17:43:43 2025 -0500 Added memory copy and allocation types, need to test kernel perfetto output commit c70ce8340a5e4603bed27d6f3f0d95bc77aad196 Author: Jonathan R. Madsen <jonathanrmadsen@gmail.com> Date: Thu Mar 27 15:16:18 2025 -0500 Misc libpyrocpd updates - support conditions on read commit 189226fb3aeaf3485137335392b271f4f1271040 Author: Ian Trowbridge <Ian.Trowbridge@amd.com> Date: Thu Mar 27 14:43:25 2025 -0500 kenrel_dispatch type added, added load function for serialize commit 07f6af65733f34c067365c94500b03a9ffff6b6b Author: Jonathan R. Madsen <jonathanrmadsen@gmail.com> Date: Thu Mar 27 14:08:30 2025 -0500 ROCPROFILER_BUILD_DEPRECATED_WARNINGS OFF by default commit b3df97af8fee651d20030dbfc8d6c635774030a7 Author: Jonathan R. Madsen <jonathanrmadsen@gmail.com> Date: Thu Mar 27 14:06:42 2025 -0500 Fix include commit 49baef7d173385154d346120313e6c9511665b68 Author: Jonathan R. Madsen <jonathanrmadsen@gmail.com> Date: Thu Mar 27 13:16:50 2025 -0500 Python interface improvements commit 7d614ed3ab07836c420e216261e80e0b629739a4 Author: Jonathan R. Madsen <jonathanrmadsen@gmail.com> Date: Thu Mar 27 13:16:35 2025 -0500 Output keys: support {...} format in addition to %...% format commit afdb63a0814f0954dfb7500abe5c95fbacdddbc2 Author: Jonathan R. Madsen <jonathanrmadsen@gmail.com> Date: Thu Mar 27 00:43:50 2025 -0500 [Release] Replace implementation usage - rocprofiler_record_dimension_info_t -> rocprofiler_counter_record_dimension_info_t commit d60ecf99334d96a75b08f99d7a5d8556588258e3 Author: Jonathan R. Madsen <jonathanrmadsen@gmail.com> Date: Thu Mar 27 00:43:04 2025 -0500 [CMake] ROCPROFILER_BUILD_DEPRECATED_WARNINGS option commit a89de9f205d500ffc9fdbef400b8b712b167782b Author: Jonathan R. Madsen <jonathanrmadsen@gmail.com> Date: Thu Mar 27 00:30:52 2025 -0500 [python rocpd] bindings updates - read_{code_objects,kernel_symbols,nodes,processes,threads} - support writing perfetto output for regions - support fallback casting of python object - defined various data types for python commit bb048b42f5828ce742947e1d3b72a35c578c0b0c Author: Jonathan R. Madsen <jonathanrmadsen@gmail.com> Date: Wed Mar 26 22:18:56 2025 -0500 [SDK] hip stream tracing update - disable hip stream tracing support for HIP compiler API functions - add "stream_value" field to rocprofiler_callback_tracing_hip_stream_data_t - data type: rocprofiler_address_t commit 392a12b20ee795ab025097066444504aac3ddd88 Author: Jonathan R. Madsen <jonathanrmadsen@gmail.com> Date: Wed Mar 26 22:02:22 2025 -0500 [output library] generateRocpd.cpp updates - generic functions for accessing fields which may/may not exist - simple timer logging - populate parent_stack_id - misc cleanup commit 86ae21d178a0c52ec47e27414b63edbd2a62a94d Author: Jonathan R. Madsen <jonathanrmadsen@gmail.com> Date: Wed Mar 26 21:46:34 2025 -0500 [public API] update cxx/serialization/load.hpp - ROCPROFILER_SDK_CXX_SERIALIZATION_LOAD_DEBUG - load definitions - rocprofiler_address_t - rocprofiler_callback_tracing_code_object_load_data_t - rocprofiler_callback_tracing_code_object_kernel_symbol_register_data_t commit 4ba6232c23296791df484d47db8268e4bc997c0d Author: Jonathan R. Madsen <jonathanrmadsen@gmail.com> Date: Wed Mar 26 21:44:52 2025 -0500 [public API] update cxx/perfetto.hpp - function: get_perfetto_category for callback tracing and buffer tracing kinds commit 4c0fcf4395f6c337cf3b955ec45b0567f9b3a477 Author: Jonathan R. Madsen <jonathanrmadsen@gmail.com> Date: Wed Mar 26 21:26:32 2025 -0500 [python] Rename rocpd/schema_data/* files - {tables,views,indexes}.sql commit 47b862ece9ed3f16d7a5ecd6b632f13b0086bc01 Author: Jonathan R. Madsen <jonathanrmadsen@gmail.com> Date: Wed Mar 26 21:23:35 2025 -0500 [external] cleanup external/CMakeLists.txt commit 4e2a71db9b224b11f4144104b4569a5efb45b6c2 Author: Jonathan R. Madsen <jonathanrmadsen@gmail.com> Date: Wed Mar 26 21:23:05 2025 -0500 [samples] Minor tweak to counter collection sample app commit c62dd55c5e82b4105bdce374dc45c6adf65a0cc4 Author: Jonathan R. Madsen <jonathanrmadsen@gmail.com> Date: Wed Mar 26 21:22:08 2025 -0500 [tool library] Rework kernel rename and stream id data - simplify and improve memory management - support stream information for memory allocation commit 2521d499664bb01c0e2f9f9454b5da5c38b29cfc Author: Jonathan R. Madsen <jonathanrmadsen@gmail.com> Date: Wed Mar 26 21:18:46 2025 -0500 [output library] Misc reorg and updates - dropped "with_stream" from various data types - add stream support to memory allocation records - generator<T> base class resize function - serialization load functions for kernel symbols, node_info - reorganization of SQL code - cereal::SQLite3InputArchive commit 70a76d6352dca84241f1749da660f8af8e89c469 Author: Jonathan R. Madsen <jonathanrmadsen@gmail.com> Date: Mon Mar 24 03:52:58 2025 -0500 Misc fixes after rebase commit 4811201c3e07ac8b5b4edc4028df9cdbc3481bd1 Author: Jonathan R. Madsen <jonathanrmadsen@gmail.com> Date: Thu Nov 14 22:30:43 2024 -0600 Python Bindings Fix rocpd graph support - remove uniqueness constraint from subclass - replace assumption of "graphExec" with "pGraphExec" in the parsed API arguments pybind11 submodule Initial roctx python bindings Fix testing Add rest of python bindings for ROCTX Add decorators and context manager for roctx Fix: restore rocprofiler_reset_python3_cache from CMakeLists work in prorgess! added stub test for pybind added subdirectory for pybind-test Fixing CMakeLists for pybind-test Pybind-test - Move simple-transpose-pybind.py to tests/bin - Execute test in CMakeLists for pybind-test Fix python format Update rocprofv3 python bindings testing - move to tests/rocprofv3/python-bindings - remove misc code - add marker.py Update roctx python bindings Update rocprofv3 python bindings test validation Update rocprofv3 python bindings test validation - Add checks for marker_api in json format - Unify checks for csv and json patch removing tmp trace_processor_python_api file after validate is run, ptherwise it blocks the work of others correction of previous commit: correct patch removed patch because the santizing will be added to perfetto reader perf reader: removing tmp user-owned file after the data are read robust handling of removing tmp file via a context minor clean up perfetto reader, fixing hip-in-libraries-test minor clean up perfetto reader, fixing hip-in-libraries-test Fix generateRocpd for older compilers Formatting Remove rocpd-db-to-chrome-tracing console script via pip Fix PYTHONPATH for rocprofv3-trace-roctx-python-bindings-* replaced context by PerfettoReader destructor to remove tmp perfetto dir after the test run formatting python formatted cmake for pybind-tests Update contexts and decorators roctx python bindings added stub jypiter notebook for python analysis formatting notebook Update jypiter notebook for python analysis - Create summary tables as views : memory_copy, kernel_dispatch, hip_api upd the generate_rocpd.py by schme reading and rocpd_node table and queue to rocpd_kernelapi comments to discuss on generateJson and generateRocpd.cpp comments to discuss on generateJson and generateRocpd.cpp remove comments because they block pull remove comments because they block pull format generate_rocpd formatted generateRocpd and generateJSON cpps formatted generateRocpd and generateJSON cpps formatted the analysis notebook reset the Rocpd aand JSon generate to the state before I added comments because my local formatting with c-lang format fails add using pandas dataframe to generate summaries added basic pytorch trace example in tests upd execute toy pytorch test pretty printing 3 temporary views bypassing pand frames set instrument cells (functions, constants) in front of usage cells added experimental routines for creating time slots via markers and counting given api calls within this time slot formatting simplified utilities more simplified utilities, added label usage in report more simplified utilities, added label usage in report fogottem commit example matrix mult added extracting copy operations by kind, added second db to compare, moved looping over database outside basic procedures marker around print replaced push.pop with just markers added duration and matrixmult flops, lists of kernel names and api names add bin directory variable to pass to generateRocpd correcting schema add bin directory variable to pass to generateRocpd debug upd for matrixmult create default timed views as copy of the original db correct table naming in the orifinal db removing rocpd_string, rocpd_metadata from views removing rocpd_string, rocpd_metadata from views removing rocpd_string, rocpd_metadata from views implemented timed views clean up and creation innotebook formatted python formatted python formatted files via make format and commented pytprch test added new line at the end of the tableSchema Add tracing script for rocpd time slicing views - Start and end options: timestamps/percentage , markers rebase and format one cpp file fix source formatting Rocpd: Time slice data - Rename tracing.py > time_slice.py - Adapt time_slice.py to use RocpdImportData, fix typo 'rocpd_api_ops', add output argument (overwrite input otherwise), update rocpd_kernelapi view - Simplifyed chrome_tracing.py : remove start+end arguments, avoid manipulating the time window/time slice Fix rebase issues [DO NOT MERGE] rocpd-schema.md Schema updates rocpd schema v3 updates/fixes rocpd schema v3 updates/fixes - samples view - kernels view - insert zero duration regions as samples generateRocp update rocpd_memory_allocate table fill Misc compilation cleanup tableSchema.sql updates Add core performance analysis views: - Add busy view for GPU utilization metrics - Add top view for overall performance summary - Add top_kernels view for kernel performance analysis - Add memory_copies view for memory transfer tracking - Add memory_allocates view for memory allocation stats - Add kernel_summary view for aggregate kernel metrics utilitySchema.sql clean up TOP view and some consistancy changes added marker views Update rocpd schema - new: rocpd_pmc_event - modified: rocpd_pmc - modified: _rocpd_memory_copy - name_id - add rocdecode API - populate rocpd_pmc and rocpd_pmc_event - slight modifications to column names for consistency :memory allocation insert for db memory allocation insert for db memory allocation insert for db commit after rebase autoincrement for memory allocation stringsanitizer applied for pmc description replaced inner join with left join for stream id, queue id, region name id Update chrome_tracing.py - chrome_tracing script updated using new schema - importer.py pending update with new schema, currently commented out Misc compilation cleanup after rebase Update rocpd/time_window.py - time_window.py script updated using new schema - Adapt chrome_tracing.py to use time windows added rocpd_arg and the join view with events, populated set back default python version to 3.6 for remote addeddereferencing stream ids and usage, some formatting refine error messages and help arguments in time_windows.py test if removing exact python version fixes integration test removed unused variable formatting and added back exact 3.6 to fix python linting (try) removed the exact python version again because it causes advanced analysis to fail revise microseconds to nanoseconds made type and name text again, put back strict version for python Reorganization of public cxx serialization headers Rework generator<T> (abstract base) + file_generator<T> - return file_generator<T> from buffered_output Add gotcha submodule Update cxx serialization headers Revert some HIP stream changes [output library] Misc updates - fix serialization of agent_info - add parseRocpd.{hpp,cpp} - move common SQL utils to sql.{hpp,cpp} [python libraries] Bindings for rocpd + reorg - move common cmake code into source/lib/python/utilities.cmake - build python bindings for rocpd (primitive implementation) - Require ROCPROFILER_BUILD_SQLITE3=OFF for rocpd python bindings - wrap sqlite3_open / sqlite3_close / sqlite3_open_v2 / sqlite3_close_v2 Update rocpd/importer.py - importer.py updated using new schema commit aff19818a5c9f9f6004013144ae00e2c31b21739 Author: Jonathan R. Madsen <jonathanrmadsen@gmail.com> Date: Thu Nov 14 22:27:39 2024 -0600 [SEPARATE PR] HIP API buffer records with args (ext) - Needs to be in separate PR - New buffer tracing domain(s) for HIP APIs which include the arguments and the return value in the buffer records Update HIP stream support for extended HIP buffer tracing Update rocprofv3 tool library and output library to use extended HIP buffer tracing recods commit 43c3f0ddd5a104346d6db77b8a1b66fd9ec2f797 Author: Jonathan R. Madsen <jonathanrmadsen@gmail.com> Date: Thu Nov 14 18:47:52 2024 -0600 [SEPARATE PR] Combo of several separate PRs [SEPARATE PR] Update correlation ID retirement - Needs to be in separate PR - correlation_id_finalize [SEPARATE PR] rocprofiler_query_intercept_table_name - Needs to be in separate PR - Function to get the name of intercept tables [SEPARATE PR] Memory copy and memory alloc updates commit e3da9738b06f974fb6b935893f4172852819b6bc Author: Jonathan R. Madsen <jonathanrmadsen@gmail.com> Date: Mon Nov 4 17:22:30 2024 -0600 Add SQLite3 build support Initial SQL writing to rocpd schema in C++ task code Implement kernel dispatch and memory copy writing to table Update generateRocpd.cpp - SQLITE3_CHECK macro - Error messages for sqlite3_exec errors - Tweaked rocpd_kernelcodeobject to use kernel ids as id - fixed some issues: - use string_entries.at(...) instead of (SELECT id FROM ... WHERE string = ...) - use custom op_idx instead of relying on correlation ID (one correlation ID can map to multiple ops) Update generate-rocpd.py - misc tweaks to mirror generateRocpd.cpp implementation - add strings for counter names - use kernel id for rocpd_kernelcodeobject Tweak tests/bin/hip-graph/hip-graph.cpp - use stream sync Update generateRocpd.cpp - made table and view schema more readable - sql_exec_callback Add source/scripts/rocprofv3-db-to-tracing.py - Script which reads a rocprofv3 rocpd database and outputs a chrome tracing format JSON Common library updates - static_tl_object: similar to static_object but for thread-local objects - additional template metaprogramming constructs - reverse - function_traits rocprofiler_stream_id_t: opaque handle for a stream - e.g. HIP stream - the same HIP stream may map to different HSA queues at different points in the application - added to: - rocprofiler_buffer_tracing_hip_api_record_t - rocprofiler_buffer_tracing_memory_copy_record_t - rocprofiler_callback_tracing_hip_api_data_t - rocprofiler_callback_tracing_memory_copy_data_t rocprofiler_stream_id_t: output support - use stream_id in generatePerfetto.cpp - use stream_id in generateRocpd.cpp rocprofiler_stream_id_t: output support - rocpd_kernelapi and rocpd_copyapi encode stream/queue as integer instead of string Update source/scripts/rocprofv3-db-to-tracing.py - Create temporary view from multiple db file - Modify write_chrome_tracing_json to use the tmp views Update rocprofv3-db-to-tracing.py - retain rocpd tables names in create_temp_view instead of appending "_tmp" - improve the GPU/Agent and Queue identifiers - make the SQL statements easier to read - remove rocpd_hsaApi usage (not necessary as HSA data resides in rocpd_api table) - directly connect to input if only one database Update generateRocpd.cpp - use dispatch id for rocpd_kernelapi id (primary key) - use dispatch_id for rocpd_op sequenceId - create copy_id for rocpd_copyapi id (primary key) - use copy_id for rocpd_op sequenceId Update tests/{async-copy-tracing,kernel-tracing,page-migration} - expand reading the RCCL and scratch memory trace data Add sdk::parse::strip function - utility function for stripping characters from beginning and end of string Preliminary rocpd_node table - rocpd_node is a meta-table for process ids - added lib/output/node_info.{hpp,cpp} - add tool::node_info instance to tool::metadata Update page-migration test Python packaging of rocpd add temporary view rocpd_copyapi Update rocpd/importer.py - Add rocpd_kernelapi and rocpd_copyapi - Create meta temporary views using RocpdSchema JSON tool handling of HIP compiler API data Misc rocpd updates - SQL formatting - remove unused imports - fix relative import lib/output metadata updates - add process_init_ns and process_fini_ns - add command_line member rocprofv3 updates: tool.cpp - simplify buffer service configuration - rocprofiler_at_intercept_table_registration -> api_timestamps_callback -> start time - record init, fini, start, and end times Added simple_timer to common library Add missing new lines at end of files f added labels for multiprocessed, tested kernel-rename, groupd by queue, type-relative of but bug in run-label - fixed static marker name to dynamic rocpd fix python linting errors Squashed commit of JSON changes, small CSV, OTF2, pftrace fixes, and rocpd commandline params Contains these original commits: commit 2c7a10771d60ad0b93073f94f6226a6e92ade4cb Author: oshkarav_amdeng <oshkarav@amd.com> Date: Wed May 14 18:57:30 2025 +0000 removed run markers reporting from otf2 as they duplicate api calls commit 826c0c13a164e7a5d2c7f1963cc437ee53416658 Author: Young Hui <young.hui@amd.com> Date: Wed May 14 12:55:51 2025 -0400 rocpd command line updates - some stats-summary params moved to generic params since applies to json and csv - added --group-by-queue for perfetto commit 33a82dcc9c05869f70219b75c76e4d0e6ae84a39 Author: acanadas <acanadas@amd.com> Date: Wed May 14 07:12:45 2025 -0500 Fix corr_id for kernel in write_csv, fix std dev in summary views commit b89dfbd02a243c3e1b5d1a4a968ab5b7c9ecb3a3 Author: acanadas <acanadas@amd.com> Date: Wed May 14 04:03:56 2025 -0500 Fix generate json merged for multiple db commit 4968f65f51a6539c95a801c1340487a5675b1f45 Author: Jin Tao <jin.tao@amd.com> Date: Wed May 14 08:02:10 2025 +0000 clean the code for json: remove host fuction, but keep the basic structure commit 0f5ed2011a6da5a6046974c03c64569a4747102e Author: Jin Tao <jin.tao@amd.com> Date: Wed May 14 07:12:23 2025 +0000 small fix commit 152c149905e98f33688fdb0cfc5ca88b3c61694d Author: Young Hui <young.hui@amd.com> Date: Tue May 13 22:13:35 2025 -0400 Schema revert rocpd_event.correlation_id back to INT - removed duplicate K.tid from kernel views - re-enabled some json.cpp code to build commit 8af5ff79b1f85cd1f2ac4c61af580dc891e6dd70 Author: Jin Tao <jin.tao@amd.com> Date: Tue May 13 11:13:56 2025 +0000 add host_functions commit 2a3e0307ab7571be0f46ffd2f706d2c9f34cdce1 Author: acanadas <acanadas@amd.com> Date: Tue May 13 06:08:10 2025 -0500 Adding summary to write_json, add extra information for sql column type error commit d339f1cc5eb856e9e3bd7461ac4d776e8265b7cd Author: oshkarav_amdeng <oshkarav@amd.com> Date: Mon May 12 17:42:47 2025 +0000 fixed counter annotations for kernel dispatch AND shadowed names in jsob.cpp commit d3a8ece3135a8cb5a78b05f7ca653793d89d5d5a Author: acanadas <acanadas@amd.com> Date: Mon May 12 09:18:56 2025 -0500 Fix node for counters in write_json commit 5153841b8d047bb4c78aeda5675848f2887aef29 Author: acanadas <acanadas@amd.com> Date: Mon May 12 08:26:22 2025 -0500 Adding counters. code_objects and kernel_symbols in write_json commit e979a1ab87d6b148d2b90ae25fc60d5e766e3251 Author: Jin Tao <jin.tao@amd.com> Date: Mon May 12 13:10:13 2025 +0000 add json counter_collection.records commit 026d022052122ee36f83022e270110840dc38aa3 Author: Young Hui <young.hui@amd.com> Date: Sat May 10 00:29:33 2025 -0400 Fix kernel_dispatch thread_id in Perfetto traces, more pftrace args supported and formatting - perfetto-backend=system results in 0KB pftrace file, but probably need the traced and perfetto daemons (have not tested with them) commit f1341487723672b407fa89d2d383944d92663c55 Author: oshkarav_amdeng <oshkarav@amd.com> Date: Fri May 9 17:08:40 2025 +0000 fixed counters for perfetto but need more testing commit 33612f9bc6121e38ad9b41d2aecd035d1d629efb Author: oshkarav_amdeng <oshkarav@amd.com> Date: Fri May 9 13:34:50 2025 +0000 rearchitectured generator tests into a dedicated rocpd-generators suit (to be extended) and jdon file-name input made a parameter commit da1e386b77bc1bbbbdc1ca8ee2ffd8c09a99ae5b Author: acanadas <acanadas@amd.com> Date: Fri May 9 08:36:00 2025 -0500 fixing buffer_record missing values in json, fix using clang_tidy commit f7adb8b28ab64127ef20ff642ced40ccace7fef8 Author: acanadas <acanadas@amd.com> Date: Fri May 9 10:08:28 2025 +0000 revise callback_records.counters_collection, still has 3 TBDs commit 49d4057e7b7e295953915b90cdeb7359948c5034 Author: acanadas <acanadas@amd.com> Date: Thu May 8 11:10:32 2025 -0500 Fixing correlation_id for csv and json commit bb37556c9d1da7fe1ccb0bc5bf0e05d4f0989476 Author: oshkarav_amdeng <oshkarav@amd.com> Date: Thu May 8 13:27:01 2025 +0000 This is a combination of 3 commits. tree node indexing updated, added extended agent struct for otf2 for efffective handling agents, agent indexes and laeled agent names rolled back tree id as process counter, tested well (source and test) commit 38dc36da08a5a505b6d7d47840690b4d6c29b429 Author: oshkarav_amdeng <oshkarav@amd.com> Date: Thu May 8 13:26:46 2025 +0000 tree node indexing updated, added extended agent struct for otf2 for efffective handling agents, agent indexes and laeled agent names commit 2958ba37abc86e5b03bc03f5d529c0efcfd2ce45 Author: acanadas <acanadas@amd.com> Date: Thu May 8 08:08:10 2025 -0500 Adding size, kind and operation for buffer_records in write_json commit 654c42d8a1c341ef1bfbb724eaf6b24e64fb5475 Author: acanadas <acanadas@amd.com> Date: Thu May 8 07:40:30 2025 -0500 Fixing merge with schema change commit 01f133bc0fed66d60cb5fe05d9e0b256938c7fb0 Author: acanadas <acanadas@amd.com> Date: Thu May 8 06:48:58 2025 -0500 fixing write_json merge commit dd3bbd5ed6faabc2207d9bf933c5c8efb8a2fe16 Author: root <root@smc300x-ccs-aus-GPUF2C5.cs-aus.dcgpu> Date: Thu May 8 04:45:56 2025 -0500 update write_json adding strings, buffer_records and counters commit 6a991305bf75f4f2e944b40e7194b43fcf1ec340 Author: Young Hui <young.hui@amd.com> Date: Thu May 8 01:29:08 2025 -0400 Schema changed rocpd_event.correlation_id to TEXT type [WIP] - when --kernel-rename is used, SQL stores correlation_id.external.value value with precision error. To store accurately, need to store as TEXT. - work-in-progress commit, still getting a few rocpd SQL conversion warnings due to TEXT type change. commit d50b058dc764394f9fc9cfc264866f43f9354da0 Author: oshkarav_amdeng <oshkarav@amd.com> Date: Fri May 2 09:32:12 2025 +0000 This is a combination of 2 commits. fixed clang tidy complaint on otf2 and perfetto, added guid-s t info names, but still 2-process DBs fail fixed mutliprocess otf2, but agents still need to be labeled by guid Misc fixes - flush less often for perfetto - metadata create_agent_index - remove sqlite3_close from write_otf2 - warning if gotcha_wrap fails - fix rocpd-generators tests - fix multi-python build config Whitespace cleanup Revert tests/rocprofv3/summary/CMakeLists.txt Remove unused scripts - generate-rocpd.py is outdated - rocprofv3-db-to-tracing.py is scratch - simple-transpose-pybind.py is no longer used Fix maybe-uninitialized compiler warning Fix maybe-uninitialized compiler warning Update rocpd.write_X functions to require RocpdImportData instance - create libpyrocpd.RocpdImportData - rocpd.importer.RocpdImportData inherits from libpyrocpd.RocpdImportData - write_csv, write_json, etc. all require instance of RocpdImportData instead of list of sqlite3.Connection output/rocpd source reorganization - move lib/output/parseRocpd.* to lib/python/rocpd/source/common.* - move lib/output/serialization to lib/python/rocpd/source/serialization - move lib/output/sql/generator.hpp to lib/python/rocpd/source/sql_generator.hpp Minor refactor of lib/output/sql/* Remove lib/output/sql.hpp - this file just included other headers Fix rocpd source reorg Update schema files - replace quotes with backticks remove short guid from perfetto fix string sanitization in generateRocpd.cpp * Update cereal submodule * Updates following rebase - remove MANIFEST.in - remove rocpd/schema_data/*.sql - use rocprofiler-sdk-rocpd * rocpd command line update to pass importData instead of connection * Update get_perfetto_category for ROCDECODE_API_EXT * data_views.sql updates - remove kernels_renamed view - remove rccl view - remove rocjpeg view - remove rocdecode view - remove api_regions view - remove api_threads view * Perfetto, OTF2, output config updates - Support kernel rename for output config - Combine memory copy and kernels into same stream track - Fix get_category_string in perfetto - Support kernel renaming in OTF2 and Perfetto * Add samples to perfetto * ORDER BY for perfetto regions/samples * Fix busy view, fix integer overflow in summary views * CSV adding --stats-per-rank and --stats-summary-per-rank reports Squash of these original commits: commit 4c0ded4efb6ef273953faf7fd100a54ce16ae00b Author: Jin Tao <jintao12@amd.com> Date: Mon Jun 2 07:25:06 2025 +0000 change all rank output to PID instead of GUID commit 969950b009ad18a156a31bfee43f64e032720262 Author: Jin Tao <jintao12@amd.com> Date: Wed May 28 11:22:29 2025 +0000 Change --stats-per-rank and --stats-summary-per-rank to csv.py commit 970f88dec3bb83663d8a84d8bf4dfcfb8857e903 Author: Jin Tao <jintao12@amd.com> Date: Thu May 22 11:39:51 2025 +0000 refactor and small bug fix commit a4c5ab4290c0a9b7b45421c3fc21150914d12110 Author: Jin Tao <jintao12@amd.com> Date: Thu May 22 10:59:13 2025 +0000 add args for --stats-per-node and --stats-summary-per-node commit 432907d0a54de886d01f962f3084c6708bdb1456 Author: Jin Tao <jintao12@amd.com> Date: Thu May 22 09:22:55 2025 +0000 add summary for marker, rccl, rocdecode, rocjpeg_calls commit ab53c9c8f4a6f5f7bf40a32bf1f93add1dbd10f5 Author: Jin Tao <jintao12@amd.com> Date: Wed May 21 13:33:04 2025 +0000 add hsa_api_stats_by_node, hip_api_stats_by_nod commit b26974bdd6d0047759b4014f9c226cf771f0a2ad Author: Jin Tao <jintao12@amd.com> Date: Wed May 21 11:14:07 2025 +0000 Add memory allocation stats by node and memory copy stats by node commit d02819c2c4bd57bfeb0537d3a42d9f22866e8600 Author: Jin Tao <jintao12@amd.com> Date: Wed May 21 08:33:08 2025 +0000 Kernels (or categories) by node completed * JSON fixes for agent in memory allocation if free in write_json, add group by nid for _summary_node views Squash of these original fixes: commit 25d8d985b319585fab16eb6d8e9ec5c91e767a91 Author: acanadas <acanadas@amd.com> Date: Thu May 22 14:00:20 2025 +0000 fix agent for memory allocation if free in write_json, add group by nid for _summary_node views commit d64905418a307db9e56b816e4b9f4de587bc3a14 Author: acanadas <acanadas@amd.com> Date: Thu May 22 08:55:21 2025 +0000 Fix typo json.cpp for memory copy src agent * CSV add domain_stats_per_rank and PID header fix Squash of these 3 commits: commit 10ec1cf40cc92a9d7ce7db84a8c6bbe20c51a340 Author: Jin Tao <jintao12@amd.com> Date: Tue Jun 3 10:06:33 2025 +0000 small fix PID header rename commit 6775fd3b2031b30ae389b165cf44b321026c1e80 Author: Jin Tao <jintao12@amd.com> Date: Mon May 26 11:47:23 2025 +0000 Add domain_stats_per_rank commit 9f3eb1793e354c1f04bab1fbc837e86f1f5b5030 Author: Jin Tao <jintao12@amd.com> Date: Thu May 22 12:11:28 2025 +0000 add domain_summary_node view CSV PID header formatting * upd otf2 generate validation test went well * fix Cmake formatting * In data_views, change P.id/T.id to P.pid/T.tid * Remove unapproved views for this PR * Remove rocpd JSON output support * Fix clang-tidy errors * Revert output_config changes + cmake + remove chrome_tracing.py * Revert * Misc rocpd cleanup - remove all CsvType per-node and stats enum values - remove rocpd::types::marker - remove rocpd/source/stats_summary.{hpp,cpp} - add `sample_regions` view - add `regions_and_samples` view - move tests/rocprofv3/rocpd-generators to tests/rocprofv3/rocpd - merge validate_perfetto.py and validate_otf2.py into single validate.py * Remove stats options from rocpd.output_config * Remove json from rocpd.__main__ * Remove rocpd.csv summary/stats options * Remove generate_from_rocpd.py * Add rocpd subparser (convert) + migrate tests to use python3 -m rocpd * Add additional tests for the rocpd command line - Check the --help flag works for: - python3 -m rocpd --help - python3 -m rocpd convert --help - python3 -m rocpd.csv --help - python3 -m rocpd.otf2 --help - python3 -m rocpd.pftrace --help * adding rocpd python shebangs and main function parameter ordering * Fix sanitizer tests + remove read_code_objects and read_kernel_symbols * Misc updates - Update CHANGELOG.md and source/lib/python/rocpd/README.md - update time_window.py - find min/max in regions_and_samples/kernels/memory_allocations/memory_copies - RocpdImportData supports all the same functions as sqlite3.Connection * Improve time_window.py - find the tables with start/end dynamically - find the tables with timestamp dynamically * Minor revert to time_window.py * Remove tests/rocprofv3/pytorch-tests * Fix python 3.6 error in time_window.py - 'type' object is not subscriptable * Fix rocpd installation --------- Co-authored-by: Jonathan R. Madsen <jonathanrmadsen@gmail.com> Co-authored-by: Young Hui <young.hui@amd.com> Co-authored-by: acanadas <acanadas@amd.com> Co-authored-by: Jin Tao <jintao12@amd.com> Co-authored-by: oshkarav_amdeng <oshkarav@amd.com> |