Misc updates for distribution (#233)

* Adding tools support * cmake formatting (cmake-format) (#227) Co-authored-by: SrirakshaNag <SrirakshaNag@users.noreply.github.com> * Checking to do rebase * Adding rocprofv2 script * cmake formatting (cmake-format) (#229) Co-authored-by: bgopesh <bgopesh@users.noreply.github.com> * Fixing build for the tool * Removing the requirement for rocm_version * Update rocprofiler_utilities.cmake * C++ filesystem fixes - added source/lib/common/filesystem.hpp - support older compilers which have <experimental/filesystem> and do not have <filesystem> - added samples/common/filesystem.hpp - samples now depend on "common" library which provides the correct filesystem header - renamed rocprofiler-stdcxxfs interface target to rocprofiler-cxx-filesystem - support old LLVM in addition to GNU - fix bin/rocprof/rocprof.cpp - was using VLA * Fix rocprofiler-drm include directories - OpenSUSE only has include/libdrm/drm.h (no include/drm/drm.h) * Tools fixes * Fix for the tools * Fix rocprofv2 script * Fixing Filesystem Issues * source formatting (clang-format v11) (#234) Co-authored-by: ammarwa <ammarwa@users.noreply.github.com> * Vlaindic/pc sampling api update (#235) * pcs: updating PC sampling API * source formatting (clang-format v11) (#232) Co-authored-by: vlaindic <vlaindic@users.noreply.github.com> --------- Co-authored-by: vlaindic <vladimir.indic@amd.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: vlaindic <vlaindic@users.noreply.github.com> * Vlaindic/pc sampling api update for ammar branch (#244) *Updating the documentation inside pc_sampling.h --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: vlaindic <vlaindic@users.noreply.github.com> * pcs: use @p in front of params * pcs: documenting struct fields updated * Fixing PC Sampling Documentation issues * Fixing PC Sampling Documentation * Relocated tools directory to source/lib/rocprofiler-tool * Fixes/updates to rocprofiler-tool - updated CMake - Fixed miscellaneous issues in the code (VLAs, etc.) - Updated rocprofv2 to reflect some minor env variables changes in rocprofiler-tool - Fixed clang-tidy warnings * Update lib/rocprofiler-tool/CMakeLists.txt - link to atomic library * Add $ORIGIN/.. RUNPATH to rocprofiler-tool * Adding readme file for tools * Renaming the tools readme file * Update ReadMe.md * Update ReadMe.md * Documentation updates - overview and explanation of design and concepts * Fix lib/rocprofiler-tool/README.md - delete ReadMe.md * Hacks for build * Update Filesystem * cmake formatting (cmake-format) (#248) Co-authored-by: ammarwa <ammarwa@users.noreply.github.com> * source formatting (clang-format v11) (#249) Co-authored-by: ammarwa <ammarwa@users.noreply.github.com> * source formatting (clang-format v11) (#250) Co-authored-by: ammarwa <ammarwa@users.noreply.github.com> * Addressing review comments on the tool readme file * Revert "Hacks for build" This reverts commit d6688cb3d1226c46fc97e37ced889a5b0d180940. * Fixes for GCC 7.5 compiler in OpenSUSE 15.4 * Update lib/rocprofiler-tool/CMakeLists.txt - link to AQL profile library * Fix lib/rocprofiler-tool/README.md - fix markdown * Fix lib/rocprofiler-tool - fix usage of hsa_ven_amd_loader_query_host_address * Fix unused variable warnings - byproduct of variables only used in assert statements * Update docs - update about.md - more "Important Changes" section here - update tool_library_overview.md - extend "Tool Library Design" section - write "Tool Initialization" section - write "Tool Finalization" section * Add ghc::filesystem submodule * Implement usage of ghc::filesystem * Add ROCPROFILER_BUILD_GHC_FS option - option to use external/filesystem (ghc) * Update samples/counter-collection - compile flags - common library - fixes for warnings * Update tests/kernel-tracing/CMakeLists.txt - change install location of kernel-tracing-test-tool and install rpath * Update samples/common/CMakeLists.txt - compile features requiring C++17 * Update lib/rocprofiler-tool/tool.cpp - remove include <filesystem> - comment out unused variable - remove unused functions - move some functions into anonymous namespace --------- Co-authored-by: Sriraksha Nagaraj <Sriraksha.Nagaraj@amd.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: SrirakshaNag <SrirakshaNag@users.noreply.github.com> Co-authored-by: gobhardw <gopesh.bhardwaj@amd.com> Co-authored-by: bgopesh <bgopesh@users.noreply.github.com> Co-authored-by: Jonathan R. Madsen <jonathanrmadsen@gmail.com> Co-authored-by: ammarwa <ammarwa@users.noreply.github.com> Co-authored-by: vlaindic <vladimir.indic@amd.com> Co-authored-by: vlaindic <vlaindic@users.noreply.github.com> Co-authored-by: Vladimir Indic <139573562+vlaindic@users.noreply.github.com> Co-authored-by: Benjamin Welton <bewelton@amd.com> Co-authored-by: Jonathan R. Madsen <jrmadsen@users.noreply.github.com> [ROCm/rocprofiler-sdk commit: fe5d074375]
2023-11-28 10:04:37 -06:00
@@ -6,6 +6,29 @@
   :maxdepth: 4
 ```

-## Overview
+## Important Changes

-TODO: Overview of rocprofiler v2
+[Roctracer](https://github.com/ROCm-Developer-Tools/roctracer) and [rocprofiler (v1)](https://github.com/ROCm-Developer-Tools/rocprofiler)
+have been combined into a single rocprofiler SDK and re-designed from scratch. The new rocprofiler API has been designed with some
+new restrictions to avoid problems that plagued the former implementations. These restrictions enable more efficient implementations
+and much better thread-safety. The most important restriction is the window for tools to inform rocprofiler about which services
+the tool wishes to use (where "services" refers to the capabilities for API tracing, kernel tracing, etc.).
+
+In the former implementations, when one of the ROCm runtimes were initially loaded, a tool only had
+to inform roctracer/rocprofiler that it wished to use its services at some point (e.g. calling `roctracer_init()`)
+and were not required to specify which services it would eventually or potentially use. Thus, these libraries had to effectively prepare for
+any service to be enable at any point in time -- which introduced unnecessary overhead when tools had no desire to use certain features and
+made thread-safe data management difficult. For example, roctracer was required to _always_ install wrappers around _every_ runtime API function
+and _always_ added extra overhead of indirection through the roctracer library and checks for the current service configuration (in a thread-safe manner).
+
+In the re-designed implementation, rocprofiler introduces the concept of a "context". Contexts are effectively
+bundles of service configurations. Rocprofiler gives each tool _one_ opportunity to create as many contexts as necessary --
+for example, a tool can group all of the services into one context, create individual contexts for each service, or somewhere in between.
+Due to this design choice change, rocprofiler now knows _exactly_ which services might be requested by the tool clients at any point in time.
+This has several important implications:
+
+- rocprofiler does not have to unnecessarily prepare for services that are never used -- if no registered contexts requested tracing the HSA API, no wrappers need to be generated
+- rocprofiler can perform more extensive checks during service specification and inform tools about potential issues very early on
+- rocprofiler can allow multiple tools to use certain services simulatenously
+- rocprofiler was able to improve thread-safety without introducing parallel bottlenecks
+- rocprofiler can manage internal data and allocations more efficiently
@@ -0,0 +1,14 @@
+# Buffered Tracing Services
+
+
+```eval_rst
+.. toctree::
+   :glob:
+   :maxdepth: 4
+```
+
+## Overview
+
+## HSA API Tracing
+
+## Kernel Tracing
@@ -0,0 +1,13 @@
+# Callback Tracing Services
+
+```eval_rst
+.. toctree::
+   :glob:
+   :maxdepth: 4
+```
+
+## Overview
+
+## Code Object Tracing
+
+## HSA API Tracing
@@ -8,4 +8,8 @@

 ## Overview

-TODO: Overview of rocprofiler v2 features
+- Improved tool initialization
+- Support for multiple tools using the same services simulatenously
+- Simplified management of enabling/disabling one or more data collection services
+- Improved error checking and logging
+- Backwards ABI compatibility (goal)
@@ -9,5 +9,9 @@
   about
   features
   installation
+   tool_library_overview
+   callback_tracing
+   buffered_tracing
+   intercept_table
   developer_api
 ```
@@ -0,0 +1,9 @@
+# Runtime Intercept Tables
+
+```eval_rst
+.. toctree::
+   :glob:
+   :maxdepth: 4
+```
+
+Discussion on how access the raw runtime intercept tables of HSA and HIP (i.e. ExaTracer requirements by LTTng).
@@ -0,0 +1,254 @@
+# Building Tool Library
+
+```eval_rst
+.. toctree::
+   :glob:
+   :maxdepth: 4
+```
+
+## Rocprofiler and ROCm Runtimes Design
+
+The ROCm runtimes are now designed to directly communicate with a new library called rocprofiler-register during their initialization. This library does cursory checks
+for whether any tools have indicated they want rocprofiler support via detection of one or more instances of a symbol named `rocprofiler_configure` (which is provided by
+the tool libraries) and/or the `ROCP_TOOL_LIBRARIES` environment variable. This design dramatically improves upon previous designs which relied solely on
+a tool racing to set runtime-specific environment variables (e.g. `HSA_TOOLS_LIB`) before the runtime initialization.
+
+## Tool Library Design
+
+When a tool has `rocprofiler_configure` visible in its symbol table, rocprofiler will invoke this function and provide information regarding
+the version of rocprofiler which invoking the function, how many tools have already been invoked, and a unique idenitifier for the tool. The tool
+returns a pointer to a `rocprofiler_tool_configure_result_t` struct, which, if non-null, can provide rocprofiler with the function it should
+call for tool initialization (i.e. the opportunity for context creation), a function is should call when rocprofiler is finalized, and a pointer
+to any data that rocprofiler should provide back to the tool when it calls the initialization and finalization functions.
+
+Rocprofiler provides a `rocprofiler/registration.h` header file which forward declares the `rocprofiler_configure` function with the necessary
+compiler function attributes to ensure that the symbol is publicly visible.
+
+```cpp
+#include <rocprofiler/registration.h>
+
+namespace
+{
+// saves the data provided to rocprofiler_configure
+struct ToolData
+{
+    uint32_t                              version;
+    const char*                           runtime_version;
+    uint32_t                              priority;
+    rocprofiler_client_id_t               client_id;
+};
+
+// tool initialization function
+int
+tool_init(rocprofiler_client_finalize_t fini_func,
+          void* tool_data_v);
+
+// tool finalization function
+void
+tool_fini(void* tool_data_v);
+}
+
+extern "C"
+{
+rocprofiler_tool_configure_result_t*
+rocprofiler_configure(uint32_t                 version,
+                      const char*              runtime_version,
+                      uint32_t                 priority,
+                      rocprofiler_client_id_t* client_id)
+{
+    // if not first tool to register, indicate tool doesn't want to do anything
+    if(priority > 0) return nullptr;
+
+    // (optional) provide a name for this tool to rocprofiler
+    client_id->name = "ExampleTool";
+
+    // (optional) create configure data
+    static auto data = ToolData{ version,
+                                 runtime_version,
+                                 priority,
+                                 client_id };
+
+    // construct configure result
+    static auto cfg =
+        rocprofiler_tool_configure_result_t{ sizeof(rocprofiler_tool_configure_result_t),
+                                             &tool_init,
+                                             &tool_fini,
+                                             static_cast<void*>(&data) };
+
+    return &cfg;
+}
+```
+
+## Tool Initialization
+
+> ***NOTE: rocprofiler does NOT support calls to any of the runtime functions (HSA, HIP, etc.) during tool initialization.***
+> ***Invoking any functions from the runtimes will result in a deadlock.***
+
+For each tool which contains a `rocprofiler_configure` function and returns a non-null pointer to a `rocprofiler_tool_configure_result_t` struct,
+rocprofiler will invoke the `initialize` callback after completing the scan for all `rocprofiler_configure` symbols. In other words, rocprofiler
+collects all of the `rocprofiler_tool_configure_result_t` instances before invoking the `initialize` member of any of these instances.
+When rocprofiler invokes this function in a tool, this is the opportunity to create contexts:
+
+```cpp
+#include <rocprofiler/rocprofiler.h>
+
+namespace
+{
+int
+tool_init(rocprofiler_client_finalize_t fini_func,
+          void* data_v)
+{
+    // create a context
+    auto ctx = rocprofiler_context_id_t{};
+    rocprofiler_create_context(&ctx);
+
+    // ... associate services with context ...
+
+    // start the context (optional)
+    rocprofiler_start_context(ctx);
+
+    return 0;
+}
+}
+```
+
+Although not strictly necessary, it is recommended that tools store the context handle(s) to control the data collection of the services associated with the context.
+
+## Tool Finalization
+
+In the invocation of the user-provided `initialize` callback, rocprofiler will provide a function pointer of type `rocprofiler_client_finalize_t`.
+This function pointer can be invoked by the tool to explicitly invoke the `finalize` callback from the `rocprofiler_tool_configure_result_t` instance:
+
+```cpp
+#include <rocprofiler/rocprofiler.h>
+
+namespace
+{
+int
+tool_init(rocprofiler_client_finalize_t fini_func,
+          void* data_v)
+{
+    // ... see initialization section ...
+
+    // function which finalizes tool after 10 seconds
+    auto explicit_finalize = [](rocprofiler_client_finalize_t finalizer,
+                                rocprofiler_client_id_t* client_id)
+    {
+        std::this_thread::sleep_for(std::chrono::seconds{ 10 });
+        finalizer(client_id);
+    };
+
+    // start the context
+    rocprofiler_start_context(ctx);
+
+    // dispatch a background thread to explicitly finalize after 10 seconds
+    std::thread{ explicit_finalize, fini_func, static_cast<ToolData*>(data_v)->client_id }.detach();
+
+    return 0;
+}
+}
+```
+
+Otherwise, rocprofiler will invoke the `finalize` callback via an `atexit` handler.
+
+## Agent Information
+
+## Contexts
+
+## Configuring Services
+
+## Synchronous Callbacks
+
+## Asychronous Callbacks for Buffers
+
+## Recommendations
+
+## Full `rocprofiler_configure` Sample
+
+All of the snippets from the previous sections have been combined here for convenience.
+
+```cpp
+#include <rocprofiler/registration.h>
+
+namespace
+{
+struct rocp_tool_data
+{
+    uint32_t                              version;
+    const char*                           runtime_version;
+    uint32_t                              priority;
+    rocprofiler_client_id_t               client_id;
+    rocprofiler_client_finalize_t         finalizer;
+    std::vector<rocprofiler_context_id_t> contexts;
+};
+
+void
+tool_tracing_callback(rocprofiler_callback_tracing_record_t record,
+                      rocprofiler_user_data_t*              user_data,
+                      void*                                 callback_data);
+
+int
+tool_init(rocprofiler_client_finalize_t fini_func,
+          void* tool_data_v)
+{
+    rocp_tool_data* tool_data = static_cast<rocp_tool_data*>(tool_data_v);
+
+    // save the finalizer function
+    tool_data->finalizer = fini_func;
+
+    // create a context
+    auto ctx = rocprofiler_context_id_t{};
+    rocprofiler_create_context(&ctx);
+
+    // save your contexts
+    tool_data->contexts.emplace_back(ctx);
+
+    // associate code object tracing with this context
+    rocprofiler_configure_callback_tracing_service(
+        ctx,
+        ROCPROFILER_CALLBACK_TRACING_CODE_OBJECT,
+        nullptr,
+        0,
+        tool_tracing_callback,
+        tool_data);
+
+    // ... associate services with contexts ...
+
+    return 0;
+}
+
+void
+tool_fini(void* tool_data);
+}
+
+extern "C"
+{
+rocprofiler_tool_configure_result_t*
+rocprofiler_configure(uint32_t                 version,
+                      const char*              runtime_version,
+                      uint32_t                 priority,
+                      rocprofiler_client_id_t* client_id)
+{
+    // if not first tool to register, indicate tool doesn't want to do anything
+    if(priority > 0) return nullptr;
+
+    // (optional) provide a name for this tool to rocprofiler
+    client_id->name = "ExampleTool";
+
+    // info provided back to tool_init and tool_fini
+    auto* my_tool_data = new rocp_tool_data{ version,
+                                             runtime_version,
+                                             priority,
+                                             client_id,
+                                             nullptr };
+
+    // create configure data
+    static auto cfg =
+        rocprofiler_tool_configure_result_t{ sizeof(rocprofiler_tool_configure_result_t),
+                                             &tool_init,
+                                             &tool_fini,
+                                             my_tool_data };
+
+    return &cfg;
+}
+```