Memory Allocation Tracking (#1142)
* Initial commit: Need to implement wrapper function to collect data and test that wrapper function is correctly replacing core HSA functions * Attempted to implement wrapper implementation for hsa memory allocation functions. Need to modify generate record files and test if implementation is working as expected * Debugging and implementing generateCSV function * Memory allocation size and starting address outputted to csv and json file formats * Formatting * Initial setup for OTF2 and Perfetto generation * Collecting agent id for memory_allocation and formatting * Modified memory_allocation.cpp to set up code for AMD_EXT commands * Support for memory_pool_allocate added * Removed accidently added file * Made flag optional and added more OTF2 and Perfetto code. Needs testing to ensure perfetto and OTF2 works * Formatting * Fixed perfetto and otf2 output * Fixed flag issue due to incorrect buffer use * Updated documentation * Small cleaning and comments * Added test for HSA memory allocation tracing * Fixed summary test validation errors due to allocation tracing. Added type to location_base to create unique event ids for allocation due to OTF2 trace error * Decreased lower limit of hip calls for test * Modified summary tests to vary number of allocate requests * Minor fixes to address comments. Still need to address OTF2 comments * Fix docs and changed OTF2 to use enum for type specified in location_base construction * Fixed schema error * Added vmem command tracking. Need to add test * Updated test to work with vmem command and updated generateCSV to output int instead of hex string. * OTF2 enum update and mispelling fix * CI does not support Virtual Memory API. Removed vmem test. Will add back if CI is modifed to suport vmem API * Update CMakeLists.txt for memory allocation test * Updated summary test * Minor fixes to address comments * Moved domain_type.hpp enum to before LAST * Fixed compile errors and formatting * Fixed stats summary domain name error * Added rocprofv3 test * Page migration test fix * Undo page migration test changes. Failures do not appear to have to do with memory allocation
Этот коммит содержится в:
itrowbri
GitHub
@@ -0,0 +1,4 @@
|
||||
"Kind","Operation","Agent_Id","Allocation_Size","Starting_Address","Correlation_Id","Start_Timestamp","End_Timestamp"
|
||||
"MEMORY_ALLOCATION","MEMORY_ALLOCATION_ALLOCATE",0,1024,140341497356288,1,65788054621500,65788055678893
|
||||
"MEMORY_ALLOCATION","MEMORY_ALLOCATION_ALLOCATE",0,1024,140341497348096,1,65788055691832,65788056666844
|
||||
"MEMORY_ALLOCATION","MEMORY_ALLOCATION_ALLOCATE",0,1024,140341497339904,1,65788056672061,65788057643457
|
||||
|
@@ -55,11 +55,11 @@ Here is the sample of commonly used ``rocprofv3`` command-line options. Some opt
|
||||
- Output control
|
||||
|
||||
* - ``-r`` \| ``--runtime-trace``
|
||||
- Collects HIP (runtime), memory copy, marker, scratch memory, and kernel dispatch traces.
|
||||
- Collects HIP (runtime), memory copy, memory allocation, marker, scratch memory, and kernel dispatch traces.
|
||||
- Application Tracing
|
||||
|
||||
* - ``-s`` \| ``--sys-trace``
|
||||
- Collects HIP, HSA, memory copy, marker, scratch memory, and kernel dispatch traces.
|
||||
- Collects HIP, HSA, memory copy, memory allocation, marker, scratch memory, and kernel dispatch traces.
|
||||
- Application Tracing
|
||||
|
||||
* - ``--hip-trace``
|
||||
@@ -78,6 +78,10 @@ Here is the sample of commonly used ``rocprofv3`` command-line options. Some opt
|
||||
- Collects memory copy traces.
|
||||
- Application tracing
|
||||
|
||||
* - ``--memory-allocation-trace``
|
||||
- Collects memory allocation traces.
|
||||
- Application tracing
|
||||
|
||||
* - ``--scratch-memory-trace``
|
||||
- Collects scratch memory operations traces.
|
||||
- Application tracing
|
||||
@@ -356,6 +360,30 @@ Here are the contents of ``memory_copy_trace.csv`` file:
|
||||
|
||||
For the description of the fields in the output file, see :ref:`output-file-fields`.
|
||||
|
||||
Memory allocation trace
|
||||
+++++++++++++++++++++++++
|
||||
|
||||
To trace memory allocations during the application run, use:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
rocprofv3 –-memory-allocation-trace -- < app_path >
|
||||
|
||||
The above command generates a ``memory_allocation_trace.csv`` file prefixed with the process ID.
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
$ cat 6489_memory_allocation_trace.csv
|
||||
|
||||
Here are the contents of ``memory_allocation_trace.csv`` file:
|
||||
|
||||
.. csv-table:: Memory allocation trace
|
||||
:file: /data/memory_allocation_trace.csv
|
||||
:widths: 10,10,10,10,10,10,20,20
|
||||
:header-rows: 1
|
||||
|
||||
For the description of the fields in the output file, see :ref:`output-file-fields`.
|
||||
|
||||
Runtime trace
|
||||
+++++++++++++++
|
||||
|
||||
@@ -374,7 +402,7 @@ memory operations (copies and scratch).
|
||||
|
||||
rocprofv3 –-runtime-trace -- < app_relative_path >
|
||||
|
||||
Running the above command generates ``hip_api_trace.csv``, ``kernel_trace.csv``, ``memory_copy_trace.csv``, ``scratch_memory_trace.csv``,and ``marker_api_trace.csv`` (if ``ROCTx`` APIs are specified in the application) files prefixed with the process ID.
|
||||
Running the above command generates ``hip_api_trace.csv``, ``kernel_trace.csv``, ``memory_copy_trace.csv``, ``scratch_memory_trace.csv``, ``memory_allocation_trace.csv``, and ``marker_api_trace.csv`` (if ``ROCTx`` APIs are specified in the application) files prefixed with the process ID.
|
||||
|
||||
System trace
|
||||
++++++++++++++
|
||||
@@ -385,7 +413,7 @@ This is an all-inclusive option to collect all the above-mentioned traces.
|
||||
|
||||
rocprofv3 –-sys-trace -- < app_relative_path >
|
||||
|
||||
Running the above command generates ``hip_api_trace.csv``, ``hsa_api_trace.csv``, ``kernel_trace.csv``, ``memory_copy_trace.csv``, and ``marker_api_trace.csv`` (if ``ROCTx`` APIs are specified in the application) files prefixed with the process ID.
|
||||
Running the above command generates ``hip_api_trace.csv``, ``hsa_api_trace.csv``, ``kernel_trace.csv``, ``memory_copy_trace.csv``, ``memory_allocation_trace.csv``, and ``marker_api_trace.csv`` (if ``ROCTx`` APIs are specified in the application) files prefixed with the process ID.
|
||||
|
||||
Scratch memory trace
|
||||
++++++++++++++++++++++
|
||||
@@ -464,6 +492,8 @@ Properties
|
||||
Dispatch Traces.
|
||||
- **``memory_copy_trace``** *(boolean)*: For Collecting Memory
|
||||
Copy Traces.
|
||||
- **``memory_allocation_trace``** *(boolean)*: For Collecting Memory
|
||||
Allocation Traces.
|
||||
- **``scratch_memory_trace``** *(boolean)*: For Collecting
|
||||
Scratch Memory operations Traces.
|
||||
- **``stats``** *(boolean)*: For Collecting statistics of enabled
|
||||
@@ -479,8 +509,8 @@ Properties
|
||||
- **``hsa_image_trace``** *(boolean)*: For Collecting HSA API
|
||||
Traces (Image-extension API).
|
||||
- **``sys_trace``** *(boolean)*: For Collecting HIP, HSA, Marker
|
||||
(ROCTx), Memory copy, Scratch memory, and Kernel dispatch
|
||||
traces.
|
||||
(ROCTx), Memory copy, Memory allocation, Scratch memory, and
|
||||
Kernel dispatch traces.
|
||||
- **``mangled_kernels``** *(boolean)*: Do not demangle the kernel
|
||||
names.
|
||||
- **``truncate_kernels``** *(boolean)*: Truncate the demangled
|
||||
@@ -990,3 +1020,18 @@ Properties
|
||||
- **`src_agent_id`** *(object, required)*: Source Agent ID.
|
||||
- **`handle`** *(integer, required)*: Handle of the agent.
|
||||
- **`bytes`** *(integer, required)*: Bytes copied.
|
||||
- **`memory_allocation`** *(array)*: Memory allocation records.
|
||||
- **Items** *(object)*
|
||||
- **`size`** *(integer, required)*: Size of the Marker API record.
|
||||
- **`kind`** *(integer, required)*: Kind of the Marker API.
|
||||
- **`operation`** *(integer, required)*: Operation of the Marker API.
|
||||
- **`correlation_id`** *(object, required)*: Correlation ID information.
|
||||
- **`internal`** *(integer, required)*: Internal correlation ID.
|
||||
- **`external`** *(integer, required)*: External correlation ID.
|
||||
- **`start_timestamp`** *(integer, required)*: Start timestamp.
|
||||
- **`end_timestamp`** *(integer, required)*: End timestamp.
|
||||
- **`thread_id`** *(integer, required)*: Thread ID.
|
||||
- **`agent_id`** *(object, required)*: Agent ID.
|
||||
- **`handle`** *(integer, required)*: Handle of the agent.
|
||||
- **`starting_address`** *(string, required)*: Starting address of allocation.
|
||||
- **`allocation_size`** *(integer, required)*: Size of allocation.
|
||||
|
||||
@@ -1374,6 +1374,90 @@
|
||||
"bytes"
|
||||
]
|
||||
}
|
||||
},
|
||||
"memory_allocation": {
|
||||
"type": "array",
|
||||
"description": "Memory allocation records.",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"size": {
|
||||
"type": "integer",
|
||||
"description": "Size of the Marker API record."
|
||||
},
|
||||
"kind": {
|
||||
"type": "integer",
|
||||
"description": "Kind of the Marker API."
|
||||
},
|
||||
"operation": {
|
||||
"type": "integer",
|
||||
"description": "Operation of the Marker API."
|
||||
},
|
||||
"correlation_id": {
|
||||
"type": "object",
|
||||
"description": "Correlation ID information.",
|
||||
"properties": {
|
||||
"internal": {
|
||||
"type": "integer",
|
||||
"description": "Internal correlation ID."
|
||||
},
|
||||
"external": {
|
||||
"type": "integer",
|
||||
"description": "External correlation ID."
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"internal",
|
||||
"external"
|
||||
]
|
||||
},
|
||||
"start_timestamp": {
|
||||
"type": "integer",
|
||||
"description": "Start timestamp."
|
||||
},
|
||||
"end_timestamp": {
|
||||
"type": "integer",
|
||||
"description": "End timestamp."
|
||||
},
|
||||
"thread_id": {
|
||||
"type": "integer",
|
||||
"description": "Thread ID."
|
||||
},
|
||||
"agent_id": {
|
||||
"type": "object",
|
||||
"description": "Agent ID.",
|
||||
"properties": {
|
||||
"handle": {
|
||||
"type": "integer",
|
||||
"description": "Handle of the agent."
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"handle"
|
||||
]
|
||||
},
|
||||
"starting_address": {
|
||||
"type": "integer",
|
||||
"description": "Starting address of allocation"
|
||||
},
|
||||
"allocation_size": {
|
||||
"type": "integer",
|
||||
"description": "allocation_size"
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"size",
|
||||
"kind",
|
||||
"operation",
|
||||
"correlation_id",
|
||||
"start_timestamp",
|
||||
"end_timestamp",
|
||||
"thread_id",
|
||||
"agent_id",
|
||||
"starting_address",
|
||||
"allocation_size"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -61,6 +61,11 @@
|
||||
"description": "For Collecting Memory Copy Traces"
|
||||
},
|
||||
|
||||
"memory_allocation_trace": {
|
||||
"type": "boolean",
|
||||
"description": "For Collecting Memory Allocation Traces"
|
||||
},
|
||||
|
||||
"scratch_memory_trace": {
|
||||
"type": "boolean",
|
||||
"description": "For Collecting Scratch Memory operations Traces"
|
||||
@@ -98,7 +103,7 @@
|
||||
|
||||
"sys_trace" : {
|
||||
"type": "boolean",
|
||||
"description": "For Collecting HIP, HSA, Marker (ROCTx), Memory copy, Scratch memory, and Kernel dispatch traces"
|
||||
"description": "For Collecting HIP, HSA, Marker (ROCTx), Memory copy, Memory allocation, Scratch memory, and Kernel dispatch traces"
|
||||
},
|
||||
|
||||
"mangled_kernels": {
|
||||
|
||||
Ссылка в новой задаче
Block a user