AI/rocm-systems
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

[SDK] Fix double buffer data race (#394)

Browse Source 
* Fix double buffer data race

- fixes relatively rare data race in double buffering scheme

In `rocprofiler::buffer::instance::emplace`, the `container::record_header_buffer::get_record_headers()` function returned a `std::vector<rocprofiler_record_header_t*>` and then invoked callback to tool. It was possible for that callback to still be executing while the buffer was being updated. This potentially introduced a scenario where the rocprofiler_record_header_t* was modified (or corrupted) before the tool processed the record. In rocprofv3, this would result in a "future" buffer record showing up among "past" buffer records. E.g., correlation id sequence of 1-15 where the buffer flushes after five values, could result in this during processing:
|     |     |     |     |      |
|:---:|:---:|:---:|:---:|:---:|
|  1 |  2 |  3 |  4 | 15 |
|  6 |  7 |  8 |  9 | 10 |
| 11 | 12 | 13 | 14 | 15 |

Because buffer A (of double buffering scheme) originally containing corr ids 1-5 stalled after process corr id 4 (e.g. write to disk), buffer B filled up with 6-10 and started flushing, causing a switch back to buffer A, and buffer A was filled with 11-15 by the time callback accessed what was originally corr id 5 but was now updated to corr id 15.

* Update CHANGELOG

* misc minor cleanup

---------

Co-authored-by: Jonathan R. Madsen <jonathanrmadsen@gmail.com>
This commit is contained in:
Madsen, Jonathan
2025-05-14 13:19:22 -05:00
committed by GitHub
parent 6ec9526475
commit 8a1ee46e47
8 changed files with 144 additions and 66 deletions
Download Patch File Download Diff File Expand all files Collapse all files
source/lib/rocprofiler-sdk/buffer.cpp
+45 -20
View File
@@ -183,27 +183,52 @@ flush(rocprofiler_buffer_id_t buffer_id, bool wait)
if(!buff_internal_v.is_empty())
{
// get the array of record headers
auto buff_data = buff_internal_v.get_record_headers();
// designates that buffer should be cleared after functor is invoked
constexpr auto clear_buffer_v = std::true_type{};
// invoke buffer callback
try
{
if(buff_v->callback)
{
buff_v->callback(rocprofiler_context_id_t{buff_v->context_id},
rocprofiler_buffer_id_t{buff_v->buffer_id},
buff_data.data(),
buff_data.size(),
buff_v->callback_data,
buff_v->drop_count);
}
} catch(std::exception& e)
{
ROCP_ERROR << "buffer callback threw an exception: " << e.what();
}
// clear the buffer
buff_internal_v.clear();
// invoke the callback within the scoped lock of process_record_headers.
auto num_processed = buff_internal_v.process_record_headers(
clear_buffer_v, [&buffer_id, &idx, &offset, &buff_v](auto&& _headers) {
// invoke buffer callback
try
{
if(buff_v->callback)
{
ROCP_INFO << fmt::format("invoking buffer callback for {} records "
"[buffer_id={}, idx={}, offset={}]",
_headers.size(),
buffer_id.handle,
idx,
offset);
buff_v->callback(rocprofiler_context_id_t{buff_v->context_id},
rocprofiler_buffer_id_t{buff_v->buffer_id},
_headers.data(),
_headers.size(),
buff_v->callback_data,
buff_v->drop_count);
}
else
{
ROCP_TRACE << fmt::format("no buffer callback for {} records "
"[buffer_id={}, idx={}, offset={}]",
_headers.size(),
buffer_id.handle,
idx,
offset);
}
} catch(std::exception& e)
{
ROCP_CI_LOG(ERROR) << "buffer callback threw an exception: " << e.what();
}
});
ROCP_INFO << fmt::format(
"completed buffer callback for {} records [buffer_id={}, idx={}, offset={}]",
num_processed,
buffer_id.handle,
idx,
offset);
}
else
{