If the execution command had a split into multiple HW operations, then runtime has to accumulate time for all operations
Change-Id: Iaba31e96250918d8190bf63adb4c07730fdfefbf
[ROCm/clr commit: 24f5362296]
Runtime can reset the last command only if it didn't change
since the query at the beginning of finish()
Change-Id: I629f2d788e9bbaa17ca4e96b1a753f8131e32463
[ROCm/clr commit: 9e74f1c7f8]
Move hidden heap creation to the kernel launch to make sure it's
allocated on the actual first usage.
Change-Id: I1b65a82fc06d9129ed45a69765bf14ea3d945b04
[ROCm/clr commit: 4975f69337]
Move StartLateDeviceInit() and FinishDeviceInit() calls in RGP
server to avoid a timeout
Change-Id: Ifd681859c7dc76259d7296aa8cc15305d563c9b2
[ROCm/clr commit: ecea224bcf]
The heap must be cleared once per device, but ROCclr doesn't
create a queue per device in HIP. Hence, the clear operation will
be performed during the first queue creation.
Change-Id: I52ceb06d67d11cde6d019c5ab510059f426a9bfb
[ROCm/clr commit: 04bfd93569]
Disable hostcall buffer in OCL for now. COv5 can add hostcallbuffer
metadata for unknown reason. OCL may fail the buffer allocation
and kernel launch.
Change-Id: I34a6a45bac86c57422b764c0d69760c96920d6c5
[ROCm/clr commit: 934149ff0a]
Disable devlib linking when runtime links multiple objects from
the app. Otherwise devlibs will be linked twice and may cause
undefined behavior with COv5.
Change-Id: I3b8640c64ff898893225fe3af5b4b4a32d42bf40
[ROCm/clr commit: c275d9b4b3]
Remove assert for kernel arg size, because COv5 reports a value
bigger than the actual usage in the most of cases
Change-Id: I8e15bc45a9e21b58a5894f9977511ca84408ce61
[ROCm/clr commit: 2be0b1e612]
With COv5 local size calculation must occur before
runtime programs kernel arguments
Change-Id: I0726c6529bde69b8fcf5360aa83986cf84e04168
[ROCm/clr commit: caa6110c29]
Add ROCR memory detection and enable arena mem object for possible
access in HIP
Change-Id: Icf86ac789176bfee4ea8d36b0970a817d4c6a2f7
[ROCm/clr commit: 28597ec5b5]
It can be too early to allocate memory at the begining of
Device::create() under PAL
Change-Id: I4bd76db7be3f6fb246243ea68022d8b0f860471d
[ROCm/clr commit: 3af3fe10de]
Currently COMGR doesn't provide global variable size and runtime
parses ELF binary directly. Avoid parsing for HIP. That can save
5% in hipModuleLoad() time.
Change-Id: I47540d1e957bdb0c2406b6b848222de2920b2504
[ROCm/clr commit: 2664d8cf9e]
Pass active queue for transfers in the cache coherency layer.
That will allow to use device transfer queue only for
cases when active queue isn't available, because using device
transfer queue from another active queue may cause a deadlock
Change-Id: Ifbe7e0303b77dbf6eeda3939ffbc25a3df7472de
[ROCm/clr commit: 95d55fdfa8]
OCL2.2 requires SPIR-V and runtime doesn't support it.
Make sure PAL backend doesn't report any SPIR-V support.
Change-Id: I8d179069674205b54f7d20d149bcb675bee5cdb0
[ROCm/clr commit: 0bf395af39]
Metadata in Codeobject version 5 is the extension of CO3 and CO4.
Add the detection of the new fields and program them in
the setup of the kernel arguments.
Change-Id: I27e58df77320ad00f4f16d35912668db803826af
[ROCm/clr commit: be6a06384e]
Report proper target id for xnack in HSAIL path. Runtime
will use ISA table and report hsailName().
Fix offline compilation path for PAL.
Change-Id: Ic0250bf6b9c193d867aec9800a319da1bf00c3ee
[ROCm/clr commit: a543d4a860]
Add a threshold for ROCR/SDMA P2P transfers. ROCR copy path
requires extra barriers in compute for synchronization. That costs
extra performance with tiny transfers.
Reduce active wait time to 10us. Tensorflow uses extra thread
per GPU with constant hipEventQuery() calls. Longer active waits
in ROCr affect CPU performance.
Change-Id: I9020358438615fa2d4617f862f00a562f0a588e7
[ROCm/clr commit: 008133cf41]
Stall in the host thread could occur earlier than the app expects.
Make sure rutnime can grow the signals to the queue size without
any stall. Also adding a new signal to the end of the pool could
break the dependency chain on signal reuse. The new logic will
insert the new signal after current to keep the chain intact.
Change-Id: I9c90b98515907db8b677528263c3e88cd9581a14
[ROCm/clr commit: 102c19adf3]
Reference for the first element can trigger an assert with
_GLIBCXX_ASSERTIONS build
Change-Id: I59c63c052831307edfe5dcc6384798a43e9596dd
[ROCm/clr commit: 6f2e7c3199]
- Create hash values for binaries
- Add the binaries into RGP trace
- Add corresponding hash value for every dispatch
Change-Id: I2c3ce004d69f37d0d46bc4744e12f24273517f5e
[ROCm/clr commit: 2a298f2ec3]
Just signal check will still submit the marker and then later
runtime will have a timeout, but the barrier packet is still
generated. Hence early timeout will allow to skip the marker.
Change-Id: Ieb7d89becbcff43a4f4c46715354ca65ab4a80b9
[ROCm/clr commit: bbb635bc32]
The optimization is controlled with ROCR_SKIP_KERNEL_ARG_COPY.
This is initial check-in for experiments. Extra changes are
necessary for full support:
- handle graph capture with the original sysmem alloc
- avoid memobject references, otherwise there is a race condition with
reusage of the arg buffer
- Remove arg setup from hip
Change-Id: Ib0af710f93e79834711fa4049a7c66093711e68b
[ROCm/clr commit: 7e12cf6318]
The kernel arg pool will be divided into 8 chunks to avoid long stalls,
when the pool will be reused.
Change-Id: I228e6ca1c09e428c1775f1e5b685220a9a5d71af
[ROCm/clr commit: f78b3a8919]
ROC_AQL_QUEUE_SIZE will control the size of AQL queue.
The current sefault value is 4096.
Change-Id: Icd2a4ee3ba554c06aa05b08defd922d2c63e43fd
[ROCm/clr commit: 7fe696b6ef]
The original logic was left after initial testing when HMM
couldn't handle xnack properly
Change-Id: I0abf01805704171e931dfba8b6d95bfe87d5fab1
[ROCm/clr commit: d17108e8d0]
Change the scope of hostcall buffer access lock during destruction.
Make sure wait() returns the signal value after timeout. That
matches ROCr behaviour for HSA signal wait.
Change-Id: I3df34207e0c2e21972ec8052777e5742bda1dca0
[ROCm/clr commit: 9a9d10a10b]
Cache coherency layer is OCL feature to support multiple devices in
single OCL context.
Change-Id: Ic66df9551fad5b0c4df95ab3e1db1da259919f25
[ROCm/clr commit: 6da9d18140]
The queue can be destroyed at the time the app will request
the event status. Hence just get the active state from the device.
Change-Id: I887ecb0cfe414c2119247228b0d1255b8308da1e
[ROCm/clr commit: f116959b54]
When unsetting runtime should use HSA_AMD_SVM_ATTRIB_AGENT_ACCESSIBLE
for the agent and not HSA_AMD_SVM_ATTRIB_AGENT_ACCESSIBLE_IN_PLACE
Change-Id: I3814802d1fb3b72c54e7566defafafed6b0d5cee
[ROCm/clr commit: d8a86e4870]
The original logic left only one slot for HW processing in the queue.
For some reason there is a race condition on CPU overwrite of the slot
before the current active. The workaround is to avoid the previous to
the current active slot for possible unfinished HW processing.
Change-Id: I565495a8feeaedffc9fc8a505edbee5ff5816975
[ROCm/clr commit: 65ddfcc6a8]
The new query MemRangeAttribute::CoherencyMode can return current
coherency mode for the provided memory region. Coherency mode can
be one of the following types: FineGrain, CoarseGrain and
Indeterminate
Change-Id: Ib66feeeb14f57a8b1cc731c65bb3d0276d297ff7
[ROCm/clr commit: 992830bab7]
There is a possible race condition when signal reuse can have
access to a destroyed Timestamp object, because the callback
was running asynchronously. Use reference counter and lock
to allow asynchronous timestamp update
Change-Id: I6224f7c62cb0a03a7466fcc512e5e5afb06736fa
[ROCm/clr commit: ec89348291]
Add ref counting to ProfilingSignal class to track the last release.
If a signal was used in the marker, then don't reuse it,
but create a new one for internal usage.
Don't rely on HSA callback for the command status update if there
are no pending dispatches.
Change-Id: I19f14ed9d80acfe79993b343b2187635f8428a20
[ROCm/clr commit: ff15c0893e]
HSA signal calback may occur during the actual marker submit. That
may cause a deadlock, because shared lock_ object. Create the new
notify_lock_ field to protect the notification.
Change-Id: I9752af84e59895530620fac3932c6fc276de8658
[ROCm/clr commit: f34c1b9ff8]
Runtime can't assign internal HSA signals for HIP events, because
HIP application can destroy the HIP stream or signal reuse may
occur internally. Switch to global HSA signals for HIP events.
Change-Id: Ieaea2d6b039e492b2e7c5112782a8f4e601e50a1
[ROCm/clr commit: ce8dad2ecc]
Add lock protection for signal processing
If signal is reused, then disable reference to it from HIP
Increase the pool signal size to 32
Change-Id: I7d529b35910f83ce577c9eca6d3386759611ccc0
[ROCm/clr commit: a1629cad26]
If AMD event contains a reference to a HW event, then runtime
could check/wait for HW event. CPU status update will occur later
after HSA signal callback, but it's not important for the result.
Change-Id: I591391a953bbdba6a25ac07e2cd98aeb17cd4596
[ROCm/clr commit: 85c70a7495]
Switch HSA_AMD_SVM_ATTRIB_READ_ONLY to
HSA_AMD_SVM_ATTRIB_READ_MOSTLY to match Cuda. The new attribute
was just exposed in ROCr/KFD.
Change-Id: I2ee522d33c347ba52a4e272d2cd7f67960490cf7
[ROCm/clr commit: 89b69638d1]
Add an extension to memory advise to disable cache coherency for
better performance
Change-Id: I283703d81d9c36ddfa2c8fffa15eef60e2195056
[ROCm/clr commit: a9a1e21445]
With HIP API callback runtime has to stall the queue until the
callback is done. Rocclr will introduce SW blocking HSA signal,
which will be released after the callback is done.
Change-Id: I6411f3efab31b468e3b87ebb5c8d155e116b613d
[ROCm/clr commit: d93df7037c]
- Make sure barrier with dependent signals issues before queue
index reservation
- Don't issue extra barrier if it's already a barrier command
with dependent signals
Change-Id: I179a8b7adac79eed698f4a4d9eca2606d8e913aa
[ROCm/clr commit: 148a5cac39]
With direct dispatch enabled make sure the queue is done before
destruction.
Change-Id: Ib80af3efb97dfb93e2dce60a11db34fb5c45f5cd
[ROCm/clr commit: a81756bba3]
Use HSA_AMD_SVM_ATTRIB_AGENT_ACCESSIBLE flag for the initial
allocation instead of HSA_AMD_SVM_ATTRIB_AGENT_ACCESSIBLE_IN_PLACE.
Change-Id: Ia52fe205563df1ea916dc2dc81e749e11c16f83d
[ROCm/clr commit: 9b3072ae12]
Device enqueue has an option to execute scheduler on the current
queue and it's enabled by default. Make sure scratch is allocated
on the current queue for that case. Add max vgpr tracking per
program to adjust scratch size accordingly.
Change-Id: I2a6d796913a4551a1e7f343a2465d589eec60d8a
[ROCm/clr commit: e553b2763a]
MT doesn't use GPU waits, but CPU for sync between engines.
Change the threshold values for CPU waits for direct dispatch.
That will bring behavior closer to MT.
Change-Id: Ia41c3cb812614962aff2746b6cf858f1bf77dda2
[ROCm/clr commit: ca2ea70a6c]
The check has to be performed inside the signal loop, because
active signals need to be processed to avoid a stale timestamp
class.
Change-Id: I26af8287aae18eb19c096d9358cd0b86cfd1c561
[ROCm/clr commit: 2d41031aec]
- With direct disaptch profiling state is enabled to trigger the
callback on HSA signal. However ROCr has very low peformance on
the first call to get the profiling info. That impacts some tiny
performance tests.
Change-Id: Idacd1b10a473fcfb5feef3074b7191d35743f769
[ROCm/clr commit: a71f7f931e]
ROCr returns some unexpected values and for now just return
invalid device back to the app
Change-Id: Ia6fb709ed2dc77d77bd5d64de80b41b3720bed76
[ROCm/clr commit: 8c513407af]
HIP requires to return AccessedBy query for all device, but ROCr
can process one per query. Hence send the queries for all
available devices and then accumulate the results in runtime.
Change-Id: I082f9adb8e31c775a8ad1bf7a5af37440ef4bd16
[ROCm/clr commit: e9c484d1ce]
Enabling DebugVMID requires a certain sequence in
PAL initialization. StartLateDeviceInit() must be called before
CommitSettingsAndInit().
Change-Id: I7385a8cc89e7a8ad97a6b56ad6acbd2cf2f29728
[ROCm/clr commit: dee99ca807]
HIP tests require HIP callbacks to be processed in another thread.
This change will use a thread from HSA signal callbacks to make
sure a HIP callback was done asynchronously.
Also process the callback before changing the status of command
Change-Id: Icef85d0e0f808663882cf6881ff1be3e5eca29ac
[ROCm/clr commit: 7f32d0b425]
- Don't notify if the batch is empty, because that means
the current command was processed already.
- Disable pinning optimization to avoid a race condition on stall.
- TS marker submition requires extra AQL barrier
to track the status.
Change-Id: I17eff4ad12ac66cfe1bb44048bebb1891805279d
[ROCm/clr commit: 24299e25bd]
Skip notification for markers with direct dispatch only,
since they are blocking always
Change-Id: I6bb17650f73371dae6e29c59fd6bb2012cc062fd
[ROCm/clr commit: a9b0e20d26]
Direct disaptch doesn't insert extra barriers for Markers if
AQL barrier was the last issued command already.
Change-Id: I00fbc658547d83dd3ee64ec391ed50e5f8a08e30
[ROCm/clr commit: 0587fb7450]
German