P2p checkpoint.

- set USE_PEER_TO_PEER=3 (requires HCC "am_memtracker_update_peers")
- when enabling peer, turn it on for previously allocated memory.
- hipDeviceCanAccessPeer is no longer self-ware (self does not qualify
  as a peer)
- device peerlist always includes self, so when we call allow_access
  we never remove self access.
- hipDeviceReset() removes old peer mappings.


[ROCm/clr commit: 9e7efd7c65]

projects/clr/hipamd/src/hip_device.cpp

@@ -174,6 +174,7 @@ hipError_t hipDeviceReset(void)
     if (device) {
         //---
         //Wait for pending activity to complete?  TODO - check if this is required behavior:
+        //TODO, also we have small window between wait and reset.
         
         device->locked_waitAllStreams();