Remove the check whether the rdcd is started by rdc user.
Add the read access check for the private key and certificates if
the authentication is enabled.
Change-Id: I0e7a7eafb7985801572f809da0cb3e4012683153
Remove the check whether the rdcd is started by rdc user.
Add the read access check for the private key and certificates if
the authentication is enabled.
Change-Id: I0e7a7eafb7985801572f809da0cb3e4012683153
[ROCm/rdc commit: 96afb24845]
Remove the * in the rdci stats
When a group is created, the GPUs can be added in the same command.
Add the support to the memory temperature.
Add the support to the memory clock.
Add the support to report the ECC errors.
Add the support to report the PCIe bandwidth throughput.
Since the RX/TX throughput may take 1 second to retreive, an async fetch is implemented
in the RdcMetricFetcherImpl.
Change-Id: If04f602fe1f2d14dbf7c2fb189549fd030523f9a
Remove the * in the rdci stats
When a group is created, the GPUs can be added in the same command.
Add the support to the memory temperature.
Add the support to the memory clock.
Add the support to report the ECC errors.
Add the support to report the PCIe bandwidth throughput.
Since the RX/TX throughput may take 1 second to retreive, an async fetch is implemented
in the RdcMetricFetcherImpl.
Change-Id: If04f602fe1f2d14dbf7c2fb189549fd030523f9a
[ROCm/rdc commit: f4a3fd4dda]
Add support for the stats subsystem in rdci
Modify the dmon system to handle the case when no GPUs in a group
Change-Id: I5a18e1201d24b5318b8e324a77551a757b108f25
Add support for the stats subsystem in rdci
Modify the dmon system to handle the case when no GPUs in a group
Change-Id: I5a18e1201d24b5318b8e324a77551a757b108f25
[ROCm/rdc commit: 096dc2dadb]
Pass in GRPC root (or use default location) for RDC to use
when building RDC components.
Change-Id: I89db2ac2be27ab6449c817d210a94c11fef965fd
[ROCm/rdc commit: 1b58033183]
Add the job stats APIs in the rdc_api_service at the server side rdcd
Add the job stats APIs for the RdcStandaloneHandler at the client side
Make the load librdc.so and librdc_client.so thread safe.
Impelement async update all fields in RdcEmbeddedHandler.
Change-Id: I659d91efb32d1094d3b7f0f2cec39518cd7336ce
Add the job stats APIs in the rdc_api_service at the server side rdcd
Add the job stats APIs for the RdcStandaloneHandler at the client side
Make the load librdc.so and librdc_client.so thread safe.
Impelement async update all fields in RdcEmbeddedHandler.
Change-Id: I659d91efb32d1094d3b7f0f2cec39518cd7336ce
[ROCm/rdc commit: fe3e75edfa]
Add the function to start and stop the job recording.
Add the function to get the job stats for each GPU and summary of multiple GPUs
Add the function to remove the jobs.
Add a class RdcLogger which can control the log level using the environment variable RDC_LOG.
This is similar to GRPC_VERBOSITY gRPC. When the customer has the issues, he can enable the verbose
log to help us to troubleshoot the issues.
Add the -u support in the rdci group, fieldgroup and dmon for connecting to rdcd without authentication.
Change-Id: I22c591823c1ee6485db106b911bed8271d1b2769
Add the function to start and stop the job recording.
Add the function to get the job stats for each GPU and summary of multiple GPUs
Add the function to remove the jobs.
Add a class RdcLogger which can control the log level using the environment variable RDC_LOG.
This is similar to GRPC_VERBOSITY gRPC. When the customer has the issues, he can enable the verbose
log to help us to troubleshoot the issues.
Add the -u support in the rdci group, fieldgroup and dmon for connecting to rdcd without authentication.
Change-Id: I22c591823c1ee6485db106b911bed8271d1b2769
[ROCm/rdc commit: a547dc7efd]
Add the support for rdci subsystem group create, delete and query
Add the support for rdci subsystem fieldgroup create, delete and query
Add the support for rdci dmon system. The dmon system may show the stats every
a few seconds until press Ctrl-C. To cleanup the resources (for example, unwatch),
a signal handler is added.
Change-Id: Ib22a8a43b7083c7c72819ca21145e22702d9ad6c
Add the support for rdci subsystem group create, delete and query
Add the support for rdci subsystem fieldgroup create, delete and query
Add the support for rdci dmon system. The dmon system may show the stats every
a few seconds until press Ctrl-C. To cleanup the resources (for example, unwatch),
a signal handler is added.
Change-Id: Ib22a8a43b7083c7c72819ca21145e22702d9ad6c
[ROCm/rdc commit: 16bce67835]
Depending on how a user starts rdcd, rdcd will either have
full monitor/control capabilities or have just monitoring
capabilties.
The only 2 user ids allowed are "rdc" and root.
Change-Id: Ie296a2f68c9723bef5945b1af1070ef99eeea93b
Depending on how a user starts rdcd, rdcd will either have
full monitor/control capabilities or have just monitoring
capabilties.
The only 2 user ids allowed are "rdc" and root.
Change-Id: Ie296a2f68c9723bef5945b1af1070ef99eeea93b
[ROCm/rdc commit: a6acf24ae7]
Implement the APIs defined in the RdcStandaloneHandler to make gRPC call to daemon
Implement the APIs defined in the RdcAPIServiceImpl to handle the gRPC calls in daemon
Add two APIs to get all GPU groups and field groups: rdc_group_get_all_ids()
and rdc_group_field_all_ids()
Those two APIs are required by the rdci group and fieldgroup
sub-modules.
Change-Id: I066091423146dea180c16af212688ed43dc44611
Implement the APIs defined in the RdcStandaloneHandler to make gRPC call to daemon
Implement the APIs defined in the RdcAPIServiceImpl to handle the gRPC calls in daemon
Add two APIs to get all GPU groups and field groups: rdc_group_get_all_ids()
and rdc_group_field_all_ids()
Those two APIs are required by the rdci group and fieldgroup
sub-modules.
Change-Id: I066091423146dea180c16af212688ed43dc44611
[ROCm/rdc commit: 7ee29b6cdd]
This will allow us to not have to use LD_LIBRARY_PATH when
packages are installed.
Change-Id: I16b4c50d400c3c7e3bbebe446c53d3605cebae53
[ROCm/rdc commit: 7084690872]
The RDC API is changed to pass the certificates to the gRPC.
Add the support to add all GPUs in the host to a group. Also before
add a GPU to a group, the RDC API will verify that GPU exists or not.
Add the support to fetch the temperature metrics.
Change-Id: I5857ef03fede233d16e8b2836be120f33172da93
The RDC API is changed to pass the certificates to the gRPC.
Add the support to add all GPUs in the host to a group. Also before
add a GPU to a group, the RDC API will verify that GPU exists or not.
Add the support to fetch the temperature metrics.
Change-Id: I5857ef03fede233d16e8b2836be120f33172da93
[ROCm/rdc commit: 66e4e790c3]
Create the skeleton implementation of rdc_client.so and rdci. Modify current rdcd to
integrate the RDC API service:
rdc.proto is changed to add a new RdcAPI service which defined the interfaces for the RDC API.
RdcStandaloneHandler.cpp is added to send the request using gRPC to the rdcd. It is built into
the rdc_client.so
rdci.cc, RdciDisCoverySubSystem.cc and RdciSubSystem.cc are added to implement skeleton rdci.
Currently, the discovery subsystem is supported.
rdc_api_service.cc is added to the server as a skeleton to implement the RdcAPI service. Currently,
only discovery API is implemented. Note: we disabled the rdc_rsmi_service, which will be removed
in the future. The original rdc_client.so is renamed to rdc_client_smi.so which should also be
removed in the future.
Add the instruction how to run the rdcd and rdci in the build folder in the README.md.
Change-Id: Id232f9f83787e5812d4a295dc8cf0daa7728b06c
Create the skeleton implementation of rdc_client.so and rdci. Modify current rdcd to
integrate the RDC API service:
rdc.proto is changed to add a new RdcAPI service which defined the interfaces for the RDC API.
RdcStandaloneHandler.cpp is added to send the request using gRPC to the rdcd. It is built into
the rdc_client.so
rdci.cc, RdciDisCoverySubSystem.cc and RdciSubSystem.cc are added to implement skeleton rdci.
Currently, the discovery subsystem is supported.
rdc_api_service.cc is added to the server as a skeleton to implement the RdcAPI service. Currently,
only discovery API is implemented. Note: we disabled the rdc_rsmi_service, which will be removed
in the future. The original rdc_client.so is renamed to rdc_client_smi.so which should also be
removed in the future.
Add the instruction how to run the rdcd and rdci in the build folder in the README.md.
Change-Id: Id232f9f83787e5812d4a295dc8cf0daa7728b06c
[ROCm/rdc commit: 020f6939f7]
Support cache manager and watch table in rdc_lib
RdcCacheManagerImpl.cc is added to implement cache of metrics. Currently, only
integer mertics are supported. The cache manager provids function to retrieve the
latest and history metrics from cache. It also provides interfaces to update and evict the cache.
RdcWatchTableImpl.cc is added to implement watch and unwatch fields. It uses the
field settings to control how frequently a field needs to be updated. We have a preliminarily
performance optimization for this class as it may be called very frequently.
RdcMetricsUpdaterImpl.cc is added to run the update at background thread when
RDC_OPERATION_MODE_AUTO is set.
After this code change, the rdcd/rdci should be able to implement basic discovery, group and dmon
function. The job management function is not implemented in the skeleton rdc_lib yet.
Change-Id: I26cff8c2ec85d1ad8e7df24c66b02f0060838d37
Support cache manager and watch table in rdc_lib
RdcCacheManagerImpl.cc is added to implement cache of metrics. Currently, only
integer mertics are supported. The cache manager provids function to retrieve the
latest and history metrics from cache. It also provides interfaces to update and evict the cache.
RdcWatchTableImpl.cc is added to implement watch and unwatch fields. It uses the
field settings to control how frequently a field needs to be updated. We have a preliminarily
performance optimization for this class as it may be called very frequently.
RdcMetricsUpdaterImpl.cc is added to run the update at background thread when
RDC_OPERATION_MODE_AUTO is set.
After this code change, the rdcd/rdci should be able to implement basic discovery, group and dmon
function. The job management function is not implemented in the skeleton rdc_lib yet.
Change-Id: I26cff8c2ec85d1ad8e7df24c66b02f0060838d37
[ROCm/rdc commit: 1ff1c7b617]
The rdc.h is modified for new discovery and grouping APIs.
The RdcGroupSettingsImpl.cc is added to implement the GPU group and
the field group management.
The RdcMetricFetcherImpl.cc is added to fetch the metrics from
rocm_smi_lib. Currently, only support power, memory, GPU utilization,
temperature, GPU clock, total device and device name.
A new example field_value_example.cc is added to demo how to record
the fields and retrieve data from cache.
Change-Id: I57acfa048fe9b3d848e2d441e768b3a63ccae3f8
The rdc.h is modified for new discovery and grouping APIs.
The RdcGroupSettingsImpl.cc is added to implement the GPU group and
the field group management.
The RdcMetricFetcherImpl.cc is added to fetch the metrics from
rocm_smi_lib. Currently, only support power, memory, GPU utilization,
temperature, GPU clock, total device and device name.
A new example field_value_example.cc is added to demo how to record
the fields and retrieve data from cache.
Change-Id: I57acfa048fe9b3d848e2d441e768b3a63ccae3f8
[ROCm/rdc commit: a5f063f8b3]
The rdc.h is the only header file will be provided to the user.
The inital version only includes the data structure and function
required for the job stats example.
The example folder has one example demonstrated how to use the API
to collect the job summary stats.
The RdcBootStrap.cc will dynamically load different libraries when user
select either the standalone or embbed mode. We also created a
dummy RdcEmbeddedHandler.cc for librdc.so.
In order to run the example after build, it needs to specify the
LD_LIBRARY_PATH. Assume current folder is the build folder:
LD_LIBRARY_PATH=$PWD/rdc_libs $PWD/example/jobstats
The folder is structured in following ways:
example
include
- rdc - rdc.h (the only header file exposed to the user)
- rdc_libs
- impl
rdc_libs
- boostrap
- src
- rdc
- src
- rdc_client
- src
- rdc_server
- src
Change-Id: Ia386ddf4cabcb2dc4fe82de6464ca0619cb3d959
The rdc.h is the only header file will be provided to the user.
The inital version only includes the data structure and function
required for the job stats example.
The example folder has one example demonstrated how to use the API
to collect the job summary stats.
The RdcBootStrap.cc will dynamically load different libraries when user
select either the standalone or embbed mode. We also created a
dummy RdcEmbeddedHandler.cc for librdc.so.
In order to run the example after build, it needs to specify the
LD_LIBRARY_PATH. Assume current folder is the build folder:
LD_LIBRARY_PATH=$PWD/rdc_libs $PWD/example/jobstats
The folder is structured in following ways:
example
include
- rdc - rdc.h (the only header file exposed to the user)
- rdc_libs
- impl
rdc_libs
- boostrap
- src
- rdc
- src
- rdc_client
- src
- rdc_server
- src
Change-Id: Ia386ddf4cabcb2dc4fe82de6464ca0619cb3d959
[ROCm/rdc commit: 85006053ed]
The rdc account will be created on installation if it does
not already exist. It will be a system account with no
home directory.
rdcd will be started as a systemd service, but change to
user "rdc". The rdc user will drop all priviliges except
CAP_DAC_OVERRIDE, permitted. This means the default mode
will have no special privileges, but have the ability to
gain write access (e.g., to sysfs) when needed.
rdc tests were being inadvertantly added to the
installation. This was adversely impacting the new
functionality, so it was corrected in this commit.
Also included are a few small formatting changes.
Change-Id: I9c6bb132fee28119fd3960594dfb97bd2e7b282a
The rdc account will be created on installation if it does
not already exist. It will be a system account with no
home directory.
rdcd will be started as a systemd service, but change to
user "rdc". The rdc user will drop all priviliges except
CAP_DAC_OVERRIDE, permitted. This means the default mode
will have no special privileges, but have the ability to
gain write access (e.g., to sysfs) when needed.
rdc tests were being inadvertantly added to the
installation. This was adversely impacting the new
functionality, so it was corrected in this commit.
Also included are a few small formatting changes.
Change-Id: I9c6bb132fee28119fd3960594dfb97bd2e7b282a
[ROCm/rdc commit: 5cc498c6aa]
Initial testing include an "id test", which really just a
template test at this point, and a temperature sensor test.
The google test code is included in this commit. It will
eventually be taken out and replaced with a pull from a google
external repo.
Change-Id: I591818a9c169f4654fc8d8f17cf648f227d72545
Initial testing include an "id test", which really just a
template test at this point, and a temperature sensor test.
The google test code is included in this commit. It will
eventually be taken out and replaced with a pull from a google
external repo.
Change-Id: I591818a9c169f4654fc8d8f17cf648f227d72545
[ROCm/rdc commit: ca4344f5fa]
* Add another Rome model
* Add gfx908 4P3L models and support
* Revert "Use cached value for detecting GDR support only once"
This reverts commit 0108a1219d.
* Skip using ibverb for GPU direct RDMA detection
* Fine tune one Rome model
[ROCm/rccl commit: a51e4071e3]
* Add another Rome model
* Add gfx908 4P3L models and support
* Revert "Use cached value for detecting GDR support only once"
This reverts commit 67c8e72ce3.
* Skip using ibverb for GPU direct RDMA detection
* Fine tune one Rome model
Bill(Shuzhou) Liu
Chris Freehill
Wenkai Du
Aryan Salmanpour