Chris Freehill
6fb4c79784
Change-Id: Id033122d0b2f74b52a95a2ace99889c5d090cab3 (cherry picked from commit 29a3aee72f9546743d25ebae8c356b33933d3657)
137 Zeilen
6.6 KiB
Markdown
137 Zeilen
6.6 KiB
Markdown
|
|
# ROCm<sup>TM</sup> Data Center Tool (RDC)
|
|
|
|
The ROCm™ Data Center Tool simplifies the administration and addresses key infrastructure challenges in AMD GPUs in cluster and datacenter environments. The main features are:
|
|
|
|
- GPU telemetry
|
|
- GPU statistics for jobs
|
|
- Integration with third-party tools
|
|
- Open source
|
|
|
|
For complete list of features and how to start using RDC from pre-built packages, please refer to [**user guide**](docs/AMD_ROCm_Data_Center_Tool_User_Guide.pdf)
|
|
|
|
# Supported platforms
|
|
Ubuntu 18.04.5 (Kernel 5.3)
|
|
CentOS v7.7 (Using devtoolset-7 runtime support)
|
|
RHEL v7.7 (Using devtoolset-7 runtime support)
|
|
SLES 15 SP1
|
|
CentOS and RHEL 8.1(Kernel 4.18.0-147)
|
|
|
|
# Building RDC from source
|
|
|
|
## Dependencies
|
|
|
|
CMake 3.15 ## 3.15 or greater is required for gRPC
|
|
g++ (5.4.0)
|
|
Doxygen (1.8.11) ## required to build the latest documentation
|
|
Latex (pdfTeX 3.14159265-2.6-1.40.16) ## required to build the latest documentation
|
|
gRPC and protoc ## required for communication
|
|
|
|
AMD ROCm platform (https://github.com/RadeonOpenCompute/ROCm)
|
|
* It is recommended to install the complete AMD ROCm platform.
|
|
For installation instruction see https://rocmdocs.amd.com/en/latest/Installation_Guide/Installation-Guide.html
|
|
* At the minimum, these two components are required
|
|
(i) AMD ROCm SMI Library (https://github.com/RadeonOpenCompute/rocm_smi_lib)
|
|
(ii) AMD ROCk Kernel driver (https://github.com/RadeonOpenCompute/ROCK-Kernel-Driver)
|
|
|
|
## Building gRPC and protoc
|
|
**NOTE:** gRPC and protoc compiler must be built when building RDC from source as pre-built packages are not available. When installing RDC from a package, gRPC and protoc will be installed from the package.
|
|
|
|
**IMPORTANT:** Building gRPC and protocol buffers requires CMake 3.15 or greater. With an older version build will quietly succeed with a *message*. However, all components of gRPC will not be installed and RDC will ***fail*** to run
|
|
|
|
The following tools are required for gRPC build & installation
|
|
|
|
automake make g++ unzip build-essential autoconf libtool pkg-config libgflags-dev libgtest-dev clang-5.0 libc++-dev curl
|
|
|
|
Download and build gRPC
|
|
|
|
$ git clone -b v1.28.1 https://github.com/grpc/grpc
|
|
$ cd grpc
|
|
$ git submodule update --init
|
|
$ mkdir -p cmake/build
|
|
$ cd cmake/build
|
|
|
|
## By default (without using CMAKE_INSTALL_PREFIX option), the following will install to /usr/local lib, include and bin directories
|
|
|
|
$ cmake -DgRPC_INSTALL=ON -DBUILD_SHARED_LIBS=ON <-DCMAKE_INSTALL_PREFIX=<install dir>> ../..
|
|
$ make
|
|
$ sudo make install
|
|
$ echo "<install dir>/lib" | sudo tee /etc/ld.so.conf.d/grpc.conf
|
|
|
|
## Building RDC
|
|
|
|
Clone the RDC source code from GitHub and use CMake to build and install
|
|
|
|
$ git clone https://github.com/RadeonOpenCompute/rdc
|
|
$ cd rdc
|
|
$ mkdir -p build; cd build
|
|
$ cmake -DROCM_DIR=/opt/rocm -DGRPC_ROOT="$GRPC_PROTOC_ROOT" <-DCMAKE_INSTALL_PREFIX=<install dir>> ..
|
|
$ make
|
|
$ make install ## default installation location is /opt/rocm
|
|
|
|
## Update System Library Path
|
|
|
|
The following commands need to be executed as root (sudo). It may be easiest to put them into a script and then run that script as root:
|
|
|
|
$ RDC_LIB_DIR=<RDC install dir>/lib
|
|
$ GRPC_LIB_DIR=<gRPC install dir>/lib
|
|
$ echo "$GRPC_LIB_DIR" > /etc/ld.so.conf.d/x86_64-librdc_client.conf
|
|
$ echo "$GRPC_LIB_DIR"64 >> /etc/ld.so.conf.d/x86_64-librdc_client.conf
|
|
$ echo "$RDC_LIB_DIR" >> /etc/ld.so.conf.d/x86_64-librdc_client.conf
|
|
$ echo "$RDC_LIB_DIR"64 >> /etc/ld.so.conf.d/x86_64-librdc_client.conf
|
|
$ ldconfig
|
|
|
|
# Running RDC
|
|
RDC supports encrypted communications between clients and servers. The
|
|
communication can be configured to be *authenticated* or *not authenticated*. The [**user guide**](docs/AMD_ROCm_Data_Center_Tool_User_Guide.pdf) has information on how to generate and install SSL keys and certificates for authentication. By default, authentication is enabled.
|
|
|
|
## Starting ROCm™ Data Center Daemon (RDCD)
|
|
For an RDC client application to monitor and/or control a remote system, the RDC server daemon, *rdcd*, must be running on the remote system. *rdcd* can be configured to run with (a) full-capabilities which includes ability to set or change GPU configuration or (b) monitor-only capabilities which limits to monitoring GPU metrics.
|
|
|
|
### Start RDCD from command-line
|
|
When *rdcd* is started from a command-line the *capabilities* are determined by privilege of the *user* starting *rdcd*
|
|
|
|
$ cd rdc_install_prefix ## If specified in Building RDC section
|
|
|
|
## To run with authentication. Ensure SSL keys are setup properly
|
|
$ ./usr/sbin/rdcd ## rdcd is started with monitor-only capabilities
|
|
$ sudo ./usr/sbin/rdcd ## rdcd is started will full-capabilities
|
|
|
|
## To run without authentication. SSL key & certificates are not required.
|
|
$ ./usr/sbin/rdcd -u ## rdcd is started with monitor-only capabilities
|
|
$ sudo ./usr/sbin/rdcd -u ## rdcd is started will full-capabilities
|
|
|
|
### Start RDCD using systemd
|
|
*rdcd* can be started by using the systemctl command. systemctl will read /lib/systemd/system/rdc.service, which is installed with rdc. This file has 2 lines that control what *capabilities* with which *rdcd* will run. If left uncommented, rdcd will run with full-capabilities.
|
|
|
|
|
|
## file: /lib/systemd/system/rdc.service
|
|
## Comment the following two lines to run with monitor-only capabilities
|
|
CapabilityBoundingSet=CAP_DAC_OVERRIDE
|
|
AmbientCapabilities=CAP_DAC_OVERRIDE
|
|
|
|
systemctl start rdc ## start rdc as systemd service
|
|
|
|
## Invoke RDC using ROCm™ Data Center Interface (RDCI)
|
|
RDCI provides command-line interface to all RDC features. This CLI can be run locally or remotely. Refer to [**user guide**](docs/AMD_ROCm_Data_Center_Tool_User_Guide) for the current list of features.
|
|
|
|
## sample rdci commands to test RDC functionality
|
|
## discover devices in a local or remote compute node
|
|
## NOTE: option -u (for unauthenticated) is required is rdcd was started in this mode
|
|
|
|
$ cd rdc_install_prefix ## If specified in Building RDC section
|
|
./opt/rocm/rdc/bin/rdci discovery -l <-u> ## list available GPUs in localhost
|
|
./opt/rocm/rdc/bin/rdci discovery <host> -l <-u> ## list available GPUs in host machine
|
|
|
|
## Troubleshooting rdcd
|
|
|
|
Log messages that can provide useful debug information.
|
|
|
|
## If rdcd was started as a systemd service, then use journalctl to view rdcd logs
|
|
journalctl -u rdc
|
|
|
|
## To run rdcd with debug log from command-line use
|
|
RDC_LOG=DEBUG ./usr/sbin/rdcd
|
|
|
|
|