doc reorganization
Change-Id: I526e5e594032299d85d995a7e6fe2d269c3621aa Signed-off-by: Galantsev, Dmitrii <dmitrii.galantsev@amd.com>
This commit is contained in:
randyh62
committed by
Galantsev, Dmitrii
Galantsev, Dmitrii
parent
234b2d835b
commit
2b815a68c8
@@ -0,0 +1,192 @@
|
||||
.. meta::
|
||||
:description: documentation of the installation, configuration, and use of the ROCm Data Center tool
|
||||
:keywords: ROCm Data Center tool, RDC, ROCm, API, reference, data type, support
|
||||
|
||||
.. _rdc-handbook:
|
||||
|
||||
***************************************************
|
||||
Building and testing RDC tool: A developer handbook
|
||||
***************************************************
|
||||
|
||||
The RDC tool is open source and available under the MIT License. This section is helpful for open source developers. Third-party integrators may also find this information useful.
|
||||
|
||||
Prerequisites for Building RDC
|
||||
==============================
|
||||
|
||||
.. note::
|
||||
The RDC tool is tested on the following software versions. Earlier versions may not work.
|
||||
|
||||
* CMake 3.15
|
||||
* g++ (5.4.0)
|
||||
* AMD ROCm, which includes AMD AMDSMI Library
|
||||
* gRPC and protoc
|
||||
|
||||
The following components are required to build the latest documentation:
|
||||
|
||||
* Doxygen (1.8.11)
|
||||
* Latex (pdfTeX 3.14159265-2.6-1.40.16)
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
$ sudo apt install libcap-dev
|
||||
$ sudo apt install -y doxygen
|
||||
|
||||
|
||||
Build and Install RDC
|
||||
=====================
|
||||
|
||||
To build and install, clone the RDC source code from GitHub and use CMake.
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
$ git clone <GitHub for RDC>
|
||||
$ cd rdc
|
||||
$ mkdir -p build; cd build
|
||||
$ cmake -DROCM_DIR=/opt/rocm -DGRPC_ROOT="$GRPC_PROTOC_ROOT"..
|
||||
$ make
|
||||
#Install library file and header and the default location is /opt/rocm
|
||||
$ make install
|
||||
|
||||
|
||||
Build Documentation
|
||||
-------------------
|
||||
|
||||
You can generate PDF documentation after a successful build. The reference manual, refman.pdf, appears in the latex directory.
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
$ make doc
|
||||
$ cd latex
|
||||
$ make
|
||||
|
||||
|
||||
Build Unit Tests for RDC Tool
|
||||
-----------------------------
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
$ cd rdc/tests/rdc_tests
|
||||
$ mkdir -p build; cd build
|
||||
$ cmake -DROCM_DIR=/opt/rocm -DGRPC_ROOT="$GRPC_PROTOC_ROOT"..
|
||||
$ make
|
||||
|
||||
# To run the tests
|
||||
|
||||
$ cd build/rdctst_tests
|
||||
$ ./rdctst
|
||||
|
||||
|
||||
Test
|
||||
----
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
# Run rdcd daemon
|
||||
$ LD_LIBRARY_PATH=$PWD/rdc_libs/ ./server/rdcd -u
|
||||
|
||||
# In another console run the RDC command-line
|
||||
$ LD_LIBRARY_PATH=$PWD/rdc_libs/ ./rdci/rdci discovery -l -u
|
||||
|
||||
|
||||
Authentication
|
||||
==============
|
||||
|
||||
The RDC tool supports encrypted communications between clients and servers.
|
||||
|
||||
Generate Files for Authentication
|
||||
---------------------------------
|
||||
|
||||
The communication between the client and server can be configured to be authenticated or unauthenticated. By default, authentication is enabled.
|
||||
|
||||
To disable authentication, when starting the server, use the "--unauth_comm" flag (or "-u" for short). You must also use “-u” in rdci to access unauth rdcd. The /lib/systemd/system/rdc.service file can be edited to pass arguments to rdcd on starting. On the client side, when calling rdc_channel_create(), the "secure" argument must be set to False.
|
||||
|
||||
Scripts
|
||||
-------
|
||||
|
||||
RDC users manage their own keys and certificates. However, some scripts generate self-signed certificates in the RDC source tree in the authentication directory for test purposes. The following flowchart depicts how to generate the root certificates using the openssl command in 01gen_root_cert.sh:
|
||||
|
||||
A picture containing sign, drawing Description automatically generated
|
||||
|
||||
.. figure:: ../data/handbook_openssl.png
|
||||
|
||||
Generation of root certificates using openssl command
|
||||
|
||||
The section where the default responses to ``openssl`` questions can be specified is included in ``openssl.conf``. To locate the section look for the following comment line:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
# < ** REPLACE VALUES IN THIS SECTION WITH APPROPRIATE VALUES FOR YOUR ORG. **>
|
||||
|
||||
|
||||
It is helpful to modify this section with values appropriate for your organization if you expect to call this script many times. Additionally, you must replace the dummy values and update the ``alt_names`` section for your environment.
|
||||
|
||||
To generate the keys and certificates using these scripts, make the following calls:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
$ 01gen_root_cert.sh
|
||||
# provide answers to posed questions
|
||||
$ 02gen_ssl_artifacts.sh
|
||||
# provide answers to posed questions
|
||||
|
||||
|
||||
At this point, the keys and certificates are in the newly created ``CA/artifacts`` directory.
|
||||
|
||||
.. important::
|
||||
You must delete this directory if you need to rerun the scripts.
|
||||
|
||||
To install the keys and certificates, access the artifacts directory and run the ``install.sh`` script as root, specifying the install location. By default, RDC expects this to be in ``/etc/rdc``:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
$ cd CA/artifacts
|
||||
$ sudo install_<client|server>.sh /etc/rdc
|
||||
|
||||
|
||||
These files must be copied to and installed on all client and server machines that are expected to communicate with one another.
|
||||
|
||||
Known Limitation
|
||||
----------------
|
||||
|
||||
The RDC tool has the following authentication limitations:
|
||||
|
||||
The client and server are hardcoded to look for the ``openssl`` certificate and key files in ``/etc/rdc``. There is no workaround available currently.
|
||||
|
||||
|
||||
Verify Files for Authentication
|
||||
===============================
|
||||
|
||||
Several SSL keys and certificates must be generated and installed on clients and servers for authentication to work properly. By default, the RDC server will look in the ``/etc/rdc`` folder for the following keys and certificates:
|
||||
|
||||
Client
|
||||
------
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
$ sudo tree /etc/rdc
|
||||
/etc/rdc
|
||||
|-- client
|
||||
|-- certs
|
||||
| |-- rdc_cacert.pem
|
||||
| |-- rdc_client_cert.pem
|
||||
|-- private
|
||||
|-- rdc_client_cert.key
|
||||
|
||||
|
||||
.. note::
|
||||
Machines that are clients and servers consist of both directory structures.
|
||||
|
||||
Server
|
||||
------
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
$ sudo tree /etc/rdc
|
||||
/etc/rdc
|
||||
|-- server
|
||||
|-- certs
|
||||
| |-- rdc_cacert.pem
|
||||
| |-- rdc_server_cert.pem
|
||||
|-- private
|
||||
|-- rdc_server_cert.key
|
||||
|
||||
@@ -0,0 +1,191 @@
|
||||
.. meta::
|
||||
:description: documentation of the installation, configuration, and use of the ROCm Data Center tool
|
||||
:keywords: ROCm Data Center tool, RDC, ROCm, API, reference, data type, support
|
||||
|
||||
.. _rdc-install:
|
||||
|
||||
******************************************
|
||||
Installing and running RDC tool
|
||||
******************************************
|
||||
|
||||
The ROCm Data Center tool (RDC) is part of the AMD ROCm software and available on the distributions supported by AMD ROCm. For RDC installation from prebuilt packages, follow the instructions in this section.
|
||||
|
||||
Prerequisites
|
||||
=============
|
||||
|
||||
The installation dependencies are described in `Dependencies in the README <https://github.com/ROCm/rdc?tab=readme-ov-file#dependencies>`_. To see the list of supported operating systems, refer to `System requirements <https://rocm.docs.amd.com/projects/install-on-linux/en/latest/reference/system-requirements.html>`_.
|
||||
|
||||
Install gRPC
|
||||
============
|
||||
|
||||
To see the instructions for building ``gRPC`` and ``protoc``, refer to `Building gRPC and protoc <https://github.com/ROCm/rdc#building-grpc-and-protoc>`_.
|
||||
|
||||
Authentication keys
|
||||
===================
|
||||
|
||||
The RDC tool can be used with or without authentication. If authentication is required you must configure proper authentication keys as described in *Authentication* in :ref:`rdc-handbook`.
|
||||
|
||||
Prebuilt packages
|
||||
=================
|
||||
|
||||
The RDC tool is packaged as part of the ROCm software repository. You must install the AMD ROCm software before installing RDC, as described in `ROCm installation <https://rocm.docs.amd.com/projects/install-on-linux/en/latest/>`_.
|
||||
|
||||
To install RDC after installing the ROCm package, use the following instructions.
|
||||
|
||||
|
||||
.. tab-set::
|
||||
|
||||
.. tab-item:: Ubuntu
|
||||
:sync: ubuntu-tab
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
$ sudo apt-get install rdc
|
||||
# or, to install a specific version
|
||||
$ sudo apt-get install rdc<x.y.z>
|
||||
|
||||
.. tab-item:: SLES 15 Service Pack 3
|
||||
:sync: sles-tab
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
$ sudo zypper install rdc
|
||||
# or, to install a specific version
|
||||
$ sudo zypper install rdc<x.y.z>
|
||||
|
||||
|
||||
Components
|
||||
==========
|
||||
|
||||
The components of RDC tool are as shown below:
|
||||
|
||||
.. figure:: ../data/install_components.png
|
||||
|
||||
High-level diagram of RDC components
|
||||
|
||||
|
||||
RDC (API) library
|
||||
-----------------
|
||||
|
||||
This library is the central piece, which interacts with different modules and provides all the features described. This shared library provides C API and Python bindings so that third-party tools should be able to use it directly if required.
|
||||
|
||||
RDC daemon (``rdcd``)
|
||||
---------------------
|
||||
|
||||
The ``rdcd`` daemon records telemetry information from GPUs. It also provides an interface to RDC command-line tool (``rdci``) running locally or remotely. It relies on the above RDC Library for all the core features.
|
||||
|
||||
RDC command-line tool (``rdci``)
|
||||
--------------------------------
|
||||
|
||||
A command-line tool to invoke all the features of the RDC tool. This CLI can be run locally or remotely.
|
||||
|
||||
AMDSMI library
|
||||
--------------
|
||||
|
||||
A stateless system management library that provides low-level interfaces to access GPU information
|
||||
|
||||
Starting RDC
|
||||
============
|
||||
|
||||
The RDC tool can be run in the following two modes. The feature set is similar in both the cases. You have the flexibility to choose the option that best fits your environment.
|
||||
|
||||
* :ref:`standalone`
|
||||
* :ref:`embedded`
|
||||
|
||||
The capability in each mode depends on the privileges you have for starting RDC. A normal user has access only to monitor (GPU telemetry) capabilities. A privileged user can run the tool with full capability. In the full capability mode, GPU configuration features can be invoked. This may or may not affect all the users and processes sharing the GPU.
|
||||
|
||||
.. _`standalone`:
|
||||
|
||||
Standalone mode
|
||||
---------------
|
||||
|
||||
This is the preferred mode of operation, as it does not have any external dependencies. To start RDC in standalone mode, RDC Server Daemon (``rdcd``) must run on each compute node. Refer to *Terminology* in :ref:`rdc-use` for more information. You can start ``rdcd`` as a ``systemd`` service or directly from the command-line.
|
||||
|
||||
Start RDC tool using ``systemd``
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
If multiple RDC versions are installed, copy `/opt/rocm-<x.y.z>/rdc/lib/rdc.service`, which is installed with the desired RDC version, to the ``systemd`` folder. The capability of RDC can be configured by modifying the ``rdc.service`` system configuration file. Use the ``systemctl`` command to start ``rdcd``.
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
$ systemctl start rdc
|
||||
|
||||
|
||||
By default, ``rdcd`` starts with full capability. To change to monitor only, comment out the following two lines:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
$ sudo vi /lib/systemd/system/rdc.service
|
||||
|
||||
# CapabilityBoundingSet=CAP_DAC_OVERRIDE
|
||||
# AmbientCapabilities=CAP_DAC_OVERRIDE
|
||||
|
||||
|
||||
.. note::
|
||||
``rdcd`` can be started by using the ``systemctl`` command.
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
$ systemctl start rdc
|
||||
|
||||
|
||||
If the GPU reset fails, restart the server. Note that restarting the server also initiates ``rdcd``. You may then encounter the following two scenarios:
|
||||
|
||||
* ``rdcd`` returns the correct GPU information to ``rdci``
|
||||
* ``rdcd`` returns the "No GPUs found on the system" error to ``rdci``. To resolve this error, restart ``rdcd`` with the following instruction:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
$ sudo systemctl restart rdcd
|
||||
|
||||
|
||||
Start RDC tool from the command-line
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
While ``systemctl`` is the preferred way to start ``rdcd``, you can also start directly from the command-line. The installation scripts create a default user - ``rdc``. Users have the option to edit the profile file (``rdc.service`` installed at ``/lib/systemd/system``) and change these lines accordingly:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
[Service]
|
||||
User=rdc
|
||||
Group=rdc
|
||||
|
||||
From the command-line, start ``rdcd`` as a user such as ``rdc``, or start it as ``root``:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
#Start as user rdc
|
||||
$ sudo -u rdc rdcd
|
||||
|
||||
# Start as root
|
||||
$ sudo rdcd
|
||||
|
||||
|
||||
In this use case, the ``rdc.service`` file mentioned in the previous section is not involved. Here, the capability of RDC is determined by the privilege of the user starting ``rdcd``. If ``rdcd`` is running under a normal user account it has the monitor-only capability. If ``rdcd`` is running as ``root`` then it has the full capability.
|
||||
|
||||
.. note::
|
||||
If a user other than ``rdc`` or ``root`` starts the ``rdcd`` daemon, the file ownership of the SSL keys mentioned in the Authentication section must be modified to allow read and write access.
|
||||
|
||||
Troubleshoot ``rdcd``
|
||||
---------------------
|
||||
|
||||
When ``rdcd`` is started using ``systemctl``, the logs can be viewed using the following command:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
$ journalctl -u rdc
|
||||
|
||||
|
||||
These messages provide useful status and debugging information. The logs can also help debug problems like ``rdcd`` failing to start, communication issues with a client, and others.
|
||||
|
||||
.. _`embedded`:
|
||||
|
||||
Embedded mode
|
||||
-------------
|
||||
|
||||
The embedded mode is useful if the end user has a monitoring agent running on the compute node. The monitoring agent can directly use the RDC library and will have a finer-grain control on how and when RDC features are invoked. For example, if the monitoring agent has a facility to synchronize across multiple nodes, it can synchronize GPU telemetry across these nodes.
|
||||
|
||||
The RDC daemon ``rdcd`` can be used as a reference code for this purpose. The dependency on ``gRPC`` is also eliminated if the RDC library is directly used.
|
||||
|
||||
.. caution::
|
||||
RDC command-line ``rdci`` will not function in this mode. Third-party monitoring software is responsible for providing the user interface and remote access/monitoring.
|
||||
Reference in New Issue
Block a user