RDC Doc formatting (#166)
* doc formatting * Update job_stats_sample.rst * Doc formatting --------- Co-authored-by: srawat <120587655+SwRaw@users.noreply.github.com>
This commit is contained in:
zatwierdzone przez
GitHub
rodzic
4a230f0180
commit
3e653b7ab3
@@ -82,27 +82,27 @@ Here are the steps to install Prometheus:
|
||||
|
||||
1. Install Prometheus client:
|
||||
|
||||
.. code-block:: shell
|
||||
.. code-block:: shell
|
||||
|
||||
$ pip install prometheus_client
|
||||
$ pip install prometheus_client
|
||||
|
||||
2. Run the Prometheus plugin:
|
||||
|
||||
.. code-block:: shell
|
||||
.. code-block:: shell
|
||||
|
||||
$ python rdc_prometheus.py
|
||||
$ python rdc_prometheus.py
|
||||
|
||||
3. Verify plugin:
|
||||
|
||||
.. code-block:: shell
|
||||
.. code-block:: shell
|
||||
|
||||
$ curl localhost:5000
|
||||
$ curl localhost:5000
|
||||
|
||||
gpu_util{gpu_index="0"} 0.0
|
||||
gpu_clock{gpu_index="0"} 300.0
|
||||
gpu_memory_total{gpu_index="0"} 4294.0
|
||||
power_usage{gpu_index="0"} 9.0
|
||||
gpu_memory_usage{gpu_index="0"} 134.0
|
||||
gpu_util{gpu_index="0"} 0.0
|
||||
gpu_clock{gpu_index="0"} 300.0
|
||||
gpu_memory_total{gpu_index="0"} 4294.0
|
||||
power_usage{gpu_index="0"} 9.0
|
||||
gpu_memory_usage{gpu_index="0"} 134.0
|
||||
|
||||
By default, the plugin runs in the standalone mode and connects to ``rdcd`` at ``localhost:50051`` to fetch fields. Ensure that the plugin uses the same authentication mode as ``rdcd``, for example, if ``rdcd`` runs with ``-u/--unauth`` option, the plugin must also use ``--rdc_unauth`` option.
|
||||
|
||||
@@ -179,11 +179,10 @@ To integrate Prometheus plugin in RDC, follow these steps:
|
||||
|
||||
2. Configure Prometheus targets:
|
||||
|
||||
Use the example configuration file ``rdc_prometheus_example.yml`` in the ``python_binding`` folder. This file refers to ``prometheus_targets.json``.
|
||||
Modify `prometheus_targets.json` to point to your compute nodes.
|
||||
Ensure that this is modified to point to the correct compute nodes.
|
||||
Use the example configuration file ``rdc_prometheus_example.yml`` in the ``python_binding`` folder. This file refers to ``prometheus_targets.json``. Modify ``prometheus_targets.json`` to point to your compute nodes.
|
||||
Ensure that this is modified to point to the correct compute nodes.
|
||||
|
||||
.. code-block:: shell
|
||||
.. code-block:: shell
|
||||
|
||||
// Sample file: prometheus_targets.json
|
||||
// Replace rdc_test*.amd.com to point the correct compute nodes
|
||||
@@ -197,15 +196,15 @@ Ensure that this is modified to point to the correct compute nodes.
|
||||
}
|
||||
]
|
||||
|
||||
.. note::
|
||||
.. note::
|
||||
|
||||
In the above example, there are two compute nodes, ``rdc_test1.adm.com`` and ``rdc_test2.adm.com``. Ensure that the Prometheus plugin is running on those compute nodes.
|
||||
In the above example, there are two compute nodes, ``rdc_test1.adm.com`` and ``rdc_test2.adm.com``. Ensure that the Prometheus plugin is running on those compute nodes.
|
||||
|
||||
3. Start the Prometheus plugin.
|
||||
|
||||
.. code-block:: shell
|
||||
.. code-block:: shell
|
||||
|
||||
% prometheus --config.file=<full path of the rdc_prometheus_example.yml>
|
||||
% prometheus --config.file=<full path of the rdc_prometheus_example.yml>
|
||||
|
||||
4. From the management node, open the URL http://localhost:9090 in the browser.
|
||||
|
||||
@@ -231,7 +230,7 @@ To install Grafana plugin, follow these steps:
|
||||
|
||||
3. To start Grafana, use:
|
||||
|
||||
.. code-block:: shell
|
||||
.. code-block:: shell
|
||||
|
||||
$ sudo systemctl start grafana-server
|
||||
$ sudo systemctl status grafana-server
|
||||
@@ -242,7 +241,6 @@ To install Grafana plugin, follow these steps:
|
||||
|
||||
.. figure:: ../data/integration_login.png
|
||||
|
||||
|
||||
Grafana integration
|
||||
--------------------
|
||||
|
||||
@@ -256,21 +254,21 @@ Grafana configuration
|
||||
|
||||
Firstly, add Prometheus as data source using the following steps:
|
||||
|
||||
1. Go to `Configuration`.
|
||||
1. Go to "Configuration".
|
||||
|
||||
.. image:: ../data/integration_config1.png
|
||||
.. image:: ../data/integration_config1.png
|
||||
|
||||
2. Select `Data Sources`.
|
||||
2. Select "Data Sources".
|
||||
|
||||
.. image:: ../data/integration_config2.png
|
||||
.. image:: ../data/integration_config2.png
|
||||
|
||||
3. Go to `Add data source`.
|
||||
3. Go to "Add data source".
|
||||
|
||||
.. image:: ../data/integration_config3.png
|
||||
.. image:: ../data/integration_config3.png
|
||||
|
||||
4. Select `Prometheus`.
|
||||
4. Select "Prometheus".
|
||||
|
||||
.. image:: ../data/integration_config4.png
|
||||
.. image:: ../data/integration_config4.png
|
||||
|
||||
.. note::
|
||||
|
||||
@@ -280,45 +278,45 @@ Firstly, add Prometheus as data source using the following steps:
|
||||
|
||||
Then, import RDC dashboard using the following steps:
|
||||
|
||||
1. Go to `+` and select `Import`.
|
||||
1. Go to "+" and select "Import".
|
||||
|
||||
2. Upload ``rdc_grafana_dashboard_example.json`` from the ``python_binding`` folder.
|
||||
|
||||
3. Select the desired compute node for visualization.
|
||||
|
||||
.. image:: ../data/integration_config6.png
|
||||
.. image:: ../data/integration_config6.png
|
||||
|
||||
Prometheus (Grafana) integration with automatic node detection
|
||||
==============================================================
|
||||
|
||||
RDC provides `Consul` to discover the ``rdc_prometheus`` service automatically. `Consul` is a service mesh solution providing a fully featured control plane with service discovery, configuration, and segmentation functionality. For more information, see `Consul <https://developer.hashicorp.com/consul/docs/intro>`_.
|
||||
RDC provides Consul to discover the ``rdc_prometheus`` service automatically. Consul is a service mesh solution providing a fully featured control plane with service discovery, configuration, and segmentation functionality. For more information, see `Consul <https://developer.hashicorp.com/consul/docs/intro>`_.
|
||||
|
||||
RDC uses `Consul` for health checks of RDC's integration with the `Prometheus` plugin (``rdc_prometheus``). These checks provide information on its efficiency.
|
||||
RDC uses Consul for health checks of RDC's integration with the Prometheus plugin (``rdc_prometheus``). These checks provide information on its efficiency.
|
||||
|
||||
With the `Consul` agent integration, a new compute node can be discovered automatically, which saves users from manually changing ``prometheus_targets.json`` to use `Consul`.
|
||||
With the Consul agent integration, a new compute node can be discovered automatically, which saves users from manually changing ``prometheus_targets.json`` to use Consul.
|
||||
|
||||
Installing the Consul agent for compute and management nodes
|
||||
------------------------------------------------------------
|
||||
|
||||
To install the latest `Consul` agent for compute and management nodes, follow these steps:
|
||||
To install the latest Consul agent for compute and management nodes, follow these steps:
|
||||
|
||||
1. To download and install the ``Consul`` agent, set up the ``apt`` repository:
|
||||
1. To download and install the Consul agent, set up the ``apt`` repository:
|
||||
|
||||
.. code-block:: shell
|
||||
.. code-block:: shell
|
||||
|
||||
$ curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -
|
||||
$ sudo apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main"
|
||||
$ sudo apt-get update && sudo apt-get install consul
|
||||
|
||||
2. Generate a key to encrypt the communication between `Consul` agents. The same key is used by both the compute and management nodes for communication.
|
||||
2. Generate a key to encrypt the communication between Consul agents. The same key is used by both the compute and management nodes for communication.
|
||||
|
||||
.. code-block:: shell
|
||||
.. code-block:: shell
|
||||
|
||||
$ consul keygen
|
||||
|
||||
For demonstration purposes, the following key is used in the configuration file:
|
||||
For demonstration purposes, the following key is used in the configuration file:
|
||||
|
||||
.. code-block:: shell
|
||||
.. code-block:: shell
|
||||
|
||||
$ consul keygen
|
||||
4lgGQXr3/R2QeTi5vEp7q5Xs1KoYBhCsk9+VgJZZHAo=
|
||||
@@ -326,7 +324,7 @@ For demonstration purposes, the following key is used in the configuration file:
|
||||
Setting up the Consul server in management nodes
|
||||
-------------------------------------------------
|
||||
|
||||
While ``Consul`` can function with one server, it's recommended to use three to five servers to avoid failure scenarios leading to data loss.
|
||||
While Consul can function with one server, it's recommended to use three to five servers to avoid failure scenarios leading to data loss.
|
||||
|
||||
.. note::
|
||||
For demonstration purposes, the configuration settings documented below are for a single server.
|
||||
@@ -335,40 +333,40 @@ To set up ``Consul`` server, follow these steps:
|
||||
|
||||
1. Create a configuration file ``/etc/consul.d/server.hcl``.
|
||||
|
||||
.. code-block:: shell
|
||||
.. code-block:: shell
|
||||
|
||||
server = true
|
||||
encrypt = "<CONSUL_ENCRYPTION_KEY>"
|
||||
bootstrap_expect = 1
|
||||
ui = true
|
||||
client_addr = "0.0.0.0"
|
||||
bind_addr = "<The IP address can be reached by client>"
|
||||
server = true
|
||||
encrypt = "<CONSUL_ENCRYPTION_KEY>"
|
||||
bootstrap_expect = 1
|
||||
ui = true
|
||||
client_addr = "0.0.0.0"
|
||||
bind_addr = "<The IP address can be reached by client>"
|
||||
|
||||
Here is how to use the variables in the configuration file:
|
||||
Here is how to use the variables in the configuration file:
|
||||
|
||||
* Run the agent in server mode by setting ``server`` to ``true``.
|
||||
* Set ``encrypt`` to the key generated in the first step.
|
||||
* The ``bootstrap_expect`` variable indicates the number of servers required to form the first `Consul` cluster. Set this variable to ``1`` to allow a cluster with a single server.
|
||||
* The ``bootstrap_expect`` variable indicates the number of servers required to form the first Consul cluster. Set this variable to ``1`` to allow a cluster with a single server.
|
||||
* The User Interface (``ui``) variable when set to ``true`` enables the Consul web UI.
|
||||
* The ``client_addr`` variable is used to connect the API and UI.
|
||||
* The ``bind_addr`` variable is used to connect the client to the server. If you have multiple private IP addresses, use the address that can connect to a client.
|
||||
|
||||
2. Start the agent.
|
||||
|
||||
.. code-block:: shell
|
||||
.. code-block:: shell
|
||||
|
||||
$ sudo consul agent -config-dir=/etc/consul.d/
|
||||
$ sudo consul agent -config-dir=/etc/consul.d/
|
||||
|
||||
3. Browse to http://localhost:8500/ on the management node to see a single instance running.
|
||||
|
||||
Setting up the Consul client in compute nodes
|
||||
---------------------------------------------
|
||||
|
||||
To set up `Consul` client, follow these steps:
|
||||
To set up Consul client, follow these steps:
|
||||
|
||||
1. Create a configuration file ``/etc/consul.d/client.hcl``.
|
||||
|
||||
.. code-block:: shell
|
||||
.. code-block:: shell
|
||||
|
||||
server = false
|
||||
encrypt = "<CONSUL_ENCRYPTION_KEY>"
|
||||
@@ -376,35 +374,36 @@ To set up `Consul` client, follow these steps:
|
||||
client_addr = "0.0.0.0"
|
||||
bind_addr = "<The IP address can reach server>"
|
||||
|
||||
.. note::
|
||||
Use the same ``CONSUL_ENCRYPTION_KEY`` as the servers. In the ``retry_join``, use the IP address of the management nodes.
|
||||
.. note::
|
||||
|
||||
Use the same ``CONSUL_ENCRYPTION_KEY`` as the servers. In the ``retry_join``, use the IP address of the management nodes.
|
||||
|
||||
2. Start the Consul agent.
|
||||
|
||||
.. code-block:: shell
|
||||
.. code-block:: shell
|
||||
|
||||
$ sudo consul agent -config-dir=/etc/consul.d/
|
||||
$ sudo consul agent -config-dir=/etc/consul.d/
|
||||
|
||||
To see if the client has joined the `Consul`, use:
|
||||
To see if the client has joined the Consul, use:
|
||||
|
||||
.. code-block:: shell
|
||||
.. code-block:: shell
|
||||
|
||||
$ consul members
|
||||
Node Address Status Type Build Protocol DC Segment
|
||||
management-node 10.4.22.70:8301 alive server 1.9.3 2 dc1 <all>
|
||||
compute-node 10.4.22.112:8301 alive client 1.9.3 2 dc1 <default>
|
||||
|
||||
3. Set up the `Consul` client to monitor the health of the RDC Prometheus plugin.
|
||||
3. Set up the Consul client to monitor the health of the RDC Prometheus plugin.
|
||||
|
||||
4. Start the RDC Prometheus plugin.
|
||||
|
||||
.. code-block:: shell
|
||||
.. code-block:: shell
|
||||
|
||||
$ python rdc_prometheus.py --rdc_embedded
|
||||
$ python rdc_prometheus.py --rdc_embedded
|
||||
|
||||
5. Add the configuration file ``/etc/consul.d/rdc_prometheus.hcl``.
|
||||
|
||||
.. code-block:: shell
|
||||
.. code-block:: shell
|
||||
|
||||
{
|
||||
"service": {
|
||||
@@ -425,30 +424,30 @@ To see if the client has joined the `Consul`, use:
|
||||
}
|
||||
}
|
||||
|
||||
.. note::
|
||||
.. note::
|
||||
|
||||
By default, the `Prometheus` plugin uses port 5000. If you don't use the default setting, change the configuration file accordingly.
|
||||
By default, the Prometheus plugin uses port 5000. If you don't use the default setting, change the configuration file accordingly.
|
||||
|
||||
6. After updating the configuration file, restart the `Consul` client agent.
|
||||
6. After updating the configuration file, restart the Consul client agent.
|
||||
|
||||
.. code-block:: shell
|
||||
.. code-block:: shell
|
||||
|
||||
$ sudo consul agent -config-dir=/etc/consul.d/
|
||||
$ sudo consul agent -config-dir=/etc/consul.d/
|
||||
|
||||
7. Enable the :ref:`Prometheus <prometheus>` integration in the management node.
|
||||
|
||||
8. In the management node, inspect the service.
|
||||
|
||||
.. code-block:: shell
|
||||
.. code-block:: shell
|
||||
|
||||
$ consul catalog nodes -service=rdc_prometheus
|
||||
$ consul catalog nodes -service=rdc_prometheus
|
||||
|
||||
Node ID Address DC
|
||||
compute-node 76694ab1 10.4.22.112 dc1
|
||||
Node ID Address DC
|
||||
compute-node 76694ab1 10.4.22.112 dc1
|
||||
|
||||
9. Create a new `Prometheus` configuration ``rdc_prometheus_consul.yml`` file for the `Consul` integration.
|
||||
9. Create a new Prometheus configuration ``rdc_prometheus_consul.yml`` file for the Consul integration.
|
||||
|
||||
.. code-block:: shell
|
||||
.. code-block:: shell
|
||||
|
||||
global:
|
||||
scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
|
||||
@@ -464,16 +463,17 @@ To see if the client has joined the `Consul`, use:
|
||||
- source_labels: [__meta_consul_service]
|
||||
target_label: job
|
||||
|
||||
.. note::
|
||||
When running the `Consul` server and `Prometheus` on the same machine, change the server under ``consul_sd_configs`` to your `Consul` server address.
|
||||
.. note::
|
||||
|
||||
When running the Consul server and Prometheus on the same machine, change the server under ``consul_sd_configs`` to your Consul server address.
|
||||
|
||||
10. Start Prometheus.
|
||||
|
||||
.. code-block:: shell
|
||||
.. code-block:: shell
|
||||
|
||||
$ ./prometheus --config.file="rdc_prometheus_consul.yml"
|
||||
$ ./prometheus --config.file="rdc_prometheus_consul.yml"
|
||||
|
||||
11. Browse the `Prometheus` UI at http://localhost:9090 on the management node and query RDC `Prometheus` metrics. Ensure that the plugin starts before running the query.
|
||||
11. Browse the Prometheus UI at http://localhost:9090 on the management node and query RDC Prometheus metrics. Ensure that the plugin starts before running the query.
|
||||
|
||||
Reliability, Availability, and Serviceability plugin
|
||||
=====================================================
|
||||
@@ -490,13 +490,13 @@ Prerequisite
|
||||
|
||||
- Ensure that the GPU supports RAS.
|
||||
|
||||
.. note::
|
||||
.. note::
|
||||
|
||||
The RAS library is installed as part of the RDC installation. No additional configuration is required for RDC.
|
||||
The RAS library is installed as part of the RDC installation. No additional configuration is required for RDC.
|
||||
|
||||
- RDC installation dynamically loads the RAS library ``librdc_ras.so``. The configuration files required by the RAS library are installed in the ``sp3`` and ``config`` folders.
|
||||
|
||||
.. code-block:: shell
|
||||
.. code-block:: shell
|
||||
|
||||
% ls /opt/rocm-4.2.0/rdc/lib
|
||||
... librdc_ras.so ...
|
||||
|
||||
@@ -27,9 +27,9 @@ Here are the steps to start RDC using ``systemctl`` command, which runs RDC in t
|
||||
|
||||
1. Copy the service file:
|
||||
|
||||
.. code-block:: shell
|
||||
.. code-block:: shell
|
||||
|
||||
sudo cp /opt/rocm/libexec/rdc/rdc.service /etc/systemd/system/
|
||||
sudo cp /opt/rocm/libexec/rdc/rdc.service /etc/systemd/system/
|
||||
|
||||
2. Configure capabilities:
|
||||
|
||||
@@ -44,30 +44,30 @@ Here are the steps to start RDC using ``systemctl`` command, which runs RDC in t
|
||||
|
||||
3. Start the service:
|
||||
|
||||
.. code-block:: shell
|
||||
.. code-block:: shell
|
||||
|
||||
sudo systemctl start rdc
|
||||
sudo systemctl status rdc
|
||||
sudo systemctl start rdc
|
||||
sudo systemctl status rdc
|
||||
|
||||
4. Modify RDCD options:
|
||||
|
||||
Edit ``/opt/rocm/etc/rdc_options`` to append any additional RDCD parameters.
|
||||
Edit ``/opt/rocm/etc/rdc_options`` to append any additional RDCD parameters.
|
||||
|
||||
.. code-block:: shell
|
||||
.. code-block:: shell
|
||||
|
||||
sudo nano /opt/rocm/etc/rdc_options
|
||||
sudo nano /opt/rocm/etc/rdc_options
|
||||
|
||||
Example configuration:
|
||||
Example configuration:
|
||||
|
||||
.. code-block:: shell
|
||||
.. code-block:: shell
|
||||
|
||||
RDC_OPTS="-p 50051 -u -d"
|
||||
RDC_OPTS="-p 50051 -u -d"
|
||||
|
||||
Flags:
|
||||
Flags:
|
||||
|
||||
- `-p 50051` : Use port 50051
|
||||
- `-u` : Unauthenticated mode
|
||||
- `-d` : Enable debug messages
|
||||
- `-p 50051` : Use port 50051
|
||||
- `-u` : Unauthenticated mode
|
||||
- `-d` : Enable debug messages
|
||||
|
||||
Starting RDC using command line as a user
|
||||
------------------------------------------
|
||||
@@ -240,7 +240,7 @@ The embedded mode is especially useful for a monitoring agent running on the com
|
||||
|
||||
The RDC daemon ``rdcd`` can be used as a reference code for this purpose. The dependency on ``gRPC`` is also eliminated, if the RDC library is directly used.
|
||||
|
||||
- To run RDC in embedded mode, use:
|
||||
To run RDC in embedded mode, use:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
@@ -261,25 +261,25 @@ The RDCD logs provide useful status and debugging information. The logs can also
|
||||
|
||||
- View logs:
|
||||
|
||||
When ``rdcd`` is started using ``systemctl``, you can view the logs using:
|
||||
When ``rdcd`` is started using ``systemctl``, you can view the logs using:
|
||||
|
||||
.. code-block:: shell
|
||||
.. code-block:: shell
|
||||
|
||||
$ journalctl -u rdc
|
||||
$ journalctl -u rdc
|
||||
|
||||
- Run RDCD with debug logs:
|
||||
|
||||
.. code-block:: shell
|
||||
.. code-block:: shell
|
||||
|
||||
RDC_LOG=DEBUG /opt/rocm/bin/rdcd
|
||||
RDC_LOG=DEBUG /opt/rocm/bin/rdcd
|
||||
|
||||
Logging levels supported: `ERROR`, `INFO`, `DEBUG`.
|
||||
Logging levels supported: `ERROR`, `INFO`, `DEBUG`.
|
||||
|
||||
- Enable additional logging messages:
|
||||
|
||||
.. code-block:: shell
|
||||
.. code-block:: shell
|
||||
|
||||
export RSMI_LOGGING=3
|
||||
export RSMI_LOGGING=3
|
||||
|
||||
If the GPU reset fails, restart the server. Note that restarting the server also initiates ``rdcd``. You might then encounter the following two scenarios:
|
||||
|
||||
@@ -311,50 +311,50 @@ Known issues
|
||||
|
||||
- dmon RocProfiler fields return zeros
|
||||
|
||||
**Solution:**
|
||||
**Solution:**
|
||||
|
||||
Set the ``HSA_TOOLS_LIB`` environment variable before running a compute job.
|
||||
Set the ``HSA_TOOLS_LIB`` environment variable before running a compute job.
|
||||
|
||||
.. code-block:: shell
|
||||
.. code-block:: shell
|
||||
|
||||
export HSA_TOOLS_LIB=/opt/rocm/lib/librocprofiler64.so.1
|
||||
export HSA_TOOLS_LIB=/opt/rocm/lib/librocprofiler64.so.1
|
||||
|
||||
**Example:**
|
||||
**Example:**
|
||||
|
||||
.. code-block:: shell
|
||||
.. code-block:: shell
|
||||
|
||||
# Terminal 1
|
||||
rdcd -u
|
||||
# Terminal 1
|
||||
rdcd -u
|
||||
|
||||
# Terminal 2
|
||||
export HSA_TOOLS_LIB=/opt/rocm/lib/librocprofiler64.so.1
|
||||
gpu-burn
|
||||
# Terminal 2
|
||||
export HSA_TOOLS_LIB=/opt/rocm/lib/librocprofiler64.so.1
|
||||
gpu-burn
|
||||
|
||||
# Terminal 3
|
||||
rdci dmon -u -e 800,801 -i 0 -c 1
|
||||
# Terminal 3
|
||||
rdci dmon -u -e 800,801 -i 0 -c 1
|
||||
|
||||
# Output:
|
||||
GPU OCCUPANCY_PERCENT ACTIVE_WAVES
|
||||
0 001.000 32640.000
|
||||
# Output:
|
||||
GPU OCCUPANCY_PERCENT ACTIVE_WAVES
|
||||
0 001.000 32640.000
|
||||
|
||||
- HSA_STATUS_ERROR_OUT_OF_RESOURCES
|
||||
|
||||
**Error message:**
|
||||
**Error message:**
|
||||
|
||||
.. code-block:: shell
|
||||
.. code-block:: shell
|
||||
|
||||
terminate called after throwing an instance of 'std::runtime_error'
|
||||
what(): hsa error code: 4104 HSA_STATUS_ERROR_OUT_OF_RESOURCES: The runtime failed to allocate the necessary resources. This error may also occur when the core runtime library needs to spawn threads or create internal OS-specific events.
|
||||
Aborted (core dumped)
|
||||
terminate called after throwing an instance of 'std::runtime_error'
|
||||
what(): hsa error code: 4104 HSA_STATUS_ERROR_OUT_OF_RESOURCES: The runtime failed to allocate the necessary resources. This error may also occur when the core runtime library needs to spawn threads or create internal OS-specific events.
|
||||
Aborted (core dumped)
|
||||
|
||||
**Solution:**
|
||||
**Solution:**
|
||||
|
||||
Follow these steps to check for missing groups:
|
||||
Follow these steps to check for missing groups:
|
||||
|
||||
1. Ensure video and render groups exist.
|
||||
1. Ensure video and render groups exist.
|
||||
|
||||
.. code-block:: shell
|
||||
.. code-block:: shell
|
||||
|
||||
sudo usermod -aG video,render $USER
|
||||
sudo usermod -aG video,render $USER
|
||||
|
||||
2. Logout and login to apply group changes.
|
||||
2. Logout and login to apply group changes.
|
||||
|
||||
@@ -56,33 +56,33 @@ With the GPU groups feature, you can create, delete, and list logical groups of
|
||||
|
||||
- To create a group, use:
|
||||
|
||||
.. code-block:: shell
|
||||
.. code-block:: shell
|
||||
|
||||
$ rdci group -c GPU_GROUP
|
||||
Successfully created a group with a group ID 1
|
||||
$ rdci group -c GPU_GROUP
|
||||
Successfully created a group with a group ID 1
|
||||
|
||||
- To add GPUs to a group, use:
|
||||
|
||||
.. code-block:: shell
|
||||
.. code-block:: shell
|
||||
|
||||
$ rdci group -g 1 -a 0,1
|
||||
Successfully added the GPU 0,1 to group 1
|
||||
$ rdci group -g 1 -a 0,1
|
||||
Successfully added the GPU 0,1 to group 1
|
||||
|
||||
- To delete a group, use:
|
||||
|
||||
.. code-block:: shell
|
||||
.. code-block:: shell
|
||||
|
||||
$ rdci group -d 1
|
||||
Successfully removed group 1
|
||||
$ rdci group -d 1
|
||||
Successfully removed group 1
|
||||
|
||||
- To list groups, use:
|
||||
|
||||
.. code-block:: shell
|
||||
.. code-block:: shell
|
||||
|
||||
$ rdci group –l
|
||||
1 group found
|
||||
$ rdci group –l
|
||||
1 group found
|
||||
|
||||
.. list-table::
|
||||
.. list-table::
|
||||
|
||||
* - **Group ID**
|
||||
- **Group name**
|
||||
@@ -99,19 +99,19 @@ The field groups feature provides you the options to create, delete, list field
|
||||
|
||||
- To create a field group, use:
|
||||
|
||||
.. code-block:: shell
|
||||
.. code-block:: shell
|
||||
|
||||
$ rdci fieldgroup -c <fgroup> -f 150,155
|
||||
Successfully created a field group with a group ID 1
|
||||
|
||||
- To list field groups, use:
|
||||
|
||||
.. code-block:: shell
|
||||
.. code-block:: shell
|
||||
|
||||
$ rdci fieldgroup -l
|
||||
1 group found
|
||||
$ rdci fieldgroup -l
|
||||
1 group found
|
||||
|
||||
.. list-table::
|
||||
.. list-table::
|
||||
|
||||
* - **Group ID**
|
||||
- **Group Name**
|
||||
@@ -123,10 +123,10 @@ The field groups feature provides you the options to create, delete, list field
|
||||
|
||||
- To delete a field group, use:
|
||||
|
||||
.. code-block:: shell
|
||||
.. code-block:: shell
|
||||
|
||||
$ rdci fieldgroup -d 1
|
||||
Successfully removed field group 1
|
||||
$ rdci fieldgroup -d 1
|
||||
Successfully removed field group 1
|
||||
|
||||
Monitor errors
|
||||
===============
|
||||
@@ -135,15 +135,15 @@ To get the Reliability, Availability, and Serviceability (RAS) Error-Correcting
|
||||
|
||||
- Correctable ECC errors:
|
||||
|
||||
.. code-block:: shell
|
||||
.. code-block:: shell
|
||||
|
||||
312 ``RDC_FI_ECC_CORRECT_TOTAL``
|
||||
312 ``RDC_FI_ECC_CORRECT_TOTAL``
|
||||
|
||||
- Uncorrectable ECC errors:
|
||||
|
||||
.. code-block:: shell
|
||||
.. code-block:: shell
|
||||
|
||||
313 ``RDC_FI_ECC_UNCORRECT_TOTAL``
|
||||
313 ``RDC_FI_ECC_UNCORRECT_TOTAL``
|
||||
|
||||
Device monitoring
|
||||
==================
|
||||
@@ -174,25 +174,25 @@ The job stats is used to display GPU statistics for any given workload.
|
||||
|
||||
- To start recording stats, use:
|
||||
|
||||
.. code-block:: shell
|
||||
.. code-block:: shell
|
||||
|
||||
$ rdci stats -s 2 -g 1
|
||||
Successfully started recording job 2 with a group ID 1
|
||||
$ rdci stats -s 2 -g 1
|
||||
Successfully started recording job 2 with a group ID 1
|
||||
|
||||
- To stop recording stats, use:
|
||||
|
||||
.. code-block:: shell
|
||||
.. code-block:: shell
|
||||
|
||||
$ rdci stats -x 2
|
||||
Successfully stopped recording job 2
|
||||
$ rdci stats -x 2
|
||||
Successfully stopped recording job 2
|
||||
|
||||
- To display job stats, use:
|
||||
|
||||
.. code-block:: shell
|
||||
.. code-block:: shell
|
||||
|
||||
$ rdci stats -j 2
|
||||
$ rdci stats -j 2
|
||||
|
||||
.. list-table::
|
||||
.. list-table::
|
||||
|
||||
* - **Summary**
|
||||
- **Executive status**
|
||||
|
||||
@@ -18,16 +18,24 @@ To install RDC from source, ensure that your system meets the following requirem
|
||||
- **Supported platforms:** AMD ROCm-supported platform. See the `list of supported operating systems <https://rocm.docs.amd.com/projects/install-on-linux/en/latest/reference/system-requirements.html#supported-operating-systems>`_.
|
||||
|
||||
- **Dependencies:**
|
||||
|
||||
- CMake >= 3.15
|
||||
|
||||
- g++ (5.4.0)
|
||||
|
||||
- gRPC and protoc
|
||||
|
||||
- libcap-dev
|
||||
|
||||
- :doc:`AMD ROCm platform <rocm:index>` including:
|
||||
|
||||
- :doc:`AMDSMI library <amdsmi:index>`
|
||||
- `ROCK kernel driver <https://github.com/ROCm/ROCK-Kernel-Driver>`_
|
||||
|
||||
For building latest documentation:
|
||||
|
||||
- Doxygen (1.8.11)
|
||||
|
||||
- LaTeX (pdfTeX 3.14159265-2.6-1.40.16)
|
||||
|
||||
.. code-block:: shell
|
||||
@@ -47,14 +55,14 @@ gRPC and Protoc must be built from source as the prebuilt packages are not avail
|
||||
|
||||
1. Install the required tools:
|
||||
|
||||
.. code-block:: shell
|
||||
.. code-block:: shell
|
||||
|
||||
sudo apt-get update
|
||||
sudo apt-get install automake make g++ unzip build-essential autoconf libtool pkg-config libgflags-dev libgtest-dev clang libc++-dev curl
|
||||
|
||||
2. Clone and build gRPC:
|
||||
|
||||
.. code-block:: shell
|
||||
.. code-block:: shell
|
||||
|
||||
git clone -b v1.61.0 https://github.com/grpc/grpc --depth=1 --shallow-submodules --recurse-submodules
|
||||
cd grpc
|
||||
@@ -77,14 +85,14 @@ Build RDC
|
||||
|
||||
1. Clone the RDC repository:
|
||||
|
||||
.. code-block:: shell
|
||||
.. code-block:: shell
|
||||
|
||||
git clone https://github.com/ROCm/rdc
|
||||
cd rdc
|
||||
|
||||
2. Configure the build:
|
||||
|
||||
.. code-block:: shell
|
||||
.. code-block:: shell
|
||||
|
||||
cmake -B build -DGRPC_ROOT="$GRPC_ROOT"
|
||||
|
||||
@@ -116,14 +124,14 @@ Build RDC
|
||||
|
||||
4. Build and install:
|
||||
|
||||
.. code-block:: shell
|
||||
.. code-block:: shell
|
||||
|
||||
make -C build -j $(nproc)
|
||||
sudo make -C build install
|
||||
|
||||
5. Update system library path:
|
||||
|
||||
.. code-block:: shell
|
||||
.. code-block:: shell
|
||||
|
||||
export RDC_LIB_DIR=/opt/rocm/lib/rdc
|
||||
export GRPC_LIB_DIR="/opt/grpc/lib"
|
||||
|
||||
Reference in New Issue
Block a user