From 3e653b7ab3a7930d0c08ea913f952484efed7464 Mon Sep 17 00:00:00 2001 From: "Rawat, Swati" Date: Mon, 5 May 2025 23:38:33 +0530 Subject: [PATCH] RDC Doc formatting (#166) * doc formatting * Update job_stats_sample.rst * Doc formatting --------- Co-authored-by: srawat <120587655+SwRaw@users.noreply.github.com> --- docs/how-to/integration.rst | 168 ++++++++++++++--------------- docs/how-to/using_RDC.rst | 102 +++++++++--------- docs/how-to/using_RDC_features.rst | 68 ++++++------ docs/install/install.rst | 20 ++-- 4 files changed, 183 insertions(+), 175 deletions(-) diff --git a/docs/how-to/integration.rst b/docs/how-to/integration.rst index 319be17f2b..a35db5b069 100644 --- a/docs/how-to/integration.rst +++ b/docs/how-to/integration.rst @@ -82,27 +82,27 @@ Here are the steps to install Prometheus: 1. Install Prometheus client: -.. code-block:: shell + .. code-block:: shell - $ pip install prometheus_client + $ pip install prometheus_client 2. Run the Prometheus plugin: -.. code-block:: shell + .. code-block:: shell - $ python rdc_prometheus.py + $ python rdc_prometheus.py 3. Verify plugin: -.. code-block:: shell + .. code-block:: shell - $ curl localhost:5000 + $ curl localhost:5000 - gpu_util{gpu_index="0"} 0.0 - gpu_clock{gpu_index="0"} 300.0 - gpu_memory_total{gpu_index="0"} 4294.0 - power_usage{gpu_index="0"} 9.0 - gpu_memory_usage{gpu_index="0"} 134.0 + gpu_util{gpu_index="0"} 0.0 + gpu_clock{gpu_index="0"} 300.0 + gpu_memory_total{gpu_index="0"} 4294.0 + power_usage{gpu_index="0"} 9.0 + gpu_memory_usage{gpu_index="0"} 134.0 By default, the plugin runs in the standalone mode and connects to ``rdcd`` at ``localhost:50051`` to fetch fields. Ensure that the plugin uses the same authentication mode as ``rdcd``, for example, if ``rdcd`` runs with ``-u/--unauth`` option, the plugin must also use ``--rdc_unauth`` option. @@ -179,11 +179,10 @@ To integrate Prometheus plugin in RDC, follow these steps: 2. Configure Prometheus targets: -Use the example configuration file ``rdc_prometheus_example.yml`` in the ``python_binding`` folder. This file refers to ``prometheus_targets.json``. -Modify `prometheus_targets.json` to point to your compute nodes. -Ensure that this is modified to point to the correct compute nodes. + Use the example configuration file ``rdc_prometheus_example.yml`` in the ``python_binding`` folder. This file refers to ``prometheus_targets.json``. Modify ``prometheus_targets.json`` to point to your compute nodes. + Ensure that this is modified to point to the correct compute nodes. -.. code-block:: shell + .. code-block:: shell // Sample file: prometheus_targets.json // Replace rdc_test*.amd.com to point the correct compute nodes @@ -197,15 +196,15 @@ Ensure that this is modified to point to the correct compute nodes. } ] -.. note:: + .. note:: - In the above example, there are two compute nodes, ``rdc_test1.adm.com`` and ``rdc_test2.adm.com``. Ensure that the Prometheus plugin is running on those compute nodes. + In the above example, there are two compute nodes, ``rdc_test1.adm.com`` and ``rdc_test2.adm.com``. Ensure that the Prometheus plugin is running on those compute nodes. 3. Start the Prometheus plugin. -.. code-block:: shell + .. code-block:: shell - % prometheus --config.file= + % prometheus --config.file= 4. From the management node, open the URL http://localhost:9090 in the browser. @@ -231,7 +230,7 @@ To install Grafana plugin, follow these steps: 3. To start Grafana, use: -.. code-block:: shell + .. code-block:: shell $ sudo systemctl start grafana-server $ sudo systemctl status grafana-server @@ -242,7 +241,6 @@ To install Grafana plugin, follow these steps: .. figure:: ../data/integration_login.png - Grafana integration -------------------- @@ -256,21 +254,21 @@ Grafana configuration Firstly, add Prometheus as data source using the following steps: -1. Go to `Configuration`. +1. Go to "Configuration". -.. image:: ../data/integration_config1.png + .. image:: ../data/integration_config1.png -2. Select `Data Sources`. +2. Select "Data Sources". -.. image:: ../data/integration_config2.png + .. image:: ../data/integration_config2.png -3. Go to `Add data source`. +3. Go to "Add data source". -.. image:: ../data/integration_config3.png + .. image:: ../data/integration_config3.png -4. Select `Prometheus`. +4. Select "Prometheus". -.. image:: ../data/integration_config4.png + .. image:: ../data/integration_config4.png .. note:: @@ -280,45 +278,45 @@ Firstly, add Prometheus as data source using the following steps: Then, import RDC dashboard using the following steps: -1. Go to `+` and select `Import`. +1. Go to "+" and select "Import". 2. Upload ``rdc_grafana_dashboard_example.json`` from the ``python_binding`` folder. 3. Select the desired compute node for visualization. -.. image:: ../data/integration_config6.png + .. image:: ../data/integration_config6.png Prometheus (Grafana) integration with automatic node detection ============================================================== -RDC provides `Consul` to discover the ``rdc_prometheus`` service automatically. `Consul` is a service mesh solution providing a fully featured control plane with service discovery, configuration, and segmentation functionality. For more information, see `Consul `_. +RDC provides Consul to discover the ``rdc_prometheus`` service automatically. Consul is a service mesh solution providing a fully featured control plane with service discovery, configuration, and segmentation functionality. For more information, see `Consul `_. -RDC uses `Consul` for health checks of RDC's integration with the `Prometheus` plugin (``rdc_prometheus``). These checks provide information on its efficiency. +RDC uses Consul for health checks of RDC's integration with the Prometheus plugin (``rdc_prometheus``). These checks provide information on its efficiency. -With the `Consul` agent integration, a new compute node can be discovered automatically, which saves users from manually changing ``prometheus_targets.json`` to use `Consul`. +With the Consul agent integration, a new compute node can be discovered automatically, which saves users from manually changing ``prometheus_targets.json`` to use Consul. Installing the Consul agent for compute and management nodes ------------------------------------------------------------ -To install the latest `Consul` agent for compute and management nodes, follow these steps: +To install the latest Consul agent for compute and management nodes, follow these steps: -1. To download and install the ``Consul`` agent, set up the ``apt`` repository: +1. To download and install the Consul agent, set up the ``apt`` repository: -.. code-block:: shell + .. code-block:: shell $ curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add - $ sudo apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main" $ sudo apt-get update && sudo apt-get install consul -2. Generate a key to encrypt the communication between `Consul` agents. The same key is used by both the compute and management nodes for communication. +2. Generate a key to encrypt the communication between Consul agents. The same key is used by both the compute and management nodes for communication. -.. code-block:: shell + .. code-block:: shell $ consul keygen -For demonstration purposes, the following key is used in the configuration file: + For demonstration purposes, the following key is used in the configuration file: -.. code-block:: shell + .. code-block:: shell $ consul keygen 4lgGQXr3/R2QeTi5vEp7q5Xs1KoYBhCsk9+VgJZZHAo= @@ -326,7 +324,7 @@ For demonstration purposes, the following key is used in the configuration file: Setting up the Consul server in management nodes ------------------------------------------------- -While ``Consul`` can function with one server, it's recommended to use three to five servers to avoid failure scenarios leading to data loss. +While Consul can function with one server, it's recommended to use three to five servers to avoid failure scenarios leading to data loss. .. note:: For demonstration purposes, the configuration settings documented below are for a single server. @@ -335,40 +333,40 @@ To set up ``Consul`` server, follow these steps: 1. Create a configuration file ``/etc/consul.d/server.hcl``. -.. code-block:: shell + .. code-block:: shell - server = true - encrypt = "" - bootstrap_expect = 1 - ui = true - client_addr = "0.0.0.0" - bind_addr = "" + server = true + encrypt = "" + bootstrap_expect = 1 + ui = true + client_addr = "0.0.0.0" + bind_addr = "" -Here is how to use the variables in the configuration file: + Here is how to use the variables in the configuration file: * Run the agent in server mode by setting ``server`` to ``true``. * Set ``encrypt`` to the key generated in the first step. - * The ``bootstrap_expect`` variable indicates the number of servers required to form the first `Consul` cluster. Set this variable to ``1`` to allow a cluster with a single server. + * The ``bootstrap_expect`` variable indicates the number of servers required to form the first Consul cluster. Set this variable to ``1`` to allow a cluster with a single server. * The User Interface (``ui``) variable when set to ``true`` enables the Consul web UI. * The ``client_addr`` variable is used to connect the API and UI. * The ``bind_addr`` variable is used to connect the client to the server. If you have multiple private IP addresses, use the address that can connect to a client. 2. Start the agent. -.. code-block:: shell + .. code-block:: shell - $ sudo consul agent -config-dir=/etc/consul.d/ + $ sudo consul agent -config-dir=/etc/consul.d/ 3. Browse to http://localhost:8500/ on the management node to see a single instance running. Setting up the Consul client in compute nodes --------------------------------------------- -To set up `Consul` client, follow these steps: +To set up Consul client, follow these steps: 1. Create a configuration file ``/etc/consul.d/client.hcl``. -.. code-block:: shell + .. code-block:: shell server = false encrypt = "" @@ -376,35 +374,36 @@ To set up `Consul` client, follow these steps: client_addr = "0.0.0.0" bind_addr = "" -.. note:: - Use the same ``CONSUL_ENCRYPTION_KEY`` as the servers. In the ``retry_join``, use the IP address of the management nodes. + .. note:: + + Use the same ``CONSUL_ENCRYPTION_KEY`` as the servers. In the ``retry_join``, use the IP address of the management nodes. 2. Start the Consul agent. -.. code-block:: shell + .. code-block:: shell - $ sudo consul agent -config-dir=/etc/consul.d/ + $ sudo consul agent -config-dir=/etc/consul.d/ -To see if the client has joined the `Consul`, use: + To see if the client has joined the Consul, use: -.. code-block:: shell + .. code-block:: shell $ consul members Node Address Status Type Build Protocol DC Segment management-node 10.4.22.70:8301 alive server 1.9.3 2 dc1 compute-node 10.4.22.112:8301 alive client 1.9.3 2 dc1 -3. Set up the `Consul` client to monitor the health of the RDC Prometheus plugin. +3. Set up the Consul client to monitor the health of the RDC Prometheus plugin. 4. Start the RDC Prometheus plugin. -.. code-block:: shell + .. code-block:: shell - $ python rdc_prometheus.py --rdc_embedded + $ python rdc_prometheus.py --rdc_embedded 5. Add the configuration file ``/etc/consul.d/rdc_prometheus.hcl``. -.. code-block:: shell + .. code-block:: shell { "service": { @@ -425,30 +424,30 @@ To see if the client has joined the `Consul`, use: } } -.. note:: + .. note:: - By default, the `Prometheus` plugin uses port 5000. If you don't use the default setting, change the configuration file accordingly. + By default, the Prometheus plugin uses port 5000. If you don't use the default setting, change the configuration file accordingly. -6. After updating the configuration file, restart the `Consul` client agent. +6. After updating the configuration file, restart the Consul client agent. -.. code-block:: shell + .. code-block:: shell - $ sudo consul agent -config-dir=/etc/consul.d/ + $ sudo consul agent -config-dir=/etc/consul.d/ 7. Enable the :ref:`Prometheus ` integration in the management node. 8. In the management node, inspect the service. -.. code-block:: shell + .. code-block:: shell - $ consul catalog nodes -service=rdc_prometheus + $ consul catalog nodes -service=rdc_prometheus - Node ID Address DC - compute-node 76694ab1 10.4.22.112 dc1 + Node ID Address DC + compute-node 76694ab1 10.4.22.112 dc1 -9. Create a new `Prometheus` configuration ``rdc_prometheus_consul.yml`` file for the `Consul` integration. +9. Create a new Prometheus configuration ``rdc_prometheus_consul.yml`` file for the Consul integration. -.. code-block:: shell + .. code-block:: shell global: scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute. @@ -464,16 +463,17 @@ To see if the client has joined the `Consul`, use: - source_labels: [__meta_consul_service] target_label: job -.. note:: - When running the `Consul` server and `Prometheus` on the same machine, change the server under ``consul_sd_configs`` to your `Consul` server address. + .. note:: + + When running the Consul server and Prometheus on the same machine, change the server under ``consul_sd_configs`` to your Consul server address. 10. Start Prometheus. -.. code-block:: shell + .. code-block:: shell - $ ./prometheus --config.file="rdc_prometheus_consul.yml" + $ ./prometheus --config.file="rdc_prometheus_consul.yml" -11. Browse the `Prometheus` UI at http://localhost:9090 on the management node and query RDC `Prometheus` metrics. Ensure that the plugin starts before running the query. +11. Browse the Prometheus UI at http://localhost:9090 on the management node and query RDC Prometheus metrics. Ensure that the plugin starts before running the query. Reliability, Availability, and Serviceability plugin ===================================================== @@ -490,13 +490,13 @@ Prerequisite - Ensure that the GPU supports RAS. -.. note:: + .. note:: - The RAS library is installed as part of the RDC installation. No additional configuration is required for RDC. + The RAS library is installed as part of the RDC installation. No additional configuration is required for RDC. - RDC installation dynamically loads the RAS library ``librdc_ras.so``. The configuration files required by the RAS library are installed in the ``sp3`` and ``config`` folders. -.. code-block:: shell + .. code-block:: shell % ls /opt/rocm-4.2.0/rdc/lib ... librdc_ras.so ... diff --git a/docs/how-to/using_RDC.rst b/docs/how-to/using_RDC.rst index 7387936a32..c4d74e30d1 100644 --- a/docs/how-to/using_RDC.rst +++ b/docs/how-to/using_RDC.rst @@ -27,9 +27,9 @@ Here are the steps to start RDC using ``systemctl`` command, which runs RDC in t 1. Copy the service file: -.. code-block:: shell + .. code-block:: shell - sudo cp /opt/rocm/libexec/rdc/rdc.service /etc/systemd/system/ + sudo cp /opt/rocm/libexec/rdc/rdc.service /etc/systemd/system/ 2. Configure capabilities: @@ -44,30 +44,30 @@ Here are the steps to start RDC using ``systemctl`` command, which runs RDC in t 3. Start the service: -.. code-block:: shell + .. code-block:: shell - sudo systemctl start rdc - sudo systemctl status rdc + sudo systemctl start rdc + sudo systemctl status rdc 4. Modify RDCD options: -Edit ``/opt/rocm/etc/rdc_options`` to append any additional RDCD parameters. + Edit ``/opt/rocm/etc/rdc_options`` to append any additional RDCD parameters. -.. code-block:: shell + .. code-block:: shell - sudo nano /opt/rocm/etc/rdc_options + sudo nano /opt/rocm/etc/rdc_options -Example configuration: + Example configuration: -.. code-block:: shell + .. code-block:: shell - RDC_OPTS="-p 50051 -u -d" + RDC_OPTS="-p 50051 -u -d" -Flags: + Flags: -- `-p 50051` : Use port 50051 -- `-u` : Unauthenticated mode -- `-d` : Enable debug messages + - `-p 50051` : Use port 50051 + - `-u` : Unauthenticated mode + - `-d` : Enable debug messages Starting RDC using command line as a user ------------------------------------------ @@ -240,7 +240,7 @@ The embedded mode is especially useful for a monitoring agent running on the com The RDC daemon ``rdcd`` can be used as a reference code for this purpose. The dependency on ``gRPC`` is also eliminated, if the RDC library is directly used. -- To run RDC in embedded mode, use: +To run RDC in embedded mode, use: .. code-block:: shell @@ -261,25 +261,25 @@ The RDCD logs provide useful status and debugging information. The logs can also - View logs: -When ``rdcd`` is started using ``systemctl``, you can view the logs using: + When ``rdcd`` is started using ``systemctl``, you can view the logs using: -.. code-block:: shell + .. code-block:: shell - $ journalctl -u rdc + $ journalctl -u rdc - Run RDCD with debug logs: -.. code-block:: shell + .. code-block:: shell - RDC_LOG=DEBUG /opt/rocm/bin/rdcd + RDC_LOG=DEBUG /opt/rocm/bin/rdcd -Logging levels supported: `ERROR`, `INFO`, `DEBUG`. + Logging levels supported: `ERROR`, `INFO`, `DEBUG`. - Enable additional logging messages: -.. code-block:: shell + .. code-block:: shell - export RSMI_LOGGING=3 + export RSMI_LOGGING=3 If the GPU reset fails, restart the server. Note that restarting the server also initiates ``rdcd``. You might then encounter the following two scenarios: @@ -311,50 +311,50 @@ Known issues - dmon RocProfiler fields return zeros -**Solution:** + **Solution:** -Set the ``HSA_TOOLS_LIB`` environment variable before running a compute job. + Set the ``HSA_TOOLS_LIB`` environment variable before running a compute job. -.. code-block:: shell + .. code-block:: shell - export HSA_TOOLS_LIB=/opt/rocm/lib/librocprofiler64.so.1 + export HSA_TOOLS_LIB=/opt/rocm/lib/librocprofiler64.so.1 -**Example:** + **Example:** -.. code-block:: shell + .. code-block:: shell - # Terminal 1 - rdcd -u + # Terminal 1 + rdcd -u - # Terminal 2 - export HSA_TOOLS_LIB=/opt/rocm/lib/librocprofiler64.so.1 - gpu-burn + # Terminal 2 + export HSA_TOOLS_LIB=/opt/rocm/lib/librocprofiler64.so.1 + gpu-burn - # Terminal 3 - rdci dmon -u -e 800,801 -i 0 -c 1 + # Terminal 3 + rdci dmon -u -e 800,801 -i 0 -c 1 - # Output: - GPU OCCUPANCY_PERCENT ACTIVE_WAVES - 0 001.000 32640.000 + # Output: + GPU OCCUPANCY_PERCENT ACTIVE_WAVES + 0 001.000 32640.000 - HSA_STATUS_ERROR_OUT_OF_RESOURCES -**Error message:** + **Error message:** -.. code-block:: shell + .. code-block:: shell - terminate called after throwing an instance of 'std::runtime_error' - what(): hsa error code: 4104 HSA_STATUS_ERROR_OUT_OF_RESOURCES: The runtime failed to allocate the necessary resources. This error may also occur when the core runtime library needs to spawn threads or create internal OS-specific events. - Aborted (core dumped) + terminate called after throwing an instance of 'std::runtime_error' + what(): hsa error code: 4104 HSA_STATUS_ERROR_OUT_OF_RESOURCES: The runtime failed to allocate the necessary resources. This error may also occur when the core runtime library needs to spawn threads or create internal OS-specific events. + Aborted (core dumped) -**Solution:** + **Solution:** -Follow these steps to check for missing groups: + Follow these steps to check for missing groups: -1. Ensure video and render groups exist. + 1. Ensure video and render groups exist. -.. code-block:: shell + .. code-block:: shell - sudo usermod -aG video,render $USER + sudo usermod -aG video,render $USER -2. Logout and login to apply group changes. + 2. Logout and login to apply group changes. diff --git a/docs/how-to/using_RDC_features.rst b/docs/how-to/using_RDC_features.rst index 9873aaad7a..b9482bd639 100644 --- a/docs/how-to/using_RDC_features.rst +++ b/docs/how-to/using_RDC_features.rst @@ -56,33 +56,33 @@ With the GPU groups feature, you can create, delete, and list logical groups of - To create a group, use: -.. code-block:: shell + .. code-block:: shell - $ rdci group -c GPU_GROUP - Successfully created a group with a group ID 1 + $ rdci group -c GPU_GROUP + Successfully created a group with a group ID 1 - To add GPUs to a group, use: -.. code-block:: shell + .. code-block:: shell - $ rdci group -g 1 -a 0,1 - Successfully added the GPU 0,1 to group 1 + $ rdci group -g 1 -a 0,1 + Successfully added the GPU 0,1 to group 1 - To delete a group, use: -.. code-block:: shell + .. code-block:: shell - $ rdci group -d 1 - Successfully removed group 1 + $ rdci group -d 1 + Successfully removed group 1 - To list groups, use: -.. code-block:: shell + .. code-block:: shell - $ rdci group –l - 1 group found + $ rdci group –l + 1 group found -.. list-table:: + .. list-table:: * - **Group ID** - **Group name** @@ -99,19 +99,19 @@ The field groups feature provides you the options to create, delete, list field - To create a field group, use: -.. code-block:: shell + .. code-block:: shell $ rdci fieldgroup -c -f 150,155 Successfully created a field group with a group ID 1 - To list field groups, use: -.. code-block:: shell + .. code-block:: shell - $ rdci fieldgroup -l - 1 group found + $ rdci fieldgroup -l + 1 group found -.. list-table:: + .. list-table:: * - **Group ID** - **Group Name** @@ -123,10 +123,10 @@ The field groups feature provides you the options to create, delete, list field - To delete a field group, use: -.. code-block:: shell + .. code-block:: shell - $ rdci fieldgroup -d 1 - Successfully removed field group 1 + $ rdci fieldgroup -d 1 + Successfully removed field group 1 Monitor errors =============== @@ -135,15 +135,15 @@ To get the Reliability, Availability, and Serviceability (RAS) Error-Correcting - Correctable ECC errors: -.. code-block:: shell + .. code-block:: shell - 312 ``RDC_FI_ECC_CORRECT_TOTAL`` + 312 ``RDC_FI_ECC_CORRECT_TOTAL`` - Uncorrectable ECC errors: -.. code-block:: shell + .. code-block:: shell - 313 ``RDC_FI_ECC_UNCORRECT_TOTAL`` + 313 ``RDC_FI_ECC_UNCORRECT_TOTAL`` Device monitoring ================== @@ -174,25 +174,25 @@ The job stats is used to display GPU statistics for any given workload. - To start recording stats, use: -.. code-block:: shell + .. code-block:: shell - $ rdci stats -s 2 -g 1 - Successfully started recording job 2 with a group ID 1 + $ rdci stats -s 2 -g 1 + Successfully started recording job 2 with a group ID 1 - To stop recording stats, use: -.. code-block:: shell + .. code-block:: shell - $ rdci stats -x 2 - Successfully stopped recording job 2 + $ rdci stats -x 2 + Successfully stopped recording job 2 - To display job stats, use: -.. code-block:: shell + .. code-block:: shell - $ rdci stats -j 2 + $ rdci stats -j 2 -.. list-table:: + .. list-table:: * - **Summary** - **Executive status** diff --git a/docs/install/install.rst b/docs/install/install.rst index 78b8a914bd..8272586738 100644 --- a/docs/install/install.rst +++ b/docs/install/install.rst @@ -18,16 +18,24 @@ To install RDC from source, ensure that your system meets the following requirem - **Supported platforms:** AMD ROCm-supported platform. See the `list of supported operating systems `_. - **Dependencies:** + - CMake >= 3.15 + - g++ (5.4.0) + - gRPC and protoc + - libcap-dev + - :doc:`AMD ROCm platform ` including: + - :doc:`AMDSMI library ` - `ROCK kernel driver `_ For building latest documentation: + - Doxygen (1.8.11) + - LaTeX (pdfTeX 3.14159265-2.6-1.40.16) .. code-block:: shell @@ -47,14 +55,14 @@ gRPC and Protoc must be built from source as the prebuilt packages are not avail 1. Install the required tools: -.. code-block:: shell + .. code-block:: shell sudo apt-get update sudo apt-get install automake make g++ unzip build-essential autoconf libtool pkg-config libgflags-dev libgtest-dev clang libc++-dev curl 2. Clone and build gRPC: -.. code-block:: shell + .. code-block:: shell git clone -b v1.61.0 https://github.com/grpc/grpc --depth=1 --shallow-submodules --recurse-submodules cd grpc @@ -77,14 +85,14 @@ Build RDC 1. Clone the RDC repository: -.. code-block:: shell + .. code-block:: shell git clone https://github.com/ROCm/rdc cd rdc 2. Configure the build: -.. code-block:: shell + .. code-block:: shell cmake -B build -DGRPC_ROOT="$GRPC_ROOT" @@ -116,14 +124,14 @@ Build RDC 4. Build and install: -.. code-block:: shell + .. code-block:: shell make -C build -j $(nproc) sudo make -C build install 5. Update system library path: -.. code-block:: shell + .. code-block:: shell export RDC_LIB_DIR=/opt/rocm/lib/rdc export GRPC_LIB_DIR="/opt/grpc/lib"