# hosttrace: application tracing with static/dynamic binary instrumentation ## Dependencies - [DynInst](https://github.com/dyninst/dyninst) for dynamic or static instrumentation - [Julia](https://julialang.org/) for merging perfetto traces ## Installing DynInst The easiest way to install Dyninst is via spack ```shell git clone https://github.com/spack/spack.git source ./spack/share/spack/setup-env.sh spack compiler find spack external find spack install dyninst spack load -r dyninst ``` ## Installing Julia Julia is available via Linux package managers or may be available via a module. Debian-based distributions such as Ubuntu can run (as a super-user): ```shell apt-get install julia ``` Once Julia is installed, install the necessary packages (this operation only needs to be performed once): ```shell julia -e 'using Pkg; for name in ["JSON", "DataFrames", "Dates", "CSV", "Chain", "PrettyTables"]; Pkg.add(name); end' ``` ## Installing hosttrace ```shell HOSTTRACE_ROOT=${HOME}/sw/hosttrace git clone https://github.com/AARInternal/hosttrace-dyninst.git cmake -B build-hosttrace -DHOSTTRACE_USE_MPI=ON -DCMAKE_INSTALL_PREFIX=${HOSTTRACE_ROOT} hosttrace-dyninst cmake --build build-hosttrace --target all --parallel 8 cmake --build build-hosttrace --target install export PATH=${HOSTTRACE_ROOT}/bin:${PATH} export LD_LIBRARY_PATH=${HOSTTRACE_ROOT}/lib64:${HOSTTRACE_ROOT}/lib:${LD_LIBRARY_PATH} ``` ## Using Hosttrace Executable ```shell hosttrace --help hosttrace -- ``` ## Hosttrace Library Environment Settings | Environment Variable | Default Value | Description | |-----------------------------|-------------------------------|----------------------------------------------------------------------------------| | `HOSTTRACE_DEBUG` | `false` | Enable debugging statements | | `HOSTTRACE_USE_PERFETTO` | `true` | Collect profiling data via perfetto | | `HOSTTRACE_USE_TIMEMORY` | `false` | Collection profiling data via timemory | | `HOSTTRACE_SAMPLE_RATE` | `1` | Invoke perfetto and/or timemory once every N function calls | | `HOSTTRACE_USE_MPI` | `true` | Label perfetto output files via rank instead of PID | | `HOSTTRACE_OUTPUT_FILE` | `perfetto-trace.%rank%.proto` | Output file for perfetto (may use `%pid`) | | `HOSTTRACE_BACKEND` | `"inprocess"` | Configure perfetto to use either "inprocess" data management, "system", or "all" | | `HOSTTRACE_COMPONENTS` | `"wall_clock"` | Timemory components to activate when enabled | | `HOSTTRACE_SHMEM_SIZE_HINT` | `40960` | Hint for perfetto shared memory buffer | | `HOSTTRACE_BUFFER_SIZE_KB` | `1024000` | Maximum amount of memory perfetto will use to collect data in-process | | `TIMEMORY_TIME_OUTPUT` | `true` | Create unique output subdirectory with date and launch time | ### Example Hosttrace Instrumentation #### Binary Rewrite Rewrite the text section of an executable or library with instrumentation: ```shell hosttrace -o app.inst -- /path/to/app ``` In binary rewrite mode, if you also want instrumentation in the linked libraries, you must also rewrite those libraries. Example of rewriting the functions starting with `"hip"` with instrumentation in the amdhip64 library: ```shell mkdir -p ./lib hosttrace -R '^hip' -o ./lib/libamdhip64.so.4 -- /opt/rocm/lib/libamdhip64.so.4 export LD_LIBRARY_PATH=${PWD}/lib:${LD_LIBRARY_PATH} ``` > NOTE: Verify via `ldd` that your executable will load the instrumented library -- if you built your executable with > an RPATH to the original library's directory, then prefixing `LD_LIBRARY_PATH` will have no effect. Once you have rewritten your executable and/or libraries with instrumentation, you can just run the (instrumented) executable or exectuable which loads the instrumented libraries normally, e.g.: ```shell ./app.inst ``` If you want to re-define certain settings to new default in a binary rewrite, use the `--env` option. This `hosttrace` option will set the environment variable to the given value but will not override it. E.g. the default value of `HOSTTRACE_BUFFER_SIZE_KB` is 1024000 KB (1 GiB): ```shell # buffer size defaults to 1024000 hosttrace -o app.inst -- /path/to/app ./app.inst ``` Passing `--env HOSTTRACE_BUFFER_SIZE_KB=5120000` will change the default value in `app.inst` to 5120000 KiB (5 GiB): ```shell # defaults to 5 GiB buffer size hosttrace -o app.inst --env HOSTTRACE_BUFFER_SIZE_KB=5120000 -- /path/to/app ./app.inst ``` ```shell # override default 5 GiB buffer size to 200 MB export HOSTTRACE_BUFFER_SIZE_KB=200000 ./app.inst ``` #### Runtime Instrumentation Runtime instrumentation will not only instrument the text section of the executable but also the text sections of the linked libraries. Thus, it may be useful to exclude those libraries via the `-ME` (module exclude) regex option. ```shell hosttrace -- /path/to/app hosttrace -ME '^(libhsa-runtime64|libz\\.so)' -- /path/to/app hosttrace -E 'rocr::atomic|rocr::core|rocr::HSA' -- /path/to/app ``` ## Miscellaneous Features and Caveats - You may need to increase the default perfetto buffer size (1 GiB) to capture all the information - E.g. `export HOSTTRACE_BUFFER_SIZE_KB=10240000` increases the buffer size to 10 GiB - The hosttrace library has various setting which can be configured via environment variables, you can configure these settings to custom defaults with the hosttrace command-line tool via the `--env` option - E.g. to default to a buffer size of 5 GB, use `--env HOSTTRACE_BUFFER_SIZE_KB=5120000` - This is particularly useful in binary rewrite mode - Perfetto tooling is enabled by default - Timemory tooling is disabled by default - Enabling/disabling one of the aformentioned tools but not specifying enabling/disable the other will assume the inverse of the other's enabled state, e.g. - `HOSTTRACE_USE_PERFETTO=OFF` yields the same result `HOSTTRACE_USE_TIMEMORY=ON` - `HOSTTRACE_USE_PERFETTO=ON` yields the same result as `HOSTTRACE_USE_TIMEMORY=OFF` - In order to enable _both_ timemory and perfetto, set both `HOSTTRACE_USE_TIMEMORY=ON` and `HOSTTRACE_USE_PERFETTO=ON` - Setting `HOSTTRACE_USE_TIMEMORY=OFF` and `HOSTTRACE_USE_PERFETTO=OFF` will disable all instrumentation - Use `timemory-avail -S` to view the various settings for timemory - Set `HOSTTRACE_COMPONENTS=""` to control which components timemory collects - The list of components and their descriptions can be viewed via `timemory-avail -Cd` - The list of components and their string identifiers can be view via `timemory-avail -Cbs` - You can filter any `timemory-avail` results via `-r -hl` ## Hosttrace Output `hosttrace` will create an output directory named `hosttrace--output`, e.g. if your executable is named `app.inst`, the output directory will be `hosttrace-app.inst-output`. Depending on whether `TIMEMORY_TIME_OUTPUT=ON` (the default when perfetto is enabled), there will be a subdirectory with the date and time, e.g. `2021-09-02_01.03_PM`. Within this directory, all perfetto files will be named `perfetto-trace..proto` or when `HOSTTRACE_USE_MPI=ON`, `perfetto-trace..proto` (assuming hosttrace was built with MPI support). You can explicitly control the output path and naming scheme of the files via the `HOSTTRACE_OUTPUT_FILE` environment variable. The special character sequences `%pid%` and `%rank%` will be replaced with the PID or MPI rank, respectively. ## Merging the traces from rocprof and hosttrace > NOTE: Using `rocprof` externally is deprecated. The current version has built-in support for > recording the GPU activity and HIP API calls. If you want to use an external rocprof, either > configure CMake with `-DHOSTTRACE_USE_ROCTRACER=OFF` or explicitly set `TIMEMORY_ROCTRACER_ENABLED=OFF` in the > environment. Use the `hosttrace-merge.jl` Julia script to merge rocprof and perfetto traces. ```shell export TIMEMORY_ROCTRACER_ENABLED=OFF rocprof --hip-trace --roctx-trace --stats ./app.inst hosttrace-merge.jl results.json hosttrace-app.inst-output/2021-09-02_01.03_PM/*.proto ``` ## Use Perfetto tracing with System Backend In a separate window run: ```shell pkill traced traced --background perfetto --out ./htrace.out --txt -c ${HOSTTRACE_ROOT}/share/roctrace.cfg ``` then in the window running the application, configure the hosttrace instrumentation to use the system backend: ```shell export HOSTTRACE_BACKEND_SYSTEM=1 ``` for the merge use the `htrace.out`: ```shell hosttrace-merge.jl results.json htrace.out ```