rocm-systems/README.md

# hosttrace: application tracing with static/dynamic binary instrumentation and perfetto


## 1. load necessary modules
```
module load julia
module load dyninst
module load rocm/VERSION
```

## 2. to install julia packages, only need it the first time.
```
$ julia
type ']' to get into pkg mode, then 
(@v1.6) pkg> add JSON, DataFrames, Dates, CSV, Chain, PrettyTables
```

## 3. hosttrace usage
```
export PATH=$PATH:$HTRACE_PATH/bin
hosttrace --help
```

## 4. instrument the binaries
```
hosttrace -L $HTRACE_PATH/bin/libhosttrace.so -o app.inst -- path_to_your_app

hosttrace -L $HTRACE_PATH/bin/libhosttrace.so -E 'hipApiName|hipGetCmdName' -o libamdhip64.so.4 --  /opt/rocm-VERSION/lib/libamdhip64.so.4

hosttrace -L $HTRACE_PATH/bin/libhosttrace.so -E 'rocr::atomic|rocr::core|rocr::HSA' -o libhsa-runtime64.so.1 --  /opt/rocm-VERSION/lib/libhsa-runtime64.so.1
```
## 5. run the app
```
export LD_LIBRARY_PATH=.:$LD_LIBRARY_PATH
rocprof --hip-trace --roctx-trace --stats ./app.inst
```

## 6. you may need to increase the buffer size if the hosttrace.perfetto-trace file is close to 1GB. for example to set it to 10GB. 
```
export HOSTTRACE_BUFFER_SIZE_KB=10240000
```

## 7. merge the traces from rocprof and hosttrace
```
julia $HTRACE_PATH/bin/merge-trace.jl results.json hosttrace.perfetto-trace*
```

## 8. another mode of Perfetto tracing is to use system backend. To do it:

### in a separate window
```
    pkill traced; traced --background; perfetto --out ./htrace.out --txt -c $HTRACE/roctrace.cfg
```

### then in the app running window do this before running rocprof or other app
```
    export HOSTTRACE_BACKEND_SYSTEM=1
```

### for the merge use the htrace.out
```
julia $HTRACE_PATH/bin/merge-trace.jl results.json htrace.out
```