AI/rocm-systems
2
0
Derivar 0
Código Questões Pedidos de integração Operações Pacotes Planeamentos Lançamentos Wiki Trabalho

Fixing kokkosp tool library packaging (#1121)

Ver a proveniência 
* Fixing kokkosp tool library packaging

* Update source/lib/rocprofiler-sdk-tool/kokkosp/CMakeLists.txt

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

* Update CMakeLists.txt

* Update CMakeLists.txt

* Component Requirement in CPack

* Adding package dependency

* Update CMakeLists.txt

* Update rocprofiler_config_packaging.cmake

* Fix rocprofiler-sdk-tool-kokkosp BUILD/INSTALL RPATH

- CMAKE_INSTALL_LIBDIR doesn't help

* Add BUILD/INSTALL RPATH to rocprofv3-trigger-list-metrics

- fixes packaging issues

* Update packaging

- core depends on rocprofiler-sdk-roctx
- add CPACK_DEBIAN_PACKAGE_SHLIBDEPS_PRIVATE_DIRS to resolve inter-package dependencies

* Fix package depends version format

* Improve tests/rocprofv3/summary/validate logging

* Update CI workflow

- prioritize roctx package in Install Packages step

* Remove setting <package-name>_VERSION in config.cmake.in

- this is automatically handled by existence of <package-name>-config-version.cmake

* Update rocprofiler-sdk-config.cmake

- relax find_package versioning requirements to same major and minor version

* Update rocprofiler-sdk-config.cmake

- relax find_package versioning requirements (remove EXACT, specify range)

* Tweak CI workflow

* Update perfetto_reader.py

- better handle failure to load trace processor

* Misc cleanup for config packaging

* Update config packaging

* Update config packaging

* Revert perfetto for core-rpm packages

* Revert perfetto for core-rpm packages

- perfetto < 0.9.0

* Tweak tests/rocprofv3/summary/validate.py

- reorder some checks

---------

Co-authored-by: Ammar Elwazir <aelwazir@useocpm2m-387-013.amd.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Jonathan R. Madsen <jonathanrmadsen@gmail.com>

[ROCm/rocprofiler-sdk commit: d762694acb]
Este cometimento está contido em:
Ammar ELWazir
2024-10-15 20:28:06 -05:00
cometido por GitHub
ascendente 3df0605296
cometimento b610c50913
10 ficheiros modificados com 86 adições e 73 eliminações
Descarregar ficheiro patch Descarregar ficheiro diff Expandir todos os ficheiros Colapsar todos os ficheiros
projects/rocprofiler-sdk/.github/workflows/continuous_integration.yml
externo
+11 -8
Ver ficheiro
@@ -86,7 +86,7 @@ jobs:
python3 ./source/scripts/run-ci.py -B build
--name ${{ github.repository }}-${{ github.ref_name }}-${{ matrix.os }}-${{ matrix.runner }}-core
--build-jobs 16
--site $(echo $RUNNER_HOSTNAME)-$(/opt/rocm/bin/rocm_agent_enumerator | sed -n '2 p')
--site ${RUNNER_HOSTNAME}
--gpu-targets ${{ env.GPU_TARGETS }}
--run-attempt ${{ github.run_attempt }}
--
@@ -133,7 +133,8 @@ jobs:
export PATH=${PATH}:/usr/local/sbin:/usr/sbin:/sbin
ls -la
ls -la ./build
for i in $(ls -S ./build/rocprofiler-sdk*.deb); do dpkg -i ${i}; done;
dpkg -i ./build/rocprofiler-sdk-roctx_*.deb
for i in $(ls -S ./build/rocprofiler-sdk*.deb | egrep -v roctx); do dpkg -i ${i}; done;
- name: Test Installed Packages
if: ${{ contains(matrix.runner, env.CORE_EXT_RUNNER) }}
@@ -181,6 +182,7 @@ jobs:
timeout-minutes: 10
shell: bash
run: |
sudo python3 -m pip install 'perfetto<0.9.0'
sudo python3 -m pip install -r requirements.txt
- name: List Files
@@ -212,7 +214,7 @@ jobs:
sudo LD_LIBRARY_PATH=./build/lib:$LD_LIBRARY_PATH python3 ./source/scripts/run-ci.py -B build
--name ${{ github.repository }}-${{ github.ref_name }}-${{ matrix.os }}-${{ matrix.runner }}-core
--build-jobs 16
--site $(echo $RUNNER_HOSTNAME)-$(/opt/rocm/bin/rocm_agent_enumerator | sed -n '2 p')
--site ${RUNNER_HOSTNAME}
--gpu-targets ${{ env.GPU_TARGETS }}
--run-attempt ${{ github.run_attempt }}
--
@@ -259,7 +261,8 @@ jobs:
export PATH=${PATH}:/usr/local/sbin:/usr/sbin:/sbin
ls -la
ls -la ./build
for i in $(ls -S ./build/rocprofiler-sdk*.deb); do dpkg -i ${i}; done;
dpkg -i ./build/rocprofiler-sdk-roctx_*.deb
for i in $(ls -S ./build/rocprofiler-sdk*.deb | egrep -v roctx); do dpkg -i ${i}; done;
- name: Test Installed Packages
if: ${{ contains(matrix.runner, env.CORE_EXT_RUNNER) }}
@@ -368,7 +371,7 @@ jobs:
python3 ./source/scripts/run-ci.py -B build
--name ${{ github.repository }}-${{ github.ref_name }}-${{ matrix.os }}-${{ matrix.runner }}-codecov
--build-jobs 16
--site $(echo $RUNNER_HOSTNAME)-$(/opt/rocm/bin/rocm_agent_enumerator | sed -n '2 p')
--site ${RUNNER_HOSTNAME}
--gpu-targets ${{ env.GPU_TARGETS }}
--coverage all
--run-attempt ${{ github.run_attempt }}
@@ -386,7 +389,7 @@ jobs:
python3 ./source/scripts/run-ci.py -B build
--name ${{ github.repository }}-${{ github.ref_name }}-${{ matrix.os }}-${{ matrix.runner }}-codecov-tests
--build-jobs 16
--site $(echo $RUNNER_HOSTNAME)-$(/opt/rocm/bin/rocm_agent_enumerator | sed -n '2 p')
--site ${RUNNER_HOSTNAME}
--gpu-targets ${{ env.GPU_TARGETS }}
--coverage tests
--run-attempt ${{ github.run_attempt }}
@@ -404,7 +407,7 @@ jobs:
python3 ./source/scripts/run-ci.py -B build
--name ${{ github.repository }}-${{ github.ref_name }}-${{ matrix.os }}-${{ matrix.runner }}-codecov-samples
--build-jobs 16
--site $(echo $RUNNER_HOSTNAME)-$(/opt/rocm/bin/rocm_agent_enumerator | sed -n '2 p')
--site ${RUNNER_HOSTNAME}
--gpu-targets ${{ env.GPU_TARGETS }}
--coverage samples
--run-attempt ${{ github.run_attempt }}
@@ -580,7 +583,7 @@ jobs:
python3 ./source/scripts/run-ci.py -B build
--name ${{ github.repository }}-${{ github.ref_name }}-${{ matrix.os }}-${{ matrix.runner }}-${{ matrix.sanitizer }}
--build-jobs 16
--site $(echo $RUNNER_HOSTNAME)-$(/opt/rocm/bin/rocm_agent_enumerator | sed -n '2 p')
--site ${RUNNER_HOSTNAME}
--gpu-targets ${{ env.GPU_TARGETS }}
--memcheck ${{ matrix.sanitizer }}
--run-attempt ${{ github.run_attempt }}
projects/rocprofiler-sdk/cmake/Templates/rocprofiler-sdk-roctx/config.cmake.in
-3
Ver ficheiro
@@ -14,9 +14,6 @@
# compute paths
get_filename_component(@PACKAGE_NAME@_CMAKE_DIR "${CMAKE_CURRENT_LIST_FILE}" PATH)
# version
set(@PACKAGE_NAME@_VERSION @PROJECT_VERSION@)
@PACKAGE_INIT@
set_and_check(@PACKAGE_NAME@_INCLUDE_DIR "@PACKAGE_INCLUDE_INSTALL_DIR@")
projects/rocprofiler-sdk/cmake/Templates/rocprofiler-sdk/config.cmake.in
+3 -9
Ver ficheiro
@@ -15,9 +15,6 @@
# compute paths
get_filename_component(@PACKAGE_NAME@_CMAKE_DIR "${CMAKE_CURRENT_LIST_FILE}" PATH)
# version
set(@PACKAGE_NAME@_VERSION @PROJECT_VERSION@)
@PACKAGE_INIT@
set_and_check(@PACKAGE_NAME@_INCLUDE_DIR "@PACKAGE_INCLUDE_INSTALL_DIR@")
@@ -58,8 +55,7 @@ else()
find_package(
amd_comgr
@amd_comgr_VERSION@
EXACT
@PACKAGE_amd_comgr_VERSION@
REQUIRED
CONFIG
HINTS
@@ -71,8 +67,7 @@ else()
find_package(
hsa-runtime64
@hsa-runtime64_VERSION@
EXACT
@PACKAGE_hsa-runtime64_VERSION@
REQUIRED
CONFIG
HINTS
@@ -84,8 +79,7 @@ else()
find_package(
hip
@hip_VERSION@
EXACT
@PACKAGE_hip_VERSION@
REQUIRED
CONFIG
HINTS
projects/rocprofiler-sdk/cmake/rocprofiler_config_install.cmake
+11
Ver ficheiro
@@ -38,6 +38,17 @@ rocprofiler_install_env_setup_files(
INSTALL_DIR ${CMAKE_INSTALL_DATAROOTDIR}
COMPONENT development)
function(compute_rocprofiler_sdk_version _VAR)
string(REGEX REPLACE "([0-9]+)\\\.([0-9]+)\\\.(.*)" "\\1.\\2" _TMP "${${_VAR}}")
set(PACKAGE_${_VAR}
"${_TMP}.0...${_TMP}.999999999999"
PARENT_SCOPE)
endfunction()
compute_rocprofiler_sdk_version(amd_comgr_VERSION)
compute_rocprofiler_sdk_version(hsa-runtime64_VERSION)
compute_rocprofiler_sdk_version(hip_VERSION)
# ------------------------------------------------------------------------------#
# install tree
#
projects/rocprofiler-sdk/cmake/rocprofiler_config_packaging.cmake
+37 -42
Ver ficheiro
@@ -2,14 +2,27 @@
# configure packaging settings
#
function(rocprofiler_set_package_depends _VARIABLE _VALUE _INFO)
function(rocprofiler_set_package_depends _VARIABLE _VALUE _INFO _REPLACE_PARENTHESIS)
string(REPLACE ";" ", " _DEPENDS "${_VALUE}")
if(_REPLACE_PARENTHESIS)
string(REGEX REPLACE "\\\(|\\\)" "" _DEPENDS "${_DEPENDS}")
endif()
set(${_VARIABLE}
"${_DEPENDS}"
CACHE STRING "${_INFO} package dependencies" FORCE)
rocprofiler_add_feature(${_VARIABLE} "${_INFO} package dependencies")
endfunction()
# Make proper version for appending Default Value is 99999
set(ROCM_VERSION_FOR_PACKAGE
"99999"
CACHE STRING "")
if(DEFINED ENV{ROCM_LIBPATCH_VERSION})
set(ROCM_VERSION_FOR_PACKAGE
"$ENV{ROCM_LIBPATCH_VERSION}"
CACHE STRING "" FORCE)
endif()
# Add packaging directives
set(CPACK_PACKAGE_NAME ${PROJECT_NAME}-sdk)
set(CPACK_PACKAGE_VENDOR "Advanced Micro Devices, Inc.")
@@ -30,15 +43,13 @@ set(ROCPROFILER_CPACK_PACKAGE_SUFFIX "")
set(CPACK_PACKAGE_FILE_NAME
"${CPACK_PACKAGE_NAME}-${ROCPROFILER_VERSION}-${ROCPROFILER_CPACK_SYSTEM_NAME}${ROCPROFILER_CPACK_PACKAGE_SUFFIX}"
)
CACHE STRING "")
if(DEFINED ENV{CPACK_PACKAGE_FILE_NAME})
set(CPACK_PACKAGE_FILE_NAME $ENV{CPACK_PACKAGE_FILE_NAME})
set(CPACK_PACKAGE_FILE_NAME
"$ENV{CPACK_PACKAGE_FILE_NAME}"
CACHE STRING "" FORCE)
endif()
set(ROCPROFILER_PACKAGE_FILE_NAME
${CPACK_PACKAGE_NAME}-${ROCPROFILER_VERSION}-${ROCPROFILER_CPACK_SYSTEM_NAME}${ROCPROFILER_CPACK_PACKAGE_SUFFIX}
)
rocprofiler_add_feature(ROCPROFILER_PACKAGE_FILE_NAME "CPack filename")
rocprofiler_add_feature(CPACK_PACKAGE_FILE_NAME "CPack filename")
get_cmake_property(ROCPROFILER_PACKAGING_COMPONENTS COMPONENTS)
@@ -64,9 +75,9 @@ set(COMPONENT_NAME_docs "rocprofiler-sdk-docs")
set(COMPONENT_NAME_tests "rocprofiler-sdk-tests")
set(COMPONENT_NAME_roctx "rocprofiler-sdk-roctx")
set(COMPONENT_DEP_core "")
set(COMPONENT_DEP_core "rocprofiler-sdk-roctx (>= ${PROJECT_VERSION})")
set(COMPONENT_DEP_docs "")
set(COMPONENT_DEP_tests "rocprofiler-sdk")
set(COMPONENT_DEP_tests "rocprofiler-sdk (= ${PROJECT_VERSION})")
set(COMPONENT_DEP_roctx "")
set(COMPONENT_DESC_core "rocprofiler-sdk libraries, headers, samples, and tools")
@@ -113,9 +124,9 @@ foreach(COMPONENT_GROUP ${ROCPROFILER_COMPONENT_GROUPS})
set(CPACK_RPM_${UCOMPONENT}_PACKAGE_NAME "${_NAME}")
rocprofiler_set_package_depends(CPACK_DEBIAN_${UCOMPONENT}_PACKAGE_DEPENDS "${_DEP}"
"Debian")
"Debian" OFF)
rocprofiler_set_package_depends(CPACK_RPM_${UCOMPONENT}_PACKAGE_REQUIRES "${_DEP}"
"RedHat")
"RedHat" ON)
foreach(COMPONENT ${COMPONENT_GROUP_${COMPONENT_GROUP}_COMPONENTS})
cpack_add_component(${COMPONENT} REQUIRED GROUP "${COMPONENT_GROUP}")
@@ -138,25 +149,20 @@ set(CPACK_DEBIAN_PACKAGE_GENERATE_SHLIBS ON) # generate list of shared libs prov
set(CPACK_DEBIAN_TESTS_PACKAGE_SHLIBDEPS OFF) # disable for tests package
set(CPACK_DEBIAN_TESTS_PACKAGE_GENERATE_SHLIBS OFF) # disable for tests package
set(CPACK_DEBIAN_PACKAGE_HOMEPAGE "${PROJECT_HOMEPAGE_URL}")
set(CPACK_DEBIAN_PACKAGE_RELEASE
"${ROCPROFILER_CPACK_SYSTEM_NAME}${ROCPROFILER_CPACK_PACKAGE_SUFFIX}")
string(REGEX REPLACE "([a-zA-Z])-([0-9])" "\\1\\2" CPACK_DEBIAN_PACKAGE_RELEASE
"${CPACK_DEBIAN_PACKAGE_RELEASE}")
string(REPLACE "-" "~" CPACK_DEBIAN_PACKAGE_RELEASE "${CPACK_DEBIAN_PACKAGE_RELEASE}")
set(CPACK_DEBIAN_PACKAGE_GENERATE_SHLIBS_POLICY ">=")
set(CPACK_DEBIAN_PACKAGE_SHLIBDEPS_PRIVATE_DIRS
${PROJECT_BINARY_DIR}/${CMAKE_INSTALL_LIBDIR}
${PROJECT_BINARY_DIR}/${CMAKE_INSTALL_LIBDIR}/${CPACK_PACKAGE_NAME})
if(DEFINED ENV{CPACK_DEBIAN_PACKAGE_RELEASE})
set(CPACK_DEBIAN_PACKAGE_RELEASE $ENV{CPACK_DEBIAN_PACKAGE_RELEASE})
set(CPACK_DEBIAN_PACKAGE_RELEASE
"$ENV{CPACK_DEBIAN_PACKAGE_RELEASE}"
CACHE STRING "" FORCE)
endif()
set(_DEBIAN_PACKAGE_DEPENDS "")
if(rocm_version_FOUND)
set(_ROCPROFILER_SUFFIX " (>= 1.0.0.${rocm_version_NUMERIC_VERSION})")
set(_ROCTRACER_SUFFIX " (>= 1.0.0.${rocm_version_NUMERIC_VERSION})")
set(_ROCM_SMI_SUFFIX
" (>= ${rocm_version_MAJOR_VERSION}.0.0.${rocm_version_NUMERIC_VERSION})")
endif()
rocprofiler_set_package_depends(CPACK_DEBIAN_PACKAGE_DEPENDS "${_DEBIAN_PACKAGE_DEPENDS}"
"Debian")
rocprofiler_set_package_depends(CPACK_DEBIAN_PACKAGE_DEPENDS
"${CPACK_DEBIAN_PACKAGE_DEPENDS}" "Debian" OFF)
set(CPACK_DEBIAN_FILE_NAME "DEB-DEFAULT")
set(CPACK_DEBIAN_PACKAGE_VERSION "${PROJECT_VERSION}")
# -------------------------------------------------------------------------------------- #
#
@@ -176,13 +182,10 @@ set(CPACK_RPM_PACKAGE_AUTOREQ
set(CPACK_RPM_PACKAGE_AUTOPROV ON) # generate list of shared libs provided by package
set(CPACK_RPM_TESTS_PACKAGE_AUTOREQ OFF) # disable for tests package
set(CPACK_RPM_TESTS_PACKAGE_AUTOPROV OFF) # disable for tests package
set(CPACK_RPM_PACKAGE_RELEASE
"${ROCPROFILER_CPACK_SYSTEM_NAME}${ROCPROFILER_CPACK_PACKAGE_SUFFIX}")
string(REGEX REPLACE "([a-zA-Z])-([0-9])" "\\1\\2" CPACK_RPM_PACKAGE_RELEASE
"${CPACK_RPM_PACKAGE_RELEASE}")
string(REPLACE "-" "~" CPACK_RPM_PACKAGE_RELEASE "${CPACK_RPM_PACKAGE_RELEASE}")
if(DEFINED ENV{CPACK_RPM_PACKAGE_RELEASE})
set(CPACK_RPM_PACKAGE_RELEASE $ENV{CPACK_RPM_PACKAGE_RELEASE})
set(CPACK_RPM_PACKAGE_RELEASE
"$ENV{CPACK_RPM_PACKAGE_RELEASE}"
CACHE STRING "" FORCE)
endif()
# Get rpm distro
@@ -197,15 +200,7 @@ set(CPACK_RPM_PACKAGE_LICENSE "MIT")
# Prepare final version for the CPACK use
#
# -------------------------------------------------------------------------------------- #
# Make proper version for appending Default Value is 99999
set(ROCM_VERSION_FOR_PACKAGE
"99999"
CACHE STRING "")
if(DEFINED ENV{ROCM_LIBPATCH_VERSION})
set(ROCM_VERSION_FOR_PACKAGE
"$ENV{ROCM_LIBPATCH_VERSION}"
CACHE STRING "" FORCE)
endif()
# Prepare final version for the CPACK use
set(CPACK_PACKAGE_VERSION
"${CPACK_PACKAGE_VERSION_MAJOR}.${CPACK_PACKAGE_VERSION_MINOR}.${CPACK_PACKAGE_VERSION_PATCH}.${ROCM_VERSION_FOR_PACKAGE}"
projects/rocprofiler-sdk/source/lib/rocprofiler-sdk-tool/CMakeLists.txt
+3 -1
Ver ficheiro
@@ -72,7 +72,9 @@ target_link_libraries(rocprofv3-trigger-list-metrics
set_target_properties(
rocprofv3-trigger-list-metrics
PROPERTIES RUNTIME_OUTPUT_DIRECTORY
${PROJECT_BINARY_DIR}/${CMAKE_INSTALL_LIBDIR}/rocprofiler-sdk)
${PROJECT_BINARY_DIR}/${CMAKE_INSTALL_LIBDIR}/rocprofiler-sdk
BUILD_RPATH "\$ORIGIN:\$ORIGIN/.."
INSTALL_RPATH "\$ORIGIN:\$ORIGIN/..")
install(
TARGETS rocprofv3-trigger-list-metrics
projects/rocprofiler-sdk/source/lib/rocprofiler-sdk-tool/kokkosp/CMakeLists.txt
+3
Ver ficheiro
@@ -20,6 +20,9 @@ set_target_properties(
rocprofiler-sdk-tool-kokkosp
PROPERTIES LIBRARY_OUTPUT_DIRECTORY
${PROJECT_BINARY_DIR}/${CMAKE_INSTALL_LIBDIR}/rocprofiler-sdk
SOVERSION ${PROJECT_VERSION_MAJOR}
VERSION ${PROJECT_VERSION}
SKIP_BUILD_RPATH OFF
BUILD_RPATH "\$ORIGIN:\$ORIGIN/.."
INSTALL_RPATH "\$ORIGIN:\$ORIGIN/..")
projects/rocprofiler-sdk/tests/pytest-packages/pytest_utils/perfetto_reader.py
+7 -3
Ver ficheiro
@@ -28,7 +28,7 @@ import time
import pandas as pd
from collections import OrderedDict
from perfetto.trace_processor import TraceProcessor
from perfetto.trace_processor import TraceProcessor, TraceProcessorConfig
class PerfettoReader:
@@ -162,15 +162,19 @@ class PerfettoReader:
def construct_trace_processor(trace_v):
for i in range(4):
try:
return TraceProcessor(trace=(trace_v))
verbosity = True if i > 0 else False
cfg = TraceProcessorConfig(verbose=verbosity)
return TraceProcessor(trace=(trace_v), config=cfg)
except Exception as e:
nwait = i + 1
sys.stderr.write(
f"{e}\n\nRetrying trace processor construction after {nwait} seconds...\n"
f"{e}\nRetrying trace processor construction after {nwait} seconds...\n"
)
sys.stderr.flush()
time.sleep(nwait)
raise RuntimeError(f"Failed to construct trace processor for '{trace_v}'")
if len(self.filename) + len(_new_filenames) != len(self.trace_processor):
self.trace_processor = [construct_trace_processor(f) for f in self.filename]
elif _new_filenames:
projects/rocprofiler-sdk/tests/rocprofv3/summary/conftest.py
+4
Ver ficheiro
@@ -33,6 +33,10 @@ def pytest_addoption(parser):
action="store",
help="Path to summary markdown file.",
)
pd.set_option("display.max_rows", None)
pd.set_option("display.max_columns", None)
pd.set_option("display.width", 2000)
pd.set_option("display.max_colwidth", None)
@pytest.fixture
projects/rocprofiler-sdk/tests/rocprofv3/summary/validate.py
+7 -7
Ver ficheiro
@@ -237,13 +237,13 @@ def test_summary_display_data(json_data, summary_data):
expected_hip_and_marker_dims = [20, 9] if hip_and_marker is not None else [0, 0]
assert get_dims(hip) == [13, 9]
assert get_dims(marker) == [7, 9]
assert get_dims(dispatch) == [3, 9]
assert get_dims(memcpy) == [2, 9]
assert get_dims(dispatch_and_copy) == [5, 9]
assert get_dims(hip_and_marker) == expected_hip_and_marker_dims
assert get_dims(total) == [22, 9]
assert get_dims(marker) == [7, 9], f"{marker}"
assert get_dims(memcpy) == [2, 9], f"{memcpy}"
assert get_dims(dispatch) == [3, 9], f"{dispatch}"
assert get_dims(dispatch_and_copy) == [5, 9], f"{dispatch_and_copy}"
assert get_dims(hip) == [13, 9], f"{hip}"
assert get_dims(hip_and_marker) == expected_hip_and_marker_dims, f"{hip_and_marker}"
assert get_dims(total) == [22, 9], f"{total}"
def test_perfetto_data(pftrace_data, json_data):