[rocpd] Adding merge and package submodules for rocpd (#164)

* adding ROCpd database merge

* adding ROCpd database merge concatenating all tables

* update merge script

  - copy all tables from files

* fix merge format

* Add package submodule, initial POC.  Need to refine

* Minor fixes and clean up duplicated code in package.py

* Revamp metadata layout, add wildcard and .rpdb parsing

* Add auto merge & package when > 5 DBs, add examples, don't use auto_merge when using sub-commands merge & package

* - Extend package/yaml inputs to all rocpd modules
- Improve handling more corner cases for bad input files when parsing input parameters (bad yaml files, bad .rpdb folder, folders as input)
- Changed to use UUID in merged filename instead of the time, in auto-merge algorithm

* Minor text fixes for consistancy between modules

* Add more wildcard support and add package, merge tests

* Make changes based on review suggestions

* Move parsing packages into importer.py, simplified adding required params to a function

* fix package test by flattening input list before processing

* Integrate merge.py changes from Jonathan to add name-collision checks, recreating indexes, foreign key check (disabled for now, due to processing time)

* Rework rocpd.<submodule>.{add_args,process_args}

- add_args function returns a functor which accepts input and args
- time_window functor returned from add_args automatically applies time windowing of input

* change merge&package limit to 1, merge should create data views

* Move files by default instead of making copies

- copying can be enabled by passing "copy=True" or --copy cmdline argument

* refactor package to make the logic cleaner, set merge limit back to 5

* Allow automerge-limit param to override limit, change default back to 1.  Tests updated to use query, much quicker

* Update --help instructions for package

---------

Co-authored-by: acanadas <acanadas@amd.com>
Co-authored-by: a-canadasruiz <Araceli.CanadasRuiz@amd.com>
Co-authored-by: Young Hui <young.hui@amd.com>
Co-authored-by: Jonathan R. Madsen <jonathanrmadsen@gmail.com>

projects/rocprofiler-sdk/source/lib/python/rocpd/time_window.py

@@ -26,10 +26,12 @@
 import argparse
 import sqlite3
 from argparse import ArgumentParser
-from typing import Optional, Tuple, Dict, Any, List
+from typing import Optional, Tuple, Dict, Any
 
 from .importer import RocpdImportData, execute_statement
 
+__all__ = ["apply_time_window", "execute", "add_args", "main"]
+
 
 def get_marker_timestamp(
     connection: sqlite3.Connection, marker_name: str, marker_type: str = "start"
@@ -265,7 +267,7 @@ def apply_time_window(connection: RocpdImportData, **kwargs: Any) -> None:
 #
 # Command-line interface functions
 #
-def add_args(parser: ArgumentParser) -> List[str]:
+def add_args(parser: ArgumentParser):
     """Add time slice arguments to an existing parser."""
 
     tw_options = parser.add_argument_group("Time window options")
@@ -307,18 +309,21 @@ def add_args(parser: ArgumentParser) -> List[str]:
         default=True,
     )
 
-    return ["start", "end", "inclusive", "start_marker", "end_marker"]
+    def process_args(input, args):
+        valid_args = ["start", "end", "inclusive", "start_marker", "end_marker"]
+        ret = {}
+        for itr in valid_args:
+            if hasattr(args, itr):
+                val = getattr(args, itr)
+                if val is not None:
+                    ret[itr] = val
 
+        if ret and input is not None:
+            apply_time_window(input, **ret)
 
-def process_args(args, valid_args):
+        return ret
 
-    ret = {}
-    for itr in valid_args:
-        if hasattr(args, itr):
-            val = getattr(args, itr)
-            if val is not None:
-                ret[itr] = val
-    return ret
+    return process_args
 
 
 def execute(input_rpd: str, **kwargs: Any) -> RocpdImportData: