[hip] Docs: Overhaul HW implementation page (#1994)

* [hip] Docs: Overhaul HW implementation page * Update hardware implementation and glossary * Update programming model * Add performance optimization * Split into how-to and understanding --------- Signed-off-by: Jan Stephan <jan.stephan@amd.com> Co-authored-by: Jan Stephan <jan.stephan@amd.com> Co-authored-by: Julia Jiang <julia.jiang@amd.com>
2025-12-11 10:52:34 +01:00
parent 9e06ea8f79
commit 256dd1963a
14 changed files with 2466 additions and 389 deletions
@@ -0,0 +1,375 @@
+.. meta::
+  :description: This chapter explains performance concepts and theoretical models for understanding AMD GPU performance
+  :keywords: AMD, ROCm, HIP, performance, theory, roofline, occupancy, bandwidth, arithmetic intensity
+
+.. _performance_optimization:
+
+*******************************************************************************
+Understanding GPU performance
+*******************************************************************************
+
+This chapter explains the theoretical foundations of GPU performance on AMD 
+hardware. Understanding these concepts helps you analyze performance 
+characteristics, identify bottlenecks, and make informed optimization decisions.
+
+For practical optimization techniques and step-by-step guidance, see 
+:doc:`../how-to/performance_guidelines`.
+
+Performance bottlenecks
+=======================
+
+A performance bottleneck is the limiting factor that prevents a GPU kernel from 
+achieving higher performance. The two primary categories are:
+
+* **Compute-bound**: The kernel is limited by arithmetic throughput
+* **Memory-bound**: The kernel is limited by memory bandwidth
+
+Understanding which category applies helps identify the appropriate optimization 
+approach. Compute-bound kernels benefit from arithmetic optimizations, while 
+memory-bound kernels benefit from memory access improvements.
+
+.. _roofline_model:
+
+Roofline model
+==============
+
+The roofline model is a visual performance analysis framework that relates 
+achievable performance to hardware limits based on arithmetic intensity.
+
+The model plots performance (FLOPS) against arithmetic intensity (FLOPS/byte) 
+with two limiting factors:
+
+1. **Memory bandwidth ceiling**: A sloped line representing peak memory bandwidth
+2. **Compute ceiling**: A horizontal line representing peak arithmetic throughput
+
+The intersection point determines the transition between memory-bound and 
+compute-bound regions. Kernels below and to the left of the intersection are 
+memory-bound, while those to the right are compute-bound.
+
+.. figure:: ../../data/understand/performance_optimization/roofline.svg
+   :alt: Roofline model diagram showing memory bandwidth ceiling and compute
+         ceiling
+   :align: center
+   :width: 100%
+
+   Roofline model showing the relationship between arithmetic intensity and
+   achievable performance. The memory bandwidth ceiling represents the GPU's
+   memory bandwidth limit, while the compute ceiling shows the maximum
+   achievable TFLOPs. Kernels falling into the area to the left of the red
+   line are memory-bound, while they are compute-bound if they fall into the
+   right area.
+
+Key characteristics:
+
+* The roofline creates an upper bound on achievable performance
+* Real applications typically achieve a significant portion of the theoretical limit
+* The model helps guide optimization efforts based on kernel characteristics
+
+.. _compute_bound:
+
+Compute-bound performance
+=========================
+
+A kernel is compute-bound when its performance is limited by the GPU's arithmetic 
+throughput rather than memory bandwidth. These kernels have high arithmetic 
+intensity and spend most cycles executing arithmetic operations.
+
+Characteristics of compute-bound kernels:
+
+* High ratio of arithmetic operations to memory accesses
+* Performance scales with GPU compute capacity
+* Limited benefit from memory bandwidth optimization
+* Can often achieve a high percentage of peak theoretical FLOPS
+
+The theoretical maximum is determined by:
+
+* Number of compute units and SIMD lanes
+* Clock frequency
+* Instruction throughput per cycle
+* Specialized unit capabilities (matrix cores, SFUs)
+
+.. _memory_bound:
+
+Memory-bound performance
+========================
+
+A kernel is memory-bound when its performance is limited by memory bandwidth 
+rather than compute capacity. These kernels have low arithmetic intensity and 
+spend significant time waiting for memory operations.
+
+Characteristics of memory-bound kernels:
+
+* Low ratio of arithmetic operations to memory accesses
+* Performance scales with memory bandwidth
+* Sensitive to memory access patterns
+* Typically achieve lower percentage of peak FLOPS
+
+The theoretical maximum is determined by:
+
+* HBM bandwidth capacity
+* Memory controller efficiency
+* Cache hierarchy effectiveness
+* Memory access pattern efficiency
+
+.. _arithmetic_intensity:
+
+Arithmetic intensity
+====================
+
+Arithmetic intensity is the ratio of floating-point operations (FLOPs) to memory 
+traffic (bytes) for a given kernel or algorithm.
+
+.. math::
+
+   \text{Arithmetic Intensity} = \frac{\text{FLOPs}}{\text{Bytes Transferred}}
+
+This metric determines whether a kernel is compute-bound or memory-bound.
+
+Key points:
+
+* Higher arithmetic intensity indicates more computation per byte transferred
+* The balance point depends on the GPU's compute-to-bandwidth ratio
+* It can be calculated theoretically or measured empirically
+* Different precision types affect both FLOPs and bytes
+
+For modern AMD GPUs:
+
+* The compute-to-bandwidth ratio varies by GPU generation
+* Higher-end models have higher ratios
+* Kernels above the GPU's specific ratio are compute-bound
+
+.. _latency_hiding:
+
+Latency hiding mechanisms
+==========================
+
+GPUs hide memory and instruction latency through massive hardware multithreading 
+rather than complex CPU techniques like out-of-order execution.
+
+How latency hiding works:
+
+* **Wavefront switching**: Context switches occur every cycle with zero overhead
+* **Multiple wavefronts per CU**: Many concurrent wavefronts supported
+* **Instruction-level parallelism**: Multiple independent instructions in flight
+
+The hardware can completely hide memory latency if there are enough active 
+wavefronts with independent work. The number of instructions required from other 
+wavefronts to hide latency depends on the specific memory latency and instruction 
+throughput characteristics of the GPU.
+
+Requirements for effective latency hiding:
+
+* Sufficient occupancy (active wavefronts)
+* Independent instructions to overlap
+* Balanced resource usage
+* Minimal divergence
+
+.. _wavefront_execution:
+
+Wavefront execution states
+==========================
+
+A wavefront can be in one of several states during execution:
+
+* **Active**: Currently executing on a SIMD unit
+* **Ready**: Eligible for execution, waiting for scheduling
+* **Stalled**: Waiting for a dependency (memory, synchronization)
+* **Sleeping**: Blocked on a barrier or synchronization primitive
+
+Understanding these states helps explain GPU utilization metrics:
+
+* **Active cycles**: Percentage of cycles with at least one instruction executing
+* **Stall cycles**: Percentage of cycles waiting for resources
+* **Idle cycles**: No wavefronts available to execute
+
+Maximizing active cycles while minimizing stall and idle cycles improves 
+performance.
+
+.. _occupancy:
+
+Occupancy theory
+================
+
+Occupancy measures the ratio of active wavefronts to the maximum possible 
+wavefronts on a compute unit.
+
+.. math::
+
+   \text{Occupancy} = \frac{\text{Active Wavefronts}}{\text{Max Wavefronts per CU}}
+
+Why occupancy matters:
+
+* Higher occupancy improves latency hiding
+* More concurrent wavefronts mask memory and instruction latency
+* Enables better utilization of execution units
+
+Limiting factors:
+
+* **Register usage**: VGPRs and SGPRs per thread
+* **Shared memory (LDS)**: Allocation per block
+* **Wavefront slots**: Hardware limit on concurrent wavefronts
+* **Block size**: Small blocks may waste resources
+
+Trade-offs:
+
+* Higher occupancy improves latency hiding but reduces resources per thread
+* Lower occupancy allows more resources per thread but may expose latency
+* Optimal occupancy depends on kernel characteristics
+* Memory-bound kernels benefit more from high occupancy
+
+.. _memory_hierarchy_theory:
+
+Memory hierarchy impact on performance
+=======================================
+
+The GPU memory hierarchy has different bandwidths and latencies:
+
+Memory types by speed:
+
+1. **Registers**: Fastest, lowest latency (per-thread storage)
+2. **LDS (shared memory)**: Very fast, on-chip (per-block storage)
+3. **L1 cache**: Fast, on-chip (per-CU cache)
+4. **L2 cache**: Moderate, on-chip (shared across CUs)
+5. **HBM (global memory)**: Slower, off-chip but high bandwidth
+
+Memory coalescing theory
+-------------------------
+
+Memory coalescing combines memory accesses from multiple threads into fewer 
+transactions. When consecutive threads access consecutive memory addresses, the 
+hardware can merge requests into efficient cache line accesses.
+
+Why coalescing matters:
+
+* Reduces number of memory transactions
+* Improves memory bandwidth utilization
+* Decreases memory access latency
+
+**Coalesced pattern**: Consecutive threads accessing consecutive addresses 
+achieve high bandwidth utilization.
+
+**Non-coalesced pattern**: Random or strided addresses result in many separate 
+transactions and low bandwidth utilization.
+
+.. _bank_conflicts_theory:
+
+Bank conflict theory
+--------------------
+
+Shared memory (LDS) is organized into banks that can be accessed independently. 
+Bank conflicts occur when multiple threads access different addresses in the 
+same bank.
+
+Why bank conflicts matter:
+
+* Conflicts serialize accesses, reducing throughput
+* LDS bandwidth drops proportionally to conflict degree
+* Can turn parallel operations into sequential ones
+
+Common patterns:
+
+* **No conflict**: Each thread accesses a different bank (full bandwidth)
+* **Broadcast**: Multiple threads read the same address (no conflict)
+* **N-way conflict**: N threads access the same bank (1/N bandwidth)
+
+.. _register_pressure_theory:
+
+Register pressure theory
+=========================
+
+Register pressure occurs when a kernel requires more registers than optimal for 
+the target occupancy.
+
+Why register pressure matters:
+
+* Reduces maximum occupancy
+* May cause register spilling to memory
+* Decreases ability to hide latency
+* Lowers overall throughput
+
+The relationship between registers and occupancy:
+
+* More registers per thread → fewer concurrent wavefronts
+* Fewer registers per thread → higher occupancy but may need memory spills
+* Optimal balance depends on kernel memory access patterns
+
+.. _performance_metrics_theory:
+
+Performance metrics explained
+==============================
+
+Understanding performance metrics helps analyze GPU behavior:
+
+Peak rate
+---------
+
+The theoretical maximum performance of a GPU:
+
+* **Peak FLOPS**: Maximum floating-point operations per second
+* **Peak bandwidth**: Maximum memory throughput
+* **Peak instruction rate**: Maximum instructions per cycle
+
+Actual performance is always below peak due to various inefficiencies.
+
+Utilization metrics
+-------------------
+
+**Pipe utilization**: The percentage of execution cycles where the pipeline is 
+actively processing instructions. Low utilization indicates stalls or insufficient 
+work.
+
+**Issue efficiency**: The ratio of issued instructions to the maximum possible. 
+Low efficiency can indicate instruction cache misses, scheduling inefficiencies, 
+or resource conflicts.
+
+**CU utilization**: The percentage of compute units actively executing work. Low 
+utilization suggests insufficient parallelism, load imbalance, or synchronization 
+overhead.
+
+**Branch efficiency**: The ratio of non-divergent to total branches. Low 
+efficiency indicates significant divergence overhead.
+
+Theoretical performance limits
+==============================
+
+Understanding theoretical limits helps set realistic performance expectations.
+
+**Peak performance bounds**
+
+Every GPU has theoretical maximum performance determined by:
+
+* Clock frequency and number of compute units
+* Instruction throughput per clock cycle
+* Memory bandwidth capacity
+* Specialized unit capabilities (matrix cores, SFUs)
+
+**Achievable performance**
+
+Real applications typically achieve a fraction of theoretical peak due to:
+
+* Imperfect resource utilization
+* Memory access inefficiencies
+* Control flow divergence
+* Synchronization overhead
+* Launch and scheduling costs
+
+The gap between theoretical and achieved performance reveals optimization 
+opportunities. The roofline model provides a framework for understanding these 
+limits and identifying which factor (compute or memory) constrains performance.
+
+Summary
+=======
+
+Understanding GPU performance requires knowledge of several interconnected concepts:
+
+* **Performance bottlenecks**: Whether compute or memory limits performance
+* **Roofline model**: Visual framework for analyzing performance limits
+* **Arithmetic intensity**: The compute-to-memory ratio of algorithms
+* **Latency hiding**: How concurrent execution masks delays
+* **Occupancy**: How wavefront concurrency affects resource utilization
+* **Memory hierarchy**: How different memory types affect bandwidth
+* **Performance metrics**: Quantitative measures for analysis
+
+These theoretical foundations inform practical optimization decisions. For 
+step-by-step optimization techniques and practical guidance, see 
+:doc:`../how-to/performance_guidelines`.