From the hardware's side, an inference run is a traffic pattern. I work on reading that pattern off real workloads, and on what follows from it: memory systems, on-chip and chip-to-chip networks, and the RTL underneath.
One decode step of a pre-norm decoder layer, drawn as traffic. Three bands: the operator order — RMSNorm, QKV projection, RoPE, KV append, attention, output projection, RMSNorm, MLP — each tagged with what it reads or writes off-chip; four HBM channels, all busy, with the served stripe sweeping them in turn; and weights multicast from the memory edge into a mesh of tiles whose partial sums reduce into an accumulator bank. A segmented gauge grows with the KV cache at every step, and the tiles stay far idler than the links. 01 · WORKLOAD — ONE DECODE STEP 02 · HBM 03 · INTERCONNECT — WEIGHT-STATIONARY TILES STEP 01/03 STEP 02/03 STEP 03/03 RMSNORM RMSNORM QKV PROJ RD W QKV PROJ RoPE RoPE KV APPEND WR KV KV APPEND ATTN RD KV ATTN O PROJ RD W O PROJ RMSNORM RMSNORM MLP RD W MLP CH0 CH1 CH2 CH3 2 PC / CH KV$ MC ACC TILES WEIGHT MULTICAST KV READ KV WRITE PARTIAL SUM ASSUMES FINE-GRAIN CHANNEL INTERLEAVE One decode step of a pre-norm decoder layer, drawn as traffic. Three bands: the operator order — RMSNorm, QKV projection, RoPE, KV append, attention, output projection, RMSNorm, MLP — each tagged with what it reads or writes off-chip; four HBM channels, all busy, with the served stripe sweeping them in turn; and weights multicast from the memory edge into a mesh of tiles whose partial sums reduce into an accumulator bank. A segmented gauge grows with the KV cache at every step, and the tiles stay far idler than the links. 01 · WORKLOAD 02 · HBM 03 · INTERCONNECT STEP 01/03 STEP 02/03 STEP 03/03 RMSNORM RMSNORM QKV PROJ RD W QKV PROJ RoPE RoPE KV APPEND WR KV KV APPEND ATTN RD KV ATTN O PROJ RD W O PROJ RMSNORM RMSNORM MLP RD W MLP CH0 CH1 CH2 CH3 2 PC / CH KV$ MC ACC TILES WEIGHT MULTICAST KV READ KV WRITE PARTIAL SUM ASSUMES FINE-GRAIN CHANNEL INTERLEAVE FIG. 01 — ONE DECODE STEP OF A PRE-NORM DECODER LAYER, DRAWN AS TRAFFIC. WEIGHTS MULTICAST FROM THE MEMORY EDGE, PARTIAL SUMS REDUCE DOWN EACH COLUMN. THE LINKS STAY BUSY WHILE THE TILES WAIT. SCHEMATIC, NOT TO SCALE. Life is a long trade, not a greedy algorithm.