Opening Remarks

Keynote I

Computer Architecture's AlphaZero Moment: Automated Discovery in an Encircled World

Karu Sankaralingam (NVIDIA & University of Wisconsin-Madison)

Abstract
With the end of Moore's Law and Dennard scaling, the “free lunch” of computing is over. Architectural innovation is now our primary—and perhaps only—lever for massive performance gains. Yet, we are tackling this critical challenge with an outdated approach. Today's human-driven architecture teams explore perhaps 50 to 100 designs per generation, sampling less than 0.001% of a practically infinite design space. This manual approach worked during the era of transistor abundance, but in today's scarcity paradigm—where architectures must deliver 2X performance on the same transistor budget—relying on human intuition alone is a severe bottleneck. In this keynote, we will explore the transition to automated idea factories. By utilizing multi-tiered evaluation pipelines and continuous machine learning from deployed telemetry, these automated systems generate and evaluate thousands of candidate architectures every week. I will share early results demonstrating how this approach compresses traditional design cycles from double-digit months down to single-digit weeks. Ultimately, we will examine a bold prediction for the industry's future: within two years, purely human-driven architecture research will be as obsolete as a human chess player sitting across from an engine. Welcome to computer architecture's AlphaZero moment.

Bio
Karu Sankaralingam is a Professor at UW-Madison and Research Director at NVIDIA Research. He is an IEEE Fellow, inventor, and entrepreneur. His research work has spanned computer architecture, compilers, and deep learning.

Session 1

• Beyond Pass/Fail: Evaluating Infrastructure Agents Across Layers, Lifecycle, and Risk

  Yuan Gao, Zeren Yang, Junnan Li, Shawn (Wanxiang) Zhong (University of Wisconsin-Madison); Ahmed Dajani, Mai Zheng (Iowa State University); Andrea Arpaci-Dusseau, Remzi Arpaci-Dusseau (University of Wisconsin-Madison)

  Abstract: Managing modern computing infrastructure has become a steadily harder problem due to the ever-increasing complexity. Recent advances in AI agents create a timely opportunity to automate infrastructure management tasks, but it remains unclear how well such agents can handle real-world infrastructure complexity. We present InfraBench, a benchmark suite for evaluating AI agents on realistic infrastructure tasks across the full system stack and full operational lifecycle with fine-grained risk assessment. Preliminary experiments with seven agent–model configurations show that even the strongest agent cannot secure a full score across all tasks. The mean scores range from 56.1% to 90.8%, and per-check scoring reveals a general failure pattern: agents may routinely satisfy short-term objectives while leaving non-durable changes, broken distributed invariants, unsafe side effects, and uncleaned state behind.

• Decision Locality: A Substrate for AI Inference and Continual Learning

  Jerry Felix (Brain-CA Technologies, Inc.)

  Abstract: The dominant AI infrastructure paradigm, built around floating-point matrix engines and dense data movement, is increasingly mismatched to the workloads now driving real deployments: streaming inference, on-device adaptation, and continual learning under tight power and latency budgets. We argue that this mismatch is structural rather than a question of compiler maturity or memory bandwidth. We propose Decision Locality as an alternative organizing principle for AI infrastructure: each decision, the state required to produce it, and the learning update that refines future decisions should be co-located in the smallest local hardware unit that can express them. We instantiate this principle with three concrete artifacts: a hardware-native binary learning primitive whose inference behavior is consistent with Bayesian belief updating using only local state and bit operations; a learner-ensemble fabric that composes these primitives into a streaming classifier with no back-propagation; and an FPGA realization that synthesizes the core unit to 48 LUTs and 8 registers at 96 MHz on a commercial accelerator card. We discuss the infrastructure-level questions this substrate raises, from scheduling and programming-model questions to per-bit energy accounting, and argue that it is best positioned not as a replacement for dense matrix engines but as a complementary substrate for the long tail of inference and continual-learning workloads that increasingly drive total infrastructure cost.

• When the LLM-Tuned Stack Misses: An Infrastructure View of Biological Foundation Model Inference Across NVIDIA and AMD

  Manpreet Singh (Embedded LLM)

  Abstract: AI inference infrastructure is increasingly tuned for one workload shape: dense, large-batch, NVIDIA-resident transformer decoding for LLMs. Biological foundation models, including protein language models, long-context genomic transformers, and structure-conditioned binders, do not fit that shape. They run at small batch sizes over short alphabets, attend over 100K+ token sequences, mix transformer blocks with geometric ops, and are increasingly deployed on whatever GPU is available, including AMD MI300X. We report a measurement study of six widely used biological models (ESM-2, ProtBERT, ProteinMPNN, AlphaGenome, Enformer, DualBind) on three GPUs spanning two vendors (NVIDIA L4, H100; AMD MI300X). We show that 60-80% of reference-implementation runtime is spent outside dense compute (in memory-bound element-wise ops, fragmented launches, and in one case a host-device round-trip the upstream code never removed), and that a single Triton kernel source tree closes most of the gap on both vendors without per-vendor forks. The results surface three infrastructure observations: (i) the dominant bottlenecks on bio workloads are different from those LLM stacks are tuned for; (ii) vendor-stack asymmetries amplify trivial systems bugs into order-of-magnitude regressions; and (iii) write-once cross-vendor portability for non-LLM AI workloads is achievable today with off-the-shelf compiler infrastructure.

Coffee Break

Session 2

• Safe CPU Oversubscription: One Size Does Not Fit All

  Jaylen Wang (Carnegie Mellon University); Fiodar Kazhamiaka (Azure Research); Pulkit Misra (Microsoft Research); Daniel Berger (Microsoft Azure & University of Washington); Akshitha Sriraman (Carnegie Mellon University)

  Abstract: Cloud providers oversubscribe CPUs to lower total cost of ownership (TCO) and carbon emissions, packing more virtual processors (VPs) onto a host than its logical processors (LPs) can run at once. Oversubscription introduces CPU Wait Time (CWT), the fraction of time a VP is runnable but unscheduled because no LP is free, which providers must keep below a quality-of-service (QoS) target. To bound CWT, providers and prior work use some form of static cap on host CPU utilization. We measure how CWT scales with host utilization and find that the safe oversubscription limit varies substantially with both platform configuration (the environment a VM runs on, e.g., scheduling constraints and LP pool size) and workload behavior (how a VM uses its VPs): platform-configuration factors shift the limit by up to 2.2× and workload-behavior factors by up to 1.5×. A single fleet-wide static cap is therefore either overly conservative, forgoing TCO and carbon savings, or unsafe, risking QoS violations; the variability motivates rethinking oversubscription policy, VM placement, and platform design to account for both factor families.

• Overdraft: Graceful Handling of ENOSPC in Multi-Tenant Storage Infrastructure

  Ahmed Dajani, Wei Xu, Mai Zheng (Iowa State University); Andrea Arpaci-Dusseau, Remzi Arpaci-Dusseau (University of Wisconsin-Madison)

  Abstract: Handling out-of-space errors remains a persistent challenge for data-intensive applications, often resulting in application crashes and misleading error messages. We present a study showing how applications react poorly to space exhaustion across diverse storage systems, revealing long error chains and inconsistent behavior. Motivated by the study and the dynamic, multi-tenant nature of modern storage infrastructures, we introduce Overdraft, a system that detects, manages, and mitigates storage capacity violations through dynamic space reallocation. Our eBPF-based prototype demonstrates low overhead and seamless integration with local and distributed file systems, enabling efficient thin provisioning while avoiding the reliability pitfalls of out-of-space conditions.

• Design Space Exploration for Optimal Silicon-Efficient Server Chiplets

  Ayan Chakraborty, Ali Ansari, Yuanlong Li, Shanqing Lin, Arunkrishna AMS, Alireza Foroodnia, August Ning (EPFL); Mohammad Alian (Cornell University); Michael Ferdman (Stony Brook University); Pejman Lotfi-Kamran (IPM); Babak Falsafi (EPFL)

  Abstract: Datacenter growth is driving unprecedented demand for both electricity and physical infrastructure. Today's servers typically rely on wide out-of-order (OoO) cores running at high nominal frequencies to meet SLOs, but rising power, silicon, memory, and cooling constraints call for rethinking this design point. We present a methodology for designing power-density-optimized CPU chiplets for server workloads. Our results show that optimal operating frequency can be up to 50% lower for air-cooled systems, and chiplets may favor in-order (InO) cores even for SLO-sensitive services, contrary to conventional trends. Overall, power-density-optimized chiplets improve performance by 1.7×–6.1× over conventional out-of-order baseline chiplets.

Lunch Break

Keynote II

KV Cache as the New AI Memory Abstraction

Junchen Jiang (Tensormesh & University of Chicago)

Abstract
Modern AI agents increasingly operate over long contexts—reading documents, executing multi-step plans, and maintaining evolving state—but today's inference engines reprocess this growing context from scratch, and this causes low throughput, high cost and latency bottlenecks. The key to tackling these bottlenecks is to elevate KV cache as a first-class memory layer for agentic systems, allowing the LLMs to reuse their past reasoning instead of recomputing it. I present LMCache, an open-source industry-adopted KV caching library that provides agents with fast, persistent, and addressable AI-native memory across steps, tasks, and model invocations, which dramatically reduces prefill time and GPU cost. I will also highlight the roadmap that brings in more research to industry adoption, in particular, KV cache compression and KV cache blending. Together, they make KV cache much smaller and reusable in many more cases. In short, KV cache is the missing substrate for scalable, efficient, and long-context AI agents, and building KV-aware memory infrastructure will be essential to the next generation of AI systems.

Bio
Junchen Jiang is CEO and Co-Founder of Tensormesh, Faculty Lead of LMCache Lab, and Associate Professor of Computer Science at the University of Chicago. He received his PhD degree from Carnegie Mellon University and Bachelor from Tsinghua University. He has received CMU Computer Science Doctoral Dissertation Award (2017), two Google Faculty Research Awards on ML Systems (2018, 2026), an NSF CAREER Award, and multiple best paper awards.

Session 3

• Reimagining LLM Inference Infrastructure with Memory-Centric KV Cache Servers

  Khyati Kiyawat, Kevin Skadron (University of Virginia)

  Abstract: Data centers are building GPU warehouses to serve a memory problem. As generative AI workloads shift from batch training to interactive, long-context, and agentic inference, the dominant bottleneck has migrated from floating-point throughput to KV cache capacity and bandwidth. Production deployments report KV cache reuse rates of 50–90% [1], yet infrastructure investments remain anchored to compute-centric accelerators whose memory is expensive, and capacity-limited. We argue that a KV cache server—disaggregated, CXL-attached memory device(s) with optional near-memory compute—is the correct infrastructure primitive for the next generation of AI data centers. We propose a spectrum of CXL-attached memory devices—from CXL DDRx expanders, through CXL-PNM, to CXL-PIM+PNM archetypes—as the substrate for a KV cache server infrastructure. Our back-of-envelope analysis grounds this vision for cost- and energy-efficient inference. We advocate architecting AI data centers around memory, not FLOPs.

• Beyond Compression Ratio: KV Cache Policies in Tiered LLM Infrastructure

  Devasena Inupakutika, Dongjoo Seo, Swarna Prabhu (Samsung Semiconductor Inc.)

  Abstract: KV cache compression is now load-bearing in LLM serving, yet most research stops at the algorithm boundary, reporting compression ratio (CR) on isolated prompts under a single-tier cache. Production stacks are multitier, where engine-side compression reshapes storage I/O and mean CR hides three effects: write-volume jitter, broken prefix reuse, and recomputation. Replaying real inference traces through a simulator, we benchmark two engine-involved scorers – KVZap (content-based learned) and TriAttention (position-based, training-free) – under fixed, percentile, and adaptive policies ± a reuse-aware bonus. Prefix preservation spans a 6× swing (10% to 61%) on tool/agent, with policy alone lifting a fixed KVZap 10% to 36%; on conversation, CR coefficient of variation swings 1.7× (0.21 to 0.12). The two scorers reach high reuse through opposite mechanisms – position-based preserves prefix by default, content-based evicts it – invisible at the algorithm-level CR × accuracy plot. Compression in tiered serving is an infrastructure design choice, not an algorithmic one.

• Is High-Bandwidth Flash All You Need?

  Vinicius Petrucci, Felippe Zacarias, Vishal Tanna (Micron Technology)

  Abstract: Large language model (LLM) inference accelerators are converging on multi-tier memory to hold multi-terabyte mixture-of-experts (MoE) weights and long-context key-value (KV) state. We use a roofline-based analytical model to compare three augmentations to high-bandwidth memory (HBM): on-package high-bandwidth flash (HBF), on-package LPDDR, and LPDDR over a chip-to-chip (C2C) link to a host CPU. We find that on-package LPDDR matches or beats every alternative at every operating point we evaluate, by up to 5.6× over HBM-only, via three properties. (1) Capacity: on-package augmentation admits compact layouts that HBM-only cannot fit; at peak throughput under the latency SLO, HBF and LPDDR both shrink the model-instance footprint 4× (16 to 4 GPUs). (2) Writability: LPDDR absorbs KV spill that HBF cannot serve, given NAND's write-rate and endurance limits, yielding up to 2.1× over HBF on KV-heavy 1M-context workloads. (3) On-package bandwidth: where C2C carries critical-path traffic, LPDDR's higher effective decode bandwidth yields 1.3×–2.0× over C2C. The larger LPDDR-over-C2C gaps are replica-shrink (property (1)): on-package LPDDR can host the cold-expert pool whereas off-package C2C cannot, positioning C2C-attached LPDDR as a KV-overflow rather than expert-offload tier.

Coffee Break

Session 4

• Store or Recompute? Characterizing the Carbon Implications of KV Cache Retention in LLM Inference

  Abnash Bassi, Jaylen Wang (Carnegie Mellon University); Fiodar Kazhamiaka, Daniel S. Berger (Microsoft); Akshitha Sriraman (Carnegie Mellon University)

  Abstract: The growing compute demand of Large Language Model (LLM) inference carries a significant environmental cost. To reduce GPU load, intermediate computation (KV caches) can be saved in non-volatile storage servers and retrieved when the session becomes active. This setup presents a fundamental tradeoff between high power and associated operational carbon emissions of recomputing the KV cache versus the high capacity requirements and embodied carbon emissions of storage. We present a first-order analytical model to identify break-even points between the two choices as a function of storage duration, comparing the embodied carbon of storage hardware against the operational carbon of GPU-based recomputation. We further examine how response time requirements interact with carbon-saving decisions, revealing scenarios where environmentally preferable strategies may conflict with performance needs. Our results demonstrate that the break-even duration varies significantly with different system parameters (e.g., the carbon intensity of the energy source), suggesting opportunities for sustainability-aware retention policies.

• Power Breakdowns and PUE Implications of Air-Cooled and Direct-Liquid-Cooled Servers

  August Ning, Xavier Ouvrard, Babak Falsafi (EPFL)

  Abstract: Power usage effectiveness (PUE) has guided years of datacenter energy efficiency improvements, but only differentiating between facility and IT power obscures insights into how IT power is used within the servers and may misdirect optimizations. In this work, we measure air-cooled and direct liquid-cooled (DLC) servers and break down their power into power supply (PSU) loss, fan, idle, and compute power via power distribution unit (PDU)-level measurements. We apply our insights to show how high ambient temperature operation affects server power profiles and distorts PUE: increasing fan power improves PUE even though energy efficiency may decrease, while transitioning to DLC can worsen PUE despite reducing total datacenter energy consumption.

• Speculative Load Micro-op Fusion

  Deepanjali Mishra (Carnegie Mellon University); Tanvir Ahmed Khan (Columbia University in the City of New York); Gilles Pokam (AMD); Heiner Litz (UC Santa Cruz); Akshitha Sriraman (Carnegie Mellon University)

  Abstract: Modern data center applications experience significant backend pipeline stalls, degrading performance and energy efficiency. One promising approach to reduce these stalls is load micro-op pair fusion, where two load micro-ops accessing the same cache line are merged into a single fused micro-op. However, existing approaches require both load micro-ops to be simultaneously visible before fusion can occur, limiting fusion coverage.

  
  

  We present I-Fuse, a profile-guided speculative load micro-op pair fusion mechanism. I-Fuse consists of two components: (1) profile-guided fusion, which identifies fusible load pairs using dynamic execution profiles, and (2) speculative fusion, which enables fusion when only the first load has been observed.

  
  

  We evaluate I-Fuse using eight backend-bound data center applications. I-Fuse improves performance by 7.4% over a system without fusion and outperforms the state-of-the-art fusion technique, Helios, by 6.8×. I-Fuse achieves 78.9% of ideal speedup on average, and up to 97.9%.