TCP and RDMA are Killing Inference Throughput; Homa can Fix It
Modern AI inferencing is shifting from monolithic requests to complex agentic workflows and disaggregated KV stores. As a result, AI network traffic is no longer just very large transfers; tiny metadata requests are becoming more and more common, and their latency has a critical impact on throughput. Unfortunately, legacy transport protocols such as TCP and RDMA perform poorly on these workloads due to poor congestion control and head-of-line blocking. This talk will discuss the problems with TCP and RDMA and provide a brief introduction to the Homa transport protocol. Homa uses receiver-driven flow control and capitalizes on priority queues in network switches to reduce short-message latency by 10x for workloads like those in AI datacenters.
About the Software Factories Track
Software Factories sessions at AI Engineer World's Fair 2026 in San Francisco.
When
Thursday, July 2, 2026
9:50 AM - 10:10 AM·20m
Where
Main Stage
Capacity: 4000 attendees
Sign in to add this talk to your schedule.
Speaker
John Ousterhout
Bosack Lerner Professor of Computer Science / Professor Emeritus
Stanford University
@johnousterhout
John Ousterhout is the Bosack Lerner Professor of Computer Science at Stanford University. He is author of the book "A Philosophy of Software Design", co-creator of the Raft consensus protocol, and creator of the Tcl scripting language and the Tk toolkit. Ousterhout received a BS degree in Physics from Yale University and a PhD in Computer Science from Carnegie Mellon University. He is a member of the National Academy of Engineering and recipient of the ACM Software System Award.