The next wave of infrastructure won’t be built around giant GPUs in data centers alone. Of course inferencing infrastructure will be a huge part of it, but as agentic workloads grow, the expectations from compute and the systems around it are changing. Here’s my take on where infrastructure is headed.
- When compute meets storage
- If Infrastructure was a Restaurant
- Human-computer interfaces and its effects on infrastructure
- The next frontier
When compute meets storage
AWS Lambda (micro VMs) made serverless compute easy and efficient, this forced a rethinking in how applications were designed. They were inherently stateless. They weren’t designed to “persist” anything between invocations. Then came other players like Cloudflare workers that also worked along the same line, each with its own strengths and downsides. In order to facilitate persistence, offerings like key value stores, external DB endpoints were used. The serverless infra was optimised to be distributed, which meant that the state/persistence layer was rarely ever on the same machine/DC/AZ.
However, at some point we realised that if we handle the user experience (make it feel snappy by introducing some sort of a loading screen or an email/notification, anything that we can use to signal and trigger an async workload) on client side, then we can push the compute nearer to the storage layer (instead of bringing the storage to the compute). Let me explain with an example. Whenever you request for an account archive from a social media site, you press the request button and it instantly says your download is being processed, there is no loading screen, but the computer now happens closer to the storage where the actual data is held instead of at the edge or on the client device. However, wouldn’t it be nice if all the user data related to each user is co-located on the same machine?
In my opinion, this is where the recent/current innovation is taking place.
Need cron triggers? Checkpointing? Fan-outs? Retries that survive process restarts? Once you put serious effort into making your application production-ready, you eventually realise that durable execution is a fundamental part of the equation.
This is where platforms like Temporal, Inngest, and AWS Step Functions come in. They offer a way to retrofit durability into your application logic. Rather than relying on the lifecycle of an individual process, they introduce an orchestration layer that persists execution state, manages retries, and coordinates work across time. They extend the traditional compute model by bringing persistence closer to execution. I view them as an extension of the storage/persistence layer.
In that sense, I believe storage is the new compute.
If Infrastructure was a Restaurant
The next logical evolution in this direction are durable objects. You can think of them like remote machines (with state and compute co-located) that are globally addressable by ID and persist state between invocations (stateful execution). Durable Objects is what you get when storage isn’t just backing compute, but becomes part of the compute primitive itself.
Durable objects and similar offering like rivet actors, celld, orbs (by ampcode), dots (openAI) or other agentic avatars/bots are the expected outcomes from this new computing infrastructure. From the user’s perspective, they’re always available, resumable and durable. From the infrastructure providers perspective its a win-win since durable objects stack really well together (high server density, scale to zero). So you can have millions of them on a single machine and only keep running whichever is active and hibernate/sleep when not, all while persisting state. You can also attach all sorts of capabilities to each of these durable objects like sandbox execution, containers, websocket support, browser use, etc. as if they’re local peripherals to the machine.
Traditional computing infrastructure is like dining at an à la carte restaurant, where each dish takes time to prepare and serve, much like the latency and round trips involved in processing each request. Durable objects are like fast food restaurant chains. They’re hyper optimised for scale, all the way from the start to packaging to even how you consume (DX). Its hyper palatable from the consumer perspective (think agents and humans too).
This new infrastructure now allows for a far more interesting and granular computing (and storage) use cases. We will continue to see increase in per-user or per use-case database or rooms/topics. Your local apps can now efficiently function offline and sync whenever they reconnect to the internet. Bots or user facing agents like dots will continue to exploit and take these primitive far ahead.
We’ll also see resumable compute go mainstream. As workflows move to these new infrastructure platforms, users will expect to start a task on one device and pick it up on another (across platforms or geographic locations) without any loss of context.
Human-computer interfaces and its effects on infrastructure
Lately I’ve been thinking about how we interact with computers and consumer devices. As speech-to-speech and speech-to-text models constantly improve, many users are finding it easier and faster to speak/dictate to their devices instead of have to use touch or typing. That shift is pushing platforms to move interaction data between users and compute with as little latency as possible, often by processing it closer to the edge. Latency was a solved problem for traditional human interface devices like typing via keyboard. Dictation, vision, etc now require thinking of how we build not just the applications on the client side but also how and where we deploy them.
I’m personally glad that realtime audio (between two devices, or conference like use case and protocols like WebRTC and SIP) is so much better than what it was a just a few years ago.
The next frontier
I think networking stack will the next frontier. We already have platforms that are now optimizing for providing quick and efficient ways to transport realtime data, with the lowest latency possible.
It will bring out a lot of innovations like MOQ (Media Over QUIC) in the networking space. As mentioned earlier, as resumability becomes a common offering for most apps and devices, we will see newer tools like Iroh or tsnet (via tailscale) being used and preferred by developers. Apps will be expected to sync more efficient between devices without going through a central endpoint all the time.
And if I’m being stretchy here, I fully expect that as models get smarter and smaller, we will eventually see them running on consumer hardware. This will be followed by a wave of BitTorrent like distributed protocols, but for inferencing. Remember folks, Pirating models will be a thing, you heard it first from me!
All things said, I’ve never been this excited about the future of infrastructure.