The Shifting Compute Frontier: How AI Is Coming Home

A thought paper on the gradual movement of artificial intelligence from the cloud to the devices we already carry.

The device in your pocket is more powerful than you think

There is something quietly strange about the way most people use AI today. If you have a recent phone, that phone in your hand almost certainly has a faster processor than the servers that powered the first generation of impressive language models just a few years ago. It has more memory than a respectable workstation from a decade ago. Its camera chip does real-time computational photography that would have required a small data centre in the 1990s. And yet, when you open an AI assistant and ask it something, your question leaves your device, travels to a building full of specialized hardware somewhere in Virginia or Oregon or Dublin, gets processed there, and the answer travels back. The whole round trip happens fast enough that it feels instant, so most people never think about it.

But that architectural choice, routing nearly all AI work through remote servers, was never inevitable. It was a product of specific conditions that existed at a specific moment in time. Those conditions are changing. And as they change, the question of where AI actually runs is becoming one of the more interesting structural questions in technology.

This piece is my attempt to think through that shift. It is not a prediction that the cloud is going away or that every AI task will run locally within five years. The reality seems more nuanced and, honestly, more interesting than either of those extremes. What is happening is a gradual redistribution of where different kinds of AI work get done, driven by hardware improvements, economic pressures, and the quietly different experience that local AI provides. Understanding that shift matters whether you are a regular technology user thinking about what your devices will feel like in a few years, or someone working in an organization trying to figure out where AI infrastructure investments are going.

Let's start with the problem that created the current situation in the first place.

The data centre tax and what it cost the rest of us

If you bought a laptop or a phone in the last few years and noticed that memory and storage felt expensive relative to what you expected, you were not imagining things. The pricing dynamics that drove that experience are worth understanding, because they connect directly to why capable on-device AI did not arrive sooner.

When the current wave of large AI models started becoming commercially serious around 2022 and 2023, the major cloud providers, Google, Microsoft, Amazon, Meta, and others, began an extraordinary buildout of specialized computing infrastructure. The hardware at the heart of this buildout was graphics processing units repurposed for AI work, and the memory that sits inside those systems, a specialized type called high-bandwidth memory. Both the chips and the memory are made by a small number of manufacturers. TSMC makes most of the advanced chips. Samsung and SK Hynix make most of the high-bandwidth memory. Those same manufacturers also produce the chips and memory that go into your phone, your laptop, and your tablet.

The problem is that chip manufacturing capacity is finite. When hyperscalers started contracting enormous volumes of high-bandwidth memory and specialized AI processors, the manufacturers shifted their most capable production toward those high-margin orders. The supply available for consumer devices tightened. Unfortunately, the pricing responded accordingly. Storage chips for ordinary laptops and phones got caught in the same supply squeeze, because the factories that make them overlap significantly with the factories making data centre components.

The effect was that at exactly the moment when AI capabilities were maturing and on-device processing hardware was theoretically becoming powerful enough to run meaningful AI workloads, the component costs for that hardware stayed elevated. Phones with enough RAM to run a capable local AI model reliably were more expensive than they would otherwise have been. Laptops with the memory bandwidth needed for smooth local inference were pushed toward the premium end of the market.

This matters because it slightly delayed the natural price point at which capable AI hardware reached ordinary consumers. The technology was not the bottleneck in the way people often assume. The supply chain economics of a specialized infrastructure buildout were.

It is also worth noting that software choices compounded the hardware constraints. The large transformer models that became the standard architecture for AI were designed from the beginning around data centre hardware. They were optimized for machines with tens or hundreds of gigabytes of high-bandwidth memory, not for devices with eight or twelve gigabytes of shared system memory. Running them efficiently on consumer hardware required significant additional engineering work, quantization techniques that reduce model size without destroying quality, new software frameworks designed for mobile and laptop chipsets, and architectural innovations that made models less memory-hungry. That work was happening in parallel, but it took a fair bit of time.

The good news is that this picture has been changing. Memory supply has expanded substantially as manufacturers added capacity. The pricing pressure on consumer components will likely start to ease. And the software engineering work to make AI run efficiently on consumer hardware has matured considerably. The conditions that delayed affordable on-device AI are fading, which brings us to the question of how quickly the alternative actually arrives.

The on-device trajectory, and when fast enough actually becomes fast enough

There is a benchmark that tends to get used informally among people who think about the feel of AI: roughly seventy-five tokens per second. A token is approximately three-quarters of a word, so seventy-five tokens per second is close to reading speed for a fast reader. At that pace, an AI generating text feels like it is thinking alongside you rather than making you wait for it. Below that threshold, there is a perceptible lag that shifts the psychological experience of using AI from conversational to transactional. You ask, you wait, you get an answer. Above it, the experience starts to feel genuinely interactive.

The question of when average consumer hardware reaches that threshold for a usefully capable model is one of the more consequential near-term questions in technology. The answer, based on current trajectories, is that the leading edge is already there, and the average is not far behind.

The highest-end smartphones available in late 2024, devices running Apple's A18 or Qualcomm's Snapdragon 8 Elite chips, could already hit or exceed that seventy-five token per second range on models in the three to seven billion parameter range. Those are models that are genuinely useful for summarization, drafting, question answering, and basic reasoning, not toy demonstrations. The flagship devices are doing real AI work, locally, at conversational speed.

The question is how quickly that capability diffuses down to mid-range hardware, which is where most people actually live. Based on how chip performance has historically improved across generations, roughly doubling in useful AI throughput every couple of years, a reasonable projection is that the majority of new smartphones and laptops sold will comfortably meet this threshold somewhere in the 2026 to 2028 window. That is not a long time.

Three things are driving this improvement simultaneously, and they compound rather than just add.

The first is raw processing power. Every major chipmaker has been integrating dedicated neural processing units into their silicon, chips specifically designed to run the matrix multiplication operations that dominate AI workloads. The TOPS ratings on those units, a measure of how many AI operations per second they can perform, have been climbing steeply with each chip generation.

The second is memory bandwidth. Transformer models, the architecture that underlies most current AI, spend a surprising amount of their inference time not computing but waiting for data to move between memory and the processor. Improving that bandwidth has an outsized effect on inference speed, and chipmakers have been making sustained progress here, particularly in the integrated designs used by Apple and increasingly by Qualcomm and others.

The third is software efficiency. The work of quantization, taking a model that was trained at high precision and compressing it into a form that uses less memory and runs faster, has become significantly more sophisticated. Models that were unusable on consumer hardware two years ago can now run at reasonable quality in compressed form. And new model architectures are emerging that are simply less memory-hungry to begin with, achieving comparable quality with a fraction of the data movement.

Together, these three improvements create a trajectory where on-device AI capability is advancing faster than most public conversation about AI acknowledges. The headlines tend to focus on the frontier, the largest models with the most impressive capabilities. The more practically significant story for most people is happening at the other end of the scale, in the hardware they are about to buy or already own.

Quiet democratization, what really changes when AI moves to the device

The word democratization gets used a lot in technology circles, often to mean "this thing got cheaper." That is part of the story with on-device AI, but it undersells what actually changes when inference moves from a remote server to a local chip.

The most obvious difference is latency. When an AI model runs on your device rather than in a data centre, the round trip time disappears. There is no network request, no queue, no response traveling back across an ocean of fiber. The processing happens in the silicon millimeters from your screen. For short tasks, summarizing a paragraph, answering a question, finishing a sentence, the practical difference between a fast cloud model and a local one is already small. But local inference does not degrade on a slow network, does not fail when you have no connection at all, and does not accumulate per-request costs that add up over a day of heavy use. It is simply always available.

The second difference is privacy, and this one tends to matter more to people the more they actually think about it. Every AI interaction that travels to a cloud server is, at minimum, passing through infrastructure operated by a third party. Most major providers have strong contractual and technical commitments around data handling. But the data does leave your device. For personal writing, sensitive work documents, medical questions, financial planning, and anything you would not want stored in a log somewhere, the knowledge that processing happens locally and data does not leave is meaningfully different. It is not just a privacy preference; for many professional and regulated contexts, it is a compliance requirement.

The third difference is more subtle but worth naming. There is something psychologically different about AI that runs on a device you own versus AI that runs on a service you subscribe to. The local version does not require a network account, does not update in ways you did not choose, and does not have terms of service that can change. For average users who are not deeply engaged with AI technology, this difference may never consciously register. But it shifts the underlying relationship from service dependency toward something that feels more like a tool you possess. And the understanding that AI is just another tool is an important distinction when defining a relationship with it.

Now, it is important not to oversell this. On-device models will trail frontier cloud models by a meaningful capability gap for years. The most sophisticated reasoning tasks, complex legal or medical analysis, multi-step research, creative tasks that require deep world knowledge, these will continue to benefit from the much larger models that only run in the cloud. The user who needs frontier-level capability will continue to reach for cloud-based tools.

But the user who needs to summarize a meeting, draft a quick email, get an answer to a factual question, or have a document edited for clarity, that user does not need frontier capability. They need something good enough, fast, available offline, and not sending their data anywhere. On-device AI in the 2026 to 2028 timeframe will clear that bar for a large fraction of everyday tasks.

The risk worth watching is a quiet bifurcation of access. If on-device AI becomes capable enough for routine tasks but requires a certain tier of hardware to do it well, a divide opens between people on recent premium or mid-range devices and people on older or budget hardware. Democratization that only reaches people who can afford the right device is a partial democratization. The industry has not fully resolved this tension yet, and I would guess it will take a while.

Did we overbuild the cloud?

This is the question that sounds provocative but is worth asking seriously. The current wave of data centre construction represents one of the largest capital investment cycles in technology history. The major cloud providers and AI companies have spent and committed to spend hundreds of billions of dollars on new infrastructure over the next several years. The implicit assumption in most of those investment theses is that AI demand will grow continuously, and that the cloud is where that demand will be served.

But what happens to those investments if a substantial share of routine AI inference gradually moves to the edge?

The answer is probably not a stranded asset crisis, but it is worth thinking through. Training large AI models will remain centralized for the foreseeable future. The energy, hardware, and data requirements for training make it implausible that model training migrates to edge devices in any meaningful way. That work belongs in large facilities with dedicated power infrastructure and purpose-built hardware.

Complex inference will also remain cloud-resident. Tasks that require very large context windows, real-time access to current information, multimodal reasoning across many sources, or the output of the very largest frontier models simply cannot be done at the edge with any reasonable hardware. The cloud's role in these workloads is not threatened by on-device improvements.

What is more vulnerable is commodity inference, the large volume of short, repetitive, relatively simple AI interactions that currently travel to cloud servers because there is nowhere else for them to go. Summarize this, finish this sentence, answer this FAQ, classify this text. These are the interactions that will migrate to on-device hardware as that hardware becomes capable enough. And they represent a very large share of total AI request volume even if they represent a modest share of revenue or compute cost.

The honest answer to whether cloud capacity is adequate for future demand is that it depends entirely on the assumption about where that demand gets served. If edge offload happens at scale, the current buildout may prove more than adequate for cloud-native tasks. If it does not happen, or happens slowly, demand continues to grow into available capacity. Neither scenario is obviously correct at this point.

What seems clear is that the assumption of indefinitely growing cloud inference demand, unchecked by any meaningful edge offset, is probably too simple. The more interesting question for infrastructure planners is what the cloud's specific job is in a world where routine tasks have migrated. The answer to that question shapes what you should be building.

The personal AI mix, now and in three years

Think about the AI interactions a reasonably active technology user has in the course of a day right now. Keyboard autocomplete on a phone. A search query that uses AI ranking. Maybe a voice assistant command. An interaction with a chatbot in a shopping app. If they are more engaged with AI tools, perhaps a conversation with an AI assistant for writing help, a coding question, an image generation request.

Of that mix, nearly everything substantive is running in the cloud today. The local processing is mostly limited to things that feel like features rather than AI: keyboard prediction, face recognition for photo organization, maybe a basic voice wake word. The gap between what the hardware could theoretically do and what it actually does locally is large.

Three years from now, that picture looks meaningfully different, not because of any single breakthrough but because of the compounding of the improvements described earlier.

Operating system vendors are the key actor here. Apple has already begun integrating on-device AI into iOS and macOS in ways that route some tasks locally and escalate more demanding tasks to cloud infrastructure, transparently to the user. Google is doing the same on Android. The intelligence of that routing will improve. The quality of the on-device tier will improve. And crucially, users will not have to make deliberate choices about which tier to use; the operating system will handle it.

For a typical smartphone user in 2028, a reasonable projection is that somewhere between forty and sixty percent of AI interactions happen on-device doesn’t feel unrealistic to me. The short, high-frequency interactions, the ones that happen dozens of times a day, tilt heavily toward local. The extended, complex, or knowledge-intensive interactions tilt toward cloud. The practical experience is an AI assistant that is simply faster for most things and maintains full cloud capability for the rest.

For people who actively choose their tools rather than using whatever the operating system provides, the mix gets more interesting. There will be more local model runners, applications that let you download and run specific models on your own hardware, giving you a set of private, always-available AI tools completely separate from any cloud service. This already exists in early form with applications like Ollama and LM Studio. In three years, the user experience of these tools will very likely be significantly more polished, the models significantly more capable, and the hardware significantly more capable of running them well.

The interesting tension in the personal space is between convenience and control. The operating system approach is seamless but opaque; you may not know or particularly care which tier handled a given request. The deliberate local approach requires some engagement but gives you something that genuinely runs on hardware you own, with models you chose, producing output that never traveled anywhere. Different users will resolve that tension differently, but the option to be deliberate about it will exist in a way it largely does not today.

The enterprise playbook, hybrid by design

Enterprises have always cared about two things in technology procurement above almost everything else: cost and control. Cloud AI creates a genuinely interesting tension with both of those values, and that tension is driving a meaningful shift in how organizations are thinking about AI architecture.

The cost dimension is straightforward. Every AI API call has a price. For an individual user making a few dozen requests a day, those costs are negligible or absorbed into a subscription. For an enterprise with thousands of employees making hundreds of requests each, the arithmetic changes significantly. Organizations that have rolled out cloud AI tools broadly are starting to see material budget lines appear, and they are beginning to ask whether every one of those interactions needed to travel to an external server and incur a per-token charge.

The control dimension is more complex. Regulated industries, financial services, healthcare, legal, government, have long had to think carefully about where sensitive data goes. Cloud AI introduces a new category of concern: not just where data is stored, but where it is processed. Even with strong data processing agreements and contractual protections, sending client information, patient data, or legally privileged content to a third-party AI service creates compliance exposure that many organizations would prefer to avoid entirely. Local inference sidesteps that exposure at the architecture level rather than managing it contractually.

These two pressures together are pushing enterprises toward hybrid AI architectures, not as a preference but as a deliberate design decision. The emerging pattern is a model where on-device or locally deployed AI handles routine, high-volume, sensitive interactions, and cloud AI is reserved for tasks that genuinely require frontier capability or that involve non-sensitive content where the per-token cost is justified by the complexity of the task.

What kinds of enterprise work fits the on-device or local deployment model well? Meeting summarization is an obvious one, both because it is high volume and because meeting content is almost always sensitive. Internal document drafting and editing is another. Basic code assistance for developers. Employee-facing Q and A based on internal knowledge bases. These are tasks that happen constantly across a large organization, involve content that should not leave the building, and do not require frontier-level reasoning to do well.

What stays in the cloud? Customer-facing AI at scale is one clear case. When an organization is running AI interactions with thousands of external customers simultaneously, the infrastructure requirements and the diversity of requests both point toward cloud deployment. Enterprise automation workflows that require complex reasoning across many data sources are another. Security operations, where AI is being used to detect threats, analyze anomalies, or correlate signals across an entire environment, benefit from the centralized view that cloud infrastructure provides. And any task that genuinely requires the most capable frontier models, complex legal analysis, sophisticated financial modeling, novel research synthesis, belongs in the cloud.

The vendor landscape implication of this shift is significant. Companies that have built businesses selling cloud AI inference as a service face pressure to also offer on-device or locally deployable versions of their models. That is a different product, a different pricing model, and a different relationship with the customer. Device manufacturers, Apple, Qualcomm, Google on the hardware side, gain leverage they have not traditionally had in the enterprise AI conversation. A CIO who is thinking about AI architecture now has to consider the silicon roadmap of the devices their employees carry alongside the API pricing of their AI vendors. That is a new kind of procurement consideration.

Conclusion: The cloud gets more specialized, not smaller

The argument that I have been building toward while I have been considering this thought paper is not that cloud AI is going away. It is that cloud AI is going to become more purposeful.

The conditions that made cloud the only viable option for serious AI work were real but temporary. The component supply chain dynamics that kept consumer hardware more expensive than it should have been should start easing. The hardware improvements that are making on-device inference fast enough to feel genuinely useful are on a predictable trajectory. The software engineering work that makes capable models run efficiently on devices most people already own or will soon own is maturing rapidly.

As those conditions resolve, the usage pattern that emerges is one where AI is distributed across a spectrum from fully local to fully cloud-based, with most of the routine daily interactions handled at the edge and the genuinely complex or frontier tasks handled in the cloud. For individuals, this means AI that is faster and more private for everyday use, with cloud capability available when you need it. For enterprises, it means a deliberate architecture decision about which tasks justify cloud exposure and cost, with a growing category of sensitive and routine work that belongs on local infrastructure.

The cloud does not shrink in this scenario. It gets more specialized. The work that remains there is harder, more complex, more valuable, and better suited to what large-scale centralized infrastructure actually does well. Training, frontier inference, complex reasoning, real-time data aggregation, customer-scale deployment. These are tasks where the cloud's advantages are durable.

What leaves the cloud, gradually and largely invisibly, is the long tail of simple, frequent, personal AI interactions that make up the majority of AI request volume but not necessarily the majority of AI value. When those interactions move to the device, users gain speed, privacy, and resilience. Enterprises gain cost control and data governance. And the cloud gains focus.

The device in your pocket is already more powerful than most people realize. Within a few years, it will be running AI for most of the things you actually use AI for. The cloud will still be there when you need it for the hard stuff. That is not a bad outcome for anyone.