CUDA and the Software Moat: What It Is Actually Made Of

“CUDA is the moat” is repeated so often that it has stopped conveying anything. The useful version names the layers, because they have very different replacement costs, and a competitor that has replaced three of them is in a completely different position from one that has replaced five.

The moat is not the language

The C-like language for writing kernels is the least defensible part. It is well documented, its execution model has been reimplemented, and a source-to-source translation of most kernels is mechanical. If the moat were a language it would have been crossed a decade ago.

What is hard is everything built on top: the tuned libraries, the collective communication implementation, the framework integration, and the enormous accumulated tail of third-party code that assumes all of it works. This page decomposes that, because deciding whether a non-incumbent platform is viable for your workload is a layer-by-layer question with checkable answers, and a general verdict is not.

There is a structural reason the upper layers resist replacement that has nothing to do with anyone’s intentions. Performance work is specific to a memory hierarchy, a tile size, a warp width and a scheduling model, so it does not transfer between architectures even when the source code does. Every technique that arrives — a new attention variant, a new quantisation format, a new sampling method — therefore has to be re-optimised per platform by someone who knows that platform intimately. The incumbent has more of those people because it has more users, and it has more users partly because it has more of those people. That loop is the moat, stated without metaphor.

Five layers, increasing difficulty

The asymmetry matters. A competitor can be excellent at layers one and two and still be unusable for a serving stack that depends on layers three and four, and it can be excellent at all four and still cost you weeks at layer five.

How the portability layers attack it

• Framework-level abstraction. If a model is written entirely in a framework’s standard operations, the framework’s backend dispatch can target a different device without changing the model. This works well for straightforward architectures and breaks the moment a model calls a hand-written kernel.
• Kernel DSLs. Languages that let you write a tiled kernel once at a higher level of abstraction and compile it for multiple targets attack layer three directly — the layer that otherwise requires an expert per architecture per technique.
• Graph compilers. Take the whole model as a graph, optimise and fuse it, emit code for whichever backend. Strong for static-shape workloads; the dynamic shapes of serving are exactly where they are least comfortable.
• Source translation. Mechanically converting kernels from one platform’s dialect to another. Effective for layer one and for simple kernels, and it inherits rather than removes the performance-tuning problem, because a translated kernel is tuned for the architecture it was written for.

A checklist you run yourself

The state of any particular alternative platform changes faster than an article can track, so this page does not assert one. Run these questions against whatever you are evaluating, on the day you evaluate it. Each has a concrete answer.

• Does the serving engine you intend to use support this backend, in its own documentation, for the model family you run? Not “is there a port” — is it a supported configuration.
• Is there an IO-aware attention implementation? Without it, long-context serving is memory-bound in the wrong way and the platform is not competitive regardless of its peak numbers.
• Are quantised kernels available at the precision you plan to serve in? A platform that only runs bf16 doubles your memory and halves your decode ceiling relative to one that runs fp8.
• Does the collectives library support the topology you will deploy on, and at what message sizes is it tuned? This is the layer that fails quietly, as poor scaling rather than as an error.
• Who fixes a kernel bug, and on what timescale? On a platform with a large community the answer is often “someone already did”. That is the value of layer five, stated concretely.

Why inference is the easier beachhead

If a platform is going to displace an incumbent it will happen in inference before training, and the reason is structural rather than commercial. Inference runs a fixed graph, so the set of operations that must be fast is small and enumerable — attention, the feed-forward matrices, a normalisation, a sampler. Training adds backward passes for every operation, optimiser states, gradient collectives at a very different scale, and a research workload that by definition uses operations nobody anticipated.

So the practical read is: a platform that covers layers one to four for the specific op set of transformer inference is viable for that job even while it remains unviable for research. That distinction is worth keeping, because “can it replace the incumbent” is the wrong question and “can it serve this model at this precision with this engine” is a question you can answer in an afternoon.

There is a second reason inference is the softer target: the surface you must match is an API rather than a research environment. If a serving stack behind an endpoint changes hardware, nothing upstream of the endpoint needs to know, so the switching cost is contained within one system rather than distributed across every team that writes model code. That is why portability is worth treating as an operational property with a value — the ability to move a workload is what turns a supply constraint or a price change from a crisis into a reconfiguration.