Inference no longer runs in one place on one kind of chip. It runs in the public cloud, the enterprise datacenter, colocation space, the edge inside a store or clinic or branch, and increasingly on the client device. It runs on GPUs from NVIDIA, AMD, and Intel, on custom accelerators from AWS and Google, on NPUs from the client world, and on CPUs when the model is small enough to justify it.

That diversity is economically rational and operationally miserable. Every accelerator vendor ships its own software stack, so every model gets ported, tuned, and re-optimized for every target. That work requires kernel engineers, who are among the scarcest people in the industry. The practical result is that hardware selection is driven by software availability rather than economics, which is backward.

And when the major chip vendors talk about heterogeneity, they are usually referencing a vertical integration of their own silicon. Not running workloads on silicon from different vendors across the datacenter.

Modular has spent four and a half years attacking exactly this. The company was founded by compiler and runtime veterans whose prior work includes LLVM, MLIR, and TensorFlow, and the architecture reflects that lineage. Mojo is a systems language with Python-like syntax for writing kernels that compile across targets. MAX is the modeling and serving framework that compiles a model for whatever hardware sits in front of it and exposes an OpenAI-compatible endpoint. Modular Cloud is the cluster layer and managed service on top.

Its ModCon event was the clearest evidence I’ve seen so far that the approach is becoming viable.

Silicon Diversity Expands to Six Vendors

Modular’s platform previously ran on GPUs from NVIDIA and AMD, plus Apple Silicon for local development. Those are different implementations of a broadly similar programming model, so supporting all three is real work but familiar work. The keynote added AWS Trainium, Google TPUs, and Qualcomm’s Dragonfly accelerators, none of which are GPUs. Each brings its own toolchain, memory model, and execution style. That’s the harder version of the portability problem, and it’s why the enablement numbers below are worth examining.The supporting announcements matter as much as the additions. Mojo 1.0 was open-sourced under the Apache 2.0 license, with the compiler and toolchain included. Modular Cloud went generally available. The MAX framework is dropping its device usage restrictions in favor of a more permissive license, with an ecosystem alliance program due by year’s end. Native Windows support for Mojo is coming through Microsoft.

Modular also described its cloud layer as a hypervisor for AI compute, with the accompanying idea that you stop buying accelerators and start buying tokens. The cluster engineering underneath that claim is real, and it’s worth understanding where it sits commercially.

Benchmarks Are Good, but Enablement Economics Rule

Most coverage will lead with performance. The more consequential number is what it now costs to support a new accelerator.

Modular says five of its engineers brought Trainium into the platform with full serving support in under five months, against an industry norm it puts at hundreds of engineers over two years. Neither figure was accompanied by methodology, so treat both as the company’s framing rather than as measured fact. But even if that number is off just a little bit, this is a big deal.

The more persuasive evidence came from outside. A services firm working with Modular described bringing up Google TPU support in roughly four months with a handful of engineers and little prior TPU knowledge. That matters more than an internal figure, because a portability layer that only its author can extend is just a product roadmap. One that others can extend is a foundation. Worth noting that the outside team required access to Modular’s closed source to do the work, which is precisely what the licensing change is meant to make unnecessary.

If enablement costs have really dropped by an order of magnitude, the downstream effects are significant. It becomes viable for a new accelerator to reach usable software support within months rather than never, giving second- and third-source silicon a path to enterprise relevance that hasn’t existed since CUDA established its position.

Cheap enablement isn’t sufficient on its own, though. Getting a model running on alternative silicon only matters if running it there is a better deal, and portability that lands you on slower or costlier hardware is an engineering achievement with no buyer. Which is why the other number from the keynote deserves attention.

A TCO Model with Teeth

Modular said that running a frontier model on the AMD MI355X, holding deployment size and latency targets constant, delivers roughly 90% of the throughput of an NVIDIA B200 at about half the cost per accelerator-hour, resulting in more than 45% lower total cost of ownership.

This is the first time I’ve seen the case for heterogeneous inference expressed as a number a CFO will notice. Performance and optionality are good messages, but they’ve never closed a budget conversation. A validated 45% delta would. Especially when talking about performance within striking distance for many inference workloads.

Again — caveat is required. This is also a vendor benchmark presented without the detail needed to evaluate it. The model, precision, batch, and concurrency settings, software versions, assumed utilization, hardware pricing source, and what’s included in the TCO calculation all move that number materially, and none were disclosed. Directionally, the argument is plausible, since MI355X has generally been available at a lower hourly rate than B200 across providers, and Modular has an obvious interest in demonstrating that portability pays. So, like the previous example of bringing accelerators online, treat 45% as an illustration of the shape of the opportunity rather than as a figure to put in a business case. But certainly ask for the methodology before you model against it.

Taken together, the two arguments imply a worthwhile discussion. If enablement is cheap and the economics of alternative silicon are competitive, then the constraint on which chips enterprises can actually use shifts from the chip to the software layer above it. That’s the shift worth watching in this market, and it’s the reason a compiler company is suddenly strategically interesting.

Low-Friction Consumption After FDE-Led Deployment

Adoption friction is the real determining factor in whether any of this reaches an enterprise, so it deserves attention.

The entry path is deliberately low friction. MAX serves an OpenAI-compatible endpoint, which, for many applications, reduces the client-side change to pointing to a different URL. That’s the application layer only. Whether a given model is available, behaves equivalently, and performs acceptably on the target silicon are separate questions, and they’re the ones that actually consume migration time. MAX itself is free to use, installs from Python packages or prebuilt containers, and runs locally on Apple Silicon against the same APIs used in production.

The most underrated demonstration was the agentic model bring-up. Modular showed an agent porting a model from PyTorch to MAX in a single pass, validating it, and serving it, and said several models came up entirely through its agent harness, with the workflows published as reusable skills. If that generalizes, it removes most of the reason an enterprise would ever need to learn Mojo. Mojo becomes a supply-side asset rather than a skills requirement on the buyer.

The enterprise path is different. Every paid tier includes forward-deployed Modular engineers who tune deployments and assist with migration. That’s a genuine benefit for early customers. It’s also a signal that the product isn’t self-service at enterprise scale yet. Normal for an infrastructure company at this stage, and a real constraint on reaching organizations whose deal size won’t support a dedicated engineer.

What’s Open (and What Isn’t) Tells the Story

The openness announcements deserve a closer read than they’ll generally get, because the boundary between what Modular gave away and what it kept is where the business is.

Open, genuinely: the Mojo language, including the compiler and toolchain, under Apache 2.0 with LLVM exceptions. That’s a real change. The standard library and much of the kernel code went open in 2025, but the compiler stayed closed until yesterday. The accelerator library and model architectures are in the same public repository under the same license. Anyone can read it, build it, and fork it.

Not equivalent: MAX. The source sits in the public repository, but usage and distribution are governed by the Modular Community License rather than by Apache 2.0. What changed at ModCon is that the device usage restrictions are being dropped, which is a meaningful loosening. It isn’t, however, the same thing as open source, and the distinction will matter to anyone whose legal team reviews dependencies.

Not open at all: the cluster layer and Modular Cloud. This is where the differentiated capability lives, including distributed KV cache management, disaggregated prefill and decode, topology-aware scheduling, prefix-aware routing, and replica scaling. It’s also where the revenue lives.

Read together, that’s a coherent commercial strategy rather than a contradiction. Give away the language to build a developer base and remove adoption risk. Loosen the framework to make hardware enablement something the ecosystem can do. Keep the layer that turns a working runtime into an operating service, and charge for it. The value has been deliberately migrated up the stack, and Modular Cloud is the destination.

One nuance on the compiler release. Modular has opened the code but isn’t yet accepting outside contributions to it, with that expected later this year. So Mojo is open in the read-and-fork sense today, and not yet open in the co-development and governance sense. Both constrain an owner differently. Forkability provides an exit and limits outright capture. Open governance provides influence over the project’s direction, which is the constraint that bites earlier and more often.

For a CIO, the practical question is which layer you’d be depending on, and the line falls in a specific place. Running models across supported silicon, including the kernels and the serving runtime, is available as a self-hosted option. What sits in the commercial layer is cluster-wide scheduling, scaling, and operational optimization, plus managed delivery. So heterogeneous execution doesn’t require Modular Cloud. But, operating it efficiently at cluster scale, with the placement intelligence that makes the economics work, currently does.

Qualcomm Ownership Requires Proof of Neutrality

Qualcomm announced the acquisition in June and closed it at the end of July. Any silicon vendor acquiring a cross-vendor software layer invites the obvious concern, which is that support quietly tiers in the parent’s favor.

Qualcomm was explicit that it didn’t acquire Modular to build a single-vendor stack, and the structural evidence supports that better than the language does. Open-sourcing the compiler after the acquisition closed, rather than before it, is a credible structural signal. Loosening the MAX device restrictions creates room for outside contributors with independent reasons to maintain parity. Presenting an AMD cost advantage over NVIDIA from a Qualcomm-owned stage three weeks after close is a stronger signal than any commitment statement.

Those protections are real but narrower than they first appear, which is where the licensing boundary matters. Apache 2.0 means Mojo can be forked, so the language is genuinely protected. MAX can’t be forked on the same terms, and the cluster layer isn’t open at all. That’s the layer where placement decisions are made, which is to say, the layer that determines which silicon serves a given request. Open licensing at the language layer doesn’t constrain behavior at the placement layer, and the placement layer is the commercial product.

None of that predicts bad behavior, and I’d expect good stewardship here. It does mean the neutrality question remains open on the merits, and that the burden of proof rises as the platform’s function shifts from executing workloads to deciding where they run.

For a CIO evaluating this, what matters most is whether NVIDIA and AMD silicon remains as well-optimized as Qualcomm’s. That’s where any tilt would actually cost you, and identical release schedules aren’t the test, since hardware cycles differ. Timely support for commercially significant competitor releases, with comparable optimization investment behind it, is.

The difficulty is that parity is hard to verify from within a single stack. Knowing your NVIDIA deployment is under-tuned means knowing what well-tuned looks like on that hardware, which means benchmarking against an alternative runtime on the same silicon. Few enterprises will do that. Which is why two structural indicators are worth watching alongside it, not because they matter more, but because they’re observable and they lead rather than lag: whether the alliance program carries governance with real outside authority, and whether another vendor repeats what the TPU services firm did without needing a special closed-source arrangement.

How Modular Helps CIOs — and what Enterprise Gaps Remain

Modular has made heterogeneous inference materially more practical than it was a year ago. For an organization facing a refresh decision that changes the calculus, because hardware selection stops being a one-way door, and mixed silicon becomes something you can run through a transition rather than around.

What it doesn’t address is everything above the runtime. There’s still no attribution of AI spend across cloud, on-premises, and edge to a model, a team, or a business unit. There’s still no normalized record of what an AI environment contains or how it behaves. There’s still no policy layer that enforces residency and access rather than documenting them. And nothing here manages infrastructure rather than workloads, so thermal throttling, firmware drift, and idle-reserved capacity remain somebody else’s problem.

That’s not a criticism, because Modular hasn’t claimed that ground. It’s an observation that meaningful progress is being made on the harder-looking half of enterprise AI infrastructure while the half that generates executive escalations remains unowned. Who builds that layer, and whether a silicon-owned company can credibly do it, is the more interesting question.

I’ll take it up in a separate field note.