Amazon OpenSearch Service grew up as a search engine, and for years it handled logs the way it handles everything else, as documents to index and find. Its new log analytics engine walks that back.
Most log work at scale is analytics. You’re counting errors, rolling up latency, grouping by service and host over long stretches of time. Some of it really is search. That’s the 2 AM hunt through raw messages for the one line that explains an outage. That work is real, and it’s why the engine keeps a search index at all.
But AWS is betting the analytics-heavy majority should set the default, with search as the part you opt into. Running everything on a search-first foundation is a big reason logs got so expensive. That cost isn’t unique to OpenSearch, though. Anyone running logs at scale hits the same wall. The whole category has been shifting to columnar storage on cheap object stores to get out from under it. AWS is making that move on its own service, and I like the bet.
Analytics Leads the New Engine, and Search Rides Along
You can see the search lineage on the bill. To keep any record instantly findable, the old design builds an inverted index across your fields. It keeps a second raw copy of every document, so it can hand the original back the moment you query it. That pays off when you need one record in a hurry, but logs rarely work that way. You wind up indexing and duplicating huge volumes you scan in bulk and rarely look up one line at a time, and you pay to keep all of it.
The new engine flips the priority. It stores everything in Apache Parquet, the columnar format the analytics world already runs on, which keeps each field in its own column, so a query that counts errors reads just that column instead of every full log line. Analytical queries run on an engine built for exactly that kind of scan.
The inverted index sticks around, but only on the fields you actually search, so you stop paying for it on every field of every log line. The engine rebuilds the document from the Parquet columns on read, which clears out the second raw copy entirely. Drop the index you weren’t using and the duplicate you didn’t need, and the storage bill falls. The analytical queries that carry most of the work get faster too.
Search still matters here, but it now rides atop an analytics engine as a feature. The old design put it at the foundation and built the rest on top.
Parquet Outlasts the Price Cut
Choosing Parquet does more than shrink the bill. It’s an open columnar format, the same one the lakehouse world stores its data in. So these logs no longer sit in a shape only OpenSearch can read. That puts this inside a bigger shift: operational data is leaving the closed observability stores. They take it in and rent it back through their own query path. Now it’s landing in the open formats the rest of the analytics stack already speaks.
Be clear about today, though. AWS hasn’t opened that access, the Parquet lives inside the managed service for now, and nothing is de-siloed yet. But the format is the foundation, and AWS has laid it. Opening that data to its other engines is the natural next move, and I’d bet they take it. Even if they never do, the cheaper engine earns its keep on cost alone. For my money, the open format is the bigger prize over time. It puts your logs in a shape you’re not locked into, sitting next to the analytics and AI services AWS wants them near. And to a buyer, an open format counts for more than any one tool’s feature list. It means the rest of your stack can pull from those logs directly, instead of routing every query through a single product.
AWS Answers the Cost Problem on Its Own Turf
Cost has been the pressure point in log analytics for years, and AWS addressed it by building the new engine into the managed service its customers already use. Customers now get the economics of the cheaper option without leaving what they know. That plays to a strength. AWS keeps taking cost out of its own data stack, designing its own chips and setting its own prices so that running data-heavy work at AI scale gets cheaper on its cloud, and a cheaper, more open log engine carries that into observability.
What it doesn’t touch is the workload itself. Log volume only grows, and faster now that fleets of agents and models are generating telemetry of their own. Retention windows keep getting longer, too. No engine makes any of that free. The new engine cuts the costs associated with indexing and copying everything the old way. The volume you generate and the window you keep it in are still yours to manage.
A couple of things would help it land wider. Today, the engine is a choice you make on a new domain, so getting an existing fleet onto it means standing one up and migrating, and I’d like to see that path get easier. The query language on which many existing dashboards were built isn’t supported yet either. That follows from the analytics-first bet, since the search-side tooling is the part still to come. Neither one dulls the move. Building a new columnar engine for a service this widely used, while leaving ingestion and the core APIs untouched, is the kind of unglamorous engineering that teams actually adopt. The open format underneath is what gives AWS somewhere to take it next.
