Research Finder
Find by Keyword
How Is AWS Reducing the Cost of Enterprise Log Analytics?
Amazon OpenSearch Service introduces the Optimized Engine to reduce storage requirements, improve query performance, and increase ingestion throughput for observability workloads while preserving full-text search and existing OpenSearch workflows.
07/06/2026
Key Highlights
- AWS introduced the Optimized Engine for Amazon OpenSearch Service, designed for observability and log analytics workloads.
- AWS estimates the Optimized Engine delivers up to 70% lower storage requirements, up to 2x faster analytical queries, and up to 2x higher ingestion throughput.
- The Optimized Engine combines compressed columnar storage and full-text search to support analytical and search workloads within a single service.
- Amazon OpenSearch Service supports more than 100,000 monthly active customers and processes more than 10 trillion requests per month, giving AWS substantial visibility into evolving search, observability, and AI workloads.
- Customers can ingest data through existing REST APIs, Bulk API workflows, client libraries, agents, and pipelines, but the Optimized Engine must be selected when creating a new domain and does not support the DSL query API at launch.
The News
AWS announced the Optimized Engine for Amazon OpenSearch Service, a new architecture for observability and log analytics workloads. The Optimized Engine combines compressed columnar storage, a new query engine, optimized EC2 instances, physical segment replication, and synchronous writes to Amazon S3 to reduce storage requirements, improve query performance, and increase ingestion throughput while preserving full-text search and compatibility with existing OpenSearch applications. For more information, read the What’s New at AWS post.
Analyst Take
AI is changing the scale and character of telemetry. Autonomous agents, APIs, and machine-to-machine interactions generate far more logs, traces, and metrics than traditional user-driven applications. Organizations want to retain more of that data, search individual events, analyze behavior over time, and do so without infrastructure costs increasing at the same pace.
AI does not just increase application activity; it increases the operational exhaust around that activity. Agentic workflows create more API calls, tool invocations, intermediate reasoning steps, retries, handoffs, and machine-to-machine interactions that need to be logged, retained, searched, and analyzed. That makes log analytics a cost-control problem and an AI readiness problem. Enterprises cannot build trustworthy AI operations if the telemetry needed to understand system behavior is too expensive to keep or too slow to query.
Amazon OpenSearch Service addresses that challenge as the AWS managed implementation of the OpenSearch project. The OpenSearch project is distributed under the Apache 2.0 license and governed by the OpenSearch Software Foundation, a member of the Linux Foundation. While both share a common software foundation, Amazon OpenSearch Service is engineered specifically for AWS infrastructure.
The Optimized Engine is designed to support both analytical queries and full-text search within the same service. OpenSearch Service stores data in Apache Parquet format and writes searchable fields to the search index. Apache Calcite parses and optimizes queries, routing analytical operations to Apache DataFusion and search predicates to Lucene. The Query Planner automatically directs each portion of a query to the appropriate storage format and returns a unified result. This allows a single query to combine search and analytical operations without requiring separate round trips, although query-language support is limited to PPL and SQL at launch. The result is an architecture designed around how customers use log data instead of requiring customers to adapt their workflows to the underlying storage engine.
HyperFRAME Research Lens: State of the Infrastructure and Operations (1H 2026) found that only 15% of organizations have achieved a single, unified observability view. Lower storage requirements and faster analytical queries can help organizations retain more telemetry, consolidate observability data, and investigate issues without proportionally increasing infrastructure costs.
Traditional observability architectures often force teams to decide whether data is optimized for incident investigation, dashboarding, long-term retention, or downstream analysis. AWS is trying to collapse more of that work into a single managed service. The tradeoff is that this is still a purpose-built engine, not a universal replacement for the general-purpose OpenSearch engine. Customers should validate query-language requirements, migration effort, workload fit, and DSL dependencies before assuming a like-for-like transition.
AWS carried the same design philosophy into the underlying infrastructure: optimized EC2 instances, physical segment replication, and synchronous writes to Amazon S3 improve ingestion throughput, reduce infrastructure overhead, and strengthen durability. Those engineering decisions help explain why Amazon OpenSearch Service does not align with the open source release cadence. AWS validates and tunes new capabilities against its own infrastructure before introducing them into the managed service.
We observe that this announcement provides insight into how AWS engineers managed infrastructure services. Amazon OpenSearch Service benefits from the scale of AWS, serving more than 100,000 monthly active customers and processing more than 10 trillion requests each month. That real-world experience, combined with customer feedback, allows AWS to identify changing workload characteristics, validate engineering changes in production, and optimize the managed service for AWS environments before making new capabilities generally available.
What Was Announced
AWS introduced the Optimized Engine for Amazon OpenSearch Service, a new deployment option designed for observability and log analytics workloads. Customers select the Optimized Engine when creating a new OpenSearch domain, while the General Engine remains available for search-centric workloads. It cannot be enabled on existing domains or applied selectively to individual indices or fields within a general-purpose domain.
The Optimized Engine introduces a new query engine, compressed columnar storage, optimized EC2 instances, physical segment replication, and synchronous writes to Amazon S3. AWS estimates the architecture delivers up to 70% lower storage requirements, 2x faster analytical query performance, and 2x higher ingestion throughput.
In AWS’s benchmark, the Optimized Engine ingested 24.4 billion documents and 9.5 TB of raw JSON, sustained 1.78 million documents per second versus approximately 647,000 documents per second for the Lucene baseline, and used roughly half the storage of the default Lucene configuration.
The Optimized Engine supports SQL and PPL queries, OpenSearch UI, and existing ingestion pipelines, but does not support the DSL query API at launch. The initial release is available on Amazon OpenSearch Service with OpenSearch 3.5. Amazon OpenSearch Service optimized for log analytics is available across 12 regions globally: US East (N. Virginia, Ohio), US West (Oregon), Canada (Central), Asia Pacific (Mumbai, Singapore, Sydney, Tokyo), and Europe (Frankfurt, Ireland, London, Spain). Technical details are available in the AWS Big Data blog post.
Looking Ahead
The Optimized Engine changes the economics of retaining and analyzing telemetry. Lower storage requirements allow organizations to keep more history immediately available instead of moving data into lower-cost archival tiers where retrieval becomes slower and more expensive. Longer retention expands opportunities for trend analysis, capacity planning, anomaly detection, security investigations, and AI evaluation using the same underlying data.
The announcement also positions Amazon OpenSearch Service as a broader analytics platform. Organizations can search for a specific event, analyze behavior across billions of events, and increasingly use the same telemetry to support AI-assisted troubleshooting, automated operations, and engineering insights.
The market for search, observability, and large-scale analytics is converging around a common objective: retaining more data, extracting insight more quickly, and reducing the infrastructure required to do both. Platforms such as ClickHouse, Databricks, Splunk, and Amazon OpenSearch Service approach the challenge from different starting points, but customers will expect search, analytical queries, AI workflows, and infrastructure telemetry to function as a connected system. Success will be measured by how effectively organizations extract insight from machine-generated data at enterprise scale.
Stephanie Walter | Practice Leader - AI Stack
Stephanie Walter is a results-driven technology executive and analyst in residence with over 20 years leading innovation in Cloud, SaaS, Middleware, Data, and AI. She has guided product life cycles from concept to go-to-market in both senior roles at IBM and fractional executive capacities, blending engineering expertise with business strategy and market insights. From software engineering and architecture to executive product management, Stephanie has driven large-scale transformations, developed technical talent, and solved complex challenges across startup, growth-stage, and enterprise environments.
Don Gentile | Analyst-in-Residence -- Storage & Data Resiliency
Don Gentile brings three decades of experience turning complex enterprise technologies into clear, differentiated narratives that drive competitive relevance and market leadership. He has helped shape iconic infrastructure platforms including IBM z16 and z17 mainframes, HPE ProLiant servers, and HPE GreenLake — guiding strategies that connect technology innovation with customer needs and fast-moving market dynamics.
His current focus spans flash storage, storage area networking, hyperconverged infrastructure (HCI), software-defined storage (SDS), hybrid cloud storage, Ceph/open source, cyber resiliency, and emerging models for integrating AI workloads across storage and compute. By applying deep knowledge of infrastructure technologies with proven skills in positioning, content strategy, and thought leadership, Don helps vendors sharpen their story, differentiate their offerings, and achieve stronger competitive standing across business, media, and technical audiences.