AWS has solidified its commitment to the Apache Iceberg open table format (OTF), integrating the technology across its storage, machine learning, and analytics portfolios. The move reflects growing demand from AWS’s vast customer base, particularly those using its popular S3 object storage, and positions Iceberg as a pivotal player in the evolving data landscape.

Iceberg, originally developed by Netflix in 2015 and donated to the Apache Software Foundation in 2018, addresses critical challenges in large-scale analytics. Unlike static file formats such as Parquet, Iceberg adds a metadata layer that allows real-time updates and seamless integration with tools like Spark, Flink, and Hive. For AWS, which controls about 23% of the enterprise data storage software market, this adoption underscores Iceberg’s versatility and appeal.

Andy Warfield, AWS’s VP and distinguished engineer, explained that the shift to Iceberg was driven by customer needs. “Our largest analytics customers on S3 have shown a clear preference for Iceberg. We’re actively contributing to its open-source development to ensure it aligns with industry demands and offers flexibility across analytics tools,” Warfield said.

The introduction of S3 Tables, a new storage bucket designed as a managed Iceberg table, exemplifies AWS’s strategy. This innovation allows users to benefit from a pre-partitioned, optimized data structure that promises a tenfold performance boost. AWS also integrates Iceberg with Sagemaker, enabling seamless interaction between structured data and machine learning applications.

While Iceberg gains momentum, competitors like Microsoft’s Delta Lake remain in the picture. Databricks, the creator of Delta, has proposed merging Delta with Iceberg, although progress remains speculative. Despite this, AWS’s adoption of Iceberg suggests it views the format as the future standard for data interoperability.

As the data ecosystem grows more complex, AWS’s Iceberg-centric approach offers businesses a unified path to harnessing diverse tools and technologies, ensuring data accessibility and flexibility remain at the forefront of its services.