Paving the Road for Large-Scale Data Processing in Production with Ray

In 2021, Amazon launched an experiment to improve the scalability, performance, cost, and operational sustainability of its S3-based data catalog�s critical change data capture (CDC) workloads using Ray on EC2. This resulted in a promising production prototype ("The Flash Compactor") and a new open source Ray ecosystem project ("DeltaCAT") that brought order-of-magnitude improvements to the latency, cost, and scalability of these workloads. However, bridging the gap from petabyte-scale prototype to the exabyte-scale critical path demands a high-availability service that can manage thousands of concurrent job runs and offers near-real-time operational health insights. It also demands a seamless migration of our existing high-availability CDC workloads to Ray without disrupting critical pipelines. In this talk, we�ll discuss our progress in meeting these demands together with the problems encountered, solutions created, critical insights gained, and anticipated future work.