Ray Summit 2022
In 2021, Amazon launched an experiment to improve the scalability, performance, cost, and operational sustainability of its S3-based data catalog�s critical change data capture (CDC) workloads using Ray on EC2. This resulted in a promising production prototype ("The Flash Compactor") and a new open source Ray ecosystem project ("DeltaCAT") that brought order-of-magnitude improvements to the latency, cost, and scalability of these workloads. However, bridging the gap from petabyte-scale prototype to the exabyte-scale critical path demands a high-availability service that can manage thousands of concurrent job runs and offers near-real-time operational health insights. It also demands a seamless migration of our existing high-availability CDC workloads to Ray without disrupting critical pipelines. In this talk, we�ll discuss our progress in meeting these demands together with the problems encountered, solutions created, critical insights gained, and anticipated future work.
Patrick Ames is a senior software engineer working on data management and optimization for big data technologies at Amazon.
Come connect with the global community of thinkers and disruptors who are building and deploying the next generation of AI and ML applications.Save your spot