Lightning Talk

Paving the Road for Large-Scale Data Processing in Production with Ray

Ray Summit 2022

In 2021, Amazon launched an experiment to improve the scalability, performance, cost, and operational sustainability of its S3-based data catalog�s critical change data capture (CDC) workloads using Ray on EC2. This resulted in a promising production prototype ("The Flash Compactor") and a new open source Ray ecosystem project ("DeltaCAT") that brought order-of-magnitude improvements to the latency, cost, and scalability of these workloads. However, bridging the gap from petabyte-scale prototype to the exabyte-scale critical path demands a high-availability service that can manage thousands of concurrent job runs and offers near-real-time operational health insights. It also demands a seamless migration of our existing high-availability CDC workloads to Ray without disrupting critical pipelines. In this talk, we�ll discuss our progress in meeting these demands together with the problems encountered, solutions created, critical insights gained, and anticipated future work.

About Patrick

Patrick Ames is a senior software engineer working on data management and optimization for big data technologies at Amazon.

Patrick Ames

Sr. Software Development Engineer, Amazon
Ray Summit 2022 horizontal logo

Ready to Register?

Come connect with the global community of thinkers and disruptors who are building and deploying the next generation of AI and ML applications.

Save your spot

Join the Conversation

Ready to get involved in the Ray community before the conference? Ask a question in the forums. Open a pull request. Or share why you’re excited with the hashtag #RaySummit on Twitter.