Data Science at Scale: From Notebook to Production with Airflow and Ray

Tuesday, June 22, 8:00PM UTC

Daniel Imberman (PMC Member, Apache Airflow | Software Engineer) & Rob Deeb (Senior Data Analytics Engineer), Astronomer.io

Since the release of Airflow 2.0 in December 2020, we at Astronomer have been working toward the next generation of ML tooling and orchestration. In this talk, we are excited to share multiple integrations with the Ray library that will allow both Airflow and Ray users access to the most powerful features of both of these products. This talk will appeal to Ray users who want to orchestrate and monitor their Ray jobs using Airflow and Airflow users who want to run distributed computation jobs using Ray.

Speakers

Daniel Imberman

PMC Member, Apache Airflow | Software Engineer, Astronomer.io

Daniel Imberman is a PMC on the Apache Airflow project and Strategy Engineer at Astronomer.io. Daniel started his open-source journey by writing the KubernetesExecutor for the Apache Airflow project during his tenure at Bloomberg LP and holds a BS/MS in Computer Science from UC Santa Barbara.

Rob Deeb

Senior Data Analytics Engineer, Astronomer.io

Rob Deeb works at Astronomer as an in-house data engineering and ML specialist. Rob uses his data science background to build internal data pipelines using Apache Airflow, Apache Superset, and other big data tools. He is an advocate for orchestrating data processing and computation using Airflow. Recently he has been a core contributor to the Airflow Provider for Ray.