Scaling Geospatial Workflows With Airflow at Overture Maps Foundation and Wherobots with Alex Iannicelli and Daniel Smith

https://is1-ssl.mzstatic.com/image/thumb/Podcasts221/v4/03/b0/38/03b03894-a103-21c1-aa77-fcd8b0d2ff65/mza_4276056948887286043.jpg/600x600bb.jpg

The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

Astronomer

78 episodes

3 days ago

Welcome to The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI— the podcast where we keep you up to date with insights and ideas propelling the Airflow community forward. Join us each week, as we explore the current state, future and potential of Airflow with leading thinkers in the community, and discover how best to leverage this workflow management system to meet the ever-evolving needs of data engineering and AI ecosystems. Podcast Webpage: https://www.astronomer.io/podcast/

Technology

RSS

All content for The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI is the property of Astronomer and is served directly from their servers with no modification, redirects, or rehosting. The podcast is not affiliated with or endorsed by Podjoint in any way.

Technology

https://files.cohostpodcasting.com/cohost/dbcdfaae-58e3-4bce-9cf7-fd6dbc27a8f5/shows/50fd8116-be73-4852-be37-94d8bc0082fa/episodes/ee50b2df-0094-40cb-aa00-08fd510e4f3a/bf2facf316.jpg

Scaling Geospatial Workflows With Airflow at Overture Maps Foundation and Wherobots with Alex Iannicelli and Daniel Smith

The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

24 minutes 3 seconds

1 month ago

Scaling Geospatial Workflows With Airflow at Overture Maps Foundation and Wherobots with Alex Iannicelli and Daniel Smith

Using Airflow to orchestrate geospatial data pipelines unlocks powerful efficiencies for data teams. The combination of scalable processing and visual observability streamlines workflows, reduces costs and improves iteration speed.

In this episode, Alex Iannicelli, Staff Software Engineer at Overture Maps Foundation, and Daniel Smith, Senior Solutions Architect at Wherobots, join us to discuss leveraging Apache Airflow and Apache Sedona to process massive geospatial datasets, build reproducible pipelines and orchestrate complex workflows across platforms.

Key Takeaways:

00:00 Introduction.

03:22 How merging multiple data sources supports comprehensive datasets.

04:20 The value of flexible configurations for running pipelines on different platforms.

06:35 Why orchestration tools are essential for handling continuous data streams.

09:45 The importance of observability for monitoring progress and troubleshooting issues.

11:30 Strategies for processing large, complex datasets efficiently.

13:27 Expanding orchestration beyond core pipelines to automate frequent tasks.

17:02 Advantages of using open-source operators to simplify integration and deployment.

20:32 Desired improvements in orchestration tools for usability and workflow management.

Resources Mentioned:

Alex Iannicelli

https://www.linkedin.com/in/atiannicelli/

Overture Maps Foundation | LinkedIn

https://www.linkedin.com/company/overture-maps-foundation/

Overture Maps Foundation | Website

https://overturemaps.org

Daniel Smith

https://www.linkedin.com/in/daniel-smith-analyst/

Wherobots | LinkedIn

https://www.linkedin.com/company/wherobots

Wherobots | Website

https://www.wherobots.com

Apache Airflow

https://airflow.apache.org/

Apache Sedona

https://sedona.apache.org/

Github repo

https://github.com/wherobots/airflow-providers-wherobots

Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow #MachineLearning