Inside Vinted’s Code-Generated Airflow Pipelines with Oscar Ligthart and Rodrigo Loredo

https://is1-ssl.mzstatic.com/image/thumb/Podcasts221/v4/03/b0/38/03b03894-a103-21c1-aa77-fcd8b0d2ff65/mza_4276056948887286043.jpg/600x600bb.jpg

The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

Astronomer

78 episodes

3 days ago

Welcome to The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI— the podcast where we keep you up to date with insights and ideas propelling the Airflow community forward. Join us each week, as we explore the current state, future and potential of Airflow with leading thinkers in the community, and discover how best to leverage this workflow management system to meet the ever-evolving needs of data engineering and AI ecosystems. Podcast Webpage: https://www.astronomer.io/podcast/

Technology

RSS

All content for The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI is the property of Astronomer and is served directly from their servers with no modification, redirects, or rehosting. The podcast is not affiliated with or endorsed by Podjoint in any way.

Technology

https://files.cohostpodcasting.com/cohost/dbcdfaae-58e3-4bce-9cf7-fd6dbc27a8f5/shows/50fd8116-be73-4852-be37-94d8bc0082fa/episodes/a755ea53-66f6-46d4-b984-504ecbf3e97d/08f781dde5.jpg

Inside Vinted’s Code-Generated Airflow Pipelines with Oscar Ligthart and Rodrigo Loredo

The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI

29 minutes 36 seconds

2 weeks ago

Inside Vinted’s Code-Generated Airflow Pipelines with Oscar Ligthart and Rodrigo Loredo

The shift from monolithic to decentralized data workflows changes how teams build, connect and scale pipelines.

In this episode, we feature Oscar Ligthart, Lead Data Engineer, and Rodrigo Loredo, Lead Analytics Engineer, both at Vinted, as we unpack their YAML-driven abstraction that generates Airflow DAGs and standardizes cross-team orchestration.

Key Takeaways:

00:00 Introduction.

05:28 Challenges of decentralization.

06:45 YAML-based generator standardizes pipelines and dependencies.

12:28 Declarative assets and sensors align cross-DAG dependencies.

17:29 Task-level callbacks enable auto-recovery and clear ownership.

21:39 Standardized building blocks simplify upgrades and maintenance.

24:52 Platform focus frees domain work.

26:49 Container-only standardization prevents sprawl.

Resources Mentioned:

Oscar Ligthart

https://www.linkedin.com/in/oscar-ligthart/

Rodrigo Loredo

https://www.linkedin.com/in/rodrigo-loredo-410a16134/

Vinted | LinkedIn

https://www.linkedin.com/company/vinted/

Vinted | Website

https://www.vinted.com/?srsltid=AfmBOor87MGR_eLOauCO93V9A-aLDaAhGYx9cnu_oN8s1SAXMlCRuhW7

Apache Airflow

https://airflow.apache.org/

Kubernetes

https://kubernetes.io/

dbt

https://www.getdbt.com/

Google Cloud Vertex AI

https://cloud.google.com/vertex-ai

Airflow Datasets & Assets (concepts)

https://www.astronomer.io/docs/learn/airflow-datasets

Airflow Summit

https://airflowsummit.org/

Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow #MachineLearning