The orchestration of data workflows at scale requires both flexibility and security. At Pattern, decoupling scheduling from orchestration has reshaped how data teams manage large-scale pipelines.
In this episode, we are joined by William Graham, Senior Data Engineer at Pattern, who explains how his team leverages Apache Airflow alongside their open-source tool Heimdall to streamline scheduling, orchestration and access management.
Key Takeaways:
00:00 Introduction.
02:44 Structure of Pattern’s data teams across acquisition, engineering and platform.
04:27 How Airflow became the central scheduler for batch jobs.
08:57 Credential management challenges that led to decoupling scheduling and orchestration.
12:21 Heimdall simplifies multi-application access through a unified interface.
13:15 Standardized operators in Airflow using Heimdall integration.
17:13 Open-source contributions and early adoption of Heimdall within Pattern.
21:01 Community support for Airflow and satisfaction with scheduling flexibility.
Resources Mentioned:
https://www.linkedin.com/in/willgraham2/
Pattern | LinkedIn
https://www.linkedin.com/company/pattern-hq/
Pattern | Website
https://pattern.com
https://airflow.apache.org
https://github.com/Rev4N1/Heimdall
https://netflix.github.io/genie/
Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.
#AI #Automation #Airflow #MachineLearning