Unsticking Ourselves from Glue: Migrating PayIt’s Data Pipelines to Argo Workflows and Hera

https://is1-ssl.mzstatic.com/image/thumb/Podcasts116/v4/56/38/f6/5638f627-2a5e-9de8-c982-3337b01b834e/mza_3964858280071841232.jpg/600x600bb.jpg

Data on Kubernetes Community

243 episodes

2 days ago

The Data on Kubernetes Community (DoKC) is where users go to run data on Kubernetes. We facilitate the creation and sharing of best practices to help users advance in their DoK journey. Here you can enjoy the audio from our livestreams and meetups. Learn more at https://dok.community/

Technology

RSS

All content for Data on Kubernetes Community is the property of Data on Kubernetes Community and is served directly from their servers with no modification, redirects, or rehosting. The podcast is not affiliated with or endorsed by Podjoint in any way.

Technology

https://d3t3ozftmdmh3i.cloudfront.net/production/podcast_uploaded_nologo/7515682/7515682-1679088616225-267a963151836.jpg

Unsticking Ourselves from Glue: Migrating PayIt’s Data Pipelines to Argo Workflows and Hera | DoKC Town Hall

Data on Kubernetes Community

23 minutes 17 seconds

1 year ago

Unsticking Ourselves from Glue: Migrating PayIt’s Data Pipelines to Argo Workflows and Hera | DoKC Town Hall

Unsticking Ourselves from Glue: Migrating PayIt’s Data Pipelines to Argo Workflows and Hera
Presented by Matt Menzenski, Senior Software Engineering Manager, Payitgov

At PayIt, we’ve been deploying applications to Kubernetes almost since the beginning of the company. Our data workloads, however, have run instead in AWS Glue. This has worked well enough for the reporting use cases that have been the main focus of this team historically. However, at the beginning of 2022, the PayIt data team began building out a new data platform, and in the process, ran into a number of challenges with Glue. In this talk, I will share the difficulties that we encountered with building, deploying, and orchestrating ETL pipelines in AWS Glue, our decision process for moving those workloads into Kubernetes, and the ELT architecture that we’ve arrived at today. Related Links DoKC Website - https://dok.community/ DoKC Meetups - https://www.meetup.com/data-on-kubernetes-community/ Join Slack - https://join.slack.com/t/dokcommunity/shared_invite/zt-1vgv7ymz7-YtLFvZicrcLP9fS3o_r2_w