In this podcast episode, we talked with Adrian Brudaru about the past, present and future of data engineering.
About the speaker:
Adrian Brudaru studied economics in Romania but soon got bored with how creative the industry was, and chose to go instead for the more factual side. He ended up in Berlin at the age of 25 and started a role as a business analyst. At the age of 30, he had enough of startups and decided to join a corporation, but quickly found out that it did not provide the challenge he wanted.
As going back to startups was not a desirable option either, he decided to postpone his decision by taking freelance work and has never looked back since. Five years later, he co-founded a company in the data space to try new things. This company is also looking to release open source tools to help democratize data engineering.
0:00 Introduction to DataTalks.Club
1:05 Discussing trends in data engineering with Adrian
2:03 Adrian's background and journey into data engineering
5:04 Growth and updates on Adrian's company, DLT Hub
9:05 Challenges and specialization in data engineering today
13:00 Opportunities for data engineers entering the field
15:00 The "Modern Data Stack" and its evolution
17:25 Emerging trends: AI integration and Iceberg technology
27:40 DuckDB and the emergence of portable, cost-effective data stacks
32:14 The rise and impact of dbt in data engineering
34:08 Alternatives to dbt: SQLMesh and others
35:25 Workflow orchestration tools: Airflow, Dagster, Prefect, and GitHub Actions
37:20 Audience questions: Career focus in data roles and AI engineering overlaps
The role of semantics in data and AI workflows
41:11 Focusing on learning concepts over tools when entering the field
45:15 Transitioning from backend to data engineering: challenges and opportunities
47:48 Current state of the data engineering job market in Europe and beyond
49:05 Introduction to Apache Iceberg, Delta, and Hudi file formats
50:40 Suitability of these formats for batch and streaming workloads
52:29 Tools for streaming: Kafka, SQS, and related trends
58:07 Building AI agents and enabling intelligent data applications
59:09Closing discussion on the place of tools like DBT in the ecosystem
🔗 CONNECT WITH ADRIAN BRUDARU
Linkedin - / data-team Website - https://adrian.brudaru.com/ 🔗 CONNECT WITH DataTalksClub
Join the community - https://datatalks.club/slack.html Subscribe to our Google calendar to have all our events in your calendar - https://calendar.google.com/calendar/... Check other upcoming events - https://lu.ma/dtc-events LinkedIn - /datatalks-club Twitter - /datatalksclub Website - https://datatalks.club/