Explorer: Data Frames in Elixir with Chris Grainger

https://is1-ssl.mzstatic.com/image/thumb/Podcasts115/v4/d0/11/11/d0111157-4c0b-04e6-b2f4-befc761c7b1a/mza_16239824909782127277.jpg/600x600bb.jpg

Elixir Wizards

SmartLogic LLC

199 episodes

2 months ago

Elixir Wizards is an interview-style podcast for anyone interested in functional programming and the Elixir Programming Language. Hosted by SmartLogic engineers and Elixirists Owen Bickford, Dan Ivovich, and Sundi Myint, this show features in-depth discussions with some of the brightest minds in the industry, discussing training and documentation in Phoenix LiveView, the evolution of programming languages, Erlang VM, and more. In the current season, we're branching out from Elixir to compare notes with thought leaders and software engineers from programming languages like JavaScript, Ruby on Rails, Go, Scala, Java, and more. Each episode will take a deep dive into a topic from Machine Learning and AI, to ECS and game development, to education and community. Learn more about how SmartLogic uses Phoenix and Elixir. (https://smartlogic.io/phoenix-and-elixir?utm_source=podcast)

All content for Elixir Wizards is the property of SmartLogic LLC and is served directly from their servers with no modification, redirects, or rehosting. The podcast is not affiliated with or endorsed by Podjoint in any way.

Technology

Education,

How To

https://assets.fireside.fm/file/fireside-images-2024/podcasts/images/0/03a50f66-dc5e-4da4-ab6e-31895b6d4c9e/episodes/6/6042bbd7-5491-4ee9-b080-8b1c58a270e6/cover.jpg?v=1

Explorer: Data Frames in Elixir with Chris Grainger

Elixir Wizards

42 minutes 55 seconds

3 months ago

Explorer: Data Frames in Elixir with Chris Grainger

In this episode of Elixir Wizards, Charles Suggs sits down with Chris Grainger, co-founder and CTO of Amplified and creator of the Explorer library. Chris explains how Explorer brings the familiar data-frame workflows of R’s dplyr and Python’s pandas into the Elixir world. We explore (pun intended!) how Explorer integrates with Ecto, Nx, and LiveView to build end-to-end data pipelines without leaving the BEAM, and how features like lazy evaluation and distributed frames let you tackle large datasets.

Whether you’re generating reports or driving interactive charts in LiveView, Explorer makes tabular data accessible to every Elixir developer. We wrap up by looking ahead to SQL-style backends, ADBC connectivity, and other features on the Explorer roadmap.

Key topics discussed in this episode:

dplyr- and pandas-inspired data manipulation in Elixir
Polars integration via Rust NIFs for blazing performance
Immutable data frames and BEAM-friendly concurrency
Lazy evaluation to work with arbitrarily large tables
Distributed data-frame support for multi-node processing
Seamless integration with Ecto schemas and queries
Zero-copy interoperability between Explorer and Nx tensors
Apache Arrow and ADBC protocols for cross-language I/O
Exploring SQL-style backends for remote query execution
Building interactive dashboards and charts in LiveView
Consolidating ETL workflows into a single Elixir API
Streaming data pipelines for memory-efficient processing
Tidy data principles and behavior-based API design
Real-world use cases: report generation, patent analysis, and more
Future roadmap: new backends, query optimizations, and community plugins

Links mentioned:

https://hexdocs.pm/explorer/Explorer.html
https://www.amplified.ai/
https://www.r-project.org/
https://vita.had.co.nz/papers/tidy-data.pdf
https://www.tidyverse.org/
https://www.python.org/
https://dplyr.tidyverse.org/
https://go.dev/
https://hexdocs.pm/nx/Nx.html
https://github.com/pola-rs/polars
https://github.com/rusterlium/rustler
https://www.rust-lang.org/
https://www.postgresql.org/
https://hexdocs.pm/ecto/Ecto.html
https://www.elastic.co/elasticsearch
https://arrow.apache.org/
Chris Grainger & Chris McCord Keynote ElixirConf 2024: https://youtu.be/4qoHPh0obv0
https://dbplyr.tidyverse.org/
https://spark.posit.co/
https://hexdocs.pm/pythonx/Pythonx.html
https://hexdocs.pm/vega_lite/VegaLite.html
10 Minutes to Explorer: https://hexdocs.pm/explorer/exploring_explorer.html
https://github.com/elixir-nx/scholar
https://scikit-learn.org/stable/
https://github.com/cigrainger
https://erlef.org/slack-invite/erlef
https://bsky.app/profile/cigrainger.bsky.social
https://github.com/cigrainger