DBT Meetup Nov 2024

Nov 21, 2024

On November 21th 2024 I was speaking at the DBT Meetup in New York. The focus was streamlining the integration between dbt and Airflow.

At BMG, enabling Analytics Engineers to schedule their dbt models efficiently has been a key focus. As the Data Platform lead, I’ve worked on reducing dependencies between Analytics and Data Engineering teams while maintaining a centralized approach.

Our first try was to use Astronomer Cosmos, a tool that simplifies rendering dbt DAGs in Airflow. While it eased the development process, challenges like long DAG-bag load times emerged. To overcome this, we transitioned to offline DAG rendering, boosting scalability and performance by decoupling dbt and Airflow dependencies.

Ingestion with dbt & DuckDB

Jul 13, 2024

Streamlined Data Ingestion with dbt and DuckDB #

Efficient file processing is crucial in data engineering. I just did a small experiment that explores integrating dbt (data build tool) with DuckDB (dbt-duckdb), enhanced by an Excel plugin. This combination of tools appears to be a simple framework for local and remote file processing and exposes itself as a powerful framework for ingestion tasks.

Why Combine dbt and DuckDB? #

By leveraging dbt and DuckDB together, we can ensure: