On November 21th 2024 I was speaking at the DBT Meetup in New York. The focus was streamlining the integration between dbt and Airflow.

At BMG, enabling Analytics Engineers to schedule their dbt models efficiently has been a key focus. As the Data Platform lead, I’ve worked on reducing dependencies between Analytics and Data Engineering teams while maintaining a centralized approach.

Our first try was to use Astronomer Cosmos, a tool that simplifies rendering dbt DAGs in Airflow. While it eased the development process, challenges like long DAG-bag load times emerged. To overcome this, we transitioned to offline DAG rendering, boosting scalability and performance by decoupling dbt and Airflow dependencies.

Analytics Engineers can now independently schedule models, streamline testing with service account impersonation on GCP, and contribute to a faster deployment cycle without compromising security. This integration has allowed BMG to continue using Airflow as the sole orchestrator for both dbt and non-dbt workloads, promoting seamless collaboration across teams.

If want to dive deeper into this workflow? Let’s connect! I have not yet had the chance to open source the code but I am willing to do so if there is interest.