DBT Meetup Nov 2024
Nov 21, 2024On November 21th 2024 I was speaking at the DBT Meetup in New York. The focus was streamlining the integration between dbt and Airflow.
At BMG, enabling Analytics Engineers to schedule their dbt models efficiently has been a key focus. As the Data Platform lead, I’ve worked on reducing dependencies between Analytics and Data Engineering teams while maintaining a centralized approach.
Our first try was to use Astronomer Cosmos, a tool that simplifies rendering dbt DAGs in Airflow. While it eased the development process, challenges like long DAG-bag load times emerged. To overcome this, we transitioned to offline DAG rendering, boosting scalability and performance by decoupling dbt and Airflow dependencies.
...