DBT Meetup Nov 2024

Nov 21, 2024

On November 21th 2024 I was speaking at the DBT Meetup in New York. The focus was streamlining the integration between dbt and Airflow.

At BMG, enabling Analytics Engineers to schedule their dbt models efficiently has been a key focus. As the Data Platform lead, I’ve worked on reducing dependencies between Analytics and Data Engineering teams while maintaining a centralized approach.

Our first try was to use Astronomer Cosmos, a tool that simplifies rendering dbt DAGs in Airflow. While it eased the development process, challenges like long DAG-bag load times emerged. To overcome this, we transitioned to offline DAG rendering, boosting scalability and performance by decoupling dbt and Airflow dependencies.

Running Airflow

Jun 21, 2023

I had the honor to contribute to the Astonomer Blog on medium with an article on Running a Multi-Tenant Airflow Cluster

The post explores how we leverage Apache Airflow in a multi-tenant setup to orchestrate diverse workloads like royalty processing, financial analytics, and marketing insights. By utilizing Google Cloud Composer, we balances cost, stability, and operational overhead across multiple teams, ensuring seamless data workflows. Key practices include workload isolation, use of short-lived credentials, atomic tasks, and cost allocation through Kubernetes namespaces. The post also covers my thoughts on continuous deployment, DAG testing, and challenges around for maintaining and upgrading Airflow environments.