Strategic Foundations #

A key observation I made for building a successful data engineering strategy is to view data engineering not as a collection of tools, but as a set of well-defined practices. Tools serve as enablers, but it is the underlying practices that drive long-term success. The focus should be on selecting tools that comprehensively support the core practices, offering a superset of features necessary to execute them effectively. When tools fall short, best practices can become cumbersome, leading to inefficiencies and friction in development workflows. For example, without a robust data testing framework, tests become very cumbersome and slow down development. The same holds true for documentation. If documentation is not written in a well defined framework, if becomes almost impossible to maintain.

Building a strong governance framework naturally follows from the idea that data engineering is about well-defined practices rather than just tools. While the right tools enable these practices, a governance framework ensures that they’re applied consistently and effectively across your organization.

You can find a few more thoughts on documentation under in my notes on Software Engineering.

I collected the most important activities for a Governance Framework for Data & BI Operations here.