The Data Engineering Manifesto presents four valuable principles on how to do Data Engineering well.
Users come first, quality right after that
Sensible tooling over custom solutions
It's tempting to try to fix any problem with an exactamundo solution. Especially for me, since I come from a software development background where anything can be solved with some Python. And glue. With this background, you know exactly what the solution needs to be, and you have your Python, Scala, etc. at hand. The tricky thing is that the problem you face by taking this approach doesn't come early on. With any luck, it'll be smooth sailing for a long time. But if you repeat this again and again, you'll soon be maintaining custom software solutions instead of operating your business. If you’re keen to learn more on how to choose the right technology for your business needs, check out our blog post on data platform selection.
This point is especially important when doing consulting and planning a solution for someone else. The situation is different when you are also the one building, maintaining, and updating the solution. But sensible tooling and make-or-buy decisions are relevant in all setups - especially with the rapid way tooling is developing currently around data. Having the right tools and updating or changing them to address changing needs in the business environment will be the better option in the long run.
Data management over independent data initiatives
Some of the frontrunners even call this part the "Data Supply Chain Management". You know what I mean. When you push your data solution to production, you're not ready yet. You are only about to turn a new page, and on that page, you will need Data Management. In fact, your need for data management should have occurred even earlier, and in some aspects, you are already late.
Data solutions (and therefore businesses) can not survive "in the wild" without Data Management. Before any initiative even starts, Data Management should direct the way initiatives are launched, what data they will include, who has ownership of that data, who are the users, and what reasoning they have for accessing the data etc. Data Management describes the way businesses organize around data and orchestrate the whole data lifecycle. It's a key building block in any data journey. And it's not business as usual if you haven't done it before.
Recordly is here to help you with data engineering
We at Recordly want to do data engineering right. If you want us to join you on your data quest, get in touch!