LITMIR.BIZ

Introduction

      Knowledge is power. Knowing more about your surroundings, like which fruit is safe to eat and which will harm you, keeps us safe and successful. Knowing more about the relationships between events, like the inverse relationship between washing hands and influenza outbreaks, minimizes inevitable difficulties. And knowing which strategies lead to success, like which mass‐reforestation strategies lead to sustainable ecosystems, helps us optimize our efforts. In the modern world information on our environment is often in the form of data.

      Whether by an individual or a team, data used intentionally and critically is a key part of success. While it's true that anyone can get lucky and stumble upon the right answer, having an idea of which tactics confer the most stability or which metrics indicate big opportunities for growth can drive teams forward quicker.

      Our thesis is this: being informed implies being successful. But what does it mean for a person or an organization to be informed exactly? If you are sufficiently informed, you can make decisions with a sense of conviction that your thinking is correct a majority of the time. In the end, your gut feeling (intuition) factors into decision making. However, with well‐organized and accessible information added into the mix, your “gut” can make more thoughtful (and better) decisions.

On the other hand, bad decisions are the result of not knowing enough: not knowing the root causes of numeric anomalies, not knowing where there is potential to invest more capital, and not knowing what the next few months could look like based on previous quarterly performance. Intuition is a powerful and necessary tool, but we believe that the individual and the company owe it to themselves to hone their intuitions. We believe that anyone with access to performance metrics should integrate that information into their decision making. Being informed is a key part of the puzzle to avoiding bad decisions. But what does it look like to be informed in the twenty‐first century?

      Companies and their operations continue to become more digitized. Companies no longer have direct access to each customer and must increasingly rely on data to improve and compete. So, organizations end up tracking a lot of data taken from many input streams. But why are so many teams struggling with organizing and leveraging insights from their data? We believe that the problem begins with tooling. We believe that the modern data workflow requires a data stack and operational structure that is ready for today's challenges.

      In this book, we will outline what the modern data stack looks like starting from a brand‐new startup all the way through a data‐driven enterprise. We will cover architectures, tools, team organizations, common pitfalls, and best practices. Armed with these, we are confident that your team can identify the evolving needs of your business and implement solutions in your data infrastructure.

Merging Business Context with Data Information

There is a chasm between the people who know the technical aspects of how to work with data and the people who know the context behind that data. The technical people can write structured query language (SQL), whereas the business people know which marketing campaigns they've run and what types of users comprise an analysis. This creates friction between the stakeholders who want access to data insights (without always knowing what that data looks like) and the analysts who know their data but not all of its history or significance.