In the latest edition of “Ask an Expert,” MessageGears’ Nick Ziech-Lopez and India Waters sat down with Rachel Bradley-Haas and Alex Dovenmuehle — co-founders of Big Time Data — to explain complex recent architectural advances. Rachel and Alex delve into a number of topics, including how their new company started, how big-time data consulting can be beneficial, and how companies can leverage the data and make it valuable.
Why did you decide to start a company together? What made you want to take that jump?
Alex: When we joined our previous company, we noticed they had a data warehouse but it wasn’t maintained, and nobody was looking at the data. Then, we started talking to colleagues at other companies, and we realized that everyone has this problem. Many companies have large amounts of data all over the place, and they don’t know what to do with it or how to organize it. Data engineers end up chasing down data-quality issues, and they don’t actually get to what you really want to get to, which is actionable valuable data. The more we started talking to people, the more we realized there are so many people who could use what we do and what we can provide.
Describe what a warehouse-first approach looks like
Alex: Companies have data in many different places, telling them all sorts of information. The data can tell them many different things, such as when a customer visits their website and where the customer is coming from. If the company has a mobile application, they can know how many times a day the customer is logging in, and many other examples. Even just pieces of that data can be valuable and really have an impact if you can not only unlock the value of it, but then tell a story by putting all the data together.
With the warehouse-first approach, you obtain all the data from all over the place to have it in one place: the data warehouse. You can then transform the data by commingling all the data from different systems and starting to push it out to other services such as Salesforce or Hubspot. By having all that data in the warehouse first, you can give an accurate state of the world to these third-party tools, where they don’t each have a different story because it’s coming from two different places.
What has changed in the past few years?
Alex: The tools for dealing with large amounts of data have been improving over time, and we’ve hit an inflection point where the majority of people know about data warehouses, and the tools themselves are easy to use. Companies don’t need a team of PhDs to run their data warehouse in order to make it valuable. When using old databases, you will eventually run into the scalability issues where you just can’t handle the amount of data that you really need to be able to. With products such as BigQuery, Snowflake, and Redshift, the whole process has been made a lot easier and user friendly. You can run queries over billions and billions of records, and your performance is extremely improved.
How often do you interface with Chief Data Officer types?
Rachel: It has been mostly underneath a COO role because you have to think about how the business operates as a whole, and data is so central to that.
Alex: Because of the fact that it needs to remain neutral for the betterment of the entire company versus one or the other, it falls underneath the COO role. Another role that we have seen is the Chief Revenue Officer and their perspective, making sure the marketing campaigns are doing well but also how they are influencing the sales pipeline and sales being closed.
Rachel: If you don’t find a neutral place, you end up driving a huge gap between product and revenue. What we are trying to do is be able to prove value to both sets of people.
How do you convey to every part of the company the value in a warehouse-first approach?
Rachel: The biggest thing is connecting the dots showing our biggest paying customers, what they can do with the product, and why you should invest in these areas.
How does warehouse-first design work with streaming individual data sources in real time?
Alex: A big factor is technology maturing and allowing streaming processing to happen in a relatively easy-to-use and scalable format. The ability to take customer data, pair it with streaming data, and then do something useful with it is an important aspect.
What is data streaming?
Rachel: The best analogy for data streaming is a shower: instead of having to fill up a bucket and dump it out, it’s a continuous stream of water. Applying that to events, instead of having to wait around a schedule, it will come over automatically, just like flowing water.
Looking forward in the data marketing and architecture space, what excites you and what will the future look like?
Alex: One of the most interesting things is Materialize, which is essentially a SQL wrapper over streams. It allows you to easily define your DBT (data-built tool) model keeping it up to date all the time, automatically updating all your models from top to bottom in real-time with good performance. With data that’s materialized in real time, you know that it’s up to date down to the second.
Rachel: The thing that really excites me is the fact that only PhDs used to be able to do this, and it was unapproachable. But now everyday people have the power to write SQL. I also love the idea of all these integrations in and out of the warehouse being seamless, being able to template in a lot of these things and enable customization to support them.
