Pythian Blog: Technical Track

The Natural Evolution of Data Warehousing—Where We Are Today

by Danil Zburivsky on Dec 4, 2017 12:00:00 AM

It’s a fact that technology is always evolving—rapidly. What’s new and hot today, may be old news and on its way to becoming obsolete tomorrow. Traditional data warehousing is no exception. We have been seeing that the old school data warehouse is on its way out and a new data platform approach is taking its place. We’re also seeing the rise of several trends and have a few thoughts on why they're happening and how we think the industry will respond over the next several years. Why are data warehouses becoming obsolete? There are several factors causing data warehouses to meet their boundaries. A few of them include:
  • An increase in demand to acquire data when it’s needed. Traditional warehouses were designed to consume flat file structures and data from other relational systems. Today, that’s no longer the case.
  • The variety of data sources has increased dramatically and this has pushed data warehouses to the edge of their capacity, in terms of how fast data sources can be acquired. Some of the legacy warehouses can't keep up with this real-time demand.
  • The type of users working with legacy warehouses has changed. Data scientists are one of the biggest and newest groups and they have very specific requirements for the data they use. They need way more processing power than a traditional warehouse may be able to give. They also need the ability to use the tool sets they're most comfortable with, which, in many cases, warehouse databases don't support. One of the top requests we hear from data scientists is they want access to raw data.
  • More automated systems consume data for analytics and existing warehouses are just not well-suited to serve this type of workload.
  • Maintaining traditional data warehouses is a capital expenditure. And maintaining a modern data warehouse on the cloud is an operational spend, which many organizations find an attractive alternative.
What are the consequences of sticking with a traditional data warehouse? All of the reasons listed above, lead to the following negative outcomes for enterprises:
  • Lost business opportunities
  • Unhappy users
  • Performance problems
  • Rising costs
  • Shadow IT
  • Limited visibility into KPIs
What is the alternative to traditional data warehouses? There are new cloud-based warehousing products and services that provide a cost-efficient way to serve and allow analytics of large amounts of data. We've been seeing a trend toward moving to the cloud in part because it takes a lot less effort to set up than it does to set up a data warehouse on-premise. We think these components of automated open-source, new types of optimization tools, and new products from cloud platforms, will demand a new approach to design and implement a new BI platform. What will the design of these platforms look like? The architecture of the modern data platform, is built on several key premises. The modern BI platform should provide both data lake and the warehouse components, meaning that it should be able to store the structure and the structure data in the same place. But it should also be able to provide a place where analytics can run queries on a more structured and curated data set, with very fast response times and the ability to use standard SQL tools or standard BI exploration tools. The right approach to BI in today's world is to stop thinking about a warehouse as being the center of the BI universe, as we thought in the past. In the model we're exploring, the warehouse is just one of the components and becomes a part of the data platform. The role of the data platform is to be able to ingest, process, and serve the data in any shape or size it comes in — to provide a combination of a data lake and warehouse technology. The benefits of having a modern data platform and not just a warehouse, is that the data platform will do all the analytics and processing for you. It will also unlock possibilities, new opportunities, and increase the velocity at which changes can be made to the data pipeline. Another benefit to moving over to a modern data platform is that you can take advantage of the best of both open source technologies and service offerings from cloud platform vendors. At Pythian, we have implemented a number of solutions based on this concept, in the form of our Kick Analytics as a Service. Making sure that a platform is trusted and that businesses are comfortable bringing more and more critical data into a solution like this, can only make an enterprise more successful. Download this full webinar to learn more about the transition from data warehouses to modern data platforms.  
Topics: Linux Site Reliability Engineering Technical Track Open Source Apache Nagios

No Comments Yet

Let us know what you think

See all

See all

Subscribe by email
Follow us
Love Your Data
Pythian Logo

© 2024 All rights reserved.