Pythian Blog: Technical Track

Dismantling data silos through cloud integration

by Ron Kennedy on Feb 23, 2019 12:00:00 AM

This is the fourth in a series of four posts on data integration and its importance for organizations. In this fourth and final post, we look at how to finally solve the data silo problem. Throughout the first three parts of this blog series, we’ve shown the importance of a centralized data management program. Not only does this approach lead to better data governance and more accurate analysis, it also helps bring down unwanted – yet inevitable – data silos within the organization. But achieving true data integration isn’t always easy, especially considering the rapidly escalating scale, pace and different types of data your system is now expected to store and process. Not to mention the pressure of new user and time demands coming from every direction, including the growing need of everyday users to perform fast, self-serve analysis via any number of third-party BI tools. We’ve seen thus far how simply using these BI tools on their own, with no master data management platform or organizational strategy to back them up, can result in a fatally flawed program. We’ve also seen that if your organization uses an on-prem data warehouse or Hadoop as a data platform, it’s likely you’ve also had to purchase additional software or services just to be able to ingest, de-duplicate, transform and unify all the information your users require. After all, the alternative is an organization peppered with data silos, leading to several versions of the truth depending on where the data originated and which department or group. So what’s the answer to the data silo problem? Hint: the answer is the cloud A cloud-native data platform is the best way organizations can cost-effectively and scalably deliver on the promise of data. It’s essentially a cloud data integration platform – a single, unified platform that ensures data quality and unity from as many sources as you can throw at it, with most of the data modifications done automatically at the source, instead of in the data hub. Modifying data at the source is a cloud integration best practice because it ensures consistency of data across systems. The cloud data platform is capable of running intensive analytics on both relational and non-relational data, and can support both traditional use cases and more exploratory or experimental activities performed by developers or data scientists. It typically includes a data warehouse, data lake and an ETL (extract, transform, load) system that provides validation, transformation and reporting along with data ingestion. The cloud data platform: the main advantages There are several advantages and key attributes to a cloud-based data platform, both in terms of ongoing, automated data integration and otherwise:
  • It includes both a data lake and a data warehouse. Data lakes are excellent at storing unstructured, semi-structured and streaming data, but don’t necessarily need the same level of data governance as your data warehouse. A data lake allows you to provide real-time information to developers and data scientists without having to worry about strict governance or warehouse performance slowdowns.
  • It’s modular in design. The cloud can help separate storage from compute, along with separating ETL functions, for greater efficiency in a world of big data demands.
  • It balances vendor-specific cloud services against multiple cloud platforms. Instead of being locked into one cloud vendor, a good data platform can easily be moved from one vendor to another. The best cloud platforms mix cloud services with open-source components, such as Spark for ETL.
  • It’s designed to be easily managed. As data becomes more valuable and more ingrained in the organization, the data platform can take advantage of automation to ensure the highest level of availability, uptime, performance, system updates, security, data scheduling and data integrity.
  • It’s designed to minimize ongoing operational costs. Performing transformations outside the data warehouse helps save on processing costs, while using open-source components helps with licensing costs. And using cloud services such as DBaaS eliminates operations costs typically associated with on-prem solutions.
However, despite the obvious advantages of a cloud data platform , it doesn’t mean your traditional, on-prem data warehouse must immediately be tossed aside. Organizations can, of course, upgrade their data platforms quickly and all in one go using a “rip and replace ” approach, but a more gradual, phased approach is also beneficial; in the short- to medium-term, under a phased approach, your traditional data warehouse can be just one of many data sources feeding your data platform. This also has the short-term benefit that existing ETLs and reporting against governed data can remain unaffected by the platform upgrade. Pythian is a world leader in consulting and developing data platforms, and has helped many organizations upgrade their traditional data warehouses into scalable, affordable and powerful cloud data platforms. Our experts can work with you to determine the approach that works best for you and your organization. Contact us or learn more about how our Enterprise Data Platform, can help you break down data silos and drive value from your data.
Topics: SQL Server Technical Track

No Comments Yet

Let us know what you think

See all

See all

Subscribe by email
Follow us
Love Your Data
Pythian Logo

© 2024 All rights reserved.