215-plus terabyte Apache HBase database, processed with the Apache Impala SQL query engine, and then fed into a proprietary neural network (advanced machine learning) application with the goal of continually refining and improving the system’s understanding and ability to respond to these interactions. However, because their HBase database was so large, it didn’t have enough capacity to support the necessary snapshots long enough to perform backups. This meant their disaster recovery capabilities were almost nonexistent, with the company facing a recovery process of several weeks if not months (along with possible significant data loss) should a disaster occur. Because it was all being run on a version of Cloudera CDH that is no longer supported, the company was facing a bill of up to $233,000 per year in license fees and other costs to upgrade to a supported release. With the organization also needing to dramatically beef up its backup and disaster recovery capabilities, they were looking at costs of closer to half a million dollars per year to continue effectively using their on- prem Hadoop cluster.

apparent that their on-premises options were limited to the expensive upgrade and licensing fee structure mentioned above, or going with an upgraded DIY Hadoop cluster that was likely to be even more expensive after staff, hardware and support costs were factored. Three main cloud options were discussed during a two-day on-site workshop with company stakeholders and Pythian consultants: Hadoop in the cloud as a service, cloud-native options that most closely fit their current HBase column store, and cloud-native options centered around an object store and column store. Public cloud services from Google Cloud Platform (GCP) and Amazon Web Services (AWS) were considered, with each having the option of either significantly minimizing or completely replacing their current Hadoop cluster:

• • AWS DynamoDB combined with Redshift was one option to replace their HBase use cases and Impala SQL query engine. This approach, while effective, was likely to be less cost-effective than other approaches. 

 

• • GCP BigTable combined with BigQuery was also considered, with BigTable performing the same function as DynamoDB and BigQuery doing the querying. Because BigTable uses the same API as HBase, this was the best fit for replicating their HBase use cases with the least disruption, and entailed costs estimated at the high end of the client’s budget. 

 

• • A completely new cloud object storage and column store option was also presented (using either AWS Simple Storage Service (S3) or Google Cloud Storage), in conjunction with either the Athena S3 Query engine or BigQuery which would automatically support multi- region redundancy for backup and disaster recovery, while being the least expensive cloud option. 

 

• • An onsite option was also discussed using onsite object and column stores like the Ceph object store and Click House column store. 

 

company’s engagement with Pythian consultants, they were able to come away with cutting-edge insights and hard data on all their on-premises and cloud options from multiple vendors, allowing them to make the most informed decision possible to upgrade their system, implement 12-hours-or-less backup and disaster recovery, and drive significant cost savings.