necessary snapshots long enough to perform Hadoop cluster backups. Data loss was a certainty if their Hadoop Distributed File System (HDFS) failed or became corrupted. There was also no disaster recovery cluster, so if the existing cluster was lost it would need to be built from scratch – a process that would likely take several weeks if not months. And although the company had previously attempted to fix the problem using Commvault, there were simply too many cluster performance issues for that initiative to succeed.

months. To achieve this while continuing to use their current on-prem cluster, Pythian proposed a new Hadoop cluster be created in another data center and fed from the backups performed on the existing cluster, or through a cluster-to cluster-sync process using CDH enterprise or MapR. Pythian also advised the client, however, that this route would require an expensive capital investment likely to exceed their desired budget. A few other on-prem options were considered -– Cloudera BDR (which requires a Cloudera Enterprise license) and WANdisco Fusion being the first two, but both were similarly deemed too expensive – along with the possibility of using Swift, Ceph, MariaDB/AX or ClickHouse open-source object storage along with the current Hadoop cluster. Pythian also outlined several cloud-native options for solving the client’s BDR dilemma, with the two most compelling and cost-effective being:

• Google Cloud BigTable combined with BigQuery, since the former supports magnetic storage and includes automatic multi-region backups to cloud storage.
• A cloud object storage/column store approach using either AWS Simple Storage Service (S3) or Google Cloud Storage, which would automatically support multi-region redundancy for backup and disaster recovery, while also being the most cost-effective option.

Pythian engagement with the confidence that comes with knowing you’ve looked at all possible options through an expert lens.