When Skillz was experiencing rising storage costs, they needed to find a data warehouse solution that would reduce the amount they were spending on storage costs. Google Cloud Platform (GCP) was identified as a possible solution, because of its pricing structure. However, since their transactional data was coming into a MySQL server, they needed to ingest data into the data warehouse in near real time to get a true and timely idea of user activity and performance. The Skillz environment had many layers and included very complex code, along with geographical data that required customization and optimization. Their current Amazon RedShift instance had latency issues and did not provide the performance they needed. In the long term, Skillz wanted to move to an ‘event-driven’ analytics structure. But they couldn’t do this with the high cost and low performance of their current solution. The data collected was critical to driving the player experience for Skillz users. “We use that data to customize offers by region or interest, segment our markets, deliver relevant campaigns, or even to perform fraud and cheater detection,” said Miriam Aguirre, VP Engineering at Skillz. “We were running these applications hourly to customize these different features, then finding ourselves wishing we could do that much more quickly.”

Skillz wanted to demonstrate the benefits of migrating analytics workloads from AWS RedShift to Google BigQuery by implementing a critical workload on BigQuery. The project required batch synchronization from MySQL to BigQuery, as well as migrating RedShift-based ETLs to BigQuery/Dataflow. But the Skillz team was a busy one, with limited time to spend to perform the complex data integration needed to test a new data warehouse platform. So Google recommended Pythian, assuring them that Pythian had the right skills and extensive experience to set up the PoC in a way that engaged the Skillz team without disrupting their core activities. Pythian stepped in and quickly became an extension of the Skillz engineering team, immediately helping to convert and validate existing ETLs and analytical queries, and providing required optimizations for BigQuery. BigQuery is now demonstrating more powerful, reliable, and cost-effective analytics. “With BigQuery we can provide better ETL from the MySQL database, and run faster queries. This will give Skillz a more accurate and up-to-date read of the activity on our platform,” according to Aguirre.

As a result of the implementation, Skillz has been seeing the advantages of near real-time analytics afforded by the BigQuery and the GCP platform. For example, queries generated by player events and interactions based on the Skillz application could be processed in minutes instead of hours. With these faster queries, Skillz can not only customize the player experience but can perform fraud and cheater detection and act on this information immediately. “Some of the queries ended up being 10 times faster on BigQuery than they had been on RedShift. We were obviously very excited to see those numbers come back. It meant we would be able to run our processes every other minute or so. This translates into a much more responsive application—whether that means improved player experience or stopping fraud and cheaters before they do any damage on the system,” Aguirre said.