A major healthcare revenue cycle technology and services company needed a way to move their data from on-prem to the cloud on a fast timeline — all with limited budget and without interrupting the pipelines that supported their business intelligence dashboards.
They had attempted an MVP version of the migration on their own, and knew they’d need to turn to the experts for the full migration.
Using our team’s expertise with cloud technologies like Azure ADLS and Snowflake, along with our own Streamliner and SQLMorph tools, phData helped the customer lift and shift their entire Hadoop data set to Snowflake within their timeline and budget.
A top healthcare revenue cycle technology & services company focuses on making the business of healthcare run better. By enabling customers to collect more data and enhance visibility into their business, the company helps frontline healthcare providers give more quality care.
In turn, they needed a cloud migration project that would let them move from Hadoop to Snowflake without interrupting their own computing power or their pipelines as they went from on-prem to a cloud data platform. They also needed to run the project on a limited timeline and limited budget.
The customer understood that Hadoop migrations provide many complexities, and they saw value in trusting the experts to get the job done. In a competitive environment for the business, they trusted in phData’s ability to execute due to our deep Hadoop understanding, as well as Snowflake expertise
To move from Hadoop to Snowflake, we had to bring together a handful of technologies, including phData’s Streamliner and SQLMorph tools as well as DistCp, Azure ADLS, SnowSQL and Bash Scripting.
Streamliner and SQLMorph were used to migrate the Hadoop table structures and views to Snowflake. The DistCp Hadoop command was used to upload the data files as-is from HDFS to Azure ADLS. Once in the cloud, we executed Copy-Into statements against external stages using SnowSQL.
Finally, we had to be mindful of when and how many concurrent uploads were performed from Hadoop due to the customer’s bandwidth limitations from moving to the cloud. Our expert team implemented a framework that managed the file uploads, maximizing the utilization of available bandwidth without overloading their network.
In a phrase, phData set up the required technology to both coordinate and automate the migration effort.
We also walked them through the synchronization process so that they could port Talend ETL jobs from Hadoop to Snowflake, running them in tandem until the migration was complete. With this combination, we completed the migration in a few months without interrupting service.
With all their data now in Snowflake, they can begin the process of shutting down their Hadoop cluster and re-allocating resources.
More than that, the synchronization process that phData set up to make the migration meant that the 12 week migration didn’t interrupt Internet bandwidth or data availability for the customer. In summary, phData helped:
Looking into Snowflake or cloud data for your organization? Learn how phData can help solve your most challenging problems.
Subscribe to our newsletter
Data Coach is our premium analytics training program with one-on-one coaching from renowned experts.
