Established Financial Company Improves Latency by 90% from Migrating its ETL Processing to Snowflake
Customer's Challenge
An established Fortune 500 financial company had recently migrated to Snowflake from Teradata but was experiencing some concerning latency issues from its existing on prem IBM DataStage environment.
phData's Solution
In just 12 weeks, phData’s data engineering team helped the client improve latency by 90 percent across the board by migrating their ETL processing from on-prem to Snowflake. Additionally, phData provided repeatable code for the client to leverage for future tables that needed implementation.
Results
phData reduced latency by 90 percent across the board. We provided the client with repeatable code and stored procedures that can be used for any future tables they need to implement.
The Full Story
Our story begins with a leading Fortune 500 financial company that had just migrated to the Snowflake Data Cloud from its outdated legacy Teradata platform. The migration work was complete, but the client was still using an on premise IBM DataStage solution to move data into Snowflake. Having all of its ETL done in IBM DataStage caused several latency issues that delayed customer report delivery.
The client was well-known in its industry for helping millions of Americans find confidence in retirement through its celebrated investment and financial services. To continue providing excellent services to its customers, the client needed to improve latency.Â
Initially, the client hoped to reduce latency by 50 percent; they just needed a partner to help make it happen.Â
Why phData?
After presenting a robust project scope and confidently answering all the client’s questions, they were convinced that phData was the right partner. Additionally, phData’s track record of completing hundreds of successful migrations to Snowflake put the client at ease.
Solving Latency Challenges​
Once the choice was made to hire phData to solve the solution, our data engineering team got to work. In a previous scoping call, we determined that a data pod approach was the best route to success.
Next, phData quickly identified that pulling data from Snowflake to IBM DataStage caused latency. The team then did a proof of concept to demonstrate the increased speed the client would get by running SQL in Snowflake vs. using DataStage. The client acknowledged the performance gains and asked phData to migrate their code, at which we proceeded to plan an approach to complete their request.
Building a Framework to Eliminate IBM DataStage
phData provided the client with a technical framework and phased plan to migrate out of DataStage for good. With renewed confidence and massive improvements in latency, the client can now move forward in its modernization to Snowflake.
Take the next step
with phData.
Learn how phData can help your organization harness the full potential of your data with our data engineering & analytics services.
