The Fortune 500 financial institution we worked with had millions of customers — and tens of millions of data points flowing into its on-prem data lake.Â
While the legacy infrastructure had kept customer information secure and centralized, employees didn’t have easy access to the customer insights, business reports and broader analytics that they needed, to serve their customers effectively.Â
The company had to run quality assurance and monitoring manually, leading to delays in key reports and higher operational costs for the business.Â
As the client realized they needed a real-time approach to the data they were processing, the need for a cloud-based data platform became clear.Â
The team at phData first joined the client to provide initial onboarding and training when the company was first making the move to Snowflake.Â
With our team’s deep expertise in both Snowflake and data engineering, the client quickly asked phData to take a more hands-on approach with the transition — they had an ambitious timeline and knew they’d need a reliable partner.Â
Through this engagement, the phData team helped the client on three different (but interconnected) fronts:
In the initial engagement, a phData Solution architect established an Information Architecture following industry best practices and standards, making the approach repeatable and sustainable. Following this initial engagement, the client needed a partner to move to the cloud.Â
phData’s team of data engineers worked quickly to put together the right framework to move the client fully into a cloud environment.Â
Beyond moving data to Snowflake, the project team built out all the interconnected tools for a future cloud-based data lake. This is how the framework took shape:Â
*Streamliner is phData’s Data Pipeline Automation tool that simplifies the process of ingesting data onto a new platform. It is not a data ingestion tool in and of itself; rather, it automates other commonly used tools that ingest and manipulate data on platforms like Snowflake
While our team’s expertise built out the framework for moving to the cloud, we didn’t stop short of getting our hands dirty.
Using both team resources and our onboarding accelerator, Tram*, we worked on behalf of the client to complete data profiling, testing, documentation, along with user provisioning and acceptance.Â
*Although the Snowflake platform does support authentication federation, accounts still need to be provisioned within Snowflake, along with databases, schemas, and roles, as well as your information architecture. phData built Tram to take the manual gruntwork out of the equation, and make onboarding users in Snowflake both seamless and painless.
Rather than setting up a data platform and then leaving the client to their own devices, our team recommended and built automation into the framework to set administrators and end users up for success in the long run.
As the phData team got the client up to speed with the new framework, they also wrote up configuration files and templates to streamline the process for moving similar data. The templates will allow the client to create additional pipelines with the same pattern by simply creating a new YAML configuration file. This will significantly reduce the time spent on new pipelines.
The client has seen four key results from this project:
Armed with more accessible data, the financial institution will be able to improve their processes in a way that flows all the way down to the customer. The organization can now identify process inefficiencies more quickly, helping customers get the products they want more easily.Â
Learn how phData can help solve your most challenging data analytics and machine learning problems.
Subscribe to our newsletter
Data Coach is our premium analytics training program with one-on-one coaching from renowned experts.
