case study

Leading U.S. Restaurant Chain Delivers Quality
At-Scale Through Machine Learning on AWS

1.

the customer:

Already doing $10+ billion in annual sales, a top-5 U.S. restaurant chain turned to machine learning to continue delivering exceptional quality and customer experience as they continued to grow.

2.

THE CHALLENGE THEY FACED:

The company needed to get hundreds of models into production using AWS and Sagemaker. They realized that highly manual processes and individually built ML pipelines were ultimately unscalable; they needed a framework to streamline the operations of AWS ML pipelines they were relying on to ensure consistent quality food and services at-scale.

3.

HOW WE HELPED:

phData created an opinionated workflow or “assembly line” for getting ML models trained, deployed,
updated, and secured both efficiently and reliably using cloud-native AWS technologies.

4.

WHAT WE GOT DONE:

The new framework has streamlined the restaurant company’s ability to get ML models into production faster, more efficiently, and with less risk — putting their ambitious goal of deploying 400+ forecasting models throughout 2020 in reach — to help them deliver superior food quality and service at increasingly massive scale.

Full story: Top-5 U.S. Restaurant Chain

A top U.S. restaurant chain, already doing $10+ billion in annual sales, knew that to sustain their aggressive growth trajectory, they needed to continue making good on their core brand promise: delivering exceptional quality and customer service at a competitive price. They weren’t historically a tech-focused company. But to ensure consistency across many hundreds of restaurant locations, they needed to become one.

They launched several machine learning (ML) projects designed to maintain their differentiation across many hundreds of locations, powered by AWS and ML technologies. Among them was a set of many computer vision models that ultimately became key to how they ensure food quality (e.g., what does a “good” sandwich look like versus a “bad” one ), as well as prototype models that will be key to improving order speed and accuracy at drive-thrus as they continue to grow.

However, it wasn’t long before the company understood that machine learning, like the restaurant business, is much more complex to do successfully as you reach an increasingly massive scale.

The ML learning curve

Originally, each ML model was hand-built in Jupyter on a laptop, deployed in a way that requires manual runs, and maintenance was completely hands-on. Because the company was, in effect, constantly reinventing the wheel, it took significant time and developer resources to get each new model into production. And for their computer vision use case — in order to recognize the many different items on their menu — they needed to train and deploy a lot of new models.

ML models were being cobbled together in random notebooks, Python programs and R scripts — without processes or controls to maintain a central repository and version controls. They had no way of tracking the life of a model from training through to production. And there was no central solution for monitoring the performance and quality of ML models over time.

The lack of these best practices contributed to serious inefficiencies and risks:

  • Longer time-to-value — Developers had to dig through random notes and lines of code to piece together what they needed, then manually train each new menu item.
  • Increased costs — In addition to the high costs of developer time, hand-built models yielded higher infrastructure costs due to over-allocation, making it hard to feasibly try out new ideas such as drive-thru speech processing for order automation.
  • Higher security risk — Each hand-built pipeline was one more chance to mistakenly make an S3 bucket public or create an over-permissive role.
  • Technical debt — Models put into production without a monitoring solution or a traceable lineage contributed to significant technical debt and a strong likelihood of solution failure in production.
Ultimately, the restaurant company saw that to deliver quality food and service at-scale, they also needed to deliver technology at-scale. But making machine learning efficient and automated was easier said than done.

Quality on the line

Solving these problems meant building a system of standards, processes, and automated workflows robust enough to get ML models into production and ensure availability — a challenge even for seasoned data scientists.

The restaurant company had been supplementing their team with university students who had talent, but not necessarily experience. And due to the complex set of variables unique to the restaurant industry — for example, the linguistic quirks of patrons ordering at the drive thru — using a hodgepodge of canned, off-the-shelf solutions wasn’t an option. Realizing they needed proven data and ML experts, they ultimately decided to partner with phData.

The phData ML team worked to understand the company’s requirements, then created an opinionated workflow or “assembly line” — based on a set of standard infrastructure including Airflow, AWS Sagemaker, and AWS Batch. The team combined this infrastructure foundation with information architecture, process, automation, and best practices — for getting ML models trained, deployed, updated, and secured.

As a result:

  • New models can be quickly plugged in and deployed using AWS CodeBuild and Jenkins.
  • Infrastructure and security is standardized using AWS CloudFormation, ensuring that every model is built and deployed according to best practices.
  • Version control and code centralization with git save time and reduce error.
  • Visibility exists in AWS SageMaker across the entire flow of turning training data into a production model.
  • A monitoring solution is in place, using AWS CloudWatch, to provide visibility into production model performance and avoid drift.
phData was able to integrate the new ML components and workflow seamlessly with the existing ecosystem of cloud and DevOps tools that the company was already using, like Jenkins and AWS CloudFormation, to manage infrastructure operations.

Scaling technology to scale quality customer experience

The new automated workflow the company created with phData has streamlined their ability to get ML models into production faster, more efficiently, and with less risk:
  • More value, faster — By transforming a highly manual process into an automated assembly line for ML models, with standardized, infrastructure automation and templatized workflows, teams can simply take new training data, drop them into storage, and run the pipeline to retrain and redeploy.
  • Lower costs, better ability to innovate — Standardized infrastructure minimizes overallocation and waste. With a more feasible cost structure, developers who no longer have to build and deploy each model by hand can focus on solving new business problems.
  • Improved reliability and risk mitigation — More automation means less human error. Infrastructure-as-code, templatized workflows, version control, and code centralization all reduce the risk of both cybersecurity and errors that impact availability and performance.
  • Minimized technical debt — Centralized monitoring and alerting, combined with highly visible lineage for each model, drastically reduced technical debt and increased the quality of the solutions in production over time.

Results

The customer now has a foundation not only for their computer vision models, but for all the ML initiatives they’re depending on to scale their brand promise of superior food quality and customer experience. They’re hitting the ground running, working toward an ambitious goal of deploying 400+ forecasting models throughout 2020.

And they’re excited to continue working with phData — harnessing data and machine learning to power a range of innovative use cases: from using speech processing to automate drive thru orders to micro-forecasting demand (accounting for complex factors like weather and traffic) to accurately predict how many baskets of fries should be cooking to improve drive-thru efficiency.

Ready to learn more about phdata? Let's chat.