Machine learning is a term almost everyone in the IT space has heard by now—but it’s not just a buzzword used in flashy presentations anymore. As machine learning has started to become more applied and less theoretical, the industry has begun to incorporate it into important projects.
In 2022, we’re past the point of proving its value. The concern now is how to properly implement a successful machine learning project and confidently push it to production.
Enter MLOps. This term may sound familiar to those who have worked on traditional software projects because it is closely tied to DevOps, the “parent” of MLOps. Let’s take a closer look at these terms and their relationship to understand how we can create a standardized workflow that engineers and data scientists can iterate on for machine learning projects.
What are DevOps and MLOps?
DevOps is a set of practices that aims to shorten a system’s development life cycle and provide continuous delivery with high software quality. Comparatively, MLOps is the process of automating and productionalizing machine learning applications and workflows. Both DevOps and MLOps aim to place a piece of software in a repeatable and fault tolerant workflow, but in MLOps that software also has a machine learning component.
You can think of MLOps as a specialized subset of DevOps for machine learning applications and projects.
What is an Ideal DevOps Cycle?
DevOps is a crucial concept in almost all successful IT projects as teams aim for a shorter code-build-deploy loop. This enables teams the freedom to deploy new features faster and therefore finish projects faster with a better end product. Without the proper DevOps practices, however, teams suffer from manual tasks, inability to test, and ultimately risky production deployments.
For a successful DevOps project, an ideal DevOps cycle will be comprised of the following five key pillars:
- Reduce organizational silos: Share ownership and accountability
- Accept failure as normal: Embrace risk and iterative development
- Implement gradual changes: Move quickly with smaller iterations to reduce the cost of failure
- Leverage tooling and automation: Use tools to automate manual tasks
- Measure everything: Define what “success” is and how it will be measured
One of the most common DevOps cycles (that includes all five of these pillars) looks like this:
Integrating development and operations reduces silos. This comes in the form of code reviews being done between teams to enable collaboration. The cyclic nature of this integration exhibits the iterative method used to develop, build, and deploy in small batches to continuously validate and fix errors. This is ideally done in an automated manner using tools like Jenkins and Git integrations. Based on predetermined definitions of success, teams then monitor the project to ensure it is meeting necessary metrics.
Using machine learning is very experimental and follows similar concepts as the ones mentioned above. Conveniently, the traits of DevOps tie seamlessly into machine learning applications. With a subset relationship in mind, let’s compare a DevOps pipeline with a MLOps pipeline.
Deep Dive Comparison of DevOps and MLOps
Cycle
Both DevOps and MLOps pipelines include a code-validate-deploy loop. But the MLOps pipeline also incorporates additional data and model steps that are required to build/train a machine learning model (see diagram below). This means MLOps ultimately has a few nuances for each component of the workflow that differ from traditional DevOps.
Although “data” and “model” are vague terms, in most cases they represent the data labeling, data transformation/feature engineering, and algorithm selection process.
Algorithms in most industry machine learning projects today are supervised. This means they have a target (or label) to learn from in the model training process. Data labeling is the process of adding the target to a chunk of data records and the model will use this as a training set.
Data transformation/feature engineering is needed because models need data to be in a certain structure in order to produce meaningful results. Selecting an algorithm depends on the nature of the prediction problem at hand.
Overall, this follows the Cross-Industry Standard Process for Data Mining (CRISP-DM) process model, which has become the most common methodology for data science projects in 2022.
The “Dev” and “Ops” parts are mostly the same at a high level. We’ll discuss low-level differences in the next few sections.
Development and CI/CD
“Development” takes on two different meanings in each concept.
On the traditional DevOps side, you’ll typically have code that creates an application or interface of some sort. The code is then wrapped up in an executable (artifact) that is deployed and validated against a series of tests. This cycle is ideally automated and continues until you have a final product.
With MLOps, in contrast, the code is building/training a machine learning model. The output artifact here is a serialized file that can have data fed into it and produce inferences. The validation would be checking how well the trained model does against test data. Similarly, this is a cycle that continues until the model performs at a certain threshold.
Version Control
Version control in a DevOps pipeline typically involves tracking changes on the code and artifacts only. In an MLOps pipeline, there are more things to track.
Model building/training involves an iterative cycle of experimenting as mentioned before. The components and metrics of each experimental run must be tracked in order to properly re-create it down the line for auditing purposes. These components include the data set used in training (train/test split), the model building code, and the model artifact. The metrics include the hyper-parameters and the model performance (e.g., error rate).
Compared with traditional software applications, this may seem like a lot of details to track. Fortunately, we have model registry tools as a tailor-made solution for versioning ML models.
Monitoring
On top of monitoring the application itself, an additional component to monitor in MLOps is model drift. Data is constantly changing and therefore your model needs to as well. Models trained on older data will not necessarily perform well on future data, especially if the data has seasonality.
In order to keep your model up to date, it will need to be re-trained regularly to gain consistent value from it.
Roles of Team Members
Roles and responsibilities differ slightly between traditional DevOps and MLOps.
In DevOps, software engineers are the ones developing the code itself while DevOps engineers are focused on deployment and creating a CI/CD pipeline. In MLOps, data scientists play the role of the application developers as they write the code to build the models. MLOps engineers (or machine learning engineers) are responsible for the deployment and monitoring of these models in production.
Key Differences Between DevOps and MLOps
Code | Artifact | Validation | Roles | |
|---|---|---|---|---|
DevOps |
| Executable JAR | Unit testing |
|
MLOps |
| Serialized file | Model performance (error rate) |
|
MLOps is not some revolutionary idea. MLOps is essentially a specific implementation of DevOps; it is DevOps for machine learning projects and pipelines. If you know DevOps, you’ll pick up the concepts of MLOps just fine if you keep in mind the specifics mentioned here.
Looking for a deeper understanding of MLOps? We literally wrote the book, or rather the eBook, on the subject and it’s available for free!
Read The Ultimate MLOps Guide: How to Deploy ML Models to Production
Frequently Asked Questions
Dataiku is an end-to-end machine learning platform that manages data preparation, machine learning, and operations steps in your pipeline in an easy-to-use interface. Sagemaker is an AWS service that serves a similar purpose.
MLflow and Kubeflow comes close with the ability to track parameters, code, metrics, and artifacts in one platform.
In most MLOps architectures we’ve seen, each component will require different tooling. For example, DVC is a popular tool for data versioning and Airflow is a good tool for orchestrating the workflow.
If the company doesn’t already have solid DevOps practices, it will have a harder time executing a proper MLOps pipeline successfully without discipline.
A good DevOps culture may need to come first and this requires an effort from the company to accept these as necessary for technical success. Although the transition might be hard at first, implementing a good DevOps pipeline will save the company time and money in the long run.
