How does Kubernetes Horizontal Pod Autoscaling Work with Custom Metrics?

A sample workflow for Kubernetes Autoscaling

Kubernetes is a great way to deploy cloud-native applications in the cloud or on-premises. One of the Kubernetes Pod Autoscaling features’ biggest advantages is to automatically scale your application based on demand. This can be extremely helpful when the load an application encounters is variable. Kubernetes has three different types of scaling: Cluster scaling, Vertical […]

How to Know if Your Data Engineering Projects Will be Successful

Data and analytics platform diagram for successful data engineering projects

To make sure data engineering and analytics projects are successful, not only do you need to pick the right technology and have the right people; you also must have the discipline to apply software engineering best practices. What sort of practices am I talking about? Make sure your requirements are clear and communicated to all […]

Trials and Tribulations Preventing Silent Data Loss

Finding silent data loss between a RDBMS and Azure Data Lake

A few weeks ago, a colleague from another project team reached out to me for help with an urgent issue. Errors and irregularities had crept into reports tied to one of their StreamSets pipelines, to the point that the business had complained about the data quality. But when the developers inspected the pipeline, they had […]