Why You Should Care About Business Metrics in Your Next ML Project

Author(s): Jonte Dancker

Originally published on Towards AI.

Imagine you have worked weeks on a project to develop a new ML model. You spent hours understanding the data, creating features, and testing different models. With all this hard work, you built a model you are proud of. It is the best model you could have developed.

Now, it is time to show your model to stakeholders. You are sure they will be as excited as you about the results. You are well prepared. You have slides showing your model’s accuracy or RMSE and how it beats all other models.

But ten minutes into the meeting, you see confused faces. They seem not convinced that your model is good. But why?

Often, stakeholders cannot relate to and do not care about ML performance metrics. They want to know how much impact the model has on the business. This impact is the true measure of model success.

The model must help convert users into buyers, make a user click on something, or improve a forecast. Hence, usually, we can assign a dollar value to a model or at least a comparable proxy. A business does not care about the model metrics. If a model does not help to make money, it is usually useless.

A technical good model might not have a business impact because it solves the wrong problem.

Hence, data scientists should be deeply concerned about the model’s business impact. We need to report the correct metrics to show our model’s value. If we cannot show our model’s business value, our entire ML project might get killed.

But how can we show that a model adds value to a business? How can we measure its impact?

We need to look at our model’s effect on business metrics. These metrics are tied to company objectives and goals. They provide clear insights into the long-term development of a company. Often used business metrics are revenue, churn rate, or click-through rate.

Business metrics relate to something graspable and thus are easy to understand, explain, and interpret. That’s why stakeholders love them. Hence, we should use them when showing the value of our models.

Business metrics emphasize understanding business needs and priorities. Hence, they show the fundamental notions of success. They can reveal the true value of our model and imperfect predictions.

We can use business metrics to develop a model with business impact. As business metrics show the big picture we can also use them for a broader risk assessment. In contrast, ML metrics do not capture the entire risk.

But why do we not use business metrics for model training?

Usually, business metrics depend on some external feedback, such as user interaction. However, this feedback is often not available during model training. Also, business metrics often do not have the mathematical properties that we need during model training.

That is why we still need our ML performance metrics.

Performance metrics are easier to obtain as they do not need external feedback. We get immediate feedback on how good our model is relative to other models, such as a baseline.

These metrics focus on the technical part of our model. They give us granular insights into model performance and effectiveness. We can use these insights to choose and finetune the best model. For example, we can detect how well the model learns and generalizes on unseen data.

However, we need a deep understanding of the metrics to interpret them correctly. Otherwise, these metrics may mislead our decisions. For example, a model that always predicts the majority class in an imbalanced data set may look good. But if the business cares about the minority class the model is useless.

Hence, performance and business metrics are essential for model development and evaluation. Using both metrics helps us build good models with business impact. We can show how our model translates to business value and thus may see a faster buy-in for our ML initiative.

But how can we combine business and performance metrics?

First, we should get a clear understanding of the business needs. What are the results the business needs? How are these needs measured? We need to answer these questions carefully. If we rush here and move too quickly to the model development phase, we might spend time working on a useless model.

Once we understand the business needs and metrics, we can translate them into performance metrics. Choosing the wrong ML metrics here can lead to a useless model from a business perspective. Hence, choosing out-of-the-box metrics might sometimes not be the best choice.

Choosing the right ML metrics is often difficult. Hence, we should understand our model’s impact on business metrics quickly. Usually, there is only one way to get such feedback. We must put our model into production and expose it to real-world data. The model can be a simple one, such as our baseline model. We can then test our assumptions and map our ML metrics to business metrics early on. With this feedback, we can iterate.

With this, we ensure that we optimize towards business impact and solve the right problem.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI