Why “Accuracy” Fails for Uplift Models (and What to Use Instead)

When it comes to uplift modeling, traditional performance metrics commonly used for other machine learning tasks may fall short.

The standard machine learning algorithms / business cases learn on the training data, predict the target on the test data and compare it to the ground truth.

However, in uplift modeling, the concept of ground truth becomes elusive since we cannot observe the impact of being treated and not treated on an individual simultaneously.

How to choose validation dataset ?

The choice of data for training and testing an uplift model depends on the available information and the specific context.

Uplift models are commonly used for marketing campaigns. Let’s illustrate how validation data is chosen from this perspective.

If we have a single campaign, we can divide the customers within that campaign into training and validation sets.

However, if there are multiple campaigns available, we can utilize some campaigns for training the model and reserve others for validation. This strategy allows the model to learn from a broader range of scenarios and potentially improves its generalization capabilities.

Without these essential components, accurately capturing uplift becomes challenging.

The main approaches

There are two main ways to assess the performance of an uplift model: Cumulative Gain and Qini. Let’s explore them:

Cumulative Gain :

The cumulative gain illustrates the incremental response rate or outcome achieved by targeting a specific percentage of the population.

To calculate cumulative gain, the individuals are ranked based on their uplift scores, and the sorted list is divided into a series of equal-sized deciles or percentile groups. The cumulative gain is then computed by summing up the outcomes or responses of individuals within each group.

N : number of clients for control (C) and treatment (T) groups for the first p% of the clients

Y : Sum of our uplift in a metric we chose for control (C) and treatment (T) groups for the first p% of the clients

For instance, CG at 20% of population targeted corresponds to the total incremental gain if we treat only the instances with top 20% highest scores.

In the example provided below, we observe that targeting the top 20% of clients with the highest scores yields a cumulative gain of 0.019.

A steeper curve indicates a better model, as it shows that a higher proportion of individuals with the highest predicted uplift are being targeted.

Qini Coefficient:

The Qini coefficient works on the same idea as the Cumulative Gain, with one key distinction.

The formula to calculate it:

That’s great but how we are going to choose between different models ? Relying solely on these curves to choose between different models might not be the most data-driven approach.

The quality metrics

There are three the most useful metrics that can help us and all of them are applicable to both Qini and Cumulative Gain approaches.

Area under Uplift (AUC-U):

Similar to the area under the ROC curve (AUC-ROC) in traditional classification, the AUC-U measures the overall performance of an uplift model. It calculates the area under the uplift / Qini curve, which represents the cumulative uplift along individuals sorted by uplift model predictions.

Uplift@K:

Uplift@K focuses on identifying the top K% of the population with the highest predicted uplift. It measures the proportion of truly responsive individuals within this selected group. A higher uplift@K value indicates a better model at targeting the right individuals.

In the example below Uplift@0.2 for the first model is roughly 0.16 and for the second model is 0.19 , and the choice of the best model is obvious.

When this metric can help ?

Uplift max:

Uplift max refers to the maximum uplift achieved by the model. It represents the difference between the treated and control groups with the highest uplift scores.

Conclusion

We have witnessed that traditional classification and regression metrics may not adequately measure uplift models’ effectiveness.

To overcome this, two primary approaches, CG and Qini, offer valuable metrics for evaluation.

It is crucial to continuously experiment with different variations and find the metrics that align best with your objectives. By exploring and refining your approach, you can effectively measure the impact of uplift models and optimize their performance.

\n

\