Platforms offering reporting often have slightly different definitions of what the displayed metrics actually mean. It’s worth taking a deeper dive into each of the displayed as Jacquard defines them to help you best understand the data presented.
Engagement rates
Jacquard offers two different lenses for analysing experiment performance: absolute performance (raw) and comparative performance (fair).
To help understand the difference, let’s consider a standard metric: email open rate.
Raw open rate answers the question: “What actually happened?”
Fair open rate answers the question: “How good is this variant compared to the control, assuming they were sent under identical conditions?”
Let’s dig into Jacquard’s definitions of raw and fair a bit more.
Raw engagement rate
The raw engagement rate is the absolute percentage of engagement. It is the historical record of exactly what occurred during the campaign.
Formula: Raw Rate = Total Engagement / Total Delivered
When to use it: Use this to measure overall audience engagement. Do not use it to compare the performance of individual variants.
Fair engagement rate
The fair engagement rate is a normalised metric calculated relative to the control variant.
In A/B testing, external factors (e.g. time of day, list quality, upcoming holidays, etc.) fluctuate. The fair rate uses the control as a baseline to isolate the quality of the content itself, stripping away the noise to tell you if a variant is genuinely outperforming the baseline.
Formula: Fair Rate = (Variant Uplift) × (Control Performance)
When to use it: Use this when making optimisation decisions, such as deciding which variant is actually performing better for your audience.
Uplift
Uplift is the measure of relative performance. It quantifies exactly how much better or worse a variant performed compared to the control.
Formula: Uplift = (Variant Rate – Control Rate) / Control Rate
Example: If the control open rate is 20% and a variant’s open rate is 24%, the uplift is calculated as (24 – 20) / 20 = +20%.
Jacquard will not calculate uplift on low-volume variants
To minimise noise, uplift is only calculated once a variant has received at least 50 engagements. Remember, engagements refer to the chosen optimisation metric’s events, not delivered messages.
For long-running experiments, simply aggregating all data into one big bucket can hide trends or skew results thanks to days with unusually high traffic.
To solve this, long experiments are divided into reporting periods. The uplift is calculated for each specific reporting period individually, and then the mean (average) of those uplifts is taken. This ensures that a variant performing consistently well over time is recognised as such, rather than being skewed by a single day of massive volume.
Let’s look at an example. In a two-day experiment, we treat each day as a separate data point to find the average performance.
Reporting Period | Control Rate | Variant Rate | Daily Uplift |
|---|---|---|---|
Day 1 | 10% | 12% | +20% |
Day 2 | 20% | 25% | +25% |
Final Uplift | +22.5% (Mean of Day 1 & 2) |
Champion uplift
The champion is the single best-performing content variant for a specific timeframe. It is identified as the variant with the highest average uplift (relative to the control) across all reporting periods in that timeframe.
Champion status is dependent on the time period you are analysing. This is because the performance of a variant can vary over time (e.g. due to language fatigue or seasonal effects). For example, a variant may be the all-time champion by average uplift, while a different variant was the champion for a specific month.
For example, let’s consider an email subject line experiment with monthly reporting windows. The table below shows uplift performance for five variants over a three-month period.
January | February | March | Average Uplift | |
|---|---|---|---|---|
Variant A | 12.0% | 13.0% | 12.0% | 12.3% |
Variant B | 5.0% | 6.0% | 7.0% | 6.0% |
Variant C | -2.0% | -4.0% | -5.0% | -3.7% |
Variant D | -6.0% | -8.0% | -5.0% | -6.3% |
Variant E | 13.0% | 12.0% | 11.0% | 12.0% |
In this example, the all-time champion is Variant A (12.3% average uplift), but the January champion is Variant E (13.0% uplift for that specific month).
Incremental engagements
Incremental engagements is a key metric that estimates the actual commercial value gained from running an experiment. It measures the absolute number of extra engagements (e.g. extra opens, extra clicks, or extra conversions) achieved compared to what the human control would have received if it was sent to the entire audience.
This metric is influenced by two factors:
Language quality - The effectiveness of the better-performing variants.
Optimisation success - The efficiency of the engine in allocating more traffic to those better variants.
Formula: Incremental Engagements = Actual Engagements – (Control Rate × Audience Size)
In the formula we define these terms as:
Actual engagements - The total number of engagements achieved by all variants (including the control) in that period.
Control rate - The engagement rate of the control variant in that specific period.
Audience size - The total number of messages sent to all variants in that period.
For long-running experiments that span multiple reporting periods, the total incremental engagements is the sum of the incremental engagements calculated in each individual reporting period.
Let’s illustrate this with an example. Consider a message that’s deployed daily where two variants are tested against a human control and our results are as follows.
Variant | Messages Sent | Open Rate | Total Opens |
|---|---|---|---|
Control | 100,000 | 10% | 10,000 |
Variant A | 150,000 | 12% | 18,000 |
Variant B | 50,000 | 8% | 4,000 |
Totals | 300,000 | 32,000 |
In this scenario, our actual opens are 32,000. The human control variant has an open rate of 10% (or 0.10). Our total audience is 300,000.
To calculate the incremental opens, Jacquard would do the following maths:
Incremental Opens = 32,000 - (0.10 × 300,000)
Incremental Opens = 32,000 - 30,000
Incremental Opens = 2,000
Therefore, in this reporting period, running this experiment generated 2,000 extra opens for the campaign.
Incremental uplift
Incremental uplift represents the realised percentage increase in engagement for your campaign. The champion uplift shows the theoretical “upper bound” of performance. In other words, it represents the potential gain if the best variant had been sent to everyone. Incremental uplift measures the actual efficiency of the experiment: the percentage increase achieved by your result relative to the baseline.
Formula: Incremental Uplift = Incremental Engagements / Baseline Engagements
Incremental engagements versus incremental uplift
This can be a confusing concept for some users, so we want to explain this very clearly.
Incremental engagements is the estimated absolute number of extra engagements generated (e.g. “We got 2,000 extra opens.”).
Whereas incremental uplift is the percentage improvement over the control's baseline performance (e.g. “We got 6.5% more opens.”).
Because incremental engagements is an absolute number and the incremental uplift is the potential upper bound of performance if the champion had gone to the entire audience, it is possible to have positive incremental uplift but negative incremental engagements. This is due to the opportunity cost of running an experiment.
To avoid this phenomenon as much as possible, following Jacquard’s testing guidelines for Dynamic Optimisation and the recommendations provided by our Data Science team is critical.