Dynamic Optimisation allows marketers to test up to 10 variants in an experiment. But this does often bring up questions.

This article is for those who've themselves wondering:

We've got information below that will clarify these questions regarding minimum testing requirements for our Dynamic Optimisation technology.

The answers to most questions regarding Dynamic Optimisation testing criteria hinge on what we refer to as optimisation events. These are simply occurrences of your chosen optimisation metric (i.e. the number opens or clicks).

For Dynamic Optimisation to work properly, there must be a certain number of optimisation events that occur within a certain time period. These differ for trigger and broadcast experiments.

Let's start with triggers. For triggers to optimise properly, Jacquard suggests a minimum of 200 optimisation events per variant per day on average.

Looking at the table below, you can see how many optimisation events are suggested per day to optimise triggers in a meaningful, statistically significant way:

If you don't have enough optimisation events on average to allow for at least five variants, Jacquard does not recommend testing on that particular campaign.

For broadcast experiments, things are a bit different. Jacquard suggests a minimum of 200,000 delivered recipients for five variants. Jacquard also recommends a minimum of four hours of optimisation time for a broadcast experiment optimising on opens.

So, when optimising on opens over four hours:

For broadcast experiments where you wish to optimise on clicks, Jacquard suggests a minimum of 2,000,000 delivered recipients for five variants. Jacquard also recommends a minimum of six hours of optimisation time for a broadcast experiment optimising on clicks.

So, when optimising on clicks over six hours:

Unless otherwise specified, we do not generally recommend testing fewer than five variants with Dynamic Optimisation.

If you don't have a large enough list to allow for at least five variants, Jacquard does not recommend testing with Dynamic Optimisation on that particular campaign type.

We certainly won't prevent you from testing as many variants as you'd like in a particular experiment, or from choosing clicks instead of opens.

However, you do risk the following outcomes if you choosing to experiment outside these minimum guidelines.

You may see a particular variant chosen as a winner when if given more data or time to mature, would not actually have been the winner overall.

Generally, we don't see a full reversal, where the declared winner ends up at the bottom of the pack. But it certainly still amounts to increased opportunity cost and leaving lots of potential uplift on the table.

When Dynamic Optimisation does not have enough data to make a proper statistically significant conclusion, it will still attempt to optimise based on the data is has received.

However, with so little data returned for each variant or not enough time to make a mature decision, the outcome is essentially a statistical guess: The equivalent of essentially picking one at random to send to more of you audience.

You may still see uplift reported when this happens. But that uplift was essentially happenstance. More often, you'll see negative uplift, which means the opportunity cost of conducting the test outweighed the benefit.

Last reviewed: 18 June 2024