An A/B test settles the argument about which banner is better with figures in hand: it shows which change genuinely moves CTR, conversion and cost per enquiry, and which only seems to work. Let’s look at how to plan such an experiment for banners and video, avoid setup mistakes and read the result correctly.
Where to start: testing priorities
Start with the elements that change people’s behaviour the most:
- The offer and benefit — price, timing, guarantee, “we’ll deliver today”.
- The opening of the video or the first carousel card — what holds attention in the first moments.
- The visual approach — background, contrast, the hero of the shot, the angle, the button style.
- Video length — a short “hook” versus a longer version.
- The call-to-action text — “Place an order” versus “Get a consultation”.
- Format — vertical or square, a concise caption or a detailed one.
The main rule: one meaningful variable changes per test. Change the image, text and button all at once, and you will never know what actually worked.
The hypothesis: wording and tracking
A handy formula: if we make [change], then [metric] will rise or fall by X%, because [observation about customers].
For example: “If we show the price in the first seconds of the video, CTR will rise by 10%, because the audience’s main doubt is the fear that it is expensive.”
It is convenient to keep all hypotheses in one table:
| No. | Comparison | Change | Metric | Expected effect | Status |
|---|---|---|---|---|---|
| H-01 | Video A vs video B | Opening: price or unboxing | CTR / CPA | CTR +10% | Test running |
How to set up a fair experiment
- Equal shares. Split traffic equally or allocate identical budgets, and keep the audience the same.
- One placement. Do not mix feed and Stories in one test: these are different viewing conditions.
- A calm period. Allow one to two weeks and do not run the test during sales or holidays.
- Auction control. If CPM or frequency changes sharply, the comparison between variants loses accuracy.
- Patience. Do not switch the test off on day two because of impressive figures — wait for the planned volume of data.
How much data you need
A practical guideline is at least 100–200 target actions per variant: clicks if you are comparing CTR, or conversions if you are comparing CVR and CPA.
For a more precise estimate at a 95% significance level and 80% power, an approximate formula is used:
n ≈ 16 · p · (1 − p) / MDE²
Here p is the current conversion rate, MDE is the minimum effect you want to detect (as a fraction), and n is the number of visitors or impressions per variant.
Example: a baseline conversion rate of 3% (p = 0.03), and we want to detect a difference of half a percentage point (MDE = 0.005). Then n ≈ 16 · 0.03 · 0.97 / 0.000025 ≈ 18,600 visitors per variant, or about 560 conversions. The smaller the effect, the larger the sample you need.
When the result can be considered reliable
- CTR and CVR are proportions; they are compared with a z-test or a chi-square test. The difference is significant if the p-value is no more than 0.05 and it holds in the main segments: on different devices and in different regions.
- CPA is assessed with bootstrapping or by comparing means when the distributions are similar. It is more reliable to look at CPA together with conversion and cost per click.
Do not forget the guardrail metrics that must not drop as CTR rises: bounce rate, time on site, pages per visit, conversion to enquiry.
Two hypothetical examples
A banner with the price
A week of interest-based impressions. Variant A has no price; variant B shows the price with a “Today only” label. B delivers a higher CTR and a cheaper enquiry (see the table below). The decision is to scale B and test two more ways of presenting the price: as a range and via a discount.
A short video versus a long one
YouTube In-Stream, ten days, comparing a short and a medium-length version. The short video is watched to the end more often (VTR rises, say, from 25% to 33%), CTR barely changes, and CPA falls thanks to completed views. Conclusion: the short version works as a “hook”, the long one for remarketing.
A one-screen report
Hypothetical figures for the banner test above:
| Metric | A | B | Change | p-value | Conclusion |
|---|---|---|---|---|---|
| Impressions | 50,000 | 50,000 | — | — | — |
| Clicks | 500 | 600 | +20% | — | — |
| CTR | 1.00% | 1.20% | +20% | 0.002 | Scale B |
| CPC | 4.00 | 3.40 | −15% | — | — |
| Enquiries | 55 | 78 | +41.8% | 0.046 | B is stronger |
| CPL | 36.4 | 26.2 | −28% | — | Move budget to B |
Below the table — two or three hypotheses for the next cycle: price as a range, a different background, a vertical version.
Testing rhythm and common mistakes
At the start of a campaign, compare two or three variants of the offer or the video opening. Once the metrics have stabilised, move on to details: background, hero, button colour, video length. Regularly look for the weak link in the funnel: if there are many clicks but few enquiries, it is the landing page that needs testing, not the banner. Test seasonal messages separately from evergreen ones.
- Too many variants for a small budget. There is not enough data for a single conclusion — test one at a time.
- Changes mid-test. Changing the audience or placement resets the comparison.
- Stopping at the peak. Early “success” often turns out to be chance — stick to the predetermined period and volume.
- CTR only. Clicks without enquiries are worth nothing — look at conversion and on-site behaviour.
- Moving “warm” creatives to a cold audience. What works in remarketing may be unclear to new people.
Before launching, check: one variable and a 50/50 split; a success metric and MDE have been chosen; the period and required volume are known; placement, audience and budgets are identical; the hypothesis is recorded in the table, and the decision rule — “scale, postpone or retest” — has been agreed in advance.
If you need an experiment plan for your budget and channels, SKY Central Asia can help you build tests as part of our digital promotion services. Find out more on our digital services page.
Leave a comment