Amazon experiments: how to run A/B tests you can trust

Every listing change is a guess until it is tested. A new main image can lift sales, do nothing, or quietly cost you, and a before-and-after comparison in your Business Report can rarely tell those apart, because the season, your advertising and your competitors changed at the same time. Amazon’s Manage Your Experiments removes most of that noise by showing two versions of your content to two groups of shoppers at once.

We have run experiments on client accounts since the tool appeared, and the honest summary is this: the tool is excellent, and it is easy to fool yourself with it. This guide covers the rules as Amazon states them for the UK, and then the part Amazon’s help pages leave to you: designing a test that can actually produce an answer.

How Manage Your Experiments works

Manage Your Experiments runs A/B tests, also called split tests, on your brand’s listing content¹. During a test, shoppers who view the product are randomly split into two groups. One group sees Version A for the whole experiment and the other sees Version B; the content is not rotated by day or by hour¹. The split is by customer account, and each shopper keeps seeing the same version whatever device they use, as long as Amazon can identify them. Visits where the customer cannot be identified are left out of the sample³.

The experimental content shows everywhere it would normally appear for that group. A test title, for example, appears in search results, on the product page and in the basket¹. That matters: a title or main image test measures whether more shoppers click as well as whether more of them buy. Amazon states that experiments do not affect search rankings¹, and results are updated once a week until the test ends³.

Who can test, and what

If a product you care about isn’t eligible, Amazon’s suggestion is to drive more traffic to it, for example with advertising¹. Our usual alternative is to test the idea on a bigger seller in the same range, then roll it out to the smaller ones and watch their conversion.

One rule from outside the tool now shapes title tests. Since 27 July 2026, UK titles must not exceed 75 characters, including spaces, for all product types except media, and a title that breaks the rules might be corrected automatically or might not appear in search⁶. Both versions of a title test must meet the current rules. Amazon’s own list of test ideas still suggests “reducing product title length to under 100 characters"⁴, which the 75-character limit has overtaken. Our listing optimisation guide covers the new title and item highlights rules, and the free listing checker tests a title against them before you submit it.

How long to run a test

Amazon gives you two choices²:

• Set duration, with fixed start and end dates. Amazon says experiments typically run for 8 to 10 weeks and recommends 8 weeks for a fixed-length test.
• To significance, where Amazon ends the experiment once results are statistically relevant, in as little as four weeks. Amazon recommends it when you are unsure, and describes it as a beta feature for title and image experiments⁵.

Whichever you choose, let it finish. Amazon’s own advice is blunt: early results “can be misleading”, and every time you peek and act on interim results you increase the chance of overestimating the effect “or even selecting the wrong version"⁴⁵. We also avoid starting a test that will straddle a major sales event unless the event is what we want to learn about, because shoppers in peak weeks behave differently.

Design a test that can give you an answer

Amazon’s tips page says experimental content must be clearly different from the existing content, because a version that is too similar “is less likely to affect customer behaviour"⁴. That is the heart of test design, and the reason is arithmetic.

Write the hypothesis first

Amazon calls the hypothesis “one of the most important parts of your experiment"². Write it as a cause and an expected effect, for example: “Showing the tablet next to the box in the main image will lift clicks from search, because shoppers can see the size.” A clear hypothesis is what lets you apply a result to other products, and what lets you learn from a test that doesn’t produce a winner.

Change one thing

A test tells you which version won, not why. If Version B has a new title, a new main image and new bullets, you learn that the bundle did better, and nothing you can reuse elsewhere. Multi-attribute experiments are useful when you want to ship a complete redesign safely; for learning, change one element per test.

Check you have enough traffic

The smaller the difference you want to detect, the more shoppers you need. A standard rule-of-thumb calculation shows the scale. It is not Amazon’s method, which is Bayesian³, but it is close enough to tell you whether a test is worth starting.

Applied to a product converting at 10% per unique visitor:

Two conclusions follow. First, an ASIN with 4,000 visitors a week can detect a 10% lift within an eight-week test, but not a 5% one. Second, the “1% improvements” that compound into big gains over a year are real, but they are invisible to any single experiment. You cannot prove a 1% change works; you can only make sure each change is sensible and that the bigger ones are tested. So test the changes most likely to move behaviour a lot, such as the main image, the title and the first two secondary images, and make small, low-risk improvements without waiting for proof.

Read the result honestly

When an experiment ends, Amazon shows, for each version, units, sales, conversion rate, units per unique visitor and the sample size (unique visitors who saw that version), plus two summary figures³:

• Probability that one version is better. Amazon calculates a range of possible impacts; “75% probability that Version A is better” means 75% of them show a positive lift from publishing A³. It does not mean A is 75% better, and it leaves a one-in-four chance that it is not better at all.
• Projected one-year impact. Amazon takes the average daily sales difference during the test and multiplies it by 365. The Likely column is the median of the possible outcomes, and Best case and Worst case show the 95% interval. Amazon notes that the projection ignores seasonality, price changes and anything else that would affect your business³.

Inconclusive results are results

An experiment can end without a confident winner. Amazon lists the usual reasons: the change was too small to alter behaviour, there was not enough traffic, both versions were similarly effective, or the change is not something most customers care about³. Read the result against your hypothesis. Amazon puts it well: an inconclusive result “can tell you that a certain type of content is not worth investing in because it does not affect customer behaviour"³.

In our experience, a large share of well-designed tests end this way, and that is not wasted effort. If a costly new set of lifestyle images can’t beat the old ones, you have saved the cost of reshooting the rest of the range.

After the test

1. Publish the winner. Amazon’s auto-publish option can do it for you; otherwise use the normal listing tools, or the A+ Content Manager for A+ and Brand Story³⁴.
2. Watch the product afterwards. In Business Reports, track Unit Session Percentage, Amazon’s units ÷ sessions measure⁷, for four weeks after publishing. If it falls back, something else was driving the result.
3. Roll out with care. Apply a winning idea to similar products, but watch each one, or test it again on the next-largest product before changing the whole range.
4. Keep a log. One row per test: ASIN, hypothesis, what changed, dates, sample size per version, probability, Likely and worst case, decision. After a year, the log is worth more than any single result.

Where to start

Over the years, the tests that have most often produced a clear answer in the accounts we manage are main image and title tests, because they change what shoppers see before they decide to click. Title length is a good example of why testing beats rules of thumb: in some categories a short, clean title wins, in others a more descriptive one does. That is our experience, not a published benchmark, so use it to choose your first tests, not to skip them.

A sensible order for a brand new to experiments:

1. Main image on your highest-traffic product: one clear change, such as showing the product next to its packaging or improving how readable the label is.
2. Title, within 75 characters: benefit-led against specification-led, keeping the search terms that bring buyers in both versions (our guide to Amazon keyword research shows how to find them).
3. The first two secondary images, on ASINs with four or fewer child ASINs.
4. A+ content, once you have a version that is genuinely different. Our A+ content blueprint covers what to build, and how to compare before and after when a product can’t be tested.

Each winning test should also lower your advertising cost, because a higher conversion rate means a lower ACoS at the same bid. Our guide to break-even ACoS shows the arithmetic, and the Profit X-Ray finds the listings with plenty of traffic and below-median conversion, which are usually the best place to run your next test.