Guide7 min readUpdated
What Is A/B Testing? A Plain Guide for Small Websites
What is A/B testing? Learn how split testing works for websites, how to test two designs, and why small sites often lack enough visitors.
Imagine a coaching centre owner in Siliguri who changes her website button from "Submit" to "Get a free demo class". Enquiries feel higher the next week. Then she remembers that an exam notification came out that same week. Was it the button or the exam news? A/B testing exists to answer exactly this kind of question.
A/B testing, also called split testing, means showing version A to some visitors and version B to other visitors at the same time, and then comparing which version gets more of the action you want. The two versions differ in one thing only. Because both run together, outside events such as a festival or an exam result affect both equally.
What is A/B testing in simple words?
A/B testing is a controlled comparison of two designs using real visitors. Half of the visitors see the current page, called A. The other half see a changed page, called B. You then count how many people in each group did the target action.
The target action is called a goal. It can be tapping "Call now", sending a WhatsApp message, filling a form or paying by UPI. You choose one goal before the test starts.
The key idea is that both versions run at the same time. If you simply change the page and compare this week with last week, you cannot tell whether the design or the calendar caused the change. Split testing removes that doubt.
What does a split testing example look like?
A clear example is testing one button on a restaurant website. Suppose a restaurant in Kolkata has a button that says "Order now". The owner wonders whether "Order on WhatsApp" would bring more orders, because customers already use WhatsApp daily.
- Version A: a green button saying "Order now" that opens a form.
- Version B: the same green button saying "Order on WhatsApp" that opens a chat with the menu message filled in.
- Goal: number of orders started.
- Rule: nothing else on the page changes. Same colour, same place, same photos.
- How long: long enough that both versions are seen across weekdays and weekends.
At the end, the owner compares orders started in each group. If version B has clearly more, the owner keeps it. If the numbers are close, the owner learns that the wording is not the main problem, and looks elsewhere. For ideas about what a button should say and where it should sit, see my post on call to action button design.
Why does A/B testing matter, and who really needs it?
A/B testing matters because it replaces opinions with evidence from your real visitors. Anyone can argue about a button colour. A test shows what your visitors actually do.
But it is not for every business. It suits sites and apps with a lot of visitors and a clear, countable goal, such as a large online store or a busy booking platform. For these, a small improvement on a page seen by many people adds up to real money.
A split test also protects you from changes that look good but hurt. A fresh design can lower sign-ups for reasons nobody expected. If you test first, you find out before the change reaches everyone.
Do small websites have enough visitors for A/B testing?
Often they do not. A test needs enough visitors and enough completed actions in each version, otherwise the difference you see may be luck.
Think of tossing a coin 10 times. You may see 7 heads and 3 tails, but nobody would say the coin is unfair. Few visitors behave like few coin tosses. If each version of your page gets only a handful of enquiries in a month, a gap of two or three enquiries tells you almost nothing.
A local business site with a few hundred visitors a month often falls into this group. A clinic, a salon, a small manufacturer or a coaching centre in a small town may get a few enquiries a week. A test on that traffic would need to run for a very long time, and during that time festivals, seasons and your own marketing would change the numbers.
I will not give you a formula here, because the right amount depends on your current numbers and the size of the change you want to detect. The honest rule is simple. If you are not sure you have enough visitors, you probably do not. Some testing tools show how sure they are. Check the current plans and the help pages of any tool before you pay.
What should you do instead if your site is small?
If your site is small, you learn faster from talking to a few people and watching them use the page than from a split test. Five people can show you most of the big problems.
- Watch 5 people try the page. Ask each person to book a table or send an enquiry, and note where they hesitate. My guide on usability testing on a small budget shows how.
- Ask customers why they chose you or left. The reasons will often point to one clear fix.
- Review the page against a checklist. A UX audit checklist and a homepage design checklist catch common problems quickly.
- Change one thing, then compare for a longer time. This is not a true test, but it is better than guessing. Write down the date and watch for events that could confuse the result.
- Use common sense on well known problems. If your phone number is hard to find on mobile, you do not need a test to fix it. See website contact page and WhatsApp CTA.
These methods find large problems cheaply. A/B testing is better at fine tuning after the large problems are gone.
How do you run an A/B test step by step?
You run a test by choosing one goal, changing one thing, splitting visitors fairly, waiting long enough and then deciding in advance what result will make you switch. Follow these steps.
- Choose one goal. For example, "sends a WhatsApp message".
- Write your guess. For example: "A button that says Order on WhatsApp will start more orders than Order now, because customers already use WhatsApp."
- Change one thing only. If you change the text, the colour and the position together, you will not know which change worked.
- Split visitors randomly. A testing tool does this for you. Do not show version A on weekdays and version B on weekends.
- Decide the length before you start. Do not stop the moment one version looks ahead. Early leads often disappear.
- Avoid special days. A big sale, a festival or a news event can change visitor behaviour.
- Decide, then record. Keep the winner, write down what you learned and choose the next test.
Work with a designer when you need good variations. The designer can make sure both versions look right and are readable on a mid-range Android phone, since most Indian traffic is mobile.
What mistakes spoil an A/B test?
The biggest mistake is stopping the test early because one version looks ahead. A lead after two days can be pure chance.
- Testing too many things at once. Change one element per test.
- Too few visitors. The result is noise, not a finding.
- Changing the page during the test. Any edit starts the test again.
- Picking a goal that does not matter. More clicks on a button are no use if enquiries do not rise.
- Ignoring the mobile view. A version can win on a laptop and lose on a phone.
- Testing a weak idea. If the page has big problems, fix them first. A small button change will not rescue a confusing page.
Can a small project skip A/B testing?
Yes. Most small projects should skip it for now. If your site gets few visitors, spend your effort on clear content, a fast mobile page and one obvious action. Then talk to real users.
You can come back to A/B testing when traffic grows and the basic design is solid. If you want a clear and tested starting point before that stage, see my website design work, or write to me through the contact section.
Frequently asked questions
What is the difference between A/B testing and split testing?
+
The two names mean the same thing. Both compare two versions of a page or element, shown to different visitors at the same time.
How many visitors do I need for an A/B test?
+
It depends on your current numbers and on how big a change you want to detect. If your site gets only a few enquiries a month, the test will probably not give a reliable answer.
How long should an A/B test run?
+
Run it for a period you decided in advance, long enough to include both weekdays and weekends. Do not stop early because one version looks ahead.
Can I A/B test a mobile app?
+
Yes, some tools support it, but the same rule applies: you need enough users and one clear goal. Check the current features of any tool before relying on it.
Working on something like this?
Tell Tithi what you are planning. You get a clear scope, a fixed price and a timeline before any work begins. I reply to every enquiry within 24 hours.