A Step-By-Step Guide to Effective Ecommerce A/B Testing

In the online shopping world, customer experience is everything. 

In fact, Statista found that 44.5% of businesses worldwide perceive customer experience to be a primary competitive differentiator. But creating a quality user experience isn’t just about using your intuition to predict what will attract and convert potential customers — you need to have tangible insights into what your target audience actually wants.

Enter A/B testing.

A/B testing is a method of determining which website design, content or functionality is more successful with your site visitors. It allows you to test a variation or element of your page that may affect your consumers’ behavior.

You’ve probably heard about basic examples of A/B testing like changing button colors and call to action copy — but many ecommerce businesses don’t know how to do A/B testing effectively.

However, when done well, A/B testing is a powerful way to improve the most important metrics in your business. If this is your first time using A/B testing, here is your personal roadmap to help you get started.

When executed well, A/B testing (aka split testing) allows you to significantly improve your customers’ shopping experience, leading to better click-through rates, more conversions, greater loyalty and much more.

Not convinced? Take a look at Amazon, which evolved from an ecommerce store that only sold books to the largest online retailer in the world — and much of this success was due to CEO Jeff Bezos’s insistence on constantly testing, experimenting and innovating.

Consider all the ways Amazon has revolutionized the online shopping experience:

• One-click ordering.
• Dash buttons that allow you to reorder products simply by pressing a button.
• Two-day and free shipping.
• Personalized recommendations.

But these innovations don’t occur by coincidence — they are the result of rigorous multivariate testing and taking action based on reliable results. 

Inevitably, you’re going to make changes to your ecommerce site at some point in the future, whether it be to your messaging, SEO tactics or checkout process. But if you want to make the right changes and convert more of your existing traffic, A/B testing is the best way to uncover insights and guide you in the right direction.

There are two primary types of A/B testing: client-side and server-side, with client-side being the more common.

1. Client-side testing involves sending the exact same iteration of a web page to every visitor, then using Javascript to make changes within the visitor’s browser before the visitor is shown the resulting page.
2. Server-side testing is when your web server shows visitors different page variations, altering them on the server before they are sent to the visitor’s browser. No modifications to the page are happening in the browser.

For example, you may send 50% of your traffic to variation A and 50% of your traffic to variation B.

Then, analyze the performance of each page variation to determine whether variation A or variation B is more effective according to a set objective (conversion), whether that’s sales, email opt-ins, click-through rates, etc.

However, in order to be confident that test results are accurate and not the result of an anomaly, a certain number of visitors — the sample size — must be reached. If this threshold isn’t met, multivariate tests can be inaccurate, leading you to implement changes that can actually hurt your site.

The implication?

If your site doesn’t generate enough traffic, you may want to consider first investing in paid advertising or SEO to increase your overall traffic numbers. Once you’ve reached the necessary level of traffic, you can implement A/B testing at any point and see a significant return on your investment.

A/B testing is all about removing consumer pain points. Whenever you can remove friction from the shopping process, it significantly increases the chances of conversions. 

In the context of your own ecommerce website, here are some common A/B testing ideas:

• Different page layouts.
• Homepage and landing page.
• Product pages, category pages and product descriptions.
• Social media posts.
• Social proof such as reviews and testimonials.
• Pop-ups. 
• CTA button and copy.
• Website photography and product images.
• Push notifications.
• SEO tactics such as meta descriptions and keyword volume.

The beauty of A/B testing is that you can systematically work through each of these areas, incrementally and consistently improving each one to increase conversions.

A/B tests can come in many shapes and sizes. Here are four common A/B test examples you might see as part of an optimization program.

Navigation A/B Test.

In this test example, the goal is to see whether consumers are able to better navigate a site and reach checkout by exposing the menu at the top level rather than through a drop-down menu. Or, you may test whether it’s easier for customers to find the right product using a “tag & filter” (like the filtering feature on Amazon) or traditional navigation system.

Product Detail Page A/B Test.

Using this test, you can determine whether placing more educational information about the product higher on the page will affect the add-to-cart rate. Or, you may test various product titles, product images or stock status messages.

Form A/B Test.

This test aims to find whether consumers are more inclined to complete a form with fewer fields and simpler placeholder text within the fields. Or, you may test whether customers are more willing to fill out the form earlier or later in the buying process.

Checkout Process Test.

If you want your customer to complete a purchase, your checkout process needs to flow as smoothly as possible — which is why A/B testing is so crucial. 

A checkout process test will help determine whether a simple multipage checkout or a single-page checkout will result in better conversions. You may also test merging several small steps within the checkout process or using breadcrumbs to tell your customers users about the next steps of the process.

Before you begin testing, it’s a good idea to conduct thorough research on your target audience and create a roadmap that will guide you specifically through what to test and in what order.

Some research tools include:

• Analytics: Identify your most popular pages, CTR, traffic sources, pages with a high bounce rate, etc.
• Heatmaps: Discover exactly where and how people navigate your site. See where they click, where they pause, where they leave, etc.
• User Tests: Personally watch users navigate your site to discover where points of friction occur. 
• Surveys: Directly ask your customers about their shopping experiences. Did they encounter any problems? What would they like to see improved?

After doing a thorough analysis of your customer’s behavior, you can begin to develop hypotheses about your current online store and how you might improve the overall customer experience. These will form the basis of your A/B testing roadmap.

As mentioned, A/B testing is a powerful tool for crafting the ideal customer experience, but that’s not the only perk you’ll come across. Let’s take a look at some other great reasons for implementing A/B testing:

• It’s entirely based on numbers: There’s no relying on gut feelings or instinct — changes are only implemented if they are validated by the data.
• It dramatically reduces risks: Instead of taking a big gamble on an entire site redesign — which could lead to your conversions tanking — you’re making small, incremental improvements.
• It cuts through the opinions of those paid the most: Your position or salary don’t determine which changes are implemented. Only the data does.

But just like anything, A/B testing also comes with its setbacks. Let’s look at a few of the challenges you may encounter when launching your first A/B test:

• It requires a fair amount of experience: Some of the specific tools used in A/B testing have a steep learning curve, and there needs to be a fundamental understanding of web design, statistical analysis and other domain-specific knowledge.
• It’s a process: A/B testing is not a silver bullet — a single test probably won’t do much for your revenue. Instead, it’s about developing a long-term plan and targeting specific goals and improvements over time.
• Traffic is required: While the amount of site visitors will vary depending on the test and objectives, you need a fair amount of traffic to ensure that your tests are statistically valid.

Given the challenges and struggles listed above, it’s understandable if you have some reservations. But when executed properly, A/B testing can have a huge impact on some of your most important KPIs:

Improved return on ad spend (ROAS).

The more friction you remove from your website, the more paid traffic turns into sales, effectively increasing your return on ad spend.

Reduced customer acquisition cost.

Similar to how testing helps improve ROAS, the money you spend on other methods of customer acquisition will benefit from the reduced friction and shrink your customer acquisition cost (CAC).

Increased lifetime customer value (LTV).

Testing makes your website more engaging, which helps drive more return business. In other words, the easier you make the shopping process, the more customers will want to come back again and again, thus increasing your customer lifetime value.

Increased email signup conversion rate.

Your email list can be a huge revenue driver. Testing different ways of inviting new subscribers will help you maximize your email list sign-ups, which increases the number of people you can promote your products and services to.

Increased average order value (AOV).

Increased testing can help you find out what product bundles or upsell opportunities could work best, increasing your average order value. At scale, this directly impacts your bottom line revenue.

Reduced cart abandonment rate.

Every time a person abandons their shopping cart, that’s money walking out your digital door. Using A/B testing to optimize your checkout process can remove friction, confusion and mistrust and reduce your cart abandonment rate.

Increased ecommerce conversion rate (CR).

Smoothing out any barriers on the path to purchase is the most effective way to increase your conversion rate.

When it comes to A/B testing, experience and skill are just as, if not more important than the tools themselves.

First, you need to determine whether you want to do your testing in-house or hire an outside agency. Here are a few best practices to help you find the best solution for your business:

Evaluate skill sets.

Regardless of whether you choose an agency or in-house, here are some of the key skills to look for:

• Data research.
• Statistical analysis.
• UX design.
• Behavioral psychology.
• Project management.
• Conversion strategy for conversion optimization.
• User research.
• Information architecture.
• HTML and CSS.
• jQuery and JavaScript.
• And more.

Each of these skills contributes uniquely to the CRO process, and the absence of any of them can lead to ineffective A/B testing.

Use The Right A/B Testing Tools.

There are a number of specific tools required to do A/B testing, the three most common being Optimizely, VWO and Google Optimize. Each offers a full suite of A/B testing tools, including the ability to easily deploy tests, analyze the results, evaluate website heatmaps and much more.

Optimizely.

Optimizely is a feature-rich, enterprise-level A/B testing tool, and is priced accordingly. It is widely considered an industry leader for larger sites.

With a Stats Engine that helps analyze A/B test results, Optimizely lets users run tests with ease and get real-time results so they can take proactive steps to enhance their ecommerce stores.

However, for many small to medium-sized businesses Optimizely can be like using a semi-truck when a pickup will do. We recommend considering whether you will realistically utilize its full suite of features before selecting this tool.

VWO.

Visual Website Optimizer (or VWO) is an effective A/B testing tool, providing a solid mix of features that will do all of your core A/B testing tasks. It offers a SmartStats analysis tool, a WYSIWYG editor for beginners as well as on-page surveys, form analytics, heatmaps, etc.

Google Optimize.

Google Optimize, formerly called Google Website Optimizer, is a free A/B testing, website testing and personalization tool for small businesses. It comes with Google Analytics and allows users to see how multiple page variations perform against a specified objective. 

Whichever tool you choose, spend time researching the capabilities, pricing and overall level of support for each. Also, if you plan on doing your testing in-house, be sure you feel some level of comfort with the software you choose.

Use a CRO Agency for A/B Testing.

A number of service-based businesses provide A/B testing and CRO (conversion rate optimization) as a “done-for-you” service — but be warned: many marketing agencies might add CRO or A/B testing to their laundry list of services yet have little more than beginner knowledge.

There is a simple way to do A/B testing and a more methodical, professional way to do it. Make sure you’re getting the latter.

Here are some things to look for in A/B testing services:

• Whether they offer CRO as an add-on to other marketing services or focus exclusively on CRO.
• Case studies that demonstrate previous CRO success. These case studies should make clear the specific processes they used to achieve their results.
• Whether they’re data-driven. Any methodology that relies on intuition, flashes of “insight” or preferences isn’t CRO — CRO should always be data-driven.
• If they have a defined CRO process. A true CRO firm should have a specific, defined CRO process that they follow again and again. 
• Do they focus on the short term or long term? Instead of taking shortcuts and not waiting for enough test results to make a proper recommendation, the best CRO firms are squarely focused on implementing a robust plan over the long run.

Bottom line: Do your research and ask the right questions before committing to a CRO agency — your research will pay off.

How to Increase Ecommerce Sales

Explore our collection of free resources designed to help you scale smarter and accelerate your online growth from $1 million to $100 million.

Access Now

Needless to say, A/B testing is one of the most powerful and effective ways to drive ecommerce growth. Rather than relying solely on intuition to predict what will convert customers, A/B testing provides tangible insights into what your audience actually wants — so you never have to play the guessing game.

Luckily, with A/B testing and CRO apps like VWO and Shogun Page Builder, BigCommerce merchants have access to a variety of tools to help you increase conversions and make your existing traffic that much more valuable.