How to do A/B Testing with Google Analytics

mobile app optimization a/b testing and a/b testing for marketing optimization and an introduction to using a/b testing for marketing optimization
OliviaCutts Profile Pic
Published Date:01-08-2017
Your Website URL(Optional)
The Ultimate Guide to A/B Testing UNBOUNCE www.unbounce.comWhat’s in the ebook? What is Unbounce? This guide will teach you everything you need to Unbounce is the DIY Landing Page platform for know about A/B tesing, including the all important Marketers. Build high converting landing pages “What to Test”. for your PPC, email, banner and social media campaigns. • Introduction to A/B Testing Find out why you should be testing.• Build & Publish Landing Pages In Minutes • Getting Started with A/B Testing Use our powerful editor to re-create your design 5 things all pages should have & a case study. from scratch, or use one of our templates for a • What to Test head start. Learn what and how to test page elements.. • 1-Click A/B Testing For Optimization • Workflow Need to solve an argument with your boss? Learn now to manage your testing workflow. Stop relying on assumptions and set up a test • Dealing with Data experiment. When to stop, and how to use your data.• Simple Analytics To Track Campaigns Our stats are powerful yet simple. It’s all about clicks, conversions and how well your campaign Feel free to share this document. is performing. START YOUR FREE 30 DAY TRIALTable of Contents 1. Intro 4. An A/B Testing Workflow a. An introduction to A/B Testing 1. Brainstorming b. What about Multivariate Testing? 2. Creating page variants c. What is the Multi-Armed Bandit? 3. Running your test 4. Promoting your winner 5. Start a new test 2. Getting Started with A/B Testing a. Building a control page 5. Dealing with the Data b. Creating variants c. What makes a champion a. How to deal with a failed test d. A case study b. When to stop testing c. What’s next? 3. What to Test 6. A&B Testing Glossary: Terminology & Tips a. Developing a hypothesis of what to test b. Page elements to test I. Headline VI. Colors II. Call to Action VII. Forms III. Images VIII. Social proof IV. Video IX. Confirmation pages V. CopyThe Language of A/B Testing & Optimization To make sure that we’re all on the same page, let’s set some basic vocabulary: • Element - a discrete unit on the page: a block of text, a form, a button, an image, etc. • Page - a web page or landing page that is considered the control page for your test. • Variation - a version of a page that has some changes made to page elements. Also referred to as a variant. • Test - a hypothesis that one version of an element will change the conversion rate in a significant, hopefully, beneficial way. • Conversion - when a visitor takes a desired action on the page. The most common means of testing to improve conversions online are A/B Testing (aka Split-testing ) and Multivariate Testing (aka MVT), which we’ll talk about in more detail below. The Ultimate Guide to A/B Testing 4What is an A/B test? When you run an A/B test, you compare one page against one or more variations that contain one major difference in an element of the control page. After a set amount of time, or visits, you compare the results to how the change affected your results. A/B testing is actually a nickname for a variety of testing that involves single element changes across VISITORS RANDOMLY DISTRIBUTED multiple variations, so it might actually be A/B/C/D VERSION VERSION testing, but it is still called A/B testing and if you see A/B/n or Split Test that is just a more accurate expression of one control with many variations. Every visitor will see one version of the page or another, and you’ll measure conversions from each set of visitors. A/B tests allow you to test one version of copy, images, forms etc. against another. Layout choices such as where you place the elements can also be tested, or functionality choices like when and how you show error messages. VERSION VERSION VERSION B IS BETTER THAN VERSION A The Ultimate Guide to A/B Testing 5How Much Traffic do You Need for a Valid Test? and customers in a controlled environment. Once you have seen the results of a test you can take the A common question is how much traffic you need new knowledge to your site wide design. when running a test. This will depend on a few factors. If you can run 100-200 visitors to your One of the best things about A/B testing is that it can pages, you can sometimes make gut decisions be very fast and dramatic. It is also less complicated about pages that are obviously underperforming and to set-up and reach significant results in order to initate a new test. But generally you’ll want to wait make informed decisions. for about 1,000 total visitors and make sure you run the test for a week to cover daily variances. Advantages of A/B Testing Use A/B Test Results for Major Site-Wide Decisions• They are fast • Advanced analytics can be installed and Split tests can also be used to temper drastic evaluated for each variation (e.g. click changes like redesigning a homepage by only tracking, heatmaps, etc.) serving it to a portion of your visitors. Companies • Can achieve more dramatic conversion rate like Twitter and Facebook use this strategy to test lift results major interface changes by only rolling out the new • Requires less traffic version to a segment of their visitors and measuring how that group reacts. Disadvantages of A/B Testing A/B test results are influential in making good • More dramatic failures decisions in site redesign because they help you see • Less specific understanding of element effects what elements are important for your audience The Ultimate Guide to A/B Testing 6What is a Multivariate test? When you run a multivariate test you use one Advantages of MVT page and dynamically supply multiple versions of multiple elements. For instance, testing 3 versions • Better measurement of element interaction of your headline, 2 versions of your call to action, • Failures are less dramatic and 3 button colors simultaneously. That would give • More comprehensive testing you 18 versions (3-headlines x 2-CTA x 3-button- color) of the page that you are testing. Disadvantages of MVT As you can see, with each successive element and • Can require many more variable combinations variation on an element you multiply the number of to be run than A/B versions that you are testing. This means that the • Requires vastly more traffic to reach statistical amount of traffic required to reach confidence significance than A/B increases. If you are testing 2 versions of 3 elements • Major layout changes are not feasible you have 8 variations (2 x 2 x 2) and if you have 3 • Less layout change is possible versions of 3 elements you have 27 variations (3 x 3 x • The restrictions of the test setup constrain 3), adding extra tests adds up very fast. marketing creativity • Dynamic elements decrease the effectiveness The purpose of multivariate testing is to refine of 3rd-party tracking tools pages that already perform well by looking for • Less layout change is possible compounding effects between elements.• The restrictions of the test setup constrain marketing creativity Multivariate testing is something that you should • Dynamic elements decrease the effectiveness only be looking at if you have large amounts of of 3rd-party tracking tools traffic. The Ultimate Guide to A/B Testing 7What is Multi-Armed Bandit Testing? There are varying opinions on whether this type of Recently a new type of testing has appeared on the testing is actually successful. Some people like Visual scene. It’s called the multi-armed bandit. This type Website Optimizer say that in the long run multi- of testing is essentially the same as an A/B split test armed bandit requires a large conversion difference and works very much the way that Google Adwords to be successful. ad delivery works. It starts with an experimental phase where it gives an even split of traffic across Why A/B testing is the right answer all versions of the page and then it goes into an most of the time exploitation phase where it give more traffic to the version that is currently performing the best. Part of why we use A/B testing is because it is less complicated for marketers that use it and is Over time if the winner starts to decline in generally more fruitful as a testing method. Chris conversion the system automatically gives Goward at Wider Funnel says that they do 8-10 traffic to another version. The drawback to A/B tests for ever 1 multivariate test that they do automated systems like this is that they can for clients, and they only do MVT on pages that make counterproductive decisions, especially if get a very large number of unique visitors. Most of they have a short experimental phase or choose the time A/B split-testing is the best answer for a conversion event that is not on target for your improving conversions and testing hypotheses, so bottom line needs. we want focus on that. The Ultimate Guide to A/B Testing 8Getting started with A/B Testing This is one of those things that is hard for a lot of people. How do you start? The simple answer is to make one page detailing the simple facts about your product or service. These simple facts are the features of your product. These features have consequences. Those consequences are benefits. Secondly, ask yourself why are you better than the other options (the competition) for solving the problem? Is it a lower price, better shipping, better return policy, etc? Online you have two primary ways to communicate information: image and copy. All good pages are going to cover the following 5 things: 1. A headline with a Unique Value Proposition 2. A demonstration of the product or the benefit - preferably in the context of how it would be used 3. A clear value for the viewer 4. Support of value 5. One action Your action and your headline are the only things that need to be copy driven, everything else is wide open. When you create your control page, make sure all 5 of these elements are represented. If you have done this you will have a solid foundation for testing. The Ultimate Guide to A/B Testing 9Building a Control Page Control pages are your first step in creating a generation of tests. Your control is the parent and variations are the children. The winning variation becomes the next generation’s control for further variation. A good place to start your control page is a wireframe. Take a look at KissMetrics’ Landing Page Blueprint (seen on the right). But, don’t let that stop you from using your own vision in how you layout your elements. Just because a certain structure is popular doesn’t mean it’s right for you, also, standing out can be a very effective strategy for good results. Once your basic structure and content is in place you will have a starting point for tests. Good structure will make a world of difference in your testing success, because it will make each change less time consuming. The Ultimate Guide to A/B Testing 10Building Page Variations into designing your test, if you are communicating Each variation should have one hypothesis of to the wrong audience, you will acquire the wrong change: color of background, different major image, customers or leads. Keep an eye on what your new different CTA in the headline, different CTA in the customers or leads do, and if they’re not the right action button, etc. It doesn’t matter how many ones, run a new test with different messaging to hypotheses you have, but you will always get the be more inline with your most important business best data by only having one hypothesis per goals. variation. A good technique for tracking your test Once you get used to how your target market performance is to keep a record of your behaves you may be able to get away with a multi- hypotheses and your results, so you know where hypothesis test, but when you start out stick to a you’re going right and where you went wrong. That framework that maximizes test cleanliness. will make your next test better, and serve as a record for stakeholders, as well as yourself. What Makes a Test Champion Note: You should use descriptive experiments names like Form Length 2012 Catalog or Call To In simple terms your champion is the page variation Action: Registration, something that makes the test that converts the best. Once you’ve crowned your parameters apparent, not Steve’s Test or PPC Test. champion though you’ll want to dig a little deeper and gauge the quality of your propects too; The reality is that no matter how much time you put The Ultimate Guide to A/B Testing 11A Case Study for A/B Testing After the launch of the Noob Guide to Online Marketing we made a landing page that used to help expand the distribution of the guide, which averaged a 13% conversion rate. We then decided to see if we could get better results from the landing page. We had gathered some data by inserting a KISSinsights (now Qualaroo) widget on the page that suggested many people (about 45%) would rather provide their email to get the PDF download, so we decided to see which would convert better, “pay with an email” or “pay with a tweet”. In addition, some feedback indicated that it was fairly common for people to pay with a tweet and then immediately delete it from their stream. Our Test Hypothesis Will allowing visitors to download the PDF by providing their email address perform better than receiving it in exchange for a tweet? Considering that not everyone has a Twitter account, or is willing to share such information with their followers. The Ultimate Guide to A/B Testing 12Pay with a Tweet Averaged Pay with e-mail averaged an 18% conversion rate. a 22% conversion rate Moving on from this test we’ve decided to discuss a possible hybrid option that might produce the best of both worlds; A single version that lets the user decide whether they want to pay with a tweet or provide their email. The Ultimate Guide to A/B Testing 13What To Test Developing a hypothesis of what to test The first step in any A/B test is to create a hypothesis for what will make the page perform better. There are two very common approaches: one revolves around data, the other involves people. The data approach uses statistical data like bounce rate, exits, conversion starts, and funnel analysis. Data driven hypothesis use tools like Clicktale, Google Analytics, Omniture, and CrazyEgg. The people approach employs tools like Qualaroo (formerly KISSinsights) and SilverBack to gather user feedback. These are all about addressing the quality of the experience on your site or specific pages. Your most effective hypotheses will come from combining both the data and people approach. On the data side you may see something like a high bounce rate and know that there is something wrong. You might look at time on page too, since this can be a good indicator of how people are interacting with your page. If you have a high bounce rate and a high time on page you probably have a problem near the bottom of the page, maybe the call to action button. If you have a high bounce rate and a short time on page it means that you have something repelling at the beginning of the page, maybe your headline. If you cannot determine from the data approach what is it specifically that is hurting your conversion rate, you should then use the people approach to gather direct user feedback to gain insight. The Ultimate Guide to A/B Testing 14So What Should You Test? The Headline Your headline is your first impression. Its success is dictated by how closely it matches what your viewer expected when they made the decision to visit your page - whether from an ad, banner or email link etc. You can try testing positive versus negative language in the way you express your value in the headline, for example: Save Time By Downloading Now vs. Stop Wasting Time, Download Now. People are impatient and will read your headline very quickly, so communicates your core value proposition in a way that makes it really obvious what your offer is. To assess problems with your headline try a 5-second test: where you flash the page in front of a person unfamiliar with your brand for 5 seconds and ask ”What is this page about?” If they don’t know, it’s not clear enough and time to revisit the messaging. Headline Test Ideas: • Long vs. short • Positive vs. negative • Benefit oriented vs. feature oriented • Single headline vs. primary and secondary supporting copy • Enhance the contrast to make your headline more visually powerful, to focus people’s attention on it The Ultimate Guide to A/B Testing 15An Example Test - Headline In this headline test by WhichTestWon version A increased visitor form fills by 27.76% by changing the headline and subhead wording. vs. VERSION A VERSION B The Ultimate Guide to A/B Testing 16Images An Example Test - Images Since images can convey complex ideas and Sometimes tests will surprise you (which is always emotions quickly they are a phenomenal source for a good way to interrupt your assumptions). In this testing. case study removing the image actually resulted in version A increasing leads by 24% by raising more of This is one of the few areas that has clear best the form above the fold. practices. Make sure you use high quality images ie. no pixelation or blurriness. If you use people, have their gaze directed to your call to action, and choose VERSION A images that enhance the understanding of your offering. It’s also been shown that image annotations are one of the first things people read on a page, so make sure you write something compelling right below your primary image. vs. The most basic image tests will be: • Color vs. black and white • People vs. product • One image vs. many images • Masculine vs. feminine VERSION B The Ultimate Guide to A/B Testing 17Video 4. Directional cues within the video: Point your Using a video on your pages can have a dramatic visitors at your landing page’s conversion goal. effect on engagement and your conversion rate. But This can be a physical action (a person pointing) what do you test? or a verbal instruction. 1. Autoplay vs. press play: Usability guidelines 5. Image vs. Video: Try a static image in one page advise against autoplay as it is an interruption variant and a video in the other. technique that annoys people and makes them click the back button. However, it can also 6. Spokesperson: Not everyone is a natural speaker, increase conversions. You’ll have to weigh up and maybe your audience doesn’t want to listen the potential negative impact on your brand’s to the CEO pitching them. Find someone who perception versus the potential for increased comes across as natural and likeable. conversions. 7. Number of videos: If you are using them for 2. Calls to action (CTA): This can be permanently testimonials, you might want to consider having visible, or be shown at strategic times throughout more than one to increase the trust factor. the video, or be presented at the end. 8. Lo-fi vs. Hi-fi: Professional videos are good 3. Length: Try short and long versions to see what at communicating your focus on quality, and your customers need. Some products may need are usually the best way to go. But with a tight a detailed study, while others might be better budget and even a camera phone, you can create suited to a 30 second “commercial” style. a convincing and authentic experience. The Ultimate Guide to A/B Testing 18Some Example Tests - Video In this case study by VidYard they show how they got a conversion lift of 100% by using video on their landing page. In this second test, adding a video made version B increase conversions by 216% vs. VERSION A VERSION B The Ultimate Guide to A/B Testing 19Copy An Example Test - Copy The typical rule of thumb is to keep your pages short Sometimes all you need to do is remove just a small and concise, but don’t dismiss the long form either bit of copy to see an increase in conversions. In this - it’s been proven to convert - but typically only for example, just removing one single block of text (not the sales letter type of page. even that significant in terms of size), resulted in a conversion lift of 62%. Whether you have 5 paragraphs or just a set of bullets it is important to focus first on benefits VERSION A vs. VERSION B and then support with facts and function. A good way to judge the effectivness of your copy (if you can’t tie your tests success or failure to it), try reading it aloud to see how it sounds and flows. Even better, get a coworker to read it to you and see how much you cringe. If you find that you are writing a lot of copy on your landing pages you should utilize the famous advice of Steve Krug, “Remove 50% of your copy, then delete half of what’s left.” Being overly verbose can hurt your conversions if your message can be described adequately in the short form. Again, test both versions to see which resonates best. The Ultimate Guide to A/B Testing 20

Advise: Why You Wasting Money in Costly SEO Tools, Use World's Best Free SEO Tool Ubersuggest.