{"id":9859,"date":"2022-06-25T11:18:41","date_gmt":"2022-06-25T11:18:41","guid":{"rendered":"https:\/\/qualaroo.com\/blog\/?p=9859"},"modified":"2026-01-14T11:26:30","modified_gmt":"2026-01-14T11:26:30","slug":"ab-testing-mistakes","status":"publish","type":"post","link":"https:\/\/web-staging.qualaroo.com\/blog\/ab-testing-mistakes\/","title":{"rendered":"The 20 A\/B Testing Errors That Are Costing You Sales"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">A website visitor leaving mid-journey is like having a customer walk into your store, shuffle around a few things, make eye contact with you, and just walk out of the store as soon as you are to greet them.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">It\u2019s a little sad, isn\u2019t it?<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong><em>But it gets even more heartbreaking if you have a website and see a big chunk of your website visitors bounce off to your competitors.<\/em><\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">A\/B Testing is a remarkable marketing strategy to help reduce bounce rates and the one you won\u2019t have to take with a grain of salt. It offers phenomenal results, but only if implemented correctly.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">So, if you don\u2019t want your website visitors to leave without completing a goal, check out these A\/B testing mistakes you should avoid while you are at it.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Let\u2019s discuss these common mistakes in A\/B testing, which take your resources down the drain. But if you have already made a few, it won\u2019t hurt to look at the solutions you can employ and avoid repeating them in the future.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Instead of simply presenting these split-testing mistakes in a list, we went a step forward to categorize these A\/B testing mistakes to avoid based on when they occur:&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Before, during,<\/strong> and <strong>after<\/strong> A\/B testing.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Let\u2019s get this show on the road!<\/p>\n\n\n\n<ol class=\"wp-block-list\"><li>Basing Your Test on Invalid A\/B Testing Hypothesis<\/li><li>Conducting Tests On A Development Site Instead of a Live One<\/li><li>Testing the Wrong Page<\/li><li>Copying Split Testing Case Studies Left and Right<\/li><li>Testing Way Too Early<\/li><li>Making It All About Conversions<\/li><li>Split Testing Too Many Elements at Once<\/li><li>Creating Several Variations<\/li><li>Showing Different Variations to Different Audiences<\/li><li>You Got the Timing Wrong<\/li><li>Testing with the Unsuitable Traffic<\/li><li>Altering Parameters in the Middle of Split Testing<\/li><li>Calling It a Day Too Early on Your Tests<\/li><li>Ignoring Periodic Radical Testing for Incremental Changes<\/li><li>Testing Multiple Tests Simultaneously<\/li><li>Not Measuring Results Carefully<\/li><li>Not Understanding Type I and Type II Errors<\/li><li>Using an Incompetent A\/B Testing Tool<\/li><li>Not Considering Small Wins<\/li><li>Wasting Your Time Testing Insignificant Elements<\/li><\/ol>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Mistakes Made Before A\/B Testing<\/strong><\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"1_Basing_Your_Test_on_Invalid_AB_Testing_Hypothesis\"><\/span>1. Basing Your Test on Invalid A\/B Testing Hypothesis<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Hypothesis building plays a pivotal role in determining how well your A\/B testing will perform and whether the results will be viable.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">In simple words, an A\/B test hypothesis refers to speculations on why you are getting specific results on your website, for example, the bounce rate on your product page, high traffic with low conversions, etc.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">If your hypothesis is invalid, so will the results from your A\/B tests. It\u2019s because the changes you made in your testing process are not valid.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>How to avoid this A\/B testing mistake<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Create a valid hypothesis.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">But how? By following these steps &#8211;<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/lh5.googleusercontent.com\/j2RYltGmmFXbd0hPfVRyrT2HncbQKbmyolqcsV4dfpBIEpFJwsJ6nFIaXhsZLLvo3aP7prlLb1F3DCtxb6d_uYoksZgE9p5v67traEV8PYDH20ITYr-fGCI1kBaPkGU9j2gDTlnw6H1yqmsCD3cOR0c\" alt=\"\"\/><figcaption><a href=\"https:\/\/blog.wishpond.com\/post\/115675436717\/ab-testing-hypothesis\">Image Source<\/a><\/figcaption><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Here are A\/B testing hypothesis examples and steps:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong><em>Step 1:<\/em><\/strong><em> Understand the problem<\/em> &#8211; why are customers not converting? Is there anything missing on your website that is crucial for customers? You can get answers to these questions in three ways:&nbsp;<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li><strong>Ask the customers directly: <\/strong>For this, you can use survey feedback tools like <a href=\"https:\/\/qualaroo.com\/\">Qualaroo<\/a> that help you create pop-up surveys which collect in-context feedback without disturbing the visitors\u2019 experience.<br><\/li><\/ul>\n\n\n\n<figure class=\"wp-block-embed-youtube wp-block-embed is-type-video is-provider-youtube wp-embed-aspect-16-9 wp-has-aspect-ratio\"><div class=\"wp-block-embed__wrapper\">\n<iframe loading=\"lazy\" title=\"What Is Qualaroo &amp; How It Works\" width=\"1120\" height=\"630\" src=\"https:\/\/www.youtube.com\/embed\/hciV49P_VSE?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share\" referrerpolicy=\"strict-origin-when-cross-origin\" allowfullscreen><\/iframe>\n<\/div><\/figure>\n\n\n\n<ul class=\"wp-block-list\"><li><strong>Use analytics: <\/strong>Explore <a href=\"https:\/\/analytics.google.com\/analytics\/web\/\">Google Analytics<\/a> to know how many people bounce from your pages, if they are clicking on your CTAs, after how long they bounce, and more.<br><\/li><li><strong>Analyze user behavior:<\/strong> Choose tools offering features like Heatmaps and session recordings for user behavior. <a href=\"https:\/\/qualaroo.com\/integrations\/sessioncam\/\">Qualaroo offers SessionCam integration<\/a> that has both of these features.<\/li><\/ul>\n\n\n\n<p class=\"wp-block-paragraph\"><strong><em>Step 2:<\/em><\/strong> Once you know what\u2019s affecting the customer behavior, for example, inaccurate CTA, low conversion, etc., you can come up with corrective changes you think will work. These changes are the elements you&#8217;ll tweak on your website or mobile app while A\/B testing.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong><em>Step 3:<\/em><\/strong> The last step is analyzing which changes will bring the results you need. For example, will you have more conversions if you change the CTA?&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">In this step, you also need to identify the measures to evaluate the success of your testing process.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">For example, you hypothesized that adding more information on your product page and updating the CTA text will increase customers&#8217; time on the page.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">So, you A\/B tested this hypothesis by adding relevant product information and updating the CTA button.&nbsp;&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">If you can reduce the bounce rate on your said page, say by 15% by the next month, that will mean your testing worked.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>CASE STUDY &#8211; HOOTSUITE<\/strong><\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/lh6.googleusercontent.com\/mu7BcqW58G8EB7bP-pIDsgl06s-XwgFGMmMRoR7gKkAvuDr6uEbxXZOnkC_pJ0AlRh-muf7wvQOzwOt_nR-4i3dYYe51CusYdAfdOSfufs3bVq1eSF6yFRuQcDlbcuilmLGkrSxQDalDk-vJoJk94uw\" alt=\"\"\/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\"><a href=\"https:\/\/qualaroo.com\/case-studies\/hootsuite\/\">Hootsuite<\/a>, a social media management platform, enables its users to access and manage their social media accounts in one place with its tailored solutions.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The company\u2019s landing page was not user optimized and had a high bounce rate. They launched surveys on their website to understand the reasons behind the visitor behavior.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">They found that the page did not have the information users were looking for. Since the team now had a viable hypothesis, they could perform website A\/B split testing to see the most needed information.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">They revamped the page to include better product information like screenshots, pricing structure, detailed features, and testimonials to help visitors make better decisions.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">On running the A\/B test, the new landing page saw an increase of 16% in the conversion rate compared to the original page.<\/p>\n\n\n\n<div class=\"banner-btn newuishow \">\n<a class=\"round_btn try-btn\" href=\"\/\">See Qualaroo in Action<\/a>\n<\/div>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"2_Conducting_Tests_On_A_Development_Site_Instead_of_a_Live_One\"><\/span>2. Conducting Tests On A Development Site Instead of a Live One<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Okay, this A\/B testing mistake might come as a shock to you, but sometimes, <strong><em>people actually test the wrong website.&nbsp;<\/em><\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Wrong as in not an entirely different website but that the developers forget to switch to a live website and continue testing on a site that\u2019s still being developed.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Not that it\u2019s a bad idea to test marketing campaigns on a development site. The only thing is that you won\u2019t get viable results since the only people visiting the website are developers and not your target audience.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"3_Testing_the_Wrong_Page\"><\/span>3. Testing the Wrong Page<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/lh4.googleusercontent.com\/wDKqN_Stv2-U4JcS23LtNiFQ5AAB8NdvzsDQT6elwjJsIpRaH66rQhQBDR-Bz1vrrZP-FzEFBuC_8WTB0YTy3pDjNyNy2or15r4aZo_hj8gxS_lABZS6F04vYf82W7mkoHH62FnYZX5ZeyeJu8nFB4o\" alt=\"\"\/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">It may sound like a not-so-common error and something obvious, but this A\/B testing mistake occurs very often. Which page you should A\/B test depends on your objective, and that\u2019s where things go south.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">For example, say visitors land on your product page, browse for a few minutes and then opt for a demo. Once they land on the demo page, they leave the website.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">In this case, you will see a high bounce rate on your demo page and you will end up A\/B testing it to understand the reason. But in reality, the product page is failing to convince the visitors to convert.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">So, the best course of action would be to understand the nuances of a buyer\u2019s journey and A\/B test your product page to boost lead conversion.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>How to avoid this A\/B testing mistake<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Generally, businesses A\/B test their best website pages, i.e., pages with the most traffic, to get the best results.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">If you aim to improve conversions, you should A\/B test the pages recommended by HubSpot, like the About page, Home page, blog, and Contact Us page.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"4_Copying_Split_Testing_Case_Studies_Left_and_Right\"><\/span>4. Copying Split Testing Case Studies Left and Right<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">You shouldn\u2019t copy A\/B testing strategies from other case studies. This is one of the very common A\/B testing mistakes people make. <strong>Your business is unique, so copying others won\u2019t bring the best results from your efforts.<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>How to avoid this A\/B testing mistake<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Analyze the case studies and derive inspiration and ideas to create an impeccable A\/B testing strategy tailored to YOUR business.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Closely analyze case studies to figure out what A\/B testing strategies others employed and for what purpose so that you can make them suitable for your business.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"5_Testing_Way_Too_Early\"><\/span>5. Testing Way Too Early&nbsp;<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/lh4.googleusercontent.com\/CxrinUUdYvx3iuCMlrIuIfs4CtWRfBfjDHaH1clftD-jvRL808doWaKv8eMInsDWCR981h5czo3kqjIkLVm4ftQnV5tAwRFmqIIOxWkoKtWjRDs6JD068bkPMBuBYmkFb-czM5_pFtesAAmmNzZWsuM\" alt=\"\"\/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">It\u2019s a thing, really. There is a right time to perform all tests, which goes for A\/B testing too. <strong>Don\u2019t just A\/B test for its sake; you need to have enough data to compare your testing results.&nbsp;<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">For example, you created a landing page for your new marketing campaign. No matter how excited you are to make the landing page more interactive and attractive to boost CTA clicks, don\u2019t start preparing for the testing just yet.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>How to avoid this A\/B testing problem<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">You need to wait for the right time until you have reliable data to draw a valid hypothesis and compare the results.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"6_Making_It_All_About_Conversions\"><\/span>6. Making It All About Conversions<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">You need to realize one thing: <strong>Conversion isn\u2019t the main focus of A\/B testing. So, if you are solely focusing on it, stop right now.&nbsp;<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Here is why: <\/strong>You might see an increase in conversions, but it may affect other areas of your business in the long term.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Let\u2019s absorb this A\/B testing mistake better with examples.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Example 1:<\/strong> If you add a new copy to your website, you may get a higher A\/B test conversion rate, but if the converted leads are of low quality, the higher conversion rate will not benefit the business.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">So, although you see more conversions now, it may not be as beneficial for your business in the longer run.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Example 2:<\/strong> Say you changed your annual plans to monthly plans during A\/B testing and instantly see a surge in conversions.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">But after a while, you&#8217;ll find you are losing money since the customers you are attracting are low-paying customers who tend to quit after a few uses. Whereas with annual plans, customers are more likely to stick around.<\/p>\n\n\n\n<hr class=\"wp-block-separator\"\/>\n\n\n\n<pre class=\"wp-block-verse\"><strong>&nbsp;\ud83d\udca1<\/strong> <strong>FUN FACT<\/strong>\nA study conducted by Profitwell on 941 SaaS companies concluded that a higher % of annual contracts and plans led to a lower churn rate.<\/pre>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/lh5.googleusercontent.com\/hUPfi-F1ZCjZUlCvNuKqquQdEhB7sUzG-ZtuCMfjzuoZRt4FoyDWR9PDxugYqibrYTzixruqPTqn_BnmpUG3YSsUiJmap_CWjfJoUY467wMpzgw6s0BaMetCzlQlm8ajp0-IJeXDJKXzZts0zhZ5FNw\" alt=\"\"\/><figcaption><a href=\"https:\/\/www.profitwell.com\/recur\/all\/why-annual-plans-are-crucial-for-reducing-your-churn\">Image Source<\/a><\/figcaption><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Mistakes Made During A\/B Testing<\/strong><\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"7_Split_Testing_Too_Many_Elements_at_Once\"><\/span>7. Split Testing Too Many Elements at Once<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/lh3.googleusercontent.com\/DMO5_9kESbhWiceFNUwlpRQVWUoSjCqWoXRO_hAW3mtTSRhDuRMz47XqoAH427pHmwuGf-PdT7v-yTX79ysI4P5k8niT6-VCECyqrcbB8IP3AAheI4NpTHN4inXa5aVL8XQWzt9UPLEvmzk3FBbF2e4\" alt=\"\"\/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Sometimes, people think it\u2019s wise to test variations of more than one element while A\/B testing so they can save time and resources.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Let us just put it out there: <strong>IT\u2019S NOT<\/strong>.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">We\u2019ll tell you why that is.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong><em>If you A\/B test multiple elements at once, you&#8217;ll have to create several variations <\/em><\/strong>(which is an error we will discuss right after). But that\u2019s not the worst part.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong><em>You won\u2019t be able to identify the root element for your results<\/em><\/strong>. What variation of what element suddenly increased the conversion rate? It irrevocably defeats the whole purpose of conducting A\/B split testing, and you&#8217;ll have to start from scratch.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>How to <\/strong><strong>solve this A\/B testing problem<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The way to bypass this A\/B testing mistake? <strong>Multivariate testing<\/strong>.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">You can modify multiple variables during multivariate testing and also track the performance of each variable. This way, you\u2019ll know which variables had the most impact during the testing.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"8_Creating_Several_Variations\"><\/span>8. Creating Several Variations<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">As we briefly discussed above, several variations for website split testing don\u2019t guarantee more valuable insights. <strong>If anything, they add to the confusion, slow down results, and risk false positives.<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">And the domino effect continues since the more variations you have, the more traffic you&#8217;ll require to get viable results. And to do this, you&#8217;ll have to run the test for a longer duration.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Now, the longer you run the test, more are your chances of getting exposed to cookie deletion.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong><em>\u201dWithin 2 weeks, you can get a 10% dropout of people deleting cookies, and that can really affect your sample quality.\u201d<\/em><\/strong><strong>\u2013 Ton Wesseling, Founder, Online Dialogue.<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">It\u2019s highly probable that participants will delete their cookies after 3 to 4 weeks, which is usually the duration of long-running tests.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Such a situation will adversely impact your result since the participants that were a part of a different variation may end up in a different one.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Another con of multiple variations is the decrease in the significance level. For instance, the accepted significance level is 0.05. Now, suppose you are running ten variations.&nbsp;<\/p>\n\n\n\n<pre class=\"wp-block-verse\">\ud83d\udca1<strong>FUN FACT<\/strong><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>At least one will be significant by chance out of these variations, i.e., 10*0.05<\/strong>. As the number of scenarios increases, the significance by chance will increase, resulting in a false positive.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">In 2009, Google tried to test several shades of blue to see which shade of blue generated more clicks on the search.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This A\/B test is notoriously known as \u201c<strong><em>Google\u2019s 41 shades of blue<\/em><\/strong>\u201d since they decided to test 41 shades (<em>here\u2019s your A\/B testing mistake<\/em>). The chance of false positives was a whopping 88%, with a 95% confidence level.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">If they had tested only ten shades, the chances of false positives would have dropped down to 40% and only 9% with three shades.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"9_Showing_Different_Variations_to_Different_Audiences\"><\/span>9. Showing Different Variations to Different Audiences&nbsp;<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Just like comparing Apples and Oranges doesn\u2019t make sense, comparing results of different variations from different audiences won\u2019t offer you anything tangible.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">To avoid this A\/B testing mistake, it would be best to always show the test variations to the same audience for comparable results.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">If you are showing a variation to a specific demographic, say only the traffic from the US, then the other variations should also be visible to the US audience only.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>How to avoid this A\/B testing mistake<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Have a common denominator for comparison.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"10_You_Got_the_Timing_Wrong\"><\/span>10. You Got the Timing Wrong<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/lh6.googleusercontent.com\/J3wBa3eYACiPXzGBP2OBpephCFiDjhJwe8VeWEchjcEGfiskmCcUyuYY_ranaj5ke0BT6-nEMr_ORFqgasvCC1r79kS3FPVIiv3AGLoe_HA5mS0C5rPfNIFIjQ6GkTsEyVYCO8oSjS1OxqYfd_NFkVE\" alt=\"\"\/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">There are multiple split testing mistakes related to timing, and we will get into each one to ensure you don\u2019t end up making them.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">As you know, timing plays a pivotal role in deciding the quality of the A\/B testing results. Here are the errors you should look out for:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li><strong><em>Concluding your test too early<\/em><\/strong><\/li><\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">Running tests for too long isn&#8217;t ideal, and it&#8217;s not beneficial to stop the tests too early.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>To achieve the industry-standard confidence rate of 95%, you need abundant data, which you can only get by running your A\/B tests for a considerable time such as a week at the least.<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list\"><li><strong><em>Comparison of different periods<\/em><\/strong><\/li><\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">If you want reliable results, you need results from a comparable period. For example, you cannot compare results from a seasonal boom period with regular days.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Another example is, say you see the most traffic on weekends. Then, you should only compare results from other weekends.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Also, say if you started running your test on Tuesday, you need to run your A\/B split testing for a whole week and end it on a Tuesday. This way, you&#8217;ll get the data for all the days, including the weekends.&nbsp;<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li><strong><em>You are testing different time delays<\/em><\/strong><\/li><\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">This A\/B testing mistake consists of trying different time delays at once on your test.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">For example, if you show your website visitor one variation after 5 seconds and the other after 15 seconds, the results are not comparable.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">It\u2019s because most customers generally stay longer than 5 seconds compared to 15 seconds. Thus, the results in this scenario are not the most accurate or reliable.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"11_Testing_with_the_Unsuitable_Traffic\"><\/span>11. Testing with the Unsuitable Traffic&nbsp;<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Along with ensuring you get the timing right, you also need the right traffic for your A\/B testing strategy to bear fruit.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">There\u2019s right traffic that is qualified and interested in your offerings and will buy from you and then there\u2019s wrong traffic that won\u2019t convert.&nbsp;&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>How to avoid this A\/B testing mistake<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Identify the right traffic and focus your A\/B tests on it.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">For example, you can segment the results into visitor types to check if the changes you made are actually helpful to your target audience.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"12_Altering_Parameters_in_the_Middle_of_Split_Testing\"><\/span>12. Altering Parameters in the Middle of Split Testing<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Changing parameters such as traffic allocation in the middle of a test is a recipe for unsuccessful A\/B testing and a big <strong>NO<\/strong>.&nbsp;<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/lh3.googleusercontent.com\/-nBL25pQ8yGu7fFm808hJvx-iPwTIIY7IZI8QiFo6szZWkVofIObPtDR03yhZmwzjJzmmwxSwRjjwyQF-4fg3IXEgxqEbw9HuGPNlr1KFpC7vy8TqRBHtDhkwWGkOGMQk-o5EGcyRnSwMos_lhwrWoA\" alt=\"\"\/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">For example, if a customer entered Variation A, they should see this variation for the entire test duration.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Changing parameters mid-test will lead this customer to see Variation B. Due to this A\/B testing mistake, the integrity of your data is compromised since this customer is involved in both tests.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Also, to give all your variations a fair chance, you need to distribute the traffic equally to get the most authentic results. Anything else besides this will only adversely affect your results.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">For example, say you divide the traffic as 70% to Control and 30% to variation. Ideally, this ratio should remain the same. But if you change it to say 50-50, then all the users will be mixed up.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Another parameter you shouldn\u2019t change is the variations themselves. Don\u2019t make further changes in the already running variations, making it impossible for you to find the cause for the results.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"13_Calling_It_a_Day_Too_Early_on_Your_Tests\"><\/span>13. Calling It a Day Too Early on Your Tests<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Going into something with the wrong expectations starts a series of mistakes. If you expect your first tests to knock it out of the park, we hate to break your bubble, but it\u2019s unlikely to happen.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Every big company performing website split testing didn\u2019t achieve the results in one go, <strong>and they certainly didn\u2019t give up on it<\/strong>.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">No test is a failed test since you always get data to learn something and do better with the next one.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>How to avoid this A\/B testing problem<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">If you don\u2019t feel good about your first tests, you can always launch another right when the first test ends.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">So, don\u2019t make this A\/B testing mistake and keep testing your page to further improve it. The more you test, the more you\u2019ll learn where you need to improve to boost conversions.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">For example, <a href=\"https:\/\/www.thetruckersreport.com\/\">TruckersReport<\/a> had to perform six rounds of A\/B testing to improve the conversions by<a href=\"https:\/\/cxl.com\/blog\/case-study-how-we-improved-landing-page-conversion\/\"> 79.3%<\/a> on their landing page.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"14_Ignoring_Periodic_Radical_Testing_for_Incremental_Changes\"><\/span>14. Ignoring Periodic Radical Testing for Incremental Changes<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Incremental changes consist of changing small elements on a page to enhance the customer experience and improve conversions. For example, changing the text of a CTA, the color of the CTA button, tweaking design elements, etc.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">A\/B testing is more than just focusing on incremental changes and one of the biggest A\/B testing mistakes is forgetting this.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Such changes may benefit large organizations with insanely high traffic, but changing a button&#8217;s color may not significantly contribute to most businesses.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">That\u2019s where <strong>Radical Testing<\/strong> comes into play, however rare. You only need periodic radical testing when incremental changes don\u2019t produce your desired results.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">For example, if you see a massive dip in the A\/B test conversion rate and the small changes aren\u2019t working, you may need a website overhaul to boost your conversions.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/lh6.googleusercontent.com\/kxoFfjG2QEfUjCfuC0uumibl_SqpKBKZm1hImqw07ZXQKbJPfdaPtjJtSKvbh_BVB-3viCKis2ip6rYZM53r4Q7cuprERvQZpYWqsjDrLZJLY9XuPIE2caQlyYztAc8lTelAN0NRnTJRIr2rvwvLQ5A\" alt=\"\"\/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">But it\u2019s not as easy as it may seem. There are a lot of factors to take into account.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong><em>Firstly, periodic radical testing requires considerable resources, investment, effort, and commitment.<\/em><\/strong><em> <\/em>It includes significant changes to your website, so you need to rethink and reconsider every decision to see significant results.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong><em>Secondly, radical testing makes it challenging to pinpoint which change brought positive results since you change many things at once.<\/em><\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Here are a few tips on how to create a hypothesis for your radical testing to bypass this A\/B testing mistake:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Challenge the current design. Instead of simply testing everything in your existing design, test elements based on hypothesis and challenge the existing assumptions.&nbsp;<\/li><li>Don\u2019t try to think on the behalf of your customers. The best way to know what needs improvement is to simply ask them. You can use online surveys to collect real-time and contextual feedback.<\/li><\/ul>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"15_Running_Multiple_Tests_Simultaneously\"><\/span>15. Running Multiple Tests Simultaneously<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Running multiple tests means simultaneously testing different pages (for instance, the home page and checkout page) with multiple versions (<em>Home page variations A and B and Checkout page variations A and B<\/em>).&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>How to avoid this A\/B testing problem<\/strong><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">You shouldn\u2019t test multiple tests with the majority of the traffic overlapping and also when and if the interactions are strong between the tests.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Mistakes Made After A\/B Testing<\/strong><\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"16_Using_an_Incompetent_AB_Testing_Tool\"><\/span>16. Using an Incompetent A\/B Testing Tool<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Did you know that even a second\u2019s delay in page load time decreases page views by <a href=\"https:\/\/www.forbes.com\/sites\/rogerdooley\/2012\/12\/04\/fast-sites\/?sh=1f808ec853cf\">11% and conversion by 7%<\/a>?&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Unfortunately, you may become a victim of these A\/B testing statistics if your A\/B testing tool isn\u2019t efficient.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Let us explain: A faulty split testing tool can also slow down your website and significantly affect the testing results. So, don\u2019t make this A\/B testing mistake and find a reliable tool to do the job.<\/p>\n\n\n\n<pre class=\"wp-block-verse\">\ud83d\udca1<strong>&nbsp;FUN FACT<\/strong><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">During one of his A\/B testing experiments, famous entrepreneur and marketing influencer <a href=\"https:\/\/neilpatel.com\/\">Neil Patel<\/a> discovered that the A\/B testing tool showed significant differences.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">However, he found no significant change in conversions upon implementing the new page. Later upon close inspection, it was deduced it was a faulty tool.<\/p>\n\n\n\n<hr class=\"wp-block-separator\"\/>\n\n\n\n<p class=\"wp-block-paragraph\">We understand how overwhelming it can be to dive into an ocean of A\/B testing tools to find the right pick for your business.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">For this reason, we meticulously created a detailed list of the <a href=\"https:\/\/qualaroo.com\/blog\/best-ab-testing-tools\/#How_to_Build_Your_AB_Testing_Hypothesis\">25 best A\/B testing tools<\/a> you can refer to and choose the one that suits your requirements.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"17_Not_Measuring_Results_Carefully\"><\/span>17. Not Measuring Results Carefully<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">It\u2019s remarkable if you didn\u2019t encounter any A\/B testing problems from above and ran successful testing.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">But don\u2019t celebrate just yet. Many A\/B testing mistakes are made at this stage while measuring and analyzing the A\/B testing results.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Once you have viable data, you need to analyze it the right way to benefit from the whole A\/B testing process.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Tools such as Google Analytics help here. To understand if your A\/B testing strategy worked, you can see the changes in conversions, bounce rate, CTA clicks, etc.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">If your tool shows averages, you can\u2019t be too sure about the data since <a href=\"https:\/\/hbr.org\/2002\/11\/the-flaw-of-averages\">averages are wrong all the time<\/a>.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Get a tool that lets you transfer your data to Google Analytics. You can use the Events and Custom Dimensions features to segregate the data and create custom reports for deeper analysis.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"18_Not_Understanding_Type_I_and_Type_II_Errors\"><\/span>18. Not Understanding Type I and Type II Errors<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">You may think you have done everything right, but these A\/B testing mistakes silently creep into your testing: Type I and Type II errors.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">So, you need to be on the lookout to detect them early in the process before they skew your results.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><strong><em>Type I error:<\/em><\/strong> This error is also known as <em>Alpha (\u03b1) error<\/em> and <strong><em>false positives<\/em><\/strong>. The tests seem to be working in this error, and the variations show results.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Unfortunately, these lifts are only temporary and won\u2019t last once you launch the winning variation universally for a significant period.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">For example, you may test some insignificant elements that may show positive results for the time being but won\u2019t get you tangible results.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img decoding=\"async\" src=\"https:\/\/lh6.googleusercontent.com\/n7BB_lu3PK3iLt7rD59UcCd-auIMpAc9w4aKBrVgefBUVm4FPJzhf5DRX1J4NKm4hzhTuFayjaZo_tXESQIQNCnqh0eLpgIppzHvmKRw1ISrT4wnxd08SzrSqlM9RAfL2IFB0LDSPxRSmfy0ZqNxp2w\" alt=\"\"\/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Type II error:<\/strong> Also known as <em>Beta (\u03b2) errors<\/em> or <strong><em>false negatives<\/em><\/strong>, it happens when a particular test seems inconclusive or unsuccessful, with the null hypothesis appearing true.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The null hypothesis here means a hypothesis you are trying to disprove but in Type II error, it appears true.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">For example, you conclude that certain elements you tested did not fetch positive results, but in reality, they did.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">In reality, the variation impacts the desired goal, but the results fail to show, and the evidence favors the null hypothesis.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">You, therefore, end up (incorrectly) accepting the null hypothesis and rejecting your hypothesis and variation.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"19_Not_Considering_Small_Wins\"><\/span>19. Not Considering Small Wins<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">If you think a 2% or 5% increase in conversion is insignificant, then look at it this way: These numbers are only from only one test.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Say you saw a 5% increase in conversion rate, but the cumulative annual conversion lift would be much more than this.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">These small gains are the reality of A\/B testing, which can translate into millions in revenue, however insignificant they seem. So ignoring them is one of the biggest A\/B testing mistakes you can make.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Major lifts such as 50% are only possible when you either perform periodic radical testing on a poorly designed website or overhaul it.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"20_Testing_Insignificant_Elements\"><\/span>20. Testing Insignificant Elements<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Changing a small element, such as the CTA button color, may not bring small companies or startups any significant results, which is a waste of time and resources.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">So, you need to analyze what elements are crucial and will bring significant results.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">For instance, instead of changing fonts on your product page, you may benefit more by adding more high-quality content and media.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">In conclusion, choose your elements for testing wisely. Here are some recommendations:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>The headline of your website; make it catchy and expressive of your brand.<\/li><li>Pricing; make it transparent and suitable for your target audience.<\/li><li>Call-to-action; make it stand out from other elements in the UI.<\/li><li>Product description; explain in detail the features of your product to make it easy for customers to make decisions.<\/li><li>Media such as images, videos, etc.<\/li><\/ul>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"21_Perform_AB_Testing_the_Right_Way\"><\/span>21. Perform A\/B Testing the Right Way<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">We hope you now have a little more clarity on the dos, don&#8217;ts, and A\/B testing mistakes you should avoid. Remember, it all starts with the customers and ends with them too.&nbsp;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">So, before basing your hypothesis on assumptions, make sure to listen to the <a href=\"https:\/\/qualaroo.com\/voice-of-customer\/\">voice of the customers<\/a> and analyze what changes will enhance their experience and, ultimately, your conversions.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><style>#sp-ea-15461 .spcollapsing { height: 0; overflow: hidden; transition-property: height;transition-duration: 300ms;}.eap_section_title_15461 { color: #444 !important; margin-bottom:  30px !important; }#sp-ea-15461.sp-easy-accordion>.sp-ea-single {border: 1px solid #e2e2e2; }#sp-ea-15461.sp-easy-accordion>.sp-ea-single>.ea-header a {color: #444;}#sp-ea-15461.sp-easy-accordion>.sp-ea-single>.sp-collapse>.ea-body {background: #fff; color: #444;}#sp-ea-15461.sp-easy-accordion>.sp-ea-single {background: #eee;}#sp-ea-15461.sp-easy-accordion>.sp-ea-single>.ea-header a .ea-expand-icon.fa { float: right; color: #444;font-size: 16px;}#sp-ea-15461.sp-easy-accordion>.sp-ea-single>.ea-header a .ea-expand-icon.fa {margin-right: 0;}<\/style><h2 class=\"eap_section_title eap_section_title_15461\"><span class=\"ez-toc-section\" id=\"Frequently_Asked_Questions\"><\/span> Frequently Asked Questions <span class=\"ez-toc-section-end\"><\/span><\/h2><div id=\"sp-ea-15461\" class=\"sp-ea-one sp-easy-accordion\" data-ex-icon=\"fa-angle-up\" data-col-icon=\"fa-angle-down\"  data-ea-active=\"ea-click\"  data-ea-mode=\"vertical\" data-preloader=\"\" data-scroll-active-item=\"\" data-offset-to-scroll=\"0\"><div class=\"ea-card ea-expand sp-ea-single\"><h3 class=\"ea-header\"><span class=\"ez-toc-section\" id=\"What_Is_AB_Testing\"><\/span><a class=\"collapsed\" data-sptoggle=\"spcollapse\" data-sptarget=#collapse154610 href=\"javascript:void(0)\"  aria-expanded=\"true\"><i class=\"ea-expand-icon fa fa-angle-up\"><\/i> What Is A\/B Testing?<\/a><span class=\"ez-toc-section-end\"><\/span><\/h3><div class=\"sp-collapse spcollapse collapsed show\" id=\"collapse154610\" data-parent=#sp-ea-15461><div class=\"ea-body\"><p><span style=\"font-weight: 400\">Split tests or A\/B testing is a process that lets\u00a0 you test an original web page called control against its slightly altered version called variation. It lets you test which one brings better results in conversions, click-through rates, engagement, etc.<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span style=\"font-weight: 400\">The A\/B testing strategy is very effective since you only change one specific element on the whole page. So, it becomes easy to analyze the results since any change will refer to the changed element.<\/span><\/p>\n<\/div><\/div><\/div><div class=\"ea-card  sp-ea-single\"><h3 class=\"ea-header\"><span class=\"ez-toc-section\" id=\"What_is_one_of_the_biggest_problems_with_AB_testing\"><\/span><a class=\"collapsed\" data-sptoggle=\"spcollapse\" data-sptarget=#collapse154611 href=\"javascript:void(0)\"  aria-expanded=\"false\"><i class=\"ea-expand-icon fa fa-angle-down\"><\/i> What is one of the biggest problems with A\/B testing?<\/a><span class=\"ez-toc-section-end\"><\/span><\/h3><div class=\"sp-collapse spcollapse \" id=\"collapse154611\" data-parent=#sp-ea-15461><div class=\"ea-body\"><p><span style=\"font-weight: 400\">People face many A\/B testing problems and make errors during testing, such as testing the wrong page and insignificant elements, running tests at the wrong time, creating too many variations, etc. <\/span><\/p>\n<\/div><\/div><\/div><div class=\"ea-card  sp-ea-single\"><h3 class=\"ea-header\"><span class=\"ez-toc-section\" id=\"What_is_one_of_the_common_mistakes_when_running_AB_tests\"><\/span><a class=\"collapsed\" data-sptoggle=\"spcollapse\" data-sptarget=#collapse154612 href=\"javascript:void(0)\"  aria-expanded=\"false\"><i class=\"ea-expand-icon fa fa-angle-down\"><\/i> What is one of the common mistakes when running A\/B tests?<\/a><span class=\"ez-toc-section-end\"><\/span><\/h3><div class=\"sp-collapse spcollapse \" id=\"collapse154612\" data-parent=#sp-ea-15461><div class=\"ea-body\"><p><span style=\"font-weight: 400\">Some of the most common mistakes in A\/B testing are: not testing on a live website but on a website under development, giving up too early on the test, testing multiple elements at once, etc.<\/span><\/p>\n<\/div><\/div><\/div><div class=\"ea-card  sp-ea-single\"><h3 class=\"ea-header\"><span class=\"ez-toc-section\" id=\"How_long_does_it_take_to_run_the_AB_test\"><\/span><a class=\"collapsed\" data-sptoggle=\"spcollapse\" data-sptarget=#collapse154613 href=\"javascript:void(0)\"  aria-expanded=\"false\"><i class=\"ea-expand-icon fa fa-angle-down\"><\/i> How long does it take to run the A\/B test?<\/a><span class=\"ez-toc-section-end\"><\/span><\/h3><div class=\"sp-collapse spcollapse \" id=\"collapse154613\" data-parent=#sp-ea-15461><div class=\"ea-body\"><p><span style=\"font-weight: 400\">The minimum recommended time for running an A\/B test is at least a week and a maximum of three to four weeks.<\/span><\/p>\n<\/div><\/div><\/div><div class=\"ea-card  sp-ea-single\"><h3 class=\"ea-header\"><span class=\"ez-toc-section\" id=\"What_is_one_of_the_common_mistakes_when_running_AB_tests-2\"><\/span><a class=\"collapsed\" data-sptoggle=\"spcollapse\" data-sptarget=#collapse154614 href=\"javascript:void(0)\"  aria-expanded=\"false\"><i class=\"ea-expand-icon fa fa-angle-down\"><\/i> What is one of the common mistakes when running AB tests?<\/a><span class=\"ez-toc-section-end\"><\/span><\/h3><div class=\"sp-collapse spcollapse \" id=\"collapse154614\" data-parent=#sp-ea-15461><div class=\"ea-body\"><p><span style=\"font-weight: 400\">One common AB testing mistake people make often is testing too many elements at the same time. It makes it harder to analyze the results and understand which element made the most impact.<\/span><\/p>\n<\/div><\/div><\/div><div class=\"ea-card  sp-ea-single\"><h3 class=\"ea-header\"><span class=\"ez-toc-section\" id=\"What_are_Type_2_errors_in_AB_testing\"><\/span><a class=\"collapsed\" data-sptoggle=\"spcollapse\" data-sptarget=#collapse154615 href=\"javascript:void(0)\"  aria-expanded=\"false\"><i class=\"ea-expand-icon fa fa-angle-down\"><\/i> What are Type 2 errors in AB testing?<\/a><span class=\"ez-toc-section-end\"><\/span><\/h3><div class=\"sp-collapse spcollapse \" id=\"collapse154615\" data-parent=#sp-ea-15461><div class=\"ea-body\"><p><span style=\"font-weight: 400\">In AB testing, a Type 2 error (also known as a false negative) occurs when you fail to reject the null hypothesis, even though the alternative hypothesis is true.\u00a0<\/span><\/p>\n<p>\u00a0<\/p>\n<p><span style=\"font-weight: 400\">In other words, it happens when you mistakenly conclude that there is no significant difference between the two variations being tested (A and B), when in fact, there is a difference.<\/span><\/p>\n<\/div><\/div><\/div><script type=\"application\/ld+json\">\n\t{\n\t  \"@context\": \"https:\/\/schema.org\",\n\t  \"@type\": \"FAQPage\",\n\t  \"mainEntity\": [{\n\t\t\t\"@type\": \"Question\",\n\t\t\t\"name\": \"What Is A\/B Testing?\",\n\t\t\t\"acceptedAnswer\": {\n\t\t\t  \"@type\": \"Answer\",\n\t\t\t  \"text\": \"Split tests or A\/B testing is a process that lets\u00a0 you test an original web page called control against its slightly altered version called variation. It lets you test which one brings better results in conversions, click-through rates, engagement, etc.\r\n\r\n\u00a0\r\n\r\nThe A\/B testing strategy is very effective since you only change one specific element on the whole page. So, it becomes easy to analyze the results since any change will refer to the changed element.\"\n\t\t\t}\n\t\t  },{\n\t\t\t\"@type\": \"Question\",\n\t\t\t\"name\": \"What is one of the biggest problems with A\/B testing?\",\n\t\t\t\"acceptedAnswer\": {\n\t\t\t  \"@type\": \"Answer\",\n\t\t\t  \"text\": \"People face many A\/B testing problems and make errors during testing, such as testing the wrong page and insignificant elements, running tests at the wrong time, creating too many variations, etc.\"\n\t\t\t}\n\t\t  },{\n\t\t\t\"@type\": \"Question\",\n\t\t\t\"name\": \"What is one of the common mistakes when running A\/B tests?\",\n\t\t\t\"acceptedAnswer\": {\n\t\t\t  \"@type\": \"Answer\",\n\t\t\t  \"text\": \"Some of the most common mistakes in A\/B testing are: not testing on a live website but on a website under development, giving up too early on the test, testing multiple elements at once, etc.\"\n\t\t\t}\n\t\t  },{\n\t\t\t\"@type\": \"Question\",\n\t\t\t\"name\": \"How long does it take to run the A\/B test?\",\n\t\t\t\"acceptedAnswer\": {\n\t\t\t  \"@type\": \"Answer\",\n\t\t\t  \"text\": \"The minimum recommended time for running an A\/B test is at least a week and a maximum of three to four weeks.\"\n\t\t\t}\n\t\t  },{\n\t\t\t\"@type\": \"Question\",\n\t\t\t\"name\": \"What is one of the common mistakes when running AB tests?\",\n\t\t\t\"acceptedAnswer\": {\n\t\t\t  \"@type\": \"Answer\",\n\t\t\t  \"text\": \"One common AB testing mistake people make often is testing too many elements at the same time. It makes it harder to analyze the results and understand which element made the most impact.\"\n\t\t\t}\n\t\t  },{\n\t\t\t\"@type\": \"Question\",\n\t\t\t\"name\": \"What are Type 2 errors in AB testing?\",\n\t\t\t\"acceptedAnswer\": {\n\t\t\t  \"@type\": \"Answer\",\n\t\t\t  \"text\": \"In AB testing, a Type 2 error (also known as a false negative) occurs when you fail to reject the null hypothesis, even though the alternative hypothesis is true.\u00a0\r\n\r\n\u00a0\r\n\r\nIn other words, it happens when you mistakenly conclude that there is no significant difference between the two variations being tested (A and B), when in fact, there is a difference.\"\n\t\t\t}\n\t\t  }]\n\t}\n\t<\/script><\/div><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Tired of making A\/B testing mistakes? Learn about these 20 common mistakes in A\/B testing &amp; how to avoid them to enjoy reliable results for your hypotheses.<\/p>\n","protected":false},"author":21,"featured_media":15606,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[738],"tags":[],"class_list":["post-9859","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ab-testing"],"_links":{"self":[{"href":"https:\/\/web-staging.qualaroo.com\/blog\/wp-json\/wp\/v2\/posts\/9859","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/web-staging.qualaroo.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/web-staging.qualaroo.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/web-staging.qualaroo.com\/blog\/wp-json\/wp\/v2\/users\/21"}],"replies":[{"embeddable":true,"href":"https:\/\/web-staging.qualaroo.com\/blog\/wp-json\/wp\/v2\/comments?post=9859"}],"version-history":[{"count":122,"href":"https:\/\/web-staging.qualaroo.com\/blog\/wp-json\/wp\/v2\/posts\/9859\/revisions"}],"predecessor-version":[{"id":22611,"href":"https:\/\/web-staging.qualaroo.com\/blog\/wp-json\/wp\/v2\/posts\/9859\/revisions\/22611"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/web-staging.qualaroo.com\/blog\/wp-json\/wp\/v2\/media\/15606"}],"wp:attachment":[{"href":"https:\/\/web-staging.qualaroo.com\/blog\/wp-json\/wp\/v2\/media?parent=9859"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/web-staging.qualaroo.com\/blog\/wp-json\/wp\/v2\/categories?post=9859"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/web-staging.qualaroo.com\/blog\/wp-json\/wp\/v2\/tags?post=9859"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}