Bob Bly Direct Response Copywriter Official Banner

In Praise of A/B Split Testing

December 9th, 2009 by Bob Bly

Despite all the sophisticated analytics and multi-variate testing tools at our disposal, sometimes the quickest and clearest way to answer a marketing question is with a good old-fashioned A/B split test.

For an online ad to build my online subscriber list, we tested two headlines:

A–“Earn $2,000 per week ‘working’ an hour a day”
B–“Earn $100,000 a year selling simple e-books online”

One outpulled the other by 25%.

Which do you think was the winner — and why?

Share

This entry was posted on Wednesday, December 9th, 2009 at 9:50 am and is filed under General. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

88 responses about “In Praise of A/B Split Testing”

  1. Twitter Trackbacks for In Praise of A/B Split Testing - bly.com blog - bly.com direct marketing blog [bly.com] on Topsy.com said:

    […] In Praise of A/B Split Testing – bly.com blog – bly.com direct marketing blog bly.com/blog/general/in-praise-of-ab-split-testing – view page – cached Despite all the sophisticated analytics and multi-variate testing tools at our disposal, sometimes the quickest and clearest way to answer a marketing question is with a good old-fashioned A/B split… Read moreDespite all the sophisticated analytics and multi-variate testing tools at our disposal, sometimes the quickest and clearest way to answer a marketing question is with a good old-fashioned A/B split test. View page […]

  2. Henry said:

    It’s usually stupid to guess which is going to do better… but I’ll do it anyway.

    I would guess Headline A outpulled B.

    $2,000 sounds more realistic even though it’s slightly more than $100K a year.

    Also, the trivializing of ‘work’ might push results toward A.

  3. Debra Dalgleish said:

    I’d guess A because the time period is shorter. We want results now, not a year from now.

  4. JB said:

    I would say that B worked better because:
    – it is specific about how the money can be made (selling e-books);
    – it sounds more believable than A, — the one week time frame in A is too short and a lead up period would be needed to make $2000/wk;
    – the words in B flow better.

  5. Bill Perry said:

    I would say A, like Henry and Debra, because it’s a smaller chunk of time and it’s also closer to the “now”.

  6. Eric S. Mueller said:

    I’d say B. The first one sounds like a scam.

  7. Gerrid Smith said:

    I agree. B is more believable and A sounds like a scam.

  8. Coach Donnelly said:

    I’d be shocked if B hadn’t performed better.

    There’s just something about “$100,000” and “simple” being used in the headline that sounds better than “working” and “$2,000”. Plus, I agree with the others about it sounding more believable.

    Having said that, I think B could perform EVEN BETTER if you replaced “Earn” with “How to Make”.

    Coach Donnelly

  9. D Michael Miller said:

    I think A pulled better. $ 2000 sounds more believable than $ 100000, even though one is in a week and the other in a year. Also, it is not specific. One has to look into it more to find out how to make $ 2000/week.

  10. Maine Copywriter said:

    I’d guess B because people are greedy. It’s more specific. And nobody likes to “work.”

  11. John W. Furst said:

    Bob,
    I like those “challenges”. I made a mistake but I already have read the previous comments. Now I am biased. So I asked my wife (she’s an artist). She believes A is the winner.

    What could be the conversations in the prospects mind?

    * $2,000 is just what I could use right now.
    * ‘work’ sounds easy, someone tells me what to do, and I do it.
    * might be a scam but I check it out anyway. maybe it’s for real.
    * “selling”? No, I won’t do that.

    * $100,000 a year selling something … sounds like a business plan.

    Of course, they don’t see the two ads next to each other. Someone with a job might go with A (if hasn’t been ‘burned’ before). Someone with a business mindset might go for B.

    A lot depends on actual source/quality of traffic.

    Can’t wait for the solution.

    Yours
    John

  12. Curtis said:

    My vote is for B. First of all, six figures is the holy grail for a lot of folks – so $100,000 carries a lot of emotion.

    So does the lack of the word ‘working’ in the headline. People say they’re willing to work for something. But, emotion rules. They want big money with the promise of little effort.

    Finally, “simple” ebooks adds to what I just mentioned above.

  13. Steve said:

    I’m surprised at the variety of responses. It has to be A. People want to know more about how to earn well without working very hard. Selling e-books to make 100K it is not clear how much work will have to be done to get there, but it’s not difficult for anyone to imagine that result with the right amount of effort.

    However 2K per week for one hour per day says all that needs to be said to those surfing for opportunity. slr

  14. Bob Bly said:

    A is the winner, and I think the reasons Steve cites above may be the correct ones.

  15. Coach Donnelly said:

    I’m shocked that A performed better and would be interested to know what kind of audience it was being advertised to.

    Personally, I see so many “make money on the Net” ads that B stands out much more as something more compelling, something more specific.

    I guess at the same time, being specific and telling people they’ll actually have to create an ebook may sound like too much work for the average Joe who sees a random ad.

    Coach Donnelly

  16. Joe said:

    I’m betting A because most people want instant gratification and $2,000 a week sounds like a quicker turn-around.

  17. Glenn Murray said:

    I’d say A was the winner because:
    1) As mentioned by a couple of commenters, it’s more immediate. Weeks and days, not years.
    2) It’s more attainable. $2000 is easier to achieve than $100k.
    3) It appeals even to people who think they can’t write or sell or work online.
    4) It’s shorter.
    5) As Henry said, it trivializes the ‘work’ involved.
    6) Also agree with Steve’s comment above: option A discusses the commitment required.

    Cheers.
    Glenn

  18. Jay Hepner said:

    I’m voting A. The quotation marks framing “working” and the need only to work “an hour a day” create a cantilever of force and reinforcement hard to pass by.

    Also, e-books don’t seem simple to many who grew up with books “classic.” There’s all that technology to master, and the non-tangiblity of the ebook, tends to further distance a good number of potential prospects from the task, and therefore from the offer.

    “Working” only “an hour a day,” there’s language that’s universally appealing.

  19. Ken Norkin - Freelance Copywriter said:

    B was the loser because it tells too much. More to the point, it tells people it’s promoting info on how to do something they probably don’t want to learn about or do — sell e-books — regardless of how much money they might make doing it.

    A, on the other hand, promises info on how to make a lot of money for very little work. Yes, it smacks of get rich quick scamminess as some have said. But those who are curious will at least respond to the ad to learn what it’s about.

    While A outpulled B, it wouldn’t surprise me if B had a higher conversion rate, since the people who responded were predisposed to learning about selling e-books. But I’ve been wrong in these kinds of guesses before.

  20. www.outlet-apparel.com said:

    We Factory wholesale Cheap Nike Shoes,Authentic Jordan sneakers, Nike Dunk,Air Force one, Jordan Fusions, Nike Air Yeezy, Nike ATO shoes, Supra Sneakers, Mauri shoes, Gucci shoes, Prada shoes, D&G, Coach, Puma Trainers, UGG, Timberland boots, Adidas, Chanel shoes, Nike Womens shoes, Nike Kids shoes, and carry some pop Hoodies, LRG, A&F, Affliction, Ed Hardy Jackets, BAPE BBC, Evisu, COOGI, True Religion Jeans, Christian Audigier, polo, Lacoste,
    Shirts,New ERA Hats, LV, COACH, CHANEL Handbag…etc, feel free to contact me at any time.
    Free shipping: (EMS/TNT/FEDEX/UPS/DHL).

    website: http://www.outlet-apparel.com

  21. Lose The Back Pain System – $46.45 Avg Payout. at Free Anger Management said:

    […] In Praise of A/B Split Testing – bly.com blog – bly.com direct … […]

  22. Èn ændring = 196% bedre konvertering said:

    […] In Praise of A/B Split Testing – bly.com blog – bly.com direct … […]

  23. Annie Daniel said:

    I go for “B” 100k yearly by simply selling ebooks online. It’s far better than the need of working an hour daily in order to earn 2k per week. Though, the choice “A” will result to much earning, it’s still tolerable cause you’ll only missed less than 5%.

  24. Ryan said:

    “A” is probably the winner because I would tend to go for “bite size chunks” than something I would have to build for a year.

  25. D Bnonn Tennant said:

    So are you gonna put us out of our misery and tell us, Bob…or leave us hanging? (:

  26. Nan Wojtkowski said:

    Superior material included in content material may make upwards to find the top e-book

  27. Latarsha Galer said:

    Hello there business owner! I located your weblog employing Google so when google search optimizer

  28. Jeanna Ratering said:

    We just couldn’t abandon your world wide web site ahead of proclaiming that we really loved your beneficial facts you provide within your guests… Will most likely be back often to check on by way of to new posts

  29. Brice Biegler said:

    Your blog is fairly wonderful, I like your articles, Its really help me to.

  30. Yong Maassen said:

    We appreciate you the straightforward, informative and simple to grasp write-up Savings

  31. Otis Garfield said:

    Great line accessible. We hardly ever at any time permit any normal shut withouth receiving not too long ago released information making use of this reference web site.

  32. Tangela Villenas said:

    Outstanding weblog below! In addition your web site tons upward very quickly! Just what number do you assume you happen to be utilizing?

  33. Kimber Zenisek said:

    I really need, understanding!

  34. Margret Baratto said:

    Like a web website owner I believe the data the following is reallyfantastic. I be thankful for the efforts. You ought to continue the very good work as soon as and for all! Good luck.

  35. gestational diabetes said:

    I believe that you know producing males and women hear whatever you have to state, specifically with an issue i am talking about so important

  36. custom cowboy boots said:

    Numerous thanks expressing your notions about this site.

  37. d3dcompiler_43.dll is missing said:

    Spot on with this write-up, I actually think this amazing site needs a great deal more attention. I’ll probably be returning to read more, thanks for the information!

  38. Jake Harper said:

    I think it has to be B cause it may not be actually possible to earn that much on the go.Software Testing Company

  39. lara said:

    nice
    top restaurant
    lung experts doctors
    advertising

  40. One Dasty said:

    Bringing home a new puppy is a wonderful experience involving the whole family, which is why we begin training early to ease the transition to their new home and family. Owning a pug puppy or French Bulldog puppy is hard work. Making make it less stressful. To provide a completely different environment for our puppies, we thoroughly researched and make sure this puppies will only go to respectable families
    https://pugslandbreeder.company.com

  41. robert said:

    https://savannahbengal.company.com/

  42. Ronzoni said:

    mouse for video editing Buy for best mouse for video editing. BEST MOUSE FOR VIDEO EDITING ON ADOBE PREMIERE PRO WITH MAC + JOG SHUTTLE WHEEL CONTROLLER.
    mouse for video editing

  43. George said:

    Thanks for sharing. Your efforts made this very easy to find
    See Ali Pur Ka Aili in PDF
    Click Here for Bachpan Ka December And
    Yaaram Novel Download
    The novel Mushaf by Nemrah Ahmed is very interesting

  44. dekhasia said:

    TamilDhool Cool Collection of Tamil Serials & Shows. TamilDhool Watch Tamil TV Serials and TV Shows Free Online,

    tamil dhool

  45. blogspot said:

    Tamilblasters is a piracy platform. Tamilblasters is an infamous piracy website offering an extensive collection of Tamil and Telugu tv show serial to its viewers for free.
    vijay tv tamildhool

  46. dekhasia.blogspot.com said:

    Sun Tv – TamilDhool Cool Collection of Tamil Serials & Shows the latest promo of popular Tamil Serial Sundari that airs on Sun TV. sun tv serial

  47. blogspot.com said:

    Watch Vijay TV Serials Online Watch latest and full episodes of your favourite tamildhool serial
    Star Vijay TV shows online on .

  48. google said:

    Watch Sembaruthi serial tamildhool zee tamil
    today’s episode, popular and all latest episodes, promos, videos and get complete details Zee Tamil is an Indian pay television Tamil.

  49. Ndure Shoes said:

    Ndure offers their customers jackets of all types, including Denim Jackets For Women, leather jackets, puffer jackets, and Sherpa denim jackets. Buy the best available winter blue jeans Jackets for women online in Pakistan from Ndure. We bring you high quality bomber jacket, Puffer Hooded Gilet for womens online.

  50. Heacox said:

    Watch latest Tamil tv show episodes on Zee5, including Tamil comedy, romance, thriller, horror & family shows and more.
    Watch your favourite shows from Star Plus, Star World, Life OK, Star Jalsha, Star Vijay, Star Pravah, Asianet, Maa TV & more online on Disney+ Hotstar.
    dekhasia
    dekhasia

  51. Prisoc said:

    Tamildhool is a video streaming website that offers more than 50 original shows and over 50,000 hours of Premium Content from leading Producers and Publishers.
    tamildhool
    tamil dhool

  52. Vandewalker said:

    ver doramas online DoramasVip Aqui podras encontrar doramas de todo el mundo traducidos al español latino Gratis DoramasVip HD doramasmp4.

    Ver en Doramasmp4 doramas gratis en emision y finalizados, capitulos completos, aqui Doramasmp4 encontraras muchos doramas, peliculas de asia gratis doramas vip

  53. Schoeb said:

    Ver online doramas coreanas, doramas estrenos, Tailandia, Japones, Estrenos Doramas con subtítulos en español gratis.

    Ver Doramas Online en Doramasflix – doramasflix en HD Doramas y Películas Sub Español , Dramas Coreanos?, Chinos, Tailandia y Japones en Doramasflix -.

  54. murtazaabbas said:

    The full type of the UTR number is a Unique Taxpayer Reference number. Given by HMRC (HM Revenue and Custom), the principle utilization of an UTR number is given to people who satisfy the expense self-evaluation and furthermore to the organizations.

  55. murtazagrn said:

    10 seconds determines a ‘knockout’ in boxing. A KO is something the likes of Tyson Fury would rather give than receive. During a fight, when a boxer falls to the floor the referee must count to 10 to determine whether they can get back up to continue. If not, their opponent wins by knockout.

  56. murtazaodint said:

    As an outsider in Singapore you can rapidly lose all sense of direction in an ocean of monetary promoting none of which might appear to be natural to you. Assuming you’re accustomed to observing ATMs from global names like Citibank, HSBC, or American Express, you’ll see that these banks have next to no presence here. Rather the best banks for outsiders in Singapore are the ones that are neighborhood but then appropriate for worldwide installments and movess
    Too read more click banks in singapore for foreigners

  57. murtazaodint said:

    The Dutch commanditaire vennootschap, or confined affiliation, is undefined from a VOF (general association) association in Netherlands. Taking everything into account, there are two kinds of colleagues: a supervising assistant who runs the association consistently and a bound or calm accessory who goes probably as a supporter. Confined associates don’t need to divulge perceiving information while spreading out a Commanditaire Vennootschapwith the KvK; the amount of calm assistants and irrefutably the capital they supply is reasonable

  58. dekhasia said:

    Watch Tamil TV Serials and TV Shows Free Online, TamilDhool • Cool <a href="colorstamil of Tamil Serials & Shows.
    Watch Tamil shows and tamil Serial Online at TamilTwist, Latest shows of vijay tv, sun tv, colors tv, zee tv and more. Check out Tamil Serials Online Today zeetamilserial.

  59. lelywa said:

    DoramasFlix
    great site for doramas in spanish sub.today i visit this site i found all doramas with latest capitulos. visit this site by clicking top or bellow button of comment and enjoyed doramas like chinese, japanese, corien, latinies and more many.
    https://doramasflixx.net

  60. boxuguka said:

    DoramasFlix
    is best for all doramas in spanish sub. visit and enjoyed all doramas with latest capitulos
    https://doramasflixx.com

  61. Vijay Tv said:

    Watch all Sun Tv Serials FREE Abhiyum Nanum , Roja , Vaanathai Pola , Poove Unakkaga , Kannana Kanne, Anbe Vaa , Chithi 2 , Thirumagal , Paandavar Illam , Maharasi , Kayal are the prime time

  62. Ghana Music Mp3 said:

    Tech and Sports News Ghana Music MP3 Download Naija Music

  63. download songs said:

    Exclusive Song: Maziyar | Iran Iran With Text And Direct Links In UpMusic

  64. qarim said:

    Aliha – “Mishe Faghat” > Old Song > Download With Text And 320 & 128 Links In Musico

  65. musicfa said:

    Exclusive Song: Maziyar | Iran Iran With Text And Direct Links In UpMusic

    Exclusive Song: Maziyar | Iran Iran With Text And Direct Links In UpMusic

  66. ????? ???? ??????? ?? ?? ????? said:

    ????? ??????? ?? ????? ?? ???? ??????? ?????? ?? ??? ????? ??? ???. ?? ???? ??????? ????? ???? ??????? ?? ?? ????? ???? ??? ????? ????? ??? ?? ?? ????????? ? ??????? ?????? ???? ?? ?????? ????? ?????? ?? ?????. ?????????? ???? ???????? ?? ??? ???? ?? ??? ????? ???????… ?? ???? ?? ??? ???? ??????. ?? ??? ???? ?? ?? ???? ???? ? ???? ???? ??????? ??? ? ??? ?? ??????? ???????? ????????? ???? ?? ????? ?????? ?? ????? ???. ?????? ???? ???? ?? ????? ????.

  67. factorywebdesign said:

    Manufacturing is not usually considered a digital industry. Historically, industrial companies allocated their budget to traditional offline marketing and advertising in industrial publications. Websites think they’re considered one dimensional…or worse, they don’t exist. With this, a basic and digitally focused website is one of the most important marketing tools for a manufacturing or industrial business. Let’s examine four reasons.

  68. Mike Rooney said:

    Awesome blog. I have read and thought about what you said. I have bookmarked it and I am looking forward to reading new articles. Keep up the good work!
    Parka Jacket Kevin Costner Season 5

  69. jonix said:

    Series y Telenovelas de Telemundo Online capitulos completos, tusmundo Capitulos nuevos de tu telenovela favorita,

  70. Highway Fashions said:

    Are you in search of a New Design Dress in Pakistan? Highway fashions offer a large variety of women’s dresses online in Pakistan. So, order your favorite dress today and enjoy.

  71. Royal Leafs said:

    If you are looking for the most appropriate hamilton marijuana store, then you are at a suitable place. Royalleafs is a premier cannabis store in Hamilton. You can find online plenty of Cannabis products such as Cannabis pre-rolls, raw and milled flowers, drinks, and accessories such as bongs and vapes of the highest quality at Royalleafs.

  72. Borjan said:

    Borjan offers the best quality shoes for men online in Pakistan. We ensure all our products are of the highest standards, emphasizing quality and style that is perfect for your hard-working lifestyle.

  73. Jazmin Clothing said:

    Jazmin Clothing presents an exquisite collection of lawn dress designs that capture the essence of grace and glamour. Choose from a variety of colors and styles that cater to every taste and occasion. Order your favorite lawn dress today.

  74. muhamad tayyab said:

    TV Host, VJ, Singer, Actor, Politician, Social Media Influencer, Entrepreneur, and Philanthropist. Waqar zakka also won many awards.

  75. Norma said:

    Mackenzie Media Group works like Specializing in SEO, website design, content creation and advertisement.

  76. dfsfsf said:

    Thank you for the good writeup. It in fact used to be.a amusement account it. Look complicated to far added agreeable from.you! By the way, how can we communicate? dating sites

  77. adad said:

    I have bookmarked your website because this site contains valuable information in it. I am really happy with articles quality and presentation. Thanks a lot for keeping great stuff. I am very much thankful for this site . Pretty good post. I just stumbled upon your blog and wanted to say that I have really enjoyed reading your blog posts. Any way I’ll be subscribing to your feed and I hope you post again soon. I think I have never seen such blogs ever before that has complete things with all details which I want. So kindly update this ever for us .

  78. ???? ???????? said:

    This was really an interesting topic and I kinda agree with what you h
    .???? ????????

  79. sdbdsb said:

    more. ??

  80. fsdff said:

    I am looking for new posts to get more precious info. Big thanks for the useful info.

  81. asvd said:

    work. ?????????

  82. ???? said:

    Nice post. I find out something very complicated on diverse blogs everyday. It will always be stimulating you just read content off their writers and practice a little there. I’d would rather use some using the content on my small blog regardless of whether you do not mind. Natually I’ll supply you with a link for your web weblog. Appreciate your sharing. ????

  83. ???? ??????? said:

    PINGU and his friend Robby get up to all sorts of mischief during a visit to the Art Museum to the annoyance of the museum curator who seeks to reprimand the two rascals. However every time he tries to rectify the damage he manages to make it worse. ???? ???????

  84. Sahil Rb said:

    Technology business refers to any company that specializes in the development, production, and sale of technology-based products or services. This can include hardware, software, telecommunications, e-commerce, artificial intelligence, robotics, and many other areas. Technology businesses often require a significant amount of investment in research and development, as well as skilled personnel to design, engineer, and manufacture their products. These companies also often require a strong marketing and sales team to effectively promote and sell their products to consumers or other businesses.

    The technology industry is constantly evolving, with new products and services being developed and released on a regular basis. This can make it both exciting and challenging for technology businesses to stay competitive in the market. Successful technology businesses are often those that are able to stay ahead of the curve by investing in emerging technologies, adapting quickly to changing market conditions, and staying focused on meeting the needs of their customers.

  85. Tusmundo Series y Novelas Online said:

    Disfruta de los contenidos de Tusmundo Novelas por ViX. Las mejores series, novelas y películas en Español y gratis.

  86. toni eka saputra said:

    ??i nh? c? h?i trúng l?n, nào, ??ng b? l? c? h?i ngàn vàng này, hãy th? ch?i slot ?ang ch?i nhi?u nhân l?n ngay hôm nay, link ??ng ký : https://www.asia99.org/

  87. Novelastv said:

    In Praise of A/B Split Testing is a powerful technique that revolutionizes decision-making and optimization in various industries. A/B Split Testing involves creating two or more versions of a webpage, advertisement, or other elements and presenting them to different audiences simultaneously. By collecting and analyzing data on user behavior and preferences, businesses can objectively determine which version performs better and yields superior results. This data-driven approach enables companies to make informed decisions, refine their strategies, and optimize their marketing efforts for maximum effectiveness. A/B Split Testing provides invaluable insights into customer preferences, helps identify the most impactful design elements, content variations, or call-to-action strategies, and ultimately leads to higher conversion rates, improved user experiences, and increased ROI. It empowers businesses to continuously iterate and refine their marketing tactics, ensuring continuous growth and success in today’s competitive landscape.

  88. El Millón said:

    Informaciones sobre el juego de El Millón de Euromillones con los códigos ganadores. Tablas de premios de El Millón todos los martes y viernes.

Leave a Reply