In Praise of A/B Split Testing
December 9th, 2009 by Bob Bly
Despite all the sophisticated analytics and multi-variate testing tools at our disposal, sometimes the quickest and clearest way to answer a marketing question is with a good old-fashioned A/B split test.
For an online ad to build my online subscriber list, we tested two headlines:
A–“Earn $2,000 per week ‘working’ an hour a day”
B–“Earn $100,000 a year selling simple e-books online”
One outpulled the other by 25%.
Which do you think was the winner — and why?
This entry was posted on Wednesday, December 9th, 2009 at 9:50 am and is filed under General. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.
December 9th, 2009 at 9:53 am
[…] In Praise of A/B Split Testing – bly.com blog – bly.com direct marketing blog bly.com/blog/general/in-praise-of-ab-split-testing – view page – cached Despite all the sophisticated analytics and multi-variate testing tools at our disposal, sometimes the quickest and clearest way to answer a marketing question is with a good old-fashioned A/B split… Read moreDespite all the sophisticated analytics and multi-variate testing tools at our disposal, sometimes the quickest and clearest way to answer a marketing question is with a good old-fashioned A/B split test. View page […]
December 9th, 2009 at 10:16 am
It’s usually stupid to guess which is going to do better… but I’ll do it anyway.
I would guess Headline A outpulled B.
$2,000 sounds more realistic even though it’s slightly more than $100K a year.
Also, the trivializing of ‘work’ might push results toward A.
December 9th, 2009 at 10:37 am
I’d guess A because the time period is shorter. We want results now, not a year from now.
December 9th, 2009 at 11:12 am
I would say that B worked better because:
– it is specific about how the money can be made (selling e-books);
– it sounds more believable than A, — the one week time frame in A is too short and a lead up period would be needed to make $2000/wk;
– the words in B flow better.
December 9th, 2009 at 11:46 am
I would say A, like Henry and Debra, because it’s a smaller chunk of time and it’s also closer to the “now”.
December 9th, 2009 at 12:36 pm
I’d say B. The first one sounds like a scam.
December 9th, 2009 at 3:51 pm
I agree. B is more believable and A sounds like a scam.
December 9th, 2009 at 6:14 pm
I’d be shocked if B hadn’t performed better.
There’s just something about “$100,000” and “simple” being used in the headline that sounds better than “working” and “$2,000”. Plus, I agree with the others about it sounding more believable.
Having said that, I think B could perform EVEN BETTER if you replaced “Earn” with “How to Make”.
Coach Donnelly
December 9th, 2009 at 6:29 pm
I think A pulled better. $ 2000 sounds more believable than $ 100000, even though one is in a week and the other in a year. Also, it is not specific. One has to look into it more to find out how to make $ 2000/week.
December 9th, 2009 at 9:15 pm
I’d guess B because people are greedy. It’s more specific. And nobody likes to “work.”
December 10th, 2009 at 6:38 am
Bob,
I like those “challenges”. I made a mistake but I already have read the previous comments. Now I am biased. So I asked my wife (she’s an artist). She believes A is the winner.
What could be the conversations in the prospects mind?
* $2,000 is just what I could use right now.
* ‘work’ sounds easy, someone tells me what to do, and I do it.
* might be a scam but I check it out anyway. maybe it’s for real.
* “selling”? No, I won’t do that.
* $100,000 a year selling something … sounds like a business plan.
Of course, they don’t see the two ads next to each other. Someone with a job might go with A (if hasn’t been ‘burned’ before). Someone with a business mindset might go for B.
A lot depends on actual source/quality of traffic.
Can’t wait for the solution.
Yours
John
December 10th, 2009 at 12:21 pm
My vote is for B. First of all, six figures is the holy grail for a lot of folks – so $100,000 carries a lot of emotion.
So does the lack of the word ‘working’ in the headline. People say they’re willing to work for something. But, emotion rules. They want big money with the promise of little effort.
Finally, “simple” ebooks adds to what I just mentioned above.
December 10th, 2009 at 12:58 pm
I’m surprised at the variety of responses. It has to be A. People want to know more about how to earn well without working very hard. Selling e-books to make 100K it is not clear how much work will have to be done to get there, but it’s not difficult for anyone to imagine that result with the right amount of effort.
However 2K per week for one hour per day says all that needs to be said to those surfing for opportunity. slr
December 10th, 2009 at 1:59 pm
A is the winner, and I think the reasons Steve cites above may be the correct ones.
December 11th, 2009 at 3:15 pm
I’m shocked that A performed better and would be interested to know what kind of audience it was being advertised to.
Personally, I see so many “make money on the Net” ads that B stands out much more as something more compelling, something more specific.
I guess at the same time, being specific and telling people they’ll actually have to create an ebook may sound like too much work for the average Joe who sees a random ad.
Coach Donnelly
December 11th, 2009 at 11:31 pm
I’m betting A because most people want instant gratification and $2,000 a week sounds like a quicker turn-around.
December 13th, 2009 at 6:22 pm
I’d say A was the winner because:
1) As mentioned by a couple of commenters, it’s more immediate. Weeks and days, not years.
2) It’s more attainable. $2000 is easier to achieve than $100k.
3) It appeals even to people who think they can’t write or sell or work online.
4) It’s shorter.
5) As Henry said, it trivializes the ‘work’ involved.
6) Also agree with Steve’s comment above: option A discusses the commitment required.
Cheers.
Glenn
December 16th, 2009 at 6:34 am
I’m voting A. The quotation marks framing “working” and the need only to work “an hour a day” create a cantilever of force and reinforcement hard to pass by.
Also, e-books don’t seem simple to many who grew up with books “classic.” There’s all that technology to master, and the non-tangiblity of the ebook, tends to further distance a good number of potential prospects from the task, and therefore from the offer.
“Working” only “an hour a day,” there’s language that’s universally appealing.
December 18th, 2009 at 12:12 am
B was the loser because it tells too much. More to the point, it tells people it’s promoting info on how to do something they probably don’t want to learn about or do — sell e-books — regardless of how much money they might make doing it.
A, on the other hand, promises info on how to make a lot of money for very little work. Yes, it smacks of get rich quick scamminess as some have said. But those who are curious will at least respond to the ad to learn what it’s about.
While A outpulled B, it wouldn’t surprise me if B had a higher conversion rate, since the people who responded were predisposed to learning about selling e-books. But I’ve been wrong in these kinds of guesses before.
December 28th, 2009 at 3:40 am
We Factory wholesale Cheap Nike Shoes,Authentic Jordan sneakers, Nike Dunk,Air Force one, Jordan Fusions, Nike Air Yeezy, Nike ATO shoes, Supra Sneakers, Mauri shoes, Gucci shoes, Prada shoes, D&G, Coach, Puma Trainers, UGG, Timberland boots, Adidas, Chanel shoes, Nike Womens shoes, Nike Kids shoes, and carry some pop Hoodies, LRG, A&F, Affliction, Ed Hardy Jackets, BAPE BBC, Evisu, COOGI, True Religion Jeans, Christian Audigier, polo, Lacoste,
Shirts,New ERA Hats, LV, COACH, CHANEL Handbag…etc, feel free to contact me at any time.
Free shipping: (EMS/TNT/FEDEX/UPS/DHL).
website: http://www.outlet-apparel.com
January 6th, 2010 at 1:57 pm
[…] In Praise of A/B Split Testing – bly.com blog – bly.com direct … […]
January 15th, 2010 at 9:22 pm
[…] In Praise of A/B Split Testing – bly.com blog – bly.com direct … […]
February 6th, 2010 at 11:09 pm
I go for “B” 100k yearly by simply selling ebooks online. It’s far better than the need of working an hour daily in order to earn 2k per week. Though, the choice “A” will result to much earning, it’s still tolerable cause you’ll only missed less than 5%.
April 4th, 2010 at 3:20 pm
“A” is probably the winner because I would tend to go for “bite size chunks” than something I would have to build for a year.
May 17th, 2010 at 6:27 pm
So are you gonna put us out of our misery and tell us, Bob…or leave us hanging? (:
May 26th, 2012 at 6:14 am
Superior material included in content material may make upwards to find the top e-book
May 26th, 2012 at 11:05 am
Hello there business owner! I located your weblog employing Google so when google search optimizer
May 26th, 2012 at 5:08 pm
We just couldn’t abandon your world wide web site ahead of proclaiming that we really loved your beneficial facts you provide within your guests… Will most likely be back often to check on by way of to new posts
May 27th, 2012 at 1:54 am
Your blog is fairly wonderful, I like your articles, Its really help me to.
June 12th, 2012 at 7:01 am
We appreciate you the straightforward, informative and simple to grasp write-up Savings
June 25th, 2012 at 11:52 pm
Great line accessible. We hardly ever at any time permit any normal shut withouth receiving not too long ago released information making use of this reference web site.
August 11th, 2012 at 9:06 pm
Outstanding weblog below! In addition your web site tons upward very quickly! Just what number do you assume you happen to be utilizing?
September 13th, 2012 at 11:43 pm
I really need, understanding!
January 23rd, 2013 at 3:42 pm
Like a web website owner I believe the data the following is reallyfantastic. I be thankful for the efforts. You ought to continue the very good work as soon as and for all! Good luck.
April 5th, 2013 at 7:22 am
I believe that you know producing males and women hear whatever you have to state, specifically with an issue i am talking about so important
June 4th, 2013 at 10:39 am
Numerous thanks expressing your notions about this site.
December 2nd, 2013 at 2:58 am
Spot on with this write-up, I actually think this amazing site needs a great deal more attention. I’ll probably be returning to read more, thanks for the information!
June 21st, 2019 at 2:09 am
I think it has to be B cause it may not be actually possible to earn that much on the go.Software Testing Company
May 29th, 2021 at 2:12 am
nice
top restaurant
lung experts doctors
advertising
June 11th, 2021 at 9:14 am
Bringing home a new puppy is a wonderful experience involving the whole family, which is why we begin training early to ease the transition to their new home and family. Owning a pug puppy or French Bulldog puppy is hard work. Making make it less stressful. To provide a completely different environment for our puppies, we thoroughly researched and make sure this puppies will only go to respectable families
https://pugslandbreeder.company.com
June 28th, 2021 at 9:14 pm
https://savannahbengal.company.com/
August 9th, 2021 at 12:08 am
mouse for video editing Buy for best mouse for video editing. BEST MOUSE FOR VIDEO EDITING ON ADOBE PREMIERE PRO WITH MAC + JOG SHUTTLE WHEEL CONTROLLER.
mouse for video editing
September 24th, 2021 at 4:45 am
Thanks for sharing. Your efforts made this very easy to find
See Ali Pur Ka Aili in PDF
Click Here for Bachpan Ka December And
Yaaram Novel Download
The novel Mushaf by Nemrah Ahmed is very interesting
November 1st, 2021 at 12:35 am
TamilDhool Cool Collection of Tamil Serials & Shows. TamilDhool Watch Tamil TV Serials and TV Shows Free Online,
tamil dhool
November 1st, 2021 at 12:36 am
Tamilblasters is a piracy platform. Tamilblasters is an infamous piracy website offering an extensive collection of Tamil and Telugu tv show serial to its viewers for free.
vijay tv tamildhool
November 1st, 2021 at 12:37 am
Sun Tv – TamilDhool Cool Collection of Tamil Serials & Shows the latest promo of popular Tamil Serial Sundari that airs on Sun TV. sun tv serial
November 1st, 2021 at 12:38 am
Watch Vijay TV Serials Online Watch latest and full episodes of your favourite tamildhool serial
Star Vijay TV shows online on .
November 1st, 2021 at 12:38 am
Watch Sembaruthi serial tamildhool zee tamil
today’s episode, popular and all latest episodes, promos, videos and get complete details Zee Tamil is an Indian pay television Tamil.
January 11th, 2022 at 10:55 am
Ndure offers their customers jackets of all types, including Denim Jackets For Women, leather jackets, puffer jackets, and Sherpa denim jackets. Buy the best available winter blue jeans Jackets for women online in Pakistan from Ndure. We bring you high quality bomber jacket, Puffer Hooded Gilet for womens online.
February 13th, 2022 at 2:01 am
Watch latest Tamil tv show episodes on Zee5, including Tamil comedy, romance, thriller, horror & family shows and more.
Watch your favourite shows from Star Plus, Star World, Life OK, Star Jalsha, Star Vijay, Star Pravah, Asianet, Maa TV & more online on Disney+ Hotstar.
dekhasia
dekhasia
February 13th, 2022 at 2:01 am
Tamildhool is a video streaming website that offers more than 50 original shows and over 50,000 hours of Premium Content from leading Producers and Publishers.
tamildhool
tamil dhool
February 20th, 2022 at 1:22 am
ver doramas online DoramasVip Aqui podras encontrar doramas de todo el mundo traducidos al español latino Gratis DoramasVip HD doramasmp4.
Ver en Doramasmp4 doramas gratis en emision y finalizados, capitulos completos, aqui Doramasmp4 encontraras muchos doramas, peliculas de asia gratis doramas vip…
February 20th, 2022 at 1:23 am
Ver online doramas coreanas, doramas estrenos, Tailandia, Japones, Estrenos Doramas con subtítulos en español gratis.
Ver Doramas Online en Doramasflix – doramasflix en HD Doramas y Películas Sub Español , Dramas Coreanos?, Chinos, Tailandia y Japones en Doramasflix -.
February 21st, 2022 at 12:09 pm
The full type of the UTR number is a Unique Taxpayer Reference number. Given by HMRC (HM Revenue and Custom), the principle utilization of an UTR number is given to people who satisfy the expense self-evaluation and furthermore to the organizations.
February 21st, 2022 at 1:37 pm
10 seconds determines a ‘knockout’ in boxing. A KO is something the likes of Tyson Fury would rather give than receive. During a fight, when a boxer falls to the floor the referee must count to 10 to determine whether they can get back up to continue. If not, their opponent wins by knockout.
February 23rd, 2022 at 5:19 am
As an outsider in Singapore you can rapidly lose all sense of direction in an ocean of monetary promoting none of which might appear to be natural to you. Assuming you’re accustomed to observing ATMs from global names like Citibank, HSBC, or American Express, you’ll see that these banks have next to no presence here. Rather the best banks for outsiders in Singapore are the ones that are neighborhood but then appropriate for worldwide installments and movess
Too read more click banks in singapore for foreigners
February 25th, 2022 at 4:58 am
The Dutch commanditaire vennootschap, or confined affiliation, is undefined from a VOF (general association) association in Netherlands. Taking everything into account, there are two kinds of colleagues: a supervising assistant who runs the association consistently and a bound or calm accessory who goes probably as a supporter. Confined associates don’t need to divulge perceiving information while spreading out a Commanditaire Vennootschapwith the KvK; the amount of calm assistants and irrefutably the capital they supply is reasonable
April 20th, 2022 at 2:56 am
Watch Tamil TV Serials and TV Shows Free Online, TamilDhool • Cool <a href="colorstamil of Tamil Serials & Shows.
Watch Tamil shows and tamil Serial Online at TamilTwist, Latest shows of vijay tv, sun tv, colors tv, zee tv and more. Check out Tamil Serials Online Today zeetamilserial.
September 13th, 2022 at 8:20 am
DoramasFlix
great site for doramas in spanish sub.today i visit this site i found all doramas with latest capitulos. visit this site by clicking top or bellow button of comment and enjoyed doramas like chinese, japanese, corien, latinies and more many.
https://doramasflixx.net
September 13th, 2022 at 8:21 am
DoramasFlix
is best for all doramas in spanish sub. visit and enjoyed all doramas with latest capitulos
https://doramasflixx.com
September 21st, 2022 at 1:11 am
Watch all Sun Tv Serials FREE Abhiyum Nanum , Roja , Vaanathai Pola , Poove Unakkaga , Kannana Kanne, Anbe Vaa , Chithi 2 , Thirumagal , Paandavar Illam , Maharasi , Kayal are the prime time
September 21st, 2022 at 1:12 am
Tech and Sports News Ghana Music MP3 Download Naija Music
December 13th, 2022 at 10:54 am
Exclusive Song: Maziyar | Iran Iran With Text And Direct Links In UpMusic
December 13th, 2022 at 10:55 am
Aliha – “Mishe Faghat” > Old Song > Download With Text And 320 & 128 Links In Musico
December 13th, 2022 at 10:58 am
Exclusive Song: Maziyar | Iran Iran With Text And Direct Links In UpMusic
Exclusive Song: Maziyar | Iran Iran With Text And Direct Links In UpMusic
December 18th, 2022 at 8:58 am
????? ??????? ?? ????? ?? ???? ??????? ?????? ?? ??? ????? ??? ???. ?? ???? ??????? ????? ???? ??????? ?? ?? ????? ???? ??? ????? ????? ??? ?? ?? ????????? ? ??????? ?????? ???? ?? ?????? ????? ?????? ?? ?????. ?????????? ???? ???????? ?? ??? ???? ?? ??? ????? ???????… ?? ???? ?? ??? ???? ??????. ?? ??? ???? ?? ?? ???? ???? ? ???? ???? ??????? ??? ? ??? ?? ??????? ???????? ????????? ???? ?? ????? ?????? ?? ????? ???. ?????? ???? ???? ?? ????? ????.
December 18th, 2022 at 8:59 am
Manufacturing is not usually considered a digital industry. Historically, industrial companies allocated their budget to traditional offline marketing and advertising in industrial publications. Websites think they’re considered one dimensional…or worse, they don’t exist. With this, a basic and digitally focused website is one of the most important marketing tools for a manufacturing or industrial business. Let’s examine four reasons.
January 25th, 2023 at 2:24 am
Awesome blog. I have read and thought about what you said. I have bookmarked it and I am looking forward to reading new articles. Keep up the good work!
Parka Jacket Kevin Costner Season 5
January 31st, 2023 at 6:47 pm
Series y Telenovelas de Telemundo Online capitulos completos, tusmundo Capitulos nuevos de tu telenovela favorita,
February 25th, 2023 at 7:15 am
Are you in search of a New Design Dress in Pakistan? Highway fashions offer a large variety of women’s dresses online in Pakistan. So, order your favorite dress today and enjoy.
February 25th, 2023 at 9:00 am
If you are looking for the most appropriate hamilton marijuana store, then you are at a suitable place. Royalleafs is a premier cannabis store in Hamilton. You can find online plenty of Cannabis products such as Cannabis pre-rolls, raw and milled flowers, drinks, and accessories such as bongs and vapes of the highest quality at Royalleafs.
March 3rd, 2023 at 6:05 am
Borjan offers the best quality shoes for men online in Pakistan. We ensure all our products are of the highest standards, emphasizing quality and style that is perfect for your hard-working lifestyle.
March 3rd, 2023 at 7:09 am
Jazmin Clothing presents an exquisite collection of lawn dress designs that capture the essence of grace and glamour. Choose from a variety of colors and styles that cater to every taste and occasion. Order your favorite lawn dress today.
March 16th, 2023 at 9:16 am
TV Host, VJ, Singer, Actor, Politician, Social Media Influencer, Entrepreneur, and Philanthropist. Waqar zakka also won many awards.
March 16th, 2023 at 9:57 am
Mackenzie Media Group works like Specializing in SEO, website design, content creation and advertisement.