Comments on: Surprising Split-Testing Results… http://www.everstatus.com/marketing/surprising-split-testing-results/ Success Blog Tue, 28 Jul 2009 16:07:33 +0000 hourly 1 http://wordpress.org/?v=3.0.1 By: Luke http://www.everstatus.com/marketing/surprising-split-testing-results/#comment-133 Luke Tue, 28 Jul 2009 16:07:33 +0000 http://bunkfree.com/?p=56#comment-133 Hi Leo, You are correct that major features like the headline or the offer typically have the highest potential of affecting conversion rates when changed. If you look at the second test listed above Page A was at 50 and Page C was at 67. That's a HUGE difference. Whether the element being tested was a headline or a font style...that's significant. I would feel good about choosing Page C at this point. However, we can see that hundreds of actions later...Page A was actually in the lead. That was really what I wanted to emphasize with this post...most people simply do not run tests long enough to make certain their results are reliable. I see "gurus" (I hate that word) declaring winners with just a handful of conversions...and that's just stupid. I just wanted to illustrate that sometimes it's very interesting to see what happens when we let split tests run longer than normal. Thanks for the input! -Luke Hi Leo,

You are correct that major features like the headline or the offer typically have the highest potential of affecting conversion rates when changed.

If you look at the second test listed above Page A was at 50 and Page C was at 67. That’s a HUGE difference. Whether the element being tested was a headline or a font style…that’s significant. I would feel good about choosing Page C at this point.

However, we can see that hundreds of actions later…Page A was actually in the lead.

That was really what I wanted to emphasize with this post…most people simply do not run tests long enough to make certain their results are reliable.

I see “gurus” (I hate that word) declaring winners with just a handful of conversions…and that’s just stupid.

I just wanted to illustrate that sometimes it’s very interesting to see what happens when we let split tests run longer than normal.

Thanks for the input!

-Luke

]]> By: Leo http://www.everstatus.com/marketing/surprising-split-testing-results/#comment-132 Leo Sat, 25 Jul 2009 17:22:54 +0000 http://bunkfree.com/?p=56#comment-132 Interesting numbers....I think that it really depends on what you are testing. For instance, if you are testing fonts, color schemes or some other MINOR design aspect, I imagine that you will find that numbers will be very close (like what you show above). However, I have found huge differences in conversions or actionable results if you test other things like headlines, adding video vs. straight text, proof placement, or even where you place the action link (above the fold, at the end...in the middle)... I would consider testing that is that close to be almost negligible though...there really is no clear cut winner if you factor in a buffer..... .02% difference with only 10k page views isn't a lot. Interesting numbers….I think that it really depends on what you are testing. For instance, if you are testing fonts, color schemes or some other MINOR design aspect, I imagine that you will find that numbers will be very close (like what you show above).

However, I have found huge differences in conversions or actionable results if you test other things like headlines, adding video vs. straight text, proof placement, or even where you place the action link (above the fold, at the end…in the middle)…

I would consider testing that is that close to be almost negligible though…there really is no clear cut winner if you factor in a buffer…..

.02% difference with only 10k page views isn’t a lot.

]]> By: Luke http://www.everstatus.com/marketing/surprising-split-testing-results/#comment-131 Luke Thu, 23 Jul 2009 14:58:27 +0000 http://bunkfree.com/?p=56#comment-131 I've used a bunch, my favorite right now is http://powersplittester.com/ It's cheap, simple and effective. I’ve used a bunch, my favorite right now is http://powersplittester.com/

It’s cheap, simple and effective.

]]> By: Andrew http://www.everstatus.com/marketing/surprising-split-testing-results/#comment-130 Andrew Wed, 22 Jul 2009 04:19:34 +0000 http://bunkfree.com/?p=56#comment-130 What software do you use to perform your split tests? What software do you use to perform your split tests?

]]>