How long should you run a website split test for?

When is the best time to call time on a split test??

One of the biggest issues when running a split test is knowing how long to run it for. Confusingly, depending on the split testing tool you use it may tell you when to stop or it may even stop the test for you based on it’s own statistical calculations. For the uninitiated that might sound like job done but in fact it’s misleading and possibly injurious to the final objective. You need to consider a few other things:

Business cycle

For the purpose of split testing, the business cycle, i.e. aggregate trade activity over a given period of time, is often thought of in terms of weeks or months. Years is rather too long to call a result even if it is likely to be statistically more robust. Days are pretty pointless, they are far too short a time frame especially when you consider that conversion rates and traffic volumes will often fluctuate on a daily basis over the course of a week; also it is still the case that for a large portion of the population that two out of seven days in the week are still non working days.

Because of that when running a split testing program the tester needs to make sure that each test is run over a complete business cycle or combination of business cycles.

Purists will argue that the run time of the test should also not favour any one part of the business cycle, so that if your cycle is a week and the test is started at 9am on a Monday morning then it should also end at 08:59 (or 9am) on a Monday morning. Doing this you will be less likely to skew the results by including one part of a business cycle (which may favour one of the variants) but not another.

Purchasing cycle (or decision-making process)

This is perhaps a footnote to the point above. On most e-commerce sites the decision-making process involved in buying a product or service may take longer than a day.

To get a better picture of how this looks go into your web analytics package and look for a report that shows you time lag and / or path length.

In Google Analytics you can find this in the Multi-Channel Funnels reporting area. These reports will show you the number of interactions (Path Length) or the number of days (Time Lag) it takes for customers to buy something. Based on this you will then have an idea of your purchasing cycle.

If you have an especially long purchasing cycle then your test time frame will need to take this into account.

Volume of data

The short answer here is ‘the more the better’. That said it’s impractical to wait for months and months to call a result and really for many e-commerce sites the purchasing cycle is not likely to be that long.

So in most cases it is considered that 300 good outcomes / sales per variant is a minimum.

BUT, if you have your testing tool hooked up to your web analytics tool specifically so you can segment your test results then 300 good outcomes might not be enough. The obvious reason here is that by segmenting a dataset with 300 good outcomes you will inevitably wind up with a smaller less robust dataset with fewer good outcomes and therefore not as reliable. So in these instances you will need to extend the minimum sample size to accommodate your segmentation.

Regression towards the mean

This is the phenomenon where a variable that was extreme on its first measurement tends to be closer to the average on its second measurement or where a variable that is extreme on its second measurement tends to have been closer to the average on its first. In split testing this often reveals itself in circumstances where both variants converge.

This process doesn’t happen overnight. So often at the start of a test the data can look fantastic with the new variant set to bring home the bacon in style, but of course:

• 
  

  1. The sample will most likely be far too small unless you have a high traffic site or you are adopting very broad targeting for the experiment.

  
  


• 
  

  2. Partly as a result of (a), as the sample grows the phenomenon of regression to the mean will reveal itself.

  
  

So by ensuring that you collect a robust sample of data AND you run for a minimum of two business cycles (preferably more) you should ensure that you will have accommodated for this phenomenon.

It’s important to remember that the criteria above are not a pick and mix set of options. It’s not the case that if you are able to collect 300 good outcomes per variant before the you have completed one business cycle (due to high traffic volumes) then you have a reliable result, even if your testing tool calls it. You should wait until you have a minimum sample size PLUS at least one full cycle (preferably two), this underscores the point that generally speaking the more data you have the more robust your results will be.

There is another, possibly rather more obscure reason, why tests need to run for longer although and this is harder to measure but is potentially linked to the final point about regression to the mean.

When running a split test you are, by definition, introducing your customers and returning visitors to a new page layout and possibly even a new process on your site. Some people will take to the newness quicker than others, but once it has been in the wild for long enough and your sample has become used to it, they may only then start to find it easier to use and only then might it start to yield a positive return when compared to the control. This is basically the inflexion point of the test.

Key takeways on planning your time to test

Of course it may be that the inflexion point is never reached but all this is to say that:

• 
  

  1. There are some clear guidelines and principles as to how long you should run a test for.

  
  


• 
  

  2. If you see your test variant performing atrociously to begin with do not think that it stands no chance of ever recovering and pull the test in order to avoid losing revenue.

  
  


• 
  

  3. Remember that things take time to bed in.

  
  

Download resource – Conversion Rate Optimisation Guide

This guide is aimed at helping more businesses get started with CRO by explaining a process and tools.

Access the Conversion Rate Optimisation Briefing

from Smart Insights http://ift.tt/1FkjrHP

via IFTTT

How effective are personalized product recommendations for Ecommerce? [#DigitalInsights]

Ecommerce in Ireland 2015

Understanding the Irish online: the E-Commerce Marketer's secret weapon

Irish online habits have changed a lot in a very short amount of time. In some respects the Irish demographic is very representative of typical EU online behaviour but in others it is quite distinct. This particular profile of Irish internet users should give companies and marketers some clear ideas of the approaches to take when selling in the Irish market online.

One of the most significant statistics in regards to the internet in Ireland is the penetration of broadband services across the country. According to Eurostat, 81% of Irish homes had broadband access in 2014. That is well above the EU average and means that more than 3 million people in Ireland are online.

Irish Internet users also spend more time on the internet than the EU average. An average of 13.5 hours are spent online per week in Ireland. Of course that breaks down unevenly across age groups with under twenties spending almost twice the average online and over 35’s spending less than half that amount of time surfing the web.

Taking the old with the new - Irish consumption of media and the internet

A recent surge in evening bandwidth usage across Ireland has been attributed to the growth of streaming services, in particular Netflix, which is thought to have as many as 200,000 subscribers in the country. Other streaming services such as RTE Player have also contributed to this growth.

This movement of consuming traditional media via the internet is supported by figures from Mediascope research that shows increasing figures for Ireland.

According to Mediascope, 81% of internet users watch TV online, above the EU average of 73%. 66% listen to the radio online and 89% read newspapers online which are both on par with the EU average. Targeted ad spots on streamed services should offer high potential in reaching Irish media consumers.

The Mediascope research also highlights large numbers accessing the internet from mobile and tablet platforms, both of which they say are 50% higher than the EU average. Mobile internet use is at 31% of the population and 12% use tablets to go online. Responsive website design is clearly a vital element in taking advantage of the Irish market.

E-Commerce appetite in Ireland

Eurostat’s 2014 figures show that 43% of Irish internet users had made an online purchase in the previous quarter. Mediascope’s figures give an overall number for those shopping online in Ireland at 97% compared to an EU wide average of 87%.

An important figure from the Mediascope report is the number of Irish shoppers who research online and purchase offline (ROPO). 88% of internet users in Ireland research products online before buying in a local store.

The research also discovered that Irish consumers had spent an average of €685 per person in the previous 6 months. Most Irish internet users considered the internet to be an important factor when deciding on purchases from a range of markets, especially travel, holidays, mobile, financial products and insurance.

Overall the figures demonstrate that Irish internet users are eager to take advantage of high broadband availability and are not averse to adopting new technologies. They also show that Irish consumers are aware of the potential the internet offers in terms of getting value for money and finding not just the best price but also the best quality.

Important online factors to consider when marketing to the Irish

When marketing your products to an Irish audience there a number of factors to consider, many of which may be applicable in your own country:

• Secure payment facilities


• A variety of payment options – Visa Debit, Master Card, PayPal


• Trust signals – credit card logos, verified supplier


• Prices in Euro


• Fast load website load times


• Positive website user experience


• Well written and descriptive product descriptions


• Mobile and responsive website design


• Crystal clear product photography


• Friendly customer service


• Easy to locate terms and conditions


• Clear refunds and returns information


• Reasonable delivery times to Ireland


• Contact details – address, phone number and email


• Real product reviews


• Social media reviews – Facebook, Twitter, Google+ local, Yelp

The Irish online market offers E-commerce retailers an engaged and savvy audience that are spending increasing amounts of time and money on the internet. The opportunities are there and 2015 will only see the Irish market grow further.

Image/Copyright: Patrick Naughton

Thanks to Patrick Naughton for sharing their advice and opinions in this post. Patrick Naughton is a digital marketing manager that specialises in international digital marketing. He works for Digital Skills Academy, a revolutionary digital technology educator with a proven track record in delivering high levels of successful employment outcomes for graduates. You can follow him on Twitter or connect on LinkedIn.

from Smart Insights http://ift.tt/1FJTTTN

via IFTTT

How to define SMART marketing objectives

From SMART to SMARTER marketing objectives

When identifying specific marketing objectives to support your long-term goals, it is common practice to apply the widely used SMART mnemonic. You will know that SMART is used to assess the suitability of objectives set to drive different strategies or the improvement of the full range of business processes.

One of the main reasons that we called our site and service SMART Insights is because we wanted to help marketers succeed through using a more structured approach to planning to give more realistic targets they could be more confident of achieving. Using SMART objectives and then measuring them through properly customised analytics reports is a big part of how we hope to help too.

With SMART objectives documented in Plans linking objectives to strategies and KPIs everyone is sure exactly what the target is, progress towards it can be quickly and regularly reviewed, for example through an Ecommerce dashboard and, if necessary, action can be taken to put the plan back on target.

How can SMART objectives help set realistic targets?

When setting future objectives for marketing such as in a marketing plan it’s useful to look hard at each measure and ask “is it essential?”. The SMART mnemonic helps as a test or filter which you can use to assess the quality of measures. My personal definition of SMART is:

• Specific – Can the detail in the information sufficient to pinpoint problems or opportunities? Is the objective sufficiently detailed to measure real-world problems and opportunities?


• Measurable – Can a quantitative or qualitative attribute be applied to create a metric?


• Actionable – Can the information be used to improve performance? If the objective doesn’t change behaviour in staff to help them improve performance, there is little point in it!


• Relevant – Can the information be applied to the specific problem faced by the marketer?


• Time-related – Can the information be viewed through time to identify trends?

Of course different people interpret define SMART differently and you can refer to the Wikipedia definition of SMART marketing objectives.

Examples of SMART objectives

Here are some typical examples of SMART objectives, including those to support objective setting in customer acquisition, conversion and retention categories for digital marketing:

• Digital channel contribution objective. Achieve 10% online revenue contribution within two years.


• Acquisition objective. Acquire 50,000 new online customers this financial year at an average cost per acquisition (CPA) of £30 with an average profitability of £5.


• Conversion objective. Increase the average order value of online sales to £42 per customer.


• Engagement objective. Increase active customers purchasing at least once a quarter to 300,000 in a market (a hurdle rate metric)

Mistakes in setting objectives

It’s worth guarding against the mistake I sometimes see with student assignments where, rather than listing objective examples like those above, the student will create separate objectives under a heading of each of SMART - this doesn’t work… Better is to group objectives in a logical way, sometimes separating out overall business and marketing objectives and digital marketing objectives.

Another mistake to avoid is a big long list of objectives - yes I have seen a whole page of bullets with no structure... Instead group them logically in a way you would present them to colleagues. We recommend structuring them based on the RACE framework as show in this table aligning objectives to strategies and KPIs

The Ten Measures Design Tests

You can add to your tests of choosing the right objectives using these 10 measure design tests developed by performance management specialist Professor Andy Neely. For SMARTER metrics, ask these questions for your KPIs as you develop them.

• 1. The truth test. Are we really measuring what we set out to measure?


• 2. The focus test. Are we only measuring what we set out to measure?


• 3. The relevancy test. Is it the right measure of the performance measure we want to track?


• 4. The consistency test. Will the data always be collected in the same way whoever measures it?


• 5. The access test. Is it easy to locate and capture the data needed to make the measurement?


• 6. The clarity test. Is any ambiguity possible in interpreting the results?


• 7. The so-what test. Can and will the data be acted upon, i.e. is it actionable?


• 8. The timeliness test. Can the data be accessed rapidly and frequently enough for action?


• 9. The cost test. Is the measure worth the cost of measurement?


• 10. The gaming test. Is the measure likely to encourage undesirable or inappropriate behaviours?

These tests show there are additional filters on top of SMART are useful to choose the best measure, I particularly like the "So-what test, another way of explaining relevance and Gaming - a common issue with target setting that isn’t considered by SMART!

Alternative SMART objectives definitions

Finally, some have developed the SMARTER objectives definition that show the need to re-examine the relevance of SMART objectives through time:

This definition certainly shows the many alternative SMART objectives definitions - you may want to compare against these!

from Smart Insights http://ift.tt/1v0MzBV

via IFTTT

Using social media marketing in B2B markets

Digital Marketing Trends 2015

Creating content the C-Suite truly values

Rethink your content strategy for engaging Senior Decision-makers

There are many frameworks and strategies that we can use to create content that educates, engages and spreads our message. For example, the Smart Insights Content Marketing Matrix helps review which content is effective in different parts of the marketing funnel. When it comes to tailoring content for our buyer personas it’s an obvious process; build content that solves a problem.

There’s another side to this though. If you’re trying to engage the C-Suite, or any senior decision maker for that matter, then the typical method of content marketing may not work. This is because they operate on a different level than your typical prospect.

There are a few reasons that you may want to target the C-Suite:

• 1. Enterprise sales; or any type of sales opportunities worth 6 to 7 figures.


• 2. Business development opportunities; connecting and partnering with complimentary organisations that would really move the needle.


• 3. Thought leadership; spread your content and authority to senior decision makers who can spread it through their network.

Whatever your goals are, the framework and methodology is the same, and you can build your content around this methodology in order to achieve these goals. This same methodology has connected companies with senior decision makers and generated 7 figure sales pipeline, often in as little as 8 weeks.

Even if you class yourself as a small business, this still applies to you. Despite your offering, there are still solutions you can offer to large organisations. Problems are always waiting to be solved.

Content engagement methodology

The methodology consists of two key components. We’ll cover these two components and then share an example of how one company has used them and the results they yielded.

• 
  

  Trust

  
  

Before we talk about the nuts and bolts of content that resonates with the C-Suite, you need to know how these senior decision makers operate. Typically, the C-Suite spends only 2% of their time each week with vendors, which is less than an hour.

There’s a lot of noise to cut through, so how do we get their attention? First, understand that these CxOs aren’t interested in pleasantries or what you and your company is about. They want to know what you can bring to them and their organisations.

When it comes to sales & marketing, most people start with what they do, moving into the benefits, making a business case and THEN they attempt to build trust. Marketing to the C-Suite works the other way around.

You need to spend the time to build the trust first. If you come in with an obvious seller stance their guard will be up and your message will be left unheard. You need to take your brand out of the spotlight and come from an independent stance.

Independence comes in two forms:

• 1. Third party endorsements; from your clients, partners and other advocates.


• 2. An independent brand; we do this in the form of an independent club that focuses on a specific challenge. There are other ways to do this, and we’ll cover these soon.

Endorsements typically come in the form of testimonials and case studies, but there’s a much more effective method to this as you’ll soon see.

It often gets lonely at the top, and because of this the C-Suite rarely find themselves challenged on the things that matter most, that being at a strategic level.

What they truly value is, being tested and because they’re surrounded by those who report to them on a day-to-day basis, their views are rarely challenged. It’s because of this that a face-to-face element in your C-Suite content delivery is important.

• 
  

  Value

  
  

Real credibility is established by being truly valuable to the C-Suite. They operate at a very strategic level, and are typically uninterested in your product or service.

Instead, they want insight, statistics and cold hard facts. There needs to be a logical argument to change, and this happens further up the buyer cycle.

You can use how-to guides in order to do this, but it’s much more effectively used as a complement to the independent entity we’ll be showing you how to create. Value in the C-Suite marketing world is much more than digital content.

It also means being different, and bringing something new to the table. Can you challenge the way they’re currently doing things? Can you bring a new point of view into their world?

We do this by taking challenging topics and aligning them with what the market is already talking about. Backed up with facts from credible sources, this is a very powerful technique.

• 
  

  Independence

  
  

Everything we’ve talked about so far has revolved around the idea of independence. As I said, it’s difficult to engage with C-Suite executives while coming from the stance of a seller. So how do we get their attention?

Our C-Suite Marketing methodology does this by way of creating an independent ‘club’, focusing the content around a pressing topic that the market is talking about and aligning it with the value proposition of our clients.

Let’s get clearer on this with an example. We recently built the Digital Resolution Club for our client Ixonos, who are a creative technology agency. They used this to position their message in the eyes of their target senior decision makers as something truly valuable, something that would help overcome their challenges.

The content was revolved around digital monetisation and the rise of second screens – a topic very close to the hearts of their target audience, being media companies. Again, this wasn’t a topic that we thought was pressing, or even a topic Ixonos themselves wanted to cover. It’s what the market is talking about, right now.

Value was delivered in an intimate, face-to-face setting with 14 senior decision makers. The low numbers are deliberate in order to ensure strong relationships were built.

The attendees included C-Suite execs from Hearst Magazines, Mail Online, BBC, The Economist as well as Metro UK and The Guardian. These were opportunities that Ixonos wouldn’t otherwise have thanks to the independent positioning and the clever alignment of a pressing topic with their value proposition.

Summary on engaging with your C-Suites

As you can see, this is not content marketing in the way we currently know it, but it is an innovative approach that merges digital and face-to-face channels in order build relationships based on intimacy.

Blending digital and face-to-face channels is how you build these relationships faster. It builds trust first and then works backwards to present a business case. Then, and only then, do you present the benefits of your solution and what it actually is.

Brands like Ixonos, as mentioned above, as well as NetSuite and SAS are using this same technique with great efficiency. Businesses of all sizes can benefit as long as you put the focus on the pressing challenges that your target market are trying to overcome.

We’ve given you our method of building an independent club brand, but the options are open to your creativity. How would you take this methodology and apply it to your own business?

Thanks to Tom Whatley for sharing his advice and opinions in this post. Tom is Growth Marketer at Seraph Science You can follow him on Twitter or connect on LinkedIn.

from Smart Insights http://ift.tt/1LarOsV

via IFTTT