Monthly Archives: September 2008

Why does Think Eyetracking sometimes use a sample of 30?

Posted on by
4
Share

Everyone knows that the more people you use in any kind of test then the better your sample is likely to represent the population as a whole. This applies to questionnaires, polls, observations, and also where people look when they look at adverts, packaging, homepages, landing pages, emails etc However, in the real world of budgetary and time constraints, you want to test with as few people as possible whilst still being confident that your results are reliable so that you can make decisions with confidence. There are ways that you can determine the appropriate numbers to achieve this – we won’t get into the statistical theory here – but it is important to note that the minimum number of people to use is dependent on what you are measuring. Traditional Market research often uses samples of 100 or more people. At Think Eyetracking we often test 30 people – why is this? Broadly it is because what we are measuring is entirely different; traditional market research usually asks the consumers conscious mind it’s opinion or attitude, at Think we interrogate what the subconscious mind ‘does’ (rather than what the conscious mind it thinks it does!) by recording fixations and saccades. Fixations and saccades are ‘things’ rather than abstract concepts such as opinions and attitudes; because we are counting ‘things’ we can often use smaller samples than traditional market research and achieve statistically reliable results. As a demonstration, here is part of a more formal Monte Carlo Simulation we did:

We asked 150 people to look through the pages of a magazine as they normally would do; there was a mix of advertisements and editorials. The Dove advertisement below was included.

We then created 4 heatmaps from 4 different (randomly assigned) groups of 30 from the total sample of 150.

You can see in these heatmaps above that each sample of 30 produced the similar pattern of visual behavior. And when you run the numbers these are essentially the same too. So that’s a fairly simple explanation, but let’s look a little closer at what’s involved. There are three key elements that can effect the necessary sample size:

  • A) the complexity of the stimuli being shown
  • B) the task being set and
  • C) the variability in the sample

A stimuli with limited complexity and a directive task both limit the need for a larger sample. For example look at figures 1 and 2 below.

It is easy to see that with a less complex stimuli as in figure 1, there will be less variability in people’s visual behavior, and so a smaller sample can be used than would be necessary for figure 2 to find common patterns. Similarly, if the task looking at figure 2 was to find the two black squares, there would be more common behavior than if it was to simply look at the image, and so a smaller sample can be used to find common visual behavior. We have been running commercial eyetracking studies for over six years now (2002 to 2008 at time of writing) and during this time we have got real good at managing elements A through C – stimuli complexity, task, and sample variability – to ensure that a sample of 30 can be enough to provide statistically significant and robust results.

That said, sometimes it is necessary to test with samples greater than 30 and you will be unsurprised to find out that this is when the differences in the things we are testing are very subtle or the image is very visually complex. In these cases we can advise on what will be an appropriate number to sample.

Posted By Lizzie Maughan and Robert Stevens.

Credit Crunch, Theft and Checks

Posted on by
12
Share

As Christmas is almost here I thought is a good time to revist a blog from last year looking at credit rating agencies Equifax and Experian. Last year we found that 97% of people could not find their Statutory Credit Report using Equifax. A report that you are legally entitled to for just £2.00.

In the intervening time Equifax has made it even harder to find your report: There were two links on Equifax’s home page to your Statutory Credit Report, now there is just one and it is effectively hidden at the bottom of the page with grey text on a grey background!

Equifax Statutory Credit Report

The rest of this blog was originally posted on 9th September 2008.

As the credit crunch worsens and identity theft increases, consumers are well advised to check their credit rating on a regular basis.

Credit Ratings are now more important than ever before, so at Think Eye Tracking we conducted independent research on two major credit reference agency websites; Equifax and Experian.

In the UK, people have a legal right to get a copy of their Credit Rating for £2 ($4US). Equifax and Experian allow people to apply for their Statutory Credit Ratings online. Both websites also offer a number of premium services related to credit ratings which range in price.

The research included eyetracking thirty people as they tried to get their Statutory Credit Rating reports.

With Equifax, of the thirty people tested, only one person saw and clicked on the statutory report. The premium services however were very engaged with. This suggests Equifax maybe pushing the premium services at the expense of the statutory one.

Some of the people tested verbalized their confusion and unhappiness:

“They are trying to force me to buy my credit rating for £12 when I have a legal right to for £2, it’s disgusting.”

“I just can’t find it. Am I being stupid?…. Is it really here?”

“I must have looked for it for four or five minutes without finding it before I gave up.”

Finding the statutory report proved difficult for 29 of the 30 consumers tested as the heatmap below shows, there is only one mouse click on the Statutory Credit Report link:

The testing returned far more positive results from the Experian site. 18 of the 30 people tested clicked the statutory report, but 27 saw it. Consumers who did not choose the statutory report made an informed decision to go for Experian’s one month free trial of their premium service.

The eye tracking heatmap above details where people tested focused on the Experian home page. Eighteen of the thirty testedclicked to access their statutory report and 27 saw it. Some of the people tested stated:

“I looked at the statutory offer but went with the free one because it seemed better.”

“Why isn’t the Exquifax one easy like that?”

Companies do need to balance their commercial interests with their moral and legal obligations. This is necessary to gain the trust and loyalty of consumers. At Think we believe Experian has got the balance right by prioritizing their premium service while also giving consumers statutory rights a significant amount of prominent screen estate alongside a compelling commercial offering.

Is Equifax profiteering from the recession?

Cuil is (currently) destined to fail

Posted on by
3
Share

 

As stated in our previous post, 97% of people we have tested use Google. It is the first and only search engine most all surfers employ. Google’s simple yet efficient display has moulded user’s subconscious search habits and preferences so strongly that new interfaces departing from its model are almost destined to fail even if they do have a compelling offering.

The two/three column approach taken by Cuil challenges the habitualised search behaviour people have developed using Google; it is not intuitive to a world of searchers with an unconscious preference for Google’s linear format.

In addition to this, people are happy with Google, a sample of 90 people recently tested by Think, 86% said they wouldn’t use anything else.

Cuil was created by two former Google employees and it boasts to have the largest search capacity to date, but is that what users are really interested in? People have learned how to search efficiently in order to minimize the time spent searching so how does a 3 column layout with paragraphs of text going to align itself with the learned behaviour?

We tested a sample of 30 people using the new search engine to better understand this.

As shown in the heatmap above, Cuil’s 3 column layout promotes localized areas of attention. Fixations are concentrated upon the headers – this is also where participants clicked to access their desired internet pages. The top results acquired the largest amount of fixations, receding gradually from left to right and top to bottom. Results appearing below the category search box suffered from a low fixation count as they were the last in the viewing order. Although limited attention was paid to the category search box, people did verbalise a sense that this would be helpful.

Breaking from the familiar linear format of Google has lead to an inconsistency in people’s visual behaviour. The gaze plots below illustrate some of the representative paths taken. The many long saccades indicate that users were confused. The page layout is different to the familiar flow of Google and so breaks with the users’ subconscious models of how to search. We believe this is a fundamental problem that will prevent users adopting Cuil.

Users also verbalised their confusion and dissatisfaction:

“When you first see it, it looks too busy. I have to choose what I want to look at.”

“I don’t like the layout – it’s too long-winded; too wordy, and I tend to look at things in a list rather than a box layout.”

“I just looked at the headings instead of the black writing. You wouldn’t need to have so much text if you knew what you were looking for. I think there’s more info than you need on it. It’s just supposed to give you a link. There’s lots of things you don’t see because you don’t bother to scroll down for it.”

Cuil’s layout challenges, rather than takes advantage of, the habitualised search behaviour people have developed through years of using Google.

For this reason we think Cuil is currently destined to fail.

Has Google gotten better?

Posted on by
55
Share

How many times do you use Google each day? Each week? Each month? Countless times! And would you say you use it in the same way you did a couple of years ago?

It’s likely that you don’t.

There has been a significant shift in search pattern behaviour; one that reflects rational and effective time management; one that is a result of habitualisation. Here at Think Eyetracking, we conducted a study into search behaviour to explore this.

In a 2005 study we found that people looked through many of the search results before clicking on one. The distribution of attention using Google can be seen in the heatmap below.

In our most recent study, we had 30 participants search for the broad term, Oasis. PEEP methodology was employed; participants were eyetracked while completing the task and later asked about their usual behaviour.

As seen in the heatmap above, fixations are studded around the top 5 results and the majority of clicks are upon the top 3 results (discounting the sponsored link). The sponsored link was actually not well attended to due to the fact that searchers are now familiar with advertiser placement within Google. The 2008 heatmap supports the recent trend observed by Cornell University (Their study found that the top 3 Google results get 79% of all clicks) and by AOL (Findings were that 63% of clicks were concentrated upon the top three search results).

Furthermore when asked afterwards what they would normally do when they couldn’t find their desired search result on the first page of Google, 87% respondents replied that they would modify the search terms or refine the search by category. 97% of people tested answered that Google was the search engine they most commonly used and out of those people, 87% stated they wouldn’t bother using anything else.

It’s not clear that Google has gotten any better, but certainly our use of it has become habitualised. Google’s popularity and dedicated following mean that a large majority of users have grown extremely familiar with the search giant and refine searches to display exactly what they need within the top 5 results. We now expect to find our required answer in the top 5 results.

The use of Google has become habit and users have optimized their behaviour accordingly; they now act in order to eliminate the need to scroll below the fold or sift through additional pages of results. Nowadays to compete competently, you must know your customer’s search words, and land in the top 5, if not top 3 results.

Why Think Eyetracking?

Posted on by
1
Share

Think helps deliver on the promise of marketing;

Create and maintain profitable relationships with consumers.

To date marketing has not delivered on its promise; approximately 80% of all new products launches fail to meet pre-launch expectations.

How can a multibillion dollar industry that employs lots of smart people and ideas get it so wrong?

The answer is simple, if not obvious: market research has largely ignored the importance of consumers’ subconscious minds. Consumers are rarely aware of what they are doing and why they are doing it – cognitive scientists have a rule of thumb “unconscious thought is 95 percent of all thought”, some believe this may be a serious underestimate. I know this may be hard to believe, so think for a moment about your daily commute today, how much of it do you remember?

Probably very little, if you take the same route to work each day. This is because your subconscious mind takes you there on autopilot; there is no need to engage your executive, conscious mind unless something novel or unexpected happens. If you need more convincing go and shop your usual weekly or monthly grocery shop at a new store. Pay attention to how it feels to need to engage your executive mind to find items when you are used to using your habitual mind. It is frustrating!

To date, market research has addressed the conscious or executive mind with questionnaires, focus groups, satisfaction surveys and the like. The problem is the conscious mind is at best only partially responsible for the many decisions people make on a daily basis. One unfortunate but real consequence of this is that 43% of smokers relapsed again within 12 months after undergoing surgery to remove cancerous parts of their lung (according to study by Walker, 2006). This is not the action of a rational conscious executive mind. It is the action of an unconscious habitual mind, stuck on a bad autopilot setting.

There is a disconnect between what people say and what they actually do, we believe this disconnect is symptomatic of the two different states of mind. It’s not that consumer lie on purpose, they usually answer truthfully, but using their executive mind and that is only part of the way we all make decisions. At Think we believe we have found a way to bridge the chasm between the executive and habitual minds to help market research deliver on the promise of marketing.

We rarely consciously control our eyes, this fact gives us the ability to see into some of the processes of the powerful habitual mind with eye tracking. Combining this with our experience, dedicated software, and methodologies, allows us to peer into the consumers’ minds to understand which (if any) mind is being engaged.

Our proven approach combines eyetracking with traditional market research to gain deeper understanding of how consumers engage with visual and written creative work on habitual and executive levels.

It’s the beginning of something special.