I hope you don't mind, but we shall look at a survey that has nothing to do with injury prevention. I happen to think it's quite an interesting story.

Shere Hite (be careful how you pronounce that) wrote a book in 1987 called "Women and Love: A Cultural Revolution in Progress"

The main findings in the book are based on a survey - results from 4,500 women were analysed and seem to tell us:

• 84% of women are "not satisfied emotionally with their relationships"
• 70% of all women "married five or more years are having sex outside of their marriages"
• 95% of women "report forms of emotional and psychological harassment from men which whom they are in love relationships"
• 84% of women report forms of condescension from the men in their love relationships.

As this is an online course, you can't see me, but I can tell you that I am male, and have been married for more than five years. What does a survey like this say to someone like me? Do you have any thoughts on this survey? If you are working online with others, perhaps you would like to discuss these results.

Sample Size

The Shere Hite survey was based on analysing records sent in by 4,500 women. Do you think this sample size is large enough?

You would usually think 4,500 was large enough for most survey purposes. It does depend how many questions you are asking though, and whether you want to analyse subgroups. 

But is there another concern about this number?

There is a lot that could be said about the technicalities of surveys. If you are interested Sharon Lohr has written an up to date account of many of the technical machinery. If not trying to do anything too intricate, 4,5000 is usually regarded as quite a substantial sample size.

However, here's the problem. 

Shere Hite sent survey forms to 100,000 women. Let's not argue about whether these 100,000 were selected correctly, let's just assume these 100,000 were a good range of women in the US. The problem is that only 4.5% responded to the survey.

Do you think the 4.5% of women who responded to a survey asking them all kinds of delicate and personal information about their relationships are typical of the 100,000 women who received a survey form? If we also mentioned that there were 127 essay type responses does that make it sound better or worse?

Low Response Rate

So, for all the marvellous and beautiful statistical technicalities that go with survey methods, it counts for very little if we only get a 4.5% response rate. Shere Hite's book should have been called:

"Thoughts of the 4,500 out of 100,000 women who were prepared to fill in an intrusive survey and Love: A Cultural Revolution in Progress"

I'll leave the rest to your own imagination. Do you think the 4,500 respondents are typical?

Joel Best, in his book "Damned Lies and Statistics" talks about "dark numbers".   In this case, this would be the 95,500 women who didn't fill in the survey.

Workforce Surveys

Consider this - do you think the 30% of your workforce that fill in staff surveys are typical?

Maybe they are all the very happy and very unhappy and you lose the rest in the middle.   If we have a poor response rate, we can't take much from a survey.

Of course it gets slightly worse than this. How well defined is the target audience?   Many of the (expensive) market research companies and opinion pollsters will use the electoral register and scientific methods to select a representative survey group. When we talk about a staff survey, it ought to be clear what we mean (all staff), but when we talk about a community survey, what do we mean by community? Residents? People who work in the area?

When thinking about surveys what do you think we need to consider?

Listed below is the four main criteria that needs to be considered when thinking about surveys.

(a) Who is asking the question (and why)

(b) Who is being asked the question (and why this group)

(c) How many of those being asked the question gave us an answer

(d) Is the group of people giving us answers likely to be the same as the group we wished to investigate

Task: Look at the following survey reported on the Brake website

01.01.08 - Royal & Sun Alliance, UK - More than half (51%) fleet drivers admit using mobiles while driving for work, and one in seven (15%) feel pressurised into committing driving offences while driving for work. One in ten UK workers feels pressurised by bosses to answer business phone calls when driving, and more than a fifth (21%) feel the need to speed to reach meetings on time.

Very briefly explain:

(a) Who is being surveyed - is it just one group of people

(b) What is the response rate - are the responders typical of all people surveyed

(c) What are the dark figures - do you think there are non responders and if so how might they have replied

(d) What does this survey tell us about the usage of mobile phones by a middle aged adult driving for social, domestic or pleasure purposes?

Random Sampling

Now for statistical preaching. Random sampling makes survey methods very powerful. If Shere Hite wanted 4,500 responders there's a better way to get them: Randomly selecting people from the list of 100,000. Then all you do is put a lot of effort into getting their replies.

Consider why we may want to take a random sample, why not just use the whole population?

Because it costs to measure things (even opinions). Forms have to be filled, data entered. So, whenever possible, a study that uses a carefully designed random sample will be better than a self-selecting sample.

Solving the non-response/ dark numbers problem

So, there are answers to the non-response / dark numbers problem:

- A little bit of care in the design of a survey can work wonders.

- Surveys that are a tolerable length.

- Surveys that seem to matter.

- Surveys where you take a smaller random sample but then spend time and effort chasing up non-responders (in the nicest possible way). 

When evaluating statistics from a survey, these are all easily checked. We have statistical methods to deal with the way we do random sampling. We have very little to help us fill in the dark numbers. You should always be wary of surveys where there is a lot of scope for dark numbers (or where the population being measured in the first place isn't clear or is very different).