I have performed dozens of jobs. I delivered phone books (back when everyone had to have one). I baked cakes (even though I’m not much of a baker). I roto-tilled garden plots, I sewed in a dress factory, and I sold fancy scarves to very rich ladies. I like to do things that give me insight; insight that pays extra cash is even better.
So when my husband passed along an article about “mystery shopping” I carefully chose an organization to sign up with. More insight…and cash to go shoe shopping with. Cool.
Much to my dismay, although I signed up with just one organization, within hours I began to receive assignment solicitations from several organizations; organizations I never heard of. I must not have read the fine print on the first one. Oh well. The real disappointment came with assignment descriptions (paraphrased and exaggerated just a little):
“Memorize the script we will send you. You can use a PDA or cell phone recorder, hidden in your purse, to take notes. Drive 10 miles through some of the worst traffic you’ve ever encountered. Find a parking spot in one of the most crowded malls in North America. Make your way into the Upscale Shoe Salon and, following the script which may have you being an ornery witch, try on several pairs of shoes. You must buy a pair (minimum $300), but if you want to keep them, you will not be reimbursed for them. If you wish to return them, you must do so the same day. After you’ve driven there, parked, shopped, bought, and returned the shoes, go back home and fill out a report with short essays on your experience. Scan and email the business card and parking receipt (to prove you were there) to us, along with your report, within 24 hours. We will pay you $14 for this. If you choose to receive a paper check, there will be a $3 processing fee.”
This got me thinking…how representative of the experience of a real shopper at this store is this “shopping assignment”? What woman who actually has a closet of fine designer shoes is going to do all of this for $14? Even if she were going to buy the shoes anyway, the time spent on the prep work and reports pays about minimum wage. If the script calls for her to be a difficult customer or a repeat-returner she’ll be burning her bridges at that store. I began to question what insight the store owner was getting from this data. (Of course, I don’t have enough information to make any judgments about it).
But it does give one pause, and not just about mystery shopper data.
Do you know - for sure - that the data collection you do is representative of the population you want to understand? Are your Facebook friends or your LinkedIn connections representative of the customers you have? Are they representative of the customers you want? If you ask patron of your restaurant to fill out a customer satisfaction survey, are the responses you receive representative of all of your customers? What do they tell you about the people you would like to have as customers, but don’t yet?
I had a colleague who used to say (he was quoting another colleague so I apologize to whomever actually said it first):
The only thing you know for sure is that the sample you have represents the population you took it from. This is not to say that the population you sampled was the population you intended to study.
Unfortunately, there is no formula to ensure a representative sample. The best defense against a bad sample is to take a critical view of a process that you understand very well. After you think you’ve nailed it, ask another subject matter expert to look at your plan with an even more critical eye. Then you’ll probably be OK.
I’ll be posting more on statistical sampling. Lots more… Follow me on Twitter or subscribe to my feed so you won’t miss the answer to the most frequently asked question about statistics: How big does my sample need to be?