“How old are you?”
“How much money do you make?”
“How much do you weigh?”
These lines have one thing in common: They’re all terrible conversation starters. And yet, each could be the first item on the next questionnaire you see.
Sadly, it’s easy to fail — hard and fast — in survey design. Happily, with a little bit of srategic attention, you can make a world of difference. Treat your next survey as a conversation and you’ll see greater engagement, better data, and higher response rates. Sound good?
Survey as conversation
Over years of conducting research studies, helping clients to manage their own projects, and training hundreds of users and colleagues, I’ve come to consider the “survey as conversation” model to be one of the most useful strategies to share. With all respect for those conducting extremely specific scientific analyses under regimented conditions and structures, the vast majority of people creating surveys (1) aren’t experts in research methodology and (2) don’t really need to be. For advice to be useful, in this case, it needs to be both accessible and practical.
Consider some of the most common survey design questions we encounter:
- How long should my survey be?
- How many questions should be on a page?
- How many survey questions should be mandatory?
- What order should I put my questions in?
- How do you make sure people only see questions they can answer?
- What’s the best question to start with?
- What’s the best question to end with?
- Do I need a welcome message for my survey?
In some cases, I hear these questions before someone’s even written a single word of the survey. In some cases, they’ve written or built the whole thing and they’re now beginning to realize that there might just be a little more to this than they thought. Either way, asking “What if this was a conversation?” can be a great follow-up to arrive at a meaningful response together.
Wait, so why not just have a conversation?
The conversational approach is so great that the only natural reaction is to consider having a conversation instead of a survey. Right? Plenty of research studies are conducted through one-on-one interviews, panel conversations, and focus groups — and sometimes through analysis of video or audio recordings. These are absolutely invaluable — authentic, fresh, direct, and generally in the participant’s own words. Depending on the degree of structure allowed in the approach, the interviewer may have some or even total freedom to decide what questions will be asked, which path or line of questioning to follow, and even how long to spend asking questions. Again depending on the structure, the person or people responding may be able to respond in any manner they choose, to take as long as they like to respond, and even to ask follow-up questions to the interviewer or side questions to others in the same group. So much freedom! So much data!
And yet… sometimes talking it through isn’t the way to go at all. A one-on-one interview can be a bit intense for some participants — what’s the difference between an interrogation and an interview? This leads to results skewed by stress. Others may fall prey to social desirability bias — that response bias that can easily nudge us to seek the approval of our interlocuters by giving the “right” responses. This leads to results skewed by, sorry to say, make-yourself-look-better-than-you-are lies. Or, the issue can lie instead with the interviewer. Researcher bias can be just as dangerous, leading the “good” participant through easy questions and offering encouraging feedback along the way while tripping up “bad” participants. (Watch out for this issue in HR interviews with candidates!)
A survey can be preferable to a conversation for many reasons.
- Efficiency is at the top of the list. If you’re conducting an in-depth qualitative case study, disregard. If you’re looking for a census study or a perception study, though, you’re probably going to want to hear from a lot more people. Let’s assume that you don’t have infinite time or resources and that an online survey allows you to collect more responses more quickly.
- Also high on the list is the confidentiality (or even total anonymity) of completing a survey online. Participants are more likely to share candid, complete responses when their results can’t be connected with them personally.
- Next, consistency. Most interviews aren’t completely structured, and even those that are semi-structured can wander off track a bit. This means that any two interviews — or group chats, or focus groups (more likely to have a solid protocol, but still!) — can end up at different destinations.
- This brings us to the final point on this list for now: analytics. If you’ve conducted even two interviews, you’ve got a lot of notes and recordings to code and analyze. Yes, valuable data, but it can take a long time to work through. If you’re using a survey platform with solid reporting capabilities, you don’t need to put together all of that data from a raw pile. Instead, it all comes together automatically. Did I mention efficiency?
Just so we can move on for now, the point is clear: A conversation is great, but there are good reasons to use surveys sometimes, too.
Where to begin?
Just like any conversation, the start of your survey is pretty critical. Let’s assume, for the moment, that your participant’s experience starts right on this screen. While it’s totally possible that they’ve clicked a link in an invitation, on your website, or elsewhere, start with what you can control on this single screen.
It’s pretty natural (see intro) to start by thinking about the key things you want to know about who this person is. After all, you probably want this information in your report. It’s not uncommon to see demographic details on forms — in a doctor’s office, for example, it would make plenty of sense to ask about your age and weight, although the income question still might seem funny.
However, imagine walking up to someone on the street and asking how old they are. While social taboos vary by culture, of course, it would be really strange to start any conversation with a question like that.
Although demographics can definitely be important, in an online survey you may be able to pre-fill and hide those details. If not, you may choose to ask them a bit later. If you’re creating a form for a doctor’s office, though, do what you need to do.
How’s everybody feeling tonight?
Okay, you don’t need to be a stand-up comedian or a lounge singer to get your audience in the right mood. You do, however, have to put some effort into warming up the crowd. Again, let’s assume they just showed up and they’re waiting to see what this whole thing is all about.
In a good conversation, you might start by introducing yourself. Think about all those times you’ve had to call a complete stranger on the phone. When they come on the line, you can’t just shout out “Do you like our blog?” That’s awkward at best. Instead, you introduce yourself and let them know what this call is about.
“Hi, is this Zak? … Hi Zak, this is Melissa from Sogolytics. Do you have a minute to chat? … I wanted to get your feedback on this blog I’m writing right now…”
Easy does it
First, it’s smart to put something out there — a little information up front to tell participants what this survey is about, what it has to do with them, how you’ll use their feedback, and why they should care. It’s also helpful to let them know some pratical things, like how long it might take them to complete the survey and if they have the opportunity to save part-way through and continue later. If there’s a deadline for the survey to be submitted, that’s also helpful.
Also, it should probably go without saying, but I’ll go ahead and say it: They should know who you are. Participants are more likely to respond to well-designed, professional looking surveys — even if your only real “design” element is having a logo at the top of the page. Your visual settings choices are definitely a factor here — who wants to participate in a survey that looks spammy? — but since our focus is more on the words, I’ll set that aside for now.
Giving participants something before asking for anything also mimics the back-and-forth nature of a real conversation.
To begin with…
Early questions in a survey often set the stage for what’s to come. Sometimes they’re qualifying questions — “decision” questions in place to enable survey logic to lead you in the right direction. After all, if you’re going to put all this work into designing the perfect conversation, you want to make sure you’re having this conversation with the right person. In the example above, if I just assumed I was talking to Zak and plowed right into my questions, I might be surprised to find that I was instead talking to Alex, who might be surprised as well.
If you’re sending email or SMS invitations, there’s a good chance you might already know the answers to some of these qualifying questions. So, rather than asking “What’s your role in the school district?” you might be able to pre-fill someone’s answer as “Parent”, “Teacher”, or both. With that question answered, you can then lead your participants down the right path. If they answer “None”, you might choose to exit them out of the survey completely to avoid asking them questions that are irrelevant to their experience — and to avoid collecting responses that are irrelevant to your research. Of course, mind your manners. Even if you ended up speaking to the wrong person on the phone, you’d probably say something like, “My apologies! Thanks anyway.” You’re much too polite to hang up on someone abruptly or — if this was in person — to simply turn around and walk away. You can set up a customized message to let “exiting” participants know what’s happening and why, or you could even ask them a few of their own follow-up questions. (How did you get to this survey anyway?)
Keep going
In the next part of this series, we’ll tackle the middle section of the survey conversation — from after you say “Hi there!” to just before “Bye!” (That’s how you end your surveys, right?)
See you soon!
