Imagine you are working with an auto company. Let’s say Volkswagen. And you are tasked with the challenge of redesigning your website to better align with the needs of Volkswagen’s customers in order to help recover online traffic from the recent emissions test fiasco. Let’s say you only have 2 weeks to perform research for this project. Your hunch tells you to create a survey. But you have questions.
Survey’s Outcomes & Project Goals: How do you ensure that the data captured in the
survey will align with the desired project goals?
Survey Participation & Completion: Once participants start the survey, will they complete it?
Survey Interpretation & Understanding: Will participants interpret the survey questions in the way you expect? Will participants understand the language conveyed in the survey?
While surveys provide an efficient means to reach out to a broad set of users and gain insights on patterns of preferences, thinking, and behavior, it can be challenging to obtain optimal participation and ensure the data captured meets the overall project objectives.
At Fuzzy Math, we’ve constructed and disseminated surveys to tens of thousands of real users for products in order to understand patterns in perceptions and behaviors and use these patterns to inform design decisions. On a recent project, Fuzzy Math created a survey for a client that was looking to redesign the content of their existing website. We encountered several challenges including keeping the survey to 20 minutes while incorporating questions that would allow us to thoroughly explore user goals, prioritization, and experiences. As we reflect on how we addressed these challenges and tackled some of the questions above, we have the following tips for survey construction.
Our Tips for Building Better Surveys
1.Identify survey goals and topics (to address these goals)
Before even beginning to draft questions for a survey, it’s important to understand the overall goals for the survey and how the data will be applied. Survey goals are shaped by the overall project goals.
If you are working to redesign the content on Volkswagen’s website, one survey goal may be to understand how users currently interact with content, their frequency of use, and how they prioritize certain application features. Information about feature usage and prioritization can help you identify how users prioritize information about driver assistance, in-vehicle navigation, and fuel efficiency and can help shape decisions about information architecture.
Another survey goal may be to understand the differences in attitudes and behaviors of different customer segments. Many products are geared towards multiple types of users or purposes. Volkswagen, for instance, builds cars that support different lifestyles and environments. A survey supporting the launch of a new Volkswagen product may require understanding differences in the activities and attitudes of these different lifestyles and environments. A customer living in an urban environment may have very different needs than a customer living in rural contexts. Consequently, it is important to incorporate questions to differentiate user groups.
2. Identify existing data that can be leveraged
Once the goals of your survey and topics have been identified, it is important to understand whether there is existing research that you can leverage from your company, client, or industry. As a survey demands participants’ time, it is important to identify existing knowledge you can use (and not necessarily ask again during the survey) and focus the survey on questions or topics that are currently unknown.
If the goal of your survey is to understand offline and online car buying habits of Volkswagen customers, it may be worthwhile to identify whether there is existing car industry research that can be leveraged. Research firms like Forrester offer industry-specific reports based on annual surveys of consumers and business leaders.
The company you are working with may also have existing data about their consumers that you can leverage. Beyond stats about customer perceptions and behaviors, larger companies would likely have information about their customer segments – and the key characteristics they use to differentiate segments. Volkswagen just may have an Analytics or Marketing team with insights about the needs of different customers including urban and rural-based customers, families and couples, and high-end and price-sensitive consumers.
3. Document questions and rationale
Survey questions should support the research goals and encourage clear analysis. To ensure the questions are distinct and purposeful, we use a grid to document our questions and the criteria for those questions. Those criteria might include: answers (if the question is not open-ended), any additional instructions, the type of question (e.g., multiple choice (multiple answer), matrix, text entry), and your rationale for each question. This becomes a working document for review as the survey is drafted.
For our imaginary Volkswagen survey, we may document one question as follows:
Question: Please describe your current living situation.
Instructions: Check all that apply.
Answers: I live alone, I live with roommates, I live with parents, relatives, or guardians, I live with a spouse/domestic partner/significant other, I live with my child/children
Reasons: To differentiate between multiple customer segments
Type of Question: Multiple Choice (Multiple Answer)
4. Consider the survey question order
As a survey’s success rests on user participation and completion, it is important to review the survey from the lens of the user. The order of the survey questions is important. If the first few questions are not interesting, participants may be reluctant to complete the survey. We suggest including questions that engage participant’s interest toward the beginning of the survey and placing less interesting questions pertaining to demographics or satisfaction toward the end. If the survey incorporates questions on multiple topics, group questions related to a similar topic together. For example, if your Volkswagen survey incorporates questions about consumers’ online car buying experiences including research, comparison, and payment activities, group questions about each activity type together (i.e. research questions, comparison questions, payment questions). This helps participants maintain their focus on a particular topic and avoid the fatigue that comes with switching back and forth between topics.
5. Test the survey and make adjustments
Aside from lack of interest or fatigue, duration and comprehension may be limiting factors in survey completion. If the survey is too long (takes more than 20 minutes) or is difficult to understand (uses jargon unfamiliar to participants), participants may not take the time to complete it.
We would suggest guerrilla testing the survey with a few participants to identify survey completion time. Survey tools like SurveyGizmo and SurveyMonkey allow you to time the survey. SurveyGizmo offers a feature that predicts the amount of time the survey would take someone to complete based on the amount and type of questions.
We would also suggest guerrilla testing the survey with a different set of participants (ideally people unfamiliar with the survey topic) to identify whether the questions can be easily understood. Remember: when drafting survey questions it is important to include questions that are simple, straightforward, and as concise as possible. Avoid industry jargon that may be unfamiliar to participants.
Building effective surveys is not easy. Striking the delicate balance between a survey that is not too long and fulfills the needs of the overarching project can be challenging. Consider these tips as a way to systematically refine a survey to support project goals and lower barriers to participation. Identifying survey goals and existing data that can be leveraged helps you narrow the focus of the survey on the topics to explore. Documenting key characteristics about each survey question like the question type, response options, and the rationale helps support continual feedback and review. Ordering the survey questions to engage participant interest and avoid fatigue promotes survey completion. And finally, testing the survey with participants unfamiliar with the topic for survey completion time and comprehension helps ensure that the survey is not too long and can be easily understood.
Happy survey building!