The seven stages of analysing open-ended questions

Open-ended survey questions are used to collect free format answers which yield insights that cannot be gained from just closed-ended questions. Because they are not restricted to a specific list of answers, they can provide a richness and level of depth that closed-ended questions are unable to provide.

For even more insight into why to use open-ended survey questions, you can read about the top 10 benefits of open-ended questions and read our complete guide to open-ended questions.

Once you’ve decided to include some open-ended survey questions in your project, you will need to consider how to process and analyse the results. Here we explain the steps taken to produce a robust, thorough and high-quality set of coding for your open-ended survey question.

As we go through the steps we will refer to our recent survey where we asked 1000 people to “Name something on your bucket list”.

How to analyse open-ended survey questions

These seven steps provide a framework for you to follow in your open-ended analysis process, with each step being important in providing a quality output to your business objectives.  Once you are clear on the steps, you can consider whether to use Excel or a dedicated coding platform such as codeit to help you perform each step efficiently, effectively and to the level of quality required.

1. Think before you start

It’s important to have the business objectives for the survey and this specific question in mind from the start. What actions will you take based on the analysis? This determines the level of detail you need for your codeframe. For example, with travel being a key theme in the ‘bucket list’ question, determining whether coding ‘travel’ is enough to fulfil the objectives, or if you need more specific detail, such as ‘travel to India’. If you are a travel business you probably want the detail, but if you are doing a PR survey, ‘travel’ may be enough.

2. Consider the question type

Is this a simple single response question, or could the respondent express multiple themes within each response? If it’s the latter, the tool you use to process and analyse your data will need to handle multiple codes for each response given. Whilst Excel can handle single coded responses, it doesn’t handle multicoded data gracefully. Beyond that, your answers could be ‘short text’ such as brands, or longer, richer verbatims, such as a “reasons why..?” question. codeit provides specific functionality to maximise the efficiency of each.

3. Review a sample of answers first

Before you can create a codeframe, you need to get a sense of the data by reading some of the answers first to see what themes are emerging. Think about how to structure your codeframe, e.g. a code ‘service’ may be more helpful if it is split into ‘good service’ and ‘poor service’. Review enough answers to design a draft codeframe. Traditionally this would be ten percent of the data set, but dedicated survey platforms such as codeit allow you to follow a much more agile approach, where you can adjust your codeframe on the fly, adding, editing, splitting and merging codes later should you need to.

4. Use ‘Nets’ and ‘Sub nets’

It's usually useful to analyse results both at a granular and high-level. Nets allow you to do this by grouping codes into overall parent categories. This approach helps you summarise your data at a high level whilst retaining the granular detail beneath.

For example, in our bucket list survey, we see that Travel is a major theme with around 31% of people citing something travel related. Beneath the overall Travel net, we can break the numbers down into further levels, such as country and city - for example, North America, Hawaii, Las Vegas, etc.

This is a key benefit of using a dedicated coding platform like codeit as these numbers will be automatically updated in real-time as you code.

5. Build your codeframe

Build up your codeframe in a logical way, adding new codes as required and grouping similar codes next to each other within a net. The example above shows a logical hierarchical structure, with high-level concepts (e.g. "Experiences", "Money", "Travel") containing more granular child elements (e.g. Region, City). A dedicated coding platform such as codeit will enable you to develop your codeframe in a fluid, flexible way as the appropriate structure evolves and becomes clear to you.

Another useful technique is to include catch-all `other` codes within your Codeframe. These can be at the overall codeframe level or added within nets (e.g. "Travel (other)" above). While you're coding, if a response doesn't fit into an existing code, you can assign it to an 'other' code temporarily as a placeholder. If you start to get a build up of similar responses in 'other' you can add a new code to your codeframe and recode these items to reassign them to the new code.

6. Start coding

Once you have your initial codeframe, begin to review each answer and assign codes to each accordingly. If a new theme appears that is not covered by the codeframe, use the search features in codeit to decide if this is a significant theme and therefore requires a new code in the codeframe. Alternatively, code the verbatim as ‘other’ and review later.

If you have a large dataset, you may want to assign additional coders to your project. For this to be effective, the team should be briefed on the research objectives to ensure they all code consistently. Dedicated coding platforms such as codeit are designed for multiple coders to login and code from anywhere in the world, and for handling multiple users working simultaneously on the same project.

Once you have coded a reasonable set of verbatims (usually around 500 or more), you can consider using the Machine Learning built into the codeit platform. The coding you have done manually serves as a set of training examples for the Machine Learning to learn from. It will use this to autocode as many items as possible, leaving anything it is unsure of for you to review manually afterwards.

7. Check quality

Once coding is complete, it should be checked. If multiple people have been working on it, then you need to ensure that each person has coded the data in a consistent manner. Typically, a random sample of ten percent should be checked. codeit has quality control tools built in, allowing verification to happen seamlessly and effectively.

If you would like to see for yourself how codeit supports these seven steps, sign up for a 30 day free trial to help you start coding and analysing your open-ended question data.