Want to know how to code mountains of messy business survey feedback into actionable insights?
Look no further…
Open-ended survey responses are a fantastic source of data. They reveal subtleties of the customer experience that multiple-choice options never will. However…
There is a problem. Most businesses end up buried under oceans of unstructured text data. If they lack a robust system for organising and coding those responses, valuable insights will get lost.
Thankfully, there are some tried-and-tested techniques to turn chaotic verbatim responses into clean, analysable data.
In this guide, we’ll cover:
- The core principles of advanced survey coding
- Four advanced techniques that actually work
- How to maintain quality as you scale the process
Why Businesses Struggle With Open-Ended Responses
Open questions give the survey participant some freedom. That freedom creates richer data. But it also creates big headaches.
Have a think about this…
When a business asks a customer the open question “What could we improve?”, the responses can come back in hundreds of different ways. Short, long, neatly written, or misspelled, with bullet points, smiley faces or even emoji.
This is where coding open ended surveys becomes an essential step in the process of making sense of qualitative customer feedback.
The process of coding qualitative data is assigning labels to pieces of text, so responses can be grouped together, counted, and even analysed statistically.
Without coding qualitative data the business has two choices:
- Spend hours, or even days, reading every single response
- Ignore the data entirely, losing all of the effort that went into collecting it.
Neither option is good, of course, which is why businesses need advanced coding techniques.
The Core Principles of Effective Survey Coding
Before we jump into the cool stuff, there are a few rules that every business must follow if they want their data analysis to be successful.
Principle 1: Create a master code frame in advance. The code frame is the list of codes, or categories, into which responses will be sorted. A code frame should have no more than 30-40 codes, any more than that, and the process starts to slow down. Coders can’t remember so many codes in their head.
Principle 2: Establish clear definitions. Each code in the code frame must have a precise definition. For important codes, include examples in the definition. The purpose is to maximise consistency between coders and over time.
Principle 3: Plan for an “Other” category. Inevitably, some responses will fail to match any code. From a quality perspective, it is a good rule of thumb that less than 10% of responses should end up in the “Other” category. More than that, and there are too many responses that are slipping through the net unlabelled.
Principle 4: Allow multiple codes per response. Open-ended survey questions often result in responses that have multiple topics, or themes, within them. For example, if someone says “The product is great but the shipping was slow”, they should get two codes: one for satisfaction with the product AND another code for a complaint about shipping.
Pretty straightforward? Yes, now let’s get to the cool stuff…
Four Advanced Coding Techniques
These are the techniques that make the difference between amateur survey analysis and professional-quality insights.
Technique 1: Iterative Code Frame Development
One mistake that businesses make is building the entire code frame before they even start coding responses. Don’t do that. Rookie mistake.
Iterative code frame development looks like this: start by reading the responses and noting common themes. At least 10% of the responses need to be read before you start calling things. Then you create the codes to describe those themes. Then you test those codes against another sample of responses. Rinse and repeat.
Iterative code frame development means that as common themes emerge in the data, they get captured with codes. This approach catches themes that wouldn’t be obvious by reading the responses once, or by simply guessing based on what you think your customers are going to say.
Technique 2: Hierarchical Coding Structures
Flat lists of codes work fine for straightforward surveys. But complex business feedback often needs multiple levels.
Organise the codes into:
- Parent categories. Broad themes, such as “Price”, “Quality”, or “Service”
- Child codes. Specific sub-themes within each parent.
The benefit of hierarchical coding is it allows analysis at different levels. Sometimes senior leadership just wants to see high-level category summaries. Sometimes, product teams need fine-grained detail. A hierarchical coding structure can support both needs.
Technique 3: Sentiment Layering
Remember, it’s not enough to know what the customers are mentioning in their responses.
Businesses also need to know how customers feel about each theme that comes up. Is it a positive mention, a negative mention, or are they being neutral or mixed in their sentiment?
Add sentiment tags to the coding process:
- Positive mentions
- Negative mentions
- Neutral/mixed mentions.
Take the earlier example of a mention of “price”. The customer could be saying your prices are too high OR they could be saying that prices are fair. Sentiment layering captures the difference between those two very different things. Without sentiment, a business loses an entire layer of emotional context that can help to drive decision-making.
Technique 4: Hybrid Manual – Automated Coding Approaches
Listen up, businesses…
The debate about whether to code manually or automatically has become outdated. The smartest approach is to use both.
According to research from NORC, AI can now categorise text data at scale more efficiently than humans can while also replicating the quality level that humans achieve. But businesses should still employ human oversight for nuanced interpretation.
A hybrid coding workflow is:
- First use automated or AI tools for initial sorting and bulk categorisation
- Apply human review to deal with ambiguous responses
- Conduct final quality checks on a random sample
The hybrid approach is superior because it gets the speed and volume benefits from automation without compromising quality. And the efficiency gains are real. 90% of workers say AI saves them time on work tasks.
Maintaining Quality as you Scale the Process
Coding 500 responses is hard, but possible. Coding 5,000 responses? That’s an entirely different level of challenge.
Some tips to get the process right when working with large volumes of qualitative
Batch your work. Code no more than 50-100 responses at a time. Cognitive fatigue starts to affect consistency after that. Take regular breaks.
Apply inter-coder reliability checks. If you have multiple coders working on the same data, there is a check to ensure quality over time. Coders independently code a random sample of the dataset. Compare results. Disagreements highlight ambiguity in the code definitions that needs correcting.
Document decisions. Note everything that requires a coding decision. When you refer back to the dataset later, documentation prevents inconsistent re-interpretation of responses.
Add quality controls to the workflow. Simple rules, like “Review any response coded to ‘Other’ before finalising” catch miscategorisation before it happens.
Scaling up doesn’t mean you lower standards. The businesses that get the most value out of open-ended surveys are the ones that maintain rigorous quality standards even as data volumes grow.
Wrapping Up
Advanced coding techniques can turn business surveys with open-ended questions from overwhelming heaps of text into strategic assets.
The fundamentals of survey coding are important. A well-built code frame, clear code definitions, and a systematic approach matter. However, adding a layer of sophisticated techniques like hierarchical structures, sentiment layering, and hybrid automation are what separate the best-in-class businesses from the also-rans.
To quickly recap:
- Iterative code frame development – build codes by reading responses
- Hierarchical coding – allows analysis at different levels of detail
- Sentiment layering – capture how customers feel about topics
- Hybrid AI approaches – combine the best of machine and human coding
- Don’t lower quality when scaling – just get smart about the process
Done well, advanced coding techniques unlock powerful competitive advantages for the businesses that do it. Because now the business doesn’t just know what their customers are thinking, they really understand it.
Read More From Techbullion
