Transforming Panels: how is Kantar using LLMs to improve panel responses?

In Kantar’s Profiles Division, we ask our panelists over 600 million open-ended questions each year, across 100+ panel markets, on behalf of our clients. This is a massive operation, managed with care and attention to ensure the highest quality data and insights for our clients. As we enter an age where respondents’ attention spans are shortening and qualitative data collection is becoming increasingly taxing on respondents, we need to leverage technology such as LLMs to help us understand the interaction between the respondent and the survey content to improve survey engagement and sample quality. Continuous improvement is the name of the game. 

In our first piece about Gen AI, we discussed what LLMs could mean for the market research industry. And in a recent article about synthetic sample, we zoomed in on the specifics of evaluating how good Generative AI is at generating responses today. In this piece, we focus on tools that we are working on behind the scenes to drive efficiency and accuracy in our panels and data analysis. 

We are creating (and preparing to create) a range of tools to support our panel operations, leveraging the ChatGPT API, aimed at bringing higher efficiency in manual tasks. It can also improve accuracy in manual tasks, further automate manual tasks, objectively measure processes and improve consistency in the data collected.  

Our tools leveraging ChatGPT support 50+ languages for our global and local clients on a multitude of projects. The possibilities are almost endless. But with new uses, we need to carefully test these projects – comparing AI and human work side by side, to understand levels of accuracy and where AI can work in a hybrid way with people to produce the best results. 

To bring our use of AI to life, here’s a brief description of some of the use cases we are working on: 

Anti-Fraud use cases: 

 

• Our OpenQ tool evaluates open-ended responses in real-time, looking at key measures of Relevancy, Originality, Completeness, and Language. The AI is trained on human responses (of course) and then can judge responses based on each of these measures and score them. 
• This means we can weed out responses that don’t count – saving lots of time. And where there’s doubt, human intervention helps to resolve these cases, and the data is fed back to the AI to improve future accuracy and precision. 
• We also use our AI to detect brands, gibberish, plagiarised answers, slang, profanity, racism, and Personally Identifiable Information (PII) in the answers that are given. We can even check for answers that have been generated using ChatGPT. These responses can be discounted or flagged, as appropriate. 

Post-analysis use cases: 

 

• For multimarket projects we can automate language translation, for example in open-ended responses where the client requires the project to be delivered in one language. We can be confident in answers, even if they are in a different language. 
• We use AI to automate the generation of a data summary. This can save hours of time, and then someone can check and refine this summary where required. 
• Incomplete Data Analysis: where data is incomplete, we can use AI to understand what’s missing, identify why the gap might be occurring, and flag it. 
• Discard Low Quality Data: we use AI to assess data quality and it can be trained to discard low quality data, for example preventing copy/pasted answers making it into the data. The AI can also efficiently identify and score low quality data for discarding. Not only that but it means we can also find reasons for low and high scores – so we can have the confidence of knowing the AI is working. 
• Conversely, the AI can also score great answers, so we can highlight them to clients and even automatically reward users for good answers. 
• The AI is trained to aggregate and discover points mentioned in open-end answers. It can also remove subjectivity from ‘judging’ answers, as to whether they are relevant, should be included in the dataset, etc., as well as ‘slice and dice’ user demographics to summarise and compare responses from different groups of respondents, for example by age breaks.  
  
  

Our own use cases within Profiles: 

 

• We can use AI to search for the right screening questions for respondents, generating the targeting qualifications by translating simple client emails and instructions into specific targeting criteria for setting up a study and generate feasibility in real-time.  
• We can also use AI to check for scripting errors; in translations, the wording, grammar and spelling. 
• AI can also be used for research question checks – If the test run shows low relevancy on most of the answers, it may be a badly phrased question where the respondents are confused or even discouraged to share insights. 
• Support emails – triage and screening: 
  
  • We use AI to sort our support emails from panelists into categories, eg: account/system/survey issues, redemption issues, GDPR account closures, etc. 
  • As with the open-ended questions, we can screen the language used by the user on support emails and remove those with profanity or racism, for example. 
• And finally, we can use AI to scan through Trustpilot, Playstore, Appstore reviews; then summarise and analyse the data to tell the business what it needs to do to improve respondent satisfaction. 
  
  

There are many credible use cases for LLMs across the market research industry.  

At Kantar we are committed to leveraging AI, using our own data to augment our products and solutions and make them more effective and faster, as well as these use cases in our panel operations. The transformation is ongoing – and it’s all for the cause of creating trustworthy insights for our clients. 

Find out more about our AI capabilities