At inVibe, we use machine learning technology to recognize and measure emotionality in our respondents’ voices when they’re speaking. To humans who have spent a lifetime decoding emotion, this task seems very simple. However, getting a computer to decode emotions like a human can is a bit more complicated than it seems. To understand exactly what we’re asking computers to do, let’s explore the nature of emotions, what it means to “recognize” one, and how humans do it.

When we think about emotions, the first ones that come to mind tend to be “basic” emotions like 

. Social scientists refer to these emotions as “basic” because—compared to other emotions—they’re simple. In fact, they’re so one-dimensional and consistent that children tend to recognize them as young as 3 to 4 years old—about 4 years before they reliably recognize more complex emotions like 

After a lifetime of interacting with others, we all have intuitions about how people express their emotions. In real life, those intuitions are generated from a variety of sources: facial expressions, body language, tone of voice, and situational context, among others. What if we couldn’t rely on all of these behaviors and contextual elements? What if, like a computer, we had to recognize emotions from just the tone of voice? No context, no body language, no facial expressions, just voice. How would we do it?

Let’s take this example out of the abstract and give it a try for ourselves. Below are 5 examples of an actor saying “He told me he was moving away” with each performed with a different basic emotion (

). Listen to the following examples and try to intuit which audio corresponds to each emotion. Once you’re ready to see the answer, click the static below the “EMOTION” header to reveal the corresponding emotion.

How’d you do? Maybe this was harder than you expected, maybe your intuitions matched all 5 correctly. Tasks like this are hard for most people—certainly harder than recognizing the emotions of someone you’re speaking with directly—and most people are only able to guess emotions at just above random chance in tasks like this (about 1 in 5 or 20%). We’re asking much more from computers, which are expected to be at least 80% accurate, identifying 4 out of 5 emotions correctly.

Let’s explore how humans decode emotionality in speech. When the only thing that most of us are aware of is that person “sounds 

,” it’s hard to tease apart how we came to that intuition. Think about your own thought process as you tried to decode the above examples. What were you listening for? What clued you into the emotionality in the audio examples? Well, if you’re like most adults, you probably noticed differences in vocal qualities like 

 (smoothness), or any other of the 30-odd other human-perceptible vocal qualities. If your intuitions are anything like mine, your descriptions for how these 5 emotions sound may look something like this.

Taken together, subtle differences across these 4 vocal qualities are enough to uniquely identify these 5 basic emotions. But the differences within each feature are just that: “subtle.” Over the course of childhood, we learn to associate certain vocal qualities with particular emotions through repeated exposure in social interactions. And, through this exposure, we learn to adapt to tremendous amounts of variation by speaker and by circumstance. Take pitch for example.

Is my ‘quieter’ the same as yours? ‘Louder?’ ‘Neutral?’

Is my ‘quieter’ right now the same as it was yesterday or it will be tomorrow?

How much louder is a ‘loud’ emotion like 

Are these pitch contrasts consistent for speakers of American English and British English? What about speakers of French? Chinese?

) be? Does it pick a quality from one emotion, take the average, something else altogether?

Whether we can articulate answers to these questions or not, we all have robust intuitions on the answers to these questions in real life conversations. The problem becomes then “How do we move beyond intuition? How can we prove our intuitions are accurate?” At inVibe, we solve these problems by using Speech Emotion Recognition (SER) technology to measure emotionality in the voice. In our next blog, I’ll discuss what SER is, how it works, and how we use it to power our linguistic insights.

If you’re interested in learning more about inVibe, our listening-centered approach to research, or how we can support your business objectives, 

Discover how inVibe uses machine learning to decode emotions in voice data, exploring the complexities of getting computers to recognize feelings as humans do.

Hearing and Intuiting Emotion in the Voice

National Breast Cancer Awareness Month (NBCAM) is observed annually in October to help spread awareness, increase early detection rates, and provide public education about breast cancer.

Breast cancer is the most common type of cancer in women in the United States aside from skin cancers. The NBCAM initiative helps to encourage individuals to get screened, donate to relevant foundations, and take part in events that support research and patient care.

Living with breast cancer involves navigating a range of challenges, from managing physical symptoms like pain as well as treatment side effects like fatigue, nausea, and hair loss to coping with the emotional and mental impacts of the living with the condition.

Let’s listen to a 41- to 45-year-old woman with breast cancer describe what it’s like for her to live with this condition and some of the many challenges that she faces.

Question 1, Response 15

Understanding the patient voice is at the heart of 

, which we approach holistically by analyzing what is said, how it’s said, and how it sounds.

Looking first at what she says and how, this patient describes going through treatment as physically, financially, and emotionally difficult. She emphasizes the burdensome nature of living with breast cancer by repeating the linguistic intensifier ‘very’ three times: “

 sick.” She stresses how the physical impacts of living with cancer have changed even how she sees herself, describing how cancer has “all of a sudden” made her “not be the same person.”

, her description of the challenges and frustrations of living with breast cancer and going through treatment is expressed with strongly negative emotionality (‘valence’), which emphasizes the multifaceted difficulty of undergoing breast cancer treatment.

Taken together, her response suggests a need for advocacy groups to move beyond awareness to understanding and addressing the problems associated with living with and treating breast cancer.

Our Listening Platform makes it simple for patients to speak with us about their experiences in an open and authentic way and easy for you to engage with their perspectives directly.

If you’re interested in hearing from more patients like this one so that you can gain insight into their experiences, beliefs, behaviors, and desires, 

In Recognition of National Breast Cancer Awareness Month (October 2024)

Anyone who has conducted research will likely agree that the process of collecting data is one of the biggest hurdles to completing the project on time and on budget, which can especially be challenging for traditional qualitative studies. Data collection often demands large-scale efforts to compile relatively small datasets with findings that some regularly dismiss for lacking statistical power.

Now, the snowballing popularity of AI-driven approaches to conducting research has given rise to a new concept to entice those who seek a lower friction method of obtaining qualitative insights: synthetic qualitative data.

You may be wondering if you read that correctly. Does “synthetic,” mean 

not spoken, written, or otherwise communicated by living, breathing participants

? Again, yes. Synthetic qualitative data, one of the latest applications of large language models (LLMs), seeks to supplement—or even supplant—qualitative datasets comprised of real responses collected from real people.

The basic process of generating synthetic qualitative data is alluringly simple. First, a researcher feeds a small sample of existing real-world data into an LLM—such as ChatGPT or Claude.ai. Then, the model generates new machine-authored outputs that—ideally—mirror the complexity and quality of the original data. While such a concept may be especially tantalizing in the high-speed field of market research—where large-scale access to consumers’ voices is always needed and timelines are short—it is critical that we pause and dip our toes in to test these unknown waters before jumping headfirst into them.

 cases in which synthetic data can be used ethically and effectively. Early touters of synthetic qualitative data and its potential value have already begun to define ground rules and best practices. They tend to emphasize reserving synthetic data for situations where no other viable options exist, such as:

Difficult-to-find or non-existent target audiences

Need for testing a new methodology or design aspect

However, these guidelines currently inhabit the realm of theory rather than reality, tending to be offered without concrete examples of how they have led to successful outcomes in actual qualitative studies. This, of course, raises the question of whether proposed guidelines will yield results comparable to those from a larger sample of human respondents.

Let’s assume that synthetic responses can be generated to mirror the complexity and quality of the real responses from which they’re generated. How, then, can we determine that they are not—at best—an amplification of the smaller, human sample’s original distribution of sentiments and perceptions? In other words, how can synthetic responses reliably achieve the intended goal of predicting 

sentiments and perceptions that researchers 

 have heard if they conducted interviews with a larger sample? To answer these questions, several studies have attempted to demonstrate that predominantly synthesized samples consistently lead to similar findings as a fully human sample of the same size. So far, they’ve had less-than-promising results.

The point here is not to completely dismiss the idea of using synthetic data. 

 data when cost, time, or other constraints make existing methodologies impossible—though we all agree it is always preferable to obtain data from human participants. However, until guidelines and best practices are developed and proven in the field, the best course of action should be to proceed with caution.

Considering again the early ground rules for ethical and effective usage of synthetic data, we at inVibe see potential for synthetic responses when testing aspects of our research designs. For example, our unique methodology requires us to carefully craft our survey prompts to elicit responses that are detailed and on-topic. To amplify the power of our 

, we can use a custom LLM to generate synthetic responses iteratively on draft questions, which will allow us to predict which versions are expected to lead to the best responses from our human respondents. In effect, synthetic responses would allow us to conduct rapid pre-tests, ensuring that our recommended design will elicit the high-quality responses that our clients have come to expect.

While new use-cases will likely continue to emerge as various industries adopt and experiment with synthetic data, inVibe’s one-of-a-kind methodology bypasses the need to conduct actual analysis on synthetic responses. Specifically, our HIPAA-compliant anonymization and secure data storage procedures eliminate concerns about data privacy. Our panel partnerships connect us to a vast network of stakeholder audiences, making even the rarest audiences accessible. And lastly, our automated voice response platform allows us to conduct 

 responses in a cost- and time-efficient manner. So, if you need to talk to real stakeholders but can’t figure out how to reach them in the numbers you require, 

Anyone who has conducted research will likely agree that the process of collecting data is one of the biggest hurdles to completing the project on time and on budget, which can especially be challenging for traditional qualitative studies. Data collection often demands large-scale efforts to compile relatively small datasets with findings that some regularly dismiss for lacking statistical power. Now, the snowballing popularity of AI-driven approaches to conducting research has given rise to a new concept to entice those who seek a lower friction method of obtaining qualitative insights: synthetic qualitative data.

Synthetic Data in Qualitative Research

World Atopic Dermatitis Day is an annual event observed on September 14th, aimed at raising global awareness about atopic dermatitis, also known as atopic eczema. Initiated in 2018 by the European Federation of Allergy and Airways Diseases Patients’ Associations (EFA) and the International Alliance of Dermatology Patient Organizations (GlobalSkin), this day is intended to help build a global movement for awareness, change, and improvement in the lives of those affected by atopic dermatitis.

Atopic dermatitis (AD) is an inflammatory skin condition characterized by severe itching, rashes, and painful flareups. AD is one of the of the most prevalent skin diseases worldwide, but its impact remains significantly under-recognized. It is often dismissed as "just a skin condition," leading to inadequate support care for patients.

Let’s listen to a 46- to 50-year-old woman with AD describe what it’s like for her to live with this condition and how it impacts her daily life.

Atopic Derm Day Q1.R7

First breaking down what she is saying and how, this patient highlights the difficulty of living with AD due to her ongoing symptoms. She intensifies both her symptom experience and its impact, describing it as “very” painful and as making it “really hard” to complete daily tasks.

, she describes her experience with strongly negative emotionality (‘valence’), particularly when talking about impact on daily life activities and the ongoing, constant nature of her symptoms.

Considering both what is being said and how it sounds, this response points to the persistent challenges patients living with Atopic Dermatitis can face, as well as the need for treatments that provide better symptom control and allow patients to live the lives they want.

makes it simple for patients to speak with us about their experiences in an open and authentic way and easy for you to engage with their perspectives directly.

World Atopic Dermatitis Day is an annual event observed on September 14th, aimed at raising global awareness about atopic dermatitis, also known as atopic eczema. Initiated in 2018 by the European Federation of Allergy and Airways Diseases Patients’ Associations (EFA) and the International Alliance of Dermatology Patient Organizations (GlobalSkin), this day is intended to help build a global movement for awareness, change, and improvement in the lives of those affected by atopic dermatitis. 

In Observation of World Atopic Dermatitis Day (September 14)

The Patient Perspective

Decoding the Patient Voice

Simple, Systematic, Scalable

Thanks for reading!

Hearing and Intuiting Emotion in the Voice

In Recognition of National Breast Cancer Awareness Month (October 2024)

Synthetic Data in Qualitative Research

/voice

/product

/resources

/use-cases

/company

@social