Overview
A robot that responds to how people feel has to answer three questions: when to be empathic, what to say, and how to show it. In this lecture I present how we answer each one on Haru, Honda's tabletop social robot.
Part 1, Responding Empathically, covers detecting when a user is seeking empathy, guiding large language models toward empathic replies with empathy cues, and expressing empathy through backchannels, facial and motor expressions, and an expressive voice. Part 2, Expressing Emotion, shows how emotion-aware co-speech gestures keep emotion visible while the robot speaks.
Slides
Videos
Full lecture
-
When, What, and How: Designing Empathic Behaviors for Social Robots
Invited lecture at the AFAR Lab, University of Cambridge, November 17, 2026. Recording coming soon.
Demos shown in the lecture
-
Part 1Haru's replies with and without empathic text generation. Papers: RO-MAN 2025, HRI 2025 YouTube
-
Part 1Classifying the behavioral empathy a listener shows from multimodal cues. Paper: IEEE Access 2026 YouTube
-
Part 1Haru expresses empathy through verbal and nonverbal cues. YouTube
-
Rhythmic-Semantic Co-Speech Gestures
Part 2Haru mixes rhythmic beat gestures with emotion-specific iconic gestures while it speaks. Papers: HAI 2026, EEITE 2026 YouTube
Papers
When to be empathic
-
When and How to Express Empathy in Human-Robot Interaction Scenarios
RO-MAN 2025IEEE International Conference on Robot and Human Interactive Communication (RO-MAN), Eindhoven, Netherlands, pp. 1070–1077
-
Empathetic Robots Using Empathy Classifiers in HRI Settings
HRI 2025Proceedings of the 2025 ACM/IEEE International Conference on Human-Robot Interaction (HRI ’25), IEEE Press, pp. 1211–1215
-
EmpatheticExchanges: Toward Understanding the Cues for Empathy in Dyadic Conversations
IEEE Access 2024IEEE Access, vol. 12, pp. 195097–195110
What to say
-
A Survey on Integration of Empathy Cues into Empathetic Text Generation
IEEE Access 2025IEEE Access, vol. 13, pp. 140791–140815
-
Behavioral Empathy Classification in Dyadic Interactions Using Multimodal Cues
IEEE Access 2026IEEE Access, vol. 14, pp. 64054–64074
Co-speech gestures
-
Rhythmic-Semantic Co-Speech Gestures for Emotional Dialogue in Robots
HAI 2026Proceedings of the 14th International Conference on Human-Agent Interaction (HAI ’26), ACM
-
Efficient Emotion-Aware Iconic Gesture Prediction for Robot Co-Speech
EEITE 20267th International Conference in Electronic Engineering & Information Technology (EEITE)
Datasets & Code
Ours
-
whEE: When and How to Express Empathy
Code, prompts, and results for detecting when a user seeks empathy and generating empathic replies. Includes EDR (Empathetic Exchanges with a Robot), conversations recorded in the presence of Haru. GitHubRO-MAN 2025
-
Conversations from EmpatheticDialogues split into speaker–listener exchanges, each labeled for behavioral empathy and annotated with empathy cues: sentiment, valence–arousal–dominance, EPITOME mechanisms, empathetic intents, and mimicry. Also comes in an extended version with more labeled conversations. GitHubIEEE Access 2024
-
mEX: Multimodal EmpatheticExchanges
Text exchanges with behavioral-empathy labels plus video-based cues (arousal and valence, mimicry, speaker status), with the analysis notebooks and multimodal baselines. GitHubIEEE Access 2026
Public datasets we use
-
Sentences labeled as seeking empathy, providing empathy, or neither. Hosseini & Caragea, “It Takes Two to Empathize: One to Seek and One to Provide,” AAAI 2021. GitHubPaperCSV in whEE
-
Tweets annotated for empathy. Hosseini & Caragea, “Distilling Knowledge for Empathy Detection,” Findings of EMNLP 2021. GitHubPaperCSV in whEE
-
Framework and dataset for the three ways empathy is expressed in text: emotional reactions, interpretations, and explorations. We use these mechanisms as empathy cues and in our text-generation prompt. Sharma et al., EMNLP 2020. GitHubPaper
-
60 hours of motion-captured co-speech gestures from 25 speakers with a wide range of emotions; the training data for our gesture models. Liu et al., CVPR 2024. Project pagePaper