In the realm of human-computer interaction, recognizing emotions and dialogue acts within conversations is crucial for developing natural and responsive dialogue systems. While numerous conversational emotion corpora exist, most lack annotations for dialogue acts. This gap hinders the comprehensive analysis of interactions, which requires understanding both emotional states and the performative functions of utterances. To address this, we worked on the research topic Emotional Dialogue Acts at the Knowledge Technology group from the University of Hamburg to propose an innovative method: using a neural ensemble to annotate dialogue acts in emotional corpora.
The Importance of Dialogue Acts
Dialogue acts are linguistic units that capture the intention behind utterances, such as questions, statements, or requests. These acts are essential for understanding the performative function of speech, making them integral to discourse analysis and the development of interactive dialogue systems. Combining dialogue acts with emotional states can significantly enhance the naturalness and appropriateness of responses in conversational agents.
The Proposed Solution & Annotating Emotional Corpora
In the research article, we have developed a neural ensemble annotation system that employs various recurrent neural network (RNN) models trained on the Switchboard Dialogue Act (SwDA) corpus. This ensemble annotates emotion corpora with dialogue act labels, enriching the datasets and enabling more nuanced conversational analysis. The system was applied to two prominent multi-modal emotion corpora: IEMOCAP and MELD.
The IEMOCAP (Interactive Emotional Dyadic Motion Capture) and MELD (Multimodal EmotionLines Dataset) datasets are well-known in the field of emotion recognition. IEMOCAP includes annotations for both discrete emotion categories (like joy, sadness, and anger) and dimensional attributes (valence, arousal, and dominance). MELD, an extension of the EmotionLines dataset, contains annotations for seven basic emotions and sentiment labels.
Neural Models for Dialogue Act Recognition
Five different neural models were utilized in the ensemble, each with distinct architectures and capabilities:
- Utt-level-1 DANA: An utterance-level RNN model with an attention mechanism.
- Context-1 DANA: A context-aware model using preceding utterances for improved accuracy.
- Utt-level-2 DANA: An utterance-level model using averaged word embeddings and a feedforward neural network.
- Context-2 DANA: Similar to Context-1 but with averaged embeddings for the input utterances.
- Context-3 DANA: Combines features of all previous models to enhance recognition capabilities.
The ensemble approach integrates the outputs of these models, checking for majority occurrences and ranking confidence values to determine the final dialogue act labels.
Results and Analysis
The annotated datasets revealed significant co-occurrences between specific dialogue acts and emotions. For example, the act of accepting or agreeing frequently aligns with joy, while apologies often accompany sadness. These findings underscore the intricate relationship between how something is said (dialogue acts) and what is being felt (emotions).
We also can see how sentiment is associated with the dialogue acts:
Conclusion and Availability
This enriched Emotional Dialogue Acts (EDA) corpus, combining emotion and dialogue act labels, offers a valuable resource for further research and development in natural language processing and human-computer interaction. The annotated datasets and the ensemble annotation tool are available for the research community, fostering advancements in creating more natural and emotionally intelligent dialogue systems. The code is available at: github.com/bothe/EDAs.
References
For detailed information on the datasets, models, and access to the EDA corpus, visit the article at: EDA: Enriching Emotional Dialogue Acts using an Ensemble of Neural Annotators.
