IS4152 / IS5452: Affective Computing

Department of Information Systems and Analytics
National University of Singapore

Semester 1, AY2021-2022

Instructor

Desmond Ong, Assistant Professor
  Contact: dco (at) nus (dot) edu (dot) sg

Teaching Assistants

  Varsha Suresh
  Gerard Yeo

Class Details

  Class Times: Tuesdays 4pm-6pm
    [from August - November 2021]
  Location: Zoom

Overview

Bulletin Listing

This module provides a broad introduction to the field of affective computing, focusing on the integration of psychological theories of emotion with the latest technologies. Students can look forward to learning about contemporary theories of emotion, empathy, emotion regulation; automated emotion recognition from video, speech, and text; automated affect generation in human-computer interaction; commercial affective computing technologies, including potential interaction with local startups. Students will work in groups on a semester-long project that may take several forms, such as the incorporation of emotion recognition into a prototype system, or critical evaluation of commercial affective computing technologies.

Course Overview

Everyone has emotions, and so, being able to use automated information systems to interface with people's emotions is becoming an important --- and commercially lucrative --- capability for many technology companies. This course provides a broad introduction to psychological theories of emotion and how modern technologies are used to recognize, reason about, and generate emotions, and there will be a strong emphasis on critically evaluating contemporary affective computing technologies, including a discussion on the latest developments.

We will begin with an introduction to scientific theories of emotion and how scientists define and measure emotions. We will then proceed with a survey of current affective computing technologies: What are the technologies that power facial emotion expression recognition, emotional speech recognition, text-based emotion analytics, as well as analytics from physiological sensors and other modalities? How do we (and how should we) design emotion-based applications, and emotional agents like chatbots, virtual characters and robots? What are the current capabilities as well as the current limitations of today's affective computing technologies, and what are the barriers to progress?

This course covers contemporary material that is at the forefront of modern psychology and technology. Throughout the course, we will rely mainly on reading recent academic papers, both theoretical and empirical. To solidify understanding and also to apply what they have learnt, students will work in groups to propose, design, and execute a semester-long project. Projects can take many forms, such as a critical evaluation of an existing technology or the development of new software or app that harnesses technology to gain more insight into their users' emotions.

Learning Objectives

By the end of the course:

  • Students will be able to discuss scientific theories of emotion, and assess the scientific validity of affective computing technologies.
  • Students will be able to discuss recent advances in affective computing technologies, including Affect Recognition technologies (from different modalities) and Affect Generation technologies.
  • Students will be able to discuss and analyse various commercial applications of affective computing.
  • Students will complete a research project that examines affective computing technologies within a certain context.

Brief outline of topics

  • Theories of Emotions and Emotion Measurement
  • Emotion Recognition
    • From images (facial expression analysis)
    • From natural language text (e.g., sentiment analysis, applicable to social media like Twitter)
    • Multimodal Emotion Recognition
  • Emotion Modelling
  • Emotion Generation
    • Virtual Agents
    • Affective Robotics
  • Ethical Issues in Affective Computing
  • Affective Computing Companies
    • Large Companies and Small Startups / Multinational and Local Companies
  • Affective Computing Applications
    • Emotion Regulation and Mental Health
    • Affective Computing for Assistive Technologies
    • Affective Tutoring
  • Special Topics (time permitting)
Depending on schedule availability, we may also invite guest lecturers from other departments, schools, or industry, to share their expertise with us.

Logistics

Pre-requisites

Students are assumed to have a high level of maturity and competency in programming and coding: no programming will be taught in this class. The course content will also require some familiarity with machine learning. Depending on the final course project they choose, students may require skills in machine learning, video and other data analytics, software development.

Readings

Readings will mostly consist of academic papers and excerpts from selected books. They will be made available on LumiNUS. Students are expected to do the readings for a particular week before class, in order to facilitate a productive and intellectually stimulating discussion in class. To incentivize keeping up with the readings, there will be short weekly quizzes.

In the reading list (appended at the end of the schedule), there is also a set of optional readings and other resources for the interested student (which could be relevant to class projects). A handy reference is the following Handbook, which summarizes many of the important topics in Affective Computing:

  • Calvo, R. A., D'Mello, S., Gratch, J. M., & Kappas, A. (Eds.). (2015). The Oxford Handbook of Affective Computing. Oxford University Press, USA.
The full text is available online via NUS libraries [link, then click Access via Oxford Handbooks Online]

Grading

  • 35% Weekly Quizzes
  • 50% Class Project
    • 10% Project Proposal : Due Week 6
    • 20% Final Project Presentation : Due Week 13 (at showcase)
    • 20% Final Project Report : Due Week 13
  • 15% Class Participation (including forum discussions)

Forum Discussions

We will be using Piazza as a forum. Please use the forum to discuss questions relevant to affective computing, project implementation issues, or share/summarize relevant articles.

Quizzes

There will be weekly quizzes conducted at the start of each class, assessed on the readings of that class. Instructions will be given in class.

Expectation / Time commitment

This will be an intensive course. There is no homework. But there are many readings which are essential to your learning, quizzes to test your understanding, and a semester-long class project. How much time the class project takes up will depend on a lot of factors: your ambition, your existing skills, the learning curve for skills you need to pick up. As with all research and development projects, I assure you that if you put in the required work, you will finish the class with a final project that you will be proud of and that showcases what you have learnt.

Academic Integrity

  • Plagiarism is unacceptable. If we find any evidence of plagiarism (for example, copying text wholesale from a website, an academic paper, or another source without proper citation) or any other forms of academic dishonesty (for example, cheating on a quiz), you will receive a zero, and you may face disciplinary action by the university.
    • Exceptions: You may use portions of your Project Proposal in your Final Report, i.e., within the class (and you can do this without citing your Proposal).
  • But: you may not use portions of any of your previous work outside this class (previous class papers; previous or current research projects) without the prior approval of the instructor.
  • Note that this also applies to code that you write. I'm a big fan of open-source code and will recommend several open-sourced resources throughout the course, but you should still cite your sources, especially if you modified or adapted a complex piece of code. If it forms a big part of your project, I expect citation in the project writeup. Otherwise, commented citations in your code will suffice. Merely citing does not license you to use copyrighted material, so do not use any copyrighted or proprietary software without appropriate permission. If in doubt, ask the instructor.
In short: don’t steal, and cite your sources, especially if you borrow or are otherwise influenced by ideas, text, or code. References and citation should be done in APA format.

Feedback

We welcome feedback on the course at any point. Feel free to email the instructor directly, or leave anonymous feedback by using the anonymous Google form (URL given in the syllabus on LumiNUS).



Class Project

The class project offers you an opportunity to apply what you have learnt in the class. The final outcome will vary depending on the choice of project, but will involve a substantial amount of research and/or development. Projects can take many forms, such as the incorporation of emotion recognition into a prototype software system or app, a research project implementing the latest techniques in machine learning to affective computing, a critical evaluation of an existing technology offering that might also include a discussion of how a particular technology is implemented and their ethical ramifications.

Students are expected to do a substantial amount of the background reading of the emotion literature in order to inform the project, or do critical research on existing technologies. You must base your project on compelling scientific prior work. Even if you decide to build something completely new that has not been studied before, there must be related work that will inform your proposed project.

The scope is open-ended, and you and your group members will come to a consensus as to a feasible (and mutually agreeable) project. This project will have you apply what you have learnt about the psychological theories of emotion with a keen sense of technology, and will solidify the learning objectives into a final, tangible product to be displayed in a public showcase. Unless students opt-out, final project write-ups (and any associated demos/material) will be made publically available and archived on the course webpage.

More information will be provided in class.

We assume that students (being final-year undergrads in a School of Computing) will be familiar with good coding practices, such as using version control. We expect that students will make all their final project code available via a public GitHub respoitory. Please get familiar with GitHub and version control if you are not already, and consider looking into requesting an education account at https://education.github.com/, which among other things, allows you to have private repositories (which some students may be more comfortable with for works-in-progress).

If you are using data (and most of the projects will), please bear in mind any restrictions about publically sharing the data. In the majority of cases, I recommend to only commit CODE to GitHub, and not to commit any data (e.g. videos) unless you have obtained the necessary permissions from the data owners or other relevant parties. When in doubt, ask the instructor. (Do not even commit data to a private repository, because you might make it public in the future).

Class Project Template

We will be following the Affective Computing and Intelligent Interaction (ACII) conference format for the project proposals and final reports. The ACII template (LaTeX and Microsoft Word) can be found here. (If the link is broken, there is a local copy here). We recommend using the Overleaf template, which is here. Overleaf is like Google Docs for LaTeX and allows you to collaborate on documents with your groupmates in real-time; NUS has a professional subscription, or you can also create a free account.

List of past projects

* Showcased at 19th STePS (SoC Term Project Showcase) on 10 Nov 2021.
  • Do Machines Discriminate? Examining the Hidden Gender and Racial Bias in Speech Emotion Recognition AI
    • Kingsley Kuan, Wong Wei Kang, Jeremy Sim, Saja Bashanam, Shao Guoxin [ github ]
  • LittleGenius: A Novel and Intelligent Chatbot for Detection and Treatment of Mental Disorders
    • Sun Haowen, Huang Yixuan, Wang Yixuan, Jing Yu, Zhou Zekai [ github ]
  • Online multimodal depression detection
    • He Qinxing, Shi Shu, Duan Yuxin, Sun Qicong, Zhu Qianru [ github ]
  • Investigating Affective States with Music
    • Akanksha Bansal, Raakhee Sadhnani, Vijaya Tripathi, Wong Zheng Yang [ github ]
  • The challenge of Affective Computing on Emotional Support Platform
    • Mia Huong Nguyen, Matthew Yong Chee Xian, Thai Lam Thuy Quyen, Ryan Wong [ github ]
  • Emotional 3D Avatar Visualization using Sentiment Analysis Model
    • Zhong Yanbin, Wang Xiaojin, Pan Yijia, Liang Jianheng [ github ]
  • Emergency Dispatcher Assistance System (EDAS)
    • Zhi Jun, Jiang Luyu, Hou Xueyu, Xi Jiri [ github ]
  • Understanding Sentiment in Memes using Knowledge-Aware Information


* Showcased at 17th STePS (SoC Term Project Showcase) on 17 Nov 2020.
  • Sentiment Analysis Dashboard for 2020 US Presidential Election
    • Cheng Lin Pak, Choon Kiat Kang, David Choo, Jia Yun Teo [ github ]
  • Investigating Transfer Learning of Affective Computing Models
    • Kong Yan San, Jonathan Soh, Jing Lin, Deng Jingyuan [ github ]
  • Multi-task BERT for Emotion Recognition from Textual Conversations
    • Varsha Suresh [ github ]
  • Multi-Agent Appraisals for Emergent Emotions in Reinforcement Learning Agents
    • Joel Huang, Gnanapoongkothai Annamalai [ github ]
  • Emotional Speech Synthesis in English Using GST-Tacotron 2