Course logo

Selected Topics in Human-Computer Interaction
 

What's new | Course Description | Schedule | Registration

 

Lecture Series by Andrea Corradini and Manish Mehta

Topic Human-Computer Interaction in Immersive Environments and for Embodied Conversational Characters
Teacher Andrea Corradini and Manish Mehta, NISLab, University of Southern Denmark, Odense
Time and place 4.-6.11.2003, University Main Building (Kalevantie 4, Tampere), Pinni A (Kanslerinrinne 1, Tampere), and Pinni B (Kanslerinrinne 1, Tampere); see maps of the university campus
Class hours
Tuesday 4.11.
12.15-14.00 Lecture room A3, main building
Wednesday 5.11.
12.15-14.00 Lecture room B4113, Pinni B
16.15-18.00 Auditorium B1097, Pinni B
Thursday 6.11.
12.15-14.00 Paavo Koli Auditorium, Pinni A
16.15-18.00 Auditorium B1096, Pinni B
Teaching The course will be structured as a mix of lectures, video and audio clips, live demonstrations, and discussions.
Materials Slides by Andrea Corradini [zip]
Slides by Manish Mehta [zip]
Objective and contents In this two-day intensive introduction course we will propose an approach to natural multimodal interaction both in immersive environments and for embodied conversational characters. In the case of immersive environments, prior research in 2D multimodal gesture/speech interfaces, the proposed approach builds upon, will be presented. We will outline an architecture and provide examples from a preliminary 3D multimodal testbed, which we have been developing to explore a way to disambiguate the user's intent by fusing symbolic and statistical information from a set of 3D gesture and speech agents. In the embodied conversational scenario, we will outline architectural requirements for embodied conversational agents, and explain the architectural approach we have been following. We will outline an approach to Natural Language Understanding and Response Generation, which we have been following under the NICE project.

An introductory overview of the following topics will be provided:

  • gesture recognition
  • natural language processing
  • multimodal system
  • multimodality fusion
  • pervasive computing
  • augmented and reality reality
  • embodied conversational agents
  • response generation
The focus of the classes will be fairly broad, covering theory, empirical evaluation, design, and applications. At the end of the course, the student should have a good appreciation of the key research topics in this area, methodology used by researchers in this field, research results to date, and open research problems.
Registration All participants are requested to register in advance.
Background reading
  • Cohen, P.R., et al. QuickSet: multimodal interaction for distributed applications. in Fifth Annual ACM International Multimedia Conference (Multimedia '97). 1997. Seattle, WA: ACM Press.
  • Oviatt, S.L. Mutual disambiguation of recognition errors in a multimodal architecture. in Human Factors in Computing Systems (CHI'99). 1999. New York: ACM Press.
  • Norman, D.A., The Design of Everyday Things. 1988, New York, NY: Currency/Doubleday. 257.
  • McNeill, D., Hand and Mind: What Gestures Reveal about Though. 1993, Chicago: University of Chicago Press.
  • Kendon, A., Language and Gesture: Unity or Duality ?, in Language and Gesture, D. McNeill, Editor. 2000, Cambridge University Press: Cambridge, UK. p. 47-63.
  • Cassell, J., and Stone, M., Living Hand to Mouth: Psychological Theories about Speech and Gesture in Interactive Dialogue Systems, in Proceedings of the AAAI 1999 Fall Symposium on Psychological Models of Communication in Collaborative Systems. 1999, AAAI Press: North Falmouth, MA. p. 34-42.
  • Johnston, M., et al. Unification-based multimodal integration. in 35th Annual Meeting of the Association for Computational Linguistics (ACL '97). 1997. Madrid, Spain: ACL Press.
  • Badler, N., Bindinganavale, R., Allbeck, J., Schuler, W., Zao, L., and Palmer, M., Paramaterized action representatin for virtual human agents, in Embodied conversational agents, J. Cassell, Sullivan, J., Prevost, S., and Churchill, E., Editor. 2000, The MIT Press: Cambridge, Massachusetts. p. 256-286.
  • Cohen, P.R., McGee, D., Oviatt, S.L., Wu, L., Clow, J., King, R., Julier, S., Rosenblum, L., Multimodal interaction for 2D and 3D environments. IEEE Computer Graphics and Applications, July/August 1999, p. 10-13
  • Corradini, A. and Cohen, P. R. On the Relationships among Speech, Gestures, and Object Manipulation in Virtual Envi-ronments: Initial Evidence. Proc. Int. CLASS Workshop on Natural, Intelligent and Effective Interaction in Multimodal Dialogue Systems, (2002), 52-61
  • Corradini, A., Mehta M., Bernsen, N.O., Martin, J.-C., Multimodal Input Fusion in Human-Computer Interaction on the Example of the on-going NICE Project, to appear in: Proceedings of the NATO-ASI conference on Data Fusion for Situation Monitoring, Incident Detection, Alert and Response Management, August 18th-29th, Yerevan (Armenia), 2003
  • Kaiser, E., D., Olwal, McGee, A., Benko, H., Corradini, A., Li, X., Cohen, P.R., Feiner, S., Mutual Disambiguation of 3D Multimodal Interaction in Augmented and Virtual Reality, to appear in: ACM 5th International Conference on Multimodal Interfaces (ICMI-PUI'03), November 5th--7th, Vancouver (BC, Canada), 2003
  • Cassell, J., Bickmore, T., Campbell, L., Chang, K., Vilhjalmsoon, H., Yan, H., Requirements for an architecture for embodies conversational characters, Computer Animation and Simulation ’99, Eurographics series, Vienna, Austria.
  • Allen, J., Ferguson, G., Stent, A., An architecture for more realistic conversational systems.
  • Cassell, J., Nudge Nudge wink wink : Elements of face to face conversation of embodied conversational agents. In Cassell, J., et al(eds), Embodied Conversational Agents, The MIT press, pp. 1-27, 2000.
  • Bickmore, T., Cassell, J., (in press) Social Dialogue with Embodied Conversational Agents In J. van Kuppevelt, L. Dybkjaer, and N. Bernsen (eds.), Natural, Intelligent and Effective Interaction with Multimodal Dialogue Systems. New York: Kluwer Academic
About the teachers Andrea Corradini received his Ph.D. in computer science from the Department of Neuroinformatics at the Technical University of Ilmenau, Germany, where he was awarded a Marie Curie Research Fellowship by the European Union. Prof. Corradini has also studied mathematics at the University of Trento, Italy, and worked at the firm, MediSYS GmbH, in Ilmenau, Germany, on a project founded by the AT&Q Leonardo European Community program. Upon moving to the United States, Prof. Corradini held first the position of postdoctoral fellow at the Center for Human-Computer Communication at the Oregon Graduate Institute of Science and Technology, Oregon, and later that of senior research associate at the Center for Human-Computer Communication at the Oregon Health & Science University, Oregon. As of January 2003, he has been working as Assistant Professor at the Natural Interactive Systems Laboratory at the University of Southern Denmark, Odense, Denmark. His research interests include multimodal interaction, gesture recognition and analysis, face detection and recognition, sensor fusion, and dialogue.

Manish Mehta received his M.S in computer science from the Department of Computer Science and Engineering at Michigan State University, US. He received his Bachelors of Engineering in Electrical and Electronics from Panjab Engineering College, Chandigarh, India. After finishing his Bachelor’s degree, Manish Mehta worked in Xansa India Ltd for one year. As of February 2003, he has been working as a research assistant at the Natural Interactive Systems Laboratory at the University of Southern Denmark, Odense, Denmark. His research interests include speech and Natural Language Processing.

 
Last update on January 13, 2004.
Kari-Jouko Räihä (kjr@cs.uta.fi)