Reading Mediated Minds: Empathy with Persons and Characters in Media and Art Works

A CCCT Summer School -- Amsterdam July 11-13, 2011

Convenor: Ed S. Tan, ASCoR

Empathy and Media
Aims and participants
Themes
Lecturers
Programme
Guidelines for sessions, presentations, and discussions
Art events
Paper proposals PhD students and masterclasses

Empathy and Media

Communication involves one mind informing another about its state, and the other mind grasping it, at least in the successful case. In daily life we are constantly reading other people’s minds, often without being aware. What we read from other minds, how and why has become a blossoming field of research in many areas spanning the entire spectrum between natural sciences and humanities, with the social sciences in between. The research has often spectacularly broken through traditional disciplinary boundaries. Just one example is neuroscientist Vittorio’s Gallese’s essay on the significance of the mirror neuron system for understanding human social identity.

Humans need to read other people’s minds almost permanently in order to live together and to collaborate. In face-to-face exchanges we use language utterances, face and body language to grasp our partners’ state of mind that is how they feel, what their beliefs are, and what they may want from us. The focus of the summer school though is not on face-tot-face exchanges but on communication through media. When we communicate through media we have to intuit states of mind of others using an impoverished supply of cues. The ubiquitous use of media has turned reading other people’s minds as expressed in media to a routine. All cultural forms and genres, from TV news show to theatre, film and comics have developed their own ways to deal with how to convey the subjective life of persons, animals and animated beings to the audience. Users of these and other media developed medium specific ways of dealing with limitations, often turning deficits into assets. And yet any complete accounts of the ways we encode our own and decode others’ feelings, desires, intentions, and beliefs through different media are at present missing. Reading Mediated Minds provides an occasion for young researchers to discuss ideas that may fill the lack.

Expressing one’s own state of mind and deciphering that of another may be automated and unconscious processes, but mediated communication is often as complex as to require explicit knowledge of how other people’s inner life is expressed (that is a “theory of mind”). Misunderstandings arise due to encoding or decoding disabilities of persons involved, or to communication channel limitations. Even more interesting for media communication is the fact that sender and receiver often share intentions to a certain degree only.  For instance, disclosure of feelings may have a strategic element to it that may or may not be perceived as deception by the receiving party. Media use involves reflective and strategic thinking and smart use of cultural conventions and formulae for self-expression and presentation. In media communication sophisticated use of expressive cues in “mind writing” and reading is the rule rather than the exception. The complexities involved are studied from many disciplinary perspectives under different names, using different materials, conceptualizations and methods. It may be the case that researchers working on problems from one discipline cannot recognize the solution value the work of colleagues from another discipline may contribute. Reading Mediated Minds intends to make up for such missing connections.

Aims and Participants

Reading Mediated Minds aims at invoking an exchange of ideas, perspectives, and methods for studying all manifestations and uses of mind reading in and through media. First, researchers studying empathy in the reception of various media and arts, namely film, visual arts, literary and narrative texts and music will present and discuss their studies. These studies address empathy and mind reading processes in the receiver as well as expressive cues in products, often but not exclusively from a psychological angle. The presentations will allow for comparisons across the different arts and media, a first and obvious aim of the summer school. A second and more adventurous aim is to organize an exchange between researchers in media and arts, on the one hand and researchers in biology  and information sciences on the other. The idea is that arts and media researchers already have adopted many insights from ethology and neuroscience about the basic mechanisms of empathy, mind reading and social connectedness, but that they could learn a lot more in direct exchanges and dialogue with these disciplines. And similarly, researchers in media and art reception have started to use computers to help them in analyzing cues in media and art products, but they may put information technology to a much more profound and intensive use in modeling and analysis of empathy and its material cues. Conversely, information technology specialists (such as affective computing researchers, but not only those) may find new challenges in studies of empathy using complex data structures, and researchers from the life sciences may be inspired by the culturally sophisticated forms that empathy assumes in the media.

Reading Mediated Minds aims at widening young researchers’ perspectives through dialogues among technological, life and social sciences

Participants: The summer school’s participant community consists of 40 graduate or PhD level students recruited from institutes all over the world for being interested in the topic, and in novel interdisciplinary approaches. Leading researchers in the disciplinary fields will be invited as teachers to present their research and engage with the community in lectures and master classes. They include international specialist guests as well as researchers from the host university, UvA and friends from other Dutch research centres. Participants will familiarize with a variety of UvA research institutes and experience UvA’s open intellectual academic atmosphere.  Participants will be selected on the basis of their current research interests and plans.

Themes

1. Biology/ ethology: Biological bases of empathy, mind reading, communication, sympathy and moral. Interspecies continuities.
1. Is the capacity for empathy with media characters uniquely human?
2. Are there cultural varieties of expression and empathy in the animal kingdom?
3. How special is the case of empathising with “real” and fictional media personae from a biologocal point of view?
4. How can we use information technology in order to analyse large samples of behaviour and expression relevant for empathy (in the actor or an observer)?
2. Informatics: Machine readable empathy in media content
1. Can machines read minds of persons, characters, authors and recipients from media contents?
2. How can empathy-provoking cues be identified in content of various media and genres?
3. How can empathy provoking utterances and expressions in media contents be tagged?
4. How can empathy be read from media users’ responses?
5. Semantic representation systems for search and retrieval
3. Neuroscience: what neural mechanisms are involved in empathy
1. Can empathy with personae in the media production be distinguished from e. with producers/ makers/ authors?
2. Are there any differences between empathy with fictional characters and with real personae in film?
3. How can we use information technology in order to analyse large samples of physiological or behavioural data relevant for empathy (in the actor or an observer)?
4. Media research and studies
1. Cognitive film studies and film psychology
1. How do different genres provoke empathy, and what is the role of film style and technology?
2. What is the role of empathy in communication between film maker and film viewer: is there mutual empathy in communication through film?
3. How are film materials analysed as to cues used by viewers in empathy and mind reading?
4. How can we use information technology in order to analyse large samples of behaviour and expression, or film style cues relevant for empathy (in the actor or an observer)
2. Visual arts and media
1. How do different genres provoke empathy, and what is the role of style and technology?
2. What is the role of empathy in communication between maker and viewer: is there mutual empathy in communication through visuals?
3. How are visual materials analysed as to cues used by viewers in empathy and mind reading?
4. How can we use information technology in order to analyse large samples of visual materials in search of features relevant for empathy (in the actor or an observer)
3. Literature and narrative discourse comprehension
1. How do different genres provoke empathy, and what is the role of language and style?
2. What is the role of empathy in communication between author and readers/ listeners: is there mutual empathy in communication through narrative?
3. How are literary texts analysed as to cues used by readers in empathy and mind reading?
4. How can we use information technology in order to analyse large samples of texts in search of textual features relevant for empathy (in the actor or an observer)
4. Psychology of Music
1. How are emotions conveyed from musicians to listeners (and back?)
2. What does it mean to feel into music?
3. How are pieces of music analysed as to cues used by listeners in empathy and mind reading?
4. How can we use information technology in order to analyse large samples of music in search of musical features relevant for empathy (in the listener)?

Lecturers

• Heidi de Mare (Amsterdam University College). Masterclass Visual Arts: Visual analysis.
• Sennay Ghebreab (Amsterdam University College; UvA Brain & Cognition; ISLA; BeTaLab). Seminar Neural Mechanism of Empathy: Neuroesthetics, the Brain and Empathy; Master classes
• Frank Hakemulder (Utrecht University) Masterclass Literature: Empirical Methods
• Christopher Honey (Princeton University) Masterclass Film: Communicating Brains
• Jeroen Kools (Amsterdam University College; UvA Brain&Cognition)
• Keith Oatley (University of Toronto), key note speech On Empathy and Fiction and Masterclass Literature: Psychological models
• Albert Ali Salah (University of Amsterdam), Seminar Machines Reading Minds and Masterclass Film: (Im)possibilities of Machine Learning
• Arnold Smeulders (University of Amsterdam), Masterclass Visual Arts: Visual Search Systems as a Tool in Analysis
• Rens Vliegenthart (University of Amsterdam), Masterclass Literature: Content Analysis with Machine Support
• Semir Zeki (University College London), Seminar Neural Mechanism of Empathy: Neuroesthetics, the Brain and Empathy
• Remko Scha (University of Amsterdam), Master class Music: Resonance in CNS and Body as Empathy
• Wouter Weerkamp (University of Amsterdam), Seminar Machines Reading Minds: Sentiment analysis as an example of large scale empathy content analysis.

Programme

Programme details

Brain & Cognition Lab (LAB) AUC harbours a lab for its students where they can set up not too complex studies on brain and behaviour in a variety of experimental tasks. The equipment is ideally suited to try out simple tests on cognitive and emotional processing of visual, film and text stimuli. We will have available installations for EEG and EMG recording, eye tracking equipment, and software for facial expression categorisation. All participants of the summer school will have access to the equipment and can try it out. Absolute beginners are especially encouraged to explore and orient on the use of techniques and measures. If you have never seen how registering and analysing brain signals, facial expressions and eye movements works you can obtain hands-on experience. Testing out the equipment may help you to envisage simple studies using these techniques to extend your research plan with on-line measures of recipient attention, empathy, emotion, reading and viewing behaviour. Participants can bring along their stimulus materials and test these.

Sennay Ghebreab is our lecturer on brain, cognition and computing at the summer school. Jeroen Kools is the lab supervisor. He will present introductions and help participants to find their way in the lab. At the start of the classes there will be a schedule where you can sign in. If you already have special questions and requests let us know at   e.s.h.tan@uva.nl.

The opening times of the lab are on the schedule that is passed around for your signing in. You may choose from the following menu:

1. our along the devices and software + Reading Powerpoint shows on:
   • general intro into methods and devices
   • applications relevant for empathy
2. Hands-on introduction to one method. Try out registrations and visualisations of your responses to selected stimuli such as films, written narratives and pictures. Select one of these methods
   • EEG
   • Heart rate and GSR
   • Facial expression
   • Eye tracking
   • Advanced analysis

Sign in on the list that will circulate on Monday morning. Please select a time where you are not in a master class where you present.

Guidelines for sessions, presentations, and discussions

Seminars

The two seminars are attended by all participants. They consist of a lecture, max. 1 hour, followed by discussion. Discussion round 1: Clarification questions to the lecturer. The experts will assist the lecturer in the explanations. Discussion Round 2: General discussion of positions, proposals, findings and other topics. Discussion Round 3: Applicability of research and methods in empathy research in arts and media. Participants are encouraged to apply topics, insights and questions arising from the lecture to their own research.

Master classes

All master classes and seminars are attended by all participants. The lecturers present a brief introduction to their field, either the use of information systems in analyzing textual, visual or musical content, or the use of brain and physiological research in studying processes of empathy and mind reading in viewing, reading or listening. They have 1.5 hrs jointly and maximally. In the second half of the master class participants present their own research. Each presentation takes 15 minutes, followed by 20 minutes discussion. Presenters assume that their research essays have been read by all participants so that they can focus on preferably these questions:

• What is meant by “empathy” in the present research, and how is it conceptualized?
• What features of materials (text, visual art works, music, film) are relevant?
• What sources are used?
• Indication of methods (i.e. approaches and techniques)

Presenters may use power point slides, handouts or both. There will be a computer with internet and a beamer.

The discussion is structured as follows: five minutes max. of clarification q&a; 15 min. on conceptual issues and methods. We always try to conclude with considering the question whether and how the analyses could be scaled up using information technology and or physiological registrations. Participants who wish to have additional feedback on their research plans may present a poster in the poster session. Note that this is not a requirement, but an additional way to get feedback on your work). All participants are encouraged to visit the Brain&Cognition Lab in order to acquaint themselves with the use of brain activity registrations, psychophysiological recordings and facial expression registration.  They can discuss how these techniques may be of help in their research during a master class or in between the sessions. A limited number of participants who do not present their work during a master class can visit the B&C lab.

Poster session

We aim to have free style posters, but the frames used pose limitations. Use portrait, not landscape setup.

• A1 60 * 84 cm, or American D
• A2 42 * 60 cm, or 2 * A3 or 4*A4, or American C

You may of course combine a manifold of usual printer paper format (in Europe A4) and paste these together into one sheet using tape. Preferably use Times Roman 24 or equivalent readability; paragraphs reporting technical details or sample texts can be set in smaller font.  Contents and format are free, so you can use standard scientific lay outs or any ones that serve your ends best as long as they are readable and informative. Remember that posters are additional to presentations and essays. You may even choose to only pose questions though your poster, or present a few tables to discuss results, or …

Art events

It's All Over Now, Baby Blue

Art event 1:  It's All Over Now, Baby Blue is a theater performance, that mixes dance, physical theater and an animation. The audience is invited to the bizarre worlds of the performance that dives trough the subconscious life, the fears and controlling behaviour. The thrilling and twisted atmosphere on the stage is created by a Finnish dancer choreographer Milla Virtanen, an animation artist Leevi Lehtinen and a composer Anssi Laiho.

Concept and Choreography: Milla Virtanen Animation: Leevi Lehtinen Sound Design/Composition: Anssi Laiho Light Design: Sjaak Rood Costume: Milla Virtanen Stage Design: Milla Virtanen & Leevi Lehtinen Production: Theater De NWE Vorst Co-Production: The Netherlands Institute for Animation Film Financial Support: Theater De NWE Vorst, The Netherlands Institute for Animation Film, The Finnish Art Council, Oulu City Board of Culture Advisers: Aurelie Camil, Wim Arts, Leon van Rooij

The Benefactor

Art event 2: The Benefactor is a project about success and hunger. It is a lecture performance on gifts, investments, parasitism and the state of the arts, developed and performed by Julian Hetzel. First of all, I support this new cabinet. But – I disagree with them on one point: we should not cut funding to the arts. We should raise it. And I’d like to talk about my memories of music and culture from my youth. But first, I think that we can pay for culture by cutting back on development aid!" Frits Bolkestein Inspired by the statement of Dutch politician Frits Bolkestein, referring to the current political situation in the field of art and culture in The Netherlands, Julian Hetzel developed a project about the radical alternative use of cultural money – by donating his budget to a starving child in Africa. He donated 2000 € into a child-sponsorship, 1 € per day. He frames this child-sponsorship as a piece of art, as a "durational intercontinental performance of 2000 days". In doing so, he raises questions about the ethics of success in society. But in fact he is first and foremost investing into his own future success… Art is created out of circumstances and not by pure inspiration. How to turn development aid into art? How to turn art into development aid? How to develop art? How to turn?

Paper proposals PhD students and master classes