Memory Reconstruction (Lucas Bietti and Alan Cienki)

Shared and distributed collective memories are utilized to create a feeling of connection and maintain a consistent feeling of identity among group members. Intimate people are strategically engaged in processes of remembering and forgetting, which are modeled according to the specific goals of a particular interaction. In these cases, members of groups construct a distributed socio-cognitive system, which is shaped by the physical and social environment in which they are located. This socio-cognitive system enables intimate people to connect, interrelate and manage individual and collective memories which are distributed among them and form part of shared past experiences. The aim of this project is to explore the ways in which a individual and collective memories are interactionally re-constructed and coordinated in everyday interactions between family members and close friends in real-time multimodal interactions. This project is funded by an Alexander von Humboldt Research Fellowship for Postdoctoral Researchers (2011-2013).

Features of horizontal entrepreneurship (Nicolina Montesano Montessori and Alan Cienki)

This project involves a multimodal analysis of videorecorded interviews with three leaders of social movements in the Netherlands: the Dutch leader of Zeitgeist, the leader of MasterPeace and the leader of Givolicious. Zeitgeist advocates a new economic order based on abundance rather than scarcity. MasterPeace wants to inspire everyone to use his or her talents for peace building. It aims to make the International Peace Day more well known and to involve everyone actively in peace-building. Givolicious spreads the view that the essence of life has to do with giving and letting go rather than accumulating and taking control. The mission is to motivate traditional companies to give free support to social companies. The analysis of the interviews relies on a methodology that combines Critical Discourse Analysis with gesture studies and entails a reflection on how these two research traditions work together (considering the benefits and challenges).

Viewpoint Expression (Kashmiri Stec)

We use space to organize our activities; we wash vegetables in the sink, and cut them on the counter; things I need to do are stacked on the left side of my desk, things I’ve already done are on the right. This kind of “basic” spatial organization is used not only for physical activities, but conceptual ones as well – such as keeping track of different characters in a story, or different lines of communication. In this project, two lines of viewpoint expression are investigated. In the first, the focus is on the non-verbal behaviors [gaze, facial displays, the orientation of different body parts] accompanying quotative utterances, specifically focusing on the spatial orientation of the body as different characters in a narrative are quoted. In the second, done in collaboration with Prof. Eve Sweetser (U.C. Berkeley), the use of space to organize discourse between pairs of English speakers is described. We focus on the layout of real space blends, and pay particular attention to how body orientation and gaze patterns correlate with different aspects of dialogue (foregrounded/backgrounded information; narrative; self-reflection/thought; repairs; meta-discourse; etc.). Our corpus is over 4 hours long, and contains semi-spontaneous auto-biographical narratives told between pairs of native US English speakers. This project is funded by a Vidi grant awarded to Dr. Esther Pascual by the Netherlands Organisation for Scientific Research (NWO).

Gesture and Cognitive Linguistics (Alan Cienki)

From the perspective of cognitive linguistics, not only speech but also other kinds of bodily movements used when speakers communicate can provide insight into language users’ ongoing conceptualizations – of the physical world and of abstract ideas. Increasing attention is being paid in cognitive linguistic research to gesture (in particular, manual gesture) with speech in relation to such topics as: conceptual metaphor; conceptual metonymy; (image) schemas; perspective-taking and construal; mental spaces; formal and conceptual integration/blending theory; and mental simulation and simulation semantics. The inclusion of gesture in cognitive linguistic research raises not only methodological issues concerning what constitutes linguistic data and what means can be used to analyze it, but also significant theoretical questions about the nature of language itself, especially in relation to its dynamic character and the degree to which grammar is multimodal.

Gesture and Construction Grammar (Suwei Wu)

Suwei’s PhD project is funded by a Joint Scholarship provided by the China Scholarship Council and VU Amsterdam. The project is on the Multimodality of Spoken Language and Construction Grammar. The constructions in our spoken language and the relationship between these constructions and our co-speech gestures are explored in this project.

Gesture and Cognitive/Functional Discourse Grammars (Kasper Kok)

In this project, we explore interfaces between three specific developments in scientific approaches to language. The first concerns the shift from viewing language as a logical system of formal symbols to approaches that take principles from cognitive psychology into account and study language in terms of underlying cognitive processes. The second concerns the growing interest in studying language as a vehicle for communication in the interactive context of face-to-face communication, rather than objectifying language as a code which an individual ‘possesses’. As a third topic of interest we consider approaches that move away from viewing language as a complete and autonomous system, but rather see language as part of a ‘multimodal’ (audio-visual) communication repertoire, taking into account co-verbal aspects of expression such as gesture and intonation patterns. To what degree can these different properties of spoken language use be integrated into one coherent model? This project will focus on two specific theories – Cognitive Grammar and Functional Discourse Grammar – to see how well they can be integrated with each other and with co-verbal properties of communication. Based on theoretical and empirical exploration of the points of convergence between these theories, the aim is to develop an integrated dynamic model of multimodal communication that can be modeled computationally and tested against relevant data. An existing data set of video-recorded conversations will be used to evaluate the model’s predictions and allow for adjustment of the model to better represent realistic communication. This project is funded by the Netherlands Organisation for Scientific Research (NWO):  PhDs in the humanities.



Past Projects


Stance-taking (Camille Debras) 

There’s a metaphor half asleep in the notion of taking stance: together with speech, people also use their bodies and the communication space to literally position themselves, with respect to both their discourse object and addressee. Specifically in the course of ethical discussion, participants constantly take stance: they pass judgement on what they/ people do and what ought to be done, justify their beliefs based on expert or common knowledge, take full responsibility for an opinion or on the contrary make mitigated evaluations. To do so, they rely on speech, intonation, facial expressions or gesture, and sometimes use many of these at once. The main goal of this project is to propose a unified method to account for the multimodal aspects of stance-taking, based on an almost 3-hour corpus of ethical discussions about environmental issues between pairs of native speakers of British English. They belong in the Cambridge Student Corpus, a collection of videotaped dyadic conversations between Cambridge University students that I recorded in 2010-2011 together with Yann Fuchs and Eric Mélac, two other PhD students from Sorbonne Nouvelle University in Paris. This project is funded by a 3-year PhD scholarship from the French Ministry of Higher Education and Research (2010-2013), during which I had the opportunity to join the Amsterdam Gesture Center as a visiting scholar for one semester in 2012. Alongside exciting discussions on the study of dynamic multimodal communication with the group, I collaborated with Alan Cienki on a study of the stance-taking functions of shrugging and lateral head tilts, which we presented at the international social computing conference SocialCom in Amsterdam in September 2012.