This paper intends to contribute to the multimodal turn-taking literature by presenting data collected in an improvisation session in the context of the performing arts and its quali-quantitative analysis, where the focus is on how gaze and the full body participate in the interaction. Five expert performers joined Portuguese contemporary choreographer, Joao Fiadeiro, in practicing his Real Time Composition Method during an improvisation session, which was recorded and annotated for this study. A micro-analysis of portions of the session was conducted using ELAN. We found that intersubjectivity was avoided during this performance, both in the performers’ bodily movements and mutual gaze; we extrapolate that peripheral vision was chiefly deployed as a regulating strategy by these experts to coordinate turn-taking. A macro-analysis comparing the data with an analogous one obtained from Non-Performers provides the context for a discussion on multimodality and decision-making.
gaze, non-verbal behavior, silent turn-taking, gesture function, decision-making, performing arts, inter-rater agreement
[1] H. Sacks, E. A. Schegloff, and G. Jefferson, “A simplest systematics for the organization of turn-taking for conversation,” Language, vol. 50, no. 4, 1974.
[2] J. Fiadeiro, “Wenn du das nicht weißt, warum fragst du dann?” in Wissen in Bewegung. Perspektiven der k¨unstlerischen und wissenschaftlichen
Forschung im Tanz, G. Sabine, P. Husemann, and von Katharina Wilke, Eds. Bielefeld: transcript Verlag, 2007, pp. 103–112.
[3] A. Camurri, B. Mazzarino, M. Ricchetti, R. Timmers, and G. Volpe, “Multimodal analysis of expressive gesture in music and dance performances,” in Gesture-based communication in humancomputer interaction. Berlin: Springer, 2004, pp. 20–39.
[4] E. Goffman, Behavior in Public Places. New York: Free Press, 1963.
[5] V. Petukhova and H. Bunt, “Who’s next? Speaker-selection mechanisms in multiparty dialogue,” in Proceedings of the Workshop on the Semantics and Pragmatics of Dialogue, Stockholm, 2009, pp. 19–26.
[6] D. Bohus and E. Horvitz, “Multiparty turn taking in situated dialog: Study, lessons, and directions,” in Proceedings of the SIGDIAL 2011 Conference, Portland, Oregon, 2011, pp. 98–109.
[7] A. Kendon, “Some functions of gaze direction in social interaction,” Acta Psychologica, vol. 26, pp. 22–63, 1967.
[8] E. Ochs, E. A. Schegloff, and S. Thompson, “Interactional units in conversation: Syntactic intonational and pragmatic resources for the management of turns,” in Interaction and Grammar. Cambridge: Cambridge University Press, 1996, pp. 134–184.
[9] L. Mondada, “Multimodal resources for turn-taking: pointing and emergence of possible next speakers,” Discourse Studies, vol. 9, no. 2, pp. 194–225, 2007.
[10] L. Cerrato and M. Skhiri, “Analysis and measurement of head movements signalling feedback in face-to-face human dialogues,” in Proceedings of the First Nordic Symposium on Multimodal Communication, P. Paggio, K. Jokinen, and J. J¨onsson, Eds., Copenhagen, 2003, pp. 43–52.
[11] B. Rahayudi, R. Poppe, and D. Heylen, “Twente debate corpus - a multimodal corpus for head movement analysis,” in International Conference on Language Resources and Evaluation (LREC 2014), vol. 106, Reykjavik, 2014, pp. 4184–4188.
[12] J. B. Bavelas, “Appreciating face-to-face dialogue,” in Audio-Visual Speech Processing, British Columbia, 2005, vol. 1.
[13] K. Jokinen, “Gaze and gesture activity in communication,” in Universal Access in Human-Computer Interaction. Intelligent and Ubiquitous Interaction Environments, C. Stephanidis, Ed. Berlin: Springer, 2009, pp. 537–546.
[14] G. Brˆone, K. Feyaerts, and B. Oben, “On the interplay between eye gaze and speech,” in Empirical Approaches to Multi-Modality and to Language Variation (AFLiCo 5), Lille, France, 2013.
[15] K. Shockley, D. C. Richardson, and R. Dale, “Conversation and coordinative strucutres,” Topics in Cognitive Science, vol. 1, pp. 305–319, 2009.
[16] J. Holler and K. H. Kendrick, Gesture, gaze, and the body in the organization of turn-taking for conversation: Insights from a corpus using new technologies. Los Angeles: 4th International Conference on Conversation Analysis (ICCA14), 2014.
[17] C. S. Peirce, “Philosophical writings,” in Logic as Semiotic: the Theory of Signs, J. Buchler, Ed., Dover, NY, 1955 [1902], pp. 98–119.
[18] H. Holle and R. Rein, “The modified Cohen’s kappa: Calculating interrater agreement for segmentation and annotation,” in Understanding
Body Movement: A Guide to Empirical Research on Nonverbal Behaviour, H. Lausberg, Ed. Frankfurt am Main: Peter
Lang Verlag, 2013, pp. 261–277.
[19] R. J. Landis and G. G. Koch, “The measurement of observer agreement for categorical data,” Biometrics, vol. 33, pp. 159–174, 1977.
[20] C. Oertel, M. Wlodarczak, J. Edlund, P.Wagner, and J. Gustafson,“Gaze patterns in turn-taking,” in Proceedings of Interspeech 2012, Portland, Oregon, 2012.
[21] C. D. Frith and T. Singer, “The role of social cognition in decision making,” Philosophical Transactions of the Royal Society of London B: Biological Sciences, vol. 363, no. 1511, pp. 3875–3886, 2008.
[22] N. Sebanz, D. Rebbechi, G. Knoblich, W. Prinz, and C. D. Frith,“Is it really my turn? An event-related fMRI study of task sharing,” Social Neuroscience, vol. 2, no. 2, pp. 81–95, 2007.
[23] R. Rico, M. Sénchez-Manzanares, F. Gil, and C. Gibson, “Team implicit coordination processes: A team knowledge-based approach,” The Academy of Management Review, vol. 33, no. 1, pp. 163–184, 2008.
[24] R. Fusaroli and K. Tylén, “Carving language for social coordination: A dynamical approach,” Interaction Studies, vol. 13, no. 1, pp. 103–124, 2012.
[25] B. Skyrms, Ed., The Stag Hunt and the Evolution of Social Structure. Cambridge: Cambridge University Press, 2004.
[26] R. A. Shweder, M. Mahapatra, and J. G. Miller, “Culture and moral development,” in Cultural psychology. Essays on Comparative Human Development, J. W. Stigler, R. A. Schweder, and G. Herdt, Eds. New York: Cambridge University Press, 1990,
pp. 130–204.
[27] Y. Tsushima, Y. Sasaki, and T. Watanabe, “Greater disruption due to failure of inhibitory control on an ambiguous distractor,” Science, vol. 314, no. 5806, pp. 1786–1788, 2006.
[28] T. Singer and E. Fehr, “The neuroeconomics of mind reading and empathy,” American Economic Review, vol. 95, no. 2, pp. 340–345, 2005.
[29] S. J. Rogers, “Mutual gaze,” in Encyclopedia of Autism Spectrum Disorders, F. R. Volkmar, Ed. New York: Springer, 2013, pp. 1966–1967.
[30] V. Evola, J. Skubisz, and C. Fernandes, “The role of gaze and other body movements in collaborative decision-making: A study on coordinating turns in a contemporary dance improvisation exercise,” in 7th Conference of the International Society for Gesture Studies (ISGS 2016), Paris.
[31] E. Paul and F. Wallace V, “The repertoire of nonverbal behavior: Categories, origins, usage, and coding,” Semiotica, vol. 1, no. 1, pp. 49–98, 1969.
[32] M. H. Krout, “Autistic gestures: An experimental study in symbolic movement.” Psychological Monographs, vol. 46, no. 4, pp. i–126, 1935.
[33] H. Lausberg and H. Sloetjes, “Coding gestural behavior with the NEUROGES-ELAN system,” Behavior Research Methods, Instruments, & Computers, vol. 3, no. 41, pp. 841–849, 2009.
[34] M. Gullberg and K. Holmqvist, “Keeping an eye on gestures. Visual perception of gestures in face-to-face communication,” Pragmatics & Cognition, vol. 7, no. 1, pp. 35–63, 1999.
[35] M. Carrasco, C. Penpeci-Talgar, and M. Eckstein, “Spatial covert attention increases contrast sensitivity across the CSF: Support for signal enhancement,” Vision Research, vol. 40, no. 10, pp. 1203–1215, 2000.
[36] M. I. Posner, “Orienting of attention,” Quarterly Journal of Experimental
Psychology, vol. 32, no. 1, pp. 3–25, 1980.
[37] R. S. Friedman, A. Fishbach, J. F¨orster, and L. Werth, “Attentional priming effects on creativity,” Creativity Reserach Journal, vol. 15, no. 2 & 3, pp. 277–286, 2003.
[38] V. Evola, “Multimodal cognitive semiotics of spiritual experiences: Beliefs and metaphors in words, gestures, and drawings,” in Form, Meaning, and Body, F. Parrill, V. Tobin, and M. Turner, Eds. Stanford: CLSI, 2010, pp. 41–60.
[39] V. Evola, “Metafore, sistemi religiosi e semiotica cognitiva multimodale. discorsi, gestualit`a e disegni di predicatori di strada cristiani e di un satanista,” Sistemi Intelligenti, no. 1, pp. 23–48, 2010.
[40] P. Bourdieu, Outline of a Theory of Practice. Cambridge: Cambridge University Press, 1977.
[41] M. Kimmel, “Embodied (micro-)skills in tango improvisation: How a collaborative behavioral arc comes about,” in Out for a walk. Das Entgegenkommende Denken, Actus et Imago. Berliner Schriften f¨ur Bildaktforschung und Verk¨orperungsphilosophie, F. Engel, S. Marienberg, and P. Schneider, Eds. Berlin: De Gruyter, 2015, vol. 15, pp. 57–74.
[42] L. Brandt, “Dance as dialogue: Metaphorical conceptualization and semantic domains,” in Sémiotique de la Musique - Music and Meaning (Signata 6), J. R. do Carmo Jr and P. A. Brandt, Eds. Liége: Presses Universitaires de Liége, 2015, pp. 231–249.
[43] H. Lausberg, “The NEUROGES coding system: Design and psychometric properties,” in Understanding Body Movement: A Guide to Empirical Research on Nonverbal Behaviour, H. Lausberg, Ed. Frankfurt am Main: Peter Lang Verlag, 2013, pp. 85–106.
[44] D. McNeill, Hand and mind: What gestures reveal about thought. Chicago: University of Chicago Press, 1992.
[45] D. McNeill, “Introduction,” in Language and Gesture: Window into Thought and Action, D. McNeill, Ed. Cambridge: Cambridge
University Press, 1992, pp. 1–10.
[46] I. Mittelberg and V. Evola, “Iconic and representational gestures,” in Body - Language - Communication. An International Handbook on Multimodality in Human Interaction (Handbooks of Linguistics and Communication Science 38.2.), C. M¨uller, A. Cienki, E. Fricke, S. L. Ladewig, D. McNeill, and J. Bressem, Eds. Berlin/Boston: De Gruyter Mouton, 2014, pp. 1732–1746.
[47] J. J. Fleiss, “Measuring nominal scale agreement among many raters,” Psychological Bulletin, vol. 5, no. 75, pp. 378–382, 1971.
[48] K. Krippendorff, Content Analysis. An Introduction to Its Methodology, 2nd ed. Thousand Oaks: Sage, 2004.
[49] J. Cohen, “A coefficient of agreement for nominal scales,” Education and Psychological Measurement, vol. 20, pp. 37–46, 1960.
[50] C. Ribeiro, V. Evola, J. Skubisz, and R. Kuffner, “Transposing formal annotations of 2D video data into a 3D environment for Gesture research,” in 7th Conference of the International Society for Gesture Studies (ISGS 2016), Paris.
