Recently, I had the opportunity to attend the Mind Art Knowledge conference, where I presented our latest research on dual-screen watching.
The core focus of our poster is the emergence of “ambient TV.” Until now, ambient TV has mostly been treated as a neat concept in cultural journalism. In our poster, we proposed a concrete, empirical way to actually test this phenomenon.
Because we presented it in a poster format, we intentionally leaned into the medium to make it a bit of a provocation…
Yesterday, I was lucky to visit the Creative Computing Institute (CCI), and specifically Tim Smith‘s team, at the University of the Arts London. Tim invited me to give a short presentation about the research I am conducting at Royal Holloway under the supervision of Adam Ganz and Szonya Durant. Besides that, Tim showed me their facilities, and we had a talk about computer vision in cinema research.
In my talk, I presented a selection of the most interesting findings from the last 18 months: a) the difference between watching videos in vertical and horizontal frames, b) a null result regarding the effect of screen time on gaze patterns (both soon to be published in Projections), c) how people simultaneously watch narrative stimuli on TV and TikTok on a tablet (article just before submission), and d) which formal elements of narrative stimuli pull visual attention from a tablet to a TV screen (what I am working on). My main argument was that, contrary to my initial assumption, TikTok (and short video platforms in general) is probably not changing how we watch films. Rather, short videos (among other things) have made us aware of the audience’s inattentiveness. However, the fact that people are distracted is not new and it was always like that. It is just more visible now.
On a personal level, it was amazing to see the Creative Computing Institute and how Tim approaches research in the arts and humanities. I know very well from CoSTAR that arts research can look very different from what I was trained in. The CCI combines research activities with teaching, and it is inherently interdisciplinary. Truly, one can hardly say where one discipline ends and another begins. An illustrative example was when Tim showed me how they connected a virtual production studio with fNIRS and used it in a dance performance. The dancer’s brain activity produced a visualization on the LED panels behind him. This single example clearly shows how art can merge with neuroscience, technology, and programming, proving that these disciplines are not only communicating but also mutually beneficial.
Here’s a poster we presented in Amsterdam. Some more details can be found here and in repository.
The third day of the conference started with James Cutting’s presentation on the basic form of stories. I like his book Movies on Our Minds and how he combines basics, such as types of shots and transitions, with a deep understanding of the evolution of film style, as well as how he works with empirical data. This was the first time I had seen him live. And in a similar way as in his book, Cutting started with the basics – the form of a joke – and continued with more and more complex stories (and not only stories) to show us that the “beginning – middle – end” form is repeated in many variations. Sometimes individual parts are divided into subsections, but the basic form is everywhere. I must say that this time I found myself missing an explanation as to why this is so surprising. I don’t doubt that stories might be segmented into smaller parts, but I don’t get the surprising point. Only in the subsequent discussion did the explanation emerge: that the segmentation into smaller parts might be caused by the demands of our memory. Then it makes sense and opens up possibilities for further research. But without an explanation like that, the segmentation itself feels more like a tool to understand individual stories, than a basic form shared across media and cultures.
Furthermore, the conference offered many interesting papers focusing on the details such as an effect of single white frames in narrative films on perception (Javier Sanz-Aznar and Juan José Caballero-Molina), or overall theories such as model of film comprehension (Tess Lankhuizen and Katalin Bálint). What really fascinated me, however, was the panel discussion at the end of the day.
In the Author-Meets-Critics format, four panelists first commented on Malcolm Turvey’s book Film, Art, and the Limits of Science: In Defence of Humanistic Explanation. After that, Malcolm Turvey rebutted their arguments. To sum up the discussion, Turvey argued that some research questions might only be answered by means of the humanities and no scientific explanation is needed. His critics objected that it is hard to say in advance which explanation will be sufficient. Turvey and his critics all agreed that the humanities need to work with empirical evidence, have systematic methods, and ensure their findings are open to revision. I haven’t read the book, so my impression is only superficial, but… what is the point of keeping the humanities and science separated, when we wipe out the methodological differences? In this case, I don’t mean “method” as using machines. In this case, the scientific method means “falsifying hypotheses based on empirical evidence”. When this is the same for various disciplines, why do we need to draw a line between them? I suppose reading Turvey’s book will offer the answer.
The fourth day of the conference was pretty short for me. I needed to work, so I attended only two panels. In one of them, Carl Plantinga presented types of engagement from one chapter of his forthcoming book. Those types are obviously inspired by Daniel Kahneman’s System 1 and System 2 thinking. Partly, this can be understood as a possible solution to problems connected with the models of narrative engagement from previous days. In both models, the individual differences of viewers were mentioned as a problem. Cynthia Cabañas, co-author of the TENCo model, immediately suggested a possible way of integrating Plantinga’s types of engagement into her model by treating them as dynamically changing ways of engagement with film.
The second day of the conference was mostly about narration for me, and how to think about it from a different perspective than what is usual in film studies. Traditional film form analysis is hard work, as a researcher needs to watch a film shot-by-shot or even frame-by-frame and manually code elements like shot size, type of shot, composition, staging, and transitions on one level. On another level of analysis, the researcher needs to think about patterns and the structure of the plot. This approach can be very effective when we want to understand how stories are constructed in cinema, but as you can imagine, it takes a lot of time and effort, and it is very demanding to cover a larger corpus of titles. Therefore, a computational breakthrough is needed.
First thing in the morning, Chiao-I Tseng and Matthias Springstein presented TIB-AV-A, their computational framework for analysing narrative patterns in films. They focus on shot sizes and characters within shots, and based on that, they were able to automatically identify several shot patterns in movies consisting of combinations of several shots. Their approach seems reasonable, especially given that there are many open-source Python libraries involving computer vision that can tell us a lot about the style of a film. When these are effectively combined, we can use computational analysis to understand some aspects of narration, too.
The second narration-focused paper was presented by Cynthia Cabañas. In her paper, she explored narrational complexity in relation to cognitive flexibility and tolerance for ambiguity. Building on the previously published TENCo model – involving cognition, affective response, appreciation of film, and individual traits – she tested if watching films causes shifts in cognitive processes. In another paper research by Gaia Yonah was based on the same model; she tried to develop and validate the Narrational Complexity Engagement (NCE) Scale – a questionnaire designed to assess engagement driven by complex narration.
Computational and psychological approaches are getting closer to understanding what narration is and how it works on the level of film form and audience engagement. I believe this is the future of research in cinema. Maybe not exactly these specific examples, but methods and tools like those presented today.
The final panel was dedicated to the 30th anniversary of Ed Tan’s Emotion and the Structure of Narrative Film – Film As An Emotion Machine. Several scholars were remembering their first encounter with Ed Tan and his book, and in the end, Ed Tan looked back and recapitulated how our understanding of emotional responses has developed since the publication of his book. According to him, despite the great efforts of researchers, there is still an explanatory gap – the one David Chalmers calls the “hard problem of consciousness.” Subsequently, Ed Tan suggested involving a phenomenological approach in cognitivist research to address the subjective quality of these experiences. While I understand his concern about understanding subjective feel of emotions, my impression is that the whole SCSMI conference is full of examples showing that we understand the mechanisms of emotions better and better. It reminded me of Daniel Dennett’s From Bacteria to Bach and Back, where he opposes Chalmers and argues that consciousness (and arguably the explanatory gap itself) can be resolved when the easy problems are taken apart and studied gradually – in a similar way how computational analysis and cognitive psychology works. I guess that emotional reactions to films can be solved in a similar, progressive way.
Furthermore, there was a poster session today. Six posters were presented, and participants of the conference voted for the best one. The final ranking was:
So, third place for our poster! Not bad when I realise that this was my first experience with an academic poster since I was an MA student.
This year’s annual conference of the Society for Cognitive Studies of the Moving Image (SCSMI) takes place at Vrije Universiteit Amsterdam. I am attending this conference for the first time, and after the first day, I am incredibly excited. In the morning, we visited the Eye Film Archive, where we had a wonderful tour. For some reason, people in our group were very curious about nitrate film stock, so we were shown how nitrates are stored and what it looks like when they decay. I noticed that, at a certain stage, the decay looks strikingly similar to eye-tracking heatmap visualizations. Which also serves as a segue from the archive to cognitivism!
Later at the conference, I attended several panels. First an interesting presentation by Joe Magliano on his empirical research into plot density and memory for serialised narratives. Joe mentioned that plot density correlates with arousal, which in turn affects memory. I definitely want to address this topic in a paper we are currently preparing on the recall of ambient TV during dual-screen viewing of television and TikTok.
Another fascinating paper was presented by Peter Kapritsias, who talked about awe and how it relates to visual edge quality in animated films. I really appreciated that Peter is both an artist and a researcher, and I loved how he seamlessly combined the phenomenological level with empirical evidence.
However, there were two absolute highlights of the first day. The first was a presentation by Jonathan Frome – or, rather, the discussion that followed it. Jonathan criticized the field of cognitive film studies for its inability to reject certain theories, specifically focusing on Noël Carroll’s theory of erotetic narration. (Jonathan published a paper on Carroll’s theory in 2020, to which Carroll responded in 2024). While I don’t want to go into the details of the criticism of the erotetic theory itself, Jonathan’s broader point is highly relevant. In our field, and many others field, there are people who believe in “pure theories” that exist outside the real world, entirely disconnected from empirical evidence. I’ve had these conversations many times and I am often unsure how to approach them. For obvious reasons, it is impossible to argue against this kind of pure theory: without empirical evidence, and sometimes even without the rules of basic logic, there is simply no way to prove it wrong. In my experience there are people like Jonathan, who actively fight against this, and those who simply ignore theories that cannot be operationalised and tested.
The second highlight was the keynote speech by Neil Cohn. It was simply fascinating. In just one hour, Neil walked us through his narrative grammar of visual storytelling. Despite its long history, I honestly thought the concept of visual grammars was dead, but Neil proved me wrong. He combined EEG data with corpus analysis to describe a theory that sounds incredibly solid. And not just for comics, which is his primary area of research, but for cinema as well. I will definitely be reading his book and following his future research.
Have you heard about it? A goldfish has longer attention span than people! Our ability to focus has shrunk because of watching short videos. Besides other things it means that you will read only the first paragraph of my blog. So you won’t learn that the whole attention span crisis is nonsense and that our research article was accepted by Projections journal.
After several years of reading about the attention span crisis, I noticed an opposite trend this year. In March there was an article in the Financial Times disputing the shrinkage of our attention span. The author is even citing Gloria Mark:
The psychologist Gloria Mark, who is often quoted in relation to our degrading attention spans — and some of whose research landed us at this median number [47 seconds] — said on a podcast last year: “I don’t believe that our basic ability to pay attention has changed.”
For those of you who don’t know Gloria Mark’s name – her research is probably the reason why we thought there was a reason to believe that something is happening to our attention. However Gloria Mark was measuring how long it takes to switch between screens on your laptop when you are working. It is not really about attention, is it? I mean we could easily assume that when working on a Mac people would be switching more, because it is easier than on a PC. But it wouldn’t automatically mean that there is a causality between having a Mac and worse attention.
But even so, you might feel that your attention is worse than it was and that today’s kids’ attention is worse than that of kids decades ago. And exactly this feeling is the main reason we are talking about the attention crisis in the first place. According to article from Nature, no published data suggest something is happening to our ability to focus when we are not distracted. However, as the article stresses, the distractions are not always bad.
And distraction itself is not always the enemy. An important finding from controlled lab studies of attention is that even when volunteers have no external distractions, their minds still wander, often without conscious awareness that they have gone off task.
Esterman, who has spent years studying these fluctuations, argues that the flickering of attention must originate in the brain as useful internal processes that can distract us, including thoughts, ruminations and worries. Periods of mind-wandering can support creativity, planning and problem-solving, enabling the brain to explore and integrate ideas.
Now you might feel a little bit calmer about your attention. However, years of misinterpretations have brought us to a point where governments of Western countries are considering bans on social media for kids and bans on smartphones in schools. The technology is once again seen as the root of evil. This was the main topic of an online roundtable “Born Digital: Social Media, Screen‑Time, and the UK’s Public Consultation” organised by William Proctor. The discussion wasn’t about attention. It was rather about – according to panelists – baseless legislation, driven by moral panic about the new technology and scaremongering about the impact on children.
These three examples of course aren’t the first to raise doubts about our shrinking attention span and bad influence of technology. Today I bought a book Unlocked: The Real Science of Screen Time (and how to spend it better) written by psychologist Pete Etchells which is supposed to be sceptical about the bad influence of screens in 2024.
In this context I am happy to announce that this week an article I co-authored with Szonya Durant and Adam Ganz was accepted by the journal Projections. One part of the article discusses the null results from a correlation analysis we conducted. Among other things, we were interested in whether there is any relation between screen time on short video platforms (ranging between 0 and 4 hours a day) and four gaze metrics (fixation duration, length of horizontal saccades, proportion of vertical saccades, fixation entropy). And we found none. Despite a small number of participants and therefore being underpowered for small effects, I am really glad that publication of null results regarding screen time was supported by reviewers and editors of the journal.
During my travels to inspiring research sites, I visited Felix Carter at the University of Bristol. I met Felix last year at a conference and was intrigued by the fact that, as a psychologist, he is involved in research into the perception of art. Last week, when I visited Felix in Bristol, we finally had time to talk about our research. It seems to me that Felix and I have very similar research interests. He places a little more emphasis on the synchronicity of the viewing experience, while I focus more on the impact of formal means in shaping the viewing experience, but in the end we are talking about the same thing. The difference is that Felix, unlike me, understands statistics.
I took two main things away from our meeting. First, it was an amazing experience when Felix showed me what they have at MyWorld cinema. MyWorld is a research center similar in focus to Co-STAR at Royal Holloway. They also specialize in technology and its use in creative industries. And people like Felix will soon have a cinema at MyWorld for researching audience reactions (not only) to films. The cinema seats 35 people and is equipped with a projector, sound equipment, and recording technology. It has several cameras to record audience faces, as well as ports for connecting sets for recording brain and heart activity. Felix said they are considering how to connect eye trackers to individual seats. Once this is done, hopefully in the second half of this year, we will be a little closer to finding out how viewers really perceive audiovisual works. We will move from laboratories to real movie theaters.
The second thing I’m taking away from Bristol is a few ideas for joint research. First, Felix and I decided to look at my older data and see if the video format (vertical/horizontal) affects the way we scan actors’ faces. According to my data, it seems that there is a difference in how we watch horizontal and vertical videos. It stands to reason that there will also be a difference in how we observe human faces in each format. And hopefully in the second half of the year we will be able to collect some new data in Felix’s cinema.
My dear supervisor recently gave his inaugural lecture. Over the past year, I have mainly discussed my research ideas with him, so it was great to hear what he himself has been thinking about. He guided us through his reflections on storytelling that arises from the place where it takes place. He started with William Blake and, via ancient myths, Hollywood films, and his own work, arrived at AI.
I’m looking forward to seeing Professor Adam Ganz’s latest film, Synthetic Sincerity.
During November, I visited several colleagues at various universities in England and while travelling by train, I read Adam Nixon’s book. Prestigious British universities and no-budget filmmaking – you may wonder why I would connect the two, but I will show you that there is a connection.
Nixon’s book is one filmmaker’s reflection on how filmmaking has changed and a report on the possibilities of film with virtually no budget. His perspective fascinates me. He is able to think about audiovisual culture in a very broad sense. In the preface, he prepares the reader for the fact that the best film of the present day may not be the work of a professional crew, let alone a film studio. We may already be living in an era where the best films are made on TikTok.
“Imagine an unknown artist is producing the best film of the current era right now. They are creating it outside centers of power, using a cheap camera with little expense. Cineastes are waiting to discover it. The question is, where will they find it? Will it debut on TikTok, Instagram, or at one of the thousands of fringe film festivals that dot the globe? The anticipation of this discovery keeps the cinema world exciting and ever evolving.” (p. viii)
Nixon’s experience as a film school teacher shows how the situation and function of schools in my field have changed. Whereas in the past they were places where students could access the best filming technology, today everyone has a 4K camera in their pocket. Access to technology alone does not make it easier to make a film. (Let’s leave sound aside.) This attitude is very similar to what my colleague from Olomouc, Tomáš Jirsa, says—students don’t go to school because of the availability of technology and don’t expect us to teach them how to use it. They have other reasons.
Nixon’s perspective shed a completely different light on what I saw at the workplaces I visited. Paolo Russo at Oxford Brookes University showed me around the campus, showed me the facilities, and we briefly discussed the technology available to their students. With Murray Smith at the University of Kent, we discussed his recent experience with interdisciplinary research and its applicability in teaching. Ceylin Ertekin from the London School of Economics showed me around their eye-tracking lab. Deborah Klika from the University of Greenwich gave me a tour of their studios, including virtual production. And everywhere I went, I thought to myself that it would take us a very long time to catch up in Czech film and humanities departments in general.
Nixon’s book puts this into a different perspective. Thanks to smartphones, we all have a camera in our pocket (in new iPhones, even with an eye tracker). Thanks to TikTok and YouTube, we all have access to a global audience. The democratization of technology may have completely redrawn the field in which we operate. The best Hollywood producers are suddenly meeting amateurs, and gatekeepers are losing power.
Don’t get me wrong. I still think technology is essential. The fact that Royal Holloway and the University of Greenwich have virtual production studios means that they have enormous opportunities in terms of research. The excellent sound technology at Oxford Brookes University offers unparalleled opportunities for their students, even if they use it to shoot on their phones. But if we stick to the educational dimension and student expectations, Nixon is probably right and access to the latest technology is not essential. After all, the best films are already being made for TikTok.
For me, Nixon’s book is mainly an inspiration to rethink what film studies are for and what their place is in today’s education system.
SNAPlab 