Recently, I had the opportunity to attend the Mind Art Knowledge conference, where I presented our latest research on dual-screen watching.
The core focus of our poster is the emergence of “ambient TV.” Until now, ambient TV has mostly been treated as a neat concept in cultural journalism. In our poster, we proposed a concrete, empirical way to actually test this phenomenon.
Because we presented it in a poster format, we intentionally leaned into the medium to make it a bit of a provocation…
Yesterday, I was lucky to visit the Creative Computing Institute (CCI), and specifically Tim Smith‘s team, at the University of the Arts London. Tim invited me to give a short presentation about the research I am conducting at Royal Holloway under the supervision of Adam Ganz and Szonya Durant. Besides that, Tim showed me their facilities, and we had a talk about computer vision in cinema research.
In my talk, I presented a selection of the most interesting findings from the last 18 months: a) the difference between watching videos in vertical and horizontal frames, b) a null result regarding the effect of screen time on gaze patterns (both soon to be published in Projections), c) how people simultaneously watch narrative stimuli on TV and TikTok on a tablet (article just before submission), and d) which formal elements of narrative stimuli pull visual attention from a tablet to a TV screen (what I am working on). My main argument was that, contrary to my initial assumption, TikTok (and short video platforms in general) is probably not changing how we watch films. Rather, short videos (among other things) have made us aware of the audience’s inattentiveness. However, the fact that people are distracted is not new and it was always like that. It is just more visible now.
On a personal level, it was amazing to see the Creative Computing Institute and how Tim approaches research in the arts and humanities. I know very well from CoSTAR that arts research can look very different from what I was trained in. The CCI combines research activities with teaching, and it is inherently interdisciplinary. Truly, one can hardly say where one discipline ends and another begins. An illustrative example was when Tim showed me how they connected a virtual production studio with fNIRS and used it in a dance performance. The dancer’s brain activity produced a visualization on the LED panels behind him. This single example clearly shows how art can merge with neuroscience, technology, and programming, proving that these disciplines are not only communicating but also mutually beneficial.
Here’s a poster we presented in Amsterdam. Some more details can be found here and in repository.
Have you heard about it? A goldfish has longer attention span than people! Our ability to focus has shrunk because of watching short videos. Besides other things it means that you will read only the first paragraph of my blog. So you won’t learn that the whole attention span crisis is nonsense and that our research article was accepted by Projections journal.
After several years of reading about the attention span crisis, I noticed an opposite trend this year. In March there was an article in the Financial Times disputing the shrinkage of our attention span. The author is even citing Gloria Mark:
The psychologist Gloria Mark, who is often quoted in relation to our degrading attention spans — and some of whose research landed us at this median number [47 seconds] — said on a podcast last year: “I don’t believe that our basic ability to pay attention has changed.”
For those of you who don’t know Gloria Mark’s name – her research is probably the reason why we thought there was a reason to believe that something is happening to our attention. However Gloria Mark was measuring how long it takes to switch between screens on your laptop when you are working. It is not really about attention, is it? I mean we could easily assume that when working on a Mac people would be switching more, because it is easier than on a PC. But it wouldn’t automatically mean that there is a causality between having a Mac and worse attention.
But even so, you might feel that your attention is worse than it was and that today’s kids’ attention is worse than that of kids decades ago. And exactly this feeling is the main reason we are talking about the attention crisis in the first place. According to article from Nature, no published data suggest something is happening to our ability to focus when we are not distracted. However, as the article stresses, the distractions are not always bad.
And distraction itself is not always the enemy. An important finding from controlled lab studies of attention is that even when volunteers have no external distractions, their minds still wander, often without conscious awareness that they have gone off task.
Esterman, who has spent years studying these fluctuations, argues that the flickering of attention must originate in the brain as useful internal processes that can distract us, including thoughts, ruminations and worries. Periods of mind-wandering can support creativity, planning and problem-solving, enabling the brain to explore and integrate ideas.
Now you might feel a little bit calmer about your attention. However, years of misinterpretations have brought us to a point where governments of Western countries are considering bans on social media for kids and bans on smartphones in schools. The technology is once again seen as the root of evil. This was the main topic of an online roundtable “Born Digital: Social Media, Screen‑Time, and the UK’s Public Consultation” organised by William Proctor. The discussion wasn’t about attention. It was rather about – according to panelists – baseless legislation, driven by moral panic about the new technology and scaremongering about the impact on children.
These three examples of course aren’t the first to raise doubts about our shrinking attention span and bad influence of technology. Today I bought a book Unlocked: The Real Science of Screen Time (and how to spend it better) written by psychologist Pete Etchells which is supposed to be sceptical about the bad influence of screens in 2024.
In this context I am happy to announce that this week an article I co-authored with Szonya Durant and Adam Ganz was accepted by the journal Projections. One part of the article discusses the null results from a correlation analysis we conducted. Among other things, we were interested in whether there is any relation between screen time on short video platforms (ranging between 0 and 4 hours a day) and four gaze metrics (fixation duration, length of horizontal saccades, proportion of vertical saccades, fixation entropy). And we found none. Despite a small number of participants and therefore being underpowered for small effects, I am really glad that publication of null results regarding screen time was supported by reviewers and editors of the journal.
The last day of the conference had two highlights for me.
The first highlight was a reading from Deborah Klika’s screenplay Czech in the Rear View Mirror. I have known Deborah since 2023 and I know that she is proud of her Czech roots. Her screenplay is the story of her parents, who emigrated to Australia. The table read of the script was a great experience, partly because it was funny, but mainly because Deborah found Czechs living in Adelaide to play the two main characters. In addition, two other ladies from the Czech club in Adelaide came to the reading. After the reading, I had the opportunity to talk to them and it was nice to hear their stories, in varying degrees of Czech language recall.
The second highlight of the day was a presentation by Alicia Butterworth. Alicia is a sound designer and she talked about how they developed and created the VR experience We Were Children Once about Holocaust survivors. I consider it the best paper of the entire conference. Alicia was able to introduce us to the details of a sound designer’s work in preparing VR and concluded her talk with a set of recommendations for VR creators. Among the recommendations were the following regarding directing the audience’s attention in 360 degrees:
– Involve sound from pre-prod – we can help direct attention.
– VR is overwhelming for the audience – attention cues need to be significantly more obvious (louder/more present) to affect the audience than in traditional disciplines.
Sound designers therefore think about attention quite consciously. This reminded me of James Cutting’s observation that filmmakers are great psychologists. This is probably even more true of sound designers. There were two contributions from sound designers at the conference, and both showed that they think about audiences quite differently from screenwriters. From screenwriters at the conference, I heard mainly reflections on the diversity of audiences and the impossibility of generalizing about them. Sound designers approach this from the opposite angle—from perception—and thus have a quite different idea of audiences based on shared sensory experience.
That’s all from this year’s conference, and see you next time in Oxford!
On Thursday, May 15, 2025, we welcomed Tim Smith from the University of the Arts in London to CoSTAR National Lab. Tim was kind enough to accept the invitation from Adam Ganz and me.
In his lecture, he talked about Continuity Editing Rules and why we, as viewers, enjoy watching (mainly) Hollywood movies so much. He also discussed how filmmakers control the audience’s attention. According to Tim Smith, filmmakers are brilliant intuitive psychologists. Without scientific knowledge of human cognition, they are able to very successfully control where viewers look in films and how they perceive them. He illustrated this with a range of data from his experiments.
I have been familiar with Tim’s work for many years. I mention several of his experiments and findings to my student in cognitive film theory classes. However, two things never occurred to me.
1) Tim refers to almost no film theorists. The exceptions are Janet Staiger, Kristin Thompson, and David Bordwell. He finds much more inspiration in the work of cognitive psychologists (J. J. Gibson, Daniel Simons, Joseph Anderson), neuroscientists (Uri Hasson), and especially filmmakers, whom he quotes extensively (Spielberg, Tarantino, Dmytrik, Eisenstein).
2) How much he loves film. He is genuinely fascinated by film and has a vast knowledge and passion for cinema.
I had the opportunity to talk to Tim before and after the lecture, and I was pleased by his vision of an ever closer connection between filmmakers and cognitive film studies. However, we also briefly touched on the topic of TikTok as a competitor to Cinematic Continuity in the battle for audience attention. So perhaps the main question is what form film will take in the future.
Exploring the Cognitive Foundations of Cinematic Continuity
Professor Tim Smith (University of the Arts London)
Thursday 15th May, 16:00
SHILLING-LT (Shilling building, Royal Holloway)
Filmmakers tell stories by selecting and emphasizing key details of an audiovisual scene through editing, cinematography and sound design. Such edited sequences instantaneously transport the viewer through space and time in ways that are physically impossible and, due to their divergence from reality should pose problems for viewer comprehension. However, filmmakers believe they have at their disposal a suite of editing techniques, known as the Continuity Editing Rules that make the viewing of film effortless. In this talk I will present a series of behavioural and eye tracking experiments investigating the cognitive foundations of film perception and provide a theoretical and methodological framework for extending filmmaker insights into audience experiences of film.
Tim J. Smith BSc. PhD. is Professor of Cognitive Data Science in the Creative Computing Institute, University of the Arts London and head of the Cognition in Naturalistic Environments (CINE) Lab. He applies empirical, computational, neuroscientific and developmental methods to questions of Media Cognition, including real-time audience experience analysis via eye tracking. He has published widely on the subject both in Psychology and Media journals and his research has informed media practices through collaborations with Dreamworks Animation, BBC, Channel 4, and the Academy of Motion Picture Arts and Sciences.Currently he is leading the UKRI cross-council multi-site research project, Animating Minds: Triangulating the age-appropriate impact of children’s media.
It’s evening. I’m not going out, but I don’t feel like reading, so I’m watching a movie on an unnamed streaming platform. When choosing, I prefer movies I’ve already seen. Why? So I don’t feel guilty about looking at my phone every now and then.
A not insignificant number of viewers are reportedly thinking along similar lines to this scenario. They watch a film unfocused, or while doing something else. In another post I addressed casual viewing and today I want to look at the changing ways of viewing audiovisual content from the perspective of watching multiple screens simultaneously.
There are several interesting studies on the topic in which researchers have looked at the viewing practice of watching multiple screens simultaneously.
All of the studies are impressive because they experimentally test how viewers behave. They use an eye-tracker (mobile or stationary) and think about setting to resemble a living room as much as possible and a laboratory as little as possible. In other words, they make sure that the experiment reflects the natural environment.
The results are then very compelling and show that viewers are primarily watching TV and paying less attention to the tablet/phone (by a ratio of 2:1 to 5:1).
The problem is that all these experiments are formulated from start to finish in such a way that it is clearly established which screen is “first” and which is “second”. Even the choice of what is played on the second screen is only complementary to what is played on the first screen. For all three studies, this approach is understandable. Their intent was to offer data to make multi-screen viewing consistent, which could then be used by media companies to develop applications. But if we are interested in the relationship between audiovisual and spectator behaviour, we necessarily feel that something has been left out. They left aside the fact that there is a competition for our attention in the media market.
I don’t know anyone who would say they watched TikTok last night and occasionally glanced at what was happening on Netflix. That’s not how we talk about leisure time. Statistics, meanwhile, show that the average U.S. adult in 2024 watched Netflix for only 3.7 minutes longer per day than TikTok (62.1 vs 58.4 minutes). Is that enough of a difference to determine which screen is hierarchically first and second? I’d say no.
It may well be that the attention paid to movies and TikTok is different when the division between first and second screen is abolished. And it’s also quite possible that some movies are less successful at attracting attention (though studies suggest that this is not the case).
I would like to build on the three studies mentioned and think about how to implement experiments in a way that reflects the battle of the screens. To start with, users will need to break the hierarchy of the first and second screens and let them be free.
SNAPlab 