Science of Chess: Seeing moves with momentum over the board

I've been into astronomy for about as long as I've been a chess player. To this day, I have a decent collection of telescopes that includes an 80mm refractor with wonderful optics, a 90mm Maksutov-Cassegraine telescope that is small enough to travel with and has a camera port for astrophotography, and a big ol' 8-inch telescope for seeing all the really cool deep-sky stuff. Though the night skies of Fargo can be a little bright sometimes, I still try to get outside every so often to see the planets, or the Pleiades, or just take in the fascinating terrain of the Moon.

One thing about star-gazing that fascinated me as a kid is that looking at celestial objects necessarily involves seeing them how theywere rather than how theyare. Light takes time to get from a planet or a star to your eye, and the vast distances involved mean that this transit time becomes fairly meaningful: Jupiter, one of my favorite telescope targets, is about 45 light-minutes away, That doesn't sound so bad, but once we're looking at things outside our Solar System it gets more interesting quickly: The Orion Nebula is another one of my favorite destinations and it's light takes about 1500 years to get to us! Some profound cosmic event may have changed what's happening in that part of the sky a millennium ago and we'd still have to wait another 500 years to see the consequences.

If your brain lags behind, use your vision to think ahead!

All of that stuff is happening on a galactic scale, but a fun fact about your visual system is that it has to cope with a similar problem. While light can get from objects and surfaces out in the world to your eye in a negligible amount of time, there is still a meaningful gap between what you're seeing and what is actually happening right now. The reason for this is that while light from the immediate environment gets to you instantly, your neurons take some time to send signals about that light from your retina to the parts of your brain that support your conscious awareness. You can see a schematic view of that pathway below, which I hope makes it easier to see that your vision is really a bit of a relay race, with cells at each stage passing the baton (neural responses) on to the next cell in line.

By Ratznium at en.wikipedia, CC BY 2.5, https://commons.wikimedia.org/w/index.php?curid=3199004

The passing of that baton turns out to take something like 50-100ms, which doesn't sound so bad compared to our astronomical time-delays. Even though the time scale is a lot shorter, it still has important consequences for us, however. A lot can happen in 1/10th of a second, after all, and I'm sure you can imagine any number of fast-moving objects like cars, projectiles, or even other people that could cover a dangerous amount of ground in that kind of time. If you were consistently off by a 10th of a second, surely there would be many more car accidents, lots more bumping into people as you were walking around, and Pickleball would be more or less impossible.

Adobe Stock Image. Good luck with your Ladder League if your vision lags!

Fortunately for us, your visual system copes with this bit of lag handily. To keep your experience of the visual world as up-to-date as possible, what reaches awareness isn't just what you actually saw - instead, you get a prediction of what is happening now based on that data. That is, your visual system uses the incoming sense data to project ahead a little bit to cover that 50-100ms gap, so you aren't constantly just a little bit behind.

The flash-lag effect - seeing the future and the past simultaneously!

I recognize that this may sound pretty wild and perhaps a little unbelievable. Besides, how on Earth could we know such a thing? Look, personally, I think the fact that professional tennis happens is in itself a reasonable piece of evidence in favor of this idea, but you may be looking for something more convincing. To that end, I give you the Flash-Lag Effect. This is a wonderful visual illusion that lays bare the predictive nature of your perception by taking advantage of a visual event that you can't predict easily, leading to a systematic error in what you see.

Many versions of the Flash-Lag Effect involve a skinny rectangle that is rotating clockwise at a fairly rapid clip. So far, this is an easy thing for your visual system to make predictions about, but we're going to add something to the display that complicates things for you. Specifically, each time this spinning rectangle reaches a particular spot (let's say the diagonal position in which it's pointing from the lower-left to the upper-right) we're going to briefly flash some new rectangles that extend beyond our spinning rectangle on both ends (see below). For a moment, this means that our spinning rectangle is lined up with some collinear partners, here's the interesting part: Because the spinning rectangle is easy to make predictions about and the flashed rectangles aren't, your visual system provides you with a prediction about the spinning rectangle but can't follow suit for the small ones! The result is that even though all three rectangles were physically aligned on-screen, what you see is a "broken" arrangement of the rectangles (see the right image below).

Image Credit: Hogendoorn (2020) - A schematic depiction of the physical set-up in a Flash-Lag experiment (left) and what observers typically see (right).

You can try out a demonstration of the Flash-Lag effect at Michael Bach's wonderful Visual Illusions website if you're not convinced. There are also many other related demonstrations of this effect that use a translating object instead, or sometimes an object that is orbiting a central point - in each case, this disjointed experience arises consistently from the discrepancy between what you can predict and what you can't.

Making visual predictions in chess (Ferrari et al., 2006)

What does all this have to do with chess? The authors of the study I want to describe for you here motivate their work by pointing out that strong play doesn't just involve understanding positions in terms of the static position of the pieces at a particular moment in time, but also understanding what sequences of moves might follow from a particular position. The ease with which strong players do this was something that I found remarkable when I first started watching chess streamers a few years ago. Hearing Hikaru or Magnus casually rattle off potential sequences of moves that they thought would be good lines from their current position felt like a clear demonstration that they had an incredible ability to see not just what was in front of them, but what could be in front of them in a few moves (and sometimes many more than just a few!)

I hope this sounds at least a little familiar: What our group of researchers were arguing for in their study is that this understanding of the dynamics of a position, or more specifically, the ability to make predictions about the next steps in a game, may be an obligatory visual process that unfolds for stronger players as they look at a position. To be clear, the idea is that this is different than explicitly calculating potential candidate moves and their consequences! Certainly strong players are good at calculating and may use visual imagery to guide them, but the big idea here is there is also an involuntary process that just happens on its own. Just like you can't help but predict the position of the spinning rectangle in the Flash-Lag effect, the proposal here is that strong chess players perhaps can't help but predict likely next moves from a position that they are considering. To put it another way, it's like their perception of the board has momentum. The question is, can we see that there are consequences of this act of prediction? Does seeing into the future a little bit make it so that players who predict see something analogous to the misaligned objects in the Flash-Lag display?

Experiment 1: Looking for evidence of predictions over short time-scales

One of the things that I was excited to see in this study was an experimental design that looked very familiar to me. During my PhD, I did a number of studies about how we perceive static images of objects that can move and that work included some examinations of how your visual system might make predictions about familiar objects like human bodies. Because we know how bodies tend to look when they're walking or running, does that mean our visual system is particularly prone to projecting forward a little in time when you see a static image of a person?

One way I decided to look for evidence of those predictions was to use a simple same-different visual task with images of people walking. For my participants, the task was very simple: First they would see one picture of someone mid-stride for 500ms, then they would see a second picture of that person after a short delay. Their job was to decide if these pictures were exactly the same or if there was any difference at all between the two pictures. The sneaky part of this study was that when the two images were physically different, half of the time this meant that the second image was a little further along in the striding motion (the Forward order condition) and half of the time the same two images were presented in the opposite order (the Backward order). My logic was that if your visual system started with the first image and predicted what it might turn into, that prediction might look a lot like the second Forward order image, leading you to mistakenly say "same" even though they were different. The same process applied to the Backward pairs would make them easier to tell apart because now that forward prediction would make the first picture look less like the second. This was part of what we found, suggesting that these predictions may be happening automatically when you looked at images of people in motion.

Adapted from Balas & Sinha, 2009.

The authors of the current study adopted very similar logic and applied it to chess boards instead of pictures of people. The idea here was to present weaker players (~1500 ELO or so) and stronger players (~1900 ELO) with positions like the ones you see below and ask them to complete a same-different task with the boards. Just like my experiment with walkers, participants were asked to indicate if the two positions that they saw in sequence were identical or had any differences between them. These authors used the same sneaky design trick I included in my study, too: When the two positions are different, half of the time the difference between them is that on the second board a typical next move following from the first position has been made. The other half of the time, participants would see those same two positions in the opposite order, as though the game was being "wound back." These two versions of what "different" positions are were referred to here as the normal order and reversed order conditions in their design. You can also see in the figure below that the authors included standard next moves and non-standard next moves, which gives them a number of interesting hypotheses to examine:

1. Stronger players may show different performance in the same-different task for normal and reversed positions due to strong predictive perception.
2. Standard positions may yield different performance than non-standard positions because predictions will be more consistent with standard next moves.
3. These two effects may interact: Strong players may have a bigger difference for normal vs. reversed boards, but only in standard positions, for example.

Adapted from Ferrari et al., (2006) Figure 1. The same position pairs can be presented in normal or reversed order to test the strength of predictions during visual inspection of chessboards by players of varying strength.

So what did they find? In their first experiment, participants got to see the first position in a single same-different pair for 5 seconds, and then had to decide if the second position that replaced it was identical or not as quickly as they could. This gave the experimenters the chance to measure players' accuracy and how quickly they made correct responses to the critical "different" pairs. You can see in the table below that the presentation order of the two boards does turn out to matter! Stronger players (but not weaker ones) were more accurate at detecting differences between boards when the two positions were presented in the normal order than when the same two positions were presented in the reverse order. This was the case for both the standard and non-standard positions, suggesting that stronger players are making obligatory predictions about these positions and that this helps them notice when the next board has a change consistent with that prediction.

Adapted from Ferrari et al. (2006) Table 2 - Expert players are better at detecting differences between positions when they are presented in an order consistent with the sequence of moves in classic openings.

Things look mostly the same when we consider how quickly players' were able to make correct decisions about "different" pairs, but with one crucial and interesting difference. If you take a look at the bar chart below, you'll see the average response time across conditions for both participant groups and all of the experimental conditions. In this plot, a higher value means worse performance insofar as it means it took you more time to get to the right answer. Here, the Expert players are slower when making correct "different" responses to the reverse order boards, and unlike the accuracy data this effect appears to be weaker in the non-standard positions! To me, this is really kind of cool as it suggests a subtler aspect of predictive visual processing: One's familiarity with specific opening sequences may make those predictions particularly strong or perhaps more impactful on your visual processing than other possible futures.

Adapted from Ferrari et al. (2006) Figure 2.

Experiment 2: Looking for evidence of predictions over longer time-scales

This is already pretty interesting stuff, but the authors have a second experiment that adds some further interesting insights into how visual predictions manifest during expert and beginner consideration of chess positions. In this second task, almost everything is the same except for one crucial difference: Instead of seeing pairs of positions one right after the other and completing a same-different judgment, now the authors asked their participants to complete a memory task. This means that now they would see a collection of positions in sequence for 5 seconds each, and then take a short break before taking a memory test. In that test, they would see some positions they had been shown before ("old" items) and some that they hadn't been shown ("new" items) - the question is, can they tell which is which?

Like Experiment 1, the researchers built in the same forward and reversed pairs, but use them somewhat differently. The idea here is a little more like my body perception study: If you can't help but predict the future a little after seeing a chess position, maybe what you store in your memory is not what you actually saw, but a guess about the future. This should mean that players who predict automatically are more likely to produce "false alarms" by saying that new test items that are one forward move along are actually old! The neat thing about this experiment is that unlike the first one, it puts stronger players in a position where the thing that serves them well over the chessboard makes them worse at performing this particular task.

Adapted from Ferrari et al. (2006) Figure 3.

This is indeed what the authors reported, and you can see the crucial data in the figure above. The most important parts of this graph are the "False alarm" sections because these indicate how often players at each level made mistakes by saying that a "new" item was actually something they had seen before. While Beginners make these mistakes at the same rate for both kinds of new items, Expert players fall for the positions that are a little further along in the sequence of moves.

Conclusions

The takeaway is that strong players do seem to make visual predictions automatically from chess positions, and depending on what you're asking them to do this can be either a good thing or a bad thing! Chess expertise thus isn't just about accumulating a sort of library of static board positions, but also about understanding sequences of play and how positions may transform from one move to the next. Over the board, as in the real world, it turns out that Wonderland's Red Queen may have been right: you've got to keep moving just to stay in one place!

Support Science of Chess posts!

Thanks as always for reading! If you're enjoying these Science of Chess posts and would like to send a small donation my way ($1-$5), you can visit my Ko-fi page here: https://ko-fi.com/bjbalas - Never expected, but always appreciated!

References

Balas, B., & Sinha, P. (2009). Learned prediction affects body perception. Visual Cognition, 17(5), 679–699. https://doi.org/10.1080/13506280802327866

Ferrari, V., Didierjean, A., & Marmèche, E. (2006). Dynamic perception in chess. Quarterly journal of experimental psychology (2006), 59(2), 397–410. https://doi.org/10.1080/17470210500151428

Hogendoorn H. (2020). Motion Extrapolation in Visual Processing: Lessons from 25 Years of Flash-Lag Debate. The Journal of neuroscience : the official journal of the Society for Neuroscience, 40(30), 5698–5705. https://doi.org/10.1523/JNEUROSCI.0275-20.2020