Online elite chess and cognitive performance during the pandemic

Does remote working increase productivity, or decrease productivity? The pandemic forced a lot of workers into remote working, so perhaps this natural experiment can give us some idea of the impacts of remote working. Do we gain more from avoiding commuting time, greater flexibility over work time and workspace, and fewer interruptions from colleagues, than we lose from reduced interaction, supervision and structure (in addition to whatever other effects might happen in either direction)? Despite the hype, the results so far are far from clear, especially in terms of what types of jobs or work improve in a remote setting.

An interesting new article by Steffen Künn, Christian Seel (both Maastricht University), and Dainis Zegners (Rotterdam School of Management), published in The Economic Journal (open access) provides a contribution towards answering those questions. Künn et al. look at the impact of the shift to online of elite chess tournaments. Specifically:

Our data consist of games from the World Rapid Chess Championships 2018–2019, played offline in Saint Petersburg and Moscow, and from the Magnus Carlsen Chess Tour and its sequel, the Champions Chess Tour, both played online from April to November 2020 on the internet chess platform chess24.com... the majority of players (20 out of 28) in the online tournaments also competed in at least one of the World Rapid Chess Championships in the years 2018–2019, enabling us to make within-player comparisons of performance for each of these 20 players.

Künn et al. measure the performance of each chess player for every move in every one of those tournaments (with a few exceptions, and excluding the first fifteen moves for each player in each game), relative to one of the top chess engines. As they explain:

To estimate the effect of playing online on chess players’ performance, we evaluate each move in each game in our sample using the chess engine Stockfish 11... 

For a given position in game g before individual move mig, the chess engine computes an evaluation of the position in terms of the pawn metric Pigm... The numerical value of the pawn metric indicates the size of the advantage from the perspective of player i, with one unit indicating an advantage that is comparable to being one pawn up...

Künn et al. use this evaluation to generate a measure of 'raw error', being the difference in the pawn metric between the player's choice of move, and the 'optimal' move as determined by Stockfish. They then compare this raw error between play in online tournaments and play in face-to-face tournaments, for the same players. They find that:

...playing online leads to a reduction in the quality of moves. The error variable... is, on average, 1.7 units larger when playing online than when playing identical moves in an offline setting. This corresponds to a 1.7% increase of the measure... or an approximately 7.5% increase in the RawError... The effect is statistically significant at the 5% level.

The effect is quite sizeable:

Playing online increases the error variable, on average, by 1.7 units, which corresponds to a loss of 130 points of Elo rating.

In reading the paper, my first thought was that the results would be contaminated by the psychological effects of the pandemic. Fear or anxiety could easily lead to suboptimal performance, and cause the observed increase in error, rather than reduced performance in the online format per se. However, Künn et al. anticipate this in their robustness checks, noting that:

...to mitigate concerns that results are related to the pandemic, we add a control variable to the regression model to capture the severity of regulations implemented in a player’s home country during the tournament times... Although the aggregate online dummy reduces in size and significance (p-value of 0.172), presumably because lockdowns occurred only during the online tournaments, the effect pattern on the separate tournament dummies remains almost identical relative to the main results...

That doesn't quite allay my concerns, for two reasons. First, it assumes that all players react similarly to the local pandemic context, since it assumes all experience the average effect on their performance. That average effect is statistically insignificant. Second, including the pandemic variable renders the impact of online play statistically insignificant. Part of the problem is that the pandemic is happening at the same time as the switch to online play (for obvious reasons). Clearly, the natural experiment is not sufficient to disentangle the effects of online play from the effects of the pandemic. That really limits what we can learn from this study.

Finally, and interestingly, the negative effects (if we accept that there are some) decrease over time. As Künn et al. note:

...the negative effect of playing online on the quality of moves is strongest for the first (and second) online tournament. Thus, the adverse effect of playing online on the quality of moves decreases over time, possibly because players adapt to the remote online setting...

Perhaps the players have adapted to the online setting, or perhaps they have adapted to the pandemic, or perhaps the pandemic is becoming less severe over time. Given that we can't disentangle the effects of pandemic or online setting, we can't really tell.

I'm not trying to pick on this study, which uses an interesting setting to try and estimate the impact of remote work, in a case where performance can be measured reasonably accurately and consistently. In theory, that should provide as clean a measure of impact as we can find. However, once you recognise the problem in this study, it is easy to see why it would be even more difficult to use the pandemic natural experiment where the data on performance are not as clear.