Flaws found in video game studies

(Medical Xpress) -- Over the past several years, many studies have found that people who regularly play action video games outperform people who don’t on tasks that involve perception and cognition. However, a new study has found that most, if not all, of these studies suffer from common pitfalls in experimental design, so that the results of the studies are undermined by methodological shortcomings.

The new study notes that, although game training could prove very useful if it works, it is actually in conflict with the typical findings of cognitive training research, which shows that training in one task rarely improves performance in other tasks. In this light, the possibility that video game training causes “broad transfer” of skills beyond gaming itself would be unusual, but potentially very constructive.

In their review article published in Frontiers of Psychology, Walter Boot and Daniel Blakely from Florida State University and Daniel Simons from the University of Illinois have discussed the methodological shortcomings of previous video game studies and offer guidelines for improved testing methods.

First, one of the biggest problems with video game studies was the recruiting process. In many studies, participants knew they were being recruited due to their gaming expertise, which could make them expect to perform well, and the expectation could in turn influence their performance. In contrast, non-gamer participants who were selected without gaming expertise would not experience the same motivation. This recruiting process violates a core principle of experimental design, which Boot, Blakely, and Simons suggest could be improved by covert recruiting. For instance, participants could be asked about their video game experience only at the end of the study or in a prescreening that is not linked to the particular experiment. Still, the review authors note that comparing gamers and non-gamers does not account for the possibility that people may become action gamers because they have the abilities required to excel at these games, or that a third factor might influence both gaming and cognitive abilities.

In other studies, all the participants are non-gamers and they are assigned to either action game training (the experimental group) or non-action game training (the control group). Participants’ performance on tasks involving perception and cognition is measured before and after their training. The problem with this set-up is that participants know which training they’re receiving and whether or not it should improve their performance on the cognitive tasks. Similar to the recruiting problem, participants trained on action video games may expect their performance to improve as a result of their training, while participants in the control group may have no such expectation. Boot, Blakely, and Simons suggest that participants should be asked whether they perceived a connection between their training and the tasks, and also whether they’re aware of past studies that have shown the benefits of gaming on cognitive performance.

The authors of the review also note that the results of some of these studies seem to conflict with one of the most fundamental principles of learning, which is that performance improves with practice. One would expect that participants who trained on non-action video games should improve at least a bit the second time they perform the cognitive tests, but many studies have found a lack of improvement. This unexpected lack of improvement may give the appearance of a greater benefit of action video game training in comparison. Interestingly, studies that do find an improvement in the control group often find no significant effect of video game training on cognitive performance. As the review authors suggest, in order to make claims about the benefits of video game training, the control group should perform as expected.

Finally, because video game training studies are costly and time-consuming, the results are often split up and published in multiple journal articles, even though they use the same training groups. Boot, Blakely, and Simons recommend that, when multiple results of a single experiment are reported in different articles, the connection should be clearly stated to avoid confusion and publication bias.

Overall, the review authors emphasize that their criticism does not mean that the results of all previous video game studies are wrong, just that their experimental methods are not compelling enough to draw strong conclusions. They emphasize that adopting a set of clinical trial best practices could illuminate the true benefits of gaming for cognitive performance. If future studies with improved methods can confirm that action video games do improve perception and cognition, then game training holds great promise for people with vision and attention disorder, as well as for remediating the effects of cognitive aging.

© 2011 PhysOrg.com