Brain Training Pop Quiz: BBC + Nature = ?

brain training pop quiz

Select the appropriate BBC Brain Training answer:

BBC’s Brain Test Britain + Nature = ?

a) A well-researched, no nonsense brain training report
b) A landmark scientific refutation of cognitive training gains
c) Results of the largest study of brain exercise published by a respected journal
d) None of the above

It’s a trick question of course. While it would seem that the BBC’s reporting and fact-checking pedigree coupled with Nature magazine’s reputation for publishing serious scientific findings would add up to something significant, here it’s not the case.

With Brain Test Britain the BBC engaged scientists to design a battery of cognitive training tasks to test the claim that such training can improve not just performance on the tasks but also general cognitive ability. The BBC then used its not inconsiderable media exposure to engage a very large number of people (11,400 or so — which is still only about .06% of the population of Britain) in the training program.

Here’s the catch: The very premise of the study ignored existing, well-respected research on the kinds of brain training that can result in transfer to general cognitive ability. In particular:

1. The training was not frequent enough (3 days per week vs. 5 days per week in successful studies)

2. The training sessions were not time-intensive enough (10 minutes vs. 30 minutes)

3. The training tasks were varied instead of focusing on training that has previously shown transfer.

On average the participants in Brain Test Britain trained for about 250 minutes over the course of six weeks on a variety of tasks. In contrast, Mind Sparke Brain Fitness Pro’s initial training period requires about double the training time over just four weeks and focuses on a single task. It’s no wonder that there was no transfer.

Dual n-back working memory training places a very specific demand on the trainee. It is not just a task that places demands on working memory, it is a task that requires committed, focused, single-minded attention for a minute at a time. Each session includes twenty such spans back to back.

But perhaps most interesting stumble of all in the Brain Test Britain methodology was in its measurement of general cognitive ability. The study used a generally available cognitive assessment battery from Cambridge Brain Sciences (I’d encourage anyone interested to register and check out the tests. It’s free.)

I just went through the test battery, and in my humble opinion it’s really not a very good measure of the kinds of cognitive ability that would be valuable and useful in everyday life. Every test is time-pressured (not that this is a bad thing per se, but in everyday life much of our cognitive processing permits us some luxury of time and thought; especially tough problems.) And although there’s a matrix test, it’s not a particularly inspired one.

This astounded me: On the two reasoning tests (essentially these were fluid intelligence tests) I scored in the 19th and 25th percentiles respectively!! Whereas, my fluid intelligence on several full scale, well-normalized tests taken over the past 28 years has been above the 98th percentile and in some cases in the 99th percentile.

I would predict that a course of training with Brain Fitness Pro might help someone score better on the Cambridge Brain Sciences tests, but I wouldn’t expect this to bring them great satisfaction, nor to be an accurate measure of their true cognitive gains. Whereas a course of training with Brain Fitness Pro will translate into everyday benefits to someone’s thinking ability and problem-solving skills in the real world, as well as their scores on tests that truly measure intellectual ability.

Tags: , , , , , ,

2 Responses to “Brain Training Pop Quiz: BBC + Nature = ?”

  1. Will says:

    Hi Martin,

    Thanks for the link to the Camridge tests. I’m a bit confused by your scores on these in light of the normal range you score in on fluid tests (as well as your capacious working memory capacity). You might want to try again as I’m very confident you can score in the range you normally do. I’d suggest that you not worry about the speeded component for the object reasoning task and just resond once you see the pattern emerge. I think the trick is to look for two patterns on the tougher ones before choosing. The time element actually makes this a good training task for quick reasoning…as you point out, though, this taks may not serve at the
    more complex levels where more time is required.

    My scores on the object reasoning and odd man out test as well as spatial digit span seem to be consistent with my other scores on WM or fluid measures of intelligence (assuming a normal distribution for these Cambridge tests which is probably a wrong assumption). However, my perceptual scores are lower than usual — I score well on highly speeded perceptual tasks over at Mybraintrainer but yet I scored average range here…maybe this is tapping some different are of the brain or maybe I am overthinking it? Not sure.

    One problem is that norming with the Cambridge tests is set against users of the site as opposed to the general population distribution…so its
    not very clear if an IQ rating would be appropriate or what the percentile
    score really means. They say a “general population of people using this site” — yet we don’t know what the general populatin consists of…if it’s average internet users, the mean of users here are likely above average intelligence. Overall, though, these Cambridge Brain Science tests seem pretty standard. While not “inspired” generally the whole idea of standard
    is to negate inspiration. That is, there’s nothing particularly inspired about
    neuropsychological batteries or standardized tests.

    As for the study itself, I think it does say something about brain training games in general as not really being the panacea for the general public that they are marketed as…for instance, I don’t think mybraintrainer has
    done much to improve my intelligence, but it might “keep me sharp” just
    as any of these other brain training games would. I think a light workout at a health club (over the span of a year) might not get one in shape per se, but then, it might help in keeping someone from becoming obsese. If someone is playing brain games, it’s better than watching American Idol but maybe not as productive as more rigorous intellectual challenges….

    As for dual n back, though, I think it stands alone as the best brain training that I’ve done above and beyond other brain training games on the market. I think the Jaeggi study did deserve mention in the study — to show that there is scientific evidence for gain in fluid intelligeence from brain training. However, mention of it might have detracted from the aim of what the authors ultimately wanted — to undercut the brain training hype. N back is not quite the baby, and generic brain training games are not the bathwater in this instance…but, there is an element of that.

  2. martin says:

    Hello, Will.

    Thank you, as always for a very thoughtful and measured response. It only occurred to me after writing the post that the Cambridge tests were likely normalized against a skewed population (although I assumed that this would include the tens of thousands of people who signed up for Brain Test Britain — which may well be a false assumption).

    You’re also probably right about my being able to improve my scores if I ignored the time pressure. I’ll have to give that a try.

    And, lastly, I think you’re correct that the study was probably intended to debunk the myth of brain training games as a panacea. But how many people will realize that? People tend to take this kind of reporting at face value. On the other other hand, how many people will remember this report in a couple of years time…


Leave a Reply

You must be logged in to post a comment.