Press muse – Cynical studies

Note: The original title of this post was “Trial and Tribulation”. Having thought about it, I believe it to be inappropriate since no trials (akin to a clinical trial) should have been referenced in the post – only surveys and studies. Within the text, I have changed the term ‘trial’ to ‘study’, which is more appropriate. The title was also swapped with something that has a little more ring to it.

Reading Ben Goldacre has done something to my news reading sensibilities. It got to me in a pathological way. When I read a news story that involves science, statistics and surveys, my bad science radar switches into tracking mode; the analytical arsenal primed and ready to dissect possible flaws.

So it happens, my sensors hit pay dirt this morning when I reached page B6 in the Home section of The Straits Times. The entire page was devoted to a single story on the connection between poor academic performance and youth addiction to Facebook and computer games. (read the full article here).

Too much time online hurts your grades: Polls, declared the headline. Lester Kok and Eisen Teo then open their story with a fairly definitive gambit: “IF YOU think that spending too much time on the computer is bad for your grades – well, it is.”

They cite a study from a doctoral student at the Ohio State University which “found that college students who used Facebook spent less time studying and scored lower grades than those who were not busy “poking” their friends online.”

They report that “Facebook users in the study of 219 undergraduates had grade point averages (GPAs) of between 3 and 3.5, while non-users had GPAs of between 3.5 and 4.”

A fairly straightforward claim, and an intuitive one too. But I’m just a little too nosy for my own good now, so I googled for this research. Funnily enough, it first hit the papers in the West in April 2009 – in British newspaper The Times (12 April) and in Time magazine (14 April), amongst others. So it’s pretty old stuff. Okay, to be fair The Straits Times reporters were using this research as a peg for presenting newer data which they had just gleaned, so maybe we can forgive these two decidedly stale paragraphs.

There is more than meets the eye about this Ohio research; it appears to be simply a correlational observation – when a group of people does A, they exhibit signs of B; another group of people doesn’t do A, and they don’t exhibit signs of B.

Yet nothing in this study indicates a definitive causal link between A and B. For example, we may observe that some people eat junk food and then become overweight, while we see others not eating junk food and not becoming overweight. But this doesn’t necessarily mean eating junk food causes people to become overweight.

It doesn’t even necessarily mean eating junk food is the most important factor. Other factors like failure to exercise, failure to balance diets with healthier food, underlying medical conditions, or lower metabolism could be just as, if not more, important.

The thing to remember is correlation doesn’t equate causation. And the Ohio researchers admitted as much in their press release (see:

“We can’t say that use of Facebook leads to lower grades and less studying – but we did find a relationship there,” said Aryn Karpinski, co-author of the study and a doctoral student in education at Ohio State University.

Later on in the press release, the researcher goes to some length to clarify the superficial relationship her initial research has revealed.

Karpinski emphasized that the results don’t necessarily mean that Facebook use leads to lower grades.

“There may be other factors involved, such as personality traits, that link Facebook use and lower grades,” she said.

It may be that if it wasn’t for Facebook, some students would still find other ways to avoid studying, and would still get lower grades. But perhaps the lower GPAs could actually be because students are spending too much time socializing online.

As such, while the Ohio study can provide a hypothesis for further study into the subject, it cannot and should not be taken as proof that increased Facebook use would harm academic performance. Not that the Straits Times deigned it necessary to let you in on that.

What the Straits Times had to say that was new is their findings on youth addiction to video games and its effect on academic performance. This consisted of survey data collected by online entertainment company Sulake.

The same Sulake that is “focused on virtual worlds and social networking”, whose “long-term strategic aim is to be a leader in community-based entertainment with a portfolio of properties addressing a wide range of target audiences”, and whose “properties include virtual world Habbo and social networking service IRC-Galleria”.

If this isn’t enough to sound off the alarm bells in your head, your ‘dodgy surveys’ radar probably needs a re-jig.

I can see it now – smirking Sulake executives thumbing smugly through their copy of today’s Straits Times, trying not to piss themselves with glee at an early Christmas. Free publicity on a national broadsheet, in an article which also slams one into the collective mugs at Facebook…oh the gratification!

Besides an obvious conflict of interest, there are more issues with the survey. The Straits Times seem to be attempting a study similar to that of the Ohio research on Facebook – looking for a relationship between gaming addiction and poor grades.

As their lede suggests rather strongly (see above: “If you think…), they think that there is a causal relationship between addiction to computer games and performing poorly at school.

If intent of the exercise is to discover whether computer gaming addiction is a causal factor in falling academic grades, you would likely approach the problem in one of two ways.

You can do a ‘case-control’ study – comparing groups which have the outcome or not (in this case, deteriorating grades) and seeing how common is the exposure to the hypothetical cause is in each group (computer gaming addiction). Alternatively, you can do a ‘cohort study’ – comparing groups which have had the exposure or not, and then see if there is any variation in the hypothetical outcome.

From the data put forth by ST in their story, it appears they have gone for the latter approach. For the survey Sulake questioned 653 teenagers, aged 12 to 18, on the social networking website Habbo Hotel. In paragraph 6, they stated:

Of those who were not addicted, 59 per cent said they scored mostly As and Bs, while 6 per cent scored mostly Cs and Ds and below. For addicts, 50 per cent scored mostly As and Bs while 14 per cent ended up with Cs and Ds and below.

Their choice of this approach is not surprising. It would have been far more problematic to conduct a ‘case-control’ study for this investigation. The potential causes of poor grades are a dime a dozen – isolating the sole factor of computer game addiction which the study is interested in while keeping all other variables under control is well-nigh impossible in a simple questionnaire-based survey.

Also, deterioration in grades is a relative thing – a highly intelligent person suffering from gaming addiction may achieve higher grades than someone not addicted but less intelligent. Trying to ascertain the relative grade deterioration experienced by survey respondents is difficult, if not impossible, given the way this survey was conducted.

Significant problems remain nonetheless. The survey sample is self-selecting, and thus limited to a very specific type of youth – internet-savvy teens who are users of a particular social networking site (Habbo Hotel). The possible influences this has on the survey outcome are as problematic as they are numerous. For a start, it is not implausible to suppose that such a group of teenagers who possess characteristics that predispose them to greater susceptibility to addiction and the consequences this has on their academic performances.

Also, the veracity of some of the results is also suspect, as far as one could tell. This survey counts on the honesty of its respondents, as well as the consistency with which they evaluate and answer questions that involve significant value judgement, such as ‘When you lose a game, do you get frustrated, angry and upset?’ and ‘Are your parents fine with how much time you spend on computer games?’ On addiction itself, we can reasonably expect that some addicts will not be self-aware and deny that they in fact have a problem.

It is not clear if the survey is designed to remove the influence of other possible contributory factors like psychological conditions, quality of teaching etc. But given that it was conducted by an online entertainment firm concerned more with PR outcomes than the scientific integrity of their work, it is not unfair to suppose that it probably isn’t.

Regardless, this whole exercise remains fundamentally flawed if the researchers truly desire to prove causality; by nature of its design, this survey can only detect observational correlation. To test the hypothesis that gaming addiction causes deterioration in academic performance, the decline in grades must be measured as a process occurring in teenagers as they become addicted to gaming.

The study would thus have to be conducted – if it were to be a cohort study – somewhere along these lines. Two groups of randomly selected teenagers hailing from similar familial, social and financial backgrounds, undergoing identical educational syllabi, teaching and examination methods, are to be selected. One group will be the control – they will go through the school year without access to computer games and their academic performance will be tracked. The other group would be introduced to computer games, and their subsequent levels of addiction (from no addiction to heavy addiction) and academic performances monitored. The study would also have to be double-blind – both the teenagers and the researchers should not know which study group individual teenagers belong to, so as to reduce external influences such as the placebo effect and observer’s bias.

If the hypothesis holds, the results should progressively show a negative correlation between the addiction level of the teenagers and their academic grades (the more addicted, the poorer their grades).

But clearly the conduct of a study as I have described can be justifiably derided as unethical, and for very good reasons. If such a study is to be conducted, compromises will likely have to be made, and the potential corruptions of the data be factored in.

Nonetheless, based on the results of their inadequate poll, the reporters make further tenuous claims:

The Straits Times poll found that gaming addiction was likely to hurt a child’s relationships with his parents and siblings, with just over half of them picking fights with relatives over the use of the computer.

According to the table provided, 54 per cent of those addicted pick fights with relatives over the computer. But this figure in itself means nothing – especially if a very similar percentage of those non-addicted are just as hostile to their relatives. To clarify their claim, the paragraph should have noted that, according to their poll, only 38 per cent of those not addicted pick fights over the computer.

However, does the 38 per cent represent a reliable baseline figure from which we can compare to addicted gamers? No. In fact, the 54 per cent figure is similarly unreliable.

We know next to nothing about the detailed personal background of the respondents. This is important because underlying factors, unseen by the survey, may have caused the alleged increased likelihood of being aggressive to their relatives. Gaming addiction could be a surrogate indicator of psychological inclinations that result in more hostile behaviour and poor interpersonal relationships, or of family issues – broken families, financial difficulties etc.

In essence, gaming addiction could be how teenagers with underlying personal problems cope with the issues in their lives. Rather than being the cause of their aggression and hostility, gaming addiction might be something that accompanies these traits, as part of their coping mechanism for their problems.

Only a simplistic interpretation of the data could back up what the reporters claim, but as we know, correlation doesn’t equate causation.

In any case, the real point of the story was not proving causality – the mere intuitive suggestion is more than sufficient. The thrust of the piece is the fingering of the new scourge of teens – online social networking – insinuating that it siphons time and energy away from more useful pursuits…like studying.

In this context, these warnings are merely a natural extension to the old cautionary tales directed against the proliferation of computer games, mobile phones, tamagochi (remember that?), glue sniffing, cards, marbles…ad infinitum.

While the answer to addiction lies more with the addicted than the offending game, product or service, the idea that the problem is really within us, is understandably hard to swallow. The temptation is therefore to externalise and objectify it – to see the addiction as an externally introduced affliction that can be eradicated, rather than a systemic “flaw” in our psychological make-up that we may never be rid of.

The castigation of Facebook and computer games is but the latest incarnation of this externalisation, informed perhaps by an inherent phobia of technology and the rapid societal changes it induces. A witchhunt that is less about understanding the problem than fulfilling the cultural meaning of the act. A ritual bequeathed by modernity to our risk society in perpetuity.


~ by spiegel2071 on November 28, 2009.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: