Parachutes may be ineffective as safety devices

A lot of what you read about research in mainstream media is taken from press releases (by universities or research institutes), or at best, is taken from the abstracts of research papers. Randomised controlled trials are the gold standard in evaluation research (especially in health), so many readers might not take a critical eye when reading the studies. This would be a mistake.

A good example of where a lack of critical reading would go horribly wrong is this article from the Christmas issue of the British Medical Journal, by Robert Yeh (Harvard Medical School) and co-authors, entitled "Parachute use to prevent death and major trauma when jumping from aircraft: randomized controlled trial". Here's the first sentence of the conclusions from the abstract:

Parachute use did not reduce death or major traumatic injury when jumping from aircraft in the first randomized evaluation of this intervention.

One could conclude from that sentence that parachutes are ineffective as safety devices. However:

Compared with individuals screened but not enrolled, participants included in the study were on aircraft at significantly lower altitude (mean of 0.6 m for participants v mean of 9146 m for non-participants; P<0.001) and lower velocity (mean of 0 km/h v mean of 800 km/h; P<0.001).

The authors tried to enrol people into their randomised controlled trial by asking people:

...whether they would be willing to be randomized to jump from the aircraft at its current altitude and velocity.

They would be randomised into making the jump either with, or without, a parachute. The only people willing to be randomised in the study, unsurprisingly, were those on stationary aircraft on the ground. This limits the external validity of their sample a little, but does allow them to conclude that:

...although we can confidently recommend that individuals jumping from small stationary aircraft on the ground do not require parachutes, individual judgment should be exercised when applying these findings at higher altitudes.

This study was, of course, a response to an early BMJ article on parachutes from 2003, which concluded that:

...the effectiveness of parachutes has not been subjected to rigorous evaluation by using randomised controlled trials.

Well, now the effectiveness of parachutes has been evaluated, and found wanting. Read both papers (they're open access); they're hilarious (as has been the case with many past papers in the Christmas issue of the British Medical Journal, such as this one I blogged about in 2017).

[HT: Thomas Lumley at StatsChat, whose pet hate is journalists quoting uncritically from press releases or abstracts of non-peer-reviewed papers]

When the biggest changes in alcohol legislation aren't implemented, you can't expect much to change

Back in October, the New Zealand Herald pointed to a new study published in the New Zealand Medical Journal:

New laws introduced to curb alcohol harm have failed to make a dent on ED admissions, new research has found.

The study, released today, showed that around one in 14 ED attendances presented immediately after alcohol consumption or as a short-term effect of drinking and that rate had remained the same over a four-year period.

Here's the relevant study (sorry I don't see an ungated version, but there is a presentation on the key results available here), by Kate Ford (University of Otago) and co-authors. They looked at emergency department admissions at Christchurch Hospital over three-week periods in November/December 2013 and in November/December 2017, and found that:

...[t]he proportion of ED attendances that occurred immediately after alcohol consumption or as a direct short-term result of alcohol did not change significantly from 2013 to 2017, and was about 1 in 14 ED attendances overall.

The key reason for doing this research was that the bulk of the changes resulting from the Sale and Supply of Alcohol Act 2012 had been implemented in between the two data collection periods. The authors note that:

...[a] key part of the Act was a provision to allow territorial authorities to develop their own Local Alcohol Policies (LAPs). The Act was implemented in stages from December 2012 onwards and subsequently, many local authorities attempted to introduce LAPs in their jurisdictions.

However, here's the kicker:

In many cases these efforts met legal obstacles, particularly from the owners of supermarket chains and liquor stores... For example, a provisional LAP in Christchurch was developed in 2013 but by late 2017 it still had not been introduced. 12 This provisional LAP was finally put on hold in 2018... Similar problems have been encountered in other regions... 

If you are trying to test whether local alcohol policies have had any effect on alcohol-related harm, it's pretty difficult to do so if you're looking at a place where a local alcohol policy hasn't been implemented. Quite aside from the fact that there is no control group in this evaluation, and that the impact of the earthquakes makes Christchurch a special case over the time period in question, it would have been better to look at ED admissions in an area where a local alcohol policy has actually been implemented (although, there have been too few local authorities that have been successful in this). To be fair though, the authors are well aware of these issues, and make note of the latter two in the paper.

However, coming back to the point at hand, whether legislation is implemented as intended is a big issue in evaluating the impact of the legislation. A 2010 article by David Humphreys and Manuel Eisner (both Cambridge), published in the journal Criminology and Public Policy (sorry I don't see an ungated version), makes the case that:

Policy interventions, such as increased sanctions for drunk-driving offenses... often are fixed in the nature in which they are applied and in the coverage of their application... To use epidemiological terminology, these interventions are equivalent to the entire population receiving the same treatment in the same dose...

However, in other areas of prevention research... the onset of a policy does not necessarily equate to the effective implementation of evidence-based prevention initiatives...

This variation underlines a critical issue in the evaluation of effects of the [U.K.'s Licensing Act 2003]...

In other words, if you want to know the effect of legislative change on some outcome (e.g. the effect of alcohol licensing law changes on emergency department visits), you need to take account of whether the legislation was fully implemented in all places.

Read more:

• Public health vs. competition in Australian alcohol regulation

Do single-sex schools make girls more competitive?

It is often argued that single sex schools are good in the sense that they reduce gender gaps (see here for a rundown of recent evidence). This recent paper in the journal Economics Letters (ungated here) by Soohyung Lee (University of Maryland), Muriel Niederle (Stanford University), and Namwook Kang (Hoseo University, Korea) caught my attention because it looks at whether the gender gap in competitiveness is narrowed by single-sex schooling.

The general problem with trying to estimate the effects of single-sex schooling on any outcomes is that students (or rather, their parents) self-select into single-sex or coed schools. So, its not generally possible to separate the effect of single-sex schooling from the unobserved student or family characteristics that are related to the choice of school. On top of that, single sex schools in many countries (like New Zealand) are more likely to be private schools that can be more selective about the students they admit.

Lee et al. exploit a unique feature of the South Korean education system - that students are randomly assigned to middle schools. From the study:

The key challenges to estimating the effect of single-sex schooling are two-fold: first, coeducational and single-sex schools often have different qualities, and second, students often select which type of school they attend. We address these challenges by examining middle school students (grades 7 to 9) in Seoul, South Korea. This experimental group is well-suited for the purpose of our study because a student is randomly assigned to a single-sex or coeducational school within a school district and all school districts have both single-sex and coeducational schools...

Therefore,  we identify the causal effect of single-sex schooling on competitiveness by estimating simple regression models controlling for school-district fixed effects and individual characteristics.

Participants in the study were asked to solve as many simple addition problems as they could in three minutes. They could then choose to participate in a tournament where they would be paid only if they were the top performer in a randomly-selected group of four students. Those who are more competitive will more likely choose the tournament (the study also includes controls for risk aversion, and for students who want to avoid denying others the chance to win the tournament). The experiment is run twice - first at the beginning of the second term of the 2011-12 academic year (August 2011), and second near the end of the academic year (February 2012).

The authors find:

...girls are less likely than boys to choose tournament: 29.9% of boys select tournament in Task 3, while 22.3 girls do (p-value of testing no gender gap: 0.032). This difference remains even after we control for students’ characteristics.

The results contrast with earlier and widely cited work in the U.K. (earlier ungated version here) by Alison Booth and Patrick Nolen (both at the University of Essex and Australian National University). However, Booth and Nolen's sample were not randomised by school type.

There are a couple of reasons that make me worry about the robustness of the results in Lee et al.'s paper. First, it is essentially an impact evaluation - what is the impact of school type on competitiveness? Given that there are three variables of interest (gender, school type, and before/after), I would have expected them to use difference-in-difference-in-differences (aka DDD - see here for a quick, but somewhat technical, description of DDD). Their simple regression controls lacks the appropriate controls for the direct effect of gender (although this might have been included in student characteristics, which weren't reported), school type interacted with before/after (in case different schools have different general effects over time), gender interacted with before/after, and the triple-interaction (which is the variable of interest in DDD). While this doesn't necessarily invalidate their results, it would be interesting to see what their results look like in a DDD analysis.

Second, the timing of the two rounds of data collection is an issue. Given that the first round occurred after the students had already commenced middle school, the results likely underestimate any impact of single-sex schooling on competitiveness. So, demonstrating a statistically insignificant effect of single-sex schooling on narrowing the gender gap doesn't demonstrate that there is no effect, because perhaps most of the effect occurs in the first term of middle school. We don't know.

I have to agree with the authors when they conclude.:

...whether policies expanding single-sex schools will promote gender equality is a question that requires more thorough empirical investigation.

For me, this paper just doesn't answer the question on whether single-sex schooling narrows gender gaps or not.