A Study Says Higher Temperatures Mean Worse Grades: Why Care About Research With an Obvious Conclusion, and What to Care About

Good research can carry strong “policy implications” — and what it means to say correlation isn't causation

A new study looks at the relationship between temperature and academic performance, and finds a significant negative correlation between the two.

Specifically, the researchers analyzed 13 years of exam scores from 10 million American middle and high school students and found that rising temperatures push scores down; the effect of heat is larger on students from low-income and minority families.

They also found that using air conditioning can almost entirely cancel out that effect.

These conclusions sound like things we'd know without any research at all. Let's start by reading the paper itself.

The Heat and Learning Study

The study, Heat and Learning, is an NBER working paper written by researchers at Harvard, UCLA, Georgia State University and the College Board (the College Entrance Examination Board).

NBER, the National Bureau of Economic Research, was founded in 1920. Most economics researchers know it well, and a good number of Nobel laureates in economics are members.

NBER working papers rank first on both the h5-index and the h5-median (think of them, roughly, as measures of citation count) in Google Scholar's business, economics and management category (as of this writing), ahead of famous economics journals such as the American Economic Review and the Quarterly Journal of Economics. But note that NBER working papers have not been through peer review — a working paper, as the name says, has not been formally published, and you have to judge the quality of the piece yourself (in theory that is true of any journal article), though most NBER working papers do end up published in journals.

The researchers claim theirs is the first study to offer evidence this clear that rising temperatures cause academic performance to fall.

They analyzed the scores of 10 million American middle and high school students who took the PSAT between 2001 and 2014, local temperature data (1996-2014), and school air-conditioning installation and use (the AC rate) collected through surveys, and drew a series of conclusions. The PSAT is a practice SAT, roughly the equivalent of a mock exam before China's gaokao; it is taken mainly by 10th and 11th graders, comparable to first- and second-year high school students in China.

The data cover a wide geographic range, including parts of the U.S. with very different climates and temperatures. The study found that the hotter it was at school during the year before the exam, the worse students scored. That held in the north and the south alike, in states that are colder on average and states that are hotter.

Specifically, the study found that, on average, every 1°F rise in temperature during school hours (about 0.56°C) lowers scores by 1%. The effect turns significant once the temperature reaches 70°F (21°C) — run the data yourself and you'll see the little asterisks (**) — and it jumps sharply at 90°F (32°C) and above. In short, the effect is not linear: the hotter it gets, the more scores suffer, and the more extreme the heat, the greater the damage.

Why?

The researchers think the likeliest reason is that heat cuts the efficiency of learning during instructional time (while students are at school). Too hot to take anything in — at least in developed economies (for the sake of academic rigor: the sample is only the U.S., after all). This is not a guess out of thin air, even if it sits close to common sense. Compared with other variables, they found that, on average, only the temperature during school hours — more precisely, “heat exposure” — affects academic performance (Effect 1). The effect of temperature during summer vacation and on weekends (Effect 2) is very small, and controlling for that portion of heat exposure does not change the size of Effect 1.

Controlling for local economic conditions (Effect 3) does not change the size of Effect 1 either, which rules out economic conditions as a mediating variable — the possibility that it is not heat lowering scores but heat-related economic conditions acting on families that hurt performance. The same logic rules out air pollution (Effect 4). Ruling out these possibilities — not that you can ever rule out all of them — lends further support to the inference that heat affects how efficiently students learn.

Why It's Worth Caring About

The size of the effect, though, varies from student to student. The study found that Black and Hispanic American students (minority students, for short) are hit three times as hard by heat as white students, and that students living in low-income areas are hit two to three times as hard as students from high-income areas.

Plenty of other factors could produce that pattern — richer students can pay for tutoring, for one — but a “simpler explanation” is that low-income students, at home and at school alike, are less likely to have air conditioning.

Racial gaps in educational attainment in the U.S. are a closely watched subject, and what people want to know is which factors create them.

The study argues that temperature is one of them. Analysis of geographic data (zip codes) shows that minority students experience temperatures 5°F (2.78°C) higher on average than white students do; and for every 1°F rise in temperature, the drop in scores is 0.002 standard deviations larger for minority students than for white students on average. Multiply the two and the gap is 0.01 standard deviations. Students have roughly 10 years of schooling before they sit the PSAT, and if these effects accumulate, the gap becomes 0.1 standard deviations — that is, the temperature effect on minority students is 0.1 standard deviations larger.

There is no detailed explanation for why the temperatures minority students experience are 5°F higher on average. But we can look at two other numbers: the survey shows that schools with mostly minority students have an air-conditioning rate 3.5% lower than mostly white schools, a gap that does not vary by region; low-income areas are 5% lower than high-income areas, a bigger gap than the one between cold and hot regions.

Now look at the outcome — take PSAT scores as a stand-in for educational attainment, and the data show minority students scoring 0.8 standard deviations below white students on average. Which means that temperature alone accounts for between 1.25% (0.01 standard deviations) and 12.5% (0.1 standard deviations) of the racial gap in American educational attainment.

This is a form of “policy simulation” — dig a little beneath an obvious conclusion and you often find something that makes you stop and think, which is why research like this is sometimes worth caring about. As for the specific measures worth considering — call them “policy implications” — we'll come to those later.

On the other side, the study found that installing and using air conditioning (the AC rate) can “almost entirely” cancel out the effect of heat on scores. The most basic model shows that at schools without air conditioning, a 1°F rise in temperature lowers scores by 0.0032 standard deviations; the interaction-term coefficient tells you that at schools with air conditioning the effect is 0.0025 standard deviations smaller — an offset of 73%, which is what they mean by “almost entirely.” No need to nitpick, though: several models that add more interaction terms and other independent variables point the same way as this one — “more air conditioning, less impact (of heat on scores)” — differing only in degree.

That still isn't convincing enough. The more important step is that they put a “change in school air conditioning” variable on the right side of the equation, in effect looking at schools that added air conditioning and comparing them with the same schools 10 years earlier without it (using only two points in time, 2016 and 2006), to see whether scores went up — that is, whether the “offsetting effect” is real. It is, and the conclusion survives several other factors that might affect the AC rate. In other words, all else equal, simply because a school installed air conditioning, heat does less damage to its students' scores.

That means changes in the AC rate are more likely to be exogenous rather than a product of cross-sectional variation itself. This matters a lot (though the small number of time points in the data is a weakness); studies that come off as muddled often handle endogeneity badly. With the support of the two points above, the paper concludes that air conditioning can substantially — or at least “partly substantially” — reduce the harm heat does.

What to Watch For When You Care

So why say all this? Isn't it enough to know that heat hurts grades and that air conditioning helps? Why not go one step further and just tell the authorities to put more air conditioning in schools? Isn't that the “policy implication” the subtitle is talking about?

Before you interpret what a conclusion implies, let alone make policy recommendations out of it — worth noting: one of the five principles NBER set for itself at its founding is to deliberately avoid giving policy advice — you have to be careful about how the study argued its way to that conclusion. That is why we've spent so much space picking out the key steps it takes to get from a correlation between heat and worse scores to heat (heat with a long list of qualifying conditions) causing worse scores. It also partly explains all the concessions and qualifying clauses in the paper.

Going from research to recommendation, two things deserve attention: 1) correlation is not causation or causality; 2) when you generalize a conclusion, mind how the mechanism actually works.

On the first point: “higher temperature, worse grades” can be read two ways, taken literally — temperature and scores are negatively correlated, or high temperatures cause worse scores (set aside for now whether directly or indirectly). In practice the first reading is easy to reach; it's only a question of what standard you use. The second is very hard. Generally speaking, establishing causality is somewhat easier than establishing causation, since the former does not require a direct cause-and-effect link.

The difficulty is sharpest in the social sciences, which rely mainly on empirical research. The gold standard for pinning down cause and effect is the randomized controlled trial (RCT) or the double-blind experiment, common in medicine. But a great deal of empirical research analyzes data that already existed before the research began. Society is not a laboratory. In a laboratory the variables are relatively controllable, and you don't have to do a mountain of later work to rule out confounders, or to sort out how they relate to the dependent variable you care about (academic performance, say) or the independent variable (temperature).

This deserves extra attention in today's “big data era,” when data is being collected on a scale and by methods never seen before — which seems to create the illusion that the truth is within reach, that correlation has become easier to come by. It does not mean that inferring causation has become any easier. Quite the opposite: inferring causation matters more than ever.

Mistaking correlation for causation is a common attribution error. As we noted in another article, research says smart people are 30% more likely to wear glasses (more precisely, to have worse eyesight), but wearing glasses will not make you smarter. The stereotype born of that mistake does turn out to be “rather useful,” though — some research shows that people still tend to think you're smarter when you wear glasses.

On the second point: most social-science research studies a very small corner of human society, but a study usually doesn't mean to stop at explaining one specific case — it is after some general explanation. That is nearly impossible, so what you see more often is a study's conclusion stretched into wider “application scenarios.” Stretching comes with conditions: if you can spell out exactly how the cause produces the effect — if the mechanism is clear and the process can be broken into parts — generalizing gets easier.

One reason this study supports a causal link between heat and worse scores is that the authors compared the effect on scores of temperature during school hours with temperature on weekends and holidays, and found the latter tiny (see the end of the first section). That is what gives real force to the mechanism they propose: heat lowers learning efficiency, which in turn lowers scores.

So if you're going to cite this study when you ask your school to put air conditioning in the dorms, remember to stress that the time you spend studying in your dorm has a significant effect on your grades overall — or, to put it another way (even though the meaning has shifted), that a good share of your dorm time goes to studying. That way your extension of the study's conclusion will stand on firmer ground. This too is “policy simulation” in practice.

Policy Implications

With all that in mind, back to policy implications.

NBER strives for neutrality and avoids policy advice, but policymakers are plainly among its readers.

In this study, the authors weighed the benefit of installing air conditioning (better academic performance) against the cost of installing and running it, and concluded it is worth doing. They also looked further out: as the planet warms, temperature's overall effect on academic performance will grow. By their calculations, installing and using air conditioning pays off more for minority and low-income students.

At this point the finding at the top of the second section — that different students are affected to different degrees — acquires a real policy implication. What's left is deciding (on the basis of more varied research) exactly what to do: whether to increase air-conditioning subsidies for schools that mostly serve these students, whether to keep improving the composition of student bodies, and so on — none of which the paper itself proposes.

The earlier section on racial gaps in educational attainment (the second section) also opens up further subjects. Take the 10-year cumulative effect of heat on minority students. Affirmative action (AA), practiced in the U.S. for more than 40 years, is much argued over in education; beyond the charge of “reverse discrimination,” there is also a live debate about which stage of education it should be applied at. Not only in the U.S.: Oxford's attempt late last year — extending the math exam by 15 minutes, which did little for female students' scores — was seen as an act of desperation. Perhaps the affirmative-action efforts more worth making are in middle and elementary school, before university. The reason being that 1.25%-12.5% cumulative effect.

Good research often carries policy implications.

Another example: in an article of ours about migrant children going to school, we mentioned that two researchers at Shanghai University of Finance and Economics argued in a 2014 paper, 《学校类型与流动儿童的教育》 (“School Type and the Education of Migrant Children”), that school type is a major cause of migrant children's poor grades. After controlling for family and individual factors and using instrumental variables to handle the endogeneity of school choice (there's that word “endogeneity” again), they found that school type matters more for students with worse grades. They also noted that Shanghai's restructuring of primary schools for migrant workers' children in recent years — merely changing the schools' type, as the report above describes — has significantly narrowed the gap between students with different hukou (household registration) status. It appears to have been changed back since.

The paper ran in China Economic Quarterly, published by Peking University's China Center for Economic Research (CCER, also known as the National School of Development). CCER and NBER have jointly held an annual China economics conference every year since 1998.

Generally speaking, though, policymakers aren't much interested in the specific mechanism. Whether that is a good thing or a bad thing, who knows.

Cover image / Giphy