How to be casual about things

[dieseltaylor]

From the BBC

Reinhart, Rogoff... and Herndon: The student who caught out the profs

By Ruth Alexander BBC News This week, economists have been astonished to find that a famous academic paper often used to make the case for austerity cuts contains major errors. Another surprise is that the mistakes, by two eminent Harvard professors, were spotted by a student doing his homework.

It's 4 January 2010, the Marriott Hotel in Atlanta. At the annual meeting of the American Economic Association, Professor Carmen Reinhart and the former chief economist of the International Monetary Fund, Ken Rogoff, are presenting a research paper called Growth in a Time of Debt.

At a time of economic crisis, their finding resonates - economic growth slows dramatically when the size of a country's debt rises above 90% of Gross Domestic Product, the overall size of the economy.

Word about this paper spread. Policymakers wanted to know more.

And so did student Thomas Herndon. His professors at the University of Massachusetts Amherst had set his graduate class an assignment - pick an economics paper and see if you can replicate the results. It's a good exercise for aspiring researchers.

Thomas chose Growth in a Time of Debt. It was getting a lot of attention, but intuitively, he says, he was dubious about its findings.

Some key figures tackling the global recession found this paper a useful addition to the debate at the heart of which is this key question: is it best to let debt increase in the hope of stimulating economic growth to get out of the slump, or is it better to cut spending and raise taxes aggressively to get public debt under control?

EU commissioner Olli Rehn and influential US Republican politician Paul Ryan have both quoted a 90% debt-to-GDP limit to support their austerity strategies.

But while US politicians were arguing over whether to inject more stimulus into the economy, the euro was creaking under the strain of forced austerity, and a new coalition government in the UK was promising to raise taxes and cut spending to get the economy under control - Thomas Herndon's homework assignment wasn't going well.

No matter how he tried, he just couldn't replicate Reinhart and Rogoff's results.

"My heart sank," he says. "I thought I had likely made a gross error. Because I'm a student the odds were I'd made the mistake, not the well-known Harvard professors."

His professors were also sure he must be doing something wrong.

"I remember I had a meeting with my professor, Michael Ash, where he basically said, 'Come on, Tom, this isn't too hard - you just gotta go sort this out.'"

So Herndon checked his work, and checked again.

By the end of the semester, when he still hadn't cracked the puzzle, his supervisors realised something was up.

"We had this puzzle that we were unable to replicate the results that Reinhart-Rogoff published," Prof Ash, says. "And that really got under our skin. That was really a mystery for us."

So Ash and his colleague Prof Robert Pollin encouraged Herndon to continue the project and to write to the Harvard professors. After some correspondence, Reinhart and Rogoff provided Thomas with the actual working spreadsheet they'd used to obtain their results.

"Everyone says seeing is believing, but I almost didn't believe my eyes," he says.

Thomas called his girlfriend over to check his eyes weren't deceiving him.

But no, he was correct - he'd spotted a basic error in the spreadsheet. The Harvard professors had accidentally only included 15 of the 20 countries under analysis in their key calculation (of average GDP growth in countries with high public debt).

Australia, Austria, Belgium, Canada and Denmark were missing.

Oops.

Herndon and his professors found other issues with Growth in a Time of Debt, which had an even bigger impact on the famous result. The first was the fact that for some countries, some data was missing altogether.

Reinhart and Rogoff say that they were assembling the data series bit by bit, and at the time they presented the paper for the American Economic Association conference, good quality data on post-war Canada, Australia and New Zealand simply weren't available. Nevertheless, the omission made a substantial difference.

Thomas and his supervisors also didn't like the way that Reinhart and Rogoff averaged their data. They say one bad year for a small country like New Zealand, was blown out of proportion because it was given the same weight as, for example, the UK's nearly 20 years with high public debt.

"New Zealand's single year, 1951, at -8% growth is held up with the same weight as Britain's nearly 20 years in the high public debt category at 2.5% growth," Michael Ash says.

"I think that's a mistaken way to examine these data."

There's no black and white here, because there are also downsides to the obvious alternatives. But still, it's controversial and it, too, made a big difference.

All these results were published by Thomas Herndon and his professors on 15 April, as a draft working paper. They find that high levels of debt are still correlated with lower growth - but the most spectacular results from the Reinhart and Rogoff paper disappear. High debt is correlated with somewhat lower growth, but the relationship is much gentler and there are lots of exceptions to the rule.

Reinhart and Rogoff weren't available to be interviewed, but they did provide the BBC with a statement.

In it, they said: "We are grateful to Herndon et al. for the careful attention to our original Growth in a Time of Debt AER paper and for pointing out an important correction to Figure 2 of that paper. It is sobering that such an error slipped into one of our papers despite our best efforts to be consistently careful. We will redouble our efforts to avoid such errors in the future. We do not, however, believe this regrettable slip affects in any significant way the central message of the paper or that in our subsequent work."

Accidents do happen, and science progresses through the identification of previous mistakes. But was this a particularly expensive mistake?

"I don't think jobs were destroyed because of this but it provides an intellectual rationalisation for things that affect how people think about the world," says Daniel Hamermesh, professor of economics at Royal Holloway, University of London.

"And how people think about the world, especially politicians, eventually affects how the world works."

Discovering a spreadsheet error was never going to end the debate over austerity - and nor should it, according to Megan McArdle, special correspondent for Newsweek and The Daily Beast.

"There is other research showing that you can have these slowdowns when you get to high levels of debt," she says. "We have a very vivid [example] in Greece."

Thomas Herndon 's view is that austerity policies are counter-productive. But right now he's delighted that the first academic paper he's ever published has made such a splash.

"I feel really honoured to have made a contribution to the policy discussion," he says.

]

It is really rather shaking that so much rests on such poorly constructed spreadsheets. I am remingded that 50% of the most quoted medical research is actually bogus or suspect. Checking up seems to be an inadequately rewarded whereas those who make mistakes or lie seem to get away with it.

[costard]

We're probably still better off with 50% bad results than with 85% bad results. Depending on the level of affect of the decisions made with bad results backing the process. It sorts itself out, eventually, one way or another.

Proof by inspection is slow, but it happens.

[dieseltaylor]

Monmouth University Chemistry Professor Tsanangurayi Tongesayi and his students, Lauren Lechner, Patrick Fedick, Christiana Brock, Arielle Le Beau and Chelsea Bray, have found that rice imported from certain countries contains high levels of lead that could pose health risks, particularly for infants and children, who are especially sensitive to lead's effects, and adults of Asian heritage who consume large amounts of rice.

Their research, which found some of the highest lead levels in baby food, was among almost 12,000 reports presented at the 245th National Meeting & Exposition of the American Chemical Society, the world's largest scientific society. Their research was also selected as the topic for a press release and a press conference and was streamed live at www.ustream.tv/channel/acslive on April 8 at 2:30 pm. The paper titled "Safety and ethics in food production and distribution: A case for lead in rice" based on the research has been accepted for publication in the JOURNAL OF ENVIRONMENTAL SCIENCE AND HEALTH, PART B: Pesticides, Food Contaminants, and Agricultural Waste.

http://www.monmouth.edu/newsdisplay.aspx?id=29695

BUT BUT BUT ......

(NaturalNews) A recent scientific paper that concluded imported rice was heavily contaminated with lead has been suddenly withdrawn by its author. Natural News has confirmed from the author, Monmouth University Chemistry Professor Tsanangurayi Tongesayi, that the paper is "recalled until further notice."

The paper, announced with great fanfare at the American Chemical Society last week, was also accepted for publication in the Journal of Environmental Science and Health, Part B: Pesticides, Food Contaminants, and Agricultural Waste. It found that some sources of rice, including rice from China, were contaminated with as much as 12,000% more lead than allowed under current safety limits for children.

http://www.naturalnews.com/039998_imported_rice_lead_contamination_retraction.html

Quite an interesting long article on how to go wrong, the lack of checking, and the difficulties of analysis when done properly.

[Michael Emrys]

They might have been right the first time. There has also recently been a flurry of apprehension about levels of arsenic in rice, another heavy metal.

http://www.consumerreports.org/cro/magazine/2012/11/arsenic-in-your-food/index.htm

Michael

[JonS]

This reminds me of the Regnerus study.

The problem isn't the bad, mistaken, or flawed studies, per se. People make mistakes, honest or otherwise, and to expect perfect answers and results straight out of the blocks badly misunderstands the process of developing knowledge. The problem is that people continue to use and bad, mistaken, or flawed studies long after they've been shown to be bad, mistaken, or flawed, wich leads to the conversation going around and around in circles. We've see exactly the same behaviour here often enough with Kettler and hi lunacy - he'll be shown exactly why his belief is wrong and his sources are terrible, go quite for a while, then pop up again in 6 or 12 months spouting exactly the same nonsense.

You can see the same thing in the above rice example - the paper is pulled, but the news report about that repeats the main findings of the study as if they're still valid, when even a moment's thought would lead to the conclusion that the paper has been pulled because the conclusions are wrong! I have little doubt that Reinhart & Rogoff's paper, or the Regnerus study, will continue to be favourably cited for many many years to come. That is a problem.

[Kanonier Reichmann]

Or the allegedly solid sources of information about how Iraq had mobile chemical weapons laboratories which was used as one of the pretexts to invade that country when the 2 individuals on whom the intelligence was based were considered "fabricators" by both French & German intelligence and this was passed on to both British & U.S. intelligence counterparts. Unfortunately, the seed had been sown previously by these sources and MI6 & the CIA ran with it regardless of the major concerns over the credibility of the individuals concerned.

I'm sure there are many, many people who still believe that the intelligence gathered to justify the war was all kosher and indisputible despite the evidence at the time & subsequent to the events that was completely to the contrary.

Regards

KR

[dieseltaylor]

MI6 - more let politicians distort and pressure to fabricate a case.

http://www.dailymail.co.uk/news/article-2025512/MI6-pressure-Blair-Campbell-produced-bogus-Iraq-war-evidence.html

Its a laugh who gets considered a war criminal. Blair comes out a millionnaire.

[dieseltaylor]

Mistakes happen I will subscribe to but what is missing is confirmed checking by an independent. It used to be that before being published peer review was required. It seems that even this elementary precaution was not taken AND the samples had not been double-checked prior to the paper being sent in.

If we are going to live in an age of people rushing to publication then surely it is better for society that we put the onus back on the researchers to go through some mandatory checking or face the consequences.