Tag: academic publishing

Robin Hanson on MS Dhoni (well, kind of)

AshishLeave a comment

Does adding the phrase “well, kind of” make a clickbait-ish title less clickbait-ish? Asking for a friend.

My favorite book to read about cricket when I was growing up was a book called “Cricket Skills and Techniques: A Comprehensive Guide to Coaching and Playing“, written by Douglas Wright. The only reason it was my favorite is because that’s the only book on cricket coaching that we had at home. Don’t ask me how we ended up having that copy at home, because nobody at home played (or coached) cricket. Still, in those bad old days of no cable television, let alone the internet, I must have read that book dozens of times.

Written in 1971, it was as stodgy about the “how to play cricket” approach as it is possible to be. The whole “back and across”, bat coming down straight, “flick on the legside only when the ball is outside of the leg stump” approach. It’s been years, but I really do doubt if the word “lofted” was used even once in the entire book.

Think of it this way: if Suryakumar Yadav were to occupy one end of the spectrum, the batting techniques in this book would lie at the other.

Would somebody steeped in thinking about batting like this ever have selected MS Dhoni to be a batsman for India, let alone the captain of the Indian Test Cricket team?

Robin Hanson says that academia suffers from the same problem:

I conclude that each typical academic journal not only sees itself as covering a limited topic area, it also sees itself as being willing to consider only a limited range of concepts and argument types. Furthermore, these limits are quite strongly selective; the vast majority of statements that ordinary people actually generate on such topics, even when they are trying to talk seriously, are seen as unsuitable.

https://www.overcomingbias.com/p/why-not-heterodox-research

There is, in other words, A Correct Way to do research. Just like there is A Correct Way to hold the bat, and A Correct Way to hit the ball. And deviations from ACW are not to be acknowledged, let alone tolerated.

And the point that Robin Hanson is making is that if an academic journal be thought of as the Indian cricket team, and a paper within it as MS Dhoni, well, that paper would never have been published – let alone be thought of as one of the best papers ever to have been published.

And the question that needs to be asked, of course, is how many Dhonis have been left out in the cold when it comes to academic research?

Hanson’s blog post goes a bit deeper, and asks why this should be so. Why are the gatekeepers in academia so very khadoos? Because, he posits, they think it to be the rational approach. They claim, he says, that:

To make progress on our topics, our discipline’s concepts and methods are quite sufficient. Sure others might in principle use other concepts and methods to draw relevant conclusions on our topics more easily than do we, but the chance of that usually seems so low that we just habitually ignore all purported candidates of this sort. There are just not usually clues that could plausibly indicate such a scenario well enough to get us to consider including heterodox articles in our journals, or to consider citing them in our articles. The enormous costs to us of evaluating the quality of such heterodox contributions completely swamps any value they might have to offer.

https://www.overcomingbias.com/p/why-not-heterodox-research

You might miss out on the odd Dhoni if you are a dyed-in-the-wool traditionalist, in other words, but that’s fine, because you will have to evaluate a million Dhoni look-alikes who simply aren’t anywhere near as good. But just looking at the few who are the best and in the traditional mould is enough to find the absolute best in the country. So why bother with all those ugly hoikers of the ball? Stick to what we know!

This “leaves money on the table”, of course. Many unorthodox cricketers (academic papers) could be unearthed if only we broadened our methods of evaluation a little bit.

For me this issue highlights the great potential of innovations in how we evaluate contributions. Today, a reviewer typically takes an hour or two to review a dense 4-8K word research paper, where anyone in a discipline needs to be qualified to evaluate any article in that discipline (again, really sub-discipline). In this case, yes, everyone in a discipline must know well the same concepts and methods, and so each discipline can’t accept many such concepts and methods.
But, it should be possible to instead have different people review different aspects of a paper, so that the concepts and methods of a paper don’t have to be limited to just one sub-discipline. Some academic reviewers could specialize in evaluating the concepts and methods of ordinary conversation, to make those available to paper authors. And it should be possible to get quick less-expert less-formal evaluations from betting markets, with bettor incentives tied to much-rarer more expert and expensive evaluations. Using such methods, academic journals should be able to consider submissions using a much wider range of concepts and methods.

https://www.overcomingbias.com/p/why-not-heterodox-research

And well, we did broaden our methods of evaluation in cricket! We did find Suryakumar Yadav, and dozens, if not hundreds, of wonderful (and unconventional) cricketers. How did we do this? As Amit Varma has pointed out so many times, by getting the correct incentives in place – by having a tournament called the IPL, in other words. Scouts for teams in the IPL are looking for people who will get the results that are needed, techniques be damned (and yes, I know Dhoni made his debut three years before 2008. The point still stands, I’d argue).

How should we get the Indian Publishing League going? Robin Hanson (and I!) would like to know:

Note, however, that such innovations have long been possible, and I have personally seen such proposals enthusiastically rejected. Turns out disciplinary authorities who have risen to the top of their fields via their mastery and control over acceptable concepts and methods may not be eager to invite competition for their prestigious positions from a wider range of people using strange concepts and methods. And as long as no parties near the academic world are able to defy the power of their prestige, this is how things will remain.

https://www.overcomingbias.com/p/why-not-heterodox-research

Elementary, My Dear Excel

AshishLeave a comment

This broke my heart:

But some researchers are calling Ariely’s large body of work into question after a 17 August blog post revealed that fabricated data underlie part of a high-profile 2012 paper about dishonesty that he co-wrote. None of the five study authors disputes that fabrication occurred, but Ariely’s colleagues have washed their hands of responsibility for it. Ariely acknowledges that only he had handled the earliest known version of the data file, which contained the fabrications.
Ariely emphatically denies making up the data, however, and says he quickly brought the matter to the attention of Duke’s Office of Scientific Integrity. (The university declined to say whether it is investigating Ariely.) The data were collected by an insurance company, Ariely says, but he no longer has records of interactions with it that could reveal where things went awry. “I wish I had a good story,” Ariely told Science. “And I just don’t.”

https://www.sciencemag.org/news/2021/08/fraudulent-data-set-raise-questions-about-superstar-honesty-researcher

I’ve been recommending Dan Ariely’s books and talks to students for years now, and with good reason. But whether he himself was responsible for this, or not, it is certainly the case that a thorough investigation is warranted, both of this specific paper, but also of his entire body of work.

But the point of this post isn’t to just point out this rather depressing fact. The blogpost that broke the story is worth reading in full for the following reasons:

  1. The admirable clarity in how it is written. Anybody who knows the very basics of math and statistics (and I do mean the very basics) will be able to understand what is going on.
  2. You don’t need to know any coding to figure out how they uncovered the fraud. Simple Excel is enough.
  3. The researchers have provided the data for you to play along with as you read the blogpost.

So if you are a student of statistics (and that is all of us, like it or not), I’d strongly encourage you to set aside a couple of hours, and work your way through the post and the Excel file(s).

And finally, a word of advice if you are a student who is just about beginning to play around with data:

  1. Don’t commit fraud. It sounds stupid, almost, to dispense this advice, but please, resist the temptation.
  2. Double check data that has been sent to you by somebody else. Triple check it! And checking means running sanity checks. There is still a chance that you will not be able to detect fraud, if it has been committed, but minimize the chances. Get better at asking questions of the data you are working with!
  3. Stuff like this is, trust me on this, the best way to learn statistics. No amount of end-of-chapter problem solving will help you get your basics clear like a statistical whodunnit. Or a what-was-done, as in this case.

A lengthy excerpt, but a necessary one. What follows are the last three paragraphs of the blogpost that broke this story:

We have worked on enough fraud cases in the last decade to know that scientific fraud is more common than is convenient to believe, and that it does not happen only on the periphery of science. Addressing the problem of scientific fraud should not be left to a few anonymous (and fed up and frightened) whistleblowers and some (fed up and frightened) bloggers to root out. The consequences of fraud are experienced collectively, so eliminating it should be a collective endeavor. What can everyone do?
There will never be a perfect solution, but there is an obvious step to take: Data should be posted. The fabrication in this paper was discovered because the data were posted. If more data were posted, fraud would be easier to catch. And if fraud is easier to catch, some potential fraudsters may be more reluctant to do it. Other disciplines are already doing this. For example, many top economics journals require authors to post their raw data [16]. There is really no excuse. All of our journals should require data posting.
Until that day comes, all of us have a role to play. As authors (and co-authors), we should always make all of our data publicly available. And as editors and reviewers, we can ask for data during the review process, or turn down requests to review papers that do not make their data available. A field that ignores the problem of fraud, or pretends that it does not exist, risks losing its credibility. And deservedly so.

https://datacolada.org/98

If you’re writing a paper, put your data up for public scrutiny. Always, and without fail. It matters.