The subject of repeatability of Computer Science papers comes up every so often. It’s such a problem that it’s being actively studied by at least one dedicated project.
Most recently, Jean Yang of CMU blogged trying to address Mike Hoye’s Twitter “rant”, see above. Let’s quickly sum up their points. Mike says:
- Mozilla has encountered, often enough to complain about it publicly, the problem of trying to repeat an experiment or verify a paper’s claims over new data
- The lack of shared information (be it code, or papers behind paywalls, etc) obstructs Mozilla from attaining the speed they could have
- Just as we lost priceless knowledge captured in artifacts we could not reproduce (my note: or straight up vanished in, say, burning libraries), we are going to lose the knowledge inside those CS papers
In response, Jean points out:
- The data shows the claim “CS researchers don’t publish data” is false. According to 1, more than 50% of researchers provide code directly or upon request
- Conferences are pushing for standardisation of the sharing process and evaluating the repeatability of results themselves. Therefore the problem is acknowledged, and being dealt with
- Academia’s job is not to produce “shippable products”, and what will be produced is the subject of discussions, and takes (a lot of) time
Now, it’s very important we agree there is a problem. Jean acknowledges it, and the Twitter heat revolves around how it’s not nice to use hyperbole to point out the existing problem, and not on denying the problem. So far, so good.
It’s equally important, however, to not underestimate the problem. There are two key points here:
-
Even if it is important to identify the researchers or institutes that adopt responsible practices (or not), it is only part of the picture
-
The same data Jean points to show the problem is huge. Even avoiding the temptation to criticise the exclusion process, we see 43% (176 out of 402) of surveyed papers do not supply repeatable results
Point #2 does not leave any wiggle room: Mike’s criticism is right on point, and Mozilla’s frustration with the availability of verifiable results is completely valid. 50% is not a good target for numbers of repeatable papers. That 50% leaves outside another half which is problematic, and we need to do better. Science is a systematic endeavour, which is firmly based on reproducibility. If a system is not backed by code, it’s backed by math, and since we provide the math, why not the code?
The criticisms, which also appear in the UoA study, about the state of shared code and its quality are not valid any more. It is about matters which are resolved trivially: hire a (team of) software engineer(s) to help you with them. That’s it. It really is a solved problem, since the industry has been putting out working code and products for years. Use their expertise.
The criticisms about IP and other concerns which inhibit sharing, like paywalled research, are a bigger problem which I would not dare go into in text. Suffice to say, as an example, anybody should find it ludicrous that the IEEE (still) attempts to profit from my colleagues’ and my ten-year-old research, when the same research is offered for free. No, of course I won’t link to their library, are you nuts?
Point #1 is about leaving parts of the problem outside the discussion. There’s already a vast amount of unreproducible claims out there, this much is certain. This phenomenon needs to stop as soon as possible. The studies point out multiple problems, and these problems cannot possibly lie exclusively with our processes or interpersonal relationships, but are deeply ethical and possibly in danger of becoming systemic. That is, if they’re not systemic already, which some would argue clearly is the case. Transparency about research should not be negotiable. Otherwise, we’re led down the path of conspiracies about bubbles, formed by paywalls and academic cliques, grown by misappropriation of hunting for government funding and perpetuated by nepotistic behaviours. Parts of those hyperbolic claims are slowly and surely becoming the norm, and it’s really baffling that everybody seems to abhor them in public discussion, but continue to feed them anyway.
« Past Future »