Bug Bounty Program

Terms for my scientific bug bounty program.

Figure 1: Bug death duster. Figure from the Internet Archive Book Images

See also my list of mistakes.

To find errors in my work, I try to make my analyses reproducible, and rely on a few best practices borrowed from the software industry, such as version control, code review, testing, and sanity checks. Still, given that I do not always have co-authors who have the time and ability to review my code, I am looking for other ways to find and correct errors in my work.

Figure 2: Get him. Figure from the Internet Archive Book Images

Putting up an anonymous submission form only yielded one measly submission (although a good one, which led to an erratum in Proceedings B, because it turned out the journal published a later version than the one I quoted and commented on).

Stepping on people’s toes with my work yielded more, but it’s not a practice I want to institutionalise.

Bug bounties

So, I’m borrowing another practice from software: a bug bounty program.¹ I’m slightly less well funded than Google and their ilk, but the Free Knowledge Fellow program by Wikimedia and the Stifterverband endowed me with some money to use for open science projects and this is how I choose to spend half of it.

So, what does this apply to, and what can you earn?

Criteria

• I’m first author or the other authors ask me to include it in the bug bounty program.²
• It’s an empirical manuscript³
  • preprints count for the full amount (until they are accepted)
  • published papers count for 3/4 the amount
  • blog posts count for 1/4 the amount unless they are tagged quick job⁴
• It was published after October 2, 2018.⁵

The different factors for preprints, publications, and blogs reflect the amount of effort I already put in them and how easy it will be for me to correct something at that point. So, because I would love to hear about errors before the publication processes “enshrines” the paper⁶, errors in preprints are worth more than those in published paper.

Budget

For now, I’m setting aside 2500€. When this runs out, I’ll have to a) improve my error checking workflow, b) get another stipend, award, donation, or tenured position, so I can afford continuing to make errors. I’ll keep this page updated, so should my budget have run out, you won’t waste too much time.

Within at most two months of the bug being reported, I’ll pay out the sum via Paypal or bank transfer or argue my case why it was not a bug. In severe cases, or if I’m ill, or out of work, it might take me longer, in these cases I will post a note quoting the bug report and my intention to respond. In all cases, I will post a note on this page, in more severe cases, I will start the process of submitting a correction, or a retraction. I cannot give a timeline on these because of co-authors.

If you report a bug anonymously, I will donate the fee to Against Malaria via the German Equivalent of Give Well. You can also choose a different charity on the GiveWell list. You can also choose to donate if you do not report anonymously.⁷

Bounties

Item	Bounty
Typos	A beer8 if we meet.
Omitted citations	A beer9 if we meet.
Lacking reproducibility10	10€
Coding error that changes a number reported in the supplement	10€
Lacking robustness11	20€
Coding error that changes a number reported in the manuscript, but not a conclusion based on the number	50€
A better way to statistically model or test something that leads to a different conclusion	100€
Coding error that changes a conclusion in the manuscript	200€
Coding error that changes the main conclusion in the abstract	400€

I will count the errors (by type, not by repetitions), not the numbers, or conclusions changed.

Please note that these bounties do not necessarily reflect only the gravity of a bug (i.e., missing a citation can be pretty bad), but also my perception which of these are actually determinable objectively. So, by reducing the sum for omitting citations, I hope to forestall drawn-out arguments by reducing both sides’ temptation to bargain.

Any one person can claim at most 5 items, so up to 760€. If multiple people notice the same error, second and third reporters only get a beer, and my heartfelt gratitude.

This policy could change anytime without notice.

Please help!

Figure 3: Help me master my bugs. Figure from the Internet Archive Book Images

Acknowledgements

Thanks to Wikimedia Germany and the Stifterverband for funding me through their fellowship “Free Knowledge” and supporting this program. Thanks to Daniel Mietchen, Nicolas Schmelling, and Benjamin Paffhausen with whom I hatched the precursor to this plan.

---

1. The computer scientist Donald Knuth has a similar program in place that I did not know about until I mentioned this idea to others and I also drew some inspiration from gwern’s Mistakes page.↩︎

2. I don’t want to scare people off of collaborating with me, so I have resolved not to badger anyone into this.↩︎

3. I know for a fact that there are bugs in my software, but users already tell me about those. Also, it’s open source and can be updated continuously, so just create an issue or send a pull request.↩︎

4. For these, payouts are capped at one beer.↩︎

5. I am still grateful for error reports for older manuscripts, but I precommitted not to pay for them, so as not to inhibit responses via my Tell me I’m wrong form. I will buy you a beer though.↩︎

6. I’m obviously not a fan of enshrining papers, but think we currently lack a good mechanism of correcting papers citing an invalidated claim, see my blog post, so it’s hard to fix on our own.↩︎

7. I will not donate more to reflect the amount I would get back in taxes if I donate, because I cannot be bothered to do the calculations.↩︎

8. Capped at 3, I don’t want you to get hungover.↩︎

9. Capped at 3, I don’t want you to get hungover.↩︎

10. e.g., I forgot to list a necessary package, code doesn’t run without fiddling with paths.↩︎

11. e.g., reasonable covariates that I left out would substantially alter estimates↩︎