WP Tavern’s Justin Tadlock Won’t Address Lack of Due Diligence With False Claims from Patchstack

Earlier this year we ran across claims from the web security company Patchstack that a bug bounty program they were running, which they were misleadingly market as a “red team”, was finding an extraordinary amount of vulnerabilities in WordPress plugins.

In May, for example, they claimed that there were 292 vulnerabilities found and that one of the submitter found 149 vulnerabilities and another found 101 vulnerabilities. Both the total and individual numbers sounded hard to believe based on our experience, both collecting up data on vulnerabilities in WordPress plugins and discovering vulnerabilities.

We simply didn’t see what you would expect going on if these results were true. When we did run into what might be the results of that, what we found was that Patchstack and their biggest submitter didn’t seem to have a basic grasp of security.

Continuing forward, huge numbers were put forward, without any evidence to support that.

Our experience going back when Patchstack was called WebARX, is that the company and its head, Oliver Sild, are often not telling the truth. It isn’t clear to what extent that this involves intentionally lying and to what extent it involves people that don’t even understand that what they are saying isn’t true. So the bar for journalist covering their claims should be high, but that doesn’t appear to be the case.

On November 2, the WP Mainline (written by the former author of the WP Tavern) published a post on Patchstack with this claim:

In 2021, within half the year, the company helped identify and fix more than 1,000 plugin vulnerabilities.

We replied on Twitter, asking what the sourcing for that was. We received no response.

On November 5, the WP Tavern published a post on Patchstack, which made a similar claim:

However, issues found in 2021 have multiplied from the previous year. Patchstack Red Team, a community bug-hunting program that pays out monthly bounties, has reported 1,182 vulnerabilities from March through October. Bounty payouts have reached $9,150 thus far.

The author of that story, Justin Tadlock, claims to be an “independent journalist”:

I am employed by Audrey Capital as an independent journalist. I have no ties to Automattic.

(Though that is hard to square with him being employed by Audrey Capital, which means he works directly for the head or WordPress and Automattic, Matt Mullenweg.)

In the comments on that post, someone asked about the vetting of the claim:

Reported, but how many actually valid? Big difference. Reported are useless unless they’re valid.

Justin Tadlock responded:

“Reported” in this sense means those that Patchstack checks, validates, and reports to the public, not reported in the sense that someone submitted it as a potential issue. I’m guessing there are far more of those. All 1,182 should be valid.

The commenter responded:

have you verified that yourself? the numbers don’t to add up

Justin Tadlock responded to that with this:

In what way do the numbers not add up? Patchstack has a public database of known vulnerabilities. If the numbers do not add up or any of the issues are false, please post your findings.

Asking someone to prove a negative isn’t what you would expect from a journalist.

When we did as Justin Tadlock had suggested to the commenter, looked at the database, we found the numbers don’t add up.

What we found was that as of November 11, if you headed back in the records of PatchStack’s database, the first entries from this year on 52 page. There are 20 entries per page and with 8 from this year on page 52, there have been a total of 1,028 entries this year. By comparison, in Justin Tadlock’s post, the claim was that the total number of entries was “over 2,000”:

These are merely the problems found through Patchstack Red Team. When combined with security issues reported through other vendors that the company tracks, the vulnerability count jumps to over 2,000.

So there is a difference of around 1,000 entries.

The head of Patchstack, Oliver Sild, replied to Justin Tadlock in that comment thread and indicated that some of the vulnerabilities they claim their red team has found, are not in their shown in the database yet:

Hey, all 1182 reports are valid. We validate them before we send reports out to the plugin developers. In some cases, it can take months to get vendors to release the fixed version. In rare cases, we’ve also merged some of the reports into single entries. So yes, you can’t see all the reports on the database yet but they will be appearing there once the triage is complete.

Even if you assume that several hundred of those are not shown yet, the numbers still don’t add up. Not only do the totals not add up, but it doesn’t make sense that the 1,028 could include both the vulnerabilities they claim to have discovered, but also vulnerabilities discovered by others. That brings up the other issue we found.

On November 11, we also pulled out the URLs for the Vulnerability Details links in the entries through page 47, which took you back through March (which is when they claim to have started this). There were 804 URLs and only 3 are from patchstack.com. By comparison, there are 342 from competing data provider, wpscan.com, 190 from wordfence.com, 76 from nintechnet.com, and 29 from codevigilant.com.

wpscan.com 342
www.wordfence.com 190
blog.nintechnet.com 76
codevigilant.com 28
m0ze.ru 18
jetpack.com 16
github.com 14
packetstormsecurity.com 12
www.exploit-db.com 9
jvn.jp 8
plugins.trac.wordpress.org 7
bit.ly 6
youtu.be 6
www.youtube.com 6
cxsecurity.com 6
www.trustwave.com 4
www.whitesourcesoftware.com 4
patchstack.com 3
www.vulnerability-lab.com 2
wordpress.org 2
blog.szfszf.top 2
medium.com 2
developer.woocommerce.com 2
www.in-spired.xyz 2
blog.sonarsource.com 2
www.syss.de 2
ithemes.com 2
secupress.me 2
0xb9.blog 2
www.jinsonvarghese.com 2
appcheck-ng.com 2
exploit.kitploit.com 2
gist.github.com 1
www.boho.or.kr 1
cve.mitre.org 1
forum.ait-pro.com 1
blog.asturhackers.es 1
cybersecurityworks.com 1
www.codevigilant.com 1
www.wpcharitable.com 1
www.pfelilpe.com 1
johnjhacking.com 1
drive.google.com 1
10up.com 1
truocphan.medium.com 1
sploitus.com 1
hidden-one.co.in 1
bentl.ee 1
n4nj0.github.io 1
www.fortiguard.com 1
ganofins.com 1
sh3llcon.org 1
mega.nz 1

If those numbers match up with their claims, it isn’t apparent to us how that would be.

Without knowing what the vulnerabilities they claim to have found, we and others can’t check on their validity. But isn’t hard to find information on Patchstack failure to fully validate things. Here was an example we wrote about 2 days before his post.

Here was another example we wrote in June.

We had presented that information in a Twitter thread on November 11, @replying to Justin Tadlock. We again asked him for a response on Twitter the next day. We didn’t get one.

Someone told us they left a reply in the previously mentioned thread in the comments of his post, asking about a response to our Twitter thread, and that was never approved to be shown there.

While isn’t a great for a journalist to not do the necessary due diligence before running with extraordinary claims, but when someone does that for you, refusing to even acknowledge it, much less rectify the situation, seems indefensible.

We reached out to Justin Tadlock by email on November 19 offering to include a comment in this post on why he acted in the way he did, but we have yet to receive a response.

Update (December 21, 2021): 13 days ago the head of Patchstack, Oliver Sid, posted on the website Product Hunt claiming that 1,300 vulnerabilities had been added to their database for this year. That is 700 less than before, but it was still more than the actual number.