In those years it's developed a reputation as one of the most hated independent testing companies around.
And it just had a pretty ugly public spat with security software vendor CrowdStrike. CrowdStrike tried, and failed, to get a temporary restraining order and injunction to stop NSS Labs from publishing results of a test that includes CrowdStrike's software and several competitors.
CrowdStrike makes cloud security software that protects PCs and networks from hackers and malware. It gained attention as a buzzy startup when it raised $156 million in venture funding, including a huge $100 million round in 2015 from Capital G, the growth investment fund from Google's parent company Alphabet. That round valued the startup at about $667 million, according to Pitchbook.
NSS came to fame years ago for its tests on browser security which found Internet Explorer to be more secure than Google Chrome. Google was not pleased, disputed the methodology, said the tests were paid for by Microsoft and that the tests didn't even use the latest version of Chrome.
CrowdStrike said in a blog post that it initially hired NSS to test its software and issue a private report to be seen only by the peeps at CrowdStrike. It said immediately disputed NSS's testing methodology (calling it "deeply flawed"). When NSS wanted to include CrowdStrike in one of the tests it would sell in a public report, and announce the results at a major conference, CrowdStrike said no.
NSS was not deterred, and according to CrowdStrike's blog post, obtained software provided by one of CrowdStrike's resellers and did the tests anyway.
CrowdStrike was not pleased. "It colluded with a reseller and engaged in a sham transaction to access our software," the company accused. The company asked a judge to block the release of the report alleging everything from breach of contract to stolen trade secrets.
The judge in Delaware district court wasn't buying those allegations, and on Monday issued a ruling siding with NSS. So the results will be published. Neither CrowdStrike or NSS immediately responded to requests for comment.
Previous challenges
This is not the first time NSS has raised the ire of the software vendors it tests. In addition to the Microsoft/Google browser hullabaloo, NSS was criticized by Palo Alto Networks and FireEye in 2014, when they questioned its testing methodology and its ethics.
Palo Alto Networks founder and CTO Nir Zuk went so far as to imply NSS charged the vendors who are tested a lot of money and gave them undue influence in how the tests were set up so that their products would shine, while products from their peers (the ones who didn't pay) would not.
NSS has denied this, saying that it doesn't take money from vendors for tests shares publicly. It says it chooses which vendors to test based on requests from IT professionals wanting to buy its reports, and if a vendor refuses to participate, it will buy the product and test it that way.
The upshot is, if you are an independent software tester and you are doing it right, you are going to anger a lot of vendors who will try to discredit your results. And if you are an independent software tester who is doing it wrong, you are going to anger a lot of vendors who will try to discredit your results.
But since the judge ruled against CrowdStrike, IT buyers will be free to see the latest test results, examine the methodology and judge for themselves.