This year has been the year when fake news became big news. But fake news isn’t really new: it has been going on for years. We see fake IT news all over the IT industry when partial truths and irrelevant benchmarks are used to sell products. The presence of fakes means that we need to assess our IT news carefully. The reality is that there is no source of news that can be trusted absolutely. You must evaluate the truth and the usefulness of each piece of IT news.
Fake news is generally created by someone with an agenda and a bias, then spread by others with the same agenda and bias. Any IT product vendor has a bias to its own product and an agenda to sell more of that product. This doesn’t mean that vendors all produce fake news or that IT news from non-vendor sources is never fake. Just like the mainstream news, IT news has varying degrees of truth and usefulness. Information coming from the same vendor, and even from the same person at that vendor, can have differing levels of truth. As an aside, truth is not an absolute here, either. A benchmark result is true because it accurately represents your application. For another business or another application in your business, the benchmark may not represent the workload. The truly fake news occurs when the benchmark does not represent any real application, yet the benchmark result is sold as valuable to all businesses.
One of the enablers for fake news is the decline of investigative journalism. There is a lot of work that needs to be done to investigate the truth and identify the real issues. In IT, this used to be done by the trade magazines. Back when there was lots of advertising revenue, the magazines had labs and did extensive comparisons. They would bring in competing solutions from multiple vendors and run a bake-off. Then, they would publish a long article on the relative merits of each product. These comparisons were never quite the same as any production deployment. Magazine readers would need to consider their own environment as well as the reviewer’s experience to identify the suitability of products. The great thing was that someone with experience and independence actually deployed the product and then told you what that experience was like. They also applied some consistent evaluation criteria to multiple products and told you the results. The Internet has changed the whole publication model.
Now, the vendors who advertised in the trade magazines have their own websites. There is no need for a hundred pages of product catalog in a magazine when advertisers just need to publish the URL and some specials. Without all that advertising revenue, the magazine cannot run its lab. The bake-off cannot be done by a grizzled technical specialist who deploys each product. Now the bake-off is done by auditing the vendor’s own deployment of the solution. The vendor does the deployment, often in its own labs. The analyst/journalist only sees the result, not the blood, sweat, and tears of the deployment. Sometimes articles are researched by reading the vendor’s marketing materials and specification sheets, with no actual deployment. These lightweight comparisons have even less foundation in a real-world deployment. Publishers are at risk of repeating any fake news that vendors put in their materials. There are many types of fake news in IT. There are benchmarks that are carefully crafted to show a storage array in the best light: so carefully crafted that the results are an order of magnitude different from the performance for real-world workloads. There are product comparisons that stack their evaluation in favor of the author’s product. There are arbitrary category definitions for comparisons. These definitions can prevent comparison between similar products that address the same customer need in different ways.
The reality is that there is much fake news in IT. To avoid false conclusions, you must evaluate the truth and relevance of the information you use. One fundamental is to read the whole analysis, not just the conclusion. Don’t stop when you have looked at the pretty picture: work out whether the product comparison matches your use. Look for the author’s bias; every author and editor have a bias, and it always colors the published information. Most of all, realize that the final responsibility for ensuring the solutions you deploy fit your business requirements is yours. The vendor team members will not lose their jobs if your company goes out of business. You may not be so lucky.