Making decisions about IT infrastructure purchases is hard. There are many interlocked decisions to be made, each of which has multiple requirements. Often, a collection of compromises must occur. For the business project owner with little IT knowledge, it may seem far too complicated. It would be great to have tools that simplify these decisions and reduce the work required to make complicated decisions. It would also be useful to have an independent analysis that supports the decisions made.
One way to simplify these decisions is to rely on the expertise of others. Before the internet and blogging, a major source of expertise was print magazines. A chunk of revenue from a magazine’s advertisers was fed into a test lab. Vendors might submit products for evaluation, or the magazine might buy products. After weeks of testing, the magazine could publish reviews that summarized its findings and analysis. Even better were bake-offs, in which several products were tested against the same criteria. The magazine’s readers could then make informed decisions about which product to buy for their needs. Alas, the days of thick, advertising-heavy magazines have gone. Those advertisers now run their own websites or simply list on Amazon. With no advertising revenue, there is no lab budget. Most of the magazines have gone online now, too.
Where does an organization go these days to find a bake-off of infrastructure products? If you are Megacorp, then you run your own: a series of proof of concept (PoC) projects, one with each of the possible solutions. The reason that only Megacorp can do this is that PoCs cost a lot of money, mostly in staff salaries. Wouldn’t it be great if Megacorp published its results? Alas, the right choice produces market differentiation for Megacorp, so the results are very secret. For smaller organizations, the cost of running multiple PoCs exceeds the cost of choosing the wrong product. It would be nice if there was a place we could go to get comparison information to help us make our decisions.
One useful site for crowdsourced product comparisons is now available. WhatMatrix provides community-driven comparisons of IT infrastructure products. It covers a range of common product categories and a pretty comprehensive set of comparison criteria. Much of the analytical work of designing comparisons and evaluating products has been completed by independent consulting organizations. I have to expect that some customers will approach these organizations for assistance with projects as a result. The website allows for comparisons of up to three products, within one category, at a time. It has nice red/yellow/green color coding for some of the comparison lines, helping you zero in on areas of difference. Overall, it’s a pretty cool site, and the results are available for free. Another nice thing about it is that you can use the site to decide what other products in the category you should evaluate, as well as possibly rule some products out before spending more time or money on further evaluation.
Now let’s consider the risks of this sort of comparison shopping. The first thing to remember is that there is (alas) no one solution that is right for every situation. Even though one product may have the highest score in a category, that doesn’t mean it is always the best choice in that category. One thing you will need to do is look at the evaluation criteria and work out whether they map to your own needs. For example, a company may have a corporate standard for a specific hypervisor. WhatMatrix will score a solution that supports multiple hypervisors higher than one that only supports a single hypervisor. This is not part of the company’s criteria, however; the business only cares that the solution supports the one hypervisor that it will be using.
Another factor to be aware of is a bias among the independent consultants. I don’t mean bias in a bad way: everyone has a bias based on their own knowledge and experience. Personally, I have done a lot of work with hyperconverged, recently with SimpliVity and in the past with Nutanix. Because I’m familiar with these products, I’m inclined to think they are good solutions for customers. I’m less familiar with using a NetApp/Cisco FlexPod, so I am less likely to see it as a solution for a problem. Everyone has a bias, and it is worth considering the bias of any analysis you use.
IT infrastructure decision making is complex, and there are usually multiple right answers. Relying on some external expertise is a reality with the decision process. WhatMatrix makes some great analysis available for free. Just make sure the analysis fits your situation.
Thanks for your insight and comments Alastair. We’d like to address your concerns in our comments below.
“One thing you will need to do is look at the evaluation criteria and work out whether they map to your own needs”
Yes, totally concur. Therefore, in the next major release we will provide a “use case” filter and customization option. In simple terms this will allow the user to prioritize or exclude features according to THEIR deployment scenario and preferences (e.g. “integration with hypervisor xyz” might be very important to you if that is what you use – then again it might not be – and therefore might currently skew your score).
“bias among the independent consultants”
We genuinely believe that we are in a much more agnostic position to begin with compared to any vendor-provided comparisons and many larger analysts but, of course, individuals by definition have a bias. Therefore, we address this point by going through a peer review (within the community) before any update, where we also allow the involved vendors to challenge us (e.g. the technical review with two storage vendors took us literally 2 weeks for a single release update). In addition, we invite the wider community to challenge any evaluation that they disagree with for a review (simply click on any feature in the matrix and select “submit change”)
One of our key goals is to continue to evolve and improve the comparison framework based on feedback – so please keep it coming!
I hope it was clear that I really like WhatMatrix.The ability to tailor the comparisons will be great and significantly increase the value of the comparisons on the site. Keep up the great work.
As for bias, I think that we need to acknowledge that bias is natural and to have disclosure around our own bias. I definitely agree that a lot of analyst firms fail to disclose bias. Analyst firms, including here at TVP, are funded by a subset of the vendors that are written about. For some, there is a strong correlation between funding and ranking. I like that WhatMatrix makes the contributors visible.