Benchmarking pros and cons
Benchmarking pros and cons
When considering buying any piece of hardware or software, it's useful to know as much as possible about that product and its direct competitors. That's why product reviews make such popular reading and also why the independence and integrity of reviewers is critical.
There's no point at all bothering to read a review if you're not 100 per cent sure that the conclusions are not being swayed by inducements from the product's maker and the reviewer, therefore, doesn't actually have your best interests at heart.
Assuming the price is appropriate, the key factors that a reviewer has to consider are how easy a product is to use and whether or not it works as claimed on the tin. Oh, and, of course, how sexy it is!
Assessing how accurately a maker's claims are met requires a reviewer to check whether the features available do actually match those that are promised and then find out how effectively and how quickly those features work.
Benchmarking programs have become key tools for assessing the performance of new hardware and software. That's because they make possible comparisons with previous generations of product and between competing offerings. They also have one practical advantage - many are automated and cut down on the reviewer's often heavy workload.
Many sites and magazines use the same suites of benchmarks as each other – albeit alongside some of their own home-brewed programs. In one way, such standardisation is a good thing. It provides a measure of consistency between tests carried out at different times and on different products. It also enables a buyer to compare results from different reviews as a way of double-checking on performance (and the reviewer's integrity!).
But there are a number problems with the current state of benchmarking – and widespread standardisation is one of them. Take a situation where, say, a graphics card maker knows that all of the key reviews of its flagship card will be carried out using the same suite of benchmarks.
The results of such tests are likely to be much the same irrespective of the reviewer if few errors are made and there are no shady goings on. If test results show the product in a good light, that should help ensure positive reviews and, logically, such reviews should lead to increased sales.
It might therefore be seen by the maker as being in its own best interest to optimise the drivers for a new card so that high scores are produced – rather than concentrate where it should do on making changes that would offer real-world benefit to most would-be purchasers.
Even though such things may do go on – and may happen in a number of product sectors – massaging numbers this way is probably one of the least important issues with benchmarking software, though it does still cast a small shadow over Intel's offering, the IDHCAT suite.
Two other factors are far more significant. The first is whether or not benchmarks are fully understandable to a majority of product buyers. The other is whether or not what's being tested provides truly useful information – even if it is widely understood.
The most widely used of today's benchmarking programs are very specialised tools. Their functions and results may be crystal clear to the technically clued-up HEXUS.core reader but they aren't fully understood even by some reviewers and are not understood at all by the man/woman in the street who is the intended purchaser of digital-home PCs.
In addition, most of the benchmarking suites currently in use are intended to give a thorough work out of just one specific component – such as a graphics card or CPU – and its allied motherboard sub-systems.
Trouble is, no single suite provides comprehensive whole-body testing of digital-home PCs, such that it predicts how well the machine can do the various multi-media tasks it will be expected to carry out.
Together, those factors explain why Intel feels it in necessary to jump-start things by introducing the IDHCAT suite.
