Introduction
The launch of Futuremark's 3DMark06, its latest 'Gamer's Benchmark', has thrown up all manner of questions and concerns about the benchmark itself, Futuremark as a company, the changes facing consumer 3D graphics in Windows Vista, and just how much weight the benchmark carries outside of the forums and hardware websites of the world.
This editorial piece attempts to discuss those questions and concerns from my personal perspective as our Group Technical Editor, and therefore the person who gets to arbitrate its usage in our published titles. The piece written by yours truly on 06's launch, published by HEXUS.core, summed up 3DMark06 as follows:
While the overall score remains fairly useless to us in some respects, that doesn't mean the benchmark has no merit.
Recent deeper analysis of the benchmark, which forms the basis of the technical perspective of this piece, has that statement stand firm. However, the analysis also throws up other facets of how the benchmark was engineered that require discussion. So to get this piece kick-started, here's my take on where 3DMark sits in the world of consumer 3D.
Futuremark's 3DMark benchmark
Since the launch of 3DMark99 and right through to recent times with 3DMark05, Futuremark's 3D benchmarks have obtained ever increasing reverence and influence, and are often held up as the last bastion of 3D performance measurement for some people. Despite being an inherently wrong way to analyse the relative merits of 3D graphics hardware, reliance on just the final score spat out by 3DMark has become ingrained into various parts of the 3D industry. And rather than weaken its position because of that, it's only served to make 3DMark's position stronger over the years.Read a review of a 3D graphics product on the web and you'll invariably see the final score used somewhere. Read online forums where 3D graphics is a strong topic of discussion and you'll see the benchmark and its magical final number rolled out to make a point, close a discussion or simply to show off to others.
However, and perhaps more sinisterly , the final score is often used - and solely used with no consideration for any other performance measurement - by system builders and OEMs when deciding what graphics card they should buy and offer in their SKUs. A ratio of 3DMark score to buy-in price is used by people spending a collective pot of money that likely reaches into the hundreds of millions of dollars.
So while 3DMark has influence at websites and online forums, itself probably not a good thing on balance, it also has influence over massive budgets and share prices.
That lofty position of influence only works if the benchmark does. It's that I'm entirely unconvinced of with 06, especially given Futuremark's own stated goals in the creation of each new version. Let's dig deeper into how Futuremark engineer their 3DMark series of benchmarks, to see if they get the basic building blocks right to justify the hammer they hold - and often weild.
