How Futuremark Engineer 3DMark
Futuremark hint that the development process for 3DMark is a collaborative one. Directly from their website at Futuremark/BDP:
From the BDP intro on their website:
Futuremark is an industry-leading provider of benchmark software and services. Our mission is: To increase growth within ICT industry by showing the performance of new PC and mobile technology, simultaneously taking into account the end user satisfaction.
Benchmark Development Program (BDP) is Futuremark's way of co-operating with leading hardware and technology manufacturers in order to design unbiased high quality benchmark software.
So Futuremark use the BDP to help create unbiased benchmarks, cooperating with the leading hardware and technology vendors in that graphical list, along with ExtremeTech, C|NET and Beyond3D from the online technology press. Futuremark also have a Media BDP, where they give media publications - like HEXUS - early access to the final revisions of the benchmark before it's publically released, soliciting their feedback on the benchmark supposedly in the interest of making the benchmark better, by way of their BDP mantra.
I've been part of the Media BDP in the past, specifically for 3DMark05, and have given my snippets of feedback on what I think could make the tool better from a hardware reviewer's perspective, shortly before the product launched. I haven't renewed my participation this time for 06, largely because Futuremark engage the Media BDP members far too late in the development process for us to make any difference to how the tests are architected, developed or otherwise engineered.
Speaking with other Media BDP members, Beyond3D, and reading posts on a public forum from ExtremeTech's Jason Cross, it seems no different this time around. Media BDP members were engaged right at the very end of the 3DMark06's development, literally days or a week before its launch. Indeed, Beyond3D and ExtremeTech - listed as tier 1 BDP members in the graphic along with C|NET - either had no BDP input at all for 06 (Beyond3D) and didn't see it before release, or were engaged as Media BDP were (ExtremeTech).
That casts doubts on how involved in the development process the hardware companies are, architecturally, in order to create the unbiased benchmark Futuremark seek to build.
Now some of the other BDP members don't have input into 3DMark at all, and are instead members because of participation in other Futuremark projects. That realistically leaves just Microsoft, ATI, NVIDIA - and possibly AMD and Intel - as the real heavyweights with significant participation in the development of 3DMark06.
It's worth noting at this point that Futuremark require a fee to be paid for membership into the main BDP by some of the companies. BDP fees for relevant parties seems to be negotiated on a company-by-company basis, each asked by Futuremark to pay a sum that likely befits their involvement in production of the Futuremark benchmark they're involved in.
Summary
Futuremark claim to, by way of collaboration with paying IHVs and industry partners, and non-paying Media BDP members, develop unbiased, high quality benchmark software. However, with little Media BDP involvement until very late in the process, literally just before it went gold in the case of 05 and 06, it questions just how involved each of the other industry 3DMark BDP members gets.Additionally, those industry partners are somewhat key to Futuremark when determining future trends in 3D graphics. Or so you'd think. The IHV's developer relations teams who spend significant time and resources working with developers world-wide, to ask them what they'll be doing with hardware in the future in terms of 3D rendering, are undoubtedly the most clued in on that subject.
With that in mind, let's have a look at what Futuremark came up with for 3DMark06.
