The things Futuremark have to contend with
The crystal ball problem
Futuremark - with the very name of the company telling you what their biggest hurdle is - have a problem somewhat unique to them in terms of 3D application development. While anyone writing a 3D game has to consider how the architecture of the renderer will affect its performance on hardware now and in the future, Futuremark get confounded by the fact that 3DMark06 is all about the business of measuring that performance.And that performance measurement using their eventually chosen set of techniques seeks to emulate (in a sense) the performance profile of any number of other popular graphics techniques during the useful life of the benchmark.
They don't have the luxury of patching to make their way around performance cliffs on one or two popular SKUs, a few months (or even years) down the road. The very fact that 3DMark is built around the Futuremark ORB, where you can upload results and compare your system to someone else's, means that heavily patching the renderer is almost absolutely out of the question.
Invalidating results in the ORB is the reason why. Futuremark therefore have to trust their research and development team who work to decide what rendering techniques they'll enter into a new version of 3DMark. The implementation of those chosen techniques not only has be the very backbone of a 3DMark score, but Futuremark have to battle with hardware availability during development, API targets and backdoors, and the maintenance of multiple codepaths for hardware they target, should one not be enough.
The nature of diverse hardware development means that all of that is, in my opinion, a fairly impossible task.
The hardware availability problem
It's quite clear that 3DMark06 was developed almost entirely on NVIDIA's Shader Model 3.0 hardware, and therefore logically with some assistance from NVIDIA's developer relations team. The timeframe of 05's release and 06's introduction shows you that.S3, ATI, Intel, and anyone else with aspirations of shipping SM3.0-able hardware into the desktop space, simply didn't have hardware available to Futuremark so that the company could do side-by-side development work.
ATI's Radeon X1000-series SM3.0 parts became available near the end of the 06 development process, although that's arguably too late for Futuremark to make sure their R&D work for 06 was validated on those ATI parts.
The API target problem
While Futuremark uses Microsoft's DirectX API to develop 3DMark, with the introduction of 3DMark05 it started using a 3D hardware feature not specifically exposed by Direct3D. 3DMark06 takes that even further with the use of multiple vendor-specific hardware features detected and used in different ways.Some of those vendor-specific features were only testable and implementable late in the day, given their availability on certain ATI SM3.0 parts.
The multiple codepath problem
Vendor-specific hardware features and performance profiles of hardware across IHV (and somewhat across vendor) now determine how Futuremark architect their renderers. Taking into account any differences in implementation or performance means maintaining a codepath block per major variation at least, depending on whether you can roll-up multiple paths into one block.It's the same problem a game developer faces, but when it concerns the lofty goal of "unbiased high quality benchmark software" you quickly run into a, "well, you added a codepath for their feature, why not one for ours?"", quandry.
The performance difference problem
Lastly there's the decision to implement API features based on performance. Since 3DMark06 targets Shader Model 3.0 hardware heavily, without everyone's Shader Model 3.0 hardware available during development they ran the risk of not using a certain API feature because it underperforms or doesn't work on the hardware they used for development.Again, the game developer faces the same issue. It's an easier problem for a game developer to solve, though. Futuremark's requirement to be rigid after release has the potential to box them into a corner.
