good question , but what if tester were testing not only the product , but its competitors as well ,would that give a more objective opinion.

should testing only be done by independant testers who have nothign to gain by quality of the test ?

Moby-Dick
good question…

indeed :)

Moby-Dick
…but what if tester were testing not only the product , but its competitors as well ,would that give a more objective opinion.

how do you think BETA-testers can be trusted to provide any independent and objective comparative buying advice on products they've specifically tried to help manufacturers make better?

Moby-Dick
should testing only be done by independant testers who have nothign to gain by quality of the test ?

i've thought it through, and can't see how, ethically, it can be done any other way…

but that's one of the points of the poll and this thread Moby - what's your take ?

cheers,

PD

you should never *review* beta hard/software. things are all too subject to change. but i don't see any harm in offering previews, based upon experience with a beta product

what's important, though, is not to let the previewer also be the reviewer - it's far too easy to get emotive if the beta and final don't match, and slant scores.

If you seperate QA testing and public beta testing - then you might have the best of both worlds ?

If the QA phase was carried out in house , you'd know that any informatino published as a result of that testing would potentially be slewed.
Even if a beta tester had nothing to gain , I dont think they'd be able to sit on the fence when it came to attempting to improve 2 competing products. So reviews / advice based on that testing should carry informatino to that effect ?

directhex
you should never *review* beta hard/software. things are all too subject to change.

agreed - and over the weekend this was a point well made by HEXUS.lifestyle Editor Bob Crabtree.

directhex
but i don't see any harm in offering previews, based upon experience with a beta product

agreed.

and that's not the same as BETA-testers specifically offering editorial that's dressed up as independent and objective comparative buying advice, on the very products they've BETA-tested to specifically try to help manufacturers make better.

directhex
what's important, though, is not to let the previewer also be the reviewer - it's far too easy to get emotive if the beta and final don't match, and slant scores.

interesting points.

do you think that BETA-testers posing as ‘reviewers’ are even more likely to “slant scores”?

and would it make a difference if the primary source of income for a publication came from the manufacturer for which it was BETA-testing?

and would it make a difference if the primary source of income for a publication came from the manufacturer for which it was BETA-testing?

I dont belive it would automatically discredit anything they produce - but you would have to bear it in mind.

*notes all the orange and red in this thread* ;)

It's a rather biased question to ask, from the phrasing used (“masquerade” has negative connetations, for example, as does the “hoodwinked” question) and the 3 no answers vs. 1 yes answer. But maybe that's just me. :)

Either way, in my opinion, no, it's not okay. But there's a difference between reviewing a beta product, and being a beta-tester. The former case is entirely allowable, I feel, so long as the reviewer has nothing to gain from doing so, but for giving his audience a look at what's to come.

Of course, if a reviewer never told anyone about being a beta-tester, no one would ever know, in the same way that if a company pays off a reviewer, no one would ever know. In both cases, there's opportunity for bias and downright corruption (Of course, accepting money for a review is already pretty corrupt). And I'd be happier, as a reader, knowing that the opportunity doesn't exist in the first place. :)

(Side note, I voted option 3 by accident, saw it was 3: 3 votes, and 4: 4 votes, clicked back, set 4, and tried again, this time with it being 3: 3 votes and 4: 5 votes. So not too sure if I voted twice, or if it was actually changed.)

I think it's too subjective question to actually select a poll option.

A beta tester can still be a reviewer as long as they have the integrity and publish an honest review rather being slanted into publishing what makes the manufacturer happier.

An astounding number of writers and reviewers of Windows 2003 server were beta-testers. That's the very nature of the product.

The difference between the reviews & reviewers is those that felt they had to give a favourable review because it's Microsoft and they beta tested it.. and those who were completely honest and beta tested it.

It's more down to the publications integrity & reviewer honesty than a hard & fast rule of yes/no.

Since when was reviewing all about

Hexus
objective comparative buying advice

?

Beta testers will have access to a product, and presumably know a lot about it, thus may be the most informed to give out information. I'm glad just to get the information in a review.

Beta testers do not always work for the company of the product they are testing, and there is not always (or even usually in some cases) any incentive to falsely promote a product in a ‘review’.

Finally Hexus itself is happy to give opinions during reviews - why should everyone else have to be objective when even Hexus isn't? I'm glad for subjective reviews sometimes - it's obvious enough if someone isn't trustworthy and then it's just the case that no-one listens to their reviews.

I'm disappointed hexus would chose to weigh such a discussion by using such language as ‘masquerading as reviewers’. Just present to us the content, but also give us (factual) information about the reviewer so we can make our own minds up.

hi kalniel,

kalniel
I'm disappointed hexus would chose to weigh such a discussion by using such language as ‘masquerading as reviewers’. Just present to us the content, but also give us (factual) information about the reviewer so we can make our own minds up.

hm… ok, lets tweak the poll question a bit…

how about:

Can BETA-test sites be trusted to provide 100% independent and objective reviews ?

?

cheers,

PD

Shorty - You raise some good points, but the problem is establishing that a reviewer is honest and has integrity in the first place. Everyone represents themselves as being honest and accurate, since that's best for business, so as a reader, it becomes difficult to tell who really is honest, and who isn't.

As I see it, there could be three categories here:
1) Reviewers who say they don't/neglect to mention that they beta-test, but do
2) Reviewers who say they do beta-test, but assure everyone that it won't affect reviews
3) Reviewers who say they don't beta-test, and don't

For me, category 2 is potentially susceptible to corruption, which unfortunately puts anyone in that category in a bad light, since I'll never be 100% certain that the review is objective. Of course, I can't really tell the difference between 1 and 3 unless something gives it away (As has been the case with one or two other review sites, in terms of paid reviews).

Eldren,

I agree with you wholeheartedly. Unless you know the publications “people” personally, there is no real assurance of honesty. That goes for any publication on any subject. They are may seem sincere but.. you know how it goes.

I still insist that whether someone beta tests a product is not a factor in whether they can fairly review it (im not a beta tester Id just to point out!). It's down to the person reviewing to provide an accurate analysis regardless of whether they tinkered with it in it's early days or now.

Gosh, I've got the strongest feeling there is something y'all aren't telling us here. You didn't wake up this morning, stare at your cheerios, and “discover” this question. So, give –why this now?

Also, “100%” objective seems a bit like putting the thumb on the scale to me to get an answer, and the options weighted as well.

So, come on fellas, give out where you're coming from on this one. . .

Thanks PD. Appreciate it.

I've found pretty much with every opensource / public beta style application, the beta testers are %100 objective and honest, to the point where I have seen beta's pulled and a whole product revision dropped and re-wrote.

I would assume the same with companies like microsoft (I use microsoft as an easy example) simpley because by the time they do a “public” beta all the really serious problems should be gone, and kept internally, however if something bad is found I'd expect to read between the lines as companies will only release as positive spin on a major problem.

I think people are pretty open with beta's though as a general rule of thumb

I'd agree that Beta testing is designed to make products better. BETA testing is only worthwhile if it comes EARLY in the process. IT has to be able to influence the product in development and as such the public should pay NO attention to the review, as it should have only a small amount of bearing on the retail product.

Will it be interesting to read beta testing reviews on the latest ECS motherboard or ATI Graphics card. YES definitely. But if its being published to the world, then i would say it would definitely have a different tact to a direct report to the manufacturer.

If Hexus did this and got paid for it - i wouldn't personally have a problem with it, but then i doubt if its the community that is most important in this decision.

A big PRO or big Negative about a Beta review might create a storm?. (Nick's review of Pacific Fighters still makes me laugh)

TiG

As most of you know, until recently I was working for a Greek magazine as a hardware reviewer. I am also a beta tester for ATI, so I think I have a saying in the questions Dave presented (sorry for not noticing the thread earlier, but I was looking for something totally irrelevant and kinda stumbled upon this :))

The reasoning is simple really, and it is indifferent whether you beta test or not. It's all about your journalistic integrity. From my point of view, beta testing a product has a very good influence on your perspective review. The reason is plain simple: you have spent a great deal of time with a product, trying hard to break it, to expose its flaws. If you are a capable beta tester, you pretty much know what it can do and what it can't do. Do you get that kind of in-depth knowledge from a (perfectly capable otherwise) reviewer, who will only review a product, most times in a hurry, in order to meet a deadline? Will he have time to do an exhaustive review, and know where to look in order to have a better view of the product? The answer, in my opinion, is a ‘No’ on both accounts.

Of course I am not saying that reviewers should beta test every hardware they review, that's not my point at all. But when reviewers review products they also beta test, their job is much easier exposing flaws than people who haven't got this much “breaking experience” with a said product. Now, there's the issue of trust, as Dave said. Well, my answer is pretty obvious. Not all journalists are beta testers, and not all beta testers are journalists. Journalists are judged by the readers with every review they publish - and that is a constant process; we are judged by what we write, what we say and what we do. If you see a journalist being biased, this probably means that he's not a good journalist in the first place, regardless whether he beta tests something he reviewed or not. If you take pride at what you do, and you want to be considered one of the best, you don't let either beta tests, personal friendships with PR Managers and any other relationships influence your judgement. And you try your best to be impartial. That's what journalism is all about really; and I believe our readers can tell impartiality when they see it.

Just my two eurocents :)

Well, personally I wouldn't think the objectivity pressures are any worse than for the smaller sites that have to scuffle for cards and have the ever present fear that they won't see another if they don't give a positive review. So I think it comes down to integrity, buttressed (hopefully) by a strong site in the first place to help keep those pressures at bay.

Tho I think if I was running the shop I'd have someone else do the review and just use the beta guy as a “resource” if questions come up.