Should we have a Metacritic that rates game reviewers, asks David Braben
"As developers, we all like to bemoan the odd game review from time to time – because games are close to our hearts, and increasingly our wallets both in terms of sales and Metacritic-based incentives.
Most reviews are targeted at what are often called ‘core’ gamers; people like us that follow games avidly, and are very experienced at playing them. Most reviewers and developers fall into this camp themselves, as do the readerships of most gaming websites and print press.
And so, entirely reasonably, those reviewers aim their reviews accordingly.
A problem starts to occur when the audiences’ tastes differ significantly from the reviewer’s – or developer’s – own tastes. This is becoming more of an issue as our industry matures to include a great many people outside this group – particularly so if the group targeted is not just this ‘core’.
I have in mind games like Animal Crossing, Nintendogs, RollerCoaster Tycoon – all huge sellers, but with noticeably lower reviews than their quality suggests.
The presumption is ‘it wasn’t aimed at me, and so it must be bad’.
Advertisement
This attitude makes sense if the audience for the review is effectively the same as the reviewer themselves, but for a review on TV, on a website for kids and adults, or in the mainstream media, it does not.
Just as it is difficult to develop games for a different audience, it is difficult to review them too."
"The overall effect – particularly for those games that don’t include this ‘core’ in their audience, is about 10 per cent of the review points. It is not something to cry about – just something to be borne in mind.
I have been delighted by the mature response by the games press over Kinectimals, for example. Almost without fail they have pointed out that the game is targeted at kids – as it was – and have reviewed it with that in mind – like the excellent ‘Meet Dave’ piece in Joystiq by Justin McElroy, for example, and the astonishingly cute review by a six year old on wearearcade.com, which gave it 100/10.
There are always the occasional outliers, either through a genuine difference of opinion, or for some other reason. Kinectimals got a couple of shockers, but as developers we have to take these things on the chin."
"In these connected times, Achievements or Trophies have been a curse for the small minority of hurried or irresponsible reviewers if their online ID is known.
Occasionally someone is caught out, or accused of being unreasonable. For example, there was the hoo-hah over the reviews of Space Giraffe and Kane and Lynch and the alleged connection to the firing of Jeff Gerstmann.
Most reviewers are excellent at what they do, and it is a very hard job with, frankly, little glory. As an industry, there is something we could do to recognise this – effectively a Metacritic for reviewers.
The best reviewers give spot-on reviews pretty soon after a game is released. They do not wait to see what others say, but nevertheless consistently come very close to the final average score. There could be a prize for the best each year.
Don’t forget – this is not intended to influence reviews – just to encourage and reward consistency – as it is not a high reviewer that gets the reward, it is the one that gets the best result.
This method could also be used for non-‘core’ games, too, with the benchmark being either eventual sales, or eventual average user reviews, as at the moment it is a real lottery for customers buying games for their younger kids – with few trustworthy reviews – which is one of the reasons, I think, so many shovel-ware games sneak under the radar in this sector.
If there were a system that tracked reviews by reviewer, not by publication, then hopefully this could reinforce the position of ‘star’ reviewers in particular sectors, which I think would be a very good thing for all concerned."
On the whole, games media IS terrible.
It's very rare that you'll find anyone reviewing games who is objective, or has any sort of understanding of the medium or its history.
Read any review in, say, Empire, and they'll talk about the director, reference relevant movies, influences, styles etc... You practically get an insight into how the movie happened, was made, where they succeed, where they fail.
Games media just doesn't have anything of this quality. Instead you get barely literate teenagers taking a guess at the facts, reviewing things which aren't even aimed at them, and trying to be funny along the way (I blame Zapp64 squarely for that)!
I read the headline, then your comment, then the headline...
Good point, that's a bit of a jerky thing for me to say! Yeah, they could win I suppose, but it would mean completely changing the way things are at the moment.
I'm one of these guys who has long since given up on games reviewers. they universally praise things which I think are terrible about games (the quality of the plot / writing in CODBLOPS a good recent example), whilst at the same time ignoring the things I like seeing in games (innovation, scope etc..)
Everyone has a different taste in games, different criteria for judging and enjoying them. Same with music. Trying to review these things by some standard criteria is never going to work. My favorite albums of all time are rated less than 30% on Metacritic!
That's why they have different magazines for different types of music. the music of Justin Beiber wouldn't go down well in Kerrang.
I find myself increasingly taking more notice of user reviews on Amazon and similar non-hardcore places.
Every time I read or hear discussions about games journalism, the chatter is always fixed on reviews and scores.
While there's nothing wrong with a publication reviewing games, of course, I struggle to understand why so many people tie product reviews so closely with the broad term 'games journalism'.
It's indicative of a lack of understanding from the audience, but that's just me.
Shit Mark Kermode reviews, to compare, would not mean the BBC News and its Magazine is failing its audience.
* [This article, of course, is solely discussing games reviewers and not the feature writers, news hacks or broadcast writers].
I feel quite bad about my original post now. Of course there's good people out there, it's just that there's far more bad. Far far more.
I personally cannot stand Metacritic. While it has some usefulness by allowing a collective view on how a game is received, I find that too often developers will follow it's criticism as the be-all end-all on the quality of their game (or worse, base major decisions on it).
I think the actual content of what a reviewer writes should be more important then the final score they put on it. Too often people will skip an entire article just to see the number they threw onto it in the end.
I dont think its necessary to review the reviewers that would just extend the problem that hes trying to combat with this opinion. I think instead gamers should remember that a review is an opinion which is going to be based on the personal likes and dislikes of the writer. I think the games industry just needs to continue making the games and the gamers should make their own minds up about the games they like and buy.
An aggregate for reviews per writer would be very beneficial: you'd get to see their opinions on many different games all in one place, facilitating the ability to find reviewers you agree with, which is an important factor in purchase decisions.
But the aggregate score Metacritic is known for is a well-intentioned misstep, or perhaps it's just given far too much weight by developers and publishers. As previous posters have said, tastes always vary wildly in subjective media and a score to gauge whether reviewers have the correct "finger on the pulse of games" is... unnecessary.
What about if there were a site which handled the balancing of scores. Perhaps users who turn up to the site enter their game preferences, submit a few of their favorite games, click if they agree or disagree with scores given for games they've played, enter their own scores... and over time, the 'ratings' get tilted towards their preferences based on all this information. So two people turning up to the same page would see a different set of scores for the same game.
That, of course, doesn't make up for badly written reviews, but if everyone does just scroll to the scores, at least they'd be more accurate to their tastes.
There's a fundamentally incorrect and disrespectful assumption that always gets trotted out by developers, including Mr Braben in this very article. It is this: "The presumption is ‘it wasn’t aimed at me, and so it must be bad’."
No. That's not it at all. Give the writers more credit.
Game reviews are not written to apply to the entire world. They are written specifically with the interests of their respective publication's readers in mind. Nintendo Power, USA Today, PC Gamer, and the Humane Society e-newsletter are read by completely different people. And if you are a reviewer who is not putting yourself in the shoes of the readers of those publications and writing your Kinectimals review with their tastes in mind, you are a poor reviewer who is not serving your readers.
However, the majority of people reading game reviews in gaming magazines and websites are enthusiasts - that's why they're seeking out specialist publications. And they tend to be less interested in kiddie stuff, casual titles, Flash and facebook games, iphone gaming (though this is changing) and so on.
That is why they score these games lower. It's not because the writers don't "get" the game or understand what makes it appealing to its target market. It's because the writers know their readers aren't part of that target market. You want great reviews in EGM? Make a game that appeals to that book's readers. Want great reviews for Kinectimals? Look in magazines that are read by 10 year-old kids and their moms.
I agree with what you're saying Eric B, but I would question why sites / magazines would review things that aren't aimed at their target audience / readership in the first place?
I've never totally relied on reviews, though multiple exceptionally low scores will give me pause for thought. When it comes down to it though, it's how much I enjoy the game demo that influences me more.
Unfortunately in recent years a lot of developers/publishers don't release demos, or in the case of several EA games they do on the consoles but not for the PC which is my main platform of choice. This results in me depending more on reviewers' opinions than I'd prefer.
So reviewers are reviewed? But then who reviews those who review the reviewers?
Herein lies the problem: if you have a Metacritic for game journos, then any platform-exclusive title/blockbuster game that's not given a 9 or 10/10 by a reviewer will be slated and ridiculed by a small, but very vocal contingent of games website readers.
The phenomena is referred to as fanboyism (I'm sure you've all heard of it). The reality of it means that any reviewer who doesn't laud the latest blockbuster FPS or PS3/Xbox 360 exclusive gets ripped apart by fanboys in comment boards.
These same fanboys would be very active on a 'review of the reviewers' Metacritic - this I am certain of, just as the sun will come up again tomorrow morning.
By using a system like this, you'd effectively be strangling innovation in the industry even further. Derivative genre-blockbusters would dominate Metacritic even more than they already do, pushing the innovative titles further out onto the periphery.
This is nonsense. How do you think a reviewer is going to review a game when he is under pressure from his publisher to get a good metacriticritic score? A high rank on metacritic means more exposure, more site traffic, more magazine sales, and more money flowing in their direction in a highly competitive market sector. Whatever criteria you produce these scores with, critics will find themselves compelled to match them. If the object is to match the overall average and publish first then all reviews will be moderated and rushed. The quality will suffer and all critics compromised. Right now they get to write what they think about a game, under your system they will all race to get a review out and do nothing but predict an average score and write it accordingly. This is totally dishonest and thoroughly unhelpful to the consumer, and the consumer is the one who reads them. Your article is wrong in every way.
Reviews are the product of a forum made up of anyone who has anything to say, those who get paid for it get to take the floor now and again and they are allowed to be disagreed with and they are allowed to disagree themselves, the system of reviewing is not there for the developers benefit, if you don't like it, make better games.
I've been reviewing games since 1998 for a few places before striking out on my own with a blog about two years ago. My own opinions have evolved over time, but I've always tried to judge games based solely on their own merits as opposed to endlessly (or unfairly in some cases) comparing every aspect of a game to another game (unless it's so close in terms of certain elements that it's hard to ignore).
It irks me when I read some overly jaded reviews from overly jaded reviewers who absolutely don't "get" that not EVERY game is going to look "state of the art", be 100% "innovative" or follow "today's standards" - three of the laziest terms used in reviews.
Nonsense such as comparing a portable version of a game point for point to an HD console version, knocking a kids game around for being too easy (or too shallow) or not understanding why a decent single player game doesn't require multiplayer modes are all things that I see otherwise fun games get slammed around for - and don't get me started on the "8-10 hours is too short" nonsense...
Sure, I love pretty graphics, but in the end, they're subjective. If you only play HD games, a DS game may look too ugly (even if it's the best-looking game on the system). On the other hand, if you have an appreciation for art and can see the hard work that went into ANY type of visuals (yes, you NEED to understand stylization!), graphics don't matter much.
If a game is fun and makes me want to keep playing (even with minor technical issues), I'll review it as such while making a suggestion here and there on what could have been done better. I actually like chatting with developers whenever possible simply because I love the medium and it's always interesting to see and hear the hows and whys of certain design choices.
Even if I don't like a game, I'll at least give it a constructive bit of criticism rather than just use it as a dartboard for lousy jokes about the folks who made it or the ones who still buy and play games that aren't so hot for any number of reasons.
I'd say reviewers need to realize that in this day and age, people who aren't part of the demographic they write for are reading their reviews because MORE people are playing games. Writing for a certain audience alone only alienates readers looking for fair opinions, period. If you slam a kids game just to get your more macho shooter readers chuckling, that's not a good review at all. That type of writer should stick to the games they cover best and leave games he or she doesn't play to those who can explain the good and bad points without being a total jerk.
Imagine if you had writers who absolutely hated (and had little understanding of) sports covering sports on a regular basis? That, I'd gather, would make for some fine flame wars... but not good journalism at the end of the day.
g.
True Story, I sed to review games for Dave "Gamesanimal" Perry years ago and he said' he'd fire anyone who'd ever rate a game 7/10 as it's the most pathetic "non-offensive" number to score a game.
As with book reviews it always helps to know who the review is aimed at, like it is said in the review. It is important that the review is directed towards the reader, and it is helpful if the reader knows this. For example, a new Football Manager might score low for fans of Football Manager and high for a new audience if the new version had all the hard core options removed and simplified for beginners. I like a review that informs *me* of how *I* might respond to the game, or that provides me with enough information to excite me or repel me from the title.
Reviewing can be a tough call, some reviewers were chastised for giving GT5 a low score because of the time it took, but then when you consider the anticipation of a long awaited sequel and look at the massive difference between GTA4 and GTA3, you can see why that *might* now be relevant.
*I'll go hide somewhere safe now*
