I think the biggest problem with Metacritic, or the whole result of using numbers from various reviews and adding them together for an average score, is that niche games tend to simply dissapear among the countless mediocre games, with scores of 60-80.
In the case of many of the mediocre games, those are the final numbers that most of the reviews ended up with, so the metacritic score average isn't all that far from what most of the reviewers had written. The games were simply flawed and unpolished, and could not reach up to that magic 80. They might be worthwhile games but they don't stand a chance against other titles.
Then we've got these niche games - games that you'll either love or hate. They tend to get reviews that either skyrocket or sink like rocks. When you've got a bunch of reviews that show in the range of 40% and others up beyond 80% and compound them into an average you'll have this illusion of the game in fact just being a "Meh" experience of mediocricy, when it might in fact have been the most engrossing experience of the year for you.
I think the biggest problem with Metacritic, or the whole result of using numbers from various reviews and adding them together for an average score, is that niche games tend to simply dissapear among the countless mediocre games, with scores of 60-80. In the case of many of the mediocre games, those are the final numbers that most of the reviews ended up with, so the metacritic score average isn't all that far from what most of the reviewers had written. The games were simply flawed and unpolished, and could not reach up to that magic 80. They might be worthwhile games but they don't stand a chance against other titles. Then we've got these niche games - games that you'll either love or hate. They tend to get reviews that either skyrocket or sink like rocks. When you've got a bunch of reviews that show in the range of 40% and others up beyond 80% and compound them into an average you'll have this illusion of the game in fact just being a "Meh" experience of mediocricy, when it might in fact have been the most engrossing experience of the year for you.
Of course a Niche game will get "Meh" raiting since after all it will only appeal to a very limited group of players. Then you have to look past the score and actually read what is said in the review to realise what the game is about and why the reviewer gave it this score.
But tbh - niche games are very often limited and thus should not get top raitings. Im still trying to understand why ppl call it "niche" just because they failed to get any real following. Atlantica for example is a "niche" game with turn based combat but still it gets very high score and alot of players enjoy it. Its diffrent - but still FUN to play and the score shows it (gamers rating).
I see these scores fairly good, in my opinion. UO is the last one for example. Well, its an old, 2D, cluncky game, at ñeast that was the feeling I got when I tried it. some vets may look at it with diferent eyes but for the most players its what it is. Take Aion or STO, they are low in the rank, it makes sense, this games has too few content or too much grinding, people talk about them everyday in these our forums. WoW the first one?, I dont play and dont like it, but it makes sense, 11 million players. I could go on and on but as I said, I see the scores right.
I actually think those scores are quite good give or take 10 points for personal preferences. I see no fault in them. Can you be so childish and think that the whole metacritic is BS just because your favorite game didn't score high? I happened to like Morrowind and Pirates of the Burning Sea even if they scored quite low. It is not the fault of the rating system, it is only my own quirkiness that I liked the games nevertheless.
I skate to where the puck is going to be, not where it has been-Wayne Gretzky
Well it is certainly not Metacritics fault, they just are passing on what gaming journalists are telling them.
I think using small sites is very important, we would not even hear about strategy or adventure games if it was up to the big sites.
I did not realise the scores were so important for the industry. It is endemic of where gaming is headed right now that anyone could take Metacritic data as statistical evidence. It is just the mean of a bunch of reviews, not even all taken from the same sites, or done by the same reviewers. That creates a lot of randomness.
I use Metacritic a lot, but never for MMO reviews, I invest to much time in a MMO not to do far more digging than just looking up the review scores.
Those scores don't seem so far off what I would have expected them to be tbh. This article seems to be more of a rant about metacritic being used within the industry as a gage of success or failure.
Originally posted by Scot Well it is certainly not Metacritics fault, they just are passing on what gaming journalists are telling them. I think using small sites is very important, we would not even hear about strategy or adventure games if it was up to the big sites. I did not realise the scores were so important for the industry. It is endemic of where gaming is headed right now that anyone could take Metacritic data as statistical evidence. It is just the mean of a bunch of reviews, not even all taken from the same sites, or done by the same reviewers. That creates a lot of randomness. I use Metacritic a lot, but never for MMO reviews, I invest to much time in a MMO not to do far more digging than just looking up the review scores.
I would agree with this and I wouldn't even blame the writers of the reviews themselves either. It's the people actually interpreting these reviews especially the ones that say "these are accurate" when a good portion of them appear way off.
Ultimately this as much as my own opinion saying the scores are inaccurate as it is theirs saying the scores are accurate. However, how many of these people that say that the scores are accurate actually have played the game? Experienced its end game? Spent more than 2 weeks playing most of those games?
It's easy to see how easy these impressions are, almost as easy as the writers of these journalists when in fact these reviews are not entirely accurate especially when MMORPG's are long term games but being reviewed as short term games, which the article is mainly pointing out.
I find it ironic that LOTRO and EQ2 are running next to each other in the scoring, that either speaks volumes of how bad lotro has gotton or how much better eq2 has made itself.
I fail to see what the issue here is, besides personal bias on the part of the author. Clearly he's mad about it - maybe something he doesn't like either scored too high, or something he did like too low. Or maybe even that a result feels too weighted vs the result given by a review here.
Only a few of the numbers on the list differ significantly from the reviews on this site, though. Most are within an acceptable range, considering the Meta scores are weighted averages. I fail to see how this makes them "wildly inaccurate" and how their scores are "broken when it comes to MMO's", especially considering a single review here is more likely to be biased than an average of several (as ghettobooste pointed out, Aion was given an 8.7 here...).
So anyway, let me guess... someone hates WoW and is angry they got the top spot, maybe? It wouldn't be the first time.
Heh i love posts from WoW fanatics like this one. I think the main reason such good articles get so many of these posts is because the author designed the layout of his article wrong. He put the review list at the top of his article, meaning while the WoW nerds begine reading the first line of the article there eyes automatically flick to the list and then they make assumptions therefore missing the articles main point of how the execs in companies like EA for example use the Metacritic score as word of god.
So yea i agree with the article that the Execs are making a big mistake in using metacritic like that when most of the reviews in metacritic are flawed by the fact that the reviewers are under pressure to get there review out before anyone else so tey can call it there 'Exclusive' review [i.e. first one out].
To make this clearer to the hard of reading among us i shall use an analogy as to why metacritic is fundimentally flawed [especially true for MMORPG's], First before the Analogy remember that a metacritic score is amde up of several reviews from multiple sources. My analogy is of how one such review is made when a game is new.
I am sitting in my office when my boss comes in carrying a Big Engineering book [about 500 pages] he gives me the book and says, "i got an urgent job for you this engineering procedure book has just come out and i want you to write a Review on it for the director of human resources". I go "ok i should have a full review within the we-", "sorry but te Director of human resources is getting a flee bath today he needs the report in 1 hour now do a good job because this book will be the basis for future engeneer training".
So i have one Hour to finish and hand in a review of a 500 page technical book which gives me 5 minuites to read it and 50 to write it and 5 to print and deliver it. So obviously my review will not be 100% accurate and probably only covers the first chapter of the book [i.e. the Introduction and index].
What i am basically saying in a long winded way is that most reviews especially the early ones are not accurate because the reviewers ahve to rush through the game in order to get in a early review otherwise known as the 'Exclusive'. And as metacritic scores are made from such reviews they are of course not reflective of the game [this is especially true of MMORPG's as the writer will not have time at the begining to play the whole thing and MMORPG's change over time anyway, some getting better or worse].
Another great example of Moore's Law. Give people access to that much space (developers and users alike) and they'll find uses for it that you can never imagine. "640K ought to be enough for anybody" - Bill Gates 1981
Those scores don't seem so far off what I would have expected them to be tbh. This article seems to be more of a rant about metacritic being used within the industry as a gage of success or failure.
Which is the whole point posting those scores is just the article writer desplaying the background to his main point of the way the Industry is using Metacritic.
Personally i like Metacritic not for its scores but as a source of reserch for my next considered gaming purachase [i.e. read the reviews].
Another great example of Moore's Law. Give people access to that much space (developers and users alike) and they'll find uses for it that you can never imagine. "640K ought to be enough for anybody" - Bill Gates 1981
Generally speaking I thought the article was interesting but really needed more specific facts and less 'feelings'.
If Metacritic chooses to use data from sites that only review one time and dip very shallow while reviewing because that is the data out there, so be it. It isn't up to them to change scores because it was 'only' someones first impression. And when you pull any information together from multiple locations with a disparate series of value systems, you have to 'do the best you can' with a master value system of your own.
If they use the same valuation system for all the scores across all the sites they pull from, then they are playing 'fair'.
Not sure why you feel these scores aren't right - they seem perfectly accurate to me.
While I wouldn't personally rate games like WoW as highly as that, I understand that as a game WoW has achieved a lot, and it's a pretty polished product.
UO, sadly, deserves to be at the bottom now.
And yea, games like Champions, which completely failed to impress me, seem to fit well in the middle.
I've learned to hate almost all review sites from my experience. I mean, the power an angry writer has whole procrastinating but having to finish this review by 10 this morning is immense, and they can abuse it. Take it from me; they sure can abuse it. Honestly, the only reviews I'll take seriously now days for purposes other than getting a quick Feel for the game are the Gametrailers ones. The final verdict alone means almost nothing to me, especially with MMOs. When looking for a new game usually I'll start with a screenshot, or the name, or genre, or something along the lines, then watch a video and maybe, after all that, read the review to see what kind of game it is. I made the mistake of reading the Bullet Witch review before watching the video, then I watched a video of its gameplay and was like, "OMG! This is like S4 but a tad more clunky, more grungy, but with spells and less trolls!"
People that put themselves above others put me in a bad mood. http://www.surrealtwilight.com/index.php ^Has nothing to do with that retarded Vampire Novel Series, I swear!^
In light of this article... Why isn't mmorpg.com a member of metacritc?
Wouldn't that help right the wrongs, as the OP sees it?
I've suggested mmorpg.com join metacritic before. Was even told by a staff member here they were looking into it. That was quite some time ago now. I guess they were either rejected, or it never happened.
Edit: Here is the link to the thread where I suggested mmorpg.com become a member of metacritc.
MikeB replied: "We are currently in the application process."
Does this article mean that the application was turned down?
I do agree with the point of the article, but I find it a bit disingenuous (or perhaps jealousy); when mmorpg.com is not even a member that helps contribute to the score used on metacritic.
What is stopping them from becoming a member?
Anyone?
I also think that an article of this subject matter should have included this link:
Very helpful to read through just how the scores are tabulated, and which publications are included in each category. In the end the article seems to come across as a bit of "sour grapes" that metacritic is given more weight within the industry than mmorpg.com. Perhaps that's just me reading between the lines a bit too much... but that's the perception the article gave me.
the only scores out of place is warhammer and ultima online so i fail to see your point about the scores. I do a agree they only review the first 3-10 hours of a game. No where near enough time to tell if the game will be good or not.
In my opinion the biggest reason why the scores shouldn't be trusted is that there is a hidden agenda giving good or bad reviews. A publishing company like Blizzard, Turbine, Codemasters and alike can make it worth a person or a site's trouble of writing a good review for them or even a bad review for others. I don't know how far it goes in this business but we all know that pharmaceutical companies pay large sums of money or give expensive gift and trips to doctors that write out their medicine instead of a rivaling companies medicine and I wonder if that also happens with MMO publishing companies.
I can see one perk that can easily be given and those are beta keys. Beta keys are a very hot item for new titles that are expected to be good and if you get to give beta keys away you will attract lots of people that want to play the game before anyone else and in the end a MMO review site wants readers because they are directly and indirectly (depending on the site) for revenue either from them if they want a 'premium' account or from ads that we all know and see. I expect that some companies will also reward the owner of a review site to place them above their main rival in the industry. So basically if you write a bad review for a MMO this year you will probably not get beta keys for the next MMO they bring out and that is not what you want so everyone scores 'good' and the ones that give more or which you can't get around score a little better.
Although there's a point in this column, I don't think it's the point the writer's trying to make...
Metacritic isn't the party at fault here. Their system works fine. The problem is that MMORPG's are, as the writer points out, difficult to review well.
But then again... isn't reviewing any game difficult? It's always hard to balance subjective and objective "facts", lots of reviewers don't play through entire single player games either, some review games within a genre they dislike, etc. etc.
So, to the writer: Yes, the scores on Metacritic can be (way) off, true. This has nothing to do with Metacritic though, it's down to the fact that reviews (of any kind, really) are simply not always correct.
What Metacritic does actually lessens this problem, because they offer an average, so the truly ridiculous should be compensated...
I got a strong feeling while reading this article that the writer has some personal beef with Metacritic, especially since he doesn't offer any ideas on how to improve the system, but simply says "you can't really review MMO's, so Metacritic sucks" which doesn't make a lot of sense....
Which leads me to discard his ideas the same as I, ironically, discard many reviews.
Is it just me, or do most of the MMORPG.com columnists seem to whine a lot? :P
I think you hit the nail on the head here.
Yeah, you got me. It does read a bit more whiney than I planned.
I really just wanted to show that Metacritic is an OK (but flawed at times) tool, and, because of it's inaccuracy, should not be used as an internal tool by publishers. You can have one, the other, but not both.
Although there's a point in this column, I don't think it's the point the writer's trying to make... Metacritic isn't the party at fault here. Their system works fine. The problem is that MMORPG's are, as the writer points out, difficult to review well. But then again... isn't reviewing any game difficult? It's always hard to balance subjective and objective "facts", lots of reviewers don't play through entire single player games either, some review games within a genre they dislike, etc. etc. So, to the writer: Yes, the scores on Metacritic can be (way) off, true. This has nothing to do with Metacritic though, it's down to the fact that reviews (of any kind, really) are simply not always correct. What Metacritic does actually lessens this problem, because they offer an average, so the truly ridiculous should be compensated... I got a strong feeling while reading this article that the writer has some personal beef with Metacritic, especially since he doesn't offer any ideas on how to improve the system, but simply says "you can't really review MMO's, so Metacritic sucks" which doesn't make a lot of sense.... Which leads me to discard his ideas the same as I, ironically, discard many reviews.
Well said.
I'm hoping that someone from mmorpg.com comments on what happened to the application process that mmorpg.com did for metacritic back in August of 2009. Surely they've heard back from them by now.... right?
This article does come off as bit of jealousy over metacritic's importance in the industry. As reventon pointed out above... metacritic simply takes an aggregate score from numerous sources.
THAT process should actually make for a much more universally agreed upon score than a single sites review. Why wasn't that point brought up in the article?
Is it just me, or do most of the MMORPG.com columnists seem to whine a lot? :P
I think you hit the nail on the head here.
Yeah, you got me. It does read a bit more whiney than I planned.
I really just wanted to show that Metacritic is an OK (but flawed at times) tool, and, because of it's inaccuracy, should not be used as an internal tool by publishers. You can have one, the other, but not both.
Note to self: less whiney next time.
Thanks for the comments
Any word on the application process that mmorpg.com did for metacritic back in August 2009?
Although there's a point in this column, I don't think it's the point the writer's trying to make... Metacritic isn't the party at fault here. Their system works fine. The problem is that MMORPG's are, as the writer points out, difficult to review well. But then again... isn't reviewing any game difficult? It's always hard to balance subjective and objective "facts", lots of reviewers don't play through entire single player games either, some review games within a genre they dislike, etc. etc. So, to the writer: Yes, the scores on Metacritic can be (way) off, true. This has nothing to do with Metacritic though, it's down to the fact that reviews (of any kind, really) are simply not always correct. What Metacritic does actually lessens this problem, because they offer an average, so the truly ridiculous should be compensated... I got a strong feeling while reading this article that the writer has some personal beef with Metacritic, especially since he doesn't offer any ideas on how to improve the system, but simply says "you can't really review MMO's, so Metacritic sucks" which doesn't make a lot of sense.... Which leads me to discard his ideas the same as I, ironically, discard many reviews.
There have been a couple of posts like this one. Which only goes to show that I didn't articulate my point very well. My bad. I'll try harder next week.
However, I'd like to stress that I don't have a problem with Metacritic. I use it all the time. It's a very useful consumer tool. I was being a bit dramatic to show that Metacritic can occassionally spit out some spurious scores. But, i f you know how it works, you can work around these scores, or just go to the review sites you trust, or ignore Metacritic altogether. No biggie.
What I really wanted to get across, and seemingly failed at, is that publishers use these scores in a very real sense. Bonuses at some studios are based on scores. Publishers use Metacritic scores in their earnings reports (which is mind boggling). As a working designer, I really don't want an aggregate review site helping determine how much compensation i get. And I really, really don't want Metacritic giving ammunition to big-multi-studio publishers looking to trim any fat.
So, sorry for being a bit confusing. And thanks to everyone who's left a comment.
Sorry, Justin. A lot of opinions are subjective. I don't like the Metacritic scores for every game there, but I have to say they do give a bit of a picture of what is going on.
First of all, WOW - yeah, it's the most popular and some might say most polished MMO. I hate it, but it IS the McDonalds of MMO's - billions served and all. Makes sense it has a top score.
WARHAMMER ONLINE I happen to love. Apparently a lot of other reviewers did, too. Sure, the game lost subs - doesn't mean its a bad game. I still sub to WAR and I have high hopes it will once again achieve its former glory. I can say that I think the PvP in WAR is the best out there.
The PvP in the two games this site LOVES to promote - Aion and EVE - I happen to *hate*. I find EVE to be about as much fun as working on an Excel spreadsheet, and Aion just isn't my cup of tea at all. You can barely read a staff article by, say, Jennings on this site without at least one plug for Aion or EVE - even his last article on STO (Good/Bad/Ugly) ended with a link to the EVE game, a move I found just unprofessional and ridiculous - really, this site needed another plug for EVE ONLINE? I think we get it - MMORPG.COM *loves* Aion and EVE. Got it. Doesn't need to be mentioned in every article or on articles for games unrelated to either.
Your comment about the "smaller sites" getting as much attention as the "big boys" (like, I assume, you see MMORPG.COM) - but frankly, sometimes the smaller sites do a better job reporting on the games. MMORPG.COM gets carried away in the hype and the staff's "favorite games" - again, which seem to be Aion and EVE. Had Aion and EVE been at the top of that list I have a feeling you would have been find with the Metacritic scores.
Now, one score I didn't agree with was Fallen Earth's score of 69. I loved that game. I notice Aion scored low - that's probably a good thing; I didn't like Aion and apparently a lot of other reviewers didn't, either.
I don't come to MMORPG.COM for staff reviews any more - you have people writing columns that are contracting out for game companies, there seems to be a huge bias for certain games here to the point where people can come into forums and bash one game ad nasuem but if you bash a "favorite" (EVE or Aion in particular) - watch out! At least, this is my perception here.
"Smaller" sites have valid opinions, too, and it's very condescending to assume/say that only the "big sites" are worth listening to, or that because MMORPG.COM is so huge, its review should be given more weight. Why? You guys rated Aion highly; I hated it. I have been finding the reviews and news over at massively.com to be more even handed, actually. You guys weren't so fond of Fallen Earth here, and I love it. And so on.
I don't know, this staff article came off as preachy and condescending. We get it. Only the big boys matter, smaller review sites don't have valid opinions, and the metacritic scores ticked you off because Aion and EVE weren't at the top of the list, right?
Although there's a point in this column, I don't think it's the point the writer's trying to make... Metacritic isn't the party at fault here. Their system works fine. The problem is that MMORPG's are, as the writer points out, difficult to review well. But then again... isn't reviewing any game difficult? It's always hard to balance subjective and objective "facts", lots of reviewers don't play through entire single player games either, some review games within a genre they dislike, etc. etc. So, to the writer: Yes, the scores on Metacritic can be (way) off, true. This has nothing to do with Metacritic though, it's down to the fact that reviews (of any kind, really) are simply not always correct. What Metacritic does actually lessens this problem, because they offer an average, so the truly ridiculous should be compensated... I got a strong feeling while reading this article that the writer has some personal beef with Metacritic, especially since he doesn't offer any ideas on how to improve the system, but simply says "you can't really review MMO's, so Metacritic sucks" which doesn't make a lot of sense.... Which leads me to discard his ideas the same as I, ironically, discard many reviews.
There have been a couple of posts like this one. Which only goes to show that I didn't articulate my point very well. My bad. I'll try harder next week.
However, I'd like to stress that I don't have a problem with Metacritic. I use it all the time. It's a very useful consumer tool. I was being a bit dramatic to show that Metacritic can occassionally spit out some spurious scores. But, i f you know how it works, you can work around these scores, or just go to the review sites you trust, or ignore Metacritic altogether. No biggie.
What I really wanted to get across, and seemingly failed at, is that publishers use these scores in a very real sense. Bonuses at some studios are based on scores. Publishers use Metacritic scores in their earnings reports (which is mind boggling). As a working designer, I really don't want an aggregate review site helping determine how much compensation i get. And I really, really don't want Metacritic giving ammunition to big-multi-studio publishers looking to trim any fat.
So, sorry for being a bit confusing. And thanks to everyone who's left a comment.
So, that totally does not come across in the article.
What is clear is that you do not like the final scoring that metacritic provides even though metacritic is really an aggregate of reviews from many places (except mmorpg.com I might add.) So, rather than focus on how mmo's are reviewed in general, the source of metacritics information, you opted to slam metacritic. And then, towards the end you decided to mention how the composite score is used within the industry as well. But, rather than say how industry shouldn't use Metacritic as a yardstick, you attack metacritic for, basically, existing.
Considering how most comments seem to reflect this, you probably need to a do a bit better than a limp apology, "sorry for being a bit confusing." Might I suggest that, in addition to writing the article, you write a separate summary of what you're aiming to communicate, then use an editor?
Comments
I think you hit the nail on the head here.
I think the biggest problem with Metacritic, or the whole result of using numbers from various reviews and adding them together for an average score, is that niche games tend to simply dissapear among the countless mediocre games, with scores of 60-80.
In the case of many of the mediocre games, those are the final numbers that most of the reviews ended up with, so the metacritic score average isn't all that far from what most of the reviewers had written. The games were simply flawed and unpolished, and could not reach up to that magic 80. They might be worthwhile games but they don't stand a chance against other titles.
Then we've got these niche games - games that you'll either love or hate. They tend to get reviews that either skyrocket or sink like rocks. When you've got a bunch of reviews that show in the range of 40% and others up beyond 80% and compound them into an average you'll have this illusion of the game in fact just being a "Meh" experience of mediocricy, when it might in fact have been the most engrossing experience of the year for you.
Of course a Niche game will get "Meh" raiting since after all it will only appeal to a very limited group of players. Then you have to look past the score and actually read what is said in the review to realise what the game is about and why the reviewer gave it this score.
But tbh - niche games are very often limited and thus should not get top raitings. Im still trying to understand why ppl call it "niche" just because they failed to get any real following. Atlantica for example is a "niche" game with turn based combat but still it gets very high score and alot of players enjoy it. Its diffrent - but still FUN to play and the score shows it (gamers rating).
I see these scores fairly good, in my opinion. UO is the last one for example. Well, its an old, 2D, cluncky game, at ñeast that was the feeling I got when I tried it. some vets may look at it with diferent eyes but for the most players its what it is. Take Aion or STO, they are low in the rank, it makes sense, this games has too few content or too much grinding, people talk about them everyday in these our forums. WoW the first one?, I dont play and dont like it, but it makes sense, 11 million players. I could go on and on but as I said, I see the scores right.
I actually think those scores are quite good give or take 10 points for personal preferences. I see no fault in them. Can you be so childish and think that the whole metacritic is BS just because your favorite game didn't score high? I happened to like Morrowind and Pirates of the Burning Sea even if they scored quite low. It is not the fault of the rating system, it is only my own quirkiness that I liked the games nevertheless.
I skate to where the puck is going to be, not where it has been -Wayne Gretzky
Well it is certainly not Metacritics fault, they just are passing on what gaming journalists are telling them.
I think using small sites is very important, we would not even hear about strategy or adventure games if it was up to the big sites.
I did not realise the scores were so important for the industry. It is endemic of where gaming is headed right now that anyone could take Metacritic data as statistical evidence. It is just the mean of a bunch of reviews, not even all taken from the same sites, or done by the same reviewers. That creates a lot of randomness.
I use Metacritic a lot, but never for MMO reviews, I invest to much time in a MMO not to do far more digging than just looking up the review scores.
Those scores don't seem so far off what I would have expected them to be tbh. This article seems to be more of a rant about metacritic being used within the industry as a gage of success or failure.
I would agree with this and I wouldn't even blame the writers of the reviews themselves either. It's the people actually interpreting these reviews especially the ones that say "these are accurate" when a good portion of them appear way off.
Ultimately this as much as my own opinion saying the scores are inaccurate as it is theirs saying the scores are accurate. However, how many of these people that say that the scores are accurate actually have played the game? Experienced its end game? Spent more than 2 weeks playing most of those games?
It's easy to see how easy these impressions are, almost as easy as the writers of these journalists when in fact these reviews are not entirely accurate especially when MMORPG's are long term games but being reviewed as short term games, which the article is mainly pointing out.
Oh, the irony of anyone working on this site complaining about metacritics rating structure.
you wanna see some really jacked up game ratings? Check out the ones on MMORPG.com
"If you want a picture of the future, imagine a robot foot stomping on a human face -- forever."
I find it ironic that LOTRO and EQ2 are running next to each other in the scoring, that either speaks volumes of how bad lotro has gotton or how much better eq2 has made itself.
Heh i love posts from WoW fanatics like this one. I think the main reason such good articles get so many of these posts is because the author designed the layout of his article wrong. He put the review list at the top of his article, meaning while the WoW nerds begine reading the first line of the article there eyes automatically flick to the list and then they make assumptions therefore missing the articles main point of how the execs in companies like EA for example use the Metacritic score as word of god.
So yea i agree with the article that the Execs are making a big mistake in using metacritic like that when most of the reviews in metacritic are flawed by the fact that the reviewers are under pressure to get there review out before anyone else so tey can call it there 'Exclusive' review [i.e. first one out].
To make this clearer to the hard of reading among us i shall use an analogy as to why metacritic is fundimentally flawed [especially true for MMORPG's], First before the Analogy remember that a metacritic score is amde up of several reviews from multiple sources. My analogy is of how one such review is made when a game is new.
I am sitting in my office when my boss comes in carrying a Big Engineering book [about 500 pages] he gives me the book and says, "i got an urgent job for you this engineering procedure book has just come out and i want you to write a Review on it for the director of human resources". I go "ok i should have a full review within the we-", "sorry but te Director of human resources is getting a flee bath today he needs the report in 1 hour now do a good job because this book will be the basis for future engeneer training".
So i have one Hour to finish and hand in a review of a 500 page technical book which gives me 5 minuites to read it and 50 to write it and 5 to print and deliver it. So obviously my review will not be 100% accurate and probably only covers the first chapter of the book [i.e. the Introduction and index].
What i am basically saying in a long winded way is that most reviews especially the early ones are not accurate because the reviewers ahve to rush through the game in order to get in a early review otherwise known as the 'Exclusive'. And as metacritic scores are made from such reviews they are of course not reflective of the game [this is especially true of MMORPG's as the writer will not have time at the begining to play the whole thing and MMORPG's change over time anyway, some getting better or worse].
Another great example of Moore's Law. Give people access to that much space (developers and users alike) and they'll find uses for it that you can never imagine. "640K ought to be enough for anybody" - Bill Gates 1981
Which is the whole point posting those scores is just the article writer desplaying the background to his main point of the way the Industry is using Metacritic.
Personally i like Metacritic not for its scores but as a source of reserch for my next considered gaming purachase [i.e. read the reviews].
Another great example of Moore's Law. Give people access to that much space (developers and users alike) and they'll find uses for it that you can never imagine. "640K ought to be enough for anybody" - Bill Gates 1981
Generally speaking I thought the article was interesting but really needed more specific facts and less 'feelings'.
If Metacritic chooses to use data from sites that only review one time and dip very shallow while reviewing because that is the data out there, so be it. It isn't up to them to change scores because it was 'only' someones first impression. And when you pull any information together from multiple locations with a disparate series of value systems, you have to 'do the best you can' with a master value system of your own.
If they use the same valuation system for all the scores across all the sites they pull from, then they are playing 'fair'.
Not sure why you feel these scores aren't right - they seem perfectly accurate to me.
While I wouldn't personally rate games like WoW as highly as that, I understand that as a game WoW has achieved a lot, and it's a pretty polished product.
UO, sadly, deserves to be at the bottom now.
And yea, games like Champions, which completely failed to impress me, seem to fit well in the middle.
I've learned to hate almost all review sites from my experience. I mean, the power an angry writer has whole procrastinating but having to finish this review by 10 this morning is immense, and they can abuse it. Take it from me; they sure can abuse it. Honestly, the only reviews I'll take seriously now days for purposes other than getting a quick Feel for the game are the Gametrailers ones. The final verdict alone means almost nothing to me, especially with MMOs. When looking for a new game usually I'll start with a screenshot, or the name, or genre, or something along the lines, then watch a video and maybe, after all that, read the review to see what kind of game it is. I made the mistake of reading the Bullet Witch review before watching the video, then I watched a video of its gameplay and was like, "OMG! This is like S4 but a tad more clunky, more grungy, but with spells and less trolls!"
People that put themselves above others put me in a bad mood.
http://www.surrealtwilight.com/index.php
^Has nothing to do with that retarded Vampire Novel Series, I swear!^
In light of this article... Why isn't mmorpg.com a member of metacritc?
Wouldn't that help right the wrongs, as the OP sees it?
I've suggested mmorpg.com join metacritic before. Was even told by a staff member here they were looking into it. That was quite some time ago now. I guess they were either rejected, or it never happened.
Edit: Here is the link to the thread where I suggested mmorpg.com become a member of metacritc.
MikeB replied: "We are currently in the application process."
www.mmorpg.com/discussion2.cfm/post/3027577#3027577
Does this article mean that the application was turned down?
I do agree with the point of the article, but I find it a bit disingenuous (or perhaps jealousy); when mmorpg.com is not even a member that helps contribute to the score used on metacritic.
What is stopping them from becoming a member?
Anyone?
I also think that an article of this subject matter should have included this link:
www.metacritic.com/about/scoring.shtml
Very helpful to read through just how the scores are tabulated, and which publications are included in each category. In the end the article seems to come across as a bit of "sour grapes" that metacritic is given more weight within the industry than mmorpg.com. Perhaps that's just me reading between the lines a bit too much... but that's the perception the article gave me.
the only scores out of place is warhammer and ultima online so i fail to see your point about the scores. I do a agree they only review the first 3-10 hours of a game. No where near enough time to tell if the game will be good or not.
In my opinion the biggest reason why the scores shouldn't be trusted is that there is a hidden agenda giving good or bad reviews. A publishing company like Blizzard, Turbine, Codemasters and alike can make it worth a person or a site's trouble of writing a good review for them or even a bad review for others. I don't know how far it goes in this business but we all know that pharmaceutical companies pay large sums of money or give expensive gift and trips to doctors that write out their medicine instead of a rivaling companies medicine and I wonder if that also happens with MMO publishing companies.
I can see one perk that can easily be given and those are beta keys. Beta keys are a very hot item for new titles that are expected to be good and if you get to give beta keys away you will attract lots of people that want to play the game before anyone else and in the end a MMO review site wants readers because they are directly and indirectly (depending on the site) for revenue either from them if they want a 'premium' account or from ads that we all know and see. I expect that some companies will also reward the owner of a review site to place them above their main rival in the industry. So basically if you write a bad review for a MMO this year you will probably not get beta keys for the next MMO they bring out and that is not what you want so everyone scores 'good' and the ones that give more or which you can't get around score a little better.
Although there's a point in this column, I don't think it's the point the writer's trying to make...
Metacritic isn't the party at fault here. Their system works fine. The problem is that MMORPG's are, as the writer points out, difficult to review well.
But then again... isn't reviewing any game difficult? It's always hard to balance subjective and objective "facts", lots of reviewers don't play through entire single player games either, some review games within a genre they dislike, etc. etc.
So, to the writer: Yes, the scores on Metacritic can be (way) off, true. This has nothing to do with Metacritic though, it's down to the fact that reviews (of any kind, really) are simply not always correct.
What Metacritic does actually lessens this problem, because they offer an average, so the truly ridiculous should be compensated...
I got a strong feeling while reading this article that the writer has some personal beef with Metacritic, especially since he doesn't offer any ideas on how to improve the system, but simply says "you can't really review MMO's, so Metacritic sucks" which doesn't make a lot of sense....
Which leads me to discard his ideas the same as I, ironically, discard many reviews.
I think you hit the nail on the head here.
Yeah, you got me. It does read a bit more whiney than I planned.
I really just wanted to show that Metacritic is an OK (but flawed at times) tool, and, because of it's inaccuracy, should not be used as an internal tool by publishers. You can have one, the other, but not both.
Note to self: less whiney next time.
Thanks for the comments
Well said.
I'm hoping that someone from mmorpg.com comments on what happened to the application process that mmorpg.com did for metacritic back in August of 2009. Surely they've heard back from them by now.... right?
www.mmorpg.com/discussion2.cfm/post/3027577#3027577
This article does come off as bit of jealousy over metacritic's importance in the industry. As reventon pointed out above... metacritic simply takes an aggregate score from numerous sources.
THAT process should actually make for a much more universally agreed upon score than a single sites review. Why wasn't that point brought up in the article?
I think you hit the nail on the head here.
Yeah, you got me. It does read a bit more whiney than I planned.
I really just wanted to show that Metacritic is an OK (but flawed at times) tool, and, because of it's inaccuracy, should not be used as an internal tool by publishers. You can have one, the other, but not both.
Note to self: less whiney next time.
Thanks for the comments
Any word on the application process that mmorpg.com did for metacritic back in August 2009?
www.mmorpg.com/discussion2.cfm/post/3027577#3027577
There have been a couple of posts like this one. Which only goes to show that I didn't articulate my point very well. My bad. I'll try harder next week.
However, I'd like to stress that I don't have a problem with Metacritic. I use it all the time. It's a very useful consumer tool. I was being a bit dramatic to show that Metacritic can occassionally spit out some spurious scores. But, i f you know how it works, you can work around these scores, or just go to the review sites you trust, or ignore Metacritic altogether. No biggie.
What I really wanted to get across, and seemingly failed at, is that publishers use these scores in a very real sense. Bonuses at some studios are based on scores. Publishers use Metacritic scores in their earnings reports (which is mind boggling). As a working designer, I really don't want an aggregate review site helping determine how much compensation i get. And I really, really don't want Metacritic giving ammunition to big-multi-studio publishers looking to trim any fat.
So, sorry for being a bit confusing. And thanks to everyone who's left a comment.
Sorry, Justin. A lot of opinions are subjective. I don't like the Metacritic scores for every game there, but I have to say they do give a bit of a picture of what is going on.
First of all, WOW - yeah, it's the most popular and some might say most polished MMO. I hate it, but it IS the McDonalds of MMO's - billions served and all. Makes sense it has a top score.
WARHAMMER ONLINE I happen to love. Apparently a lot of other reviewers did, too. Sure, the game lost subs - doesn't mean its a bad game. I still sub to WAR and I have high hopes it will once again achieve its former glory. I can say that I think the PvP in WAR is the best out there.
The PvP in the two games this site LOVES to promote - Aion and EVE - I happen to *hate*. I find EVE to be about as much fun as working on an Excel spreadsheet, and Aion just isn't my cup of tea at all. You can barely read a staff article by, say, Jennings on this site without at least one plug for Aion or EVE - even his last article on STO (Good/Bad/Ugly) ended with a link to the EVE game, a move I found just unprofessional and ridiculous - really, this site needed another plug for EVE ONLINE? I think we get it - MMORPG.COM *loves* Aion and EVE. Got it. Doesn't need to be mentioned in every article or on articles for games unrelated to either.
Your comment about the "smaller sites" getting as much attention as the "big boys" (like, I assume, you see MMORPG.COM) - but frankly, sometimes the smaller sites do a better job reporting on the games. MMORPG.COM gets carried away in the hype and the staff's "favorite games" - again, which seem to be Aion and EVE. Had Aion and EVE been at the top of that list I have a feeling you would have been find with the Metacritic scores.
Now, one score I didn't agree with was Fallen Earth's score of 69. I loved that game. I notice Aion scored low - that's probably a good thing; I didn't like Aion and apparently a lot of other reviewers didn't, either.
I don't come to MMORPG.COM for staff reviews any more - you have people writing columns that are contracting out for game companies, there seems to be a huge bias for certain games here to the point where people can come into forums and bash one game ad nasuem but if you bash a "favorite" (EVE or Aion in particular) - watch out! At least, this is my perception here.
"Smaller" sites have valid opinions, too, and it's very condescending to assume/say that only the "big sites" are worth listening to, or that because MMORPG.COM is so huge, its review should be given more weight. Why? You guys rated Aion highly; I hated it. I have been finding the reviews and news over at massively.com to be more even handed, actually. You guys weren't so fond of Fallen Earth here, and I love it. And so on.
I don't know, this staff article came off as preachy and condescending. We get it. Only the big boys matter, smaller review sites don't have valid opinions, and the metacritic scores ticked you off because Aion and EVE weren't at the top of the list, right?
There have been a couple of posts like this one. Which only goes to show that I didn't articulate my point very well. My bad. I'll try harder next week.
However, I'd like to stress that I don't have a problem with Metacritic. I use it all the time. It's a very useful consumer tool. I was being a bit dramatic to show that Metacritic can occassionally spit out some spurious scores. But, i f you know how it works, you can work around these scores, or just go to the review sites you trust, or ignore Metacritic altogether. No biggie.
What I really wanted to get across, and seemingly failed at, is that publishers use these scores in a very real sense. Bonuses at some studios are based on scores. Publishers use Metacritic scores in their earnings reports (which is mind boggling). As a working designer, I really don't want an aggregate review site helping determine how much compensation i get. And I really, really don't want Metacritic giving ammunition to big-multi-studio publishers looking to trim any fat.
So, sorry for being a bit confusing. And thanks to everyone who's left a comment.
So, that totally does not come across in the article.
What is clear is that you do not like the final scoring that metacritic provides even though metacritic is really an aggregate of reviews from many places (except mmorpg.com I might add.) So, rather than focus on how mmo's are reviewed in general, the source of metacritics information, you opted to slam metacritic. And then, towards the end you decided to mention how the composite score is used within the industry as well. But, rather than say how industry shouldn't use Metacritic as a yardstick, you attack metacritic for, basically, existing.
Considering how most comments seem to reflect this, you probably need to a do a bit better than a limp apology, "sorry for being a bit confusing." Might I suggest that, in addition to writing the article, you write a separate summary of what you're aiming to communicate, then use an editor?