A scoring system similar to what one tv show does with movies would probably work best for me. The movie scores are "buy", "rent", or "pass". With MMOs it could be "buy", "free trial", or "pass". A number score doesn't really tell you how the reviewer really feels about the game. I have read numerous reviews for games where I got to the end and thought "wow this reviewer really like/hated this game" then I got to the score and thought it was the complete opposite.
Basically a buy rating would mean the reviewer was really enjoying the time spent in game. This should mean it is not overly buggy and has plenty of interesting things to do from the reviewers point of view.
A free trial rating would mean it seemed like a cool idea but maybe was a little too buggy or didn't quite fit the reviewers playstyle.
Pass would mean the game seemed like a bug ridden pile of contentless garbage.
As a previous poster pointed out it would be nice to have a bio on each reviewer stating at the very least what three games he/she has spent the most time playing as well as what the reviewer is currently playing at time of review. Including prefered playstyle may also be nice as in crafter, pve, pvp, solo, group, casual, etc.
I haven't read a game review since YouTube was invented. I watch gameplay videos on YouTube and decide if I'll try the game or not. No review I've ever read has given me more insight about a game than watching a 5 minute gameplay video has given me.
We live in the Web 2.0 era, you're behind the times already only having written reviews. If you really want to distingush yourself from the rest of the gaming sites on the web than video reviews is the way to go. Sure, some sites are already doing this, like Game Trailers, but they aren't MMO specific, and honestly, the reviews I've seen on there aren't great.
As far as the actual grading/reviewing process I would recommend a combination of a lot of what I've seen here.
First thing to do would be to get rid of any grading scale at all. You can still have ratings (because the majority of people need that), but make it very simple. I would suggest a "Buy", "Try" or "Skip" rating scale. Self explanitory, no room for confusion.
Second the review needs to be 3 parts, as many have suggested. Start with a first impressions on day of release, or a week or so later.
Two weeks later, more indepth information on how the game plays, technical information, etc.
Then after 30 days give the final review with the Buy, Try, or Skip rating attached at the end. This would be a review of parts 1 and 2 + additional information like progression info, griding scale, etc.
And of course, ALL IN VIDEO FORMAT!
Edit: The reason I suggest a 3 part review is because if you do one on day 1, and then one on day 30, people will have lost interest. Having a shorter time span between each part keeps people's attention.
I would actually watch the reviews on here if they were done like this!
Ok. Now I like this idea. This makes perfect sense
Take a look at the Reviews tab. All the games are rated in the six to eight range with a couple of fives.
What good is using a 10 point scale when you never use one two three four nine and ten??? Some of the games rated in the seven range are either mediocre or okay. The eight-point range seems to describe games that are good to excellent. IMO this is a huge disparity from seven to eight.
Bottom Line: Rating games using this system takes away the credibility of the entire MMORPG reviewing system. It's actually laughable with some people I've talked to. "I wonder Rift will score at MMORPG...OMGWTFBBQ!!! It's an eight! Shocker!" It's almost as if the scale is set up to be intentionally vague so as not to piss off any potential advertisers.
If you're only going to use four numbers to rate a game then use a four-star type system please! And no half stars either.
1 star - Represents a game that is in serious disarray. Missing key components and has unplayable sections. Should never have been released.
2 stars - Represents a game that has some redeeming qualities but should only be played by hard-core fans of the IP. The game is lacking in content and is buggy. A game that has potential after a nice revamp and series of patches.
3 stars - This level will represent games that excel in their particular genre. The game should have been release worthy and released with a nice amount of content and should contain stable code. Although the game will have some minor flaws and perhaps lack innovation in certain areas.
4 stars - A must play MMO! For a MMO to receive four stars it should be an industry leader in its genre and contain industry innovations. This type of game would have a fantastic support staff and supporting website and features. It should be very well tested and virtually bug free (no game will ever be completely bug free of course) and the bugs that exist should be address as quickly as possible.
This folks, is a rating system that's believable. Not many games will receive a four, but if they do, you know they'll be great. Three stars means you should have confidence that you're playing a quality MMO. Two starts means that the game needs some work, but if you're a huge fan of the IP, you'll be able to ignore some of its faults. One star means that a game is garbage and MMORPG needs to not be afraid to say so. This type of honesty will lend credibility to your review system and staff of writers.
The current system is soft and I hope you consider a change.
How I would have rated some of the latest released MMO's:
Rift - 3 stars. A quality MMO with an interesting Rift system worthy of subs from theme park-style MMO fans. Still, not enough innovation to warrant 4 stars as much of the game play can be found in previous titles. (Wow-Clone...there I said it)
Earthrise - 1 star. Terrible playability. Seriously lacking in content. Clunky combat. Bugs galore. Old concepts rehashed poorly.
DCUO - 2 stars. Excellent character customization. Combat is fluid although I can seem like a console button-masher at times. Great world design does justice to the IP. Bugs are still prevalent. The chat interface is awful. Lack of economy and proper functioning AH takes away from the MMO feel of the game and makes it more of a console experience. Potential is there for a 3-star rating at the 6 month re-review.
Perpetuum - 2 stars. Finally, a release worthy MMO just in time for 2011! Excellent base game mechanics and original ideas although some of the mechanics will be readily familiar to Eve Online players. Shady internet connections to the server. Buggy and incomplete systems. Completely unoptomized for performance. Great MMO for fans that want to stick it out. Perpetuum has 3-star potential in a year or two as it will take time for the small independent developers to make the necessary improvements rapidly.
Stick to the 10 Point scale. As you say, it works well with MetaCritic and you want your reviews to appear over there as well. But [1] enforce consistency and assign each number with a [2] proper meaning, which you should also [3] spell out right next to the number or by providing a link to a rating policy page, where you may also conveniently explain your criteria. The reviewer should first have a verdict, only then translate it to the scale. Here is a take with keeping in mind that 5 should be the true average.
10 — Perfect
09 — Excellent
08 — Very Good
07 — Good
06 — Above Average
05 — Average
04 — Below Average
03 — Bad
02 — Atrocious
01 — Abysmal
Categories
You could use categories and rate on each separately and weight the categories differently, making Character Creation more or less important than something else, depenend on your criteria. These weighted scores are then aggregated into the final score which should line up with the verdict as listed above, when your criteria and weights are set correctly that is. You only need to setup an Excel table and it would be easy for your staff to use it. I won't overdo the categories though. A few would be enough.
Character Creation: breadth (many options), depth (option has many settings)
Usability: from installing to playing, how convenient and smooth does it feel like. Is it intuitive, clunky, responsive ...
Sophistication: in lieu of a better word: is it ripe, how many and how severe are bugs, how feature complete is it?
Core Game: there is a play pattern at the heart of any MMO that is repeated over and over, mostly combat wrapped in quests. How is the quality of that? Too many skills to use? It is too repetetive? Does it grow old quickly?
Scope: breath and depth are adequate terms to describe what to expect as well. Does it offer more than other games on release? Does it have few large and awesome features? Is it more a collection of many minigames? Has it enough PVP and PVP. What about crafting?
Balance: There are many kinds of balances, not just between classes. How about the ratio between action and downtimes. What about difficulty on your level? Can you outheal the opponent? Are other classes so much easier to play?
Presentation: I'd lump graphics and sound into that one, and other production values. Be careful with rating graphics merely from a technical point of view. MMOs are geared at large audiences and state-of-the-art is usually not the priority. Where it ends up depends on many factors, some deliberate (maybe the game is aimed at an older demographic expected to have more outdated machines) and some more or less coincidential (development time, rule of thumb etc.). It would be unfair to rate down a game for a certain deliberate strategy. Rather, look at art style, concistency, quality in the substance (lighting, atmosphere, personality etc.) and the like.
Innovation: I am bit hesistant with this one. I think it is overrated. A genre simply has certain genre characteristics and the community and some reviewers on this site places too much emphasis on it. You would have to find out whether its bad to have the basic DikuMUD Gameplay and if different approaches should yield better scores. Every game is different and most games have their own take on known features at least. So I remain skeptical with this.
Fun: The game could totally suck at anything and still be super fun. This might be the Joker card that influences the final score the most.
Social: I feel this should show up somewhere as well, but don't judge whether most players of this game are douche, but at how game features condense in the community. Keep in mind that drama is good in some games and detrimental in others. Also, some things are really opinions, like if mandatory grouping is a good thing. Remain impartial on that one. But if grouping is strongly discouraged, dungeon finder teleporting makes it so that you never see anyone anymore and the like, there are too many player hubs so there is none, these are reasons to consider. Also did the devs introduce rally points (factions, alliances, guild features) to identify with, and does it pan out well? Do the emotes have a positive impact?
Target Players
You have a box with the Pros and Cons, which I think is great. You could expand this to recommend it to a certain player type, you think this game caters to the most. Examples:
Comparison
Players who liked Game X and Game Y may also like this Game.
Bartle-Class Recommendations
They are typically depicted using the playing cards symbols which is also an excuse to make it look nice: Explorers (♠), Killers (♣), Socializers (♥), Achievers (♦). These are motivations and every player has them in varying degrees.You could use them to describe where the game is strong at. Lot's of PVP and very twitch based? Then Killers should look into it. Or is the game very item centric with a huge achievement system? Is it an open-world game or very instanced? This pretty much influences to whose motivations the game will cater to.
Nick Yee's Classification
It's bit differently and essencially three-sided. Achievement, Social and Immersion with some subcategories. You could combine it with the rating suggested above, or very quickly indicate strong points and weak points by just pointing out the words, thus complementing the Pros and Cons. You should keep this standardized and place a longer overview page behind the words. Please note that you do not rate the game itself, but how you assume the game will cater to players who are motivated by certain factors. E.g, Game X: "Advancement, Teamwork, Competition"
MMO (Sub) Subgenre / Distinctions
As anyone and his mom knows there are several traditions of MMORPGs and they can make the gameplay dramatically different. The two broad categories are of course the Sandbox vs Themepark distinction, actually rather Simulation vs. Game. Themepark actually means a game world that is based on attractions in well-crafted thematically different areas (think Magic Kingdom style, "Adventureland", "Tomorrowland" etc.). This is of course the hallmark of World of Warcraft, which seems to be modelled fairly close to original Disneyland (no surprise since their Irvine office is only a tossed dwarf away). Games that are more sandbox-y, that is simulationist, are rather based on emergent gameplay, Ultima Online, EVE and the like. In the former developer control is emphazised, player control in the latter.
Another one would be Open-World vs. Lobby/Tunnel . Open-World games have a more or less freely accessible game (over) world, where you may find mobs of your level range in corners of many different zones. Games with a lobby are almost entirely instanced and the instances may be rather linear tunnels to fight till the end boss. Sandbox games are usually open-world as well, but the distinctions should not be confused (WOW is open world as well).
The next distinction, again extreme ends, would be Twitch vs Turn. One places heavy emphasis on the players inherent abilities like hand-eye coordination (twitch) so that you cannot be a master archer if you aren't in real life, so to speak. The other extreme are turn based games or MMORPGs where the old Wargaming roots show through and only your strategical/tactical choices matter, so you can be master archer even as real life paraplegic, because your character sheet says you are (this is also at the heart of "What means RPG", which is not speaking with a pirate accent all day).
Most MMORPGs are actually somewhere in the middle, where some reflexes are required even in barely concealed quasi turn based games on one end, and the newer action-oriented MMORPGs which still resemble DikuMUD combat, but feature a more free-form targeting and faster gameplay. Essentially, Boring vs. Carpal Tunnel Syndrome.
Anyhow, the core characteristcs could be summed up as well, getting rid of comments that somehow demand the opposite type of gameplay.
The Review
Generally, many reviews I've read had too much focus on the reviewer, what the person personally likes and dislikes rather than strongly reflected on the game itself (wait. I get to that). It often read like apologizing in advance or dodging bullets that commentors may shoot. Personal preferences could be outsourced to introduction / profile pages of the reviewer, possibly adhering to some standards, like the classifications mentioned above. So you don't want to read too much of "I am personally fine with Kill Ten Rat Quests, but Jade Doe may not like this." The key information here would be that the game is full of these quests and the reader will know what to make of it. Likewise, I don't want the reviewer to point out that players who don' like PVP will not like this game. I want them to point out that PVP is mandatory or that there is little else beside it. If things are merely described, keep it tight. If you analyze, stay away from "No Shit, Sherlock?" territory. If you offer an opinion, it should be "reasonable". Generally, I have no qualms with that one, usually seems to be the case..
I miss a certain standard of journalism in gaming and I guess games are not considered a matured medium, unless game journalism is more mature as well. It could also be more unexpected at times, like Gonzo Journalism in Middle Earth. You could also show more Genre Savvy-ness at times. A word or two could be said about all the corny clichés that reappear in any new next-gen incarnation. It's boring to poke fun at Elves and Orcs per se, why not go step further and make fun of all the ridiculous "conventions". I already gave you some links to start looking (but be warned, it will ruin your life).
To be honest, I don't really have a single problem with the reviews here. I think you guys are pretty clear and concise with your scoring system, the icon-system for quick-viewing, the labels you give different kinds of reviews.
In my opinion, any mature adult that uses reviews as deciding factor of whether they buy a game is being foolish.
I use reviews to make a more informed decision. So I tend to look for mostly facts in MMORPG.com's reviews, and so far I'm happy enough with that information.
What I want from a review is a more indepth discussion of the information I can read on the developer's FAQ. Whether the reviewer likes a certain feature or not is beside the point, so as long as you guys and girls are giving accurate descriptions of the features in a game, then I'm a happy reader.
One off-topic thing, though, not related to reviews so much: please stop using GamerTube for your vid streams. The quality is just awful!!
The reason this is brought up, is because of nearly every reason mentioned here. But the fundemental reason, is you don't have a set-standard of criteria.
No matter what scale, or rating system you use, won't matter if the criteria is different for each review. So that goes with YOU setting the stage, & holding these game companies to those standards.
However close we (the MMORPG.com community) agree on those standards (yours), doesn't matter. Only that YOUR criteria is consistant, then readers can weigh their bias based on deltas known to differ from YOUR opinion.
Lastly, it is quite clear this website has grown rapidly & is not to professional at all. I'm seeing reviews from newbies (to me) that draw references to games mechanics, without knowing the said mechanic's origins, or it's depth of intracies.. or, how shallow they are represented in said game. It hurts to read so much ignorance.
Then, we are barraged with glitzy & flashy banner adds aimed at 12 year old girls & teen boys..? What does half of that garbage have to do with roleplaying games? Their just MMO's..
Start a freak'n sister site MMO.com and put all that^ kiddie stuff over there and turn this site around.
Adults don't need hype, we want reviews that cuts the core out of developers & makes them fess up to their vision, not cash deals that seem oh so apperent. We want articles that are pages deep and throught with adventure & details.
Or, how about well written Chronologies (ie: roleplayed) that are month-long journeys into the classic MMORPGs. Some type of feature set, so that young & old... all readers can learn about the history of MMORPG's. The progeny and how their built, & how IT and server farms make them work...
educate these kids... their completly lost and are looking for a new game and cannot express themselves. They need to have a knowlege base of something more than World of Warcraft.
Maybe then, YOUR reviews will be heralded as tag lines on the cover of Magazines, instead of chatter in teenage online chat rooms.
"No they are not charity. That is where the whales come in. (I play for free. Whales pays.) Devs get a business. That is how it works."
I prefer knowing the categories. I couldn't care less that a game is an 8 as much as I care where the game failed to not make it a 10 and where it was well made to achieve an 8 in the first place. I like it when a reviewer grades the categories seperate and if needed gives an average.
On that note though, I don't see why all categories must be weighted the same. Sure I'd rather have great gameplay then say have great music (when I'll usually turn it off and play my own regardless).
I think Stormwatch has a great idea (read up for his post) but may be a bit more time consuming. Although in an ideal world I'd go with his approach or one close to it.
Bro you are playing LOTRO like it owes you money - Grunt187 Anticipation : TERA
Have as many previews as you want, just make it very clear that’s what they are. Then your launch review on or very close to the first day of launch. Having any review after this encourages MMO companies to launch before they are ready, so don’t do one! If say six months to a year down the line a major content update occurs, just do what you normally do and have a review of the update.
In my humble opinion. One to ten is allready representative of the letter system, anything below a six is failure. (10A+)(9A)(8B)(7C)(6D)(5F). Obviously a game should be judged as a whole, rather than fully in certain areas. Leaving the judgemant open for interpretation is sort of ridiculous. Now a lot of people who lack self discipline are turned hostile so.
(Elaboration) I haven't played the game, the information gathered by the first hand account i.e. the reviewer is his alone. I can never truly make a verdict. It is my theory that the burden rests on the shoulders of the first hand acountant. Reading about it gives me an idea of the game, but never the experience. Good critiques will give good judgemants and ones that I would trust over my own. That's why I'm the guy reading the review, not the other way around.
Cuting the lower portion of the scale out because no ones been their, is like cutting the top out because no ones climbed the mountain. Which further confuses me about the scaleing system, scales are useful and they need to be somewhat accurate. Their are deffinately some failing games out their, why they don't deserve a fail grade IDK. Saying something is a failure isn't cruel and I disagree with the idea that games don't fail. They do and if you don't think so, I don't want to argue the sky blue.
Summary gradeing should cut a game into pieces. Each piece being portioned equally combined together would be the score. 1-10 is perfect and a mathmatical wonder seeing as it manifests in many things.
Know what grinds my gears is grinding my gears. For gods sake video games are video games. If your not drawn in by a picture or even the first few minutes of gameplay who cares quit. Posting useless reviews on the internet it a waste of time when now most people can just watch game play videos. Cuts down all the reading to letting your eyes visually scan the game and see how it comes off to your eye site.
When did you start playing "old school" MMO's. World Of Warcraft?
The most imprtant thing you can do to improve your reviews is have actual players review the games. It is quickly apparent when reading any MMORPG.com review that the writer has barely played the game. They have geenrally only played superficially enough to get a flavor. A prime example is Bill Murphy. He writes some good stuff but its quickly apparent the man has never actually played a MMO other than on the surface. To really a play a MMO and understand one needs to play mroe than 20 levels. IMO one needs to play more than 50-100 days played to fully understand a game. But I know realistically that will not happen. But your reviews are written mainly by people who are only guessing at how games are actually played. Basically they should be called 1st impresion form a casual player.
2. An even point system does not work
It is nice and pretty but an even point system does not work. One cannot weigh different aspects like combat and roleplay for example evenly. You cannot break it down and it add up like that. The best thing IMO would be review the game seperately for many different qualities that actually matter to players such as Fun factor, completeness, polish, quality of content, endgame content, fluidity of design, innovation, raiding, group dungeons, combat mechanics, crafting economy, PvP, Community building tools. Plus an overall score based on the feeling the reviewer has, not a simple summation fo the parts as everything is simply not equal. People look for different hings in a game. Some cry for innovation while otehrs will be simply happy for good endgame content.
Keep it numerical so it's clear how averages are computed.
Give each category a standard weight
Allow registered users to determine their own weights.
Average game score is computed dynamically based on the weights.
So for example gameplay has a standard weight of 3 and sound a standard weight of 1. RndUsr is always listening to music anyway so in his preferences he sets the weight of sound to 0.
RndUsr visits this site without being logged in and when computing the average score the website counts gameplay 3 times and sound once for computing the average. Now RndUsr logs in, because his settings changed the next score computed doesn't count sound at all. Causing games with a high soundscore to actually decrease in average rating ( for him. ) and games with low soundscores to increase in average rating. Because for RndUsr sound just isn't important.
Same thing could be done for PvPscores and such. If you absolutely need PvP then set the weigth for that score very high. If you don't care about PvP set it at zero.
This way the average score is actually determined by the reader himself, he decides which categories are important and which aren't.
Only downside would be that you'd probably have to add a few more categories.
We are the bunny. Resistance is futile. ''/\/\'''''/\/\''''''/\/\ ( o.o) ( o.o) ( o.o) (")("),,(")("),(")(")
Know what grinds my gears is grinding my gears. For gods sake video games are video games. If your not drawn in by a picture or even the first few minutes of gameplay who cares quit. Posting useless reviews on the internet it a waste of time when now most people can just watch game play videos. Cuts down all the reading to letting your eyes visually scan the game and see how it comes off to your eye site.
This is a good point Vid Reviews allow a lot of information and impression. But I also think written reviews by mmorpg critics (well written!) with mmorpg punter's responses are also part of the process. Discussion, debate and disagreement are essential.
In addition one scoring system is only measuring one thing in time and in one thing in game/genre terms.
A score for genre (comparitative to similar games)
A score for production (gameplay, graphics, sound, service etc)
A personal, professional game critics score (you peeps)
Meta-scores across the game world/web AND/OR Player rating score for this website.
Finally a re-review assessment due to change in players numbers, developer patches and oc a lot more feedback from players.
Video Reviews would be a great addition, I agree. But I'm not sure we have the manpower for that undertaking. It's something we're toying with though, and is still in its infancy.
But folks, the real thing this column is about is whether you think we should alter the grading system for our games. Jon and I both would love to do away with scores completely. Let players be the ones to assign a score, and let us just write reviews without a number. But is that feasible? Or should we merely alter the scoring system? Make it a Grade Scale, make it 1-5, or make it an average with a revised set of categories.
That's the question we're really asking. What do you think?
Try to be excellent to everyone you meet. You never know what someone else has seen or endured.
The scale is fine. 1-10 works and it's about how everything in life is graded. It's similar to how our papers are graded in school. It's roughly how we refer to cute guys/girls. Numbers are fine but the numbers need to be approrpriate.
Average games need average ratings, so I have to say again. Move away from the rating curve and leeway.
Try pairing random reviewers together. Have two people independently review the game and combine their reviews, especially for the major reviews--what people are most interested in.
Spec'ing properly is a gateway drug. 12 Million People have been meter spammed in heroics.
But folks, the real thing this column is about is whether you think we should alter the grading system for our games. Jon and I both would love to do away with scores completely. Let players be the ones to assign a score, and let us just write reviews without a number.
+1 for doing away with the grading system. It has simply lost it's meaning and most games always end up between 7.5 to 8.5, with this alone it's simply too hard to tell if the game is truly good or not because there is never a score that will stand out from the rest. It also, sometimes, seems simply inconsistent with the reviews hurting a bit the credibility of either the reviewer or the website all together. Would also help avoid the typical reaction of "Game got a high score when it sucks, MMORPG.com are being bought!". Members should also be made aware that reviews are suggestives and that it is simply impossible to make unbiased reviews (as seen in this thread, it's not very clear for some which leads to more typical reaction).
Doing away with it would be perfect. As for user-based scoring system, as I mentionned in my first post, I'd prefer users be able to simply choose options like "Was this review helpful [yes] [no]" with the possibility to write a commentary (similar to the user-generated game Ratings) . With MMORPG.com's current community, you'd see ridiculous scores of 3-5 for games that are actually good, but hated deeply by some disgruntled gamers, in which case the scoring system would be yet again simply useless (Wouldn't be suprised if WoW were to drop below 0 on this website should this ever happen ).
I'd like to see each review broken down into scores by headings (numeric, % based) and reasoning given for it. If you like, at the end give a couple of paragrphs over to the "feel" of the game (that nebulous, subjective notion) and the game an overall grade based on performance & feel, that could be your A / B / C / D grade.
What the reviews seem to suffer from most here is lack of an overall structure. It allows forumgoers too much leeway for poking holes in the reviews "You didn't even mention the glorious sounds / buggy animations / hellacious lag", whatever.
If all reviewers are working off the same template, the review process is rendered more transparent and less liable to someone proclaiming a review "fail" because something got left out, or claiming that you were paid to mention well-worked out feature A while ignoring teeth-grindingly awful problem Y.
If you use a combination like the one above (or one suggested by others in thread) people can see the scores, and then read through the "opinion" part to better understand why the game recieved the grade it did.
I understand completely the pressure to have a review out asap at launch, but I think it would be a good plan to revisit each game (particularly AAA titles) maybe six months / a year down the road and see whether it has improved / stagnated over that time.
I know, speaking for myself, I can be very hyped up for a game through beta and launch, only to walk away a month or two later because there was something in the later game that I couldn't stomach. Revisiting would mean less chance of an earlier rave review coming back to bite you on the behind - or vice versa.
I do believe a number system should be used,but you need to understand what the gaming industry is capable of doing before weighing in with numbers.
One example would be in the use of textures or model poly count.I have heard time and again how the dev is trying to cater to more people,so one needs to mention that in the rating,don't just hand out 9/10 for low end qaulity because the dev was aiming for that.
There was something missing from the OP,and i feel it very important to mention it.The problem with MANY reviews is they"appear" to be biased or more of a marketing campaign.Most readers are not stupid,they can see where a post is going or coming from.I am not saying this is the case for ALL posts/reviews i am just saying there is at times the appearance of being a skewed review.
SO i think before a review is posted,the writer needs to read it over a few times and determine if it looks skewed in the eyes of the potential readers.
I have also seen at times,reviews recinded or changed ,just to keep the majority of fanbopis happy,this does not look good from a readers stand point.
Because there is always going to be the potential to show bias in a review,i feel the author needs to make sure detailed infoprmation is used.Way tooo often i see geralization terms used that really leave no room for factual rating.I will use one example i see posted in these forums A LOT !
here goes....."Cataclysym has made Wow a great game ,better than it was but not quite as good as vanilla Wow".Well with no detail to back up that statement,it is just a vague generalization that really offers nothing to warrant any rating at all,if used in a review rating.We need to avoid these vague statements when giving reviews.
Never forget 3 mile Island and never trust a government official or company spokesman.
I have seen a couple people say things like this, but I am for a 3-part review. Almost all games get some kind of Beta that you guys will be invited to, play that, and make a release day review, Something that we can read the day it comes out, maybe day before, and decide, "Yeah I am going to go buy it!" or "Not for me, thanks guys." We are all adults (most of us.....at least the ones you sould be caring about.) and can understand that this review is off of your opinions from the beta, and you don"t have some super crystal ball that allows you to see into the future, your just telling us what it is, at the moment. You can list promises made by the developer, but don't put that into consideration when scoring.
Then do another review after the 30-days, should we sub? What evidence do you have to back up that claim? have any promises been kept? Bugs squashed? Community population? All those things we care about. Score it for the month, let us know what changed from the initial impression, and create a new score for the enjoyment of the free month.
Then after a fair amount of time, maybe after the first quarter, or year, 3 ,major content packages, whatever standard you want to set, come back, review it again, for longevity. LINK ALL THE REVIEWS TOGETHER! Make it easy to see how the impression changed throughout the reviews.
As for scoring, I would just stick with 1-10, because it keeps you inline with your competitors, we all know that 3=F 5=F 8=B 10=A+, so no need for that, as far as a multicatagory system, if you do that, just make sure that each catagory is weighted right, sound should not have an equal share as gameplay. maybe some scores get counted twice or three times in the average.
Gameplay=5
sound =10
graphics=8
then to average it you would go 5+5+5+10+8+8=41
41/6 = 6.8 instead of 7.6
its an idea, and would solve the problems discussed with a multi score system.
Main thing Jon, treat us like savvy consumers, you give us these letters where you talk down to earth with us, don't treat us this way now, and like little kids when it comes to review time. You obviously have thick skin, so through those reviews out there, make it clear what the review is reviewing, if its a review right out of beta something to give us on release day, sweet, its more than most sites offer. A month later, Someone wants to buy it, OMG I get another review to see if it still holds value? I'm sure you get it, hopefully its a system you are leaning towards.
Now there is the issue of purchasing three reviews per game, to publish on the site, how do you go about that? Not sure there, my publishing operation pales in comparison to what your doing, If you cut the cost of each review by 66% they would cost the same as if you were just doing 1 per game, but then the quality you get might not be on par for the standard your setting, those writers don't do this out of the kindness of their heart.
So I really do understand the situation your in, if you make three reviews per game, then there is alot of games that won't get reviewed. Maybe there could be a threshold, like, games that get below a 6 or 7 on a review don't get another, so everything gets a first review out the box, and not all of them are going to automatically get the next.
You'd have to do some math on how to figure out expenses on that, you probably have the numbers on how how many games on average pass or fail.
Just some ideas for you, and letting you know we understand the situation your in, its not just as simple as wave your magic wand.
Video Reviews would be a great addition, I agree. But I'm not sure we have the manpower for that undertaking. It's something we're toying with though, and is still in its infancy.
But folks, the real thing this column is about is whether you think we should alter the grading system for our games. Jon and I both would love to do away with scores completely. Let players be the ones to assign a score, and let us just write reviews without a number. But is that feasible? Or should we merely alter the scoring system? Make it a Grade Scale, make it 1-5, or make it an average with a revised set of categories.
That's the question we're really asking. What do you think?
now your talkin'! that would be a great idea. it woudl force ppl to read the review & understand the deeper aspects of the game instead of just looking at the score & saying "oops, 5.0, game sucks, moving on" which IMO is the problem with score based reviews, it encourages "at a glance" decisions by the reader.
1. Defined review criteria that are consistent across all reviews
2. Weighted categories that reflect the importance of each. If the site truly believes that publishers must stop pushing half finished buggy game on us, make sure that a category exists to grade the game and that the category is weighted very highly
3. Never EVER give points for "potential". That's a preview not a review. Feel free to re-review the game in a few months time.
All time classic MY NEW FAVORITE POST! (Keep laying those bricks)
"I should point out that no other company has shipped out a beta on a disc before this." - Official Mortal Online Lead Community Moderator
Proudly wearing the Harbinger badge since Dec 23, 2017.
Coined the phrase "Role-Playing a Development Team" January 2018
"Oddly Slap is the main reason I stay in these forums." - Mystichaze April 9th 2018
3. Never EVER give points for "potential". That's a preview not a review. Feel free to re-review the game in a few months time.
This is good! If you think opinions are subjective (and of course they are), what a person thinks a game might potentially do is far far more subjective.
Paragraphs about "Potential" or "Developer response time" are fine but should have 0 impact on how the game is rated (that's not say bugs or flaws should have no impact). If a deveoper responds in a way you like to something you consider an issue, you'll want to rait them higher. If they don't respond or in a negative way, you'll want to rate them low. Grading for these is like grading a company for having nice employees.
- - -
Again, 7.5, 75%, etc give the impression "The game is pretty good.", and most games are not.
Spec'ing properly is a gateway drug. 12 Million People have been meter spammed in heroics.
3. Never EVER give points for "potential". That's a preview not a review. Feel free to re-review the game in a few months time.
This is good! If you think opinions are subjective (and of course they are), what a person thinks a game might potentially do is far far more subjective.
Paragraphs about "Potential" or "Developer response time" are fine but should have 0 impact on how the game is rated (that's not say bugs or flaws should have no impact). If a deveoper responds in a way you like to something you consider an issue, you'll want to rait them higher. If they don't respond or in a negative way, you'll want to rate them low. Grading for these is like grading a company for having nice employees.
- - -
Again, 7.5, 75%, etc give the impression "The game is pretty good.", and most games are not.
It's actually worse. It's like grading a company because they might someday in the future have nice employees.. and you think it might happen. Well... like I said.. when it does happen go and re-review it.
All time classic MY NEW FAVORITE POST! (Keep laying those bricks)
"I should point out that no other company has shipped out a beta on a disc before this." - Official Mortal Online Lead Community Moderator
Proudly wearing the Harbinger badge since Dec 23, 2017.
Coined the phrase "Role-Playing a Development Team" January 2018
"Oddly Slap is the main reason I stay in these forums." - Mystichaze April 9th 2018
Maybe you could have two different types of reviewers review each game and give two seperate scores. One "hardcore" reviewer, and one "casual" reviewer, that way you can effectively give each side of the spectrum how it would play out for them and anyone in the middle can draw from each side appropriately to determine how exactly it would play out for them. Distinguishing between hardcore and casual is important. Think of reviewing Everquest in 2000. As a hardcore player, you'd have a blast, but as a casual the game is unprogessable. It's grindy, long, and very punishing, with a high focus on gear.
Comments
A scoring system similar to what one tv show does with movies would probably work best for me. The movie scores are "buy", "rent", or "pass". With MMOs it could be "buy", "free trial", or "pass". A number score doesn't really tell you how the reviewer really feels about the game. I have read numerous reviews for games where I got to the end and thought "wow this reviewer really like/hated this game" then I got to the score and thought it was the complete opposite.
Basically a buy rating would mean the reviewer was really enjoying the time spent in game. This should mean it is not overly buggy and has plenty of interesting things to do from the reviewers point of view.
A free trial rating would mean it seemed like a cool idea but maybe was a little too buggy or didn't quite fit the reviewers playstyle.
Pass would mean the game seemed like a bug ridden pile of contentless garbage.
As a previous poster pointed out it would be nice to have a bio on each reviewer stating at the very least what three games he/she has spent the most time playing as well as what the reviewer is currently playing at time of review. Including prefered playstyle may also be nice as in crafter, pve, pvp, solo, group, casual, etc.
Ok. Now I like this idea. This makes perfect sense
I wrote a thread on this subject on 4/04/11 http://www.mmorpg.com/discussion2.cfm/thread/312136/MMORPG-needs-a-new-review-scoring-system.html I'm just going to paste it here.
Take a look at the Reviews tab. All the games are rated in the six to eight range with a couple of fives.
What good is using a 10 point scale when you never use one two three four nine and ten??? Some of the games rated in the seven range are either mediocre or okay. The eight-point range seems to describe games that are good to excellent. IMO this is a huge disparity from seven to eight.
Bottom Line: Rating games using this system takes away the credibility of the entire MMORPG reviewing system. It's actually laughable with some people I've talked to. "I wonder Rift will score at MMORPG...OMGWTFBBQ!!! It's an eight! Shocker!" It's almost as if the scale is set up to be intentionally vague so as not to piss off any potential advertisers.
If you're only going to use four numbers to rate a game then use a four-star type system please! And no half stars either.
1 star - Represents a game that is in serious disarray. Missing key components and has unplayable sections. Should never have been released.
2 stars - Represents a game that has some redeeming qualities but should only be played by hard-core fans of the IP. The game is lacking in content and is buggy. A game that has potential after a nice revamp and series of patches.
3 stars - This level will represent games that excel in their particular genre. The game should have been release worthy and released with a nice amount of content and should contain stable code. Although the game will have some minor flaws and perhaps lack innovation in certain areas.
4 stars - A must play MMO! For a MMO to receive four stars it should be an industry leader in its genre and contain industry innovations. This type of game would have a fantastic support staff and supporting website and features. It should be very well tested and virtually bug free (no game will ever be completely bug free of course) and the bugs that exist should be address as quickly as possible.
This folks, is a rating system that's believable. Not many games will receive a four, but if they do, you know they'll be great. Three stars means you should have confidence that you're playing a quality MMO. Two starts means that the game needs some work, but if you're a huge fan of the IP, you'll be able to ignore some of its faults. One star means that a game is garbage and MMORPG needs to not be afraid to say so. This type of honesty will lend credibility to your review system and staff of writers.
The current system is soft and I hope you consider a change.
How I would have rated some of the latest released MMO's:
Rift - 3 stars. A quality MMO with an interesting Rift system worthy of subs from theme park-style MMO fans. Still, not enough innovation to warrant 4 stars as much of the game play can be found in previous titles. (Wow-Clone...there I said it)
Earthrise - 1 star. Terrible playability. Seriously lacking in content. Clunky combat. Bugs galore. Old concepts rehashed poorly.
DCUO - 2 stars. Excellent character customization. Combat is fluid although I can seem like a console button-masher at times. Great world design does justice to the IP. Bugs are still prevalent. The chat interface is awful. Lack of economy and proper functioning AH takes away from the MMO feel of the game and makes it more of a console experience. Potential is there for a 3-star rating at the 6 month re-review.
Perpetuum - 2 stars. Finally, a release worthy MMO just in time for 2011! Excellent base game mechanics and original ideas although some of the mechanics will be readily familiar to Eve Online players. Shady internet connections to the server. Buggy and incomplete systems. Completely unoptomized for performance. Great MMO for fans that want to stick it out. Perpetuum has 3-star potential in a year or two as it will take time for the small independent developers to make the necessary improvements rapidly.
Thanks for listening!
A couple of suggestions from my side.
Score
Stick to the 10 Point scale. As you say, it works well with MetaCritic and you want your reviews to appear over there as well. But [1] enforce consistency and assign each number with a [2] proper meaning, which you should also [3] spell out right next to the number or by providing a link to a rating policy page, where you may also conveniently explain your criteria. The reviewer should first have a verdict, only then translate it to the scale. Here is a take with keeping in mind that 5 should be the true average.
10 — Perfect
09 — Excellent
08 — Very Good
07 — Good
06 — Above Average
05 — Average
04 — Below Average
03 — Bad
02 — Atrocious
01 — Abysmal
Categories
You could use categories and rate on each separately and weight the categories differently, making Character Creation more or less important than something else, depenend on your criteria. These weighted scores are then aggregated into the final score which should line up with the verdict as listed above, when your criteria and weights are set correctly that is. You only need to setup an Excel table and it would be easy for your staff to use it. I won't overdo the categories though. A few would be enough.
Character Creation: breadth (many options), depth (option has many settings)
Usability: from installing to playing, how convenient and smooth does it feel like. Is it intuitive, clunky, responsive ...
Sophistication: in lieu of a better word: is it ripe, how many and how severe are bugs, how feature complete is it?
Core Game: there is a play pattern at the heart of any MMO that is repeated over and over, mostly combat wrapped in quests. How is the quality of that? Too many skills to use? It is too repetetive? Does it grow old quickly?
Scope: breath and depth are adequate terms to describe what to expect as well. Does it offer more than other games on release? Does it have few large and awesome features? Is it more a collection of many minigames? Has it enough PVP and PVP. What about crafting?
Balance: There are many kinds of balances, not just between classes. How about the ratio between action and downtimes. What about difficulty on your level? Can you outheal the opponent? Are other classes so much easier to play?
Presentation: I'd lump graphics and sound into that one, and other production values. Be careful with rating graphics merely from a technical point of view. MMOs are geared at large audiences and state-of-the-art is usually not the priority. Where it ends up depends on many factors, some deliberate (maybe the game is aimed at an older demographic expected to have more outdated machines) and some more or less coincidential (development time, rule of thumb etc.). It would be unfair to rate down a game for a certain deliberate strategy. Rather, look at art style, concistency, quality in the substance (lighting, atmosphere, personality etc.) and the like.
Innovation: I am bit hesistant with this one. I think it is overrated. A genre simply has certain genre characteristics and the community and some reviewers on this site places too much emphasis on it. You would have to find out whether its bad to have the basic DikuMUD Gameplay and if different approaches should yield better scores. Every game is different and most games have their own take on known features at least. So I remain skeptical with this.
Fun: The game could totally suck at anything and still be super fun. This might be the Joker card that influences the final score the most.
Social: I feel this should show up somewhere as well, but don't judge whether most players of this game are douche, but at how game features condense in the community. Keep in mind that drama is good in some games and detrimental in others. Also, some things are really opinions, like if mandatory grouping is a good thing. Remain impartial on that one. But if grouping is strongly discouraged, dungeon finder teleporting makes it so that you never see anyone anymore and the like, there are too many player hubs so there is none, these are reasons to consider. Also did the devs introduce rally points (factions, alliances, guild features) to identify with, and does it pan out well? Do the emotes have a positive impact?
Target Players
You have a box with the Pros and Cons, which I think is great. You could expand this to recommend it to a certain player type, you think this game caters to the most. Examples:
Comparison
Players who liked Game X and Game Y may also like this Game.
Bartle-Class Recommendations
They are typically depicted using the playing cards symbols which is also an excuse to make it look nice: Explorers (♠), Killers (♣), Socializers (♥), Achievers (♦). These are motivations and every player has them in varying degrees.You could use them to describe where the game is strong at. Lot's of PVP and very twitch based? Then Killers should look into it. Or is the game very item centric with a huge achievement system? Is it an open-world game or very instanced? This pretty much influences to whose motivations the game will cater to.
Nick Yee's Classification
It's bit differently and essencially three-sided. Achievement, Social and Immersion with some subcategories. You could combine it with the rating suggested above, or very quickly indicate strong points and weak points by just pointing out the words, thus complementing the Pros and Cons. You should keep this standardized and place a longer overview page behind the words. Please note that you do not rate the game itself, but how you assume the game will cater to players who are motivated by certain factors. E.g, Game X: "Advancement, Teamwork, Competition"
MMO (Sub) Subgenre / Distinctions
As anyone and his mom knows there are several traditions of MMORPGs and they can make the gameplay dramatically different. The two broad categories are of course the Sandbox vs Themepark distinction, actually rather Simulation vs. Game. Themepark actually means a game world that is based on attractions in well-crafted thematically different areas (think Magic Kingdom style, "Adventureland", "Tomorrowland" etc.). This is of course the hallmark of World of Warcraft, which seems to be modelled fairly close to original Disneyland (no surprise since their Irvine office is only a tossed dwarf away). Games that are more sandbox-y, that is simulationist, are rather based on emergent gameplay, Ultima Online, EVE and the like. In the former developer control is emphazised, player control in the latter.
Another one would be Open-World vs. Lobby/Tunnel . Open-World games have a more or less freely accessible game (over) world, where you may find mobs of your level range in corners of many different zones. Games with a lobby are almost entirely instanced and the instances may be rather linear tunnels to fight till the end boss. Sandbox games are usually open-world as well, but the distinctions should not be confused (WOW is open world as well).
The next distinction, again extreme ends, would be Twitch vs Turn. One places heavy emphasis on the players inherent abilities like hand-eye coordination (twitch) so that you cannot be a master archer if you aren't in real life, so to speak. The other extreme are turn based games or MMORPGs where the old Wargaming roots show through and only your strategical/tactical choices matter, so you can be master archer even as real life paraplegic, because your character sheet says you are (this is also at the heart of "What means RPG", which is not speaking with a pirate accent all day).
Most MMORPGs are actually somewhere in the middle, where some reflexes are required even in barely concealed quasi turn based games on one end, and the newer action-oriented MMORPGs which still resemble DikuMUD combat, but feature a more free-form targeting and faster gameplay. Essentially, Boring vs. Carpal Tunnel Syndrome.
Anyhow, the core characteristcs could be summed up as well, getting rid of comments that somehow demand the opposite type of gameplay.
The Review
Generally, many reviews I've read had too much focus on the reviewer, what the person personally likes and dislikes rather than strongly reflected on the game itself (wait. I get to that). It often read like apologizing in advance or dodging bullets that commentors may shoot. Personal preferences could be outsourced to introduction / profile pages of the reviewer, possibly adhering to some standards, like the classifications mentioned above. So you don't want to read too much of "I am personally fine with Kill Ten Rat Quests, but Jade Doe may not like this." The key information here would be that the game is full of these quests and the reader will know what to make of it. Likewise, I don't want the reviewer to point out that players who don' like PVP will not like this game. I want them to point out that PVP is mandatory or that there is little else beside it. If things are merely described, keep it tight. If you analyze, stay away from "No Shit, Sherlock?" territory. If you offer an opinion, it should be "reasonable". Generally, I have no qualms with that one, usually seems to be the case..
I miss a certain standard of journalism in gaming and I guess games are not considered a matured medium, unless game journalism is more mature as well. It could also be more unexpected at times, like Gonzo Journalism in Middle Earth. You could also show more Genre Savvy-ness at times. A word or two could be said about all the corny clichés that reappear in any new next-gen incarnation. It's boring to poke fun at Elves and Orcs per se, why not go step further and make fun of all the ridiculous "conventions". I already gave you some links to start looking (but be warned, it will ruin your life).
TLDR.
To be honest, I don't really have a single problem with the reviews here. I think you guys are pretty clear and concise with your scoring system, the icon-system for quick-viewing, the labels you give different kinds of reviews.
In my opinion, any mature adult that uses reviews as deciding factor of whether they buy a game is being foolish.
I use reviews to make a more informed decision. So I tend to look for mostly facts in MMORPG.com's reviews, and so far I'm happy enough with that information.
What I want from a review is a more indepth discussion of the information I can read on the developer's FAQ. Whether the reviewer likes a certain feature or not is beside the point, so as long as you guys and girls are giving accurate descriptions of the features in a game, then I'm a happy reader.
One off-topic thing, though, not related to reviews so much: please stop using GamerTube for your vid streams. The quality is just awful!!
Lets be real here...
The reason this is brought up, is because of nearly every reason mentioned here. But the fundemental reason, is you don't have a set-standard of criteria.
No matter what scale, or rating system you use, won't matter if the criteria is different for each review. So that goes with YOU setting the stage, & holding these game companies to those standards.
However close we (the MMORPG.com community) agree on those standards (yours), doesn't matter. Only that YOUR criteria is consistant, then readers can weigh their bias based on deltas known to differ from YOUR opinion.
Lastly, it is quite clear this website has grown rapidly & is not to professional at all. I'm seeing reviews from newbies (to me) that draw references to games mechanics, without knowing the said mechanic's origins, or it's depth of intracies.. or, how shallow they are represented in said game. It hurts to read so much ignorance.
Then, we are barraged with glitzy & flashy banner adds aimed at 12 year old girls & teen boys..? What does half of that garbage have to do with roleplaying games? Their just MMO's..
Start a freak'n sister site MMO.com and put all that^ kiddie stuff over there and turn this site around.
Adults don't need hype, we want reviews that cuts the core out of developers & makes them fess up to their vision, not cash deals that seem oh so apperent. We want articles that are pages deep and throught with adventure & details.
Or, how about well written Chronologies (ie: roleplayed) that are month-long journeys into the classic MMORPGs. Some type of feature set, so that young & old... all readers can learn about the history of MMORPG's. The progeny and how their built, & how IT and server farms make them work...
educate these kids... their completly lost and are looking for a new game and cannot express themselves. They need to have a knowlege base of something more than World of Warcraft.
Maybe then, YOUR reviews will be heralded as tag lines on the cover of Magazines, instead of chatter in teenage online chat rooms.
"No they are not charity. That is where the whales come in. (I play for free. Whales pays.) Devs get a business. That is how it works."
-Nariusseldon
I prefer knowing the categories. I couldn't care less that a game is an 8 as much as I care where the game failed to not make it a 10 and where it was well made to achieve an 8 in the first place. I like it when a reviewer grades the categories seperate and if needed gives an average.
On that note though, I don't see why all categories must be weighted the same. Sure I'd rather have great gameplay then say have great music (when I'll usually turn it off and play my own regardless).
I think Stormwatch has a great idea (read up for his post) but may be a bit more time consuming. Although in an ideal world I'd go with his approach or one close to it.
Bro you are playing LOTRO like it owes you money - Grunt187
Anticipation : TERA
Have as many previews as you want, just make it very clear that’s what they are. Then your launch review on or very close to the first day of launch. Having any review after this encourages MMO companies to launch before they are ready, so don’t do one! If say six months to a year down the line a major content update occurs, just do what you normally do and have a review of the update.
In my humble opinion. One to ten is allready representative of the letter system, anything below a six is failure. (10A+)(9A)(8B)(7C)(6D)(5F). Obviously a game should be judged as a whole, rather than fully in certain areas. Leaving the judgemant open for interpretation is sort of ridiculous. Now a lot of people who lack self discipline are turned hostile so.
(Elaboration) I haven't played the game, the information gathered by the first hand account i.e. the reviewer is his alone. I can never truly make a verdict. It is my theory that the burden rests on the shoulders of the first hand acountant. Reading about it gives me an idea of the game, but never the experience. Good critiques will give good judgemants and ones that I would trust over my own. That's why I'm the guy reading the review, not the other way around.
Cuting the lower portion of the scale out because no ones been their, is like cutting the top out because no ones climbed the mountain. Which further confuses me about the scaleing system, scales are useful and they need to be somewhat accurate. Their are deffinately some failing games out their, why they don't deserve a fail grade IDK. Saying something is a failure isn't cruel and I disagree with the idea that games don't fail. They do and if you don't think so, I don't want to argue the sky blue.
Summary gradeing should cut a game into pieces. Each piece being portioned equally combined together would be the score. 1-10 is perfect and a mathmatical wonder seeing as it manifests in many things.
Know what grinds my gears is grinding my gears. For gods sake video games are video games. If your not drawn in by a picture or even the first few minutes of gameplay who cares quit. Posting useless reviews on the internet it a waste of time when now most people can just watch game play videos. Cuts down all the reading to letting your eyes visually scan the game and see how it comes off to your eye site.
When did you start playing "old school" MMO's. World Of Warcraft?
Two big improvements would be:
1. Hire people that actually play MMO's
The most imprtant thing you can do to improve your reviews is have actual players review the games. It is quickly apparent when reading any MMORPG.com review that the writer has barely played the game. They have geenrally only played superficially enough to get a flavor. A prime example is Bill Murphy. He writes some good stuff but its quickly apparent the man has never actually played a MMO other than on the surface. To really a play a MMO and understand one needs to play mroe than 20 levels. IMO one needs to play more than 50-100 days played to fully understand a game. But I know realistically that will not happen. But your reviews are written mainly by people who are only guessing at how games are actually played. Basically they should be called 1st impresion form a casual player.
2. An even point system does not work
It is nice and pretty but an even point system does not work. One cannot weigh different aspects like combat and roleplay for example evenly. You cannot break it down and it add up like that. The best thing IMO would be review the game seperately for many different qualities that actually matter to players such as Fun factor, completeness, polish, quality of content, endgame content, fluidity of design, innovation, raiding, group dungeons, combat mechanics, crafting economy, PvP, Community building tools. Plus an overall score based on the feeling the reviewer has, not a simple summation fo the parts as everything is simply not equal. People look for different hings in a game. Some cry for innovation while otehrs will be simply happy for good endgame content.
Could be a crazy idea but how about this:
Go back to the standard rating per category.
Keep it numerical so it's clear how averages are computed.
Give each category a standard weight
Allow registered users to determine their own weights.
Average game score is computed dynamically based on the weights.
So for example gameplay has a standard weight of 3 and sound a standard weight of 1. RndUsr is always listening to music anyway so in his preferences he sets the weight of sound to 0.
RndUsr visits this site without being logged in and when computing the average score the website counts gameplay 3 times and sound once for computing the average. Now RndUsr logs in, because his settings changed the next score computed doesn't count sound at all. Causing games with a high soundscore to actually decrease in average rating ( for him. ) and games with low soundscores to increase in average rating. Because for RndUsr sound just isn't important.
Same thing could be done for PvPscores and such. If you absolutely need PvP then set the weigth for that score very high. If you don't care about PvP set it at zero.
This way the average score is actually determined by the reader himself, he decides which categories are important and which aren't.
Only downside would be that you'd probably have to add a few more categories.
We are the bunny.
Resistance is futile.
''/\/\'''''/\/\''''''/\/\
( o.o) ( o.o) ( o.o)
(")("),,(")("),(")(")
This is a good point Vid Reviews allow a lot of information and impression. But I also think written reviews by mmorpg critics (well written!) with mmorpg punter's responses are also part of the process. Discussion, debate and disagreement are essential.
In addition one scoring system is only measuring one thing in time and in one thing in game/genre terms.
A score for genre (comparitative to similar games)
A score for production (gameplay, graphics, sound, service etc)
A personal, professional game critics score (you peeps)
Meta-scores across the game world/web AND/OR Player rating score for this website.
Finally a re-review assessment due to change in players numbers, developer patches and oc a lot more feedback from players.
http://www.gdcvault.com/play/1014633/Classic-Game-Postmortem
Video Reviews would be a great addition, I agree. But I'm not sure we have the manpower for that undertaking. It's something we're toying with though, and is still in its infancy.
But folks, the real thing this column is about is whether you think we should alter the grading system for our games. Jon and I both would love to do away with scores completely. Let players be the ones to assign a score, and let us just write reviews without a number. But is that feasible? Or should we merely alter the scoring system? Make it a Grade Scale, make it 1-5, or make it an average with a revised set of categories.
That's the question we're really asking. What do you think?
Try to be excellent to everyone you meet. You never know what someone else has seen or endured.
My Review Manifesto
Follow me on Twitter if you dare.
The scale is fine. 1-10 works and it's about how everything in life is graded. It's similar to how our papers are graded in school. It's roughly how we refer to cute guys/girls. Numbers are fine but the numbers need to be approrpriate.
Average games need average ratings, so I have to say again. Move away from the rating curve and leeway.
Try pairing random reviewers together. Have two people independently review the game and combine their reviews, especially for the major reviews--what people are most interested in.
Spec'ing properly is a gateway drug.
12 Million People have been meter spammed in heroics.
+1 for doing away with the grading system. It has simply lost it's meaning and most games always end up between 7.5 to 8.5, with this alone it's simply too hard to tell if the game is truly good or not because there is never a score that will stand out from the rest. It also, sometimes, seems simply inconsistent with the reviews hurting a bit the credibility of either the reviewer or the website all together. Would also help avoid the typical reaction of "Game got a high score when it sucks, MMORPG.com are being bought!". Members should also be made aware that reviews are suggestives and that it is simply impossible to make unbiased reviews (as seen in this thread, it's not very clear for some which leads to more typical reaction).
Doing away with it would be perfect. As for user-based scoring system, as I mentionned in my first post, I'd prefer users be able to simply choose options like "Was this review helpful [yes] [no]" with the possibility to write a commentary (similar to the user-generated game Ratings) . With MMORPG.com's current community, you'd see ridiculous scores of 3-5 for games that are actually good, but hated deeply by some disgruntled gamers, in which case the scoring system would be yet again simply useless (Wouldn't be suprised if WoW were to drop below 0 on this website should this ever happen ).
I'd like to see each review broken down into scores by headings (numeric, % based) and reasoning given for it. If you like, at the end give a couple of paragrphs over to the "feel" of the game (that nebulous, subjective notion) and the game an overall grade based on performance & feel, that could be your A / B / C / D grade.
What the reviews seem to suffer from most here is lack of an overall structure. It allows forumgoers too much leeway for poking holes in the reviews "You didn't even mention the glorious sounds / buggy animations / hellacious lag", whatever.
If all reviewers are working off the same template, the review process is rendered more transparent and less liable to someone proclaiming a review "fail" because something got left out, or claiming that you were paid to mention well-worked out feature A while ignoring teeth-grindingly awful problem Y.
If you use a combination like the one above (or one suggested by others in thread) people can see the scores, and then read through the "opinion" part to better understand why the game recieved the grade it did.
I understand completely the pressure to have a review out asap at launch, but I think it would be a good plan to revisit each game (particularly AAA titles) maybe six months / a year down the road and see whether it has improved / stagnated over that time.
I know, speaking for myself, I can be very hyped up for a game through beta and launch, only to walk away a month or two later because there was something in the later game that I couldn't stomach. Revisiting would mean less chance of an earlier rave review coming back to bite you on the behind - or vice versa.
I do believe a number system should be used,but you need to understand what the gaming industry is capable of doing before weighing in with numbers.
One example would be in the use of textures or model poly count.I have heard time and again how the dev is trying to cater to more people,so one needs to mention that in the rating,don't just hand out 9/10 for low end qaulity because the dev was aiming for that.
There was something missing from the OP,and i feel it very important to mention it.The problem with MANY reviews is they"appear" to be biased or more of a marketing campaign.Most readers are not stupid,they can see where a post is going or coming from.I am not saying this is the case for ALL posts/reviews i am just saying there is at times the appearance of being a skewed review.
SO i think before a review is posted,the writer needs to read it over a few times and determine if it looks skewed in the eyes of the potential readers.
I have also seen at times,reviews recinded or changed ,just to keep the majority of fanbopis happy,this does not look good from a readers stand point.
Because there is always going to be the potential to show bias in a review,i feel the author needs to make sure detailed infoprmation is used.Way tooo often i see geralization terms used that really leave no room for factual rating.I will use one example i see posted in these forums A LOT !
here goes....."Cataclysym has made Wow a great game ,better than it was but not quite as good as vanilla Wow".Well with no detail to back up that statement,it is just a vague generalization that really offers nothing to warrant any rating at all,if used in a review rating.We need to avoid these vague statements when giving reviews.
Never forget 3 mile Island and never trust a government official or company spokesman.
I have seen a couple people say things like this, but I am for a 3-part review. Almost all games get some kind of Beta that you guys will be invited to, play that, and make a release day review, Something that we can read the day it comes out, maybe day before, and decide, "Yeah I am going to go buy it!" or "Not for me, thanks guys." We are all adults (most of us.....at least the ones you sould be caring about.) and can understand that this review is off of your opinions from the beta, and you don"t have some super crystal ball that allows you to see into the future, your just telling us what it is, at the moment. You can list promises made by the developer, but don't put that into consideration when scoring.
Then do another review after the 30-days, should we sub? What evidence do you have to back up that claim? have any promises been kept? Bugs squashed? Community population? All those things we care about. Score it for the month, let us know what changed from the initial impression, and create a new score for the enjoyment of the free month.
Then after a fair amount of time, maybe after the first quarter, or year, 3 ,major content packages, whatever standard you want to set, come back, review it again, for longevity. LINK ALL THE REVIEWS TOGETHER! Make it easy to see how the impression changed throughout the reviews.
As for scoring, I would just stick with 1-10, because it keeps you inline with your competitors, we all know that 3=F 5=F 8=B 10=A+, so no need for that, as far as a multicatagory system, if you do that, just make sure that each catagory is weighted right, sound should not have an equal share as gameplay. maybe some scores get counted twice or three times in the average.
Gameplay=5
sound =10
graphics=8
then to average it you would go 5+5+5+10+8+8=41
41/6 = 6.8 instead of 7.6
its an idea, and would solve the problems discussed with a multi score system.
Main thing Jon, treat us like savvy consumers, you give us these letters where you talk down to earth with us, don't treat us this way now, and like little kids when it comes to review time. You obviously have thick skin, so through those reviews out there, make it clear what the review is reviewing, if its a review right out of beta something to give us on release day, sweet, its more than most sites offer. A month later, Someone wants to buy it, OMG I get another review to see if it still holds value? I'm sure you get it, hopefully its a system you are leaning towards.
Now there is the issue of purchasing three reviews per game, to publish on the site, how do you go about that? Not sure there, my publishing operation pales in comparison to what your doing, If you cut the cost of each review by 66% they would cost the same as if you were just doing 1 per game, but then the quality you get might not be on par for the standard your setting, those writers don't do this out of the kindness of their heart.
So I really do understand the situation your in, if you make three reviews per game, then there is alot of games that won't get reviewed. Maybe there could be a threshold, like, games that get below a 6 or 7 on a review don't get another, so everything gets a first review out the box, and not all of them are going to automatically get the next.
You'd have to do some math on how to figure out expenses on that, you probably have the numbers on how how many games on average pass or fail.
Just some ideas for you, and letting you know we understand the situation your in, its not just as simple as wave your magic wand.
My Thoughts on Content Locust
now your talkin'! that would be a great idea. it woudl force ppl to read the review & understand the deeper aspects of the game instead of just looking at the score & saying "oops, 5.0, game sucks, moving on" which IMO is the problem with score based reviews, it encourages "at a glance" decisions by the reader.
I just want to reiterate again:
1. Defined review criteria that are consistent across all reviews
2. Weighted categories that reflect the importance of each. If the site truly believes that publishers must stop pushing half finished buggy game on us, make sure that a category exists to grade the game and that the category is weighted very highly
3. Never EVER give points for "potential". That's a preview not a review. Feel free to re-review the game in a few months time.
All time classic MY NEW FAVORITE POST! (Keep laying those bricks)
"I should point out that no other company has shipped out a beta on a disc before this." - Official Mortal Online Lead Community Moderator
Proudly wearing the Harbinger badge since Dec 23, 2017.
Coined the phrase "Role-Playing a Development Team" January 2018
"Oddly Slap is the main reason I stay in these forums." - Mystichaze April 9th 2018
This is good! If you think opinions are subjective (and of course they are), what a person thinks a game might potentially do is far far more subjective.
Paragraphs about "Potential" or "Developer response time" are fine but should have 0 impact on how the game is rated (that's not say bugs or flaws should have no impact). If a deveoper responds in a way you like to something you consider an issue, you'll want to rait them higher. If they don't respond or in a negative way, you'll want to rate them low. Grading for these is like grading a company for having nice employees.
- - -
Again, 7.5, 75%, etc give the impression "The game is pretty good.", and most games are not.
Spec'ing properly is a gateway drug.
12 Million People have been meter spammed in heroics.
It's actually worse. It's like grading a company because they might someday in the future have nice employees.. and you think it might happen. Well... like I said.. when it does happen go and re-review it.
All time classic MY NEW FAVORITE POST! (Keep laying those bricks)
"I should point out that no other company has shipped out a beta on a disc before this." - Official Mortal Online Lead Community Moderator
Proudly wearing the Harbinger badge since Dec 23, 2017.
Coined the phrase "Role-Playing a Development Team" January 2018
"Oddly Slap is the main reason I stay in these forums." - Mystichaze April 9th 2018
I think 1-10 is great.
Maybe you could have two different types of reviewers review each game and give two seperate scores. One "hardcore" reviewer, and one "casual" reviewer, that way you can effectively give each side of the spectrum how it would play out for them and anyone in the middle can draw from each side appropriately to determine how exactly it would play out for them. Distinguishing between hardcore and casual is important. Think of reviewing Everquest in 2000. As a hardcore player, you'd have a blast, but as a casual the game is unprogessable. It's grindy, long, and very punishing, with a high focus on gear.
He who keeps his cool best wins.