Why Are We Still Using Review Scores?

Neoyoshi

Great video

Sovrath

Just listened to that.

Yeah, review "scores" are sort of just an entryway into someone reading the review. But people just want to see a number. A number that has no standard metric (per the video) and is supposed to represent someone's subjective review.

Neoyoshi

I wish i had just a fraction of Sabastian's quippy charm.

Scot

We have to have review scores, we just need the wonderful gaming public at large to understand they are just one persons estimate, not written on eternal tablets of stone.

cameltosis

Neoyoshi said:

Great video

Any chance you could post a little summary of the video for those of us who won't / don't / can't watch them?

I NEVER watch videos posted on forums, so when someone just dumps a link and leaves it can be quite hard to follow along or contribute.

More importantly, I don't really care what someone says in a video. I care about what the people on this forum think as they (and you) are the people I actually interact with.

Kyleran

cameltosis said:

Neoyoshi said:

Great video

Any chance you could post a little summary of the video for those of us who won't / don't / can't watch them?

I NEVER watch videos posted on forums, so when someone just dumps a link and leaves it can be quite hard to follow along or contribute.

More importantly, I don't really care what someone says in a video. I care about what the people on this forum think as they (and you) are the people I actually interact with.

Amen, preach on brother. If I was a moderator here I would remove any posts which just dropped a video without any sort of short summary or commentary.

Like you I normally ignore such posts unless they are funny cat videos.

Aeander

Well, scores are still useful for review aggregators like Metacritic, Open Critic, and the user reviews of Steam and other storefronts. It's a good short-hand for whether a game is generally seen as good, or bad, or amazing.

The only real problems are that we assign the opinions of individual reviewers as the opinions of an outlet, and user reviews are often worthless because we tend to use them to voice outrage.

Vrika

We are using review scores because I'm not going to spend 8 minutes of my life finding out what something contains - like for example I didn't bother watching that video.

Give me a quick summary and I may be interested.

Review score is the most important part of a quick summary that may or may not get me interested enough to look at the actual review.

[Deleted User]

The user and all related content has been deleted.

thelawoflogic

Because of money

Wargfoot

Video Score: 67.9 out of 75.

Vrika

Wargfoot said:

Video Score: 67.9 out of 75.

Sounds interesting. Where can I read your review of the video?

Scot

When you consider how players react to reviews; I don't know how the staff here sees this, but if I was new to the office and was given a review to do I would wonder if I had done something to upset the editor.

Dammam

templarga said:

I actually pay attention to the bulleted points now and summary in reviews and ignore the multiple pages of experience and number scores. They are used because they are quick and easy and we associate them with other review scores. An 8 is good. A 2 is not. We know this without having to know it.

One of the biggest issues though with numbers is that they mean different things. I bought my last vehicle from a dealership and they sent me a review request. So I treated a 0-10 scale like its meant to be used 0-3 is bad, 3-6 is ok and 7-9 is good and 10 is fantastic (that's my interpretation). The OWNER of the dealership called me because I gave them scores they had never gotten before. For them a 7 was bad. I told them then they should have a 0-3 scale and that they were artificially inflating their numbers on their review metric and it was worthless. Needless to say, I never bought from them again.

You touch on a major challenge of creating and conveying information with scores that I think is often missed - the meaning of the score changes depending on what you are using it for and your tolerance for that thing. In your example, you used the 0-10 scale to answer a multi-class problem, meaning the answer could be any of the four classes you listed - bad, ok, good, or fantastic. But when someone uses the score to make a judgement, it may very well be a binary judgement of good vs bad or to buy vs not to buy. How your grading scale maps to their judgement call is not straightforward, because it's not obvious where their binary threshold would be. Are they buying if the score is above a 5? Or is their minimum acceptable score an 8?

One way of assuming customers' thresholds is to compare to the average scores of similar options, since people often make judgement calls relative to their options. If the average score of your competitors' products is a 7, then those customers may treat your score of 7 like a 5 on a 0-10 scale or simply "average" rather than the intended "good". Good would have to be better than average. So baked into the score and abstracted away is not just your judgement and the readers' judgements, but all the other context that could influence that interpretation. It's truly difficult to convey valuable information this way.

Kyleran

I think we should use a binary Pass/Fail rating.

For MMORPGS there wouldn't have been any that passed in the past 15 years or so.

Scot

Kyleran said:

I think we should use a binary Pass/Fail rating.

For MMORPGS there wouldn't have been any that passed in the past 15 years or so.

SWTOR, ESO and TSW in 15 years for me and TSW closed before it came back ever easier.

Kyleran

Scot said:

Kyleran said:

I think we should use a binary Pass/Fail rating.

For MMORPGS there wouldn't have been any that passed in the past 15 years or so.

SWTOR, ESO and TSW in 15 years for me and TSW closed before it came back ever easier.

ESO was passable when I played it 7 years after release, at launch I didn't last 3 months.

Playing EVE starting in 2006 damaged me, after that I kept looking for a better version of it, yet another theme park (which ESO was at launch) just didn't cut it anymore)

What it comes down to is I don't enjoy MMORPGS where progression is reset continuously and there are darn few out there that don't follow the original formula set up by EQ and those that followed it.

What's strange is I'm now playing 7D2D like an MMORPG, Day 475 which you'll find is almost unheard of for most who play it.

I think when the time comes to reset, probably to try out one of the challenge mods like Darkness Falls, or from the next alpha release I'll probably be fine with it.

I'm coming to view that more like rerolling a new alt which I always did enjoy in traditional MMORPGS.

Wargfoot

Scot said:

Kyleran said:

I think we should use a binary Pass/Fail rating.

For MMORPGS there wouldn't have been any that passed in the past 15 years or so.

SWTOR, ESO and TSW in 15 years for me and TSW closed before it came back ever easier.

Nothing has come close to TSW for atmosphere.

Scot

Wargfoot said:

Scot said:

Kyleran said:

I think we should use a binary Pass/Fail rating.

For MMORPGS there wouldn't have been any that passed in the past 15 years or so.

SWTOR, ESO and TSW in 15 years for me and TSW closed before it came back ever easier.

Nothing has come close to TSW for atmosphere.

And its uniqueness in both setting and theme, mind you I am not saying three good MMOs in 15 years is actually great here. We used to get one good really solid MMO every couple of years.

Dammam

Scot said:

Kyleran said:

I think we should use a binary Pass/Fail rating.

For MMORPGS there wouldn't have been any that passed in the past 15 years or so.

SWTOR, ESO and TSW in 15 years for me and TSW closed before it came back ever easier.

I'd add GW2 to the list of passed for me. 

In a pass/fail system, there isn't much nuance in a single score, so many imperfect games might get a pass alongside much better ones. But aggregating many people's pass/fail scores can show some of that nuance. A game that was playable but not everyone's cup of tea might get some mix of passes and fails, while an utterly unplayable game would only get passes from those who backed it on kickstarter.

lotrlore

Scot said:

When you consider how players react to reviews; I don't know how the staff here sees this, but if I was new to the office and was given a review to do I would wonder if I had done something to upset the editor.

Most of us just don't read the comments.

 Honestly, most of us don't like the scores either. It's literally the last thing any of us think about, though, when reviewing. When I took over the site in 2019, one of the first things I did was change how we review games so that the content of the review was more important than that score. We also changed up how we score now, as before everyone had their own certain formula to determine the final number (an aggregate of all the different parts, for example). Now, we use the term the number stands for to help determine the actual score. Would you describe this game, on the whole, as a "great" game when talking about it with a friend? Well, Great on our scale is at least an 8, and so on.

But the content of the review should always be the most important aspect of the whole. But we also know that many people want that shorthand and don't want to actually read. But scores themselves are effectively required if you want a mid-sized outlet like ours to get any traction in Google. It's also good for Metacritic and OpenCritic linking, which is valuable for SEO. Honestly, though, if we could, I would simply do away with scores. But at this point it's just never going to be possible, at least until the major industry players (IGN, GameSpot, PCGamer, etc) do away with them first.

Scot

lotrlore said:

Scot said:

When you consider how players react to reviews; I don't know how the staff here sees this, but if I was new to the office and was given a review to do I would wonder if I had done something to upset the editor.

Most of us just don't read the comments.

 Honestly, most of us don't like the scores either. It's literally the last thing any of us think about, though, when reviewing. When I took over the site in 2019, one of the first things I did was change how we review games so that the content of the review was more important than that score. We also changed up how we score now, as before everyone had their own certain formula to determine the final number (an aggregate of all the different parts, for example). Now, we use the term the number stands for to help determine the actual score. Would you describe this game, on the whole, as a "great" game when talking about it with a friend? Well, Great on our scale is at least an 8, and so on.

But the content of the review should always be the most important aspect of the whole. But we also know that many people want that shorthand and don't want to actually read. But scores themselves are effectively required if you want a mid-sized outlet like ours to get any traction in Google. It's also good for Metacritic and OpenCritic linking, which is valuable for SEO. Honestly, though, if we could, I would simply do away with scores. But at this point it's just never going to be possible, at least until the major industry players (IGN, GameSpot, PCGamer, etc) do away with them first.

Most players would undoubtably say "we need the scores". Although I like the idea you have tried to standardize the scoring (it is what I would try to do), I have seen many times the problems inherent with trying to standardize peoples definitions, so genuinely good luck with that.

What we need is more of an appreciation of what a review score is; how it depends very much on the reviewer and what they like and don't like in gaming. But also on their own beliefs, I don't think anything should get a 10 because I don't think a scoring system should ever mark something as perfect. Most people would disagree with me.

This is why the same game can get a score of 60 and 90, people do not all see a game through the same pair of eyes, which is only quite natural.

And yes most players don't want to read; be it reviews, guides, quest text or lore, but that's where we are now they simply want to be told "yes or no" and play without any need to understand anything about the game, which is quite worrying.

Wargfoot

I think scores can be helpful.

A 2.0 tells me the game probably isn't worth investigating while rave reviews mean it may be worth watching a couple of YouTube videos.

The difference between 8.3 and 8.5 is meaningless to me, tho.

Sovrath

Vrika said:

We are using review scores because I'm not going to spend 8 minutes of my life finding out what something contains - like for example I didn't bother watching that video.

Give me a quick summary and I may be interested.

Review score is the most important part of a quick summary that may or may not get me interested enough to look at the actual review.

And for me the review score is just an invitation to read/listen the review. I've read reviews that had higher scores only to read/listen to them and think "why is that good?" Only to read something with a lesser score but that I find out is amazing based off what is said.

Scot

Sovrath said:

Vrika said:

We are using review scores because I'm not going to spend 8 minutes of my life finding out what something contains - like for example I didn't bother watching that video.

Give me a quick summary and I may be interested.

Review score is the most important part of a quick summary that may or may not get me interested enough to look at the actual review.

And for me the review score is just an invitation to read/listen the review. I've read reviews that had higher scores only to read/listen to them and think "why is that good?" Only to read something with a lesser score but that I find out is amazing based off what is said.

The review score and I am including "positive" on Steam here makes me go to the next stage of checking a game out. For me it is a set of quick hurdles the game has to pass before I read a review and then watch a video guide. The video guide is more for how it looks really, rather than how it plays which I will more or less have worked out by that stage.

Mackaveli44

Reviews are just opinions. Opinions are like assholes, everyone has one. 

Stop listening to reviews. Form your own opinion on things.  Quit being followers.