Scenario Depot - Balance Ratings

[xerxes]

One thing that bugs me about the scenario depot rating system is the balance rating. As it is now people estimate balance by their one playing of a scenario. Quite frankly, it is not possible to estimate balance from a single playing between two players.

As a point in case, my scenario "Recon! Point Counter Point" has received inaccurate low balance ratings. How do I know the ratings are inaccurate? Because the scenario was used in a tournament where it was played 16 times.

In the tournament there were 8 battles in which the Romanian scored more points and 8 battles in which the Soviets scored more points. All of these battles were between "good-excellent" CMBB players. The exact results can be found here:

Recon battle results

I think that instead of having players rate balance they should just put in the score of the game. The average score should be reported as the "balance" for a scenario. I think that would be far more accurate then subjective judgements.

Just my 2 cents.

btw, I think the scenario depot is a fantastic resource and Keith has done a bangup job with it.

[ November 13, 2003, 10:02 AM: Message edited by: xerxes ]

[Keith]

I strongly disagree. I find the balance rating the most important rating that I use to decide if a scenario is suitable for PBEM. I sure as heck want to know what other players experienced before investing a month of time swapping emails only to discover a scenario is way out of balance. People are intelligent and look at the cummulative ratings, and each review is a but a single data point. Give consumers some credit for knowing how to discern the data.

[xerxes]

I disagree with your disagreeing. Actually I do agree that balance is a very important characteristic.

My point is that in a single play of a scenario it is extremely difficult for the participants to accurately assess balance. From my experience the further away from a draw, the lower the balance rating. The problem is that each rating is based on a single point sample. To accurately extrapolate from 1 point to an entire distribution is simply not possible.

Most people assume that if there is a lopsided result in a scenario it means the scenario is unbalanced. This is simply not true. I've seen cases in which the exact same scenario was declared unwinnable from both sides.

[dugfromthearth]

I would agree that asking people who played a single battle, seeing only their side, whether or not a scenario is balanced is going to get you meaningless data.

How could they possibly know from that if a scenario was balanced?

[Keith]

If you disagree with a review, simply provide author feedback and provide data that you just talked about (ex. played 14 times in tournament with x/y win loss ratio). That is what the scenario author comments capability is for.

I just read the reviews of the above mentioned battle and my overall impression, having not played the scenario, is that it is a fun, small scale recon battle designed for PBEM. Most people found it balanced but a couple people thought it favored the Russians too much. So as a consumer of this info I would perhaps see if my potential opponent finds the map objectionable and suggest the more experienced player play the Romanians. People are intelligent and know how to interpret the data. Yes each review is a single vote, but we all have brains and can read ALL of the reviews to make informed decisions.

You WILL get the occasional bad review, but I would take it with a grain of salt and use it to examine how you could perhaps improve your battle.

The biggest problem I see with the rating system is the fanboy phenomena where buddies of the author pile on superlative reviews. I have learned the hard way to only trust the opinions of certain reviewers.

[ November 17, 2003, 03:04 PM: Message edited by: Keith ]

[DavidFields]

I try not to rate a scenario until I have played it from both sides. Still, it can be difficult to judge, because the first time I play it is "blind" and the second time, necessarily, I know the force structure. (Always single-player) I think a good reviewer will simply note this in the comments section--as a scientist would note his/her methods.

[Michael Dorosh]

Originally posted by Keith:

The biggest problem I see with the rating system is the fanboy phenomena where buddies of the author pile on superlative reviews. I have learned the hard way to only trust the opinions of certain reviewers.

Can you add a feature like they have at amazon, where we can rate the reviews themselves - ie click Yes this was helpful, No this was not?

[Keith]

Good idea! And you should only be able to vote once per review. Maybe have a rating of good reviewers based on the number of votes they get. Anyways, I'm a different Keith than Admiral Keith, the owner of the website.

[GJK]

Originally posted by Michael Dorosh:

Can you add a feature like they have at amazon, where we can rate the reviews themselves - ie click Yes this was helpful, No this was not?

[Ace Pilot]

I think xerxes’ idea of putting in your score is an excellent idea. I think it should be taken a step further and there should be three sets of scores – one for head-to-head play, one for play against the Axis AI and one for play against the Allied AI.

We all know there is no such thing as the perfectly play balanced scenario, so the best we can do is try to beat what the average score is. In my experience, the only reliable average scores I’ve come across are those generated in tournaments. If the Scenario Depot started recording scores and calculating the average, we’d have a source of average scores beyond those generated in tournaments.

[Sergei]

I usually find the textual reviews more telling than numbers.

[Thin Red Line]

Originally posted by Michael Dorosh:

Can you add a feature like they have at amazon, where we can rate the reviews themselves - ie click Yes this was helpful, No this was not?

It would lead to an lesser number of reviews.

[Admiral Keth]

All,

I'm definitely open to new ideas on how we can:

1) Improve or modify the rating system

2) Elicit more reviews from players

3) Elicit more constructive reviews from players

However, right now I'm elbow-deep in my 10-hour-per-day day job, building two complete websites for paying clients, and remodeling the bathroom. I haven't even made beer in the last 3 months. Therefore, any code-based suggestions to modify The Scenario Depot may need to wait until the beginning of next year to implement.

That said, it needs to be remembered that each reviewer is unique. Everyone is going to have a different perspective on the value of any aspect of a scenario. That's why one reviewer will rate a scenario all 10's, while another may rate the same scenario all 1's. When reading reviews, the reader must take into account numerous factors, including:

1) Does the reviewer consistently rate every scenario they review low or high, or do they actually rate each scenario on its own merits?

2) Does the reviewer provide credible and constructive feedback, or do they simply post the numeric values and leave it at that?

What needs to occur is for everyone, authors and players alike, to engage in debate and agree upon a comprehensive and reliable method by which scenarios can be reviewed/rated...and I mean a step-by-step "Here are the values on the form which the reviewer fills out, and here is how these values interact with each other to form a consistent rating." The new system must also include methods by which authors and players can encourage other players to constructively review scenarios.

I do not want to get into coding a system upon with which few will agree, and reviewers won't use.

Something else which must be considered is what to do with the old ratings/reviews. Integrate them or discard them? If integrate, there must be either a volunteer to collate the data and resolve it to the new system, or some automated method to include the ratings/reviews...or just simply attach them as archive material.

The Scenario Depot is for you guys...the players and the authors. You need to tell me what you need, but you also need to help in describing how it can be accomplished.