Scenario Balance Indication List (CMBB)

[GreenAsJade]

Over at We Band Of Brotheres, a fantastic list of scenario game results is available - the result of every game played by the club members.

Using that (with great help from the club President in getting the raw data), I've been able to collate this

<a href="http://www.gregories.net/CMBB/CMBB-Summary.htm" target="_blank">

Scenario Balance Ratings List</a>.

What this list shows the "Imbalance" of each scenario, determined by how far from 50% the average Axis and average Allied scores are for that scenario.

So, for example, a scenario where the average Axis score is 49% and the average Allied score is 52% has an imbalance of 3.

If a balanced scenario is what you're looking for, then I think you're probably after an "Imbalance" of less than 10 or 15.

I've put the scenarios first into groups depending on how many games were played - this determines the "Significance" of the result. If only 3 games have been played and the Imbalance is showing up as "40" I wouldn't consider that very "Significant". If 10 games have been played, I'd be considering it quite significant.

Within each group, the table is ordered by increasing imbalance.

This means that the scenarios at the top are pretty sure to be balanced scenarios.

I hope this is valuable to someone - I'm certainly using it to pick candidate PBEM games...

Congrats to Hans on topping the "most balanced" list with his designed-to-be-balanced-and-fun "Pershotraveneve Ridge". It's nearly the most played, and conclusively the least Imbalanced...

GaJ.

(scenarios where less than 4 games were played are not shown on this list, because the result is unlikely to tell us anything conclusive about the scenario)

[ July 01, 2005, 06:46 PM: Message edited by: GreenAsJade ]

[GreenAsJade]

One mysterious result is "Two Hostile Brothers".

This scenario, which my opponent and I did not enjoy enough to play to the end, is supposedly a complete mirror image map.

I wonder what has led to it resulting in such an imbalanced outcome?

[GJK]

Is this populated dynamically? It appears to be static. I ask because it would be nice to be able to resort the list by column header.

This is very similar to the old ASL ROAR list, which is a very useful tool for ASL players.

Of course, the results are very dependant upon the skill levels and balance of the players themselves. I think a small tale tell sign should also be the min-max values. What to think of scenarios that have a minimum score of 6% and a max score of 90%?

Good job!

Edit: I see that this is a straight Excel-web transfer, thus answering my first question. Easy enough to copy/paste into Excel for your own sorting, but that's not an option for everybody of course.

[Ace Pilot]

Thanks for posting this, GaJ. Looks very useful.

Could you explain how the Significance value was calculated?

Just to be clear, this is all PBEM or TCP/IP games, correct? No play against the AI?

[GreenAsJade]

Correct - the information is about human vs human games.

The "Significance" groups Scenarios together based on the number of games played for that Scenario. Scenarios are grouped in bigger groups when they have more games played (because instinctively there is a lot of difference between an average of 4 games and the average of 8, but not much different between the average of 8 and the average of 12 games).

IE scenarios with 4-7 games played are grouped together, scenarios with 8-14 games are grouped together, scenarios with 15-30 games are grouped together.

The choice of grouping is utterly arbitrary - it's what turned out to be easy at the time and felt kinda right. Any suggestions for improvement welcome.

GaJ.

[GreenAsJade]

Originally posted by GJK:

Of course, the results are very dependant upon the skill levels and balance of the players themselves. I think a small tale tell sign should also be the min-max values. What to think of scenarios that have a minimum score of 6% and a max score of 90%?

This is why an average of lots of games is important. These results are coming from effectively a random selection of players: people in a club who decide to play against each other. There will be results skewed far by unbalanced opponents, but these will become less significant on the average.

I don't see a problem with a Scenario where the Max Allied score is 90% and the Min Allied Score is 6% ... this means that opponents of vastly different skills played Allies on different occasions. On average, though, if the Allied Score is 80%, then mostly the Allies win, which to me is not a balanced scenario.

[GreenAsJade]

Originally posted by GJK:

Is this populated dynamically? It appears to be static. I ask because it would be nice to be able to resort the list by column header.

If there's enough interest, I could set the page up with the columns sortable.

(I'd have to figure out how!)

In the mean time, as you say: paste into a spreadsheet and go for it.

GaJ.

[Treeburst155]

This is a great thing you have done!

Extremes don't matter. Only the average score for the sides after MANY plays of the scenario by different pairs of players. By the time a scenario is played a dozen times by a dozen different pairs of players, I'd say balance has been fairly well established.

Treeburst155 out.

[Spookster]

Neat, but...

You've created a circular self-selection problem in rating your games. As it stands now, people tend to play games that are "Balanced", and stay away from the "Imbalanced" ones, even though most the imbalanced ones (many of which are my favorites)do NOT have a big enough sample size to make a determination on their relative fairness.

Another problem, and I've found to be a big one: Maximizing balance does not always maximize enjoyment. It is a huge component, but I've played PBEM games with good balance that were not necessarily better than an "imbalanced" one, like: "Message from Geotz", which has a ton of drama, intensity and charm.

You know all this, I'm sure. And, of course, you are interest in measuring the best head to head scenarios for your player rankings. How about this?:

Allow players to play all the scenarios, even the so-called imbalanced ones. Win or lose, nothing is recorded until that scenario has a significant sample (which I assume your stats guy can determine). Then, previous games can be compared against a meaningful average. Wins and losses can be determined relative to other games.

That way, players can play a greater diversity of scenarios and be less afraid of "imbalance" issues.

Just my 2 cents.

-Spookster

p.s. BTW: Great job with your site. If I ever played as much as I did in the BO days, I'd join.

[GreenAsJade]

Note that this list has no input into what scenarios are played other than the players themselves choosing to use it as a guide. Players can choose whichever scenario they like from wherever they can find it. This list just records the results after the games.

And, as you said: balance is only one factor in picking a game to play. I would fully expect people to also research the scenario via the reviews at the depot etc. Also, the sheer number of times a scenario has been played is some indication of how good it is (another reason for putting scenarios with lots of games above those with fewer).

My intention with this list was to at least give people the opportunity to get some better information about balance while choosing a scenario.

In theory this could lead to less data about scenarios that initially appear imbalanced because people shy away from them, but in practice I doubt if this is a problem. The reason is that there are still so few scenarios that appear to be balanced! After you've played the few scenarios with a significant balanced result, you will then find yourself looking down the list for other less-played scenarios, and this will improve the data for those scenarios.

Possibly I should require _more_ games played before displaying a result, so that if the first couple of results make a scenario appear imbalanced when really it was the players who were imbalanced then that scenario doesn't get mistaken for an imbalanced one. However, I think that by the time a scenario has been played 4 times you are starting to get some indication of the way that it goes.

Cheers,

GaJ.

[Spookster]

Greenie,

No slight intended. Cleary your data is helpful for the reasons we both agree. I also imagine, since yours is a close-knit group, word of mouth does spread regarding most balanced scenarios and, since (like it or not) the interest in CM is tapering off, I doubt you can get big samples on the assumed "imbalanced" scenarios.

Cheers,

Mike

[GSX]

I just dont think the samples of any other than maybe the ones played over 20 times can give a representative picture of balance or not. The trouble with this as i see it is the results of games can be scewed by many factors. If a scenario had been played 4 times but twice by the same player, who played the same side and got a bigger winning margin, its not a pointer to balance, but experience etc. Again the experience levels of players would have to be taken into account. Its a very usefull pointer to have but Im not sure how representative it really is. Unless of course really large samples are used.

[GreenAsJade]

It's just another piece of information that you now have that you didn't have before about how balanced a scenario might be.

It's certainly not definitive. To make that clear, I've changed the topic of this thread to "Scenario Balance Indication List" instead of "Rating List".

However, here is a challenge for you, if you don't believe the results. Go look at the force balance ratings that players have subjectively given the scenarios at The Scenario Depot. See how well they match the results reported here.

I think you'll find that the level of correlation is suprisingly good for games with far less than 20 played.

In fact, so far only a couple of results have stuck out to me as "questionable - needs a closer look". A discussion on other anomalies that you think you can find would be interesting....

In other words, instead of just saying "you don't think these results mean much", how about an example to show why?

GaJ.

[Joachim]

Originally posted by Treeburst155:

This is a great thing you have done!

Extremes don't matter. Only the average score for the sides after MANY plays of the scenario by different pairs of players. By the time a scenario is played a dozen times by a dozen different pairs of players, I'd say balance has been fairly well established.

Treeburst155 out.

Hmmm... Extremes don't matter but as a statistician I'd like somehting more than just an average (where the extremes do count).

GaJ could you add the variance? It can tell more than the extremes.

Or even better I'd like to have the median and the deciles. Knowing the range where 80% of games ended would tell more than the extremes - and it is easier to comprehend than the variance. An advantage is that scens played less than ten times have their extremes as deciles - which automatically leads to a huge range which in turn is interpreted as "no secured information from this sample".

(Lower Decile = at least 90% are bigger or equal, at least 10% are smaller or equal. Upper decile = at least 90% smaller or equal, at least 10% are bigger or equal. E.g. look for the percentile operator in Excel)

Gruß

Joachim

[GreenAsJade]

In principle I can do anything you ask, it's all just code

I'd like to do something that ends up being easy to present and understand in the limited display format that I have available.

Can you suggest something specific that would make sense, and that would not result in a massive table of numbers for people to sift through?

(I'm also contemplating weighting balance results by player rating disparity.... I may have the player ratings available for each player that participated in each battle. These ratings are only issued once players have 10 results, so they are pretty good. )

GaJ.

[Josh Coady]

Has this list move to another URL? The one given in the original post seems to be broken. TIA!

-Josh

[GreenAsJade]

Sorry about that - gregories.net changed host recently, and there was this minor hiccup. I've edited the link above - should be right now.

GaJ