The "Analysis of Nabla scoring system" thread

[GreenAsJade]

I do think it means that the Nabla system needs an extra scaling step based on the maximum possible score differential.

2. The difference between a player's score and the median score for the side he played will be determined for all scenarios.

The maximum possible value of this difference needs to be calculated, and the final Nabla scores of each player in that side for that scenario divided by it.

This means that the maximum Nabla score for a player for a scenario would be 1.0 ... independent of how balanced the scenario is. At the moment the maximum Nabla score you can achieve is unbounded, and determined by how unbalanced the scenario is.

In retrospect its obvious that this adjustment is necessary: if you can demonstrate that Nabla can result in different maximum scores for different scenarios (which we have done) then you've demonstrated that it doesn't factor out scenario imbalance.

GaJ.

[GreenAsJade]

(third post in a row, sorry!)

... I would suggest that a scoring type person try my adjustment to the ROW scores.

I bet you will find that the result mix looks more balanced, and that the obvious good players still stand out ... but *also* that some of the excellent performances on the "easy" side of unbalanced scenarios get a bit more recognition.

IE I would expect to see some Axis players in the higher ratings of Wet after applying this adjustment, because I'm sure some of them did an outstanding job of holding off all those tanks, yet they had limited ability to outshine the rest of the mob scorewise for doing so.

GaJ.

[ June 11, 2005, 09:40 PM: Message edited by: GreenAsJade ]

[Europa]

I agree with GAJ. If the score isn't normed on both sides of a given scenario then the entire system falls appart. A 100% victory must be given the same score. The tricky part is only to distribute the score between 0 and 100%. The only tweaking needed if that isn't the case is to divide the score with the max of each side resp. and get a -1 -> 1 scoring system.

[Bigduke6]

Here's a possible non-mathermatical explanation of the syndrome:

1. Good players are good precisely because they know how to milk the maximum amount of combat effectiveness out of each unit.

2. All other things being equal the more units a good player has under his control, the more likely it is his superior "unit use efficiency" will allow him to defeat his opponent.

Conversely, the smaller the force, the less likely a good player will be able to squeeze superior combat performance out of his command, as the effects of luck are magnified. Good players and bad players are affected by luck equally.

Along the same lines, the skills of good players have more scope when they have intiative, i.e., when they are attacking.

3. Allied forces in the ROWV scenarios are the larger force, with the exception of Highlanders.

Thus, if my theory is correct, we would expect good players slotted as Allies to outperform the mean less well in Highlanders, than in any other scenario.

Further, at least logically one should expect the good player to outperform the mean most when his numerical superiority is most overwhelming. This obviously is Moeltke.

Along the same lines, we would expect a good player to outperform the norm when he has a wider "mix" of weapons and units to play with. Thus one would expect a good player playing Allies to outperform the norm a bit less well than Moeltke in Wet Triangle, reason being the Allies in Wet Triangle are uniform.

I have no idea whether all this is really true or just a logical exercise made meaningless by the small size of the sample. Some one interested could check the numbers, I suppose. I suspect random stuff make my speculation here meaningless.

But It would seem to me that if you factor the idea that "good players do better if they have numbers and initiative", and add into that random factors like an excellent player winding up on the weak side of an unbalanced scenario, and that might help explain the "strange" result.

For my part the scoring system seems reasonable and I see no reason to question it. I think Kingfish ran a hell of a tournament and I can only hope he changes his mind about ROW V being the last hurrah of the CMBB/CMAK engine. I would play in tournaments like that forever, if some one would let me.

[ June 11, 2005, 11:26 PM: Message edited by: Bigduke6 ]

[GreenAsJade]

BigDuke, you've explained some ways in which different scenarios can give the players different opportunities to score differing amounts.

Confirming, as it were, the phenomenon.

Confirming that Nabla doesn't deal with it entirely.

Having recognised that this happens, in a way that was unanticipated - subtle - and not noticed till now, my thinking is that "someone should do somefink".

I've proposed a solution.

Can anyone shoot it down?

GaJ

[JonS]

I can make it a bit simpler, I think.

Basically, scale all scores for the Allies for any given scen between 0 and 1. Then do the same for the Axis. So, for each scen, one axis player will score '1' (the guy who socred the highest), one axis player will score 0 (the guy who scored the lowest), and everyone else will score something in between.

So, in the case of Malame, JonL would have scored '1' (real score 96), BigDog944 '0' (real score 10), and dieseltaylor '0.6977' (real score 70. 0.6977 = (70-10)/(96-10). And so on.

Same for the Allies.

Note that in this case opponents scores would not be the reciprocal of each other. A players NABLA score would be based purely on his comparative score against other players on his side.

Having said all that, it probably doesn't make any difference. I think that the existing system is fair, and that the best player from each group has gone through. In that sense it doesn't matter that Walpurgis scored more than JonL, NABLA-wise, in Malame.

Jon

[JonS]

Bugger. On a quick play with the Malame scores, that doesn't work. The total of all allied players points comes to 24.3953, while the total of the axis players points is only 11.6047, meaning that the avearage allied player gets twice as many points as the average axis player. Aka bias.

Re-normalising those results against each other might work (so that the total points available to the allies and the axis is the same).

I guess that's the point isn't it? The total points available for each of the Allies and the Axis within each scenario should be the same, and furhter that each scenario should contribute (in the case of ROW V) 20% of the total pot available to each side.

[ June 12, 2005, 12:49 AM: Message edited by: JonS ]

[JonS]

Originally posted by JonS:

Note that in this case opponents scores would not be the reciprocal of each other. A players NABLA score would be based purely on his comparative score against other players on his side.

That's not true either Each opponent pair of scores sums to 1.

[Flenser]

If I follow the argument correctly here (and forgive me if I don't, its rather late) you want both sides to be able to achieve the same score.

If you do this, then you _will_ invariably punish the side having the harder time of it in the scenario.

Take an extreme case (maybe not that extreme given some of the scores out there) where a scenario the Allies score an average of 90%, Axis 10%. Now put some Walpurgis-like player on the Axis side who works like mad, fights like hell, gets a little lucky, etc and scores 100% for axis. By your reckoning, that score would be the same as Allied player who does a mere 10% better than average.

Seems a bit unfair to me.

Unbalanced scenarios should yield unbalanced scores. I think NABLA does reasonably well in this regard, but there may be improvements that could be made. Its too late for me to delve into it. I am certain, though, that its the right direction to take.

[GreenAsJade]

Flens, you've hit the heart of it. What is fair in this situation you describe?

Note that this is a tourney. You've just given WalpugisLike a score of 10 points in a scenario because he scored 100% in a game where the average for his side was 10% , and the LondonerLike a score of 1 because he was on the opposite side, and scored 100% in a game where the average for his side is 90%.

Now tell me how that's fair.

Due to the scenario balance, LondonerLike could not have scored any more than 10% more than the average. This means that the pure luck of which side he drew ruled him out from moving to the finals while WalpurgisLike goes to the top with his fantastic score.

Again - how is that fair?

I would argue its fairer that the person who did the bestest on the side WalpurgisLike was on should get the same score from that scenario as the person who did bestest on the opposite side.

Nabla scoring is INTENDED to allow use of unbalanced scenarios in a tournament. That is what it and ROW are all about. It's not as if unbalanced scenarios are throwing it all out. Unbalanced scenarios are supposed to be taken care of.

But we've just seen that even with the current Nabla method, unbalanced scenarios give unbalanced results: in fact with the system as it is, it is a big advantage to be assigned to the "worst" side in an unbalanced scenario. It's overcompensated.

I guess this debate is about how to find the best way (if any) of compensating properly.

GaJ

PS

Unbalanced scenarios should yield unbalanced scores.

No, that's just wrong. ROW and Nabla are supposed to allow a "fair" competition even with unbalanced scenarios. The person who by luck is assigned to the easy side in an unbalanced scenario should have no more chance of winning the tourney than the person who lucked the hard side.

[JonS]

Originally posted by Flenser:

If I follow the argument correctly here (and forgive me if I don't, its rather late) you want both sides to be able to achieve the same score.

Yes, but it's a bit more subtle than that. The best score from each side would score the max available, and the worst score from each side would score the least available. So if the scen was horribly unbalanced, and someone had to fight like buggery to eek out a meagre draw, they would score the max available. OTOH, and from the other side, you would have to do exceptionally well to score the max available from the 'easy' side. In other words, players on the easy side would have to fight like buggery too.

[Kanonier Reichmann]

Originally posted by JonS:

Bugger. On a quick play with the Malame scores, that doesn't work. The total of all allied players points comes to 24.3953, while the total of the axis players points is only 11.6047, meaning that the avearage allied player gets twice as many points as the average axis player. Aka bias.

Re-normalising those results against each other might work (so that the total points available to the allies and the axis is the same).

I guess that's the point isn't it? The total points available for each of the Allies and the Axis within each scenario should be the same, and furhter that each scenario should contribute (in the case of ROW V) 20% of the total pot available to each side.

I think this summary is the best I've seen to date. I'm sure calculating the difference between the total Nabla points available for the Axis & Allied sides in Wet Triangle would produce similar large differences. Therefore, to a significant degree in unbalanced scenarios, the ability to score really well in a battle comes down to which side you're lucky enough to draw at the outset.

This doesn't seem right to me from a logical point of view. In theory, a player could perform exceptionally well in just 2 scenarios that happened to be unbalanced playing the non-favoured side and obtain scores of around 2 to 2.5 in each but then simply hold his own or even nominally lose the 3 remaining scenarios that were relatively balanced. The end result would be a very good score of around 4.0 that should be enough to win the group yet another player in that group could have consistently done well in all 5 scenarios winning all 5 with good margins yet miss out because he wasn't lucky enough to play the unbalanced scenarios as the non-favoured side.

Note: this is simply trying to highlight the flaw in the Nabla scoring system but in no way should be interpreted as a request for a review of the current scores since we all agreed to the scoring system from the outset in ROW V. However, if it can be adjusted for future iterations of ROW tournaments by somehow scaling back the potential extreme results in unbalanced scenarios that would be a good thing in my opinion.

Regards

Jim R.

[Sivodsi]

The main issue here is: does the Nabla system achieve what it is intended to do?

At the moment it is intended to choose the one strongest player in a group of 5 people. We can say that it probably does this, since it gives a bonus to those who do better than we might expect them to do, and the better they do the bigger the bonus. Even if those playing the disadvantaged side have the potential to gain more Nabla points, whether or not a player is in that situation is decided at random.

[Edited to add: after I did the figures below I realize I am talking bollocks here. My post below shows that is a large difference between the top scorers on boths sides in an unbalanced scenario. The difference between doing well on the unfavored side vs the favored side could well be decisive]

The wider issue (and one for the next tournament) is; does the Nabla system choose the best players in the tournament?

Here we have a tantalizing glimpse at comparability between the groups because they are all worked out in relation to everybody's scores, so it seems like we ought to be able to. But to answer this question we need somebody with a deeper understanding of the statistical methods involved (Er, not me).

I suspect that somebody with a knowledge of 'Rasch analysis' would be able to help us with a system that allows us to identify the strongest players overall without having to just peel off the best single person in each group.

Anybody know a professor of statistics ?

[ June 12, 2005, 04:31 AM: Message edited by: Sivodsi ]

[Sivodsi]

"Even if those playing the disadvantaged side have the potential to gain more Nabla points, whether or not a player is in that situation is decided at random"

Damn! as soon as you say these things you realise its not as straight as it seems.

When I said this, what I mean is that a strong player will do better than expected whether he/she is playing with the disadvantaged side or not, but if the strong player is with the disadvantaged side they stand to gain more. How many more points you could gain is an important question [edited to add: see figures in my next post]

I guess the problem comes when they are two strong players in a group, who could equally go through. In this case, under the current system, the one playing the weaker of the unbalanced sides would have the advantage of a greater potential to gain points.

[edited embarrassing grammatical mistake]

[ June 12, 2005, 04:32 AM: Message edited by: Sivodsi ]

[Sivodsi]

Okay here are the top Nabla scores for each side per scenario, with the median scores in brackets:

Wet Triangle

Axis (72): 1

Allies (27): 1.94

P to M

Axis (78): 0.72

Allies (22): 2.74

St Ed

Axis (52): 1.41

Allies (48): 1.81

Highlander hell

Axis (51): 1.86 [oh! that's me!]

Allies (49): 2.01

Bridge

Axis (45): 1.31

Allies (54): 2.59

Here you can see that the biggest difference is in Push to Maleme where the top scoring allies score is 3.8 times the top axis score. The question that Nabla forces us to to ask is "was the best allied performance in this scenario really 3.8 times better than the best axis performance?"

But of course, the point is that some people had the chance to play the sides with the potential to gain this many points, whereas others did not.

Whether this lead to anybody winning the group instead of another person is something I'd rather not look into.

[dieseltaylor]

RoW is great and I have no problem with the results of RoW5. I think it is a brilliant concept and a worthy international tournament.

That there is a problem with Nabla is indisputable in that , as shown in the Maleme example, that maximum scores are limited by which side you play.

I am hypothesising that if each player played four games and each scenario was approximately the same amount biased then providing everyone plays the strong side twice and the weak side twice the Nabla system would work properly. However this is arranging scenarios to fit the scoring system which is not really the most obvious route to go.

It does have the virtue of recognising the current deficiency.

There does seem to be mileage in the concept of looking at the percentage of possible points gained above the median rather then absolute points gained over the median score.

I will ponder more : )

[dieseltaylor]

A little doodling shows that the 5 winning Allied scores amounted to 12.10 points and the Axis to 6.30.

Obviously you get to play both sides so the worst possibly combination where you are the top winner in all your games is 6.85. This is based on the playing three Axis and two Allied and taking the lowest topmost score.

Funny old thing statistics : ). May I reiterate my appreciation and support of the splendid organisers. This look at the stats. is simply a look at the possibility of future improvements if possible/necessary etc.

[Sivodsi]

"Obviously you get to play both sides...

I'm sure I read somewhere that the side you play is totally random, so there is a chance that someone has to play axis or allies all five games.

Not sure whether this actually happened though.

[Malakovski]

Originally posted by dieseltaylor:

RoW is great and I have no problem with the results of RoW5. I think it is a brilliant concept and a worthy international tournament.

First, let's underscore that again. It's really a testament to both the scoring system and the tourney that it's been through five iterations without anyone noticing this potential flaw. It also took WN's special gift for scoring over 90% in any and all circumstances to really underline it.

I've only got a minute here, but lots of good stuff was posted after I went to bed and I wanted to comment on a couple.

Originally posted by JonS:

Bugger. On a quick play with the Malame scores, that doesn't work. The total of all allied players points comes to 24.3953, while the total of the axis players points is only 11.6047, meaning that the avearage allied player gets twice as many points as the average axis player. Aka bias.

This is an interesting stat, but not exactly relevent. The issue is the potential for big scores being present for one side and not the other. There's no question that very few players will be able to get such scores, and hence that the totals look like the above is not a big surprise.

Originally posted by dieseltaylor:

A little doodling shows that the 5 winning Allied scores amounted to 12.10 points and the Axis to 6.30.

This is the relevant stat for our problem.

I think the problem has been well established in this discussion. The only topic left is what--if anything--can be proposed to address it. Now that we have the raw scores, it will be easy to test out any proposed system on real results.

Well I'm off to breakfast. I'll be back to ponder this more later...

[Elmar Bijlsma]

Originally posted by Sivodsi:

"Obviously you get to play both sides...

I'm sure I read somewhere that the side you play is totally random, so there is a chance that someone has to play axis or allies all five games.

Not sure whether this actually happened though.

I don't think so. The couple of people I heard from have been playing with one side four times and just once with the other.

[Malakovski]

Originally posted by Elmar Bijlsma:

I don't think so. The couple of people I heard from have been playing with one side four times and just once with the other.

Yes, I drew Axis four out of five. It's bound to happen to a fair number of people. Odds of drawing all the same side are only 32:1 or so.

[dieseltaylor]

And the flip side is that that score is debited to his opponent. So in Maleme WN opponent was -2.74 but then BigDog944 won all four of his remaining games and yet ends up at -0.43.

That would suggest looking at the overall list that he is not that good which seems a travesty given his subsequent victories.

[Flenser]

Originally posted by GreenAsJade:

ROW and Nabla are supposed to allow a "fair" competition even with unbalanced scenarios. The person who by luck is assigned to the easy side in an unbalanced scenario should have no more chance of winning the tourney than the person who lucked the hard side.

So your choices are:

1) A normalized system (all players score from 0-1) attempts to get rid of the "luck of the draw", but punishes the player "lucky" enough to draw the harder side to play.

2) A NABLA like system, which, perhaps, punishes the player "lucky" enough to draw the easy side, though I prefer to see it as "rewarding the tougher side" (semantics).

Either way, the luck of the draw is a factor, as is who gets thrown into your section, and there is no way around that unless you want the tourney to last a decade and require the resources of a small nation.

Given that luck is a factor in both systems, I'd prefer the NABLA method, personally, as it tends to reward outstanding achievement. How much that reward should be is up for debate.

[Malakovski]

Originally posted by Flenser:

Either way, the luck of the draw is a factor, as is who gets thrown into your section, and there is no way around that unless you want the tourney to last a decade and require the resources of a small nation.

True, luck cannot be eliminated, and a fully normalized system would have it's drawbacks.

I think the solution would be to curtail the extent of the big nabla scores without going over to a fully normalized system.

A big score is one thing, but at present the scores are so big that one big win is enough to win a section, or to lose a section if you get beat. Getting a 90% win in a scenario in which the average for your side is 30% is a major accomplishment, but should a victory in one scenario decide an entire round of the tourney? It didn't happen this time, but you could have a case in which someone got a big win and then lost by a small margin in all the others and still won the section.

[Malakovski]

Well judging from the other thread, this issue is getting out of hand.

I suggest we postpone further discussion of the nabla system until ROW V is concluded.