AI cheats! (with real data)

[Warren Peace]

Reduced chris's range to 6 vs. 6 battle (took out pill-boxes). Played six times each side.

As Russians

28 t34 vs. 9 stug

As Axis

23 T34 vs. 14 stug

chi2 is only 0.2, but numbers are still small. Still, identical trend as with my small battle set up.

[CombinedArms]

Originally posted by Treeburst155:

Surprising development!! The human turns in his best performance yet. Not only that, the AI has its worst round!

After 1,000 shots, the AI first round HIT percentage is 37.6%

The human first round hit percentage after 1,000 shots is up to 34.2%

The percentage difference is now only 3.4!

Clearly, the AI and I will have to each fire 500 more times. Maybe even 1,000 more times.

Treeburst155 out.

I think the significant factor here is that the human (or the AI) player can HAVE a GOOD ROUND over as many as 200 trials. In other words, in a sample of this size, there can be significant variability in the outcome. I'm no mathematician (I can't even spell it) but that suggests that even 1000 tests just may not be enough to yield a definitive result. Looks to me like the AI may have just started with a couple of really good rounds, and now we're settling back toward parity. Only TB knows if its worth the effort to him to truly find out. In any case, I think even if the AI has a 3.5% advantage, I can live with it.

[Treeburst155]

You are right, Combined Arms. The gap is too small to be trusted with ONLY 1,000 rounds fired. I shall press on to the truth! Geez, I just love blue collar statistics.

BTW, the worst human performance was 60/200. The best was 77/200. Worst AI performance 70/200, the best was 82/200.

Treeburst155 out.

[Pascal DI FOLCO]

Originally posted by Battlefront.com:

Warren Peace,

</font><blockquote>quote:</font><hr />This advantage appears magnified in small battles. Are you sure this is not possible? Not a cheat but more a possible undiscovered programing oversight.

I don't see how it can be possible. The way the code is set up there is no difference. The gunnery and ballistics calculations don't know who is controlling the forces they are calculating. The one part of the code asks another part "this tank is shooting at that tank, what happens?". The other part of the code states an outcome and that is that. There is no way that I can see how this system would have some sort of (unititional) bias introduced. I'll ask Charles, but I think the chance of there being a problem here is just about zero. If there are some differences in the numbers you guys are seeing there must be another reason for it.

Steve</font>

[Lt. Kije]

Originally posted by Treeburst 155:

BTW, the worst human performance was 60/200. The best was 77/200. Worst AI performance 70/200, the best was 82/200.

I picture a pair of bell shaped curves, one shifted a bit to the right of the other.

-- Lt. Kije

[ October 31, 2002, 04:58 PM: Message edited by: Lt. Kije ]

[Guest Mike]

TB in your initial post you said you'd figured out a way of couting hits - what is that - Mk1 eyeball of each mad minute, or somethign more devious and intersting??

[Treeburst155]

Mike,

I'm counting the number of detailed armor hit messages. The last messages appear towards the end of the 7th second. The first ones disappear late in the 9th second. By counting the messages at all three points I can be sure of the hit count. Probably checking the 8th second would be good enough, but I look at 7 and 9 too.

Also, I've checked dozens of times to make sure all rounds were fired by the 7th second. That is definitely the case.

I've also run dozens of turns to completion to see if there were ever any late hits. It never happened.

It's actually quite easy to count the hits, especially with all the practice I'm getting.

_______________________________________________

The human continues to chip away at the AI lead with another strong performance. Is the AI folding under pressure, or is the human bearing down?!

After 1,200 shots the human first shot HIT percentage is 34.75% (up from 34.2)

The AI slips just a bit with a first round hit percentage of 37.5% (down from 37.6) after 1,200 shots.

The percentage difference is now only 2.75, after 1,200 shots.

Treeburst155 out.

[ October 31, 2002, 05:44 PM: Message edited by: Treeburst155 ]

[karch]

Jeez, you'd think that after 1200 shots they'd have the range nailed and all get first round hits wouldn't you?

[gunnersman]

Stop, just stop for the love of God people! Please! Oh the humanity! Save yourselves while you can! Dont let it consume you! There is hope!

Hey! Look! A T34/85 mod!

[ October 31, 2002, 06:27 PM: Message edited by: gunnersman ]

[Treeburst155]

Do T34-85 mods affect first round hit percentage? I'll have to test that next.

Both the human and the AI managed to raise their first round HIT percentage in the latest round; but the gap was still closed ever so slightly by the human.

Human first round HIT percentage after 1,400 shots: 34.86%

AI first round HIT percentage after 1,400 shots: 37.57%

Percentage difference after 1,400 shots: 2.71

Treeburst155 out.

[L.Tankersley]

It's sounding more and more like the (true) hit probability is equal for AI and human. This is what I expected; as Steve mentioned, the ballistics code should be the same for AI and human. Where I think there might be a discrepancy is in the time it takes to get a shot off (even if the human issues a targetting order). Unfortunately it's not easy to gather accurate data on just how long it takes to get a shot off because of the granularity of the CMBB movie clock. However, I think it would probably be feasible (for someone with altogether too much time on his hands, such as the estimable Treeburst) to set up an experiment to see which tank of a pair fires first.

I would suggest a setup similar to Treebursts' but with each tank (AI and human) both having a single round of AP. By watching the turn movie from high overhead (view 8 or 9) with unit size at +4 you should be able to easily see the smoke plume when a shot is fired. (It might help to turn on fast, compatible smoke with shift-I.) There would be three possible outcomes for each trial (pair of vehicles): AI shoots first, human shoots first, or shot fired simultaneously (within the ability of the tester to judge, anyway). Identical vehicles should definitely be used, and the test should probably be run from both sides (human as Axis and as Allied). If a few hundred trials result in essentially equal chances, then I think we could probably put the issue to bed and write the various "curious" results we have seen off to confounding factors rather than fundamental differences in the code.

[Treeburst155]

How about this to test time to first shot:

Helpless targets at 200 meters so hit % is high. Veteran crews, again for hit percentage. Let the AI fire (20 lanes), and record the second in which the first detailed armor hit appears. Then, I will play the firing side and see if there is a difference. Of course this will have to be repeated many, many times. Differences of less than a second might be difficult to notice if they do not span two distinct seconds of the turn.

BTW, my current first round HIT test already includes only one AP round per firing vehicle.

I'm still not quite satisfied yet with my test. I'll move on to a "time to shoot" test after I'm finished with this one.

Treeburst155 out.

[Treeburst155]

The gap continues to close.

After 1,600 shots the AI first shot HIT percentage is: 37.06%

The human is continuing to improve with 34.94% after 1,600 rounds.

The gap: 2.12

Treeburst155 out.

[Silvio Manuel]

Originally posted by Battlefront.com:

There is no way that I can see how this system would have some sort of (unititional) bias introduced. I'll ask Charles, but I think the chance of there being a problem here is just about zero. If there are some differences in the numbers you guys are seeing there must be another reason for it.

Steve

I would say, Hi Mom!, except I think the Stat Grogs are having a field day, three-legged races and all.

[Treeburst155]

Inexorably the gap closes as we approach the truth. The AI cannot keep up it's artificially high percentage. On the other hand, the artificially low human percentage continues to rise with even moderately good showings in a given round. Even an extremely good showing by the AI would be just a hiccup with this many shots behind us. The percentages will continue to converge IMO.

Human first round hit percentage after 1,800 rounds: 35.11%

AI first round hit percentage after 1,800 rounds: 36.94%

The difference: 1.83 percentage points

I am now satisfied that the AI does not have an advantage over a human when it comes to the probability of a first round HIT against a sitting duck target given the EXACT same situation.

Treeburst155 out.

[ October 31, 2002, 09:01 PM: Message edited by: Treeburst155 ]

[Warren Peace]

Ok, my final post on this topic

I tested the idea that the difference between the AI and human is related to global morale.

I used my modified Chris scenario with only 6 tanks per side. There were two situations. In one I added 2000 points worth of pillboxes to each side and in the other their was 0 points added. I played each scenario 12x from both sides. I used manual targeting on all tests.

Results

With no pillboxes

Human as Axis

47 T34 vs. 28 Sugs

Allies as Axis

57 T34 vs. 18 Stugs

THis result is consistant with what I have seen all along.

With 2000 in added pill boxes

As Axis

56 T34 vs. 19 stugs

As Allies

49 T34 vs. 32 Stugs

This is the first time I've seen results in which the human actually performed better then the computer (although I think this is probably just statistical noise).

I suspect that the global morale effect is only working for the human and is not engaged by the AI. I suggest that BFC examine this possibility.

No more testing for me. I'll be happy to send me scenarios to someone if they want to further explore these observations.

[Treeburst155]

Whe Gets The First Shot? Test #1

The Test:

1 Captured T-34/85 faces off against 1 Russian crewed T-34/85 at 200 meters. Both crews are veterans. Both are unbuttoned. Time is June '44 in hopes that a Russian vet is as good as a German vet by then. The hit chance is 87% for both (Kill Good).

Human German player manually targets Russian vehicle. The German armor fired first 18 out of 20 times. In the two instances where the Russians fired first, the Germans still managed to get off a round. They were just a half second or so behind the Russians. In four of the 18 German "firsts", the Russians managed to get off a round a fraction of a second later (before dying).

This preliminary test would seem to show that a manually targetting human gets a quicker response than the AI, given the same vehicles and crew quality on both sides.

Other possible causes for the lopsided results:

Russian vets may not be as quick as German vets, even in June '44.

The German SIDE may have a quicker response time regardless of whether it is AI or human controlled, and regardless of what vehicle is used (the same vehicle for both sides, of course).

The captured armor may not be the exact same vehicle as the Russian armor. The penetration charts are different. Perhaps the ROF is also different due to German improvements. This may tie in with the possible quicker response time for the German SIDE mentioned above.

Not enough samples? This one is REALLY lopsided, but maybe things would level out with a bigger sample.

Now to test with the Russian side under human control.

Treeburst155 out.

[ November 01, 2002, 01:26 AM: Message edited by: Treeburst155 ]

[Louie the Toad]

What all of this is telling me is that I should only move my forces and let the AI do the shooting for me. RIGHT !?

OR does the AI playing on its own, do better than the AI / human player partnership?

Do you ever 'sense' that when the AI plays your forces (as it does from time to time) vs the AI opponent (which is of course itself), that you come out on the short end?

twilight zoning.......... Toad

[Treeburst155]

Interesting results here from the human controlled Russian side.

The Russians got off the first and only shot 9 times.

Close calls going to the Russians as described above in the other test occurred 5 times.

German first and only shots occurred only 1 time!

The remaining 5 trials resulted in virtually simultaneous fire. If I had to choose who fired first I would give it to the Russians 3-2 for these "simultaneous" shots. We're talking a tenth of a second here I think.

The interesting thing is that the Germans only got off the first and only round ONE TIME when the human switched to the Russian side. Compare this to the 14 times the Germans got off the ONLY shot while the human was playing the Germans.

The poor AI is getting cheated bad here! Apparently, where the human goes, is where the majority of first shots will go, at least when the human manually targets.

OK, statisticians, how many samples do I need from each side for this one.

Treeburst155 out.

[ November 01, 2002, 01:09 AM: Message edited by: Treeburst155 ]

[Michael Emrys]

Originally posted by Louie the Toad:

What all of this is telling me is that I should only move my forces and let the AI do the shooting for me. RIGHT !?

Not unless losing is what gives you a thrill. All the evidence seems to point that if you want to do as well as the AI all by itself, you should manually target if you have the chance to.

Michael

[Guest Sgt. Emren]

... 47 T34 vs. 28 Sugs

Allies as Axis...

Okay, Warren, you may take a break now!

[thewood]

What I find most interesting is just the passion in the thread itself. I have played many wargames from ASL up to the GIC demo and I have never seen as passionate and intelligent discussion, as well as unbiased, as what I see SOMETIMES on this board. While this board, like any, ranges from immature to PENG, I am continually amazed at threads like these where people are here to learn and teach without a bad attitude.

I continue to beleive that BFC has created one of the few true communities for gaming, although Falcon4 has done it despite its developers stupidity. Anyone who doubts me, take a look at the "I know you are, but what am I?" threads over at GIC. I have followed CMBO since its original vision as Computer ASL. I continue to see the evolution of the community.

To foster the community, I hope someday that Steve and Charles have a "user conference" in Bangor, Maine. Its a great central location between the west coast and Europe. Its cheap to fly in and its near the original birth place of CM. I would love to meet some of these guys in person and share a Sam Adams.

[Soddball]

I ran a couple of tests.

First I fired up a few 500pt QBs vs the AI.

Then I tested my reaction times reaching the fridge for beer. I did this twelve times.

Then I fired up a few more 500pt QBs vs the AI.

I got ****ed over badly. This goes to prove that the AI cheats by not drinking beer.

[Jarmo]

Interesting. Apparently you do need more than 1000 repeats to get "close enough" to the truth.

Will be really hard to prove anything about CM now on..

Treeburst, I assume you have the hit scores in Excel or something similar.

Could you check what the hit probabilities are if you ignore the 500 first tests?

Seems the AI got tired of cheating after the beginning.

(Perhaps worried she might get caught? Clever biatch.)

[Treeburst155]

LOL at Soddball!!

Your conclusions are wrong. The AI does not cheat because it doesn't drink beer. You are just bonkers for going into combat drunk.

Treeburst155 out.