Warren Peace
-
Posts
205 -
Joined
-
Last visited
Never
Posts posted by Warren Peace
-
-
Treeburst,
I counted vehicles that routed off the map as kills. I never encountered crews in the act of bailing by the time the third turn was over. They either have bailed by then or their not going to.
Warren
-
Treeburst
Did you get the scenario? I E-mailed it to you.
Any tank that the crew has bailed out of or has been destroyed is counted as killed.
Warren
-
Nothing of the sort. I play the scenario until all firing stops (usually two; at most three turns. I see how many of my tanks are alive. I surrender and then see how many of my oppenents are alive. Then I play again. With 6 tanks it does not take very long.
My comment about abandonment is simply an observation that often after the first turn I will have an immobile tank with the crew inside, and on the next turn the crew will immediately abandon the tank rather than continue to shoot.
-
Elbrus,
Actually if you sum up all of the six-tank tests I have performed I have done at least 30 for each side. The result is very consistent. I'll be happy to send you the scenario for your own testing if you wish.
Treeburst,
20 is too many tanks. Global morale issues will be swamped out because each tank destroyed will have a smaller effect on morale. Try a six vs. six tank battle. I will send you my scenario when I get home.
Warren
[ November 01, 2002, 03:38 PM: Message edited by: Warren Peace ]
-
Steve,
I think everything is totally consistant with a global morale problem. I am simply counting how frequently tanks are abandoned. If global morale were lower wouldn't that lead to crews panicking quicker and abandoning their tanks quicker? I have noticed that if tanks get immobilized in these battles the crews almost always abandon them before the tank actually gets penetrated.
Warren
[ November 01, 2002, 03:14 PM: Message edited by: Warren Peace ]
-
Treeburst,
I will be happy to send you my scenario's for replication purposes if you wish. I don't have a website I can post them on, but I could E-mail them to you.
Replication is the key to science.
Warren
-
I had frosted mini-wheats!
-
Ok, my final post on this topic
I tested the idea that the difference between the AI and human is related to global morale.
I used my modified Chris scenario with only 6 tanks per side. There were two situations. In one I added 2000 points worth of pillboxes to each side and in the other their was 0 points added. I played each scenario 12x from both sides. I used manual targeting on all tests.
Results
With no pillboxes
Human as Axis
47 T34 vs. 28 Sugs
Allies as Axis
57 T34 vs. 18 Stugs
THis result is consistant with what I have seen all along.
With 2000 in added pill boxes
As Axis
56 T34 vs. 19 stugs
As Allies
49 T34 vs. 32 Stugs
This is the first time I've seen results in which the human actually performed better then the computer (although I think this is probably just statistical noise).
I suspect that the global morale effect is only working for the human and is not engaged by the AI. I suggest that BFC examine this possibility.
No more testing for me. I'll be happy to send me scenarios to someone if they want to further explore these observations.
-
Reduced chris's range to 6 vs. 6 battle (took out pill-boxes). Played six times each side.
As Russians
28 t34 vs. 9 stug
As Axis
23 T34 vs. 14 stug
chi2 is only 0.2, but numbers are still small. Still, identical trend as with my small battle set up.
-
Not to be picky but I don't think you are using the chi square properly. Your "expected" value assumes that the average of the two trials is the real expected value. This may or may not be true. In fact the proper way to do this is to simply due the chi2 using one set of values as the expected and the other as the experimental. Remember, the test simply asks what is the liklihood that the results obtained could have come from the same underlying distibution.
If one does this the chi2 comes out to 3.73 and the P<.053
Warren
-
One last thing
If I am right that size is key, I will test this by modifying Chris's scenario to reduce it to only six tanks per side. If I get the same sort of data that I got with my scenario this would show that size is a key variable.
Warren
-
Steve:
No one (including me) thinks the AI cheats. However, there is concern that the AI may have some sort of advantage in certain situations. This advantage appears magnified in small battles. Are you sure this is not possible? Not a cheat but more a possible undiscovered programing oversight.
Warren
-
I hope this posts. Third try.
In order to test the idea that immobilization my play a role, I modified Chris’s superb range by replacing the broken terrain with open. I did his range 3x as both Allies and Axis. Aggregate data is as follows.
Human as Allies
90 T34 vs. 51 Stugs (Kill ratio is 1.76:1)
Human as Germans
92 T34 vs. 47 Stugs (Kill ratio is 1.96:1)
I won’t bother with the statistics, they look identical.
I have played my small scenario (6 bank range) 9 times from each side (not including initial 6 in first post) and have found the following
Human vs. Allies
44 T34 vs. 11 Stugs (Kill ratio is 4:1!)
Human vs German
35 T34 vs. 21 Stugs (Kill ratio is 1.67:1)
Chi2 is 4.15; P<0.05
My best guess is that the weirdess I observe may be related to some sort of global morale effect as smaller battles would amplify this effect as each kill is a higher percentage of total morale. Perhaps global morale is not effecting the AI as it should? Just a thought.
Warren
-
Chris,
Here is my data using your range. Extreme FOG. Used N key to target each lane.
Human as Russians
26 T34 vs. 21 Stug
34 T34 vs. 14 Stug
29 T34 vs. 14 Stug
Human as Jerry
34 T34 vs. 15 Stug
36 T34 vs. 11 Stug
34 T34 vs. 14 Stug
Also did test without Targeting.
Human as Russians
31 T34 vs. 14 Stug
32 T34 vs. 12 Stug
27 T34 vs. 18 Stug
Human as Jerry
31 T34 vs. 14
26 T34 vs 19
26 T34 vs 20
Based on these numbers I do not see an AI advantage like I did in my scenario. I did play my scenario three times from each side and got the following.
As Russians
6 T34 vs. 0
5 T34 vs 2
4 T34 vs 2
As Germans
2 T34 vs 4
4 T34 vs 2
5 T34 vs 1
Clearly these results are not statistically significant, but again I see a trend that the AI does better then the human.
WHy might this be true in my scenario and not yours? I see two differences. First, I did not control for global morale. Second my tanks are free to move around.
Unfortunately I too must go to bed. I will try to get more data from my scenario tomarrow.
-
Clarification:
Although I did say that the AI cheats in the title, I hope you all realize that simply was a "advertising" tool to get you to look at my result. I am certain that the advantage the AI may have is of a unforeseen nature.
W
-
CHris I can't seem to acess your scenario. THe link is dead.
Warren
-
Maastrician:
Your results seem consistent with my initial observations. But you have extended them with the all important hot seat experiment.
Seems to support an AI advantage.
Warren
-
Limberg,
What statistic do you suggest? Chi square seems an appropriate test in this situation. However, if you would like to use some other method I'm all ears.
Your comment on variables is interesting. If it is a sub-routine problem that lets the AI shoot quicker or hit more often, I'd think that qualifies as an AI cheat, dont' you think.
Warren
PS Marlow mentioned that this was observed in CMBO. I'd like to find that thread.
Warren
-
OK, I see I have stirred up the pot.
I encourage all of you to set up similar tests and see if you get similar results. I am a scientist by training (PhD in biochemistry) and I run a molecular genetics research program. I know something of scientific methods and I am also quite aware of the importance of independent validation to science. I do use statistics regularly, but I am not a statistician. For my P value I simply plugged my numbers into a 2x2 table and the statistics program I use (Statistica) gave me the value using a McNemer Chi2.
I will respond to some of the comments and questions.
1) Each battle is actually 6 identical sub-battles. (This is what the trees are for). There is no variability in the starting positions of the tanks, although they are free to move. (However, this rarely happens before they are destroyed). The only user imput is the initial targeting line to the only visible enemy tank. I have tried not targeting and this appeared to make no difference.
Most of the battles end within the first minute, with an occasional duel lasting longer. Thus there is a total of 36 identical battles so I think this is sufficient for chi 2 analysis.
Many people keep mentioning other uncontrolled variables. Please tell me what these might be?
Several people have mentioned that I should do the experiment in hot-seat mode and see what happens. I too thought of this, but I had to go to work! I will try and get to it tonight, unless someone else beats me too it!
Warren
-
To CM player:
The P value says that these results would be observed 3/1000 times by chance. For most lines of research this is considered a quite significant value. What number do you think is significant? Do you have any background in statistics?
Warren
-
Test is a six-lane gunnery range (tall pines between lanes). I have six StugIIIG's vs. Six T34/85. All with regular crews. Range is about 730 meters.
I played six times as Axis and six times as allies. Played until one side or another had all vehicles destroyed. Whichever side I was playing I would manually target the enemy vehicle on the first turn.
Found the following aggregate data.
As the Allies I lost 30 T34/85 and the axis lost 8 stugIII. (numbers don't add up to 36 because of occasional double kills).
As the Axis I killed 17 T34/85 and lost 20 StugIII's.
The P value using a chi square is 0.003
ALthough I did not follow this rigourously, it appeared that the AI's first shot had a higher hit % than the human first shot, and that this may be the cause of the difference.
Let the discussion begin!
-
I think their is something strange going on. I set up a gunnary range (trees, 900Meters). STUGF/8 owns T34/85. For some reason T34/85 shells do not penetrate STUG front armor. THe chart suggests that 85 should have no problem penetrating 80mM at this range. I also engineered the test to take away all the AP ammo from the STugs to keep the T34/85s alive. What I found is that an unusually large number of hits resulted in "broken" shot.
Does this sound right? Is the Stug Armor "extra-hard".
-
1) How Many shermans were sent to England?
2) How Many sherman's were converted to firefly's?
3) How Many were sent to Russia?
4) What was the total number of sherman's lost in combat in Europe?
Warren
-
I don't think your results are typical. I just set up a scenario where I had 9 t34 (late 43) battle 9 pzIIIM's. Battle ended after three turns with 8 PzIII destroyed against 1 t34. Initial range was 600 Meters.
Seems about right to me!
Warren
PS All crews were regular
[ October 27, 2002, 10:05 PM: Message edited by: Warren Peace ]
AI cheats! (with real data)
in Combat Mission Archive #4 (2002)
Posted
Treeburst:
I played tonight as well. Six times on each side.
Human as Axis
11 Stugs vs. 26 T34
Human vs. Allies
12 Stugs vs. 25 T34
Guess what? I didn't see any difference! I also set up a battle where instead of STugs I used captured t34/85.
As Germans
21:21
As Russians
20:20
I guess my earlier observations were just a fluke
Oh well, thats how science works sometimes
PS Kind of anticlimatic, but boy what a great thread!