AI cheats! (with real data)

[Lt. Kije]

Wow!

To sum up the most recent developments.

1. Maastrician has created a test bed and invites replication. Go thou, my children, and be fruitful. The 'yer n is too small' folks will be mollified and I hope that many new anecdotal observations will result.

2. Maastrician offers data from his test bed, but there is not widespread agreement on whether it supports Theory A (the AI does nothing we cannot, nor does it do anything we do with superhuman effectiveness) or Theory B (the AI is doing something different; it has some advantage over humans). It seems a much clearer analytic framework will be needed to make sense of data coming from the test bed. Perhaps I need to set up a simple chi-square two by two table for people to place their data into? It won't help if 100 people submit 100 idiosyncratic data reports.

3. Lt. Tankersley offers data, but again interpretation of these seems to depend on who is doing the interpreting. To some, an AI advantage seems apparent. Tankersley sees it as either a wash or a very small AI advantage. In this case, n really is too small.

Editorial note: when you do find an effect with a small n, then n is not too small, since the statistical test requires bigger and bigger effects as n gets smaller and smaller. But, and this is a very big but, if you do NOT find an effect, then a too-small n may well be the reason. See above. "Bigger effects required as n gets smaller," means that it is easy to fail to detect an effect with a small n even when the effect exists.

4. Maastrician offers a very important clarification. If the AI turns out empirically, by actual experimental findings, to have some advantage in hit or win percentages, this does not imply that BTS intentionally built in any 'cheat' routines to aid the AI.

5. There seems to be a consensus developing that the human side performs best when the human designates his own targets, rather then sitting back and letting the TacAI do it. Warren Peace's original experiment was conducted with human designating targets.

6. Theory B (the AI is doing something better than humans) seems to becoming centered on something causing the AI to target better and thus to get off shots faster. Especially the first shot.

-----------------------

Editorial note: I made a comment about a member's understanding of statistics that seems to have been very offensive. I really regret this. No denigration was intended. I even took pains to observe that this member's posts on matters of content were invariably intelligent and informative. I look forward to reading those posts in the future and I'll work at being less of a smarta** . My bad.

-- Lt. Kije

Scorekeeper and Historian

[ October 30, 2002, 05:23 PM: Message edited by: Lt. Kije ]

[Conscript Bagger]

Originally posted by Cameroon:

I take it you haven't yet gotten all the way through the thread yet Those results, and others that we're seeing, are due to the computer player choosing targets during the plotting phase.

That's what happens when people post while you're replying

I see now that you originally said your test's German Panthers were slow acquiring Soviet-manned captured Panthers, but they still fired first when under AI control (right?). In terms of my hypothesis, that does not match what happened in Tankersley's no-manual-targeting tests, where the Allied T-34s were slow firing whether under human or AI control.

[ October 30, 2002, 05:16 PM: Message edited by: Offwhite ]

[Pascal DI FOLCO]

Maastrician you did a very good job . Can't you rerun the H vs AI test with targeting orders issued ?

Cameroon,

Looks like you didn't get the whole point : checking if results of a given H vs AI fight are the same that if you reverse side, tactics notwithstanding... As long as the fight isn't lopsided (say KT ve T-26 lol), you can choose any opponents ..

Overall,

I hesitate between calling this thread "Statistics PhD course #1" or "The madmen are running wild". Opinions ?

[dalem]

I would have done much better in my statistics course if the book had more examples of Panthers and T-34s and fewer white ball/black ball problems.

One day someone will invent the "customizable textbook" for mathematics, where you can choose a few general subjects at the beginning of a course and the textbook is printed with a certain portion of the problem sets geared towards those subjects.

-sigh-

-dale

[Cameroon]

Here's the test I ran, I'll leave the relative merits up to the rest to debate for a bit before I make any more comments

Pz VA (early) vs Pz VA (early, captured), 742m, no FOW, regular crews, immobilized on rough terrain. The soviets had no ammo, the germans had nothing but HE and AP. To-hit: 50%, kill fair. Each tank had only one target, they were separated by raised terrain (they on level 7 with level 10 ridges between). There were ten shooters/targets. Thus this test was run 10 times for 100 shots. Only first shot stats were recorded.

First, I ran 100 firings with the AI firing at the de-clawed Soviets. Hits: 55, Misses 45

Then, I ran 100 firings with "me" firing at the de-clawed Soviets. Hits; 49, Misses 51. I hit the "N" key for the target.

Those look pretty close to the advertised 50% to-hit chance with the Target/LOS tool. So I'm comfortable that the AI isn't getting any bonuses.

When the AI was firing, the shots would go off in the first 5 seconds. The same occured when I fired. The time results would not be the same had I let the TacAI select the target. Again, I see no bonus given to the AI.

This has been my point all along. A lot of the misleading results (such as faster shot) are due entirely to the fact that the "plotting AI" picks a target during its "orders" phase. Just like we could do. The "AI's" TacAI is not getting an edge in acquiring targets.

Pascal DI FOLCO, no I got it when I realized what they were doing. I used identical units and units that couldn't fire back to test the first shot hit percentage. That was the original remark that WP made, that there was an "enhancement" for the first shot for the AI. I believe that my results and tests indicate that this is not true.

[ October 30, 2002, 05:44 PM: Message edited by: Cameroon ]

[L.Tankersley]

Hey, I forgot to mention something I noticed while building my second test scenario: the scenario editor modifies elevation data on its own sometimes.

If you recall, my setup had all tiles set at elevation 12 except for a single elevation 0 tile near each end of each lane. When I opened the scenario again to edit it, the elevation 0 tiles were now at elevation 9. I replicated this a few times. This seems to happen when you load the scenario into the editor (because when I saved the scenario with elevation 0 and ran the test, the tiles were actually at 0, not 9) and not when you save the file.

Not sure what's going on here. Has anybody else noticed odd changes in elevation (particularly around abrupt elevation changes)?

[CombinedArms]

Originally posted by Marlow:

I saw this problem in some CMBO testing, and it was even more pronounced (i.e. lopsided towards the AI) if you allow the tac AI to do the targeting.

I think the KEY question is RIGHT HERE in Marlow's statement! Do tanks perform better in tank duels when the TacAI does the targeting as opposed to manual targeting?

Here's a potential hypothesis:

1. Manual targeting imposes a firing delay when compared to the TacAI's targeting, assuming one is firing in both cases at the same target (I don't know this to true, but it might be true and seems consistent with the data.)

2. Who fires first is often the crucial question in an armored duel. The first shot might be a kill, ending the battle right there. If not, the second shot is likely to be more accurate, and therefore a kill. Whoever gets in the first shot also beats the opponent to the second shot, etc., etc.

3. So by imposing a firing delay via manual targeting, Warren Peace may be significantly disadvantaging the tanks he controls.

4. And that accounts for the difference in outcome--because of TacAI targeting, the AI is getting in more first shots, and therefore scoring a higher proportion of kills.

The way to test this would be for Warren Peace to play from one side first, and then the other with the same vehicles as before (e.g. Allied first, then Axis), but WITHOUT manually targeting for his own tanks. If the results turn out to be more equal between human and AI, then the problem is almost certainly with a delay imposed by manual targeting. This could also be tested by measuring the delay before firing with the same tank w/ and w/o manual targeting.

This would be a very important thing to know: it would suggest than in a tank duel, we should avoid manual targeting and let the TacAI do the targeting (unless, perhaps, we disagree w/ the target chosen--e.g. HT instead of potentially lethal tank.)

Just my suggestion.

[Cameroon]

Anyone wishing to use my updated scenario for testing, here it is.

[Lumbergh]

Originally posted by CombinedArms:

</font><blockquote>quote:</font><hr />Originally posted by Marlow:

I saw this problem in some CMBO testing, and it was even more pronounced (i.e. lopsided towards the AI) if you allow the tac AI to do the targeting.

I think the KEY question is RIGHT HERE in Marlow's statement! Do tanks perform better in tank duels when the TacAI does the targeting as opposed to manual targeting?

Here's a potential hypothesis:

1. Manual targeting imposes a firing delay when compared to the TacAI's targeting, assuming one is firing in both cases at the same target (I don't know this to true, but it might be true and seems consistent with the data.)

2. Who fires first is often the crucial question in an armored duel. The first shot might be a kill, ending the battle right there. If not, the second shot is likely to be more accurate, and therefore a kill. Whoever gets in the first shot also beats the opponent to the second shot, etc., etc.

3. So by imposing a firing delay via manual targeting, Warren Peace may be significantly disadvantaging the tanks he controls.

4. And that accounts for the difference in outcome--because of TacAI targeting, the AI is getting in more first shots, and therefore scoring a higher proportion of kills.

The way to test this would be for Warren Peace to play from one side first, and then the other with the same vehicles as before (e.g. Allied first, then Axis), but WITHOUT manually targeting for his own tanks. If the results turn out to be more equal between human and AI, then the problem is almost certainly with a delay imposed by manual targeting. This could also be tested by measuring the delay before firing with the same tank w/ and w/o manual targeting.

This would be a very important thing to know: it would suggest than in a tank duel, we should avoid manual targeting and let the TacAI do the targeting (unless, perhaps, we disagree w/ the target chosen--e.g. HT instead of potentially lethal tank.)

Just my suggestion.

</font>

[Maastrictian]

Originally posted by Lt. Kije:

2. Maastrician offers data from his test bed, but there is not widespread agreement on whether it supports Theory A (the AI does nothing we cannot, nor does it do anything we do with superhuman effectiveness) or Theory B (the AI is doing something different; it has some advantage over humans). It seems a much clearer analytic framework will be needed to make sense of data coming from the test bed. Perhaps I need to set up a simple chi-square two by two table for people to place their data into? It won't help if 100 people submit 100 idiosyncratic data reports.

I'm happy to do the math if people will just post their data.

-- Lt. Kije

Scorekeeper and Historian

Thanks for summing up things so far. That really clarifies the thread.

Pascal DI FOLCO

Maastrician you did a very good job . Can't you rerun the H vs AI test with targeting orders issued ?

I'd love to but I don't have time right now. I will tomorrow.

CameroonI used identical units and units that couldn't fire back to test the first shot hit percentage. That was the original remark that WP made, that there was an "enhancement" for the first shot for the AI. I believe that my results and tests indicate that this is not true.

Your tests look very good. I agree with your conclusions after doing the chi square, at least for your 100 trials. Don't forget that 100 trials is too low, as even Madmatt says.

Chi Square works out as:

Chi Square = ((55-52)^2/52) + ((49-52)^2/52) = .346

Which gives a P (for one degree of freedom) = .556

Which looks really good and sugests there is no difference between AI and Human performance.

Can you e-mail me (dinosaur@noct.net) your scenario or simply put it online so others can exhaustively test it?

I also want to point out that if there is no difference between the to hit chance of a human controled tank and an AI controled one there still may be a difference in target aquisition speed or some other factor. This is a very good test, but I also think "to the death" tests should be done as well as there are other places the AI could have an advantage.

--Chris

[CombinedArms]

I may have gotten things backwards as suggested above--anyway I did a simple observation on my own gunnery range and realized that I'd made an incorrect assumption when I suggested that the TacAI would fire faster than manual targeting. That's just wrong, I'm afraid, based on observation of a reg PzIIIM platoon firing down individual lanes at T-34's with manual targeting and another reg PzIIIM platoon firing at T-34's w/o manual targeting.

The manually targeted platoons get their shots off in about 5 seconds. The TacAI platoons take a couple of seconds to acquire the target, then get their shot off--the total time is now about seven seconds. So, in fact, manual targeting actually speeds up the firing of human controlled units, suggesting that their performance should actually be a little better, if anything, though a two second difference may not be enough to matter, since the T-34s were getting off first shots at both sets of Pz IIIs.

[ October 30, 2002, 06:25 PM: Message edited by: CombinedArms ]

[Michael Emrys]

Originally posted by jamgra:

I, for one, am a bit skeptical as I do not believe that BTS would not mislead us as to any advantages the AI may have.

This sentence, quoted in its entirety, states the equivalent to "...I believe that BTS would mislead us..." Did you mean for it to say that, or did you just lose track of how many negatives you had put in?

Michael

[OddjobXL]

Aside from TacAI targeting and manual targeting there are also 'covered arcs' now which serve to tighten up the area the TacAI looks for targets in. I don't know how that might effect the speed of acquisition or if the AI even uses covered arcs but it might be another variable worth considering?

Maybe not. Interesting discussion though. I'll kick back and leave it to the experts now.

[Guest Sgt. Emren]

Tankersley,

Interesting test results. Maybe the AI loads its guns while its "thinking", and this is what is giving it a faster first shot? It would be interesting to use other tank models, to see if the Rate of Fire has some influence on the timing of the first shot.

I was one of those who argued that there where "too many variables", from my own post:

I don't believe it's very sensible to measure the number of kills. It makes more sense to measure chance-to-hit-%. Anything beyond that, there are WAAY too many variables to say anything meaningful in a statistical sense.

I think it's better to run tests using only the first shot fired, and counting actual hits. What happens after this is not interesting in a statistical sense (this is where the many variables enter) because of the "secret" hit-percentages on the various parts of the tank hull, the randomness of armor penetration etc. I realize that some tests currently being conducted actually do count first-hits, and I heartily applaud the effort. Call me a maso-statistical idiot, but I have enjoyed this thread a lot!

[Bruno Weiss]

Michael emrys wrote:

This sentence, quoted in its entirety, states the equivalent to "...I believe that BTS would mislead us..." Did you mean for it to say that, or did you just lose track of how many negatives you had put in?

Ah hem, ah excuse me, young man! This is obviously the advanced Triggernomitry class. Perhaps the Western Articulations and Intercourse class is down the hall.

[ October 30, 2002, 07:45 PM: Message edited by: Bruno Weiss ]

[Michael Emrys]

Originally posted by Bruno Weiss:

Michael emrys wrote:

</font><blockquote>quote:</font><hr /> This sentence, quoted in its entirety, states the equivalent to "...I believe that BTS would mislead us..." Did you mean for it to say that, or did you just lose track of how many negatives you had put in?

Ah hem, ah excuse me, young man! This is obviously the advanced Triggernomitry class. Perhaps the Western Articulations and Intercourse class is down the hall. </font>

[Michael Emrys]

One thing I wonder about that could possibly throw off the test results, and that is the question of how random the supposedly random factors are in all the various calculations that the program makes. Someone offered the point in a thread a few weeks back that the random number generator in CM might be wonky and, for instance on certain dates, might be cranking out numbers that aren't entirely random. This would not be a cheat but a bug, but it could by happenstance appear to favor one side over another in any specific test or series of tests conducted on a given date.

Michael

[zukkov]

i did a little test of my own with 12 t34/85s and 12 stugIIIgs(late). 3 tests on each side in a 1 player setup. as russian, i lost all 3 engagements by a total of 36 losses to 24 kills. as the german i lost all 3 engagements 36 losses to 21 kills. there definitely seems to be something to his claim. i noticed in all engagements the computer started firing first and achieved kills before my guys started firing back...

[Michael Emrys]

Originally posted by zukkov:

i did a little test of my own...

But you forgot to tell us whether you were manually targetting or not.

Michael

[Bruno Weiss]

You didn't say Simon says.

[Lumbergh]

Originally posted by Michael emrys:

One thing I wonder about that could possibly throw off the test results, and that is the question of how random the supposedly random factors are in all the various calculations that the program makes. Someone offered the point in a thread a few weeks back that the random number generator in CM might be wonky and, for instance on certain dates, might be cranking out numbers that aren't entirely random. This would not be a cheat but a bug, but it could by happenstance appear to favor one side over another in any specific test or series of tests conducted on a given date.

Michael

This is an excellent point, although it would be pretty strange if the random number generation specific to CM is wonky (since it is just a simple call in C++ or whatever CM is programmed in to get a random number). However, and I hope there is an EE out there who knows the theory better than I, supposedly random number generation from a computer is never truly random.

I have no idea as to how serious this problem is, but the fact that random calls in computer programs are used all the time would point to it not being a biggie.

Maybe we're making the process non-random? If you've never seen this concept before, it is pretty interesting: the human impact on machine outcomes. It is the Heisenberg uncertainty principle to the nth degree.

Not to get all spooky. I don't actually think this is what's happening.

[Ace Pilot]

Originally posted by Cameroon:

</font><blockquote>quote:</font><hr />Originally posted by Ace Pilot:

</font><blockquote>quote:</font><hr />Originally posted by Cameroon:

One flaw that I see in the majority of these test scenarios (I commented on it above, but in an edit so it may be missed), is that the German vehicles have better optics.

That's why captured vehicles should be used.

Maybe I'm missing the point, but I thought only German vehicles had different types of optics modeled, and the other nationalities (even using captured equipment) only had the same "standard" level of optics. So, a German Panther will have better optics than a Russian captured Panther. True?

[Edited to practice my spelling skills.]</font>

[zukkov]

sorry, when i read the first few threads i just started the test. seemed like there was too much thinkin' and not enough action. anyway, i didn't manually target. then i went back and manually targeted and did notice that the human forces did fire faster and actually got a draw in one of the tests...

[Cameroon]

Originally posted by zukkov:

sorry, when i read the first few threads i just started the test. seemed like there was too much thinkin' and not enough action. anyway, i didn't manually target. then i went back and manually targeted and did notice that the human forces did fire faster and actually got a draw in one of the tests...

Heh, maybe I'll get it across sometime if I say it enough

What you were seeing wasn't an advantage that the AI got. In many of the tests, when we didn't manually target the enemy, we left it up to the TacAI. The computer player, however, was not leaving it up to the TacAI. It was selecting targets in the "orders" phase.

So our choice not to select a target, and the computer's choice to select a target, caused a difference in the "targeting" time during the turn. Since the computer player started off with targets chosen, it got a 2 or 3 second advantage When you selected the target, you noticed that your 1st shot fire times were right in line with the computer player's.

And I apologize if you'd figured that out, it just seems like this fact has been overlooked repeatedly in this thread and its just "one of those things" where I feel like I've got to say "but wait!"

[Herr Oberst]

Originally posted by Lumbergh:

... However, and I hope there is an EE out there who knows the theory better than I, supposedly random number generation from a computer is never truly random.

...

The "random number" algorithm is running on a deterministic computer, and, given the same initial state, will produce the same pseudo-random sequence. Better algorithms produce longer sequences before the repitition becomes evident through analysis of the generated numbers, and generally try to pull in as many "extraneous" variables as possible as some fraction of the input. The more you can mangle the initial seed with orthogonal (non-related) values, the better off you are, generally speaking.

Unfortunately, most system-supplied random number generators are linear congruential generators, which generate a long sequence on integers between 0 and the maximum value representable by the data type, m (8 bit, 16 bit, 32 bit, 64 bit, etc). This typically generates a recurrence with a period no greater than m.

In the end, you can apply all sorts of borg-brained adjustments to your algorithms, but they just extend the period, or repetition cycle, out to very large intervals, like 2 times 10 to the 18th power (damn UBB) for some relatively simple algorithms. Enormous prime numbers are treasured for this ability, and their subsequent effects on cryptographic algorithms.

No, I'm not an EE, but I did stay at a Holiday Inn last night.

[ October 30, 2002, 10:18 PM: Message edited by: Herr Oberst ]