The research system is flawed. I have some extensive testing to back it up!

[zappsweden]

I will post the test results i had, very soon.

[ January 06, 2003, 06:08 PM: Message edited by: zappsweden ]

[zappsweden]

Since the 1.06 patch i have been curious of the catch-up effects that were invented. When i read in the "SC version changes" file

"

- normalized research via 5->4->3->2->1 slow down and relationship with enemy

research levels to represent captured equipment, spying, etc.

** New Formula **

(5% + highest enemy level - current level) * applicable chits

Restriction: highest enemy level - current level > 0

"

I thought i should test the system and see if it worked. My plan was to see how fast USA, Russia and UK would reach level 5 if they already were at level 4 and their enemies (Italy and Germany) were at level 0. However, as the tests showed, I already got confused by the result i had in my first test where every nation *(allies and axis) started at the SAME tech level.

[ January 06, 2003, 07:37 PM: Message edited by: zappsweden ]

[zappsweden]

THE TESTS I MADE:

The tests are about monitoring the advances in Jet Aircraft and Advanced Bombers and changing the tech levels to see what happens. Russia, USA and UK have 10 chits and put 5 chits on Jets and 5 chits on Bombers i.e a total of 30 chits on effective research. In all the tests I checked these things about 5-6 times to see if they were right set. In all the tests I used the hotseat play and picked my scenario that i had modified from the 1943 scenario.

TEST ONE:

I took the 1943 scenario and put all nations (except for France and Italy) on level 4 jets and level 4 bombers. When the scenario started i put 5 chits on jets and 5 chits on bombers for the US, Russia and UK. If the highest enemy has the same as the nation in question then every chit should mean 5% chanse of advance right?

Every nation with 10 chits then has 10*5=50% chanse of advancing. The chanse that all 3 nations (USA, Russia and UK) fails is then 0.5*0.5*0.5= 1/8 (one in eigth).

To my surprise, i only got 2 advances in the 20 allied turns that i did. Since i got the first advance already in turn 2, the following 18 turns had reduced probability (since one was already at level 5) so for the remaining 18 turns, the probability for failing a turn was 0.75*0.5*0.5=0.1875. The odds that it took so long to get 2 advances out of 6 (2 reserach areas*3 nations) must have been HUGE. Believe me, i checked the chits and the levels many times to make sure that it was ok.

TEST TWO:

I did the same thing, but this time i let all nations start at level 0 jets and level 0 bombers. The result was TOTALLY different. In 6 turns i already had 6 advance so I quit the test there.

TEST THREE:

I put all nations on level 4 (Jet and Bomber) again and also included Italy and France on that tech level (in case there were some strange calculations there).

In the first 8 turns i got 3 advances. So that makes 3 out of six since only those left on level 4 could be improved. One country had 2 advances, one had 1 advance and one had 0. From this point the chanse that all 3 nations fail have the probability 1.0*0.75*0.50=0.375 i.e 37.5%.

My goal was now to see if those 3 advances were just pure luck, so i decided i should go on for many turns to see the nig picture. I kept on for roughly 28 turns more before the next advance came, and the probability to fail was 37.5% so that is very strange.

In test 3 i got a total of 4 advances in roughly 36 turns though i had 3 nations with 10 chits each. Surely, the chanses were halved when 3 of 6 advances had already occured but still there were 15 chits active and 10 chits should in normal circumstances mean a 50-50 chanse of advance which probably was not the case.

[ January 06, 2003, 07:51 PM: Message edited by: zappsweden ]

[zappsweden]

The conclusions:

In general, it seems that the spread in advances seem too big because you just too often have long, long perios with no advances and suddenly 3-4 advances occur in some turns. The very low probabilities for repeated turns without advances are very small. They should not occur that often in the small number of games i have been in myself. It is like if you had played 100 round of poker (1 change) and already got 3 Royal Straight Flush.

I am a programmer myself (and i know some statistics) and I do not know how Hubert made the random numbers in this game. If the strange results has something to do with the scenario editor, the optimisations of the code or the "ending the turn without moving", I do not know. In fact, I moved my pieces in the first test) and attacked a few units occasionally (so that the testing method itself did not affect the outcome) but I got a very long perios of no advances anyway. The random does not seem as reliable as a rolling a die.

Even if the total number of advances in 1000 turns may be approximately correct, the random spread may still be faulty because of the randomizer. It is like playing a game were you often roll a die 6 three times in a row but suddenly you try 25 times without even get one die 6.

In my games, it has often seemed (ever since 1.05 patch) that the 5% per chit does not give an average of 1 advance in 20 turns, but I am not sure. Now that I have shown some strong indications on a fawlty research system, the randomzer should not be assumed error free until propoerly tested (please test it someone with patch 1.05 so there are no catch-up effects). Especially, it seems (with 1.05) that when you have 5 chits (25% chanse i.e 1 advance in 4 turns) it almost always take more than the average 4*5=20 turns to go from level 0 to level 5. I could test it sometime.

Already on patch 1.05 (without this new catch-up system) I noticed that too often you could have 5 chits (i.e 25% chanse of advance and 75% chanse of fail) for VERY long periods (15-25 turns) without any luck and suddenly when the advances come, they seem to be several of them in just some turns.

Since i got rapid advances in test 2 (where every nation started on level 0) contrary to test 2 where I had very long periods with no advances, you could question if the formula for deciding the percentage of every chit (on a given occasion) works. That thing about "the highest enemy level" is particularily interesting since it could mean that highest enemy level always counts as 0 i.e advancing from level 4 gives 5-4=1% per research chit!

The result where i had no luck on research was more like a 5-4=1% chanse per chit. That would mean there is 70% of a fail, since 30 chits become 30% chanse of success. If you combine that 1% theory chanse there is still very little chanse that you can have 15-20 turns without any advance BUT combined with a "not so random" randomiser it could happen.

[ January 06, 2003, 07:49 PM: Message edited by: zappsweden ]

[zappsweden]

Dammit, do not you have testers that can check this out before the patch is released?

Soon you have to hire me to be a tester because of all my hard work and statistical knowledge.

[ January 06, 2003, 06:55 PM: Message edited by: zappsweden ]

[Hubert Cater]

Originally posted by zappsweden:

THE TESTS I MADE:

TEST ONE:

I took the 1943 scenario and put all nations (except for France and Italy) on level 4 jets and level 4 bombers. When the scenario started i put 5 chits on jets and 5 chits on bombers for the US, Russia and UK. If the highest enemy has the same as the nation in question then every chit should mean 5% chanse of advance right?

I only read this far and noticed the above assumption is incorrect and may affect the rest of your tests/text below, you will have to let me know.

Basically the Restriction: highest enemy level - current level > 0 needs to be considered. I believe I wrote it as >= 0 in the Update Notes but it should be > 0. In this case the restriction fails so the highest enemy level in the formula will be assigned a 0 value and thus each chit will be 1% chance instead.

I realize now that the first example I used for the new formula uses Level-0 for both your enemy and yourself so the example ends up a little misleading. Sorry about that

Hubert

[zappsweden]

I might have missed the point with the restriction, but was not the meaning that you should slow down 5->4->3->2->1 for every level AHEAD of the enemy and if you were not ahead you should have the usual 5% instead?

Now it seems that instead, research becomes almost impossible when you are ahead AND on level 3-4 EVEN if the enemy is just one level behind. Meanwhile, it seems that the enemy will get a 5->6->7->8->9 system so that he catches up better, but that is not described (and i have not tested it, but i will) but it seems that way on the formula.

[pzgndr]

do not you have testers that can check this out before the patch is released?

We did check this out. One beta version had a research tracking log that provided tech advance probabilities based on how many chits were invested and what the tech level differential was. The math checked out fine, turn after turn.

The whole random number generation thing is another issue. I'm still not 100% convinced that this is working the way it should. Statistically, we should be seeing a normal distribution of results based on probabilities and number of game turns. That means some good luck and some bad luck, but mostly average results. It's not happening, at least not consistently for me and others. I've noticed differences between my solo games and tcp/ip games and it's been bugging me, because research advances are important in this game. But it's very difficult to say what the problem might be since sometimes it appears to be working fine.

Hopefully Hubert will continue investigating this and find something to fix.

[kenfedoroff]

Originally posted by Bill Macon:

The whole random number generation thing is another issue. I'm still not 100% convinced that this is working the way it should....

Thanks for sharing your concerns on this issue. I was beginning to wonder if there are other factors involved.

Sincerely,

"Paranoid in Plymouth"

[zappsweden]

Originally posted by Bill Macon:

Statistically, we should be seeing a normal distribution of results based on probabilities and number of game turns. [/QB]

I knew there was something fishy about the normal distribution in this game, now atleist i get it confirmed.

[Hubert Cater]

I might have missed the point with the restriction, but was not the meaning that you should slow down 5->4->3->2->1 for every level AHEAD of the enemy and if you were not ahead you should have the usual 5% instead?

Not quite, the point of the new research system was to slow it down entirely to get to the next level using the 5->4->3->2->1 system. The inclusion of the spying based on enemy research was to quicken the gap when lagging behind but otherwise no difference if you are ahead or at the same level.

Take a look at the samples again and you'll see what I mean.

Hubert

[Hubert Cater]

Statistically, we should be seeing a normal distribution of results based on probabilities and number of game turns. [/QB]

I knew there was something fishy about the normal distribution in this game, now atleist i get it confirmed.

[Jim Boggs]

Hubert, I am reminded of an old Dilbert strip in which he announced to Dogbert that he was going to invent a security system for the internet that would prevent teenagers from accessing porn on the internet. Dogbert's reply was something like, "So, you are going to match your programming talents against the hormonal drive of millions of teenagers? Hmmmmm."

You have created a wonderful game that is FUN to play, but the grognard world is relentless in their pursuit of perfection.

Hang in there!

[Hubert Cater]

I was wondering if anyone else was noticing this!

Hubert

[pzgndr]

OK, OK, I'm not going to push this one. If 10,000 loops says it's fine, I'll hit the "I believe" button. Statistically speaking then, I have one helluva lucky game to look forward to one of these days!

Perhaps I'll dust off an old Probability and Statistics book and offer some alternate ideas for SC2. Maybe a standard distribution could be used at the time a research decision is made or changed to predetermine when the next advancement occurs. That would ensure an expected result. Or maybe start increasing the chances when the average time has been exceeded.

The research issue has been simmering for a long time. It should have a random element for replayability, but not be PURELY random. As it is, you could possibly get to L5 within a year or perhaps never see a L1 advance. We need reasonable expectations.

[Friedrich_der_2]

Nice to know that in 10.000 games I should statistically get X advances but who has enough time to play 10.000 games?

I work in the research area and I know that often there are unexspected breakthroughs and long periods without much advances. But I think the random factor in SC is to great. You should ensure that there is an advance e.g. all 20 turns for one reseach point. The research points are very expensive with patch 1.06. I think too expensive. Just in TCP/IP games you often see an early decision and I experienced in my TCP/IP games research becomes only important in long games. With patch 1.05 research was always more important.

If you change the research system I would suggest that you get first research success faster and cheaper. The highest level should be much more difficult to reach, perhaps only if you invest 4 or 5 points in this field of research. So research becomes more important also in the shorter games which ends and it is also more realistic. With an minimum investement in a research field you never reach the highest knowledge.

So please not too much statistics. If you have always bad luck in one game it can destroy the fun. And this is the reason why we play. The statistics must work in ONE game then it is a good game design.

Sven

[zappsweden]

It seems to work alright now that Hubert has explained the things about setting the highest enemy at 0 when the restriction does not come true.

It was my intuition and the text in the "SC version changes" that made me believe something was wrong.

I gotta say that it seems a bit chaotic to have 5 chits equal 5*1=5% when being on level 4 while the enemy might have level 3 and get 5*6=30% using the same number of chits.

This makes long term focused research VERY hard. I used to lay low and research Jets with 3-5 chits being UK and by 1942 being able to surprise the Axis and posses the air (hence bombing the hell out of him). Now, the best thing to do is simply to buy just a few chits and spread them out on Jets, Long-Range, Bomber, sonar, Gun Laying and hope they advance sometime.

Even if i could get a rarely seen level 3 vs level 0 superiority in air versus axis he could just put 3 chits on Jets and it makes 3*8=24%. In just 5-10 turns he almost catch-up the research i have been doing secretly for 2 years. The shift seem so radical from how the game used to work. Before it was "Jets, Jets, Jets" and "industrial, industrial, industrial" and now it is "do not do anything today, that you can do tomorrow"

My beliefs are that there are 2 alternatives that can work good:

ALTERNATIVE 1:

Slowing down (5->4->3->2->1) dependant only on the current level (i.e on level 4 the chits will be on 1% each) but NO catch-up needed.

ALTERNATIVE 2:

both catch-up and slow-down compared to the highest enemy level (i.e if i have level 4 and the enemy has level 3 i will have 4% per chit and he will have 6% per chit).

I strongly would go for the first alternative.

THE CURRENT 1.06 PATCH ALTERNATIVE:

Slowing down (5->4->3->2->1) dependant only on the current level.

catching up dependant on highest enemy level i.e if i have level 2 and the enemy level 5 i will get 5+3=8% per chit instead of the usual 5%.

[ January 07, 2003, 06:21 PM: Message edited by: zappsweden ]

[zappsweden]

Another problem with a catch-up contrary to a slow-down is that whatever i research as UK, i will give to the Axis hence hurting Russia when they enter.

If i get level 2 Jets with UK, then probably Axis will benefit because of it and give Russia level 0 Jets a hard time when they enter the war.

Do not EVER EVER research Industrial with UK because it will benefit Germany most because of the catch-up

[pzgndr]

Some more comments on this. After geeking out on some probability theory, here's what I came up with. The current randomness produces a cumulative success chance that initially increases quickly but tapers off to asymptotically approach 1. It does not look like a normal distribution centered about a mean, which would give you an S-curve. For 1 chit at 5%, the mean is 13.51 turns to reach a 50% chance of success. However, you have a 14.3% chance of seeing success by turn 3 and a whopping 35.8% chance of NOT seeing success by turn 20. This leads to a very wide spread of results and a bit of frustration.

We don't want research to be deterministic and give us predictable results every time, but something closer to the mean would be more reasonable. Assuming most players would prefer a normal distribution, it should be possible for the power of the computer to crank out better results.

Each research area would need to have an internal counter to increment the number of turns since research began for a new tech level. Base probability for success would be the same as currently determined in the game (5-4-3-2-1 plus bonus, or whatever other changes to this may happen.) OK. Using a binomial distribution formula, determine the mean number of turns for the given base probability. Normalize this mean to a z-distribution (eg, 5% = 13.51 turns = 0.50) Determine the current z number based on the turn counter (eg, Turn 8 = 0.296) Determine the current probability based on the current z number, which progressively gets better. Perform random number check. Increment the turn counter and repeat as necessary until successful.

This sounds terribly complicated, but all the math can be coded to produce the desired results without adding a smidgen of complexity to what players see now. Since it would run every turn, any changes to number of chits or enemy research levels would be accounted for every turn.

Now, since the mean is 68.97 turns for 1%, 34.31 for 2%, and 22.76 for 3%, it may make sense to shift those research percentages to 6-5-4-3-2 or something else so L5 is attainable during the game. But that's another issue once the normal distribution is resolved. I think my math is correct, but another math geek is welcome to back me up.

[ January 13, 2003, 01:48 PM: Message edited by: Bill Macon ]

[pzgndr]

Another geeky comment about research. Both the current random system and the normal distribution system I previously discussed have the same average time to success. Only the distribution looks different for limited trials (ie, << 10,000 loops!) So how quickly should we be allowed to progress through the research tree?

Assume the case where Germany starts with L1 jets in 1940 and wants to advance to L5, which historically wasn't achieved until 1944/45. Say 4 game years or about 52 turns. And we're talking about effective deployment of advanced jets, not when they started testing prototypes.

With 5 chits, the average time to advance from L1 to L5 using the straight 4-3-2-1 formula is about 27 turns. That's only 2 years in the game, and much too fast. With 4 chits, it takes 35 turns. 3 chits, 47 turns; 2 chits, 142 turns; and 1 chit, forever. Based on these numbers, I'd recommend 3 chits as the limit for any one area rather than the current limit of 5. It would still permit reaching L5 levels by 1944 and maybe by 1943, but probably not sooner. This is a simple change that would significantly limit ahistorical results.

[zappsweden]

Originally posted by Bill Macon:

Another geeky comment about research. Both the current random system and the normal distribution system I previously discussed have the same average time to success. Only the distribution looks different for limited trials (ie, << 10,000 loops!) So how quickly should we be allowed to progress through the research tree?

Assume the case where Germany starts with L1 jets in 1940 and wants to advance to L5, which historically wasn't achieved until 1944/45. Say 4 game years or about 52 turns. And we're talking about effective deployment of advanced jets, not when they started testing prototypes.

With 5 chits, the average time to advance from L1 to L5 using the straight 4-3-2-1 formula is about 27 turns. That's only 2 years in the game, and much too fast. With 4 chits, it takes 35 turns. 3 chits, 47 turns; 2 chits, 142 turns; and 1 chit, forever. Based on these numbers, I'd recommend 3 chits as the limit for any one area rather than the current limit of 5. It would still permit reaching L5 levels by 1944 and maybe by 1943, but probably not sooner. This is a simple change that would significantly limit ahistorical results.

That is why i would prefer a 4->3->2->1->1 system instead of the 5->4->3->2->1. Also, i want NO CATCH-UP EFFECTS then because they are not needed. That would probably make advancing from level 0 -> level 5, take 4 years instead.

[ January 29, 2003, 09:50 AM: Message edited by: zappsweden ]

[pzgndr]

Catch-up effects were added to represent captured equipment, spying, etc. And a boost to close gaps so one lucky side doesn't have a huge advantage for too long. It's abstract and worth reconsidering for SC2. Maybe the research differential could be limited to +1 or +2. Or just make this an on/off game option.

Something else to reconsider is which techs should affect other techs for the differential. Tanks could affect both tank and AT research, for instance. Subs could affect both subs and sonar. Stuff like that. :cool:

In general, it seems that the spread in advances seem too big because you just too often have long, long periods with no advances and suddenly 3-4 advances occur in some turns.

5-4-3-2-1 seems to work OK. It's simple and adequately represents progressively more difficult research. IMHO, tweaking the distributions and changing the limits from 5 to 3 would smooth out research and provide more historical results. That should resolve the original concern.

[zappsweden]

In my opinion a slow-down 5->4->3... or whatever will make your enemies close the gaps even without a specific catch-up bonus.

The problem with the catch-up now is that no nation would want to be the leading nation and do a long-term research EARLY in the war. It seems much better to lay low and come from the behind using the "free ride" that the catch-up effects give.

I do not care if the new catch-up system benefit the underprivilieged Allies (like some player claim), because if that is an argument then there is sure something wrong with the game balance. Strengthening the Allies can be done much simpler by including some MPP off-map production for USA and UK.

the fun thing BEFORE the patch was the long-term tech race which now unfortunately does not seem to exist.

Changing the system to 4->3->2->1->1 instead of changing the max chits to 3 is just a matter of taste. The important thing is that something is done about the fast research in the game and that long-term tech races can occur, not just boring catch-ups that make every nation look the same.

If Germany develops Tiger tanks (level 5) or real Jets (level 5) let them have that advantage. If Italy develops the best navy let them have it.

In real WW2, the Allies could not match the German Tiger Tanks even in 1945. In THIS game, they would get the Tiger Tanks 4-8 months after the Germans so it does not seem right.

[ January 14, 2003, 02:05 PM: Message edited by: zappsweden ]

[zappsweden]

It is the high plunders and the high income from occupied neutral countries that give the Germans the upper hand not the research itself.