How come PBEM file size balloons from 1.5MB to 66 MB

[Broadsword56]

An issue has come up with a PBEM and file sizes. We're wondering what causes PBEM e-mail files to suddeenly increase vastly in file size, after the deployment turns have been exchanged and the first player uploads the first WEGO move.

Here's the situation:

I created a 4km x 4km historical master map recently for a large swath of Normandy (see this thread for the background info on it)

The master map has just tiles and roads -- virtually no objects on it at all, except for a couple of small forests here and there.

Now, on top of the master map, I've marked out a battle area of about 1.25 km x 1.25 km and detailed it with objects, to make a scenario. The opposing forces are only company (+) versus company (-).

Both players exchanged their initial PBEM turns. Each file was 1.53 MB. So far, so good.

But then one player made his initial moves and uploaded the turn file. Suddenly the file size ballooned to 65.99 MB !!

Needless to say, this file crashes the game when only about 18 to 70 percent of the turn has loaded.

Just to see if object density is an issue, I made another test scenario on the "clear" 4km x 4km master map, without the detailed smaller battle area on it. Same thing happened -- the first actual move had a 63.27MB file size.

So, I realize that a huge map size must be causing this. But we're curious about what makes the file size increase so dramatically, just because a move turn has been played. Why the difference?

Could it have to do with pathfinding? I suppose it's possible that if you plot a short move of 50 meters from point A to Point B, the unit still has to "know" where the entire boundary of the map is, to know its location and reference the waypoint you've ordered it to. And that would be a strain on the computer to take the entire 16 square km into account to locate two particular action spots, no matter how close together they might be.

We're also puzzled because Huzzar is a very large map and is fully detailed with objects from end to end -- not quite 4x4 km, but still it was a biggie and played just fine. Quickbattle PBEM files with battalion vs. battalion on a "large rough bocage" map averaged 14-16MB per turn, and played fine too.

I was able to create and preview the master map with no problem, and it worked fine after I added the detailed battle area and played a bit of the scenario in 1 player real-time format -- just got a little drop in frame rates now and then.

If I were to cut down the master map area around the detailed battle area by 25%, would that give us a 25% decrease in game file size? I'm not sure whether this is a 1:1 ratio here, or whether it's something else. I may try that, but I just wanted to throw this matter out to the community and see what the experts say.

[c3k]

Well, just to toss out a bit of math, if it's a geometric progression (I'm not saying it is...), a 25% reduction in dimension would be greater than a 25% decrease in size.

Yes, I know you've said "size" not "dimension". But, lets say it's every action spot times every action spot. 3/4's of the action spots times 3/4's of the actions spots gives 9/16's of the original action spot relationships.

If there are higher power's than mere squaring involved, increase your savings as needed.

An interesting set of information would be to let us know what happens if you reduce the size by 25%, 33%, 50%. The same if you reduce the DIMENSIONS by the same ratios.

Ken

[sburke]

I ran a test using my map which is roughly 2x4 km. The initial files generated are 600+k. File 3, the first move turn jumps to 30mg. My suspicion is that saves 1 and 2 only create a base game with passwords. Essentially an initial handshake, but not the full data set. The 3rd file is the full data set and that is your true file size. I selected a minimal number of units to run the map test and while I have a significant portion of very rugged terrain elevation items, there are no trees yet. I suspect my file size will go up once I finish them and and place a full unit roster, but not significantly. It seems more that the base size of your map creates a basic data file size that is quite large in a 4x4 map.

I ran it as a hth game as well and achieved the same approx. file size so it doesn't appear there is much generated specifically by selecting PBEM.

[Broadsword56]

So the bottom-line question is: What is the true maximum playable size of a map on CMBN, if the theoretically available 4km x 4km size or even 2km x 4km isn't really feasible? Can anyone at BFC help enlighten us or give us some insight into this problem?

[sburke]

Send me your turn and password. let me see if I can open it. It could simply be one of those..depends. May be simply an issue of processor speed and overall oomph of your computer.

[__Yossarian0815[jby]]

The exponential growth of spotting info makes the files grow IMO. Then when the bullets fly all those trajectories are added on top of that, hence the shift to a larger file in the middle of the game often signals that the sh*t has just hit the fan )

[winkelried]

So the bottom-line question is: What is the true maximum playable size of a map on CMBN, if the theoretically available 4km x 4km size or even 2km x 4km isn't really feasible? Can anyone at BFC help enlighten us or give us some insight into this problem?

i develop some scenarios on large maps - here some of my experience:

map 1 4000x3000m -quite detailed - loaded in 3D view, but didn't work with troops had to reduce to 3200x2200m to make it work with 2x reinforced inf batallions (first map in my 415th Infantry campaign)

map 2 3440x2800m - lots of details - loaded in 3D view, but not couldn't get this one to work with troops either. had to reduce to 1800x2800m to make it work with 4x reinforced mixed batallion (1.5 red 2.5 blue).

loading times are very long (usually enough time make a good cup of tea) - but playing is quite ok in WEGO. Usually save every 5 moves to make sure i can recover after a crash.

[sburke]

Some additional data.

I took my same map and added 2 battalions of US infantry and 1 battalion of German. When I went to run the first move save, I got an out of memory error. Afterwards I realized I hadn't assigned set up zones and basically all units would have been intermingled. I think my CPU simply freaked out. I then went back and assigned set up zones and assigned 2 battalions to each side

Save

001 American game creation 630k

002 German password assignment and unit purchase 634k

003 American unit purchase and move plan 30,829k

004 German move plan first turn computed 51,318k

005 American replay turn 1 and move plan turn 2 51,311k

006 german replay and move plan turn 2. 41,224k

So the map itself (a 2x4km) is the bulk of the file up until save 4. The unit purchases themselves didn't drive that much. The first action sequence however added another 20 meg and that is with no units in sight of one another. The opposite end of the spectrum was 3 battalions at point blank range which seems to have made a laughing stock of my CPU.

I was suprised in that the turns themselves loaded pretty quickly and the turn computation was quick, but again there was no fighting. In a heavy engagement I expect this one would just not run though having 3 battalions at point blank range on a billiard surface is unlikely. It isn't quite the Civil War. The good thing about my map is, it is really unlikely that units are going to spot each other in large groups. Let's just say the recent thread on forest fighting should be on everyone's mind - Can we sat Huertgen?

[Broadsword56]

In light of the above, I'm cutting the master map sides around my battle area (La Nicollerie scenario) down to a much smaller size. Then we can try it again as see if that helps.

[sburke]

Sounds like a plan. My map is pretty bare so the additional space you have even though not part of the scenario is still adding to the base computation. I think the thread of our battle is reflecting somewhat different than my test. I expect because ours is not a PBEM style, the turn I sent back to you at 65mg is the first action sequence. If you can send me your password I can see if I can at least open it (I promise not to look at your forces ). I am curious to see just how many issues relate to the threshold of a usable file size. I don't remember the size of the map for QB 80, but we both had over a battalion of heavily engaged infantry.

[JonS]

;1309724']The exponential growth of spotting info makes the files grow IMO. Then when the bullets fly all those trajectories are added on top of that' date=' hence the shift to a larger file in the middle of the game often signals that the sh*t has just hit the fan )[/quote']

Yep.

http://www.battlefront.com/community/showpost.php?p=1228302&postcount=463

[GreenAsJade]

Needless to say, this file crashes the game when only about 18 to 70 percent of the turn has loaded.

Why is this "needless to say"? I don't think I've seen one quite this big, but I'm not clear on why there's anything bad about this other than a longer loading time... is it established that there is some terminal file size afterwhich the game crashes?

GaJ

[ClarkWGriswold]

It doesn't make sense to me that a bigger map would make for a bigger turn file. Only the number of units on the map (and what they can see or shoot at) should matter.

If you have one German tank and one American tank on a map, and they can see each other, it shouldn't matter if the map is tiny or giant. You're still only dealing with two action spots and two units - meaning a very limited amount of spotting and firing.

I can't test it from work, but if you put a small number of forces on the huge 4km X 4km map and the exact same forces on a tiny map, are the turn files of similar size? I would expect the bigger map to make larger turn files only on the very first turn, where the map info is exchanged.

[sburke]

Why is this "needless to say"? I don't think I've seen one quite this big, but I'm not clear on why there's anything bad about this other than a longer loading time... is it established that there is some terminal file size afterwhich the game crashes?

GaJ

The game 004 save file definitely isn't loading for either of us and I have another one with a 50+ meg size that doesn't seem to have an issue. Whether that is file size or possibly something within the file that is creating a problem is not clear.

[sburke]

It doesn't make sense to me that a bigger map would make for a bigger turn file. Only the number of units on the map (and what they can see or shoot at) should matter.

If you have one German tank and one American tank on a map, and they can see each other, it shouldn't matter if the map is tiny or giant. You're still only dealing with two action spots and two units - meaning a very limited amount of spotting and firing.

I can't test it from work, but if you put a small number of forces on the huge 4km X 4km map and the exact same forces on a tiny map, are the turn files of similar size? I would expect the bigger map to make larger turn files only on the very first turn, where the map info is exchanged.

In our case I don't think we have even generated the first turn that would require spotting or ballistics. As to the actual unit quantity, we actually are only dealing in company + strength. I have another test run that has 4 battalions in a 2x4 km map that seems to be okay so far.

[JonS]

I can't test it from work, but if you put a small number of forces on the huge 4km X 4km map and the exact same forces on a tiny map, are the turn files of similar size?

Basically; yes. I have a large (2.5km x 2.5km), very complex map which is very sparsely populated with recon units. The file size is recockulous.

[sburke]

Basically; yes. I have a large (2.5km x 2.5km), very complex map which is very sparsely populated with recon units. The file size is recockulous.

recockulous, yeah that's about right.

In my map the base file for the 2x4 km and 4 battalions of infantry prior to moves was about 30meg. It jumps to 50+ even without any units spotting one another on the first move turn. (It seems to drop to 40 meg the second turn, but I'd have to run a lot more tests to really confirm what all is affecting the file size.)

My suspicion is GaJ may be partly right in that normally this would just be a matter of load time assuming your CPU can handle it, but we may also be seeing some issue with the file itself. Maybe there is something about the map that is impacting the file creation. Too early to say yet.

Broadsword is going to edit the battle to slice off unneeded portions of the master map, then we will see what happens to file size and our ability to have a go at this battle.

[Vanir Ausf B]

So the bottom-line question is: What is the true maximum playable size of a map on CMBN, if the theoretically available 4km x 4km size or even 2km x 4km isn't really feasible? Can anyone at BFC help enlighten us or give us some insight into this problem?

The answer to this may differ depending on whether the machine in question is a Mac or PC given that the Mac executable is 64-bit.

[hoolaman]

If the dusty recesses of my memory serve, the spotting data is made a lot more complex and therefore bigger files, by a lot of changes in elevation. Would be easy enough to check by flattening out the same map and looking at the file sizes.

[LongLeftFlank]

It would be nice to hear from BFC on this issue before I do any scenario design work on my own 2600 x 2800 master map. Is it a bug or something we need to live with? What are the functional limits on playable map size? (the Manual says 4 x 4). If a bug, is it on the list to be fixed?

For example, some scenarios I was toying with the idea of having a long map with the first third built out in detail for the battle, the middle half a featureless and impassable "unfinished" portion then the remainder being the distant rise around Le Carillon where German OPs and "88s" sit in overwatch.

EDIT: For comparison, the save game files for my CMSF "JOKER THREE" scenario, which is 1km x 600m and flat but has buildings, walls and doodads covering almost every square foot, top out at 3.5MB -- the game .BTT file itself is 1.3MB with all the briefing maps and whatnot included. Smaller test scenarios on the full sized 2km x 750m Ramadi map top out at about 6.5MB. So this metastatic file growth may be new with CMBN -- maybe that will provide some clues as to cause.

[George MC]

This thread has details about Out of Memory issues (OOM):

http://www.battlefront.com/community/showthread.php?t=99521

A couple of things worth keeping in mind:

Everyone's PCs are differant (no branier eh?) but therein lays lot's of possible configurations etc which may or may not have an affect on the game.

In that respect, like when CMX1 came out some people with older or less highly spec machines may find that there are some scenarios they just cannot play i.e. their PC cannot cope with the demands the scenario makes on the game and hence their system.

So some OOM issues are down to a low spec PC plain and simple. Other OOM occur on higher spec machines that should, in theory, play the game. That's the puzzling bit - which, looking at the thread, is being looked into.

CMBN maps are way more graphic intensive than most CMSF maps - all that greenery and variety of 'stuff' comes at a graphic price. Also I think players are starting to look at playing with larger OOBs - more units doing more shooting = more demands on the game engine to compute.

Cheery!

[ClarkWGriswold]

Basically; yes. I have a large (2.5km x 2.5km), very complex map which is very sparsely populated with recon units. The file size is recockulous.

How big (file size) is the actual map? I'm wondering if the very first turn or two exchange the entire map so that both opponents have it. That could explain the ginormous file size.

[Frankster65]

Well said George_MC. With my old machine, I had to be careful how big a scenario I would play in CMSF. With my new machine, I can play anything with any number of troops and equipment in the scenario. Currently, I'm playing a CMSF scenario where the common PBEM file sizes are 40MB. A lot of terrain and a lot of equipment shooting it out.

[JonS]

How big (file size) is the actual map? I'm wondering if the very first turn or two exchange the entire map so that both opponents have it. That could explain the ginormous file size.

The scen file is 1.8MB. But in general the file size profile is much the same as the graph in my earlier post; the first couple of files are tiny, the next couple are an order of magnitude larger, then when things actually start happening file sizes get huge. But the final magnitude does seem - to me - to be more dependant on the map size than the number of troops. That is a large map with few troops will generate a larger file than a small map with loads of troops. Of course, that assumes that map complexity (number of contours, number of flavour objects, number of trees, fences, roads, buildings, hedges, etc) scales geometrically with the size of the map. I wouldn't be too surprised if a large but very simple map generated smaller files than a tiny but dense map.

[sburke]

Just an FYI, Broadsword cut most of the unneeded portion of the master map and with the same forces and detailed area of the map we are now running on a 20+meg file.