uncam: Combat Mission campaign extractor

[Rokko]

Hello,

I had pretty much given up on CM back in 2016 for various reasons and mostly lost interest in anything CM related as well. The announcment of CMSF2 has brought me back somewhat and, coincidentally, I am currently learning a new programming language. So I decided to write a little application I had wanted to write even way back then, which can extract scenarios from campaign files so you can look at them in the editor or edit them. Mad Mike's ScAnCaDe is a great tool for parsing scenario files, extracting campaigns and visualizing scenario and campaign metadata. However, it does not work with the 'legacy' CM games Shock Force 1 and Afghanistan. Also, who knows if Mad Mike is still around to update it anymore for future versions. He did a great job deciphering the various campaign file formats that have existed throughout CM's lifetime and much of what I did is based on his groundwork (which involves a lot of staring at hundreds of rows of bytecode and figuring out which bytes stand for which variables), so credit where credit is due. And while according to the devs you should be able to play any old campaign in CMSF2 out of the box, now you can also extract the scenario files from these campaigns and update them to the new engine standards, even if the original creators are now longer around.

I decided to focus on the extraction part and didn't bother with any of the metadata listing and visualizing. The tool should be able to extract any campaign file from any CM game so far and should also work with CMSF2 once it's released. I originally wanted to extract every bit of information required to also recompile the campaign from the extracted data without any further input, but that proved to be more difficult than I anticipated because some of that information is compressed or decrypted within the campaign files and I have not found a way yet how to extract that. This includes the campaign victory/loss texts and the complete core units and campaign briefing data. Maybe I will manage to add this at a later point. For now the scenario extraction works really well however, but I need some feedback from others. I don't have any CM games installed currently besides Shock Force and thus have only been able to  thorougly test the CMSF stock campaign as well as many campaigns downloaded from the Scenario Depot. So if anyone finds a campaign that does not work, make sure to post or send me the error log file that should be created. Usage of the application is as simple as it can be, just put the .exe file in a directory that contains any number of campaign files and start the executable.

 

uncam-0.9.zip

[sburke]

Thanks Rokko. This will definitely help folks who want to work on the old campaigns and bring them up tp 4.0 standards even on a map level.

[benpark]

Yes, a great tool. Thank you.

[Badger73]

5 hours ago, Rokko said:

Hello,

I had pretty much given up on CM back in 2016 for various reasons and mostly lost interest in anything CM related as well. The announcment of CMSF2 has brought me back somewhat and, coincidentally, I am currently learning a new programming language. So I decided to write a little application I had wanted to write even way back then, which can extract scenarios from campaign files so you can look at them in the editor or edit them. Mad Mike's ScAnCaDe is a great tool for parsing scenario files, extracting campaigns and visualizing scenario and campaign metadata. However, it does not work with the 'legacy' CM games Shock Force 1 and Afghanistan. Also, who knows if Mad Mike is still around to update it anymore for future versions. He did a great job deciphering the various campaign file formats that have existed throughout CM's lifetime and much of what I did is based on his groundwork (which involves a lot of staring at hundreds of rows of bytecode and figuring out which bytes stand for which variables), so credit where credit is due. And while according to the devs you should be able to play any old campaign in CMSF2 out of the box, now you can also extract the scenario files from these campaigns and update them to the new engine standards, even if the original creators are now longer around.

I decided to focus on the extraction part and didn't bother with any of the metadata listing and visualizing. The tool should be able to extract any campaign file from any CM game so far and should also work with CMSF2 once it's released. I originally wanted to extract every bit of information required to also recompile the campaign from the extracted data without any further input, but that proved to be more difficult than I anticipated because some of that information is compressed or decrypted within the campaign files and I have not found a way yet how to extract that. This includes the campaign victory/loss texts and the complete core units and campaign briefing data. Maybe I will manage to add this at a later point. For now the scenario extraction works really well however, but I need some feedback from others. I don't have any CM games installed currently besides Shock Force and thus have only been able to  thorougly test the CMSF stock campaign as well as many campaigns downloaded from the Scenario Depot. So if anyone finds a campaign that does not work, make sure to post or send me the error log file that should be created. Usage of the application is as simple as it can be, just put the .exe file in a directory that contains any number of campaign files and start the executable.

uncam-0.9.zip

Please consider uploading this to CMODS III (http://cmmodsiii.greenasjade.net/) or PM @Bootie to coordinate making this generally available.  Thank you.

[jackal263]

It would be fantastic if you can also extract from saved files the exact number of troops lost / enemy killed by single unit !

Do you plan something like that ?  

[para]

Can you alter the mission time with this?

[Rokko]

19 hours ago, Badger73 said:

Please consider uploading this to CMODS III (http://cmmodsiii.greenasjade.net/) or PM @Bootie to coordinate making this generally available.  Thank you.

I intend to do so, I just wanted to wait if someone reports any bugs or issues before doing so.

17 hours ago, jackal263 said:

It would be fantastic if you can also extract from saved files the exact number of troops lost / enemy killed by single unit !

Do you plan something like that ?  

No not really. The format for campaign files is pretty simple, it's basically just a bunch of files glued together, so all you have to do is find the glue. As I said Scenario (and savegame files) are much more complex and involve some sort of compression or decryption mechanism. I'm sure some reverse engineering expert could figure it out, but probably not me

14 hours ago, para said:

Can you alter the mission time with this?

Not sure what you mean. You can extract scenario files from campaigns. You can open these files in the regular scenario editor then and alter whatever you like (or whatever can be altered with the editor). Repacking these (altered) scenario files back into a campaign files requires a little more input unfortunately, unless you don't mind losing the campaign briefing text/images.

[banned]

When failing campaigns and coming to a premature end, does this allow me to extract the last missions in order to play them?

[Rokko]

Yes you can, but the status of your or the enemies' core units will not be persistent with your savegames. In other words, all core units will be at full strength.

[banned]

Great, thanks

[Rokko]

Due to a specific request I've compiled a 32-bit version of the executable as well. No other changes so far, see the attachment.

uncam-0.9.zip

 

[37mm]

Thanks Rokko for the great work.

You've just doubled the number of available scenarios for CMA players!

 

[danfrodo]

I tried running this but it just flashes a window & crashes.  Anyone else see this?  Do I just double click on it or do I need to run from command line?

Certain version of python or some such needed, perhaps?

[Rokko]

Should work without any external software. Just put it in the same folder with some campaign files and double click the .exe

Some questions:

- was an error log file created? If so, please send it to me or post it here

- did you use the correct version? There are executables for 32-bit and 64-bit Windows

- could you tell me specifically which campaign(s) you tried to extract?

[danfrodo]

Ah, I think I know the problem.  I ran it from wrong place, thinking it would ask me what campaign to unpack. Thanks! 

[Mattis]

Thanks for providing this.

I hope that you guys one day manage to find a way to extract the compressed and decrypted information which is required to recompile the campaign file.

[markus544]

I ran that program and placed the files into the folder.  It unpacked the NATO files but not the other american files.  The screen flashed and I got an error report which is displayed below.

 

In file: C:\Users\Miste\Desktop\Shock Force 22\Semper Fi Syria.cam
Invalid number read: Attempted to parse core units size
The application attempted to read a number value in the current campaign file was significantly larger than what could be be considered sensible.
    Heuristic for maximum value: 100000000
                     Read value: 2164227200
Something probably went wrong with parsing the specific file format and the cursor was not positioned correctly.
Alternatively, the heuristic may not have been set conservatively enough and should be increased.

In file: C:\Users\Miste\Desktop\Shock Force 22\Task Force Thunder.cam
Invalid number read: Attempted to parse campaign image size
The application attempted to read a number value in the current campaign file was significantly larger than what could be be considered sensible.
    Heuristic for maximum value: 100000
                     Read value: 3904958953
Something probably went wrong with parsing the specific file format and the cursor was not positioned correctly.
Alternatively, the heuristic may not have been set conservatively enough and should be increased.

In file: C:\Users\Miste\Desktop\Shock Force 22\Training Campaign.cam
Attempted to read a required size value which was zero.
 

[Hapless]

I've gotten that error too, when trying to uncam the Task Force Thunder and Semper Fi campaigns.

It'd be a fun experiment to play the CMSF2 campaigns in multiplayer!

[Rokko]

Thanks for the report. I've noticed this as well and think it should be easy to fix, will probably get around to it in a couple of days. The TF Thunder and Semper Fi campaigns were reworked for the initial CMSF2 release so they are the ones causing problems as of now, while all the other campaigns are still identical to their CMSF1 versions and can still be unpacked.

As far as I understand, ScAnCaDe can unpack the reworked campaigns out of the box.

[CMFDR]

@Rokko awesome tool, thanks a lot!

I might have a brain fart here but is it any way to extract the Campaign briefing and the Core Units file too?

[Falaise]

2 hours ago, CMFDR said:

@Rokko awesome tool, thanks a lot!

yes

[Rokko]

On 2/4/2019 at 6:37 PM, CMFDR said:

@Rokko awesome tool, thanks a lot!

I might have a brain fart here but is it any way to extract the Campaign briefing and the Core Units file too?

Thanks,

unfortunately, no. There is a big blob of bytes in every campaign file that somehow encodes the core units of a campaign, but I have been unable to figure out how to extract that in a way that it can be loaded in the editor. The thing is, scenario files use some kind of compression algorithm (or maybe even encryption), i.e. you can save the same the same file a bunch of times and large parts of the resulting files will be different each time. It might be possible to make an empty scenario file and somehow glue in the core unit bytes at the right place and hope it loads, but I haven't done any experiments for that yet. In fact haven't done any work on the application in the last weeks since I'm pretty much 100% preoccupied with finishing my Masters thesis. Once that is out of the way I'll probably have time to do some experiments.

For now, uncam can do two things really well, which Mad Mike's tool (I always forget the spelling) can't and that is extracting CMA and CMSF1 campaigns and his tool is the only one able to extract updated CMSF2 campaigns, so everyone should be covered.

[CMFDR]

Thanks for the explanations Rokko, go back to your thesis man!

[Vergeltungswaffe]

@Rokko have you ever gotten the chance to upgrade this for unpacking the updated SF2 campaigns?

I'd really like to not have Java on this new computer as scancade is the only thing that would use it.

[Vergeltungswaffe]

I'll take it that's a no...