Update on Engine 4 patches

[Battlefront.com]

Patches for all five existing CM2 Families are coming right on the heals of the CMSF2 release.  This is a lot later than they should be out to you guys, and we are very sorry for it, however we feel realistically that's the best we can do at this point.  That's the short answer for you.

The longer explanation is the nature of CM2 has thrown us a bit of a curveball that requires us to change the way we patch.  In hindsight we should have figured this out months ago, however with our plates as full as they are we didn't see where this would end up until recently.

Since the very beginning of CM1 we pretty much worked on one game at a time.  We'd fix things for the next game, make sure they were working fine, release that game, then patch the previous game.  That worked fine up until the five Family Game Engine 4 Upgrade came out and we now had to make sure that fixes worked equally well for five games all at the same time.  Given the complexity of CM2 games there's a lot that goes into making sure we're not putting out a bad patch and having to go right back into patching again.

The primary problem is with a dozen or two fixes queued up, the chances that a fix might not work correctly, or in fact cause a new problem, for one or more of the five (soon to be six) games is pretty good.  Especially when you consider that the Game Engine code is always being worked on, which means we have to make sure brand new code doesn't mess up things for older releases.  That's a lot of work for our testers.

The solution we came up with is to cut down on the complexity of any one patch.  Instead of having dozens of fixes in one patch we'll keep it down to the bare minimum, release a patch fairly quickly, then let some time go by and patch another modest number of fixes, and repeat as often as needed.  It's too late to adopt this strategy now, so we have to wait until CMSF2 is out the door before going through the patch process.

OK, I can hear some of saying that an alternative solution is to put out "beta patches".  Sorry, no.  As soon as gamers are given something that is called "beta" they start thinking they are beta testers and wanting to have their 2 cents of input on everything.  Not just stuff we're trying to fix but stuff that isn't even broken.  This creates entitlement that we can't possibly live up to, which means we get slammed.  In short, we try to do something good and get treated like bad guys.  No, that's not a road we want to go down.

There you have it.  The long explanation of how we got to this point and our plans for addressing it in the future.

Steve

[Ithikial_AU]

8 minutes ago, Battlefront.com said:

Since the very beginning of CM1 we pretty much worked on one game at a time.  We'd fix things for the next game, make sure they were working fine, release that game, then patch the previous game.  That worked fine up until the five Family Game Engine 4 Upgrade came out and we now had to make sure that fixes worked equally well for five games all at the same time.  Given the complexity of CM2 games there's a lot that goes into making sure we're not putting out a bad patch and having to go right back into patching again.

Many blue moons ago when the whole Upgrade system was announced I mentioned that long term it was going to turn into a bit of a logistical nightmare trying to balance multiple number of game 'families'. At the time there was only two families (I think), now we're heading for six.

Has this experience changed BF's thinking around release strategy? For instance bolting on on more modules to CMRT to push that front back to say Kursk, rather than releasing a whole new family for July 1943 - May 1944 eastern front? I'm no programmer so no idea if it would make a difference man hour wise but logistically/project management wise wouldn't there be some gains?

[IICptMillerII]

A frustrating but completely understandable situation. Very much appreciate the explanation as well as the plan for going forward. I think having more incremental patches instead of single large ones is a good direction to go in. 

Thanks for the update! Sounds like summer is going to be quite the season of releases. Very much looking forward to it all!

Can you elaborate at all on what the patch will be fixing/tweaking?

[Heirloom_Tomato]

4 minutes ago, Ithikial_AU said:

Many blue moons ago when the whole Upgrade system was announced I mentioned that long term it was going to turn into a bit of a logistical nightmare trying to balance multiple number of game 'families'. At the time there was only two families (I think), now we're heading for six.

Has this experience changed BF's thinking around release strategy? For instance bolting on on more modules to CMRT to push that front back to say Kursk, rather than releasing a whole new family for July 1943 - May 1944 eastern front? I'm no programmer so no idea if it would make a difference man hour wise but logistically/project management wise wouldn't there be some gains?

Or connecting CMFB with CMBN? 

Thanks for the update Steve, it is nice to know some of the backstory behind the delay.

Edited June 4, 2018 by Heirloom_Tomato

[TGarner]

Thank you for the update and the honesty.  

You guys do what you need to do, I can only speak for myself but the issue was never about getting the patch ASAP it was more of an open and honest communication issue and you have addressed that here.  I personally very much appreciate that.  I hope you guys(Devs) don't just look at us as a bunch of entitled kids here yelling for more stuff.  It is easy for the testers in this community who know what's going on behind the scenes to keep a cool head and in some cases mock and point fingers at those of us who are not testers for seeking information they themselves generally have. 

I hope you and the rest of the team understand our frustration and from the sound of your post you do.  We love you guys(not in a gay way) and obviously love your product.  Hell this game series has been a major part of my entire gaming life, so it is important.

Keep up the good work and thank you again for the honest communication. That goes a LONG way!

 

Edited June 4, 2018 by TGarner

[Battlefront.com]

8 minutes ago, Ithikial_AU said:

Many blue moons ago when the whole Upgrade system was announced I mentioned that long term it was going to turn into a bit of a logistical nightmare trying to balance multiple number of game 'families'. At the time there was only two families (I think), now we're heading for six.

We did anticipate the concept that things would become more complex as we moved forward.  However, we could not know the specifics until we got into it.  Things were deceivingly fine until Upgrade 4 came out.  The Upgrades themselves didn't feel like a huge deal either, though now it's clear why... we dedicated 100% of our development time to making those Upgrades.  So yeah, duh, of course it didn't seem like a big distraction from other things because we didn't have other things going on!

8 minutes ago, Ithikial_AU said:

Has this experience changed BF's thinking around release strategy? For instance bolting on on more modules to CMRT to push that front back to say Kursk, rather than releasing a whole new family for July 1943 - May 1944 eastern front? I'm no programmer so no idea if it would make a difference man hour wise but logistically/project management wise wouldn't there be some gains?

It's definitely entered into our thinking, though I'm not sure how the future will proceed.  The primary dilemma we have is that the price of what we release needs to be proportional to the effort that went into it.  CMFB, for example, was a major project for us and charging $35 as a Module for CMBN would not have been worth it to us.

That said, I think our perspective on Families and expansions has evolved over the past 10 years.  I do expect to see changes from us in the future.

5 minutes ago, IICptMillerII said:

A frustrating but completely understandable situation. Very much appreciate the explanation as well as the plan for going forward. I think having more incremental patches instead of single large ones is a good direction to go in. 

Thanks.  Our tradition of being comprehensive seems to have bitten us in the buttocks this time around.  Best we can do is learn from that and adapt.

5 minutes ago, IICptMillerII said:

Thanks for the update! Sounds like summer is going to be quite the season of releases. Very much looking forward to it all!

Can you elaborate at all on what the patch will be fixing/tweaking?

Oooo... good question!  Well, the one I can say for sure is fixed is the one you are most concerned with.  And that is TacAI behavior in relation to enemy fire and cover (in particular Fortifications).  What looked to be a fairly straight forward and binary problem instead turned out to be something that was very nuanced and difficult to pin down.  Sure, we made improvements pretty quickly, but there were all kinds of corner cases where things didn't work as intended or in fact were made worse by the fixes.  It took quite a long time to get a handle on it and develop fixes (plural) to address the underlying problems.  Not surprisingly, the root of this came from the Game Engine 4 feature that introduced proactive TacAI behavior under fire.

There's oodles of fixes, small and large, that cover such things as a specific formation in a specific game not having something it should, something being mislabeled in another game, etc.

Steve

[Battlefront.com]

4 minutes ago, TGarner said:

I hope you guys(Devs) don't just look at us as a bunch of entitled kids here yelling for more stuff.

Trust me, that's definitely not the case.  As I said in the first of the posts that I put out, you guys were left in an info vacuum for too long.  That's my fault.  We've tried some things over the past year that worked in some ways but not in others.  This was one of the things that definitely didn't work out so well.  Knowing there's a problem is more than 1/2 the battle, so definitely expect to see improvements going forward.

I'll be honest, it is time consuming and very distracting to keep conversations going with you guys.  It's not just the time that's involved (which is cumulatively significant), it's the mental energy that goes along for the ride.  It's not helped by the fact that both value and enjoy the conversations here, which I have to be careful doesn't get in the way of the other ton of things that needs doing.  Being here is not exactly like a big slice of Boston Cream Pie with a side scoop of vanilla ice cream kind of temptation, but in game development terms it's kinda in the same ballpark   Tough to strike the right balance sometimes.

Steve

 

[TGarner]

Steve no doubt I am sure it is hard maintaining ongoing communication here and getting all the work you guys have ahead of you done.

Your participation here is very much appreciated but I think most everyone understands you have work to do, heck considering the age group of CM fans I am sure we all have other things we need to be doing   But hanging out here of course is more fun.

Like I said before and not to repeat myself but I really do personally appreciate your honesty and the information you have provided here tonight.  I am super duper times infinity excited about the patch and most of all SF2!  I might even find some sand to dump in my underwear just to get the full immersion when that baby comes out!

Have a good week!

 

Edited June 4, 2018 by TGarner

[IICptMillerII]

31 minutes ago, Battlefront.com said:

Thanks.  Our tradition of being comprehensive seems to have bitten us in the buttocks this time around.  Best we can do is learn from that and adapt.

It's completely understandable in context. Between the major engine upgrade, the new website, new game releases, staff turnover, a new model on using 'contractors' for textures and the like, upgrading Shock Force, new modules and other projects like defense contracts (phew!) there has been a whole lot of transition this past year or so. That's bound to cause some friction. I can only speak for myself, but I think you've more than made up for the understandable information vacuum. 

37 minutes ago, Battlefront.com said:

Oooo... good question!  Well, the one I can say for sure is fixed is the one you are most concerned with.  And that is TacAI behavior in relation to enemy fire and cover (in particular Fortifications).  What looked to be a fairly straight forward and binary problem instead turned out to be something that was very nuanced and difficult to pin down.  Sure, we made improvements pretty quickly, but there were all kinds of corner cases where things didn't work as intended or in fact were made worse by the fixes.  It took quite a long time to get a handle on it and develop fixes (plural) to address the underlying problems.  Not surprisingly, the root of this came from the Game Engine 4 feature that introduced proactive TacAI behavior under fire.

There's oodles of fixes, small and large, that cover such things as a specific formation in a specific game not having something it should, something being mislabeled in another game, etc.

This is fantastic news! I can only begin to imagine the nightmare it was trying to isolate the specific issue in the code that needed fixing. That, and I know how much time and effort must be sunken into patching, and the relatively little return in value developers get from such work. I really do appreciate the efforts that must have gone into finally squashing that bug.

Can't wait to see what other tweaks are in store with this patch, and for Shock Force 2 and beyond! Never thought I would wish for the summer to hurry up and get a move on. 

[Anson Pelmet]

The heads-up is much appreciated. Been playing Combat Mission since Beyond Overlord was your only game, and nothing sampled since has come close to matching its realism. Especially looking fwd to the WW2 East Front addition. I and others appreciate your honesty, and the integrity of your product!

[Aragorn2002]

1 hour ago, Anson Pelmet said:

The heads-up is much appreciated. Been playing Combat Mission since Beyond Overlord was your only game, and nothing sampled since has come close to matching its realism. Especially looking fwd to the WW2 East Front addition. I and others appreciate your honesty, and the integrity of your product!

+1

[Bulletpoint]

4 hours ago, Battlefront.com said:

We'd fix things for the next game, make sure they were working fine, release that game, then patch the previous game.  That worked fine up until the five Family Game Engine 4 Upgrade came out and we now had to make sure that fixes worked equally well for five games all at the same time. 

I'm a bit confused ... Wasn't one of the big advantages of engine 4.0 that all games now run on the same code base? If so, doesn't that mean that you only have to fix things in one place now, instead of several?

[Holman]

Thanks for the update, Steve!

I never doubted it, but it's good to hear that progress is being made.

[Artkin]

26 minutes ago, Holman said:

I never doubted it

Pfff it's unlikely that he would stick an extra unit or two in your TOE 

[Ghost of Charlemagne]

Great news about the patch and CM:SF2, it was long awaited.

Any chance of a special effects pack/module in the future?
It would include full 3d explosions, improved explosion craters based on the kind of ground (i.e. on hard ground like a paved road or rocky ground the crater would be much more shallow), more detailed building destruction (i.e. each building floor is divided into numerous "destruction-sections", instead of just one), flammable buildings, terrain and flora, blood and gore and animations for wounded pixeltruppen.

Would anyone else be interested in such a pack/module?

[Dynaman216]

5 hours ago, Bulletpoint said:

I'm a bit confused ... Wasn't one of the big advantages of engine 4.0 that all games now run on the same code base? If so, doesn't that mean that you only have to fix things in one place now, instead of several?

They took care of the problem with adding any new models and such was a nightmare.  Now they have the "problem" that the TACAI is very complex.  The biggest problem right now is the infantry running away from entrenchments but that same fix also gave us infantry that spread out when moving forward instead of moving in a line to get shot like ducks in a row, Shooting around corners, and a couple other items I forgot.  I can easily see how that code base is a pain to work with.

[A Canadian Cat]

6 hours ago, Bulletpoint said:

I'm a bit confused ... Wasn't one of the big advantages of engine 4.0 that all games now run on the same code base? If so, doesn't that mean that you only have to fix things in one place now, instead of several?

Since I cannot comment on BFC development I'll comment on my experience (25 years as a product software developer) and a typical scenario that exists right now with my company. We release new versions of our software with new features and at any given time we have, lets say, six versions in use by our customers. In our case each of those versions are extensions of the previous software. In our case that leads to one source tree with six branches from a single trunk. The older ones are further way from the most current versions. This is not exactly the same situation that BFC has but it should do to give you insight into what is involved. If a customer using, worst case, the oldest version hits a problem and we find a fix we can patch that older version. If we also need to fix it in the newer versions then someone needs to merge that change into each branch. Sometimes that's easy because it is in an area that has not changed much other times - well it can suck. Either way each one of those patches needs to be tested independently. For one reason you should never ship software that just compiled - even if it is "just the same" as another version but more importantly changes in other ares of the product can sometimes be effected in surprising ways or worse new features may have been built based on behaviour that turns out was not quite right. All of which means that the testing needs to cover more than just that one little thing that was a problem. We have to at least touch the other features and depending on the fix we add more tests if we are afraid of the side effects or actually find a problem during initial testing. That's a lot of effort.

We typically try to not patch everywhere and fix things in the branch the customer hit the problem and in the latest release we are working on. And we fix just one thing since it can limit the amount of testing. Those strategies plus encouraging customers to move to newer versions - especially if they find a bug that we already fixed - tries to minimize the patching work we do. This is such a significant effort that we actually have a separate team that patches and tests software in production so that we can let the main development team stay focused on new features. Obviously our small team does sometimes interrupt the main development team when we need additional help but our goal is to manage it without distracting from new features.

[Muzzleflash1990]

My simplistic interpretation is that consolidating all games into one codebase simplifies development. But each development change now affect all games, so instead of maintaing N codebases where each change has to tested in one game, instead we have  1 codebase where each (some) change needs to be tested in N games. I don't know if this is actually the case, but I could see it sorta make sense since there is only one main developer to work on the code base, but a larger collection of (underappreciated) beta testers available to test the changes.

[Txema]

Steve,

Thank you very much for the update on the engine 4 patch. I really appreciate it.

Looking forward to the patch :-)

Txema

[Sequoia]

Will CMSF II be released with the patch fixes already in?

[sttp]

To add something more specific and CM-related to IanL's informative post: yes, it does at first seem like one AI change to the "trunk" of the tree should propagate across all 5 main titles. And I stubbornly thought this until maybe a few days ago....

Now I think about it this way: the AI coding would have to be slightly different for, let's say,  those 10-person, un-splittable, ****tily-equipped Italian squads. The code which says "this is when you stay with your crappy 10-man squad" vs. "this is when you get out of Dodge" would have to be different than is found in other CM titles... because other CM titles do not have that force structure. So it's easy to see, based upon 'only' the composition of a formation, how the little bits of code pushing and pulling the pixeltruppen behavior would have to be tweaked separately for each game.

Edited June 4, 2018 by sttp

[Oliver_88]

Thanks for update! Can't wait to see those Bren guns giving them bursts!

Am hoping for the mortar acquire bug, icon updating bug, finding unseen units exploit. Though maybe some of those came too late to have been included?

Quote

cover such things as a specific formation in a specific game not having something it should

Am hoping that could mean airborne and infantry formations getting their vehicles in quick battles just like they do in scenarios.

[Vergeltungswaffe]

[The Steppenwulf]

Thanks Steve for your frank and honest communication about the TacAi problems and the patch. It's really appreciated.

[Battlefront.com]

10 hours ago, Bulletpoint said:

I'm a bit confused ... Wasn't one of the big advantages of engine 4.0 that all games now run on the same code base? If so, doesn't that mean that you only have to fix things in one place now, instead of several?

Yes and no.  As Ian explained, having a common set of code doesn't mean everything works for all releases exactly the same way.  Each specific version, in our case a Family, has it's own quirks.  The consolidation of the families with Game Engine 4 vastly reduced the divergence, with CMFI being a big investment of time and CMSF2 being a HUGE one.  Using the same code (and licensing methodology too, BTW) makes things vastly easier than they were before.  However, easier does not mean easy

Here's some more on that...

3 hours ago, Muzzleflash1990 said:

My simplistic interpretation is that consolidating all games into one codebase simplifies development. But each development change now affect all games, so instead of maintaing N codebases where each change has to tested in one game, instead we have  1 codebase where each (some) change needs to be tested in N games. I don't know if this is actually the case, but I could see it sorta make sense since there is only one main developer to work on the code base, but a larger collection of (underappreciated) beta testers available to test the changes.

Exactly!  Things like shaders, sound code, user input, etc. are pretty universal and probably will work equally well for all games without any problem.  But there's plenty of things that are specific to a game or, even more commonly, specific to a single unit within a game.  In order to make sure that tweaking something for A doesn't adversely affect B, C, D, E, or F we have to test all 6 iterations just to be sure.

To keep on this theme of explanation...

3 hours ago, sttp said:

To add something more specific and CM-related to IanL's informative post: yes, it does at first seem like one AI change to the "trunk" of the tree should propagate across all 5 main titles. And I stubbornly thought this until maybe a few days ago....

Now I think about it this way: the AI coding would have to be slightly different for, let's say,  those 10-person, un-splittable, ****tily-equipped Italian squads. The code which says "this is when you stay with your crappy 10-man squad" vs. "this is when you get out of Dodge" would have to be different than is found in other CM titles... because other CM titles do not have that force structure. So it's easy to see, based upon 'only' the composition of a formation, how the little bits of code pushing and pulling the pixeltruppen behavior would have to be tweaked separately for each game.

This is an excellent example.  Those massive, clunky Italian formations are unique to CMFI.  Charles could make a change to the TacAI that works brilliantly for every single other unit in all the games but does something horrible to the Italians only.  If we don't test properly then the first customer that pops open a scenario with Italians in it would say "HEY!!  This is broken!!!  Fix it!!".  So then we rush out a patch to fix that one problem in that one game only to see 3 days later someone find a totally different quirk in a totally different game and say "HEY!! This is broken!!! Fix it!!".

Another great example is something like a damage effect.  Charles makes some adjustment so the modern 122mm Wingdingit now causes a bit more damage to Superduper Armor.  The testers go for it and find it works great.  Ooops... but the WW2 version of the 122m Wingdingit now has modern properties to it and incorrectly slices through things it should bounce off of.  Maybe it was because Charles didn't see how the code would apply to WW2, or it could be simply that he "fat fingered" a value while fixing the modern version.  Either way, not doing comprehensive testing practically assures us that something like this will slip through into a patch.

Every single patch comes with a semi-fixed amount of administrative and development overhead to it.  Whether it's one fix of a hundred, the overhead remains constant.  We have to minimize the amount of patching we do or we'll spend too much of our time patching and not making new stuff.

3 hours ago, Sequoia said:

Will CMSF II be released with the patch fixes already in?

Yup!  The fixes made to the Engine 4 code are automatically included, which is why we can't retroactively patch the AI proactive fire behavior without also patching in untested new code.  Hence the need to do pretty intensive testing.

After the next round of patches we're going to approach things differently.  When a serious issue is found with the released games we'll stop what we're doing to the Game Engine for the next game, make patches based on the current code, and get them out to you.  What this does is limit the divergence of the released code (what you guys have to play with) and the current state of the Game Engine (what we're working on internally).  Limiting the divergence means an easier time verifying the patches are good to go.

Steve