Any plans for 1.12?

[Guest]

I've written tens - if not hundreds - of thousands of automated tests in industries where legal or technical requirements demanded them. I've also fixed hundreds of bugs in CM. I know that automated testing would catch very few of the bugs that really cause us trouble. I also know that retroactively adding significant levels of automated tests to our existing code base would cripple our ability to expand CM, fix big bugs, and add features.

(And, as an aside, if you think issues like the wandering crew member problem are really that easily boxed in, I think perhaps you need to revisit your assumptions. You're looking at dozens of variables, at best, very few of which are captured in saved games after the fact.)

I don't think Steve is calling anyone a "professor". He's saying that not every development technique - regardless of whether various industries view them as best practices - fits our available development time, or development needs. We do code defensively, we test automatically at critical junctures, we do a very good job on a very complex piece of code, but we have very, very little development time. Certainly not enough to plan and code automated testing for every major piece of code.

I can also tell you with certainty that game companies at our staffing levels where I have acquaintances don't do mass automated testing. Even with larger staffs it's a rarity... they may do it for tools, or critical pieces of code, but they leave the rest to their QA. I suspect it's because QA for games is necessarily more holistic and encompassing than in other disciplines, but don't quote me on that.

[Battlefront.com]

This is most unfortunately. The right thing for me to do at this point would be to walk away from this thread. But when my credibility is attacked I'm compelled to respond.

Hmm... well, you're the one that called our credibility into question. So I was simply correcting you because, frankly, you don't know what you are talking about when it comes to game development. You probably know a lot about something else, but clearly you are confusing how you think things should work from how they actually work. A common enough mistake, so no ill will towards you for thinking one size fits all.

I *think* I know a little about software engineering and clearly I am not a professor and I think have proven that I can survive in the real world and understand market pressures.

Have you written any games as a paying gig lately? I have 20 years of experience in the games industry, which last time I checked is the only thing that's relevant to this discussion. I am telling you, emphatically, that automated testing is NOT DONE. No game developer in their right mind would pay for such work. A little here and there? No doubt. But systematically? No. Which means either all games programmers and their employers are idiots, or there are issues which make it an unwise choice and that's why they don't go that route.

If I were writing software where people's lives and safety were at risk I'd be singing a different tune. I also wouldn't be selling the end product for $45 a pop.

[calming down]

If you can't have a calm, rational intellectual discussion with someone that has a different viewpoint... well... you're definitely not in the right place.

Like I said its unfortunately that you made such a rash assumption. Certainly you don't have to agree with anything posted here. I just felt that I wanted the record straight regarding my background.

I will not respond to the rest of your comments directly nor replying to this thread any longer since it clearly would not be productive.

You are correct about one thing... it's not productive. When it comes to the realities of game programming you don't have a clue what you're talking about. When it comes to DoD contract programming... I don't have a clue. The difference is I'm not telling you what your industry is like, but you are telling me what mine is.

Steve doesn't know what he's talking about here, and it's actually embarassingly obvious and painful to read - for every software engineer with some experience who has ever worked on big projects in the real world.

How much experience do you have with game development? Just curious because you are saying you know more about it than I do.

Mind baffling response by Battlefront, really. :eek:

Ask any game developer, and I mean any, if they do automated testing and you'll find the same "baffling" response.

Oh, I see Phil just posted. I guess if my response was baffling, his will be downright insane.

Steve

[Battlefront.com]

Exactly.

So instead of having different codebases for 1.x and 2.x you use the 2.x codebase for both and just disable a couple new toys to warrant the price difference. That way there's no code porting nightmare.

And so you are saying that Charles and Phil didn't think of this before hand? You know, that they're blithering idiots for not seeing something obvious? I just want to get it straight what your thinking here is.

I don't think there was anything in 1.x that is gone in 2.x.

From an end user feature standpoint? Nothing. But the graphics code, for one, was effectively rewritten. The code handling soldier models and textures was completely rewritten. The models themselves are incompatible with the old system. Significant sections of UI code were rewritten. Networking code was seriously altered. Etc.

Sure, in theory Charles and Phil could keep track of all changes, big and small, and maintain a single code base. But c'mon... you should know better than most that at some point keeping track of a thousand minor to major changes over the course of multiple unique sub projects over multiple years turns into a huge drag on development compared to just starting fresh.

Steve

[MikeyD]

I know an old college buddy who some years ago found himself at an event sitting at the same table as an editor for 'Consumer Reports' magazine. My buddy commenced to tell this poor editor in excruciating detail exactly how their product tests are all muddled and precisely what formula should be followed for optimal product testing. It wasn't until three decades later that the psychiatric community came up with a name for the condition 'Asperger's syndrome'.

[Battlefront.com]

II don't think Steve is calling anyone a "professor". He's saying that not every development technique - regardless of whether various industries view them as best practices - fits our available development time, or development needs.

Exactly. The game programmers I've worked with have all said academic schooling wasn't much use for them in practical terms. I'm sure more than a few would recommend to an aspiring high school programmer to get a degree in math or physics and not programming. At least for the sorts of programming skills we tap into.

I can also tell you with certainty that game companies at our staffing levels where I have acquaintances don't do mass automated testing. Even with larger staffs it's a rarity... they may do it for tools, or critical pieces of code, but they leave the rest to their QA. I suspect it's because QA for games is necessarily more holistic and encompassing than in other disciplines, but don't quote me on that.

Yes, it is that and the fact that game companies don't care if there are bugs in their software. Not inherently. Especially because they calculate the customer will lose interest in the game itself before they either find or are significantly bothered by the bugs that slip through. This is not a cynical calculation, it is an economic one.

Even with $50,000,000 budgets (that is the current ballpark for the big games) there are loads of bugs in their programs. Which should say either they don't do automated testing or they suck at it. I know it's the former and not the latter.

Steve

[Wicky]

Dr. Temple Grandin - Animal Handling Inventions

http://www.women-inventors.com/Dr-Temple-Grandin.asp

Purnell (1998) quotes Temple Grandin as saying “The best facilities and the latest technology make handling cattle easier but they don’t make the manager; and until the owner or manager is convinced that proper handling practices pay off economically, it’s unlikely that employees will follow procedures day-in and day-out. The manager that is most effective in maintaining high humane standards is involved enough in day-to-day operations to know and care, but not so involved that he or she becomes numb and desensitized (Grandin, 1994).

I know an old college buddy who some years ago found himself at an event sitting at the same table as an editor for 'Consumer Reports' magazine. My buddy commenced to tell this poor editor in excruciating detail exactly how their product tests are all muddled and precisely what formula should be followed for optimal product testing. It wasn't until three decades later that the psychiatric community came up with a name for the condition 'Asperger's syndrome'.

[Battlefront.com]

Dr. Temple Grandin - Animal Handling Inventions

http://www.women-inventors.com/Dr-Temple-Grandin.asp

Sounds like game development

Steve

[Redwolf]

And so you are saying that Charles and Phil didn't think of this before hand? You know, that they're blithering idiots for not seeing something obvious? I just want to get it straight what your thinking here is.

From an end user feature standpoint? Nothing. But the graphics code, for one, was effectively rewritten. The code handling soldier models and textures was completely rewritten. The models themselves are incompatible with the old system. Significant sections of UI code were rewritten. Networking code was seriously altered. Etc.

Sure, in theory Charles and Phil could keep track of all changes, big and small, and maintain a single code base. But c'mon... you should know better than most that at some point keeping track of a thousand minor to major changes over the course of multiple unique sub projects over multiple years turns into a huge drag on development compared to just starting fresh.

Steve

I still don't understand why you want to keep the technical aspects of the 1.x code.

I understand that you'd like to have a 1.x and a 2.x line of patches so that you can charge for the latter and can use the former for plain fixes at no cost when problems pop up. (and to state for those who don't know, I am in favor of that scheme)

But why do you think the old rendering needs to stick around? Why not use the 2.x codebase for the free 1.x fixes, too, and simply disable a couple select features (but do not go with two renderers, of course).

[ASL Veteran]

I understand that you'd like to have a 1.x and a 2.x line of patches so that you can charge for the latter and can use the former for plain fixes at no cost when problems pop up. (and to state for those who don't know, I am in favor of that scheme)

Always trying to slip something in there under the radar eh? The 2.0 upgrade was $10, but patches to anything in version 2.0 will be at no charge. So now you will have version 1x patches at no charge and version 2x patches at no charge (after making the initial upgrade purchase of course). Sure, the upgrade had some problems in it, but you shouldn't be conflating the 2.0 upgrade itself with any subsequent patches that will be applied to the 2.0 upgrade. Of course, you claim to be in favor of that 'scheme' though so I guess we shouldn't think that you are taking another back handed slap at the BFC crew should we?

[Redwolf]

Always trying to slip something in there under the radar eh? The 2.0 upgrade was $10, but patches to anything in version 2.0 will be at no charge. So now you will have version 1x patches at no charge and version 2x patches at no charge (after making the initial upgrade purchase of course). Sure, the upgrade had some problems in it, but you shouldn't be conflating the 2.0 upgrade itself with any subsequent patches that will be applied to the 2.0 upgrade. Of course, you claim to be in favor of that 'scheme' though so I guess we shouldn't think that you are taking another back handed slap at the BFC crew should we?

You are confused.

There is nothing back-handed or slippery about me saying that BFC taking on even more code bases than they have to is insane. Have you made a count lately?

[Wicky]

Hire her to do the Normandy moo cow modelling for the CM-BN odds and sods pack

Sounds like game development

Steve

[Rocky Balboa]

I still don't understand why you want to keep the technical aspects of the 1.x code. .....

But why do you think the old rendering needs to stick around? Why not use the 2.x codebase for the free 1.x fixes, too, and simply disable a couple select features (but do not go with two renderers, of course).

What I fail to understand is why so many people have a trouble telling the difference between an upgrade and a patch. Patches are free fixes to old code and updates are paid upgrades to new features/new code. I think as Steve said there is a significant difference in the two code bases, not just disabling a few features. Many of the differences in the code bases have to do with how the 2 versions interact with the graphic models hence the reason why so many of the bitmaps and models changed in the 2 versions.

Doing what you suggest would mean BF would have to also put together a free patch to 1.x that would include all of those updated models. This would mean that you would have some players who paid for code in and upgrade that other players would get for free in a patch. Can you imagine the problems and angst within the community this would cause? Of course you can....

[Redwolf]

What I fail to understand is why so many people have a trouble telling the difference between an upgrade and a patch. Patches are free fixes to old code and updates are paid upgrades to new features/new code. I think as Steve said there is a significant difference in the two code bases, not just disabling a few features. Many of the differences in the code bases have to do with how the 2 versions interact with the graphic models hence the reason why so many of the bitmaps and models changed in the 2 versions.

Doing what you suggest would mean BF would have to also put together a free patch to 1.x that would include all of those updated models. This would mean that you would have some players who paid for code in and upgrade that other players would get for free in a patch. Can you imagine the problems and angst within the community this would cause? Of course you can....

I didn't suggest anything like that. You would disable a suitable amount of stuff so that the result is a good compromise between (1) keeping code differences low and (2) provide upgrade value.

Just for the heck of it, why don't we look at how much BFC has on their plate, with 2 programmers:

• CMBN
  
  • Windows version
    
  • Mac version
    
    • Vanilla
      
    • Mac store
      

  [*]CMSF

  • Windows version
    
  • Mac version (thankfully no mac store yet)
    

  [*]Mobile (CM touch)

  • IOS
    
  • Android
    
    • Amazon shop
      
    • Google play
      

Not all of these have different codebases, but these are the things you need a separate release process for.

Within the space of the main CMBN codebase, it is split like this, all in active development:

• 1.x version still maintained
  
• 2.x version is active fix development
  
• 3.x version is what carries combat mechanics development, except the MG fix is in the 2.x version so there's cross-branch pollution already
  

Anybody else sees a problem here when comparing to the development resources at hand?

Cutting out the 1.x branch might "leak" some candy to people "for free" but you have to weight that against saved development time.

P.S. exercise to the reader: put in games and modules here.

[mjkerner]

You are confused.

There is nothing back-handed or slippery about me saying that BFC taking on even more code bases than they have to is insane. Have you made a count lately?

Redwolf, you are undoubtedly no dummy, and know eons more about computers under the hood than me, but why don't you let BFC figure out their own development plans and schemes. They have shown that they thoroughly plan ahead (alot) and you're hints, complaints and suggestions aren't going to make one whit of difference to them. No disrespect meant, but please, just let it go.

[Battlefront.com]

Redwolf is correct about one thing. In theory having a single code base would be the best way to go. And it is exactly why all v2.0 games will come out of a single code base, not 6 or so (one for each Family). It would be a nightmare to try and do that, not to mention then support an equal number of code bases for the previous version.

Which is why we are sticking as close to theory as possible. One code base for the current version for all Families, one code base for the previous version for all Families. Which means basically 2 code bases instead of 12 or more when we get to v3.0. Sure, it's not 1 instead of 12, but that's because theory is never perfect.

There are literally thousands of differences between CM v1.x and CM 2.x. They are scattered all over the place. It is not as simple as toggling something on or off. Feature A might expect a value of 2 in one place for v1.x, but because of a redesign in v2.x the expected value is 1.99. Over time these sorts of extremely minor, simple changes would become a nightmare to keep track of. Worse, it greatly increases the chances of causing new bugs that would otherwise not be there. For example...

v1.x has a feature that uses a value of 5 that comes from a separate function. Other functions were written based on that assumption. The feature is rewritten for v2.x and in the process the value is changed to 10. The programmer now has to a) keep in mind that the old function won't work with the new value and keep separate sub-sections working the old way c) document them so 2 years later he knows what they are for d) hope there isn't unintended consequences and e) have to deal with all bug fixes and future work based on two sets of elements instead of one.

Sure, isolated here and there this isn't a big deal. But CM is huge and it's inner workings are inherently interconnected in ways that defy simplistic cause/effect relationships. Trying to maintain these relationships with even one version is a difficult enough task on its own. To expect trying to maintain two (or more) sets of relationships within a single code base is really asking for trouble.

Nope, the vastly superior approach is to have one code base where anything is possible. Make as many changes as you want whenever you want with no special consideration for backwards compatibility consequences. Then, when you want to update the older version you select isolated changes and go to the older code base and put them in without worrying about negatively affecting the current code base. Since we're talking about fixes instead of major new features, the workload is extremely manageable because the scope is narrow.

And that is why Redwolf's theory doesn't stand up to reality. And that gets us back to my original point... either Charles and Phil are idiots or Redwolf lacks the practical experience managing a project of this size and scope to understand where theory and reality diverge. Based on our 20 years of experience making games you guys obviously think are a cut above the rest, I think the answer is pretty clear that we're not idiots.

Steve

[slysniper]

Redwolf, you are undoubtedly no dummy, and know eons more about computers under the hood than me, but why don't you let BFC figure out their own development plans and schemes. They have shown that they thoroughly plan ahead (alot) and you're hints, complaints and suggestions aren't going to make one whit of difference to them. No disrespect meant, but please, just let it go.

Plus, learn who you are talking about.

I can tell you they are not doing the touch coding at all, that is a outside firm that has taken the game and coded it for touch. They just make a little money off it. So that wrecks your one concept of whats on their pallet

[Battlefront.com]

Just for the heck of it, why don't we look at how much BFC has on their plate, with 2 programmers:

Your list is all wrong. We have only two code bases:

CM v2.0 with toggles for Mac and Windows specific features. Additional toggles for Module/Theater specific features and content. Additional toggles for App store specific needs.

CM v1.0 with toggles for Mac and Windows specific features. Additional toggles for Module/Theater specific features and content. Additional toggles for App store specific needs.

CM Touch is a totally separate product created and maintained by a 3rd party. It also has toggles for different OSes and sales mechanisms.

When CMx3 comes out we will still only have two code bases:

CM v3.0 with toggles for Mac and Windows specific features. Additional toggles for Module/Theater specific features and content. Additional toggles for App store specific needs.

CM v2.0 with toggles for Mac and Windows specific features. Additional toggles for Module/Theater specific features and content. Additional toggles for App store specific needs.

There will be no v1.0 maintained. It will be a "dead project" just like CMx1 is.

Redwolf, you are undoubtedly no dummy, and know eons more about computers under the hood than me, but why don't you let BFC figure out their own development plans and schemes. They have shown that they thoroughly plan ahead (alot) and you're hints, complaints and suggestions aren't going to make one whit of difference to them. No disrespect meant, but please, just let it go.

Quite Redwolf is welcome to code his mega complex games over the next 10 years any way he wants to. We'll stick to doing ours the way that works best for us based on our experience.

Steve

[Mad Mike]

I've written tens - if not hundreds - of thousands of automated tests in industries where legal or technical requirements demanded them. I've also fixed hundreds of bugs in CM. I know that automated testing would catch very few of the bugs that really cause us trouble. I also know that retroactively adding significant levels of automated tests to our existing code base would cripple our ability to expand CM, fix big bugs, and add features.

I can believe that it would cripple your ability.

(And, as an aside, if you think issues like the wandering crew member problem are really that easily boxed in, I think perhaps you need to revisit your assumptions. You're looking at dozens of variables, at best, very few of which are captured in saved games after the fact.)

I didn't make any assumption on any specific bug being caught be automated testing. But it will catch bugs, including some which will have an impact on the overall stability or gameplay.

I don't think Steve is calling anyone a "professor". He's saying that not every development technique - regardless of whether various industries view them as best practices - fits our available development time, or development needs. We do code defensively, we test automatically at critical junctures, we do a very good job on a very complex piece of code, but we have very, very little development time. Certainly not enough to plan and code automated testing for every major piece of code.

So you do test automatically after all .. why is that, if it doesn't suit your development style?

I can also tell you with certainty that game companies at our staffing levels where I have acquaintances don't do mass automated testing. Even with larger staffs it's a rarity... they may do it for tools, or critical pieces of code, but they leave the rest to their QA. I suspect it's because QA for games is necessarily more holistic and encompassing than in other disciplines, but don't quote me on that.

Yeah, but in your special case the "QA" is just not working so well - barbed wire crashes, quick battle crashes, wandering crewman .. not a track record show case advocating the QAs effectiveness.

[Mad Mike]

You are correct about one thing... it's not productive. When it comes to the realities of game programming you don't have a clue what you're talking about. When it comes to DoD contract programming... I don't have a clue. The difference is I'm not telling you what your industry is like, but you are telling me what mine is.

You're in the software engineering business as well, which is quite universal in its principles. The basic failure or bug causing reasons are the same for every piece of software, no matter what it is programmed to do.

I also think you don't have a clue about programming or software engineering, but of course you have huge experience as a game designer, which I'm obviously not questioning.

How much experience do you have with game development? Just curious because you are saying you know more about it than I do.

I know more about software engineering than you, without a doubt. About game development, not really .. but in the end it is all software development which follows universal principles. Just because it is a game doesn't make it so special - that's just a chip on your shoulder.

Ask any game developer, and I mean any, if they do automated testing and you'll find the same "baffling" response.

Oh, I see Phil just posted. I guess if my response was baffling, his will be downright insane.

Steve

Nope, still quite the opposite, while Phil's response was quite balanced even with the admission that some automated testing seems to be done already, yours is still trying to paint everything in black and white with the justification that your software product is a game - which doesn't really matter.

[sfhand]

I'd like to comment here... I once started reading a book on C++ programming (never finished), and the author wrote "In theory, theory and practice are the same; in practice, they never are".

But that is an aside comment, my main purpose for this post is to update the relevant information for my fellow data miners. Here is the relevant data mining information gleaned from this thread:

v3.0 target date: Late 2013

expected v2.0 titles: "6 or so"

current v2.0 titles: 2

currently announced future v2.0 titles: CMSF2 CMEF1

So, unless Steve considers 4 "6 or so" we have 1,2, or 3 unannounced titles to look forward to by the end of the year (unless BF is going to develop v2.0 titles after v3.0 is released).

[Guest]

I can believe that it would cripple your ability.

But you're aware of how much effort it would take to retroactively add exhaustive automated testing to a very large code base.

And you know how much development time we have available to us.

So... how would this NOT cripple our ability to turn out features?

I didn't make any assumption on any specific bug being caught be automated testing. But it will catch bugs, including some which will have an impact on the overall stability or gameplay.

Great. No one doubts that. None of the major bugs from the last two years or so would have been caught, though - and *not* caught at what cost?

So you do test automatically after all .. why is that, if it doesn't suit your development style?

You must know that when I say "we test automatically at critical junctures" I'm talking about very different levels of testing from what's being discussed in this thread. Defensive automated testing in highly critical / suspicious areas of code is so commonplace I can't believe you'd conflate it with full-on automated testing - unit testing, scripted run-throughs - which is what everyone here has talked about so far.

AND, I didn't say it didn't fit our development "style". I said it didn't fit our needs or resources. Which are very different things.

Yeah, but in your special case the "QA" is just not working so well - barbed wire crashes, quick battle crashes, wandering crewman .. not a track record show case advocating the QAs effectiveness.

And the hundreds of bugs they do catch? You're picking exceptions and saying they prove your rule. They *are* exceptions, though. And putting "QA" in quotes isn't helping me see your side of things, Mike. We can have a reasonable discussion about this, but if you're going to pick or prod it's going to end pretty quickly.

[Mad Mike]

But you're aware of how much effort it would take to retroactively add exhaustive automated testing to a very large code base.

And you know how much development time we have available to us.

So... how would this NOT cripple our ability to turn out features?

I actually did not try to be ironic. I really believe that something like this would cripple your ability. So there is no argument here. Things are the way they are, especially real constraints like time and resources.

Great. No one doubts that. None of the major bugs from the last two years or so would have been caught, though - and *not* caught at what cost?

You must know that when I say "we test automatically at critical junctures" I'm talking about very different levels of testing from what's being discussed in this thread. Defensive automated testing in highly critical / suspicious areas of code is so commonplace I can't believe you'd conflate it with full-on automated testing - unit testing, scripted run-throughs - which is what everyone here has talked about so far.

AND, I didn't say it didn't fit our development "style". I said it didn't fit our needs or resources. Which are very different things.

But style follows needs and resources, surely.

And statements like " I am telling you, emphatically, that automated testing is NOT DONE." are not mine (also, admittedly, not yours). But we can only discuss things which are being stated, so maybe some care should go into these statements.

And the hundreds of bugs they do catch? You're picking exceptions and saying they prove your rule. They *are* exceptions, though. And putting "QA" in quotes isn't helping me see your side of things, Mike. We can have a reasonable discussion about this, but if you're going to pick or prod it's going to end pretty quickly.

Sure, as I'm the customer and not involved in your development. The bugs which remain are all I have to go by in my judgement. As such, I'm not satisfied with the bugs which are in the game. Nothing more, nothing less.

But those bugs I mentioned should have been caught, especially since they are in areas which are common in a lot of typical games of CMx2. Start a QB via PBEM and check if everything works.

Create a map with every terrain feature and placeable object (mines etc.) and check if everything works, in play.

I apologise for the quoted "QA", it was indeed a cheap shot on my part.

I'm interested, does your QA have a catalogue or list of points and functional areas (user workflow) of the game which have to be checked before the game or a patch is released?

[Battlefront.com]

You're in the software engineering business as well, which is quite universal in its principles.

Sure, like all mammals have the same basic organic composition and anatomy. But saying this is good enough to conclude that a dairy cow could manage the farm's finances is unsound.

Like any profession, there is absolutely no "universal" skill set, nor approaches. Everything is customized for it's particular end goals given certain resources and design limitations. Charles could no sooner program the Mars Rover than the Mars Rover programmer could make Combat Mission using the same "principles". It's silly to suggest otherwise.

The basic failure or bug causing reasons are the same for every piece of software, no matter what it is programmed to do.

Sure, just like things fall to the ground because gravity isn't relative. But the reasons why two objects fall at different velocities depends on the specifics, does it not?

I also think you don't have a clue about programming or software engineering, but of course you have huge experience as a game designer, which I'm obviously not questioning.

You are digging yourself into a bigger hole with each post. You're wrong again because I work with game programmers every day. And have done so for 20 years. I've been responsible for their hiring and firing based on their capabilities. What's more important, I have to design to their capabilities, which means I must understand what their capabilities and limitations are. Or do you think a civil engineer can design a sky scraper one day and a bridge the next without understanding construction techniques and issues unique to each?

What's more, if you were paying attention you'd see that Phil does know game software engineering. And he's saying you're wrong and that I'm correct.

Now, could I program a game? Nope. I gave up programming 25 years ago when I realized I would never get better than basic functionality. I still do scripting, which is a form of programming, but it's not in the same class. But I must have a pretty good understanding of game programming to be able to do my job as both a game designer and as the business manager for the products we produce. If you haven't been in a position like mine before I can understand you not understanding this.

I know more about software engineering than you, without a doubt.

No argument at all.

About game development, not really .. but in the end it is all software development which follows universal principles.

But economic and practical principles are not.

Just because it is a game doesn't make it so special - that's just a chip on your shoulder.

No chip on my shoulder. I am interested in any technique that will make my life, and my bank account, happier. But I am not fool enough to think there's a reason game developers out should use automated testing to the degree being suggested here in this thread. That seems to be a chip on your shoulder.

Nope, still quite the opposite, while Phil's response was quite balanced even with the admission that some automated testing seems to be done already, yours is still trying to paint everything in black and white with the justification that your software product is a game - which doesn't really matter.

I also said we use automated testing in places. I also stated that there are gains to be had from it. What I am saying, and Phil is also saying, is that the sorts of automated testing being advocated here won't work because they are not viable for this application.

Tell you what. Find one game programmer who has worked on multiple, published games of any significant size/complexity that agrees with you. I am confident you will never find such a programmer. And there is a reason for that.

Steve

[Battlefront.com]

I actually did not try to be ironic. I really believe that something like this would cripple your ability. So there is no argument here.

Then what the heck are you arguing about?

And statements like " I am telling you, emphatically, that automated testing is NOT DONE." are not mine (also, admittedly, not yours). But we can only discuss things which are being stated, so maybe some care should go into these statements.

Cherry picking single lines out of context doesn't win you any points in my eye. Why not quote the rest of that paragraph?

So much for your position that I'm in conflict with Phil or that I'm viewing this as Black and White. What's more, even this more expanded quote is out of context of the discussion.

Sure, as I'm the customer and not involved in your development. The bugs which remain are all I have to go by in my judgement. As such, I'm not satisfied with the bugs which are in the game. Nothing more, nothing less.

Are you satisfied with the bugs in the operating system you are using? How about the video card in your computer? How about other games? How much did you pay for that stuff and how much does other things you buy and do with them rely upon them?

But those bugs I mentioned should have been caught, especially since they are in areas which are common in a lot of typical games of CMx2. Start a QB via PBEM and check if everything works.

In theory bugs should not exist in any piece of software or any hardware platform. How well does that theory work in reality?

Create a map with every terrain feature and placeable object (mines etc.) and check if everything works, in play.

And how many specific circumstances work as intended? Thousands. Success to failure ratio is definitely relevant to this discussion unless you believe in absolutes only.

I'm interested, does your QA have a catalogue or list of points and functional areas (user workflow) of the game which have to be checked before the game or a patch is released?

In my previous life as a corporate gaming QA Manager we used to have checklists like this. We do not use checklists like that now. Based on my experience I say we have fewer problems without the checklist mentality than we had with ridged sign offs on checklists in my old job. And that was with vastly more simple games.

What we do is have a wide enough array of testers playing the game different ways with some guidance from us what elements should be tested. At times it is general, sometimes it is specific. When we are satisfied with the results we release.

It's the standard rule of diminishing returns. The amount of energy/time/resources we would have to put into hunting for a few extra bugs is not worth the investment. Not for us, not for you. Because all effort put into defensive programming means effort not put into expanding the capabilities of the game. I am sure if we took a survey the vast majority of our customers would agree that having a few bugs, which do get fixed, is an acceptable tradeoff for getting a bunch of new game functionality they would not otherwise get.

As I said earlier, if I was in charge of software development where people's lives were on the line I would have a completely different approach. And I would be charging appropriately for that approach as well.

Steve

[Mad Mike]

In my previous life as a corporate gaming QA Manager we used to have checklists like this. We do not use checklists like that now. Based on my experience I say we have fewer problems without the checklist mentality than we had with ridged sign offs on checklists in my old job. And that was with vastly more simple games.

What we do is have a wide enough array of testers playing the game different ways with some guidance from us what elements should be tested. At times it is general, sometimes it is specific. When we are satisfied with the results we release.

That would not be good enough for me, but of course I'm not going to tell you how you should run things.

I can only tell you that I'm not really satisfied with the final result with regards to stability and fatal, game-preventing bugs.

I am sure if we took a survey the vast majority of our customers would agree that having a few bugs, which do get fixed, is an acceptable tradeoff for getting a bunch of new game functionality they would not otherwise get.

Steve

That would be an interesting survey. I am not sure one way or the other. But I would definetely like to see some survey like this.