Battlefront.com Posted January 16, 2013 Share Posted January 16, 2013 Regarding professional software. The company I work for pays around 30000 US$ licence fee for a major finite element software package. I was unable to use the latest versions of this software for two years, because they - simply put - confused x and y coordinates for one output variable. Back, when I reported this bug, I thought that it would be fixed immediately, because it could cause postprocessing programs to give totally wrong results. Instead, it took them two major releases to finally fix this. And the fix looks more like a hack, anyway. Puts complaining about a PBEM game bug in a $50 program that will be fixed within about a month into perspective, doesn't it? I will also remind the critics that NASA plowed a $125m satellite into Mars because one engineering team used Metric and another used English values. If NASA isn't using automated testing good enough to detect that... well, I think maybe we should be cut a little more slack for the barbed wire save bug. Steve 0 Quote Link to comment Share on other sites More sharing options...
Wicky Posted January 16, 2013 Share Posted January 16, 2013 I will also remind the critics that NASA plowed a $125m satellite into Mars because one engineering team used Metric and another used English values. If NASA isn't using automated testing good enough to detect that... well, I think maybe we should be cut a little more slack for the barbed wire save bug. Fatally floored it was 0 Quote Link to comment Share on other sites More sharing options...
Buzz Posted January 16, 2013 Share Posted January 16, 2013 Puts complaining about a PBEM game bug in a $50 program that will be fixed within about a month into perspective, doesn't it? I will also remind the critics that NASA plowed a $125m satellite into Mars because one engineering team used Metric and another used English values. If NASA isn't using automated testing good enough to detect that... well, I think maybe we should be cut a little more slack for the barbed wire save bug. Steve Exactly what gets me about some business comments.:confused: I spend way more than $50 for ammunition on a range day. If I get a misfire, I confirm it is not a weapons system issue to be sure. I chuck the bad round and keep going down range. It is a "range day" not a focus on 1 misfire in 500 rounds day. 0 Quote Link to comment Share on other sites More sharing options...
Mad Mike Posted January 16, 2013 Share Posted January 16, 2013 Puts complaining about a PBEM game bug in a $50 program that will be fixed within about a month into perspective, doesn't it? I will also remind the critics that NASA plowed a $125m satellite into Mars because one engineering team used Metric and another used English values. If NASA isn't using automated testing good enough to detect that... well, I think maybe we should be cut a little more slack for the barbed wire save bug. Steve One final comment (hopefully, I can already hear people groaning ). I cut you slack for bugs, they do happen. But I would never go so far as saying that I'm not unsatisfied by them being in the product. Also, after reflecting what has been written here over the past days, this discussion for me is not about comparisons and what other companies, which sometimes make shiploads of money, do equally bad, but what you - Battlefront - could do better. I understand now, as a result of this discussion, that you don't see any way of expanding your automated testing. This is due to constraints in resources/budget and a perceived return on investment which is too small. But pointing to others as examples for bad practices and things gone wrong to paint own mistakes in a better light is not really a constructive way of thinking. That's why I tried not to compare and rather speak about general principles. I can see that this seems like preaching about theory, but it is the only starting point really. In my opinion, it is fair enough to say: "In principle, automated unit testing helps the developer to create cleaner code and enables him to implement and check changes reliably and quicker, especially for unintended side effects in the code unit behaviour. Due to the special nature of the software project under development and a resource/budget constraint, we made the decision to not do this and try to ensure overall quality by other measures". It is not fair enough to say: "Since NASA uses automated testing and they still had bugs, this whole automated testing stuff can not be worth much and we should therefore not do it." It is simply a wrong argument since it completely neglects that the automated testing has probably caught thousands of smaller and bigger bugs and errors in this NASA project and the "metric and English values" anecdote is rooted in faulty application of specifications. After reading the wikipedia entry about this example, automated testing could have picked this up - but was probably not implemented correctly. Still, this discussion has been worthwhile and educational for me, since I now understand your way of working a little bit better. I still don't agree with the automated testing part of it, but it's of course not really my concern. 0 Quote Link to comment Share on other sites More sharing options...
JonS Posted January 16, 2013 Share Posted January 16, 2013 Hi Mike, I tried not to compare and rather speak about general principles. I can see that this seems like preaching about theory, but it is the only starting point really. But ... pointing to principles IS inherently comparing. Principles don't spring fully formed out of nowhere, they're based on prior experience which may or may not be relevant. By pointing favourably at those principles, you are in effect saying that BFC's (and every other game company's) business is comparable to the situation in which those principles were developed. You might be right, although an entire industry is suggesting that maybe you aren't. It is not fair enough to say: "Since NASA uses automated testing and they still had bugs, this whole automated testing stuff can not be worth much and we should therefore not do it." ... automated testing could have picked this up - but was probably not implemented correctly. Sure. And that can be said about everything that fails, especially when it fails so catastrophically and publically: "it probably wasn't implemented correctly." And that is, I think, the exact point that Steve and Phil have been trying to make. Yes, in principle, given unconstrained budgets and resources, an automated testing regime could be put in place and implemented correctly, and CM might be better for that. Meanwhile, in this universe, budgets and resources are constrained, and the odds of perfectly implementing an ATR is zero. The point of the NASA example is that even with all their intellectual, physical, and financial resources, they still fell victim n00b mistake which all their testing completely failed to notice, destroyed the mission, and wasted practically ALL the resources spent on it. If the best that ATR has to offer is "well, maybe your project won't crash and burn?" AND it costs a ton of resources that then can't be used to create game features ... why bother? Especially when the stakes are so much more modest than life saving medical equipment, inteplanetary travel, or the like? tl;dr: Horses for courses. Regards Jon 0 Quote Link to comment Share on other sites More sharing options...
Battlefront.com Posted January 17, 2013 Share Posted January 17, 2013 I cut you slack for bugs, they do happen. But I would never go so far as saying that I'm not unsatisfied by them being in the product. And I would NEVER expect you to be happy about experiencing bugs. Why should you be? Which means I expect you guys to voice concern, frustration, etc. when you experience a bug that significantly affects your enjoyment of the game (well, at least reasonably!). But this hasn't been the point of your posts. Not at all. Instead... But pointing to others as examples for bad practices and things gone wrong to paint own mistakes in a better light is not really a constructive way of thinking. Neither is judging someone's work based on a theoretical practice which much better resourced organizations, with a LOT more riding on their mistakes, can't measure up to. Worse, when I showed that theory doesn't work very well in the real world, and explained why, you said I was the one that was clueless. That I, not you, had no clue how games were made in the real world, not to mention the games produced under by my own company. You don't see that being slightly bad logic on your part? That's why I tried not to compare and rather speak about general principles. As JonS pointed out, you can NOT separate these two things when you elect to hold someone accountable to those theoretical practices. I can see that this seems like preaching about theory, but it is the only starting point really. Correct. Which is why I pointed out that game programmers are more than likely to say that what they learned in expensive university programming courses doesn't help them out much in the real world. In my opinion, it is fair enough to say: "In principle, automated unit testing helps the developer to create cleaner code and enables him to implement and check changes reliably and quicker, especially for unintended side effects in the code unit behaviour. Due to the special nature of the software project under development and a resource/budget constraint, we made the decision to not do this and try to ensure overall quality by other measures". Still flawed because you presume that there is a practical choice being made. As Phil and I have both tried to explain, it's not a choice. Not in the real world. It is not fair enough to say: "Since NASA uses automated testing and they still had bugs, this whole automated testing stuff can not be worth much and we should therefore not do it." It is simply a wrong argument since it completely neglects that the automated testing has probably caught thousands of smaller and bigger bugs and errors in this NASA project and the "metric and English values" anecdote is rooted in faulty application of specifications. After reading the wikipedia entry about this example, automated testing could have picked this up - but was probably not implemented correctly. My point was to demonstrate that the assumption that if we had rigorous automated testing in place that there wouldn't be the PBEM bug or the old barbed wire bug, is demonstrably false. This was a position that was not put forward by me, but by you and two others. Still, this discussion has been worthwhile and educational for me, since I now understand your way of working a little bit better. I still don't agree with the automated testing part of it, but it's of course not really my concern. Correct, because you're not a games programmer If you got a job as a games programmer you would either a) change your view on automated testing or get fired. There's no third path I can see after my 20 years of being in this industry. Steve 0 Quote Link to comment Share on other sites More sharing options...
fatehunter Posted January 17, 2013 Share Posted January 17, 2013 Gentlemen, I am not a games programmer. I do work in risk assessment in the field of marine transportation. Simplified, what we do is compare the 'likelihood' of a potential event against the 'consequence' of that potential event. Then we spend our limited resources on changing the conditions that allow those potential events that with a combined score are unacceptable down to something we call 'as low as reasonably practicable'. We never fix everything. We fix it down to an acceptable and manageable level. I think this has a direct corollary with computer programming. This means some potential events are never addressed if they are highly unlikely, or if the consequences are very minor. With-out such a ranking scale, there is no way to systematically make an effective reduction in the risk you face every-day. This is just a systemic way of spending your limited resources wisely. It sounds to me like this is exactly what Steve does. Man this thread is getting tiresome. 0 Quote Link to comment Share on other sites More sharing options...
Michael Emrys Posted January 17, 2013 Share Posted January 17, 2013 Man this thread is getting tiresome. Amen. I hope that everybody who has something to say on this particular subject is finished so that we can move on. Michael 0 Quote Link to comment Share on other sites More sharing options...
Georgie Posted January 17, 2013 Share Posted January 17, 2013 Hmmm, what can we learn from this about getting Steve's attention? 0 Quote Link to comment Share on other sites More sharing options...
sburke Posted January 17, 2013 Share Posted January 17, 2013 Hmmm, what can we learn from this about getting Steve's attention? The subjects that 99% of us want to hear about get 1% of Steve's time.... 0 Quote Link to comment Share on other sites More sharing options...
Vanir Ausf B Posted January 17, 2013 Share Posted January 17, 2013 LOL, Yep. ... 0 Quote Link to comment Share on other sites More sharing options...
dan/california Posted January 17, 2013 Share Posted January 17, 2013 He could surely scatter out a bone or two if he really wanted too. It does wonders for enlivening conversations hereabouts. I have been unreliably informed that the something nice is a robotic nanny that actually give me time to play the bloody game 0 Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.