Mad Mike Posted January 15, 2013 Share Posted January 15, 2013 Then what the heck are you arguing about? Cherry picking single lines out of context doesn't win you any points in my eye. Why not quote the rest of that paragraph? So much for your position that I'm in conflict with Phil or that I'm viewing this as Black and White. What's more, even this more expanded quote is out of context of the discussion. Are you satisfied with the bugs in the operating system you are using? How about the video card in your computer? How about other games? How much did you pay for that stuff and how much does other things you buy and do with them rely upon them? In theory bugs should not exist in any piece of software or any hardware platform. How well does that theory work in reality? And how many specific circumstances work as intended? Thousands. Success to failure ratio is definitely relevant to this discussion unless you believe in absolutes only. In my previous life as a corporate gaming QA Manager we used to have checklists like this. We do not use checklists like that now. Based on my experience I say we have fewer problems without the checklist mentality than we had with ridged sign offs on checklists in my old job. And that was with vastly more simple games. What we do is have a wide enough array of testers playing the game different ways with some guidance from us what elements should be tested. At times it is general, sometimes it is specific. When we are satisfied with the results we release. It's the standard rule of diminishing returns. The amount of energy/time/resources we would have to put into hunting for a few extra bugs is not worth the investment. Not for us, not for you. Because all effort put into defensive programming means effort not put into expanding the capabilities of the game. I am sure if we took a survey the vast majority of our customers would agree that having a few bugs, which do get fixed, is an acceptable tradeoff for getting a bunch of new game functionality they would not otherwise get. As I said earlier, if I was in charge of software development where people's lives were on the line I would have a completely different approach. And I would be charging appropriately for that approach as well. Steve Sorry for the double reply, but if I may be allowed to try to summarise your point of view on this, I would say that: You are quite happy with the stability and bug-to-feature ratio of the game. All bugs, should they occur, are collateral damage which can not be avoided in any way given the constraints and even if they could be avoided, it doesn't matter since the customer actually accepts this state of affairs and any further modification of the way you're testing is therefore not necessary. I would not agree with this as there are always measures for improvement, a defined set of "test points" or (even selective) automated unit testing being two of them. If my summary is incorrect, I would like to know where I misinterpreted what has been said. 0 Quote Link to comment Share on other sites More sharing options...
Battlefront.com Posted January 15, 2013 Share Posted January 15, 2013 You are quite happy with the stability and bug-to-feature ratio of the game. All bugs, should they occur, are collateral damage which can not be avoided in any way given the constraints and even if they could be avoided, it doesn't matter since the customer actually accepts this state of affairs and any further modification of the way you're testing is therefore not necessary. Yeah, totally off the mark because of that chip on your shoulder. I'll summarize things below... I would not agree with this as there are always measures for improvement, a defined set of "test points" or (even selective) automated unit testing being two of them. I agree there is always room for improvement. Sometimes it is even practical. But we do not work in a vacuum with unlimited time and resources. We also know that overall customer expectations of quality and functionality are, in general, unreasonably high. Either from practical or technical standpoints. Which means we must keep all aspects of software development in mind when we make decisions. We can not make engineering decisions as if the other elements don't exist. If my summary is incorrect, I would like to know where I misinterpreted what has been said. Wrong in the sense that you are mischaracterize our motivations. Let me rework you words instead of writing new ones: ----- We are generally satisfied with the stability and bug-to-feature ratio of the game. Most bugs, should they occur, are collateral damage which practically can not be avoided in any reasonable way given our constraints. Other bugs, given hindsight, could have been avoided. Where a pattern of weakness emerges we invest in preventing future problems of a similar nature. Reasonable customers actually accepts this state of affairs provided the bugs are not excessive and that the serious ones addressed through free patches in a timely manner. Fundamental modification of the way we're testing is not practical, therefore we do the best we can with what we have to work with. ----- When I worked for Impressions (Sierra) my job was manage a QA department of 5 people full time. Our ONLY job was to find bugs and other sorts of flaws. That was probably a $300-400k annual expense for the company when you consider direct and indirect overhead. One game we tested almost exclusively every day for the better part of a year. When it shipped there was more than 1000 known bugs in it, ranging from fairly serious to really trivial. And more were found after release we hadn't found before hand. At least one serious one. Battlefront doesn't have the resources to do such testing. And even if it did, I'm not sure the bug count would be much lower than it is now. I can say that you'd be paying a lot more for your games though. Steve 0 Quote Link to comment Share on other sites More sharing options...
Javaslinger Posted January 15, 2013 Share Posted January 15, 2013 My god.. are you people still arguing about this??? I had always assumed that this company line that 'spending time on the forums takes too much time away from programming' as being a bunch of hooey... But now that I see it in action, I have to agree... 0 Quote Link to comment Share on other sites More sharing options...
sfhand Posted January 15, 2013 Share Posted January 15, 2013 A point of reference: Battlefield 3 Made by DICE and published by EA with a huge development team and even larger budget. I know for a fact that both BFBC2 and BF3 went through "public beta". If automated testing is superior to actual users with a variety of hardware and playing styles, then why would DICE and EA bother with the "public beta" after their "internal beta"? Sure, you could say marketing, and I wouldn't ignore that as one of many reasons, but the fact that their games have all had "bugs" on release should be somewhat revealing as to the pettiness of your dissatisfaction with CM's current state and Steve's assertion that all game releases have bugs (an assertion that holds up in my experience). 0 Quote Link to comment Share on other sites More sharing options...
Battlefront.com Posted January 15, 2013 Share Posted January 15, 2013 I can only tell you that I'm not really satisfied with the final result with regards to stability and fatal, game-preventing bugs. How satisfied are you with the bugs in the OS you are using? What's the most expensive piece of software you own? Are you satisfied with the level of bugs in that? Our artists have to regularly reformat their hard drives and reinstall everything because of the bugs in the 3D software we use. And that software sells for many thousands of Dollars, depending on configuration. There's an old saying. Those who have unreasonable expectations are going to be more disappointed in their experience than those with reasonable expectations. That would be an interesting survey. I am not sure one way or the other. But I would definetely like to see some survey like this. Something I definitely know more about than you is how our customers think, as a group. But I think it's pretty much a no brainer which they would choose: 1. An automated testing process that might, but only might, catch a serious PBEM bug ahead of time. A bug that we can easily catch after and patch for free within a reasonable amount of time. 2. TCP/IP WeGo, a Follow Command, and coding for flamethrowers (not the artwork) I have no idea how much time it would take to do #1, and only #1, but based on my experience I think the above choice is fairly realistic. Steve 0 Quote Link to comment Share on other sites More sharing options...
sfhand Posted January 15, 2013 Share Posted January 15, 2013 My god.. are you people still arguing about this??? I had always assumed that this company line that 'spending time on the forums takes too much time away from programming' as being a bunch of hooey... But now that I see it in action, I have to agree... I'd much rather have Steve comment on the current ROADMAP... 0 Quote Link to comment Share on other sites More sharing options...
Battlefront.com Posted January 15, 2013 Share Posted January 15, 2013 A point of reference: Battlefield 3 Made by DICE and published by EA with a huge development team and even larger budget. I know for a fact that both BFBC2 and BF3 went through "public beta". If automated testing is superior to actual users with a variety of hardware and playing styles, then why would DICE and EA bother with the "public beta" after their "internal beta"? Sure, you could say marketing, and I wouldn't ignore that as one of many reasons, but the fact that their games have all had "bugs" on release should be somewhat revealing as to the pettiness of your dissatisfaction with CM's current state and Steve's assertion that all game releases have bugs (an assertion that holds up in my experience). The great thing about thinking in a vacuum is you don't have to pay attention to realities like the above. Holding us to a standard that a $50,000,000 budgeted game can't live up to isn't very reasonable or realistic. Steve 0 Quote Link to comment Share on other sites More sharing options...
Mad Mike Posted January 15, 2013 Share Posted January 15, 2013 Yeah, totally off the mark because of that chip on your shoulder. I'll summarize things below... I agree there is always room for improvement. Sometimes it is even practical. But we do not work in a vacuum with unlimited time and resources. We also know that overall customer expectations of quality and functionality are, in general, unreasonably high. Either from practical or technical standpoints. Which means we must keep all aspects of software development in mind when we make decisions. We can not make engineering decisions as if the other elements don't exist. Wrong in the sense that you are mischaracterize our motivations. Let me rework you words instead of writing new ones: ----- We are generally satisfied with the stability and bug-to-feature ratio of the game. Most bugs, should they occur, are collateral damage which practically can not be avoided in any reasonable way given our constraints. Other bugs, given hindsight, could have been avoided. Where a pattern of weakness emerges we invest in preventing future problems of a similar nature. Reasonable customers actually accepts this state of affairs provided the bugs are not excessive and that the serious ones addressed through free patches in a timely manner. Fundamental modification of the way we're testing is not practical, therefore we do the best we can with what we have to work with. ----- When I worked for Impressions (Sierra) my job was manage a QA department of 5 people full time. Our ONLY job was to find bugs and other sorts of flaws. That was probably a $300-400k annual expense for the company when you consider direct and indirect overhead. One game we tested almost exclusively every day for the better part of a year. When it shipped there was more than 1000 known bugs in it, ranging from fairly serious to really trivial. And more were found after release we hadn't found before hand. At least one serious one. Battlefront doesn't have the resources to do such testing. And even if it did, I'm not sure the bug count would be much lower than it is now. I can say that you'd be paying a lot more for your games though. Steve Steve, thanks for the reply. For the sake of letting you get back to work instead of arguing with me, I can only say that I have to remain one of your unreasonable customers. I hope you can agree that this is certainly not the worst imaginable outcome of this small debate. 0 Quote Link to comment Share on other sites More sharing options...
MikeyD Posted January 15, 2013 Share Posted January 15, 2013 In a story in my area a local baseball hero tried to start his own computer game company. After just two years he filed for Chapter 7 bankruptcy with one hundred fifty million dollars in liabilitys, layed off all 400 employees and had to give up his huge office space. 400 employees? Huge office space? Say what you want about BFC, at least their business model isn't suicidal! :eek: 0 Quote Link to comment Share on other sites More sharing options...
Mad Mike Posted January 15, 2013 Share Posted January 15, 2013 A point of reference: Battlefield 3 Made by DICE and published by EA with a huge development team and even larger budget. I know for a fact that both BFBC2 and BF3 went through "public beta". If automated testing is superior to actual users with a variety of hardware and playing styles, then why would DICE and EA bother with the "public beta" after their "internal beta"? Sure, you could say marketing, and I wouldn't ignore that as one of many reasons, but the fact that their games have all had "bugs" on release should be somewhat revealing as to the pettiness of your dissatisfaction with CM's current state and Steve's assertion that all game releases have bugs (an assertion that holds up in my experience). Automated testing, especially for games, is not superior in any way to testing by users. If you interpreted my comments in this way, I can only clarify that automated testing can complement the testing by the users already in place, ensuring that certain bugs will not reach those users so that they may concentrate on testing the integrated product. I hope this is an understandable way of approaching software testing - testing from small chunks or individual parts towards more complex, user driven testing of the whole game. I wouldn't say my dissatisfaction is petty. I just don't agree with the "release it now, fix it later" mentality from a customer point of view, which is hopefully quite natural. Wholesale subscription to this approach as a customer, no matter which game developer does it (and they all do in these days), I would describe as "gullible". 0 Quote Link to comment Share on other sites More sharing options...
Javaslinger Posted January 15, 2013 Share Posted January 15, 2013 I'd much rather have Steve comment on the current ROADMAP... LOL!!! I know! He's finally active on the forums on a minute by minute basis defending the honor of the companies programming and QA but still mute on what 99% of us actually care about... Stop pandering to the tightass, whiny 1%! We are the 99% and we want some Road Map details!! 0 Quote Link to comment Share on other sites More sharing options...
Battlefront.com Posted January 15, 2013 Share Posted January 15, 2013 Steve, thanks for the reply. For the sake of letting you get back to work instead of arguing with me, I can only say that I have to remain one of your unreasonable customers. Understood I hope you can agree that this is certainly not the worst imaginable outcome of this small debate. Indeed. And I am hoping that you have a little more appreciation for our grasp of the situation. We wouldn't be a thriving niche company if we didn't have a sound understanding of what's going on. And as I said before, if I didn't understand the engineering aspects of programming I can promise you we'd not be having this discussion now. We'd have gone out of business eons ago. Designers without technical understanding are more correctly called "Dreamers". Doesn't take much imagination to understand which one you want making games. Steve 0 Quote Link to comment Share on other sites More sharing options...
Battlefront.com Posted January 15, 2013 Share Posted January 15, 2013 In a story in my area a local baseball hero tried to start his own computer game company. After just two years he filed for Chapter 7 bankruptcy with one hundred fifty million dollars in liabilitys, layed off all 400 employees and had to give up his huge office space. 400 employees? Huge office space? Say what you want about BFC, at least their business model isn't suicidal! :eek: My brother-in-law worked for them. I knew more than a year before the implosion that it was coming. Not because I had detailed information (the employees were clueless until the end), but because all the signs were there for a one way ride to oblivion if you knew how to read them. I wouldn't say my dissatisfaction is petty. I just don't agree with the "release it now, fix it later" mentality from a customer point of view, which is hopefully quite natural. We don't like it either. Patches are a PITA to do and they cost us time away from other things. But it comes down to balancing all factors. Would customers pay more to have a few less bugs? Would they wait longer for a few less bugs? Our experience is the vast majority of customers would answer NO and NO. Which means holding back our products and raising their prices to MAYBE eliminate a few nasty bugs is against our customers' wishes. Wholesale subscription to this approach as a customer, no matter which game developer does it (and they all do in these days), I would describe as "gullible". Believing that it can be different is naive. Expecting it to be different is only going to disappoint. LOL!!! I know! He's finally active on the forums on a minute by minute basis defending the honor of the companies programming and QA but still mute on what 99% of us actually care about... Stop pandering to the tightass, whiny 1%! We are the 99% and we want some Road Map details!! Read it a few days ago and didn't see much reason to comment. I'll go find that thread and change that. Steve 0 Quote Link to comment Share on other sites More sharing options...
ASL Veteran Posted January 15, 2013 Share Posted January 15, 2013 Shogun 2 has been out for at least a year or two now and Total War has released several DLC for different factions. I've been playing it a lot lately and I have to say that the naval battles are completely broken. How did this state of affairs come to pass with a game company that has a budget so large that they have their own motion capture studio and they could hire a programmer who did nothing but code water for an entire year? The problems with naval battles? Let me list a few of them 1. Enemy ships line up on the opposite side of the map, but don't move until your ships get into bow range. This allows me to move my ships all up to one flank and put the whipping on the enemy because they don't react until I've already taken my time and set up my trap. 2. Frequently, enemy ships run for the map edge even though they are in good order. For some reason the ships get 'stuck' on the map edge and can't leave. I typically let them sit there while I finish off all their friends, then I send my entire fleet to the map edges to pick off these stragglers who are stuck on the edge. 3 The Sengoku Bune, the sailing ship with no oars, when choosing to board one of my ships after I've positioned my ships UPWIND seem to travel at speeds appropriate for ships with an engine and a propeller shaft in them rather than a sailing vessel going upwind. They move at this speed in any direction when routing as well. 4. The pathing is pretty suspect at times. If there are a lot of surrendered or burning ships littering the sea (or even when there is open ocean this happens sometimes) then many times if you want to move a ship a few meters in one direction they end up doing a circular motion sailing around in a circle for, like fifty meters, before getting to that location that was originally about five meters away. The ironic thing is that they allow the ships to turn in place sometimes. 5. Ships will auto target when an enemy ship gets in range, but I've noticed that there is a lot of target 'fixation' where they will target the first ship that gets in range and they will continue to fire on that ship even though another ship might be approaching with the intent to board and is now closer than the first target which might be routing away. 6. If your ships aren't moving (these are oared vessels so you can just sit in place if you want to) then after they auto target the first vessel that comes in range they will stop shooting when that ship gets out of range. Those ships will then not auto target any other vessels that get in range, and this forces the player to manually target enemy ships or move your ships a little bit if you don't want to get fired upon without returning fire. How naval battles got past the testing phase is a complete mystery to me. Anyone who went in and played one could have seen all the problems with them. They haven't even fixed this problem yet either, and this game has been out for a while. They are now mostly working on Rome 2 I guess and creating new DLC faction packs and skins to charge people for. I suppose having working naval battles just isn't a priority for those guys. I wonder what that guy who was coding water for a year is doing now? Maybe he could do some bug fixing instead? 0 Quote Link to comment Share on other sites More sharing options...
Pvt. Ryan Posted January 15, 2013 Share Posted January 15, 2013 I'd much rather have Steve comment on the current ROADMAP... http://www.battlefront.com/community/showpost.php?p=1416431&postcount=12 0 Quote Link to comment Share on other sites More sharing options...
Rocky Balboa Posted January 15, 2013 Share Posted January 15, 2013 Originally Posted by Javaslinger View Post My god.. are you people still arguing about this??? I had always assumed that this company line that 'spending time on the forums takes too much time away from programming' as being a bunch of hooey... But now that I see it in action, I have to agree... Leave the man alone and let him talk. Everytime he posts, he drops little bones for us to gnaw on. Originally Posted by Battlefront Something I definitely know more about than you is how our customers think, as a group. But I think it's pretty much a no brainer which they would choose: 1. An automated testing process that might, but only might, catch a serious PBEM bug ahead of time. A bug that we can easily catch after and patch for free within a reasonable amount of time. 2. TCP/IP WeGo, a Follow Command, and coding for flamethrowers (not the artwork) 0 Quote Link to comment Share on other sites More sharing options...
Battlefront.com Posted January 15, 2013 Share Posted January 15, 2013 How naval battles got past the testing phase is a complete mystery to me If anybody wants to see what pushing a product out the door for money reasons looks like... well, you did a good job giving an example. Our customers complain when we miss projected ship dates. Either you get the game when you expect it, in whatever state it's in, or you wait and get it when it's ready. "Ready" of course is a tricky problem. Technically we have never sold any software which is "ready" if "ready" means bug free. And we never will because it's impossible, even if because other software will make sure something of ours doesn't work as expected (video card drivers... I'm pointing at you in particular!). Steve 0 Quote Link to comment Share on other sites More sharing options...
Redwolf Posted January 15, 2013 Share Posted January 15, 2013 Quite Redwolf is welcome to code his mega complex games over the next 10 years any way he wants to. Nah, a game I would do would be a lot simpler than this. I would never go with 1:1 representation without 1:1 control. In fact there are code pieces in CMx1 that I would simply delete. 0 Quote Link to comment Share on other sites More sharing options...
Battlefront.com Posted January 15, 2013 Share Posted January 15, 2013 After 10 years, you must be well along with it Steve 0 Quote Link to comment Share on other sites More sharing options...
Michael Emrys Posted January 16, 2013 Share Posted January 16, 2013 2. TCP/IP WeGo, a Follow Command, and coding for flamethrowers (not the artwork) Winner, hands down. Michael 0 Quote Link to comment Share on other sites More sharing options...
Thomm Posted January 16, 2013 Share Posted January 16, 2013 Not sure what kind of automated testing is written about here. One feature that could be helpful would be sped-up AI-vs-AI play. And again, there are certainly more important items than that. Regarding professional software. The company I work for pays around 30000 US$ licence fee for a major finite element software package. I was unable to use the latest versions of this software for two years, because they - simply put - confused x and y coordinates for one output variable. Back, when I reported this bug, I thought that it would be fixed immediately, because it could cause postprocessing programs to give totally wrong results. Instead, it took them two major releases to finally fix this. And the fix looks more like a hack, anyway. Best regards, Thomm 0 Quote Link to comment Share on other sites More sharing options...
Redwolf Posted January 16, 2013 Share Posted January 16, 2013 After 10 years, you must be well along with it Still waiting to get fired so that I have time 0 Quote Link to comment Share on other sites More sharing options...
sburke Posted January 16, 2013 Share Posted January 16, 2013 Let the results of the business model/plan speak for themselves. All subjective opinions aside, with no knowledge at all of BF's books and order quantities - Just looking at what they have been able to produce the past two years and the forecast going forward shows a company who's strategy appears to finally be hitting the stride they were shooting for. It is obvious to anyone who isn't simply refusing to see what is in front of them that BF has hit a qualitative point in their business plan. 2011 CMBN 2012 CMBN 1st Module CW - 2013 CMFI Version 2 upgrade and 1.11 patch for CMBN 2013 Our schedule is a bit flexible, but as of now for 2013: Normandy 2.01 patch Normandy 1.12 patch Italy 1.02 patch Something Nice CMBN Market Garden CMEF 1 (1944-1945, starting with Bagration) CMSF 2 (introduces v3.0) For 2014: CM Bulge CMEF 2 (1943-1944, starting with Kursk) Various Packs sprinkled in here and there along the way. But it's too difficult to say what and when specifically. We have no plans on doing a North Africa game at the moment, but I wouldn't rule it happening. Steve The negativity crowd and doomcriers are going to need new material as facts are running over them like an M1 over a speedbump. Facts trump opinion every day of the week and twice on Sundays. Version 3 better include some ability to affect the time space continuum or I am going to have to figure out an early retirement plan. It is hard enough just trying to decide whether to launch CMBN, CMFI or CMSF. 0 Quote Link to comment Share on other sites More sharing options...
Barkhorn1x Posted January 16, 2013 Share Posted January 16, 2013 In a story in my area a local baseball hero tried to start his own computer game company. After just two years he filed for Chapter 7 bankruptcy with one hundred fifty million dollars in liabilitys, layed off all 400 employees and had to give up his huge office space. 400 employees? Huge office space? Say what you want about BFC, at least their business model isn't suicidal! :eek: Interesting you bring up Curt Schilling. Some ASL vets may know him as he bought the ASL rights from Hasbro after AH Hill went bust and enjoyed some success w/ Multiman Publishing selling ASL products and other board games. I guess the PC gaming biz is a bit...ah..."different", which is the point Steve is making. 0 Quote Link to comment Share on other sites More sharing options...
Battlefront.com Posted January 16, 2013 Share Posted January 16, 2013 Still waiting to get fired so that I have time Heh. I think of all the great stuff that people could be doing but can't because their day jobs squash them. Which is why I quit the only two jobs I had post college. First one was in 1993 and I started up my own game company. That got crushed by the evils of retail distribution, the second was in 1997 when I quit Impressions/Sierra to start Battlefront. Neither would have happened if I kept working for The Man (though I really liked working for Impressions. Sierra? Not so much). Let the results of the business model/plan speak for themselves. The great thing is... The negativity crowd and doomcriers are going to need new material as facts are running over them like an M1 over a speedbump. Facts trump opinion every day of the week and twice on Sundays. It took nearly 30 years for people to stop predicting that Apple was going to go out of business the next day. We've only had the fun of proving people utterly wrong for 12 years, so we probably have a ways to go yet. What is clear to us is some people need a different hobby. If you're not good at something you should probably move onto something else I guess the PC gaming biz is a bit...ah..."different", which is the point Steve is making. Precisely Steve 0 Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.