What sucks about this board!!!!

[Guest MantaRay]

Well I think that the more I read it the more I want the game to come out. I mean it is not fair that people such as Fionn get to play the full version and then tell us all about it. I wanna play too!!!!!

------------------

[Fionn]

Oh, it's fair...

You don't see all the hard work testers put in every day and week. You just hear about "fun stuff". Beta testing and scenario design isn't a whole heap of fun.

I'd be willing to bet you'd drop out of testing after only a couple of weeks . I heard a person from a large wargaming company once say that only 10% of testers who they initially picked stuck through the long haul and delivered good work over the entire course of the project.

That puts the work into perspective a little.

------------------

___________

Fionn Kelly

Manager of Historical Research,

The Gamers Net - Gaming for Gamers

[Thomm]

I mean it is not fair that people (such as Fionn ) get to play the full version and are not allowed to talk about it !!!

[This message has been edited by Thomm (edited 01-17-2000).]

[Berlichtingen]

I've beta tested my share of software... it sucks

[Guest kevi]

"Beta testing and scenario design isn't

a whole heap of fun"

Why do users demand editors?

Why do folks volunteer to beta test?

I think it's a blast.

Unless one draws an significant income

from development (i.e. a job) the

whole thing must be fun (i.e. a

hobby)to participants. I think the

drop out rate of volunteers is high

in all walks of life. Wargaming is not special in that regard. It’s not a

question of "fun" per say, but time.

If an activity does not put food on

the table then anything can and will

get in the way of completing commitments.

I bet most beta testers and designers

feel bad when they are forced to drop

out because its a challenging commitment

that is fun to them.

- Kevin

[Bullethead]

Kevi said:

<BLOCKQUOTE>quote:</font><HR>I bet most beta testers and designers

feel bad when they are forced to drop

out because its a challenging commitment

that is fun to them.<HR></BLOCKQUOTE>

I've been testing wargames pretty much continuously since the early 80s (as in being an official tester, not just a shafted customer ). In all that time, I've only had to drop out once and that was the fault of the developers for assembling the test group and then not giving us the game for 6 months (BTW, the game hit the shelves about 3 weeks after I dropped out...). I felt bad about it, but only because it ruined my perfect record . I had long since given up hope of this game being any good.

IMHO, the main reason people drop out of wargame testing is due to problems with developers similar to the above. In my experience, wargame developers go to some trouble to try to only pick guys who have a genuine interest in the genre and really really want a good game of this type due to their scarcity on the market. So most testers I've worked with have been knowledgeable, enthusiastic (at first), and dedicated (at first). Just the type of guys customers would want doing the job, guys who will be happy to devote a lot of unpaid late hours to the project.

But then the developers/publishers shaft them. Reported programming (as opposed to modeling) bugs, even very serious ones, don't get fixed. Modeling errors (as opposed to design constraints or deliberate abstractions), proved by thorough research, don't get changed. And perhaps the most damaging, the testers often don't get credit for their work--no mention in the manual, no free copy of the retail version, etc. "It's by baubles that men are led," you know. And then the game hits the shelves, and sucks, and the poor testers get blamed for all its problems (which is one reason I usually don't mind being left out of the manual ).

No wonder folks drop out in-process, or don't volunteer for another project. I doubt very many feel bad about this, either.

This, of course, varies with developers and publishers. In my experience, the bigger the company, the worse they treat testers. In some cases, such as the one I bailed on, I've gotten the impression that the testers' main purpose was to serve as a practice group for training the customer support personnel to ignore everything. If they can be deaf to very knowledgeable, enthusiastic people honestly trying to help, they can handle any type of complaints from customers

OTOH, I've really enjoyed working with some smaller companies and lone wolves.

-Bullethead

[This message has been edited by Bullethead (edited 01-17-2000).]

[Mark IV]

I have lots of experience with beta testing and testers, but (unfortunately) not with wargames.

However, in our little industry our control software is kind of fun, if this is how you make your living. And new software is mostly driven by demand from the field and pressure from competitors.

With that said, I have to point out that very few people have the dedication to stick with testing or give us useful feedback. This includes the people who clamor for the new features in the first place.

We have the fault-finders, who gloat every time they can make something crash but can't document what they were doing at the time and don't take accurate (or any) notes. There is also a sub-category who are unable to understand the meaning of the word "beta test" and are genuinely diappointed and bitter if anything goes wrong (decorum and the high standards of the board prevent me from sharing the name of this category).

We have the ostriches, who give the app to customers and never follow up. When asked how things are going, they say "must be fine or we woulda heard something by now".

We have flakes who boot it up, click around a couple screens, and report no problem found as though they'd run it through its paces. This category is usually field sales people who are afraid that if they report any bugs, it will hold up the release date.

There are also folks who are so in awe of the project that they will accept things which are obviously wrong as gospel because it's on the screen in front of them.

There is no way to write a complex piece of interactive software and hypothesize every possible combination of things a user might try to do with it. Getting someone with the mental discipline to go over and over the same things in detail and DOCUMENT what they are doing is very difficult.

In fact, it is because of these insights that I have placed my spare time at BTS' disposal though my test version must still be in the mail....

By the way, the other truly amazing thing is the pressure from our customers to release new software as soon as they see a beta version- "how bad can it be? You can just fix it later...". And when it takes a dump and a system quits working, O the howling and pain and accusation... "don't you guys test this stuff?"

[Guest teutonic]

Well as far as beta testing has gone I had applied with a lot of companies to beta test and have only tested one game.

Unfortunately that game was Flight Commander from SSI. I spent the time trying things and then when it crashed or did something it shouldn't have I also spent the time writing the problem up. The thing with this game was, almost every beta tester was saying the same thing to the company : "This game needs more work". The company then went and released it right before Christmas time, still buggy and incomplete. They knew it was that way but the pressure from up high was to release in one of the most lucrative times for software sales.

I have to say if I was asked again I probably wouldn't test again for them.

Anyone else have this kind of experience?

Teutonic

Dang still a junior member

[This message has been edited by teutonic (edited 01-17-2000).]

[Lee]

Having beta tested, I can say for a fact that there is

a lot of work involved. So it's important that

you be a very big enthusiast of the subject matter

to help you get through all the crashes and trying

to track down what is causing them. It also helps a

lot if you have a high quality, dedicated, knowledgeable

group of testers to work with.

[Guest MantaRay]

Actually Fionn, I have beta tested quite a few games, and I was even a paid employee for a company for 2 years(perfect college job). But you have it much better than I have on most of the projects I have done, as the developers actually listen and try to implement the changes that you as a beta-tester and fan of the game make.

And Fionn, I was just joking anyway, and the post was just really made in jest...you lucky dog.

And a little hard work never hurt anyone.

------------------

[Bullethead]

Mark IV said:

<BLOCKQUOTE>quote:</font><HR>There is no way to write a complex piece of interactive software and hypothesize every possible combination of things a user might try to do with it. Getting someone with the mental discipline to go over and over the same things in detail and DOCUMENT what they are doing is very difficult.<HR></BLOCKQUOTE>

No argument. I've been involved with a number of "open beta" projects as well and all the bad tester types you mentioned have posted copiously on the related message boards. And it's just as bad from my POV as yours, because it seems to me that often quantity of posting is what sways company opinion, not quality. I'm sure this is a decision made by short-sighted marketing dweebs who will never have to use the software or deal with its consequences, but in any case it results in crap for me as the user, more work for you as the producer, and much heartburn all around. A thousand curses on open betas.

<BLOCKQUOTE>quote:</font><HR>In fact, it is because of these insights that I have placed my spare time at BTS' disposal though my test version must still be in the mail....<HR></BLOCKQUOTE>

Yeah, I too am using this thread as advertising for my ability and willingness to test CM2

<BLOCKQUOTE>quote:</font><HR>By the way, the other truly amazing thing is the pressure from our customers to release new software as soon as they see a beta version- "how bad can it be? You can just fix it later...". And when it takes a dump and a system quits working, O the howling and pain and accusation... "don't you guys test this stuff?"<HR></BLOCKQUOTE>

Hehehe, same here. Such customers shoot at everybody. You for "rushing" it out, me for "willfully negligent testing".

-Bullethead

[Bullethead]

Teutonic said:

<BLOCKQUOTE>quote:</font><HR>I have to say if I was asked again I probably wouldn't test again for them. Anyone else have this kind of experience?<HR></BLOCKQUOTE>

Here are a couple of "good" ones:

360 Software:

Harpoon, from the very beginning in the early 80s until 360 went tango uniform. This was a game where they built the graphics engine first, then tried to build the combat engine around it. EVERY ONE of the major bugs (crashes, teleporting air bases, planes homing in on the north pole, ships sailing through land, etc) was in beta #1 way back when. And never ever fixed. Subsequent versions merely added new bugs. I'm not shown in the docs as a tester, and I don't usually mention that I was.

SSI:

Fighting Steel was the game where they assembled the beta test group in November 1998. "We'll have a beta for you real soon, but for now please comment on the gunnery model." Then nothing more--despite previews appearing in magazines, they kept telling us those were all alphas and they didn't have it to beta form yet. By May 1999 when I no longer had the free time required for testing, we STILL had not received the game. Nobody in the beta group ever did. Yet the game came out a few weeks later. It listed "beta testers" in the docs, but none of them had been in the group.

-Bullethead

[Renaud]

I am very thankful for all the unpaid beta testers and scenarios designers out there helping make this game a great one. The idea of beta testing/scenario design for the final product has some superficial appeal until I think of the thrill of seeing the complete package with no spoilers and not having already been worn out by interminable bug hunts and scripted play-testing. Thank you beta testers!

-Ren

[Mark IV]

<BLOCKQUOTE>quote:</font><HR>Yeah, I too am using this thread as advertising for my ability and willingness to test CM2 <HR></BLOCKQUOTE>

Busted. Guilty as charged. Though from the beta selection point of view, I thought mine was better crafted and I should be first

As a somewhat technical long-sighted marketing dweeb (meaning my time-horizon extends beyond the next happy hour, if I'm REALLY focused), who is also often responsible for the original spec, I certify that my above statements are true and so are yours. The whole process is filled with humans and that's the real problem.

The big difference between my experience and beta-testing wargames is the "nobody listens" complaint, though we get that and it's sometimes true. It's just that if our stuff doesn't work, systems die, and BIG cash payers notice.

Most of my biggest sales efforts are done in our own R&D: "honest guys, the users love it. They just don't write C+. Couldn't we maybe just put a button there?" Programmers can be a sulky lot and are often bitter when the stupid, incompetent and underqualified end-users don't quite intuit what they're supposed to.

BTS is obviously exceptional in several of the above regards (and lord, their Kiwi tastes good ). Tomorrow morning I plan to storm into our own Engineering and demand rifle grenades. And I suspect I'll get some....

[Bullethead]

<BLOCKQUOTE>quote:</font><HR>It's just that if our stuff doesn't work, systems die, and BIG cash payers notice.<HR></BLOCKQUOTE>

Yup, that's the key difference between "real world" software and wargaming. Not only do you have a lot more competition to live up to, but customers are in effect trusting you with their whole livelihood. If your stuff doesn't perform, people can get fired, businesses can fail, lawsuits can crop up, all that.

I must say that my experience testing commercial software has been much more positive than with most game companies. I've had people actually call me on the phone offering help and wanting suggestions for improvements, for one thing. For another, some of these suggestions actually made it into the next version. It was almost as good as the way BTS does things .

-Bullethead

[Fionn]

Beat this,

I am listed in a manual by a company as "Historical Consultant" to their game yet I NEVER got a copy of the game . Mucho hours of unpaid work went into that and I didn't even get a copy.. Now, it's not that I'm totally hung up on getting a copy since I probably wouldn't play it in any case BUT the fact that I got one "beta/gold" CD and that was it really bummed me about the amount of effort I put in.

Companies that do that kind of stuff get bottom of the barrel testers and it serves them right IMO.

Bullethead, you know the one I mean ..

One thing I have noticed in some companies is a willingness to ignore most testers and just focus on the reports of a small group of testers... That's fine if you're in the small group whom they value due to some other work you've done for them etc but I must admit it leads to worse games.

Honestly, there are a few wargaming companies out there which have reputations which are terrible vis a vis treatment of and attention paid to their beta testers. That's why they have such a high turn-over IMO and never get a good quality team they can bring from project to project.

------------------

___________

Fionn Kelly

Manager of Historical Research,

The Gamers Net - Gaming for Gamers

[guachi]

Fionn,

Even I know which game you are talking about.

It wouldn't happen to be a game about things that float on water, would it?

Jason

[Moon]

Duck Hunter?

[guachi]

Ah, but what else floats on water?

Jason

(kepping fingers crossed that somebody gets my movie reference...)

[Cormand]

Very small pebbles?

[Das Boot]

Wood (PS: She turned me into a newt)

[civdiv]

Cormand, get it right: 'Very small rocks.'

[Mark IV]

<BLOCKQUOTE>quote:</font><HR>One thing I have noticed in some companies is a willingness to ignore most testers and just focus on the reports of a small group of testers<HR></BLOCKQUOTE>

Now, that is a great truth. I enjoy the psychology of why these things happen. In this case, you usually have a vocal, more technical sub-group who become a sort of "in crowd" with the development team. These guys all talk, out of school, because they all know each other. They reinforce one another's opinions and prejudices, and often will go to the mat for a single feature that a year later is insignificant and forgotten.

The development group is frightened to death of "missing the market" and will overreact to this group. Also, because of greater familiarity with this group, their complaints are given greater credence, and not so readily dimissed as "operator error".

[Tom Manning]

What else floats on water.....why everyone knows it's a bay ruth candy bar!

[Doug Beman]

"EEwwww! Dirty dirty!!"

{Bill Murray picks it up, sniffs it, takes a big bite}

"S'okay!"

DjB

ps you left out bread and churches.