How do y'all decide on your scenario ratings?

[SurlyBen]

I've reviewed a few scenarios over at the scenario Depot, and at the start I would just rate a scenario based on general feel. Then I decided that this was arbitrary in a way that was maybe unfair to the scenario designers (Ok, I designed a scenario and saw how hard it was to get it right...) So I've started to use a more consistent approach where I assume everything starts at 10 and mark off points from that (this is similar to the way one might grade a test).

A scenario would have to be pretty vile to score a one or a two (non existent briefing, or a briefing with lots of swearing and l337 d00dz spelling would get a 0 or 1 respectively) Most of my scores will be in the 6-10 range.

Then it occurred to me that some of you might be using 5 as baseline for an 'average' scenario, in which case your scores will be in the 3-7 range, even for a pretty good scenario.

I'm not suggesting that changes be made (I doubt we could reach a consensus ), I'm just kind of curious as to what methods (and/or criteria) people are using when they hadn out those ratings.

(also this give me another chance to plug my Knife fight at Cannes scenario. Also is it "knife fight", "knife-fight" or "knifefight"? My dictionary didn't say so I assumed the first, but it always seems strange to write. )

Surlyben

[Ligur]

Hmm, hear, hear, I score pretty much like you do. For a decent effort, i.e. a respectable scenario, obviously tested, but with no extra flavor will get a general score of 7 from me.

Lacking aspects will lower the score to 6, or 5. For example a scenario touted as a human vs human that infact is an automatic massacre for the other player would get a 5 or 4 point PBEM rating.

A map that has no effort (resembling a quick battle map) would get a score of 6 or 5 from me.

And so on.

For scenario goodness, like examplary play balance, a map that has been designed with passion, a long and inspiring briefing, superb or even good playability vs AI and such will get very good scores from me, most often a 9. I rate only the best I have played as 10.

I try to forget personal preferences when it is clear that the effort has been excellent, but just not my style. For example, if I wouldn't like long briefings, I would not rate the briefing as a "5" if it IS long, since the author obviously has tried (btw, I like long briefings) his best.

Still, I would encourage everyone to use the 10 available numbers, not just slap a 9 or 10 to everything that is kinda fun and 5 to everything that doesn't suit your own gaming preferences.

[Wild Bill Wilder]

I'll have to agree on the arbitrary choice of a "1" because someone does not like one's writing styles or feels he has too much info, or too little.

I could live with a "5" even but a consistent "1" on scenario briefings seems quite harsh. It is done, however, and rather consistently, with certain authors. And so it goes.

[CombinedArms]

Wild Bill,

Your briefings are always interesting--though sometimes deliberately deceptive! (Rumblings here from the Rumblings of War, where I--and my opponents!--have been burned more than once). I can't imagine you've gotten many 1's however--or even 5's, in your rating history.

Evaluating briefings can be a tricky, however. Some are almost obsessively detailed (they may give you too much information) whereas others are deliberately short. Esp. in human vs. human designed scenarios, the style of some prolific and talented designers (I'm thinking particularly of Rune here), is deliberately brief, leaving both players largely to their own devices. Also, some of the byte battles have witty but very short briefings. How does one rate such a one-or two sentence briefing on a scale of 1-10? You might give it a high rating because it's appropriate to the game. Or a low rating because of the lack of detail. I suppose one could always leave the space blank in such a situation? Anyway, that's one case I've found hard to handle.

[WWB]

Well, at first I tended to be guilty of score inflation. But now I use 5 as a baseline 'adequate but not good' rating. If there are real problems, or it is somehow unfinished, I will rate lower. But if it is better then I will rate higher.

WWB

[Splash]

Perhaps the base should be the value of a QB. That being say 2 of 10. Now I can improve any QB by giving the player -30% so say that is a 4.

The problem is that this game cannot produce a 1.

The "gamey" factor is another consideration.

Gamey at it's 2 poles are:

A.) the truly annoying

B.) that's annoying I now have to learn something new.

Personally I think the scale should be reversed scaled out of 4-- in that 4 is item A.) and 1 is item B.)

(Altho' for simplicity sake they might be reversed- but somehow I think "truly annoying" is more satisfactorily expressed by higher digits (even into the millions!)).

So that a rating of 9/1 is outstanding. but a 9/4 is descriptive of gamey (re: unsatisfactory) situations.

This would eliminate all this decimal stuff which seems an attempt to parse these elements.

So you could have 4/2 as the baseline classic good QB.

How's that?

[Splash]

Perhaps the base should be the value of a QB. That being say 2 of 10. Now I can improve any QB by giving the player -30% so say that is a 4.

The problem is that this game cannot produce a 1.

The "gamey" factor is another consideration.

Gamey at it's 2 poles are:

A.) the truly annoying

B.) that's annoying I now have to learn something new.

Personally I think the scale should be reversed scaled out of 4-- in that 4 is item A.) and 1 is item B.)

(Altho' for simplicity sake they might be reversed- but somehow I think "truly annoying" is more satisfactorily expressed by higher digits (even into the millions!)).

So that a rating of 9/1 is outstanding. but a 9/4 is descriptive of gamey (re: unsatisfactory) situations.

This would eliminate all this decimal stuff which seems an attempt to parse these elements.

So you could have 4/2 as the baseline classic good QB.

How's that?

[Ligur]

By the way, I'd prefer a 1-5 points scoring.

[WWB]

So far we have five distinct replies to what is a damn good question (thanks Ben): just how are scenarios rated at the depot?

The real answer, as shown above, is everyone has his own definition. But, given that we have a nice database with normalizeable numeric scoring, we should probably be on the same page. Admiral Keth has given us some fine scoring parameters, but they are encased behind a 'review tips' window. It might well be easier to score a scenario if those could be displayed with the definitions. Possibly by popping a separate window, where the ratings definitions are displayed beside the score scroll down.

Also, it might make sense to update the scoring definitions, since they do not reflect some of the more updated features.

Or another option is to take this time to create succinct (1-2 sentence) definitions of each value to be displayed to the end user when they are updating the DB. Going to a five point system, as Ligur suggests, is a good idea IMHO, but probably very painful at least from a DB/web design point of view. We could, however, concatenate ranges of values, so as to effectively make it a five point scale with half scores.

But the bottom line is the definitions need to somehow be in one's face when one inputs his opinion.

WWB

[Wild Bill Wilder]

Thanks, CA. I really appreciate those thoughts. And you are perfetly right about the briefing. Some want little or nothing. Others want it all and then some.

The truth be known, I think the silent majority want enough information to know just what is supposed to be going on, what they just might face (leave a few surprises here - ), and how they got to this place.

In my thinking, the lack of some sort of background turns it simply into an arcade style-shoot-em-up. A little history to give one the feel of the situation is mandatory, at least for me.

I detest nothing more than a scenario with little or no background, nothing to put me "in the mood."

Now I don't need background for Wolfenstein, or Pac-Man. I do need it when I am about to enter a battle. Imagine a poor commander thrust out on a battlefield knowing nothing of the enemy, his own forces or what possibly might be out there! Poor guy!

Hard call, anytime. Maybe criteria for evaluating briefings might be in order.

1. Historical flavor

2. Grammar and Spelling

3. Interest level

4. Intelligence (infomration)

5. Size

Does it make me feel like I'm really going to war? Can I hear shells falling in my mind as I read it?

Does this person know how to spell? If he doesn't spell check his document or read it over to be sure it is right, what confidence do I have that he did not do the same in his design?

Does it make me want to play the battle? Does it pique my interest?

Does it tell me too little, too much? Do I feel that it is adequate for the task before me?

Is it too short, too long? Do I feel like I'm reading a book or a briefing?

Offering two points for each of those and you can come up with a decent evaluation of the briefing. If everyone would do that with mine, I could take the score a lot better. If they still gave it a "one," at least I would know that they evaluated it fairly on more than one point.

I'm off the pulpit, guys...WB

[SlowMotion]

I would also like the 1-5 point rating more than 1-10.

One thing about briefings that seems to be missing from this discussion so far, is that for non English speaking people writing really good briefings is much more difficult than for native speakers. And reading them probably works the same way. It doesn't set the mood as well if you don't know the meaning of each word

A good map and force balance can be created by anyone.

[SlowMotion]

I meant:

A good map and force balance can be created by people for whom English is a foreign language.

[Berlichtingen]

You may want to head over to the Scenario Depot and check out the forum. There has been a long discussion about changes to the scoring system. IIRC they include dropping replayablity from the final score (being expressed asa simple yes or no) and dropping the 'weight' of the briefing from the final score. That would mean that map, balance and playability would determine what the final score is. Drop by and post your thoughts

[Keith]

I think the briefing is an ESSENTIAL part of the gaming experience. This IS an historical wargame after all - not Mortal Kombat. Getting rid of the briefing score is like getting rid of the swimsuit competition in the Miss Universe contest. Just because some authors can't be bothered to put in the 1/2 hour effort into writing a briefing doesn't mean that the briefing score should be eliminated. The briefing sets the tone of the battle and provides essential info on reinforcements, force allocation, intelligence, etc.

As a reviewer, if you don't want to rate a category, just put a zero in. For myself, I always put a comment in my reviews on the briefing quality. The briefing might seem like window dressing to others but to history buffs like myself the briefing is an important part.

[Andreas]

<blockquote>quote:</font><hr>Originally posted by Keith:

I think the briefing is an ESSENTIAL part of the gaming experience. This IS an historical wargame after all - not Mortal Kombat. Getting rid of the briefing score is like getting rid of the swimsuit competition in the Miss Universe contest. Just because some authors can't be bothered to put in the 1/2 hour effort into writing a briefing doesn't mean that the briefing score should be eliminated. The briefing sets the tone of the battle and provides essential info on reinforcements, force allocation, intelligence, etc.

As a reviewer, if you don't want to rate a category, just put a zero in. For myself, I always put a comment in my reviews on the briefing quality. The briefing might seem like window dressing to others but to history buffs like myself the briefing is an important part.<hr></blockquote>

What he said, basically.

[Berlichtingen]

<blockquote>quote:</font><hr>Originally posted by Keith:

As a reviewer, if you don't want to rate a category, just put a zero in.<hr></blockquote>

I thought a zero only worked for the playability scores. Does it work for everything?

[WWB]

I do agree briefings are an essential part of the experience, and should still be numberically rated and counted overall. And lord knows that I can't write them.

What we really need is a rather general scale easily viewable while people rate scenarios, to keep them on the same page so far as ratings go.

I looked at the very interesting discussion over on the depot's BBS, lots of good ideas there. Probably the best one is to make replayability a yes/no category, as it really is not that sliding a scale.

WWB

[Keith]

I'll give an example of how the lack of a briefing can adversly affect the playing of a scenario. I just played a scenario with no briefing where for the first five turns I had nothing on the board but one platoon of infantry. It turns out this was a recon platoon and in the following turns roughly a battalion of reinforcements showed up. I had no clue this would happen not to mention the fact that the enemy actually outnumbered me for several turns. I would have handled things differently by initially attacking less aggressively if there had been a briefing.

[ 12-04-2001: Message edited by: Keith ]</p>

[SlowMotion]

Assuming this was a purely fictional scenario how long do you think the briefing should have been in order to get an OK rating? I mean: could a ten line briefing be as good as a 20 line one?

[SurlyBen]

Well, I'll give a briefing an OK rating if it tells me what I have, what my objectives are, and a rough idea of what I am expected to face. Unusual victory conditions should also be listed (ie: you only "really" win if Major Beery makes it to the tavern alive.) That takes, like, two sentences. And it's ok (encouraged, even) to lie about reinforcements and enemy strength.

Anything beyond that (history, suggested reading, a nice little story) only adds.

I'll mark down for spelling if it's terrible, but a typo or two is no big deal (it's not one of my pet peeves (unlike bumpy water) )