Epiphany regarding the Scenario Depot

[Michael Dorosh]

It is a common scene - a scenario designer states explicitly at the Scenario Depot that a scenario is intended solely for play vs. a human opponent, when along comes a reviewer (perhaps well-intentioned, perhaps just addled), tries it versus the AI, loses, doesn't bother to play it again to see if it was simply luck, then procedes to rate the game a 3.0 as "vs. AI" and then posting a pronounciamento alluding to the suckiness of said scenario.

Scenario (possibly well crafted and balanced for PBEM) slides to bottom of ratings and interest in it is lost.

I've been dissatisfied with the rating system for a while, though the site itself is still a terrific tool, and most reviewers do a good job of rating them.

But consider this

how about letting the scenario submitter block off certain categories?

Ie if the scenario is not intended for AI play, how about giving him the option of not allowing any reviewers to assign a grade for that?

If you wanted to reflect the number of allowable categories, you could perhaps indicate this by a number in brackets after the total average score

ie 8.60 (4) indicating a score of 8.60 calculated from four allowable categories?

Just a random late night thought....probably posted in the wrong place, but rather than just emailing the Admiral, I thought I would open it up here.

Some other scenario designers have been talking to me privately and expressing some concerns about the Depot and the hazards of putting scenarios up for review in an environment where there is potential for unfair criticism due to technical concerns - I would classify the rating of a PBEM scenario with regards to AI play as one of these concerns.

It's like my review of one scenario at the depot - I was set to critique it with regards to the AI's ability to set up the defender and realized that it was very possible that the scenario designer had specifically mentioned "stick to scenario default", which I never do. As a result I almost rated it down because the AI set up two bunkers way back on the map (incidentally, a note to other designers - I see that defensive heavy weapons like bunkers and AT guns very often do get set up way at the back of the map by the AI - very mystifying and something one needs to watch for) but had the scenario actually specifically said not to do that...it would be most unfair.

In any event, I love the Scenario Depot and would love to see more people interacting there. A return to allowing authors to respond to reviews would be a great start. A renewed discussion of how to rate the scenarios would be another; I still don't think the current rating system is flexible or far reaching enough. But I have few ideas on improvements. One that I do like comes from a game site's review of CMBB (or was it GIC?) - a TILT option. One more category with which you can skew the final average as the reviewer sees fit.

Let's take the example above again. It plays poorly against the AI because of the set up bug. But the map is great and the forces pretty good, and the location of the AT guns really isn't a factor. So I rate vs. AI playability as a 4, but hey, the rest of it was so good - I give the TILT a 9 so the average at the end is a bit higher.

[ November 22, 2002, 02:37 AM: Message edited by: Michael Dorosh ]

[Thin Red Line]

I fully subscribe to the above.

If only scenarios had more reviews, it would minimize the effect of 'weird' or faulty ratings.

[Panzer Leader]

Why don't you use "stick to scenario default" for computer forces? The designer often puts a lot of thought into setting up positions for the enemy to maximize the player's struggle.

[Bruno Weiss]

Epiphany! Now lemme see here, I heard of that afore. Epiphany, epiphany, hey that's up New York state ain't it.

[xerxes]

I think blocking out invalid dimensions would be great. Particularly the "vs ai" or "pbem".

I also would rather see an average of actual victory levels rather than a balance rating. Balance is really almost impossible to tell from a single playing. I realize now that a scenario I rated as unbalanced is actually quite balanced. My game was terribly lopsided, but after seeing the results of a number of games, the scenario is balanced.

I'd also like to be able to look at all the ratings from a single reviewer. If I know someone who has my "taste" I'm much more interested in which scenarios that player likes.

[tracer]

Well I try to rate all the scenarios I get from the depot, And I rated one called ASL Hube's Pocket" and yes, it said "best played 2 player" and it also said play second as "axis vs. AI" I did the second, and I rated it according to what I thought, if it hurt some ones feelings, well then don't put it at the depot because it will get rated because there is a rating system, and be it that the rating system sucks to some or doesn't to others, I do it out of respect of getting to go to the Depot to get "FREE" battles and Ops,if people don't want a rating done on thier scenario being played as human Vs, AI, I suggest that they not put it in the briefing. I try to rate honestly, and not try to kiss butt being that I don't need to (I don't need a job,not by any means}I'm set just fine. So I guess nuff said.

[Admiral Keth]

Michael,

An excellent suggestion regarding the authorial disablement of specific non-applicable rating categories. Let me do some investigation as to the degree of difficulty to implement this. If I get distracted and you don't hear back from me regarding this topic after a couple of weeks, send me a mnemonic device. As for the Author feedback - it's been on my To Do list for a while, along with several other tasks. It's 85% complete, I just need to finish testing and put it into place.

My day job has been remarkably brutal of late (10 to 12 hours+ per day), thus sapping energy which would otherwise be spent adding features to The Scenario Depot. I also have several web-based clients whom I am attempting to wean off by the end of this year.

All of this means that by the beginning of 2003, I will be able to dedicate a significant amount of time to enhancing The Scenario Depot. The next month or so will find me tying up loose ends, but after that a large portion of my off hours will be consumed behind the keyboard.

I am absolutely open to any and all suggestions from the authors and players on specific improvements to The Scenario Depot, the rating system, the feedback system, or any other aspect.

[ November 22, 2002, 10:11 AM: Message edited by: Admiral Keth ]

[Admiral Keth]

With regards to TILT...

1) Is TILT relative to a single rating, or applied to the Overall Rating?

2) Is TILT applied by the author to a specific review, or by the reviewer?

[Panzer Leader]

Oooh! I've got a suggestion!

How about an auto-upload feature for scenarios/maps/operations and screenshots? (Like CM Mod Database has - very easy to use!)

Seems like then you wouldN'T have to hand enter every one (If I am understanding how it works properly)

[EDITED]

[ November 22, 2002, 10:37 AM: Message edited by: Panzer Leader ]

[Ken Talley]

What I have noticed is that each reviewer seems to have different criteria on how to rate a scenario. For example, some will take an average scenario and use 5 as representing average. Others will take the same average scenario and use a 7 or 8 as average. The numbers aren't really tied to average or excellent or anything beyond what the reviewer wishes to use.

I think it would help if there was a brief guide describing how the numbers should be used to ensure coherent results. If this guide were at the top of the review form, everyone would start from the same point.

I sometimes think a poor, average, good, excellent system might work better than a pure number system.

[Admiral Keth]

Originally posted by Ken Talley:

What I have noticed is that each reviewer seems to have different criteria on how to rate a scenario. For example, some will take an average scenario and use 5 as representing average. Others will take the same average scenario and use a 7 or 8 as average. The numbers aren't really tied to average or excellent or anything beyond what the reviewer wishes to use.

I think it would help if there was a brief guide describing how the numbers should be used to ensure coherent results. If this guide were at the top of the review form, everyone would start from the same point.

I sometimes think a poor, average, good, excellent system might work better than a pure number system.

Ken,

Your statement is fairly true. Most people have a highly subjective view of each of the parameters. Therefore any rating system has to be taken with a grain of salt.

There is a guide which was meant to give some direction to reviewers, but it seems as if everyone has missed it:

Reviewers should click on the "Review Tips" button for guidelines on what the various ratings mean. While this page may not be up to date, it still gives some idea on what to do.

A short while back I requested a volunteer to write a "How to Review" guide based on author suggestions and feedback, and alas I received no takers. The job is still open for anyone willing to help out a beleagered developer.

In regards to changing the rating system from numerical to textual...that might be significantly more painful than creating a comprehensive guide upon which everyone agrees to utilize.

[Le Tondu]

Originally posted by Panzer Leader:

Why don't you use "stick to scenario default" for computer forces? The designer often puts a lot of thought into setting up positions for the enemy to maximize the player's struggle.

What for? If I had stuck to the scenario default the first time I played Jagermeister, I would've had my butt handed to me. Instead, I handed the AI it's collective butt.

I guess it all boils down to why we wargame. Do we wargame to experience more struggle? I can get that at the local grocery store or driving across town. No, I wargame to have fun -plain and simple. That's not to say that I'm against having an interesting challenge and I do sometimes use the scenario default.

It's just that so many scenario designers just lay out the forces in some weird way knowing that the player will set them up as he sees fit. Say, isn't that what real commanders go after?

As for rating a scenario in ways that it shouldn't be rated, I'm against that as well. It is only a mistake when someone rates a scenario for play against the AI when it is meant for only play against a human. It's rather dumb too.

[Eden Smallwood]

Originally posted by Panzer Leader:

Why don't you use "stick to scenario default" for computer forces? The designer often puts a lot of thought into setting up positions for the enemy to maximize the player's struggle.

And the AI's pillboxes and stuff won't be on the backmost pixel, at a minimum!

Also, something which is important to me, but not everyone apparently- I don't like being in contact on turn one. I've found some scenarios will do that if I let the AI place units, but not in the default setup. So for all these reasons a scenario should be played with default setup if one is going to review it.

Otherwise, a player's opinion of the Balance doesn't reflect what the designer put into it, at least. As discussed here, the whole rating thing is problematic enough already, without adding complications.

Eden

[Abbott]

So for all these reasons a scenario should be played with default setup if one is going to review it.

Unless it is stated "computer placement".

[Ben Galanti]

Originally posted by Le Tondu:

It's just that so many scenario designers just lay out the forces in some weird way knowing that the player will set them up as he sees fit. Say, isn't that what real commanders go after?

I think sticking to scenario default is only for the AI side. The player always has the option to move his forces. If you turn this option on, the AI will always set up in the same palce (the place were the scenario designer put them), if you have it off, the AI will move it's forces as it sees fit during the setup phase. This will give you a different fight each time, but human designed placement will generally be better.

I'm pretty sure this doesn't affect human players at all though.

Ben

[Michael Dorosh]

Originally posted by Admiral Keth:

With regards to TILT...

1) Is TILT relative to a single rating, or applied to the Overall Rating?

2) Is TILT applied by the author to a specific review, or by the reviewer?

I swear I responded to this...

anyway, thanks Admiral for listening, and sorry about real life getting in the way - my untended websites suffer from the same problem; I do note with gratitude the flow of new scenario download links is constant, and that's a good thing!

Tilt is applied by the reviewer, and is applied to the overall rating - think of it as another category that goes to the total score. The reviewer uses it to balance out the negative or positive comments he feels will unbalance his review. ie - imagine a scenario has a great map, terrific briefings, but has such an inbalance of forces that it is unplayable. Once you score the scenario a 9 for map, 10 for briefings, and say a 4 for force balance, you still have an average of 7.66. So you apply a tilt of 2, and get an overall score of 6.25

Not the most elegant suggestion in the world, but a thought. Great map+great briefings+poor balance don't necessarily make for a 7 or 8 overall scenario, but given the equal weights of the various categories, that is how it sometimes works out.

Of course, you could also simply let the reviewers (or the designers...?) apply weight as they see fit - ie rate a map as worth 5 percent of the overall score, briefings as 10, gameplay 80 and replayability as 10 - and let everyone assign their own weights. That has its pitfalls too - but given the example of replayability, say - I have a scenario up at the depot that has 120 turns in it. It is designed primarily to be experienced once - and if anyone wants to play through all 120+ turns more than once, more power to them, but I don't think replayability will be as big a deal as say a clear briefing outlining the objectives of the scenario. As the designer, I would like to then assign replayability, say, a 0% weight on the overall score, since it wasn't designed with that in mind.

I'd like to see some other categories too, like innovation - if a scenario is trying to be innovative (say it has "house rules" to determine victory rather than using the flag/exit/casualty system built in), or has a terrific map like Clubfoot's Pointe du Hoc that let you climb the cliffs - I would rate that as highly innovative.

By contrast, yet another company vs company meeting engagement on a computer generated farmland map isn't very innovative- but if it isn't the designers intent, why rate him down for it? I'd give him the option of turning off the "innovative" category since it wasn't the intent, or of weighting it to 0 percent. On the other hand, maybe this company vs company battle on the "boring" map is perfectly balanced and thus infinitely replayable. This should get more weight in the review.

Etc....

[Franko]

I could not agree with you more: the ratings system at the Scenario Depot needs an overhaul. First, the 1-10 system allows for too much variance: only one numb-nut has to give a review a "1", and then the scenario gets unfairly slammed.

One guy reviewed my August Bank Holiday scenario, and gave it a poor review because the map was too bare: even though I worked off of several topographical maps, one hand-drawn map, and actually WALKED the battlefield! Talk about unfair.

If the Depot doesnt change its system, the people in charge should at least EDIT the reviews: this would be responsible. If a scenario is NOT INTENDED to be balanced, then it should not be REVIEWED for balance. If it is NOT INTENDED to be a 2-player game, it should not be REVIEWED as a two-player game. For example, many of my scenarios are purely "historical", and I do not give a hoot about balance. Furthermore, real battles are rarely "balanced". So long as the designer is clear about his intentions, he should not be exposed to a bad (or good) review in such matters. To be fair, 'balance' is a separate discussion full of pitfalls, which I would rather not get into.

I propose a simple five star system (or something to that effect), which rates scenarios as Excellent, Good, Average, Fair, and Poor in various categories. There should be one more "rating", i.e., NOT APPLICABLE (which would not relect on the overall score.) Such categories could be the following (with comments):

MAP -- Overall attention to detail, topographical realism, etc. Does it look like real ground? Or is a jumbled-mish mash of terrain features?

FUN -- Did you have fun playing it? Were you afraid you were gonna lose? Did you have interesting tactical problems to solve? Did it give you a "historical feel?" for the time and place of the battle?

HISTORICAL -- This could be hotly debated. Basically, its real difficult to get accurate orders of battle on a given day (or a given hour) of a battle. Glaring errors (e.g., King Tigers in 1941) should be penalized. If there's snow in the Kuban in August, rate it "Poor". If people do not feel qualified to answer, they should put "N/A".

ORDER OF BATTLE/Unit Editor: This is probably a good one. Did the designer pay attention to things such as troop quality, ammo levels, etc?

REPLAY VALUE: I think this is a goofy category. Really good single-player scenarios are probably ruined after one sitting (you wont be fooled again, for example). And its just CHEATING to play twice two-player, unless both players have played the scenario before.

OVERALL RATING: This should not be an "average" of all the other scores, because this would imply that all categories are equal (I mean, who cares if T-34 M43's were "largely" unavailable in December 1942, or whatever). I think "FUN" would be worth more, for example.

In fact, I would dump all the categories in favor of one overall rating, with a place for the reviewer's comments. However, I would encourage everyone to be mindful of the designers wishes: If he does not intend to be historical, dont punish him if he isnt. If he intends to be, hold him to that standard.

A word about "historical" scenarios: I define "historical" scenario as one that attempts to, as much as reasonably practicable, to simulate a historical event. In other words, a good faith effort has been made to develop an accurate map based on historical texts and/or maps; an accurate order of battle (without a wooden reliance on TOE's), and historical parameters (e.g., weather, time of day) based on reliable historical sources. I think this is more difficult in CMBB than CMBO (blame the commie propagandists and the self-serving, sore-loser German officer corps), so I've "loosened" the definition a bit for my CMBB scenarios.

In sum, I strongly recommend we go to a simpler rating system, with more aggressive editing on the part of those who "host" these scenario review sites. I like the Scenario Depot, but there is nothing so disheartening for a scenario designer (who, after all, does this out of passion, not out of pay) to get slammed in an unfair way by some dumbass who doesnt know where we are coming from.

Of course, its then OUR fault if we dont tell the reviewer WHERE we are coming from. I would encourage all designers to put "designers notes" in their briefings to explain their rationale for various design elements (while carefully trying to avoid spoilers). I think they are interesting to the player, and would help the reviewer in coming up with a more-informed review.

Rant finished.

Franko Out.

[Panzer Leader]

Could I jump in for just a second to ask a quick question? Thanks. Where can I host my QB Maps? I'd like to put them at Scenario Depot, but I need to have the hosting link, could someone clear this up for me?

[Admiral Keth]

Originally posted by Panzer Leader:

Could I jump in for just a second to ask a quick question? Thanks. Where can I host my QB Maps? I'd like to put them at Scenario Depot, but I need to have the hosting link, could someone clear this up for me?

PanzerLeader,

I can accomodate hosting for you. My web server has 2GB of free space left. Submit the map parameters to the database, then zip up the maps and email them to me. I'd be happy to post and host them

[Panzer Leader]

Thanks Keth, emailing you a batch right now!

[Lawyer]

Originally posted by Franko:

For example, many of my scenarios are purely "historical", and I do not give a hoot about balance. Furthermore, real battles are rarely "balanced". So long as the designer is clear about his intentions, he should not be exposed to a bad (or good) review in such matters. To be fair, 'balance' is a separate discussion full of pitfalls, which I would rather not get into.

Franko Out.

I generally agree with your post, Franko. Your scenario grading system makes sense to me.

But your point on historical scenarios deserves some further comment, so I'm going to start a separate forum thread. And blame it on you, of course.

BTW, I've played of few of your scenarios. They made me realize the value of a good map for enjoying the game, so now I only play maps that look authentic. I know when I choose a Franko scenario that the map will be authentic and the historical research will be accurate. Thank you.

"August Bank Holiday" was simply too large for me to enjoy. It took forever to finish by PBEM, and facing the many moves in each turn when I came home from work became burdensome. I was glad to throw in the towel by the end. I'm not criticizing the scenario itself, but it did make me realize that I don't enjoy such big battles. For all the big size, it still came down to some AI shootout decisions whose overall effect on the outcome could have happened in a smaller scale scenario. In other words, the large size didn't add any meaningful tactical options that I could see.

In contrast, "Augen Zu" was quite good for me. Interesting map and objectives. Better tactical options. As the Germans, I lost at the wire by PBEM due to a couple of AI decisions that went against me. But it was a nail biting and manageable scenario.

After you advised me by email that "Parkers Crossroads" could not be "won" by the US forces, I decided not to play because I couldn't see the point of spending a lot of time playing an "unbalanced" scenario where I had no chance to be a victor. Later, I visited the real Parkers Crossroads on vacation in Belgium twice in 2001 and 2002 for a detailed look. Your map was very authentic, particularly considering the limitations of the map editor in CMBO. Excellent mapping.

Keep up the good work. So what scenarios do you suggest for CMBB that will give each side a fair shot at victory?