Jump to content

CMAK Imminent - can we fix the Scenario Depot Rating System Beforehand?


Recommended Posts

Originally posted by Michael Dorosh:

Any word on this?

I'm still waiting for _any_ kind of concensus. I see we have at least a dozen opinions, pretty much all numeric-based (which I believe I stated that it was my preference to not implement.

Once I wrap up building a shopping cart system for a paying client (within 2 to 3 weeks), I'll be turning my full attention to this task. In the mean time, I'd like to remind people of my set of criteria.

The rating system should be either text-based (Excellent, Good, Average, Poor, Abominable), or graphic-based (1 to 5 stars).

The rating system must non-cumulative, i.e., each review stands on its own merits and does not contribute to an overall rating. Therefore, any values associated with any aspect of the rating system _must_ be self-explanatory and understandable by any reviewer.

Authors should be deciding what aspects (map design, force balance, PBEMability) of a scenario they want rated, as well as the values (as above) for each of the aspects.

What should be done about historical reviews/ratings?

I also like WWB's idea on page 1 regarding the "I played this scenario...". That's definitely going in.

In the mean time, i'm continuing to read everyone's input...please continue to do so, but everyone needs to start agreeing on a final format.

Lastly, when this new review system is finished being implemented, authors and players must necessarily be able to accept whatever reviews are posted; good, bad, and indifferent reviews will need to stand as posted regardless of the perceived intent, intellect, or integrity of the reviewer.

Link to comment
Share on other sites

  • Replies 198
  • Created
  • Last Reply

Top Posters In This Topic

The rating system must non-cumulative, i.e., each review stands on its own merits and does not contribute to an overall rating. Therefore, any values associated with any aspect of the rating system _must_ be self-explanatory and understandable by any reviewer.
Admiral-

Does this mean there will be no lists?

Thanks.

[ January 12, 2004, 08:30 AM: Message edited by: Flammenwerfer ]

Link to comment
Share on other sites

1) Go with registration

2) Use the "Played As" checkboxes. Choices should be: Axis vs AI, Allies vs AI, Two-player blind, and Two-player NON-blind

3) Allow the reviewer to give only a letter grade (A,B,C,D,F) after checking his "reviewer's perspective" in #2 above.

Reviewers should be allowed to give a separate grade letter for each checkbox category which is treated as a totally separate review.

The Scenario Depot visitor would browse reviews by selecting the checkbox category he is interested in. If he wants to know how a scenario does in the blind two-player category, he gets that list of reviews, etc..

Link to comment
Share on other sites

Originally posted by Treeburst155:

1) Go with registration

2) Use the "Played As" checkboxes. Choices should be: Axis vs AI, Allies vs AI, Two-player blind, and Two-player NON-blind

3) Allow the reviewer to give only a letter grade (A,B,C,D,F) after checking his "reviewer's perspective" in #2 above.

Reviewers should be allowed to give a separate grade letter for each checkbox category which is treated as a totally separate review.

The Scenario Depot visitor would browse reviews by selecting the checkbox category he is interested in. If he wants to know how a scenario does in the blind two-player category, he gets that list of reviews, etc..

I agree with all of this. I think WWB balked at the idea of allowing someone to sit on the fence with a "C" grade, though, which is a good point. Is that a commonly held feeling?
Link to comment
Share on other sites

If a scenario sits on the fence, why not give a grade that reflects that fact? smile.gif The average scenario should get a C. Most scenarios should get a C. If a reviewer had an especially good time doing a blind PBEM of a scenario, he might want to give it an A or a B depending on just how memorable the experience was.

The grade should be based on the reviewer's enjoyment of the experience. Different reviewers are going to give different grades. The scenario shopper wades through all these opinions, taking special note of reviewers who tend to agree with him on what constitutes a good scenario.

BTW, I love the reviewer incentive idea where reviewer's reviews get rated as useful or not. This is great incentive for thoughtfully grading a scenario IMO. I doubt you'd get scenario shoppers to vote on the usefulness of reviews however.

Link to comment
Share on other sites

One thing that could be paid attention is giving some common guidelines for rating. For instance, how does one rate balance? I think it should be rated according to what the designer aimed to achieve and how well the outcome reaches that goal. If he tried to make a mirrored game but one side has an M4A2 and the other a Tiger, then the balance is not very well thought out. But if it's meant as a "David vs. Goliath" kind of situation where the Allied side has to come up with a plan of how to kill that Tiger, then the balance actually might be suitable for the scenario and thus be given a favourable rating. For a normal scenario, I think a perfect balance is such that makes the scenario most interesting to play, even if that means a bit of imbalance by conventional means (and no, I don't find mirrored games very interesting). Of course the designer has to express his intentions for the reviewers to know.

Link to comment
Share on other sites

Originally posted by xerxes:

I rely on what I know about author's to pick scenarios that I'll like. I don't care for some designer's style, others I tend to like very much.

In other words, you are biased. I like Treeburst's anonymity idea very much.

If you really do think you have a designer's style pegged, then it would be obvious during gameplay (or simply reading the briefing :D ); unless you are really referring to personality conflicts, in which case, you're really not judging the scenario fairly.

Not that we don't all have personality issues from time to time, I don't mean to sound accusatory.

But I think designers are too ego-driven to ever submit willingly to anonymity...except for the really good ones, of course... :cool:

Link to comment
Share on other sites

The scenario grade should be assigned by the reviewer based on the "fun factor". I think most people would consider a highly unbalanced scenario to be not much fun most of the time.

Balance in CM is virtually impossible to achieve. The scenario need only be balanced enough to be enjoyable for both sides. That is all that can be expected, and all that is really needed.

Anonymous scenarios......yeah, wouldn't that be fun.

Link to comment
Share on other sites

Originally posted by Treeburst155:

Anonymous scenarios......yeah, wouldn't that be fun.

Maybe CMX2 needs a toggle in the scenario briefings; instead of just a text file, you have to fill out a form in the editor itself, and then when you go to play it, you can toggle "don't show historical aftermath" or "don't show designer's name" on and off, or "only show after conclusion of scenario."

That would be kinda neat.

Then the SD could have the same toggle - show designers, don't show designers.

Link to comment
Share on other sites

Word would get around on who designed what. Only the designers themselves could really protect their anonymity. This would mean hotseat playtesting by themselves, or with a select few highly trusted playtesters.

With anonymity, new designers would be on equal footing with the established pros. They would have a better chance of getting their scenarios played and reviewed.

Link to comment
Share on other sites

No, I'm not biased, I'm not sure how you would come to that conclusion. If you'd played two scenarios by a designer that had fundamentally the same design flaw (or feature) the odds are that a third scenario would exhibit the same design flaw. It's simply statistics. From my experience a number of scenario designers have certain tendancies that manifest themselves in their designs. Not really all that surprising.

Link to comment
Share on other sites

Originally posted by Treeburst155:

The scenario grade should be assigned by the reviewer based on the "fun factor". I think most people would consider a highly unbalanced scenario to be not much fun most of the time.

Balance in CM is virtually impossible to achieve. The scenario need only be balanced enough to be enjoyable for both sides. That is all that can be expected, and all that is really needed.

Then why not call it something else? But the same goes with other features. If you're rating map, then what are you rating? How pretty it is, or how historic you think it is? As a Finn I might come across a scenario set in Finland but in which the terrain looks more like Ukraine (there wouldn't be anything bad about the map by itself). Should I give the map a poor rating? Or maybe I should just try to think how well the map supports the scenario idea. Without any guidelines, I can't tell, and if reviewers don't know what they really are rating, then those ratings can't be given real value as they are random.
Link to comment
Share on other sites

Originally posted by xerxes:

No, I'm not biased, I'm not sure how you would come to that conclusion.

By your very words! LOL! :D Unless you don't know what biased means?

If you'd played two scenarios by a designer that had fundamentally the same design flaw (or feature) the odds are that a third scenario would exhibit the same design flaw. It's simply statistics. From my experience a number of scenario designers have certain tendancies that manifest themselves in their designs. Not really all that surprising.
Can you give a for instance? I mean, I don't like armour-heavy scenarios or monster scenarios - but I can generally weed these out by reading the descriptions first. I'm not sure I agree with your assertion. I can look at the flaws in my own work - and I like to think I learn from mistakes and negative feedback and try and mix up the types of scenarios I design - some small, some big, some armour heavy, some infantry, some probes, some MEs etc.

I can give an example of a designer I like; Andreas - and I've played scenarios of his that were small, some were medium, some were purely infantry, some had armour. I haven't noticed any tendencies, whatever is meant by that. On the other hand, he's always strived to be innovative. The immobilized Russian tank in one scenario, for example, or the trenches in the church in another. I must admit, I'm a big fan of innovation like that - stuff you don't expect. But I have always played more for the experience than for the competition.

I also admit I'm biased in his favour; I enjoy his posts on the forum tremendously, and if I ever thought a scenario of his truly sucked, I would never say it in a review. Especially since I perceived a bit of (understandable) upset on his part at the way some of his scenarios have been reviewed or commented on in the past. I suppose I'm sensitive to his feelings far more than I would be if he were anonymous.

So I am certainly guilty of bias myself. It's not an accusation, it's a fact. And I think scenario designers tend to be more biased when reviewing other designers than those who don't design at all. Actually, I think you prove my point. ;)

Those that aspire to know the ins and outs of scenario design are naturally harsher critics, whether wittingly or not.

Link to comment
Share on other sites

Sergei,

Every reviewer has his own likes and dislikes when it comes to scenarios. They will grade a scenario differently for different reasons. Still, given enough reviews, a picture will begin to develop. The key to scenario shopping will be this big picture, and also how your favorite reviewers scored the scenarios. What makes a favorite reviewer? Someone who grades scenarios as you would, and therefore likes the same type of scenarios you do.

Link to comment
Share on other sites

But then we get to another issue. A scenario that is given a bad rating, for whatever reason, won't be played as much, and the likelihood of it getting more reviews is weakened. Except of course if the first review is ridiculously low, in which case suspicions arise. It works better if the opening review is praising, then others will be interested to try it out, and eventually people will give reviews, especially if the original one was misleading.

Having "favourite reviewers" can be handy for a designer IF he has designed a lot of scenarios, which is necessary for this kind of relationship to develop. But what about beginners? Don't they need just as much, if not more, of good feedback? And what about players who are using the same system to find good catches?

To another topic, what would be the aspects in scenarios that we would be rating? I don't remember there was any conclusion to that, except that 'replayability' is useless.

Link to comment
Share on other sites

Sergei,

"Favorite Reviewers" would be something scenario shoppers would have to help them pick scenarios.

Rating criteria? How about "FUN"? This will mean different things to different people; but that's OK.

To get noticed, unknown designers not affiliated with a design group need to blow their own horns loudly. Simply uploading a scenario to The Depot with no fanfare is almost guaranteed to result in few downloads and few reviews for the unknown designer. If you've created something you really believe is worth playing, announce it to the forum. Be proud of it! The shy unknown will always be unknown. There are just too many scenarios out there for it to be any other way.

Berlichtingen,

I would like to see a fairly high percentage of "C" scenarios. This allows the Good, Great, Bad, and Terrible ones to stick out like a sore thumb. This would not keep me from playing a "C" scenario, especially if my "favorite reviewers" were giving it higher marks.

[ January 11, 2004, 01:20 AM: Message edited by: Treeburst155 ]

Link to comment
Share on other sites

Originally posted by Treeburst155:

Rating criteria? How about "FUN"? This will mean different things to different people; but that's OK.

Okay. But shouldn't that then be made clear to everyone? E.g. substitute "balance" with "fun", because otherwise people are going to be confused. Of course, is that then necessary as a separate rating from playability vs. Ai and human? Also, 'fun' is very subjective, often winning an uneven game can be fun even if the opponent wouldn't agree.

Basically I'm against anything that is easy to misinterprete, because it also makes rating scenarios harder as you have to think what the "playability", "balance" or "replayability" actually mean. Just a while ago I was reviewing a scenario but had to leave it without ratings as I wasn't sure about the meaning of those.

Link to comment
Share on other sites

Sergei,

I think the review grade should simply be the reviewer's overall opinion of the scenario based on his point of view (the "Played As" checkboxes). Nothing more complicated than that. The reviewer simply grades his enjoyment of the experience. The review process should be kept very simple to promote more reviews. Asking players to rate a dozen scenario aspects is asking too much of most people I think.

The whole thing hinges on getting lots of reviews. If this happens, you will know a bad scenario, and a good one, without having to play it first.

I might give a scenario a "D" because it is too short. Someone else might give it a "B" for the same reason. What I see as "too short", they see as a "frantic, exciting scramble". This is all fine. With enough opinions players will be able to judge scenarios they will likely enjoy, which is the whole point.

Determining reviewers who enjoy what you enjoy will be necessary.

[ January 11, 2004, 02:27 AM: Message edited by: Treeburst155 ]

Link to comment
Share on other sites

Perhaps people who register at The Scenario Depot should be asked to briefly supply in a profile what their likes and dislikes are concerning scenarios. This info would then be available to scenario shoppers.

My "Reviewer Profile" would be something like this:

I don't like running out of time with a significant number of combat capable troops still on the map.

I don't like flags near map edges or in the middle of open terrain.

I think the luck factor in CM is more than enough without adding planes or low percentage reinforcement arrivals.

I don't like tiny scenarios.

These profiles would help shoppers determine who their favorite reviewers are.

[ January 11, 2004, 02:44 AM: Message edited by: Treeburst155 ]

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Unfortunately, your content contains terms that we do not allow. Please edit your content to remove the highlighted words below.
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


×
×
  • Create New...