CMAK Imminent - can we fix the Scenario Depot Rating System Beforehand?

[Michael Dorosh]

I am at work and should really email this to Adm Keth, but thought some public discussion would be in order anyhow.

One of my CMBB scenarios got rated recently; most reviewers gave it a 7 to 8 rating; I just had one between 5 and 6. With little to no explanation of why the ratings are given. (Someone else had rated the same one between 4 and 5, with no explanation either).

Now, I am inclined to just dismiss reviews with no explanation out of hand, but that does me no good as a designer.

The reviewer did email me and told me he thought it was an average scenario and thus rated it accordingly. In his scheme, he never rates higher than an 8 for anything.

I think he highlighted the current problems, discussed here often enough.

Perhaps we can put a new system in place before CMAK comes out - start fresh? It would be impossible to retrofit the CMBB and CMBO reviews, but let's not even think about that for now.

My suggestion would be to make the different marks consistent with an established standard. Currently, I mark 10s for categories I thought fulfilled my own personal criterion. A 9 is a scenario that just missed the mark. Really awful ones get 1, 2 or 3 - but with no rhyme or reason attached.

Here is my suggestion vis a vis categories and numerical ratings

First of all - the person posting the scenario gets the option of disabling some, none or all of the following criteria so mouth breathing morons don't rate him down on "historical accuracy" and "playable vs AI" when he intended it as a fictional encounter for PBEM.

All categories would have a 0 rating as current, if the reviewer did not want to rate on that aspect.

BRIEFING (GENERAL)

1 - Briefing absent

2- Bare bones briefing provided, some necessary information missing

3 - Briefing provides all NECESSARY information to play the scenario (note, some briefings may leave out info by design, this should be considered)

4 - Briefing goes beyond functional and sets the stage well for the player

BRIEFING (Execution)

1 - Briefing absent

2 - Briefing indecipherable, contains so many grammatical, puncutation, or spelling errors, or formatting problems, that the briefing is useless.

3 - Minor spelling mistakes, grammar and punctuation errors, but don't detract from reading

4 - Perfect spelling, punctuation and grammar.

MAP (Execution)

1 - Major errors; bridges don't match up to roads or end in dead ends, reinforcements arrive behind impassable terrain, no passable route for units marked for exit

2 - Minor errors; one or two minor glitches that effect game play (routes for exiting units)

3 - Minor errors that don't effect game play or no errors, but plain and functional

4 - No errors, exceptionally good looking map

MAP/TIME LIMIT (Gameplay)

1 - map too large or too small for forces involved and terrain not suited to force makeup (unless this is an historical choice) - for example deep snow used but player only has 20 turns to traverse a 2000 metre map for exit - drastic changes needed

2 - map physically adequate for forces involved or time limit not adequate - major changes needed

3 - map and time limit barely adequate for forces involved; minor tweaks needed (for example, widening the map by 100 metres or adding 5 turns)

4 - map and time limit adequate for player to have a creditable chance of victory

FORCE MIX (Game Play)

1 - at least one side is completely missing equipment required to complete the scenario successfully (ie no vehicles are provided to transport them off an exit scenario, despite turn limit requiring it), no assault boats are provided in a river crossing scenario, designer forgot to give German infantry panzerfausts or AT weapons when clearly they need it

2 - all equipment is present but quality has been poorly thought out; US airborne troops are uniformly Conscript, for example; starving men in a besieged town are rated as FIT, ammunition levels are low (without a historical reason stated in the breifing) and units run out before half the game is over

3 - each player has the general forces required to complete his victory conditions successfully, though minor tweaks might make for a better game (ie one more anti-tank gun would provide a more even chance of victory for one side)

4 - Force mix well suited to the scenario in question

FORCE MIX (Historical Accuracy)

1 - Units used outside of historical context or timeline, hodgepodge of forces thrown together with no reference to historical employment

2 - Units generally correct, some interesting and ahistorical variations (unless designer admits to same in briefing)

3 - Units historically accurate, or only one side has well researched units as for 4

4 - Designer has gone extra mile and provided names for units, or researched in detail both sides of the OOB and provided sources

BALANCE (remember, the designer may not want to be going for this, if he does enable this rating, then assume he wants this considered - also, if designed for play against the AI, the "player" should be considered as only the player playing the side against the AI that the designer intended)

1 - Player has no possibility of winning this scenario in any circumstances

2 - Player has only a marginal chance of winning this scenario and will rely on luck

3 - Player can achieve victory given the resources at hand, but will require some luck to do it

4 - Scenario can be won by either side in normal circumstances demonstrating skill

VICTORY CONDITIONS

1 - victory conditions did not match designers intent (for example designer wanted to have the attacker capture the hill regardless of casualties, but provied only 1 small 100 point victory flag)

2 - victory conditions not well matched to the scenario (for example player was required to exit at least half his force but few of his units were marked for exit

3 - victory conditions adequate to the scenario

4 - scenario designer has gone extra mile in providing elegant victory conditions that truly influenced game play and reflected the intent of the designer

CHALLENGE

1 - scenario required no work at all for one side, just hit GO and watch the action

2 - scenario required minimal planning on the part of the player; no alternate routes for attackers, no possibility of a mobile defence for defenders

3 - scenario provided the player with adequate opportunity to influence the outcome of the game, or provided one side (only) with many challenges as in 4

4 - scenario provided both sides with many challenges and tough decisions to make

DESIGN FOR AI - HUMAN PLAYER AS ALLIES (only if recommended as such)

1 - design does not allow for intelligent AI play at all; no flags to guide AI attacks, poor placement of enemy units beforehand, set up zones inadequate for the AI player

2 - AI set up and guided, but scenario still does not play well against the AI; forces perhaps not correctly proportioned

3 - AI performs adequately in attack or defence, but minor tweaks could be performed

4 - Elegant design incorporates weaknesses of AI and adjusts force size, victory flags, exit zones, etc. to compensate for them

DESIGN FOR AI - HUMAN PLAYER AS AXIS

Identical to above; some games will be designed only for one side as AI - the designer can disable one set in that case and make sure he isn't rated down for it

"Playable via PBEM" is kind of irrelevant and tied to things like balance, game length - these factors are either stated up front or reviewed above - players can make their own choice from that as to whether or not it is going to work vs the AI or via PBEM. I think the depot should assume any scenario is to be played vs another human via either PBEM, TCP or Hotseat, and treat vs. AI play as a special case.

Ditto replayability - this will be made obvious through other factors and is kind of subjective - many players refuse to replay games, others insist on switching sides and trying again.

You then get an overall 4 point scale, as an average of the above ratings.

4.0 - A

3.7 A minus

3.3 B plus

3.0 B

2.7 B minus

2.3 C plus

2.0 C

1.7 C minus

1.3 D plus

1.0 D

Below 1.0 - Not possible

I think this would be far preferable to the system currently in place, and getting it online now prevents having to redo things once CMAK is in everyone's hands.

If we all have the same criteria in mind, the system will work much better.

It also helps fulfill the second reason for the reviews - ie useful feedback for creators. The reviewer will be forced to sit and actually think about why he is giving the ratings, rather than being subjective, which is precisely what is happening now.

Subjectiveness should be left to individual players, the review process should simply identify which scenarios have been prepared for specific types of gameplay, and how well these preparations have been made.

I've been guilty of subjective reviews myself, but have also caught myself on many occasions - ie I commented on one operation that I hate night scenarios, but realized I couldn't rate the operation down simply because it included one. I did, however, rate it down because the briefing excluded mention of it, something I thought essential to gameplay.

So stuff like "I hate Stalingrad scenarios" would become irrelevant, and popularity contests as well as shilling (if I may use that word incorrectly here) for friends would hopefully be diminished.

Given specific criteria for numerical ratings, the only danger is that reviewers would feel less of a need to justify their decisions via comments - I would suggest comment fields under each and every category above, with a general comment field for the very end of the review.

Perhaps as a final part of the review, players could decide what type of game it is best suited for - the first thing I look at is size; I think many players prefer certain size games to others - some like monsters, I prefer company sized. Perhaps, without assigning a numerical value, the following could be added also

GAME TYPE (for example)

a) Beer and Pretzels - easily played in a short time; small map, few units, few complicated choices to make. Probably decided early in the game due to nature of forces

Chess game - small but involving more complicated choices, thought and planning - may also require detailed movement orders, etc. In other words, small, but much attention to detail needed. Most likely a nail biter with victory determined late in the game.

c) Battalion Action - medium-large game, will take longer to play de to size of map and forces

d) Monster Scenario - will require a very large investment of time on the players' part due to not just size of forces but the amount of planning and detail required on the part of players

Of course, reviewers may find several descriptions adequate - for example a Monster Scenario could also be a Nail Biter....

[ November 24, 2003, 05:09 PM: Message edited by: Michael Dorosh ]

[GreenAsJade]

Two points come to mind:

1) The current rating system works well for

me as a customer of scenarios. When I want

something that will definitely be good, I only

choose high rating ones, and I have not yet

been disappointed unduly.

When looking for a good one, I choose

sceanrios with ample number of reviews,

pretty decent average score, then I _read_

the reviews.

2) The system proposed looks like an improvement

over something that is already OK.

I only feel a little perturbed at loosing

the out-of-10 scale. Would it be OK to use

the descriptions for 1, 4, 6, 10 and let

people interpolate if they want to?

GaJ.

[Admiral Keth]

All,

The recently-acquired company for whom I work is gearing up for a major changeover to their new identity on 11.30.03 (yep, I get to work on Sunday). I'm pretty much wall-to-wall with regards to time until then. I'll respond to any points made here in detail after this weekend.

[Michael Dorosh]

Originally posted by GreenAsJade:

Two points come to mind:

1) The current rating system works well for

me as a customer of scenarios. When I want

something that will definitely be good, I only

choose high rating ones, and I have not yet

been disappointed unduly.

When looking for a good one, I choose

sceanrios with ample number of reviews,

pretty decent average score, then I _read_

the reviews.

2) The system proposed looks like an improvement

over something that is already OK.

I only feel a little perturbed at loosing

the out-of-10 scale. Would it be OK to use

the descriptions for 1, 4, 6, 10 and let

people interpolate if they want to?

GaJ.

First to Keth - congratuations!

(?)

Green Jade - perhaps I'm being overly egotistic; admittedly I am looking at this very much from the standpoint of a designer. I think the Scenario Depot is a great place, and in fact find the majority of scenarios that I play PBEM there. I won't say I haven't been disappointed by some of the reviews that have led me down the garden path, but in the end we all have different desires and ideas on what makes for a satisfying experience.

I do think my ideas would be of benefit to those simply looking to play a decent game, too, not just egomaniacs like myself who get in a huff at "average" scores that match "below average" scores depending on whom the reviewer is.

I think I understand your point about 1,4, 6 anc 10 - ie - define what these scores are and then let people "tilt" these by using 2, 3, 5, 7, 8, 9 etc. as appropriate.

It has merit. I kinda like the 4.0 scale myself having used it to track my grades for seven (long) years of university.

Perhaps allowing decimals to be used in the scoring system would allow for this - ie if you think a scenario is more a 4 than a 3, rate it a 3.7 - same idea, just taking it from a 10 point scale to a very rigidly defined 4 point one. Still room for variation between the 4 points, but less room for unconsidered whim to totally derail a scenario.

My thoughts, anyway.

[Fredrock1957]

Hi Michael...

Based upon the rating system you described, I believe that your rating system would be a bigtime benefit to the scenario designer. A lot more then the current rating system.

Being fairly new to the CM series (and also being fond of designing), when I first went to the Scenario Depot it appeared to me that the site wasn't really intended for feedback to the designer but more for the multitude of players looking to pick up different scenario's to play ( I think that Admiral Keth has done an awesome job with that site by the way).

After posting 4 scenario's at the Scenario Depot and watching each get downloaded about 100-200 times without feedback I went on a search. Thats where I discovered The Proving Grounds of which one portion of the site was specifically developed with the designer in mind. Right now there are about 40-60 very active members (mostly designers) that are giving, discussing, and receive quality feedback (via Playtesting) while developing the best possible scenario's they can for posting at The Scenario Depot. I believe your rating system could be very advantagous to TPG and I hope GJK will look at it closely.

Thank you for the detailed breakdown of the ratings.. I think you hit the "nail" on the head

Regards,

FredRocker

[John O'Reilly]

Michael,

Just a clarification as said reviewer once more. I do rate 8s or higher, but not as frequently as others. If everything is rated towards the higher end of the scale then we have a very coarse system which doesn't seem very useful . We might be better off with a 5 point scale in the first place.

One of the problems inherent to the current system is that scores such as replayability can drag down the average. For example, in a scenario designed to be played double-blind with plenty of surprises, the score for replayability can only be average to low. This drags down the rating for what might be an incredible scenario. I suppose I could put "0" in such a case, as I have noticed others are doing.

I would like to see a score for "enjoyment". After all, that is what people are really looking for and might be more meaningful when quickly perusing a large number of potential downloads. The other scores could retain some functionality without detracting from the overall rating.

All the best,

John

[GreenAsJade]

In case it wasn't clear in my previous post, I'm in favour of what Dorosh has proposed.

I did think it important to recognise that the current system is not entirely broken... and I acknowledge that it probably works better for the scenario shopper than the supplier.

I guess Admiral will make a call on 4 or 10

Cheers,

GaJ.

[xerxes]

One change I'd like to see is a "rate this review as helpful". Just like Amazon does. I think this would help things. Ideally you could search by reviewer but I know this is technically not possible without requiring registration for reviewers.

I know that when I'm shopping for a new scenario I pay particular attention to reviewers that share the same biases that I do.

[Michael Dorosh]

Originally posted by xerxes:

One change I'd like to see is a "rate this review as helpful". Just like Amazon does. I think this would help things.

Definitely; a few of us have mentioned this now.

I agree with John's points above too - I don't know that everyone rating high is a problem so much as people rating inconsistently. For me an 8 is average, for some 8 is well above average. Fair enough, but ultimately confusing.

[ November 26, 2003, 02:00 PM: Message edited by: Michael Dorosh ]

[Spookster]

Originally posted by xerxes:

One change I'd like to see is a "rate this review as helpful". Just like Amazon does.

Xerxes,

Good point, but I'd take it a step further. I'd ID individuals with a knack for writing useful commentary and give them special status (e.g. "Recon Critic".) For example, if you write, let's say, ten reviews and you gather a net of, say, 40 "useful" designations, you are a one star critic. (The formula, of course, would have to be worked out.)

There are at least two benefits to such a system.

A. Some reviewers may seek to join this elite group by improving their evaluations. Better evaluations = better feedback = ... better scenarios.

B. Players will better identify the scenarios they want to play.

I think after literally years of grumbling from the designers regarding the SD rating system, we have to realize that we cannot trust every reviewer to give us thoughtful feedback or give the players a fair evaluation. Let's give incentives to create an elite group of reviewers whose words carry more weight...because they earned that right.

What say you?

[Michael Dorosh]

Originally posted by Spookster:

What say you?

Yes.

[Spookster]

Originally posted by Michael Dorosh:

</font><blockquote>quote:</font><hr />Originally posted by xerxes:

One change I'd like to see is a "rate this review as helpful". Just like Amazon does. I think this would help things.

Definitely; a few of us have mentioned this now.

I agree with John's points above too - I don't know that everyone rating high is a problem so much as people rating inconsistently. For me an 8 is average, for some 8 is well above average. Fair enough, but ultimately confusing. </font>

[Michael Dorosh]

Originally posted by Spookster:

</font><blockquote>quote:</font><hr />Originally posted by Michael Dorosh:

</font><blockquote>quote:</font><hr />Originally posted by xerxes:

One change I'd like to see is a "rate this review as helpful". Just like Amazon does. I think this would help things.

Definitely; a few of us have mentioned this now.

I agree with John's points above too - I don't know that everyone rating high is a problem so much as people rating inconsistently. For me an 8 is average, for some 8 is well above average. Fair enough, but ultimately confusing. </font>

[Spookster]

Even with each numberic value explained in detail, complications arise in the thought process. "Causual" reviewers will likely pass.

But I think we agree...we need more quality reviews than quantity.

p.s. Wish I was young.

[jwxspoon]

Hey guys!

Sorry it took me so long to notice this thread, I generally run away from Michael Dorosh authored threads as fast as I can :eek: (just kidding Mike! Actually after seeing pics of your jeep I am working hard to get mine up to snuff)

Rune and I were talking the other day about what changes we would like to see in the rating system at the Scenario Depot. Both of us wholeheartedly support Admiral Keith and the Scenario Depot. It has become the best online repository of scenarios for the CM series. However, the current numbering system does need to be changed.

I would like to see a switch from a numbering system to a simple poll-type display. A reviewer would simply pick from the following choices:

1. Highly Recommended

2. Somewhat Recommended

3. Reviewer is Neutral

4. Somewhat Negative

5. Extremely Negative

The results could be displayed in a percentage type basis. Perhaps there could still be a section for individual text comments by the reviewers, as well as a rebuttal section for the scenario author.

jw

[xerxes]

The scale 'spoon proposes is very good. Unanchored (just numbers) rating scales are known to produce less reliable measurements. I'd also like to see an explicit "no comment/opinion" option. The "0" does that now but a new reviewer can easily miss that.

I'd also like the reviewer to be able to post the score from the scenario that they played.

[ December 02, 2003, 02:52 PM: Message edited by: xerxes ]

[Admiral Keth]

All,

I've managed to weather the initial onslaught of company conversion, so I'm taking a minute to put forth my thoughts on the subject.

I would like to remind everyone that many hands contributed to the design of the existing system when it was implemented. Do a search; you'll see the reason for the current design. Thus, we will try this experiment one more time...authors and players, now is the time to voice both opinions and solutions.

After much discussion with other individuals, as well as reading numerous past and present posts, I am forming the opinion that the numerical rating system must be scrapped.

The reasons are:

1) Any numerical system is strictly subjective. Literally up to the whim and perspective of the person posting the review. Interpretaion of the numerical values by the reviewer, no matter how well explained, can be misinterpretted or ignored. Therefore any system which uses a numerical system will necessarily be skewed to individual values, and it would take a large number of excessively good reviews to avarage out a single bad review.

2) Numerical systems can be abused. Several people have pointed out that numerous scenarios are receiving inordinate scores based on whatever clique happens to be rating them. This unfairly affects groups of scenarios over individual designers.

3) Anonymity. Reviews are being posted by everybody and their brother under the assumption that they are anonymous. Therefore reviews and ratings were being posted which did not truly reflect the true value of a particular scenario. I am leaning toward implementing a system similar to the one mentioned by jwxspoon. These values would not be accumulate into any kind of total. Each review would need to stand on its own. Therefore reviews which simply stated "It sucks" can be mentally eliminated by the reader.

I am strongly considering making The Scenario Depot a registration/login-based site. No more anonymous posting of reviews. If you post a review which seems out of touch with the other reviews and do not give valid reasons for your review, then the author will have the capability to contact you. You can then work out amongst yourselves.

The Lists, while a decent idea at the time, have become so badly abused by inflated ratings as to become useless. The Lists will be modified to no longer include the numerical values. Instead, I am strongly considering simply showing statistics such as Top 25 Downloads (Total, Past Month, Past Quarter, Past Year). I would like to hear what other non-rating releated statictics authors would be interested in seeing. Keep in mind that any statistic-based information would start fresh from day of implementation; there is no method by which I can determine historical values.

In the short term, I'll be finishing the CMAK submission and search pages before release, plus some other tasks. Be aware that I am also working on web sites for paying clients, so they necessarily get the bulk of my time.

It's still going to take a few more days here at my day job to conduct damage control for the changes we've made, so I'll respond as often as time permits.

I would be interested in chatting with anyone who has PHP/MySQL experience who would want to assist in these tasks. Email me directly, please, if you are interested.

Lastly, if you are simply going to say "It's broken, fix it!", then the door is over there. I'm tired of people complaining, yet never offering any sort of assistance. If you are going to say "It's broken, and here are some real world methods by which it can be fixed with which everyone will be made happy.", then I'm all ears.

[Pzman]

Great points AK, too bad I don't know how to use those codes since I have a lot of free time right now.

[WWB]

Been meaning to put in my 2c on the subject, but got sidetracked and have needed to collect my thoughts.

First and foremost, I wholly support the Admiral Keth and Scenario Depot. Even as it stands now it is a wonderful resource. Without it the trade of making scenarios would be in serious difficulty and the "little guy" without a website would have nowhere to post his work. I also fully undestand the paying clients situation. Designing a database-driven website is not a simple matter of making a few dreamweaver templates and manufacturing pages. Getting all the programming to play well together, especially when dealing with a large body of existing data and accepting updates from the general public is not an easy job.

That said, it is pretty clear that time has proven that the ratings on the depot are somewhat inaccurate. I like Jeff's proposal alot. On the other hand, I also like having multiple ratings for different categories. Different players like different things in battles, and I think that reviews should reflect that.

There also is the question of how to handle the existing data. This is possibly the most cruical part of this process. If one chooses to just drop or archive it, one has alot more freedom of action. Whereas if one chooses to update it in some form than one is, by definition, somewhat tied to former design decisions.

The problems with the current system as I see them are thus:

1) Too much gets lumped into the PBEM/AI playability ratings. Basically, this rating is the bulk of the review. Everything else is pretty much eye candy. This rating covers balance, fun factor, overall design and just about anything else the four remaining categories don't cover.

2) Replayability really should be a boolean value, and not averaged into scores. A scenario either is or is not replayable. Given that, AFAIK, most people will play a battle once, it should not be rated and averaged into the total.

3) As Keith notes, the anonimity lets people bump up their friends, take down their enemies and otherwise be general nuicances.

I cannot quite claim to have my thoughts together enough to issue a specific proposal, but I shall discuss the options on the table already.

While Grog Dorosh's system covers just about every concievable base, it is a bit ardurous at best. Where there were once five ratings there would be nearly a dozen. That is a bit overblown, not to mention a very large break with the past. People already tend not to review battles with comparatively little to think about. I suspect fewer would put up with filling out more fields. In addition, it should be noted that, no matter how clearly the criteria are laid out, any review is subjective. Also, I think dropping multiplayer ratings is a grave mistake. The people who care enough about a game to go seek out, and review, scenarios are far more likely to play multiplayer. And I know of alot of folks who choose their scenarios based on the multiplayer ratings at the depot. I think the best path lies somewhere in the middle.

I do think the idea of making it a registration-based site is the right course for the future. This does present a nasty problem with existing reviews. Once one moves into such a registered system, all the pre-existing, non-registered reviews are by definition invalidated. But it might well be worth it, for it would stife trolls to some extent while also allowing one to do things like search by reviewer. In any case, such a move is alot of coding work, as one is almost starting from scratch, even if one has a very good set of existing data.

One other ancilliary, and technical note. One can store the data as numerical, but present text to the end users. This might go a long ways towards normalizing the scores. In any case, I am pondering useful suggestions.

WWB

PS: Rob, PHP is very easy to learn, especially if you already know javascript. MySQL is just a database, like many others. Would not be a bad skillset to pickup.

[Berlichtingen]

Originally posted by jwxspoon:

Rune and I were talking the other day about what changes we would like to see in the rating system at the Scenario Depot. Both of us wholeheartedly support Admiral Keith and the Scenario Depot. It has become the best online repository of scenarios for the CM series. However, the current numbering system does need to be changed.

I would like to see a switch from a numbering system to a simple poll-type display. A reviewer would simply pick from the following choices:

1. Highly Recommended

2. Somewhat Recommended

3. Reviewer is Neutral

4. Somewhat Negative

5. Extremely Negative

The results could be displayed in a percentage type basis. Perhaps there could still be a section for individual text comments by the reviewers, as well as a rebuttal section for the scenario author.

jw

I've got to say that I really like this approach. I get the impression that at least some reviewers scew existing scores to achieve the one combined score they want. Let's face it, if you play a sceanrio that you think is great, or you think sucks, what difference does the map make?

I also think the idea of having the site a registered site is a fantastic idea for a couple of reasons. One, obviously, has already been stated. Two, and of more importance to me, is that with registered reviewers I would think it would be possible to pull up a list of reviews by a specific reviewer. So, if I find that WWB consistantly gives good reviews to scenarios I like I can hunt down the scenarios he has reviewed.

As for lists... I only really pay attention to the recent review list anyway. Perhaps, as a registered user, we could have saved searches? Example... WWB's reviews. Is that doable?

As for trying to retain existing reviews... I'd say do whatever is easiest. I wouldn't suggest wasting a lot of extra effort on it

[WWB]

Berli--I think you just unmuddled my thoughts.

Here is my official suggestion:

Archive the text portions of the old reviews, and maybe the overall ratings.

Go registration required.

Make reviews a simple jefe-style thumbs up/thumbs down. But add a few checkboxes for the following:

I Played This Scenario:

[]Allied vs the AI

[]Axis vs the AI

[]Multiplayer

That way people could search for reviews relevent to how they play the game.

WWB

[Michael Dorosh]

Originally posted by WWB:

Berli--I think you just unmuddled my thoughts.

Here is my official suggestion:

Archive the text portions of the old reviews, and maybe the overall ratings.

Go registration required.

Make reviews a simple jefe-style thumbs up/thumbs down. But add a few checkboxes for the following:

I Played This Scenario:

[]Allied vs the AI

[]Axis vs the AI

[]Multiplayer

That way people could search for reviews relevent to how they play the game.

WWB

But if you only played it as Allied via PBEM, wouldn't that be relevent, too?

I will add my voice to the registration requirement - including a valid email address to confirm the registration, and not allowing multiple accounts on the same email address.

[Spookster]

Concerns:

1) After CMBO, CMBB,the recently released CMAK, the time it will take AK et. al. (us) to design a new system, and all the time AK will take to implement the new system, will the CM craze be past its zenith. Will it be worth it for AK to put this together for FREE?

2) Should we (designers/players) pitch in $20 to a pool to aid in AK's efforts? I understand there is a "free rider" problem to this solution, but I'm willing, and I've only designed and posted five scenarios. (I've already gotten much > $20 out of the current system.)

3) All the suggestions I've read above and in other discussions do not completely address the problem of "fan boy" reporting, "hate mail" reporting (which is strongly correlated to how badly the reviewer got spanked in the scenario), and overall scenario bias (e.g. I hate city scenarios, love tank battles...etc.)

Let me suggest a more modest fix to the system that a) grants the designer better feedback involves less bias c) keeps this particular message board in business d) uses the system that AK has in place.

SUGGESTION

Keep the SD as is, but allow scenarios to be tagged as follows: recommended (bronze star), highly recommended (silver star), elite (gold star).

Where do these tags come from?

People on this board.

Three will be elected (using this message board) to a board of review (a Cabal) and one BACKUP. If two of three recommend a scenario, then it gets a bronze star. If two of three highly recommend a scenario it gets a silver star. If all members believe it is an elite scenario, then it gets a gold star.

(Note: The Cabal could include five, seven, or more members...but this system needs DEDICATED CMers to work...can we find THREE?)

For a scenario to be considered, it must be nominated by someone on the Cabal.

If a scenario is considered, someone on the board (or a trusted proxy) should submit a quick review outlining weaknesses/strengths (though if the Cabal discussions are posted here, that may not be necessary.) Scenarios may be reconsidered.

The elections for the board will be yearly. Memebers of the board may resign at any time between elections...the retiring (or vacationing)member and remaining members of the board will elect a replacement. A member of the Cabal may not nominate or recommend his own scenarios -- the BACKUP will vote instead.

Pros:

a) Good feedback for the designers. As the Cabal discusses a particular scenario on this messageboard, non-Cabal members can inject their own two cents.

Keeps this message board hopping. (It seems a bit slow lately.)

c) Cheaper than a re-programing the SD.

d) More useful to gamers in choosing scenarios.

Cons:

a) Can we find three honorable and dedicated CMers for the Cabal?

This system is useful for CMAK, but can we go back and use it for BB and BO?

-Mike

[Admiral Keth]

spookster,

Excellent ideas all around. I think the Cabal concept desreves further exploration. I can foresee potential issues with regards to politics and existing affilliations of authors with current design groups. All of these will need to be addressed within this forum.

Others have also mentioned a star-based rating system. However, unless the methodology by which reviewers post values is uncorruptable, then it will eventually face the same issues in a year's time.

Prior to me putting one byte of code into place, I want as many authors to contribute to this thread as possible. Contibute ideas and methodologies. Those of us that can code, at least try to put authors' ideas into logical sequences (if-elseif-else-then, do-while, switch type statements) that I'll be able to quickly translate in PHP/MySQL. I want as large a concensus as possible before work begins.

I have also had the idea put forth to me regarding moderators. Each moderator would be responsible for keeping an eye on various sections (CMBO/CMBB/CMAK/Authors) and notifying me when issues arise. If they are proficient in PHP/MySQL, then I would allow access to specific sections to expedite resolution of issues. Moderators would also double as problem-fixers, i.e., broken links re-associated, missing scenario authors contacted, screen captures for

Previews, etc.

I am also considering allowing authors to upload there own files. I have always held the opinion that by my receipt of the files, I can absolutely guarantee that the file being download by a player is safe. If we open it up to authorial upload, this introduces the capability for malicious intent; all it takes is for one virus to be uploaded.

I'd like to hear opinions on this matter, as well as methodologies by which everyone can be absolutely guaranteed that the file being downloaded is safe.

I am not averse to donations at this point. I have never had much faith in the contribution system; unless there is a tangible exchange of goods or services, people tend to take more than give. Cumulative contributions could change the priority under which work would be accomplished at The Scenario Depot. I could post a link to my PayPal account, if the community is willing to support this. Opinions, please.

[xerxes]

So if we pay more, we get the changes we want!

I think the cabal idea really has some serious merit. I'd think there would need to be more like 10 members in it to make sure we get to three members playing though a particular scenario in a reasonable length of time. Given a cabal made up of a diverse set of the CM population and obviously very interested in scenarios I think the bias issue would be moot. We could even exclude scenario designers from membership in the cabal (or limit the designer to half the positions or something). A member who is a friend of the scenario designer should obviously exclude themselves from rating that particular scenario.