Jump to content
Sign in to follow this  
Michael Dorosh

CMAK Imminent - can we fix the Scenario Depot Rating System Beforehand?

Recommended Posts

OT - For those that care...

The handle I use pretty much everywhere is one that was _given_ to me by my Star Fleet Battles group. Ahhh, SFB, now there's a thinking man's game.

I predominantly played Klingons, and was moderately good. On two different non-tournament occassions I managed to beat supposed Rated Aces without breaking a sweat. On another occassion, I successfully captured a Lyran Starbase without firing a shot in anger. Yes, I have reliable witnesses.

So, the "rank" is not so much wet navy rank as it is fictional callsign (which are given, not taken) than anything else.

Back to the debate...

Share this post


Link to post
Share on other sites
Originally posted by Admiral Keth:

However, everyone still seems hung up on developing an aggregating numeric-based system. I am increasingly adverse to the implementation of any kind of system which can be subverted or corrupted.

_________________________________________________

I believe that ANY numbers based review system can be corrupted. If we use a simple Excellent-Very Good-Good-Poor-Very Poor rating system I believe that a better choice. Or something aa simple as "I liked it" or "I didn't like it".

The object of the system is to be easy to both use and implement. So the simplier the system that meets the requirements of a review is the best. What needs to be in a review? Does the reviewer like the scenario. Anything else is gravy. "Oh yes, this was a great scenario because...."

It is all opinion related. Whatever the opinion the reviewer has, is related AFTER he tells you, either he liked it or he didn't. So reviews should be along those same lines.

_________________________________________________

Changes that _will_ be made are as follows:

1) Remove the Awards line - Maybe re-implement at a future date with better graphics.

2) Change the location of the download link - Some players have difficuly in locating this link.

3) Add the capability to search for vs. AI, and PBEM battles.

4) Add A PayPal donation link.

_________________________________________________

Could we get a search criteria for type of battles as well such as Historical and Semi-Historical? Possibly size related as well? Lots of players play a particular type of battle that they are well versed in and favor.

_________________________________________________

In addition, how do the authors want to handle historical rankings? Simply archive the lot and start fresh? Leave them in place as is and ignore them for future ratings? This aspect needs to be handled in a logical and simple fashion, plus not invalidate the effort everyone has put into placing reviews over the past couple of years.

_________________________________________________

My thoughts would be to get rid of the lists entirely, they aren't updated at the moment with correct values anyway, and then keep the historical download data on each. Keep it simple.

_________________________________________________

In summary, the aggregating numeric system is soon to go the way of the Dodo. Let us concentrate our design efforts to a more simple and effective system. Once again, there must be a concensus, first from the authors, and then from the players, on how the system is going to work. This board system has the capability to post polls. Perhaps we can beg/wheedle/cajole MaddMatt into activating that feature for a one-time vote. [/QB]

It would be interesting to see the results of that particular poll.

Panther Commander

Share this post


Link to post
Share on other sites
Originally posted by John O'Reilly:

[QB] Michael,

A 10-point system allows one greater discrimination. Is a map slightly above average but not something you can consider "good"? Fine then it's a 6. Bog standard? It's a 5. Pretty straightforward.

Given Admiral Keth's comments, discussion on this is now moot.

As for my rating, I can only believe you are now being either obtuse or overly sensitive as I already explained this to you in a personal communication. The Russian briefing was functional (noted by the reviewer just before me by the way) and the German one was good. Where does that leave me overall? Well slightly above average which equals a 6.
None of which was stated in the briefing on the site for others to read and understand why you rated it. Given a half dozen ratings averaging 9, and your 6, the obvious question that forms in anyone reading the reviews is - "why the difference?" Not that I don't think briefing reviews were over-weighted in any event (usually 20% of the final ranking was based on the briefing, far too high for my liking) and it would work in reverse, too. Had everyone else rated it a 6 and you a 10, I'd be equally curious as to why. The fact that a different ranking was left was tantalizing; you will have to forgive people from being interested enough to know why. Or, to expect you to have had a reason for it.

Most people do not detail the thinking behind the rating system in whatever text they enter, you are a notable exception.
Spook was always good for these, too, as was Agua and one or two others. Detailing ratings isn't a requirement, don't know if it should be, but I know whose reviews I would pay more attention to. As stated earlier, given a choice between a "10" scenario with two reviews saying "wow, cool" and a scenario with 1 review of 6 with a detailed breakdown of what the reviewer liked - I'll go for the 6 every time.

I think the Proving Grounds may be quite useful - perhaps designers should really be getting their detailed feedback from that site, and leave the depot as simply "one stop shopping" for game players.

I also agree with the Admiral's (and hey, Keth, don't sweat the name it looks good on you!!) current proposals, as suggested by Jeff (reluctantly, but I grudgingly admit it would be the most workable).

There is probably some merit in using the Proving Grounds to perfect scenarios anyway, allowing finished products to make their way to the depot to be subjected to the tender mercies of others. Then the designers get what they want, and the players get improved "product".

[ December 08, 2003, 12:17 PM: Message edited by: Michael Dorosh ]

Share this post


Link to post
Share on other sites
In addition, how do the authors want to handle historical rankings? Simply archive the lot and start fresh? Leave them in place as is and ignore them for future ratings? This aspect needs to be handled in a logical and simple fashion, plus not invalidate the effort everyone has put into placing reviews over the past couple of years.
I have pondered this a bit, and I think I have come to a workable solution. Given that the movement is afoot to also tie reviews to logins, I propose doing the following:

1) Old reviews scores are divided by 2 and rounded to the nearest integer to give a score on the 1 to 5 scale. Comments field is kept as the new comments field.

2) Given that email is a required field in both reviews and scenarios, it might make sense to make email address the login name. One can then SELECT DISTINCT emails from both reviews and scenarios, add in a random password, and generate an initial users table. Then this could be tied back to the scenarios & review tables by JOINing things on email.

3) One then emails the list of addresses their new logins and temporary passwords. I have a blast email system that does customized messages and can send it for you if need be.

4) People can login, setup their accounts and such, and it will tie back nicely to the existing body of reviews, all of which will appear in the new system.

One other big question--how deep will the registration system go? Obviously one will need it to upload or review battles. But do we want to require a login to download a battle? My vote is no--I dont think putting impediments in the way of people getting the battles is a good idea.

WWB

PS: GaJ--I understand your proposal. I just find the idea of authors directing reviews of their work by anything other than the quality of said work to be fundamentally flawed.

PS: AK, I can try and lean on matt for a poll if need be, when the time comes.

Share this post


Link to post
Share on other sites

Well, after watching all of the discussion so far, I figure it's time to weigh in with my five canuck bucks (or 2 cents US)...

{Warning: the following is my opinion. Nothing more, nothing less.}

First of all...why do we even have the Depot? What is it? It seems to me that it exists simply to have a central, searchable location for all CM scenarios. Period. Anything else is gravy. What I think it is Not is a place for designer feedback, ego-inflating/deflating ratings, flamewars over subjective scores or opinions or anything else that might interfere with the simplicity of the site.

As a player, I could really give a hoot about weighting, numerical score vs stars or what kind of feedback the designer might get from the Deopt. What I want is to be able to quickly find a scenario that meets the criteria that I'm looking for at that particular moment, whether it be a huge fight with Finns in the snow against the AI or a quick tcp armor fight in the desert. As an added bonus I'd like to be able to see at a glance what other people thought of the scenario so that I could be reasonably sure that I would be playing a (hopefully) high quality scenario.

So, if there is to be a change, in my opinion it should consist of the following...

1. Be simple for Keith to implement. Nothing else matters really, because if he can't or won't do it, it ain't gonna happen.

2. Use a 5-Star system with the checkboxes as has been mentioned. Why? Because it provides useful information at a glance. I do Not want to read through 15 pages of numerical scores accurate to 2 decimal places, spoilers, reviews, designer responses and flamewars just to find out that approximately 90% of the people who played it liked it. 1 5-star, 8 4-star, 1 1-star and 1 3-star ratings will tell me the same thing with a lot less anguish for everyone involved. All I (remember me...the player?) want is to know if it's crap or good. Again...simple.

3. Include the date/time the score was entered, and log the IP (if possible). If a flood of good/bad reviews come in at the same time, or occur at the same time every day, or originate from the same IP etc., then there is probably some abuse happening. No system will be infallible, but simpler is still better.

4. Lose the text review & author's response, or move it to a separate page like the AAR's are now. They generally have spoilers in them, and I'm also incredibly Not interested in reading a designer vs player flamewar while I'm quickly trying to find a scenario for a tcp game in the chatroom. I just want to see a page that shows if the scenario is generally considered to be good or not. See the trend here...it's simple.

5. Make it easy for me to score a scenario. I don't want to have to decide if I liked the briefing 7.5 times out of 10, worry that the designer will send me hate mail because I didn't like the way the map looked, or be concerned that if I don't give constructive enough criticism I'll be condemned to playing mediocre scenarios for the rest of my life. I just want to score it from Flushable to Frameable, and if there is anything that stands out (good or bad) I'll let the author know about it. Or not. That should be my option, not requirement.

I fully understand that there is very little (or better yet none at all) feedback for the designer in what I have typed here, and that is intentional. I personally don't think that the Depot is the place for it, as it does nothing for me when I'm trying to find a scenario, which again (in my opinion) is the whole point of the Depot.

Harv

Share this post


Link to post
Share on other sites
Originally posted by Panther Commander:

_________________________________________________

Could we get a search criteria for type of battles as well such as Historical and Semi-Historical? Possibly size related as well? Lots of players play a particular type of battle that they are well versed in and favor.

_________________________________________________

Historicality and Size search criteria are already in place for CMBB and CMAK Battles. Size is already in place for CMBO Battles, as well as CMBO/CMBB/CMAK Operations, but it seems as I somehow left off Historicality. I'll pop that in whilst I am under the hood.

Originally posted by Panther Commander:

________________________________________________

My thoughts would be to get rid of the lists entirely, they aren't updated at the moment with correct values anyway, and then keep the historical download data on each. Keep it simple.

_________________________________________________

The Lists will definitely undergo some kind of revision. What that revision is remains to be seen.

Originally posted by Panther Commander:

________________________________________________

It would be interesting to see the results of that particular poll.

________________________________________________

I agree...Steve/Matt/Moon; any chance of this happening?

Share this post


Link to post
Share on other sites

By the by, any registration system which forces people to log in may potentially break direct links to scenarios.

A fine point of this topic is:

What kind of system do people want implemented?

Do you want:

1) a system that requires login to view/post for the entire site?

2) a system which allows view only of the scenario synopsis, allows anonymous downloading of the scenario file, but requires login to post reviews?

3) a system which allows view only of the scenario synopsis, but requires login to download files and post reviews?

IP tracking is a definite yes.

Share this post


Link to post
Share on other sites
Originally posted by Harv:

Anything else is gravy.

I personally don't think that the Depot is the place for it, as it does nothing for me when I'm trying to find a scenario, which again (in my opinion) is the whole point of the Depot.

Harv

If you don't like the gravy don't eat it...but don't ask the hostess not to serve it.

I think I am leaning to your suggestions - so far as the Proving Ground remains in place. But tell me - did the detailed text reviews really prove to be an impediment to you? If so, how?

Share this post


Link to post
Share on other sites
Originally posted by Michael Dorosh:

If you don't like the gravy don't eat it...but don't ask the hostess not to serve it.

I think I am leaning to your suggestions - so far as the Proving Ground remains in place. But tell me - did the detailed text reviews really prove to be an impediment to you? If so, how?

Ah, poor choice of words on my part. Replace "gravy" with "mostly unnecessary fluff and stuff". ;)

The text reviews were an impediment when looking for a scenario, as I had to be very careful not to look at any spoilers while scrolling down the page to look at the scores. The averaged results are fine, but I prefer to see each individual score.

As someone who was going to review a scenario I didn't like text reviews simply because sometimes I just wanted to score something, but not discuss it in any way. Some scenarios are just worthy of an "It Rocks!!" or "It Sucks!!", and some just need a score and nothing else really needs to be said. Either way, a separate optional page would solve both concerns rather nicely.

I do think you've hit the nail directly on the head with your thoughts about using the Proving Grounds as the test site and the Depot as the "finished" location. That way the designers get what they need from the Proving Grounds, and the players (at least this one) get what they want from the Depot.

Harv

Share this post


Link to post
Share on other sites

I probably shouldn't throw any more fuel on the fire, but what about a system which allows the author to select whether they want just textual reviews, just numerical reviews or both for a particular scenario. This would enable the author to have ultimate control over the nature of the feedback. In addition, they could select whether they wanted each review to stand on its own, or have any numerical values accumulated into an aggregate value which would have no relationship to any other values associated with other scenarios.

Thus, if an author chose to aggregate and average numerical review values for a single scenario he was fine tuning, any numerical value (10, 8, 5, 1) would only have weight against _that_ scenario, and have no wirght against any other scenario comparably valued. At any point, the author would then be able to deactivate the numeric system for that scenario, relying only on textual reviews.

Lastly, an author could simply deactivate the numeric and textual review system altogether, and only use The Scenario Depot as a file host.

This is actually fairly easy to implement in a couple of solid 8 hour days of work.

Opinions...

Share this post


Link to post
Share on other sites

I like Jeff's and Harv's ideas Admiral keth. I don't want it to be what it is nor do I want it overly complicated. What I suggest is Jeff's, a description and the numbers 1-5 and just leave it so I can click on the author'sname to leave feedback in text form. The world need not see it, and prvents spoilers. Allow the author to have it on or off, so if he doesn't want to respond he doesn't have to or receive it. So, a 1-5 scale and that is what you see, along with the ability to send text to the author alone via email. No top tens, so people do not rush in to abuse the system, and last, ip tracking as you stated.

For the users, they see at a glance if the scenario was fun or not, and the authors can still get good feedback/improvement feedback via email.

Rune

Share this post


Link to post
Share on other sites

Let me have a say too, will ya! smile.gif Is it possible to have two sections in a review where one is a general one and the other is an AAR style section. The AAR section, if it is used, stays hidden and shows up only when clicked on (it could be a pop up). That way you could see what previous players have said about the scenario before you try it yourself, but won't risk seeing any spoilers.

Share this post


Link to post
Share on other sites
Originally posted by Sergei:

Let me have a say too, will ya! smile.gif Is it possible to have two sections in a review where one is a general one and the other is an AAR style section. The AAR section, if it is used, stays hidden and shows up only when clicked on (it could be a pop up). That way you could see what previous players have said about the scenario before you try it yourself, but won't risk seeing any spoilers.

AAR's are already implemented. There is an AAR button beneath the Review Average Ratings column. This button will pop up an AAR screen, where you can read and submit detailed AAR's. A few people actually use them, as shown in the indeces AAR column. People _should_ be placing detailed information in this section, as opposed to including the in the reviews. Such is not always the case.

Implementing a feature where the "I Liked It!" or 1 - 5 rating is displayed along with a "Read More About It" link is easily done. The link would then popup the textual review, if one was submitted. How many will then complain about "Awww, I gotta click a billion links to read the reviews. Fix It!".

Who was it that mentioned something about pleasing all the people all the time? :D

Share this post


Link to post
Share on other sites

Alright Keith...why do you insist on interrupting us while we decide what you're going to be doing with your website? tongue.gif

I'm not sure if having different options for different review system types is a good idea, as it may become somewhat daunting for us dumb dert farmer types to figure out which scenarios we can review, and then which system we should use once we get past Step A. And then we'd probably have to type sumfink after Step B. Or C. See, I'm confused already. ;)

I also (and I can't believe I'm saying this) agree with Rune's ideas he managed to blurt out with (onto?) his keyboard. And without lists and rankings, a logon system might not even be necessary.

Just more thoughts in any case.

Harv

Share this post


Link to post
Share on other sites
Originally posted by Flammenwerfer:

I hope top 10 lists of some sort are kept...no system will completely stop trolls or abuse.

I completely agree - please don't take away a way of relative ratings for scenarios, and ways to ask to see the ones that people thought were best.

That is a valuable part of Scenario Depot as it is now, and it is _not broken_.

Sure, there might be a scenario that should be 3rd instead of 4th or something, but only the author cares about that. The rest of the public uses the system to find the kinds of scenarios that lots of people liked. That is important.

GaJ.

Share this post


Link to post
Share on other sites
Originally posted by WWB:

Sigh. My proposal was all about reviewing the quality of the work. It was also about putting appropriate emphasis on different aspects of a piece of work.

Its as if you would be in favour of a book review including a rating for "creativity" in a documentary, or "factual basis" in a work of imaginative fantasy.

If this were the "Book Depot", people would be intersted in "Creativity" for fantasy works and would laugh at the idea of a "factual basis" rating for imaginative fantasy.

This is no different.

I know that the proposals have moved on now, but I could not let that point go by.

I hope we don't throw the baby out with the bathwater in the new system.

It's great having detailed numerical ratings on the scenarios, even if there is some author angst from time to time.

GaJ.

Share this post


Link to post
Share on other sites

One thing about Top Ten lists and the like...they only really work if the scenarios are scored using a system accurate to 1/100.

A five or even ten point grading system won't allow enough discrimination between scenarios. For example, if 60 scenarios are scored at 4 Stars, or 45 are rated at 4.5 out of five (9 out of 10), which ones should be in the top 10?

If Keith moves to a "less precise" system of rating scenarios, the lists are essentially worthless by default.

Harv

Share this post


Link to post
Share on other sites
Originally posted by Harv:

One thing about Top Ten lists and the like...they only really work if the scenarios are scored using a system accurate to 1/100.

A five or even ten point grading system won't allow enough discrimination between scenarios. For example, if 60 scenarios are scored at 4 Stars, or 45 are rated at 4.5 out of five (9 out of 10), which ones should be in the top 10?

If Keith moves to a "less precise" system of rating scenarios, the lists are essentially worthless by default.

Harv

And unless the same reviewers are reviewing ALL the scenarios, the larger the number spread, the more variance in how people assign those numbers - ie you may consider an average scenario a 45, while I might consider 80 an "average" score. So if you rate my scenario a 55 thinking it "above average" and I rate rune's scenario an 85 thinking it "above average" in the same way - without a detailed breakdown of how the numbers were assigned, we have the same problem. Then the players come along and perhaps one of them refuses to try anything less then a 90, based on his own definition of what "average" is.

Not that this isn't all normal, but the five point system may at least minimize the frustration for people just looking for something enjoyable.

http://www.battlefront.com/cgi-bin/bbs/ultimatebb.cgi?ubb=get_topic;f=9;t=010907

Check out this post on EYSA and the discussion of the 100 point review system used by PC Gamer. GI Combat sucked, but one reviewer gave it a 59% - some felt that was too high. EYSA has received 86 - see the other comments for a look at the troubles of the percentage system.

[ December 09, 2003, 10:22 AM: Message edited by: Michael Dorosh ]

Share this post


Link to post
Share on other sites
Originally posted by GreenAsJade:

I completely agree - please don't take away a way of relative ratings for scenarios, and ways to ask to see the ones that people thought were best.

That is a valuable part of Scenario Depot as it is now, and it is _not broken_.

At the moment it is broken. The lists are not at all accurate. There are some scenarios that are not on the lists that should be in the top three. I know of a scenario that should be the #1 PBEM Battle and it isn't even on the list. So yes, it is broken, at the moment. By the way it isn't my scenario, just to forestall all of those comments.

Sure, there might be a scenario that should be 3rd instead of 4th or something, but only the author cares about that. The rest of the public uses the system to find the kinds of scenarios that lots of people liked. That is important.

If you find the lists important at the moment they aren't representative of the review system at all. I believe AK hasn't fixed that until this issue is resolved but I won't speak for him.

Panther Commander

Share this post


Link to post
Share on other sites
Originally posted by Panther Commander:

That is a valuable part of Scenario Depot as it is now, and it is _not broken_.

No, its not broken... It never worked in the first place. That's why the registed user idea has real merit if, as a registered user, you can customize the homepage (saved searches, that sort of thing). I find the list of what's been added and what's been reviewed recently helpful... the top 10 lists have always been a waste of space

Share this post


Link to post
Share on other sites
the top 10 lists have always been a waste of space
I would bet that is a minority opinion...the lists though in need of refinement, are a practical place for someone to start on their quest for a good battle.

What do the downloaders want? Maybe the opinions of scenario designers(mine included) should come second.

[ December 09, 2003, 10:35 PM: Message edited by: Flammenwerfer ]

Share this post


Link to post
Share on other sites
Originally posted by Flammenwerfer:

</font><blockquote>quote:</font><hr />the top 10 lists have always been a waste of space

I would bet that is a minority opinion...the lists though in need of refinement, are a practical place for someone to start on their quest for a good battle.

What do the downloaders want? Maybe the opinions of scenario designers(mine included) should come second. </font>

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Sign in to follow this  

×
×
  • Create New...