CMAK Imminent - can we fix the Scenario Depot Rating System Beforehand?

[Berlichtingen]

Originally posted by GreenAsJade:

The lists ARE VALUABLE because ON THE WHOLE they contain AN EXCELLENT reference for people looking for scenarios.

The lists would have value if the users could define them. As it is, each list has wildly dissimilar scenarios. Looking at the CMBB Top 10 battles vs the AI, there are only two that I would even bother with. I personally don't care what the score is, I'm more interested in subject matter and turn length for AI games

The average reader IS NOT BOTHRED if a scenario has one rating of 9.5 and one of 6. (Only the designer cares). IT IS INTERESTING to read the differing opinions, and why they are the way they are!

You make my point for me... the players will never see one rated 6

[General Bolt]

Originally posted by Admiral Keth:

By the by, any registration system which forces people to log in may potentially break direct links to scenarios.

A fine point of this topic is:

What kind of system do people want implemented?

Do you want:

1) a system that requires login to view/post for the entire site?

2) a system which allows view only of the scenario synopsis, allows anonymous downloading of the scenario file, but requires login to post reviews?

3) a system which allows view only of the scenario synopsis, but requires login to download files and post reviews?

IP tracking is a definite yes.

Personally I think loggin in is a waste of time, except for scenario developers. And the reason for that is to give credit where credit is due.

So what, if someone reviews a scenario 97 times and rates it with all 10's. I have more important things to do with my life then stuff a CM scenario ballot box. Who benefits and who is hurt? I may then down load a really crappy game that was rated 9.995, but, big deal its just a freakin game. ...Just a freakin game... did I say that? It the only 3 games I play

Loggin for a download? Why? The only reason for that is catch some weasel rat b@$tard that may have a bootleg copy of the game.

On top of all that its just another password for me to forget. AK, if you do descide to require loggin in please have the remember my id/pw option so I only need to log in once.

[GJK]

Nothing personal, but you're exactly the reason why many sites do have login requirements.

[Michael Dorosh]

Originally posted by GJK:

Nothing personal, but you're exactly the reason why many sites do have login requirements.

Too true.

[GreenAsJade]

Originally posted by Berlichtingen:

</font><blockquote>quote:</font><hr />Originally posted by GreenAsJade:

The lists ARE VALUABLE because ON THE WHOLE they contain AN EXCELLENT reference for people looking for scenarios.

The lists would have value if the users could define them.

</font>

[flammenwerfer]

You make my point for me... the players will never see one rated 6

I think Berli raises a good point as very few scenarios are spotlighted by the current lists..maybe more types of lists are need for historic, armor, infantry only, combined arms, or specific battles, regions or time frames, as examples..

Also, I believe certain battles are not showing up on the current lists because of a bug..?

I believe many of the SD problems are a result of the scarcity of reviews. Would a single 10 point review system encourage more reviews from the DLers?

I suggest a 10 point system where each value from 10 down to 1 is clearly described. As an example:

10- A classic. Innovative or Exceptional design...

9-

7- Good.

5- Average. Minor flaws

3- Needs revision major flaws.

1- crap

This is a rough layout, maybe someone much more analytic could help with the descriptions.

Also, the review 'screen' would be prominently displayed, perhaps where the news section is currently.

[ December 14, 2003, 02:09 PM: Message edited by: Flammenwerfer ]

[Michael Dorosh]

Originally posted by Flammenwerfer:

</font><blockquote>quote:</font><hr /> You make my point for me... the players will never see one rated 6

I think Berli raises a good point as very few scenarios are spotlighted by the current lists..maybe more types of lists are need for historic, armor, infantry only, combined arms, or specific battles, regions or time frames, as examples..

Also, I believe certain battles are not showing up on the current lists because of a bug..?

I believe many of the SD problems are a result of the scarcity of reviews. Would a single 10 point review system encourage more reviews from the DLers?

I suggest a 10 point system where each value from 10 down to 1 is clearly described. As an example:

10- A classic. Innovative or Exceptional design...

9-

7- Good.

5- Average. Minor flaws

3- Needs revision major flaws.

1- crap

This is a rough layout, maybe someone much more analytic could help with the descriptions.

Also, the review 'screen' would be prominently displayed, perhaps where the news section is currently. </font>

[flammenwerfer]

If you can't provide a clear example of what you're talking about, why bother?...

So your ten point scale is as meaningless as the current one, unless you want to flesh out the proposal a bit.

My ideas are very clearly stated. The details need to fleshed out, with the appropriate language (for the descriptions) ageed upon by a consensus...

If others think this is a good idea(and will increase the number of reviews), I will dedicate more time to fleshing it out. If any one wants to help(Michael included) let me know.

[ December 15, 2003, 05:20 PM: Message edited by: Flammenwerfer ]

[Berlichtingen]

Originally posted by Flammenwerfer:

I suggest a 10 point system where each value from 10 down to 1 is clearly described. As an example:

10- A classic. Innovative or Exceptional design...

9-

7- Good.

5- Average. Minor flaws

3- Needs revision major flaws.

1- crap

I don't see where this would be better than a 5 point scale...

5 - Exceptional

4 - Good

3 - Average

2 - Needs work

1 - Crap

Adding 5 extra ratings to that just adds to confusion

[Michael Dorosh]

Originally posted by Berlichtingen:

</font><blockquote>quote:</font><hr />Originally posted by Flammenwerfer:

I suggest a 10 point system where each value from 10 down to 1 is clearly described. As an example:

10- A classic. Innovative or Exceptional design...

9-

7- Good.

5- Average. Minor flaws

3- Needs revision major flaws.

1- crap

I don't see where this would be better than a 5 point scale...

5 - Exceptional

4 - Good

3 - Average

2 - Needs work

1 - Crap

Adding 5 extra ratings to that just adds to confusion </font>

[Berlichtingen]

Originally posted by Michael Dorosh:

Even a five point system will work better with definition;

5 - Well above average - may be innovative, or simply elegantly designed for maximum enjoyment when played as prescribed (ie vs AI, PBEM, tournament, etc.)

4 - Above average - Briefing well written and involving, forces realistic/well researched (if this matters), fun to play

3 - Average - All necessary elements are in place (ie adequate briefing is included, map is adequate to the size of the game, forces are balanced (if that was the intent). May not be "above average" because it is not fun to play (for example, requires lengthy road moves of trucks and much micromanagement/plotting, or a very long approach march in which the first half of the game is spent simply moving to contact)

4 - Below Average - elements missing, or poorly done (incomplete briefing, for example, forces not suited to the task, map not sized correctly, turn length not adjusted)

5 - Unplayable/Unwinnable - major elements missing from the game or not well thought out.

Good descriptions, but that's only a 3 point system (3, 4, 5)

[Sergei]

How about a binary feedback system:

1 - make more like these

2 - go kill yourself

That surely should give clear signals to designers.

[Berlichtingen]

Originally posted by Sergei:

How about a binary feedback system:

1 - make more like these

2 - go kill yourself

That surely should give clear signals to designers.

ROTFL!!!!!!

I like it! maybe we should go with a 3 point system

1 - make more like these

2 - Better than nothing

3 - go kill yourself

[Pzman]

Originally posted by Berlichtingen:

I like it! maybe we should go with a 3 point system

1 - make more like these

2 - Better than nothing

3 - go kill yourself

Maybe number three should be... Don't quit your day job.

[flammenwerfer]

Originally posted by Berlichtingen:

</font><blockquote>quote:</font><hr />Originally posted by Flammenwerfer:

I suggest a 10 point system where each value from 10 down to 1 is clearly described. As an example:

10- A classic. Innovative or Exceptional design...

9-

7- Good.

5- Average. Minor flaws

3- Needs revision major flaws.

1- crap

I don't see where this would be better than a 5 point scale...

5 - Exceptional

4 - Good

3 - Average

2 - Needs work

1 - Crap

Adding 5 extra ratings to that just adds to confusion </font>

[Harv]

Originally posted by Flammenwerfer:

I think the problem with a 5 point scale (as others have already mentioned in this thread) is that it will not provide enough discrimination, and you'll end up with hundreds of scenarios with a meaningless 3.2 rating. A proper distribution is needed.

If we can increase the # of reviews, than that one '6' review(which so rankles the designer)will average out with all the other '9's given.

Also, clearly defining the values from 10 to 1 will clear up the current confusion.

Can you please clarify a few things that come to mind from your post...

Why doesn't a 5 point system discriminate between scenarios enough? From my perspective as someone looking for something halfway decent to play, I would say anything rated at 4 or 5 should be good enough for me. Having something rated at 8.01 out of 10 doesn't make it any better than something scored as 4 out of 5.

Why is a score of 3.2 meaningless? To me it means the scenario is considered average, just as a more defined score of 5.23 , or even a range from 4-6 would in a 10 point system.

Not trying to start an argument here, but as I'm (obviously) someone who firmly believes that a 5 point system is much simpler, clearer and easier, and would increase the number of reviews posted because of those points, I'm interested in knowing why people in favor of a 10 (or more) point system think it would work better in the long run?

Harv

[Frenchy]

Originally posted by Sergei:

How about a binary feedback system:

1 - make more like these

2 - go kill yourself

That surely should give clear signals to designers.

Shouldn't a binary system be:

0 - make more like them

1 - go kill yourself

Sorry couldn't resist....

[GJK]

Play with this if you want, but something like this came up on a web developer's list that I'm subscribed to:

[math]SCORE=([Avg_Of_All_Reviews]*.7)+([Times_Reviewed]*.3)[/math]

[!]-- Averages are worth 70% of the final score and Count of Reviews are worth 30% . Adjust as needed, of course --[/!]

Scenario A:

---------

AVG=8 or 4.5

REV=1

[score]=5.9 or 3.45

Scenario B:

---------

AVG=7 or 4

REV=100

[score]=34.9 or 32.8

Scenario C:

---------

AVG=10 or 5

REV=1000

[score]=307 or 303.5

Or something like that....

[ December 15, 2003, 07:16 PM: Message edited by: GJK ]

[flammenwerfer]

Originally posted by Harv:

</font><blockquote>quote:</font><hr />Originally posted by Flammenwerfer:

I think the problem with a 5 point scale (as others have already mentioned in this thread) is that it will not provide enough discrimination, and you'll end up with hundreds of scenarios with a meaningless 3.2 rating. A proper distribution is needed.

If we can increase the # of reviews, than that one '6' review(which so rankles the designer)will average out with all the other '9's given.

Also, clearly defining the values from 10 to 1 will clear up the current confusion.

Can you please clarify a few things that come to mind from your post...

Why doesn't a 5 point system discriminate between scenarios enough? From my perspective as someone looking for something halfway decent to play, I would say anything rated at 4 or 5 should be good enough for me. Having something rated at 8.01 out of 10 doesn't make it any better than something scored as 4 out of 5.

Why is a score of 3.2 meaningless? To me it means the scenario is considered average, just as a more defined score of 5.23 , or even a range from 4-6 would in a 10 point system.

Not trying to start an argument here, but as I'm (obviously) someone who firmly believes that a 5 point system is much simpler, clearer and easier, and would increase the number of reviews posted because of those points, I'm interested in knowing why people in favor of a 10 (or more) point system think it would work better in the long run?

Harv </font>

[Michael Dorosh]

Originally posted by Flammenwerfer:

A 3.2 score is meaningless, in my opinion, because 90% of the battles will have that score. It will push all the battles into the middle.

So what? 90 percent of scenarios are playable or above average, are they not?

Or are you demanding a 20 percent distribution among the 5 grades?

How exactly, then, would you mark on the curve? :confused:

[flammenwerfer]

Originally posted by Michael Dorosh:

</font><blockquote>quote:</font><hr />Originally posted by Flammenwerfer:

A 3.2 score is meaningless, in my opinion, because 90% of the battles will have that score. It will push all the battles into the middle.

So what? 90 percent of scenarios are playable or above average, are they not?</font>

[Michael Dorosh]

Originally posted by Flammenwerfer:

</font><blockquote>quote:</font><hr />Originally posted by Michael Dorosh:

</font><blockquote>quote:</font><hr />Originally posted by Flammenwerfer:

A 3.2 score is meaningless, in my opinion, because 90% of the battles will have that score. It will push all the battles into the middle.

So what? 90 percent of scenarios are playable or above average, are they not?</font>

[Berlichtingen]

Originally posted by Flammenwerfer:

A 3.2 score is meaningless, in my opinion, because 90% of the battles will have that score. It will push all the battles into the middle. The scores will look like a snake that swallowed a large rat and not a rainbow of opinions.

Uh... it should look like a snake that swallowed a large rat. The average should be the majority. All increasing the range does is increase its vagueness

[WWB]

Actually, the more I think about it, the more I realize we need a six point scale. That way people cannot cop out and choose the middle--they have to say it was either above or below average.

Regarding searchability, I think we really need to look at allowing authors to attach categories to their battles. Having designed an open-ended search or two, I must say they are quite a bear. And in reality 90% of the searches are for a small subset of what you built out the search to do. Oftentimes adding such "metadata" helps alot.

This metadata could also be used for the lists, presuming they continue. Instead of just pulling randomly based on scoring, the author could choose a "primary group", such as TCP Battles, for his battle to be potentially ranked in. In addition, I think requiring at least 10 reviews to be ranked in anything is a good idea.

WWB

[Hans]

Reform is a good idea

I'd like to be able to search by reviewer, I find that certain reviewers are more believable than others. It might also be a good idea to keep track of how reviewers review, ie a guy who always give a 10-10-10-10-10-10 would reflect that in his rating score. A reviewer who rates low would also be reflected in his score.

I'm for multiple catagories (to reduce egos mainly) and a 'cabal' to officially rate scenarios while the riff-raff public can add only general comments.

Log in? Yes definitely.