Jump to content

CMAK Imminent - can we fix the Scenario Depot Rating System Beforehand?


Recommended Posts

I say have more areas to review and then throw out the highest and lowest rating. Oh, no I don't - reviewers probably don't want more categories to review (it's a chore already to many) and throwing out higher/lower will bunch everyone in the middle again.

Nobody liked my idea above, the 70-30 thing? I played with the numbers last night, and if you put it at 99-1 then ratings are adjusted within a 2 pt range up or down. Seems it would work on a 10 pt system.

Link to comment
Share on other sites

  • Replies 198
  • Created
  • Last Reply

Top Posters In This Topic

Originally posted by Berlichtingen:

</font><blockquote>quote:</font><hr />Originally posted by Flammenwerfer:

A 3.2 score is meaningless, in my opinion, because 90% of the battles will have that score. It will push all the battles into the middle. The scores will look like a snake that swallowed a large rat and not a rainbow of opinions.

Uh... it should look like a snake that swallowed a large rat. The average should be the majority. All increasing the range does is increase its vagueness </font>
Link to comment
Share on other sites

If anyone is still listening, this is my proposal-

10- Exceptional

9- Very Good

8- Good

7- Above Average

6- Average

5- Mediocre

4- Poor

3- Very Poor

An overall score with written comments for scenario reviews.

Also, the first step in the review process would be this Question:

How did you first play the Scenario?

1- Axis vs. AI

2- Allied vs. AI

2- 2 player

If the above response does not match the Designers recomendation(s), then the numeric review will not count, only the written comments. This info will also be used to create top 10 list for AI/2 player battles.

[ December 22, 2003, 06:03 PM: Message edited by: Flammenwerfer ]

Link to comment
Share on other sites

Originally posted by Berlichtingen:

I still maintain that anything larger than a five point scale is unnessessary fluff

Would you please elaborate. A broader scale provides more discrimination and the ability to assess differences.

What provides more information- Do you like? Yes or No Vs. Like Very Much, Like Somewhat, Do Not Like).

[ December 22, 2003, 06:36 PM: Message edited by: Flammenwerfer ]

Link to comment
Share on other sites

Originally posted by Flammenwerfer:

</font><blockquote>quote:</font><hr />Originally posted by Berlichtingen:

I still maintain that anything larger than a five point scale is unnessessary fluff

Would you please elaborate. A broader scale provides more discrimination and the ability to assess differences.

What provides more information- Do you like? Yes or No Vs. Like Very Much, Like Somewhat, Do Not Like). </font>

Link to comment
Share on other sites

I agree the lower end of the spectrum is not very important, discriminating between the abysmal and the poor is not the issue, what is important is determing what is simply good from what is great.

A 5 point system, in effect is a 3 point system and will lump everything together. Keep in mind a scenarios 'score' is an aggregate of several reviews.

The important point is not the quality of the review system, it is the quality of the reviewer.
Wrong. The quality of the reviewer cannot be controlled for, our task is the system.

[ December 22, 2003, 08:55 PM: Message edited by: Flammenwerfer ]

Link to comment
Share on other sites

Originally posted by Flammenwerfer:

If anyone is still listening, this is my proposal-

10- Exceptional

9- Very Good

8- Good

7- Above Average

6- Average

5- Mediocre

4- Poor

3- Very Poor

What would the difference be between "mediocre","poor" and "very poor". You don't say. This system means nothing without some sort of rationale. What is the difference between "good" and "above average"? They mean exactly the same thing.

At least the 5/6 point system can have a bit of definition to it.

Link to comment
Share on other sites

What would the difference be between "mediocre","poor" and "very poor". You don't say. This system means nothing without some sort of rationale. What is the difference between "good" and "above average"? They mean exactly the same thing
The rationale is very clear. If you're troubled by the terms, then just go by the numbers. Would you rather go out with a girl who is a '9' or a '6'.

[ January 10, 2004, 03:08 PM: Message edited by: Flammenwerfer ]

Link to comment
Share on other sites

Originally posted by Flammenwerfer:

A broader scale provides more discrimination and the ability to assess differences.

Bull. That is only true with things that can be measured. There is no 'yard stick' for messuring scenario quality. In reality, all scenarios fall into THREE categories... Good, So-So, Bad. A five point scale allows for some discrimination while still keeping the values at a point that just about anyone can understand (Wow! I really liked it!...5, Hmm... better'n most...4, Average...3, Not really all that fun...2, Damn! I wouldn't force my mother-in-law to play this one!...1). With the ten point scale, you just get confusion (Wow! This one's great...9? 10?, This one's so-so... 5? 6? 7?). I have not seen any evidence that a 10 point scale provides ANY benefit at all.
Link to comment
Share on other sites

Originally posted by Flammenwerfer:

</font><blockquote>quote:</font><hr />What would the difference be between "mediocre","poor" and "very poor". You don't say. This system means nothing without some sort of rationale. What is the difference between "good" and "above average"? They mean exactly the same thing

The rationale is very clear. If you're troubled by the terms, than just go by the numbers. Would you rather go out with a girl who is a '9' or a '6'. </font>
Link to comment
Share on other sites

Originally posted by Flammenwerfer:

I agree the lower end of the spectrum is not very important, discriminating between the abysmal and the poor is not the issue, what is important is determing what is simply good from what is great.

A 5 point system, in effect is a 3 point system and will lump everything together. Keep in mind a scenarios 'score' is an aggregate of several reviews.

</font><blockquote>quote:</font><hr /> The important point is not the quality of the review system, it is the quality of the reviewer.

Wrong. The quality of the reviewer cannot be controlled for, our task is the system. </font>
Link to comment
Share on other sites

[You may want to rethink your argument that "the quality of the reviewer cannot be controlled for." (Such procedures happen all the time in the social sciences.) Sorry to be so snippy, but when someone writes "wrong" without half a clue, it bothers my tender ego.]
Spookster

I apologize for the snippiness. I meant to say, incorrect IMHO. smile.gif Though I stand by what I said.

How do we define A 'quality' reviewer? One that gives high scores for the battles you like or writes eloquent remarks, or writes a review every weekend. You say one that is more dedicated..Trolls are very dedicated, with a complex system you'll get the very motivated from both sides of the spectrum. Let's avoid this can of worms and just say that all reviews and opinions are valid.

I agree with you that we need to increase the number of reviews made at the SD. I dont think a 5 point scale vs a 7 or even 9 point scale will increase the number of reviews, as you assert.

quote:

--------------------------------------------------------------------------------

Originally posted by Flammenwerfer:

A broader scale provides more discrimination and the ability to assess differences.

--------------------------------------------------------------------------------

Bull. That is only true with things that can be measured. There is no 'yard stick' for messuring scenario quality. In reality, all scenarios fall into THREE categories

I disagree, imho. Peoples' subjective opinions regarding goods and services are constanly measured in the marketplace. If as you say 1/3 of all scenarios are good, why not give the dler more information as to which scen. where rated at the head of the pack.

These anchored scales are easy to interpret and tend to be used by people in the same way. From my days in marketing research we would use various scales like these:

Strongly Disagree-9-8-7-6-5-4-3-2-1-Strongly Agree

Strongly Like 7-8-6-5-4-3-2-1 Strongly Dislike

Exceptional 5-4-3-2-1 Poor

Various types of scales are used. There is no wright or wrong in this discussion, just preferences.

I say we compromise and go with the 7.

[ December 23, 2003, 03:15 PM: Message edited by: Flammenwerfer ]

Link to comment
Share on other sites

Originally posted by Flammenwerfer:

[QB]

Spookster

"I apologize for the snippiness. I meant to say, incorrect IMHO. smile.gif Though I stand by what I said."

Apology accepted.

"How do we define A 'quality' reviewer? One that gives high scores for the battles you like or writes eloquent remarks, or writes a review every weekend. You say one that is more dedicated..Trolls are very dedicated, with a complex system you'll get the very motivated from both sides of the spectrum. Let's avoid this can of worms and just say that all reviews and opinions are valid."

By quality, I mean someone who takes the time to respond in a thoughtful fashion (be he an idiot or Einstein.) Surely, you agree that a review that uses specifics is more useful than a "gosh, I liked it, but you suck" review?

A more complex system (all things being equal) requires more dedication. Using extremes, how many people do you suppose would review scenarios at the SD IF they were required to write a 5 page essay? And...if one person did so for each scenario, would it be worth it?

"I agree with you that we need to increase the number of reviews made at the SD. I dont think a 5 point scale vs a 7 or even 9 point scale will increase the number of reviews, as you assert."

Here, I may agree with you. If we had a sample of potential reviewers that approached infinity, I'd disagree. At the margin, a five point scale's simplicity may attract more reviews. But if we are talking about a few dozen possible reviewers a month, if that, it may not matter. But I honestly don't know.

As far as the 1-10 scale adding more variation for the reviewer, I agree that in a perfect world it does, but in reality, people limit themselves to 6-10, as I do.

Will they limit themselves to 3-5 in a 1-5 scale? If they do, then a change from the 1-10 scale is not necessary.

I just don't know. So do should we make a change?

Link to comment
Share on other sites

Originally posted by dugfromthearth:

a 3 point system is all that is required:

3 really liked it

2 it was okay

1 didn't like it

like grade inflation, a rating system only works if everyone uses the same system. If you have a 1-10 scale and some people see a 7 as "okay" and others see 5 as "okay" then you might have a rating spread but it is meaningless.

I am liking this but I vote for a 4 point scale. That way people cannot cop out and pick 2. Either you liked it or you did not, completely neutral should not be an option.

WWB

Link to comment
Share on other sites

  • 3 weeks later...

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Unfortunately, your content contains terms that we do not allow. Please edit your content to remove the highlighted words below.
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


×
×
  • Create New...