GJK Posted December 16, 2003 Share Posted December 16, 2003 I say have more areas to review and then throw out the highest and lowest rating. Oh, no I don't - reviewers probably don't want more categories to review (it's a chore already to many) and throwing out higher/lower will bunch everyone in the middle again. Nobody liked my idea above, the 70-30 thing? I played with the numbers last night, and if you put it at 99-1 then ratings are adjusted within a 2 pt range up or down. Seems it would work on a 10 pt system. 0 Quote Link to comment Share on other sites More sharing options...
Berlichtingen Posted December 17, 2003 Share Posted December 17, 2003 Originally posted by GJK: Nobody liked my idea above, the 70-30 thing? To be honest, I didn't understand your 70-30 thing 0 Quote Link to comment Share on other sites More sharing options...
flammenwerfer Posted December 17, 2003 Share Posted December 17, 2003 Originally posted by Berlichtingen: </font><blockquote>quote:</font><hr />Originally posted by Flammenwerfer: A 3.2 score is meaningless, in my opinion, because 90% of the battles will have that score. It will push all the battles into the middle. The scores will look like a snake that swallowed a large rat and not a rainbow of opinions.Uh... it should look like a snake that swallowed a large rat. The average should be the majority. All increasing the range does is increase its vagueness </font> 0 Quote Link to comment Share on other sites More sharing options...
Spookster Posted December 21, 2003 Share Posted December 21, 2003 Any concensus? 0 Quote Link to comment Share on other sites More sharing options...
flammenwerfer Posted December 22, 2003 Share Posted December 22, 2003 If anyone is still listening, this is my proposal- 10- Exceptional 9- Very Good 8- Good 7- Above Average 6- Average 5- Mediocre 4- Poor 3- Very Poor An overall score with written comments for scenario reviews. Also, the first step in the review process would be this Question: How did you first play the Scenario? 1- Axis vs. AI 2- Allied vs. AI 2- 2 player If the above response does not match the Designers recomendation(s), then the numeric review will not count, only the written comments. This info will also be used to create top 10 list for AI/2 player battles. [ December 22, 2003, 06:03 PM: Message edited by: Flammenwerfer ] 0 Quote Link to comment Share on other sites More sharing options...
Berlichtingen Posted December 23, 2003 Share Posted December 23, 2003 I still maintain that anything larger than a five point scale is unnessessary fluff 0 Quote Link to comment Share on other sites More sharing options...
flammenwerfer Posted December 23, 2003 Share Posted December 23, 2003 Originally posted by Berlichtingen: I still maintain that anything larger than a five point scale is unnessessary fluff Would you please elaborate. A broader scale provides more discrimination and the ability to assess differences. What provides more information- Do you like? Yes or No Vs. Like Very Much, Like Somewhat, Do Not Like). [ December 22, 2003, 06:36 PM: Message edited by: Flammenwerfer ] 0 Quote Link to comment Share on other sites More sharing options...
Spookster Posted December 23, 2003 Share Posted December 23, 2003 Originally posted by Flammenwerfer: </font><blockquote>quote:</font><hr />Originally posted by Berlichtingen: I still maintain that anything larger than a five point scale is unnessessary fluff Would you please elaborate. A broader scale provides more discrimination and the ability to assess differences. What provides more information- Do you like? Yes or No Vs. Like Very Much, Like Somewhat, Do Not Like). </font> 0 Quote Link to comment Share on other sites More sharing options...
Mad Russian Posted December 23, 2003 Share Posted December 23, 2003 Originally posted by Spookster: I think the simpler the system the better. We need more reviews. To get that done an easier, simpler system, would work better. IMHO. I agree whether it is a five, three or two point system would be better than what we have now. KISS is the way to go. Panther Commander 0 Quote Link to comment Share on other sites More sharing options...
flammenwerfer Posted December 23, 2003 Share Posted December 23, 2003 I agree the lower end of the spectrum is not very important, discriminating between the abysmal and the poor is not the issue, what is important is determing what is simply good from what is great. A 5 point system, in effect is a 3 point system and will lump everything together. Keep in mind a scenarios 'score' is an aggregate of several reviews. The important point is not the quality of the review system, it is the quality of the reviewer. Wrong. The quality of the reviewer cannot be controlled for, our task is the system. [ December 22, 2003, 08:55 PM: Message edited by: Flammenwerfer ] 0 Quote Link to comment Share on other sites More sharing options...
Michael Dorosh Posted December 23, 2003 Author Share Posted December 23, 2003 Originally posted by Flammenwerfer: If anyone is still listening, this is my proposal- 10- Exceptional 9- Very Good 8- Good 7- Above Average 6- Average 5- Mediocre 4- Poor 3- Very Poor What would the difference be between "mediocre","poor" and "very poor". You don't say. This system means nothing without some sort of rationale. What is the difference between "good" and "above average"? They mean exactly the same thing. At least the 5/6 point system can have a bit of definition to it. 0 Quote Link to comment Share on other sites More sharing options...
flammenwerfer Posted December 23, 2003 Share Posted December 23, 2003 What would the difference be between "mediocre","poor" and "very poor". You don't say. This system means nothing without some sort of rationale. What is the difference between "good" and "above average"? They mean exactly the same thing The rationale is very clear. If you're troubled by the terms, then just go by the numbers. Would you rather go out with a girl who is a '9' or a '6'. [ January 10, 2004, 03:08 PM: Message edited by: Flammenwerfer ] 0 Quote Link to comment Share on other sites More sharing options...
Berlichtingen Posted December 23, 2003 Share Posted December 23, 2003 Originally posted by Flammenwerfer: A broader scale provides more discrimination and the ability to assess differences.Bull. That is only true with things that can be measured. There is no 'yard stick' for messuring scenario quality. In reality, all scenarios fall into THREE categories... Good, So-So, Bad. A five point scale allows for some discrimination while still keeping the values at a point that just about anyone can understand (Wow! I really liked it!...5, Hmm... better'n most...4, Average...3, Not really all that fun...2, Damn! I wouldn't force my mother-in-law to play this one!...1). With the ten point scale, you just get confusion (Wow! This one's great...9? 10?, This one's so-so... 5? 6? 7?). I have not seen any evidence that a 10 point scale provides ANY benefit at all. 0 Quote Link to comment Share on other sites More sharing options...
Berlichtingen Posted December 23, 2003 Share Posted December 23, 2003 Originally posted by Flammenwerfer: ...what is important is determing what is simply good from what is great.4 5 0 Quote Link to comment Share on other sites More sharing options...
Berlichtingen Posted December 23, 2003 Share Posted December 23, 2003 Originally posted by Spookster: I agree that less is more when it comes to simplifying the rating scale. As it stands now, the 1-10 scale has become a 6-10 scale (6=poor, 7=average, 8=good, 9=very good, 10=great).Oh very well put! 0 Quote Link to comment Share on other sites More sharing options...
Michael Dorosh Posted December 23, 2003 Author Share Posted December 23, 2003 Originally posted by Flammenwerfer: </font><blockquote>quote:</font><hr />What would the difference be between "mediocre","poor" and "very poor". You don't say. This system means nothing without some sort of rationale. What is the difference between "good" and "above average"? They mean exactly the same thing The rationale is very clear. If you're troubled by the terms, than just go by the numbers. Would you rather go out with a girl who is a '9' or a '6'. </font> 0 Quote Link to comment Share on other sites More sharing options...
Spookster Posted December 23, 2003 Share Posted December 23, 2003 Originally posted by Flammenwerfer: I agree the lower end of the spectrum is not very important, discriminating between the abysmal and the poor is not the issue, what is important is determing what is simply good from what is great. A 5 point system, in effect is a 3 point system and will lump everything together. Keep in mind a scenarios 'score' is an aggregate of several reviews. </font><blockquote>quote:</font><hr /> The important point is not the quality of the review system, it is the quality of the reviewer. Wrong. The quality of the reviewer cannot be controlled for, our task is the system. </font> 0 Quote Link to comment Share on other sites More sharing options...
flammenwerfer Posted December 23, 2003 Share Posted December 23, 2003 [You may want to rethink your argument that "the quality of the reviewer cannot be controlled for." (Such procedures happen all the time in the social sciences.) Sorry to be so snippy, but when someone writes "wrong" without half a clue, it bothers my tender ego.] Spookster I apologize for the snippiness. I meant to say, incorrect IMHO. Though I stand by what I said. How do we define A 'quality' reviewer? One that gives high scores for the battles you like or writes eloquent remarks, or writes a review every weekend. You say one that is more dedicated..Trolls are very dedicated, with a complex system you'll get the very motivated from both sides of the spectrum. Let's avoid this can of worms and just say that all reviews and opinions are valid. I agree with you that we need to increase the number of reviews made at the SD. I dont think a 5 point scale vs a 7 or even 9 point scale will increase the number of reviews, as you assert. quote: -------------------------------------------------------------------------------- Originally posted by Flammenwerfer: A broader scale provides more discrimination and the ability to assess differences. -------------------------------------------------------------------------------- Bull. That is only true with things that can be measured. There is no 'yard stick' for messuring scenario quality. In reality, all scenarios fall into THREE categories I disagree, imho. Peoples' subjective opinions regarding goods and services are constanly measured in the marketplace. If as you say 1/3 of all scenarios are good, why not give the dler more information as to which scen. where rated at the head of the pack. These anchored scales are easy to interpret and tend to be used by people in the same way. From my days in marketing research we would use various scales like these: Strongly Disagree-9-8-7-6-5-4-3-2-1-Strongly Agree Strongly Like 7-8-6-5-4-3-2-1 Strongly Dislike Exceptional 5-4-3-2-1 Poor Various types of scales are used. There is no wright or wrong in this discussion, just preferences. I say we compromise and go with the 7. [ December 23, 2003, 03:15 PM: Message edited by: Flammenwerfer ] 0 Quote Link to comment Share on other sites More sharing options...
Spookster Posted December 23, 2003 Share Posted December 23, 2003 Originally posted by Flammenwerfer: [QB] Spookster "I apologize for the snippiness. I meant to say, incorrect IMHO. Though I stand by what I said." Apology accepted. "How do we define A 'quality' reviewer? One that gives high scores for the battles you like or writes eloquent remarks, or writes a review every weekend. You say one that is more dedicated..Trolls are very dedicated, with a complex system you'll get the very motivated from both sides of the spectrum. Let's avoid this can of worms and just say that all reviews and opinions are valid." By quality, I mean someone who takes the time to respond in a thoughtful fashion (be he an idiot or Einstein.) Surely, you agree that a review that uses specifics is more useful than a "gosh, I liked it, but you suck" review? A more complex system (all things being equal) requires more dedication. Using extremes, how many people do you suppose would review scenarios at the SD IF they were required to write a 5 page essay? And...if one person did so for each scenario, would it be worth it? "I agree with you that we need to increase the number of reviews made at the SD. I dont think a 5 point scale vs a 7 or even 9 point scale will increase the number of reviews, as you assert." Here, I may agree with you. If we had a sample of potential reviewers that approached infinity, I'd disagree. At the margin, a five point scale's simplicity may attract more reviews. But if we are talking about a few dozen possible reviewers a month, if that, it may not matter. But I honestly don't know. As far as the 1-10 scale adding more variation for the reviewer, I agree that in a perfect world it does, but in reality, people limit themselves to 6-10, as I do. Will they limit themselves to 3-5 in a 1-5 scale? If they do, then a change from the 1-10 scale is not necessary. I just don't know. So do should we make a change? 0 Quote Link to comment Share on other sites More sharing options...
dugfromthearth Posted December 24, 2003 Share Posted December 24, 2003 a 3 point system is all that is required: 3 really liked it 2 it was okay 1 didn't like it like grade inflation, a rating system only works if everyone uses the same system. If you have a 1-10 scale and some people see a 7 as "okay" and others see 5 as "okay" then you might have a rating spread but it is meaningless. 0 Quote Link to comment Share on other sites More sharing options...
Michael Dorosh Posted December 24, 2003 Author Share Posted December 24, 2003 Originally posted by dugfromthearth: like grade inflation, a rating system only works if everyone uses the same system. If you have a 1-10 scale and some people see a 7 as "okay" and others see 5 as "okay" then you might have a rating spread but it is meaningless. I've read that somewhere before.... :confused: 0 Quote Link to comment Share on other sites More sharing options...
WWB Posted December 24, 2003 Share Posted December 24, 2003 Originally posted by dugfromthearth: a 3 point system is all that is required: 3 really liked it 2 it was okay 1 didn't like it like grade inflation, a rating system only works if everyone uses the same system. If you have a 1-10 scale and some people see a 7 as "okay" and others see 5 as "okay" then you might have a rating spread but it is meaningless. I am liking this but I vote for a 4 point scale. That way people cannot cop out and pick 2. Either you liked it or you did not, completely neutral should not be an option. WWB 0 Quote Link to comment Share on other sites More sharing options...
Michael Dorosh Posted January 9, 2004 Author Share Posted January 9, 2004 Any word on this? 0 Quote Link to comment Share on other sites More sharing options...
Sergei Posted January 9, 2004 Share Posted January 9, 2004 How about instead of giving a number or writing a review, people could just draw a picture depicting their feelings after playing the scenario? 0 Quote Link to comment Share on other sites More sharing options...
Andreas Posted January 9, 2004 Share Posted January 9, 2004 Originally posted by Sergei: How about instead of giving a number or writing a review, people could just draw a picture depicting their feelings after playing the scenario? 0 Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.