Korsun Relief Scenario

[ASL Veteran]

Originally posted by Spook:

But where I go with Andreas on this is that there is a world of difference to labeling a specific opinion as "pathetic" versus applying that term to another poster on a direct, personal level, something we can all be mindful of.

Perhaps we should be mindful of calling people spineless cowards too.

I am also not naive enough to believe that his intent in bringing up ASL was simply to discuss the merits of ASL scenarios within the context of this discussion.

[ December 11, 2002, 02:10 PM: Message edited by: ASL Veteran ]

[rune]

Matti brings up a good point. IF it was 1 bad review and 50 good ones, things average out. Since that is far from the case, a bad review carrys a lot of weigh. Why is this a concern? Because if a human error is made, a scenario gets panned.

The problems as I see it, and this is only my opinion:

1. Too few reviews.

2. The 0-9 system is not understood by the reviewers.

3. How many people know if you don't want to rate something, give it a 0.

4. As Dan said, too many people using a part of the review to balance out the score.

5. How many people understand that a 5 is an average review?

6. Too many people not taking the time to understand a scenario and don't ask. A fictional scenario is exactly that. A Night scenario is exactly that. Etc.

I think I make a good case on how not liking 1 part of 1 scenario can cause the entire rating to be a low lower then what it should be. So...how do we fix?

Rune

[Spook]

Originally posted by ASL Veteran:

Perhaps we should be mindful of calling people spineless cowards too.

True, thus the basis of my "all" qualifier. But I will go on a limb and state, IMO, that Andreas wouldn't had written the following:

You crack me up just like this Cpt. Kloss chap - is there something you want to say, or are you just a little coward ASL Vet?

....if he wasn't branded as "pathetic" first.

[Cpt Kernow]

Whow

everyone calm down and chill.

Andreas, dont take this the wrong way (please) but to this neutral observer your posts seem the most aggresive.(Using the word liar is most definately Schroener like)

Lets keep this on track which is a dialogue basicaly about how reviewers can/do score scenarios at the scen depot.

If it was to go to the five star system, well I think this would excerbate (grrrr) the problems made clear by Rune. E.g. scenario skipping due to low marks. I assume the five stars would have to be an aggregate of reviews. Theres no reason to assume that a few bad reviews could ruin a star rating and thus lead to skipping. If it wasnt an aggregate who would determine the intial star rating? The scen designer?

Hardly ideal.

I base my arguement on the following and would apprecaite comments on these points.

1. As a downloader I am happy with 95% of the scenarios I download from the scenario depot.These downloads are based on the comments of other reviewers. If I was "skipping" other scenarios would only be relevant in this context if I was skipping "good" to download "bad" scenarios. This is not the case.

Therefore on this count we must consider the scenario depot a success.

2. I can only conjecture on the experience of the uploader (scen designer) but given the huge number of scens uploaded to the scen depot I can only conclude that general satisfaction is the experience of the uploader (scen designer.)

Therefore on this count the scen depot must also be considered a success.

Therefore from the perspective of uploader and downloader the scenario depot is a success.

To use a charming american aphorism "if it aint broke why fix it."

Kernow.

[ December 11, 2002, 02:17 PM: Message edited by: Cpt Kernow ]

[ASL Veteran]

Originally posted by Spook:

if he wasn't branded as "pathetic" first.[/QB]

His intent in bringing up ASL was not to discuss it's merits within the context of the discussion. If it was, he would have put forward a more insightful analysis of the various flaws inherent within that genre of scenarios rather than the cavalier remark that he chose to make instead. I feel that my response was both appropriate and justified.

[Andreas]

Originally posted by rune:

Matti brings up a good point. IF it was 1 bad review and 50 good ones, things average out. Since that is far from the case, a bad review carrys a lot of weigh. Why is this a concern? Because if a human error is made, a scenario gets panned.

The problems as I see it, and this is only my opinion:

1. Too few reviews.

2. The 0-9 system is not understood by the reviewers.

3. How many people know if you don't want to rate something, give it a 0.

4. As Dan said, too many people using a part of the review to balance out the score.

5. How many people understand that a 5 is an average review?

6. Too many people not taking the time to understand a scenario and don't ask. A fictional scenario is exactly that. A Night scenario is exactly that. Etc.

I think I make a good case on how not liking 1 part of 1 scenario can cause the entire rating to be a low lower then what it should be. So...how do we fix?

Rune

Rune, FWIW, I fully agree with the assessment. ISTR that when the Depot first came out there was some discussion about ratings too.

Regarding the point that this is just a discussion started by some designers who can not hack bad reviews - with a 1-5 system, it will still be possible for anyone to give a bad review if they so desire. They give it a '1'. Introducing a 'fun' rating, will also increase the possibility to give a bad review - just rate it '1'. Some reviewers (e.g. Big Dog) have used their own 1-5 rating system (the famous x out of 5 WOOFs) , showing that they seemed to feel a need for a different system.

If nothing else, I thing my little run around the block with ASL Vet has shown that there is a wide variety of opinions on what constitutes good scenarios. I am sure I would enjoy rating those four ASL scenarios '1' for fun, in the same way I suspect him to enjoy rating most of mine '1' for replayability.

This discussion in my mind is about increasing the options, and making the reviews more relevant to both the designers and the players. The current mathematical average is just that, a mathematical average. I would not be surprised if it actually forces reviewers to give inflated scores on some aspects in scenarios that they enjoyed, and want to mark highly (if e.g. they had a lot of fun and want to recommend it, but the briefing was not too good). Thus a case can be made that the mathematical average has the potential to distort the individual score. Which in my mind is another reason to do away with it.

My suggestion would be a hand-entered overall rating, and an additional rating for 'fun'. I would also restrict ratings from 1-5, not the current 1-10, which seems to turn into a 5-10 most of the time anyway. I think 1-10 gives an illusion of exactitude that simply does not exist. When we review tenders at work, we just have a 0-3 rating, and that has been enough.

I would not apply any of this to past scenarios, sodd comparability, that is just extra work.

[rune]

Cpt,

Because unfortuantely not everyone is like you. Look at my CMBO scenarios. The ones with the highest rating got the most downloads [over 2000] Yet, another scenario, rated a point lower has only 200 downloads. To say the points don't make a difference would be in error.

Since I named at least six scenario designers who DO feel there is a problem, then maybe there is a problem. Also members of Boot and Tracks and member of Der Kessel have problems with it too. Not everyone, but enough for me to take notice. I could just walk away from the depot....but why punish Admiral and Bigdog for the good they are trying to do?

Yes, a 1-5 average WOULD help overall, especially if the scores were not averaged, but made the user read the comments. Just as there are people who download everything I make, and the people who won't touch a scenario I make, there are people on the fence trying to decide. Give them the proper tools to decide.

Rune

[Andreas]

Originally posted by Cpt Kernow:

2. I can only conjecture on the experience of the uploader (scen designer) but given the huge number of scens uploaded to the scen depot I can only conclude that general satisfaction is the experience of the uploader (scen designer.)

Therefore on this count the scen depot must also be considered a success.

Therefore from the perspective of uploader and downloader the scenario depot is a success.

To use a charming american aphorism "if it aint broke why fix it."

Kernow.

Kernow, I think you maybe confusing two issues here. The real service of the Depot to those who upload directly to it is that it gives them an instant 'client' base for their work. The reviews are probably just an extra bonus for many of them. I am fairly convinced you would see the same number of uploads if there was no review feature at all.

For designers in groups like 'Die Sturmgruppe' and 'B&T' this is not an issue at all, we are more concerned about the reviews. With 'concerned' being the keyword here for a few of us (but not all).

In an ideal world, an upload would happen for both reasons.

[Spook]

Originally posted by rune:

The problems as I see it, and this is only my opinion:

1. Too few reviews.

2. The 0-9 system is not understood by the reviewers.

3. How many people know if you don't want to rate something, give it a 0.

4. As Dan said, too many people using a part of the review to balance out the score.

5. How many people understand that a 5 is an average review?

6. Too many people not taking the time to understand a scenario and don't ask. A fictional scenario is exactly that. A Night scenario is exactly that. Etc.

I think I make a good case on how not liking 1 part of 1 scenario can cause the entire rating to be a low lower then what it should be. So...how do we fix?

Rune

To #1: Human nature is given to downloads first, feedback second. If more feedback in scenario reviews is desired, then it should be stated more plainly, and more obviously, than what I could make out presently at the site. Something like, "Please provide feedback as a means of showing appreciation of someone else's effort to provide you a scenario to play."

#2: Most people can figure that the higher the number, the better the rating, in most cases, but a default explanation to numerical scoring wouldn't hurt to "idiot-proof" the scoring application. The alternate is to go to a "star" system of 1-5 stars, allowing half-stars, as this is applied pretty consistently at movie and book review sites.

#6: Again, if a reviewer is desired to uphold some given standard, the review portion of the website has to have it explained explicitly as to what these standards should be. For example, if an overview of the scenario introduction notes is desired prior to reviewing the "historicity" of the scenario, then it should state so.

[Scott B]

Originally posted by rune:

The problems as I see it, and this is only my opinion:

1. Too few reviews.

2. The 0-9 system is not understood by the reviewers.

3. How many people know if you don't want to rate something, give it a 0.

4. As Dan said, too many people using a part of the review to balance out the score.

5. How many people understand that a 5 is an average review?

6. Too many people not taking the time to understand a scenario and don't ask. A fictional scenario is exactly that. A Night scenario is exactly that. Etc.

I think I make a good case on how not liking 1 part of 1 scenario can cause the entire rating to be a low lower then what it should be. So...how do we fix?

I agree, and I think that the above are some important points to consider in seeking a new rating system. I have a few issues with the system; while it's good, it could be better. I would like to see the numeric scoring system revised - and the 'overall score' that Rune and others have proposed is a step in the right direction.

While I personally would welcome a move to all text reviews of scenarios - it's the only way I think it could be completely fair - I recognize that some people want numbers, even though they are sometimes meaningless, subjective, and/or irrelevant to the designer's intent. I like the idea of an area to list pros and cons of a scenario, and then a text description of what is liked and not liked. And when it gets down to it I'd rather rate a historical scenario primarily on its historical accuracy; I'd rather rate a two player battle primarily on its force balance, etc. - that's a sliding scale that is difficult to accomodate in any format.

Any improvements to the current rating system would be welcome, if possible, and discussion (such as the above, where it has occurred) is very useful toward this end.

Scott

[Cpt Kernow]

Rune.

Its simple.

If the current review process means I download good scenarios then the current process works.

Yes I do download good scenarios.

Are scenarios designers unhappy with the current system.

Not evinced by the huge amount of scenarios uploaded.

In short, it dosnt matter to me what system is used as long as the system:

1.Means I download good scenarios.

2.Scenario desiginers continue to post scenarios.

Current system fullfils those crtiteria.

Kernow.

[ December 11, 2002, 02:43 PM: Message edited by: Cpt Kernow ]

[rune]

Spook, I can agree with your points BUT, if I can't get someone to read the briefing correctly, how the heck can I be sure they read the depot instructions? Someone posted a review and complained about tanks that weren't even in the scenario! [i think it was one of Dan's]. I know of a lot of players that don't want to read the details, just as there are those that want to know each and every name of the soldiers on the battlefield. Too many different styles...

It is why I think we go with an overall fun rateing. Do we need 1/2 stars/tanks/dead scenario designers rateings? Not really...unless that is what everyone wants to do.

As I said, wish I knew the answer...but at least we are talking about it...and you see other scenario designers think the system could be revised.

Rune

[Iconoclast]

Originally posted by rune:

One addition to this...on a 0-9 scale system, 5 should be the average. How many people out there won't look at a scenario below 7? Below 8? How many people really understand a 5 is average?

As I said, wish I knew the answer. Just can't help but feel the system, as is, is flawed.

Rune

Rune,

It's funny! Just to show you how complicated this whole numbers thing can be, I'm pretty sure we're in raging agreement that we both believe the current rating scheme to be FUBAR-- yet it seems we're each interpreting it differently! I'd have said the rating scale is 1-10, with 0 being Not Applicable (as it clearly states on the scenario review form). When even folks who really care about the numbers and the ratings-- either because they are scenario designers trying to figure out what the hell the gaming public thinks about their efforts, or because they're reflexive number-crunchers (my hand goes up in confession)-- can't seem to agree about what a scale means, it becomes next to meaningless. This is on top of the scale being subject to distortion because of the small N, sample skew (i.e, the only ones who care enough to rate are at the two tails of the bell curve on opinion), etc.

Part of the confusion about the scale is whether folks are using 1-10 as a genuine bell curve with 5 being average, or equating it to academic grades, where 70% (more or less) equals "F" and a failure.

Here's a proposal, Rune. I'm nearly finished playing a double-blind PBEM game of another one your designs-- a Battle of Minors. I have some specific "issues" (OK, criticisms) I'd like to raise in my feedback. You know my position on scenario reviews by now-- that if something doesn't make sense or feel "right" to this 30+ year grog, then in my mind it is fair game to point these things out in review comments *without* asking the designer to explain them first, since the odds are that John Q. Gamer won't have "gotten it" either. (That's the fundamental difference between FEEDBACK and a REVIEW in my mind.)

What I would couple this with is a) providing a fairly lengthy assessment of WHY I found these elements problematic, I will provide NO NUMERICAL RATINGS at all, just comments, and c) I will include things I *liked* and do a summation. (Not sure if I'll try an "X stars out of 5" approach or not-- I doubt it.) The only way a prospective player of the scenario would see my opinions is if he scrolls down the reviews column after having clicked on the scenario. This way there's no risk that I artificially skew your numbers, yet I say my piece. Sound worth trying to you, since we're both not happy with the status quo?

Jim

[ December 11, 2002, 03:00 PM: Message edited by: Iconoclast ]

[Cpt Kernow]

I am here basically repeating my post from above but from a different angle.

1. Do bad reviews lead to downloaders choosing bad reviews.

not in my exprience.Happy with 95% of downloads. 5% unhappy with not attributable to bad reviews but in fact to overly good ones leading me to choose a scenario I think is foobar.

2. Is the presence of bad reviews acting as a disincentive to scenario designers uploading there scenario's.

Again not evinced by GIAGANTIC amount of scenarios uploaded to the depot.

Conclusion:

Bad reviews are not a problem.

Current system works.

Cheers.

Kernow

[flammenwerfer]

What about a survey style system with the numerical quant. calculated internally for each of the categories- Briefing, Map, Force Balance, Play vs AI, PBEm Play.

Very Poor(1)- Poor(2) - Average(3) - Good(4) - Very Good(5) - Exceptional(6)

Something like this...

[ December 11, 2002, 02:54 PM: Message edited by: Flammenwerfer ]

[Spook]

Originally posted by rune:

Spook, I can agree with your points BUT, if I can't get someone to read the briefing correctly, how the heck can I be sure they read the depot instructions? Someone posted a review and complained about tanks that weren't even in the scenario! [i think it was one of Dan's]. I know of a lot of players that don't want to read the details, just as there are those that want to know each and every name of the soldiers on the battlefield. Too many different styles...

True, and some "outlier" viewpoints that don't really help to give compelling reviews are probably unavoidable.

I will, however, add this as a personal perspective. I first visited the noted site only just today. And on this first-entry basis, I felt as that I had to "wade around" a bit just to figure out how to post a review. And I am stating this as someone predisposed to offer a review, let alone someone just looking for downloads.

IMO, the site needs some improvement on the "navigation" aspect, but I will have to mull a bit before offering suggestions. Is it you who is hosting it, or others?

[rune]

Jim,

Do so here. Would love to see feedback. As for the 30 years of wargaming experience, I think I got you by a few years. Also, would give me an example of how you think scenario ratings should go.

Cpt.,

For you, there is no problem, for me, there is. Yes, the scenarios are there for feedback.. To make my point do I have to stop sending scenarios there? Should all of us who think there is a problem quit posting scenarios? What good would that serve? Would I get better feedback? Would no feedback make me a better designer? Nope. again, the case where a scenario is downloaded 2000 times vs a 1 point lower score 200 times shows me the ratings do impact the downloads. The depot is not just a place to upload and download, I could place the scenarios on boot and tracks and forget the depot...but the point is I don't get accurate feedback, good and bad, that way.

Rune

[Ken Talley]

I think everyone can agree that whether a scenario is fun or not is the key to the success of a scenario.

The current system adds a lot of good info of the parts of a scenario. It is important to know whether a scenario has replayability, a good map, etc. I would keep the current system in place to provide that individualized information.

However I would replace the overall rating composed of the average of the existing ratings with a single rating of "fun". It could be a five star "fun" rating completely independent of replayability, maps, etc. Is the scenario bad, poor, average, great or superb "fun"?

When I look at the scenario, I would see the "fun" rating. Then I could look at the individual ratings of replayability, maps, etc. Then I could read the comments about why it was fun. I think most people can explain why it was fun and you can decide if it might be fun or not to you as well.

Fun is what it is all about-at least to me.

[ December 11, 2002, 03:09 PM: Message edited by: Ken Talley ]

[rune]

Spook,

Admiral Keth and Bigdog's site. I try not to impose on them as they do this out of the goodness of their hearts. It is why i brought the discussion here...if we can get the majority to agree on something, then maybe we could then approach them.

Jim,

Forgot to comment....Yes, we both say the stats are flawed but from different angle...and I think that can be added to the list of problems. To me a 0-9 scale the average is 5, to you it is 70%. Now how many other people rating it different ways?

Also we will have to agree to disagree on feedback/review. You say you review it as if others would not get the point either. My reply is how do you know they didn't get the point? What happens to the times when the majority got it, and one or two didn't? It is why I still think the designer could be asked. If i reply yes, i had a lot of complaints on that, you know it is an issue, if i say it was the first i heard of it, then maybe the majority got the point or just didn't care. However, this requires trust that I would respond honestly, and the other scenario designers as well. Hmm... another quandry...

Rune

[ December 11, 2002, 03:16 PM: Message edited by: rune ]

[Marlow]

Originally posted by Cpt Kernow:

Are scenarios designers unhappy with the current system.

Not evinced by the huge amount of scenarios uploaded.

I for one no longer post scenarios. It isn't so much the current format, which I don't really mind, but rather the lack of common courtesy of those who download, but don’t review the battles (despite frequent requests from the designers). These reviews provide at least some of the satisfaction of creating the battles. Obviously there is no requirement that people review the scenarios they play, but if the designer puts forth the effort to make it, the right thing to do is to take a couple of minutes to review it. I uploaded 4 battles for CMBO, but only ever got comments on 2 of them. I have a few finished CMBO battles, and a couple of historical/semi-historical battles for CMBB about Russian airborne operations during the Moscow counteroffensive, but given the lack of response, I haven’t had much incentive to get around to writing the final briefing, and uploading it.

[Conscript Bagger]

I've followed this thread with interest, and now I'm going to help muddy the waters.

Rune - I respectfully suggest that you not assume that a one-point difference in rating is the cause of a tenfold difference in downloads. I have no idea which two scenarios you're talking about, but perhaps the "ignored" one had conditions that scared off casual players (time, date, weather, force mix, PBEM suitability, fictional vs. historic), or perhaps it was a general situation that folks had already seen in other scenarios and they were looking for something fresh (that's not a reflection on your scenario, just a relative thing - when I've had pizza six meals in a row, a burger starts to sound good). Granted, the lower rating probably didn't help matters any, but my point is that many factors were probably at work.

Spook - isn't a five-star system allowing half-stars really a ten-point system? But your mention of movie reviews goes along with what I was about to suggest for the Depot: that review scores not be averaged until there is a minimum number of reviews (I think IMDb does this). That would help alleviate the problem of one bad score torpedoing the average.

Another suggestion I liked was made by more than one person, and that was to provide the separate, hand-entered "Reviewer's Gut Reaction" score. I would suggest that, the Depot add this category and average only this score for the overall rating that is displayed on the scenario list page (besides requiring a minimum number of reviews before averaging). On the scenario detail page, I would still like to see the averaged scores in the current categories (whether they stay with the ten-point system or not) and I always read the reviewers' comments as well.

[rune]

Offwhite, while that is a possibility for some, it would not explain a 10 fold difference. Go onto the depot and look at CMBO scenarios. Search by my name. Go through the scenarios, see how many were rated, how many were, look at the scores and look at the amount fo downloads. I think you will be surprised by the trend.

Ken and Offwhite, both are good possibilities. I think replayability should be junked. Not sure on some of the others...I had map ratings of 5, even though the map is compeltely accurate. Blame God, or the Russians, as I didn't create the terrain, just represented it. A Battle of Minors and my tractor works are both good examples. Both are based on accurate maps. So, it still will not prevent that sort of thing. I got maps sent to me from Warphead [bless him], and wwb and James went into the national archives and i paid for copies of maps for the designers out of my pocket. So no, it was NOT a quick battle modified map.

Rune

[Ken Talley]

I think replayability is an important category. I know a couple of people have indicated that scenarios aren't meant for replays. Philosophical difference, but I disagree strongly. A map with a variety of approaches provides multiple offensive strategies. Have you ever wondered why you didn't put your HMG in that heavy building or that ATG in that little patch of woods? Wow, I wish I had structured my defense differently. Certain map types and force structures can provide multiple playings. I know all of my scenarios are specifically designed for replayability. I think the "Jagermeister" scenario is a good example of a scenario with replayability. I can see multiple choices as to launching both infantry and armor attacks into the town.

I think it is very difficult in a purely historical scenario to design for replayability when the terrain is absolutely known. But in either semi-historical or fictional, flexibility exists for ensuring the terrain can allow multiple strategies and thus replayability.

As a category, perhaps it isn't as important as balance or PBEM suitability but does provide good information.

[JPS]

I would second Offwhite's suggestion.

In addition, I would specifically like to keep the "replayability" as one subcriteria, as that is something I seek and value in _some_ scenarios.

[Andreas]

Originally posted by Ken Talley:

I think replayability is an important category. I know a couple of people have indicated that scenarios aren't meant for replays. Philosophical difference, but I disagree strongly. A map with a variety of approaches provides multiple offensive strategies. Have you ever wondered why you didn't put your HMG in that heavy building or that ATG in that little patch of woods? Wow, I wish I had structured my defense differently. Certain map types and force structures can provide multiple playings. I know all of my scenarios are specifically designed for replayability. I think the "Jagermeister" scenario is a good example of a scenario with replayability. I can see multiple choices as to launching both infantry and armor attacks into the town.

I think it is very difficult in a purely historical scenario to design for replayability when the terrain is absolutely known. But in either semi-historical or fictional, flexibility exists for ensuring the terrain can allow multiple strategies and thus replayability.

As a category, perhaps it isn't as important as balance or PBEM suitability but does provide good information.

Ken, I think we could have a good discussion about this. E.g. on what you would do with smaller scenarios, or with any of the points I brought up earlier, or the absence of FOW in general on replay.

That is not the point however - pretty much the same arguments could be had about almost any aspect of the rating system.

Nobody on the side of those arguing for some change asks that any category be dropped, on the contrary, the argument goes for increasing the available categories. If, say, I design a scenario with no regard to replayability (as I usually do, because FOW is one of the most important variables in my design approach), and say so in the briefing, do you think it would be fair that a low replayability rating should drag down the scenario rating overall, regardless of how much you enjoyed (or did not enjoy) it otherwise? Say you enjoyed it, but are peeved it is not replayable, would the rating system offer you a better service if it allowed you to give an overall 4 that is not affected by a 1 for replayability?

Extending the rating system in such a way would improve it somewhat IMO. It would not be perfect however, since there is no perfect rating system.

Fundamentally though the main flaw of the Depot is one that is very difficult to address - a lack of reviews, as Mace pointed out. I fully agree with him, and have come to the same conclusion as him. I am speaking as someone who is blessed (well, most of the time ) with a lot of reviews though, but I am still very frustrated about this.

The bottomline is that while the Capt. Kernow approach of 'all is reasonably well' works for many designers and players, it no longer works for me. There are not enough reviews, and that needs to change as far as I am concerned. I have tried pretty much all I can to encourage reviews (provision of the link to the Depot and my email in the briefing, linked in my sig), but to no avail. That is a real shame.

YMMV, before I am being told I am pathetic, or have eyes rolled at me again.