Jump to content
Sign in to follow this  
GreenAsJade

Scenario Depot Blues

Recommended Posts

Another consideration...

Authors may choose to elicit numeric only, text only, numeric and text, or no reviews.

How will text only and no review scenarios weigh against those which are being numerically rated? Should they even count, i.e., if an author wants their scenarios to be ranked against each other on the Top 10 lists, should they be compelled to use numeric ratings?

If so, this will require that there be hard-coded numeric categories (Map Quality), as well as author-defined numeric categories.

If there are hard-coded numeric categories, should they _always_ be mandatory fields on the review page, in additon to anything the author may define?

So far, the design I'm envisioning goes something like this:

* 4 hard-coded ratings for aspects that should be in _any_ scenario. Regardless of what options the author selects (Numeric Only, Text Only, Text & Numeric), these four values will always be displayed and prompted for when a reviewer posts a review. If the author elects to _not_ receive reviews, then the scenario will never be on any Top 10 List.

These ratings will be:

Playable vs Human Opponent

Playable vs. AI

Briefing Quality

Map Design

Again, zero values in any category are not included in calculations and do not count against the average score. These are the values that are used in the Top 10 Lists.

The author can define 0 to 6 additional rating categories which _are not_ included in the cumulative score. These values are for the author's evaluation of the scenario, and are never used to rate one scenario against another.

In addition, scenarios will only be compared to one another based on historicality. Thus fictional scenarios will be in their own Top 10 List of Playable vs. Human.

Share this post


Link to post
Share on other sites

Admiral...

Too bad a 1 for 1 (review for download) system couldn't have been instituted.

jw

Before a person can download a scenario have him submit a review. I am no coding guru but there must be a way.

Share this post


Link to post
Share on other sites

In my opinion, for what it's worth...

You want to make it as easy as possible for the reviewer or the reviewie.

The author can choose to include the numerical value for the review or not. Written text stays.

Readers will read all regardless, most of the time. If a designer has discluded several scores it will show as they are flagged.

Include all in the top ten lists after modification, or disclude if they have discluded X number of scores. You need 3 to be in the top 10?

The reviewer must earn the right to review, as JWX suggested. There is an accumulative average score for any reviewer, against an overall total of reviews, so readers know where they stand with any given review and the reviewer will gain credit by being evenhanded as the number of reviews grows.

I would steer clear of further disemination of scoring than you already have. The harder you make it the less people will review.

Can I ask if you get more reviews since v2 or less reviews?

[ October 22, 2004, 11:26 AM: Message edited by: Richie ]

Share this post


Link to post
Share on other sites
Originally posted by Frenchy:

Admiral...

</font><blockquote>quote:</font><hr /> Too bad a 1 for 1 (review for download) system couldn't have been instituted.

jw

Before a person can download a scenario have him submit a review. I am no coding guru but there must be a way. </font>

Share this post


Link to post
Share on other sites
Originally posted by Richie:

Can I ask if you get more reviews since v2 or less reviews?

v1 was in operation for 21 months with a total of 5879 reviews (CMBO/CMBB/CMAK battles/operations/maps) = 279.95/month

v2 has been in operation for 4 months with a total of 470 reviews = 117.5/month

Share this post


Link to post
Share on other sites

Adm - thanks so much for listening to our feedback and being prepared to contemplate change yet again!

There will be as many different system suggestions as people responding, I'm sure.

The issue of the person being reviewedm not liking reviews is not unique. Even EBay has it, and copes, with an (mostly) automated system.

Here is my suggestion...

1) Get rid of all existing numeric scales and replace them with one: recommendation for the scenario - a rating of 1 to 5.

This is really the most useful thing that you can record a number for - to help users quickly find highly recommended scenarios. This is the thing that has gone away in V2.

2) Have a check box to indicate whether the recommendation is for PBEM or AI, and make both the PBEM and AI ratings of scenarios available.

People are looking for either PBEM or AI, not both at the same time (usually).

3) Have a text review section where people are encouraged to discuss map design, play balance, briefing etc, but not give ratings to those.

SO:

When I review a scenario, I log in, click "PBEM" or "AI", select a number from 1 to 5 as recommendation rating, and optionally provide a text review.

The rating is _clearly_ subjective (so we don't have arguments about the details of how to come up with it) and the review process is made very easy for those who want to provide feedback quickly.

4) Make it so that you have to register to review.

The _designers_ can give the _reviewers_ a rating.

One rating from each designer for each reviewer - a "fairness" number, and some text to explain why. The designers can probably change their rating if things change. Everyone can see a reviewer's ratings, just like EBay. Maybe list the "Top 10 review contributors" and "Top 10 fair reviewers".

...And the average of a reviewer's rating is used to weight their recommendation rating on the overall score for every scenario they review.

Ideally, the overall recommendation calculation for each scenario that a reviewer has reviewed would be revised each time that reviewer's rating changes.

Hence repeat offenders become less and less significant in ratings, while becoming more and more obvious in their own rating page.

This moves the argument away from one bad review ... one bad review: put up with it... if they are really a troll, it will emerge.

Whadaya reckon?

GaJ.

[ October 22, 2004, 04:23 PM: Message edited by: GreenAsJade ]

Share this post


Link to post
Share on other sites

IMO the current system is good. That said, I spend most of my CM time designing/testing so I don't play scenario's made by others all that often, just don't have to time really. I think we could have the current system based on most reviews along with another list with the highest scoring reviews. Then it is up to the player, on how they want to choose. As I said to start though, the current system seems fine to me.

Share this post


Link to post
Share on other sites
Originally posted by Admiral Keth:

</font><blockquote>quote:</font><hr />Originally posted by Frenchy:

Admiral...

</font><blockquote>quote:</font><hr /> Too bad a 1 for 1 (review for download) system couldn't have been instituted.

jw

Before a person can download a scenario have him submit a review. I am no coding guru but there must be a way. </font>

Share this post


Link to post
Share on other sites
Originally posted by Pzman:

IMO the current system is good. That said, I spend most of my CM time designing/testing so I don't play scenario's made by others all that often, just don't have to time really. I think we could have the current system based on most reviews along with another list with the highest scoring reviews. Then it is up to the player, on how they want to choose. As I said to start though, the current system seems fine to me.

I don't doubt that the current system is fine for people who just want to put scenarios _in_.

It's just not as good as it used to be for people who want to quickly find good ones to take _out_.

Share this post


Link to post
Share on other sites

Gotta say I like the depot the way it is now. I think the reviews now are a lot more honest now that there is no motivation to either promote or demote scenarios to make the list. Would also think that a particualary good or bad scenario would tend to make the top 20 most reviewed list that is currently featured.

Also I would have to think that now that you are required to log in and give out your e-mail address to post a review that trolling has pretty much dwindled off (have not seen much of Tommy Tiddlywinks or whatever his name was). Before it was entirely possible for anyone to make up a name and post a review and I think that this could be one of the reasons for less reviews now.

Not sure that all folks would want to have to post a review in order to download another scenario. Sometimes I download a bunch but then only play the ones that appeal to me. Would think that this might lead to people posting bogus reviews just so they could download again.

And as for the "best scenario" lists well one mans treasure is another ones trash. And most players only play a scenario once, even though anybody who thouroughly playtests a battle or op knows a CM scenario can play out a hundred different ways. At the CM level of combat balance of a scenario often depends on the success of one or two tanks or priority units. Score a lucky hit on a critical enemy unit and suddenly people are griping about how unbalanced your scenario is. I know from playtesting ops that if you play it 50 times you are likely to see a full spread of results, just as if you were rolling dice.

Having said all that if you do decide to intitute a top 10 list perhaps the worst review or 2 could be ignored in the scoring (but not deleted). Another possibility is that when a review is posted it could be set up to not officially affect the scenario score until the author can reply to it. Then have it set up so the reviewer would have to reply again to confirm or change his review depending on their satisfaction of the authors explanation and rebuttal.

Admiral I certainly appreciate and applaud your efforts for the CM community. There is no other site like yours

Share this post


Link to post
Share on other sites
Originally posted by GreenAsJade:

I don't doubt that the current system is fine for people who just want to put scenarios _in_.

It's just not as good as it used to be for people who want to quickly find good ones to take _out_.

I don't understand you.

1) I download scenarios, and it's not harder than before; you always have to read the reviews to make sure it's not just crap praised high by someone with very different taste before you download it

2) If ratings are not comparable, and there's the chance that you'll miss a gem just because one low rating lowers the total score, then why do you want to compare the ratings?

3) You really should be playing more scenarios that haven't been reviewed!

Share this post


Link to post
Share on other sites

1) Yes, you have to read the reviews, but how many do you want to look at before you find ones saying good things? How did you choose which ones to look at?

Before, if you wanted a highly recommended scenario, you could find a set easily, then read a few reviews to finalise your choice.

2) The ratings _are_ comparable. They are just not trying to be scientifically precise.

There are some people who can't handle imprecision. Do we have to have no useful ratings at all because of those few people's limitation?

3) Sometimes I will feel like playing a scenario blind. Sometimes I will want to quickly find a food one.

With the new arrangements I can still find scenarios blind... but I can no longer quickly find a highly recommended one.

Is the problem with the word "difficult"? It's not more "difficult" in that you still just press your browswer button to download. It is, however, far more time consuming to find highly recommended scenarios.

I really miss that - it was what made it good before.

Anyone else agree, or do y'all wonder what I'm talking about, like Sergei?

GaJ.

Share this post


Link to post
Share on other sites

Would it be possible to see the entire list of scenarios for each game ranked instead of just the top 20? Or maybe the top 100, since there are 757 CMBB scenarios at this very moment.

What about a feature that shows the scenarios that have gotten the most downloads, for a set period, such as 30 or 60 days? That would be another indicator players could use to choose scenarios.

One line of reasoning, that often comes out in these threads, that puzzles me, are the comments about why cater to the scenario designers. Unless I am mistaken, and I could be, that is why everybody goes to the Sceanrio Depot. To download the scenarios they have uploaded. If you don't make them happy there won't be anything to get all upset over. Now will there? :confused:

Just a puzzle that continues to pop up now and again.

Good Hunting.

Share this post


Link to post
Share on other sites

Yes it is for downloading, but its not a popularity contest either. I like the idea about the top 100 or even top 50 or something along those lines.

As for number ratings why not just have each players rating standing on its own, like now but with numbers rather than a graphic because it's same thing really. Some may like the graphic others the number, why not have both?

Share this post


Link to post
Share on other sites
Originally posted by GreenAsJade:

Adm - thanks so much for listening to our feedback and being prepared to contemplate change yet again!

There will be as many different system suggestions as people responding, I'm sure.

The issue of the person being reviewedm not liking reviews is not unique. Even EBay has it, and copes, with an (mostly) automated system.

Here is my suggestion...

1) Get rid of all existing numeric scales and replace them with one: recommendation for the scenario - a rating of 1 to 5.

This is really the most useful thing that you can record a number for - to help users quickly find highly recommended scenarios. This is the thing that has gone away in V2.

2) Have a check box to indicate whether the recommendation is for PBEM or AI, and make both the PBEM and AI ratings of scenarios available.

People are looking for either PBEM or AI, not both at the same time (usually).

3) Have a text review section where people are encouraged to discuss map design, play balance, briefing etc, but not give ratings to those.

SO:

When I review a scenario, I log in, click "PBEM" or "AI", select a number from 1 to 5 as recommendation rating, and optionally provide a text review.

The rating is _clearly_ subjective (so we don't have arguments about the details of how to come up with it) and the review process is made very easy for those who want to provide feedback quickly.

4) Make it so that you have to register to review.

The _designers_ can give the _reviewers_ a rating.

One rating from each designer for each reviewer - a "fairness" number, and some text to explain why. The designers can probably change their rating if things change. Everyone can see a reviewer's ratings, just like EBay. Maybe list the "Top 10 review contributors" and "Top 10 fair reviewers".

...And the average of a reviewer's rating is used to weight their recommendation rating on the overall score for every scenario they review.

Ideally, the overall recommendation calculation for each scenario that a reviewer has reviewed would be revised each time that reviewer's rating changes.

Hence repeat offenders become less and less significant in ratings, while becoming more and more obvious in their own rating page.

This moves the argument away from one bad review ... one bad review: put up with it... if they are really a troll, it will emerge.

Whadaya reckon?

GaJ.

I reckon you said it all my man!

Share this post


Link to post
Share on other sites
Originally posted by Pzman:

Yes it is for downloading, but its not a popularity contest either. I like the idea about the top 100 or even top 50 or something along those lines.

As for number ratings why not just have each players rating standing on its own, like now but with numbers rather than a graphic because it's same thing really. Some may like the graphic others the number, why not have both?

I don't think it should be a popularity contest either. All too often I see any ills attributed to those whining designers. (Take a look at a couple of these posts for instance...)Whining from either designers, reviewers or players is counterproductive, and not everything that is wrong with the world originated with a CM scenario designer... smile.gif;)tongue.gif

I think everyone agrees that the system for putting the scenarios on the site is fine. Where the problems start are with the reviews. The designers see them as 90% accurate and honest. That's not bad. With the current system there have been less complaints that I have noticed. Not that I would see them all, but the community isn't bashful about voicing it's opinion, about the number one archive for CM either.

There needs to be a bit more common ground between the designers and the reviewers at the moment I think. The current process is:

1. Post the scenario.

2. Someone reviews it.

3. Designer responds.

That works well when the review is a good one and everybody agrees. The problems arise when the designer doesn't exactly agree with what the review states. IMHO,that is the issue as far as most designers are concerned.

By the way, I post scenarios to the SD, and I review scenarios, so I have an idea about both sides of this fence.

I really like the idea of a reveiwer having some responsibility for his review. I have had a few of the infamous reviews that were bad, as reviewers ripped some of my scenarios, that were for instance, intended for 'Axis vs AI' play and he played it as the Allied player vs the AI. The designer doesn't have to totally lose it and can pretty much address the issue in positive terms.

BUT if there was a way for the designer to rate the reviewer, as Richie suggests, then both parties can be part of the process. A simple check sheet with a comment section is pretty much what we have now. What is lacking, is any form of recourse for the designer, once the review is done.

You have to also allow for a reviewer that simply didn't like the scenario. That isn't a sin. He CAN not like it. I've had some reviews that I attributed to them not liking, not only a single scenario, but quite possibly my designing technique overall. Guess what? I'm never going to make that reviewer happy, because he/she really doesn't like my work to begin with. That's okay. My work isn't going to rock everybody's world. But he/she has a right to that opinion about my work. They played it and took the time to do a review on it.

I have more than the average amount of scenarios posted by other designers on the site. So I see all kinds of reviews. I think in the long run a bad review is balanced out in the wash. If a scenario gets 7 reviews and 1 is bad it won't hurt it that much. What I think is worse is the designers response to a bad review. If all you do is rip the reviewer for daring to state that there may be something less than appealing with your creation you have done the entire CM communtiy but especially yourself a disservice.

I put at least 40 hours a week into CM and that includes what time I spend at the Scenario Depot.

That is probably above average too as most people's job situation won't allow that. (There are times I wish mine didn't.)I see every review put on the SD and at TPG. I read them all and all the responses. I want to learn what I can about the hobby.

I have read EVERY REVIEW ever posted on the Scenario Depot, to see what I can learn, where other designers made mistakes, to try to avoid making those same mistakes myself. Of course, I was in search of scenarios to play, as well. After awhile, you find certain reviewers that you agree with, and those that you don't.

I applaud Keith for wanting to improve The Scenario Depot and for all of you for airing your concerns and ideas. This is constructive and a very healthy way to make improvements. Just griping to the ends of the earth is counterproductive.

If there were more ways to view the data that reviews generate like;

1)a top 50 or 100 download list,

2)a list that shows the most downloaded scenarios for a set period for say 30-60-90 days,

3)a reviewer list so I could go and see if my favorite reviewer has played something he likes, and let the designers have a say at some sort of feedback for the reviewer.

4)there could also be a list of designers that show the number of scenarios they have done by game CMBO-CMBB-CMAK and how many reviews that they have. It could give you an idea of whether that designer is popular or not. If they will reveiw your work, they either really like it, or really don't like it. Reading the reviews will tell you quickly which it is.

There could be a number of things done, to improve the lot of designers as well, but I really think that they are unneccessary. The designers bring their sceanrios to The Scenario Depot in droves now, there are over 700 CMBB battles alone... :D:D

Good Hunting.

[ October 23, 2004, 12:59 PM: Message edited by: Panther Commander ]

Share this post


Link to post
Share on other sites

The main thing I like in a game is balance and a good fight. Walk overs suck and being walked on REALLY sucks.

I like to see a review where the user reports type of game (AI or 2 human) and percentage of victory for each side.

Allied H-64% vs Axis-AI 36%

Allied-H 29% vs Axis-H 81%.

There could be a list of all reviews and a total/averaging of the reports.

5 AI battles - Allied 67% Axis 33%

3 two man battles - Allied45% Axis avg 55%

If there was a good number (5 or more) of reports on each scenario then a downloader could determine which scenarios were balanced and which were not.

The newbie could pick a battle he'd be sure to win. The vet/regular could pick something balanced. And the elite CM player could play the same battle as the newbie but pick the difficult side.

Additionally, the author could review these stats and revise his scenario to make it more balanced.

Share this post


Link to post
Share on other sites

Admiral Keith,

First, thx for the great job of keeping Scenario Depot at live !

You're absolutely right with the Review scoring !

My Scenario (Stalingrad The Moviescene) which got quite good reviews, received two 0 point reviews by an ***hole who quoted having made the same, although mine was already there for a year and a half !

Now there are maybe 8-10 CMBB Scenarios coming in a week and it really doesn't take much time to download 100kb. Most of the time a look at the map already tells a lot about the game, and whether you may like it or not.

Greets

Daniel

Share this post


Link to post
Share on other sites
Originally posted by General Colt:

The main thing I like in a game is balance and a good fight. Walk overs suck and being walked on REALLY sucks.

I like to see a review where the user reports type of game (AI or 2 human) and percentage of victory for each side.

Allied H-64% vs Axis-AI 36%

Allied-H 29% vs Axis-H 81%.

There could be a list of all reviews and a total/averaging of the reports.

5 AI battles - Allied 67% Axis 33%

3 two man battles - Allied45% Axis avg 55%

If there was a good number (5 or more) of reports on each scenario then a downloader could determine which scenarios were balanced and which were not.

The newbie could pick a battle he'd be sure to win. The vet/regular could pick something balanced. And the elite CM player could play the same battle as the newbie but pick the difficult side.

Additionally, the author could review these stats and revise his scenario to make it more balanced.

I cannot see that system working since not everyone plays the game at the same level so scores like that are based on the players ablities, not the quality of the battle itself.

Share this post


Link to post
Share on other sites
Originally posted by General Colt:

The main thing I like in a game is balance and a good fight. Walk overs suck and being walked on REALLY sucks.

....

Additionally, the author could review these stats and revise his scenario to make it more balanced.

This is all a matter of having a scenario playtested before it is sent to TSD, IMHO. A perfect world would have all scenarios sent to TSD having been playtested for balance so that the reviews left at TSD would be directly related to the fun factor of the scenario only.

Share this post


Link to post
Share on other sites

First off - another big hats off to Admiral Keth for the great site he has given to the CM community.

Being both a designer and a reviewer I believe the site as it is now is functioning just fine (I also believed that the previous version was fine also)...

When I go to TSD I am looking for the following things as a Designer ...

</font>

  • Ability to upload a scenario with ease.</font>
  • Be able to provide the community with pertinant information about the scenario.</font>
  • Ability to see reviewers feedback.</font>
  • Ability to provide reviewer with designers feedback.</font>
All of these seem to work very well...

From the Designers side if someone doesn't like my work... so be it... its not going to annoy me... I am not in the business to have the 'most scenarios' or the 'most popular scenarios' or any other competitive type rating.. I design scenarios for fun... and the community should consider themselves lucky that the designers are willing to share their work with others...

Now from the point of view as a Reviewer/Player ...
</font>
  • The ability to download a scenario with some knowledge of what it is about (PBEM or Historical or Large or Small, etc).</font>
  • I want to see what others think about said scenario.</font>
  • The ability to provide the designer with feedback about his scenario.</font>

These items all seem to work for me...

As a reviewer/player I feel that TSD provides me with the information I need to make an informative choice for the specific type scenario I am looking for... For me I don't really care what others have said about a scenario (in fact I dont like to look at that)... I can read the briefings, look at the map and make my own determination if its good or bad... in fact, in playing some scenarios that have not been rated I have found some really hidden jewels... I also always try to provide feedback on scenarios that I have chosen to play...

So what are the issues...

Best I can tell is the designers want their scenarios reviewed more... That would be nice... but if you restrict downloading to those who are only going to submit reviews then I think you are hurting the idea of TSD...

The reviewers want easier ways to pick scenarios... IMHO it doesnt get much eaiser then it is now... Lets face it in a perfect world.. everytime a scenario would be downloaded, it would get a perfect review, cause it is a perfect scenario, and the everybody would be happy... NOT...

Ok, little things can be refined or tweaked to enhance almost anything... but IMHO TSD works just fine...

Now that I rambled (after having a 10 cup'er of coffee this AM) I will get back to my original thought... THANK YOU KEITH MILLER FOR THE AWESOME SITE it is a pleasure to have people like you, GJK from The Proving Grounds, Todd Johnston from Combat Mission Modification Database and all the others who donate their time and effort to make this such a great community...

That is all...

-FR

Share this post


Link to post
Share on other sites

What about using the old system with a challenge provision, allowing scenario authors to 'challenge' any review that is trollish or did not follow the battle guidlines, for example.

Also, reviewers would give an overall score from 1-10 which would be culmative.

I think we should keep things as simple as possible, for the reviewer.

Share this post


Link to post
Share on other sites

Unfortunately so few people post reviews we may be fretting for naught.

I use the author's description more than the reviews. Sometimes if there is a bad review I will try the scen to see if the review is bogus.

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Sign in to follow this  

×
×
  • Create New...