Jump to content
Sign in to follow this  
GreenAsJade

Scenario Depot Blues

Recommended Posts

Good discussion.

A few points I would like to add or reiterate.

* True -- few people do submit reviews. I like the idea of encouraging people to submit reviews, but making it manditory on a 1-1 basis will be a total nuisance and most users of the site will howl with complaints.

* To sort out the crack pots who submit lame or bogus reviews, you may institite some sort of "rate the review" system much like you would see on Amazon.com. A simple 1-5 scale works. A user who accumulates enough "low" votes that the reviews he submits are worthless would suffer some sort of sanctioning. Maybe all his future rating are "null" and simply are not counted in the total. Maybe his future posts carry some sort of red flag or a clown face icon. Maybe something else.

* An important point for ANY rating system for reviews or scenarios -- don't disclose ALL the rules or factors that go in to the ratings. This way, you make harder for crack pots to "game" the rating system. Using the example above, don't disclose that it might take 20 (or whatever number) "low" votes before a user's ratings are null. This way, this crack pot can go right on being a crack pot but not know that his lowball rankings are not having an effect.

* A "top #" list is a good idea. Maybe a "top downloads" or "top rated". Is there a way to not count downloads more than once from the same user? (Once again, if some crack pot wants to keep hitting the download button in an effort to jack his numbers up, don't disclose that the system only counts one per user. Let him waste his time.)

* Whatever you do, keep it simple! Getting a couple of moderators to help is a great idea as well.

Share this post


Link to post
Share on other sites

Anyone other than me think it's harder now to find good scenarios to play? We've heard from people who think it's still fine. What about any others?

And what about the detailed suggestion I made for change: nice & simple, and addresses the problems. Any comments on that? What's wrong with what I suggested?

GaJ.

Share this post


Link to post
Share on other sites

I think the review system that is in place is fine, if people want to post a review then they do so, if not, thats fine also. Everybody has a different idea of what or how a battle should play out like. If you only download a battle because of a review then I would call that stupid. If I'm made to review a battle before I can download other battles then I won't be around that site anymore.

As far as I'm concerned, what needs a major overhaul at SD is in the author administration area, this as it is, is a nightmare. As it is right now I'm not even going to attempt to upload anymore battles, or update the ones I have there, for that matter either. I only hope that the things keith has said would be done in that area, well be done, because that to me needs immediate attention.

Those things are the ability to overwrite previous battles, able to delete previous battles. able to change data fields in the scenario data page without having to upload the battle again.

I hope your job prospects have worked out well for you Keith.

Share this post


Link to post
Share on other sites
Originally posted by DEY:

I think the review system that is in place is fine, if people want to post a review then they do so, if not, thats fine also. Everybody has a different idea of what or how a battle should play out like. If you only download a battle because of a review then I would call that stupid. If I'm made to review a battle before I can download other battles then I won't be around that site anymore.

Point taken.

Originally posted by DEY:

As far as I'm concerned, what needs a major overhaul at SD is in the author administration area, this as it is, is a nightmare. As it is right now I'm not even going to attempt to upload anymore battles, or update the ones I have there, for that matter either. I only hope that the things keith has said would be done in that area, well be done, because that to me needs immediate attention.

Yep, it's a little awkward right now; that's wht it's being updated.

Originally posted by DEY:

Those things are the ability to overwrite previous battles, able to delete previous battles. able to change data fields in the scenario data page without having to upload the battle again.

These items are being coded as we speak...er, write.

Originally posted by DEY:

I hope your job prospects have worked out well for you Keith.

So far, no dice. Although they have extended my employment to the end of next January, it's only a Stay of Execution, not a Reprieve.

Share this post


Link to post
Share on other sites
Originally posted by GreenAsJade:

Adm - thanks so much for listening to our feedback and being prepared to contemplate change yet again!

No worries, mate! Every forum member who cares about supporting the author's ability to refine and enhance their work should be participating in this discussion.

Originally posted by GreenAsJade:

There will be as many different system suggestions as people responding, I'm sure.

True, but the simpler, the better. I'm looking for systems which have a broad degree of usefullness to both players and authors, yet are easy to implement and use.

Originally posted by GreenAsJade:

The issue of the person being reviewedm not liking reviews is not unique. Even EBay has it, and copes, with an (mostly) automated system.

Here is my suggestion...

1) Get rid of all existing numeric scales and replace them with one: recommendation for the scenario - a rating of 1 to 5.

This is really the most useful thing that you can record a number for - to help users quickly find highly recommended scenarios. This is the thing that has gone away in V2.

Something similar to (I know it's atrocious English, but...):

My Scenario Recommendation

1 Excellent

2 Good

3 Fair

4 Below Average

5 Poor

Everyone MUST keep in mind that ANY numeric scale is subjective to the reader. Thus, it will be incumbent upon the reviewer to clarify their position through the text review portion.

Originally posted by GreenAsJade:

2) Have a check box to indicate whether the recommendation is for PBEM or AI, and make both the PBEM and AI ratings of scenarios available.

People are looking for either PBEM or AI, not both at the same time (usually).

Simple enough.

Originally posted by GreenAsJade:

3) Have a text review section where people are encouraged to discuss map design, play balance, briefing etc, but not give ratings to those.

OK, although eliciting that information from people is usually like pulling teeth. This can be alleviated by me providing more explicit instructions above the text area, which ddescribes the type of information the player should be discussing. Conversely, I could create field-level help which provides information specific to the field in question.

Originally posted by GreenAsJade:

SO:

When I review a scenario, I log in, click "PBEM" or "AI", select a number from 1 to 5 as recommendation rating, and optionally provide a text review.

The rating is _clearly_ subjective (so we don't have arguments about the details of how to come up with it) and the review process is made very easy for those who want to provide feedback quickly.

4) Make it so that you have to register to review.

The _designers_ can give the _reviewers_ a rating.

One rating from each designer for each reviewer - a "fairness" number, and some text to explain why. The designers can probably change their rating if things change. Everyone can see a reviewer's ratings, just like EBay. Maybe list the "Top 10 review contributors" and "Top 10 fair reviewers".

I'm liking this, although I also forsee implementation of the "Was This Review Helpful?" for the players. Thus, both players and authors get to rate the review.

Originally posted by GreenAsJade:

...And the average of a reviewer's rating is used to weight their recommendation rating on the overall score for every scenario they review.

Ideally, the overall recommendation calculation for each scenario that a reviewer has reviewed would be revised each time that reviewer's rating changes.

It's early and I'm having some trouble following this. I'll have to re-read it more carefully later on. I'm also a little concerned about page load time with the number of calculations that would need to be performed.

Originally posted by GreenAsJade:

Hence repeat offenders become less and less significant in ratings, while becoming more and more obvious in their own rating page.

This moves the argument away from one bad review ... one bad review: put up with it... if they are really a troll, it will emerge.

Whadaya reckon?

GaJ.

Many of these thoughts do have significant merit. Let's see how they either stand on their own, or can be meshed with other ideas.

Share this post


Link to post
Share on other sites
Originally posted by GreenAsJade:

Heck, make it so you have to register to download too, so we can see who's downloading heaps and reviewing little...

GaJ.

Is this something authors really want? Although it is easy to implement, what would the authors want to do with the data? Sure, the authors can get a list of people who downloaded your scenario, but do the authors want to then begin pestering players for reviews?

Even if there is simply a list of usernames and download counts ratioed to review counts, that's a feature that might cause many people to stop downloading and playing altogether.

This will really have to be justified by the authors and approved by a large number of players prior to implementation.

Share this post


Link to post
Share on other sites
Originally posted by GJK:

You've got to give credit when a member joins or else they wont be able to download a scenario in order to review it.

Agreed. So far the credit count is 10 free downloads prior to needing to input 1 review. Anyone want a different ratio? Perhaps 1 review = 2 DL credits.

Originally posted by GJK:

Perhaps a "Reviewers Guideline" is in order as well. I personally tend to just go middle of the road unless something exceptional stood out in a positive or negative way, but I can guarantee that if I stacked up a couple of my reviews together, I would see several instances of "hmmm, gave that map a 5 and this one a 4, yet the second one was actually better in my opinion".

Field level help would go far to assisting with this idea. This would require criteria from authors as to the nature of the information they want they reviewer to enter.

Share this post


Link to post
Share on other sites
Originally posted by Emar:

Gotta say I like the depot the way it is now. I think the reviews now are a lot more honest now that there is no motivation to either promote or demote scenarios to make the list. Would also think that a particualary good or bad scenario would tend to make the top 20 most reviewed list that is currently featured.

The implicit desire for some authors to have their scenarios _always_ at the top of the lists was one of the stronger reasons why the Cumulative Rating System was abolished. There _were_ authors who wanted especially low rating reviews excised in order to increase their standing.

Originally posted by Emar:

Also I would have to think that now that you are required to log in and give out your e-mail address to post a review that trolling has pretty much dwindled off (have not seen much of Tommy Tiddlywinks or whatever his name was). Before it was entirely possible for anyone to make up a name and post a review and I think that this could be one of the reasons for less reviews now.

Trolling has pretty much been nipped in the bud with having to register, as well as my ability to ban people completely from the site. I have not had to do this yet, but the feature is in place, waitng to utilized.

Originally posted by Emar:

Not sure that all folks would want to have to post a review in order to download another scenario. Sometimes I download a bunch but then only play the ones that appeal to me. Would think that this might lead to people posting bogus reviews just so they could download again.

That's why I'm thinking a different ratio; 1 review = 2 DL's, or 1 review = 5 DL's. I would want to strike a fair balance between requiring reviews and allowing downloads, especially if the player wants to just get a couple for entertainment purposes.

You are correct with regards to the 'bogus review' issue, but that is a possibility now.

Originally posted by Emar:

And as for the "best scenario" lists well one mans treasure is another ones trash. And most players only play a scenario once, even though anybody who thouroughly playtests a battle or op knows a CM scenario can play out a hundred different ways. At the CM level of combat balance of a scenario often depends on the success of one or two tanks or priority units. Score a lucky hit on a critical enemy unit and suddenly people are griping about how unbalanced your scenario is. I know from playtesting ops that if you play it 50 times you are likely to see a full spread of results, just as if you were rolling dice.

True, a player's opinion of a scenario may be biased based on his experience level. A scenario which the author intentionally made challenging for experienced players may overwhelm a novice, thus coloring their response to how the sceanrio was actually to be played. Therefore, they may return an unfavorable review.

To counter this, what about attaching some kind of experience level to scenarios? For example, the author may indicate that this scenario should be played by beginners to CM, or by experienced grogs.

Originally posted by Emar:

Having said all that if you do decide to intitute a top 10 list perhaps the worst review or 2 could be ignored in the scoring (but not deleted). Another possibility is that when a review is posted it could be set up to not officially affect the scenario score until the author can reply to it. Then have it set up so the reviewer would have to reply again to confirm or change his review depending on their satisfaction of the authors explanation and rebuttal.

Hmm. A bit too much rigmarole for people to deal with. Ease of use for both the author and the reviewer is the watchword here.

Originally posted by Emar:

Admiral I certainly appreciate and applaud your efforts for the CM community. There is no other site like yours

Emar, thanks. It's a challenge to design a system which is at once useful, easy, and informative.

Share this post


Link to post
Share on other sites

Not wanting to complicate this further, but I would like a user guideline too. I stink at some tactics. I have started to play some very well done battles, and found them to be way over my abilities. A newbie, average, expert rating would be nice.

Share this post


Link to post
Share on other sites
Originally posted by Panther Commander:

Would it be possible to see the entire list of scenarios for each game ranked instead of just the top 20? Or maybe the top 100, since there are 757 CMBB scenarios at this very moment.

The page return time on a list like that would be prohibitive. It all has to do with the load being placed on the server-side PHP engine, and what it can realistically return in a respectable amount of time before the person who clicked on the link gets bored or thinks it locked up.

Originally posted by Panther Commander:

What about a feature that shows the scenarios that have gotten the most downloads, for a set period, such as 30 or 60 days? That would be another indicator players could use to choose scenarios.

Possible, but again, I'd be concerned about page return time. I can code it up and give it a try, but this one will go somewhere near the bottom of The Giant List of Sceanrio Depot v3 Things To Do.

Originally posted by Panther Commander:

One line of reasoning, that often comes out in these threads, that puzzles me, are the comments about why cater to the scenario designers. Unless I am mistaken, and I could be, that is why everybody goes to the Sceanrio Depot. To download the scenarios they have uploaded. If you don't make them happy there won't be anything to get all upset over. Now will there? :confused:

Just a puzzle that continues to pop up now and again.

Good Hunting.

There must be a balance between the author's ability to glean useful information from reviews, and players ability to place those reviews, offset by some burden upon the player to come back and contribute. So far reviews have been 100% voluntary, but there seems to be some call for semi-compulsory reviews. The jury is still out.

Share this post


Link to post
Share on other sites

I have to agree with GreenAsJade to some extent. I've only used the Scenario Depot a few times (being a relatively recent and fairly infrequent CM player) but I found the previous system much more convenient in terms of finding a good scenario of the type I want to play. Today, for example, I want to look for an Allied Attack scenario (Medium size), which can be pretty well vs an Axis AI defender. That gives me 62 options, which I then have to look at one by one to find which can be played well vs Axis AI defender. With v1 of the Scenario Depot, it was much easier to find such a scenario quickly. It could be ameliorated to some extent by allowing an extra search field for 'playable vs Axis AI', and it being up to the designer to give fill in some kind of field for whether a scenario is suitable for that kind of play.

My other problem is that I haven't been able to use the scenario depot at all from the computer I actually have Combat Mission installed on. It's Windows XP, pretty much straight out of the box, but clickingon those nice buttons such as 'download', 'submit review', or anything else with that style of button completely fails to do anything. Now although this is almost certainly the fault of my security settings in some way, it's also the only site on which I've noticed any problems at all.

Share this post


Link to post
Share on other sites
Originally posted by vulture:

My other problem is that I haven't been able to use the scenario depot at all from the computer I actually have Combat Mission installed on. It's Windows XP, pretty much straight out of the box, but clickingon those nice buttons such as 'download', 'submit review', or anything else with that style of button completely fails to do anything. Now although this is almost certainly the fault of my security settings in some way, it's also the only site on which I've noticed any problems at all.

I use a computer that has XP straight out of the box too. I have CMBB and CMAK loaded on it and have no problems accessing the Scenario Depot. I don't think XP is your problem.

Good Hunting.

Share this post


Link to post
Share on other sites
Originally posted by Admiral Keth:

</font><blockquote>quote:</font><hr />Originally posted by GreenAsJade:

Heck, make it so you have to register to download too, so we can see who's downloading heaps and reviewing little...

GaJ.

Is this something authors really want? Although it is easy to implement, what would the authors want to do with the data? Sure, the authors can get a list of people who downloaded your scenario, but do the authors want to then begin pestering players for reviews?

Even if there is simply a list of usernames and download counts ratioed to review counts, that's a feature that might cause many people to stop downloading and playing altogether.

This will really have to be justified by the authors and approved by a large number of players prior to implementation. </font>

Share this post


Link to post
Share on other sites

There already is a top ten download list, but that does seem a little short. A rank system sounds somewhat interesting to a point. As for bad scenarios, well that may be so, but last time I checked there were no requirements for the level of design required to post there, nor IMO should there be. As has been said before, one mans garbage is another mans tresure.

To me any rating system, although useful, means that a lot of good scenarios may be missed because nobody takes the time to review them. There are still hundreds of battles without reviews, but that does not mean they are any better or worse than the others.

Share this post


Link to post
Share on other sites

Incidentally, another argument against requiring reviews before downloading more scenarios: Consider someone off for a few weeks with their laptop and no internet access guaranteed. They might just want to download a reasonable groups of scenarios (say 5 in one go), with the intention of playing (and reviewing) them all over the next few weeks.

Share this post


Link to post
Share on other sites
Originally posted by DEY:

I think the review system that is in place is fine, if people want to post a review then they do so, if not, thats fine also. Everybody has a different idea of what or how a battle should play out like. If you only download a battle because of a review then I would call that stupid.

I guess you'll have to call me stupid. It worked really well for me though: downloading highly recommended scenarios always got a good result,

and quickly.

It amazes me how much time some people have to go about searching for a good one and reading reviews to make the selection. Good luck to you. I would like to have a quicker way to get to the good ones!

If I'm made to review a battle before I can download other battles then I won't be around that site anymore.

Although I think I've reviewed every battle I've played, I have to agree that forcing people to do reviews won't do anyone any good. It will just generate rubbish reviews. I want to read reviews by people who wanted to write them.

GaJ.

Share this post


Link to post
Share on other sites
Originally posted by junk2drive:

Unfortunately so few people post reviews we may be fretting for naught.

Actually, this is a regular "complaint" ... "we don't get enough reviews", but I would say 100+ a month is not too shabby...

Share this post


Link to post
Share on other sites
Originally posted by Admiral Keth:

</font><blockquote>quote:</font><hr />Originally posted by GreenAsJade:

Heck, make it so you have to register to download too, so we can see who's downloading heaps and reviewing little...

GaJ.

Is this something authors really want? Although it is easy to implement, what would the authors want to do with the data? Sure, the authors can get a list of people who downloaded your scenario, but do the authors want to then begin pestering players for reviews?

Even if there is simply a list of usernames and download counts ratioed to review counts, that's a feature that might cause many people to stop downloading and playing altogether.

This will really have to be justified by the authors and approved by a large number of players prior to implementation. </font>

Share this post


Link to post
Share on other sites

Looking at other organizations that try to judge independent artistic creation. I would think an ad hoc group, selected by the Admiral. Would conduct an academy awards type arrangement. Rating games would come from three aveneues

1. The Grog core, restricted to those designers with say more than ten scenarios and selected by one another (very cliquely) They would recommend scenarios for ratings in various catagories.

2. All registered Forum users (using a voting system to be added-perhaps a top ten?)

3. Anyone who puts in a review. (using the existing system)

Share this post


Link to post
Share on other sites
Originally posted by Admiral Keth:

Something similar to:

My Scenario Recommendation

1 Excellent

2 Good

3 Fair

4 Below Average

5 Poor

Simple and effective: works for me.

(Though I would make a good score higher

than a bad one)

Or

5 Must Play

4 Highly Recommended

3 Worth Playing

2 You might like it, I didn't.

1 Fatally flawed: don't bother.

Originally posted by GreenAsJade:

4) Make it so that you have to register to review.

The _designers_ can give the _reviewers_ a rating.

(snip) Everyone can see a reviewer's ratings, just like EBay. Maybe list the "Top 10 review contributors" and "Top 10 fair reviewers".

...And the average of a reviewer's rating is used to weight their recommendation rating on the overall score for every scenario they review.

Ideally, the overall recommendation calculation for each scenario that a reviewer has reviewed would be revised each time that reviewer's rating changes.

It's early and I'm having some trouble following this. I'll have to re-read it more carefully later on. I'm also a little concerned about page load time with the number of calculations that would need to be performed.

I thought an example would illustrate best, so I emailed one to Admiral. Then I thought "what the heck why not let everyone pick it to bits. So here it is:

Joe the reviewer has posted 3 reviews. The designers of each scenario he reviewed

gave him ratings of 4, 5 and 3 (out of 5) respectively as a reviewer. They thought his reviews were pretty good.

So his personal reviewer rating is (4+5+3)/3 = 4.

Troll the reviewer has posted 2 reviews. The designers of each scenario he

reviewed gave him 0, 1 respectively.

He was obviously a troll.

So his personal reviewer rating is (0 + 1)/2 = 0.5.

Joe and Troll each come to review a new scenario.

Joe gives the scenario a recommendation of 4 ("highly recommended").

Troll gives the scenario a recommendation of 1 ("don't bother playing it").

The net recommendation for the scenario from these

two reviews is the is the average of Joe's recommendation and Troll's recommendation

weighted by their reviewer rating .

Calculation:

[ (Joe rating * Joe Score) + (Troll rating * Troll score) ] /

(Joe rating + Troll rating)

which is

[ (4 * 4) + (1 * 0.5) ] / (4 + 0.5) = 3.7

See how this scenario ends up being nearly highly recommended (3.7 is nearly 4), as

rated by Joe, because Joe's recommendation counted 8 times as much as Troll's. But it's not quite a 4 on the off chance that Troll was pointing out a real flaw.

Now suppose Joe goes psycho, and starts submitting bad reviews. Before long

his own reviewer score average drops to Troll like levels and he can no longer

significantly affect the ratings of scenarios.

Note how all this happens without a moderator having to be involved.

By rating reviewers each time they submit a review, the scenario designers have the power over time to rule out trolls,

without individual reviews having to be assessed and argued about.

Why I'm putting this forward:

1) It gives a system to let people quickly find highly recommended scenarios.

2) It makes reviewers accountable for their reveiws.

3) It lets questionable reviews be dealt with without a moderator having to put effort into sorting through an argument about individual reviews.

GaJ.

Share this post


Link to post
Share on other sites

So if someone posts low scores more than once that makes them a troll? Hmm I don't know, but it must take a while to figure it out. I mean even trolls will like some scenarios and give them 3-5 of 5. One thing that bugged me with the old system for example was when someone would post the same low scoring review twice. Now maybe there was a double post problem like there is here sometimes. If that is the case, rather than being marked as a troll, the person doing the review should have the option of deleting their own review. Or another option would be to only allow one review per person per battle.

Share this post


Link to post
Share on other sites

I have a theory about generational differences.

Akula2 and I had a discussion about fixed or locked down placement of units in setup. He believes the player should always have the choice of where to place his units in setup. I can see a designer, after playtesting, deciding to lock units for balance of gameplay.

I am in my 50s. I don't know about Akula2 and it doesn't matter. There is nothing wrong with his manner of playing or his preferences.

My theory is that I grew up (sort of) with checkers, chess, and games with defined placement of pieces. Baseball allows some placement of players, but not total freedom. So I am use to being told, so to speak, where to start from. In fact it is more enjoyable sometimes, than fretting over what to do. Younger people raised on console games, may have a different outlook on what wargaming is. No right or wrong here, just different.

How would these two types of players rate the same battle?

Share this post


Link to post
Share on other sites

It doesn't matter how each of them rates it.

It's a shame that designers will feel so passionate about any given individual rating (because it is about their work, afterall) when for the purposes of selecting highly recommended sceanrios its all about averages.

If you give it 4 and Akula2 gives it 2 then it will show up as 3. That will at least position it and allow someone to start making an assessment. Someone else will read _why_ you gave it 4, and will choose to play it because you made sense to them. They will give it 4. It's average will move up to 3.3 (or whatever). Now it is already rated better than average because at least _some_ people liked it, so it is a more logical choice for someone looking for a scenario than one which no-one liked.

It's all about averages. Don't get hung up on individual score.

GaJ.

Share this post


Link to post
Share on other sites

Sigh - the joys of written communication!

Originally posted by Pzman:

So if someone posts low scores more than once that makes them a troll?

Of course not.

Notice that I didn't say what Troll gave as scores at all. What I said was that the designers gave _him_ low scores, because they thought his reviews stank.

If someone posts a well thought through and explained review where they give a low score, hopefully most designers will be mature enough to give the reviewer a reasonable rating.

What makes someone a troll is a review with a low score and text that says "This scenario sucks. The author's mother stank of elderberry wine."

Don't confuse the rating that someone gives to a scenario in a review with the rating they get back in turn from the designer.

In my example above it was assumed that the two low reviewer_ ratings that Troll got were because his reviews stank, not just because they were low scores.

GaJ.

[ October 30, 2004, 05:10 AM: Message edited by: GreenAsJade ]

Share this post


Link to post
Share on other sites

This kind of stuff is self correcting, by the way. That's the beauty of it.

If I give your scenario a long well thought out review explaining why it wasn't enjoyable, and you give me a low reviewer rating, I'm not going to review any more of your scenarios, am I? Problem solved: I get good reviewer ratings from people who appeciate my review style and you don't suffer from reviews from me that you hate.

GaJ.

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Sign in to follow this  

×
×
  • Create New...