GreenAsJade Posted June 30, 2009 Share Posted June 30, 2009 Hi All, Today cmmods.greenasjade.net experienced a massive spike in downloads (or at least, page accesses). Because I'm still rebuilding my laptop, I can't look into the details. If this is you downloading a whole pile of mods, no problems but PLEASE let me know. Otherwise I have to assume it's an attack, and I may need to take the site offline (to prevent myself suddenly getting $100s of data charges). GaJ 0 Quote Link to comment Share on other sites More sharing options...
GreenAsJade Posted June 30, 2009 Author Share Posted June 30, 2009 Well, it was just a spike, not an ongoing assault. Leave me wondering WTF though!? GaJ 0 Quote Link to comment Share on other sites More sharing options...
Wicky Posted June 30, 2009 Share Posted June 30, 2009 Search engine bots having a trawl? Looking at your Robots.txt file http://cmmods.greenasjade.net/robots.txt # See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file # # To ban all spiders from the entire site uncomment the next two lines: # User-Agent: * # Disallow: / I suspect it should be: # See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file # To ban all spiders from the entire site uncomment the next two lines: User-Agent: * Disallow: / As the hash # indicates a comment 0 Quote Link to comment Share on other sites More sharing options...
Redwolf Posted June 30, 2009 Share Posted June 30, 2009 Might want to think about that because then - obviously - the search engine don't find anything anymore. Disallow download of pictures but allow text files, that seems more reasonable if search engines are indeed the problem. 0 Quote Link to comment Share on other sites More sharing options...
Wicky Posted July 1, 2009 Share Posted July 1, 2009 I agree, at present it's a free for all, while the intention was to block everything. IMHO should allow the bots at minimum access of a cannily worded home page that spells out what the site is about to give a clue to motivated mod hunters. Access to pic folder would only be of some benefit if the pictures were accurately referred to by their content - but would it be worth the precious bandwidth? Might want to think about that because then - obviously - the search engine don't find anything anymore. Disallow download of pictures but allow text files, that seems more reasonable if search engines are indeed the problem. 0 Quote Link to comment Share on other sites More sharing options...
Redwolf Posted July 1, 2009 Share Posted July 1, 2009 The server logfiles should tell you what was being downloaded, and the search engines mark themselves in there. 0 Quote Link to comment Share on other sites More sharing options...
GreenAsJade Posted July 1, 2009 Author Share Posted July 1, 2009 Yeah - I just need to have my whole environment back up (linux) so I can get accses to the service provider to see the logs! I was having a dang of a time recovering the linux partitions, and now I'm travelling with business, which has put me out of commission a bit for this stuff Grrrrr. The only information I have from google analytics is that the "unique visitors" count sprang from an average of 50 per day up till Mon June 29, then on June 29 for some reason there were 260 unique visitors, most of which were "direct" (not referred) then it went back to normal the next day. WTF eh? While we're talking stats, y'all may be interested to know that cmmods.greenasjade.net averages 1000 page views per day. GaJ 0 Quote Link to comment Share on other sites More sharing options...
GreenAsJade Posted July 1, 2009 Author Share Posted July 1, 2009 Note - this peak in usage definitely wan't normal robots, because it comes from google analytics, which is java based, so it is "non-robot" data, AFAIK. GaJ 0 Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.