While all is calm on the surface we continue to be plagued by bad actors. Much of our capacity to host the site is being consumed by various bots that relentlessly hammer the site. We have no idea what they are trying to find. Jay is constantly battling these things. Today’s report
I got one of the two bots that have been hammering the CCC Forum by using my user agent blacklist, and it seems to have helped a little bit with the other one, but it still is getting through. Yesterday, it did 339,069 hits to pages and 2,466 hits to the robots.txt file for a total of 13.55 GB of bandwidth. Obviously a bot. It’s using multiple IP addresses, but it looks like most of them are in the CIDR XX.XX.XXX.X/XX, so I am going to try blocking that in CCC’s xxxxxx file rather than globally at the firewall.
That CIDR belongs to Microsoft in Boyden, Virginia, so I may get legitimate users, but I kind of doubt it. They don’t look like an ISP, but a provider of servers.
Kudos to to what you guys do, I have very limited knowledge on how you guys keep everything in line but what I do know bandwidth equals money and I felt the need to kick in a few bucks. That’s the least I could do for all the help and knowledge I have learned here over the past couple years. Keep up the good fight boys!
This is an automatic informational message to notify you that domain (classiccougarcommunity.com) has used 59.12% of its bandwidth allotment under your current plan.
Once you reach 100% utilization of your bandwidth your site will be interrupted.
What sucks is that a big part of our bandwidth is consumed by various bots. The Google and Bing bots at least have a purpose and identify themselves in most cases for what they are. Then we have the strange ones that we can’t identify at all. Where a real live user looks at the unread posts these things tend to hit every page on the site.
We are probably going to be okay this month, but that warning (it begins when we hit 50%) is coming earlier every month.
Hey Bill, would it help if you required all of us legitimate users to enter a Capcha code every time we wanted to get in? Or do the bots get around that too? Just my $0.02 worth …
Bing is a search engine, like Google. The bots “index” the site and add it to the searchable data base.
Most people find the site via search so they are a necessary evil. There are many other bots or spiders that crawl the web and look for things or are used to build other kinds of data bases. There really isn’t a good way to block them without also blocking a bunch of legitimate users including legitimate bots.
Catcha features are good at stopping bots that want to act like users. We use the sign up quiz that seems to stop most, although some have gotten through.
Jay has come up with an ingenious new strategy to help resolve this… until the bot keepers figure it out.
After installation this is what we are seeing.
This is an automatic informational message to notify you that domain (classiccougarcommunity.com) has used 61.35% of its bandwidth allotment under your current plan.
Once you reach 100% utilization of your bandwidth your site will be interrupted.
Jay is F’ng brilliant. But we knew that already. His latest modification has turned into a clean kill.
Here is the latest from this morning:
This is an automatic informational message to notify you that domain (classiccougarcommunity.com) has used 64.70% of its bandwidth allotment under your current plan
What this means is that we are now down to a very reasonable less than 2% per day!
You are doing all the right things already. What really makes the site work are people sharing their knowledge and experiences. Can’t ask for more than that.