The Halloween Horror Strikes Again

[Keith Robinson]

Just as a temporary measure, I restricted access to the search function of the forums. Spammers have been known to exploit the search function by running automated remote searches, thus clogging things up, causing more bandwidth, etc. I thought I'd restrict access for a while to see what happens.

Note that this restriction is primarily meant for the search box, but it also affects other things like active topics, unanswered posts, and of course when you try to view new posts if you happen to have that saved as a bookmark. The forums themselves are unrestricted if you browse from the main board index.

So it's open to all, but admittedly a bit too restricted for regular users. If I can find a way to restrict the search box but keep everything else open, I'll do so. But this may be temporary anyway. I just wanted to try it and see if it helps!

[Keith Robinson]

By the way, I found that one particular IP address is causing most of the trouble. When the stats show one visitor (which is actually one unique visit in the space of an hour) with over a million hits to the site, that's a serious problem and probably the cause of it all. The IP originates in China, where "more than a third of all forum spam comes from" according to many sources.

I've tipped HostGator to this and hope they just go ahead and block the IP from the server. I can easily block the IP from the site and forums, but that's AFTER it's "hit" the server, which is no good. The problem we've been having is actual hits to the server in a very short space of time.

[Julie2owlsdene]

Thanks for all these updates, Keith. Seems it's quite a problem. Let's hope the HostGator takes this on board and prevents the IP address from posting!

[Eddie Muir]

Thanks for all your updates, Keith. It's good to know that you understand the problems, as all the technology is beyond me.

[Anita Bensoussane]

Thanks very much for looking into things further and thinking about ways to resolve the issues, Keith. Let's hope HostGator are able to block that IP - and help us prevent a similar thing happening again. Fingers crossed that service will soon be back to normal!

[pete9012S]

I read this interesting report about HostGator on wiki;

October 2014 Blackout

During the morning of October 29, 2014 Hostgator suffered another major outage that affected their reseller accounts. The company did not immediately announce the cause of the outage, though they stated, on Facebook and their forum, that an OS upgrade was the cause of the issue. (They made the reseller sections of their forum inaccessible to the public, visible only to registered users.)[24] As of 6:10pm PDT, the outage persisted, and HostGator became non-responsive on Facebook, Twitter, and its own support forum. Users reported being hung up on via live chat and phone, despite waiting hours to be connected. As of 9:30pm PDT, the outage continued, though HostGator Support stated they are making progress [25] and were offering uptime credit to customers that request it.[26] As of 6:45am EDT on October 30, 2014, there was still no resolution for sites with mysql databases. No ETA announced on most recent forum update. As of 10:00 am EDT on November 1, 2014, many reseller accounts were still without service or with only intermittent service. Still no public statement of how, when, or if service would be back to 100%. Many users reported that databases had been restored, but had been wound back to week-old backups, with all newer data lost. Others reported being unable to access their accounts to correct problems with the restored databases. Email service was also disrupted.[27] The server-level problems weren't resolved until shortly before 11:00 pm EST on November 2, 2014, almost five days after the outage began.[28] By 4:00 am EST on November 4, 2014, users were still reporting widespread account-level database problems.[29]

http://en.wikipedia.org/wiki/HostGator" onclick="window.open(this.href);return false;

[Keith Robinson]

Yes, I consider myself lucky that the vast majority of my stuff wasn't affected!! I saw one roll-back on a site, meaning they lost a slight update made that week, but otherwise I'm not aware of anything.

I think in this case HostGator were just bombarded left, right and center and had to shut down the phones and support and actually deal with the issue, all hands on deck. I've always found them very responsive and helpful. This was an unusual situation.

[Moonraker]

View New Posts is bookmarked for me, and I can still gain access through that route. My IP address is blocked on a forum that has banned me, so I wouldn't have thought it was that difficult to do.

[sixret]

It's very easy to check who has a multinick by looking at their IP address.

I know someone has used a multinick by looking at their IP address.

[Moonraker]

Great to be back again. Let's hope the spamming Chinaman has been eliminated.

[sixret]

Hope there will be no more problem.

[Keith Robinson]

Finally!

HostGator said "...it appears that the site was flooded with a massive number of page requests by the sqlmap/1.0-dev crawler. Thus to help prevent this issue from recurring, I have denied subsequent requests from this bot..."

They said nothing about the Chinese IP address with an abnormally high amount of hits. I stopped that one in its tracks by denying it, and lo and behold, another Chinese IP address reared up in its place. I blocked that one too. I'll see if another comes up, and if so, I might just go ahead and block China! (Well, maybe not, but there's a whole range of known IPs from Chinese spammers I can block.)

Of course, I mentioned this IP address in a separate ticket, and HG just might not have got to that one yet!

This attack came at a really bad time. HostGator said, "My apologies for the response time, as our monitoring ticket queue has been abnormally high." I'm still ticked that it took so long to get around to investigating and implement a simple fix, but I'm glad we're back online again.

[pete9012S]

Thank you Keith for all the man hours and hard work you put into resolving this problem.

Best Wishes

Pete

[Eddie Muir]

It's great to have the website back. Many thanks Keith.

[Daisy]

Great to be back and it has seemed a lot longer than 4 days!