Issues with your account? Bug us in the Discord!

Firstones Server issues...

JackNJackN <font color=#99FF99>Lightwave Alien</font>
Getting REALLY slow Server Response, Timeouts and 500 Server Errors in the last few weeks...

Anything we should check into?
«1

Comments

  • Random ChaosRandom Chaos Actually Carefully-selected Order in disguise
    I only noticed that the last day. Looked into it a little tonight. The server host is reporting the issue as resolved. Let me know if anyone is still having issues and I can put in a trouble ticket.

    Host info on our server, as of today, 9pm EST (6pm PST):
    [quote]
    Disk errors surfaced after the last reboot. A filesystem check is being performed at the moment. All services ( http, ftp, etc ) will remain offline until the filesystem check completes. We hope to have the machine back online shortly. We sincerely apologize for any inconvenience this has caused and appreciate your continued patience. Please check back for future updates.
    [B]Update Sept 6th 5:07pm PST: [/B] The server has started, and apaches are up and running. If you’re still seeing problems, please let us know ASAP. — Brian H

    [/quote]
  • Random ChaosRandom Chaos Actually Carefully-selected Order in disguise
    Grr.

    Got a 500 error on posting that. Looking into it.
  • Random ChaosRandom Chaos Actually Carefully-selected Order in disguise
    Testing
  • Random ChaosRandom Chaos Actually Carefully-selected Order in disguise
    Hmm, looks to be better suddenly. I'm going to hold off on the support ticket until it starts acting up again.
  • Random ChaosRandom Chaos Actually Carefully-selected Order in disguise
    Things are running smoother, but it is nothing I did. I sent a support request in anyway in case the smoother running is a fluke. Given that they asked for us to report problems, went ahead and did so.
  • JackNJackN <font color=#99FF99>Lightwave Alien</font>
    Working much better than it has for the last few days. :)

    Did I say weeks, I meant days... :s
  • It's still running slow for me and I'm getting Internal Server Errors.
  • MessiahMessiah Failed Experiment
    Ive only had one problem, and that was yesterday when I tried to view this thread and it timed out several times ironically..
  • The problems started for me on Monday, this is actually the first time I've made it on site since then.
  • Random ChaosRandom Chaos Actually Carefully-selected Order in disguise
    According to Dreamhost we are going over our memory limits for PHP, causing the scripts to die.

    Looks like the culprit might be the wiki, specifically a massive amount of spam on it combined with search engines choking on trying to index that spam.

    Therefore, temporarily:

    * Disabled new user creation on the wiki
    * Disabled editing on the wiki
    * Blocked search engine indexing of the wiki

    I am going to try to do some mass cleanup of the wiki by editing the database over the next few days to get rid of this spam, then I will re-enable search engine indexing and editing. I will need to investigate if we can do a better job of blocking spam accounts from being created before re-enabling new user creation.
  • ShadowDancerShadowDancer When I say, "Why aye, gadgie," in my heart I say, "Och aye, laddie." London, UK
    Careful RC! People might start thinking you actually have a use! :p
  • Random ChaosRandom Chaos Actually Carefully-selected Order in disguise
    I have cleaned up most of the spam on the wiki, banned all users created since the spamming started, and re-enabled editing.

    I know there are some defaced pages still floating around the wiki, but it will be a bit more involved to track them down than simple mass editing via the database.

    There were around 6000 spam pages, some of them serving images uploaded to our server via MediaWiki, which I still need to locate and cull on disk. 761 images...yuck. All the database entries are gone so no one can find them at least.
  • Random ChaosRandom Chaos Actually Carefully-selected Order in disguise
    All images culled. 8427 total removed. Those include variant sizes of each one (most images had 2-3 sizes created).

    Removed several hundred MBs of spam data from the harddrive.

    And now it is time for dinner, an hour and a half late thanks to fixing Firstones :p
  • I'm still having the same problems.
  • Random ChaosRandom Chaos Actually Carefully-selected Order in disguise
    I am monitoring the logs right now for 500 errors. We've had a few dozen since I cleaned up the wiki, but nothing like what we were having before that. I'll continue to try and track down other culprits, but so far I don't have any leads.
  • FreejackFreejack Jake the Not-so-Wise
    Wait,,,we have a server?

    I thought we just existed in the nebulous of the net...

    Jake
  • JackNJackN <font color=#99FF99>Lightwave Alien</font>
    Hey RC, Bob over on Foundation3D installed a module for VBB or whatever the hell he uses that will kill new users who miraculously create an account within several seconds (bots I guess...)

    There might be something similar for the wiki...

    Just thinking out loud.

    And do you have to restart the server to clear all that indexed crap in the database out of memory to get the reults?
  • Random ChaosRandom Chaos Actually Carefully-selected Order in disguise
    Well, the underlying issue isn't really spam related. It's bot indexing of said spam.

    I've been monitoring the logs for the past several hours. Every time we start getting 500 errors, Microsoft BingBot is busy doing a massive indexing binge. It will query several hundred (or thousand) pages for changes over a couple minute period, during which 500 errors sprout like rabbits for just about anyone else accessing the site at the same time.

    I've added a crawl delay configuration directive to robots.txt for Bingbot only which will hopefully slow it down, but it may take a day or so before it actually gets to rechecking robots.txt.

    In the interim, not much I can do about speeding it up.
  • We should not allow robots to crawl our index!

    That's exactly what happened in Terminator...
  • JackNJackN <font color=#99FF99>Lightwave Alien</font>
    Microsoft Bing great, we should block that f*cker altogether...
  • Vertigo1Vertigo1 Official Fuzzy Dice of FirstOnes.com
    I don't suppose IP blocking the bingbot for a few days, would help?
  • JackNJackN <font color=#99FF99>Lightwave Alien</font>
    Just like Microsoft to cause a Denial of Service attack on our server...
  • OK, now everything is working just as good as it used to.
  • Random ChaosRandom Chaos Actually Carefully-selected Order in disguise
    There are still occasional 500 errors, but not as often. One or two every few hours, per the logs. Still trying to determine if there is anything more I can do to stop them.
  • The site loads for me now though, and is MUCH faster!
  • MessiahMessiah Failed Experiment
    I couldnt get on here at all yesterday afternoon..
  • If something good came out of those 500 -errors, its this thread. One of the most active ones for awhile :)
  • And the error appears to be back, not so frequent, but still.
  • MessiahMessiah Failed Experiment
    Excellent, then we can talk a bit more about it. :)
  • Random ChaosRandom Chaos Actually Carefully-selected Order in disguise
    Actually, it never stopped. Still have not figured out every underlying cause. I just stopped the bulk of them so most of the time it works.
Sign In or Register to comment.