Site Outage 18-19 October 2022

Read-only messages from the Burning Man Staff
Locked
User avatar
trilobyte
Site Admin
Posts: 17257
Joined: Tue Mar 02, 2004 10:54 pm
Burning Since: 2004
Camp Name: Atomic Octopus
Location: Las Vegas
Contact:

Site Outage 18-19 October 2022

Post by trilobyte » Wed Oct 19, 2022 8:34 pm

Hi everyone, I wanted to take a moment to apologize for the outages we've experienced over the last couple days, as well as the general lack-of-performance recently. While things may have had a bit of an air of 'burned down, fell over, then sank into the swamp' about them, the tech team and I were scrambling to get things back up and running.

We appeared to have been caught up in something of a perfect storm. Not just garden variety spammos and the ever-increasing wave of nogoodnicks (which started a few months before Russia invaded Ukraine), but also the need to refresh database indexes, resynchronize some info, purging some cache, and play wack-a-mole with the site's extensions to find the culprit. In a nutshell, we ran into an issue with a misbehaving extension that we use internally for reporting spam to a group that monitors that activity. In the interim, the offending extension has been disabled, and we will work with the developer to get things resolved and then give it some rigorous testing before bringing it back in the future. I'm sorry for all the headaches and hassles and blank screens and SQL errors over the last few days, we should be back to whatever passes for normal around here :-)

The site was not attacked (the waves of spam we see are more of an excessive nuisance than any form of denial-of-service attack), the site was not hacked, and absolutely no user info was in any way compromised.


User avatar
trilobyte
Site Admin
Posts: 17257
Joined: Tue Mar 02, 2004 10:54 pm
Burning Since: 2004
Camp Name: Atomic Octopus
Location: Las Vegas
Contact:

Re: Site Outage 18-19 October 2022

Post by trilobyte » Wed Oct 19, 2022 8:39 pm

I know I also probably owe a bunch of people responses on a number of emails and private messages over the last several weeks, and I will get to them. My life outside Burning Man has been quite busy lately, the admin side of managing the board has been very time-consuming of late, and I am still digging through quite a lot of messages.

Locked

Return to “ePlaya Announcements”