Site Maintenance & Downtime (July 16/17, 2013)

We're doing it wrong...we know
User avatar
trilobyte
Site Admin
Posts: 15958
Joined: Tue Mar 02, 2004 10:54 pm
Burning Since: 2004
Camp Name: Eridu Society
Location: San Francisco
Contact:

Site Maintenance & Downtime (July 16/17, 2013)

Postby trilobyte » Tue Jul 09, 2013 8:33 pm

Further investigation into the wonkiness we've been experiencing over the last several days has revealed that we are in fact, awesome. Okay, well it's revealed that the problems were a combination of some work being done with our hosts, a change in the server backup strategy we had in place, and quite a bit more traffic on the boards these days (as opposed to the olden days). :D

Anyhow, the three things combined spelled intermittently terrible performance, and connection timeouts happening a lot for some users. Even after our cloud server overlords finished up their stuff, we've wrestled with performance issues. For the short term we've reverted to the previous server backup strategy, and some behind-the-scenes tweaks (and a good swift kick or two) have helped improve performance a little. For a longer term solution (significantly increasing our capacity), we need to take the board down for a bit.

We're scheduling that down time for the night of July 16. Starting at 10pm 9pm PDT on July 16, and continuing until around 2am PDT on the morning of July 17, ePlaya will be offline for maintenance and improvements. Sorry for any inconvenience that the downtime will cause, but the increased capacity will keep us from experiencing performance problems or outages as we head into the home stretch for this year's burn.

Many many thanks to Cat and Backbone on the IT team for their help and support.

User avatar
trilobyte
Site Admin
Posts: 15958
Joined: Tue Mar 02, 2004 10:54 pm
Burning Since: 2004
Camp Name: Eridu Society
Location: San Francisco
Contact:

Re: Site Maintenance & Downtime (July 16/17, 2013)

Postby trilobyte » Mon Jul 15, 2013 10:37 pm

Bump, and update! The site will be down for maintenance and improvements tomorrow night. I will be taking the board offline at 9pm PDT (not 10pm, as originally announced). I apologize in advance for any inconvenience, but look forward to the improved performance we'll see once we finish up in the early hours of Wednesday morning.

User avatar
trilobyte
Site Admin
Posts: 15958
Joined: Tue Mar 02, 2004 10:54 pm
Burning Since: 2004
Camp Name: Eridu Society
Location: San Francisco
Contact:

Re: Site Maintenance & Downtime (July 16/17, 2013)

Postby trilobyte » Wed Jul 17, 2013 12:03 am

And...we're back!

MASSIVE thanks to Backbone for all his help and technological greatness!

Thanks everybody for your patience during the downtime, as well as over the last couple weeks with the board's slow performance. We should hopefully be good to go for the foreseeable future.

User avatar
trilobyte
Site Admin
Posts: 15958
Joined: Tue Mar 02, 2004 10:54 pm
Burning Since: 2004
Camp Name: Eridu Society
Location: San Francisco
Contact:

Re: Site Maintenance & Downtime (July 16/17, 2013)

Postby trilobyte » Wed Jul 17, 2013 12:10 am

Google's DNS servers still have the old IP address cached, so some users may not be able to access the board for another hour or two.

User avatar
Elliot
Posts: 7278
Joined: Sun Dec 18, 2005 8:41 pm
Burning Since: 2006
Camp Name: Elliot’s Bicycle Service
Location: Clearlake, Northern California
Contact:

Re: Site Maintenance & Downtime (July 16/17, 2013)

Postby Elliot » Wed Jul 17, 2013 7:21 am

If we still believed in gods, you wizards of electric computing machines would be it.
Elliot's Naked Bicycle Service, Piano Bar & Body Painting
- Noon-6 daily, in the 4:30 Plaza at G -

-----Bike come unglued? Take it to the nude dude!-----

Website: http://www.elliotsbikes.org

Awesome? Yes, Ma'am, we do Awesome.

Image

User avatar
Aurelia
Posts: 2287
Joined: Mon Aug 20, 2007 8:34 pm
Burning Since: 1986
Camp Name: ALC
Location: San Fracisco, Bay Area

Re: Site Maintenance & Downtime (July 16/17, 2013)

Postby Aurelia » Wed Jul 17, 2013 7:27 am

Thank you Trilo !
formerly aserendipity

User avatar
trilobyte
Site Admin
Posts: 15958
Joined: Tue Mar 02, 2004 10:54 pm
Burning Since: 2004
Camp Name: Eridu Society
Location: San Francisco
Contact:

Re: Site Maintenance & Downtime (July 16/17, 2013)

Postby trilobyte » Wed Jul 17, 2013 11:29 am

Things may be sporadically herky-jerky (yes, that's a technical term!) for the next dozen or so hours. Turns out that Google still hasn't updated all of it's cache, that could end up taking up to 24 hours. Your mileage may vary, but what I've noticed before is that 80% of the time page loads are pretty close to instantaneoous (I'm on a fast connection). The other 20% is still sluggish, or the page may even time out. Once things in DNS land are sorted, we should be back to whatever passes for normal around here :)

User avatar
Canoe
Posts: 2734
Joined: Sun Jul 31, 2011 8:01 pm

Re: Site Maintenance & Downtime (July 16/17, 2013)

Postby Canoe » Wed Jul 17, 2013 11:31 am

I'm getting response times all over the map. One page will load in < 2 seconds. The next, 45 seconds (most typical long time). The next will timeout with no response.
Just came back to the site and I was listed as still signed in...

ah
your post beat mine

I was wondering about all the bots I was seeing.
Jar Jar Sith Lord
.
Odd. No bears in the dump. Oh well, lets go across the road & pick blueberries.
.
... but don't harm the red dragon that frequents the area from time to time. He and I have an agreement.

User avatar
trilobyte
Site Admin
Posts: 15958
Joined: Tue Mar 02, 2004 10:54 pm
Burning Since: 2004
Camp Name: Eridu Society
Location: San Francisco
Contact:

Re: Site Maintenance & Downtime (July 16/17, 2013)

Postby trilobyte » Wed Jul 17, 2013 11:59 am

There's been no change in the kinds and types of bots that we allow on the site (we let in the usual suspects...Google, Bing, Yahoo, the Alexa crawler, etc). You may be seeing more of them on the bottom of the page because I've opened up our 'Users Online' setting back to 30 minutes, which is what we had it set to for a long time.

User avatar
Elderberry
Moderator
Posts: 14017
Joined: Tue Jul 17, 2007 10:00 pm
Burning Since: 2007
Camp Name: Mudskipper Cafe
Location: Palm Springs
Contact:

Re: Site Maintenance & Downtime (July 16/17, 2013)

Postby Elderberry » Wed Jul 17, 2013 2:08 pm

Canoe wrote:I'm getting response times all over the map. One page will load in < 2 seconds. The next, 45 seconds (most typical long time). The next will timeout with no response.
Just came back to the site and I was listed as still signed in...

ah
your post beat mine

I was wondering about all the bots I was seeing.


I'm getting the same here.

Are you sure this is a DNS issue? There could be a problem accessing the site for the first time, but once the requester's ISP DNS server has received the correct information,it is cached and all further requests are then answerd by that server. Doesn't make sense to me.
JK
Image
http://www.mudskippercafe.com
When I was a kid I used to pray every night for a new bicycle.
Then I realised that the Lord doesn't work that way so I stole one and asked Him to forgive me.

User avatar
lemur
Posts: 3599
Joined: Fri Feb 13, 2009 2:07 pm
Camp Name: Plug N Play Camp
Location: Madagascar

Re: Site Maintenance & Downtime (July 16/17, 2013)

Postby lemur » Wed Jul 17, 2013 2:36 pm

yeah! or something.

it doesnt seem like a regular DNS propogation issue to me, i.e. the TTL on something needs to expire in a cache before the new IP shows up on a DNS resolve request



the TTL on eplaya.burningman.org seems to be 15 seconds..

if ya sit there and ping eplaya.burningman.org a few times (like in different requests..) after about 15 seconds you might encounter a part where it says it doesnt exist

~: ping eplaya.burningman.org
PING eplaya.burningman.org (54.213.17.175) 56(84) bytes of data.
^C
--- eplaya.burningman.org ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3505ms

~: ping eplaya.burningman.org
ping: unknown host eplaya.burningman.org
~: ping eplaya.burningman.org
ping: unknown host eplaya.burningman.org


it looks to me like after 15 seconds its hitting up another one of the resolvers listed in the WHOIS:

Domain servers in listed order:
NS3.BURNINGMAN.COM 65.161.206.37
NS.HYPERREAL.ORG
NS1.BURNINGMAN.COM 209.237.226.70

(i dont know if the middle one not having an IP address attached to it may cause problems)

all 3 of the resolvers report 'the correct' ip address for eplaya..

~: dig 65.161.206.37 eplaya.burningman.org
eplaya.burningman.org. 15 IN A 54.213.17.175

~: dig NS.HYPERREAL.ORG eplaya.burningman.org
eplaya.burningman.org. 15 IN A 54.213.17.175

~: dig 209.237.226.70 eplaya.burningman.org
eplaya.burningman.org. 15 IN A 54.213.17.175



my guess is that.. some systems are balking at the 2nd nameserver entry in the WHOIS being blank.

and with such a short TTL.. youre ending up with that 2nd nameserver (without a IP addy) in the WHOIS often..



just a guess!


(in other news: i think the previous issues were unrelated to this)

as for WHOIS records... THOSE may take a while to update as the domain delegater updates them on their own schedule, or not..
Don't link to anything here!

User avatar
Bless
Posts: 1067
Joined: Tue Jul 16, 2013 8:43 pm
Burning Since: 2012
Location: District of Columbia

Re: Site Maintenance & Downtime (July 16/17, 2013)

Postby Bless » Wed Jul 17, 2013 7:12 pm

This site isn't working on my desktop computer, only my iPhone. I tried three different browsers and tried clearing my history and caches. Please help!
FUCK YOU I'M A WIZARD. FUCK YOU I'M A SHARK.

User avatar
trilobyte
Site Admin
Posts: 15958
Joined: Tue Mar 02, 2004 10:54 pm
Burning Since: 2004
Camp Name: Eridu Society
Location: San Francisco
Contact:

Re: Site Maintenance & Downtime (July 16/17, 2013)

Postby trilobyte » Wed Jul 17, 2013 7:29 pm

The problem should work itself out by around midnight PDT tonight, as the the various DNS servers driving the web replace the old info with the new.

msj2u
Posts: 336
Joined: Mon Feb 04, 2013 10:17 pm
Burning Since: 2003
Location: Inland Empire, CA

Re: Site Maintenance & Downtime (July 16/17, 2013)

Postby msj2u » Wed Jul 17, 2013 8:13 pm

Trilobyte,

Should have said this earlier, thanks to you and the team for the work on the site!

MSJ
The older I get, the less patience I have for stupid.

I can't be muted

User avatar
Elderberry
Moderator
Posts: 14017
Joined: Tue Jul 17, 2007 10:00 pm
Burning Since: 2007
Camp Name: Mudskipper Cafe
Location: Palm Springs
Contact:

Re: Site Maintenance & Downtime (July 16/17, 2013)

Postby Elderberry » Wed Jul 17, 2013 9:21 pm

pinging is never a reliable indicator as it is often blocked either on the server or the firewall.
JK
Image
http://www.mudskippercafe.com
When I was a kid I used to pray every night for a new bicycle.
Then I realised that the Lord doesn't work that way so I stole one and asked Him to forgive me.

User avatar
lemur
Posts: 3599
Joined: Fri Feb 13, 2009 2:07 pm
Camp Name: Plug N Play Camp
Location: Madagascar

Re: Site Maintenance & Downtime (July 16/17, 2013)

Postby lemur » Thu Jul 18, 2013 12:47 am

the point of that wasnt the ping ..it was the name resolution
Don't link to anything here!

User avatar
trilobyte
Site Admin
Posts: 15958
Joined: Tue Mar 02, 2004 10:54 pm
Burning Since: 2004
Camp Name: Eridu Society
Location: San Francisco
Contact:

Re: Site Maintenance & Downtime (July 16/17, 2013)

Postby trilobyte » Thu Jul 18, 2013 9:44 am

If anyone's still experiencing issues with connecting to the site, or maintaining that connection, please send me a private message with the details.

User avatar
Bless
Posts: 1067
Joined: Tue Jul 16, 2013 8:43 pm
Burning Since: 2012
Location: District of Columbia

Re: Site Maintenance & Downtime (July 16/17, 2013)

Postby Bless » Thu Jul 18, 2013 10:05 am

I was having sporadic problems this morning, but it seems to be all fixed now. Thank you so much!
FUCK YOU I'M A WIZARD. FUCK YOU I'M A SHARK.


Return to “ePlaya Feedback”

Who is online

Users browsing this forum: No registered users and 0 guests