STEP has launched... but server crashed immediately

[bradtem]

I gotta say, I have always felt first-come first-served is not so different from a lottery sometimes.

It is different, of course. It does favour eagerness and spare time, as well as luck. And people do attribute failure to win more to themselves than to luck sometimes. It is much less prone to oversubscription (though as prone, if not more, to various established scalper systems which have been tuned for FCFS ticket systems.) Your bad luck is a bit more apparent -- you see that you didn't get in because you didn't get one of the lucky slots within the ticket server's capacity, as opposed to the opaque "the random number generator did not smile on you."

But neither system is particularly good for this purpose.

It should be noted that Amazon Web Services has immense capacity -- but you have to plan to use it, and pay for it. If your app doesn't know what to do with the thousands of instances it can buy on AWS, it won't make could use of them. Based on data, there were about 70,000 lottery entries (many for 2 tickets) and about 24,000 winners. So there could be at most around 45,000 people who might try to join STEP. Don't know what fraction of them were pounding on servers at noon. Of course, once you get slow, you get even slower as people keep retrying and reloading on you.

[lemur]

I REFRESHED 517 TIMES.. WHY ISNT IT WORKING?!

[vargaso]

The Ticketing for STEP seemed to be hosted by Amazon Web Services, yes that Amazon... Did you notice the URL theat was in the addrwess of the links sent?

No matter, even Amazon is slow at times I guess.

Amazon Web Services
http://www.amazonaws.com/
Amazon Web Services (AWS) delivers a set of services that together form a reliable, scalable, and inexpensive computing platform “in the cloud”.

The way it appeared to me was it was bouncing around several servers. The initial emailed link was to BM's servers. Which redirected to Amazon. The "SAVE" button page looked to be hosted on Amazon. However, hitting save seemed to redirect traffic back to burningman.com. I noticed that eplaya was very unresponsive during all of this. I got to the amazon page several times but it kept timing out when trying to redirect back to BM's servers. Eventually I got the confirmation message, but I think the big issue was the connection to BM's servers, not Amazon. Eplaya is working just fine again, so I would guess they shared the same connection. I doubt BM has as huge a pipe as Amazon.

Hitting save redirected you back to the confirmation screen on burningman.com, but the Save request went to Amazon servers. As others have said, it's not Amazon, it's inTicketing. Amazon servers can handle almost unlimited traffic depending on how you configure it for your purposes. All they provide are the physical boxes, it's up to the client (inTicketing) to configure it.

[trilobyte]

Every other person whose ever worked in an IT department or said hello to the IT folks in their company all claim to have the experience to resolve the problem of handling the crush of on-sale demand with hot ticket events. If that were the case, then why in the world aren't you making big bucks at any one of the ticketing vendors showing them just how easy it is to solve a problem that they spend vast fortunes to cope with? If InTicketing were the only ones who had those problems, then it would be easier to believe it was all their fault.

[RevDusty]

Every other person whose ever worked in an IT department or said hello to the IT folks in their company all claim to have the experience to resolve the problem of handling the crush of on-sale demand with hot ticket events. If that were the case, then why in the world aren't you making big bucks at any one of the ticketing vendors showing them just how easy it is to solve a problem that they spend vast fortunes to cope with? If InTicketing were the only ones who had those problems, then it would be easier to believe it was all their fault.

'Cuz I'm making big bucks where I am solving these and similar problems. Seriously.

[RevDusty]

Every other person whose ever worked in an IT department or said hello to the IT folks in their company all claim to have the experience to resolve the problem of handling the crush of on-sale demand with hot ticket events. If that were the case, then why in the world aren't you making big bucks at any one of the ticketing vendors showing them just how easy it is to solve a problem that they spend vast fortunes to cope with? If InTicketing were the only ones who had those problems, then it would be easier to believe it was all their fault.

And, Trilo, TicketBastard, as much as I loathe them, don't have this problem. I don't get kicked off the servers. I may not get the ticket ,but I don't get booted off after getting through. Big difference.

[lemur]

and as we know! even the best doods in the world end up fucking up their config ....even the people with plenty of capacity end up with bottlenecks and issues that just didnt come up in smaller scale testing.... even if you were able to simulate 50,000 people hitting it at once, odds are you probably also werent accoutning for 25,000 of them constantly hitting refresh over and over..

theres basically no real way to simulate the real thing beforehand, you just gotta hope for the best .. hope you did it right..

bonnaroo got screwed over by their implementation when their tickets went on sale.. and they surely have a bigger budget than burning man.. have 15+ corp sponsors (including DELL and StubHub) who surely allow them to be able to dedicate more cash to the project than burning man can.. and they still had problems..

s0 yeah, it sucks that they had problems, but it wasnt like issues would be unexpected..... and as bradtem mentions, as soon as things start getting slow the problem becomes compounded because you not only have 50,000 people hitting the site at once.... but you have 50,000 people refreshing over and over numerous times.. basically making themselves feel to the servers like double, or triple.. or even more than that.

[vargaso]

Every other person whose ever worked in an IT department or said hello to the IT folks in their company all claim to have the experience to resolve the problem of handling the crush of on-sale demand with hot ticket events. If that were the case, then why in the world aren't you making big bucks at any one of the ticketing vendors showing them just how easy it is to solve a problem that they spend vast fortunes to cope with? If InTicketing were the only ones who had those problems, then it would be easier to believe it was all their fault.

'Cuz I'm making big bucks where I am solving these and similar problems. Seriously.

The tools and methodologies exist, they really do (online banking for instance, WAY more complicated and much larger user base), it's just a matter of forking over the dough to get the right people in there. I don't fault BMORG this year, it's all new, even though some of us have been suggesting they find a new vendor for awhile now. If next year the same issues happen, then I don't know. And personally, I don't have the expertise, I'm a front-end developer, but I've worked on top of massively powerful systems (online banking and health care) that handle stupid amounts of traffic.

[lemur]

banks and healthcare providers also have billion $ budget.

[theCryptofishist]

banks and healthcare providers also have billion $ budget.

That's what you "99%ers" claim, anyway.

[RevDusty]

banks and healthcare providers also have billion $ budget.

Yes, they do.

But Amazon (and others) greatly equalize out those capabilities, especially when we are discussing renting machines for 30 or so minutes.

[vargaso]

banks and healthcare providers also have billion $ budget.

That is true. Even so, this process could have been handled better without having to spend millions. Tens of thousands, yes, but not millions.

[suicyco]

Every other person whose ever worked in an IT department or said hello to the IT folks in their company all claim to have the experience to resolve the problem of handling the crush of on-sale demand with hot ticket events. If that were the case, then why in the world aren't you making big bucks at any one of the ticketing vendors showing them just how easy it is to solve a problem that they spend vast fortunes to cope with? If InTicketing were the only ones who had those problems, then it would be easier to believe it was all their fault.

I don't think it was any sort of huge disaster. It just failed ungracefully. The 2000 who got in, did so in a relatively short amount of time. It could have been designed to fail in a more friendly way, however it did its purpose.

I work in IT and design systems that handle orders of magnitude larger datasets and network traffic. However, these systems are for much deeper pockets than BM. I'm not complaining though, it is what it is. I mean it only really lasted for what, 30 minutes? Frustrated some folks but hey, its over, you got in or you didn't, and thats that. I personally have very slim hopes of receiving tickets via the STEP program. I am working other avenues as best I can. I already have a ticket for myself if I cannot get one through STEP, however our camp needs two more.

[lemur]

banks and healthcare providers also have billion $ budget.

That is true. Even so, this process could have been handled better without having to spend millions. Tens of thousands, yes, but not millions.

how do you know ?

[hellolinds]

Really Inticketing doesn't give a fuck! and they don't have to. The Borg keeps using them again and again. Fail after fail. Failed last year? Here, have a go at it again. oopsie, cheaped out a bit on that 30 minute rush.

and still, there are 2,000 peeps that got through (happy I wouldn't say, just hanging onto small pathetic hopes...at least that's where I'm at) And so BMORG doesn't care either. It's been a total shitshow, how much worse could it get for them?

[lemur]

banks and healthcare providers also have billion $ budget.

Yes, they do.

But Amazon (and others) greatly equalize out those capabilities, especially when we are discussing renting machines for 30 or so minutes.

they still need all of the resources to design a system using the amazon infrastructure.... it doesnt build itself!

[vargaso]

banks and healthcare providers also have billion $ budget.

Yes, they do.

But Amazon (and others) greatly equalize out those capabilities, especially when we are discussing renting machines for 30 or so minutes.

This too. That's the genius of Amazon's cloud service, you don't have to buy and maintain anything, just rent. And has RevDusty says, it can literally be in 30 minute increments. You can rent a tiered solution for these kinds of things where your capacity is at maximum the first 30 minutes, then gradually scales down, as does the pricing.

With that said, hats off to the ticket dept., this has to have been the most stressful month of their BM lives, dealing with unprecedented demand and scrutiny. I don't envy them (unless they get a ticket out of the deal, ha!).

[vargaso]

banks and healthcare providers also have billion $ budget.

That is true. Even so, this process could have been handled better without having to spend millions. Tens of thousands, yes, but not millions.

how do you know ?

Because I work in this industry and have built the front-end aspect of contingency plans for server overload scenarios, wherein users get redirected to a landing page with explanations and instructions on what to do, so that people don't just sit there with a blank screen thinking their request is being processed.

[lemur]

this process could have been handled better without having to spend millions. Tens of thousands, yes, but not millions.

how do you know ?

Because I work in this industry and have built the front-end aspect of contingency plans for server overload scenarios, wherein users get redirected to a landing page with explanations and instructions on what to do, so that people don't just sit there with a blank screen thinking their request is being processed.

so... you dont know what resources burning man/inticketing have available.... and therefore you dont know if this process could have been handled any better.

experience elsewhere doesnt exactly translate to knowing what you could do if put in the shoes of the burningman/inticketing people today...

yeah, i can imagine a way it could be handled better too.... if i had an unlimited budget and as many resources as i could imagine.. but, i dont know what their budget is and i dont know what resources are available... perhaps this is the best that could have been done, perhaps it isnt... but none of us really know unless we are actually on the inside

[RevDusty]

That is true. Even so, this process could have been handled better without having to spend millions. Tens of thousands, yes, but not millions.

how do you know ?

Because I work in this industry and have built the front-end aspect of contingency plans for server overload scenarios, wherein users get redirected to a landing page with explanations and instructions on what to do, so that people don't just sit there with a blank screen thinking their request is being processed.

...and ultimately, if you do that right, you reduce the load on the servers from people who don't know what has happened to their requests.

[JungoRoad]

Dearest Troll-oh-bite,

"Moderator" is from the root "moderate".

Jungo Road

[beer]

Look familiar ?




I started clicking at 12:00 and did so at least 40-50 times. At least 5 of those times I got the page with the button to click to confirm, and got the same message after clicking it.
At 12:18 I finally got thru, and received the email a few minutes later that we're on the list (I requested 2 tickets).

It would be interesting to see how long it took to fill the queue. 20 minutes sound about right ?


k.

[vargaso]

so... you dont know what resources burning man/inticketing have available.... and therefore you dont know if this process could have been handled any better.

experience elsewhere doesnt exactly translate to knowing what you could do if put in the shoes of the burningman/inticketing people today...

yeah, i can imagine a way it could be handled better too.... if i had an unlimited budget and as many resources as i could imagine.. but, i dont know what their budget is and i dont know what resources are available... perhaps this is the best that could have been done, perhaps it isnt... but none of us really know unless we are actually on the inside

The thing is, Burning Man tickets are about the most expensive tickets for any event on the planet. People pay an assload of dough for the whole thing. I give BM WAY more slack that I would any other entity because of what it is and how much it means to me. But at some point, they have to step (har!) up and face reality. Like I've said, this year is a learning experience for everyone and I don't fault them one bit. I'm hoping things improve next year.

[Sassy Britches]

Because I work in this industry and have built the front-end aspect of contingency plans for server overload scenarios, wherein users get redirected to a landing page with explanations and instructions on what to do, so that people don't just sit there with a blank screen thinking their request is being processed.

That was the frustrating part.

I don't know much at all about how technology works, and I also really don't think that the BMORG has ill intentions - I think their hearts are with the community - but that doesn't negate the feelings of frustration and sadness that are prevalent among our friends and so many others who want so much to be home on the Playa this year... Nor does it stop us from hoping that all who do want to be there will find their way to face value tickets.

[remi]

banks and healthcare providers also have billion $ budget.

BM makes a few bucks too.

[theCryptofishist]

The thing is, Burning Man tickets are about the most expensive tickets for any event on the planet.

I'm gonna have some fun with this one...

[lemur]

im guessing all of the people who suggest they know how to make it better have already filled out a volunteer questionnaire... because they really want to help make it better!

http://www2.burningman.com/people/ volunteer questionnaire

and! they probably have also volunteered their assistance with the 'tech team'


http://www.burningman.com/participate/techweb.html

and they might have already emailed techie-volunteers (at) burningman (dot) com and told them all about their expertise in the field and offered themselves up to make this process better..

volunteer!

[theCryptofishist]

banks and healthcare providers also have billion $ budget.

BM makes a few bucks too.

Billions to spend on IT compared to an event that has ~20 million for everything. Meaning that the IT budgets we're talking about are 500 times as large--at a minimum.

[drutter]

this year is a learning experience for everyone and I don't fault them one bit. I'm hoping things improve next year.

That's a lovely sentiment, but once you've been through 3 or 4 years, it becomes harder and harder to say genuinely. "It was a learning experience, next time will be better!" works well for some things, like riding a bike through a sand-trap on the playa. Not as well for things like parachuting onto the Esplanade. Some of us are getting tired of re-using that expression when it comes to the ticket process.

[vargaso]

im guessing all of the people who suggest they know how to make it better have already filled out a volunteer questionnaire... because they really want to help make it better!

http://www2.burningman.com/people/ volunteer questionnaire

and! they probably have also volunteered their assistance with the 'tech team'


http://www.burningman.com/participate/techweb.html

and they might have already emailed techie-volunteers (at) burningman (dot) com and told them all about their expertise in the field and offered themselves up to make this process better..

volunteer!

I volunteered for the Web Team in 2009, I sat in on a couple meetings via phone, but at that point, tickets had never sold out and most of the talk was around migrating burningman.com to a new content management system. I am absolutely volunteering for next year.

Edited to add:
Not that I think I'm a genius and/or that the current team isn't up to the job or that I can magically fix things. The major thing I could offer is some outside perspective on similar use cases and acting as an information architect, if they don't already have one. Someone like RevDusty, who sounds like a heavy duty software/server architect, would be great too.