Not bad, right?

See our annotations on Heroku’s Bamboo Routing Performance

While it’s true that the switch to random routing didn’t “degrade” performance on Cedar, that’s only because Cedar always used random routing and therefore always had terrible performance for Rails applications on thin (and other non-concurrent web servers)

Note that thin is the default web server on Cedar, so by default your Rails Cedar app will perform poorly. And of course you won’t even know it because the degradation in performance is caused by queuing that’s not measured by Heroku’s logs or New Relic

Click here for instructions on how to make New Relic report correct queue times

“Yesterday” (2/13/2013) is inaccurate – I started discussing the issue with Adam Wiggins (Heroku’s CTO) on 2/5/2013, and sent him the simulation results on 2/8/2013.

On 2/11/2013 he responded with an email that said:

I’m convinced that the best path forward is for one of your developers to work closely with [redacted] to modernize and optimize your web stack. If you invest this time I think it’s very likely you’ll end up with an app that performs the way you want it to at a price within your budget.

Also, Tim Watson approached Heroku about this in February 2011. Tim links to Adam’s response:

Tim,

You’re correct, the routing mesh does not behave in quite the way described by the docs. We’re working on evolving away from the global backlog concept in order to provide better support for different concurrency models, and the docs are no longer accurate. The current behavior is not ideal, but we’re on our way to a new model which we’ll document fully once it’s done.

In the meantime, you shouldn’t have any difficulties as long as you keep your web requests short (less than about 500ms), which is good practice anyway.

Sorry for any difficulty or confusion, and thanks for digging in and providing such a detailed analysis.

Adam

Fuck it, I’m gonna paste the whole thing in!!!

Shit’s getting CRAY.

Over the past 3 years we’ve grown from nothing to the biggest hip hop site on the Internet. Now, we’ve raised 15,000 racks from Andreessen Horowitz to become the biggest site PERIOD.

But we need more geniuses to make this happen – right now there are only 5 members of the RG fam who program!

SO: if you are

1. A genius Rails programmer
2. A genius iOS / Android programmer
3. A genius front end developer who can also design

AAANDDDDDD

You’re down to help build the Internet Talmud (aka the best hip hop site, the best lyrics site, the best music social network, the best wikipedia for explanations of all of text, etc, etc)

HIT UP jobs@rapgenius.com with:

1. Your name and online identity – i.e., your Twitter, Github, Blog, Stackoverflow account, personal website, etc. The more info the better.

2. Location – you can’t get the true RG cult effect working remotely, so only apply if you live in NYC or are willing to relocate (our office is in Williamsburg)

3. Educational background and most recent job (or just send your resume or whatever)

4. A little bit about your technical background, including something brief about what you like and dislike about the technologies you’ve used. We use Rails and Javascript (jQuery) extensively, so try to say something about both. Here are some other technologies we use: Git, Heroku, SCSS, Facebook API, Twitter API

5. If you come in for an interview, you’ll start by making a short technical presentation to the team and taking their questions. Generally this takes the form of “throw up a screenful of code and explain what it does / why it’s cool”, tho I’m down for whatever as long as it’s technical. What will you present?

6. THE MOST IMPORTANT: What have you built online that you’re proud of? Any cool designs to show off? Maybe send me some code?

This job has some pretty serious bennies!:

• SICK salary & equity. You won’t be taking a pay cut to work here (unless you work for a hedge fund..)
• Free bomb health insurance
• Unlimited free Seamless
• Gym in the building
• Whatever computer setup you want
• Free bike parking in the building
• We pay your phone bill and when you join we’ll upgrade you to the newest iPhone (or, IF YOU INSIST, Droid)

SWA-SWAAG!

As the coiner of this little phrase I’ve just gotta get it out there: DESPITE EVERYTHING I STILL LOVE HEROKU!!! Still a sick technology. But you just gotta be real with your customers.

Much love.

And of course the naive router scales much worse too. Here are the results from re-running the same simulation with 18,000 requests per minute instead of 9,000 requests per minute:

Strictly speaking this doc only applies to Heroku’s Bamboo stack, but Bamboo and Cedar behave exactly the same when it comes to routing requests for Rails

It’s non-zero because the router DOES queue requests when all your dynos are down or restarting (such as when you deploy)

Here’s a typical line of log output:

2013-02-13T20:47:22+00:00 heroku[router]: at=info method=GET path=/annotations/for_song_page?song_id=1373 host=rapgenius.com fwd=“129.138.136.18” dyno=web.112 queue=0 wait=0ms connect=2ms service=5983ms status=200 bytes=64144

Notice queue is 0 (it’s always 0) and the time spent queuing in the dyno is (I think) lumped in with overall service time