Unexplainable, high latencies for some requests Routing Performance Update by Jesper Joergensen (Ft. Heroku & LEMON) 9

I.e., when requests that are normally fast queue behind very slow requests at the dyno level:

This video is processing – it'll appear automatically when it's done.

But until this week, we failed to see a common thread among these reports Routing Performance Update by Jesper Joergensen (Ft. Heroku & LEMON) 9

This is hard to believe. When Tim Watson approached Heroku in February 2011 Heroku’s response was:

Tim,

You’re correct, the routing mesh does not behave in quite the way described by the docs. We’re working on evolving away from the global backlog concept in order to provide better support for different concurrency models, and the docs are no longer accurate. The current behavior is not ideal, but we’re on our way to a new model which we’ll document fully once it’s done.

In the meantime, you shouldn’t have any difficulties as long as you keep your web requests short (less than about 500ms), which is good practice anyway.

Sorry for any difficulty or confusion, and thanks for digging in and providing such a detailed analysis.

Adam

This video is processing – it'll appear automatically when it's done.

Green links Bamboo Routing Performance by Oren Teich (Ft. Heroku & LEMON) 4

Not bad, right?

This video is processing – it'll appear automatically when it's done.

UPDATE 2/15: Click to read our response to Heroku's apology Heroku's Ugly Secret by James Somers (Ft. Andrew Warner, ATodd, Chrissy & LEMON) 41

See our annotations on Heroku’s Bamboo Routing Performance

This video is processing – it'll appear automatically when it's done.

Ruby on Rails apps running on Bamboo have experienced a degradation in performance over the past 3 years as we have scaled. Bamboo Routing Performance by Oren Teich (Ft. Heroku & LEMON) 4

While it’s true that the switch to random routing didn’t “degrade” performance on Cedar, that’s only because Cedar always used random routing and therefore always had terrible performance for Rails applications on thin (and other non-concurrent web servers)

Note that thin is the default web server on Cedar, so by default your Rails Cedar app will perform poorly. And of course you won’t even know it because the degradation in performance is caused by queuing that’s not measured by Heroku’s logs or New Relic

Click here for instructions on how to make New Relic report correct queue times

This video is processing – it'll appear automatically when it's done.

Yesterday, one of our customers let us know about significant performance issues they have experienced on Heroku. Bamboo Routing Performance by Oren Teich (Ft. Heroku & LEMON) 4

“Yesterday” (2/13/2013) is inaccurate – I started discussing the issue with Adam Wiggins (Heroku’s CTO) on 2/5/2013, and sent him the simulation results on 2/8/2013.

On 2/11/2013 he responded with an email that said:

I’m convinced that the best path forward is for one of your developers to work closely with [redacted] to modernize and optimize your web stack. If you invest this time I think it’s very likely you’ll end up with an app that performs the way you want it to at a price within your budget.

Also, Tim Watson approached Heroku about this in February 2011. Tim links to Adam’s response:

Tim,

You’re correct, the routing mesh does not behave in quite the way described by the docs. We’re working on evolving away from the global backlog concept in order to provide better support for different concurrency models, and the docs are no longer accurate. The current behavior is not ideal, but we’re on our way to a new model which we’ll document fully once it’s done.

In the meantime, you shouldn’t have any difficulties as long as you keep your web requests short (less than about 500ms), which is good practice anyway.

Sorry for any difficulty or confusion, and thanks for digging in and providing such a detailed analysis.

Adam

This video is processing – it'll appear automatically when it's done.

Rap Genius is hiring! Heroku's Ugly Secret by James Somers (Ft. Andrew Warner, ATodd, Chrissy & LEMON) 41

Fuck it, I’m gonna paste the whole thing in!!!

Shit’s getting CRAY.

Over the past 3 years we’ve grown from nothing to the biggest hip hop site on the Internet. Now, we’ve raised 15,000 racks from Andreessen Horowitz to become the biggest site PERIOD.

But we need more geniuses to make this happen – right now there are only 5 members of the RG fam who program!

SO: if you are

  1. A genius Rails programmer
  2. A genius iOS / Android programmer
  3. A genius front end developer who can also design

AAANDDDDDD

You’re down to help build the Internet Talmud (aka the best hip hop site, the best lyrics site, the best music social network, the best wikipedia for explanations of all of text, etc, etc)

HIT UP jobs@rapgenius.com with:

  1. Your name and online identity – i.e., your Twitter, Github, Blog, Stackoverflow account, personal website, etc. The more info the better.

  2. Location – you can’t get the true RG cult effect working remotely, so only apply if you live in NYC or are willing to relocate (our office is in Williamsburg)

  3. Educational background and most recent job (or just send your resume or whatever)

  4. A little bit about your technical background, including something brief about what you like and dislike about the technologies you’ve used. We use Rails and Javascript (jQuery) extensively, so try to say something about both. Here are some other technologies we use: Git, Heroku, SCSS, Facebook API, Twitter API

  5. If you come in for an interview, you’ll start by making a short technical presentation to the team and taking their questions. Generally this takes the form of “throw up a screenful of code and explain what it does / why it’s cool”, tho I’m down for whatever as long as it’s technical. What will you present?

  6. THE MOST IMPORTANT: What have you built online that you’re proud of? Any cool designs to show off? Maybe send me some code?

This job has some pretty serious bennies!:

  • SICK salary & equity. You won’t be taking a pay cut to work here (unless you work for a hedge fund..)
  • Free bomb health insurance
  • Unlimited free Seamless
  • Gym in the building
  • Whatever computer setup you want
  • Free bike parking in the building
  • We pay your phone bill and when you join we’ll upgrade you to the newest iPhone (or, IF YOU INSIST, Droid)

SWA-SWAAG!

This video is processing – it'll appear automatically when it's done.

December 13th, 2013

i guess i’m gona have to build a resume. i’ll be back shortly.

Add a comment

Heroku Swindle Factor™ Heroku's Ugly Secret by James Somers (Ft. Andrew Warner, ATodd, Chrissy & LEMON) 41

As the coiner of this little phrase I’ve just gotta get it out there: DESPITE EVERYTHING I STILL LOVE HEROKU!!! Still a sick technology. But you just gotta be real with your customers.

Much love.

This video is processing – it'll appear automatically when it's done.

mjm
February 14th, 2013

Every request we be hustlin

Add a comment

To bring queuing down to an acceptable level (<10ms), you’d need to crank your app to 4,000 dynos, or fifty times your allotment in the intelligent case. Heroku's Ugly Secret by James Somers (Ft. Andrew Warner, ATodd, Chrissy & LEMON) 41

And of course the naive router scales much worse too. Here are the results from re-running the same simulation with 18,000 requests per minute instead of 9,000 requests per minute:

This video is processing – it'll appear automatically when it's done.

Elsewhere in their current docs Heroku's Ugly Secret by James Somers (Ft. Andrew Warner, ATodd, Chrissy & LEMON) 41

Strictly speaking this doc only applies to Heroku’s Bamboo stack, but Bamboo and Cedar behave exactly the same when it comes to routing requests for Rails

This video is processing – it'll appear automatically when it's done.