Rap Genius is hiring! Heroku's Ugly Secret by James Somers (Ft. Andrew Warner, ATodd, Chrissy & LEMON) 41

This video is processing – it'll appear automatically when it's done.

“It ain’t where you’re from, it’s where you’re at” Heroku's Ugly Secret by James Somers (Ft. Andrew Warner, ATodd, Chrissy & LEMON) 41

This video is processing – it'll appear automatically when it's done.

Weibull distribution Heroku's Ugly Secret by James Somers (Ft. Andrew Warner, ATodd, Chrissy & LEMON) 41

We found that the Weibull distribution with shape parameter = 0.46 is a reasonable approximation

So if you want to run this at home, you should do:

  • wshape = 0.46
  • wlambda = 50 / (log(2) ^ (1 / wshape))
  • req_durations_in_ms = pmin(30000, pmax(10, ceiling(rweibull(nreqs, wshape, wlambda))))

This video is processing – it'll appear automatically when it's done.

That's right — if your app needs 80 dynos with an intelligent router, it needs 4,000 with a random router. Heroku's Ugly Secret by James Somers (Ft. Andrew Warner, ATodd, Chrissy & LEMON) 41

If that seems like an outrageous figure, recall the birthday paradox, which shows that you only need to get 23 people in a room before there’s a 50% chance that two of them will share a birthday. The birthday paradox is called a paradox because nobody quite believes the number’s going to be that small; our minds vastly underestimate the probability of hash collisions.

See also the below graph, which plots the probability of all dynos being open (i.e., no collisions/queuing) given a certain number of simultaneous requests. This is with a naive router. (With an intelligent router, of course, this would look like a flat line at 1.0, since you’d never having any queuing, no matter how many requests you had.)

Here’s another “intuition pump.” Imagine you’re playing a game of Russian roulette. You’re holding a revolver with n chambers. All but one are loaded.

Before you put the barrel to your temple and pull the trigger, you’re given a choice between two methods for spinning the cylinder: (A) spin until you reach the first empty chamber, or (B) spin randomly. Which do you choose?

Unless you have a death wish, you choose A. Of course you choose A. And the larger n gets, the more you want to choose A, because as n grows, so too does the probability that a random spin will land on a bullet.

The analogy back to Heroku’s two routing regimes should be clear.

This video is processing – it'll appear automatically when it's done.

September 24th, 2013

In the case of the Russian roulette, IMHO, as n grows, the less you want to choose A (even though it’s always better to choose A over B). The probability you land on a bullet in A is 1/n (uniform law) which, over [1, +inf], decreases and tends to 0. In B, the probability is constant and equals 0. Your will to choose is A over B is then 1/n-0, which also decreases. Thought?

Marko
December 11th, 2013

“All but one are loaded” — means all chambers are full with bullets, but one. So, as N grows (chambers full with bullets) you want A more and more, because your probability diminishes (1 empty chamber in the ever growing N — chapmbers full of bullets).

Add a comment

“So what so what so what’s the scenario?” Heroku's Ugly Secret by James Somers (Ft. Andrew Warner, ATodd, Chrissy & LEMON) 41

This video is processing – it'll appear automatically when it's done.

Defines Heroku's Ugly Secret by James Somers (Ft. Andrew Warner, ATodd, Chrissy & LEMON) 41

This video is processing – it'll appear automatically when it's done.

Those machines at the Whole Foods checkout line Heroku's Ugly Secret by James Somers (Ft. Andrew Warner, ATodd, Chrissy & LEMON) 41

Whole Foods saw a big improvement in wait times at its busiest stores when it implemented a checkout line management system exactly analogous to Heroku’s old intelligent routing scheme.

This video is processing – it'll appear automatically when it's done.

Concurrency can be significantly less than the number of dynos Heroku's Ugly Secret by James Somers (Ft. Andrew Warner, ATodd, Chrissy & LEMON) 41

Another nasty consequence being that Heroku’s 30s request time limit, which used to be based on request time alone, is now based on request AND queue time. Which means that a fast request, taking say 2s, could still trigger an “H12” timeout error if it gets caught in a large queue at a dyno.

Which is exactly what happens — in our case, more than than a thousand times an hour.

This video is processing – it'll appear automatically when it's done.

February 14th, 2013

I am in the same boat with you, James. What strikes me hard is that heroku support told us they’ve not seen this with other apps and it is due to our application code, which is not. :(

Add a comment

Doc Heroku's Ugly Secret by James Somers (Ft. Andrew Warner, ATodd, Chrissy & LEMON) 41

This video is processing – it'll appear automatically when it's done.

Heroku log format Heroku's Ugly Secret by James Somers (Ft. Andrew Warner, ATodd, Chrissy & LEMON) 41

As described on this page

This video is processing – it'll appear automatically when it's done.