lil_tee

According to British politician Norman Fowler’s Wikipedia bio:

Fowler later resigned from the cabinet as Employment Secretary in January 1990, becoming the first politician to cite “to spend more time with my (his) family” as his reasoning

Fowler wrote about the ordeal in The Guardian nearly 20 years later:

Some [politicians] had been sacked in one of the regular Thatcher reshuffles and one or two others … had resigned on issues of principle. But up to that point no one had actually left to “spend more time with their family”.

In truth, I did not use those by now much quoted words. In my letter of resignation to the prime minister, I wrote: “I have a young family and for the next years I should like to devote more time to them while they are still so young.”

See: https://github.com/kch/rack-timeout

The R programming language is a great choice for putting together a quick simulation script. You can find a gist of the script we used here

Rap Genius contributor Mark writes:

Or deadlock a few dynos and see if request timeouts increase? That would take a few minutes.

In December 2010, when Salesforce.com acquired Heroku, Salesforce founder Parker Harris wrote on his blog:

But most of all, it was Heroku’s dedication to their developers – the customers of their service — that impressed me. They were clearly application developers themselves, who understood the needs and desires of Ruby app devs in a direct and visceral way. Their website, the way they communicated with developers, and their product itself was pitch perfect.

https://twitter.com/StewFeagleton/status/302511807325687808

UPDATE 2/18: See bottom of this annotation

UPDATE 2/14: We received some helpful suggestions that we have added to our simulations.

Choice of two routing is the naive method from before, with the twist that when you assign a request to a random dyno, if that dyno is already busy then you reassign the request to a second random dyno, with no regard for whether the second dyno is busy

Unicorn allows a single dyno to process multiple requests simultaneously, and each dyno sends its requests to its workers in the intelligent style. For example, say dyno D has 2 workers: A and B. If request R is sent to D, and both A and B are already busy, D will intelligently assign R to the first worker, A or B, that becomes available

We also received a lot of feedback that our response times are too slow, and we need to “kill the tail” of our slow responses. We wholeheartedly agree, and we work every day to improve it, but it seems that given any even remotely realistic distribution of request times, the random routing becomes a problem as you scale.

Here are some graphs that show the 4 routing styles: intelligent, naive, naive with choice of two, and naive with unicorn. The first graph uses Rap Genius’s empirical distribution of request times (median 46 ms, 99th percentile 3144 ms)

This second graph uses a simulated Weibull distribution with median 50 ms, 99th percentile 537 ms, 99.9th percentile 898 ms

The graphs show that both choice of two and unicorn offer meaningful improvements over naive routing, but even with the much faster response times profile, you will still see non-trivial queueing as your traffic grows. Here’s a final graph with the faster response times profile, but doubling the requests per minute to 18,000

UPDATE 2/18:

Peter Bailis writes:

FWIW, I’m pretty sure that your “Choice of Two” load balancing differs from standard “Power of Two Choices” load balancing

The key difference is that, in standard “Power of Two Choices” balancing, you query two Dynos and send the request to the less loaded: “each request is placed in the least loaded of d >= 2 Dynos chosen independently and uniformly at random”

So I updated results to include this “Power of two” approach. The choice of two and power of two have the same % of requests queued, which makes sense because in either case, if one of the two dynos is free, the request will get assigned to that dyno.

However, the power of two method lowers the average queue time, because in the case where both selected dynos are busy, the “choice” method picks randomly, while the “power” method assigns to the dyno with fewer requests queued (note that fewer requests queued does not necessarily mean it will be available first, but more often than not that will be true)

Here are results using a scenario with 9,000 requests per minute, following our simulated Weibull distribution:

The up-to-date R code is available on Github

We also ran the simulation using a faster distribution of request times. If you use a Weibull distribution with shape = 0.8 and scale = 79.06, you this distribution of request times:

Here is what the simulations look like with these faster requests:

Naive Dynos

Intelligent Dynos

and a table of results:

So when the requests are faster, you only need 30 intelligent dynos to keep up, but 30 naive dynos still results in 44% of requests getting queued

An example of a self-referential annotation: the first result in a Google search for the phrase “ready for dome” is this annotation

On the other hand, the first search result for “ready to bone” is an UnderstandRap link…

Classic corporate retreat moment at the end where the audience has no idea how to react, which somehow results in a half-hearted standing O

If “higher standards” involve attempting to foreclose on people who don’t even have mortgages, then I’d hate to see what lower standards are…