We love Heroku, they allowed us to focus on building an amazing site, they hooked us up Money Trees (Rap Genius Response to Heroku) by LEMON 12

From Heroku’s success story about Rap Genius:

Lehman explained that handling the increase in traffic was made simple with Heroku. Rather than focusing on the intracacies of managing server load, Heroku helped him to continue focusing on developing a solid application.

“There are some aspects of Heroku that make it trivial to scale. You just increase the number of dynos you have and that allows you to scale up without having to make any sort of logical changes to what your applications is doing,” said Lehman. “There’s no way to abstract away all of the details of increasing performance, but from a technical perspective it’s been pretty painless. Heroku guides you through it in a sense.”

This video is processing – it'll appear automatically when it's done.

We spent a bunch of time optimizing performance in the wrong places Money Trees (Rap Genius Response to Heroku) by LEMON 12

This was often egged on by Heroku. See this ticket between me and a Heroku engineer from over a year ago in which the engineer advised me to upgrade my database plan to solve my H12 problems (which were actually caused by in-dyno queuing which at the time I thought was impossible):

(click for bigger)

This video is processing – it'll appear automatically when it's done.

Click here to read our entire correspondence Money Trees (Rap Genius Response to Heroku) by LEMON 12

This video is processing – it'll appear automatically when it's done.

I documented for Adam both our simulation results and the many inaccuracies in Heroku’s docs, revealing to him all the same information that went into our final article Money Trees (Rap Genius Response to Heroku) by LEMON 12

Read it yourself in the full transcript of our email chain:

(Click for bigger)

This video is processing – it'll appear automatically when it's done.

Https://help.heroku.com/tickets/75238 Money Trees (Rap Genius Response to Heroku) by LEMON 12

You can’t access https://help.heroku.com/tickets/75238 unless you work at Heroku, but here’s a screenshot of the whole thing

(Click for bigger)

This video is processing – it'll appear automatically when it's done.

Heroku’s extensive apology and technical explanation of the queuing problems its Rails customers are facing Money Trees (Rap Genius Response to Heroku) by LEMON 12

This video is processing – it'll appear automatically when it's done.

However, here’s how New Relic should have looked: Money Trees (Rap Genius Response to Heroku) by LEMON 12

We made it look this way by patching New Relic. You can do this too if you install the heroku-true-relic gem that we wrote

This video is processing – it'll appear automatically when it's done.

Oren Teich, Heroku’s GM & COO Money Trees (Rap Genius Response to Heroku) by LEMON 12

Oren was the author of Heroku’s first apology blog post

This video is processing – it'll appear automatically when it's done.

In fact, Rap Genius just released a gem that makes New Relic display the actual time your app spends queuing rather than always displaying 0 Money Trees (Rap Genius Response to Heroku) by LEMON 12

Download heroku-true-relic today!

These new accurate queue numbers confirm the results of our simulations: we are currently running 250 dynos (monthly bill: $27,000) with an average throughput of ~11000 requests per minute

A simulation with those numbers estimates that the average queue time should be around 290ms, which is very close to the 324ms average New Relic now reports. We don’t have a ton of data yet with the accurate request queueing, but that’s pretty close!

This video is processing – it'll appear automatically when it's done.

Justin George
February 18th, 2013

Used to work on the New Relic Agent, this is not entirely a correct solution. Many machines on AWS experience clock skew, leading to greater (or negative) queue time length — that’s what the queue wait parameter was a work around for.

The correct solution would be to have the parameter added at the edge of the dyno, not at the beginning in the front-end routing framework. This is not a user-accessible area within heroku, however.

Just be aware that your queue time may be (perhaps massively) over or underreported using this method.

February 18th, 2013

Great point! This is far from a perfect solution, but it’s the best we’ve got now. It seems roughly accurate in the aggregate, at least for us, based on http://rapgenius.com/1506509. We’re continuing this discussion over on github for anybody who is curious — https://github.com/RapGenius/heroku-true-relic/issues/1.

Our main goal is to get people the right information, so that they can work on optimizing the right code paths in their app. Thanks Justin!

Add a comment

Simulated both routing algorithms as Rap Genius did Money Trees (Rap Genius Response to Heroku) by LEMON 12

Read more about the simulations in our original article

This video is processing – it'll appear automatically when it's done.