Little green friend

Posted by Matt Jankowski

Jul 18

Here’s a brief introduction to a new application we’ve been working on. The application is called hoptoad, and it’s a hosted web service which can be used as an error reporting and analysis tool by Rails applications. The service is running in the Engine Yard cloud and will be completely free when it launches, so go ahead and sign up to that mailing list and we’ll let you know when it’s live.

For the academically curious, the word “hoptoad” is railroad industry slang for a derailed train. At least, it used to be when the railroad industry had slang.

Background

We’ve been using Jamis Buck’s “Exception Notifier” Rails plugin for years now, and for the most part it’s accomplished what we wanted. For the uninitiated, the Exception Notifier plugin catches all errors that haven’t already been caught, and sends an email with a formatted error report about what happened, where it happened, the session data, the params, the environment info, etc.

We’ve used this with great success, probably on ~30 projects or so, and it’s been a reliable way to know when there is an issue with one of the production applications you are maintaining.

Problems

The problems we ultimately ran into here…

We would occasionally get completely hammered with a stream of app error emails. In most cases, this was something completely innocent (like a bunch of RoutingError’s caused by a new release version changing the url structure, or by the rails2 upgrade sending InvalidAuthenticityToken, or by a search engine doing something weird like sending us an HTTP ‘OPTIONS’ verb in a request, etc). In other cases, we had legitimately screwed up and had an app which was throwing off errors left and right – but in that case, once the first 10 emails trickled in, we already knew there was an urgent problem to solve, and the next 1000 emails didn’t make it any more urgent, they just clogged up our inboxes.
There’s no way to measure success over time. Am I correcting the errors I had last deploy? Am I getting more or less errors each month?
The errors are a big stream. If I get error A 10 times, error B 40 times and error C 50 times – sure, I have 100 errors – but I really want to know that I have three errors, and how many times eacbh of them have occurred.

Solution

As you may have guessed, we’ve solved these problems, and replaced the Exception Notifier in most of the apps we manage, with a homegrown solution called hoptoad. Hoptoad is two things:

A smallish Rails plugin, which installs into vendor/plugins, and defines a #rescue_action_in_public method which does nothing but report the error it received (via HTTP POST to a web service), and then return control to the framework so that Rails can render the 500 (or 422, or 404, or whatever) page that’s appropriate for the error.
The web service, which accepts incoming error reports from various applications running the hoptoad plugin, and stores them for later review.

Plugin

By default, the hoptoad notifier plugin requires only one configuration option: the API key for the project the plugin is installed into, so that it can be properly stored. In addition to that, it is possible to specify a set of errors that you’d like the plugin to ignore (it has a default set to ignore and not report on, which you may override). The plugin, and sample configuration, are available from the project configuration area within the application.

Web application

The application is responsible for grouping errors, marking inactive errors, providing rss feeds of groups of errors, managing user access to error reports, handling incoming errors from the plugin via an API, etc.

We expect to the public launch to be sometime next week, so stay tuned.

If my train goes off the track, 
pick it up, pick it up, pick it up back back 
back to the grill again, the grill again 
friends 
how many MC's must get dissed?

-KRS One

Thoughtbot

thoughtbot is a technology consulting firm that provides web application development and design services. We focus on building modern systems, embracing good ideas and delivering elegant solutions.

Interested in learning Rails?

Comments on this post

George Anderson said,

The links in this post to http://www.hoptoadapp.com are dead. But http://hoptoadapp.com is good. Perhaps some quick DNS config is in order.

Looking forward to the release.

Matt Jankowski said,

Thanks George, links updated.

Colin said,

Good stuff guys, we’ve been talking about putting something like this together internally since we first started using Exception Notifier. This going to be something free and open source? Or a commercial service?

The plugin will be open by requirement (because you’ll have to install it in your app). The collection service and webapp will be free to use when we launch, but only available as a hosted service, not as something you can run yourself.

Both of those things ($ cost and hosted-only) are subject to re-evaluation as we continue to expand the application.

Wyatt Greene said,

Wow, that looks really useful!

Tim Gourley said,

I can’t wait to check it out!

Dr J said,

Cool. Looking forward to something like this

Gaius said,

It’d be really wonderful if you made this SOA compatible by supporting logging calls via WS-*. Logging is going to be one of many services that become more important as SOA grows. (Plus, it’d really start to show those corporate people that Ruby is enterprise worthy.)

Gaius – we do have a very basic logging mechanism built in, so that you can send the service more than just error data. This is something we need to use ourselves more before knowing what the correct solution is.

Jul 19

chris finne said,

Sounds great, but what happens if hoptoad goes down, gets overloaded, DNS server doing the lookup for your service gets overloaded, etc. ?

Probably the same thing that can happen to the Exception Notifier if a mail server is having issues: the Rails process hangs. (this was a painful lesson for me)

With Exception Notifier, I prefer to have it use a local sendmail setup which will returns immediately and queues up the mail to be sent (or resent) outside the Rails stack.

Any plans to have some sort of async method to send it?

Darragh Curran said,

Sounds great – there’s another very similar app worth checking out by some peeps I know – http://www.getexceptional.com/ great minds think alike

@chris – good point, that’s another thing that’s burned us while using the EN. I’m not sure if this will be in place for launch, but we do intend to introduce some sort of local queue so that the plugin can handle the error and immediately return control to rails, and then asynchronously deal with making sure the errors get reported to the server.

Magnus Holm said,

How hard would it be to add support to other frameworks?

@magnus – we’re launching this as a rails-only, and the plugin we distribute will be specifically for use in rails applications. With very few exceptions, all of the applications we’re maintaining that are in production are all rails applications, so that’s where our need is.

That being said, there’s nothing we’re doing to block out the future possibility of supporting other frameworks. I suspect that adding support for a non-rails ruby web framework (merb, camping, etc) would be fairly straightforward.

Jul 20

Nicolás Sanguinetti said,

Matt, regarding other frameworks, is there something on your end that disallows exceptions that don’t come from rails?

I don’t see how difficult could it be to produce something for other frameworks/libraries/straight-out ruby scripts that will just post to hoptoad using your API, “stolen” from the rails plugin.

Just dreaming a little (I’m not sure it can be done this way, and we have to wait for you guys to see =P), some rack middleware could be written and voila, all rack-based frameworks can cascade down from this.

Jul 21

Nicolas – no, there’s nothing specifically that makes me think it would be hard. We just haven’t thought about it yet.

Jul 22

zoomn said,

rap-nerd that i am i just have to point out that the quote at the end is not solely from krs but partly from black sheep(lines 1+2) and mc serch (line 3). nice plugin though!

sorry, application not plugin…

Sorry, comments are closed for this article.

Come “ride the toad” on Hoptoad, the app error app.
Thunder Thimble: Brand monitoring for social media.
Widgetfinger: Simple content management for simple websites.
Tee-Bot, funny shirts your friends won't understand!
Umbrella Today: “It’s like totally the simplest weather report ever, Julie.”

GIANT ROBOTS SMASHING INTO OTHER GIANT ROBOTS