September 18, 2014
I was debugging an issue today and I ran into a dead end. Typically, when an error occurs in our system it is logged in sentry. This one was not. I had a screen shot from a customer, but it did not include any useful information either. I was finally able to find some relevant information in the web server logs that proved that the customer receiving the error was getting HTTP 400s. This was interesting, but also not helpful. There are a number of different reasons that a 400 could be returned.
I was trying to avoid diving right into the code, which is typically my goto move (and we all know that gotos are bad, right?). I knew that the system was working for other customers, so there had to have been something unique about this customer causing this specific error. After exhausting nearly all possibilities I gave in and dove into the code, hoping that I would not end up going down a (wrong) rabbit hole. It turned out one of the very first pieces of code that I saw was checking for a specific exception and returning a 400. Unfortunately, because that exception was being "gracefully" handled and not bubbling all the way up sentry was not capturing any information about the error.
Since it is relatively simple to configure raven as a sentry logger I got that all set up so that rather than just returning the error to the client I also logged it in sentry. I guess the moral of the story is to make sure that when you are writing error handling code make sure that you are not only communicating useful information and nice error messages to the customer but also to yourself.
Tell me and I forget. Teach me and I remember. Involve me and I learn. - Benjamin Franklin