Postmark Letter Parsing

Overview: This post focuses on our experiences building on Postmark’s Inbound feature.

Postmark is Our Trusted Mail Carrier

Here at Sprout we use Postmark to deliver the emails that come out of our application. An example is the welcome email we send after someone signs up. Rather than build and maintain our own infrastructure to deliver emails, we gladly delegate that to them!

Most engineers think initially, “why wouldn’t you just run your own Postfix?” The answer is that deliverability is hard. Getting a scalable email infrastructure up and running isn’t too daunting. That’s especially true for us as one of our Dev Ops engineers built the Postfix infrastructure that underlies Rackspace Cloud Sites. However, ensuring deliverability of millions of emails is scary as hell, on our own. Thankfully, Postmark knows the tricks and they give really clear reporting to prove it. Our results have been outstanding, hovering at around 99.7% of our emails reaching their destination. That’s even more than 37signals gets using their home-grown infrastructure. Besides, the “cloud” is hot for a reason: specialization of labor. We’re small and every minute we spend not working on our core competency, our product, is not time well spent.

Now They Deliver to Our House Too

So we love Postmark and were stoked when they announced support for inbound emails. The timing couldn’t have been better. We had just started spec’ing out an enhancement to our Tasks feature (now released), which would bring email notifications. The gist is that users can task social media content for a teammate to handle; for example, responding to a customer question. Team members may also communicate with each other, in Sprout, on a thread related to the task at hand. Given the complexities of receiving emails, we expected to start with just a one-way, no reply email notification when some task activity happened. That would allow users to get quick visibility, but would leave them helpless until they got back to a computer. Postmark’s feature made it so damned easy, building the reply feature now was a no brainer.

Task Email Notifications

Here is the gist of how the inbound feature works from a high level, if you are not familiar. A user receives an email from Sprout (via Postmark). The user then replies to that email, which is sent back to Postmark using a Postmark-generated From address. That From has a unique hash that lets Postmark know the response is intended for you. Postmark then converts that reply into JSON which they POST to a public HTTP endpoint that you specified ahead of time. Your endpoint then receives the HTTP POST with the Postmark-generated JSON representation of the email the user sent. In our case, we use the inbound JSON to create comments on the given task within Sprout. You can get more details on how this process works at Postmark’s site.

The One Gotcha

Our experience with Postmark’s inbound feature was relatively painless besides one medium-sized caveat: Postmark does not parse out custom reply header text. Here’s what we mean. Open your email client and hit ‘reply’ on a message. Above the original message text you will notice a string that is pre-pended. Here is an example of what gmail prepends:

On Fri, Feb 17, 2012 at 1:49 PM, Sprout Social  wrote:

We love user experience at Sprout, and so stripping this line and all of the text that will appear below it, is ideal. Remember, the text above that line is the content we want to capture, not what’s below. Who wants to see a messy email thread at the bottom of their one line comment? From a programming perspective, this seems like no big deal, but when you delve into the variations across email clients, you find there are many cases to handle: Gmail, Hotmail, Yahoo! Mail, several Outlook versions, Mac Mail, Thunderbird, etc. Online stats reveal just how many clients are popular and some are really surprising for 2012. As one source, the fantastic Litmus published their take on the breakdown of email client usage.

While they don’t parse it out for you, Postmark at least offers some guidance, by passing the X-Mailer header in their JSON representation. That could direct what regex to use, but it’s not comprehensively provided. For example, although X-Mailer is very prevalent, Gmail doesn’t send it. Jerks. For us, what it all boiled down to is a class that searches the available headers in the inbound payload and then uses regular expressions to parse out the particular version of the line we want to strip. Here is an example of a regular expression that matches the above prepended reply text:

gmail_regex = ".*\"?Sprout Social\"?\s+\s*wrote:.*"

This isn’t an ideal solution as we now have to maintain which email client sends what headers and how their pre-pended reply string appears. Presumably, this can change over time too. It would be great if Postmark handled this mess once and for all, else each developer will have to reinvent the wheel.

In closing, we’re big fans of the feature, and indirectly, our users will be too.

— Rob (@rmadd3n) & Aaron (@aaronrankin)