Aug 13, 2015 by Matt Terwilliger

Revamping MailChimp’s List Import Process

One of the first things a new MailChimp user does to get started is import a subscriber list. In fact, they’re asked during the signup process if they have a list they’d like to import to make it that much easier for them to get started sending campaigns. In the past month alone we've processed over 200,000 imports.

Unfortunately, the list import process that was in place when I began my internship was, in the words of our VP of Product, “crusty and outdated.” So my fellow intern Stephen and I set out to redesign and refactor it. We had two goals in mind: create a better first impression for our new users, and empower existing customers to use our app more efficiently.

Designing the imports experience

The code for the list import feature was among the oldest in the app. Since it was last touched, the rest of the app evolved so much that list import looked ancient by comparison. Stephen was responsible for both a functional and visual revamp of the process, which meant reading up on our pattern library, learning how to use our MVC framework, and talking to our front-end engineers and designers about the way things are done at MailChimp.

It all boils down to one concept: we want to improve the workflow of our users. Whether it’s improving the app's performance or streamlining the user experience, we’re always on the lookout for positive changes we could make—changes that will save our users time and frustration. In the first month, Stephen drafted ideas and helped one of our product designers, Tyrick, create InVision comps using assets from our Photoshop pattern library.

list-import-source

An early draft of the new list imports wizard

We scrapped a lot of early ideas to the parts bin, but that's fine, since iteration is an important and necessary part of our creative process. In fact, we considered a number of solutions before deciding on the wizard you see in the app today. Functionally, the wizard format helps a user focus on the step at hand. Imports now fit right in with other major MailChimp components like campaign creation.

86754777
The new and improved list imports wizard

Refactoring the old code

From a fundamental level, the process seems simple: take my data, put it into MailChimp. But we also run user lists through a number of different systems to make all of those powerful list tools work properly. The whole thing is much more complicated than a cursory glance reveals. I'm happy to say that imports have come a long way, though.

Firstly, the old code was substantially refactored. As an application grows, so do its dependencies. While good design minimizes the effort needed to extend software, it's impossible to predict the future. There comes a point where it's good to sit down and evaluate if you've outgrown the existing structure. With all of the list imports code approaching 6 years, we hit that point. MailChimp's come a long way since 2009, and list imports were overdue for an overhaul.

The refactor also exposed inefficiencies in the import process. With an import, because you're executing a set of actions for a group of email addresses, things like superfluous database queries become quite time consuming. Using the Percona Toolkit, we were able to quantify this. As it turns out, 3 expensive, unnecessary queries represented 39% of the total query time. Killing those off was a huge performance improvement.

After that, the bottleneck was creating the various email objects to store in our databases. So we went a step further and tweaked imports to run concurrently. Now, after a MailChimp user uploads their list, we actually split it up into smaller lists, processing those simultaneously. With these 2 optimizations, list imports are running ~80% faster as a whole. Nice!

By refactoring list imports from top to bottom, we were able to identify weaknesses, improve upon them, and create a solid foundation that MailChimp will build upon for years to come. And hey, if we get to make our users feel a little bit like wizards in the process, that's great too.

Discussion

  • Barry Ryan

    08.13.2015 - reply

    im on forever free

    I tried the new import this morning with a CSV that worked ok on July 30th

    It failed to match the field names automatically, only processed 10 of 13 columns and 70% of the records.

    Happy to provide more details but there is no support service for forever free? Even for bug reports?

    • Matt Terwilliger

      08.13.2015 - reply

      Hey Barry, sorry for the trouble! Someone from our support team will be in touch.

  • Robert

    08.14.2015 - reply

    I have had issues with importing and field matching. After a day with chat support, the conclusion you came to is that my problem is local. Clear your cache, use a different browser, etc. No matter what I do, how carefully I match fields, several fields do not match in the review details part of the upload and the final data after upload. I am working with 4,000 or so records. Going at it again, day 2. BTW I am not bashing support here… they were on it and helpful.

    • Ted Wood @ The Digital Orchard

      08.17.2015 - reply

      Did they offer to receive your raw import list and take a look at it first-hand? I think that would shortcut the troubleshooting process dramatically.

Comment