Obama tech team talks winning, disrupting and sticking to basics

By Jo Piazza / current.com / @jopiazza

The developers behind President Barack Obama's 2012 presidential campaign spent Saturdays and Sundays pretending the world was ending.

At 9 a.m. on Saturday mornings they would receive an email with their script for the next two days.

In it, everything they had prepared for no longer worked. Their tasks were the complete opposite of what they had planned to release. 

"Those were incredibly stressful days, but those saved us and they gave us a lot of faith in our tools when we went into Election Day. I don’t think any of us went in cocky and arrogant, but we trusted our tools," said Jason Kunesh, the user interface/user experience lead for Obama for America.  

During one of those disaster preparedness weekends, Mother Nature reared her nasty head in the real world as Hurricane Sandy began to migrate up the East Coast. 

"We picked up the infrastructure and migrated it from the East Coast data center to the West Coast data center for Amazon," Scott VanDenPlas, the devops tech lead, explained.

They pulled the requisite all-nighters and kept the campaign online. That was just another day in the office. 

By now, the technological prowess of the Obama presidential campaign has been thoroughly lauded.

(More from Current:Digital literacy in the US: 3 ways the Obama administration is fueling the tech revolution) 

From the start of 2012, the top-secret Project Narwhal, which could link separate depositories of information to build a more complete picture of the potential voter, was hailed as something that would revolutionize how campaigns operated. Narwhal certainly accomplished that, but it was also the campaign’s focus on the basics, its efficiency and its having the foresight to bring in a disparate team of engineers from the startup world, all of which helped to close the gap for the incumbent.

We spent an evening with the developers behind the Obama for America campaign: VanDenPlas, tech lead for devops; Kunesh, UI/UX lead; Ryan Kolak, Narwhal integration tech lead; Chris Gansen, dashboard tech lead; Nick Leeper, finance tech lead. It was an inside look into why the Obama campaign hired a bunch of startup guys to disrupt traditional voter outreach and fundraising in 2012 and how they beat the Romney team — and its incredibly expensive counter-effort to Narwhal, Project Orca — with a vastly smaller budget.

"We had this crazy technology team that existed to disrupt stuff. The goal for an incumbent president was not to f**k up," said Gansen, the dashboard tech lead.  

To that end, VanDenPlas bought a cake at the end of the campaign that said exactly that: "Don't f**k this up."

"People walked by the cake thinking I meant, 'Don’t f**k the cake up.' So no one ate the cake. I think the first person to cut into it was Eric Schmidt [Google executive chairman and Obama campaign adviser]," Vandenplas said.

There were themes to the campaign's technological success — disrupting the system, sticking to basics and preparing for the worst.

The questions compiled here come from crowdsourcing our own development team, the audience and New Relic, the host of the evening's event. Material has been edited for space.

Disrupting the system

Why did the campaign decide it wanted its own in-house development and engineering team instead of turning to vendors to provide their technology?

Jason Kunesh (UI/UX lead): The campaign realized there was enormous value to reaching out via new media, whether it was reaching out to old ladies on Facebook or the ability to fundraise and take donations online. When it came to 2012, they wanted to take it a step forward and have an in-house engineering culture to accomplish all that.

We were there to bring some startup-type qualities to the campaign and find ways to disrupt and engage voters in different ways.

In 2008, people basically managed this stuff by managing databases or Google docs. As a UX person, I saw so many different Google docs from so many states that it nearly broke Google Docs.

What was your mobile strategy versus desktop?

Chris Gansen (dashboard tech lead): When we were building the dashboard, our target device was an iPad because that was what a lot of people inside the organization had and what our end users were comfortable with, so we figured if we could build it well to the iPad it would scale up to the desktop and scale down to the iPhone.

Jason: It worked well for our demographic because if it were simple enough to work on a mobile for the college-aged users, then it would be simple enough that someone who is a little bit older and not as familiar with computers could use without coaching.

Why were you brought in if an incumbent campaign is risk averse?

Scott VanDenPlas (tech lead for devops): It's 2012. Why is technology risky? I don't think technology is any more risky than anything else. I have no idea how people did this without technology.

Ryan Kolak (integration tech lead): I think it would be more risk averse not to try technology. So we were brought in to make things easier.

What did Narwhal do for the campaign?

Ryan: One of the main things for the Narwhal Project was working with our vendors. Campaigns are very compressed. You don't have a candidate until the primary season is over, and then you have three or four months to build the whole enterprise. So people turn to vendors to build pieces. What the Narwhal Project did was interact with all our vendors' APIs to pull information into a central database and then build an API on top of that using an integrated database.

Nick Leeper (finance tech lead): We had all these pictures of how you vote and what you do and how you do it, but what Narwhal really did was show us a complete profile of you.

Ryan: As the election got closer, everything was built into Narwhal. We still had a lot of people in the field offices using legacy tools because change is hard, and we had to choose which problems we solved.

What was the purpose of the dashboard for your team?

Jason: To unite people online and offline to re-elect the president. To get connected with your neighborhood team, you would go through this platform. The campaign term is "ladder of engagement." Back in '08, Sarah Palin derided the president by calling him a community organizer. It's true. He is a community organizer.

Chris: The biggest challenge we faced on a day-to-day basis was figuring out what the application was going to do. When we started, we had a massive product road map. We had enough to keep a team four times our size busy for twice as long.

Scott: That's why we worked 20 hours a day.

Chris: We were trying to whittle it down to how we could make people's lives easier. The real challenge there was, I think, we swung wide and we built this big bloated application, and we had to pare it down and figure out what wasn't working. The process of figuring out what wasn't working was huge.

Scott: It's that famous problem where you have a bunch of people come to the table and they each have a different concept of what they plan to build, but if you don't get people on the same page, you lose focus.

Chris: One more challenge, and this is a big one — we had significant product challenges. We went in with a complete green field, and we had them build the Narwhal platform while they built all the apps on top of it.

Scott: And, P.S., we had no idea how people were going to use it.

Chris: We were trying to figure out how it was going to work, where it was going to work and who was going to use it on a significantly compressed timeline. You ask about use cases and you get 10 different answers depending on who you ask.

Sticking to basics

In a world that isn't familiar with agility, how did you take the scope hammer and break it down?

Jason: We got outside the headquarters. Everyone who was inside headquarters had a viewpoint. They knew what it was like to be in the field previously, but they didn't know what it was like now. The goal in every state was different. The easiest thing to do was interview users, see what they were doing, measure what we were doing. I got a lot of gruff from press for putting a giant road map on the wall. We had a bunch of sticky notes and a color-coded system, and we stuck it on the wall and we had a lot of different releases. But that way we could start to have a dialogue with people and break them into bite-sized releases and surface the features that were actually important.

Chris: Being very open and transparent about the product road map was key. We said, "This is everything everyone in the campaign has asked for, and this is what we are going to deliver."

Scott: In this process, the first release is never going to be good. People are like, "Why does this look like crap? You guys are the worst engineers ever." They're not used to daily and weekly cycles. They're used to a perfect product.

Chris: The only way we got beyond the resistance to iterative development and deployment was to show them that it worked. That put all the pressure on us to not f**k up.

Scott: We're not campaign people. We're startup people. There is definitely the culture clash there. I know far too much about how campaigns work at this point.

Chris: Being able to show our business owners and people calling the shots actual metrics about what was going on, actually being able to quantify all the work we were doing instead of the finger in the air emotional decisions, was huge. We logged and measured everything. I spent a solid 30 percent of my day staring at charts and Google Analytics and making engineering decisions based on that. I can't imagine working any other way.

Scott: Too much of engineering has been gut checks and "I think that this solves that." It was measure, figure out what was going on, do the analysis and fix it.

Jason: We went out in the field and did field tests and usability tests, and we talked to people and made sure those voices percolated up. It wasn't something that usually happened. Campaigns tend to be hierarchical structures. Ultimately, we all wanted the same thing; we just came from radically different cultures. Most of us showed up in T-shirts and shorts, and the interns from Yale showed up in three-piece suits. We were significantly outdressed by the intern staff.

What about bringing in money online? How much money came into the website?

Nick: We had about $670 million online. During the (Democratic National Convention), right after the president left the stage we were doing $2 million an hour.

Scott: And there were, like, 60,000 tweets a second.

Ryan: And the donations were not large. They were people giving $3 to $5. That was a lot of transactions.

What kind of A/B testing were you doing?

Ryan: Since we built our own payment process, we could get our own stats. Our email team, they sent out a lot of email, but they could send out a test email with different messages, and with what we built, they could see in near-real time how things were running and make decisions based on that.

Scott: Our sample size was enormous. We're, like, "Let's A/B test this on 400,000 people," when we're used to sample sizes of like 100.

So there has been a lot of coverage of Orca versus Narwhal. From your experience and knowledge, how was the Romney campaign's engineering team different?

Jason: In 2008, the [Obama] campaign had a system called Houdini, and the idea of Houdini goes back to what happens on Election Day. You want lawyers on site in places where there might be trouble. The second thing you want to have happen is the people you are hoping to vote for your candidate turn out. You've got this giant army of volunteers and you want to make sure they are calling places and knocking on doors. So in '08, there was a system called Houdini, and it was supposed to do all this — and it failed in about 40 minutes on Election Day. In '12, we wanted to build super-scalable apps. Narwhal had nothing to do with any of that but back it. The thing we used on Election Day was Gordon, because Gordon was the guy who killed Houdini. The budget was maybe $10 million. The Romney campaign had a different take, which was to pay a lot of consultants roughly $40 million, and part of what they did was build Orca. With our toolset, we knew most of our volunteers were 18 to 25 or mostly old ladies. That is proven by data. We're not ageist. We knew they would struggle with tech and we knew technology had to scale and be simple for them. For the most part, when we deployed stuff, the concerns from leadership were twofold: Does it make people's lives easier and is it fast? The Orca team started doing things like sending 80-page PDFs to retirees so they could print them out on their home printers, and they tried the whole system on Election Day.

Nick: We had gone through every scenario that could happen to our apps.

Ryan: We used all the early voting days to test these issues. So we were really ready at Election Day, and Election Day for us was four days before the election and we pretty much worked 24/7 that whole time.

Jason: I was the tech liaison in the boiler room in case something went wrong, and outside of pulling some data on voter suppression, it was the most boring four days of my whole life.

What is your legacy?

Scott: There is nothing we did that was revolutionary. What we ended up doing was something a lot of us take for granted. We practiced solid fundamentals. We put together a smart team of honest and passionate people and gave them the right tools and the right environment to produce some great results. The mistake of forgetting these fundamental pieces is so much more tragic than the positive result of doing something revolutionary.

Chris: We didn't come up with a revolutionary new sorting algorithm. We focused on the fundamentals. We built solid apps. If we can build a platform that makes data accessible and reliant and build apps on top of that to get the job done to re-elect the president, just showing that is possible meant we won't have to fight that battle in the next campaign. We won't have as much resistance from the hot-shot startup engineer. We can look back and say, "This worked for 2012 and you should continue on this model and expand on it."

Scott: This is the Orca problem. They rebuilt something that failed in the past. We are not in a world where we can build something four years in advance. We just can't do it. If anyone tells you they can figure it out, they're lying.

Jason: This won't happen again. By 2016, we'll all be wearing Google goggles and won't even have infrastructure.

Ryan: By building a simple API, we knew that if it failed, it failed in a graceful way such that we knew where it failed and it didn't impact the rest of the system. One of the decisions we made was deciding to write the front end in Ruby and the back end in Python. (This) was to force ourselves to go through the APIs, and there were some painful moments in doing that, but it was better in the end.

Scott: We have technology agnosticism. I don't care what I am hosting. It doesn't matter to me. Use something reasonable and we will work on it. I'm not going to tell an engineer to write in a specific language. We had Ruby on Rails, we had Java apps, we had Python with Flask and some Django. You name it, we touched it in some way. When you have an environment that is accepting like that, it works out pretty well.

Preparing for the worst

When you are collecting $2 million an hour, how do you prepare for potential problems?

Nick: One of the things we designed was redundancy. We thought, "How can we make this as compartmentalized as possible?" We built each piece to fail gracefully. We thought, "What happens if the database goes away? Who cares? We will pull all the data into a queue."

Scott: We had three different data centers, two of which were cloud based. All of the forms were in Jekyll, which was S3-based. You have this environment where S3 pushes things around through multiple regions. The application was smart enough to know if the database is going to fail gracefully.

Ryan: When the president announced his support for gay marriage, it was the first time we had a whole lot of load on our servers.

Scott: We found out about that announcement about the time that everyone else did. I think I had maybe a minute. We broke 4K Gbps. Only about half of that was video.

How do you build to have things fail gracefully?

Ryan: I love our vendors. Kind of. They didn't necessarily always have the most reliable APIs. Every single piece of data went through the integration side of things through an SQS queue. Honestly, we had pretty much an unlimited capacity to throw up more boxes. When I told our vendors we were using SQS, they said they considered it and they decided it was too expensive. I think 10,000 messages cost a penny. Throughout the campaign we used about a billion messages so it was about $1,000, and being able to use that and throw things into a queue and have that be something we could build on was the backbone.

Chris: We built in a lot of pressure release valves. We built the ability to turn off features. If one feature was causing a lot of stress on the system, we could turn that off individually. We could put up warning messages.

Scott: The maintenance page was awesome too. It was Joe Biden looking sad.

Chris: The ability to tweak the user experience to respond to system loads saved us tremendous amounts of grief. I don't think we would be able to do what we did if we didn't have the working relationship between the product teams and the engineering teams and the devops team.

Scott: My budget total was only a couple hundred thousand. My budget was $450,000 at the end of it all. I think the comparable side on the Romney campaign was like $14.5 million … but I am sure they did it right.

What was your structure and team size like?

Jason: Our team structure was screwed up because we were a transplanted kidney into a campaign. Overall the campaign struggled to figure out where to place us because we were startup guys, but that wasn't the goal. The goal was to make the trains run on time. We had 40 people on our team and a bunch of volunteers who came in. But it was fluid. We started to organize our own product teams. By organizing around products, you sort of broke through a lot of silos.

Chris: Our CTO hired almost exclusively senior-level people, so we didn't have many junior-level developers straight out of college. Everyone came from backgrounds where they knew how to ship code.

I was working with a team of four or five engineers who were probably more skilled than I was. They wrote better code than I did. We all knew what the goals were and what we had to accomplish. We worked in short iterations, and we released every week. There was no room to slip. We couldn't tolerate people dropping the ball. I think everyone knew exactly what was expected of them, and everyone delivered top-notch work.

Scott: There was a handful of times we needed a feature out and I said it to a team of engineers, and the next day it was done.

What is your advice to aspiring developers?

Ryan: Find something you are passionate about and jump in.

Scott: As engineers we tend to have a focus on solving technical problems, and in a campaign we have real-world problems that we need to build toolsets to fix. I think if I were to tell you one thing to do, it's don't ever stop, just keep going. There is no end.

Chris: Find people doing interesting work and introduce yourself and ask what they are doing, because I think engineers like to talk about what they are doing. I learned a lot on the campaign from working with a lot of people, and every day I learned new things by asking them questions. Surround yourself with smart people and don't let go.

How involved was the president?

Ryan: He ultimately approved all the decisions on what we were building. On a couple of occasions, he came into the office, but there was a lot of delegation.