Pages

Friday, October 25, 2013

Obamacare Roll-Out: Ezra Klein Interviews IT Whiz, Fred Trotter


Ezra Klein: Are you surprised by the rollout of HealthCare.gov?
Fred Trotter: I've been in health IT for years. I have this data set where I look at the health-care system as a whole and model it out. I have experience as a government contractor working on VISTA for the VA. So it doesn't surprise me at all. What you need to run a massive consumer web site is the latest in horizontal scaling and that's hardly approved software for the federal government. The federal government has very conservative mechanisms for purchasing off-the-shelf software and creating new software. That puts a lot of constraints on them.

EK: Explain what horizontal scaling is.
FT: When you get a certain amount of traffic going to any site on the internet a single computer can't handle it. In order to handle tremendous amounts of traffic you have to have more than one computer sharing a task. At modern sites like Amazon and Ebay and Google, the main innovation they've pioneered is using lots of computers at the same time to answer one query to the web site.

But it's a different problem for different tasks. If the federal government wanted to sell billions of books online that's fairly understood. They could just look at Amazon. But they're trying to do something entirely new. And that means that what they're talking about is the invention of something new. The way Congress looks at software is that there are these sites that do these amazing things and we should do that, too. They don't realize that a tremendous amount of invention has gone on at these scaled web sites to handle these processes.

EK: I've been reading the new biography of Jeff Bezos, as he is now my master and overlord, and an ongoing theme is that Amazon's tech had to keep being reinvented as the site grew. And that process, it's clear, was really hard, and getting it right was really important to why Amazon and not some other e-commerce site won. How much more difficult of a problem did the federal government give itself when they tried to unveil a single piece of tech that could handle huge traffic on day one?
FT: They screwed themselves twice. The first thing they did that was very foolish was to go at scale. Usually when the government understands the problem of that they do things in phases. They didn't draft everyone for Vietnam all at once. That's the model they should've used. They should've said people born in January can now get health insurance. Then it should've expanded to everyone born in the first quarter. And so on. But they presumed scale was easy. That was the first mistake. The second was assuming invention was easy. And scaling something that hasn't been invented yet -- that's technological suicide.

EK: Tell me a bit about your experience as a government contractor.
FT: I worked with the VA innovations lab. My company partnered with a local VA hospital to build this new amazing thing. But in the second contract to deploy the thing we built the VA changed the definition of the kind of company that could go for the contract -- potentially excluding us. So they did this thing to encourage innovation but as soon as the innovation was done they went back to this very traditional process.
I realized I could figure out how to develop these very complex, very new software programs or I could figure out how to contract with the government. And so I chose to do the thing that was innovative. I have thousands of hospitals that want to use my software but if I become a government contractor and they don't want to use what I'm using I'm out of a job.

The farce of today's testimony is Congress is basically saying that we created this ridiculous structure in how the government relates to technology and you contractors developed the expertise to jump through our hoops -- why didn't you also develop the expertise to be a great technology company?

EK: How would you fix the way government relates to technology?
FT: One of the jokes we have in health IT that doctors have no idea what they want and we've been giving it to them for years. And I think that's a fair assessment of technology procurement. The government has no idea what it wants and contractors have been giving it to them for years. There needs to be a much more balanced approach to saying that when the federal government understands that it doesn't understand a problem it needs to act very differently. When you're doing a procurement and you know you don't actually know how to solve the problem you need to take a much more organic, evolutionary, and open-source approach. Hire four vendors and require all of them to release the code as open source so you can see who is solving the problem how and then compare approaches.

NHS has just gotten tremendously burned by a rollout of electronic health record systems. It's probably the biggest catastrophe before HealthCare.gov. They also had a big bang, all-at-once, one-big-vendor strategy. That did not work. And now they're going with an innovative approach and creating an open-source ecosystem and allowing multiple vendors to go in so they can not just solve the problem but learn how to solve the problem. I think the U.S. government will have to learn the same lesson. When you don't know what you're building hiring a black-box vendor doesn't work. That's just rolling the dice.

EK: One thing that happens in these contracts though is that Congress and agencies wants to be able to specify what they're getting. That's in tension with this iterative, evolutionary, organic approach to development that can drive towards many different possible outcomes. So how do those balance?
FT: Right now they should be in a position to say they hired five vendors to work on the web site five ways. That sounds like it should be more expensive but in reality you reduce the size of every one. You intentionally create a resource-constrained environment. Being risk tolerant when you look at an ecosystem means being okay with some of the contractors failing. What you're saying is this contractor must succeed in order for the government to succeed. And by letting contractors fail you let them become more risk tolerant because they know they're in a competition they may not win and so they need the best solution, not just a solution, that makes them much more technologically aggressive about getting things done.

EK: One of the problems on the other side of this is that these processes are run by the government and the government's IT departments just aren't very good and the people in them aren't very up on modern software development principles.
FT: There's a problem in government IT departments in general where you get somebody who got a job in their 20s and didn't really have any reason to improve their skill set or change their approach over 20 years but now they're in charge of a department. People think if you are a geek and a technologist and that's the way it is. But if I knew what I knew four years ago today and that's all I knew today I'd be out of a job. Technology moves so fast that you need to either be job hopping or learning the way you do when you're job hopping. Otherwise you just get left behind. But in government there's no good incentive to keep up. And that's a very difficult problem.



No comments:

Post a Comment