Fixers and spinners, repairing the healthcare.gov project

In early autumn, long before holidays: four fixers fixing, three spinners spinning, two layers laying--one rotten egg! That's how healthcare.gov turned out when it went live October 1, 2013: a Web-site disaster built for the Obama administration's health-care reform program. [1] A story spun by Pres. Obama's staff was that nobody knew disaster was brewing--not at all likely. [2] Someone frequently communicating with the White House did know and likely did say. Crowned heads chose to ignore warnings, and they got what they deserved. [3] [4]

A key figure is Henry Chao, the deputy director and deputy chief information officer in the Office of Information Services, an agency of the Centers for Medicare and Medicaid Services (CMS), a bureau of the Department of Health and Human Services. He appears to have enlarged the usual scope of information services staff. He also stepped beyond the typical role of software architect, his designation in the healthcare.gov project. Mr. Chao doesn't have an engineering degree, a business administration degree, hands-on experience developing commercial software or business experience managing software development. However, working at CMS for nearly 20 years, he is reported to have led the final design and implementation stages for several software systems--the sort of on-the-job training that once was common among software developers. One of his projects had a rocky ride: the initially faulty online information for Medicare Part D. [5] It's not clear what part Mr. Chao played in that project.

Several news reports and Mr. Chao's testimony and appearance at a televised hearing of the House Oversight and Government Reform Committee on November 13, 2013, showed a person concerned about the progress of contractors building healthcare.gov and its back-end data import, access, analysis and export software--monitoring tasks as a diligent project manager would. At the hearing, he sounded like a seasoned software development manager, trying to avoid being a spokesperson for others. He was careful to distinguish his technical responsibilities from the activities of policymakers and from the operations and finance scopes of program directors. However, an issue that repeatedly stymied him in trying to answer questions was that no one in the Obama administration looks to have been clearly delegated the role of program director. [6]

The government customer for the software is another CMS agency, the Center for Consumer Information and Insurance Oversight. Its director since August, 2012, has been Gary Cohen, a lawyer who also serves as a deputy administrator of CMS. The Congressional committee invited to its hearing neither Mr. Cohen nor Mr. Chao's boss Tony Trenkle, the department's chief information officer. The Department of Health and Human Services was represented by Frank Baitman, the deputy assistant secretary for information services, and by Mr. Chao. [6] It's unusual for Congress to hear from workers that deep in the government ranks. Mr. Baitman had little to say and got few questions. Mr. Chao looked to have put on extra pounds--maybe a sign of his occupation's usual drugs of abuse: sugar and caffeine.

Writing for Commonwealth Fund of New York City, Jane Norman, HealthBeat associate editor for Congressional Quarterly, had reported the previous March about an insurance industry conference held a few days earlier in Washington, DC. Mr. Cohen and Mr. Chao were the speakers for the opening session of the conference. [7] Ms. Norman wrote, "Chao was frank about the stress and tension of the compressed time frame involved in setting up the exchanges." She quoted him as saying, "We are under 200 days from open enrollment, and I'm pretty nervous." [8] Another quote from Mr. Chao, widely circulated in business media, was ignored by general-interest news writers at the time: "Let's just make sure it's not a third-world experience."

According to Ms. Norman, at the industry meeting Mr. Cohen said, "I think it's only prudent not to assume everything is going to work perfectly on day one and to make sure that we've got plans in place to address things that may happen...Everyone recognizes that day one will not be perfect." About six weeks later, Mr. Cohen appeared before a subcommittee of the House Committee on Energy and Commerce. To that audience, he said, "We are on schedule, and I am confident that Americans in all states will enjoy the benefits of the Affordable Care Act...Beginning on October 1, 2013, when consumers visit the Web site of their marketplace, they will be able to submit an application...." [9] Different strokes for different folks.

The House Oversight hearing on November 13 produced theatre. Committee chair Darrell Issa (R, CA) played Carlos the Jackal--snide, bullying and juvenile--snickering at his own jokes. Todd Park, U.S. Chief Technology Officer since March, 2012, played Godfather--perhaps a old role for him since founding health IT companies AthenaHealth and Castlight Health starting at age 24. After the White House refused to send him voluntarily as a witness, Rep. Issa had issued a subpoena for Mr. Park to appear. He mentioned his Korean ancestry and deep regard for U.S. government. [6] As can happen with Congressional committees, the hearing produced little new information. Questions from reactionary committee members often turned rhetorical. Some committee members sounded foggy.

Rep. John Tierney (D, MA) developed the only sustained and intelligent dialog with a witness. At about two and three-quarters hours into the hearing, he asked Mr. Chao how a decision to disable the Web site's "anonymous shopper" feature had occurred, shortly before October 1. Chairman Issa tried to cut him off, asking, "Will the gentleman yield?" Rep. Tierney simply said, "No," a rare event in the ossified world of Congress. [6] Mr. Chao explained that getting the "plan compare" feature working, essential to submitting an insurance application, took priority over fixing bugs in "anonymous shopper." Throughout, the witnesses maintained a serious demeanor that mocked diatribes from reactionary committee members. Several times the Godfather mentioned "incredibly hard work" being put in by the "project team." No committee member thought to ask him who the team leader was.

With the wash hung out to dry after October 1, no hearing witness would offer a blanket guarantee that the Administration's goal of making the Web site function by November 30 would be reached, nor was any willing to estimate the cost of the repair work. However, committee members seemed more frustrated that no witness would say what had been conveyed to the White House before October 1 about software problems and failures to achieve schedule goals. In response to a direct question, Mr. Chao said such issues should probably go to Ms. Tavenner. Marilyn Tavenner, the CMS administrator, and Mr. Chao had been witnesses at another hearing held by the same committee June 17, but that hearing never reached such questions. Ms. Tavenner had said then, "I want to assure you that October 1, 2013, the health-insurance marketplace will be open for business." [10]

Missing in action at the November 13 Congressional committee hearing were representatives of the contractors working on healthcare.gov and Jeffrey Zients, who was appointed the new fixer-in-chief by Pres. Obama October 22. Mr. Zients was formerly deputy director for management at the Office of Management and Budget. He organized a 3-day review, then announced that the Web site would be in full service by November 30 and that one of the project's contractors, Quality Software Services, would "oversee repairs" as a "general contractor"--not a term ordinarily used in software development. [11] It is also not clear how a government agency can legally delegate its contract management responsibilities to a "general contractor" when a project is already underway.

Quality Software Services built the data access hub for healthcare.gov, which also serves the online exchanges run by states. This complex software is intended to provide a single, secure point of access to government data needed to verify and process health-care insurance applications, and it forwards data among the federal and state exchanges and the participating insurance companies. At the November, 2013, Congressional committee hearing, Mr. Chao claimed the hub was tested and working, although it had become the object of many complaints about poor performance and garbled information. [12] It is not clear whether Quality Software Services assumed the role of system integrator being performed by CMS. Had the Congressional committee really wanted to find out what was going wrong with the project and how defects would be repaired, it missed an opportunity to assemble some of the most knowledgeable people.

CMS and its contractors failed to conduct a reasonable program of testing before opening healthcare.com to public access. [13] After integration with its data access hub, the complex Web site received only about two weeks of testing--rather than several months needed for even minimal assurance of reliability. Nevertheless, on September 27, 2013, CMS Administrator Marilyn Tavenner approved open release of the largely untested software, with predictably disastrous consequences. [14] She ignored a well known rule taken to heart by professional software developers for decades: "What you haven't tested doesn't work." Government memos made available to the public in redacted form show a working environment near panic in July, 2013--the name of Marilyn Tavenner looking to have been blacked out. [15] In a message dated July 16, Henry Chao asked, "Did you see my other email about first just talking to [Marilyn] to convey just how low the confidence level and then pile on top of that the request for more money when we constantly struggle to get a release done, vacillating on delivery by due dates, and worse of all poor [quality assurance]...." [sic]

Efforts to build the federal health-care insurance exchange and its healthcare.gov Web site seem to have been smaller in scale than legions of workers sometimes imagined. [16] CGI Federal has been the main contractor for the Web site, as distinguished from the data access hub. As of July, 2013, one news writer found only ten software developers from CGI Federal working on the "plan compare" feature. [17] However, CGI Federal was raking in big bucks. Contrary to popular impression, it never "won" a contract to build the Web site, because there has been no such contract. Instead, it has a continuing contract awarded in 2007, during the Walker Bush administration, to provide Health and Human Services with a broad range of software development and maintenance. During 2013, the total contract payments to CGI Federal were growing from under $100 million to nearing $300 million. The Administration exploited the legacy contract with CGI Federal, avoiding competitive bidding on a new contract for healthcare.gov and following a hazardous trail blazed by the former Federal Systems Division of IBM.

During the first Reagan administration, also under a continuing contract, Federal Systems began to develop a pie-in-the-sky concept called Advanced Automation System (AAS), aimed to replace air traffic controllers who had gone on strike with computers. Over 13 years, about $3.7 billion was spent on AAS, but little useful was ever produced. During the first Clinton administration, the Federal Systems contract was terminated, and the AAS project was cashiered. The second Clinton administration then developed the NextGen air-traffic automation program, which was designed to start with a project much less ambitious than AAS, called En Route Automation Modernization (ERAM). During its first year, the Walker Bush administration awarded ERAM to Lockheed Martin--a company with key contacts in that administration--under a sole-source, no-bid contract. [18] During over 12 years of development, through 2013, ERAM has never worked well enough to be placed in full and regular service.

On a more encouraging note, while the November 13 Congressional committee hearing was underway, a session with healthcare.gov found the Web site working and responsive. It proved much brisker than Web sites for the House Oversight Committee, the New York Times or the Washington Post. By that point, the "anonymous shopping" feature had become available, although no committee member seemed to know about it. When asked for information about available health-care insurance plans, for New York and Massachusetts, healthcare.gov exited to Web sites of the exchanges run by those states. For New Hampshire, a federal-partnership state, healthcare.gov displayed 11 plans offered by Anthem, the only participating insurer for that state, along with monthly prices.

---

[1] Tony Jewell, Descent into madness: an account of one man's visit to healthcare.gov, October 2, 2013, at http://thehealthcareblog.com/blog/2013/10/02/descent-into-madness-an-account-of-one-mans-visit-to-healthcare-gov-october-1st/

[2] Greg Botelho and Holly Yan, Sebelius says Obamacare Web site problems blindsided the President, CNN, October 23, 2013, at http://www.cnn.com/2013/10/23/politics/obamacare-sebelius-interview/

[3] Gabriel Debenedetti and Susan Cornwell, Official who made big health-care Web site decision a frequent White House visitor, Reuters, October 25, 2013, at http://www.reuters.com/article/2013/10/25/us-usa-healthcare-chao-idUSBRE99O01F20131025

[4] Juliet Eilperin and Sandhya Somashekhar, Private consultants warned of risks before healthcare.gov launch, October 1, Washington Post, November 18, 2013, at http://www.washingtonpost.com/politics/private-consultants-warned-of-risks-before-healthcaregovs-oct-1-launch/2013/11/18/9d2db5f4-5096-11e3-9fe0-fd2ca728e67c_story.html

[5] Brett Norman, Health official involved in Obamacare site also had role in Medicare rollout, Politico, October 25, 2013, at http://www.politico.com/story/2013/10/henry-chao-obamacare-site-medicare-rollout-98874.html

[6] Obamacare implementation, House Committee on Oversight and Government Reform, November 13, 2013, at http://oversight.house.gov/hearing/obamacare-implementation-rollout-healthcare-gov/

[7] 2013 Exchange Conference, America's Health Insurance Plans, Ritz Carlton, Washington, DC, March 14, 2013. Schedule at a Glance, at http://www.ahip.org/Conferences/Exchange2013/Schedule/

[8] Jane Norman, HHS working on contingency plans in case exchanges not ready in time, Commonwealth Fund (New York, NY) Newsletter, March 18, 2013, at http://www.commonwealthfund.org/Newsletters/Washington-Health-Policy-in-Review/2013/Mar/March-18-2013/HHS-Working-on-Contingency-Plans.aspx

[9] Statement of Gary Cohen, JD, concerning the Center for Consumer Information and Insurance Oversight and the implementation of the Patient Protection and Affordable Care Act, Subcommittee on Oversight and Investigations, U. S. House Committee on Energy and Commerce, April 24, 2013, at http://docs.house.gov/meetings/IF/IF02/20130424/100743/HHRG-113-IF02-Wstate-CohenG-20130424.pdf

[10] Privacy, security and fraud, House Committee on Oversight and Government Reform, June 17, 2013, at http://oversight.house.gov/hearing/evaluating-privacy-security-and-fraud-concerns-with-obamacares-information-sharing-apparatus/

[11] Caroline Humer and Sharon Begley, White House says 'Obamacare' Web site will be fixed by end of November, Reuters, October 25, 2013, at http://www.reuters.com/article/2013/10/25/us-usa-healthcare-exchanges-idUSBRE99O0SS20131025

[12] Robert Pear, Sharon LaFraniere and Ian Austen, From the start, signs of trouble in federal project, New York Times, October 13, 2013, at http://www.nytimes.com/2013/10/13/us/politics/from-the-start-signs-of-trouble-at-health-portal.html

[13] Robert Pear, Tests of Web site only two weeks before opening, New York Times, October 25, 2013, at http://www.nytimes.com/2013/10/25/us/politics/bipartisan-dismay-over-health-plan-woes-at-house-hearing.html

[14] James Kerr and Henry Chao to Marilyn Tavenner, Re federally facilitated marketplace, undated, available at http://www.scribd.com/doc/180387053/CMS-memo-Approved-and-Signed-by-Marilyn-Tavenner-Accepting-Security-Risk-of-Untested-Obamacare-Website-to-Authorize-Activation-27-Sep-2013 as a nonsearchable scanned image. The document includes approval of the authority to operate healthcare.gov, signed by CMS Administrator Marilyn Tavenner, dated September 27, 2013, with reviews acknowledged by Teresa Fryer, Tony Trenkle and HHS COO Michelle Snyder. Searchable text at http://libertysflame.com/cgi-bin/readart.cgi?ArtNum=35567 also posted pseudonymously.

[15] Nonsearchable images for a selection of redacted e-mail messages sent during July, 2013, concerning the healthcare.gov project, released by the House Committee on Oversight and Government Reform, November, 2013, at http://energycommerce.house.gov/sites/republicans.energycommerce.house.gov/files/20131115-HHSemails-July2013.pdf without source citations or explanations. Abbreviations in some messages documented in [13].

[16] Elise Hu, Internal e-mails reveal warnings healthcare.gov wasn't ready, National Public Radio, November 15, 2013, at http://www.npr.org/blogs/alltechconsidered/2013/11/15/245399200/internal-emails-reveal-warnings-healthcare-gov-wasnt-ready

[17] Sharon Begley, As Obamacare tech woes mounted, contractor payments soared, Reuters, October 17, 2013, at http://www.reuters.com/article/2013/10/17/us-usa-healthcare-technology-insight-idUSBRE99G05Q20131017

[18] Matthew L. Wald, FAA to skip bids on air traffic system, New York Times, March 7, 2001, at http://www.nytimes.com/2001/03/06/business/faa-to-skip-bids-on-traffic-system.html