Thursday, November 21, 2013

The Gravy Plane, costly failures for air-traffic control


On October 1, 2013, the federal government's Web site, healthcare.gov, crashed and burned at launch--probably attracting more attention than any other government software failure. It provides public access to the federal health-care insurance exchange, a key element of Pres. Obama's health-care reform program. The government said the site attracted an average of two million visitors a day during its first four days, while only six managed to enroll in insurance plans the first day. [1] [2]

Nevertheless, that was just a Punch-and-Judy show compared with a horrible reign of errors lasting 32 years to date: repeated failures by the U.S. government to produce well integrated and reliable automation in support of air-traffic control. From a perspective of public service, those efforts have been a bipartisan disaster: the offspring of Reagan, Herbert Bush, Clinton, Walker Bush and Obama administrations. For two major government contractors, however, an air-traffic automation disaster became "The Gravy Plane"--a lush source of high-flying government spending--now around $6 billion and climbing. [3] [4] [5]

Tragegy of mismanagement
So far, this has been a tragedy in two acts. The first attempt by the Federal Aviation Administration (FAA) at improving automation for air-traffic control--Advanced Automation System (AAS)--collapsed after 13 years work and $3.7 billion spent, with little useful delivered. It began as a pie-in-the-sky concept ordered up during the first Reagan administration, in the wake of a 1981 strike by federal air-traffic controllers. At 7 am EDT on August 3 of that year, nearly 13,000 of 17,500 members of the former Professional Air Traffic Controllers Organization walked off the job. Over 11,000 failed to return and were fired two days later. [6] [7]

Former Pres. Reagan, a technology buff who bought into "Star Wars," expected that computers could supplant air-traffic controllers. After a long siege of planning and prototyping--at government expense--IBM's former Federal Systems Division won a large continuing contract over its chief competitor, the former Hughes Aircraft. However, some promises made for AAS proved beyond capabilities of the era's main technologies, and project management was flagrantly bungled by both FAA and Federal Systems. Eventually, during the first Clinton administration, the Federal Systems contract was ended and the 13-year AAS project cashiered. [8] [9]

The second Clinton administration released so-called "NextGen" air-traffic control plans, a long-term program for automation to be carried out in a sequence of stages. NextGen was to start with an effort less ambitious than AAS. In 1998, FAA began planning the En Route Automation Modernization (ERAM) project. During its first year, the first Walker Bush administration channeled the ERAM project to Lockheed Martin--a company with key contacts in that administration--under a sole-source, no-bid contract. [10]

During more than 12 years of implementation through 2013, ERAM has never worked well enough to be placed in full and regular service. [5] Although books have been published on problems of AAS, so far no comparable account has been published on problems of ERAM. Unlike AAS, but like healthcare.gov, ERAM may emerge from a morass of problems and may in time be marshalled into a full-service system. However, FAA had initially advertised full and regular ERAM service by 2007. In 2012 its inspector general found achieving the goal to be unlikely before 2016--making that project 18 years in planning plus implementation. [4] [11] [12]

Top dogs and watchdogs
The U.S. government was able to land men on the Moon in eight years, using technologies of the 1960s that were much less capable than those available 20 and 50 years later. Why would it take more than three decades to improve earth-based air-traffic control automation? Part of the answer looks to be manipulation of contracts, in a program far less visible than a Moon landing, by large companies that had little to lose when failing. In neglecting to plan and manage programs intelligently, monitor work rigorously and prepare alternatives, a rigid and foolish agency allowed those companies to function as monopolies--leaving the government with no other ready sources of services.

The government has several watchdogs expected to uncover mismanagement. They include the General Accountability Office, the Office of Management and Budget, the Congressional Budget Office and, since 1973, the inspectors general assigned to cabinet departments and later to agencies. The armed services have had inspectors general since the Revolutionary War, but the Department of Agriculture was the first civilian agency to utilize one. However, while all the watchdogs exhibit financial skills, few have management skills and none have technical design skills or competence with automation technologies. They live up to their reputations as "bean counters" and often fail to uncover mismanagement, if only because they just don't understand what's going on.

Software and turkeys
Computer software has sometimes been considered a field of engineering. However, unlike development of designs in other engineering fields, with software the final design is the product. There is no physical product to inspect and measure. Moreover, ultimate software designs--"code"--are often thousands of times more complex than other final engineering designs. It took about 30 years for the critical lessons about how to cope with these challenges to emerge within the software professions. So far those lessons have rarely penetrated the heads of government watchdogs. They visit software shops as foreigners, literally not speaking the languages. Thus, back in the shop, bean counters have long been called "turkeys"--not the brightest birds.

The inspector general responsible for FAA in 2005 displayed an exemplary turkey profile. Here the bean counter was foolishly preoccupied with "lines of code." The report revealed that Lockheed Martin was reusing code from legacy systems. It must somehow be interfacing new Ada code with ancient code in Jovial. It must never have performed a first-principles analysis of communication design. Both are major barriers to success, yet the bean counter was busy counting "lines of code." The bean counter did worry over the cost of finding software expertise with Ada although not with Jovial. Both emerged from military environments, with neither attracting much other use. Strong skill with Jovial today would be comparable to fluency in ancient Phoenician, while Ada might be compared with Old High German. [11] [12]

The bean counter's main sally was to recommend a "value-engineering analysis" to see if fewer new computers could be bought than one per long-range ("en route") control center. Even a bean counter might be able to compare at least $2 billion going for software with perhaps $20 million for computers and see that potential savings on computers were not going to be much of a benefit. That was where the turkey profile sharpened, but the bean counter had already given the game away with the title of the report: "FAA's En Route Modernization program is on schedule." Later events showed the project was effectively years behind schedule by that point. The bean counter had either been enlisted or been hoodwinked.

Air-traffic control automation
U.S. air-traffic control is a distributed system, currently with 22 long-range control centers, 164 regional-range centers and a local-range center for each large and medium-size airport. These are known, respectively, as "air route traffic control" or "en route" centers, "terminal radar approach control" or "Tracon" centers, and "airport traffic control towers" or just "towers." FAA oversees all the facilities, and it operates the en route centers, the Tracon centers and some of the towers, known as the "federal control towers." [13] [14] [15]

Starting in the 1960s, air-traffic control automation was developed by FAA personnel to assist the air-traffic controllers. It has undergone upgrades and computer replacements, but there has been no change to its basic structure since the 1960s. Like military backgrounds from which many FAA personnel came, but also like computers typical of the era in which it originated, the structure is hierarchical. A central coordinating computer communicates with "en route" center computers, which communicate with Tracon center computers, which communicate with radars serving their regions. Air traffic controllers coordinate handovers of flights between Tracons and towers by voice. Officially called the National Airspace System, the software is informally known as Host--an emblem of the era in which it originated.

The Host and its keepers
In the beginning was the Host. The first model of distributed computing, emerging from the regimented 1950s, was God-fearing and hierarchical. At the center was a "host computer" that coordinated activities. It took many years for software developers to recognize and overcome liabilities of the model: its susceptibility to overloading communications channels and its vulnerability to failure of host computers. If the Internet were to depend on a hierarchical organization with host computers, it would frequently fail. Instead, it was designed in the late 1980s using a robust model with distributed, independent coordination. FAA. however, continues to operate U.S. air-traffic control on the back of first-generation automation, scrambling to keep a brittle, creaky system working.

Between 1969 and 1977, Host was mounted on IBM model 9020 computers--a specialized version of the System 360, model 65, introduced in 1966. It was programmed by FAA personnel in Jovial--an early, high-level language derived from IAL, later called Algol-58--and in IBM 360 machine code. In 1982, amid fears that equipment would become unmaintainable, FAA began plans to renovate the system. A key problem was software, written partly in a little known, obsolete high-level language and partly in machine code for an obsolete computer. Because of a sense of urgency, that effort became independent of AAS and was treated as a short-term measure to sustain operations. In 1985, IBM was awarded a contract. IBM replaced the original computers with model 3083, rewrote machine code for the new computers and adapted Jovial language support to run on them. [16]

After termination of the AAS project, IBM sold its Federal Systems division to Loral, which later sold it to Lockheed Martin. As an arm of Lockheed Martin, the former Federal Systems division continued to do business with the federal government, maintaining air-traffic control under Host for FAA. Meanwhile, both Clinton administrations tried to sweep cobwebs out of FAA and to develop a sustainable architecture for air-traffic control automation. Yet a 1997 report from the (then) Government Accounting Office still complained, "FAA lacks a system architecture." Yes, indeed. That was more honking from bean counters and turkeys. Few people like those who wrote the legacy Host software, starting in the 1960s, would have recognized the term "architecture" as meaningful in their work. Like the builders of Roman roads, they became skillful at using the resources available to them to do the jobs they were assigned. By 1997, however, FAA had produced an architecture to guide future air-traffic control--NextGen--and was about to plan its first stage: ERAM. [17]

Unfortunately, time ran out. No contract for ERAM was awarded before the second Clinton administration expired. When the opportunity to let a contract for ERAM fell into the lap of the new Walker Bush administration, Lockheed Martin turned up in first post position. Norman Mineta, the new secretary of Transportation, had been senior vice president and managing director at Lockheed Martin. His deputy, Michael Jackson, had been vice president and general manager at Lockheed Martin IMS, Transportation Systems and Services. Lynne Cheney, wife of Vice President Dick Cheney, had been a Lockheed Martin director until shortly before Mr. Cheney took office. The fix was in; the new air-traffic control architecture was mostly out. [10]

Chiseling a contract
Lockheed Martin lacked practical interests in NextGen architecture except as an advertisement. To the contrary, its interests lay in preserving cranky and poorly documented Host software, for which the government had paid the company to develop unique expertise. Other organizations would be unable to compete with Lockheed Martin at projects based on Host, because they would lack the gradually acquired, hands-on expertise. The Walker Bush administration negotiated a sole-source, no-bid ERAM contract with Lockheed Martin in 2001. While it appeared even-handed, with fixed-price deliverables and performance incentives, it had an escape clause for Lockheed Martin: get the government to "accept" ERAM and then subsequent work is billed at cost-plus. [4] [18] [19]

During the Walker Bush administrations, FAA failed to require, and of course Lockheed Martin did not perform, a first-principles analysis of communication design--to find an optimum approach taking best advantage of decades of progress since Host was produced. Instead, Lockheed Martin grafted ERAM onto the legacy Host software, committing the government to indefinite support of a marginally reliable antique, housed inside a new shell. The company pushed the envelope to deliver ERAM early, promising to deploy it to the FAA "en route" control centers starting in 2005. [20]

With Lockheed Martin at work, ERAM began reproducing vulnerabilities of legacy air-traffic control. Although the company accumulated a substantial number of problem reports, nevertheless in October, 2007, FAA signed off on government acceptance. The acceptance was based on bench testing only, at the FAA Technical Center, without using ERAM in a field setting, much less using it for air-traffic control. Qualifying for "early delivery" got the company out of jail financially and justified a large "performance incentive" fee. Through 2011, the ERAM project paid Lockheed Martin over $150 million in incentive fees. [21] [22]

Truth or consequences
With the start of the first Obama administration, Lockheed Martin lost its key allies at FAA, and FAA got a new administrator in J. Randolph "Randy" Babbitt, a former commercial airline pilot and former head of the Air Line Pilots Association. Mr. Babbitt's background prepared him to take on chronic morale problems at the agency but left him unequipped to deal with decades of mismanaged automation technology. He made the mistake of assuming, just because the FAA Technology Center said ERAM was OK, that it must be. In the spring of 2009, he authorized the "en route" center in Salt Lake City to test ERAM with live air traffic--never having exercized the software by using it only to provide a backup. [23]

Once again, a government factotum ignored a well known rule taken to heart by professional software developers for decades: "What you haven't tested doesn't work." It didn't. Luck was with Mr. Babbitt, the crews and the passengers: the first three system crashes in the wee hours of weekend mornings attracted mainly protests from air-traffic controllers who could see, first hand, the dangers that were caused. They, in turn, got the ear of Utah senators, who forced a delay in further testing. It didn't help much; the bugs in ERAM were too deeply embedded to be readily detected or corrected. The next live test got more attention. At 5 am ET on November 19, FAA systems crashed so badly that air-traffic controllers had to send information between centers by Fax and hold up flights that were not already airborne. Delays rippled nationwide for most of the day. [24] [25]

The highly public consequences of the November, 2009, ERAM crash led to a stop-and-go pattern of further tests and repairs persisting through 2013, four years later, and probably for at least a few more years. The most congested regional centers--New York, Washington, Atlanta and Miami--still can't use ERAM, and seven other centers are able to use it only in low-traffic conditions, mostly at night and on weekends. Most deployment for the NextGen program remains on hold until ERAM becomes a full-service system. When and if it does, it will still lack the architectural integrity that was intended for NextGen in the mid-1990s and therefore lack the long-term reliability and extensibilty that should have accompanied a sound program. [5] [26]



[1] Tim Mullaney, Obama adviser says demand overwhelmed healthcare.gov, USA Today, October 6, 2013, at http://www.usatoday.com/story/news/nation/2013/10/05/health-care-website-repairs/2927597/

[2] Susan Cornwell and David Morgan, Documents show enrollment in Obamacare very small in first days, Reuters, October 31, 2013, at http://www.reuters.com/article/2013/11/01/us-usa-healthcare-surge-idUSBRE99U16R20131101

[3] Robert L. Glass, ed., Software Runaways: Monumental Software Disasters, Prentice-Hall, 1998, p. 71

[4] Jeffrey B. Guzzetti, Weaknesses in program and contract management contribute to ERAM delays and put other NextGen initiatives at risk, USDOT Report No. AV-2012-179, September 13, 2012, at http://www.oig.dot.gov/sites/dot/files/ERAM%20Final%20Report%5E9-13-12.pdf

[5] Jeffrey B. Guzzetti, FAA has made progress fielding ERAM, but critical work on complex sites and key capabilities remains, USDOT Report No. AV-2013-119, August 15, 2013, at http://www.oig.dot.gov/sites/dot/files/DOT%20OIG%20ERAM%20Report%20508.pdf

[6] Rebecca Pels, The pressures of PATCO: Strikes and stress in the 1980s, Essays in History 37 (University of Virginia), 1995, at http://www.essaysinhistory.com/articles/2012/121

[7] Willis J. Nordlund, Silent Skies: The Air Traffic Controllers Strike. Praeger, 1998

[8] Mark Lewyn, Flying in place: The FAA's air control fiasco, Business Week, April 25, 1993, at http://www.businessweek.com/stories/1993-04-25/flying-in-place-the-faas-air-control-fiasco

[9] Robert Britcher, The Limits of Software: People, Projects and Perspective, Addison-Wesley, 1999

[10] Matthew L. Wald, FAA to skip bids on air traffic system, New York Times, March 7, 2001, at http://www.nytimes.com/2001/03/06/business/faa-to-skip-bids-on-traffic-system.html

[11] David A. Dobbs, FAA's En Route Modernization program is on schedule, USDOT Report No. AV-2005-066. June 29, 2005, at http://www.oig.dot.gov/sites/dot/files/pdfdocs/av2005066.pdf

[12] Mike Paglione, Metrics-based approach for evaluating air-traffic control automation of the future, Federal Aviation Administration, 2006, at http://acgsc.org/Meetings/Meeting_99/SubcommitteC/8.1subc.ppt

[13] Federal Aviation Administration, Air Route Traffic Control Centers, 2013, at http://www.faa.gov/about/office_org/headquarters_offices/ato/artcc/

[14] Federal Aviation Administration, Terminal Radar Approach Control Facilities, 2013, at http://www.faa.gov/about/office_org/headquarters_offices/ato/tracon/

[15] Federal Aviation Administration, Airport Traffic Control Towers, 2013, at http://www.faa.gov/about/office_org/headquarters_offices/ato/atct/

[16] John Andelin, Review of FAA's 1982 National Airspace System plan, Office of Technology Assessment, 1982, available at http://www.princeton.edu/~ota/disk3/1982/8222/8222.PDF

[17] Randolph C. Hite, Complete and enforced architecture needed for FAA, General Accounting Office, February 3, 1997, at http://www.gpo.gov/fdsys/pkg/GAOREPORTS-AIMD-97-30/html/GAOREPORTS-AIMD-97-30.htm

[18] Calvin L. Scovel, III, Challenges in meeting FAA's long-term goals for the Next Generation air transportation system, April 21, 2010, Testimony to Subcommittee on Aviation, House Committee on Transportation and Infrastructure, at http://www.oig.dot.gov/sites/dot/files/WEB%20FILE_NextGen%20Testimony.pdf

[19] Anthony N. Palladino, Protest of Raytheon Company, FAA docket no. 01-ODRA-00180, June 15, 2001, available at http://www.pubklaw.com/rd/other/01-ODRA-00180.pdf

[20] Unattributed, Lockheed reports good early progress on en route projects, World Aviation (Beijing, China), August 16, 2004, at http://www.huisi02.25u.com/ac/ac0400/ac0403_content.asp?id=618

[21] James K. Reagan, Independent assessment of the ERAM program, MITRE Lincoln Laboratory, October, 2010, available at http://assets.fiercemarkets.com/public/sites/govit/mitreindependentassessment_eram.pdf

[22] John Sheridan, FAA remains quiet on ERAM budget overruns and delays, Aviation International News, December, 2011, at http://www.ainonline.com/aviation-news/aviation-international-news/2011-12-02/faa-remains-quiet-eram-budget-overruns-delays

[23] Sholnn Freeman, FAA asked to do more to fix morale, Washington Post, December 1, 2009, at http://www.washingtonpost.com/wp-dyn/content/article/2009/11/30/AR2009113004066.html

[24] Joan Lowy, Associated Press, Utah lawmakers say air traffic computer not ready, Salt Lake Tribune, June 13, 2009, at http://www.sltrib.com/news/ci_12584509

[25] Matthew L. Wald, Backlog of flight delays after computer problems, New York Times, November 20, 2009, at http://www.nytimes.com/2009/11/20/us/20air.html

[26] John Sheridan, ERAM development is reminiscent of failed AAS program, Aviation International News, October, 2013, at http://www.ainonline.com/aviation-news/aviation-international-news/2013-10-02/eram-development-reminiscent-failed-aas-program

Friday, November 15, 2013

Fixers and spinners, repairing the healthcare.gov project


In early autumn, long before holidays: four fixers fixing, three spinners spinning, two layers laying--one rotten egg! That's how healthcare.gov turned out when it went live October 1, 2013: a Web-site disaster built for the Obama administration's health-care reform program. [1] A story spun by Pres. Obama's staff was that nobody knew disaster was brewing--not at all likely. [2] Someone frequently communicating with the White House did know and likely did say. Crowned heads chose to ignore warnings, and they got what they deserved. [3] [4]

A key figure is Henry Chao, the deputy director and deputy chief information officer in the Office of Information Services, an agency of the Centers for Medicare and Medicaid Services (CMS), a bureau of the Department of Health and Human Services. He appears to have enlarged the usual scope of information services staff. He also stepped beyond the typical role of software architect, his designation in the healthcare.gov project. Mr. Chao doesn't have an engineering degree, a business administration degree, hands-on experience developing commercial software or business experience managing software development. However, working at CMS for nearly 20 years, he is reported to have led the final design and implementation stages for several software systems--the sort of on-the-job training that once was common among software developers. One of his projects had a rocky ride: the initially faulty online information for Medicare Part D. [5] It's not clear what part Mr. Chao played in that project.

Several news reports and Mr. Chao's testimony and appearance at a televised hearing of the House Oversight and Government Reform Committee on November 13, 2013, showed a person concerned about the progress of contractors building healthcare.gov and its back-end data import, access, analysis and export software--monitoring tasks as a diligent project manager would. At the hearing, he sounded like a seasoned software development manager, trying to avoid being a spokesperson for others. He was careful to distinguish his technical responsibilities from the activities of policymakers and from the operations and finance scopes of program directors. However, an issue that repeatedly stymied him in trying to answer questions was that no one in the Obama administration looks to have been clearly delegated the role of program director. [6]

The government customer for the software is another CMS agency, the Center for Consumer Information and Insurance Oversight. Its director since August, 2012, has been Gary Cohen, a lawyer who also serves as a deputy administrator of CMS. The Congressional committee invited to its hearing neither Mr. Cohen nor Mr. Chao's boss Tony Trenkle, the department's chief information officer. The Department of Health and Human Services was represented by Frank Baitman, the deputy assistant secretary for information services, and by Mr. Chao. [6] It's unusual for Congress to hear from workers that deep in the government ranks. Mr. Baitman had little to say and got few questions. Mr. Chao looked to have put on extra pounds--maybe a sign of his occupation's usual drugs of abuse: sugar and caffeine.

Writing for Commonwealth Fund of New York City, Jane Norman, HealthBeat associate editor for Congressional Quarterly, had reported the previous March about an insurance industry conference held a few days earlier in Washington, DC. Mr. Cohen and Mr. Chao were the speakers for the opening session of the conference. [7] Ms. Norman wrote, "Chao was frank about the stress and tension of the compressed time frame involved in setting up the exchanges." She quoted him as saying, "We are under 200 days from open enrollment, and I'm pretty nervous." [8] Another quote from Mr. Chao, widely circulated in business media, was ignored by general-interest news writers at the time: "Let's just make sure it's not a third-world experience."

According to Ms. Norman, at the industry meeting Mr. Cohen said, "I think it's only prudent not to assume everything is going to work perfectly on day one and to make sure that we've got plans in place to address things that may happen...Everyone recognizes that day one will not be perfect." About six weeks later, Mr. Cohen appeared before a subcommittee of the House Committee on Energy and Commerce. To that audience, he said, "We are on schedule, and I am confident that Americans in all states will enjoy the benefits of the Affordable Care Act...Beginning on October 1, 2013, when consumers visit the Web site of their marketplace, they will be able to submit an application...." [9] Different strokes for different folks.

The House Oversight hearing on November 13 produced theatre. Committee chair Darrell Issa (R, CA) played Carlos the Jackal--snide, bullying and juvenile--snickering at his own jokes. Todd Park, U.S. Chief Technology Officer since March, 2012, played Godfather--perhaps a old role for him since founding health IT companies AthenaHealth and Castlight Health starting at age 24. After the White House refused to send him voluntarily as a witness, Rep. Issa had issued a subpoena for Mr. Park to appear. He mentioned his Korean ancestry and deep regard for U.S. government. [6] As can happen with Congressional committees, the hearing produced little new information. Questions from reactionary committee members often turned rhetorical. Some committee members sounded foggy.

Rep. John Tierney (D, MA) developed the only sustained and intelligent dialog with a witness. At about two and three-quarters hours into the hearing, he asked Mr. Chao how a decision to disable the Web site's "anonymous shopper" feature had occurred, shortly before October 1. Chairman Issa tried to cut him off, asking, "Will the gentleman yield?" Rep. Tierney simply said, "No," a rare event in the ossified world of Congress. [6] Mr. Chao explained that getting the "plan compare" feature working, essential to submitting an insurance application, took priority over fixing bugs in "anonymous shopper." Throughout, the witnesses maintained a serious demeanor that mocked diatribes from reactionary committee members. Several times the Godfather mentioned "incredibly hard work" being put in by the "project team." No committee member thought to ask him who the team leader was.

With the wash hung out to dry after October 1, no hearing witness would offer a blanket guarantee that the Administration's goal of making the Web site function by November 30 would be reached, nor was any willing to estimate the cost of the repair work. However, committee members seemed more frustrated that no witness would say what had been conveyed to the White House before October 1 about software problems and failures to achieve schedule goals. In response to a direct question, Mr. Chao said such issues should probably go to Ms. Tavenner. Marilyn Tavenner, the CMS administrator, and Mr. Chao had been witnesses at another hearing held by the same committee June 17, but that hearing never reached such questions. Ms. Tavenner had said then, "I want to assure you that October 1, 2013, the health-insurance marketplace will be open for business." [10]

Missing in action at the November 13 Congressional committee hearing were representatives of the contractors working on healthcare.gov and Jeffrey Zients, who was appointed the new fixer-in-chief by Pres. Obama October 22. Mr. Zients was formerly deputy director for management at the Office of Management and Budget. He organized a 3-day review, then announced that the Web site would be in full service by November 30 and that one of the project's contractors, Quality Software Services, would "oversee repairs" as a "general contractor"--not a term ordinarily used in software development. [11] It is also not clear how a government agency can legally delegate its contract management responsibilities to a "general contractor" when a project is already underway.

Quality Software Services built the data access hub for healthcare.gov, which also serves the online exchanges run by states. This complex software is intended to provide a single, secure point of access to government data needed to verify and process health-care insurance applications, and it forwards data among the federal and state exchanges and the participating insurance companies. At the November, 2013, Congressional committee hearing, Mr. Chao claimed the hub was tested and working, although it had become the object of many complaints about poor performance and garbled information. [12] It is not clear whether Quality Software Services assumed the role of system integrator being performed by CMS. Had the Congressional committee really wanted to find out what was going wrong with the project and how defects would be repaired, it missed an opportunity to assemble some of the most knowledgeable people.

CMS and its contractors failed to conduct a reasonable program of testing before opening healthcare.com to public access. [13] After integration with its data access hub, the complex Web site received only about two weeks of testing--rather than several months needed for even minimal assurance of reliability. Nevertheless, on September 27, 2013, CMS Administrator Marilyn Tavenner approved open release of the largely untested software, with predictably disastrous consequences. [14] She ignored a well known rule taken to heart by professional software developers for decades: "What you haven't tested doesn't work." Government memos made available to the public in redacted form show a working environment near panic in July, 2013--the name of Marilyn Tavenner looking to have been blacked out. [15] In a message dated July 16, Henry Chao asked, "Did you see my other email about first just talking to [Marilyn] to convey just how low the confidence level and then pile on top of that the request for more money when we constantly struggle to get a release done, vacillating on delivery by due dates, and worse of all poor [quality assurance]...." [sic]

Efforts to build the federal health-care insurance exchange and its healthcare.gov Web site seem to have been smaller in scale than legions of workers sometimes imagined. [16] CGI Federal has been the main contractor for the Web site, as distinguished from the data access hub. As of July, 2013, one news writer found only ten software developers from CGI Federal working on the "plan compare" feature. [17] However, CGI Federal was raking in big bucks. Contrary to popular impression, it never "won" a contract to build the Web site, because there has been no such contract. Instead, it has a continuing contract awarded in 2007, during the Walker Bush administration, to provide Health and Human Services with a broad range of software development and maintenance. During 2013, the total contract payments to CGI Federal were growing from under $100 million to nearing $300 million. The Administration exploited the legacy contract with CGI Federal, avoiding competitive bidding on a new contract for healthcare.gov and following a hazardous trail blazed by the former Federal Systems Division of IBM.

During the first Reagan administration, also under a continuing contract, Federal Systems began to develop a pie-in-the-sky concept called Advanced Automation System (AAS), aimed to replace air traffic controllers who had gone on strike with computers. Over 13 years, about $3.7 billion was spent on AAS, but little useful was ever produced. During the first Clinton administration, the Federal Systems contract was terminated, and the AAS project was cashiered. The second Clinton administration then developed the NextGen air-traffic automation program, which was designed to start with a project much less ambitious than AAS, called En Route Automation Modernization (ERAM). During its first year, the Walker Bush administration awarded ERAM to Lockheed Martin--a company with key contacts in that administration--under a sole-source, no-bid contract. [18] During over 12 years of development, through 2013, ERAM has never worked well enough to be placed in full and regular service.

On a more encouraging note, while the November 13 Congressional committee hearing was underway, a session with healthcare.gov found the Web site working and responsive. It proved much brisker than Web sites for the House Oversight Committee, the New York Times or the Washington Post. By that point, the "anonymous shopping" feature had become available, although no committee member seemed to know about it. When asked for information about available health-care insurance plans, for New York and Massachusetts, healthcare.gov exited to Web sites of the exchanges run by those states. For New Hampshire, a federal-partnership state, healthcare.gov displayed 11 plans offered by Anthem, the only participating insurer for that state, along with monthly prices.



[1] Tony Jewell, Descent into madness: an account of one man's visit to healthcare.gov, October 2, 2013, at http://thehealthcareblog.com/blog/2013/10/02/descent-into-madness-an-account-of-one-mans-visit-to-healthcare-gov-october-1st/

[2] Greg Botelho and Holly Yan, Sebelius says Obamacare Web site problems blindsided the President, CNN, October 23, 2013, at http://www.cnn.com/2013/10/23/politics/obamacare-sebelius-interview/

[3] Gabriel Debenedetti and Susan Cornwell, Official who made big health-care Web site decision a frequent White House visitor, Reuters, October 25, 2013, at http://www.reuters.com/article/2013/10/25/us-usa-healthcare-chao-idUSBRE99O01F20131025

[4] Juliet Eilperin and Sandhya Somashekhar, Private consultants warned of risks before healthcare.gov launch, October 1, Washington Post, November 18, 2013, at http://www.washingtonpost.com/politics/private-consultants-warned-of-risks-before-healthcaregovs-oct-1-launch/2013/11/18/9d2db5f4-5096-11e3-9fe0-fd2ca728e67c_story.html

[5] Brett Norman, Health official involved in Obamacare site also had role in Medicare rollout, Politico, October 25, 2013, at http://www.politico.com/story/2013/10/henry-chao-obamacare-site-medicare-rollout-98874.html

[6] Obamacare implementation, House Committee on Oversight and Government Reform, November 13, 2013, at http://oversight.house.gov/hearing/obamacare-implementation-rollout-healthcare-gov/

[7] 2013 Exchange Conference, America's Health Insurance Plans, Ritz Carlton, Washington, DC, March 14, 2013. Schedule at a Glance, at http://www.ahip.org/Conferences/Exchange2013/Schedule/

[8] Jane Norman, HHS working on contingency plans in case exchanges not ready in time, Commonwealth Fund (New York, NY) Newsletter, March 18, 2013, at http://www.commonwealthfund.org/Newsletters/Washington-Health-Policy-in-Review/2013/Mar/March-18-2013/HHS-Working-on-Contingency-Plans.aspx

[9] Statement of Gary Cohen, JD, concerning the Center for Consumer Information and Insurance Oversight and the implementation of the Patient Protection and Affordable Care Act, Subcommittee on Oversight and Investigations, U. S. House Committee on Energy and Commerce, April 24, 2013, at http://docs.house.gov/meetings/IF/IF02/20130424/100743/HHRG-113-IF02-Wstate-CohenG-20130424.pdf

[10] Privacy, security and fraud, House Committee on Oversight and Government Reform, June 17, 2013, at http://oversight.house.gov/hearing/evaluating-privacy-security-and-fraud-concerns-with-obamacares-information-sharing-apparatus/

[11] Caroline Humer and Sharon Begley, White House says 'Obamacare' Web site will be fixed by end of November, Reuters, October 25, 2013, at http://www.reuters.com/article/2013/10/25/us-usa-healthcare-exchanges-idUSBRE99O0SS20131025

[12] Robert Pear, Sharon LaFraniere and Ian Austen, From the start, signs of trouble in federal project, New York Times, October 13, 2013, at http://www.nytimes.com/2013/10/13/us/politics/from-the-start-signs-of-trouble-at-health-portal.html

[13] Robert Pear, Tests of Web site only two weeks before opening, New York Times, October 25, 2013, at http://www.nytimes.com/2013/10/25/us/politics/bipartisan-dismay-over-health-plan-woes-at-house-hearing.html

[14] James Kerr and Henry Chao to Marilyn Tavenner, Re federally facilitated marketplace, undated, available at http://www.scribd.com/doc/180387053/CMS-memo-Approved-and-Signed-by-Marilyn-Tavenner-Accepting-Security-Risk-of-Untested-Obamacare-Website-to-Authorize-Activation-27-Sep-2013 as a nonsearchable scanned image. The document includes approval of the authority to operate healthcare.gov, signed by CMS Administrator Marilyn Tavenner, dated September 27, 2013, with reviews acknowledged by Teresa Fryer, Tony Trenkle and HHS COO Michelle Snyder. Searchable text at http://libertysflame.com/cgi-bin/readart.cgi?ArtNum=35567 also posted pseudonymously.

[15] Nonsearchable images for a selection of redacted e-mail messages sent during July, 2013, concerning the healthcare.gov project, released by the House Committee on Oversight and Government Reform, November, 2013, at http://energycommerce.house.gov/sites/republicans.energycommerce.house.gov/files/20131115-HHSemails-July2013.pdf without source citations or explanations. Abbreviations in some messages documented in [13].

[16] Elise Hu, Internal e-mails reveal warnings healthcare.gov wasn't ready, National Public Radio, November 15, 2013, at http://www.npr.org/blogs/alltechconsidered/2013/11/15/245399200/internal-emails-reveal-warnings-healthcare-gov-wasnt-ready

[17] Sharon Begley, As Obamacare tech woes mounted, contractor payments soared, Reuters, October 17, 2013, at http://www.reuters.com/article/2013/10/17/us-usa-healthcare-technology-insight-idUSBRE99G05Q20131017

[18] Matthew L. Wald, FAA to skip bids on air traffic system, New York Times, March 7, 2001, at http://www.nytimes.com/2001/03/06/business/faa-to-skip-bids-on-traffic-system.html

Tuesday, November 12, 2013

New baby in government's nursery of software disasters


In October, 2013, New York Times reporters Robert Pear, Sharon LaFraniere and Ian Austen first reported gross conflicts and disorganization among contractors developing the health-care Web site, healthcare.com, and their supervisors who were federal government employees. [1] While the Times described problems soon after a crisis became public, its reporters did not explain how the problems developed. Three weeks later, Washington Post reporters Amy Goldstein and Juliet Eilperin traced them to blunders committed by lawyers who were serving as government officials but had no significant operations backgrounds, technical competence or business experience--their authority underwritten directly by Pres. Obama. [2]

Neither the Times nor the Post has yet described how a senior Administration official behind the blunders, Nancy-Ann Min DeParle--former director of the Office of Health Reform at the White House and from 2011 to 2013 Pres. Obama's deputy chief of staff for policy--was allowed to quit the government before the health-care reform program began operating. An ambitious person, regarded as a health-care policy expert, she had served in prominent positions in the federal government and the state government of Tennessee, where she spent much of her youth and graduated from college. Her most obvious blunder, failing to set and then freeze program requirements, allowed a stream of changes ordered when efforts were already gravely behind schedule. [3]

Not much remembered years later, Ms. DeParle was Pres. Obama's substitute for former Sen. Tom Daschle (D, SD), whom but for a dust-up over business activities and misreporting of taxable income he would have appointed health-care "czar" for his administration--that is, both as secretary of Health and Human Services and as chief Presidential advisor on health care. Kathleen Sebelius, former governor of Kansas, got the former post and Ms. DeParle the latter. [4]

Ms. DeParle served as the Obama administration's main contact with Congress during the 2009 health-care reform effort. She had been administrator of the U.S. Health Care Finance Administration, predecessor to the Centers for Medicare and Medicaid Services, during the second Clinton administration. Before that, she worked in the Office for Management and Budget, participating in the first Clinton administration's ill-fated foray into health-care reform. Previously, she had served as Tennessee commissioner for Human Services from 1987 to 1989, an appointee of former Gov. Ned McWherter. She is married to New York Times reporter Jason DeParle. [5]

According to some observers, Ms. DeParle left the Obama administration after Pres. Obama declined to appoint her chief of staff, following Jack Lew's promotion to become Secretary of the Treasury. Her position of deputy chief of staff was filled by Rob Nabors, formerly director of legislative affairs. Denis McDonough, formerly foreign policy advisor to Mr. Obama while he was a U.S. Senator and later the President's deputy national security advisor, took the position as chief of staff for the second Obama administration. [6]

Ms. DeParle's departure as deputy chief of staff for policy left the Administration's health-care reform programs rudderless. In 2011, the Administration had abolished the former Office of Health Reform she had headed, and since then there has been no one else close to the President working specifically on health-care policy. Jeanne Lambrew, a policy analyst, moved from the Department of Health and Human Services to the Domestic Policy Council, headed by director Melody Barnes, as a deputy assistant to the President for health policy. [7]

Following Ms. DeParle's exit, the Administration's health-care efforts look to have become increasingly chaotic. By then the Centers for Medicare and Medicaid Services had been given charge. Henry Chao, deputy chief information officer of the department, was designated chief architect of software, but he seems to have been treated as an errand-boy. His warnings starting early in 2013 that the software had big problems and was badly behind schedule were clearly ignored. When Chief Information Officer Tony Trenkle signed an authorization to operate the healthcare.com Web site on September 3, 2013, Mr. Chao was left out of the loop. [8] In early November, he told Congressional investigators that he had never been told of potential problems found during a audit of security issues. A week after Ms. Sebelius testified to Congress, apologizing for program failures, Mr. Trenkle resigned. [9] [10]

Given bias against business experience and technical competence in the Obama administration, it is unlikely anyone who could have made the federal health-care insurance Web site function would have been given authority to oversee the job. Little known to the public, similar blunders occurred with the Administration's initiative in electronic health records, leaving a patchwork of vague regulations that require many software interfaces but do not detail how they must work. As with the rollout of health-care reform, key advisors for electronic health records left the Administration before efforts they had overseen became operational. The Obama administration has also extended a long series of blunders in automating air-traffic control--an explosive legacy inherited from the Reagan, Herbert Bush, Clinton and Walker Bush administrations. [11] [12] [13]



[1] Robert Pear, Sharon LaFraniere and Ian Austen, From the start, signs of trouble in federal project, New York Times, October 13, 2013, at http://www.nytimes.com/2013/10/13/us/politics/from-the-start-signs-of-trouble-at-health-portal.html

[2] Amy Goldstein and Juliet Eilperin, HealthCare.gov: How political fear was pitted against technical needs, Washington Post, November 2, 2013, at http://www.washingtonpost.com/politics/challenges-have-dogged-obamas-health-plan-since-2010/2013/11/02/453fba42-426b-11e3-a624-41d661b0bb78_story.html

[3] Sharon LaFraniere, Ian Austen and Robert Pear, Specialists see weeks of work ahead on federal health-care exchange, New York Times, October 21, 2013, at http://www.nytimes.com/2013/10/21/us/insurance-site-seen-needing-weeks-to-fix.html

[4] Unattributed, Former Obama adviser Nancy-Ann DeParle joins N.Y. private equity firm, Nashville Tennessean, August 8, 2013, at http://www.tennessean.com/article/20130809/BUSINESS01/308090077/

[5] Michael Collins, Nancy-Ann DeParle, Roane native, leaves White House team, Knoxville News Sentinel, January 26, 2013, at https://www.knoxnews.com/news/2013/jan/26/nancy-ann-deparle-roane-native-leaves-white-team/

[6] Deborah Waroff, Resignation of White House point-person on ME/CFS, International Association for CFS/ME, January 15, 2013, at http://www.meassociation.org.uk/2013/01/resignation-of-white-house-point-person-on-mecfs-from-deborah-waroff-on-co-cure-15-january-2013/

[7] Matthew DoBias, White House reshuffles energy and health reform offices, National Journal, March 1, 2011, at http://www.nationaljournal.com/whitehouse/white-house-to-reshuffle-energy-health-reform-offices-20110301

[8] Robert Pear, Official at health-care site says he didn't know of risks, New York Times, November 12, 2013, at http://www.nytimes.com/2013/11/12/us/official-at-health-site-says-he-didnt-know-of-potential-risk.html

[9] Robert Pear, Sebelius apologizes for health-care Web site malfunctions, New York Times, October 31, 2013, at http://www.nytimes.com/2013/10/31/us/politics/sebelius-apologizes-for-health-sites-malfunctions.html

[10] Stephanie Baum, First casualty of healthcare.gov debacle, CMS CIO resigns, Med City News, November 7, 2013, at http://medcitynews.com/2013/11/first-casualty-healthcare-gov-debacle-cms-cio-resigning/

[11] Jeffrey B. Guzzetti, Weaknesses in program and contract management contribute to ERAM delays and put other NextGen initiatives at risk, USDOT Report No. AV-2012-179, September 13, 2012, at http://www.oig.dot.gov/sites/dot/files/ERAM%20Final%20Report%5E9-13-12.pdf

[12] Jeffrey B. Guzzetti, FAA has made progress fielding ERAM, but critical work on complex sites and key capabilities remains, USDOT Report No. AV-2013-119, August 15, 2013, at http://www.oig.dot.gov/sites/dot/files/DOT%20OIG%20ERAM%20Report%20508.pdf

[13] Sean Reilly, Air traffic controllers making more mistakes, inspector general finds, Federal Times, March 6, 2013, at http://www.federaltimes.com/article/20130306/AGENCY02/303060006/