Software
Failures
The Most Highly Publicized Software Failure In History
by Ted Smillie
Who Killed the Virtual Case File?
One of the top 10 articles at IEEE Spectrum is Who Killed the Virtual Case File? [1], an engrossing 11 page read about the FBI's Virtual Case File (VCF), which the article describes as "the most highly publicized software failure in history." Unluckily for the FBI, its antiquated IT systems were in the spotlight following the September 11, 2001, terrorist attacks, when counterterrorism became the top priority. This resulted in a series of reviews by the Office of the Inspector General (OIG), U.S. Department of Justice and testimony to the US Senate Committee on the Judiciary which was highly embarrassing to the FBI.
The VCF was in the third phase of the Trilogy Information Technology Modernization Project, which was to upgrade the FBI's IT infrastructure and replace its outmoded and largely paper-based case management system with a modern online system. Trilogy comprised: 1) the Information Presentation Component upgrade, 2) the Transportation Network Component, a communication network upgrade and.3) the User Applications Component which would replace the FBI's most important investigative applications, including the Automated Case Support (ACS) system, the FBI's current case management system.
The Trilogy infrastructure upgrades were completed 22 months late and at a cost of US$337 million, which was close to US$100 million over budget. The US$170 million VCF project was cancelled in March 2005 and is being replaced by a new information technology project called Sentinel.
The VCF was always going to be a disaster but without 9/11 there would probably never have been such a detailed and public analysis. Using interviews with key players and quotes from official reports, Who Killed the Virtual Case File? gives a step by step account of how the project went pear-shaped even though it appeared to be using "tried and true" software development and project management approaches. These included hiring a seasoned project manager and using Joint Application Development (JAD) and a spiral/iterative methodology.
What Went Wrong?
So what went wrong? In its own words, the article "paints a picture of an enterprise IT project that fell into the most basic traps of software development, from poor planning to bad communication." This opinion is reinforced by various warnings and reports throughout the project. As usual, there were wiser heads inside and outside the project whose warnings were ignored. One of the best examples is Matthew Patton.
Like Cassandra before the fall of Troy, the VCF had Matthew Patton, a security expert who was taken on by the VCF Contractor, SAIC in 2002 and left after a few months. From the start, Matthew had some serious reservations about the project, which he raised with his supervisor but to no effect. In frustration, he posted a cri de coeur (blew the whistle) on a web discussion board in October 2002. His comments included: "As a 2-bit journeyman I can't seem to get anyone to pay the slightest attention nor do they apparently (want to) understand just how flawed the whole design is from the get go."
Matthew also has a side-swipe at the cost of the project:" A year+ from now the FBI will have fielded a MAJOR national-security/law-enforcement impacting system at an incredibly high price tag (I've personally done systems of roughly comparable complexity with a staff of eight, not 200 persons) with but a figleaf for security (and an entertaining disaster recovery plan to boot)." Matthew concludes, "If 'trained experts' are not allowed to pull the emergency brake and force a reality check, what chance is there EVER of changing the appalling security in the gov't IT landscape regardless of how many millions get thrown at the problem?"
The immediate result of Matthew's action was that he was taken off the project and left SAIC shortly thereafter. Matthew's posting is an interesting read in its own right and is reprinted below.
NRC Report
One of the key problems was that the FBI had no enterprise architecture. As the article notes: "This cavalier approach to software development would prove fatal to the VCF." Again, there were warnings. Special Agent Larry Depew, who took on a project management role, thought that the rush to implement the VCF meant "shooting from the hip."
In September 2002, the FBI asked the National Research Council (NRC) for help in reviewing Trilogy. The NRC convened various experts to meet with the FBI for briefings and to respond to the FBI as individuals on various aspects of Trilogy. A later NRC report (2004) notes plaintively: "In hindsight, many of these individually provided comments presaged the more formal findings and conclusions presented in this report."
A September 2003 GAO report titled FBI Needs an Enterprise Architecture to Guide Its Modernization Activities [2] notes: "Given the state of the FBI's enterprise architecture management efforts, the bureau is at Stage 1 of GAO's enterprise architecture management maturity framework.. Organizations at Stage 1 are characterized by architecture efforts that are ad hoc and unstructured, lack institutional leadership and direction, and do not provide the management foundation necessary for successful architecture development and use as a tool for informed IT investment decision making. "
For the 2004 report, the Computer Science and Telecommunications Board (CSTB) of the NRC selected a committee composed largely but not exclusively of the experts convened for the September 2002 meeting. (The FBI privately referred to this committee as the "graybeards.") The FBI had asked for a review that could be done quickly and relatively inexpensively. Briefings to the committee on October 27-28, 2003, and December 15-16, 2003 constituted the factual base for the 2004 report, IT-Related Issues for the FBI Requiring Immediate Action [3].
THE NRC report raised a number of significant issues requiring concerted FBI action in four major clusters: 1) Enterprise architecture; 2) System design; 3) Program and contract management; and 4) Skills, resources, and external factors . The report notes: "The issues in each of these clusters are serious in and of themselves. Taken in aggregate, the detrimental impact of inattention to these issues on the FBI's IT modernization efforts is enormous despite the progress that has been made. "
The full NRC report is worth reading from an enterprise architecture perspective as it gives a lucid explanation of why an enterprise architecture is necessary, explains the structure of an enterprise architecture and provides some case studies and references. . To quote a brief extract:
"An enterprise architecture characterizes the enterprise's missions, tasks, and operational processes, and relates these tasks, processes, and operational objectives to IT strategy, investment, and design. It provides substantial detail on the structure and standards used to implement the IT system. The enterprise architecture is the framework that describes the way in which an organization such as the FBI conducts its mission(s), how it organizes and uses technology to accomplish its goals and execute key operational processes, and how the IT system is structured and designed in detail to achieve these objectives. In general, it should also include documentation that explains the rationale behind important decisions and why certain alternatives were chosen and others rejected."
OIG July 2005 Testimony
The OIG Inspector General's July 2005 testimony to the Committee on the Judiciary [4] covers various FBI activities, including the VCF. The Inspector General noted some general concerns as well as specific factors which contributed to the VCF's demise. In summary, his general concerns were:
1. The first is the urgent need to upgrade the FBI's information technology systems. I believe this is one of the most critical challenges facing the FBI.
2. The FBI faces challenges in the human capital area. Between November 2001 and February 2005, 15 different key information technology managers have been involved with the Trilogy project, including 5 FBI Chief Information Officers and 10 individuals serving as project managers for various aspects of Trilogy.
3. A third critical challenge facing the FBI is its need to share intelligence and law enforcement information efficiently, both within the FBI and with its law enforcement and intelligence partners.
4. Fourth, I believe the FBI must value and support to a greater degree FBI staff with technical skills. For example, until recently, the FBI did not adequately value the contributions of intelligence analysts. Special agents historically were promoted to technical leadership positions within the FBI, such as handling information technology upgrades or leading scientific efforts in the laboratory.
Commenting on a February 2005 Trilogy audit report, the OIG Inspector General noted that " the FBI was unable to create and deploy the VCF after more than 3 years and $170 million budgeted for the project. The OIG audit report concluded that the VCF either would require substantial additional work or would need to be scrapped and replaced by a new system. Moreover, at the time of the audit, the FBI had not provided a realistic timetable or cost estimate for implementing a workable VCF or a successor system."
The Inspector General noted that the OIG audit had identified a variety of causes for the delays and cost increases in the Trilogy project, including:
- poorly defined and slowly evolving design requirements for Trilogy,
- weak information technology investment management practices at the FBI,
- weaknesses in the way contractors were retained and overseen,
- the lack of management continuity at the FBI on the Trilogy project,
- unrealistic scheduling of tasks on Trilogy, and
- inadequate resolution of issues that warned of problems in Trilogy's development.
The OIG report assigned responsibility for Trilogy's success (or lack of it) to several parties: the FBI; the Department; FEDSIM (the component of the General Services Administration that awarded Trilogy contracts on behalf of the FBI); and the two contractors; Computer Sciences Corporation for the two infrastructure components, and Science Applications International Corporation for the user applications component that included the VCF. The report concluded that "these entities, to varying degrees, did not effectively contract for, manage, monitor, or implement the Trilogy project."
While the blame was shared, the OIG report faulted the FBI "for moving forward with contracting for this complex project without providing or insisting upon defined requirements, specific milestones, critical decision review points, and penalties for poor contractor performance." The audit concluded that because of the inability to develop and deploy the VCF, the FBI continued to lack critical tools necessary to maximize the performance of both its criminal investigative and national security missions.
The FBI had acknowledged and was addressing most of the Inspector General's concerns. In its Response to the Draft Report [5], the FBI noted that National security remained uncompromised by the delay of VCF. The FBI's response listed a number of IT improvements and concluded "In short, the FBI's capacity to access, analyze, and share data internally and externally has improved considerably since the OIG began this audit, strengthening our ability to predict and prevent acts of terrorism and otherwise supporting our national security mission. Additional improvements currently underway will further strengthen these capabilities over the next few months."
The FBI was confident it would do better with the Sentinel project. They had been planning it for a year, evaluating commercial off-the-shelf software, creating an enterprise architecture, and establishing a number of IT management oversight boards. They had also provided project management training to 80 IT staff members.
Ken Orr, one of the CSTB "greybeards" thought they were kidding themselves about buying and installing something within a year. He believed the FBI needed to work out "how to bring these new software programs online incrementally and train more than 30 000 people to use them. Then they could focus on converting millions of paper records as well as all of the audio, video, photographic, and physical evidence that has piled up over the years, which will continue to grow at an increasing rate to support the bureau's counterterrorism mission. " His guess was that it would be closer to 2010 or 2011 before the FBI had the complete system up and running.
OIG May 2006 Testimony
So how has the new Sentinel project actually fared since its inception in May 2005? The OIG's May 2006 testimony [6] is encouraging but there are still a few issues. It noted that the OIG's March 2006 audit found the FBI has taken important steps to help prevent the types of problems encountered in the Virtual Case File project. and had developed information technology planning processes that, if implemented as designed, could help the FBI successfully complete Sentinel. Notable improvements included establishing Information Technology Investment Management processes, developing a more mature Enterprise Architecture, and establishing a Program Management Office dedicated to the Sentinel project.
But there were still a few areas of concern. To quote:
- (1) the incomplete staffing of the Sentinel Program Management Office,
- (2) the FBI's ability to reprogram funds to complete the second phase of the project without jeopardizing its mission-critical operations,
- (3) Sentinel's ability to share information with external intelligence and law enforcement agencies and provide a common framework for other agencies' case management systems,
- (4) the lack of an established Earned Value Management process,
- (5) the FBI's ability to track and control Sentinel's costs, and
- (6) the lack of complete documentation required by the FBI's Information Technology Investment Management processes.
The OIG was also concerned that while the FBI had considered its internal needs in developing Sentinel's requirements, it had not yet adequately examined or discussed Sentinel's ability to connect with external systems. The testimony notes: "If such connectivity is not built into Sentinel's design, other agencies could be forced into costly and time-consuming modifications to their systems to allow information sharing with the Sentinel system".
Wikipedia Summary
The VCF gets a write-up in Wikipedia [7], which gives provides a good overall summary of the reasons for failure from a software engineering point of view, as follows:
- The project demonstrated a systematic failure of software engineering practices:
- Lack of a strong blueprint from the outset led to poor architectural decisions.
- Repeated changes in specification.
- Repeated turnover of management, which contributed to the specification problem.
- Micromanagement of software developers.
- The inclusion of many FBI Personnel who had little or no formal training in computer science as managers and even engineers on the project.
- Scope creep as the requirements were continually added to the system even as it was falling behind schedule.
- Code bloat due to changing specifications and scope creep. At one point it was estimated the software had over 700,000 lines of code.
- Addition of more people and resources to the project as it was falling behind, a violation of Brooks' law.
- Planned use of a flash cutover deployment which made it difficult to adopt the system until it was perfected.
References:
1. Who Killed the Virtual Case File? http://www.spectrum.ieee.org/sep05/1455
2. FBI Needs an Enterprise Architecture to Guide Its Modernization Activities http://www.fas.org/irp/gao/gao-03-959.pdf.
3. NRC Report, IT-Related Issues for the FBI Requiring Immediate Action http://darwin.nap.edu/books/NI000561/html/1.html
4. OIG Inspector General's 2005 testimony to the Committee on the Judiciary, http://judiciary.senate.gov/testimony.cfm?id=1589&wit_id=4561
5. The FBI's Response to the Draft Report, http://www.usdoj.gov/oig/reports/FBI/a0507/app7.htm
6. OIG Inspector General's 2006 testimony to the Committee on the Judiciary, http://darwin.nap.edu/books/0309092248/html/16.html
7. Wikipedia, http://en.wikipedia.org/wiki/Virtual_Case_File