Anatomy of an IT disaster: How the FBI blew it

Anatomy of an IT disaster: How the FBI blew it

Some U.S. Federal Bureau of Investigation (FBI) agents ruefully refer to the trilogy project, a massive initiative to modernize the FBI's aging technology infrastructure, as the 'Tragedy' project. It certainly has all the earmarks of tragedy: the best intentions, catastrophic miscommunication, staggering waste.

Some U.S. Federal Bureau of Investigation (FBI) agents ruefully refer to the trilogy project, a massive initiative to modernize the FBI's aging technology infrastructure, as the "Tragedy" project. It certainly has all the earmarks of tragedy: the best intentions, catastrophic miscommunication, staggering waste.

Trilogy, as the name suggests, had three parts: an enterprisewide upgrade of desktop hardware and software; deployment of a modern network infrastructure; and an integrated suite of software for entering, finding, sharing, and analyzing case information. In a congressional hearing last month, FBI Director Robert Mueller was careful to note that the first two parts of Trilogy have been completed: no less than 30,000 computers, 4,000 printers, 1,600 scanners, 465 servers, and 1,400 routers were deployed as of April 2004.

After more than four years of hard work and half a billion dollars spent, however, Trilogy has had little impact on the FBI's antiquated case-management system, which today remains a morass of mainframe green screens and vast stores of paper records. As Senator Judd Gregg observed, "the software, which runs the hardware, is a huge problem."

The problem with that software, known as VCF (Virtual Case File), is that it isn't in production and may never be. VCF may be one of the most extreme examples of requirements bloat in IT history. What began as a fairly modest software project swelled into an all-encompassing replacement for a panoply of woefully outmoded applications and procedures. Along the way, the FBI went through five different CIOs, 10 project managers, and 36 contract changes. The result, said Senator Patrick Leahy at February's Senate Appropriations Committee hearings, "has been a kind of train wreck in slow motion."

Accounting for US$170 million of Trilogy's $581 million price tag, VCF fell afoul of extraordinary circumstances --notably, the Sept. 11 attacks, which piled enormous pressure onto the Trilogy project and altered the course of VCF dramatically.

FBI representatives declined to be interviewed for this story. But thanks to the Senate hearings, a report from the U.S. Department of Justice's Office of the Inspector General (OIG), and interviews with the FBI contractor that developed VCF, InfoWorld has been given a rare glimpse into the inner workings of a colossal IT failure.

Green screens and filing cabinets

Sen. Leahy offered another, more whimsical analogy for Trilogy: the 1993 movie Groundhog Day, in which Bill Murray wakes up each morning to relive the same day. Since 1997, proposals for modernizing the FBI's technology and processes have emerged again and again, culminating with Trilogy. Trilogy itself then underwent a cyclic series of evaluations and funding requests until Congress finally learned that its third leg, VCF, might never materialize.

For the foreseeable future, that leaves the FBI with its obsolete, mainframe-based ACS (Automated Case Support) system, which requires the user to traverse a dozen 3270 green screens to upload a single document. Worse, according to the OIG's report, "the ACS only serves as a backup to the FBI's paper file system (and) information within that system cannot be changed or updated."

By the year 2000, aging infrastructure alone -- including 386-based desktop PCs and 12-year-old interoffice networks -- was hobbling the FBI. In September 2000, the FBI proposed FITUP (FBI Information Technology Upgrade Project), for which Congress allocated $379.8 million, spread over an estimated three-year effort. Two months later the project was divided into three parts and renamed Trilogy.

Trilogy's requirements for desktop hardware and network infrastructure stayed relatively stable throughout the life of the project (although an additional $100 million was allocated to accelerate completion, to no avail). The software portion was a different story.

What we now know as VCF was preceded by Trilogy's original and quite different proposal, UAC (User Applications Component). The objective of UAC was simple: "Webify" five of the 42 mainframe applications employed by FBI agents in the course of investigations. On the face of it, this seems like a sensible first step toward modernization, but it can also be seen as the crucial error of the entire Trilogy project.

Putting a pretty front end on green screens did nothing to change the underlying processes, in which paper records were primary. Also, according to the OIG report, the choice of which applications to Webify was not based on objective research and evaluation. Instead, FBI IT managers simply picked the five applications used most often by agents.

Disastrous turns

The FBI realized it didn't have the internal IT expertise to pull off Trilogy. So in mid-2001, it contracted with DynCorp (later purchased in early 2003 by consulting giant Computer Sciences Corp.) for the desktop and infrastructure components and SAIC (Science Applications International Corp.) for the software.

Only a few months after the ink was dry on these contracts, the Sept. 11 tragedy struck, reshaping the mission of the FBI. No longer would the Bureau be concerned merely with law enforcement. Instead, to protect against terrorism on U.S. soil, the FBI needed to get into the intelligence business.

This shift turned the requirements for UAC inside out. Instead of beautifying old mainframe apps, the charter changed to replacing those applications with a new, collaborative environment for gathering, sharing, and analyzing evidence and intelligence data.

"About six months into the contract, SAIC was essentially told by the FBI that 'we really want to change what you're doing on this contract,' " recounts SAIC Group President Mark Hughes. " 'Set aside the work you've done to date and what we'd like you to do is work with us to build a brand new case-management system for the FBI. And by the way, we don't have any requirements for that yet, so we need you to work with us to help develop those requirements.' "

According to Hughes, those requirements were almost impossible to nail down, because the FBI's new mission set in motion a rolling transformation of internal processes.

One requirement was a so-called "flash cutover." In other words, says Hughes, "you build this big system, and when it's ready, you turn the old one off and you turn the new one on. To anybody who knows anything about big systems, that's a very risky way to introduce a new system."

SAIC claims that it pushed back on this and other unrealistic requirements. At the time, however, SAIC had a cost-plus-award contract. According to Gartner Fellow John Pescatore, these types of contracts estimate the real cost of a project and add a profit margin that is awarded annually to the contractor -- in full, in part, or not at all, depending on the government's rating of the contractor's performance for that year.

"So here's what happens," Pescatore says. "In the beginning, you never want to say no, because you'll get a bad rating. It essentially incents the contractor to be much more accepting of out-of-scope changes. It's kind of like a mass-suicide pact, except you're hoping a miracle is going to occur later on."

The prototype from hell

At the February 2005 hearing, Mueller said that the FBI delivered "finalized" requirements for VCF in June 2002, which included integrating the functionality of the five original ACS applications with the new system. But according to Hughes, the changes kept coming at a rate of more than one per day. Cli ck for larger view.

What followed was the mother of all misunderstandings. In 2002, Hughes says, SAIC offered a proposal identifying December 2003 as the deployment date for VCF. He maintains, though, that as the changes rolled in, SAIC alerted the FBI that the cost and delivery date would be seriously affected. Ultimately, instead of a final version, what SAIC delivered in December 2003 was an incomplete system for evaluation.

The result, according to Hughes, was consternation. "Apparently the communication about what was going on in the project had not gone up the chain in the FBI and in (the Department of) Justice and in Congress;" he says, "and when we didn't actually deploy a completed system in December 2003, there was a lot of surprise at those levels."

Mueller voiced his bitter disappointment over the December version of VCF in the hearings. "When SAIC delivered the first product in December 2003, we immediately identified a number of deficiencies, 17 at the outset. That soon cascaded to 50 or more and ultimately to 400 problems with that software."

Hughes still bristles at having that prototype tested as a final version, but he does admit to some culpability. "That tells you one thing we should have done better: talking at every level in the government to make sure everybody was on the same page."

Downsized delivery

In January 2004, the FBI hired a new acting CIO, Zalmai Azmi, its fifth in four years. Azmi officially took the CIO job in May 2004. An intense back-and-forth between Azmi and SAIC ensued during the first half of that year, with SAIC determined to nail down an unchanging set of requirements and Azmi pushing for delivery based on a contract that was performance-based, rather than cost-plus-award.

Both parties got their wish in June 2004, when Azmi and SAIC worked out a two-track plan. In December 2004, SAIC would deliver an IOC (initial operating capability), a workflow application that would automate the case-management document approval process. According to the OIG report, the FOC (full operating capability) was simply a new effort to "identify new requirements for developing a functional case management system to replace the ACS."

The IOC delivery is still a source of pride for Hughes. "That system contained 100 percent of the requirements that the customer said they wanted in the IOC system," he says.

The IOC system is currently being field-tested, but the FBI's take on it was somewhat different. At the hearings, Mueller and Azmi repeatedly alluded to a still-classified report by the nonprofit Aerospace Corporation that tallied up a long list of deficiencies in the IOC. Azmi also stated that the IOC represents only one-tenth of the VCF's intended capability, a claim that SAIC Executive Vice President Arnold Punaro rejected in interviews with reporters after the hearing, arguing that VCF never had a baseline to begin with.

Groundhog Day redux

As SAIC labored on the IOC, still other plans were afoot. In September 2004, the FBI and the Department of Homeland Security began planning an interagency FICMS (Federal Investigative Case Management System) that would make the still-unfinished VCF obsolete.

The FICMS proposal sends FBI case management back to the drawing board at a time when, according to Hughes, SAIC's version of the VCF is further along than it may appear. "There are other capabilities that we have already designed and coded, and in fact the code is actually embedded in the IOC system -- but we turned it off and didn't test it or integrate it, because they didn't want it as part of the IOC."

Hughes estimates it will take three years for FICMS to become a working system. Meanwhile, he says, tests have already proven that SAIC's system, if deployed in stages, can scale to do what the FBI needs it to do. "I think it's crazy not to deploy it, regardless of how they want to go in the future," he says.

Today, however, everything remains uncertain. The pilot program for the IOC ends this month. The FBI says it will award a contract for the development of FICMS next month. If it does, then the FBI will likely abandon VCF.

After the fact, members of the Appropriations Committee have raised many objections to the way the VCF project proceeded: off-the-shelf software should have been considered, management should have been more forthright about problems, SAIC should have had its feet held to the fire.

The reality, though, is that IT backwaters like the FBI can't modernize without an extensible enterprise architecture, which the FBI admits it's only beginning to develop. Meanwhile, Gartner's Pescatore says only 76 percent of FBI personnel are using secure e-mail. Such outrages mean that, before any grand architecture rolls out, point solutions such as VCF must have their day. Side bar interview Peeking under the hood of VCF

The details of how the FBI does its business and the systems that support the bureau are classified. Nonetheless, Executive Editor at Large Eric Knorr did his best to get a rough description of VCF (Virtual Case File) -- the $170 million system that never launched -- from SAIC Group President Mark Hughes and Technical Director Frank Perry.

IW: I know I'm not going to get a diagram, but can you thumbnail it in words?

Frank Perry: Without going into any details, I think what you would find is a pretty standard, three-tier, Web-based, enterprise-scale application for an enterprise of tens of thousands of people.

IW: Java-based?

FP: Yes.

IW: The most alarming thing I heard about this project was the "flash cutover." I imagine on the back end of that three-tier architecture you would still integrate with the legacy system; you wouldn't just shut down all the old mainframes. Or was that what was planned?

Mark Hughes: That's what was planned initially, but when they decided to do the incremental deployment, it's just as you say. In fact, the IOC system has been integrated to some degree with their legacy systems, called ACS.

IW: How did you integrate with the mainframes? Did you use screen scraping?

FP: I don't think we can go into too much detail, but at the level that we did the integration for the IOC capability -- it's a data-level integration.

IW: Can you say anything about the security?

MH: All I can say about security is that the security requirements of the FBI are really quite unique. On the one hand, they're dealing with pressures to share information. On the other hand, there are sources and methods and things like that, which really need to be classified. And then the FBI had some espionage problems, like the (Robert Philip) Hanssen case, where an insider was using their systems to get information. So the systems have to be designed to deal with that kind of thing, too.

IW: Would it be fair to characterize the security as identity based?

FP: I don't think we can really get into that. -- InfoWorld (US)

Join the CIO New Zealand group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.

Join the newsletter!

Error: Please check your email address.
Show Comments