Refusing to Misunderstand the Defense Acquisition Problem

On October 2, the Permanent Subcommittee on Investigations of the Senate Committee on Homeland Security and Government Affairs released a document titled "Defense Acquisition Reform: Where Do We Go from Here? A Compendium of Views by Leading Experts."

The Compendium is the work product of Senators Carl Levin and John McCain and their staffs. There is a descriptive summary of the document at National Defense Magazine.

The effort collected essays from 31 defense acquisition specialists, varying from current and former Pentagon managers to think tank and GAO analysts.

While it surely reflects the conventional wisdom at the Levin/McCain Permanent Subcommittee on Investigations, and the Senate Armed Services Committee where those two senators also reign, the National Defense Magazine summary article contains a remarkable sentence: "That military weapon acquisitions continue to be plagued by cost overruns and schedule delays is particularly frustrating to many lawmakers who expected major problems would end after Congress passed to great fanfare the 2009 Weapon Systems Acquisition Reform Act."

While the authors of the 2009 Weapon Systems Acquisition Reform Act (WSARA), such as Senators Levin and McCain, did indeed loudly ballyhoo their legislation as the solution to the Pentagon's massive (and deepening) acquisition troubles, the complete failure of their Act to do anything to end massive cost growth, seemingly endless schedule delays and gigantic performance disappointments in DOD acquisition should come as a surprise to no one.

Indeed, that failure came as no surprise to one of the authors who contributed to the Compendium. Thomas Christie spent almost 50 years in the DOD acquisition bureaucracy, ending his career as George W. Bush's selection to run DOD's independent operational testing office, known as "DOT&E." Explaining--in passing--why WSARA failed, Christie further explains why the predictable prognostications for DOD acquisition reform persistently fail. Solving the problem is not a matter of still more Pentagon reorganizations, regulation and stature "streamlining," a perverse "culture" in the bowels of the bureaucracy, or better training for mid-level bureaucrats, as the Leven/McCain description of their Compendium would have you believe. Instead, the problem resides--in the most fundamental manner--in the people who make the decisions that permit all the problems to continue, even expand--as they have since WSARA.

In the calmest of tones, and with compelling evidence, Christie explains in his essay for the Levin/McCain Compendium. His article is titled "Developing, Buying and Fielding Superior Weapon Systems." It follows--in its original form as submitted to the PSI. I urge you to read it.

Developing, Buying and Fielding Superior Weapon Systems

by Thomas Christie

The current Defense Department acquisition process that develops, tests, and procures new weapons for U.S. combat forces has evolved over the past five decades in response to multiple defense management strategy initiatives, external reform proposals and lessons-learned from the field. The conventional wisdom notwithstanding, the process as spelled out in DoD's directives and instructions is fundamentally sound and could avoid its unending cost overruns, delays and performance failures, if it were implemented in an better informed and rigorously disciplined manner. The problem is not nearly as much in the laws and regulations as it is in the execution by the people who have been operating the system.

We should not waste time in this short essay re-inventing bromides for the bureaucracy to cogitate and self-appointed reformers to contrive. Essential ingredients to a viable weapons acquisition system include -

• budgeting with truly independent estimates of program development, procurement and support costs;

• an evaluation process, again independent, to find and correct reliability problems earlyand throughout the entirety of a program's life cycle;

• conducting combat realistic operational tests of weapons and honest and complete reports to permit decision makers inside and outside the Pentagon to make properly informed judgments.

Anyone familiar with the way the system has broken down up to now knows these are needed, but there is also more. There are other features of the process that need attention and must be executed, not circumvented, to achieve successful weapons at affordable cost in a reasonable time. These other aspects include -

• insisting on discipline throughout the decision-making process;

• cleaning up the front end of the process where dubious requirements and buy-in cost and schedule promises are greeted without criticism and committed to;

• demonstrating - through empirical field testing, not success oriented modeling and simulation - new technologies before each major decision-maker approval point;

• establishing and carrying out event-based strategies accompanied by realistic pass/fail criteria for each phase of a program;

• conducting continuous independent evaluations of programs all the way through development, testing, production, and even after introduction in the field - to include training exercises and combat results, and

• feeding back all such results completely and accurately to the entire acquisition community.

Nothing in today's laws and regulations prevent any of the above; most are actually called for, and yet they almost never happen.

The Need for Reform Is Not New

Proceeding with any new weapon system development, production and fielding with the Pentagon acquisition process as currently implemented (or, perhaps more appropriately, not implemented) will only continue the debacles of the past. Both past and present Pentagon leadership has been painfully aware that "Something's wrong with the system," as Secretary of Defense Donald Rumsfeld told Congress in 2005.

More recently, Secretary of Defense Robert Gates was perceptive in stating -

"First, this department must consistently demonstrate the commitment and leadership to stop programs that significantly exceed their budget or which spend limited tax dollars to buy more capability than the nation needs.

Second, we must ensure that requirements are reasonable and technology is adequately mature to allow the department to successfully execute the programs.

Third, realistically estimate program costs, provide budget stability for the programs we initiate, adequately staff the government acquisition team, and provide disciplined and constant oversight.

We must constantly guard against so-called "requirements creep," validate the maturity of technology at milestones, fund programs to independent cost estimates, and demand stricter contract terms and conditions."

Once again, there is nothing wrong with the assertions, but even with Secretary Gates' many subsequent program alterations, a few actual cancellations, and some modest overhead savings, can anyone say that the Pentagon has transformed into what Gates said he wanted? More, much more, actual implementation is required.

Congress has behaved similarly - with words more grandiose than actions. In 2009, it weighed in with its latest attempt to rescue the Pentagon's acquisition processes: the Weapon Systems Acquisition Reform Act of 2009 (WSARA 2009). In addition to reestablishing test and evaluation and system engineering capabilities eliminated by the Clinton administration with Congress' consent, WSARA 2009 directed the application of several ideas that had been advocated for decades; these included independent cost assessments; evaluating trade-offs of cost, schedule and performance; and competitive prototype development and testing.

But will the Pentagon actually follow what Congress says it intends with this legislation, or will it exercise the many loopholes that Congress consciously inserted into virtually every requirement - at the explicit request of top DOD management - to permit circumvention of most, or all, of these reforms? History suggests the latter.

The Problem is Not Lack of Study

It is difficult to find another process that has been studied more than DOD acquisition. Every three to four years, yet another high-level study is commissioned to review DOD management in general and the acquisition process in particular, or Congress steps in and legislates, in great detail, how the Pentagon should organize and carry out its mission. The commissions, studies, and statutes are many.

The common goal for many of these efforts has been "streamlining" the acquisition process. Typical techniques recommended were efforts, not always successful, to reduce management layers, eliminating reporting requirements, replacing regulated procurement with Commercial off the Shelf (COTS) purchasing, reducing oversight from within as well as from outside DoD, and eliminating perceived duplication of testing.

Advertised as reform, most of these efforts had the real effect of reducing oversight. While oversight by government agencies and the associated reporting requirements can indeed be burdensome, they are not the causes of the continuing miserable record of program stretch outs and cost growth. This is true independent of whether those agencies and their reporting requirements are internal to DOD, such as the Defense Contract Management Agency (DCMA), independent cost analysis groups, operational test and evaluation organizations; or external entities, such as the congressional committees and the Government Accountability Organization (GAO). This finding is borne out by my decades of participation in the acquisition process and some of the more competent official reviews, such as that done in 1990, a streamlining review performed by the Defense Science Board (DSB).

That 1990 DSB review, covering some 100 major defense acquisition programs, concluded that failure to identify and admit to technical problems, as well as real costs, before entry into what was known as Full-Scale Engineering Development (FSED) - now referred to as Engineering and Manufacturing Development (EMD) -- was the overwhelming cause for subsequent schedule delays, often years, and the resulting cost growth. Oversight enabled the discovery and reporting of test failures during FSED/EMD that often necessitated additional time and dollars for system redesign, testing and retesting of fixes, as well as costly retrofits of those fixes. It is a viable question, however, whether these delays caused more, or less, schedule alteration to utility in the field than discovering the problems late, after deployment; without question, testing and finding problems early, before serial production, is cheaper - by a very large margin.

After all the reforms of previous decades, here we are in 2010 and what's demonstrably different from the past? Major defense programs are taking 20 to 30 years to deliver less capability than planned, very often at two to three times the cost. It all may be worse now than ever before.

The basic problem is not the official directives. Instead, Pentagon acquisition officials too often have violated the regulation's intent by approving "low-balled" estimates of the costs and time required to deliver new capabilities, and ignoring independent assessments that were often available and more realistic. Time and again, early-on funding for building and testing prototypes to better understand technical and operational issues has gone by the wayside. A powerful - overwhelming - factor in the making of these slipshod decisions is the competition for dollars inside the bureaucracy: approve the money now, lest it be grabbed by another program.

A typical hardware program will involve three to five administrations and ten, or more, congresses. By the time the technical and cost issues finally become known in the current system, few, if any, of those involved initially are still around, and those who are refuse to admit they had been wrong, to cut their losses before the problems worsen, or to discipline the system by making an example of program officials and their contractors who have sold the department and the taxpayers a bill of goods.

What Is Needed?

There isn't much that knowledgeable observers of, and participants in, this process haven't already identified as problems and have proposed solutions for. They all appear in existing acquisition directives and instructions. Implementing them, rather than exercising their loopholes, is the starting point for fixing the process.

With the current national fiscal environment and the lack of significant threats projected for the foreseeable future, waivers of the procedures and criteria for success that the regulations were designed to uphold should be few and far between, if they occur at all. In addition, they should be escalated to the Secretary of Defense for major, and even some lesser, programs. Finally, the Defense Department should not proceed with any program with waived requirements until the Congress and its independent arm, the GAO, have evaluated the rationale for the requested waivers, and the appropriate Congressional committees give explicit, statutory approval to proceed. There is no rationale for not taking the necessary time for scrupulous analyses to determine whether we should embark on a new program, and the responsibility and accountability must be clearly established and accepted at the top of the system.

The Front End: Setting Requirements

Hard-nosed discipline on the part of decision-makers at the front end of the process is crucial to reining in the appetite of the requirements community and precluding ill-informed development decisions based on immature technologies and optimistic projections of system costs, schedule, and performance. Upfront realistic cost estimates and technical risk assessments, developed by independent organizations outside the chain of command for major programs, should inform Defense Acquisition Executives. The requirement for those assessments to be independent, not performed by organizations already controlled by the existing self-interested sections of the bureaucracy - as is the case now, even after WSARA 2009 - is essential.

The existing process has heartily approved presumed quantum leaps in claimed capability that are reflected in high-risk, often unattainable, technical and operational requirements. Many of these system performance goals have resulted from the salesmanship of the DOD research and development communities, combined with industry lobbying, in successfully convincing the user and the rest of the acquisition community that the hypothetical advanced capabilities could be delivered rapidly and cheaply

In case after case, Pentagon decision-makers have acquiesced to programs entering FSED/EMD and even low-rate initial production before technical problems are identified, much less solved; before credible independent cost assessments are made and included in program budget projections; and before the more risky requirements are demonstrated in testing. This is nothing more than a "buy-in" to "get the camel's nose under the tent."

The MV-22 is a good example of a major program that encountered technical and cost problems after entering EMD in 1986, yet was approved to enter low-rate initial production (LRIP). In 1999, the urgency of replacing aging CH-46s drove decisions to severely reduce development testing before its completion, to enter operational testing prematurely and to gain approval LRIP.

In April 2000, an MV-22 crashed during an operational test resulting in the deaths of 19 Marines. The official investigation into this tragic accident reported that the Flight Control System Development and Flying Qualities Demonstration (FCSDFQD) Test Plan investigating the phenomenon known as power settling was reduced from 103 test flight conditions to 49, of which only 33 were actually flight-tested with particularly critical test points not flown at all.

This series of events, culminating in the April 2000 accident and another crash in December of that year, brought the program to halt, nearly resulting in termination.

However despite these setbacks, the program continued in low-rate production while Pentagon leadership debated whether to continue the program. In the end, the MV-22 program recovered, executed the full range of technical testing that should have been done previously, and was introduced into Marine Corps medium-lift forces in 2005, nearly 25 years after the decision to initiate the program. In the meantime, some 70 or more MV-22s had been procured, many of which required expensive modifications to correct deficiencies discovered in testing.

Among the Many False Reforms

The process has become even more cumbersome with increased involvement of the Joint Chiefs of Staff (JCS). Over the years, the Joint Requirements Oversight Council (JROC) and the Joint Capabilities Integration and Development System (JCIDS) process were established to ostensibly provide the combat forces a greater voice in setting requirements. There is, however, little evidence that the "reformed" process has made any significant changes to programs as originally proposed by the advocates.

Real Reform: Considering Alternatives

Approval to proceed with any new development should depend on requirements, both technical and operational, that are attainable, affordable and testable and are based on realistic threat and funding projections. Most crucial to an effective new start is the conduct of an independent Analysis of Alternatives (AOA) that explores other approaches to meeting an identified need. The proposed solutions should run the gamut from continuing existing systems, to incremental improvements to those systems, to launching the development and procurement of a new system. DOD's regulations in Instruction 5000.2 call for AOAs to be completed and/or updated before each of "Milestone" review, but in reality they have been few.

A thorough AOA should be a hard and fast prerequisite for any Milestone review. It should focus on an independent life cycle cost estimate (R&D, procurement and operating and support) and on the affordability of the various alternatives. It should also include realistic projections into the outyears for cost, force levels, manpower support requirements, total procurement quantities, and affordable annual procurement rates. Done properly, an AOA should generate cost and schedule thresholds as well as key performance parameters (including measures of effectiveness, survivability, interoperability, and reliability and maintainability thresholds) upon which the rationale for a new program is based and where it fails in comparison to others. The performance thresholds should include both technical and operational measures that, in turn, should guide the planning and execution of both development and operational testing focused on those key parameters that constitute the justification for proceeding with the new program.

These independent analyses should be updated at regular intervals, not just for each program milestone. The process should be one of continuous evaluation, incorporating updated cost estimates and system performance projections, based on experience in development and testing to-date.

Periodic program assessments should weed out programs that are falling behind schedules, growing in cost and falling short of key measures of effectiveness and suitability.

Real Reform: Fly-Before-Buy/Competitive Prototype Demonstration

The "Fly-before-Buy" philosophy should be the mandated standard for all programs. Perhaps a better term would be "Fly-before-Decide." Done properly, it will demand the demonstration, through actual field testing of new technologies, subsystems, concepts, etc. to certain success criteria before proceeding at each milestone, not just the production decision. Accordingly, successful and competitive prototype development and testing should be a hard and fast prerequisite for any new development proceeding into the FSED/EMD phase. The Achilles heel of many a defense program has been its failure to adhere to this strategy, resulting in technical difficulties and costly development delays that could have been avoided had the decision-maker demanded successful completion of realistic prototype testing and evaluation.

Critical to the success of such a strategy is allocating sufficient up-front funding and schedule to permit a robust comparative evaluation of prototype systems in an operational environment during the Demonstration/Validation (Dem/Val) phase. The Department has paid only lip service in the past to the competitive prototype demonstration requirement spelled out in its own directives. DOD should establish, adequately fund, and maintain operational units (e.g., aircraft squadrons, ground force brigades/battalions), independent of R&D organizations, to conduct tests and experiments to effect this concept.

Real Reform: Event-Based, Not Schedule-Based Decisions

DOD's experience with systems entering operational testing prior to completion of sufficient development testing is chronicled in innumerable GAO and several Defense Science Board (DSB) reports in recent years.A May 2008 DSB Task Force Report on Development Test and Evaluation found that, in the ten year period between 1997 and 2006, over two-thirds of Army systems failed to meet their reliability requirements in operational testing. In almost all these cases, the systems had entered OT&E with little or no chance of success, based on the failures demonstrated during development testing. These programs had not met the criteria for successful completion of development testing and had entered OT&E doomed to fail.

The acquisition decision authority should impose an event-based strategy on programs with meaningful and realistic pass/fail criteria for each stage of development and production. Only if the criteria are satisfied (through actual testing where applicable) should the decision-maker allow a program to proceed to its next phase. For example, when a program is approved at Milestone B to move into EMD, approval to successfully pass a future Milestone C and proceed into low-rate initial production should be predicated on the program demonstrating specific performance/reliability/cost thresholds.Failure to achieve these goals should result in program termination or at least significant restructure until they are met.

Real Reform: Continuous Evaluations

As a new program begins, a process of continuous and independent evaluation must be established to track the program through development, testing and production, and eventual fielding and employment by operational forces. In the early stages, such evaluations should be based on emerging test results and updated cost estimates and should focus on those attributes or capability measures that formed the basis for program approval. These evaluations should be updated with results presented to senior leadership on a routine basis, certainly at least annually. Such evaluations should inform decisions whether or not to proceed with the program or to restructure the program goals and acquisition strategy.

It is extremely important that this process of continuous evaluation extend beyond development. Organizations, independent of both the development and operational communities, should be established and maintained to track experience with new and existing systems in the field, evaluating data gathered in training sorties and exercises as well as in combat, where applicable. Assessments should include not only the usual measures of system performance, but also all aspects of system supportability, to include reliability, availability and maintainability (RAM), as well as safety, training, and human factors.

Feedback loops from the field to the requirements and acquisition communities should be established and maintained throughout the life of a weapon or system. Such arrangements should take full advantage of operational experience in developing plans and requirements for starting a new program, determining needed fixes for deficiencies encountered in the field, continuing and/or upgrading existing systems. Such lessons learned should be invaluable to the acquisition community in shaping its approach to the development of new systems as well as to the T&E and analytic communities in structuring their evaluations of similar systems in the future.

Conclusion

As the country enters what promises to be a prolonged period of fiscal austerity, it can no longer afford the extravagance of spending hundreds of billions of dollars and not receiving the capabilities it paid for. Fortunately, we have an extensive base of experience, derived from both military and commercial programs that we can draw upon to avoid the mistakes of the past. These lessons have been codified in DoD regulations, and the evidence shows that the vast majority of cost overruns and schedule delays come from avoiding their requirements, particularly in the initial stages of a program.

We are also fortunate that there is no need to rush new systems into development and procurement in order to counter some imminent new threat. The F-16, for example, entered operational service in 1980 and is still in production. It and the remaining A-10s in the Air Force's inventory are more than adequate aircraft for existing missions in Afghanistan and for conventional threats, should they arise. There is no projected threat on the horizon that would justify taking additional risk by compressing development schedules for any new system (such as the highly problematic F-35 program). Moreover, compressing prescribed schedules when real threats actually exist, such as during the Cold War, has proven to be a huge cost and performance disaster - and to save no time.

We have the tools and expertise we need to make substantial reductions in the cost overruns, performance disappointments, and schedule slips that plague our weapon programs. What we do not have, or have not had consistently, is the determination to apply the available tools, especially when it means canceling programs that are generating careers in the Pentagon and jobs and votes outside it.