Dangerous Testing Plan for the F-15EX

Disastrous weapons programs like the Littoral Combat Ship, the F-35, and the Army’s Future Combat System serve as master classes in how not to develop, test, and purchase military hardware. In each of those programs the Pentagon committed to buying unproven systems and paid the price in schedule delays and increased costs. Rather than applying the painful lessons learned over the past 60-plus years to its acquisition of an extensively modified new air-to-ground version of the F-15 fighter, the Air Force is doubling down on its worst testing and concurrency practices.

Just as most of us would never buy a car without a test drive, the best practice for buying new systems should be to fly before we buy. In the Pentagon, that means conducting combat-realistic operational tests before a weapon system is put into production. In a misguided attempt to weaken oversight and speed up procurement, the Air Force’s F-15EX program officers plan to merge the plane’s development and its operational tests (combined DT/OT). This is, in fact, a recipe for slowing down the discovery and correction of design deficiencies — and, even worse, it will reduce the realism of the operational tests that are meant to determine whether the plane is worth buying. When the procurement process is done correctly, developmental testing is conducted first to uncover any design flaws and to ensure those flaws are corrected, and then prototypes of the product go through operational test teams to see if the completed design works properly in the hands of the troops.

Why? Because development tests are done by engineering test pilots, on behalf of the developer, to uncover any deficiencies in meeting engineering specifications, then to cure and retest those deficiencies as quickly as possible. These aims are often incompatible with operational testing, which is necessarily done on behalf of the combat user, by tactical pilots evaluating the combat suitability of the fully production-ready, deficiency-free plane under stressful, realistic, non-engineering conditions. These combat suitability evaluations must be done without regard to the developer’s engineering specifications, which are rarely direct measures of combat effectiveness. For an example of this, look no further than the F-35’s flawed maintenance network: It was supposed to increase the program’s reliability, but ended up drastically reducing the program’s overall effectiveness.

F-15EX program officials plan to combine the developmental and operational testing in a misguided attempt to speed up the procurement process. To allow this, Pentagon leaders classified the F-15EX program as a “Middle Tier Acquisition.” As such, it is subject to substantially reduced oversight from Congress and the secretary of defense than major defense acquisition programs like the F-35 — meaning there will be even less scrutiny to ensure it is effective and safe. The major defense acquisition program process involves multiple formalized reviews to inform the decisions made by the secretary of defense and congressional committees. The reviews include an Analysis of Alternatives, to determine if other weapons could perform the same mission; an Independent Cost Estimate; and a Test and Evaluation Master Plan to evaluate how the new weapon system performs in combat.

Despite the new buzzwords describing the F-15EX’s program plan — “Combined DT/OT,” “Middle Tier Acquisition,” “Agile Acquisition Framework” — the concepts behind them are nothing new at all. The new buzzwords are just a different way of concealing the fact that the department and the defense industry are yet again trying to lock in funding for a potentially problem-ridden weapon system.

The F-15EX program, as is the case for many currently troubled major acquisitions, is exploiting service-alleged urgency to begin production before the weapon is operationally tested or before the design and development are even completed. Secretary of the Air Force Frank Kendall, who once called the F-35 “acquisition malpractice,” could back up his earlier tough words by insisting that the F-15EX adhere to best practices and begin operational testing only after fully developed, combat-ready test planes are in hand. Instead, his office now seeks to double down on the concurrency “acquisition malpractice” that is bedeviling the F-35 and many other of today’s troubled acquisitions. The Air Force’s push to combine F-15EX developmental testing and operational testing further exacerbates the problems created by the concurrency on those programs.

Fortunately, the office of the Director of Operational Test and Evaluation (DOT&E) has used its independent authorities under federal law and Congress’s statutory mandate to include the F-15EX program in its portfolio for adequate operational testing. According to sources within the testing community, DOT&E is attempting to work with the F-15EX program office to ensure a sufficiently combat-realistic operational test plan. Unfortunately, DOT&E will probably not succeed if other department officials permit the Air Force to continue combining and overlapping developmental and operational testing. Maintaining the independence of operational testing will be an early test for the new DOT&E nominee, Nickolas Guertin, should he be confirmed.

The F-15EX program office is pushing to conduct operational testing before the new, highly complex defensive and offensive systems are fully integrated into test planes that are deficiency-free and production-representative. Fully developed planes with all components are absolutely essential for rigorous, combat-realistic testing that will produce a reliable data-based analysis and report to Congress. If the Air Force’s concurrent and truncated testing schedule — which would prevent meaningful operational testing — is approved, any decision to enter full production will be based on the performance of an underdeveloped prototype rather than what the troops should expect to see in the field.

Independent analysis has shown that industry and service complaints about operational testing have been largely meritless. In June 2015, at the behest of the House Armed Services Committee, the Government Accountability Office (GAO) investigated whether independent operational testing had resulted in significant cost and schedule increases. The answer was a clear no. Recognizing the inherent incompatibility between the developer’s goals (keep the money flowing and ramp up production as soon as possible) and the user’s operational test goals (determine combat suitability), the GAO found that less than 10% of cases resulted in significant disputes over operational test realism and schedules. In nearly every case the disputes were resolved and “only a few had considerable cost or schedule impacts.”

Operational testing supports the acquisition program’s ostensible purpose of delivering a weapon with superior military effectiveness. An independent operational evaluation report tells the users, the secretary of defense, and Congress if the weapons the Pentagon wants to buy will actually work in the hands of the troops. Combining the development and the operational tests inevitably leads to inadequate testing of incomplete weapon systems — thus delivering incomplete answers to the crucial question, will the weapon work in combat?