Bad Watchdog Season 2 launches June 20.

Championing Responsible National Security Policy
|
Analysis

F-35 Program Stagnated in 2021 but DOD Testing Office Hiding Full Extent of Problem

The F-35 Joint Strike Fighter program appears to be in a state of suspended development, with little progress made in 2021 toward improving its lackluster performance. The latest report by the Pentagon’s Director, Operational Test & Evaluation (DOT&E) reveals stagnation and even backsliding in some fleet reliability measures.

And that’s just the public DOT&E report.

In an unprecedented move, DOT&E is concealing many of the key details of the F-35’s poor performance. For the first time ever, the testing office created a non-public “controlled unclassified information” version of its report, and although there is much overlap between the two versions, the meaningful details about the ever-troubled program are only included in the non-public one.

One thing to note about so-called controlled unclassified information: It is not classified. The label is a tool that some in the federal government misuse to conceal information that could be embarrassing to them, but because the information does not damage national security, they can’t hide it under a classification label. The Project On Government Oversight obtained a copy of the non-public report, and what it clearly shows is that the F-35 program has made few fixes to many of the reliability and performance problems that have prevented the aircraft from meeting the needs of the services. This is information the public must have in order to pressure policymakers to correct the problems that, if uncorrected, could harm U.S. service members and the U.S. national defense mission.

In an unprecedented move, DOT&E is concealing many of the key details of the F-35’s poor performance.

Despite more than 20 years and approximately $62.5 billion spent so far on research and development alone, program officials still haven’t been able to deliver an aircraft that can fly as often as needed or to demonstrate its ability to perform in combat, which places military personnel in jeopardy. Most of the important details can only be found in the non-public version of the report, but some key findings are available in the public version:

  • The F-35’s availability rates “plateaued” over most of 2021 and then declined in the final months of the year.
  • Program leaders abandoned the efforts to complete the troubled Autonomic Logistics Information System (ALIS) and instead decided to build a new network called Operational Data Integrated Network (ODIN). The new system, meant to anticipate maintenance problems and track parts and repair processes, runs faster and is more deployable than ALIS but is already behind schedule and has some of the same cyber vulnerabilities.
  • The Joint Simulation Environment, meant to be a high-fidelity and fully validated and verified simulator to test the F-35’s high-end capabilities, is now more than four years behind schedule. A full-production decision can’t be made until the planned 64 tests in the simulation can be completed.
  • The F-35 program’s modernization effort, an effort to complete the delivery of capabilities that should have been included under the original development contract, is behind schedule and has done little to reduce the high number of unresolved design flaws.

The Department of Defense awarded Lockheed Martin with the coveted Joint Strike Fighter development contract on October 26, 2001. The high expectations for an affordable futuristic fighter jet quickly crashed against reality. Despite the fact that development costs have more than doubled and delays have set the F-35 back by nearly a decade, the program has yet to deliver a fully developed aircraft.

Underwhelming Fleet Performance

While the F-35 program experienced a few marginal improvements in some reliability categories, the overall trend shows that the fleet’s performance remains below the Defense Department’s standards and is even getting worse in some categories.

According to the non-public version of the testing report, F-35 aircraft availability rates “plateaued” in 2021 and then declined starting in June. The services set an availability rate goal of 65%. For an aircraft fleet, 65% is a low bar since 75% to 80% is the accepted standard for other programs. The fleet-wide average availability rate for the F-35 was 61%. The services calculate an aircraft fleet’s availability rate as the percentage of aircraft that are mission capable and in the possession of its operating squadron, not in depot-level maintenance.

The testing office made no mention at all about the program’s mission capable rates or the even more relevant performance metric for a multi-role aircraft like the F-35, full mission capable rates. The mission capable rate is calculated by each military unit and is the percentage of aircraft ready to perform at least one of its assigned missions. An aircraft just able to take off is counted as being mission capable whether it can perform any of its actual combat roles or not. The services prefer to cite this measurement over full mission capable rates because the former is a much lower standard and easier to meet. A fully mission capable aircraft is one that is able to perform all of its assigned missions.

That information is vital to determine how effective the program truly is. Luckily, other federal agencies provided some of that information. The Government Accountability Office included them in a July 2021 report. The GAO found that the entire F-35 fleet averaged a full mission capable rate of 39% in 2020, which was an improvement from the 32% the year before. The Air Force’s F-35A variant performed the best with a fleet average of 54% that year, a rate of performance that is still far below the 80% mission capable rate needed for an effective aircraft fleet (and even significantly below the program’s low 65% availability standard). The Marine Corps’ fleet of short takeoff and landing F-35Bs and the Navy’s fleet of F-35Cs, which are tailored for use on aircraft carriers, lag far behind. The F-35B fleet’s full mission capable rate got worse between 2019 and 2020, dropping from 23% to 15%. The F-35C fleet showed some improvement during that period, but that is not saying much. That fleet’s rate went from 6.4% to 6.8%.

The Congressional Budget Office also provided useful information the testing office omitted from its unclassified reports this year. In January 2022, it released a report specifically about availability rates of aircraft in the Air Force and the Navy. The report concluded that the aircraft fleets of the Air Force, Navy, and Marine Corps (part of the Department of the Navy), declined across the board but that “the decline was more marked” in the Navy.

Amazingly, the Congressional Budget Office reported that the Air Force’s own system for tracking availability figures, the Reliability and Maintainability Information System or REMIS, “does not accurately track availability or flying hours” for the conventional take off variant F-35A. The office went on to report that the data for the Marine Corps’ F-35B and the Navy’s F-35C “did not match other reports of the availability of those aircraft.” For these reasons, the Congressional Budget Office simply excluded the F-35 program, currently the highest profile aircraft program, from its analysis.

The non-public DOT&E report explains that the F-35 fleet’s availability rates increased temporarily because program officials surged spare parts to some units, and because the newest aircraft delivered from the factory had the effect of reducing the percentage of aircraft being pulled from service to send to the maintenance depots for modifications or major repairs. What that means is the F-35 did not suddenly become a more reliable aircraft in 2021. It means that it takes extraordinary effort to keep the fleet operating even close to the required levels and suggests that those availability rates are not sustainable long-term.

The availability of spare parts has long been a serious issue with the F-35 program, resulting in aircraft sitting idle on tarmacs as maintenance crews waited for spares to be delivered and driving down readiness rates. DOT&E reports that it continued to be a problem in 2021 because the systemic issues have not been addressed. According to the non-public report, the “lack of spares inventory, and limited component-level depot repair capacity, contribute to the shortfalls in spares supply.” First, the services are buying new F-35s so fast that the manufacturers of replacement parts can’t keep up with demand. Second, the entire F-35 enterprise relies on a complicated global spares pool scheme that makes F-35 operators around the world, including the United States, compete against each other for replacement parts.

The competition for replacement engines is perhaps the program’s most acute challenge today. According to DOT&E, by the end of the week of September 27, 2021, 52 F-35s sat idle because they didn’t have functioning engines. The problem is particularly prevalent in the Air Force since “almost all aircraft requiring an engine are F-35A variants.”

Program leaders also expected the Pratt & Whitney F135 engine used in the F-35 to be more reliable than they have proven to be. Service officials based plans on the promised performance rather than on the demonstrated performance and so underestimated the maintenance depot capacity they were going to need to repair broken engines. The unanticipated volume of work has overwhelmed the maintenance crews, and the backlog has grounded much of the fleet, including many aircraft being used in operational units. “Although the program and the Services manage engine spares by prioritizing combat-coded units over test and training units,” the report said, “the shortage of spare engines has adversely affected deployed combat units as well.”

A fighter jet program that can’t get off the ground is of limited value.

It’s not just how frequently the F-35 breaks down that impacts the overall fleet performance but also how long it takes to fix the problems. Maintenance crews found that on average it takes more than twice as long as the program’s contract specifications to complete needed repairs. The F-35’s engine assembly, canopy, and stealth coating are a few of the components listed as the top drivers of the longer-than-planned repairs.

The world recently saw evidence of how fragile the F-35’s stealth coating is and the challenges of maintaining it properly. Photos released by the Pentagon’s own in-house multimedia outlet, Defense Visual Information Distribution Service, show some of the F-35Cs aboard the USS Carl Vinson covered in what appears to be rust while on deployment in the Pacific.

A fighter jet program that can’t get off the ground is of limited value. This is obviously true in times of war, but it’s also equally true during peacetime. Pilots need to be able to fly the aircraft they would take into combat to get enough flight hours to develop and increase their skills. Without enough flight hours, a pilot’s skills degrade. As one former fighter pilot wrote, when a pilot doesn’t get enough experience in the air, “the likelihood they will be effective or even survive in combat drops precipitously.”

Cyber Threats and a Scrapped Logistics and Maintenance Network

F-35 boosters enjoy telling people that the program is more than just a fighter jet. “A computer that happens to fly,” “The Most Advanced Node in the 21st Century Warfare,” and “The Most Lethal, Survivable, And Connected Fighter Jet Ever Built” are just a few of the hyperbolic marketing slogans used to sell and protect the F-35 program. While these slogans may look good on a brochure or splashed across the pages of trade publications, the connected nature of the F-35 aircraft and support networks on the ground may ultimately prove to be the program’s biggest conceptual flaw. The entire F-35 enterprise remains dangerously vulnerable to cyberattacks despite years of warnings.

Cybersecurity testing completed in 2021 revealed a number of “vulnerabilities that must be addressed to ensure secure ALIS, training systems, United States Reprogramming Lab (USRL), and air vehicle (AV) operations.” The testing office is effectively reporting that there is no aspect of the F-35 program protected against a cyberattack.

The F-35 actually has plenty of company in this regard. In the cover letter accompanying the annual report, Operational Test & Evaluation Director Nikolas Guertin said that less than 10% of the 81 weapon programs evaluated in 2021 were sufficiently hardened against a cyberattack. “[Test & Evaluation] continued to show that DOD networks and systems supporting critical missions are not secure,” he wrote. “Improving DOD’s cybersecurity posture will require network defenders, system users, and mission commanders to be equipped with innovative tools and training to successfully detect and rapidly respond to nation-state cyber-attacks.”

F-35 boosters worked diligently to reassure Congress and the American people that the program’s all-encompassing maintenance and logistics network, the Autonomic Logistics Information System (ALIS) would work as advertised and transform the way the services would perform maintenance. The $16.7 billion ALIS network was supposed to be able to use the aircraft’s embedded diagnostics functions to detect and even anticipate maintenance problems. When the system identified a problem, the replacement parts needed would automatically be ordered and then tracked through the supply chain. Once the parts arrived, ALIS would then guide maintenance crews through the steps to correct the problem. Lockheed Martin says ALIS “turns data from many sources into actionable information, enabling pilots, maintainers and military leaders to make proactive decisions to keep jets flying.”

The government surrendered a great deal of control of the largest weapon program in history — leaving no choice but to rely on Lockheed Martin.

At least that was what ALIS was supposed to do.

What it really did was create more work for pilots and maintenance crews. ALIS consistently reported breakdowns that weren’t, prompting maintainers to spend time investigating alerts that didn’t exist. The reporting system often produced bad data that required administrators to spend time on tedious workarounds.

Lockheed Martin not only designed and built ALIS, but also controlled and operated it under a concept known as Total System Performance Responsibility. From the very beginning of the F-35 program, the government surrendered a great deal of control of the largest weapon program in history because officials at the time believed that the contractor could more efficiently provide support. The company retained the data rights and intellectual property necessary to effectively sustain the F-35 program throughout its lifespan. The government could not competitively bid the lucrative sustainment contracts because no other firm could access the data they would need to do the job. The services have no choice but to rely on Lockheed Martin as a result.

The problems with ALIS became so acute that Pentagon leaders decided in 2020 to scrap the entire project and replace it with a new cloud-based system called the Operational Data Integrated Network, or ODIN.

After Lockheed Martin failed to deliver effective maintenance and logistics with ALIS, the company is now being paid again to build ODIN.

At this point in the whole F-35 saga, it’s doubtful anyone would be surprised to know that after Lockheed Martin failed to deliver effective maintenance and logistics with ALIS, the company is now being paid again to build ODIN. Program officials had little choice but to award the contract to Lockheed because the company controls the data necessary for the job. And true to form, the project is already behind schedule. At the time of the ODIN announcement, Pentagon officials claimed the new system would not add any additional costs and would be fielded by 2022. ODIN development has “stagnated” due to funding constraints and the need to husband ALIS along as an interim solution through the transition. DOT&E reports that questions remain about ODIN’s ability to withstand cyberattacks despite the recognized need for all new information systems to be developed with cybersecurity as a fundamental design feature.

DOT&E does offer some good news about ODIN. The Base Kit, the hardware that is set up wherever the F-35 operates, runs faster and is much smaller than the equivalent ALIS standard operating unit. The ALIS version weighed 891 pounds. The ODIN Base Kit weighs approximately 202 pounds, which does make the system more deployable. Beyond that, little information about the effectiveness of ODIN has been reported to date.

Unless or until Lockheed Martin and the F-35 Joint Program Office can produce a working support network, the F-35 will not be an effective aircraft program. Joint Program Office officials claim the F-35 can operate for 30 days without connecting to ALIS. It must be noted that DOT&E does not report that claim as an objective fact. Testing office officials have said for years that F-35 program leaders need to ensure the aircraft can operate independently of ALIS — and now ODIN — in the event the system is compromised or unavailable and called for the program officials to conduct tests to prove it. DOT&E reports that the “ALIS Contingency Operations Plan” test was to happen in late 2021 or early 2022. If such a test happened, the results have not been made public, even in the non-public version of the most recent report.

For now, the biggest flaw of the F-35 program may not be a vulnerability to an enemy weapon, but the program’s own information architecture.

Unfinished Testing Simulator

The major reason the F-35 program remained stagnant in 2021 is that the designers and programmers of a key simulation facility were unable to complete their work. The program office needs a specialized simulation facility to fully test the F-35’s ability to fight and survive in the heavily defended airspace a sophisticated adversary like Russia or China would create. The original designers of the F-35 had just such an extreme scenario in mind when they drafted the initial requirement documents, and it is the ability to persevere in this type of extreme scenario for which the American people are paying the premium price of the F-35 program.

For the past seven years, a team led by the Navy has labored to build the facility, called the Joint Simulation Environment (JSE), at Maryland’s Naval Air Station Patuxent River. DOT&E reports that the facility is now more than six years behind schedule and that “significant” work remains before F-35 testing can be completed.

The F-35’s Initial Operational Test & Evaluation master plan calls for 64 test scenarios to be “flown” by pilots in the simulator. The tests include 11 defensive counter air missions to test the F-35’s ability to defeat enemy aircraft attacking friendly forces or facilities, 22 cruise missile defense missions, and 31 missions testing the F-35’s capabilities to penetrate enemy airspace to shoot down enemy aircraft, destroy ground targets, and defeat air defense assets.

Engineers finished building the Joint Simulation Environment facility years ago. The long delays now are because designers haven’t been able to complete the simulation software. The remaining operational tests require the simulation to accurately reproduce the real-world performance of all the F-35’s mission systems and how they respond when they detect signals from the myriad potential simulated radar systems, aircraft, and anything else the real F-35s would encounter on a mission. DOT&E once described the Joint Simulation Environment as “the only venue available, other than actual combat against near-peer adversaries, to adequately evaluate the F-35 due to inherent limitations associated with open-air testing.”

Pentagon leaders cannot even provide an estimated completion date as they’ve done in the past.

The simulation software programmers have so far not been able to finish their work. Pentagon leaders cannot even provide an estimated completion date as they’ve done in the past. DOT&E can only report that “significant work remains to complete the necessary verification and validation process, which compares JSE component and system-level performance to F-35 flight test data to accredit the JSE for operational test trials.”

The Joint Simulation Environment has fallen so far behind schedule that program leaders brought in outside experts to review the project. Teams from Johns Hopkins Applied Physics Laboratory, Carnegie Mellon University Software Engineering Institute, and the Georgia Tech Research Institute evaluated the facility. The teams completed their work in May 2021 and concluded that the simulator effort needed more funding and personnel “along with strong support from all stakeholders to support [Initial Operational Test & Evaluation] requirements” to finish the needed work.

The simulated tests will only effectively show how well the F-35 will work in combat if the realism of the simulations is properly validated. Designers validate the simulator by taking data gathered during real F-35 flights over test ranges. Each test aircraft is equipped with instruments that register how the onboard mission systems react when triggered by a radar signal or other threat. Software engineers must program the simulations to accurately reflect how the F-35’s mission systems react to the real stimulus.

The Joint Simulation Environment is important even beyond the program’s operational testing phase. Last year’s testing report says the facility “will be an invaluable resource” for both high-end training and tactics development.

Sources involved in the process tell POGO that they doubt the Joint Simulation Environment will ever work the way the program’s leaders promised. If the facility can’t work properly, the formal operational testing plan will either have to be amended or cut off prematurely. Per federal law, a major defense acquisition program cannot legally enter into full-rate production without completing its initial operational test and evaluation plan. The testing director has the authority to scrap the approved testing plan and declare that his office has enough data to make an assessment of the program’s combat effectiveness, which would clear the path for a final production decision. If Guertin were to do that in this case without the simulated tests, the first time the F-35’s full capabilities would be tested could potentially occur over the skies of an enemy’s capital.

Uncompleted Development Masquerading as Modernization

If and when the F-35 program does clear all the hurdles to move into the production stage, the F-35 will still require extensive development to finish all the work that should have been completed long before making it to that point. “Significant operational deficiencies (classified) were identified by the operational test units and field units in CY2020 [calendar year 2020] that required software modifications,” the non-public report stated. As a result, the program “continues to field immature, deficient, and insufficiently tested mission systems software to fielded units without adequate operational testing.”

Because the Joint Program Office and Lockheed Martin were unable to meet all the requirements for a fully functional aircraft within the time and budget of the first, second, or third program baselines, officials decided to simply rename the remaining work. The ongoing “modernization” effort is, in reality, an F-35 initial development do-over.

The F-35 program still has 845 unresolved deficiencies — with six still classified as serious design flaws.

Engineers have their work cut out for them. DOT&E reports that the F-35 program still has 845 unresolved deficiencies with six still classified as Category I, or design flaws so serious that it “may cause death, severe injury, or severe occupational illness; may cause loss or major damage to a weapon system; critically restricts the combat readiness capabilities of the using organization; or result in a production line stoppage.”

The 845 reported design flaws in this year’s report seem like a slight improvement over the 871 identified in last year’s report. But without knowing what the design flaws were or how they were resolved, it is difficult to gauge if real progress was made. Program officials hold Deficiency Review Board meetings where the members will determine what, if any, steps will be taken to correct design flaws. The minutes of one such meeting held in 2018 showed that the board downgraded a Category I deficiency to a Category II with “no plan to correct.”

Neither the program office nor Lockheed Martin appear to be making the most of their second chance. “The program has not sufficiently funded the developmental test (DT) teams to adequately test, analyze data, or perform comprehensive regression testing to assure that unintentional deficiencies are not embedded in the software prior to delivery,” DOT&E reports.

Program leaders promised to deliver regular design updates in six-month increments when they announced the “modernization,” or Continuous Capability and Development Delivery (C2D2), plan. That scheme quickly fell apart as designers and the testing teams could not keep up with such an aggressive schedule. DOT&E now reports that the program leaders have given up on the six-month updates plan and are now operating on a one-year timeline for updates.

Along with the delays come extra costs. Program officials initially said it would cost $10.8 billion to complete the additional development work plus an additional $5.4 billion to upgrade F-35s purchased in the years before engineers completed their design work. The Government Accountability Office reported in March 2021 that costs for the new development work have risen to $14 billion, meaning that when the costs of retrofitting the fleet are added in, the “modernization” effort will cost nearly $20 billion — a third as much as has been spent so far on development alone.

Undue Secrecy

While the program’s lack of progress that was revealed in DOT&E’s non-public testing report was bad enough, controversy surrounded the DOT&E itself because what is supposed to be an independent testing office caved to pressure from the service regarding publicly releasable information.

Federal law mandates the testing office write an annual report about the weapon programs on its oversight list. The law states that if the testing office submits a classified report to Congress, they must submit an unclassified version as well. DOT&E produced three versions of the 2021 report: a classified version, a publicly released unclassified version, and a newer third version stamped Controlled Unclassified Information. This last version contained the information previously included in earlier unclassified publicly released reports. Although the publicly available version of this year’s report provided an overall assessment of the F-35 program, the authors stripped it of many figures and details that gave a full picture of how the program performed in 2021.

The difference in the level of detail in each report is obvious even in just the section heading for the F-35. And it doesn’t get better from there. For instance, the public version states that the program identified deficiencies requiring “software modifications and additional time and resources.” The non-public version states that the program had deficiencies in “weapons, fusion, communication and navigation, cybersecurity, and target processes” requiring software modifications and additional time and resources.

The public version of the report (top), states the F-35 program identified deficiencies requiring “software modifications and additional time and resources.” The non-public version (bottom) details these deficiencies. (Emphasis added for clarity.)

The DOT&E’s reports in earlier years were full of charts with data about the fleet’s readiness rates and maintenance data. The public version of this year’s report contained only two charts, while the non-public version had eight. One of the tables omitted from the publicly released version detailed the fleet’s availability rates at each F-35 base. The tables DOT&E did include in the public version of the report were also of less value because they were presented with vague labels and without detailed explanations. The public version included a chart titled “F-35 Reliability Metrics.”

The chart includes several acronyms such as “MFHBCF” but does not include a key to explain what the acronym means. Someone trying to make sense of it would have to look to another source to find that it stands for “Mean Flight Hours Between Critical Failures.” The entry for that column in the public version of the chart provides the contract standard the fleet is supposed to achieve but doesn’t provide the actual observed performance. Instead, it only includes arrows showing the up or down trend for each of the three F-35 variants and a “yes” or “no” indicating whether that part of the overall fleet is meeting the standard.

The non-public version of the report broke out the reliability data into six separate charts. Those charts helpfully spelled out the acronyms and provided actual data rather than opaque trend arrows. It’s only in these charts that the reader learns that, for example, the Air Force’s F-35A fleet is not only missing its goal of 20 hours standard for Mean Flight Hours Between Critical Failures but also that the demonstrated performance is only 11.2 hours. Where the public version only showed that the fleet’s trend in this measure is worsening, the more detailed chart shows that the time between critical failures shortened, dropping from 16.8 hours in 2020.

Although Raymond O’Toole, who was the acting testing director when the reports were released, defended the creation of a non-public version of the unclassified report by saying “I thought it very important to provide Congress and the Secretary the test evaluation details that shouldn’t wind up in our adversaries’ hands, hence the new CUI version of the annual report,” this is disingenuous at best.

Certainly there is information that must be classified. But the details DOT&E stripped from this year’s public version of its report are not the sort that would provide an adversary a technological advantage.

Operational testing, when done properly, will find design flaws — like poor engine reliability and the fact that the F-35A’s gun doesn’t work properly — so they can be fixed before an enemy can take advantage of them.

Certainly there is information that must be classified, such as data about the F-35’s radar cross section, specific software deficiencies, and any number of other deficiencies that could be exploited. But the details DOT&E stripped from this year’s public version of its report are not the sort that would provide a potential adversary a technological advantage. Instead, it is the kind of information that, although embarrassing to the Pentagon and its contractors, the public should know in order to pressure F-35 stakeholders to take the steps necessary to correct the problems. It’s the information all public DOT&E reports have provided until now.

“The office was set up to be independent of the services,” said Tom Christie, DOT&E during George W. Bush’s first term. “The testing director has the authority to ignore the service’s directives because he answers directly to Congress.”

Members of Congress have expressed their concern about the excessive secrecy surrounding the testing results through multiple letters sent to Pentagon leaders. Congress needs to keep up the pressure to protect the testing office’s independence by making sure no future unclassified reports are stamped with any kind of phony information designation.

Conclusion

More than twenty years into the F-35’s development, the aircraft remains in every practical and legal sense nothing more than a very expensive prototype. The simple fact that the contractors and the program office haven’t been able to deliver an aircraft whose effectiveness has been proven through a full operational testing program suggests the original Joint Strike Fighter concept was flawed and beyond any practical technological reality. With little progress and significant regression in 2021, it seems that the F-35 program will remain in its current stagnant state for the foreseeable future.

Congress held firm during the fiscal year 2022 budget debates by refusing to authorize additional new F-35s beyond the Pentagon’s request. This was the first time since F-35 low-rate initial production began in 2007 that Congress didn’t increase the yearly buy. It appears that the official patience with what has been the slowly unfolding disaster that is the F-35 program is beginning to wane. Congress should continue to hold firm by limiting further F-35 purchases until program officials can complete a design that can be proven effective and suitable for service through the operational test process.