Mr. Trump, Don’t Forget to Nominate a Tough Operational Testing Director

President-elect Trump is now looking for more than 4,000 individuals to fill all the appointed positions in the executive branch. Parlor speculation about his potential nominees began months ago, and there will be plenty of news coverage about his appointments in the next few weeks as the transition team begins announcing nominees. Nominations for Secretaries of State and Defense and for Attorney General are sure to receive a great deal of attention. But often overlooked are the positions that seem minor but in fact are key to effective oversight.

One such position is of enormous consequence for military service members, as well as taxpayers, and will impact our military capabilities for decades to come: the Department of Defense’s Director of Operational Test and Evaluation (DOT&E). DOT&E is charged with ensuring that the major weapons we buy will be effective and suitable (i.e. reliable) in the hands of our troops in combat. The Government Accountability Office (GAO) estimates the Pentagon’s current weapon portfolio will cost taxpayers $1.44 trillion—and the estimates are based on the Pentagon’s deliberately optimistic projections. The costs will likely grow, as we have seen time and again, as these systems are developed and the familiar pattern of cost and schedule overruns is repeated. But the operational testing process, when led properly, can head off some of these problems by identifying early in the process the systems that aren’t living up to expectations.

The current DOT&E, Dr. Michael Gilmore, has proven himself to be a highly principled and dedicated public servant. But with the impending change of administration, there is no guarantee he will continue in his presidentially appointed, Senate-approved position.

As the Project On Government Oversight (then known as the Project on Military Procurement) reported in the 1980s, weapons systems like the Army’s M-1 tank and Air Force’s AWACS airborne radar plane were not being realistically challenged in tests. The results, even of compromised tests, were also not being accurately reported. All systems were being tested by the services, with the tests and results being supervised by weapon system advocates in the military services and by their supporters in the office of the Secretary of Defense. The fear of the services and their advocates was that realistically testing these systems and reporting those test results truthfully could jeopardize production.

“The Pentagon is like the student grading its own exam paper in these operational test situations, and we are going to have to end that,” former Senator David Pryor (D-AR), the primary author of the DOT&E bill, said. A GAO report, a deluge of press articles, extensive testimony to Congress, and even a Blue Ribbon Presidential Commission found that most weapons were being fielded before they had “fully demonstrated their capabilities under representative combat conditions.'' Senators Pryor, Bill Roth (R-DE), Nancy Kassebaum (R-KS), and Mark Andrews (R-ND), with very active support from POGO, worked to create a testing office outside of the military service and DoD development chain of command.

An effective operational test program can prevent the military from making large and costly mistakes. A good example of this is the Army Helicopter Improvement Program (AHIP). In the early 1980s, the Army asked the defense industry to design an advanced helicopter to be used for attack, scouting, and artillery spotting missions. Bell Helicopter won the competition and produced an upgraded version of its earlier OH-58 Kiowa helicopter. Following several years of development, the production representative models began operational testing in September 1984.

During testing, the helicopter crews struggled to identify targets, a serious problem for a scout helicopter. The pilots flying the missions were familiar with the terrain of the test range and knew where the target arrays were, but they still had difficulty identifying these arrays using the helicopter’s forward looking infrared receiver. This and other demonstrated limitations prompted DOT&E to report that the OH-58 was operationally effective, but only as an artillery spotting platform. Plans to use the helicopter in the scouting and attack roles were scrapped. The Army cut its originally planned fleet of new OH-58Ds from 578 to 179, saving taxpayers nearly $3 billion.

More recently, POGO reported on another example that benefited immensely from DOT&E testing:

DOT&E testing on the Marine Corps’ Enhanced Combat Helmet found that “inward deformation of the helmet shell during testing presented a serious risk of injury or death.” The Marine Corps resisted these testing procedures, which had been revised with input from the Armed Forces Medical Examiner, because it “forced changes to test procedures it had intended to use for the Enhanced Combat Helmet” and prevented the expedited production initially planned. But once the program passed the new testing requirements, the Marine Corps “found the Enhanced Combat Helmet preferable to the existing lightweight helmet and proceeded to full-rate production and fielding.”

Congress created DOT&E in 1983 because Members knew they weren’t getting the kind of oversight that was needed. Congress and the Secretary of Defense were not receiving accurate information about the real-world performance of the systems they were buying. Up to 1983, the information Congress received was biased. It had been filtered through the very bureaucracy that had a stake—financial and otherwise--in making sure weapon systems proceeded into full production and deployment, regardless of whether or not they would perform effectively in combat.

Congress’s decision to establish the DOT&E was a step toward getting unbiased information, but it only went so far. Ensuring the office is led by an effective leader is what will guarantee that the information is unbiased. There have been a few directors who were truly independent and who provided quality oversight, including the most recent director, Dr. Gilmore. Whether Congress will continue to get independent evaluations will depend on the next person selected to occupy the position.

The Ideal Operational Testing Director

For this process to work correctly, DOT&E needs a strong leader.

Any prospective testing director must first have a strong background in operational (i.e. combat-related) defense issues. He or she must understand how military forces are organized and how they operate in combat to evaluate how new weapons will work in the field.

A strong background in science and engineering is desirable, but not required. The weapons testing process produces reams of technical data which the director must not only understand, but also be able to explain to both Congress and the Secretary. The people underneath the Director are almost as important. They should be objective experts without conflicting financial or career-oriented interests.

The director must also have upstanding moral character. Because of the high national security and fiscal stakes involved in the weapons buying process, the DOT&E can quickly accumulate enemies by delivering bad news that might imperil a system’s continued existence. Attacks on the person occupying that office can come from defense contractors who stand to lose money, from lawmakers that stand to lose jobs and votes, and from defense program advocates who stand to lose their credibility and future job prospects. It is one thing for a DOT&E candidate to say he or she will stand up to such advocacy, it is quite another to demonstrate such character through past experience in dealing with political or corporate bureaucracies.

“He has to be ready to stand his ground,” said Tom Christie, who served as DOT&E from 2001 to 2005 after a four-decade career in the Office of the Secretary of Defense (OSD) bureaucracy. Before assuming the office, he repeatedly demonstrated his willingness to challenge service, corporate, and congressional program advocacy. He warned that any potential nominee for this position must be braced to “be vilified and be ready to let that run off like water off a duck’s back.”

DOT&E nominees must be fully vetted to ensure there are no conflicts of interest. At the beginning of his administration, President Obama issued an Executive Order to restore integrity and ethics to government, but famously issued the first waiver from such requirements to nominate former Raytheon lobbyist William Lynn as Deputy Secretary of Defense. (Lynn promptly became chairman and CEO of defense contractor DRS Technologies upon leaving OSD.)

President-elect Donald Trump recently proposed expanding ethics reforms further by stopping the revolving door by instituting a five-year ban on government officials joining lobbying firms, but it remains to be seen whether he will keep Obama’s Executive Order in place. POGO joined a coalition of good government organizations urging Trump (and Democratic candidate Secretary Hillary Clinton) to expand ethics reforms. Integrity is paramount for any DOT&E, and their financial disclosures must be scrutinized to ensure the candidate doesn’t have any vested interests in companies seeking or holding government contracts.

Their employment histories should weigh heavily in the selection process. Prior military service may appear to be an advantage, but parochial loyalties and personal relationships with people still in uniform could potentially lead to biases while assessing programs.

Past work with defense contractors need not disqualify a nominee, but a record of repeatedly traveling through the revolving door between the government and the defense industry should. Contractors have been known to place their own people in influential positions within the government, even going so far as to reward employees with stock bonuses upon leaving their jobs for a government post.

It is critical that whoever occupies this office is working on behalf of not just the taxpayers, but the men and women whose lives will depend on the weapons they take into battle. “When I was DOT&E, I was trying to get not just my office, but the service’s operational test agency, to understand that your customer is not just the acquisition community,” said Christie. “Your customer is the troops in the field in combat.” A demonstrated record for standing up for the troops should be considered the criteria sine qua non for the DOT&E job.

Phil Coyle, DOT&E during the 1990s, said it is also helpful for future directors to have a good understanding of the Pentagon’s acquisition culture. He said knowledge of the

“games” program officials play during the process is important because managers will try to exploit an outsider’s unfamiliarity with the complex weapons buying process to avoid or delay testing. This they do by claiming procedural impediments to testing where none exist.

“An important qualification for the next Director is to have understanding of, and experience with, the testing and evaluation of complex systems,” said Coyle. “But most of all, the dynamic range of DOT&E responsibilities is enormous, and the Director needs the intelligence, toughness, and leadership to handle them all."

The Role of Operational Testing

All new weapons systems are supposed to go through two separate test and evaluation processes before they are handed over to our troops in the field. The first is developmental testing, which occurs when components of the overall system are still being engineered. When these components don’t work according to their own specifications, engineers identify and implement fixes and then test again to ensure the problem has been resolved. This test-fix-test process is designed to work out as many kinks in the system as possible prior to moving to the next step. When the process works correctly, a full production model of the system is produced and the entire system is then handed over to the operational testers and DOT&E.

The services each have their own operational testing branches that conduct the actual tests. DOT&E’s role is to review and approve the testing plans and then evaluate the results. This is done to provide an independent assessment of whether a new weapon is useable, reliable, and capable of performing effectively in combat. For example, DOT&E will make sure a new rifle can not only hit a target on the range, but can also prove lethal and reliable against combat-realistic targets and be easily cleaned and maintained in the field.

Operational suitability is a vitally important concept because it is what generally determines the real-world effectiveness of any weapon system. Working in a controlled laboratory environment is helpful; operating in all the mud and chaos of combat is essential. The Pentagon’s acquisition office acknowledged this in its latest annual report, saying that operational tests “show that major programs are often effective when they tested as operationally suitable, but the converse is not true.” A soldier in direct combat with the enemy will find little comfort in the knowledge that his rifle worked perfectly in the antiseptic conditions of a laboratory if it jams the instant it’s exposed to moisture and dirt on the battlefield.

After the tests are completed, DOT&E evaluates the results and writes a report about the weapons suitability that goes directly to the Secretary of Defense and to Congress. These reports are supposed to include forthright analyses and reporting of problems encountered during the testing process. By law, a system can’t proceed to full production until DOT&E submits a report stating “the results of such test and evaluation confirm that the items or components actually tested are effective and suitable for combat and the test itself was deemed adequate to draw such conclusions.”

DOT&E evaluation before production is supposed to prevent “buy before fly.” Sadly, the Pentagon has redefined “low rate initial production” in such a way as to allow hundreds of systems to be produced without developmental or combat testing, like in the case of the F-35 Joint Strike Fighter.

Defending the Authority and Integrity of Operational Testing

The defense industry and status quo advocates in the Pentagon fiercely opposed the creation of DOT&E, and they can be expected to continue their efforts to undermine it. They fear honest, realistic operational testing evaluated by an independent office because this process can expose systems that aren’t performing as advertised and are not ready for production. The GAO highlighted this in a 2015 report:

“Postponing difficult tests or limiting open communication about test results can help a program avoid unwanted scrutiny because tests against criteria can reveal shortfalls, which may call into question whether a program should proceed as planned.”

Because of this, there have been many efforts over the years to undermine the office. Recently, there have been attempts to include language in the defense authorization bill to require DOT&E to “consider” cost increases and schedule delays when creating test plans. Industry advocates try to blame increases and delays on testing, when the true cause is compromised tests held late in the process, often after full-rate production, that require extensive reworking, retesting and even reproduction. Known as “concurrency,” this slows programs, increases costs, and results in fielding unworkable systems. A study by the Congressional Budget Office (CBO) confirmed this contributes to weapon system cost growth, and in highly concurrent programs contributed to schedule delays. Concurrency has been specifically cited as causing cost increases and schedule delays for the C-17 transport aircraft, the Littoral Combat Ship, and the F-35 Joint Strike Fighter, among many others. The GAO recently evaluated 43 programs and found that “for the programs we assessed, as concurrency increases, so does total acquisition cost growth.”

Undermining Operational Testing Through Infiltration

Current efforts on Capitol Hill and by industry show that it isn’t necessary to cut the legs out from under the testing office through legislation to get what system advocates want. All they need to do is have one of their own installed as the director. Even if they’re not a stooge for the industry, anyone who is averse to the bureaucratic infighting that is often necessary to make sure problems are identified and fixed will seriously undermine the effectiveness of our weapon systems.

Beyond simply avoiding ugly bureaucratic battles, another way a director working on behalf of system advocates typically undermines the operational testing process is by approving a weak Test & Evaluation Master Plan (TEMP). This document serves as a guide for the specific tests throughout the life-cycle of the system. The system’s Program Management Office and DOT&E work together to create this plan. Because the TEMP must be approved by DOT&E, an industry insider can simply sign off on a less rigorous series of tests, thereby guaranteeing success: in other words, letting the students write their own exams.

There is also the matter of rigging tests to produce positive outcomes. Perhaps the most notorious example of that occurred with the Army’s DIVAD, or Divisional Air Defense system, in the early 1980s. Designed to be a self-propelled radar-guided anti-aircraft gun, the DIVAD struggled through the development process. Prototypes of the system were sent to Fort Bliss, Texas for testing in July 1984. The DIVAD failed to spot a hovering helicopter drone flying in an empty sky. It was only after several radar amplifiers were mounted to the drone that the DIVAD found it at all. As journalist Gregg Easterbrook said at the time, the test was like “testing a bloodhound’s ability to track a man covered with beefsteaks standing alone and upright in the middle of a parking lot.”

A corrupt, compromised, or ineffective DOT&E wouldn’t need to resort to watering down a system’s testing program, however. They could simply submit incomplete or inaccurate reports to the Secretary of Defense and Congress. There have been cases, such as the MX missile program, where previous testing officials have signed off on results that were falsified or misrepresented. An even simpler tactic would be to bury poor results deep inside a lengthy report.

Test results and reports have also been compromised by allowing system advocates, sometimes even corporate representative, to “score” the data. This occurs when a panel of “experts” (often corporate representatives) is convened to determine which results are allowable (valid) and which are not, or what result is “successful” or not. The Army did this with the M-1 tank by throwing out reliability results which showed the early models were breaking down every 34 miles. A scoring conference later found that some of the maintenance actions weren’t really that serious and the tank would only have to stop every 94 miles, which was conveniently just above the test standard of 90 miles. Such a panel could conveniently determine that testing results showing a system isn’t performing as advertised aren’t really that important and therefore should be left out of the report.

Conclusion

While DOT&E’s core mission is to ensure our military receives effective weapons, DOT&E also provides an excellent return on investment for taxpayers. Uncovering problems with weapon systems early allows them to be fixed before a system moves into full-scale production. The GAO estimates concurrency in the F-35 program will cost $1.7 billion to “rework and retrofit aircraft with design changes needed as a result of test discoveries.” The cost may increase as planes continue to come off the production line in advance of operational testing. More and more problems are sure to be uncovered as the operational testing process unfolds. Indeed, by the end of initial operational testing, well over 500 F-35s will have been produced, and it is unknown what will become of them all should DOT&E find the system ineffective or unsuitable. This issue is common to virtually all acquisition programs in the DoD. A strong independent testing office, like Michael Gilmore’s, working to identify these problems performs an invaluable service to our men and women in uniform and to the taxpayers. The DOT&E process, were it to be properly followed, would save billions of dollars.

President-elect Trump’s possible selections for lower-tier Cabinet positions are just as important as who he will choose to fill the top spots. Should President Trump select as DOTE&E an industry go-along or a military service system advocate who lacks a demonstrated history of fighting for objectivity, independence, and integrity, we cannot expect to see the reliable assessments that we have come to expect from Michael Gilmore. Ultimately, service members will suffer the consequences of ineffective weapons while taxpayers are stuck with the bill. That’s what you get when you allow the students to write, take, and grade their own exams.