U.S. flag

An official website of the United States government

Dot gov

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Https

Secure .gov websites use HTTPS
A lock () or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

Evaluation Lessons From Live Fire Testing

Breadcrumb

  1. Home
  2. Evaluation Lessons From Live Fire Testing
Evaluation Lessons  From Live Fire Testing

Several key Live Fire Testing and Evaluation (LFT&E) lessons were learned from the prime contractor perspective on the F-35/Joint Strike Fighter (JSF) program. The F-35 fighter jet effort includes three variants that increase overall program complexity and risk. The LFT&E component of this development effort is critical to the overall success of the F-35 program.

LFT&E is a critical element of the system engineering and test and evaluation processes for Department of Defense (DoD) systems. The current ACQuipedia article on LFT&E provides this straightforward explanation of LFT&E as part of the DoD acquisition process:

A test process that evaluates the vulnerability and/or lethality aspects of a conventional weapon or conventional weapon system. LFT&E is a statutory requirement (Title 10 U.S.C. [U.S. Code] § 2366) for covered systems, major munitions programs, missile programs, or product improvements to a covered system, major munitions programs, or missile programs before they can proceed Beyond Low Rate Initial Production (BLRIP). By law, a covered system is any vehicle, weapon platform, or conventional weapon system that includes features designed to provide some degree of protection to users in combat and that is an Acquisition Category (ACAT) I or ACAT II program. (Note: The term “covered system” can also be taken to mean any system or program that is covered by Title 10 U.S.C. § 2366, including major munitions and missile programs.)

LFT&E focuses on evaluating the survivability and lethality of a system. With regard to the F-35 program, these two attributes are paramount to the success of this system operating in its intended environment. Although they are similar, each F-35 variant has its own unique survivability and lethality requirements as well, making this an even bigger challenge to getting it right.

As the DoD moves closer to a full-rate production decision for F-35, the lessons learned from the LFT&E efforts of the F-35 industry team led by the prime contractor, Lockheed Martin Corporation, can provide other acquisition organizations with valuable insight into how best to conduct LFT&E on their respective programs.

LFT&E Lessons Learned From an Industry Perspective
The F-35 LFT&E program was one of the most comprehensive in fixed-wing procurements. The F-35 program consisted of 61 test series, with more than 1,500 events against ballistic threats. The F-35 LFT&E lessons learned from the prime contractor perspective may be grouped into two general categories: Government-Industry Teamwork and Limiting Scope.

Government-Industry Teamwork
Lines of Communication. LFT&E requires the efforts of at least four primary entities: The Program Office, Director; Operational Test and Evaluation (DOT&E) Representatives; the primary weapon system contractors; and the government test facility organizations. Contractual relationships provide a formal flow between the four primary entities involved in Live Fire Testing (LFT) (Figure 1), but the formal lines of communication lack the ability to build a team capable of effectively and efficiently executing the LFT Program.
In a previous LFT&E program, the prevailing wisdom was to keep control of the program by only allowing the formal lines of communication between entities; in particular, the intention was to limit communication to DOT&E representatives. Much of the success of the F-35 LFT&E program can be attributed to the open communication and informal information flows that were created and maintained throughout (Figure 2).

Diverse Organizations. Each of these organizations have differing goals and constraints that sometimes make teamwork difficult. Industry partners want to limit company costs and risk while meeting contractual obligations. The Program Office wants to limit program impacts while delivering value to the warfighter. DOT&E’s objective is to thoroughly test. Their success is sometimes dependent on “findings”: discovering shortfalls or unexpected results. Ultimately, these disparate organizations must come together to produce a test program that meets the objectives while living within the constraints. It is important that each organizational member is at least made aware of the varying goals of the other members.

Roles and Responsibilities. It is imperative that roles and responsibilities are established early in the program. On F-35, we determined that the prime contractor should have the responsibility to create all test plans and reports. Test article construction is an activity that should be shared by the contractor and test facilities. The government test facilities have tremendous abilities to quickly design and construct test articles, particularly if these articles are sub-assemblies, and not required to be production representative. The contractors must design and construct the more complicated articles, but can be less efficient in building the simple ones. The test facilities must be the final technical approval for the test plans as they are the ones that must ultimately execute the test. DOT&E representatives must provide timely reviews and constructive comments on each test plan, along with formal signoff on the plans utilizing full-scale and Full-Up, System-Level articles.

Establishing Trust. Trust between team members will be established only through time, with open communication, honest discussions, and mutual respect. On F-35, we established weekly telephone calls with all organizations to foster trust. We also established a collaborative workspace on the F-35 Data Library in order to share technical information in the form of test plans, analyses and test reports. These two forums kept the informal lines of communication open throughout the program.

Limiting the Scope
Scope Creep. One of the most difficult problems encountered with F-35 LFT&E was the need to prevent scope creep. The cost of each test series is highly dependent on the objectives, test matrix, and complexity of the test article. Open communication helps in that each team’s organization is able to express their objectives, concerns and constraints. The objectives of the test, if defined in detail, will allow the team to limit the complexity of the test article. Spares to replace damaged components must also be taken into account. The order of the events on each test article requires much coordination but will yield the most data without requiring needless replacements and repairs. There was a transition from simple Test Data Sheets on previous programs to full-up Test Plans which were used on the F-35; these comprehensive documents went a long way to limiting objectives and setting expectations for each test series.

Objectives. LFT&E issues and sub-issues are provided to the contractors via the Test and Evaluation Master Plan (TEMP), but the issues provided are very general in nature, leaving much room for interpretation. Therefore, each F-35 test series was designed to specifically address a particular set of objectives within the list of LFT&E sub-issues.

Threats. Similarly, the potential threat list is also daunting. It was important early on to define threat types and to limit the scope by addressing only those potential threats (with some exceptions). An underlying purpose of the F-35 LFT&E plan was that the tests were going to meet the objectives in the TEMP as well as provide insights to the F-35 design team. The tests would also provide missing/inadequate data to improve the F-35 vulnerability analysis.

Controlled Damage Tests. The F-35 LFT&E team utilized more than just ballistic tests to address LFT&E issues. Wind Tunnel tests were conducted to determine the controllability of the F-35 after loss of a complete or partial control surface. Man-in-the-loop simulation was used to verify the loss of multiple flight control and electrical power components that were in close proximity. The F-35’s Fuel System Simulator was used to provide data on fuel loss and fuel tank explosion prevention. These tests were much less expensive than some of the more complex ballistic test articles, and provided a wealth of information to address many LFT&E issues.

Modeling and Simulation (M&S). M&S, once shunned by the Live Fire Test Office within OT&E, is now being used in a significant way. In the F-35 program, a symbiotic relationship was established early in the program between M&S and LFT&E. Early in the program, the contractor team conducted a unique vulnerability uncertainty analysis that gave insights into which damage mechanisms had the largest potential to affect the F-35’s vulnerability assessment results. These uncertainties were folded into the LFT&E test plans, and provided a context for discussions concerning the relative importance of individual test events. For example, testing to determine the vulnerability of the F-35’s flight control computers was determined to be relatively unimportant due to system redundancy, which rendered the potential for loss-of-aircraft to be relatively small. All test events were preceded by test predictions, most of which were conducted via M&S. Tests verified the ability of the M&S in some cases, while providing critical data to allow improvements to be made.

Conclusion
DoD acquisition program success hinges on the partnership between both government and industry in the execution of a robust systems engineering process to deliver effective solutions to the warfighter. LFT&E is a key component of this systems engineering effort. This article offers valuable lessons learned from the industry partner perspective on how to effectively execute LFT&E on a very complex acquisition program—the F-35. In the end, the success of our efforts, both government and industry, will be based on our strong partnership, effective communication and teamwork to meet the needs of the warfighter.

 


Steve Mills is a professor of Program Management and Cybersecurity at Defense Acquisition University.

Mark Stewart is a Lockheed Martin Fellow specializing in Aircraft Survivability/Vulnerability and Technical Lead of the F-35 Vulnerability Analysis.
and Live Fire Test Team.

The authors can be contacted at [email protected] and [email protected].

Click here to download a PDF copy of this article.​


subscribe