Section 913 of the 2018 National Defense Authorization Act (NDAA), calls for increased use of data analytics to improve acquisition outcomes and promote collaboration across military Services and agencies internal and external to DoD.
The commitment to make data a central focus for decision makers is reflected by Stacy Cummings, Principal Deputy Assistant Secretary of Defense, “Acquisition Enablers”:
“Our goal for the near and medium term is to look at data across the Enterprise, to bring it together in a way that will allow us to use advanced analytic capabilities and tools to look at the data where it resides.”
This article discusses how a data-based approach will enable informed decisions on critical matters, leading to the more effective use of the Department of Defense (DoD) budget, and increasing the likelihood of successful acquisition outcomes. Decisions made solely from experience often are prone to human error and bias (halo effect, groupthink, sunk cost fallacy, confirmation bias, and small numbers fallacy, to name a few). Evidence-based decision-making and the use of objective data in particular, are vital to improved agility, risk reduction, and efficiency.
The New Framework
The ongoing “agile” release of the Adaptive Acquisition Framework (AAF) is part of the further empowerment of Program Office personnel.
Intrinsically, DoD acquisition continues its evolution in the quest to deliver the most capable and effective warfighting capabilities to our nation’s military forces. Compelled by the resurgence of threats to this nation’s security and that of our allies, the 2018 U.S. National Defense Strategy (NDS) discusses the urgency and magnitude of these threats and hence the need for the acquisition workforce to deliver required capabilities at a faster pace and in the most cost-effective manner possible. To improve our competitive edge, the DoD will pursue three lines of effort:
- Build a more lethal, resilient, agile, and ready Joint Force.
- Strengthen alliances and attract new partners.
- Reform for greater performance and affordability.
A specific instruction to the DoD was to “shed outdated management practices and structures while integrating insights from business innovation.”
Doing more with less is a decades-old refrain for defense acquisition. However, as articulated in the NDS, although affordability is important, speed of delivery is now perhaps of greater importance due to our adversaries’ rapid technological advances.
To that end, the Office of the Under Secretary of Defense for Acquisition and Sustainment (OUSD[A&S]) is developing and refining innovative approaches to delivering capabilities to our warfighters. Their efforts correlate with 14 enabling tenets, all of which are rooted in their organizational Mission Statement, as shown in Table 1.
These tenets serve as a foundation for the recently unveiled Adaptive Acquisition Framework (AAF). The AAF is comprised of six “Pathways” of embarkation. These pathways allow for tailoring, combining, and transitioning between themselves to develop the nominal program framework, with data-driven decisions serving as the interconnecting crossroads between the pathways.
Business Intelligence “Democratizes” Information, Adding Visibility and Accessibility
“Online Analytical Processing (OLAP.com)” describes business intelligence (BI) as: “[The] technologies, applications and practices for the collection, integration, analysis, and presentation of business information […] to support better business decision making. Put simply, BI systems are data-driven Decision Support Systems (DSS).”
Proper execution and implementation of analytical tools such as BI will significantly improve programmatic decision-making, as promulgated by Section 913 of the NDAA. Furthermore, the rise of self-service BI, such as visualizations in dashboards, provides the opportunity to share data and insight to all employees throughout the entire organization. BI no longer requires knowledge of advanced programming languages or computer science. Such visibility and accessibility of information provides exciting opportunities to break down data silos in a way that very much supports the AAF initiative. As previously described in the quote by Stacy Cummings, it also promotes collaboration through sharing, reduces program risk by making insight available to inform decisions, and empowers everyone to be a data-driven critical thinker.
Senior DoD officials acknowledge that there is a current lack of data-informed decision making. The acquisition workforce must perform more advanced data analyses, centralize data, and promote its use in decision making in order to ultimately accelerate and optimize programs’ delivery of enhanced capability to warfighters. The need for additional tools is recognized and being worked. One such tool is “ADVANA” (Advanced Analytics), developed by DoD to be “a single authoritative source for audit and business data analytics”; it is currently taking in data from more than 120 DoD systems. It appears promising, but wider application is pending.
DoD Data and Analysis Today
Most existing DoD business systems were established by statutes and policies for reporting instead of analysis. Table 2 lists several of these information systems.
According to Chris Nott (Chief Technology Officer, Government, IBM Europe), although this satisfies regulatory requirements and archives what has happened, it fails to indicate the correlations and causal relationships that would establish why it has happened and predict what will happen. These relationships and predictions require advanced analysis. A culture shift and associated action is long overdue. Data is not just for reporting compliance to statutes and regulations; it is an asset for decision making and strategizing, enabling and driving decisions at the ever-accelerating “speed of relevance.”
Acquisition professionals always have done data analyses—including risk analysis, cost estimation, Test and Evaluation, etc. However, the tools and relevant insights are not sophisticated enough for today’s operational environment. On more than one assessment, data analytics within the DoD as a whole lacks sophistication and maturity:
“We benchmarked ourselves, and we had external parties benchmark us, and we uniformly came in at Level One mature, the lowest possible level of maturity.”
—Michael Conlin, DoD Chief Data Officer
Using a survey developed by Gartner Research, RAND rated the DoD at Level 2: “Opportunistic” on a continuum that ranges from Level 1: “Basic” to Level 5: “Transformational.” The DoD and many other corporations have significant room to advance in this area. Of the 196 organizations Gartner surveyed, 60 percent rated within the lowest three levels of maturity.
Complexity of “Data Analysis Techniques” within the DoD acquisition community range from simple data archives to predictive analysis (Figure 1). Many DoD systems lack integration of data sources and analytic components.
Now, Defense Acquisition Visibility Environment (DAVE), a database of databases, is the database of choice cited by the AAF. It is structured to allow easy integration by user interface developers who are building capabilities in DAVE, or those who want to align with DAVE’s data-first design approach. “DAVE” also is a data governance framework, creating a common way of understanding data.
DoD’s acquisition data goals are iterative as represented in Figure 2.
DoD should look at its exemplars to develop other systems to equivalent levels.
Near-Term Exemplars
The Air Force Project Management Resource Tools (PMRT) and the Army Leader Dashboard (ALD) are effective examples of integrated databases with dashboard analytics. PMRT is a robust platform that automatically pulls in data from DoD and Air Force acquisition, finance, contracts, foreign military sales, and logistics systems. In 2018, this tool was enhanced with the integration of the Qlik Sense product, a self-service BI tool. Through its dashboard, users can make sense of the data through visualizations and probe for trends or abnormalities that may influence their program. Dashboards give transparency and accessibility to the data, making it ripe for management interpretation and decision making.
Similarly, the Army is developing its ALD, which will break down silos of data into a centralized source that will give widespread insight from personnel to acquisitions to logistics. PMRT is used by the Army as a prototype for the business management portion of the acquisition data domain. The benefit of dashboards is that they can often update in real-time. Acquisition policies generally only require acquisition data to be disseminated at the time of milestone decisions. However, an automated system will allow data to be available at any time during the program’s life cycle, thereby giving decision makers enhanced insight and allow users to perform their jobs more effectively.
Moving Toward Artificial Intelligence
The Fiscal Year 2019 NDAA, Section 238g defines Artificial Intelligence (AI) as “Any artificial system that performs tasks under varying and unpredictable circumstances without significant human oversight, or that can learn from experience and improve performance when exposed to data sets.”
A recent count reveals DoD has more than 3,000 stand-alone business systems. This type of fragmented approach creates information overload. A Knowledge Graph is a tool that stimulates new ideas and identifies previously undetected cause and effect connections. Knowledge Graphs are all around us in the form of personalized interactive experiences (e.g., Amazon, Apple, Google/Youtube), each providing intuitive prompts and recommendations. Imagine applying this same AI deep learning technology to the AAF. … Are you researching a tough acquisition issue? As you search the AAF, results of searches by other program personnel—including successes, failures, and false starts—will be manifested as well as derived recommendations for your consideration in making data-driven decisions that typically result in goal-directed behaviors.
The kind of technological leap we are talking about was greatly accelerated by a match of “Go” (an ancient abstract strategy board game for two players, in which the aim is to surround more territory than the opponent with small black or white disks). In his book, AI Super-Powers, Kai Fu Lee describes when Google’s DeepMind “AlphaGo Master” computer program beat Ke Jie, the reigning “Go” world champion at the time, it became a “Sputnik” moment for China. As a result, in 2017 China’s central government established a goal to equal the United States in AI theory, technology, and application by 2025 and become the center of global innovation in AI by 2030. So far, they are on target! An example of AI has already been applied in China’s banking sector to aid decisions of whether to approve loans. Even for a loan that takes less than 5 minutes to obtain, the bank’s system can take into account more than 5,000 personal characteristics derived from just a few entries. One consideration is how tentatively you typed your application. Another is how low your phone battery is. Both of these are high-risk indicators.
By design, AAF is in a “perpetual” state of process improvement. The ideal solution captures end-user data via the AAF and other tools or resources similar to the commercial relational databases alluded to earlier. If we know which “sites” team members are browsing, clicking on, or contributing to, that information can become the basis for knowledge, understanding, and wisdom. And these can take the form of undirected and unsolicited recommendations, cautions, and advice to programs facing similar challenges.
Another RAND study, conducted in response to Section 238(e) of the NDAA of 2019 (“The Department of Defense Posture for Artificial Intelligence”), cited mission support areas, such as acquisition, as prime candidates for consideration as potential “low-hanging fruit” for the implementation of AI; “…focusing on classification and prediction.” It also recognized that AI would have its own challenges. In particular, the report cited the need for close coordination, and cooperation between data generators, data collectors, and data managers. It also recognized that this will introduce inherent vulnerabilities and attack surfaces that will “demand careful consideration.” The study concluded that DoD pursuit of AI at scale by 2025 is realistic but would “require fundamentally transforming DoD’s culture into a data-enabled one that values and uses data to the full potential—a monumental endeavor but one that could enhance efficiencies across the board.”
Fostering a Data-Centric Organization
All of the necessary systems could be in place but they make no difference if they are not being used to their potential. The aforementioned RAND study documents not only analytical capability shortcomings, but decisional issues. Specifically, decisions can be made that are contrary to the available data when there is “a general reluctance to accept unfavorable information.” To place data at the center of business decisions requires a supportive culture and a determination to derive deeper insights.
An example of how best to overcome decisional challenges comes from Alan Mulally, the former CEO of Ford Motors and Mark Fields, Ford’s former “president of the Americas”: When Mulally arrived in 2006, he was confronted with the constant reporting of “Green” on the stoplight charts during project reviews. Prior to the launch of the sport-utility-vehicle “Edge” in Canada, Fields reported during a review in the “Taurus Room” that “we’re red on Edge.” In a low-trust environment, the CEO may have scolded the project lead. Instead, Mulally applauded and said “thank you!” He further elaborated that when people speak up they are saving the company time, energy, and millions of dollars in scrap and rework, not to mention the potential liability associated with a faulty product along with possible damage to the Ford brand. Through this positive reaction to a negative situation, Mulally fostered a culture of information sharing. It was just as important that Fields as the project lead did not dismiss unfavorable data that suggested that there could be an issue with the hydraulic actuator and instead made the unpopular recommendation to delay the launch of Edge. Mulally was happy that Fields provided a realistic assessment from which he could lead more effectively.
Trust and communication, as described in the Ford Edge example, are essential. However, without knowing the context and understanding “why” it happened, the decision maker cannot demonstrate the judgment and “wisdom” to lead projects in the most effective direction. While it is easy to claim to be data driven from merely collecting and presenting data, to truly derive deep levels of meaning depends on analysis. Through analysis, data is processed and organized in five distinctive constructs of increasing value for the decision maker:
- Data: “Stuff!” A collection of letters, numbers, charts, pictures …
- Information: Contextualized data; the point at which data becomes meaningful and useful.
- Knowledge: Derived from organized information into categories of similarity and relationships (e.g. cause and effect).
- Understanding: Where intelligence meets knowledge; being able to discern why the similarities, relationships or patterns as we know them (obtained through knowledge) are manifested.
- Wisdom: Using our understanding to make intelligent decisions.
These components tend to correlate with the complexity continuum in Figure 1. That is, data archives do nothing more than hold data; databases can provide users basic information; more advanced analytic techniques can provide the “how” things work (knowledge) and “why” a mistake occurred (understanding). While AI is promising, wisdom currently is not as programmable since it involves anticipation of long-term outcomes. Information, knowledge, and understanding can help decision makers do things right; Russell Ackoff explains, “doing the right thing is wisdom…”
By all measures discussed, DoD is low on the scale for data maturity and analytics. The hope and promise of AAF includes a data-centric approach! As acquisition professionals, we need to capture, report, and utilize accurate data, even when that data is less than favorable. These practices will result in data-driven decisions, a more engaged workforce, and improved acquisition outcomes.
Moschler is a professor of Systems Acquisition Management at the Defense Acquisition University in California, Maryland, where McGhee is a professor of Acquisition Management and D’Amore is Lead Management Analyst.
The authors can be contacted at
[email protected];
[email protected]; and
Alex.D’[email protected].
